2024-05-10T10:44:15.302 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-10T10:44:15.302 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_dparmar@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/dbench}} --archive /home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922 --name dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi --verbose -- /home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922/orig.config.yaml 2024-05-10T10:44:15.361 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-10T10:44:15.462 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922 branch: wip-63896-from-wip-rishabh-testing-20240501.193033 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7700922' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-63896-from-wip-rishabh-testing-20240501.193033 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: d39fa08867836ac070c47cb66c782fa559fd480b subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d39fa08867836ac070c47cb66c782fa559fd480b kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-63896-from-wip-rishabh-testing-20240501.193033 sha1: d39fa08867836ac070c47cb66c782fa559fd480b owner: scheduled_dparmar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1698 sha1: d39fa08867836ac070c47cb66c782fa559fd480b sleep_before_teardown: 0 subset: 434/512 suite: fs suite_branch: wip-63896-from-wip-rishabh-testing-20240501.193033 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d39fa08867836ac070c47cb66c782fa559fd480b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d39fa08867836ac070c47cb66c782fa559fd480b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO2ZeqFr+Mi+lBGNyRN7rbKcvPbOnBb/7SSjSi6s2U5//nBpwQsVcTvYwpV0Os35WOflGwLkBwQXpI8zBfUYG+U= smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGL9P8oSr9KcANsT+M0gFeZcBb//34wZrtiDDy36Lg63cjxFIIGT1Es+W0vTyOazAQG8uy1pCIhBQwEYQ54mkYQ= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPrg6vrjBqnla3DtydUdfBE5rN1oXn7F5V3VLnN5wlwfqevepwq1iNMwwjWJ1ukVHXWhR4gOslxjsA0HNdIDaiY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-10_07:35:43 tube: smithi user: dparmar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-10T10:44:15.462 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d39fa08867836ac070c47cb66c782fa559fd480b/qa; will attempt to use it 2024-05-10T10:44:15.463 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d39fa08867836ac070c47cb66c782fa559fd480b/qa/tasks 2024-05-10T10:44:15.463 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-10T10:44:15.465 INFO:teuthology.task.internal:Checking packages... 2024-05-10T10:44:15.489 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd39fa08867836ac070c47cb66c782fa559fd480b' 2024-05-10T10:44:15.489 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-10T10:44:15.489 INFO:teuthology.packaging:ref: None 2024-05-10T10:44:15.489 INFO:teuthology.packaging:tag: None 2024-05-10T10:44:15.489 INFO:teuthology.packaging:branch: wip-63896-from-wip-rishabh-testing-20240501.193033 2024-05-10T10:44:15.489 INFO:teuthology.packaging:sha1: d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T10:44:15.489 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-63896-from-wip-rishabh-testing-20240501.193033 2024-05-10T10:44:15.641 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3449.gd39fa088 2024-05-10T10:44:15.642 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-10T10:44:15.648 INFO:teuthology.task.internal:no buildpackages task found 2024-05-10T10:44:15.649 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-10T10:44:15.660 INFO:teuthology.task.internal:Saving configuration 2024-05-10T10:44:15.682 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-10T10:44:15.687 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-10T10:44:15.708 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-10 10:37:51.320570', 'locked_by': 'scheduled_dparmar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO2ZeqFr+Mi+lBGNyRN7rbKcvPbOnBb/7SSjSi6s2U5//nBpwQsVcTvYwpV0Os35WOflGwLkBwQXpI8zBfUYG+U='} 2024-05-10T10:44:15.730 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-10 10:37:51.317038', 'locked_by': 'scheduled_dparmar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGL9P8oSr9KcANsT+M0gFeZcBb//34wZrtiDDy36Lg63cjxFIIGT1Es+W0vTyOazAQG8uy1pCIhBQwEYQ54mkYQ='} 2024-05-10T10:44:15.751 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-10 10:37:51.318981', 'locked_by': 'scheduled_dparmar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPrg6vrjBqnla3DtydUdfBE5rN1oXn7F5V3VLnN5wlwfqevepwq1iNMwwjWJ1ukVHXWhR4gOslxjsA0HNdIDaiY='} 2024-05-10T10:44:15.751 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-10T10:44:15.757 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-10T10:44:15.757 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-10T10:44:15.757 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-10T10:44:15.757 INFO:teuthology.run_tasks:Running task console_log... 2024-05-10T10:44:15.849 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8bf16a50d0>, signals=[15]) 2024-05-10T10:44:15.850 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-10T10:44:15.856 INFO:teuthology.task.internal:Opening connections... 2024-05-10T10:44:15.856 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-05-10T10:44:15.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-10T10:44:15.942 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2024-05-10T10:44:15.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-10T10:44:16.019 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-05-10T10:44:16.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-10T10:44:16.096 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-10T10:44:16.103 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-05-10T10:44:16.122 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-05-10T10:44:16.122 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-05-10T10:44:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-05-10T10:44:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-05-10T10:44:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-05-10T10:44:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-05-10T10:44:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-05-10T10:44:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-05-10T10:44:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-10T10:44:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-05-10T10:44:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-05-10T10:44:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-10T10:44:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-05-10T10:44:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-10T10:44:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-10T10:44:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-10T10:44:16.181 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-05-10T10:44:16.203 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2024-05-10T10:44:16.221 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2024-05-10T10:44:16.221 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2024-05-10T10:44:16.277 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream" 2024-05-10T10:44:16.278 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="9" 2024-05-10T10:44:16.278 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2024-05-10T10:44:16.278 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2024-05-10T10:44:16.278 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="9" 2024-05-10T10:44:16.278 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el9" 2024-05-10T10:44:16.278 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-10T10:44:16.278 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2024-05-10T10:44:16.278 INFO:teuthology.orchestra.run.smithi076.stdout:LOGO="fedora-logo-icon" 2024-05-10T10:44:16.278 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-10T10:44:16.278 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/" 2024-05-10T10:44:16.278 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-10T10:44:16.278 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-10T10:44:16.279 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-10T10:44:16.279 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2024-05-10T10:44:16.303 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-05-10T10:44:16.321 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-05-10T10:44:16.321 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-05-10T10:44:16.378 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-05-10T10:44:16.378 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-05-10T10:44:16.378 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-05-10T10:44:16.378 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-05-10T10:44:16.378 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-05-10T10:44:16.379 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-05-10T10:44:16.379 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-10T10:44:16.379 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-05-10T10:44:16.379 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-05-10T10:44:16.379 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-10T10:44:16.379 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-05-10T10:44:16.379 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-10T10:44:16.379 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-10T10:44:16.379 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-10T10:44:16.379 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-05-10T10:44:16.402 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-10T10:44:16.414 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-10T10:44:16.453 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-10T10:44:16.453 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-05-10T10:44:16.456 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2024-05-10T10:44:16.458 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-05-10T10:44:16.474 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-10T10:44:16.480 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-10T10:44:16.480 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-05-10T10:44:16.514 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2024-05-10T10:44:16.516 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-05-10T10:44:16.574 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-10T10:44:16.651 INFO:teuthology.run_tasks:Running task kernel... 2024-05-10T10:44:16.665 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-05-10T10:44:16.665 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-05-10T10:44:16.665 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-05-10T10:44:16.666 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-10T10:44:16.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T10:44:16.683 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-05-10T10:44:16.927 DEBUG:teuthology.task.kernel:sha1 for client.0 is fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e 2024-05-10T10:44:16.927 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e"... 2024-05-10T10:44:16.928 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-05-10T10:44:16.945 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-437.el9.x86_64 2024-05-10T10:44:16.945 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e 2024-05-10T10:44:16.946 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-10T10:44:16.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum upgrade -y linux-firmware 2024-05-10T10:44:20.580 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.1 MB 00:01 2024-05-10T10:44:22.556 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-05-10T10:44:26.682 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.2 MB 00:00 2024-05-10T10:44:28.524 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-05-10T10:44:30.237 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-05-10T10:44:35.593 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-05-10T10:44:37.749 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-10T10:44:37.750 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-05-10T10:44:37.750 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-10T10:44:37.870 INFO:teuthology.task.kernel:Downloading kernel fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e on client.0... 2024-05-10T10:44:37.870 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e 2024-05-10T10:44:38.101 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e/centos/9/flavors/default/x86_64/ 2024-05-10T10:44:38.102 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_gfb9a6a44107e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-10T10:44:38.104 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi027.front.sepia.ceph.com to complete... 2024-05-10T10:44:50.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10 10:44:50 URL:https://2.chacra.ceph.com/r/kernel/testing/fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_gfb9a6a44107e-1.x86_64.rpm [357387737/357387737] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-10T10:44:50.207 INFO:teuthology.orchestra.run.smithi027.stderr:FINISHED --2024-05-10 10:44:50-- 2024-05-10T10:44:50.207 INFO:teuthology.orchestra.run.smithi027.stderr:Total wall clock time: 12s 2024-05-10T10:44:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:Downloaded: 1 files, 341M in 12s (28.9 MB/s) 2024-05-10T10:44:50.211 INFO:teuthology.task.kernel:Installing kernel fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e on client.0... 2024-05-10T10:44:50.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-10T10:44:52.438 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying... ######################################## 2024-05-10T10:44:52.577 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing... ######################################## 2024-05-10T10:44:52.713 INFO:teuthology.orchestra.run.smithi027.stdout:Updating / installing... 2024-05-10T10:45:00.998 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-6.9.0_rc3_gfb9a6a44107e-1 ######################################## 2024-05-10T10:45:41.050 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-10T10:45:41.051 DEBUG:teuthology.orchestra.run.smithi027:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-10T10:45:41.138 INFO:teuthology.orchestra.run.smithi027.stdout:/boot/System.map-6.9.0-rc3-gfb9a6a44107e 2024-05-10T10:45:41.139 INFO:teuthology.orchestra.run.smithi027.stdout:/boot/config-6.9.0-rc3-gfb9a6a44107e 2024-05-10T10:45:41.139 INFO:teuthology.orchestra.run.smithi027.stdout:/boot/vmlinuz-6.9.0-rc3-gfb9a6a44107e 2024-05-10T10:45:41.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/System.map 2024-05-10T10:45:41.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/config 2024-05-10T10:45:41.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel 2024-05-10T10:45:41.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch 2024-05-10T10:45:41.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86 2024-05-10T10:45:41.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto 2024-05-10T10:45:41.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-10T10:45:41.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-10T10:45:41.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kernel 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kernel/cpu 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kernel/cpu/mce 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kvm 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kvm/kvm.ko 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto 2024-05-10T10:45:41.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/ansi_cprng.ko 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/anubis.ko 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/arc4.ko 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx/async_memcpy.ko 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx/async_pq.ko 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx/async_tx.ko 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx/async_xor.ko 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx/raid6test.ko 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/authenc.ko 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/authencesn.ko 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/blake2b_generic.ko 2024-05-10T10:45:41.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/blowfish_common.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/blowfish_generic.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/camellia_generic.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/cast5_generic.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/cast6_generic.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/cast_common.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/ccm.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/cmac.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/crc32_generic.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/crypto_engine.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/crypto_user.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/cts.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/des_generic.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/dh_generic.ko 2024-05-10T10:45:41.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/echainiv.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/essiv.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/fcrypt.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/khazad.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/lz4.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/lz4hc.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/md4.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/michael_mic.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/pcbc.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/pcrypt.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/rmd160.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/seed.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/serpent_generic.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/tcrypt.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/tea.ko 2024-05-10T10:45:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/twofish_common.ko 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/twofish_generic.ko 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/vmac.ko 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/wp512.ko 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/xcbc.ko 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/xor.ko 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/xxhash_generic.ko 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/acpi_pad.ko 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/ec_sys.ko 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/platform_profile.ko 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/sbs.ko 2024-05-10T10:45:41.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/sbshc.ko 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/video.ko 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/acard-ahci.ko 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/ahci_platform.ko 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/ata_generic.ko 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/libahci_platform.ko 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_acpi.ko 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_ali.ko 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_amd.ko 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_artop.ko 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_atiixp.ko 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_atp867x.ko 2024-05-10T10:45:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_cmd640.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_cmd64x.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_cypress.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_efar.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_hpt366.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_hpt37x.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_it8213.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_it821x.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_jmicron.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_marvell.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_mpiix.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_netcell.ko 2024-05-10T10:45:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_ninja32.ko 2024-05-10T10:45:41.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_ns87410.ko 2024-05-10T10:45:41.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_ns87415.ko 2024-05-10T10:45:41.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_oldpiix.ko 2024-05-10T10:45:41.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_opti.ko 2024-05-10T10:45:41.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_optidma.ko 2024-05-10T10:45:41.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_pcmcia.ko 2024-05-10T10:45:41.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-10T10:45:41.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-10T10:45:41.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_piccolo.ko 2024-05-10T10:45:41.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_rdc.ko 2024-05-10T10:45:41.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_sch.ko 2024-05-10T10:45:41.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_serverworks.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_sil680.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_sis.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_sl82c105.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_triflex.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_via.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pdc_adma.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_inic162x.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_mv.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_nv.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_promise.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_qstor.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_sil.ko 2024-05-10T10:45:41.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_sil24.ko 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_sis.ko 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_svw.ko 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_sx4.ko 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_uli.ko 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_via.ko 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_vsc.ko 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/auxdisplay 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/auxdisplay/ks0108.ko 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/base 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/base/regmap 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/bcma 2024-05-10T10:45:41.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/bcma/bcma.ko 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/aoe 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/aoe/aoe.ko 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/brd.ko 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/drbd 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/drbd/drbd.ko 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/floppy.ko 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/loop.ko 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/mtip32xx 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/nbd.ko 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/null_blk 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/null_blk/null_blk.ko 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/pktcdvd.ko 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/rbd.ko 2024-05-10T10:45:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/virtio_blk.ko 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/xen-blkback 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/xen-blkfront.ko 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/zram 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/zram/zram.ko 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hangcheck-timer.ko 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hw_random 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hw_random/via-rng.ko 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi 2024-05-10T10:45:41.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/lp.ko 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/mwave 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/mwave/mwave.ko 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ppdev.ko 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tlclk.ko 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm/tpm.ko 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-10T10:45:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-10T10:45:41.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-10T10:45:41.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-10T10:45:41.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-10T10:45:41.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/uv_mmtimer.ko 2024-05-10T10:45:41.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/virtio_console.ko 2024-05-10T10:45:41.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq 2024-05-10T10:45:41.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-10T10:45:41.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-10T10:45:41.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-10T10:45:41.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-10T10:45:41.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-10T10:45:41.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-10T10:45:41.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto 2024-05-10T10:45:41.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/ccp 2024-05-10T10:45:41.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-10T10:45:41.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/ccp/ccp.ko 2024-05-10T10:45:41.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/intel 2024-05-10T10:45:41.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/intel/qat 2024-05-10T10:45:41.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/intel/qat/qat_common 2024-05-10T10:45:41.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-10T10:45:41.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-10T10:45:41.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/padlock-aes.ko 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/padlock-sha.ko 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/virtio 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dca 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dca/dca.ko 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/devfreq 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dma 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dma/dw 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dma/ioat 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-10T10:45:41.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac 2024-05-10T10:45:41.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/amd64_edac.ko 2024-05-10T10:45:41.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/e752x_edac.ko 2024-05-10T10:45:41.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/edac_mce_amd.ko 2024-05-10T10:45:41.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i3000_edac.ko 2024-05-10T10:45:41.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i3200_edac.ko 2024-05-10T10:45:41.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i5100_edac.ko 2024-05-10T10:45:41.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i5400_edac.ko 2024-05-10T10:45:41.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i7300_edac.ko 2024-05-10T10:45:41.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i7core_edac.ko 2024-05-10T10:45:41.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i82975x_edac.ko 2024-05-10T10:45:41.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/ie31200_edac.ko 2024-05-10T10:45:41.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/sb_edac.ko 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/x38_edac.ko 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/firmware 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/firmware/edd.ko 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/gpio 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/gpio/gpio-ich.ko 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-appleir.ko 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-aureal.ko 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-axff.ko 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-dr.ko 2024-05-10T10:45:41.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-elecom.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-elo.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-emsff.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-gaff.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-gt683r.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-gyration.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-holtekff.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-hyperv.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-icade.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-keytouch.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-kye.ko 2024-05-10T10:45:41.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-lcpower.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-led.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-lenovo.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-multitouch.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-ortek.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-penmount.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-petalynx.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-picolcd.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-pl.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-plantronics.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-primax.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-rmi.ko 2024-05-10T10:45:41.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-10T10:45:41.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-common.ko 2024-05-10T10:45:41.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-10T10:45:41.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-10T10:45:41.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-10T10:45:41.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-10T10:45:41.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-10T10:45:41.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-10T10:45:41.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-10T10:45:41.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-10T10:45:41.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-10T10:45:41.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat.ko 2024-05-10T10:45:41.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-saitek.ko 2024-05-10T10:45:41.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-samsung.ko 2024-05-10T10:45:41.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-10T10:45:41.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-sjoy.ko 2024-05-10T10:45:41.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-sony.ko 2024-05-10T10:45:41.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-speedlink.ko 2024-05-10T10:45:41.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-steelseries.ko 2024-05-10T10:45:41.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-sunplus.ko 2024-05-10T10:45:41.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-10T10:45:41.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-tivo.ko 2024-05-10T10:45:41.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-tmff.ko 2024-05-10T10:45:41.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-topseed.ko 2024-05-10T10:45:41.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-twinhan.ko 2024-05-10T10:45:41.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-uclogic.ko 2024-05-10T10:45:41.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-waltop.ko 2024-05-10T10:45:41.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-wiimote.ko 2024-05-10T10:45:41.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-xinmo.ko 2024-05-10T10:45:41.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-zpff.ko 2024-05-10T10:45:41.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-zydacron.ko 2024-05-10T10:45:41.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/uhid.ko 2024-05-10T10:45:41.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/wacom.ko 2024-05-10T10:45:41.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hv 2024-05-10T10:45:41.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hv/hv_balloon.ko 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hv/hv_utils.ko 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hv/hv_vmbus.ko 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/abituguru.ko 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/abituguru3.ko 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ad7414.ko 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ad7418.ko 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adc128d818.ko 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adm1025.ko 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adm1026.ko 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adm1029.ko 2024-05-10T10:45:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adm1031.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adm9240.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ads7828.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adt7410.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adt7411.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adt7462.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adt7470.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adt7475.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adt7x10.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/amc6821.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/applesmc.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/asb100.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/asc7621.ko 2024-05-10T10:45:41.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-10T10:45:41.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/atxp1.ko 2024-05-10T10:45:41.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/coretemp.ko 2024-05-10T10:45:41.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-10T10:45:41.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/dme1737.ko 2024-05-10T10:45:41.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ds1621.ko 2024-05-10T10:45:41.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ds620.ko 2024-05-10T10:45:41.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/emc1403.ko 2024-05-10T10:45:41.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/emc6w201.ko 2024-05-10T10:45:41.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/f71805f.ko 2024-05-10T10:45:41.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/f71882fg.ko 2024-05-10T10:45:41.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/f75375s.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/fam15h_power.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/fschmd.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/g760a.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/g762.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/gl518sm.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/gl520sm.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/i5500_temp.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/i5k_amb.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ibmaem.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ibmpex.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ina209.ko 2024-05-10T10:45:41.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ina2xx.ko 2024-05-10T10:45:41.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/it87.ko 2024-05-10T10:45:41.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/k10temp.ko 2024-05-10T10:45:41.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/k8temp.ko 2024-05-10T10:45:41.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lineage-pem.ko 2024-05-10T10:45:41.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm63.ko 2024-05-10T10:45:41.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm73.ko 2024-05-10T10:45:41.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm75.ko 2024-05-10T10:45:41.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm77.ko 2024-05-10T10:45:41.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm78.ko 2024-05-10T10:45:41.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm80.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm83.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm85.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm87.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm90.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm92.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm93.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm95234.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm95241.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm95245.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc2945.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc4151.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc4215.ko 2024-05-10T10:45:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc4222.ko 2024-05-10T10:45:41.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc4245.ko 2024-05-10T10:45:41.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc4260.ko 2024-05-10T10:45:41.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc4261.ko 2024-05-10T10:45:41.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max16065.ko 2024-05-10T10:45:41.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max1619.ko 2024-05-10T10:45:41.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max1668.ko 2024-05-10T10:45:41.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max197.ko 2024-05-10T10:45:41.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max6639.ko 2024-05-10T10:45:41.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max6650.ko 2024-05-10T10:45:41.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max6697.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/mcp3021.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/nct6683.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/nct6775-core.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/nct6775.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/nct7802.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pc87360.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pc87427.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pcf8591.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-10T10:45:41.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/powr1220.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/sch5627.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/sch5636.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/sht15.ko 2024-05-10T10:45:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/sht21.ko 2024-05-10T10:45:41.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/shtc1.ko 2024-05-10T10:45:41.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/sis5595.ko 2024-05-10T10:45:41.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/smsc47b397.ko 2024-05-10T10:45:41.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/smsc47m1.ko 2024-05-10T10:45:41.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/smsc47m192.ko 2024-05-10T10:45:41.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/thmc50.ko 2024-05-10T10:45:41.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/tmp102.ko 2024-05-10T10:45:41.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/tmp103.ko 2024-05-10T10:45:41.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/tmp401.ko 2024-05-10T10:45:41.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/tmp421.ko 2024-05-10T10:45:41.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/via-cputemp.ko 2024-05-10T10:45:41.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/via686a.ko 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/vt1211.ko 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/vt8231.ko 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83627ehf.ko 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83627hf.ko 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83781d.ko 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83791d.ko 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83792d.ko 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83793.ko 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83795.ko 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83l785ts.ko 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83l786ng.ko 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c 2024-05-10T10:45:41.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/algos 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-10T10:45:41.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/i2c-dev.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/i2c-mux.ko 2024-05-10T10:45:41.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/i2c-smbus.ko 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/i2c-stub.ko 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/ib_core.ko 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw 2024-05-10T10:45:41.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/cxgb4 2024-05-10T10:45:41.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-10T10:45:41.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/mlx4 2024-05-10T10:45:41.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-10T10:45:41.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/mlx5 2024-05-10T10:45:41.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-10T10:45:41.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/mthca 2024-05-10T10:45:41.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-10T10:45:41.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp 2024-05-10T10:45:41.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/ipoib 2024-05-10T10:45:41.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/iser 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/isert 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/srp 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/srpt 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/gameport 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/gameport/gameport.ko 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/joydev.ko 2024-05-10T10:45:41.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/keyboard 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/appletouch.ko 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/bcm5974.ko 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/cyapatp.ko 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/sermouse.ko 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/rmi4 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/serio 2024-05-10T10:45:41.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/serio/altera_ps2.ko 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/serio/arc_ps2.ko 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/serio/serio_raw.ko 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/sparse-keymap.ko 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/leds-blinkm.ko 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/leds-lm3530.ko 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/leds-lp3944.ko 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/leds-ss4200.ko 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-10T10:45:41.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/bcache 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/bcache/bcache.ko 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-bio-prison.ko 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-cache-smq.ko 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-cache.ko 2024-05-10T10:45:41.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-crypt.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-delay.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-flakey.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-log-userspace.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-multipath.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-queue-length.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-raid.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-round-robin.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-service-time.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-switch.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-thin-pool.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-verity.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/persistent-data 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-10T10:45:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/raid0.ko 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/raid1.ko 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/raid10.ko 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/raid456.ko 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptbase.ko 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptctl.ko 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptfc.ko 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptlan.ko 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptsas.ko 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptscsih.ko 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptspi.ko 2024-05-10T10:45:41.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/lpc_ich.ko 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/lpc_sch.ko 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/mfd-core.ko 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/sm501.ko 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/viperboard.ko 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/vx855.ko 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/wl1273-core.ko 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/altera-stapl 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/apds9802als.ko 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/apds990x.ko 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/bh1770glc.ko 2024-05-10T10:45:41.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/cb710 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/cb710/cb710.ko 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/eeprom 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/eeprom/at24.ko 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/eeprom/max6875.ko 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/enclosure.ko 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/hpilo.ko 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/isl29003.ko 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/isl29020.ko 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/lis3lv02d 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/mei 2024-05-10T10:45:41.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/mei/mei-me.ko 2024-05-10T10:45:41.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/mei/mei-txe.ko 2024-05-10T10:45:41.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/mei/mei.ko 2024-05-10T10:45:41.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/sgi-gru 2024-05-10T10:45:41.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-10T10:45:41.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/sgi-xp 2024-05-10T10:45:41.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-10T10:45:41.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-10T10:45:41.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-10T10:45:41.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/tifm_7xx1.ko 2024-05-10T10:45:41.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/tifm_core.ko 2024-05-10T10:45:41.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/tsl2550.ko 2024-05-10T10:45:41.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/vmw_balloon.ko 2024-05-10T10:45:41.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/vmw_vmci 2024-05-10T10:45:41.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-10T10:45:41.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net 2024-05-10T10:45:41.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/bonding 2024-05-10T10:45:41.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/bonding/bonding.ko 2024-05-10T10:45:41.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa 2024-05-10T10:45:41.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa/b53 2024-05-10T10:45:41.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-10T10:45:41.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-10T10:45:41.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-10T10:45:41.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa/mv88e6xxx 2024-05-10T10:45:41.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-10T10:45:41.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dummy.ko 2024-05-10T10:45:41.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/eql.ko 2024-05-10T10:45:41.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet 2024-05-10T10:45:41.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/3com 2024-05-10T10:45:41.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-10T10:45:41.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/8390 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/adaptec 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/agere 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/alteon 2024-05-10T10:45:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/altera 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/amd 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/alx 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-10T10:45:41.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atlx 2024-05-10T10:45:41.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-10T10:45:41.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-10T10:45:41.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom 2024-05-10T10:45:41.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-10T10:45:41.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-10T10:45:41.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-10T10:45:41.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-10T10:45:41.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-10T10:45:41.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/genet 2024-05-10T10:45:41.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-10T10:45:41.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-10T10:45:41.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/brocade 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/brocade/bna 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-10T10:45:41.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/cisco 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/cisco/enic 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dlink 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-10T10:45:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dnet.ko 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/emulex 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/emulex/benet 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/ethoc.ko 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/fealnx.ko 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/e1000 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/e1000e 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/fm10k 2024-05-10T10:45:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-10T10:45:41.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/i40e 2024-05-10T10:45:41.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-10T10:45:41.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/iavf 2024-05-10T10:45:41.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-10T10:45:41.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/igb 2024-05-10T10:45:41.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-10T10:45:41.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/igbvf 2024-05-10T10:45:41.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-10T10:45:41.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-10T10:45:41.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-10T10:45:41.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-10T10:45:41.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-10T10:45:41.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/jme.ko 2024-05-10T10:45:41.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/marvell 2024-05-10T10:45:41.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-10T10:45:41.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-10T10:45:41.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-10T10:45:41.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox 2024-05-10T10:45:41.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-10T10:45:41.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-10T10:45:41.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-10T10:45:41.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-10T10:45:41.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-10T10:45:41.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-10T10:45:41.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/micrel 2024-05-10T10:45:41.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-10T10:45:41.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/myricom 2024-05-10T10:45:41.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-10T10:45:41.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-10T10:45:41.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/natsemi 2024-05-10T10:45:41.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-10T10:45:41.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-10T10:45:41.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/neterion 2024-05-10T10:45:41.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-10T10:45:41.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/nvidia 2024-05-10T10:45:41.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-10T10:45:41.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/packetengines 2024-05-10T10:45:41.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-10T10:45:41.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-10T10:45:41.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/qlogic 2024-05-10T10:45:41.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-10T10:45:41.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-10T10:45:41.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-10T10:45:41.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-10T10:45:41.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-10T10:45:41.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/rdc 2024-05-10T10:45:41.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-10T10:45:41.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/realtek 2024-05-10T10:45:41.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-10T10:45:41.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-10T10:45:41.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-10T10:45:41.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-10T10:45:41.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/rocker 2024-05-10T10:45:41.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-10T10:45:41.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sfc 2024-05-10T10:45:41.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-10T10:45:41.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/silan 2024-05-10T10:45:41.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-10T10:45:41.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sis 2024-05-10T10:45:41.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-10T10:45:41.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-10T10:45:41.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/smsc 2024-05-10T10:45:41.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-10T10:45:41.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-10T10:45:41.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-10T10:45:41.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-10T10:45:41.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/stmicro 2024-05-10T10:45:41.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-10T10:45:41.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-10T10:45:41.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-10T10:45:41.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sun 2024-05-10T10:45:41.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-10T10:45:41.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-10T10:45:41.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-10T10:45:41.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-10T10:45:41.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/tehuti 2024-05-10T10:45:41.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-10T10:45:41.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/ti 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/via 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/wiznet 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/xircom 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/geneve.ko 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/hyperv 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ieee802154 2024-05-10T10:45:41.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ifb.ko 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ipvlan 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/macvlan.ko 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/macvtap.ko 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/mdio 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/mdio.ko 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/mii.ko 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/net_failover.ko 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/netconsole.ko 2024-05-10T10:45:41.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/nlmon.ko 2024-05-10T10:45:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/pcs 2024-05-10T10:45:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-10T10:45:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-10T10:45:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy 2024-05-10T10:45:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/amd.ko 2024-05-10T10:45:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-10T10:45:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-10T10:45:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-10T10:45:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/bcm87xx.ko 2024-05-10T10:45:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/broadcom.ko 2024-05-10T10:45:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/cicada.ko 2024-05-10T10:45:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/davicom.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/dp83640.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/et1011c.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/icplus.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/lxt.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/marvell.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/micrel.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/national.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/phylink.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/qsemi.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/realtek.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/smsc.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/ste10Xp.ko 2024-05-10T10:45:41.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/vitesse.ko 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/ppp_async.ko 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/pppoe.ko 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/pppox.ko 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/pptp.ko 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/slip 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/slip/slhc.ko 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/slip/slip.ko 2024-05-10T10:45:41.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/sungem_phy.ko 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/tap.ko 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team/team.ko 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team/team_mode_random.ko 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/tun.ko 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/veth.ko 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/virtio_net.ko 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/vmxnet3 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-10T10:45:41.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/vxlan 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/vxlan/vxlan.ko 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/xen-netback 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/xen-netfront.ko 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ntb 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ntb/ntb.ko 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/nvme 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/nvme/host 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/nvme/host/nvme-core.ko 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/nvme/host/nvme.ko 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/parport 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/parport/parport.ko 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/parport/parport_cs.ko 2024-05-10T10:45:41.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/parport/parport_pc.ko 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/parport/parport_serial.ko 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pci 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pci/hotplug 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pci/pcie 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pci/xen-pcifront.ko 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pcmcia 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pcmcia/i82092.ko 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pcmcia/pd6729.ko 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pinctrl 2024-05-10T10:45:41.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pinctrl/intel 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/acerhdf.ko 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-10T10:45:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/hdaps.ko 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/intel 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/intel_ips.ko 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-10T10:45:41.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/wmi.ko 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/powercap 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pps 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pps/clients 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pps/clients/pps_parport.ko 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pps/pps_core.ko 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ptp 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ptp/ptp.ko 2024-05-10T10:45:41.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ptp/ptp_kvm.ko 2024-05-10T10:45:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ras 2024-05-10T10:45:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ras/amd 2024-05-10T10:45:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ras/amd/atl 2024-05-10T10:45:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-10T10:45:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ras/amd/fmpm.ko 2024-05-10T10:45:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc 2024-05-10T10:45:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-10T10:45:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-10T10:45:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-10T10:45:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-10T10:45:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-10T10:45:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-em3027.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-max6900.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-10T10:45:41.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-x1205.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/3w-9xxx.ko 2024-05-10T10:45:41.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/3w-sas.ko 2024-05-10T10:45:41.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/3w-xxxx.ko 2024-05-10T10:45:41.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/BusLogic.ko 2024-05-10T10:45:41.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/a100u2w.ko 2024-05-10T10:45:41.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/aacraid 2024-05-10T10:45:41.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-10T10:45:41.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/advansys.ko 2024-05-10T10:45:41.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/aic7xxx 2024-05-10T10:45:41.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-10T10:45:41.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-10T10:45:41.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/am53c974.ko 2024-05-10T10:45:41.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/arcmsr 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/atp870u.ko 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/be2iscsi 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/bfa 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/bfa/bfa.ko 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/bnx2fc 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/bnx2i 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/ch.ko 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/csiostor 2024-05-10T10:45:41.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/cxgbi 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/dc395x.ko 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/device_handler 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-10T10:45:41.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/dmx3191d.ko 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/esas2r 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/esp_scsi.ko 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/fcoe 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/fnic 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/fnic/fnic.ko 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/hpsa.ko 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/hptiop.ko 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/hv_storvsc.ko 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/initio.ko 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/ipr.ko 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/ips.ko 2024-05-10T10:45:41.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/isci 2024-05-10T10:45:41.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/isci/isci.ko 2024-05-10T10:45:41.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-10T10:45:41.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-10T10:45:41.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/libfc 2024-05-10T10:45:41.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/libfc/libfc.ko 2024-05-10T10:45:41.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/libiscsi.ko 2024-05-10T10:45:41.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-10T10:45:41.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/libsas 2024-05-10T10:45:41.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/libsas/libsas.ko 2024-05-10T10:45:41.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/lpfc 2024-05-10T10:45:41.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-10T10:45:41.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/megaraid 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/megaraid.ko 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/mpt3sas 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/mvsas 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/mvumi.ko 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/pm8001 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/pmcraid.ko 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/qla1280.ko 2024-05-10T10:45:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/qla2xxx 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/qla4xxx 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/raid_class.ko 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/scsi_debug.ko 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/ses.ko 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/st.ko 2024-05-10T10:45:41.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/stex.ko 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/sym53c8xx_2 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/virtio_scsi.ko 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/wd719x.ko 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ssb 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ssb/ssb.ko 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/iscsi 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/loopback 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/target_core_file.ko 2024-05-10T10:45:41.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/target_core_iblock.ko 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/target_core_mod.ko 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/target_core_pscsi.ko 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/target_core_user.ko 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/tcm_fc 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-10T10:45:41.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-10T10:45:41.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/ipwireless 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/n_gsm.ko 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/n_hdlc.ko 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/nozomi.ko 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/serial 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/serial/8250 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/serial/arc_uart.ko 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/serial/jsm 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ufs 2024-05-10T10:45:41.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ufs/core 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ufs/host 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/uio 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/uio/uio.ko 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/uio/uio_aec.ko 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/uio/uio_cif.ko 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/uio/uio_pci_generic.ko 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/uio/uio_sercos3.ko 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/class 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/class/cdc-acm.ko 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-10T10:45:41.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/class/usblp.ko 2024-05-10T10:45:41.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/class/usbtmc.ko 2024-05-10T10:45:41.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/host 2024-05-10T10:45:41.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-10T10:45:41.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/image 2024-05-10T10:45:41.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/image/mdc800.ko 2024-05-10T10:45:41.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/image/microtek.ko 2024-05-10T10:45:41.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc 2024-05-10T10:45:41.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/adutux.ko 2024-05-10T10:45:41.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/appledisplay.ko 2024-05-10T10:45:41.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/emi26.ko 2024-05-10T10:45:41.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/emi62.ko 2024-05-10T10:45:41.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/ezusb.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/idmouse.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/iowarrior.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/ldusb.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/legousbtower.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/sisusbvga 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/usb3503.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/usblcd.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/uss720.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/yurex.ko 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/phy 2024-05-10T10:45:41.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/phy/phy-generic.ko 2024-05-10T10:45:41.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial 2024-05-10T10:45:41.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/aircable.ko 2024-05-10T10:45:41.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ark3116.ko 2024-05-10T10:45:41.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-10T10:45:41.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ch341.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/cp210x.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/cyberjack.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/empeg.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/io_ti.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ipaq.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ipw.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ir-usb.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-10T10:45:41.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/keyspan.ko 2024-05-10T10:45:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-10T10:45:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-10T10:45:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-10T10:45:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/mct_u232.ko 2024-05-10T10:45:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/mos7720.ko 2024-05-10T10:45:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/mos7840.ko 2024-05-10T10:45:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/navman.ko 2024-05-10T10:45:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/omninet.ko 2024-05-10T10:45:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/opticon.ko 2024-05-10T10:45:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/option.ko 2024-05-10T10:45:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/oti6858.ko 2024-05-10T10:45:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/pl2303.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/qcaux.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/qcserial.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/quatech2.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/safe_serial.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/sierra.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ssu100.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/symbolserial.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/usb_debug.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/visor.ko 2024-05-10T10:45:41.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/whiteheat.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/uas.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-karma.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-10T10:45:41.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-10T10:45:41.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-10T10:45:41.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-10T10:45:41.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/usb-storage.ko 2024-05-10T10:45:41.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/usbip 2024-05-10T10:45:41.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-10T10:45:41.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-10T10:45:41.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-10T10:45:41.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vfio 2024-05-10T10:45:41.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vfio/pci 2024-05-10T10:45:41.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-10T10:45:41.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-10T10:45:41.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vfio/vfio.ko 2024-05-10T10:45:41.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-10T10:45:41.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vhost 2024-05-10T10:45:41.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vhost/vhost.ko 2024-05-10T10:45:41.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-10T10:45:41.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vhost/vhost_net.ko 2024-05-10T10:45:41.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vhost/vhost_scsi.ko 2024-05-10T10:45:41.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/video 2024-05-10T10:45:41.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/video/backlight 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/video/backlight/apple_bl.ko 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/video/backlight/lcd.ko 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio.ko 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio_balloon.ko 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio_mmio.ko 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio_pci.ko 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio_ring.ko 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/masters 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/masters/ds2482.ko 2024-05-10T10:45:41.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/masters/ds2490.ko 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-10T10:45:41.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/wire.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/hpwdt.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/i6300esb.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/ib700wdt.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/ibmasr.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/it87_wdt.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/machzwd.ko 2024-05-10T10:45:41.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/nv_tco.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/softdog.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/via_wdt.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/wdt_pci.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/xen_wdt.ko 2024-05-10T10:45:41.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-evtchn.ko 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-gntalloc.ko 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-gntdev.ko 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-pciback 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-privcmd.ko 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xenfs 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/9p 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/9p/9p.ko 2024-05-10T10:45:41.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/affs 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/affs/affs.ko 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/befs 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/befs/befs.ko 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/binfmt_misc.ko 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/btrfs 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/btrfs/btrfs.ko 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/cachefiles 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/cachefiles/cachefiles.ko 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ceph 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ceph/ceph.ko 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/coda 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/coda/coda.ko 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/cramfs 2024-05-10T10:45:41.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/cramfs/cramfs.ko 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/dlm 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/dlm/dlm.ko 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ecryptfs 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/f2fs 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/f2fs/f2fs.ko 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fat 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fat/fat.ko 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fat/msdos.ko 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fat/vfat.ko 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fuse 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fuse/cuse.ko 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fuse/fuse.ko 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/gfs2 2024-05-10T10:45:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/gfs2/gfs2.ko 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/hfs 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/hfs/hfs.ko 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/hfsplus 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/hfsplus/hfsplus.ko 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/isofs 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/isofs/isofs.ko 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/jfs 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/jfs/jfs.ko 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/lockd 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/lockd/lockd.ko 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/minix 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/minix/minix.ko 2024-05-10T10:45:41.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/netfs 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/netfs/netfs.ko 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/blocklayout 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/filelayout 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/flexfilelayout 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/nfs.ko 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/nfsv3.ko 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/nfsv4.ko 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs_common 2024-05-10T10:45:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs_common/grace.ko 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs_common/nfs_acl.ko 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfsd 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfsd/nfsd.ko 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nilfs2 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nilfs2/nilfs2.ko 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-celtic.ko 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-centeuro.ko 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-croatian.ko 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-cyrillic.ko 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-gaelic.ko 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-greek.ko 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-iceland.ko 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-inuit.ko 2024-05-10T10:45:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-roman.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-romanian.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-turkish.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp1250.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp1251.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp1255.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp737.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp775.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp850.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp852.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp855.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp857.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp860.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp861.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp862.ko 2024-05-10T10:45:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp863.ko 2024-05-10T10:45:41.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp864.ko 2024-05-10T10:45:41.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp865.ko 2024-05-10T10:45:41.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp866.ko 2024-05-10T10:45:41.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp869.ko 2024-05-10T10:45:41.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp874.ko 2024-05-10T10:45:41.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp932.ko 2024-05-10T10:45:41.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp936.ko 2024-05-10T10:45:41.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp949.ko 2024-05-10T10:45:41.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp950.ko 2024-05-10T10:45:41.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_euc-jp.ko 2024-05-10T10:45:41.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-1.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-13.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-14.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-15.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-2.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-3.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-4.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-5.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-6.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-7.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-9.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_koi8-r.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_koi8-ru.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_koi8-u.ko 2024-05-10T10:45:41.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_utf8.ko 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/cluster 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/dlm 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/dlmfs 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/ocfs2.ko 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/overlayfs 2024-05-10T10:45:41.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/overlayfs/overlay.ko 2024-05-10T10:45:41.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/pstore 2024-05-10T10:45:41.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/pstore/ramoops.ko 2024-05-10T10:45:41.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/reiserfs 2024-05-10T10:45:41.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/reiserfs/reiserfs.ko 2024-05-10T10:45:41.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/romfs 2024-05-10T10:45:41.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/romfs/romfs.ko 2024-05-10T10:45:41.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/smb 2024-05-10T10:45:41.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/smb/client 2024-05-10T10:45:41.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/smb/client/cifs.ko 2024-05-10T10:45:41.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/smb/common 2024-05-10T10:45:41.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/smb/common/cifs_arc4.ko 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/smb/common/cifs_md4.ko 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/squashfs 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/squashfs/squashfs.ko 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/sysv 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/sysv/sysv.ko 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/udf 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/udf/udf.ko 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ufs 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ufs/ufs.ko 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/xfs 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/xfs/xfs.ko 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/kernel 2024-05-10T10:45:41.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/kernel/rcu 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/kernel/rcu/rcutorture.ko 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/kernel/torture.ko 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/kernel/trace 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/asn1_encoder.ko 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/crc-itu-t.ko 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/crc8.ko 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/crypto 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/crypto/libarc4.ko 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/crypto/libdes.ko 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/libcrc32c.ko 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/lru_cache.ko 2024-05-10T10:45:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/lz4 2024-05-10T10:45:41.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/lz4/lz4_compress.ko 2024-05-10T10:45:41.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/lz4/lz4hc_compress.ko 2024-05-10T10:45:41.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/math 2024-05-10T10:45:41.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/math/cordic.ko 2024-05-10T10:45:41.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/raid6 2024-05-10T10:45:41.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/raid6/raid6_pq.ko 2024-05-10T10:45:41.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/reed_solomon 2024-05-10T10:45:41.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-10T10:45:41.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/ts_bm.ko 2024-05-10T10:45:41.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/ts_fsm.ko 2024-05-10T10:45:41.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/ts_kmp.ko 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/zstd 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/zstd/zstd_compress.ko 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/mm 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/mm/hwpoison-inject.ko 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/6lowpan.ko 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_dest.ko 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_fragment.ko 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_hop.ko 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_mobility.ko 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_routing.ko 2024-05-10T10:45:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_udp.ko 2024-05-10T10:45:41.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/802 2024-05-10T10:45:41.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/802/garp.ko 2024-05-10T10:45:41.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/802/mrp.ko 2024-05-10T10:45:41.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/802/p8022.ko 2024-05-10T10:45:41.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/802/psnap.ko 2024-05-10T10:45:41.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/802/stp.ko 2024-05-10T10:45:41.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/8021q 2024-05-10T10:45:41.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/8021q/8021q.ko 2024-05-10T10:45:41.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/9p 2024-05-10T10:45:41.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/9p/9pnet.ko 2024-05-10T10:45:41.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/9p/9pnet_fd.ko 2024-05-10T10:45:41.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/9p/9pnet_rdma.ko 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/9p/9pnet_virtio.ko 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/batman-adv 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/batman-adv/batman-adv.ko 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/br_netfilter.ko 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/bridge.ko 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-10T10:45:41.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-10T10:45:41.260 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebtables.ko 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ceph 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ceph/libceph.ko 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/core 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/core/failover.ko 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/core/pktgen.ko 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dccp 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dccp/dccp.ko 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dccp/dccp_diag.ko 2024-05-10T10:45:41.261 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dccp/dccp_ipv4.ko 2024-05-10T10:45:41.262 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dccp/dccp_ipv6.ko 2024-05-10T10:45:41.262 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dns_resolver 2024-05-10T10:45:41.262 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dns_resolver/dns_resolver.ko 2024-05-10T10:45:41.262 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dsa 2024-05-10T10:45:41.262 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dsa/dsa_core.ko 2024-05-10T10:45:41.262 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dsa/tag_brcm.ko 2024-05-10T10:45:41.262 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dsa/tag_dsa.ko 2024-05-10T10:45:41.262 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dsa/tag_none.ko 2024-05-10T10:45:41.262 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dsa/tag_trailer.ko 2024-05-10T10:45:41.262 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ieee802154 2024-05-10T10:45:41.262 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ieee802154/6lowpan 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ieee802154/ieee802154.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/ah4.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/esp4.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/fou.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/gre.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/inet_diag.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/ip_gre.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/ip_tunnel.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/ip_vti.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/ipcomp.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/ipip.ko 2024-05-10T10:45:41.263 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-10T10:45:41.264 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_bic.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_dctcp.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_diag.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_highspeed.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_htcp.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_hybla.ko 2024-05-10T10:45:41.265 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_illinois.ko 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_lp.ko 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_scalable.ko 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_vegas.ko 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_veno.ko 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_westwood.ko 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_yeah.ko 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tunnel4.ko 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/udp_diag.ko 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/udp_tunnel.ko 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/ah6.ko 2024-05-10T10:45:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/esp6.ko 2024-05-10T10:45:41.267 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/fou6.ko 2024-05-10T10:45:41.267 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/ip6_tunnel.ko 2024-05-10T10:45:41.267 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-10T10:45:41.267 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/ip6_vti.ko 2024-05-10T10:45:41.267 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/ipcomp6.ko 2024-05-10T10:45:41.267 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter 2024-05-10T10:45:41.267 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-10T10:45:41.267 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-10T10:45:41.267 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-10T10:45:41.267 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-10T10:45:41.267 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-10T10:45:41.267 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-10T10:45:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/sit.ko 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/tunnel6.ko 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/key 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/key/af_key.ko 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_core.ko 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_eth.ko 2024-05-10T10:45:41.269 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_ip.ko 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_ip6.ko 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_netlink.ko 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_ppp.ko 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/llc 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/llc/llc.ko 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/mac802154 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/mac802154/mac802154.ko 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/mpls 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/mpls/mpls_gso.ko 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set.ko 2024-05-10T10:45:41.270 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-10T10:45:41.271 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-10T10:45:41.271 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-10T10:45:41.271 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-10T10:45:41.271 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-10T10:45:41.271 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-10T10:45:41.271 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-10T10:45:41.271 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-10T10:45:41.271 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-10T10:45:41.271 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-10T10:45:41.271 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-10T10:45:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conncount.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-10T10:45:41.273 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_flow_table.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_log_syslog.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_nat.ko 2024-05-10T10:45:41.274 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_nat_irc.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_nat_sip.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_tables.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nfnetlink.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nfnetlink_log.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_chain_nat.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_compat.ko 2024-05-10T10:45:41.275 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_ct.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_fib.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_hash.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_limit.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_log.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_masq.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_nat.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_numgen.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_queue.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_quota.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_redir.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_reject.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_reject_inet.ko 2024-05-10T10:45:41.276 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_socket.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_tunnel.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_AUDIT.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_CT.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_DSCP.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_HL.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_HMARK.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_LED.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_LOG.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_NETMAP.ko 2024-05-10T10:45:41.277 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_NFLOG.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_RATEEST.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_SECMARK.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_TEE.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_TPROXY.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_TRACE.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_addrtype.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_bpf.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_cgroup.ko 2024-05-10T10:45:41.278 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_cluster.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_comment.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_connbytes.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_connlabel.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_connlimit.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_connmark.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_conntrack.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_cpu.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_dccp.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_devgroup.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_dscp.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_ecn.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_esp.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_hashlimit.ko 2024-05-10T10:45:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_helper.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_hl.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_ipcomp.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_iprange.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_ipvs.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_l2tp.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_length.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_limit.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_mac.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_mark.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_multiport.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_nat.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_nfacct.ko 2024-05-10T10:45:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_osf.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_owner.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_physdev.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_pkttype.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_policy.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_quota.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_rateest.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_realm.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_recent.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_sctp.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_set.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_socket.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_state.ko 2024-05-10T10:45:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_statistic.ko 2024-05-10T10:45:41.282 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_string.ko 2024-05-10T10:45:41.282 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_tcpmss.ko 2024-05-10T10:45:41.282 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_time.ko 2024-05-10T10:45:41.282 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_u32.ko 2024-05-10T10:45:41.282 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netlink 2024-05-10T10:45:41.282 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netlink/netlink_diag.ko 2024-05-10T10:45:41.282 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/nsh 2024-05-10T10:45:41.282 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/nsh/nsh.ko 2024-05-10T10:45:41.282 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/openvswitch 2024-05-10T10:45:41.282 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/openvswitch/openvswitch.ko 2024-05-10T10:45:41.282 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/openvswitch/vport-geneve.ko 2024-05-10T10:45:41.282 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/openvswitch/vport-gre.ko 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/openvswitch/vport-vxlan.ko 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/packet 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/packet/af_packet_diag.ko 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/rds 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/rds/rds.ko 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/rds/rds_rdma.ko 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/rds/rds_tcp.ko 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_csum.ko 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_gact.ko 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_mirred.ko 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_nat.ko 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_pedit.ko 2024-05-10T10:45:41.283 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_police.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_simple.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_skbedit.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_vlan.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/cls_basic.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/cls_bpf.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/cls_flow.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/cls_fw.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/cls_route.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/cls_u32.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/em_cmp.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/em_ipset.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/em_meta.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/em_nbyte.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/em_text.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/em_u32.ko 2024-05-10T10:45:41.284 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_choke.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_codel.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_drr.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_fq.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_gred.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_hfsc.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_hhf.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_htb.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_ingress.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_mqprio.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_mqprio_lib.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_multiq.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_netem.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_pie.ko 2024-05-10T10:45:41.285 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_plug.ko 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_prio.ko 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_qfq.ko 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_red.ko 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_sfb.ko 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_sfq.ko 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_tbf.ko 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_teql.ko 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sctp 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sctp/sctp.ko 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sctp/sctp_diag.ko 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc/auth_gss 2024-05-10T10:45:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc/sunrpc.ko 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc/xprtrdma 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/tipc 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/tipc/tipc.ko 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/tipc/tipc_diag.ko 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/unix 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/unix/unix_diag.ko 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/vmw_vsock 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/vmw_vsock/vsock.ko 2024-05-10T10:45:41.287 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-10T10:45:41.288 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-10T10:45:41.288 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/xfrm 2024-05-10T10:45:41.288 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-10T10:45:41.288 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/security 2024-05-10T10:45:41.288 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/security/keys 2024-05-10T10:45:41.288 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/security/keys/encrypted-keys 2024-05-10T10:45:41.288 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-10T10:45:41.288 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/security/keys/trusted-keys 2024-05-10T10:45:41.288 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/security/keys/trusted-keys/trusted.ko 2024-05-10T10:45:41.288 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.alias 2024-05-10T10:45:41.289 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.alias.bin 2024-05-10T10:45:41.289 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.builtin 2024-05-10T10:45:41.289 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.builtin.alias.bin 2024-05-10T10:45:41.289 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.builtin.bin 2024-05-10T10:45:41.289 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.builtin.modinfo 2024-05-10T10:45:41.289 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.dep 2024-05-10T10:45:41.289 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.dep.bin 2024-05-10T10:45:41.289 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.devname 2024-05-10T10:45:41.289 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.order 2024-05-10T10:45:41.289 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.softdep 2024-05-10T10:45:41.289 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.symbols 2024-05-10T10:45:41.289 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.symbols.bin 2024-05-10T10:45:41.290 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/vmlinuz 2024-05-10T10:45:41.294 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-gfb9a6a44107e 2024-05-10T10:45:41.294 DEBUG:teuthology.orchestra.run.smithi027:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-10T10:45:41.317 INFO:teuthology.orchestra.run.smithi027.stdout:postinstall scriptlet (using /bin/sh): 2024-05-10T10:45:41.317 INFO:teuthology.orchestra.run.smithi027.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-10T10:45:41.317 INFO:teuthology.orchestra.run.smithi027.stdout: /usr/bin/kernel-install add 6.9.0-rc3-gfb9a6a44107e /lib/modules/6.9.0-rc3-gfb9a6a44107e/vmlinuz 2024-05-10T10:45:41.317 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-05-10T10:45:41.317 INFO:teuthology.orchestra.run.smithi027.stdout:for file in vmlinuz System.map config; do 2024-05-10T10:45:41.317 INFO:teuthology.orchestra.run.smithi027.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-gfb9a6a44107e/${file}" "/boot/${file}-6.9.0-rc3-gfb9a6a44107e"; then 2024-05-10T10:45:41.317 INFO:teuthology.orchestra.run.smithi027.stdout: cp "/lib/modules/6.9.0-rc3-gfb9a6a44107e/${file}" "/boot/${file}-6.9.0-rc3-gfb9a6a44107e" 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout: fi 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout:done 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout:if [ -d "/lib/modules/6.9.0-rc3-gfb9a6a44107e/dtb" ] && \ 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-gfb9a6a44107e/dtb" "/boot/dtb-6.9.0-rc3-gfb9a6a44107e" >/dev/null 2>&1; then 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout: rm -rf "/boot/dtb-6.9.0-rc3-gfb9a6a44107e" 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout: cp -r "/lib/modules/6.9.0-rc3-gfb9a6a44107e/dtb" "/boot/dtb-6.9.0-rc3-gfb9a6a44107e" 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.dep" ]; then 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout: /usr/sbin/depmod 6.9.0-rc3-gfb9a6a44107e 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-10T10:45:41.318 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-install remove 6.9.0-rc3-gfb9a6a44107e 2024-05-10T10:45:41.319 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-05-10T10:45:41.319 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2024-05-10T10:45:41.391 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2024-05-10T10:45:41.391 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2024-05-10T10:45:41.391 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.06 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 68.el9 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:Group : Unspecified 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 8264254 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : builder@centos.org 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-10T10:45:41.392 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2024-05-10T10:45:41.393 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2024-05-10T10:45:41.393 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:45:41.393 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-10T10:45:41.393 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-10T10:45:41.393 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-10T10:45:41.393 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2024-05-10T10:45:41.393 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:45:41.393 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2024-05-10T10:45:41.394 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-10T10:45:41.394 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 6.9.0-rc3-gfb9a6a44107e 2024-05-10T10:45:41.394 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-10T10:45:41.917 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-05-10T10:45:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-10T10:45:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-05-10T10:45:43.270 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-05-10T10:45:43.289 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.wGxFhAfPXW 2024-05-10T10:45:43.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wGxFhAfPXW 2024-05-10T10:45:43.355 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.wGxFhAfPXW 2024-05-10T10:45:43.493 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.wGxFhAfPXW is 6KB 2024-05-10T10:45:43.541 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.wGxFhAfPXW 2024-05-10T10:45:43.556 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2024-05-10T10:45:43.624 INFO:teuthology.orchestra.run.smithi027.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-10T10:45:43.624 INFO:teuthology.orchestra.run.smithi027.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-10T10:45:43.624 INFO:teuthology.orchestra.run.smithi027.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-10T10:45:43.624 INFO:teuthology.orchestra.run.smithi027.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-gfb9a6a44107e.conf 2024-05-10T10:45:43.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-gfb9a6a44107e 2024-05-10T10:45:43.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2024-05-10T10:46:13.788 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-10T10:46:13.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-10T10:46:13.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-10T10:46:32.223 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-05-10T10:46:41.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-10T10:46:41.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-10T10:46:44.290 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-05-10T10:46:56.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-10T10:46:56.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-10T10:46:59.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-05-10T10:47:14.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-10T10:47:14.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-10T10:47:17.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-05-10T10:47:35.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-10T10:47:35.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-10T10:48:35.442 DEBUG:teuthology.orchestra.remote:timed out 2024-05-10T10:48:56.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-10T10:48:56.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-10T10:48:57.219 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-05-10T10:48:57.859 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-10T10:48:57.859 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-gfb9a6a44107e"... 2024-05-10T10:48:57.859 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-05-10T10:48:57.914 INFO:teuthology.orchestra.run.smithi027.stdout:6.9.0-rc3-gfb9a6a44107e 2024-05-10T10:48:57.914 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-gfb9a6a44107e vs 6.9.0-rc3-gfb9a6a44107e 2024-05-10T10:48:57.915 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-10T10:48:57.915 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-10T10:48:58.916 DEBUG:teuthology.parallel:result is None 2024-05-10T10:48:58.917 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-10T10:48:58.923 INFO:teuthology.task.internal:Creating test directory... 2024-05-10T10:48:58.924 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-10T10:48:58.931 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-10T10:48:58.934 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-10T10:48:58.983 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-10T10:48:58.989 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-10T10:48:59.073 INFO:teuthology.task.internal:Creating archive directory... 2024-05-10T10:48:59.073 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-10T10:48:59.080 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-10T10:48:59.082 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-10T10:48:59.163 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-10T10:48:59.170 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-10T10:48:59.170 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-10T10:48:59.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T10:48:59.186 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-10T10:48:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T10:48:59.203 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-10T10:48:59.211 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-10T10:48:59.229 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-10T10:48:59.272 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-10T10:48:59.276 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-10T10:48:59.290 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-10T10:48:59.290 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-10T10:48:59.330 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-10T10:48:59.382 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-10T10:48:59.387 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-10T10:48:59.394 INFO:teuthology.task.internal:Configuring sudo... 2024-05-10T10:48:59.394 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-10T10:48:59.401 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-10T10:48:59.404 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-10T10:48:59.485 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-10T10:48:59.494 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-10T10:48:59.495 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-10T10:48:59.502 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-10T10:48:59.505 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-10T10:48:59.553 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-10T10:48:59.637 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-10T10:48:59.745 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-10T10:48:59.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-10T10:48:59.930 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T10:48:59.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-10T10:49:00.024 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-10T10:49:00.067 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-10T10:49:00.120 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-10T10:49:00.204 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-10T10:49:00.248 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-05-10T10:49:00.248 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-10T10:49:00.315 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-10T10:49:00.361 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-10T10:49:00.424 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-10T10:49:00.470 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-10T10:49:00.510 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-10T10:49:00.510 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-10T10:49:00.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-05-10T10:49:00.584 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2024-05-10T10:49:00.586 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-05-10T10:49:00.618 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-10T10:49:00.655 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-10T10:49:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-10T10:49:01.215 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-10T10:49:01.223 INFO:teuthology.task.internal:Starting timer... 2024-05-10T10:49:01.223 INFO:teuthology.run_tasks:Running task pcp... 2024-05-10T10:49:01.234 INFO:teuthology.run_tasks:Running task selinux... 2024-05-10T10:49:01.243 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-10T10:49:01.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-05-10T10:49:01.362 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-05-10T10:49:01.367 DEBUG:teuthology.orchestra.run.smithi076:> sudo service auditd rotate 2024-05-10T10:49:01.424 INFO:teuthology.orchestra.run.smithi076.stdout:Rotating logs: 2024-05-10T10:49:01.426 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-05-10T10:49:01.493 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-05-10T10:49:01.495 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-10T10:49:01.495 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-05-10T10:49:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-05-10T10:49:01.560 DEBUG:teuthology.orchestra.run.smithi076:> /usr/sbin/getenforce 2024-05-10T10:49:01.591 INFO:teuthology.orchestra.run.smithi076.stdout:Permissive 2024-05-10T10:49:01.592 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-05-10T10:49:01.635 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-05-10T10:49:01.635 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi076.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-05-10T10:49:01.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-10T10:49:01.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T10:49:01.724 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-10T10:49:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T10:49:01.757 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-10T10:49:01.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T10:49:01.786 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-10T10:49:01.787 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-05-10T10:49:01.873 DEBUG:teuthology.orchestra.run.smithi076:> sudo /usr/sbin/setenforce permissive 2024-05-10T10:49:01.898 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-05-10T10:49:01.923 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-10T10:49:03.019 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-10T10:49:03.031 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-10T10:49:03.213 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-10T10:49:03.228 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-10T10:49:03.230 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi076.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-10T10:54:51.543 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi076.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-05-10T10:54:51.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-10T10:54:51.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-10T10:54:51.663 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-05-10T10:54:51.832 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-10T10:54:51.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-05-10T10:54:51.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-10T10:54:51.905 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-05-10T10:54:51.983 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-05-10T10:54:51.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-05-10T10:54:51.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-10T10:54:52.060 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-05-10T10:54:52.146 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-05-10T10:54:52.147 INFO:teuthology.run_tasks:Running task clock... 2024-05-10T10:54:52.160 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-10T10:54:52.160 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-10T10:54:52.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-10T10:54:52.169 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-10T10:54:52.169 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-10T10:54:52.171 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-10T10:54:52.172 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-10T10:54:52.198 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-10T10:54:52.213 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-10T10:54:52.216 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-10T10:54:52.230 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-10T10:54:52.237 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: ntpd: command not found 2024-05-10T10:54:52.248 INFO:teuthology.orchestra.run.smithi076.stdout:506 Cannot talk to daemon 2024-05-10T10:54:52.253 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-05-10T10:54:52.255 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-10T10:54:52.261 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-10T10:54:52.264 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-05-10T10:54:52.274 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-10T10:54:52.277 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-10T10:54:52.290 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-10T10:54:52.301 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-10T10:54:52.339 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-05-10T10:54:52.343 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-10T10:54:52.343 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-05-10T10:54:52.343 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T10:54:52.344 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T10:54:52.344 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T10:54:52.344 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T10:54:52.344 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-05-10T10:54:52.347 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-10T10:54:52.347 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-05-10T10:54:52.347 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T10:54:52.347 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T10:54:52.347 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T10:54:52.347 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T10:54:52.382 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-05-10T10:54:52.415 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-05-10T10:54:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-10T10:54:52.501 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-10T10:54:52.654 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-05-10T10:54:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-10T10:54:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-05-10T10:54:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T10:54:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T10:54:52.663 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T10:54:52.663 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T10:54:52.664 INFO:teuthology.run_tasks:Running task install... 2024-05-10T10:54:52.673 DEBUG:teuthology.task.install:project ceph 2024-05-10T10:54:52.673 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd39fa08867836ac070c47cb66c782fa559fd480b'}} 2024-05-10T10:54:52.674 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd39fa08867836ac070c47cb66c782fa559fd480b'} 2024-05-10T10:54:52.674 INFO:teuthology.task.install:Using flavor: default 2024-05-10T10:54:52.679 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-10T10:54:52.679 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-10T10:54:52.680 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd39fa08867836ac070c47cb66c782fa559fd480b', 'tag': None, 'wait_for_package': False} 2024-05-10T10:54:52.680 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T10:54:52.682 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd39fa08867836ac070c47cb66c782fa559fd480b', 'tag': None, 'wait_for_package': False} 2024-05-10T10:54:52.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T10:54:52.683 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd39fa08867836ac070c47cb66c782fa559fd480b', 'tag': None, 'wait_for_package': False} 2024-05-10T10:54:52.683 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T10:54:52.825 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/ 2024-05-10T10:54:52.826 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3449.gd39fa088 2024-05-10T10:54:52.901 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/ 2024-05-10T10:54:52.901 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3449.gd39fa088 2024-05-10T10:54:52.914 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/ 2024-05-10T10:54:52.914 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3449.gd39fa088 2024-05-10T10:54:52.939 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-10T10:54:52.939 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-10T10:54:52.939 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-10T10:54:52.980 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-10T10:54:52.980 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d39fa08867836ac070c47cb66c782fa559fd480b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-10T10:54:53.023 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-10T10:54:53.023 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-05-10T10:54:53.023 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-10T10:54:53.045 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-10T10:54:53.049 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-10T10:54:53.049 DEBUG:teuthology.orchestra.run.smithi076:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d39fa08867836ac070c47cb66c782fa559fd480b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-10T10:54:53.075 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-10T10:54:53.075 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T10:54:53.075 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-10T10:54:53.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-10T10:54:53.123 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-10T10:54:53.170 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-10T10:54:53.170 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d39fa08867836ac070c47cb66c782fa559fd480b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-10T10:54:53.186 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-05-10T10:54:53.188 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-05-10T10:54:53.189 DEBUG:teuthology.orchestra.run.smithi076:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-10T10:54:53.253 INFO:teuthology.orchestra.run.smithi076.stdout:check_obsoletes = 1 2024-05-10T10:54:53.255 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum clean all 2024-05-10T10:54:53.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-10T10:54:53.414 INFO:teuthology.orchestra.run.smithi189.stdout:54 files removed 2024-05-10T10:54:53.424 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-10T10:54:53.441 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-10T10:54:53.479 INFO:teuthology.orchestra.run.smithi076.stdout:54 files removed 2024-05-10T10:54:53.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-10T10:54:53.507 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-05-10T10:54:53.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-05-10T10:54:53.853 INFO:teuthology.orchestra.run.smithi027.stdout:54 files removed 2024-05-10T10:54:53.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-10T10:54:54.117 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 202 kB/s | 84 kB 00:00 2024-05-10T10:54:54.251 INFO:teuthology.orchestra.run.smithi076.stdout:ceph packages for x86_64 174 kB/s | 84 kB 00:00 2024-05-10T10:54:54.493 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-05-10T10:54:54.635 INFO:teuthology.orchestra.run.smithi076.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-05-10T10:54:54.869 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-05-10T10:54:54.930 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 147 kB/s | 84 kB 00:00 2024-05-10T10:54:55.003 INFO:teuthology.orchestra.run.smithi076.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-05-10T10:54:55.473 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 27 kB/s | 12 kB 00:00 2024-05-10T10:54:56.057 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-05-10T10:54:56.223 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 6.8 MB/s | 8.1 MB 00:01 2024-05-10T10:54:56.444 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.1 MB 00:01 2024-05-10T10:54:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.1 MB 00:00 2024-05-10T10:54:59.139 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-05-10T10:55:01.078 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 5.4 MB/s | 19 MB 00:03 2024-05-10T10:55:04.523 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.2 MB 00:01 2024-05-10T10:55:05.854 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.2 MB 00:01 2024-05-10T10:55:06.049 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-05-10T10:55:07.126 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-05-10T10:55:07.405 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-05-10T10:55:08.532 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-05-10T10:55:09.314 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 1.7 MB/s | 19 MB 00:11 2024-05-10T10:55:12.344 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-05-10T10:55:13.657 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-05-10T10:55:14.190 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.2 MB 00:00 2024-05-10T10:55:14.604 INFO:teuthology.orchestra.run.smithi076.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-10T10:55:14.604 INFO:teuthology.orchestra.run.smithi076.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-10T10:55:14.615 INFO:teuthology.orchestra.run.smithi076.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-10T10:55:14.616 INFO:teuthology.orchestra.run.smithi076.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-10T10:55:14.617 INFO:teuthology.orchestra.run.smithi076.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-10T10:55:14.618 INFO:teuthology.orchestra.run.smithi076.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-10T10:55:14.619 INFO:teuthology.orchestra.run.smithi076.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-10T10:55:14.621 INFO:teuthology.orchestra.run.smithi076.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-10T10:55:14.622 INFO:teuthology.orchestra.run.smithi076.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-10T10:55:14.683 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-05-10T10:55:14.697 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2024-05-10T10:55:14.697 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-05-10T10:55:14.697 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2024-05-10T10:55:14.697 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-05-10T10:55:14.697 INFO:teuthology.orchestra.run.smithi076.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-10T10:55:14.698 INFO:teuthology.orchestra.run.smithi076.stdout: ceph x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 6.6 k 2024-05-10T10:55:14.698 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 5.4 M 2024-05-10T10:55:14.698 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 890 k 2024-05-10T10:55:14.698 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 145 k 2024-05-10T10:55:14.698 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 1.5 M 2024-05-10T10:55:14.698 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 142 k 2024-05-10T10:55:14.698 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 3.5 M 2024-05-10T10:55:14.698 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 7.4 M 2024-05-10T10:55:14.698 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 49 k 2024-05-10T10:55:14.698 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 11 M 2024-05-10T10:55:14.698 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 49 M 2024-05-10T10:55:14.698 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 273 k 2024-05-10T10:55:14.699 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 763 k 2024-05-10T10:55:14.699 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-mirror x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 225 k 2024-05-10T10:55:14.699 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-shell x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 27 k 2024-05-10T10:55:14.699 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-top noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 18 k 2024-05-10T10:55:14.699 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-10T10:55:14.699 INFO:teuthology.orchestra.run.smithi076.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-10T10:55:14.699 INFO:teuthology.orchestra.run.smithi076.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-10T10:55:14.699 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 32 k 2024-05-10T10:55:14.699 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 746 k 2024-05-10T10:55:14.699 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 127 k 2024-05-10T10:55:14.699 INFO:teuthology.orchestra.run.smithi076.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-10T10:55:14.699 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-10T10:55:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-10T10:55:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-10T10:55:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-10T10:55:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-10T10:55:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 161 k 2024-05-10T10:55:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 321 k 2024-05-10T10:55:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 298 k 2024-05-10T10:55:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 99 k 2024-05-10T10:55:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 85 k 2024-05-10T10:55:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 3.1 M 2024-05-10T10:55:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 170 k 2024-05-10T10:55:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-10T10:55:14.701 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrading: 2024-05-10T10:55:14.701 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-10T10:55:14.701 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-10T10:55:14.701 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-10T10:55:14.701 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-10T10:55:14.701 INFO:teuthology.orchestra.run.smithi076.stdout: librados2 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 3.4 M 2024-05-10T10:55:14.701 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 3.1 M 2024-05-10T10:55:14.701 INFO:teuthology.orchestra.run.smithi076.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-10T10:55:14.701 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-10T10:55:14.701 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-05-10T10:55:14.701 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-10T10:55:14.702 INFO:teuthology.orchestra.run.smithi076.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-10T10:55:14.702 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-10T10:55:14.702 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 21 M 2024-05-10T10:55:14.702 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 34 k 2024-05-10T10:55:14.702 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 2.4 M 2024-05-10T10:55:14.702 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 274 k 2024-05-10T10:55:14.702 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 4.7 M 2024-05-10T10:55:14.702 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 17 M 2024-05-10T10:55:14.702 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 16 k 2024-05-10T10:55:14.703 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 25 k 2024-05-10T10:55:14.703 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-10T10:55:14.703 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-10T10:55:14.703 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-10T10:55:14.703 INFO:teuthology.orchestra.run.smithi076.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-10T10:55:14.703 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-10T10:55:14.703 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-10T10:55:14.703 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-10T10:55:14.703 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-10T10:55:14.703 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 163 k 2024-05-10T10:55:14.703 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-10T10:55:14.703 INFO:teuthology.orchestra.run.smithi076.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-10T10:55:14.704 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-10T10:55:14.704 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-10T10:55:14.704 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-10T10:55:14.704 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 500 k 2024-05-10T10:55:14.704 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-10T10:55:14.704 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 5.4 M 2024-05-10T10:55:14.704 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-10T10:55:14.704 INFO:teuthology.orchestra.run.smithi076.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-10T10:55:14.704 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-10T10:55:14.704 INFO:teuthology.orchestra.run.smithi076.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-10T10:55:14.704 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-10T10:55:14.704 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 45 k 2024-05-10T10:55:14.705 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 135 k 2024-05-10T10:55:14.706 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-10T10:55:14.706 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-10T10:55:14.706 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-10T10:55:14.706 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-10T10:55:14.706 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-10T10:55:14.706 INFO:teuthology.orchestra.run.smithi076.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-10T10:55:14.706 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-10T10:55:14.707 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-10T10:55:14.707 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-10T10:55:14.707 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-10T10:55:14.707 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-10T10:55:14.707 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-10T10:55:14.707 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-10T10:55:14.707 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-10T10:55:14.707 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-10T10:55:14.708 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-10T10:55:14.708 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-10T10:55:14.708 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-10T10:55:14.708 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-10T10:55:14.708 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-10T10:55:14.708 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-10T10:55:14.708 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-10T10:55:14.708 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-10T10:55:14.708 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-10T10:55:14.708 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-10T10:55:14.709 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-10T10:55:14.709 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-10T10:55:14.709 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-10T10:55:14.709 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-10T10:55:14.709 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-10T10:55:14.709 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-10T10:55:14.709 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-10T10:55:14.709 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-10T10:55:14.709 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-10T10:55:14.709 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-10T10:55:14.709 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-10T10:55:14.709 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-10T10:55:14.710 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-10T10:55:14.710 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-10T10:55:14.710 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-10T10:55:14.710 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-10T10:55:14.710 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-10T10:55:14.710 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-10T10:55:14.710 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-10T10:55:14.710 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-10T10:55:14.710 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-10T10:55:14.710 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-10T10:55:14.710 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-10T10:55:14.710 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-10T10:55:14.711 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-10T10:55:14.711 INFO:teuthology.orchestra.run.smithi076.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-10T10:55:14.711 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-10T10:55:14.711 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-10T10:55:14.711 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-10T10:55:14.711 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-10T10:55:14.711 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-10T10:55:14.711 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-10T10:55:14.711 INFO:teuthology.orchestra.run.smithi076.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-10T10:55:14.711 INFO:teuthology.orchestra.run.smithi076.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-10T10:55:14.711 INFO:teuthology.orchestra.run.smithi076.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-10T10:55:14.711 INFO:teuthology.orchestra.run.smithi076.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-10T10:55:14.712 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-10T10:55:14.712 INFO:teuthology.orchestra.run.smithi076.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-10T10:55:14.712 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2024-05-10T10:55:14.712 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-10T10:55:14.712 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:55:14.712 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-05-10T10:55:14.712 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2024-05-10T10:55:14.712 INFO:teuthology.orchestra.run.smithi076.stdout:Install 148 Packages 2024-05-10T10:55:14.712 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrade 8 Packages 2024-05-10T10:55:14.712 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:55:14.713 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 226 M 2024-05-10T10:55:14.713 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-05-10T10:55:15.537 INFO:teuthology.orchestra.run.smithi076.stdout:(1/156): ceph-19.0.0-3449.gd39fa088.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-05-10T10:55:15.942 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-10T10:55:15.942 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-10T10:55:15.954 INFO:teuthology.orchestra.run.smithi189.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-10T10:55:15.954 INFO:teuthology.orchestra.run.smithi189.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-10T10:55:15.955 INFO:teuthology.orchestra.run.smithi189.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-10T10:55:15.956 INFO:teuthology.orchestra.run.smithi189.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-10T10:55:15.957 INFO:teuthology.orchestra.run.smithi189.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-10T10:55:15.960 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-10T10:55:15.961 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-10T10:55:16.022 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-05-10T10:55:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-05-10T10:55:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-05-10T10:55:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 6.6 k 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 5.4 M 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 890 k 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 145 k 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 1.5 M 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 142 k 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 3.5 M 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 7.4 M 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 49 k 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 11 M 2024-05-10T10:55:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 49 M 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 273 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 763 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 225 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 27 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 18 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 32 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 746 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 127 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-10T10:55:16.045 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 161 k 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 321 k 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 298 k 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 99 k 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 85 k 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 3.1 M 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 170 k 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-05-10T10:55:16.046 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 3.4 M 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 3.1 M 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 21 M 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 34 k 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 2.4 M 2024-05-10T10:55:16.047 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 274 k 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 4.7 M 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 17 M 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 16 k 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 25 k 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-10T10:55:16.048 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 163 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 500 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 5.4 M 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-10T10:55:16.049 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 45 k 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 135 k 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-10T10:55:16.050 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-10T10:55:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-10T10:55:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-10T10:55:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-10T10:55:16.054 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-10T10:55:16.055 INFO:teuthology.orchestra.run.smithi189.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-10T10:55:16.055 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-10T10:55:16.055 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-10T10:55:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:Installing weak dependencies: 2024-05-10T10:55:16.055 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-10T10:55:16.055 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:55:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-05-10T10:55:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-05-10T10:55:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:Install 148 Packages 2024-05-10T10:55:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 8 Packages 2024-05-10T10:55:16.055 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:55:16.056 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-05-10T10:55:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 226 M 2024-05-10T10:55:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-05-10T10:55:16.354 INFO:teuthology.orchestra.run.smithi076.stdout:(2/156): ceph-fuse-19.0.0-3449.gd39fa088.el9.x8 1.1 MB/s | 890 kB 00:00 2024-05-10T10:55:16.696 INFO:teuthology.orchestra.run.smithi076.stdout:(3/156): ceph-common-19.0.0-3449.gd39fa088.el9. 15 MB/s | 21 MB 00:01 2024-05-10T10:55:16.721 INFO:teuthology.orchestra.run.smithi076.stdout:(4/156): ceph-immutable-object-cache-19.0.0-344 395 kB/s | 145 kB 00:00 2024-05-10T10:55:16.797 INFO:teuthology.orchestra.run.smithi076.stdout:(5/156): ceph-base-19.0.0-3449.gd39fa088.el9.x8 3.6 MB/s | 5.4 MB 00:01 2024-05-10T10:55:16.855 INFO:teuthology.orchestra.run.smithi076.stdout:(6/156): ceph-mds-19.0.0-3449.gd39fa088.el9.x86 15 MB/s | 2.4 MB 00:00 2024-05-10T10:55:16.866 INFO:teuthology.orchestra.run.smithi189.stdout:(1/156): ceph-19.0.0-3449.gd39fa088.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-05-10T10:55:17.384 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-05-10T10:55:17.656 INFO:teuthology.orchestra.run.smithi076.stdout:(7/156): ceph-mgr-19.0.0-3449.gd39fa088.el9.x86 1.6 MB/s | 1.5 MB 00:00 2024-05-10T10:55:17.781 INFO:teuthology.orchestra.run.smithi076.stdout:(8/156): ceph-mon-19.0.0-3449.gd39fa088.el9.x86 4.8 MB/s | 4.7 MB 00:00 2024-05-10T10:55:17.842 INFO:teuthology.orchestra.run.smithi189.stdout:(2/156): ceph-fuse-19.0.0-3449.gd39fa088.el9.x8 913 kB/s | 890 kB 00:00 2024-05-10T10:55:17.925 INFO:teuthology.orchestra.run.smithi189.stdout:(3/156): ceph-immutable-object-cache-19.0.0-344 1.7 MB/s | 145 kB 00:00 2024-05-10T10:55:17.931 INFO:teuthology.orchestra.run.smithi076.stdout:(9/156): ceph-osd-19.0.0-3449.gd39fa088.el9.x86 16 MB/s | 17 MB 00:01 2024-05-10T10:55:17.956 INFO:teuthology.orchestra.run.smithi076.stdout:(10/156): ceph-selinux-19.0.0-3449.gd39fa088.el 143 kB/s | 25 kB 00:00 2024-05-10T10:55:18.048 INFO:teuthology.orchestra.run.smithi076.stdout:(11/156): cephfs-mirror-19.0.0-3449.gd39fa088.e 2.4 MB/s | 225 kB 00:00 2024-05-10T10:55:18.157 INFO:teuthology.orchestra.run.smithi076.stdout:(12/156): cephfs-shell-19.0.0-3449.gd39fa088.el 249 kB/s | 27 kB 00:00 2024-05-10T10:55:18.266 INFO:teuthology.orchestra.run.smithi076.stdout:(13/156): libcephfs-devel-19.0.0-3449.gd39fa088 293 kB/s | 32 kB 00:00 2024-05-10T10:55:18.549 INFO:teuthology.orchestra.run.smithi076.stdout:(14/156): libcephfs2-19.0.0-3449.gd39fa088.el9. 2.6 MB/s | 746 kB 00:00 2024-05-10T10:55:18.642 INFO:teuthology.orchestra.run.smithi076.stdout:(15/156): libcephsqlite-19.0.0-3449.gd39fa088.e 1.7 MB/s | 163 kB 00:00 2024-05-10T10:55:18.767 INFO:teuthology.orchestra.run.smithi076.stdout:(16/156): librados-devel-19.0.0-3449.gd39fa088. 1.0 MB/s | 127 kB 00:00 2024-05-10T10:55:18.951 INFO:teuthology.orchestra.run.smithi076.stdout:(17/156): libradosstriper1-19.0.0-3449.gd39fa08 2.7 MB/s | 500 kB 00:00 2024-05-10T10:55:19.017 INFO:teuthology.orchestra.run.smithi189.stdout:(4/156): ceph-mds-19.0.0-3449.gd39fa088.el9.x86 2.2 MB/s | 2.4 MB 00:01 2024-05-10T10:55:19.517 INFO:teuthology.orchestra.run.smithi189.stdout:(5/156): ceph-mgr-19.0.0-3449.gd39fa088.el9.x86 3.0 MB/s | 1.5 MB 00:00 2024-05-10T10:55:20.551 INFO:teuthology.orchestra.run.smithi189.stdout:(6/156): ceph-mon-19.0.0-3449.gd39fa088.el9.x86 4.5 MB/s | 4.7 MB 00:01 2024-05-10T10:55:20.626 INFO:teuthology.orchestra.run.smithi076.stdout:(18/156): librgw2-19.0.0-3449.gd39fa088.el9.x86 3.2 MB/s | 5.4 MB 00:01 2024-05-10T10:55:20.718 INFO:teuthology.orchestra.run.smithi076.stdout:(19/156): python3-ceph-argparse-19.0.0-3449.gd3 493 kB/s | 45 kB 00:00 2024-05-10T10:55:20.827 INFO:teuthology.orchestra.run.smithi076.stdout:(20/156): python3-ceph-common-19.0.0-3449.gd39f 1.2 MB/s | 135 kB 00:00 2024-05-10T10:55:20.935 INFO:teuthology.orchestra.run.smithi076.stdout:(21/156): python3-cephfs-19.0.0-3449.gd39fa088. 1.5 MB/s | 161 kB 00:00 2024-05-10T10:55:21.111 INFO:teuthology.orchestra.run.smithi076.stdout:(22/156): python3-rados-19.0.0-3449.gd39fa088.e 1.8 MB/s | 321 kB 00:00 2024-05-10T10:55:21.227 INFO:teuthology.orchestra.run.smithi076.stdout:(23/156): python3-rbd-19.0.0-3449.gd39fa088.el9 2.5 MB/s | 298 kB 00:00 2024-05-10T10:55:21.336 INFO:teuthology.orchestra.run.smithi076.stdout:(24/156): python3-rgw-19.0.0-3449.gd39fa088.el9 918 kB/s | 99 kB 00:00 2024-05-10T10:55:21.426 INFO:teuthology.orchestra.run.smithi189.stdout:(7/156): ceph-base-19.0.0-3449.gd39fa088.el9.x8 1.1 MB/s | 5.4 MB 00:04 2024-05-10T10:55:21.528 INFO:teuthology.orchestra.run.smithi076.stdout:(25/156): ceph-radosgw-19.0.0-3449.gd39fa088.el 2.7 MB/s | 11 MB 00:03 2024-05-10T10:55:21.554 INFO:teuthology.orchestra.run.smithi076.stdout:(26/156): rbd-fuse-19.0.0-3449.gd39fa088.el9.x8 393 kB/s | 85 kB 00:00 2024-05-10T10:55:21.735 INFO:teuthology.orchestra.run.smithi189.stdout:(8/156): ceph-common-19.0.0-3449.gd39fa088.el9. 4.2 MB/s | 21 MB 00:05 2024-05-10T10:55:21.777 INFO:teuthology.orchestra.run.smithi189.stdout:(9/156): ceph-selinux-19.0.0-3449.gd39fa088.el9 609 kB/s | 25 kB 00:00 2024-05-10T10:55:21.988 INFO:teuthology.orchestra.run.smithi076.stdout:(27/156): ceph-test-19.0.0-3449.gd39fa088.el9.x 12 MB/s | 49 MB 00:04 2024-05-10T10:55:22.013 INFO:teuthology.orchestra.run.smithi076.stdout:(28/156): rbd-nbd-19.0.0-3449.gd39fa088.el9.x86 371 kB/s | 170 kB 00:00 2024-05-10T10:55:22.046 INFO:teuthology.orchestra.run.smithi076.stdout:(29/156): ceph-grafana-dashboards-19.0.0-3449.g 586 kB/s | 34 kB 00:00 2024-05-10T10:55:22.097 INFO:teuthology.orchestra.run.smithi076.stdout:(30/156): ceph-mgr-cephadm-19.0.0-3449.gd39fa08 1.7 MB/s | 142 kB 00:00 2024-05-10T10:55:22.472 INFO:teuthology.orchestra.run.smithi076.stdout:(31/156): ceph-mgr-dashboard-19.0.0-3449.gd39fa 8.3 MB/s | 3.5 MB 00:00 2024-05-10T10:55:22.556 INFO:teuthology.orchestra.run.smithi076.stdout:(32/156): ceph-mgr-modules-core-19.0.0-3449.gd3 3.2 MB/s | 274 kB 00:00 2024-05-10T10:55:22.631 INFO:teuthology.orchestra.run.smithi076.stdout:(33/156): ceph-mgr-rook-19.0.0-3449.gd39fa088.e 659 kB/s | 49 kB 00:00 2024-05-10T10:55:22.698 INFO:teuthology.orchestra.run.smithi076.stdout:(34/156): rbd-mirror-19.0.0-3449.gd39fa088.el9. 2.6 MB/s | 3.1 MB 00:01 2024-05-10T10:55:22.723 INFO:teuthology.orchestra.run.smithi076.stdout:(35/156): ceph-prometheus-alerts-19.0.0-3449.gd 179 kB/s | 16 kB 00:00 2024-05-10T10:55:22.807 INFO:teuthology.orchestra.run.smithi076.stdout:(36/156): cephadm-19.0.0-3449.gd39fa088.el9.noa 9.0 MB/s | 763 kB 00:00 2024-05-10T10:55:22.835 INFO:teuthology.orchestra.run.smithi189.stdout:(10/156): ceph-osd-19.0.0-3449.gd39fa088.el9.x8 7.4 MB/s | 17 MB 00:02 2024-05-10T10:55:22.840 INFO:teuthology.orchestra.run.smithi076.stdout:(37/156): ceph-volume-19.0.0-3449.gd39fa088.el9 1.9 MB/s | 273 kB 00:00 2024-05-10T10:55:22.866 INFO:teuthology.orchestra.run.smithi076.stdout:(38/156): cephfs-top-19.0.0-3449.gd39fa088.el9. 310 kB/s | 18 kB 00:00 2024-05-10T10:55:22.894 INFO:teuthology.orchestra.run.smithi189.stdout:(11/156): cephfs-mirror-19.0.0-3449.gd39fa088.e 3.8 MB/s | 225 kB 00:00 2024-05-10T10:55:22.944 INFO:teuthology.orchestra.run.smithi189.stdout:(12/156): cephfs-shell-19.0.0-3449.gd39fa088.el 543 kB/s | 27 kB 00:00 2024-05-10T10:55:22.994 INFO:teuthology.orchestra.run.smithi189.stdout:(13/156): libcephfs-devel-19.0.0-3449.gd39fa088 635 kB/s | 32 kB 00:00 2024-05-10T10:55:23.008 INFO:teuthology.orchestra.run.smithi076.stdout:(39/156): fuse-2.9.9-15.el9.x86_64.rpm 481 kB/s | 80 kB 00:00 2024-05-10T10:55:23.092 INFO:teuthology.orchestra.run.smithi076.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-05-10T10:55:23.103 INFO:teuthology.orchestra.run.smithi189.stdout:(14/156): libcephfs2-19.0.0-3449.gd39fa088.el9. 6.7 MB/s | 746 kB 00:00 2024-05-10T10:55:23.134 INFO:teuthology.orchestra.run.smithi076.stdout:(41/156): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2024-05-10T10:55:23.161 INFO:teuthology.orchestra.run.smithi189.stdout:(15/156): libcephsqlite-19.0.0-3449.gd39fa088.e 2.7 MB/s | 163 kB 00:00 2024-05-10T10:55:23.178 INFO:teuthology.orchestra.run.smithi076.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-05-10T10:55:23.220 INFO:teuthology.orchestra.run.smithi189.stdout:(16/156): librados-devel-19.0.0-3449.gd39fa088. 2.1 MB/s | 127 kB 00:00 2024-05-10T10:55:23.235 INFO:teuthology.orchestra.run.smithi076.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-05-10T10:55:23.293 INFO:teuthology.orchestra.run.smithi076.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-05-10T10:55:23.312 INFO:teuthology.orchestra.run.smithi189.stdout:(17/156): libradosstriper1-19.0.0-3449.gd39fa08 5.3 MB/s | 500 kB 00:00 2024-05-10T10:55:23.360 INFO:teuthology.orchestra.run.smithi076.stdout:(45/156): python3-cryptography-36.0.1-4.el9.x86 5.4 MB/s | 1.2 MB 00:00 2024-05-10T10:55:23.386 INFO:teuthology.orchestra.run.smithi076.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-05-10T10:55:23.419 INFO:teuthology.orchestra.run.smithi076.stdout:(47/156): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-05-10T10:55:23.628 INFO:teuthology.orchestra.run.smithi076.stdout:(48/156): ceph-mgr-diskprediction-local-19.0.0- 4.8 MB/s | 7.4 MB 00:01 2024-05-10T10:55:23.670 INFO:teuthology.orchestra.run.smithi076.stdout:(49/156): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 689 kB 00:00 2024-05-10T10:55:23.762 INFO:teuthology.orchestra.run.smithi076.stdout:(50/156): bison-3.7.4-5.el9.x86_64.rpm 2.7 MB/s | 943 kB 00:00 2024-05-10T10:55:23.787 INFO:teuthology.orchestra.run.smithi076.stdout:(51/156): elfutils-libelf-devel-0.190-2.el9.x86 184 kB/s | 21 kB 00:00 2024-05-10T10:55:23.804 INFO:teuthology.orchestra.run.smithi076.stdout:(52/156): boost-program-options-1.75.0-8.el9.x8 608 kB/s | 107 kB 00:00 2024-05-10T10:55:23.812 INFO:teuthology.orchestra.run.smithi189.stdout:(18/156): librgw2-19.0.0-3449.gd39fa088.el9.x86 11 MB/s | 5.4 MB 00:00 2024-05-10T10:55:23.830 INFO:teuthology.orchestra.run.smithi076.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 313 kB 00:00 2024-05-10T10:55:23.855 INFO:teuthology.orchestra.run.smithi076.stdout:(54/156): flexiblas-3.0.4-8.el9.x86_64.rpm 484 kB/s | 33 kB 00:00 2024-05-10T10:55:23.862 INFO:teuthology.orchestra.run.smithi189.stdout:(19/156): python3-ceph-argparse-19.0.0-3449.gd3 905 kB/s | 45 kB 00:00 2024-05-10T10:55:23.906 INFO:teuthology.orchestra.run.smithi076.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-05-10T10:55:23.913 INFO:teuthology.orchestra.run.smithi189.stdout:(20/156): python3-ceph-common-19.0.0-3449.gd39f 2.6 MB/s | 135 kB 00:00 2024-05-10T10:55:23.947 INFO:teuthology.orchestra.run.smithi076.stdout:(56/156): flexiblas-openblas-openmp-3.0.4-8.el9 155 kB/s | 18 kB 00:00 2024-05-10T10:55:23.971 INFO:teuthology.orchestra.run.smithi189.stdout:(21/156): python3-cephfs-19.0.0-3449.gd39fa088. 2.7 MB/s | 161 kB 00:00 2024-05-10T10:55:23.973 INFO:teuthology.orchestra.run.smithi076.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-05-10T10:55:23.998 INFO:teuthology.orchestra.run.smithi076.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 901 kB/s | 45 kB 00:00 2024-05-10T10:55:24.009 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-05-10T10:55:24.038 INFO:teuthology.orchestra.run.smithi189.stdout:(22/156): python3-rados-19.0.0-3449.gd39fa088.e 4.7 MB/s | 321 kB 00:00 2024-05-10T10:55:24.040 INFO:teuthology.orchestra.run.smithi076.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 662 kB 00:00 2024-05-10T10:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout:(60/156): libtool-2.4.6-45.el9.x86_64.rpm 7.5 MB/s | 582 kB 00:00 2024-05-10T10:55:24.097 INFO:teuthology.orchestra.run.smithi189.stdout:(23/156): python3-rbd-19.0.0-3449.gd39fa088.el9 5.0 MB/s | 298 kB 00:00 2024-05-10T10:55:24.099 INFO:teuthology.orchestra.run.smithi076.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 434 kB/s | 25 kB 00:00 2024-05-10T10:55:24.124 INFO:teuthology.orchestra.run.smithi076.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 940 kB/s | 47 kB 00:00 2024-05-10T10:55:24.139 INFO:teuthology.orchestra.run.smithi189.stdout:(24/156): python3-rgw-19.0.0-3449.gd39fa088.el9 2.3 MB/s | 99 kB 00:00 2024-05-10T10:55:24.150 INFO:teuthology.orchestra.run.smithi076.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-05-10T10:55:24.175 INFO:teuthology.orchestra.run.smithi076.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 188 kB 00:00 2024-05-10T10:55:24.189 INFO:teuthology.orchestra.run.smithi189.stdout:(25/156): rbd-fuse-19.0.0-3449.gd39fa088.el9.x8 1.7 MB/s | 85 kB 00:00 2024-05-10T10:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout:(65/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.0 MB/s | 3.0 MB 00:00 2024-05-10T10:55:24.259 INFO:teuthology.orchestra.run.smithi076.stdout:(66/156): openblas-0.3.21-2.el9.x86_64.rpm 323 kB/s | 35 kB 00:00 2024-05-10T10:55:24.343 INFO:teuthology.orchestra.run.smithi076.stdout:(67/156): postgresql-13.14-1.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-05-10T10:55:24.410 INFO:teuthology.orchestra.run.smithi076.stdout:(68/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-05-10T10:55:24.473 INFO:teuthology.orchestra.run.smithi189.stdout:(26/156): rbd-mirror-19.0.0-3449.gd39fa088.el9. 11 MB/s | 3.1 MB 00:00 2024-05-10T10:55:24.486 INFO:teuthology.orchestra.run.smithi076.stdout:(69/156): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-05-10T10:55:24.511 INFO:teuthology.orchestra.run.smithi076.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 133 kB 00:00 2024-05-10T10:55:24.523 INFO:teuthology.orchestra.run.smithi189.stdout:(27/156): rbd-nbd-19.0.0-3449.gd39fa088.el9.x86 3.4 MB/s | 170 kB 00:00 2024-05-10T10:55:24.544 INFO:teuthology.orchestra.run.smithi076.stdout:(71/156): postgresql-contrib-13.14-1.el9.x86_64 4.4 MB/s | 911 kB 00:00 2024-05-10T10:55:24.565 INFO:teuthology.orchestra.run.smithi189.stdout:(28/156): ceph-grafana-dashboards-19.0.0-3449.g 820 kB/s | 34 kB 00:00 2024-05-10T10:55:24.595 INFO:teuthology.orchestra.run.smithi076.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-05-10T10:55:24.615 INFO:teuthology.orchestra.run.smithi189.stdout:(29/156): ceph-mgr-cephadm-19.0.0-3449.gd39fa08 2.8 MB/s | 142 kB 00:00 2024-05-10T10:55:24.820 INFO:teuthology.orchestra.run.smithi076.stdout:(73/156): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 6.0 MB 00:00 2024-05-10T10:55:24.854 INFO:teuthology.orchestra.run.smithi076.stdout:(74/156): protobuf-3.14.0-13.el9.x86_64.rpm 2.9 MB/s | 1.0 MB 00:00 2024-05-10T10:55:24.879 INFO:teuthology.orchestra.run.smithi076.stdout:(75/156): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-05-10T10:55:24.921 INFO:teuthology.orchestra.run.smithi076.stdout:(76/156): python3-jsonpatch-1.21-16.el9.noarch. 615 kB/s | 26 kB 00:00 2024-05-10T10:55:24.946 INFO:teuthology.orchestra.run.smithi076.stdout:(77/156): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-05-10T10:55:24.949 INFO:teuthology.orchestra.run.smithi189.stdout:(30/156): ceph-mgr-dashboard-19.0.0-3449.gd39fa 11 MB/s | 3.5 MB 00:00 2024-05-10T10:55:25.031 INFO:teuthology.orchestra.run.smithi076.stdout:(78/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-05-10T10:55:25.105 INFO:teuthology.orchestra.run.smithi076.stdout:(79/156): postgresql-server-13.14-1.el9.x86_64. 9.3 MB/s | 5.8 MB 00:00 2024-05-10T10:55:25.131 INFO:teuthology.orchestra.run.smithi076.stdout:(80/156): python3-markupsafe-1.1.1-12.el9.x86_6 346 kB/s | 35 kB 00:00 2024-05-10T10:55:25.164 INFO:teuthology.orchestra.run.smithi076.stdout:(81/156): python3-jsonpointer-2.0-4.el9.noarch. 79 kB/s | 19 kB 00:00 2024-05-10T10:55:25.190 INFO:teuthology.orchestra.run.smithi076.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-05-10T10:55:25.215 INFO:teuthology.orchestra.run.smithi076.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-05-10T10:55:25.240 INFO:teuthology.orchestra.run.smithi076.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-10T10:55:25.266 INFO:teuthology.orchestra.run.smithi076.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 839 kB/s | 42 kB 00:00 2024-05-10T10:55:25.350 INFO:teuthology.orchestra.run.smithi076.stdout:(86/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 5.1 MB 00:00 2024-05-10T10:55:25.375 INFO:teuthology.orchestra.run.smithi076.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-05-10T10:55:25.401 INFO:teuthology.orchestra.run.smithi076.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 1.6 MB/s | 269 kB 00:00 2024-05-10T10:55:25.426 INFO:teuthology.orchestra.run.smithi076.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-10T10:55:25.451 INFO:teuthology.orchestra.run.smithi076.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-05-10T10:55:25.485 INFO:teuthology.orchestra.run.smithi076.stdout:(91/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.0 MB/s | 279 kB 00:00 2024-05-10T10:55:25.510 INFO:teuthology.orchestra.run.smithi076.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 709 kB/s | 42 kB 00:00 2024-05-10T10:55:25.535 INFO:teuthology.orchestra.run.smithi076.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 877 kB/s | 44 kB 00:00 2024-05-10T10:55:25.561 INFO:teuthology.orchestra.run.smithi076.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-10T10:55:25.586 INFO:teuthology.orchestra.run.smithi076.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 713 kB/s | 36 kB 00:00 2024-05-10T10:55:25.599 INFO:teuthology.orchestra.run.smithi189.stdout:(31/156): ceph-radosgw-19.0.0-3449.gd39fa088.el 2.5 MB/s | 11 MB 00:04 2024-05-10T10:55:25.611 INFO:teuthology.orchestra.run.smithi076.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-05-10T10:55:25.691 INFO:teuthology.orchestra.run.smithi189.stdout:(32/156): ceph-mgr-diskprediction-local-19.0.0- 10 MB/s | 7.4 MB 00:00 2024-05-10T10:55:25.704 INFO:teuthology.orchestra.run.smithi076.stdout:(97/156): socat-1.7.4.1-5.el9.x86_64.rpm 2.5 MB/s | 305 kB 00:00 2024-05-10T10:55:25.729 INFO:teuthology.orchestra.run.smithi076.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 494 kB/s | 58 kB 00:00 2024-05-10T10:55:25.741 INFO:teuthology.orchestra.run.smithi189.stdout:(33/156): ceph-mgr-rook-19.0.0-3449.gd39fa088.e 990 kB/s | 49 kB 00:00 2024-05-10T10:55:25.754 INFO:teuthology.orchestra.run.smithi076.stdout:(99/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-05-10T10:55:25.767 INFO:teuthology.orchestra.run.smithi189.stdout:(34/156): ceph-mgr-modules-core-19.0.0-3449.gd3 1.6 MB/s | 274 kB 00:00 2024-05-10T10:55:25.780 INFO:teuthology.orchestra.run.smithi076.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-10T10:55:25.784 INFO:teuthology.orchestra.run.smithi189.stdout:(35/156): ceph-prometheus-alerts-19.0.0-3449.gd 391 kB/s | 16 kB 00:00 2024-05-10T10:55:25.805 INFO:teuthology.orchestra.run.smithi076.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 887 kB/s | 45 kB 00:00 2024-05-10T10:55:25.830 INFO:teuthology.orchestra.run.smithi076.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 443 kB/s | 22 kB 00:00 2024-05-10T10:55:25.859 INFO:teuthology.orchestra.run.smithi189.stdout:(36/156): cephadm-19.0.0-3449.gd39fa088.el9.noa 10 MB/s | 763 kB 00:00 2024-05-10T10:55:25.884 INFO:teuthology.orchestra.run.smithi189.stdout:(37/156): ceph-volume-19.0.0-3449.gd39fa088.el9 2.3 MB/s | 273 kB 00:00 2024-05-10T10:55:25.901 INFO:teuthology.orchestra.run.smithi189.stdout:(38/156): cephfs-top-19.0.0-3449.gd39fa088.el9. 437 kB/s | 18 kB 00:00 2024-05-10T10:55:25.968 INFO:teuthology.orchestra.run.smithi189.stdout:(39/156): fuse-2.9.9-15.el9.x86_64.rpm 962 kB/s | 80 kB 00:00 2024-05-10T10:55:26.019 INFO:teuthology.orchestra.run.smithi189.stdout:(40/156): libgfortran-11.4.1-3.el9.x86_64.rpm 6.7 MB/s | 803 kB 00:00 2024-05-10T10:55:26.061 INFO:teuthology.orchestra.run.smithi189.stdout:(41/156): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-05-10T10:55:26.081 INFO:teuthology.orchestra.run.smithi076.stdout:(103/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-05-10T10:55:26.111 INFO:teuthology.orchestra.run.smithi189.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-05-10T10:55:26.122 INFO:teuthology.orchestra.run.smithi076.stdout:(104/156): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-05-10T10:55:26.186 INFO:teuthology.orchestra.run.smithi189.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-05-10T10:55:26.228 INFO:teuthology.orchestra.run.smithi189.stdout:(44/156): python3-cryptography-36.0.1-4.el9.x86 7.3 MB/s | 1.2 MB 00:00 2024-05-10T10:55:26.287 INFO:teuthology.orchestra.run.smithi189.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 135 kB 00:00 2024-05-10T10:55:26.320 INFO:teuthology.orchestra.run.smithi189.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-05-10T10:55:26.398 INFO:teuthology.orchestra.run.smithi076.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-05-10T10:55:26.623 INFO:teuthology.orchestra.run.smithi076.stdout:(106/156): abseil-cpp-20211102.0-3.el9.x86_64.r 691 kB/s | 548 kB 00:00 2024-05-10T10:55:26.648 INFO:teuthology.orchestra.run.smithi076.stdout:(107/156): gperftools-libs-2.9.1-2.el9.x86_64.r 544 kB/s | 309 kB 00:00 2024-05-10T10:55:26.697 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-10T10:55:26.699 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-10T10:55:26.721 INFO:teuthology.orchestra.run.smithi027.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-10T10:55:26.722 INFO:teuthology.orchestra.run.smithi027.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-10T10:55:26.724 INFO:teuthology.orchestra.run.smithi027.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-10T10:55:26.725 INFO:teuthology.orchestra.run.smithi027.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-10T10:55:26.728 INFO:teuthology.orchestra.run.smithi027.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-10T10:55:26.731 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-10T10:55:26.733 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-10T10:55:26.766 INFO:teuthology.orchestra.run.smithi076.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-05-10T10:55:26.787 INFO:teuthology.orchestra.run.smithi189.stdout:(47/156): ceph-test-19.0.0-3449.gd39fa088.el9.x 9.8 MB/s | 49 MB 00:05 2024-05-10T10:55:26.791 INFO:teuthology.orchestra.run.smithi076.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 341 kB/s | 49 kB 00:00 2024-05-10T10:55:26.811 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-10T10:55:26.813 INFO:teuthology.orchestra.run.smithi189.stdout:(48/156): python3-urllib3-1.26.5-5.el9.noarch.r 408 kB/s | 215 kB 00:00 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 6.6 k 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 5.4 M 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 890 k 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 145 k 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 1.5 M 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 142 k 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 3.5 M 2024-05-10T10:55:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 7.4 M 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 49 k 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 11 M 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 49 M 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 273 k 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 763 k 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 225 k 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-shell x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 27 k 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 18 k 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 32 k 2024-05-10T10:55:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 746 k 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 127 k 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 161 k 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 321 k 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 298 k 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 99 k 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 85 k 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 3.1 M 2024-05-10T10:55:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 170 k 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 3.4 M 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 3.1 M 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-10T10:55:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 21 M 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 34 k 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 2.4 M 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 274 k 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 4.7 M 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 17 M 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3449.gd39fa088.el9 ceph-noarch 16 k 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 25 k 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-10T10:55:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 163 k 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 500 k 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 5.4 M 2024-05-10T10:55:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-10T10:55:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 45 k 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:19.0.0-3449.gd39fa088.el9 ceph 135 k 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-10T10:55:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-10T10:55:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-10T10:55:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-10T10:55:26.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:55:26.842 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-10T10:55:26.843 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-05-10T10:55:26.843 INFO:teuthology.orchestra.run.smithi027.stdout:Install 148 Packages 2024-05-10T10:55:26.843 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 8 Packages 2024-05-10T10:55:26.843 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:55:26.843 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 226 M 2024-05-10T10:55:26.843 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-10T10:55:26.916 INFO:teuthology.orchestra.run.smithi076.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 448 kB/s | 67 kB 00:00 2024-05-10T10:55:26.941 INFO:teuthology.orchestra.run.smithi076.stdout:(111/156): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-05-10T10:55:27.101 INFO:teuthology.orchestra.run.smithi076.stdout:(112/156): libarrow-9.0.0-11.el9.x86_64.rpm 6.3 MB/s | 4.4 MB 00:00 2024-05-10T10:55:27.151 INFO:teuthology.orchestra.run.smithi076.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.5 MB/s | 839 kB 00:00 2024-05-10T10:55:27.193 INFO:teuthology.orchestra.run.smithi076.stdout:(114/156): python3-asyncssh-2.13.2-5.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-05-10T10:55:27.235 INFO:teuthology.orchestra.run.smithi076.stdout:(115/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 324 kB/s | 43 kB 00:00 2024-05-10T10:55:27.286 INFO:teuthology.orchestra.run.smithi076.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 240 kB/s | 32 kB 00:00 2024-05-10T10:55:27.340 INFO:teuthology.orchestra.run.smithi076.stdout:(117/156): python3-certifi-2021.10.8-2.el9.noar 102 kB/s | 15 kB 00:00 2024-05-10T10:55:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 822 kB/s | 172 kB 00:00 2024-05-10T10:55:27.469 INFO:teuthology.orchestra.run.smithi076.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 1.9 MB/s | 358 kB 00:00 2024-05-10T10:55:27.503 INFO:teuthology.orchestra.run.smithi076.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.8 MB/s | 303 kB 00:00 2024-05-10T10:55:27.562 INFO:teuthology.orchestra.run.smithi076.stdout:(121/156): python3-colorama-0.4.4-12.el9.noarch 301 kB/s | 35 kB 00:00 2024-05-10T10:55:27.563 INFO:teuthology.orchestra.run.smithi189.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 142 kB/s | 107 kB 00:00 2024-05-10T10:55:27.604 INFO:teuthology.orchestra.run.smithi076.stdout:(122/156): python3-google-auth-2.29.0-1.el9.noa 1.5 MB/s | 206 kB 00:00 2024-05-10T10:55:27.647 INFO:teuthology.orchestra.run.smithi189.stdout:(50/156): automake-1.16.2-8.el9.noarch.rpm 520 kB/s | 689 kB 00:01 2024-05-10T10:55:27.704 INFO:teuthology.orchestra.run.smithi076.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-05-10T10:55:27.730 INFO:teuthology.orchestra.run.smithi076.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 85 kB/s | 11 kB 00:00 2024-05-10T10:55:27.771 INFO:teuthology.orchestra.run.smithi076.stdout:(125/156): python3-grpcio-1.46.7-10.el9.x86_64. 7.6 MB/s | 2.0 MB 00:00 2024-05-10T10:55:27.772 INFO:teuthology.orchestra.run.smithi189.stdout:(51/156): bison-3.7.4-5.el9.x86_64.rpm 958 kB/s | 943 kB 00:00 2024-05-10T10:55:27.822 INFO:teuthology.orchestra.run.smithi076.stdout:(126/156): python3-jaraco-classes-3.2.1-5.el9.n 150 kB/s | 18 kB 00:00 2024-05-10T10:55:27.847 INFO:teuthology.orchestra.run.smithi076.stdout:(127/156): python3-jaraco-collections-3.0.0-8.e 197 kB/s | 23 kB 00:00 2024-05-10T10:55:27.881 INFO:teuthology.orchestra.run.smithi076.stdout:(128/156): python3-jaraco-functools-3.5.0-2.el9 178 kB/s | 19 kB 00:00 2024-05-10T10:55:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:(1/156): ceph-19.0.0-3449.gd39fa088.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-05-10T10:55:27.939 INFO:teuthology.orchestra.run.smithi076.stdout:(129/156): python3-jaraco-text-3.2.0-6.el9.noar 168 kB/s | 20 kB 00:00 2024-05-10T10:55:27.981 INFO:teuthology.orchestra.run.smithi076.stdout:(130/156): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-05-10T10:55:28.015 INFO:teuthology.orchestra.run.smithi076.stdout:(131/156): python3-logutils-0.3.5-21.el9.noarch 345 kB/s | 46 kB 00:00 2024-05-10T10:55:28.065 INFO:teuthology.orchestra.run.smithi076.stdout:(132/156): python3-more-itertools-8.12.0-2.el9. 627 kB/s | 79 kB 00:00 2024-05-10T10:55:28.108 INFO:teuthology.orchestra.run.smithi076.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 458 kB/s | 58 kB 00:00 2024-05-10T10:55:28.147 INFO:teuthology.orchestra.run.smithi189.stdout:(52/156): elfutils-libelf-devel-0.190-2.el9.x86 37 kB/s | 21 kB 00:00 2024-05-10T10:55:28.166 INFO:teuthology.orchestra.run.smithi076.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-05-10T10:55:28.178 INFO:teuthology.orchestra.run.smithi027.stdout:(2/156): ceph-fuse-19.0.0-3449.gd39fa088.el9.x8 3.6 MB/s | 890 kB 00:00 2024-05-10T10:55:28.191 INFO:teuthology.orchestra.run.smithi076.stdout:(135/156): python3-portend-3.1.0-2.el9.noarch.r 130 kB/s | 16 kB 00:00 2024-05-10T10:55:28.225 INFO:teuthology.orchestra.run.smithi076.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 766 kB/s | 90 kB 00:00 2024-05-10T10:55:28.253 INFO:teuthology.orchestra.run.smithi027.stdout:(3/156): ceph-immutable-object-cache-19.0.0-344 1.9 MB/s | 145 kB 00:00 2024-05-10T10:55:28.284 INFO:teuthology.orchestra.run.smithi076.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 240 kB/s | 28 kB 00:00 2024-05-10T10:55:28.318 INFO:teuthology.orchestra.run.smithi076.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 244 kB/s | 31 kB 00:00 2024-05-10T10:55:28.351 INFO:teuthology.orchestra.run.smithi076.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-05-10T10:55:28.372 INFO:teuthology.orchestra.run.smithi189.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-05-10T10:55:28.393 INFO:teuthology.orchestra.run.smithi076.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 541 kB/s | 59 kB 00:00 2024-05-10T10:55:28.461 INFO:teuthology.orchestra.run.smithi076.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 251 kB/s | 36 kB 00:00 2024-05-10T10:55:28.486 INFO:teuthology.orchestra.run.smithi076.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 380 kB/s | 51 kB 00:00 2024-05-10T10:55:28.519 INFO:teuthology.orchestra.run.smithi076.stdout:(143/156): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-05-10T10:55:28.522 INFO:teuthology.orchestra.run.smithi189.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 358 kB/s | 313 kB 00:00 2024-05-10T10:55:28.576 INFO:teuthology.orchestra.run.smithi027.stdout:(4/156): ceph-base-19.0.0-3449.gd39fa088.el9.x8 6.0 MB/s | 5.4 MB 00:00 2024-05-10T10:55:28.586 INFO:teuthology.orchestra.run.smithi076.stdout:(144/156): python3-websocket-client-1.2.3-2.el9 714 kB/s | 90 kB 00:00 2024-05-10T10:55:28.612 INFO:teuthology.orchestra.run.smithi076.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-05-10T10:55:28.637 INFO:teuthology.orchestra.run.smithi076.stdout:(146/156): python3-zc-lockfile-2.0-10.el9.noarc 170 kB/s | 20 kB 00:00 2024-05-10T10:55:28.671 INFO:teuthology.orchestra.run.smithi027.stdout:(5/156): ceph-mds-19.0.0-3449.gd39fa088.el9.x86 5.7 MB/s | 2.4 MB 00:00 2024-05-10T10:55:28.721 INFO:teuthology.orchestra.run.smithi076.stdout:(147/156): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-05-10T10:55:28.754 INFO:teuthology.orchestra.run.smithi027.stdout:(6/156): ceph-mgr-19.0.0-3449.gd39fa088.el9.x86 8.5 MB/s | 1.5 MB 00:00 2024-05-10T10:55:28.763 INFO:teuthology.orchestra.run.smithi076.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-05-10T10:55:28.956 INFO:teuthology.orchestra.run.smithi189.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 33 kB/s | 18 kB 00:00 2024-05-10T10:55:29.073 INFO:teuthology.orchestra.run.smithi189.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 20 kB/s | 11 kB 00:00 2024-05-10T10:55:29.114 INFO:teuthology.orchestra.run.smithi076.stdout:(149/156): NetworkManager-1.47.5-1.el9.x86_64.r 6.6 MB/s | 2.3 MB 00:00 2024-05-10T10:55:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:(150/156): NetworkManager-libnm-1.47.5-1.el9.x8 8.4 MB/s | 1.8 MB 00:00 2024-05-10T10:55:29.332 INFO:teuthology.orchestra.run.smithi189.stdout:(57/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.5 MB/s | 3.0 MB 00:01 2024-05-10T10:55:29.372 INFO:teuthology.orchestra.run.smithi076.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-05-10T10:55:29.431 INFO:teuthology.orchestra.run.smithi076.stdout:(152/156): NetworkManager-tui-1.47.5-1.el9.x86_ 4.2 MB/s | 248 kB 00:00 2024-05-10T10:55:29.590 INFO:teuthology.orchestra.run.smithi076.stdout:(153/156): openssl-3.2.1-1.el9.x86_64.rpm 8.5 MB/s | 1.3 MB 00:00 2024-05-10T10:55:29.613 INFO:teuthology.orchestra.run.smithi027.stdout:(7/156): ceph-mon-19.0.0-3449.gd39fa088.el9.x86 5.0 MB/s | 4.7 MB 00:00 2024-05-10T10:55:29.741 INFO:teuthology.orchestra.run.smithi189.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 68 kB/s | 45 kB 00:00 2024-05-10T10:55:29.783 INFO:teuthology.orchestra.run.smithi189.stdout:(59/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 194 kB/s | 160 kB 00:00 2024-05-10T10:55:29.815 INFO:teuthology.orchestra.run.smithi076.stdout:(154/156): librbd1-19.0.0-3449.gd39fa088.el9.x8 2.9 MB/s | 3.1 MB 00:01 2024-05-10T10:55:29.857 INFO:teuthology.orchestra.run.smithi076.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 9.0 MB/s | 2.4 MB 00:00 2024-05-10T10:55:29.872 INFO:teuthology.orchestra.run.smithi027.stdout:(8/156): ceph-common-19.0.0-3449.gd39fa088.el9. 9.6 MB/s | 21 MB 00:02 2024-05-10T10:55:29.939 INFO:teuthology.orchestra.run.smithi027.stdout:(9/156): ceph-selinux-19.0.0-3449.gd39fa088.el9 384 kB/s | 25 kB 00:00 2024-05-10T10:55:30.283 INFO:teuthology.orchestra.run.smithi189.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 718 kB/s | 662 kB 00:00 2024-05-10T10:55:30.358 INFO:teuthology.orchestra.run.smithi189.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 44 kB/s | 25 kB 00:00 2024-05-10T10:55:30.666 INFO:teuthology.orchestra.run.smithi189.stdout:(62/156): libtool-2.4.6-45.el9.x86_64.rpm 629 kB/s | 582 kB 00:00 2024-05-10T10:55:30.730 INFO:teuthology.orchestra.run.smithi027.stdout:(10/156): ceph-osd-19.0.0-3449.gd39fa088.el9.x8 8.5 MB/s | 17 MB 00:01 2024-05-10T10:55:30.814 INFO:teuthology.orchestra.run.smithi027.stdout:(11/156): cephfs-mirror-19.0.0-3449.gd39fa088.e 2.7 MB/s | 225 kB 00:00 2024-05-10T10:55:30.881 INFO:teuthology.orchestra.run.smithi027.stdout:(12/156): cephfs-shell-19.0.0-3449.gd39fa088.el 412 kB/s | 27 kB 00:00 2024-05-10T10:55:30.942 INFO:teuthology.orchestra.run.smithi189.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 72 kB/s | 47 kB 00:00 2024-05-10T10:55:30.948 INFO:teuthology.orchestra.run.smithi027.stdout:(13/156): libcephfs-devel-19.0.0-3449.gd39fa088 482 kB/s | 32 kB 00:00 2024-05-10T10:55:31.090 INFO:teuthology.orchestra.run.smithi027.stdout:(14/156): libcephfs2-19.0.0-3449.gd39fa088.el9. 5.2 MB/s | 746 kB 00:00 2024-05-10T10:55:31.217 INFO:teuthology.orchestra.run.smithi189.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 341 kB/s | 292 kB 00:00 2024-05-10T10:55:31.265 INFO:teuthology.orchestra.run.smithi027.stdout:(15/156): ceph-radosgw-19.0.0-3449.gd39fa088.el 6.4 MB/s | 11 MB 00:01 2024-05-10T10:55:31.299 INFO:teuthology.orchestra.run.smithi027.stdout:(16/156): libcephsqlite-19.0.0-3449.gd39fa088.e 788 kB/s | 163 kB 00:00 2024-05-10T10:55:31.332 INFO:teuthology.orchestra.run.smithi027.stdout:(17/156): librados-devel-19.0.0-3449.gd39fa088. 1.9 MB/s | 127 kB 00:00 2024-05-10T10:55:31.383 INFO:teuthology.orchestra.run.smithi027.stdout:(18/156): libradosstriper1-19.0.0-3449.gd39fa08 5.9 MB/s | 500 kB 00:00 2024-05-10T10:55:31.434 INFO:teuthology.orchestra.run.smithi189.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 246 kB/s | 188 kB 00:00 2024-05-10T10:55:31.467 INFO:teuthology.orchestra.run.smithi027.stdout:(19/156): python3-ceph-argparse-19.0.0-3449.gd3 545 kB/s | 45 kB 00:00 2024-05-10T10:55:31.542 INFO:teuthology.orchestra.run.smithi027.stdout:(20/156): python3-ceph-common-19.0.0-3449.gd39f 1.8 MB/s | 135 kB 00:00 2024-05-10T10:55:31.551 INFO:teuthology.orchestra.run.smithi189.stdout:(66/156): openblas-0.3.21-2.el9.x86_64.rpm 58 kB/s | 35 kB 00:00 2024-05-10T10:55:31.617 INFO:teuthology.orchestra.run.smithi027.stdout:(21/156): python3-cephfs-19.0.0-3449.gd39fa088. 2.1 MB/s | 161 kB 00:00 2024-05-10T10:55:31.692 INFO:teuthology.orchestra.run.smithi027.stdout:(22/156): python3-rados-19.0.0-3449.gd39fa088.e 4.2 MB/s | 321 kB 00:00 2024-05-10T10:55:31.767 INFO:teuthology.orchestra.run.smithi027.stdout:(23/156): python3-rbd-19.0.0-3449.gd39fa088.el9 3.9 MB/s | 298 kB 00:00 2024-05-10T10:55:31.834 INFO:teuthology.orchestra.run.smithi027.stdout:(24/156): python3-rgw-19.0.0-3449.gd39fa088.el9 1.5 MB/s | 99 kB 00:00 2024-05-10T10:55:31.901 INFO:teuthology.orchestra.run.smithi027.stdout:(25/156): rbd-fuse-19.0.0-3449.gd39fa088.el9.x8 1.3 MB/s | 85 kB 00:00 2024-05-10T10:55:32.182 INFO:teuthology.orchestra.run.smithi076.stdout:(156/156): librados2-19.0.0-3449.gd39fa088.el9. 981 kB/s | 3.4 MB 00:03 2024-05-10T10:55:32.184 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-05-10T10:55:32.184 INFO:teuthology.orchestra.run.smithi076.stdout:Total 13 MB/s | 226 MB 00:17 2024-05-10T10:55:32.194 INFO:teuthology.orchestra.run.smithi027.stdout:(26/156): librgw2-19.0.0-3449.gd39fa088.el9.x86 6.2 MB/s | 5.4 MB 00:00 2024-05-10T10:55:32.310 INFO:teuthology.orchestra.run.smithi027.stdout:(27/156): rbd-mirror-19.0.0-3449.gd39fa088.el9. 7.6 MB/s | 3.1 MB 00:00 2024-05-10T10:55:32.344 INFO:teuthology.orchestra.run.smithi027.stdout:(28/156): rbd-nbd-19.0.0-3449.gd39fa088.el9.x86 1.1 MB/s | 170 kB 00:00 2024-05-10T10:55:32.369 INFO:teuthology.orchestra.run.smithi027.stdout:(29/156): ceph-grafana-dashboards-19.0.0-3449.g 592 kB/s | 34 kB 00:00 2024-05-10T10:55:32.401 INFO:teuthology.orchestra.run.smithi189.stdout:(67/156): openblas-openmp-0.3.21-2.el9.x86_64.r 4.0 MB/s | 4.7 MB 00:01 2024-05-10T10:55:32.428 INFO:teuthology.orchestra.run.smithi027.stdout:(30/156): ceph-mgr-cephadm-19.0.0-3449.gd39fa08 1.7 MB/s | 142 kB 00:00 2024-05-10T10:55:32.585 INFO:teuthology.orchestra.run.smithi189.stdout:(68/156): postgresql-13.14-1.el9.x86_64.rpm 1.5 MB/s | 1.6 MB 00:01 2024-05-10T10:55:32.693 INFO:teuthology.orchestra.run.smithi189.stdout:(69/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.5 MB/s | 4.4 MB 00:01 2024-05-10T10:55:32.813 INFO:teuthology.orchestra.run.smithi027.stdout:(31/156): ceph-mgr-dashboard-19.0.0-3449.gd39fa 8.0 MB/s | 3.5 MB 00:00 2024-05-10T10:55:32.879 INFO:teuthology.orchestra.run.smithi027.stdout:(32/156): ceph-mgr-modules-core-19.0.0-3449.gd3 4.1 MB/s | 274 kB 00:00 2024-05-10T10:55:32.946 INFO:teuthology.orchestra.run.smithi027.stdout:(33/156): ceph-mgr-rook-19.0.0-3449.gd39fa088.e 751 kB/s | 49 kB 00:00 2024-05-10T10:55:33.014 INFO:teuthology.orchestra.run.smithi027.stdout:(34/156): ceph-prometheus-alerts-19.0.0-3449.gd 250 kB/s | 16 kB 00:00 2024-05-10T10:55:33.180 INFO:teuthology.orchestra.run.smithi027.stdout:(35/156): ceph-volume-19.0.0-3449.gd39fa088.el9 1.6 MB/s | 273 kB 00:00 2024-05-10T10:55:33.357 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-05-10T10:55:33.410 INFO:teuthology.orchestra.run.smithi189.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 161 kB/s | 133 kB 00:00 2024-05-10T10:55:33.452 INFO:teuthology.orchestra.run.smithi189.stdout:(71/156): postgresql-contrib-13.14-1.el9.x86_64 868 kB/s | 911 kB 00:01 2024-05-10T10:55:33.509 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-05-10T10:55:33.509 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-05-10T10:55:33.755 INFO:teuthology.orchestra.run.smithi027.stdout:(36/156): ceph-test-19.0.0-3449.gd39fa088.el9.x 13 MB/s | 49 MB 00:03 2024-05-10T10:55:33.805 INFO:teuthology.orchestra.run.smithi027.stdout:(37/156): cephadm-19.0.0-3449.gd39fa088.el9.noa 1.2 MB/s | 763 kB 00:00 2024-05-10T10:55:33.833 INFO:teuthology.orchestra.run.smithi027.stdout:(38/156): cephfs-top-19.0.0-3449.gd39fa088.el9. 246 kB/s | 18 kB 00:00 2024-05-10T10:55:33.961 INFO:teuthology.orchestra.run.smithi189.stdout:(72/156): postgresql-server-13.14-1.el9.x86_64. 4.7 MB/s | 5.8 MB 00:01 2024-05-10T10:55:33.972 INFO:teuthology.orchestra.run.smithi027.stdout:(39/156): ceph-mgr-diskprediction-local-19.0.0- 4.8 MB/s | 7.4 MB 00:01 2024-05-10T10:55:33.998 INFO:teuthology.orchestra.run.smithi027.stdout:(40/156): fuse-2.9.9-15.el9.x86_64.rpm 420 kB/s | 80 kB 00:00 2024-05-10T10:55:34.090 INFO:teuthology.orchestra.run.smithi027.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-05-10T10:55:34.140 INFO:teuthology.orchestra.run.smithi027.stdout:(42/156): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-05-10T10:55:34.152 INFO:teuthology.orchestra.run.smithi189.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 125 kB/s | 87 kB 00:00 2024-05-10T10:55:34.165 INFO:teuthology.orchestra.run.smithi027.stdout:(43/156): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-05-10T10:55:34.199 INFO:teuthology.orchestra.run.smithi027.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-05-10T10:55:34.233 INFO:teuthology.orchestra.run.smithi027.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-05-10T10:55:34.266 INFO:teuthology.orchestra.run.smithi027.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-05-10T10:55:34.300 INFO:teuthology.orchestra.run.smithi027.stdout:(47/156): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-05-10T10:55:34.384 INFO:teuthology.orchestra.run.smithi027.stdout:(48/156): python3-cryptography-36.0.1-4.el9.x86 4.2 MB/s | 1.2 MB 00:00 2024-05-10T10:55:34.428 INFO:teuthology.orchestra.run.smithi189.stdout:(74/156): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:01 2024-05-10T10:55:34.517 INFO:teuthology.orchestra.run.smithi027.stdout:(49/156): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 689 kB 00:00 2024-05-10T10:55:34.542 INFO:teuthology.orchestra.run.smithi027.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 678 kB/s | 107 kB 00:00 2024-05-10T10:55:34.576 INFO:teuthology.orchestra.run.smithi027.stdout:(51/156): elfutils-libelf-devel-0.190-2.el9.x86 373 kB/s | 21 kB 00:00 2024-05-10T10:55:34.626 INFO:teuthology.orchestra.run.smithi027.stdout:(52/156): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 943 kB 00:00 2024-05-10T10:55:34.693 INFO:teuthology.orchestra.run.smithi027.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 2.0 MB/s | 313 kB 00:00 2024-05-10T10:55:34.735 INFO:teuthology.orchestra.run.smithi027.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 445 kB/s | 18 kB 00:00 2024-05-10T10:55:34.760 INFO:teuthology.orchestra.run.smithi027.stdout:(55/156): flexiblas-3.0.4-8.el9.x86_64.rpm 178 kB/s | 33 kB 00:00 2024-05-10T10:55:34.786 INFO:teuthology.orchestra.run.smithi027.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 225 kB/s | 11 kB 00:00 2024-05-10T10:55:34.878 INFO:teuthology.orchestra.run.smithi027.stdout:(57/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-05-10T10:55:34.903 INFO:teuthology.orchestra.run.smithi027.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 389 kB/s | 45 kB 00:00 2024-05-10T10:55:34.929 INFO:teuthology.orchestra.run.smithi027.stdout:(59/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 958 kB/s | 160 kB 00:00 2024-05-10T10:55:34.953 INFO:teuthology.orchestra.run.smithi189.stdout:(75/156): python3-devel-3.9.18-3.el9.x86_64.rpm 305 kB/s | 244 kB 00:00 2024-05-10T10:55:34.979 INFO:teuthology.orchestra.run.smithi027.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-05-10T10:55:35.004 INFO:teuthology.orchestra.run.smithi027.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 342 kB/s | 25 kB 00:00 2024-05-10T10:55:35.030 INFO:teuthology.orchestra.run.smithi027.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 960 kB/s | 47 kB 00:00 2024-05-10T10:55:35.063 INFO:teuthology.orchestra.run.smithi027.stdout:(63/156): libtool-2.4.6-45.el9.x86_64.rpm 3.6 MB/s | 582 kB 00:00 2024-05-10T10:55:35.097 INFO:teuthology.orchestra.run.smithi027.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-05-10T10:55:35.122 INFO:teuthology.orchestra.run.smithi027.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-05-10T10:55:35.147 INFO:teuthology.orchestra.run.smithi027.stdout:(66/156): openblas-0.3.21-2.el9.x86_64.rpm 425 kB/s | 35 kB 00:00 2024-05-10T10:55:35.182 INFO:teuthology.orchestra.run.smithi189.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 4.9 MB/s | 6.0 MB 00:01 2024-05-10T10:55:35.256 INFO:teuthology.orchestra.run.smithi027.stdout:(67/156): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-05-10T10:55:35.279 INFO:teuthology.orchestra.run.smithi189.stdout:(77/156): python3-jinja2-2.11.3-5.el9.noarch.rp 291 kB/s | 248 kB 00:00 2024-05-10T10:55:35.373 INFO:teuthology.orchestra.run.smithi027.stdout:(68/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-05-10T10:55:35.481 INFO:teuthology.orchestra.run.smithi027.stdout:(69/156): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-05-10T10:55:35.515 INFO:teuthology.orchestra.run.smithi027.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 947 kB/s | 133 kB 00:00 2024-05-10T10:55:35.557 INFO:teuthology.orchestra.run.smithi027.stdout:(71/156): postgresql-contrib-13.14-1.el9.x86_64 3.0 MB/s | 911 kB 00:00 2024-05-10T10:55:35.575 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-05-10T10:55:35.575 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-05-10T10:55:35.624 INFO:teuthology.orchestra.run.smithi027.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 9.4 MB/s | 1.0 MB 00:00 2024-05-10T10:55:35.649 INFO:teuthology.orchestra.run.smithi027.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 961 kB/s | 87 kB 00:00 2024-05-10T10:55:35.679 INFO:teuthology.orchestra.run.smithi189.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 35 kB/s | 26 kB 00:00 2024-05-10T10:55:35.716 INFO:teuthology.orchestra.run.smithi027.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-05-10T10:55:35.754 INFO:teuthology.orchestra.run.smithi189.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 33 kB/s | 19 kB 00:00 2024-05-10T10:55:35.806 INFO:teuthology.orchestra.run.smithi027.stdout:(75/156): python3-jinja2-2.11.3-5.el9.noarch.rp 2.7 MB/s | 248 kB 00:00 2024-05-10T10:55:35.858 INFO:teuthology.orchestra.run.smithi027.stdout:(76/156): python3-jsonpatch-1.21-16.el9.noarch. 496 kB/s | 26 kB 00:00 2024-05-10T10:55:35.975 INFO:teuthology.orchestra.run.smithi027.stdout:(77/156): postgresql-server-13.14-1.el9.x86_64. 12 MB/s | 5.8 MB 00:00 2024-05-10T10:55:36.001 INFO:teuthology.orchestra.run.smithi027.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 136 kB/s | 19 kB 00:00 2024-05-10T10:55:36.034 INFO:teuthology.orchestra.run.smithi027.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-05-10T10:55:36.059 INFO:teuthology.orchestra.run.smithi027.stdout:(80/156): python3-markupsafe-1.1.1-12.el9.x86_6 602 kB/s | 35 kB 00:00 2024-05-10T10:55:36.118 INFO:teuthology.orchestra.run.smithi027.stdout:(81/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.3 MB/s | 250 kB 00:00 2024-05-10T10:55:36.137 INFO:teuthology.orchestra.run.smithi189.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 207 kB/s | 172 kB 00:00 2024-05-10T10:55:36.227 INFO:teuthology.orchestra.run.smithi027.stdout:(82/156): python3-babel-2.9.1-2.el9.noarch.rpm 9.9 MB/s | 6.0 MB 00:00 2024-05-10T10:55:36.294 INFO:teuthology.orchestra.run.smithi027.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.3 MB/s | 222 kB 00:00 2024-05-10T10:55:36.305 INFO:teuthology.orchestra.run.smithi189.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 56 kB/s | 35 kB 00:00 2024-05-10T10:55:36.336 INFO:teuthology.orchestra.run.smithi027.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 716 kB/s | 77 kB 00:00 2024-05-10T10:55:36.353 INFO:teuthology.orchestra.run.smithi027.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 704 kB/s | 42 kB 00:00 2024-05-10T10:55:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 269 kB 00:00 2024-05-10T10:55:36.428 INFO:teuthology.orchestra.run.smithi027.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-05-10T10:55:36.462 INFO:teuthology.orchestra.run.smithi027.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-05-10T10:55:36.487 INFO:teuthology.orchestra.run.smithi027.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 884 kB/s | 51 kB 00:00 2024-05-10T10:55:36.513 INFO:teuthology.orchestra.run.smithi027.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-05-10T10:55:36.638 INFO:teuthology.orchestra.run.smithi027.stdout:(91/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.5 MB/s | 5.1 MB 00:00 2024-05-10T10:55:36.663 INFO:teuthology.orchestra.run.smithi027.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 278 kB/s | 42 kB 00:00 2024-05-10T10:55:36.688 INFO:teuthology.orchestra.run.smithi027.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 896 kB/s | 44 kB 00:00 2024-05-10T10:55:36.731 INFO:teuthology.orchestra.run.smithi027.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.3 MB/s | 221 kB 00:00 2024-05-10T10:55:36.789 INFO:teuthology.orchestra.run.smithi027.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 360 kB/s | 36 kB 00:00 2024-05-10T10:55:36.815 INFO:teuthology.orchestra.run.smithi027.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 573 kB/s | 47 kB 00:00 2024-05-10T10:55:36.848 INFO:teuthology.orchestra.run.smithi027.stdout:(97/156): socat-1.7.4.1-5.el9.x86_64.rpm 5.2 MB/s | 305 kB 00:00 2024-05-10T10:55:36.873 INFO:teuthology.orchestra.run.smithi027.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-05-10T10:55:36.907 INFO:teuthology.orchestra.run.smithi027.stdout:(99/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-05-10T10:55:36.932 INFO:teuthology.orchestra.run.smithi027.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-10T10:55:36.946 INFO:teuthology.orchestra.run.smithi189.stdout:(82/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.3 MB/s | 5.1 MB 00:01 2024-05-10T10:55:36.958 INFO:teuthology.orchestra.run.smithi027.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-05-10T10:55:36.971 INFO:teuthology.orchestra.run.smithi189.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 301 kB/s | 250 kB 00:00 2024-05-10T10:55:37.088 INFO:teuthology.orchestra.run.smithi189.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 284 kB/s | 222 kB 00:00 2024-05-10T10:55:37.333 INFO:teuthology.orchestra.run.smithi027.stdout:(102/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-05-10T10:55:37.358 INFO:teuthology.orchestra.run.smithi027.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 52 kB/s | 22 kB 00:00 2024-05-10T10:55:37.458 INFO:teuthology.orchestra.run.smithi027.stdout:(104/156): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-05-10T10:55:37.492 INFO:teuthology.orchestra.run.smithi027.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 3.4 MB/s | 548 kB 00:00 2024-05-10T10:55:37.517 INFO:teuthology.orchestra.run.smithi027.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 335 kB/s | 19 kB 00:00 2024-05-10T10:55:37.559 INFO:teuthology.orchestra.run.smithi027.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 637 kB/s | 26 kB 00:00 2024-05-10T10:55:37.589 INFO:teuthology.orchestra.run.smithi189.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 68 kB/s | 42 kB 00:00 2024-05-10T10:55:37.601 INFO:teuthology.orchestra.run.smithi027.stdout:(108/156): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2024-05-10T10:55:37.622 INFO:teuthology.orchestra.run.smithi189.stdout:(86/156): python3-packaging-20.9-5.el9.noarch.r 114 kB/s | 77 kB 00:00 2024-05-10T10:55:37.643 INFO:teuthology.orchestra.run.smithi027.stdout:(109/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-05-10T10:55:37.685 INFO:teuthology.orchestra.run.smithi027.stdout:(110/156): luarocks-3.9.2-1.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-05-10T10:55:37.819 INFO:teuthology.orchestra.run.smithi027.stdout:(111/156): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-05-10T10:55:37.869 INFO:teuthology.orchestra.run.smithi027.stdout:(112/156): protobuf-compiler-3.14.0-13.el9.x86_ 948 kB/s | 863 kB 00:00 2024-05-10T10:55:37.911 INFO:teuthology.orchestra.run.smithi027.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.6 MB/s | 839 kB 00:00 2024-05-10T10:55:37.923 INFO:teuthology.orchestra.run.smithi189.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 323 kB/s | 269 kB 00:00 2024-05-10T10:55:37.945 INFO:teuthology.orchestra.run.smithi027.stdout:(114/156): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-05-10T10:55:37.970 INFO:teuthology.orchestra.run.smithi027.stdout:(115/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 435 kB/s | 43 kB 00:00 2024-05-10T10:55:37.995 INFO:teuthology.orchestra.run.smithi027.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 386 kB/s | 32 kB 00:00 2024-05-10T10:55:38.021 INFO:teuthology.orchestra.run.smithi027.stdout:(117/156): python3-certifi-2021.10.8-2.el9.noar 200 kB/s | 15 kB 00:00 2024-05-10T10:55:38.054 INFO:teuthology.orchestra.run.smithi027.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-05-10T10:55:38.088 INFO:teuthology.orchestra.run.smithi027.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 3.9 MB/s | 358 kB 00:00 2024-05-10T10:55:38.113 INFO:teuthology.orchestra.run.smithi027.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 612 kB/s | 35 kB 00:00 2024-05-10T10:55:38.147 INFO:teuthology.orchestra.run.smithi027.stdout:(121/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-05-10T10:55:38.172 INFO:teuthology.orchestra.run.smithi027.stdout:(122/156): python3-google-auth-2.29.0-1.el9.noa 2.4 MB/s | 206 kB 00:00 2024-05-10T10:55:38.206 INFO:teuthology.orchestra.run.smithi027.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-05-10T10:55:38.231 INFO:teuthology.orchestra.run.smithi027.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 185 kB/s | 11 kB 00:00 2024-05-10T10:55:38.277 INFO:teuthology.orchestra.run.smithi027.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 251 kB/s | 18 kB 00:00 2024-05-10T10:55:38.298 INFO:teuthology.orchestra.run.smithi027.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 350 kB/s | 23 kB 00:00 2024-05-10T10:55:38.315 INFO:teuthology.orchestra.run.smithi027.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 527 kB/s | 19 kB 00:00 2024-05-10T10:55:38.332 INFO:teuthology.orchestra.run.smithi027.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 601 kB/s | 20 kB 00:00 2024-05-10T10:55:38.366 INFO:teuthology.orchestra.run.smithi027.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-05-10T10:55:38.381 INFO:teuthology.orchestra.run.smithi189.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 200 kB/s | 159 kB 00:00 2024-05-10T10:55:38.425 INFO:teuthology.orchestra.run.smithi027.stdout:(130/156): python3-kubernetes-26.1.0-2.el9.noar 9.5 MB/s | 1.0 MB 00:00 2024-05-10T10:55:38.440 INFO:teuthology.orchestra.run.smithi189.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 341 kB/s | 279 kB 00:00 2024-05-10T10:55:38.441 INFO:teuthology.orchestra.run.smithi027.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-05-10T10:55:38.467 INFO:teuthology.orchestra.run.smithi027.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-05-10T10:55:38.500 INFO:teuthology.orchestra.run.smithi027.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-05-10T10:55:38.526 INFO:teuthology.orchestra.run.smithi027.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 284 kB/s | 16 kB 00:00 2024-05-10T10:55:38.551 INFO:teuthology.orchestra.run.smithi027.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-05-10T10:55:38.557 INFO:teuthology.orchestra.run.smithi189.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 81 kB/s | 51 kB 00:00 2024-05-10T10:55:38.576 INFO:teuthology.orchestra.run.smithi027.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 572 kB/s | 28 kB 00:00 2024-05-10T10:55:38.601 INFO:teuthology.orchestra.run.smithi027.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 623 kB/s | 31 kB 00:00 2024-05-10T10:55:38.668 INFO:teuthology.orchestra.run.smithi027.stdout:(138/156): python3-grpcio-1.46.7-10.el9.x86_64. 3.7 MB/s | 2.0 MB 00:00 2024-05-10T10:55:38.694 INFO:teuthology.orchestra.run.smithi027.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-05-10T10:55:38.780 INFO:teuthology.orchestra.run.smithi027.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 333 kB/s | 59 kB 00:00 2024-05-10T10:55:38.811 INFO:teuthology.orchestra.run.smithi027.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 253 kB/s | 36 kB 00:00 2024-05-10T10:55:38.836 INFO:teuthology.orchestra.run.smithi027.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 932 kB/s | 51 kB 00:00 2024-05-10T10:55:38.861 INFO:teuthology.orchestra.run.smithi027.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-05-10T10:55:38.895 INFO:teuthology.orchestra.run.smithi027.stdout:(144/156): python3-webob-1.8.7-6.el9.noarch.rpm 2.0 MB/s | 230 kB 00:00 2024-05-10T10:55:38.921 INFO:teuthology.orchestra.run.smithi027.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 344 kB/s | 20 kB 00:00 2024-05-10T10:55:38.954 INFO:teuthology.orchestra.run.smithi027.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-05-10T10:55:38.988 INFO:teuthology.orchestra.run.smithi027.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-05-10T10:55:39.041 INFO:teuthology.orchestra.run.smithi189.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 81 kB/s | 54 kB 00:00 2024-05-10T10:55:39.063 INFO:teuthology.orchestra.run.smithi027.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-05-10T10:55:39.183 INFO:teuthology.orchestra.run.smithi189.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 67 kB/s | 42 kB 00:00 2024-05-10T10:55:39.289 INFO:teuthology.orchestra.run.smithi027.stdout:(149/156): NetworkManager-1.47.5-1.el9.x86_64.r 10 MB/s | 2.3 MB 00:00 2024-05-10T10:55:39.372 INFO:teuthology.orchestra.run.smithi027.stdout:(150/156): librbd1-19.0.0-3449.gd39fa088.el9.x8 8.2 MB/s | 3.1 MB 00:00 2024-05-10T10:55:39.464 INFO:teuthology.orchestra.run.smithi027.stdout:(151/156): librados2-19.0.0-3449.gd39fa088.el9. 6.7 MB/s | 3.4 MB 00:00 2024-05-10T10:55:39.489 INFO:teuthology.orchestra.run.smithi027.stdout:(152/156): NetworkManager-team-1.47.5-1.el9.x86 360 kB/s | 42 kB 00:00 2024-05-10T10:55:39.564 INFO:teuthology.orchestra.run.smithi027.stdout:(153/156): NetworkManager-libnm-1.47.5-1.el9.x8 6.6 MB/s | 1.8 MB 00:00 2024-05-10T10:55:39.598 INFO:teuthology.orchestra.run.smithi027.stdout:(154/156): NetworkManager-tui-1.47.5-1.el9.x86_ 1.8 MB/s | 248 kB 00:00 2024-05-10T10:55:39.657 INFO:teuthology.orchestra.run.smithi027.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2024-05-10T10:55:39.699 INFO:teuthology.orchestra.run.smithi189.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 67 kB/s | 44 kB 00:00 2024-05-10T10:55:39.723 INFO:teuthology.orchestra.run.smithi027.stdout:(156/156): openssl-3.2.1-1.el9.x86_64.rpm 5.8 MB/s | 1.3 MB 00:00 2024-05-10T10:55:39.731 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-10T10:55:39.731 INFO:teuthology.orchestra.run.smithi027.stdout:Total 18 MB/s | 226 MB 00:12 2024-05-10T10:55:40.016 INFO:teuthology.orchestra.run.smithi189.stdout:(94/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-05-10T10:55:40.067 INFO:teuthology.orchestra.run.smithi189.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 250 kB/s | 221 kB 00:00 2024-05-10T10:55:40.417 INFO:teuthology.orchestra.run.smithi189.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 50 kB/s | 36 kB 00:00 2024-05-10T10:55:40.684 INFO:teuthology.orchestra.run.smithi189.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 74 kB/s | 47 kB 00:00 2024-05-10T10:55:40.901 INFO:teuthology.orchestra.run.smithi189.stdout:(98/156): socat-1.7.4.1-5.el9.x86_64.rpm 366 kB/s | 305 kB 00:00 2024-05-10T10:55:41.076 INFO:teuthology.orchestra.run.smithi189.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 88 kB/s | 58 kB 00:00 2024-05-10T10:55:41.384 INFO:teuthology.orchestra.run.smithi189.stdout:(100/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 134 kB/s | 94 kB 00:00 2024-05-10T10:55:41.468 INFO:teuthology.orchestra.run.smithi189.stdout:(101/156): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-05-10T10:55:41.585 INFO:teuthology.orchestra.run.smithi189.stdout:(102/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 93 kB/s | 64 kB 00:00 2024-05-10T10:55:41.702 INFO:teuthology.orchestra.run.smithi189.stdout:(103/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-05-10T10:55:41.727 INFO:teuthology.orchestra.run.smithi189.stdout:(104/156): protobuf-compiler-3.14.0-13.el9.x86_ 3.3 MB/s | 863 kB 00:00 2024-05-10T10:55:41.785 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-05-10T10:55:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-10T10:55:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-10T10:55:41.962 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-10T10:55:41.985 INFO:teuthology.orchestra.run.smithi189.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-05-10T10:55:41.997 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-10T10:55:42.011 INFO:teuthology.orchestra.run.smithi189.stdout:(106/156): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-05-10T10:55:42.022 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-10T10:55:42.036 INFO:teuthology.orchestra.run.smithi189.stdout:(107/156): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-05-10T10:55:42.065 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-10T10:55:42.187 INFO:teuthology.orchestra.run.smithi189.stdout:(108/156): libarrow-9.0.0-11.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2024-05-10T10:55:42.212 INFO:teuthology.orchestra.run.smithi189.stdout:(109/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 130 kB/s | 26 kB 00:00 2024-05-10T10:55:42.237 INFO:teuthology.orchestra.run.smithi189.stdout:(110/156): liboath-2.6.7-2.el9.x86_64.rpm 244 kB/s | 49 kB 00:00 2024-05-10T10:55:42.262 INFO:teuthology.orchestra.run.smithi189.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-05-10T10:55:42.304 INFO:teuthology.orchestra.run.smithi189.stdout:(112/156): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-05-10T10:55:42.338 INFO:teuthology.orchestra.run.smithi189.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-05-10T10:55:42.363 INFO:teuthology.orchestra.run.smithi189.stdout:(114/156): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-05-10T10:55:42.380 INFO:teuthology.orchestra.run.smithi189.stdout:(115/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 596 kB/s | 43 kB 00:00 2024-05-10T10:55:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-10T10:55:42.397 INFO:teuthology.orchestra.run.smithi189.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 552 kB/s | 32 kB 00:00 2024-05-10T10:55:42.422 INFO:teuthology.orchestra.run.smithi189.stdout:(117/156): python3-certifi-2021.10.8-2.el9.noar 254 kB/s | 15 kB 00:00 2024-05-10T10:55:42.448 INFO:teuthology.orchestra.run.smithi189.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-05-10T10:55:42.473 INFO:teuthology.orchestra.run.smithi189.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-05-10T10:55:42.490 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librados2-2:19.0.0-3449.gd39fa088.el9.x86_64 8/164 2024-05-10T10:55:42.498 INFO:teuthology.orchestra.run.smithi189.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-05-10T10:55:42.515 INFO:teuthology.orchestra.run.smithi189.stdout:(121/156): python3-colorama-0.4.4-12.el9.noarch 533 kB/s | 35 kB 00:00 2024-05-10T10:55:42.549 INFO:teuthology.orchestra.run.smithi189.stdout:(122/156): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-05-10T10:55:42.612 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:19.0.0-3449.gd39fa088.el9.x86_64 8/164 2024-05-10T10:55:42.624 INFO:teuthology.orchestra.run.smithi189.stdout:(123/156): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-05-10T10:55:42.649 INFO:teuthology.orchestra.run.smithi189.stdout:(124/156): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-05-10T10:55:42.667 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs2-2:19.0.0-3449.gd39fa088.el9.x86_64 9/164 2024-05-10T10:55:42.675 INFO:teuthology.orchestra.run.smithi189.stdout:(125/156): python3-jaraco-8.2.1-3.el9.noarch.rp 86 kB/s | 11 kB 00:00 2024-05-10T10:55:42.700 INFO:teuthology.orchestra.run.smithi189.stdout:(126/156): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-05-10T10:55:42.741 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs2-2:19.0.0-3449.gd39fa088.el9.x86_64 9/164 2024-05-10T10:55:42.742 INFO:teuthology.orchestra.run.smithi189.stdout:(127/156): python3-jaraco-collections-3.0.0-8.e 348 kB/s | 23 kB 00:00 2024-05-10T10:55:42.767 INFO:teuthology.orchestra.run.smithi189.stdout:(128/156): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-05-10T10:55:42.771 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rados-2:19.0.0-3449.gd39fa088.el9.x86_64 10/164 2024-05-10T10:55:42.821 INFO:teuthology.orchestra.run.smithi189.stdout:(129/156): python3-jaraco-text-3.2.0-6.el9.noar 162 kB/s | 20 kB 00:00 2024-05-10T10:55:42.859 INFO:teuthology.orchestra.run.smithi189.stdout:(130/156): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-05-10T10:55:42.885 INFO:teuthology.orchestra.run.smithi189.stdout:(131/156): python3-logutils-0.3.5-21.el9.noarch 765 kB/s | 46 kB 00:00 2024-05-10T10:55:42.910 INFO:teuthology.orchestra.run.smithi189.stdout:(132/156): python3-more-itertools-8.12.0-2.el9. 921 kB/s | 79 kB 00:00 2024-05-10T10:55:42.914 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-10T10:55:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-10T10:55:42.935 INFO:teuthology.orchestra.run.smithi189.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 763 kB/s | 58 kB 00:00 2024-05-10T10:55:42.960 INFO:teuthology.orchestra.run.smithi189.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-05-10T10:55:42.977 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-10T10:55:42.985 INFO:teuthology.orchestra.run.smithi189.stdout:(135/156): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-05-10T10:55:43.005 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephsqlite-2:19.0.0-3449.gd39fa088.el9.x86_64 13/164 2024-05-10T10:55:43.011 INFO:teuthology.orchestra.run.smithi189.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-05-10T10:55:43.036 INFO:teuthology.orchestra.run.smithi189.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-05-10T10:55:43.061 INFO:teuthology.orchestra.run.smithi189.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-05-10T10:55:43.087 INFO:teuthology.orchestra.run.smithi189.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-05-10T10:55:43.112 INFO:teuthology.orchestra.run.smithi189.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 784 kB/s | 59 kB 00:00 2024-05-10T10:55:43.126 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3449.gd39fa088.el9.x86_64 13/164 2024-05-10T10:55:43.137 INFO:teuthology.orchestra.run.smithi189.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 475 kB/s | 36 kB 00:00 2024-05-10T10:55:43.148 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libradosstriper1-2:19.0.0-3449.gd39fa088.el9.x86 14/164 2024-05-10T10:55:43.163 INFO:teuthology.orchestra.run.smithi189.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-05-10T10:55:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-10T10:55:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-10T10:55:43.188 INFO:teuthology.orchestra.run.smithi189.stdout:(143/156): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-05-10T10:55:43.213 INFO:teuthology.orchestra.run.smithi189.stdout:(144/156): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-05-10T10:55:43.228 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3449.gd39fa088.el9.x86 14/164 2024-05-10T10:55:43.239 INFO:teuthology.orchestra.run.smithi189.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-05-10T10:55:43.250 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-10T10:55:43.264 INFO:teuthology.orchestra.run.smithi189.stdout:(146/156): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-05-10T10:55:43.268 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-10T10:55:43.306 INFO:teuthology.orchestra.run.smithi189.stdout:(147/156): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-05-10T10:55:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-10T10:55:43.456 INFO:teuthology.orchestra.run.smithi189.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-05-10T10:55:43.546 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-10T10:55:43.632 INFO:teuthology.orchestra.run.smithi189.stdout:(149/156): NetworkManager-1.47.5-1.el9.x86_64.r 13 MB/s | 2.3 MB 00:00 2024-05-10T10:55:43.690 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-10T10:55:43.690 INFO:teuthology.orchestra.run.smithi189.stdout:(150/156): librados2-19.0.0-3449.gd39fa088.el9. 8.0 MB/s | 3.4 MB 00:00 2024-05-10T10:55:43.749 INFO:teuthology.orchestra.run.smithi189.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 716 kB/s | 42 kB 00:00 2024-05-10T10:55:43.750 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-10T10:55:43.841 INFO:teuthology.orchestra.run.smithi189.stdout:(152/156): NetworkManager-libnm-1.47.5-1.el9.x8 8.8 MB/s | 1.8 MB 00:00 2024-05-10T10:55:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-10T10:55:43.866 INFO:teuthology.orchestra.run.smithi189.stdout:(153/156): NetworkManager-tui-1.47.5-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-05-10T10:55:43.944 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-10T10:55:43.983 INFO:teuthology.orchestra.run.smithi189.stdout:(154/156): openssl-3.2.1-1.el9.x86_64.rpm 9.5 MB/s | 1.3 MB 00:00 2024-05-10T10:55:44.058 INFO:teuthology.orchestra.run.smithi189.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-05-10T10:55:44.153 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-10T10:55:44.280 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-10T10:55:44.300 INFO:teuthology.orchestra.run.smithi189.stdout:(156/156): librbd1-19.0.0-3449.gd39fa088.el9.x8 3.2 MB/s | 3.1 MB 00:00 2024-05-10T10:55:44.304 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-05-10T10:55:44.304 INFO:teuthology.orchestra.run.smithi189.stdout:Total 8.0 MB/s | 226 MB 00:28 2024-05-10T10:55:44.379 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-10T10:55:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-10T10:55:44.452 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-10T10:55:44.525 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-10T10:55:44.560 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-10T10:55:44.603 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-10T10:55:44.636 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-10T10:55:44.664 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-10T10:55:44.733 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-10T10:55:44.796 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-10T10:55:44.826 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-10T10:55:44.870 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-10T10:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-10T10:55:44.980 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-common-2:19.0.0-3449.gd39fa088.el9. 33/164 2024-05-10T10:55:45.024 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-argparse-2:19.0.0-3449.gd39fa088.el 34/164 2024-05-10T10:55:45.053 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cephfs-2:19.0.0-3449.gd39fa088.el9.x86_6 35/164 2024-05-10T10:55:45.124 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-10T10:55:45.168 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-10T10:55:45.242 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-10T10:55:45.376 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-10T10:55:45.430 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-10T10:55:45.483 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-05-10T10:55:45.635 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-05-10T10:55:45.636 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-05-10T10:55:45.886 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-10T10:55:45.887 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-10T10:55:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-10T10:55:46.043 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-10T10:55:46.072 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-10T10:55:46.115 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-10T10:55:46.153 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librados-devel-2:19.0.0-3449.gd39fa088.el9.x86_6 45/164 2024-05-10T10:55:46.195 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-10T10:55:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-10T10:55:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-10T10:55:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-10T10:55:46.527 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-10T10:55:46.557 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-10T10:55:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-10T10:55:46.641 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-10T10:55:46.677 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-10T10:55:46.720 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-10T10:55:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-10T10:55:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-10T10:55:46.952 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-10T10:55:46.982 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-10T10:55:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-10T10:55:47.120 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-10T10:55:47.158 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-10T10:55:47.198 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-10T10:55:47.224 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-10T10:55:47.248 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-10T10:55:47.324 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-10T10:55:47.437 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-05-10T10:55:47.438 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-05-10T10:55:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-10T10:55:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-10T10:55:47.528 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-10T10:55:47.562 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-10T10:55:47.601 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-10T10:55:47.633 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-10T10:55:47.662 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-10T10:55:47.697 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-10T10:55:47.730 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-10T10:55:47.772 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-10T10:55:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-10T10:55:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-10T10:55:48.146 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-10T10:55:48.179 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-10T10:55:48.224 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-10T10:55:48.323 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-10T10:55:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-10T10:55:49.105 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-10T10:55:49.174 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-10T10:55:49.208 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-10T10:55:49.308 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-10T10:55:49.366 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-10T10:55:49.454 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-10T10:55:49.485 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-10T10:55:49.663 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-10T10:55:49.691 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-10T10:55:49.769 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-10T10:55:50.419 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-10T10:55:50.548 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-10T10:55:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-10T10:55:51.864 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-10T10:55:51.888 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-10T10:55:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-10T10:55:52.232 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-10T10:55:52.284 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librbd1-2:19.0.0-3449.gd39fa088.el9.x86_64 101/164 2024-05-10T10:55:52.399 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:19.0.0-3449.gd39fa088.el9.x86_64 101/164 2024-05-10T10:55:52.422 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rbd-2:19.0.0-3449.gd39fa088.el9.x86_64 102/164 2024-05-10T10:55:52.458 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-10T10:55:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-10T10:55:52.938 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-10T10:55:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-10T10:55:52.979 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librgw2-2:19.0.0-3449.gd39fa088.el9.x86_64 105/164 2024-05-10T10:55:53.039 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-10T10:55:53.049 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librgw2-2:19.0.0-3449.gd39fa088.el9.x86_64 105/164 2024-05-10T10:55:53.072 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rgw-2:19.0.0-3449.gd39fa088.el9.x86_64 106/164 2024-05-10T10:55:53.157 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-10T10:55:53.211 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-10T10:55:53.250 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-10T10:55:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-10T10:55:53.766 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-05-10T10:55:53.834 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-10T10:55:53.846 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-10T10:55:53.884 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-10T10:55:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-10T10:55:53.980 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-10T10:55:53.998 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:19.0.0-3449.gd39fa088.el9.x86_64 8/164 2024-05-10T10:55:54.005 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-10T10:55:54.048 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-10T10:55:54.222 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:19.0.0-3449.gd39fa088.el9.x86_64 8/164 2024-05-10T10:55:54.257 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:19.0.0-3449.gd39fa088.el9.x86_64 9/164 2024-05-10T10:55:54.368 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-10T10:55:54.438 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:19.0.0-3449.gd39fa088.el9.x86_64 9/164 2024-05-10T10:55:54.482 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:19.0.0-3449.gd39fa088.el9.x86_64 8/164 2024-05-10T10:55:54.483 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:19.0.0-3449.gd39fa088.el9.x86_64 10/164 2024-05-10T10:55:54.594 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:19.0.0-3449.gd39fa088.el9.x86_64 8/164 2024-05-10T10:55:54.624 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:19.0.0-3449.gd39fa088.el9.x86_64 9/164 2024-05-10T10:55:54.708 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:19.0.0-3449.gd39fa088.el9.x86_64 9/164 2024-05-10T10:55:54.737 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:19.0.0-3449.gd39fa088.el9.x86_64 10/164 2024-05-10T10:55:54.818 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-10T10:55:54.888 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-10T10:55:54.894 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-10T10:55:54.920 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:19.0.0-3449.gd39fa088.el9.x86_64 13/164 2024-05-10T10:55:54.942 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-10T10:55:54.970 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:19.0.0-3449.gd39fa088.el9.x86_64 13/164 2024-05-10T10:55:55.074 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3449.gd39fa088.el9.x86_64 13/164 2024-05-10T10:55:55.096 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:19.0.0-3449.gd39fa088.el9.x86 14/164 2024-05-10T10:55:55.121 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3449.gd39fa088.el9.x86_64 13/164 2024-05-10T10:55:55.142 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3449.gd39fa088.el9.x86 14/164 2024-05-10T10:55:55.154 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:19.0.0-3449.gd39fa088.el9.x86 14/164 2024-05-10T10:55:55.164 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-10T10:55:55.183 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-10T10:55:55.275 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3449.gd39fa088.el9.x86 14/164 2024-05-10T10:55:55.306 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-10T10:55:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 107/164 2024-05-10T10:55:55.365 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-10T10:55:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 107/164 2024-05-10T10:55:55.388 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-10T10:55:55.496 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-10T10:55:55.645 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-10T10:55:55.685 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-10T10:55:55.705 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-10T10:55:55.826 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-10T10:55:55.890 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-10T10:55:55.923 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 107/164 2024-05-10T10:55:55.955 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-base-2:19.0.0-3449.gd39fa088.el9.x86_64 108/164 2024-05-10T10:55:55.999 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-10T10:55:56.038 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-base-2:19.0.0-3449.gd39fa088.el9.x86_64 108/164 2024-05-10T10:55:56.038 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-10T10:55:56.038 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-10T10:55:56.038 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:55:56.072 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-selinux-2:19.0.0-3449.gd39fa088.el9.x86_64 109/164 2024-05-10T10:55:56.092 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-10T10:55:56.162 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-10T10:55:56.195 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-10T10:55:56.267 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-10T10:55:56.280 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-10T10:55:56.329 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-10T10:55:56.363 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-10T10:55:56.427 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-10T10:55:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-10T10:55:56.462 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-10T10:55:56.506 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-10T10:55:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-10T10:55:56.538 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-10T10:55:56.575 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-10T10:55:56.642 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-10T10:55:56.721 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-10T10:55:56.753 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-10T10:55:56.788 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-10T10:55:56.852 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-10T10:55:56.864 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-10T10:55:56.899 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:19.0.0-3449.gd39fa088.el9. 33/164 2024-05-10T10:55:56.934 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:19.0.0-3449.gd39fa088.el 34/164 2024-05-10T10:55:56.954 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:19.0.0-3449.gd39fa088.el9.x86_6 35/164 2024-05-10T10:55:57.023 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-10T10:55:57.069 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-10T10:55:57.073 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-10T10:55:57.143 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-10T10:55:57.201 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-10T10:55:57.264 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-10T10:55:57.275 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-10T10:55:57.315 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-10T10:55:57.330 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-10T10:55:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-10T10:55:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-10T10:55:57.533 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-10T10:55:57.593 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-10T10:55:57.628 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-10T10:55:57.782 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-10T10:55:57.853 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-10T10:55:57.858 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-10T10:55:57.899 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-10T10:55:57.943 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-10T10:55:57.959 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-10T10:55:57.971 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-10T10:55:58.006 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-10T10:55:58.044 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:19.0.0-3449.gd39fa088.el9.x86_6 45/164 2024-05-10T10:55:58.078 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-10T10:55:58.085 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-10T10:55:58.118 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:19.0.0-3449.gd39fa088.el9. 33/164 2024-05-10T10:55:58.245 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-10T10:55:58.304 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:19.0.0-3449.gd39fa088.el 34/164 2024-05-10T10:55:58.305 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-10T10:55:58.353 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:19.0.0-3449.gd39fa088.el9.x86_6 35/164 2024-05-10T10:55:58.359 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-10T10:55:58.432 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-10T10:55:58.464 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-10T10:55:58.494 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-10T10:55:58.500 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-10T10:55:58.547 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-10T10:55:58.565 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-10T10:55:58.584 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-10T10:55:58.626 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-10T10:55:58.668 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-10T10:55:58.680 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-10T10:55:58.803 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-10T10:55:58.857 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-10T10:55:58.859 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-10T10:55:58.896 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-10T10:55:58.932 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-10T10:55:58.953 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-10T10:55:59.047 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-10T10:55:59.079 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-10T10:55:59.120 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-10T10:55:59.145 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-10T10:55:59.170 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-10T10:55:59.235 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-10T10:55:59.363 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-10T10:55:59.410 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-10T10:55:59.438 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-10T10:55:59.472 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-10T10:55:59.512 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-10T10:55:59.543 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-10T10:55:59.573 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-10T10:55:59.608 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-10T10:55:59.637 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-10T10:55:59.641 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-10T10:55:59.674 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-10T10:55:59.709 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-10T10:55:59.752 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-10T10:55:59.796 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-10T10:55:59.798 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-10T10:55:59.846 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-10T10:55:59.847 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-10T10:55:59.879 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-10T10:55:59.895 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:19.0.0-3449.gd39fa088.el9.x86_6 45/164 2024-05-10T10:55:59.925 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-10T10:55:59.971 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-10T10:56:00.024 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-10T10:56:00.080 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-10T10:56:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-10T10:56:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-10T10:56:00.343 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-10T10:56:00.441 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-10T10:56:00.484 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-10T10:56:00.524 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-10T10:56:00.621 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-10T10:56:00.667 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-10T10:56:00.732 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-10T10:56:00.797 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-10T10:56:00.807 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-10T10:56:00.852 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-10T10:56:00.886 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-10T10:56:00.984 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-10T10:56:01.007 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-10T10:56:01.036 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-10T10:56:01.072 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-10T10:56:01.108 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-10T10:56:01.130 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-10T10:56:01.159 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-10T10:56:01.180 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-10T10:56:01.294 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-10T10:56:01.342 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-10T10:56:01.358 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-10T10:56:01.394 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-10T10:56:01.399 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-10T10:56:01.441 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-10T10:56:01.472 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-10T10:56:01.474 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-10T10:56:01.688 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-10T10:56:01.871 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-10T10:56:01.913 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-10T10:56:01.952 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-10T10:56:02.023 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-10T10:56:02.084 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-10T10:56:02.120 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-10T10:56:02.133 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-10T10:56:02.183 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-10T10:56:02.228 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-10T10:56:02.233 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-10T10:56:02.279 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-10T10:56:02.355 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-10T10:56:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-10T10:56:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-10T10:56:02.631 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-10T10:56:02.677 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-10T10:56:02.749 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-10T10:56:02.911 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-10T10:56:03.008 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-10T10:56:03.486 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-10T10:56:03.565 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-10T10:56:03.601 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-10T10:56:03.634 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-10T10:56:03.952 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-10T10:56:03.997 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:19.0.0-3449.gd39fa088.el9.x86_64 101/164 2024-05-10T10:56:04.012 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-10T10:56:04.064 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-10T10:56:04.071 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:19.0.0-3449.gd39fa088.el9.x86_64 101/164 2024-05-10T10:56:04.098 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-10T10:56:04.102 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:19.0.0-3449.gd39fa088.el9.x86_64 102/164 2024-05-10T10:56:04.138 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-10T10:56:04.217 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-10T10:56:04.285 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-10T10:56:04.393 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-10T10:56:04.441 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-10T10:56:04.620 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-10T10:56:04.650 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:19.0.0-3449.gd39fa088.el9.x86_64 105/164 2024-05-10T10:56:04.686 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-10T10:56:04.712 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-10T10:56:04.721 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:19.0.0-3449.gd39fa088.el9.x86_64 105/164 2024-05-10T10:56:04.743 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:19.0.0-3449.gd39fa088.el9.x86_64 106/164 2024-05-10T10:56:04.820 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-10T10:56:05.762 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-10T10:56:05.930 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-10T10:56:07.047 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 107/164 2024-05-10T10:56:07.115 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 107/164 2024-05-10T10:56:07.539 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3449.gd39fa088.el9.x86_64 109/164 2024-05-10T10:56:07.539 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /sys 2024-05-10T10:56:07.539 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /proc 2024-05-10T10:56:07.539 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /mnt 2024-05-10T10:56:07.539 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /var/tmp 2024-05-10T10:56:07.539 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /home 2024-05-10T10:56:07.540 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /root 2024-05-10T10:56:07.540 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /tmp 2024-05-10T10:56:07.540 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:56:07.613 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-10T10:56:07.659 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 107/164 2024-05-10T10:56:07.693 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:19.0.0-3449.gd39fa088.el9.x86_64 108/164 2024-05-10T10:56:07.709 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-volume-2:19.0.0-3449.gd39fa088.el9.noarch 110/164 2024-05-10T10:56:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-10T10:56:07.723 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-volume-2:19.0.0-3449.gd39fa088.el9.noarch 110/164 2024-05-10T10:56:07.723 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:07.723 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-10T10:56:07.723 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:56:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-10T10:56:07.790 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:19.0.0-3449.gd39fa088.el9.x86_64 108/164 2024-05-10T10:56:07.790 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-10T10:56:07.790 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-10T10:56:07.790 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-10T10:56:07.819 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:19.0.0-3449.gd39fa088.el9.x86_64 109/164 2024-05-10T10:56:08.208 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-10T10:56:08.281 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:19.0.0-3449.gd39fa088.el9.x86_64 101/164 2024-05-10T10:56:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:19.0.0-3449.gd39fa088.el9.x86_64 101/164 2024-05-10T10:56:08.447 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:19.0.0-3449.gd39fa088.el9.x86_64 102/164 2024-05-10T10:56:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-10T10:56:09.090 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-10T10:56:09.117 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:19.0.0-3449.gd39fa088.el9.x86_64 105/164 2024-05-10T10:56:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-osd-2:19.0.0-3449.gd39fa088.el9.x86_64 111/164 2024-05-10T10:56:09.237 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-osd-2:19.0.0-3449.gd39fa088.el9.x86_64 111/164 2024-05-10T10:56:09.237 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:09.237 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-10T10:56:09.237 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-10T10:56:09.237 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-10T10:56:09.237 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:56:09.276 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:19.0.0-3449.gd39fa088.el9.x86_64 105/164 2024-05-10T10:56:09.302 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:19.0.0-3449.gd39fa088.el9.x86_64 106/164 2024-05-10T10:56:09.463 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mds-2:19.0.0-3449.gd39fa088.el9.x86_64 112/164 2024-05-10T10:56:09.484 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mds-2:19.0.0-3449.gd39fa088.el9.x86_64 112/164 2024-05-10T10:56:09.484 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:09.484 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-10T10:56:09.484 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-10T10:56:09.484 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-10T10:56:09.485 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:56:09.929 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mon-2:19.0.0-3449.gd39fa088.el9.x86_64 113/164 2024-05-10T10:56:09.952 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mon-2:19.0.0-3449.gd39fa088.el9.x86_64 113/164 2024-05-10T10:56:09.952 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:09.952 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-10T10:56:09.952 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-10T10:56:09.952 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-10T10:56:09.952 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:56:10.068 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-10T10:56:10.125 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-10T10:56:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-10T10:56:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-10T10:56:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-10T10:56:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-10T10:56:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-10T10:56:10.737 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-10T10:56:10.779 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-10T10:56:10.812 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-10T10:56:10.839 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-10T10:56:11.309 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 125/164 2024-05-10T10:56:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 125/164 2024-05-10T10:56:11.370 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3449.gd39fa088.e 126/164 2024-05-10T10:56:11.431 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3449.gd39fa088. 127/164 2024-05-10T10:56:11.460 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3449.gd39fa088.el9.noa 128/164 2024-05-10T10:56:12.363 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 107/164 2024-05-10T10:56:12.399 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 107/164 2024-05-10T10:56:12.495 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3449.gd39fa088.el9.noa 128/164 2024-05-10T10:56:12.537 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3449.gd39fa088.el9.n 129/164 2024-05-10T10:56:13.083 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 107/164 2024-05-10T10:56:13.128 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:19.0.0-3449.gd39fa088.el9.x86_64 108/164 2024-05-10T10:56:13.332 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.0.0-3449.gd39fa088.el9.x86_64 108/164 2024-05-10T10:56:13.332 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-10T10:56:13.332 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-10T10:56:13.333 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:13.389 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:19.0.0-3449.gd39fa088.el9.x86_64 109/164 2024-05-10T10:56:13.508 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3449.gd39fa088.el9.n 129/164 2024-05-10T10:56:13.537 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3449.gd39 130/164 2024-05-10T10:56:13.651 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3449.gd39 130/164 2024-05-10T10:56:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3449.gd39fa088.el 131/164 2024-05-10T10:56:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-2:19.0.0-3449.gd39fa088.el9.x86_64 132/164 2024-05-10T10:56:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3449.gd39fa088.el9.x86_64 132/164 2024-05-10T10:56:13.862 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:13.862 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-10T10:56:13.862 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-10T10:56:13.862 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-10T10:56:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:56:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-rook-2:19.0.0-3449.gd39fa088.el9.noarch 133/164 2024-05-10T10:56:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3449.gd39fa088.el9.noarch 133/164 2024-05-10T10:56:14.015 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-2:19.0.0-3449.gd39fa088.el9.x86_64 134/164 2024-05-10T10:56:14.063 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 135/164 2024-05-10T10:56:14.991 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-10T10:56:15.034 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-radosgw-2:19.0.0-3449.gd39fa088.el9.x86_64 137/164 2024-05-10T10:56:15.056 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3449.gd39fa088.el9.x86_64 137/164 2024-05-10T10:56:15.056 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:15.057 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-10T10:56:15.057 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-10T10:56:15.057 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-10T10:56:15.057 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:56:15.101 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3449.gd39fa 138/164 2024-05-10T10:56:15.123 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3449.gd39fa 138/164 2024-05-10T10:56:15.123 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:15.123 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-10T10:56:15.123 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:56:15.168 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 139/164 2024-05-10T10:56:15.190 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 139/164 2024-05-10T10:56:15.190 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:15.191 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-10T10:56:15.191 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-10T10:56:15.191 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-10T10:56:15.191 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:56:15.494 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 140/164 2024-05-10T10:56:15.516 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 140/164 2024-05-10T10:56:15.516 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:15.517 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-10T10:56:15.517 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-10T10:56:15.517 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-10T10:56:15.517 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:56:19.494 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3449.gd39fa088.el9.x86_64 109/164 2024-05-10T10:56:19.494 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-05-10T10:56:19.494 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-05-10T10:56:19.494 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-05-10T10:56:19.494 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-05-10T10:56:19.495 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-05-10T10:56:19.495 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /root 2024-05-10T10:56:19.495 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-05-10T10:56:19.495 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:19.678 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-volume-2:19.0.0-3449.gd39fa088.el9.noarch 110/164 2024-05-10T10:56:19.694 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-volume-2:19.0.0-3449.gd39fa088.el9.noarch 110/164 2024-05-10T10:56:19.694 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:19.694 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-10T10:56:19.694 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:20.097 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-test-2:19.0.0-3449.gd39fa088.el9.x86_64 141/164 2024-05-10T10:56:20.138 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 142/164 2024-05-10T10:56:20.179 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-nbd-2:19.0.0-3449.gd39fa088.el9.x86_64 143/164 2024-05-10T10:56:20.211 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-10T10:56:20.239 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-10T10:56:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-shell-2:19.0.0-3449.gd39fa088.el9.x86_64 146/164 2024-05-10T10:56:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-10T10:56:20.523 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs-devel-2:19.0.0-3449.gd39fa088.el9.x86_ 148/164 2024-05-10T10:56:20.552 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-10T10:56:20.598 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-10T10:56:21.124 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-10T10:56:21.162 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-10T10:56:21.182 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-10T10:56:21.189 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:19.0.0-3449.gd39fa088.el9.x86_64 111/164 2024-05-10T10:56:21.215 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:19.0.0-3449.gd39fa088.el9.x86_64 111/164 2024-05-10T10:56:21.215 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:21.215 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-10T10:56:21.215 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-10T10:56:21.215 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-10T10:56:21.215 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:21.440 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:19.0.0-3449.gd39fa088.el9.x86_64 112/164 2024-05-10T10:56:21.463 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:19.0.0-3449.gd39fa088.el9.x86_64 112/164 2024-05-10T10:56:21.464 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:21.464 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-10T10:56:21.464 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-10T10:56:21.464 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-10T10:56:21.464 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:21.908 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:19.0.0-3449.gd39fa088.el9.x86_64 113/164 2024-05-10T10:56:21.930 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:19.0.0-3449.gd39fa088.el9.x86_64 113/164 2024-05-10T10:56:21.931 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:21.931 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-10T10:56:21.931 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-10T10:56:21.931 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-10T10:56:21.931 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:22.031 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-top-2:19.0.0-3449.gd39fa088.el9.noarch 152/164 2024-05-10T10:56:22.038 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-10T10:56:22.095 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-10T10:56:22.099 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-10T10:56:22.141 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-10T10:56:22.151 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-10T10:56:22.233 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-10T10:56:22.277 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-10T10:56:22.303 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-10T10:56:22.303 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-10T10:56:22.339 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-10T10:56:22.384 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-10T10:56:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-10T10:56:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-10T10:56:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-10T10:56:22.461 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-10T10:56:22.513 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-10T10:56:22.538 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-10T10:56:22.543 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-10T10:56:22.543 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-10T10:56:22.549 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-10T10:56:22.688 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-10T10:56:22.726 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-10T10:56:22.726 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-10T10:56:22.734 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-10T10:56:22.766 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-10T10:56:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-10T10:56:22.794 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-10T10:56:22.865 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-10T10:56:23.204 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 125/164 2024-05-10T10:56:23.263 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 125/164 2024-05-10T10:56:23.300 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3449.gd39fa088.e 126/164 2024-05-10T10:56:23.361 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3449.gd39fa088. 127/164 2024-05-10T10:56:23.399 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3449.gd39fa088.el9.noa 128/164 2024-05-10T10:56:24.451 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3449.gd39fa088.el9.noa 128/164 2024-05-10T10:56:24.492 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3449.gd39fa088.el9.n 129/164 2024-05-10T10:56:25.469 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3449.gd39fa088.el9.n 129/164 2024-05-10T10:56:25.501 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3449.gd39 130/164 2024-05-10T10:56:25.611 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3449.gd39 130/164 2024-05-10T10:56:25.772 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3449.gd39fa088.el 131/164 2024-05-10T10:56:25.803 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:19.0.0-3449.gd39fa088.el9.x86_64 132/164 2024-05-10T10:56:25.827 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3449.gd39fa088.el9.x86_64 132/164 2024-05-10T10:56:25.827 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:25.827 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-10T10:56:25.827 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-10T10:56:25.827 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-10T10:56:25.827 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:25.879 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:19.0.0-3449.gd39fa088.el9.noarch 133/164 2024-05-10T10:56:25.890 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3449.gd39fa088.el9.noarch 133/164 2024-05-10T10:56:25.988 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:19.0.0-3449.gd39fa088.el9.x86_64 134/164 2024-05-10T10:56:26.036 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 135/164 2024-05-10T10:56:26.966 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-10T10:56:26.998 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:19.0.0-3449.gd39fa088.el9.x86_64 137/164 2024-05-10T10:56:27.021 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3449.gd39fa088.el9.x86_64 137/164 2024-05-10T10:56:27.021 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:27.021 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-10T10:56:27.021 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-10T10:56:27.021 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-10T10:56:27.021 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:27.026 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3449.gd39fa088.el9.x86_64 109/164 2024-05-10T10:56:27.026 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-05-10T10:56:27.027 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-05-10T10:56:27.027 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-05-10T10:56:27.027 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-05-10T10:56:27.027 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-05-10T10:56:27.027 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /root 2024-05-10T10:56:27.027 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-05-10T10:56:27.028 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:27.065 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3449.gd39fa 138/164 2024-05-10T10:56:27.087 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3449.gd39fa 138/164 2024-05-10T10:56:27.088 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:27.088 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-10T10:56:27.088 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:27.133 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 139/164 2024-05-10T10:56:27.155 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 139/164 2024-05-10T10:56:27.155 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:27.155 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-10T10:56:27.155 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-10T10:56:27.155 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-10T10:56:27.156 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:27.271 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:19.0.0-3449.gd39fa088.el9.noarch 110/164 2024-05-10T10:56:27.321 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:19.0.0-3449.gd39fa088.el9.noarch 110/164 2024-05-10T10:56:27.321 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:27.321 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-10T10:56:27.322 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:27.467 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 140/164 2024-05-10T10:56:27.489 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 140/164 2024-05-10T10:56:27.489 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:27.489 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-10T10:56:27.489 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-10T10:56:27.489 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-10T10:56:27.490 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-10T10:56:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-2:19.0.0-3449.gd39fa088.el9.x86_64 1/164 2024-05-10T10:56:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-base-2:19.0.0-3449.gd39fa088.el9.x86_64 2/164 2024-05-10T10:56:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 3/164 2024-05-10T10:56:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 4/164 2024-05-10T10:56:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3449.gd39fa 5/164 2024-05-10T10:56:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mds-2:19.0.0-3449.gd39fa088.el9.x86_64 6/164 2024-05-10T10:56:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-2:19.0.0-3449.gd39fa088.el9.x86_64 7/164 2024-05-10T10:56:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mon-2:19.0.0-3449.gd39fa088.el9.x86_64 8/164 2024-05-10T10:56:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-osd-2:19.0.0-3449.gd39fa088.el9.x86_64 9/164 2024-05-10T10:56:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-radosgw-2:19.0.0-3449.gd39fa088.el9.x86_64 10/164 2024-05-10T10:56:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-selinux-2:19.0.0-3449.gd39fa088.el9.x86_64 11/164 2024-05-10T10:56:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-test-2:19.0.0-3449.gd39fa088.el9.x86_64 12/164 2024-05-10T10:56:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 13/164 2024-05-10T10:56:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-shell-2:19.0.0-3449.gd39fa088.el9.x86_64 14/164 2024-05-10T10:56:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs-devel-2:19.0.0-3449.gd39fa088.el9.x86_ 15/164 2024-05-10T10:56:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs2-2:19.0.0-3449.gd39fa088.el9.x86_64 16/164 2024-05-10T10:56:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephsqlite-2:19.0.0-3449.gd39fa088.el9.x86_64 17/164 2024-05-10T10:56:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados-devel-2:19.0.0-3449.gd39fa088.el9.x86_6 18/164 2024-05-10T10:56:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libradosstriper1-2:19.0.0-3449.gd39fa088.el9.x86 19/164 2024-05-10T10:56:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librgw2-2:19.0.0-3449.gd39fa088.el9.x86_64 20/164 2024-05-10T10:56:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3449.gd39fa088.el 21/164 2024-05-10T10:56:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-common-2:19.0.0-3449.gd39fa088.el9. 22/164 2024-05-10T10:56:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cephfs-2:19.0.0-3449.gd39fa088.el9.x86_6 23/164 2024-05-10T10:56:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rados-2:19.0.0-3449.gd39fa088.el9.x86_64 24/164 2024-05-10T10:56:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rbd-2:19.0.0-3449.gd39fa088.el9.x86_64 25/164 2024-05-10T10:56:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rgw-2:19.0.0-3449.gd39fa088.el9.x86_64 26/164 2024-05-10T10:56:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 27/164 2024-05-10T10:56:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 28/164 2024-05-10T10:56:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-nbd-2:19.0.0-3449.gd39fa088.el9.x86_64 29/164 2024-05-10T10:56:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3449.gd39fa088. 30/164 2024-05-10T10:56:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3449.gd39fa088.el9.noa 31/164 2024-05-10T10:56:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3449.gd39fa088.el9.n 32/164 2024-05-10T10:56:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3449.gd39 33/164 2024-05-10T10:56:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3449.gd39fa088.el 34/164 2024-05-10T10:56:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3449.gd39fa088.el9.noarch 35/164 2024-05-10T10:56:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3449.gd39fa088.e 36/164 2024-05-10T10:56:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-volume-2:19.0.0-3449.gd39fa088.el9.noarch 37/164 2024-05-10T10:56:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 38/164 2024-05-10T10:56:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-top-2:19.0.0-3449.gd39fa088.el9.noarch 39/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-10T10:56:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-10T10:56:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-10T10:56:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-10T10:56:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-10T10:56:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-10T10:56:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-10T10:56:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-10T10:56:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-10T10:56:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-10T10:56:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-10T10:56:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-10T10:56:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-10T10:56:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-10T10:56:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-10T10:56:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-10T10:56:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-10T10:56:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-10T10:56:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-10T10:56:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-10T10:56:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-10T10:56:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-10T10:56:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-10T10:56:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-10T10:56:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-10T10:56:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-10T10:56:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-10T10:56:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-10T10:56:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-10T10:56:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-10T10:56:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-10T10:56:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-10T10:56:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-10T10:56:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-10T10:56:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-10T10:56:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-10T10:56:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-10T10:56:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-10T10:56:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-10T10:56:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-10T10:56:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-10T10:56:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-10T10:56:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-10T10:56:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-10T10:56:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-10T10:56:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-10T10:56:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-10T10:56:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-10T10:56:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:19.0.0-3449.gd39fa088.el9.x86_64 149/164 2024-05-10T10:56:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-10T10:56:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:19.0.0-3449.gd39fa088.el9.x86_64 151/164 2024-05-10T10:56:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-10T10:56:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-10T10:56:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-10T10:56:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-10T10:56:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-10T10:56:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-10T10:56:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-10T10:56:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-10T10:56:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-10T10:56:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-10T10:56:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-10T10:56:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-10T10:56:29.006 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:19.0.0-3449.gd39fa088.el9.x86_64 111/164 2024-05-10T10:56:29.091 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.0.0-3449.gd39fa088.el9.x86_64 111/164 2024-05-10T10:56:29.091 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:29.091 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-10T10:56:29.091 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-10T10:56:29.091 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-10T10:56:29.091 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:29.357 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:19.0.0-3449.gd39fa088.el9.x86_64 112/164 2024-05-10T10:56:29.431 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.0.0-3449.gd39fa088.el9.x86_64 112/164 2024-05-10T10:56:29.432 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:29.432 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-10T10:56:29.432 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-10T10:56:29.432 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-10T10:56:29.432 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:29.966 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:19.0.0-3449.gd39fa088.el9.x86_64 113/164 2024-05-10T10:56:30.066 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.0.0-3449.gd39fa088.el9.x86_64 113/164 2024-05-10T10:56:30.067 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:30.067 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-10T10:56:30.067 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-10T10:56:30.067 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-10T10:56:30.067 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:30.209 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-10T10:56:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-10T10:56:30.349 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-10T10:56:30.602 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-10T10:56:30.671 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-10T10:56:30.762 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-10T10:56:30.877 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-10T10:56:31.046 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-10T10:56:31.095 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-10T10:56:31.135 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-10T10:56:31.168 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-10T10:56:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-10T10:56:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:56:31.313 INFO:teuthology.orchestra.run.smithi076.stdout:Upgraded: 2024-05-10T10:56:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-10T10:56:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-10T10:56:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-10T10:56:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-10T10:56:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: librados2-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-10T10:56:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-10T10:56:31.314 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: automake-1.16.2-8.el9.noarch 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: bison-3.7.4-5.el9.x86_64 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-shell-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-top-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-10T10:56:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: flex-2.6.4-9.el9.x86_64 2024-05-10T10:56:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-10T10:56:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-10T10:56:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-10T10:56:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-10T10:56:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-10T10:56:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-10T10:56:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-10T10:56:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-10T10:56:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-10T10:56:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-10T10:56:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-10T10:56:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-10T10:56:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-10T10:56:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-10T10:56:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-10T10:56:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-10T10:56:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-10T10:56:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: lua-5.4.4-4.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-13.14-1.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-10T10:56:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-10T10:56:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-10T10:56:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply-3.11-14.el9.noarch 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-10T10:56:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-10T10:56:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-10T10:56:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-10T10:56:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-10T10:56:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-10T10:56:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-10T10:56:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-10T10:56:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-10T10:56:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-10T10:56:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-10T10:56:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-10T10:56:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-10T10:56:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: re2-1:20211101-3.el9.x86_64 2024-05-10T10:56:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-10T10:56:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-10T10:56:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-10T10:56:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-10T10:56:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-10T10:56:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-10T10:56:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-05-10T10:56:31.326 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-05-10T10:56:31.725 DEBUG:teuthology.parallel:result is None 2024-05-10T10:56:31.883 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 125/164 2024-05-10T10:56:31.922 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 125/164 2024-05-10T10:56:31.978 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3449.gd39fa088.e 126/164 2024-05-10T10:56:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3449.gd39fa088. 127/164 2024-05-10T10:56:32.128 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:19.0.0-3449.gd39fa088.el9.x86_64 141/164 2024-05-10T10:56:32.131 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3449.gd39fa088.el9.noa 128/164 2024-05-10T10:56:32.169 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 142/164 2024-05-10T10:56:32.210 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:19.0.0-3449.gd39fa088.el9.x86_64 143/164 2024-05-10T10:56:32.242 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-10T10:56:32.270 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-10T10:56:32.442 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-shell-2:19.0.0-3449.gd39fa088.el9.x86_64 146/164 2024-05-10T10:56:32.492 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-10T10:56:32.521 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:19.0.0-3449.gd39fa088.el9.x86_ 148/164 2024-05-10T10:56:32.551 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-10T10:56:32.597 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-10T10:56:33.252 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-10T10:56:33.295 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-10T10:56:33.316 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-10T10:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3449.gd39fa088.el9.noa 128/164 2024-05-10T10:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3449.gd39fa088.el9.n 129/164 2024-05-10T10:56:34.158 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-top-2:19.0.0-3449.gd39fa088.el9.noarch 152/164 2024-05-10T10:56:34.242 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-10T10:56:34.284 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-10T10:56:34.365 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-10T10:56:34.403 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-10T10:56:34.429 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-10T10:56:34.430 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-10T10:56:34.514 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-10T10:56:34.547 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-10T10:56:34.547 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-10T10:56:34.631 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-10T10:56:34.656 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-10T10:56:34.662 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-10T10:56:34.662 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-10T10:56:34.706 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3449.gd39fa088.el9.n 129/164 2024-05-10T10:56:34.741 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3449.gd39 130/164 2024-05-10T10:56:34.842 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-10T10:56:34.842 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-10T10:56:34.908 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3449.gd39 130/164 2024-05-10T10:56:34.934 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-10T10:56:34.998 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-10T10:56:35.089 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3449.gd39fa088.el 131/164 2024-05-10T10:56:35.127 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:19.0.0-3449.gd39fa088.el9.x86_64 132/164 2024-05-10T10:56:35.231 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3449.gd39fa088.el9.x86_64 132/164 2024-05-10T10:56:35.231 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:35.231 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-10T10:56:35.231 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-10T10:56:35.231 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-10T10:56:35.231 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:35.286 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:19.0.0-3449.gd39fa088.el9.noarch 133/164 2024-05-10T10:56:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3449.gd39fa088.el9.noarch 133/164 2024-05-10T10:56:35.502 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:19.0.0-3449.gd39fa088.el9.x86_64 134/164 2024-05-10T10:56:35.576 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 135/164 2024-05-10T10:56:36.678 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-10T10:56:36.705 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:19.0.0-3449.gd39fa088.el9.x86_64 137/164 2024-05-10T10:56:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3449.gd39fa088.el9.x86_64 137/164 2024-05-10T10:56:36.779 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:36.780 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-10T10:56:36.780 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-10T10:56:36.780 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-10T10:56:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:36.831 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3449.gd39fa 138/164 2024-05-10T10:56:36.903 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3449.gd39fa 138/164 2024-05-10T10:56:36.904 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:36.904 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-10T10:56:36.904 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:36.981 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 139/164 2024-05-10T10:56:37.092 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 139/164 2024-05-10T10:56:37.092 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:37.092 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-10T10:56:37.092 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-10T10:56:37.092 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-10T10:56:37.093 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:37.466 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 140/164 2024-05-10T10:56:37.539 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 140/164 2024-05-10T10:56:37.539 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-10T10:56:37.539 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-10T10:56:37.540 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-10T10:56:37.540 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-10T10:56:37.540 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:40.450 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-10T10:56:40.451 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:19.0.0-3449.gd39fa088.el9.x86_64 1/164 2024-05-10T10:56:40.451 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:19.0.0-3449.gd39fa088.el9.x86_64 2/164 2024-05-10T10:56:40.451 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 3/164 2024-05-10T10:56:40.451 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 4/164 2024-05-10T10:56:40.451 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3449.gd39fa 5/164 2024-05-10T10:56:40.451 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:19.0.0-3449.gd39fa088.el9.x86_64 6/164 2024-05-10T10:56:40.451 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:19.0.0-3449.gd39fa088.el9.x86_64 7/164 2024-05-10T10:56:40.451 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:19.0.0-3449.gd39fa088.el9.x86_64 8/164 2024-05-10T10:56:40.451 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:19.0.0-3449.gd39fa088.el9.x86_64 9/164 2024-05-10T10:56:40.451 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:19.0.0-3449.gd39fa088.el9.x86_64 10/164 2024-05-10T10:56:40.451 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:19.0.0-3449.gd39fa088.el9.x86_64 11/164 2024-05-10T10:56:40.451 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:19.0.0-3449.gd39fa088.el9.x86_64 12/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 13/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-shell-2:19.0.0-3449.gd39fa088.el9.x86_64 14/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:19.0.0-3449.gd39fa088.el9.x86_ 15/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:19.0.0-3449.gd39fa088.el9.x86_64 16/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:19.0.0-3449.gd39fa088.el9.x86_64 17/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:19.0.0-3449.gd39fa088.el9.x86_6 18/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:19.0.0-3449.gd39fa088.el9.x86 19/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:19.0.0-3449.gd39fa088.el9.x86_64 20/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3449.gd39fa088.el 21/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:19.0.0-3449.gd39fa088.el9. 22/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:19.0.0-3449.gd39fa088.el9.x86_6 23/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:19.0.0-3449.gd39fa088.el9.x86_64 24/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:19.0.0-3449.gd39fa088.el9.x86_64 25/164 2024-05-10T10:56:40.452 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:19.0.0-3449.gd39fa088.el9.x86_64 26/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 27/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 28/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:19.0.0-3449.gd39fa088.el9.x86_64 29/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3449.gd39fa088. 30/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3449.gd39fa088.el9.noa 31/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3449.gd39fa088.el9.n 32/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3449.gd39 33/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3449.gd39fa088.el 34/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3449.gd39fa088.el9.noarch 35/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3449.gd39fa088.e 36/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-volume-2:19.0.0-3449.gd39fa088.el9.noarch 37/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 38/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-top-2:19.0.0-3449.gd39fa088.el9.noarch 39/164 2024-05-10T10:56:40.453 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-10T10:56:40.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-10T10:56:40.455 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-10T10:56:40.456 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-10T10:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-10T10:56:40.458 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-10T10:56:40.459 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-10T10:56:40.460 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-10T10:56:40.461 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:19.0.0-3449.gd39fa088.el9.x86_64 149/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:19.0.0-3449.gd39fa088.el9.x86_64 151/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-10T10:56:40.462 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-10T10:56:42.884 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:19.0.0-3449.gd39fa088.el9.x86_64 141/164 2024-05-10T10:56:42.941 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 142/164 2024-05-10T10:56:43.006 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:19.0.0-3449.gd39fa088.el9.x86_64 143/164 2024-05-10T10:56:43.072 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-10T10:56:43.111 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-10T10:56:43.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-10T10:56:43.150 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:43.150 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: automake-1.16.2-8.el9.noarch 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: bison-3.7.4-5.el9.x86_64 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-10T10:56:43.151 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: flex-2.6.4-9.el9.x86_64 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-10T10:56:43.153 INFO:teuthology.orchestra.run.smithi189.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-10T10:56:43.154 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: lua-5.4.4-4.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-10T10:56:43.155 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-10T10:56:43.156 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-10T10:56:43.156 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-10T10:56:43.156 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-13.14-1.el9.x86_64 2024-05-10T10:56:43.156 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-10T10:56:43.156 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-10T10:56:43.156 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-10T10:56:43.156 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-10T10:56:43.156 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-10T10:56:43.156 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-10T10:56:43.156 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-10T10:56:43.156 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-10T10:56:43.156 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-10T10:56:43.157 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-10T10:56:43.158 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply-3.11-14.el9.noarch 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-10T10:56:43.159 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-10T10:56:43.160 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: re2-1:20211101-3.el9.x86_64 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-10T10:56:43.161 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-10T10:56:43.162 INFO:teuthology.orchestra.run.smithi189.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-10T10:56:43.162 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-10T10:56:43.162 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-10T10:56:43.162 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-10T10:56:43.162 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T10:56:43.162 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-05-10T10:56:43.416 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-shell-2:19.0.0-3449.gd39fa088.el9.x86_64 146/164 2024-05-10T10:56:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-10T10:56:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:19.0.0-3449.gd39fa088.el9.x86_ 148/164 2024-05-10T10:56:43.533 DEBUG:teuthology.parallel:result is None 2024-05-10T10:56:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-10T10:56:43.607 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-10T10:56:44.752 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-10T10:56:44.812 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-10T10:56:44.875 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-10T10:56:46.322 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-top-2:19.0.0-3449.gd39fa088.el9.noarch 152/164 2024-05-10T10:56:46.460 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-10T10:56:46.537 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-10T10:56:46.686 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-10T10:56:46.728 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-10T10:56:46.766 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-10T10:56:46.767 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-10T10:56:46.919 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-10T10:56:46.965 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-10T10:56:46.966 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-10T10:56:47.111 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-10T10:56:47.149 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-10T10:56:47.182 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-10T10:56:47.182 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-10T10:56:47.537 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-10T10:56:47.537 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-10T10:56:47.717 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-10T10:56:47.841 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-10T10:56:48.128 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-10T10:56:48.129 INFO:teuthology.orchestra.run.smithi027.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-10T10:56:48.129 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:54.185 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:19.0.0-3449.gd39fa088.el9.x86_64 1/164 2024-05-10T10:56:54.185 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:19.0.0-3449.gd39fa088.el9.x86_64 2/164 2024-05-10T10:56:54.185 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 3/164 2024-05-10T10:56:54.186 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 4/164 2024-05-10T10:56:54.186 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3449.gd39fa 5/164 2024-05-10T10:56:54.186 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:19.0.0-3449.gd39fa088.el9.x86_64 6/164 2024-05-10T10:56:54.186 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:19.0.0-3449.gd39fa088.el9.x86_64 7/164 2024-05-10T10:56:54.186 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:19.0.0-3449.gd39fa088.el9.x86_64 8/164 2024-05-10T10:56:54.186 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:19.0.0-3449.gd39fa088.el9.x86_64 9/164 2024-05-10T10:56:54.186 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:19.0.0-3449.gd39fa088.el9.x86_64 10/164 2024-05-10T10:56:54.186 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:19.0.0-3449.gd39fa088.el9.x86_64 11/164 2024-05-10T10:56:54.186 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:19.0.0-3449.gd39fa088.el9.x86_64 12/164 2024-05-10T10:56:54.186 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 13/164 2024-05-10T10:56:54.186 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-shell-2:19.0.0-3449.gd39fa088.el9.x86_64 14/164 2024-05-10T10:56:54.186 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:19.0.0-3449.gd39fa088.el9.x86_ 15/164 2024-05-10T10:56:54.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:19.0.0-3449.gd39fa088.el9.x86_64 16/164 2024-05-10T10:56:54.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:19.0.0-3449.gd39fa088.el9.x86_64 17/164 2024-05-10T10:56:54.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:19.0.0-3449.gd39fa088.el9.x86_6 18/164 2024-05-10T10:56:54.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:19.0.0-3449.gd39fa088.el9.x86 19/164 2024-05-10T10:56:54.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:19.0.0-3449.gd39fa088.el9.x86_64 20/164 2024-05-10T10:56:54.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3449.gd39fa088.el 21/164 2024-05-10T10:56:54.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:19.0.0-3449.gd39fa088.el9. 22/164 2024-05-10T10:56:54.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:19.0.0-3449.gd39fa088.el9.x86_6 23/164 2024-05-10T10:56:54.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:19.0.0-3449.gd39fa088.el9.x86_64 24/164 2024-05-10T10:56:54.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:19.0.0-3449.gd39fa088.el9.x86_64 25/164 2024-05-10T10:56:54.188 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:19.0.0-3449.gd39fa088.el9.x86_64 26/164 2024-05-10T10:56:54.188 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 27/164 2024-05-10T10:56:54.188 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 28/164 2024-05-10T10:56:54.188 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:19.0.0-3449.gd39fa088.el9.x86_64 29/164 2024-05-10T10:56:54.188 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3449.gd39fa088. 30/164 2024-05-10T10:56:54.188 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3449.gd39fa088.el9.noa 31/164 2024-05-10T10:56:54.188 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3449.gd39fa088.el9.n 32/164 2024-05-10T10:56:54.189 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3449.gd39 33/164 2024-05-10T10:56:54.189 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3449.gd39fa088.el 34/164 2024-05-10T10:56:54.189 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3449.gd39fa088.el9.noarch 35/164 2024-05-10T10:56:54.189 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3449.gd39fa088.e 36/164 2024-05-10T10:56:54.189 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:19.0.0-3449.gd39fa088.el9.noarch 37/164 2024-05-10T10:56:54.189 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 38/164 2024-05-10T10:56:54.189 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-top-2:19.0.0-3449.gd39fa088.el9.noarch 39/164 2024-05-10T10:56:54.189 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-10T10:56:54.189 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-10T10:56:54.189 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-10T10:56:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-10T10:56:54.191 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-10T10:56:54.191 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-10T10:56:54.191 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-10T10:56:54.191 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-10T10:56:54.191 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-10T10:56:54.191 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-10T10:56:54.191 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-10T10:56:54.191 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-10T10:56:54.191 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-10T10:56:54.191 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-10T10:56:54.191 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-10T10:56:54.191 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-10T10:56:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-10T10:56:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-10T10:56:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-10T10:56:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-10T10:56:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-10T10:56:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-10T10:56:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-10T10:56:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-10T10:56:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-10T10:56:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-10T10:56:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-10T10:56:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-10T10:56:54.193 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-10T10:56:54.194 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-10T10:56:54.194 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-10T10:56:54.194 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-10T10:56:54.194 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-10T10:56:54.194 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-10T10:56:54.194 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-10T10:56:54.194 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-10T10:56:54.194 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-10T10:56:54.194 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-10T10:56:54.194 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-10T10:56:54.194 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-10T10:56:54.194 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-10T10:56:54.195 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-10T10:56:54.196 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-10T10:56:54.197 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:19.0.0-3449.gd39fa088.el9.x86_64 149/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:19.0.0-3449.gd39fa088.el9.x86_64 151/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-10T10:56:54.198 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-10T10:56:54.199 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-10T10:56:54.199 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-10T10:56:54.199 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-10T10:56:54.199 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-10T10:56:54.199 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-10T10:56:54.199 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-10T10:56:56.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-10T10:56:56.438 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:56.438 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-05-10T10:56:56.438 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-10T10:56:56.438 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-10T10:56:56.438 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-10T10:56:56.438 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-10T10:56:56.438 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.438 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.438 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-10T10:56:56.438 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-10T10:56:56.438 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: automake-1.16.2-8.el9.noarch 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: bison-3.7.4-5.el9.x86_64 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.439 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:56.440 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:56.440 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:56.440 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:56.440 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:56.440 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.440 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.440 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:56.440 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.440 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.440 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.440 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-shell-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top-2:19.0.0-3449.gd39fa088.el9.noarch 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: flex-2.6.4-9.el9.x86_64 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-10T10:56:56.441 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-10T10:56:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-10T10:56:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-10T10:56:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-10T10:56:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-10T10:56:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-10T10:56:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-10T10:56:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-10T10:56:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.442 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-10T10:56:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-10T10:56:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-10T10:56:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-10T10:56:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-10T10:56:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-10T10:56:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: lua-5.4.4-4.el9.x86_64 2024-05-10T10:56:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-10T10:56:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-10T10:56:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-10T10:56:56.443 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-13.14-1.el9.x86_64 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-10T10:56:56.444 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-10T10:56:56.445 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-10T10:56:56.446 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-10T10:56:56.446 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-10T10:56:56.446 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-10T10:56:56.446 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-10T10:56:56.446 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-10T10:56:56.446 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-10T10:56:56.446 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-10T10:56:56.446 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-10T10:56:56.446 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-10T10:56:56.446 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-10T10:56:56.446 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-10T10:56:56.446 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply-3.11-14.el9.noarch 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-10T10:56:56.447 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-10T10:56:56.448 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.449 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:19.0.0-3449.gd39fa088.el9.x86_64 2024-05-10T10:56:56.450 INFO:teuthology.orchestra.run.smithi027.stdout: re2-1:20211101-3.el9.x86_64 2024-05-10T10:56:56.450 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-10T10:56:56.450 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-10T10:56:56.450 INFO:teuthology.orchestra.run.smithi027.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-10T10:56:56.450 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-10T10:56:56.450 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-10T10:56:56.450 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-10T10:56:56.450 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:56:56.450 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-10T10:56:57.278 DEBUG:teuthology.parallel:result is None 2024-05-10T10:56:57.279 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T10:56:57.422 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-10T10:56:57.491 INFO:teuthology.orchestra.run.smithi027.stdout:19.0.0-3449.gd39fa088.el9 2024-05-10T10:56:57.492 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3449.gd39fa088.el9 2024-05-10T10:56:57.492 INFO:teuthology.task.install:The correct ceph version 19.0.0-3449.gd39fa088 is installed. 2024-05-10T10:56:57.494 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T10:56:57.708 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-10T10:56:57.733 INFO:teuthology.orchestra.run.smithi076.stdout:19.0.0-3449.gd39fa088.el9 2024-05-10T10:56:57.734 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3449.gd39fa088.el9 2024-05-10T10:56:57.734 INFO:teuthology.task.install:The correct ceph version 19.0.0-3449.gd39fa088 is installed. 2024-05-10T10:56:57.736 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T10:56:57.956 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-10T10:56:57.982 INFO:teuthology.orchestra.run.smithi189.stdout:19.0.0-3449.gd39fa088.el9 2024-05-10T10:56:57.982 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3449.gd39fa088.el9 2024-05-10T10:56:57.982 INFO:teuthology.task.install:The correct ceph version 19.0.0-3449.gd39fa088 is installed. 2024-05-10T10:56:57.984 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-10T10:56:57.984 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T10:56:57.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-10T10:56:58.060 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-05-10T10:56:58.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-10T10:56:58.090 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-10T10:56:58.091 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-10T10:56:58.121 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-10T10:56:58.122 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T10:56:58.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-05-10T10:56:58.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-10T10:56:58.301 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-05-10T10:56:58.302 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/daemon-helper 2024-05-10T10:56:58.330 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-10T10:56:58.396 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-10T10:56:58.396 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-05-10T10:56:58.426 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-10T10:56:58.492 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-10T10:56:58.492 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T10:56:58.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-10T10:56:58.557 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-10T10:56:58.661 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-05-10T10:56:58.661 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-10T10:56:58.686 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-10T10:56:58.761 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-10T10:56:58.761 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-10T10:56:58.786 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-10T10:56:58.855 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-10T10:56:58.855 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T10:56:58.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-05-10T10:56:58.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-10T10:56:59.025 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-05-10T10:56:59.025 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/stdin-killer 2024-05-10T10:56:59.049 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-10T10:56:59.113 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-10T10:56:59.113 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-05-10T10:56:59.137 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-10T10:56:59.200 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-10T10:56:59.304 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'd39fa08867836ac070c47cb66c782fa559fd480b', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-05-10T10:56:59.304 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T10:56:59.305 INFO:tasks.cephadm:Cluster fsid is 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:56:59.305 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-10T10:56:59.305 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.27', 'mon.b': '172.21.15.76', 'mon.c': '172.21.15.189'} 2024-05-10T10:56:59.305 INFO:tasks.cephadm:First mon is mon.a on smithi027 2024-05-10T10:56:59.305 INFO:tasks.cephadm:First mgr is x 2024-05-10T10:56:59.306 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-10T10:56:59.306 DEBUG:teuthology.orchestra.run.smithi027:> sudo hostname $(hostname -s) 2024-05-10T10:56:59.373 DEBUG:teuthology.orchestra.run.smithi076:> sudo hostname $(hostname -s) 2024-05-10T10:56:59.397 DEBUG:teuthology.orchestra.run.smithi189:> sudo hostname $(hostname -s) 2024-05-10T10:56:59.420 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-10T10:56:59.420 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T10:56:59.558 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd39fa08867836ac070c47cb66c782fa559fd480b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79273/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3449-gd39fa088', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3449.gd39fa088'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-10 00:52:03.112070', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-63896-from-wip-rishabh-testing-20240501.193033', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-10T10:56:59.774 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-63896-from-wip-rishabh-testing-20240501.193033, sha1 d39fa08867836ac070c47cb66c782fa559fd480b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T10:56:59.775 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/x86_64/flavors/default/cephadm 2024-05-10T10:56:59.776 DEBUG:teuthology.orchestra.run.smithi027:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-10T10:57:00.236 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r--. 1 ubuntu ubuntu 782738 May 10 10:57 /home/ubuntu/cephtest/cephadm 2024-05-10T10:57:00.236 DEBUG:teuthology.orchestra.run.smithi076:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-10T10:57:00.651 INFO:teuthology.orchestra.run.smithi076.stdout:-rw-r--r--. 1 ubuntu ubuntu 782738 May 10 10:57 /home/ubuntu/cephtest/cephadm 2024-05-10T10:57:00.651 DEBUG:teuthology.orchestra.run.smithi189:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-63896-from-wip-rishabh-testing-20240501.193033/d39fa08867836ac070c47cb66c782fa559fd480b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-10T10:57:01.255 INFO:teuthology.orchestra.run.smithi189.stdout:-rw-r--r--. 1 ubuntu ubuntu 782738 May 10 10:57 /home/ubuntu/cephtest/cephadm 2024-05-10T10:57:01.256 DEBUG:teuthology.orchestra.run.smithi027:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-10T10:57:01.305 DEBUG:teuthology.orchestra.run.smithi076:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-10T10:57:01.323 DEBUG:teuthology.orchestra.run.smithi189:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-10T10:57:01.359 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b on all hosts... 2024-05-10T10:57:01.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b pull 2024-05-10T10:57:01.366 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b pull 2024-05-10T10:57:01.369 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b pull 2024-05-10T10:57:02.905 INFO:teuthology.orchestra.run.smithi076.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b... 2024-05-10T10:57:02.998 INFO:teuthology.orchestra.run.smithi189.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b... 2024-05-10T10:57:04.448 INFO:teuthology.orchestra.run.smithi027.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b... 2024-05-10T10:57:31.683 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-05-10T10:57:31.683 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_version": "ceph version 19.0.0-3449-gd39fa088 (d39fa08867836ac070c47cb66c782fa559fd480b) squid (dev)", 2024-05-10T10:57:31.683 INFO:teuthology.orchestra.run.smithi076.stdout: "image_id": "ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5", 2024-05-10T10:57:31.683 INFO:teuthology.orchestra.run.smithi076.stdout: "repo_digests": [ 2024-05-10T10:57:31.684 INFO:teuthology.orchestra.run.smithi076.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f" 2024-05-10T10:57:31.684 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-05-10T10:57:31.684 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-05-10T10:57:34.739 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-05-10T10:57:34.739 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 19.0.0-3449-gd39fa088 (d39fa08867836ac070c47cb66c782fa559fd480b) squid (dev)", 2024-05-10T10:57:34.739 INFO:teuthology.orchestra.run.smithi189.stdout: "image_id": "ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5", 2024-05-10T10:57:34.740 INFO:teuthology.orchestra.run.smithi189.stdout: "repo_digests": [ 2024-05-10T10:57:34.740 INFO:teuthology.orchestra.run.smithi189.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f" 2024-05-10T10:57:34.740 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-05-10T10:57:34.740 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-05-10T10:57:47.785 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T10:57:47.786 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-3449-gd39fa088 (d39fa08867836ac070c47cb66c782fa559fd480b) squid (dev)", 2024-05-10T10:57:47.786 INFO:teuthology.orchestra.run.smithi027.stdout: "image_id": "ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5", 2024-05-10T10:57:47.786 INFO:teuthology.orchestra.run.smithi027.stdout: "repo_digests": [ 2024-05-10T10:57:47.786 INFO:teuthology.orchestra.run.smithi027.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f" 2024-05-10T10:57:47.786 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-10T10:57:47.786 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T10:57:47.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph 2024-05-10T10:57:47.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph 2024-05-10T10:57:47.980 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph 2024-05-10T10:57:48.012 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /etc/ceph 2024-05-10T10:57:48.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /etc/ceph 2024-05-10T10:57:48.133 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /etc/ceph 2024-05-10T10:57:48.161 INFO:tasks.cephadm:Writing seed config... 2024-05-10T10:57:48.162 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-10T10:57:48.163 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-10T10:57:48.164 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-10T10:57:48.165 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-05-10T10:57:48.166 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T10:57:48.166 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-10T10:57:48.226 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 063c9e18-0ebc-11ef-bc98-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-10T10:57:48.227 DEBUG:teuthology.orchestra.run.smithi027:mon.a> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.a.service 2024-05-10T10:57:48.272 DEBUG:teuthology.orchestra.run.smithi027:mgr.x> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.x.service 2024-05-10T10:57:48.318 INFO:tasks.cephadm:Bootstrapping... 2024-05-10T10:57:48.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b -v bootstrap --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.27 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-10T10:57:48.598 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-10T10:57:48.598 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b', '-v', 'bootstrap', '--fsid', '063c9e18-0ebc-11ef-bc98-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.27', '--skip-admin-label'] 2024-05-10T10:57:48.599 INFO:teuthology.orchestra.run.smithi027.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-10T10:57:48.599 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying podman|docker is present... 2024-05-10T10:57:48.656 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-10T10:57:48.656 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying lvm2 is present... 2024-05-10T10:57:48.657 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying time synchronization is in place... 2024-05-10T10:57:48.673 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-10T10:57:48.673 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-10T10:57:48.693 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-10T10:57:48.693 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-05-10T10:57:48.719 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-05-10T10:57:48.741 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-05-10T10:57:48.741 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-05-10T10:57:48.742 INFO:teuthology.orchestra.run.smithi027.stdout:Repeating the final host check... 2024-05-10T10:57:48.806 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-10T10:57:48.807 INFO:teuthology.orchestra.run.smithi027.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-10T10:57:48.807 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl is present 2024-05-10T10:57:48.807 INFO:teuthology.orchestra.run.smithi027.stdout:lvcreate is present 2024-05-10T10:57:48.831 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-10T10:57:48.831 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-10T10:57:48.854 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-10T10:57:48.855 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-05-10T10:57:48.882 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-05-10T10:57:48.905 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-05-10T10:57:48.906 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-05-10T10:57:48.906 INFO:teuthology.orchestra.run.smithi027.stdout:Host looks OK 2024-05-10T10:57:48.906 INFO:teuthology.orchestra.run.smithi027.stdout:Cluster fsid: 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:57:48.907 INFO:teuthology.orchestra.run.smithi027.stdout:Acquiring lock 140329822227904 on /run/cephadm/063c9e18-0ebc-11ef-bc98-c7b262605968.lock 2024-05-10T10:57:48.907 INFO:teuthology.orchestra.run.smithi027.stdout:Lock 140329822227904 acquired on /run/cephadm/063c9e18-0ebc-11ef-bc98-c7b262605968.lock 2024-05-10T10:57:48.909 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 3300 ... 2024-05-10T10:57:48.910 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 6789 ... 2024-05-10T10:57:48.910 INFO:teuthology.orchestra.run.smithi027.stdout:Base mon IP(s) is [172.21.15.27:3300, 172.21.15.27:6789], mon addrv is [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-05-10T10:57:48.927 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.27 metric 100 2024-05-10T10:57:48.927 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.27 metric 100 2024-05-10T10:57:48.940 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-10T10:57:48.941 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-05-10T10:57:48.941 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-05-10T10:57:48.953 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-10T10:57:48.953 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-10T10:57:48.954 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-10T10:57:48.954 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-05-10T10:57:48.954 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-05-10T10:57:48.954 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-10T10:57:48.957 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-05-10T10:57:48.957 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-05-10T10:57:48.957 INFO:teuthology.orchestra.run.smithi027.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-10T10:57:48.960 INFO:teuthology.orchestra.run.smithi027.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-10T10:57:48.963 INFO:teuthology.orchestra.run.smithi027.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b... 2024-05-10T10:57:49.680 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 2024-05-10T10:57:49.681 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b... 2024-05-10T10:57:49.681 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Getting image source signatures 2024-05-10T10:57:49.681 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:bf5b95bd68251da39c6cd2eb96c6ba4138bb37a62bdc39287908b25e92047d7e 2024-05-10T10:57:49.681 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:5401d9dfdf7b5b1eb30ca8a50398217fe5e68f9c8026e432d9728743b160edef 2024-05-10T10:57:49.681 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying config sha256:ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 2024-05-10T10:57:49.681 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-10T10:57:53.322 INFO:teuthology.orchestra.run.smithi027.stdout:ceph: stdout ceph version 19.0.0-3449-gd39fa088 (d39fa08867836ac070c47cb66c782fa559fd480b) squid (dev) 2024-05-10T10:57:53.323 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph version: ceph version 19.0.0-3449-gd39fa088 (d39fa08867836ac070c47cb66c782fa559fd480b) squid (dev) 2024-05-10T10:57:53.323 INFO:teuthology.orchestra.run.smithi027.stdout:Extracting ceph user uid/gid from container image... 2024-05-10T10:58:02.138 INFO:teuthology.orchestra.run.smithi027.stdout:stat: stdout 167 167 2024-05-10T10:58:02.138 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial keys... 2024-05-10T10:58:03.514 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQC6/T1m4djRKhAA7Y0sxHOH40OWT5Lh8+v0PA== 2024-05-10T10:58:04.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQC7/T1mrKgjOhAA9NgH/YxTav1Ozp800rEThQ== 2024-05-10T10:58:05.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQC9/T1mDIrKCRAA6jQWpt+mjfpV4qFGlaVAkQ== 2024-05-10T10:58:05.778 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial monmap... 2024-05-10T10:58:07.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-10T10:58:07.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-10T10:58:07.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:58:07.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-10T10:58:07.620 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool for a [v2:172.21.15.27:3300,v1:172.21.15.27:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-10T10:58:07.621 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-05-10T10:58:07.621 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: set fsid to 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:58:07.621 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-10T10:58:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:58:07.621 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mon... 2024-05-10T10:58:08.754 INFO:teuthology.orchestra.run.smithi027.stdout:create mon.a on 2024-05-10T10:58:09.204 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-10T10:58:09.459 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-10T10:58:09.770 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-063c9e18-0ebc-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-063c9e18-0ebc-11ef-bc98-c7b262605968.target. 2024-05-10T10:58:09.771 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-063c9e18-0ebc-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-063c9e18-0ebc-11ef-bc98-c7b262605968.target. 2024-05-10T10:58:10.178 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.a 2024-05-10T10:58:10.178 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.a.service: Unit ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.a.service not loaded. 2024-05-10T10:58:10.492 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-063c9e18-0ebc-11ef-bc98-c7b262605968.target.wants/ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.a.service → /etc/systemd/system/ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@.service. 2024-05-10T10:58:11.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:10 smithi027 podman[29665]: 2024-05-10 10:58:10.962542849 +0000 UTC m=+0.197325941 container create 97ee5cb265e2deeb517c341bdaa67d3fd9e4fb869f4239c8726ca38f1ada29b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-10T10:58:11.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:11 smithi027 podman[29665]: 2024-05-10 10:58:11.20777672 +0000 UTC m=+0.442559819 container init 97ee5cb265e2deeb517c341bdaa67d3fd9e4fb869f4239c8726ca38f1ada29b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-10T10:58:11.338 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-10T10:58:11.338 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-05-10T10:58:11.357 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-10T10:58:11.358 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-05-10T10:58:11.358 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-05-10T10:58:11.358 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-05-10T10:58:11.360 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon to start... 2024-05-10T10:58:11.360 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon... 2024-05-10T10:58:11.513 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:11 smithi027 podman[29665]: 2024-05-10 10:58:11.218830661 +0000 UTC m=+0.453613726 container start 97ee5cb265e2deeb517c341bdaa67d3fd9e4fb869f4239c8726ca38f1ada29b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-10T10:58:11.513 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:11 smithi027 ceph-mon[29683]: mkfs 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:58:11.513 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:11 smithi027 bash[29665]: 97ee5cb265e2deeb517c341bdaa67d3fd9e4fb869f4239c8726ca38f1ada29b5 2024-05-10T10:58:11.513 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:11 smithi027 ceph-mon[29683]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-10T10:58:11.513 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:11 smithi027 systemd[1]: Started Ceph mon.a for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T10:58:12.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:12 smithi027 ceph-mon[29683]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-10T10:58:12.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:12 smithi027 ceph-mon[29683]: monmap epoch 1 2024-05-10T10:58:12.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:12 smithi027 ceph-mon[29683]: fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:58:12.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:12 smithi027 ceph-mon[29683]: last_changed 2024-05-10T10:58:06.888512+0000 2024-05-10T10:58:12.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:12 smithi027 ceph-mon[29683]: created 2024-05-10T10:58:06.888512+0000 2024-05-10T10:58:12.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:12 smithi027 ceph-mon[29683]: min_mon_release 19 (squid) 2024-05-10T10:58:12.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:12 smithi027 ceph-mon[29683]: election_strategy: 1 2024-05-10T10:58:12.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:12 smithi027 ceph-mon[29683]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-05-10T10:58:12.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:12 smithi027 ceph-mon[29683]: fsmap 2024-05-10T10:58:12.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:12 smithi027 ceph-mon[29683]: osdmap e1: 0 total, 0 up, 0 in 2024-05-10T10:58:12.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:12 smithi027 ceph-mon[29683]: mgrmap e1: no daemons active 2024-05-10T10:58:12.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:12 smithi027 ceph-mon[29683]: from='client.? 172.21.15.27:0/1682468062' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-10T10:58:12.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout cluster: 2024-05-10T10:58:12.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout id: 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:58:12.706 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-10T10:58:12.707 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:12.707 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout services: 2024-05-10T10:58:12.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.733388s) 2024-05-10T10:58:12.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-10T10:58:12.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-10T10:58:12.713 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:12.713 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout data: 2024-05-10T10:58:12.713 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-10T10:58:12.713 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-10T10:58:12.713 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-10T10:58:12.713 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pgs: 2024-05-10T10:58:12.713 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:12.713 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.938+0000 7f9f99400700 1 Processor -- start 2024-05-10T10:58:12.713 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.938+0000 7f9f99400700 1 -- start start 2024-05-10T10:58:12.713 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.938+0000 7f9f99400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 0x7f9f94083030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:12.713 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.938+0000 7f9f99400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9f94083600 con 0x7f9f94082c10 2024-05-10T10:58:12.713 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.939+0000 7f9f93e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 0x7f9f94083030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.939+0000 7f9f93e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 0x7f9f94083030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35458/0 (socket says 172.21.15.27:35458) 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.939+0000 7f9f93e00700 1 -- 172.21.15.27:0/690413544 learned_addr learned my addr 172.21.15.27:0/690413544 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.940+0000 7f9f93e00700 1 -- 172.21.15.27:0/690413544 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f94110540 con 0x7f9f94082c10 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.940+0000 7f9f93e00700 1 --2- 172.21.15.27:0/690413544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 0x7f9f94083030 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9f840098a0 tx=0x7f9f84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c07af6a08fd57a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.940+0000 7f9f92a00700 1 -- 172.21.15.27:0/690413544 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f84004160 con 0x7f9f94082c10 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.941+0000 7f9f92a00700 1 -- 172.21.15.27:0/690413544 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f84004300 con 0x7f9f94082c10 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.941+0000 7f9f92a00700 1 -- 172.21.15.27:0/690413544 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f840036a0 con 0x7f9f94082c10 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.941+0000 7f9f99400700 1 -- 172.21.15.27:0/690413544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 msgr2=0x7f9f94083030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.941+0000 7f9f99400700 1 --2- 172.21.15.27:0/690413544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 0x7f9f94083030 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9f840098a0 tx=0x7f9f84009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.941+0000 7f9f99400700 1 -- 172.21.15.27:0/690413544 shutdown_connections 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.941+0000 7f9f99400700 1 --2- 172.21.15.27:0/690413544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 0x7f9f94083030 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.941+0000 7f9f99400700 1 -- 172.21.15.27:0/690413544 >> 172.21.15.27:0/690413544 conn(0x7f9f9407e230 msgr2=0x7f9f94080690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.942+0000 7f9f99400700 1 -- 172.21.15.27:0/690413544 shutdown_connections 2024-05-10T10:58:12.714 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.942+0000 7f9f99400700 1 -- 172.21.15.27:0/690413544 wait complete. 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.943+0000 7f9f99400700 1 Processor -- start 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.943+0000 7f9f99400700 1 -- start start 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.943+0000 7f9f99400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 0x7f9f941a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.943+0000 7f9f99400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9f94110a10 con 0x7f9f94082c10 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.943+0000 7f9f93e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 0x7f9f941a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.943+0000 7f9f93e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 0x7f9f941a2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35460/0 (socket says 172.21.15.27:35460) 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.943+0000 7f9f93e00700 1 -- 172.21.15.27:0/1682468062 learned_addr learned my addr 172.21.15.27:0/1682468062 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.944+0000 7f9f93e00700 1 -- 172.21.15.27:0/1682468062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f941a2e40 con 0x7f9f94082c10 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.944+0000 7f9f93e00700 1 --2- 172.21.15.27:0/1682468062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 0x7f9f941a2900 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9f84000c00 tx=0x7f9f84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.944+0000 7f9f90c00700 1 -- 172.21.15.27:0/1682468062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f84003db0 con 0x7f9f94082c10 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.945+0000 7f9f90c00700 1 -- 172.21.15.27:0/1682468062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f84011630 con 0x7f9f94082c10 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.945+0000 7f9f99400700 1 -- 172.21.15.27:0/1682468062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f941a30d0 con 0x7f9f94082c10 2024-05-10T10:58:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.945+0000 7f9f90c00700 1 -- 172.21.15.27:0/1682468062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f84011950 con 0x7f9f94082c10 2024-05-10T10:58:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.945+0000 7f9f99400700 1 -- 172.21.15.27:0/1682468062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f941a5dc0 con 0x7f9f94082c10 2024-05-10T10:58:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.945+0000 7f9f90c00700 1 -- 172.21.15.27:0/1682468062 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9f84011af0 con 0x7f9f94082c10 2024-05-10T10:58:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.945+0000 7f9f90c00700 1 -- 172.21.15.27:0/1682468062 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9f8401c450 con 0x7f9f94082c10 2024-05-10T10:58:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.946+0000 7f9f99400700 1 -- 172.21.15.27:0/1682468062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f9406a5d0 con 0x7f9f94082c10 2024-05-10T10:58:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:11.950+0000 7f9f90c00700 1 -- 172.21.15.27:0/1682468062 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9f84032020 con 0x7f9f94082c10 2024-05-10T10:58:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:12.030+0000 7f9f99400700 1 -- 172.21.15.27:0/1682468062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f9f9402e690 con 0x7f9f94082c10 2024-05-10T10:58:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:12.031+0000 7f9f90c00700 1 -- 172.21.15.27:0/1682468062 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f9f84004260 con 0x7f9f94082c10 2024-05-10T10:58:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:12.032+0000 7f9f99400700 1 -- 172.21.15.27:0/1682468062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 msgr2=0x7f9f941a2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:12.032+0000 7f9f99400700 1 --2- 172.21.15.27:0/1682468062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 0x7f9f941a2900 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9f84000c00 tx=0x7f9f84004750 comp rx=0 tx=0).stop 2024-05-10T10:58:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:12.032+0000 7f9f99400700 1 -- 172.21.15.27:0/1682468062 shutdown_connections 2024-05-10T10:58:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:12.032+0000 7f9f99400700 1 --2- 172.21.15.27:0/1682468062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f94082c10 0x7f9f941a2900 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:12.032+0000 7f9f99400700 1 -- 172.21.15.27:0/1682468062 >> 172.21.15.27:0/1682468062 conn(0x7f9f9407e230 msgr2=0x7f9f94199600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:12.032+0000 7f9f99400700 1 -- 172.21.15.27:0/1682468062 shutdown_connections 2024-05-10T10:58:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:12.033+0000 7f9f99400700 1 -- 172.21.15.27:0/1682468062 wait complete. 2024-05-10T10:58:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:mon is available 2024-05-10T10:58:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:Assimilating anything we can from ceph.conf... 2024-05-10T10:58:13.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:13 smithi027 ceph-mon[29683]: from='client.? 172.21.15.27:0/3260867509' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-10T10:58:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:13 smithi027 ceph-mon[29683]: from='client.? 172.21.15.27:0/3260867509' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-10T10:58:14.024 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:14.024 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-05-10T10:58:14.024 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:58:14.024 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-10T10:58:14.024 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-05-10T10:58:14.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-10T10:58:14.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-10T10:58:14.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-10T10:58:14.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-10T10:58:14.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:14.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-10T10:58:14.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-10T10:58:14.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:14.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-05-10T10:58:14.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-10T10:58:14.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-10T10:58:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-10T10:58:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-10T10:58:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-10T10:58:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.345+0000 7fa642400700 1 Processor -- start 2024-05-10T10:58:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.345+0000 7fa642400700 1 -- start start 2024-05-10T10:58:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.345+0000 7fa642400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 0x7fa63c107fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.345+0000 7fa642400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c108570 con 0x7fa63c107b80 2024-05-10T10:58:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.346+0000 7fa641000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 0x7fa63c107fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.346+0000 7fa641000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 0x7fa63c107fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35468/0 (socket says 172.21.15.27:35468) 2024-05-10T10:58:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.346+0000 7fa641000700 1 -- 172.21.15.27:0/2396210435 learned_addr learned my addr 172.21.15.27:0/2396210435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.346+0000 7fa641000700 1 -- 172.21.15.27:0/2396210435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63c108d80 con 0x7fa63c107b80 2024-05-10T10:58:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.347+0000 7fa641000700 1 --2- 172.21.15.27:0/2396210435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 0x7fa63c107fa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa6300098a0 tx=0x7fa630009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cefffe2ed649ad server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:14.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.347+0000 7fa63b400700 1 -- 172.21.15.27:0/2396210435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa630004160 con 0x7fa63c107b80 2024-05-10T10:58:14.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.347+0000 7fa63b400700 1 -- 172.21.15.27:0/2396210435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa630004300 con 0x7fa63c107b80 2024-05-10T10:58:14.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.347+0000 7fa63b400700 1 -- 172.21.15.27:0/2396210435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6300036a0 con 0x7fa63c107b80 2024-05-10T10:58:14.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.348+0000 7fa642400700 1 -- 172.21.15.27:0/2396210435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 msgr2=0x7fa63c107fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:14.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.348+0000 7fa642400700 1 --2- 172.21.15.27:0/2396210435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 0x7fa63c107fa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa6300098a0 tx=0x7fa630009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:14.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.348+0000 7fa642400700 1 -- 172.21.15.27:0/2396210435 shutdown_connections 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.348+0000 7fa642400700 1 --2- 172.21.15.27:0/2396210435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 0x7fa63c107fa0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.348+0000 7fa642400700 1 -- 172.21.15.27:0/2396210435 >> 172.21.15.27:0/2396210435 conn(0x7fa63c103100 msgr2=0x7fa63c105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.348+0000 7fa642400700 1 -- 172.21.15.27:0/2396210435 shutdown_connections 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.348+0000 7fa642400700 1 -- 172.21.15.27:0/2396210435 wait complete. 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.349+0000 7fa642400700 1 Processor -- start 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.349+0000 7fa642400700 1 -- start start 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.350+0000 7fa642400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 0x7fa63c19c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.350+0000 7fa642400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c1092b0 con 0x7fa63c107b80 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.350+0000 7fa641000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 0x7fa63c19c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.350+0000 7fa641000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 0x7fa63c19c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35484/0 (socket says 172.21.15.27:35484) 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.350+0000 7fa641000700 1 -- 172.21.15.27:0/3260867509 learned_addr learned my addr 172.21.15.27:0/3260867509 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.350+0000 7fa641000700 1 -- 172.21.15.27:0/3260867509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63c19c8a0 con 0x7fa63c107b80 2024-05-10T10:58:14.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.351+0000 7fa641000700 1 --2- 172.21.15.27:0/3260867509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 0x7fa63c19c360 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa630000c00 tx=0x7fa630004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.351+0000 7fa639600700 1 -- 172.21.15.27:0/3260867509 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa630003db0 con 0x7fa63c107b80 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.351+0000 7fa639600700 1 -- 172.21.15.27:0/3260867509 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa630011630 con 0x7fa63c107b80 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.351+0000 7fa639600700 1 -- 172.21.15.27:0/3260867509 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa630011950 con 0x7fa63c107b80 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.351+0000 7fa642400700 1 -- 172.21.15.27:0/3260867509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63c19cb30 con 0x7fa63c107b80 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.351+0000 7fa642400700 1 -- 172.21.15.27:0/3260867509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa63c19f7d0 con 0x7fa63c107b80 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.351+0000 7fa639600700 1 -- 172.21.15.27:0/3260867509 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa630011af0 con 0x7fa63c107b80 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.351+0000 7fa639600700 1 -- 172.21.15.27:0/3260867509 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa63001c450 con 0x7fa63c107b80 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.352+0000 7fa642400700 1 -- 172.21.15.27:0/3260867509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa63c06a5d0 con 0x7fa63c107b80 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.355+0000 7fa639600700 1 -- 172.21.15.27:0/3260867509 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa630032020 con 0x7fa63c107b80 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.435+0000 7fa642400700 1 -- 172.21.15.27:0/3260867509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fa63c02e690 con 0x7fa63c107b80 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.445+0000 7fa639600700 1 -- 172.21.15.27:0/3260867509 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7fa630004260 con 0x7fa63c107b80 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.446+0000 7fa639600700 1 -- 172.21.15.27:0/3260867509 <== mon.0 v2:172.21.15.27:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa63001c6a0 con 0x7fa63c107b80 2024-05-10T10:58:14.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.449+0000 7fa642400700 1 -- 172.21.15.27:0/3260867509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 msgr2=0x7fa63c19c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:14.034 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.449+0000 7fa642400700 1 --2- 172.21.15.27:0/3260867509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 0x7fa63c19c360 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa630000c00 tx=0x7fa630004750 comp rx=0 tx=0).stop 2024-05-10T10:58:14.034 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.449+0000 7fa642400700 1 -- 172.21.15.27:0/3260867509 shutdown_connections 2024-05-10T10:58:14.034 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.449+0000 7fa642400700 1 --2- 172.21.15.27:0/3260867509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63c107b80 0x7fa63c19c360 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:14.034 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.449+0000 7fa642400700 1 -- 172.21.15.27:0/3260867509 >> 172.21.15.27:0/3260867509 conn(0x7fa63c103100 msgr2=0x7fa63c192fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:14.034 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.449+0000 7fa642400700 1 -- 172.21.15.27:0/3260867509 shutdown_connections 2024-05-10T10:58:14.034 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:13.449+0000 7fa642400700 1 -- 172.21.15.27:0/3260867509 wait complete. 2024-05-10T10:58:14.034 INFO:teuthology.orchestra.run.smithi027.stdout:Generating new minimal ceph.conf... 2024-05-10T10:58:15.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:14 smithi027 ceph-mon[29683]: from='client.? 172.21.15.27:0/149383320' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T10:58:15.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.699+0000 7feccfe00700 1 Processor -- start 2024-05-10T10:58:15.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.700+0000 7feccfe00700 1 -- start start 2024-05-10T10:58:15.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.700+0000 7feccfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 0x7fecd0107270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:15.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.700+0000 7feccfe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fecd0076270 con 0x7fecd0075dc0 2024-05-10T10:58:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.701+0000 7feccea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 0x7fecd0107270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.701+0000 7feccea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 0x7fecd0107270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45260/0 (socket says 172.21.15.27:45260) 2024-05-10T10:58:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.701+0000 7feccea00700 1 -- 172.21.15.27:0/18971724 learned_addr learned my addr 172.21.15.27:0/18971724 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.701+0000 7feccea00700 1 -- 172.21.15.27:0/18971724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd00763f0 con 0x7fecd0075dc0 2024-05-10T10:58:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.701+0000 7feccea00700 1 --2- 172.21.15.27:0/18971724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 0x7fecd0107270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fecc00098a0 tx=0x7fecc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c77718f19c9f75 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.702+0000 7feccd600700 1 -- 172.21.15.27:0/18971724 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fecc0004160 con 0x7fecd0075dc0 2024-05-10T10:58:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.702+0000 7feccd600700 1 -- 172.21.15.27:0/18971724 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fecc0004300 con 0x7fecd0075dc0 2024-05-10T10:58:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.702+0000 7feccd600700 1 -- 172.21.15.27:0/18971724 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fecc0011af0 con 0x7fecd0075dc0 2024-05-10T10:58:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.703+0000 7feccfe00700 1 -- 172.21.15.27:0/18971724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 msgr2=0x7fecd0107270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.703+0000 7feccfe00700 1 --2- 172.21.15.27:0/18971724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 0x7fecd0107270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fecc00098a0 tx=0x7fecc0009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.703+0000 7feccfe00700 1 -- 172.21.15.27:0/18971724 shutdown_connections 2024-05-10T10:58:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.703+0000 7feccfe00700 1 --2- 172.21.15.27:0/18971724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 0x7fecd0107270 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.703+0000 7feccfe00700 1 -- 172.21.15.27:0/18971724 >> 172.21.15.27:0/18971724 conn(0x7fecd01028a0 msgr2=0x7fecd0104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.703+0000 7feccfe00700 1 -- 172.21.15.27:0/18971724 shutdown_connections 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.703+0000 7feccfe00700 1 -- 172.21.15.27:0/18971724 wait complete. 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.704+0000 7feccfe00700 1 Processor -- start 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.704+0000 7feccfe00700 1 -- start start 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.704+0000 7feccfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 0x7fecd019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.704+0000 7feccfe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fecd0108c30 con 0x7fecd0075dc0 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.705+0000 7feccea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 0x7fecd019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.705+0000 7feccea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 0x7fecd019ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45262/0 (socket says 172.21.15.27:45262) 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.705+0000 7feccea00700 1 -- 172.21.15.27:0/149383320 learned_addr learned my addr 172.21.15.27:0/149383320 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.705+0000 7feccea00700 1 -- 172.21.15.27:0/149383320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd019f290 con 0x7fecd0075dc0 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.706+0000 7feccea00700 1 --2- 172.21.15.27:0/149383320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 0x7fecd019ed50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fecc0003f40 tx=0x7fecc0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.706+0000 7fecd4a00700 1 -- 172.21.15.27:0/149383320 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fecc00038b0 con 0x7fecd0075dc0 2024-05-10T10:58:15.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.706+0000 7fecd4a00700 1 -- 172.21.15.27:0/149383320 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fecc0003a50 con 0x7fecd0075dc0 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.706+0000 7fecd4a00700 1 -- 172.21.15.27:0/149383320 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fecc001a5c0 con 0x7fecd0075dc0 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.706+0000 7feccfe00700 1 -- 172.21.15.27:0/149383320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecd019f520 con 0x7fecd0075dc0 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.706+0000 7feccfe00700 1 -- 172.21.15.27:0/149383320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd01a2210 con 0x7fecd0075dc0 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.707+0000 7fecd4a00700 1 -- 172.21.15.27:0/149383320 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fecc001ac00 con 0x7fecd0075dc0 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.707+0000 7fecd4a00700 1 -- 172.21.15.27:0/149383320 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fecc0019700 con 0x7fecd0075dc0 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.708+0000 7feccfe00700 1 -- 172.21.15.27:0/149383320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecd006a5d0 con 0x7fecd0075dc0 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.711+0000 7fecd4a00700 1 -- 172.21.15.27:0/149383320 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fecc00199a0 con 0x7fecd0075dc0 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.788+0000 7feccfe00700 1 -- 172.21.15.27:0/149383320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fecd002e800 con 0x7fecd0075dc0 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.789+0000 7fecd4a00700 1 -- 172.21.15.27:0/149383320 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fecc0019b80 con 0x7fecd0075dc0 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.790+0000 7feccfe00700 1 -- 172.21.15.27:0/149383320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 msgr2=0x7fecd019ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.790+0000 7feccfe00700 1 --2- 172.21.15.27:0/149383320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 0x7fecd019ed50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fecc0003f40 tx=0x7fecc0011890 comp rx=0 tx=0).stop 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.791+0000 7feccfe00700 1 -- 172.21.15.27:0/149383320 shutdown_connections 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.791+0000 7feccfe00700 1 --2- 172.21.15.27:0/149383320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd0075dc0 0x7fecd019ed50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.791+0000 7feccfe00700 1 -- 172.21.15.27:0/149383320 >> 172.21.15.27:0/149383320 conn(0x7fecd01028a0 msgr2=0x7fecd0195b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:15.465 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.791+0000 7feccfe00700 1 -- 172.21.15.27:0/149383320 shutdown_connections 2024-05-10T10:58:15.465 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:14.791+0000 7feccfe00700 1 -- 172.21.15.27:0/149383320 wait complete. 2024-05-10T10:58:15.465 INFO:teuthology.orchestra.run.smithi027.stdout:Restarting the monitor... 2024-05-10T10:58:15.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:15 smithi027 systemd[1]: Stopping Ceph mon.a for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T10:58:16.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:15 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a[29679]: 2024-05-10T10:58:15.976+0000 7febe1800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T10:58:16.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:15 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a[29679]: 2024-05-10T10:58:15.976+0000 7febe1800700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-10T10:58:16.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:16 smithi027 podman[29924]: 2024-05-10 10:58:16.018804367 +0000 UTC m=+0.162457463 container died 97ee5cb265e2deeb517c341bdaa67d3fd9e4fb869f4239c8726ca38f1ada29b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, ceph=True) 2024-05-10T10:58:16.906 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:16 smithi027 podman[29924]: 2024-05-10 10:58:16.701151736 +0000 UTC m=+0.844804829 container cleanup 97ee5cb265e2deeb517c341bdaa67d3fd9e4fb869f4239c8726ca38f1ada29b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux ) 2024-05-10T10:58:16.906 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:16 smithi027 bash[29924]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a 2024-05-10T10:58:17.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:17 smithi027 podman[29937]: 2024-05-10 10:58:17.01262195 +0000 UTC m=+0.973875622 container remove 97ee5cb265e2deeb517c341bdaa67d3fd9e4fb869f4239c8726ca38f1ada29b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-10T10:58:17.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:17 smithi027 systemd[1]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-10T10:58:17.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:17 smithi027 systemd[1]: Stopped Ceph mon.a for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T10:58:17.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:17 smithi027 systemd[1]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.a.service: Consumed 1.848s CPU time. 2024-05-10T10:58:17.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:17 smithi027 systemd[1]: Starting Ceph mon.a for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T10:58:17.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:17 smithi027 podman[30034]: 2024-05-10 10:58:17.567453916 +0000 UTC m=+0.042948597 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T10:58:17.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:17 smithi027 podman[30034]: 2024-05-10 10:58:17.738368532 +0000 UTC m=+0.213863249 container create 36f5ec0cc8e03f01a68dfa9bcb3f3cb19ae644191fc8fe46296d0323e4fbfa9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-10T10:58:17.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:17 smithi027 podman[30034]: 2024-05-10 10:58:17.94153555 +0000 UTC m=+0.417030300 container init 36f5ec0cc8e03f01a68dfa9bcb3f3cb19ae644191fc8fe46296d0323e4fbfa9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-10T10:58:18.026 INFO:teuthology.orchestra.run.smithi027.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-10T10:58:18.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:17 smithi027 podman[30034]: 2024-05-10 10:58:17.950159165 +0000 UTC m=+0.425653838 container start 36f5ec0cc8e03f01a68dfa9bcb3f3cb19ae644191fc8fe46296d0323e4fbfa9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-10T10:58:18.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:17 smithi027 ceph-mon[30054]: set uid:gid to 167:167 (ceph:ceph) 2024-05-10T10:58:18.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:17 smithi027 ceph-mon[30054]: ceph version 19.0.0-3449-gd39fa088 (d39fa08867836ac070c47cb66c782fa559fd480b) squid (dev), process ceph-mon, pid 2 2024-05-10T10:58:18.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:17 smithi027 ceph-mon[30054]: pidfile_write: ignore empty --pid-file 2024-05-10T10:58:18.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: load: jerasure load: lrc 2024-05-10T10:58:18.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: RocksDB version: 7.9.2 2024-05-10T10:58:18.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Git sha 0 2024-05-10T10:58:18.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Compile date 2024-05-10 01:38:55 2024-05-10T10:58:18.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: DB SUMMARY 2024-05-10T10:58:18.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: DB Session ID: ZDD49TJ595DM4L8D4AE7 2024-05-10T10:58:18.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: CURRENT file: CURRENT 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: IDENTITY file: IDENTITY 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105126 ; 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.error_if_exists: 0 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.create_if_missing: 0 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.paranoid_checks: 1 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.env: 0x55a239513cc0 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.fs: PosixFileSystem 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.info_log: 0x55a25d4dfde0 2024-05-10T10:58:18.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_file_opening_threads: 16 2024-05-10T10:58:18.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.statistics: (nil) 2024-05-10T10:58:18.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.use_fsync: 0 2024-05-10T10:58:18.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_log_file_size: 0 2024-05-10T10:58:18.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-10T10:58:18.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-10T10:58:18.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.keep_log_file_num: 1000 2024-05-10T10:58:18.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.recycle_log_file_num: 0 2024-05-10T10:58:18.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.allow_fallocate: 1 2024-05-10T10:58:18.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.allow_mmap_reads: 0 2024-05-10T10:58:18.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.allow_mmap_writes: 0 2024-05-10T10:58:18.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.use_direct_reads: 0 2024-05-10T10:58:18.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-10T10:58:18.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.create_missing_column_families: 0 2024-05-10T10:58:18.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.db_log_dir: 2024-05-10T10:58:18.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.wal_dir: 2024-05-10T10:58:18.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-10T10:58:18.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-10T10:58:18.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-10T10:58:18.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.advise_random_on_open: 1 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.db_write_buffer_size: 0 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.write_buffer_manager: 0x55a25d4b10e0 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.rate_limiter: (nil) 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.wal_recovery_mode: 2 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.enable_thread_tracking: 0 2024-05-10T10:58:18.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.enable_pipelined_write: 0 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.unordered_write: 0 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.row_cache: None 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.wal_filter: None 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.allow_ingest_behind: 0 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.two_write_queues: 0 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.manual_wal_flush: 0 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.wal_compression: 0 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.atomic_flush: 0 2024-05-10T10:58:18.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.log_readahead_size: 0 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.best_efforts_recovery: 0 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.allow_data_in_errors: 0 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.db_host_id: __hostname__ 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_background_jobs: 2 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_background_compactions: -1 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_subcompactions: 1 2024-05-10T10:58:18.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_total_wal_size: 0 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_open_files: -1 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bytes_per_sync: 0 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_readahead_size: 0 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_background_flushes: -1 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Compression algorithms supported: 2024-05-10T10:58:18.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: kZSTD supported: 0 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: kXpressCompression supported: 0 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: kLZ4HCCompression supported: 1 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: kZlibCompression supported: 1 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: kSnappyCompression supported: 1 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: kLZ4Compression supported: 1 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: kBZip2Compression supported: 0 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.merge_operator: 2024-05-10T10:58:18.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_filter: None 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_filter_factory: None 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.sst_partitioner_factory: None 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a25d4dffa0) 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks: 1 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_top_level_index_and_filter: 1 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_type: 0 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_index_type: 0 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_shortening: 1 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-10T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi027.stdout: checksum: 4 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: no_block_cache: 0 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache: 0x55a25d4e31f0 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_name: BinnedLRUCache 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_options: 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: capacity : 536870912 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_shard_bits : 4 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: strict_capacity_limit : 0 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: high_pri_pool_ratio: 0.000 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_compressed: (nil) 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: persistent_cache: (nil) 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size: 4096 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size_deviation: 10 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_restart_interval: 16 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_block_restart_interval: 1 2024-05-10T10:58:18.212 INFO:journalctl@ceph.mon.a.smithi027.stdout: metadata_block_size: 4096 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: partition_filters: 0 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: use_delta_encoding: 1 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: filter_policy: bloomfilter 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: whole_key_filtering: 1 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: verify_compression: 0 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: read_amp_bytes_per_bit: 0 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: format_version: 5 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_index_compression: 1 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_align: 0 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: max_auto_readahead_size: 262144 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: prepopulate_block_cache: 0 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: initial_auto_readahead_size: 8192 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_file_reads_for_auto_readahead: 2 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.write_buffer_size: 33554432 2024-05-10T10:58:18.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_write_buffer_number: 2 2024-05-10T10:58:18.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compression: NoCompression 2024-05-10T10:58:18.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bottommost_compression: Disabled 2024-05-10T10:58:18.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.prefix_extractor: nullptr 2024-05-10T10:58:18.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-10T10:58:18.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.num_levels: 7 2024-05-10T10:58:18.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-10T10:58:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-10T10:58:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-10T10:58:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-10T10:58:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-10T10:58:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-10T10:58:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-10T10:58:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-10T10:58:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-10T10:58:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-10T10:58:18.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compression_opts.level: 32767 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compression_opts.strategy: 0 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compression_opts.enabled: false 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-10T10:58:18.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.target_file_size_base: 67108864 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-10T10:58:18.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.arena_block_size: 1048576 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.disable_auto_compactions: 0 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-10T10:58:18.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.inplace_update_support: 0 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.bloom_locality: 0 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.max_successive_merges: 0 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.paranoid_file_checks: 0 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.force_consistency_checks: 1 2024-05-10T10:58:18.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.report_bg_io_stats: 0 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.ttl: 2592000 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.enable_blob_files: false 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.min_blob_size: 0 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.blob_file_size: 268435456 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.blob_file_starting_level: 0 2024-05-10T10:58:18.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0e8f53f9-d4b4-484b-83a2-42e87ea2fadc 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715338698008187, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715338698010140, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100293, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 98178, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14169, "raw_average_key_size": 49, "raw_value_size": 90462, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715338698, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0e8f53f9-d4b4-484b-83a2-42e87ea2fadc", "db_session_id": "ZDD49TJ595DM4L8D4AE7", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715338698010321, "job": 1, "event": "recovery_finished"} 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a25d600000 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: DB pointer 0x55a25d5ec000 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-10T10:58:18.221 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** DB Stats ** 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: L0 2/0 99.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 79.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-10T10:58:18.222 INFO:journalctl@ceph.mon.a.smithi027.stdout: Sum 2/0 99.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 79.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 79.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 79.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-10T10:58:18.223 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative compaction: 0.00 GB write, 13.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval compaction: 0.00 GB write, 13.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache BinnedLRUCache@0x55a25d4e31f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: starting mon.a rank 0 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 bash[30034]: 36f5ec0cc8e03f01a68dfa9bcb3f3cb19ae644191fc8fe46296d0323e4fbfa9d 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: mon.a@-1(???) e1 preinit fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: mon.a@-1(???).mds e1 new map 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: mon.a@-1(???).mds e1 print_map 2024-05-10T10:58:18.224 INFO:journalctl@ceph.mon.a.smithi027.stdout: e1 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout: legacy client fscid: -1 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout: No filesystems configured 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 systemd[1]: Started Ceph mon.a for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: monmap epoch 1 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:58:18.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: last_changed 2024-05-10T10:58:06.888512+0000 2024-05-10T10:58:18.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: created 2024-05-10T10:58:06.888512+0000 2024-05-10T10:58:18.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: min_mon_release 19 (squid) 2024-05-10T10:58:18.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: election_strategy: 1 2024-05-10T10:58:18.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-05-10T10:58:18.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: fsmap 2024-05-10T10:58:18.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: osdmap e1: 0 total, 0 up, 0 in 2024-05-10T10:58:18.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:18 smithi027 ceph-mon[30054]: mgrmap e1: no daemons active 2024-05-10T10:58:19.480 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.729+0000 7f4b28c00700 1 Processor -- start 2024-05-10T10:58:19.480 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.729+0000 7f4b28c00700 1 -- start start 2024-05-10T10:58:19.480 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.730+0000 7f4b28c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 0x7f4b2410bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:19.480 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.730+0000 7f4b28c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2407d6c0 con 0x7f4b24109950 2024-05-10T10:58:19.480 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.731+0000 7f4b21e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 0x7f4b2410bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:19.481 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.731+0000 7f4b21e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 0x7f4b2410bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45266/0 (socket says 172.21.15.27:45266) 2024-05-10T10:58:19.481 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.731+0000 7f4b21e00700 1 -- 172.21.15.27:0/171319669 learned_addr learned my addr 172.21.15.27:0/171319669 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:19.481 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.732+0000 7f4b21e00700 1 -- 172.21.15.27:0/171319669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2410c280 con 0x7f4b24109950 2024-05-10T10:58:19.481 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.732+0000 7f4b21e00700 1 --2- 172.21.15.27:0/171319669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 0x7f4b2410bd40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4b140098a0 tx=0x7f4b14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7fcd619d0a89f8e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:19.481 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.733+0000 7f4b20a00700 1 -- 172.21.15.27:0/171319669 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b14004160 con 0x7f4b24109950 2024-05-10T10:58:19.481 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.733+0000 7f4b20a00700 1 -- 172.21.15.27:0/171319669 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4b14004300 con 0x7f4b24109950 2024-05-10T10:58:19.482 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.733+0000 7f4b20a00700 1 -- 172.21.15.27:0/171319669 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b14011af0 con 0x7f4b24109950 2024-05-10T10:58:19.482 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.733+0000 7f4b28c00700 1 -- 172.21.15.27:0/171319669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 msgr2=0x7f4b2410bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:19.482 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.733+0000 7f4b28c00700 1 --2- 172.21.15.27:0/171319669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 0x7f4b2410bd40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4b140098a0 tx=0x7f4b14009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:19.482 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.734+0000 7f4b28c00700 1 -- 172.21.15.27:0/171319669 shutdown_connections 2024-05-10T10:58:19.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.734+0000 7f4b28c00700 1 --2- 172.21.15.27:0/171319669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 0x7f4b2410bd40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:19.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.734+0000 7f4b28c00700 1 -- 172.21.15.27:0/171319669 >> 172.21.15.27:0/171319669 conn(0x7f4b24103360 msgr2=0x7f4b24105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:19.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.734+0000 7f4b28c00700 1 -- 172.21.15.27:0/171319669 shutdown_connections 2024-05-10T10:58:19.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.734+0000 7f4b28c00700 1 -- 172.21.15.27:0/171319669 wait complete. 2024-05-10T10:58:19.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.735+0000 7f4b28c00700 1 Processor -- start 2024-05-10T10:58:19.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.735+0000 7f4b28c00700 1 -- start start 2024-05-10T10:58:19.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.735+0000 7f4b28c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 0x7f4b2419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:19.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.735+0000 7f4b28c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2410d1f0 con 0x7f4b24109950 2024-05-10T10:58:19.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.735+0000 7f4b21e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 0x7f4b2419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:19.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.735+0000 7f4b21e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 0x7f4b2419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45276/0 (socket says 172.21.15.27:45276) 2024-05-10T10:58:19.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.736+0000 7f4b21e00700 1 -- 172.21.15.27:0/4016271718 learned_addr learned my addr 172.21.15.27:0/4016271718 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:19.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.736+0000 7f4b21e00700 1 -- 172.21.15.27:0/4016271718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2419f4f0 con 0x7f4b24109950 2024-05-10T10:58:19.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.736+0000 7f4b21e00700 1 --2- 172.21.15.27:0/4016271718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 0x7f4b2419efb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4b14011ac0 tx=0x7f4b14011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:19.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.736+0000 7f4b1aa00700 1 -- 172.21.15.27:0/4016271718 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b14003b20 con 0x7f4b24109950 2024-05-10T10:58:19.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.737+0000 7f4b1aa00700 1 -- 172.21.15.27:0/4016271718 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4b1401a8c0 con 0x7f4b24109950 2024-05-10T10:58:19.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.737+0000 7f4b28c00700 1 -- 172.21.15.27:0/4016271718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b2419f780 con 0x7f4b24109950 2024-05-10T10:58:19.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.737+0000 7f4b1aa00700 1 -- 172.21.15.27:0/4016271718 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b140248d0 con 0x7f4b24109950 2024-05-10T10:58:19.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.737+0000 7f4b28c00700 1 -- 172.21.15.27:0/4016271718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b241a02a0 con 0x7f4b24109950 2024-05-10T10:58:19.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.738+0000 7f4b1aa00700 1 -- 172.21.15.27:0/4016271718 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4b14022020 con 0x7f4b24109950 2024-05-10T10:58:19.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.738+0000 7f4b1aa00700 1 -- 172.21.15.27:0/4016271718 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4b1402d9c0 con 0x7f4b24109950 2024-05-10T10:58:19.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.738+0000 7f4b28c00700 1 -- 172.21.15.27:0/4016271718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b24064b00 con 0x7f4b24109950 2024-05-10T10:58:19.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.741+0000 7f4b1aa00700 1 -- 172.21.15.27:0/4016271718 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4b1401aa70 con 0x7f4b24109950 2024-05-10T10:58:19.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.815+0000 7f4b28c00700 1 -- 172.21.15.27:0/4016271718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f4b2406bcb0 con 0x7f4b24109950 2024-05-10T10:58:19.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.818+0000 7f4b1aa00700 1 -- 172.21.15.27:0/4016271718 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f4b1402d3f0 con 0x7f4b24109950 2024-05-10T10:58:19.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.819+0000 7f4b28c00700 1 -- 172.21.15.27:0/4016271718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 msgr2=0x7f4b2419efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:19.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.819+0000 7f4b28c00700 1 --2- 172.21.15.27:0/4016271718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 0x7f4b2419efb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4b14011ac0 tx=0x7f4b14011ba0 comp rx=0 tx=0).stop 2024-05-10T10:58:19.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.819+0000 7f4b28c00700 1 -- 172.21.15.27:0/4016271718 shutdown_connections 2024-05-10T10:58:19.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.819+0000 7f4b28c00700 1 --2- 172.21.15.27:0/4016271718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b24109950 0x7f4b2419efb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:19.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.819+0000 7f4b28c00700 1 -- 172.21.15.27:0/4016271718 >> 172.21.15.27:0/4016271718 conn(0x7f4b24103360 msgr2=0x7f4b24103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:19.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.819+0000 7f4b28c00700 1 -- 172.21.15.27:0/4016271718 shutdown_connections 2024-05-10T10:58:19.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:18.820+0000 7f4b28c00700 1 -- 172.21.15.27:0/4016271718 wait complete. 2024-05-10T10:58:19.486 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-10T10:58:19.505 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-10T10:58:19.505 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mgr... 2024-05-10T10:58:19.506 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-10T10:58:19.507 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-10T10:58:19.946 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.x 2024-05-10T10:58:19.946 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.x.service: Unit ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.x.service not loaded. 2024-05-10T10:58:19.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:19 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4016271718' entity='client.admin' 2024-05-10T10:58:20.214 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-063c9e18-0ebc-11ef-bc98-c7b262605968.target.wants/ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@.service. 2024-05-10T10:58:20.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:20 smithi027 systemd[1]: Starting Ceph mgr.x for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T10:58:20.879 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:20 smithi027 podman[30269]: 2024-05-10 10:58:20.523668385 +0000 UTC m=+0.035261939 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T10:58:20.879 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:20 smithi027 podman[30269]: 2024-05-10 10:58:20.62507177 +0000 UTC m=+0.136665357 container create 6a7db7f103443646d6582db1deeb2c50d544ab2c1d1adfb11df37f9eb19cca1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-10T10:58:20.879 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:20 smithi027 podman[30269]: 2024-05-10 10:58:20.878608312 +0000 UTC m=+0.390201908 container init 6a7db7f103443646d6582db1deeb2c50d544ab2c1d1adfb11df37f9eb19cca1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507) 2024-05-10T10:58:21.003 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-10T10:58:21.003 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-05-10T10:58:21.020 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-10T10:58:21.020 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-05-10T10:58:21.020 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-05-10T10:58:21.020 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-05-10T10:58:21.046 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-10T10:58:21.046 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-05-10T10:58:21.064 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-10T10:58:21.065 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-05-10T10:58:21.065 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-05-10T10:58:21.065 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-10T10:58:21.065 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr to start... 2024-05-10T10:58:21.065 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr... 2024-05-10T10:58:21.153 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:20 smithi027 podman[30269]: 2024-05-10 10:58:20.889765072 +0000 UTC m=+0.401358648 container start 6a7db7f103443646d6582db1deeb2c50d544ab2c1d1adfb11df37f9eb19cca1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-10T10:58:21.153 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:20 smithi027 bash[30269]: 6a7db7f103443646d6582db1deeb2c50d544ab2c1d1adfb11df37f9eb19cca1c 2024-05-10T10:58:21.153 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:20 smithi027 systemd[1]: Started Ceph mgr.x for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T10:58:21.408 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:21 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:21.151+0000 7f22a967e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-10T10:58:21.663 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:21 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:21.428+0000 7f22a967e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-10T10:58:21.663 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:21 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:21.565+0000 7f22a967e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-10T10:58:22.002 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:21 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1286003012' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-10T10:58:22.368 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:22 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:22.000+0000 7f22a967e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-10T10:58:22.369 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:22 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:22.123+0000 7f22a967e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-10T10:58:22.420 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:22.421 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-10T10:58:22.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T10:58:22.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-05-10T10:58:22.422 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-10T10:58:22.425 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-10T10:58:22.425 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-10T10:58:22.425 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:22.426 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-10T10:58:22.426 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-10T10:58:22.426 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-05-10T10:58:22.426 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-10T10:58:22.426 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-10T10:58:22.426 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-05-10T10:58:22.426 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-10T10:58:22.426 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-05-10T10:58:22.426 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-10T10:58:22.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:22.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-10T10:58:22.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-10T10:58:22.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:22.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-10T10:58:22.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:22.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-10T10:58:22.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-10T10:58:22.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-10T10:58:22.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-10T10:58:22.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-10T10:58:22.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-10T10:58:22.438 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:22.438 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-10T10:58:22.438 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-10T10:58:22.438 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-10T10:58:22.438 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-10T10:58:22.438 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-10T10:58:22.438 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-10T10:58:22.438 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-10T10:58:22.438 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-10T10:58:22.438 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-10T10:58:22.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-05-10T10:58:22.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-10T10:58:22.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-10T10:58:22.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:22.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-10T10:58:22.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:22.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-05-10T10:58:11.300584+0000", 2024-05-10T10:58:22.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-10T10:58:22.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:22.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-10T10:58:22.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-10T10:58:22.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.716+0000 7fae84800700 1 Processor -- start 2024-05-10T10:58:22.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.716+0000 7fae84800700 1 -- start start 2024-05-10T10:58:22.441 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.717+0000 7fae84800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 0x7fae7c10bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:22.441 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.717+0000 7fae84800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c07cdb0 con 0x7fae7c1099a0 2024-05-10T10:58:22.441 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.718+0000 7fae81e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 0x7fae7c10bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:22.441 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.718+0000 7fae81e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 0x7fae7c10bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45302/0 (socket says 172.21.15.27:45302) 2024-05-10T10:58:22.441 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.718+0000 7fae81e00700 1 -- 172.21.15.27:0/474389171 learned_addr learned my addr 172.21.15.27:0/474389171 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:22.441 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.718+0000 7fae81e00700 1 -- 172.21.15.27:0/474389171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae7c10c2d0 con 0x7fae7c1099a0 2024-05-10T10:58:22.441 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.719+0000 7fae81e00700 1 --2- 172.21.15.27:0/474389171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 0x7fae7c10bd90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fae700098a0 tx=0x7fae70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9720bc2d6fea8b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:22.441 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.719+0000 7fae80a00700 1 -- 172.21.15.27:0/474389171 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae70004160 con 0x7fae7c1099a0 2024-05-10T10:58:22.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.719+0000 7fae80a00700 1 -- 172.21.15.27:0/474389171 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fae70004300 con 0x7fae7c1099a0 2024-05-10T10:58:22.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.719+0000 7fae80a00700 1 -- 172.21.15.27:0/474389171 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae70011af0 con 0x7fae7c1099a0 2024-05-10T10:58:22.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.720+0000 7fae84800700 1 -- 172.21.15.27:0/474389171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 msgr2=0x7fae7c10bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:22.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.720+0000 7fae84800700 1 --2- 172.21.15.27:0/474389171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 0x7fae7c10bd90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fae700098a0 tx=0x7fae70009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:22.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.720+0000 7fae84800700 1 -- 172.21.15.27:0/474389171 shutdown_connections 2024-05-10T10:58:22.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.720+0000 7fae84800700 1 --2- 172.21.15.27:0/474389171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 0x7fae7c10bd90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:22.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.720+0000 7fae84800700 1 -- 172.21.15.27:0/474389171 >> 172.21.15.27:0/474389171 conn(0x7fae7c1032b0 msgr2=0x7fae7c105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:22.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.720+0000 7fae84800700 1 -- 172.21.15.27:0/474389171 shutdown_connections 2024-05-10T10:58:22.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.720+0000 7fae84800700 1 -- 172.21.15.27:0/474389171 wait complete. 2024-05-10T10:58:22.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.721+0000 7fae84800700 1 Processor -- start 2024-05-10T10:58:22.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.721+0000 7fae84800700 1 -- start start 2024-05-10T10:58:22.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.722+0000 7fae84800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 0x7fae7c19ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:22.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.722+0000 7fae84800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c10d1d0 con 0x7fae7c1099a0 2024-05-10T10:58:22.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.722+0000 7fae81e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 0x7fae7c19ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:22.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.722+0000 7fae81e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 0x7fae7c19ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45308/0 (socket says 172.21.15.27:45308) 2024-05-10T10:58:22.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.722+0000 7fae81e00700 1 -- 172.21.15.27:0/1286003012 learned_addr learned my addr 172.21.15.27:0/1286003012 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:22.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.723+0000 7fae81e00700 1 -- 172.21.15.27:0/1286003012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae7c19d410 con 0x7fae7c1099a0 2024-05-10T10:58:22.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.723+0000 7fae81e00700 1 --2- 172.21.15.27:0/1286003012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 0x7fae7c19ced0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fae70011ac0 tx=0x7fae70011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:22.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.723+0000 7fae76a00700 1 -- 172.21.15.27:0/1286003012 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae70003970 con 0x7fae7c1099a0 2024-05-10T10:58:22.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.723+0000 7fae76a00700 1 -- 172.21.15.27:0/1286003012 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fae70003b10 con 0x7fae7c1099a0 2024-05-10T10:58:22.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.723+0000 7fae84800700 1 -- 172.21.15.27:0/1286003012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae7c19d6a0 con 0x7fae7c1099a0 2024-05-10T10:58:22.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.723+0000 7fae76a00700 1 -- 172.21.15.27:0/1286003012 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae70024610 con 0x7fae7c1099a0 2024-05-10T10:58:22.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.723+0000 7fae84800700 1 -- 172.21.15.27:0/1286003012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae7c19e080 con 0x7fae7c1099a0 2024-05-10T10:58:22.444 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.724+0000 7fae76a00700 1 -- 172.21.15.27:0/1286003012 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fae70024c20 con 0x7fae7c1099a0 2024-05-10T10:58:22.444 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.724+0000 7fae76a00700 1 -- 172.21.15.27:0/1286003012 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fae70019700 con 0x7fae7c1099a0 2024-05-10T10:58:22.444 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.725+0000 7fae84800700 1 -- 172.21.15.27:0/1286003012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae7c064b00 con 0x7fae7c1099a0 2024-05-10T10:58:22.444 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.727+0000 7fae76a00700 1 -- 172.21.15.27:0/1286003012 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fae70043e50 con 0x7fae7c1099a0 2024-05-10T10:58:22.444 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.807+0000 7fae84800700 1 -- 172.21.15.27:0/1286003012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fae7c19e2b0 con 0x7fae7c1099a0 2024-05-10T10:58:22.444 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.808+0000 7fae76a00700 1 -- 172.21.15.27:0/1286003012 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fae70032020 con 0x7fae7c1099a0 2024-05-10T10:58:22.444 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.809+0000 7fae84800700 1 -- 172.21.15.27:0/1286003012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 msgr2=0x7fae7c19ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:22.444 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.809+0000 7fae84800700 1 --2- 172.21.15.27:0/1286003012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 0x7fae7c19ced0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fae70011ac0 tx=0x7fae70011ba0 comp rx=0 tx=0).stop 2024-05-10T10:58:22.444 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.810+0000 7fae84800700 1 -- 172.21.15.27:0/1286003012 shutdown_connections 2024-05-10T10:58:22.444 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.810+0000 7fae84800700 1 --2- 172.21.15.27:0/1286003012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7c1099a0 0x7fae7c19ced0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:22.444 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.810+0000 7fae84800700 1 -- 172.21.15.27:0/1286003012 >> 172.21.15.27:0/1286003012 conn(0x7fae7c1032b0 msgr2=0x7fae7c103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:22.444 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.810+0000 7fae84800700 1 -- 172.21.15.27:0/1286003012 shutdown_connections 2024-05-10T10:58:22.445 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:21.810+0000 7fae84800700 1 -- 172.21.15.27:0/1286003012 wait complete. 2024-05-10T10:58:22.445 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (1/15)... 2024-05-10T10:58:24.118 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:23 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:23.817+0000 7f22a967e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-10T10:58:24.119 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:24 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:24.056+0000 7f22a967e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-10T10:58:24.599 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:24 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:24.282+0000 7f22a967e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-10T10:58:24.869 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:24 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:24.697+0000 7f22a967e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-10T10:58:24.869 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:24 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:24.832+0000 7f22a967e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-10T10:58:25.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:25 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:25.102+0000 7f22a967e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-10T10:58:25.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:25 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2209954846' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-10T10:58:25.465 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:25 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:25.372+0000 7f22a967e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-10T10:58:25.783 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:25.784 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-10T10:58:25.784 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T10:58:25.784 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-05-10T10:58:25.784 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-10T10:58:25.785 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-10T10:58:25.785 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-10T10:58:25.785 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:25.785 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-10T10:58:25.785 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-10T10:58:25.785 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-05-10T10:58:25.785 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-10T10:58:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-10T10:58:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-05-10T10:58:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-10T10:58:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-05-10T10:58:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-10T10:58:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-10T10:58:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-10T10:58:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-10T10:58:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-10T10:58:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-10T10:58:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-10T10:58:25.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-10T10:58:25.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-10T10:58:25.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-10T10:58:25.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:25.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-10T10:58:25.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:25.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-05-10T10:58:11.300584+0000", 2024-05-10T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.057+0000 7f5a6f800700 1 Processor -- start 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.058+0000 7f5a6f800700 1 -- start start 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.058+0000 7f5a6f800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 0x7f5a6807d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.058+0000 7f5a6f800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6807d870 con 0x7f5a6807ee40 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.059+0000 7f5a6ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 0x7f5a6807d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.059+0000 7f5a6ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 0x7f5a6807d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45354/0 (socket says 172.21.15.27:45354) 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.059+0000 7f5a6ce00700 1 -- 172.21.15.27:0/3238126941 learned_addr learned my addr 172.21.15.27:0/3238126941 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.059+0000 7f5a6ce00700 1 -- 172.21.15.27:0/3238126941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6807d9f0 con 0x7f5a6807ee40 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.060+0000 7f5a6ce00700 1 --2- 172.21.15.27:0/3238126941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 0x7f5a6807d2a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0098a0 tx=0x7f5a5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ac3fd7947779b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.060+0000 7f5a67400700 1 -- 172.21.15.27:0/3238126941 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a5c004160 con 0x7f5a6807ee40 2024-05-10T10:58:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.060+0000 7f5a67400700 1 -- 172.21.15.27:0/3238126941 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5a5c004300 con 0x7f5a6807ee40 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.060+0000 7f5a67400700 1 -- 172.21.15.27:0/3238126941 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a5c011af0 con 0x7f5a6807ee40 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.061+0000 7f5a6f800700 1 -- 172.21.15.27:0/3238126941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 msgr2=0x7f5a6807d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.061+0000 7f5a6f800700 1 --2- 172.21.15.27:0/3238126941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 0x7f5a6807d2a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0098a0 tx=0x7f5a5c009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.061+0000 7f5a6f800700 1 -- 172.21.15.27:0/3238126941 shutdown_connections 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.061+0000 7f5a6f800700 1 --2- 172.21.15.27:0/3238126941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 0x7f5a6807d2a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.061+0000 7f5a6f800700 1 -- 172.21.15.27:0/3238126941 >> 172.21.15.27:0/3238126941 conn(0x7f5a68104400 msgr2=0x7f5a68106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.061+0000 7f5a6f800700 1 -- 172.21.15.27:0/3238126941 shutdown_connections 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.061+0000 7f5a6f800700 1 -- 172.21.15.27:0/3238126941 wait complete. 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.062+0000 7f5a6f800700 1 Processor -- start 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.063+0000 7f5a6f800700 1 -- start start 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.063+0000 7f5a6f800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 0x7f5a6819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.063+0000 7f5a6f800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6810f3f0 con 0x7f5a6807ee40 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.063+0000 7f5a6ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 0x7f5a6819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:25.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.063+0000 7f5a6ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 0x7f5a6819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45360/0 (socket says 172.21.15.27:45360) 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.063+0000 7f5a6ce00700 1 -- 172.21.15.27:0/2209954846 learned_addr learned my addr 172.21.15.27:0/2209954846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.064+0000 7f5a6ce00700 1 -- 172.21.15.27:0/2209954846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6819f580 con 0x7f5a6807ee40 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.064+0000 7f5a6ce00700 1 --2- 172.21.15.27:0/2209954846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 0x7f5a6819f040 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009fb0 tx=0x7f5a5c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.064+0000 7f5a65600700 1 -- 172.21.15.27:0/2209954846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a5c0036a0 con 0x7f5a6807ee40 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.064+0000 7f5a65600700 1 -- 172.21.15.27:0/2209954846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5a5c01a430 con 0x7f5a6807ee40 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.064+0000 7f5a65600700 1 -- 172.21.15.27:0/2209954846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a5c019550 con 0x7f5a6807ee40 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.064+0000 7f5a6f800700 1 -- 172.21.15.27:0/2209954846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a6819f810 con 0x7f5a6807ee40 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.064+0000 7f5a6f800700 1 -- 172.21.15.27:0/2209954846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a681a0330 con 0x7f5a6807ee40 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.065+0000 7f5a65600700 1 -- 172.21.15.27:0/2209954846 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5a5c01a8e0 con 0x7f5a6807ee40 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.065+0000 7f5a65600700 1 -- 172.21.15.27:0/2209954846 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5a5c02d780 con 0x7f5a6807ee40 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.066+0000 7f5a6f800700 1 -- 172.21.15.27:0/2209954846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a68064b00 con 0x7f5a6807ee40 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.070+0000 7f5a65600700 1 -- 172.21.15.27:0/2209954846 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5a5c043e50 con 0x7f5a6807ee40 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.150+0000 7f5a6f800700 1 -- 172.21.15.27:0/2209954846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f5a681a0760 con 0x7f5a6807ee40 2024-05-10T10:58:25.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.151+0000 7f5a65600700 1 -- 172.21.15.27:0/2209954846 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f5a5c032030 con 0x7f5a6807ee40 2024-05-10T10:58:25.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.152+0000 7f5a6f800700 1 -- 172.21.15.27:0/2209954846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 msgr2=0x7f5a6819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:25.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.152+0000 7f5a6f800700 1 --2- 172.21.15.27:0/2209954846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 0x7f5a6819f040 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009fb0 tx=0x7f5a5c011940 comp rx=0 tx=0).stop 2024-05-10T10:58:25.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.152+0000 7f5a6f800700 1 -- 172.21.15.27:0/2209954846 shutdown_connections 2024-05-10T10:58:25.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.152+0000 7f5a6f800700 1 --2- 172.21.15.27:0/2209954846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6807ee40 0x7f5a6819f040 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:25.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.152+0000 7f5a6f800700 1 -- 172.21.15.27:0/2209954846 >> 172.21.15.27:0/2209954846 conn(0x7f5a68104400 msgr2=0x7f5a68104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:25.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.153+0000 7f5a6f800700 1 -- 172.21.15.27:0/2209954846 shutdown_connections 2024-05-10T10:58:25.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:25.153+0000 7f5a6f800700 1 -- 172.21.15.27:0/2209954846 wait complete. 2024-05-10T10:58:25.802 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (2/15)... 2024-05-10T10:58:26.520 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:26 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:26.220+0000 7f22a967e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-10T10:58:26.868 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:26 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:26.519+0000 7f22a967e200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-10T10:58:27.368 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:27 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:27.097+0000 7f22a967e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-10T10:58:27.368 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:27 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:27.231+0000 7f22a967e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-10T10:58:28.069 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:27 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:27.713+0000 7f22a967e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-10T10:58:28.325 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:28 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:28.208+0000 7f22a967e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-10T10:58:28.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:28 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2741469419' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-10T10:58:28.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:28 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:28.341+0000 7f22a967e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-10T10:58:29.050 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:28 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:28.711+0000 7f22a967e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-10T10:58:29.050 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:28 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:28.912+0000 7f22a967e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-10T10:58:29.050 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:29 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:29.048+0000 7f22a967e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-10T10:58:29.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:29.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-10T10:58:29.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T10:58:29.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-05-10T10:58:29.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-10T10:58:29.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-10T10:58:29.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-10T10:58:29.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:29.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-10T10:58:29.111 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-10T10:58:29.112 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-05-10T10:58:29.112 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-10T10:58:29.112 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-10T10:58:29.112 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-10T10:58:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-10T10:58:29.114 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-10T10:58:29.114 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:29.114 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-10T10:58:29.114 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-10T10:58:29.121 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-10T10:58:29.122 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-10T10:58:29.122 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-10T10:58:29.122 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-10T10:58:29.122 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-10T10:58:29.122 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-10T10:58:29.122 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-10T10:58:29.122 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:29.122 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-10T10:58:29.122 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:29.122 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-10T10:58:29.122 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-10T10:58:29.122 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:29.123 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-10T10:58:29.123 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-10T10:58:29.123 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-10T10:58:29.123 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-10T10:58:29.123 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-10T10:58:29.123 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-10T10:58:29.123 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-05-10T10:58:29.123 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-10T10:58:29.123 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-10T10:58:29.123 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:29.123 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-10T10:58:29.123 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:29.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-05-10T10:58:11.300584+0000", 2024-05-10T10:58:29.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-10T10:58:29.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:29.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-10T10:58:29.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-10T10:58:29.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.345+0000 7fa4cfe00700 1 Processor -- start 2024-05-10T10:58:29.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.346+0000 7fa4cfe00700 1 -- start start 2024-05-10T10:58:29.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.346+0000 7fa4cfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 0x7fa4d0107fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:29.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.346+0000 7fa4cfe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0108570 con 0x7fa4d0107b80 2024-05-10T10:58:29.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.347+0000 7fa4cea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 0x7fa4d0107fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:29.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.347+0000 7fa4cea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 0x7fa4d0107fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45372/0 (socket says 172.21.15.27:45372) 2024-05-10T10:58:29.124 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.347+0000 7fa4cea00700 1 -- 172.21.15.27:0/3578251770 learned_addr learned my addr 172.21.15.27:0/3578251770 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.347+0000 7fa4cea00700 1 -- 172.21.15.27:0/3578251770 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d0108d80 con 0x7fa4d0107b80 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.348+0000 7fa4cea00700 1 --2- 172.21.15.27:0/3578251770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 0x7fa4d0107fa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00098a0 tx=0x7fa4c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e3a11b633da4562 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.348+0000 7fa4cd600700 1 -- 172.21.15.27:0/3578251770 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4c0004160 con 0x7fa4d0107b80 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.348+0000 7fa4cd600700 1 -- 172.21.15.27:0/3578251770 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa4c0004300 con 0x7fa4d0107b80 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.348+0000 7fa4cd600700 1 -- 172.21.15.27:0/3578251770 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4c0011af0 con 0x7fa4d0107b80 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.349+0000 7fa4cfe00700 1 -- 172.21.15.27:0/3578251770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 msgr2=0x7fa4d0107fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.349+0000 7fa4cfe00700 1 --2- 172.21.15.27:0/3578251770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 0x7fa4d0107fa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00098a0 tx=0x7fa4c0009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.349+0000 7fa4cfe00700 1 -- 172.21.15.27:0/3578251770 shutdown_connections 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.349+0000 7fa4cfe00700 1 --2- 172.21.15.27:0/3578251770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 0x7fa4d0107fa0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.349+0000 7fa4cfe00700 1 -- 172.21.15.27:0/3578251770 >> 172.21.15.27:0/3578251770 conn(0x7fa4d0103120 msgr2=0x7fa4d0105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.349+0000 7fa4cfe00700 1 -- 172.21.15.27:0/3578251770 shutdown_connections 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.349+0000 7fa4cfe00700 1 -- 172.21.15.27:0/3578251770 wait complete. 2024-05-10T10:58:29.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.351+0000 7fa4cfe00700 1 Processor -- start 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.351+0000 7fa4cfe00700 1 -- start start 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.351+0000 7fa4cfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 0x7fa4d019ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.351+0000 7fa4cfe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d007bfb0 con 0x7fa4d0107b80 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.352+0000 7fa4cea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 0x7fa4d019ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.352+0000 7fa4cea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 0x7fa4d019ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45382/0 (socket says 172.21.15.27:45382) 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.352+0000 7fa4cea00700 1 -- 172.21.15.27:0/2741469419 learned_addr learned my addr 172.21.15.27:0/2741469419 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.352+0000 7fa4cea00700 1 -- 172.21.15.27:0/2741469419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d019f2d0 con 0x7fa4d0107b80 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.352+0000 7fa4cea00700 1 --2- 172.21.15.27:0/2741469419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 0x7fa4d019ed90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0000c00 tx=0x7fa4c0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.353+0000 7fa4c7400700 1 -- 172.21.15.27:0/2741469419 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4c0003b20 con 0x7fa4d0107b80 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.353+0000 7fa4c7400700 1 -- 172.21.15.27:0/2741469419 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa4c001a8c0 con 0x7fa4d0107b80 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.353+0000 7fa4c7400700 1 -- 172.21.15.27:0/2741469419 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4c0024940 con 0x7fa4d0107b80 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.353+0000 7fa4cfe00700 1 -- 172.21.15.27:0/2741469419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d019f560 con 0x7fa4d0107b80 2024-05-10T10:58:29.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.353+0000 7fa4cfe00700 1 -- 172.21.15.27:0/2741469419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d01a2210 con 0x7fa4d0107b80 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.353+0000 7fa4c7400700 1 -- 172.21.15.27:0/2741469419 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa4c0022020 con 0x7fa4d0107b80 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.354+0000 7fa4c7400700 1 -- 172.21.15.27:0/2741469419 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa4c002dd90 con 0x7fa4d0107b80 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.354+0000 7fa4cfe00700 1 -- 172.21.15.27:0/2741469419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d006a5d0 con 0x7fa4d0107b80 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.357+0000 7fa4c7400700 1 -- 172.21.15.27:0/2741469419 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa4c0042e50 con 0x7fa4d0107b80 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.458+0000 7fa4cfe00700 1 -- 172.21.15.27:0/2741469419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa4d002e8c0 con 0x7fa4d0107b80 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.458+0000 7fa4c7400700 1 -- 172.21.15.27:0/2741469419 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7fa4c0031020 con 0x7fa4d0107b80 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.459+0000 7fa4cfe00700 1 -- 172.21.15.27:0/2741469419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 msgr2=0x7fa4d019ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.459+0000 7fa4cfe00700 1 --2- 172.21.15.27:0/2741469419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 0x7fa4d019ed90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0000c00 tx=0x7fa4c0011b70 comp rx=0 tx=0).stop 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.460+0000 7fa4cfe00700 1 -- 172.21.15.27:0/2741469419 shutdown_connections 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.460+0000 7fa4cfe00700 1 --2- 172.21.15.27:0/2741469419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d0107b80 0x7fa4d019ed90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.460+0000 7fa4cfe00700 1 -- 172.21.15.27:0/2741469419 >> 172.21.15.27:0/2741469419 conn(0x7fa4d0103120 msgr2=0x7fa4d0105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.460+0000 7fa4cfe00700 1 -- 172.21.15.27:0/2741469419 shutdown_connections 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:28.460+0000 7fa4cfe00700 1 -- 172.21.15.27:0/2741469419 wait complete. 2024-05-10T10:58:29.127 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (3/15)... 2024-05-10T10:58:29.869 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:29 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:29.467+0000 7f22a967e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-10T10:58:29.869 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:29 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:29.598+0000 7f22a967e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-10T10:58:30.699 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:30 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:30.401+0000 7f22a967e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-10T10:58:31.119 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:30 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:30.698+0000 7f22a967e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-10T10:58:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: Activating manager daemon x 2024-05-10T10:58:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: mgrmap e2: x(active, starting, since 0.0115305s) 2024-05-10T10:58:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-10T10:58:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-10T10:58:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-10T10:58:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-10T10:58:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-10T10:58:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-10T10:58:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T10:58:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-10T10:58:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: Manager daemon x is now available 2024-05-10T10:58:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-10T10:58:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-10T10:58:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:30 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' 2024-05-10T10:58:32.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:31 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' 2024-05-10T10:58:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:31 smithi027 ceph-mon[30054]: from='mgr.14100 172.21.15.27:0/537143138' entity='mgr.x' 2024-05-10T10:58:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:31 smithi027 ceph-mon[30054]: mgrmap e3: x(active, since 1.02182s) 2024-05-10T10:58:32.881 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:32.881 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-10T10:58:32.882 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-05-10T10:58:32.883 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-10T10:58:32.883 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:32.883 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-10T10:58:32.883 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-10T10:58:32.883 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:32.883 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-10T10:58:32.883 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:32.883 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-10T10:58:32.883 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-10T10:58:32.883 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-10T10:58:32.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-10T10:58:32.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-10T10:58:32.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-10T10:58:32.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:32.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-10T10:58:32.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-05-10T10:58:11.300584+0000", 2024-05-10T10:58:32.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.839+0000 7f9e33e00700 1 Processor -- start 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.839+0000 7f9e33e00700 1 -- start start 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.840+0000 7f9e33e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 0x7f9e34107fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.840+0000 7f9e33e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9e34108570 con 0x7f9e34107b80 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.840+0000 7f9e32a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 0x7f9e34107fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.841+0000 7f9e32a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 0x7f9e34107fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45450/0 (socket says 172.21.15.27:45450) 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.841+0000 7f9e32a00700 1 -- 172.21.15.27:0/2108880706 learned_addr learned my addr 172.21.15.27:0/2108880706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.841+0000 7f9e32a00700 1 -- 172.21.15.27:0/2108880706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e34108d80 con 0x7f9e34107b80 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.842+0000 7f9e32a00700 1 --2- 172.21.15.27:0/2108880706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 0x7f9e34107fa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e240098a0 tx=0x7f9e24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607b8b531c036a34 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.842+0000 7f9e31600700 1 -- 172.21.15.27:0/2108880706 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e24004160 con 0x7f9e34107b80 2024-05-10T10:58:32.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.842+0000 7f9e31600700 1 -- 172.21.15.27:0/2108880706 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9e24004300 con 0x7f9e34107b80 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.842+0000 7f9e31600700 1 -- 172.21.15.27:0/2108880706 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e24011af0 con 0x7f9e34107b80 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.843+0000 7f9e33e00700 1 -- 172.21.15.27:0/2108880706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 msgr2=0x7f9e34107fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.843+0000 7f9e33e00700 1 --2- 172.21.15.27:0/2108880706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 0x7f9e34107fa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e240098a0 tx=0x7f9e24009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.843+0000 7f9e33e00700 1 -- 172.21.15.27:0/2108880706 shutdown_connections 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.843+0000 7f9e33e00700 1 --2- 172.21.15.27:0/2108880706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 0x7f9e34107fa0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.843+0000 7f9e33e00700 1 -- 172.21.15.27:0/2108880706 >> 172.21.15.27:0/2108880706 conn(0x7f9e34103100 msgr2=0x7f9e34105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.843+0000 7f9e33e00700 1 -- 172.21.15.27:0/2108880706 shutdown_connections 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.843+0000 7f9e33e00700 1 -- 172.21.15.27:0/2108880706 wait complete. 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.844+0000 7f9e33e00700 1 Processor -- start 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.844+0000 7f9e33e00700 1 -- start start 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.844+0000 7f9e33e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 0x7f9e34192cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.844+0000 7f9e33e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3410c130 con 0x7f9e34107b80 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.845+0000 7f9e32a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 0x7f9e34192cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.845+0000 7f9e32a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 0x7f9e34192cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45456/0 (socket says 172.21.15.27:45456) 2024-05-10T10:58:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.845+0000 7f9e32a00700 1 -- 172.21.15.27:0/1705933554 learned_addr learned my addr 172.21.15.27:0/1705933554 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.845+0000 7f9e32a00700 1 -- 172.21.15.27:0/1705933554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e34193230 con 0x7f9e34107b80 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.845+0000 7f9e32a00700 1 --2- 172.21.15.27:0/1705933554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 0x7f9e34192cf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9e24003f40 tx=0x7f9e24011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.846+0000 7f9e2b400700 1 -- 172.21.15.27:0/1705933554 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e240038b0 con 0x7f9e34107b80 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.846+0000 7f9e2b400700 1 -- 172.21.15.27:0/1705933554 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9e24003a50 con 0x7f9e34107b80 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.846+0000 7f9e2b400700 1 -- 172.21.15.27:0/1705933554 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e2401a5c0 con 0x7f9e34107b80 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.846+0000 7f9e33e00700 1 -- 172.21.15.27:0/1705933554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e34191420 con 0x7f9e34107b80 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.846+0000 7f9e33e00700 1 -- 172.21.15.27:0/1705933554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e34191960 con 0x7f9e34107b80 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.847+0000 7f9e33e00700 1 -- 172.21.15.27:0/1705933554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e3406a5d0 con 0x7f9e34107b80 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.847+0000 7f9e2b400700 1 -- 172.21.15.27:0/1705933554 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 3) ==== 50105+0+0 (secure 0 0 0) 0x7f9e24024430 con 0x7f9e34107b80 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.848+0000 7f9e2b400700 1 --2- 172.21.15.27:0/1705933554 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f9e1803e3d0 0x7f9e18040890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.848+0000 7f9e2b400700 1 -- 172.21.15.27:0/1705933554 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9e240537c0 con 0x7f9e34107b80 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.848+0000 7f9e32000700 1 --2- 172.21.15.27:0/1705933554 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f9e1803e3d0 0x7f9e18040890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.849+0000 7f9e32000700 1 --2- 172.21.15.27:0/1705933554 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f9e1803e3d0 0x7f9e18040890 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9e20006fd0 tx=0x7f9e20006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:32.897 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:31.855+0000 7f9e2b400700 1 -- 172.21.15.27:0/1705933554 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9e2400bcb0 con 0x7f9e34107b80 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:32.187+0000 7f9e33e00700 1 -- 172.21.15.27:0/1705933554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9e34196a70 con 0x7f9e34107b80 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:32.188+0000 7f9e2b400700 1 -- 172.21.15.27:0/1705933554 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9e24019e10 con 0x7f9e34107b80 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:32.191+0000 7f9e33e00700 1 -- 172.21.15.27:0/1705933554 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f9e1803e3d0 msgr2=0x7f9e18040890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:32.191+0000 7f9e33e00700 1 --2- 172.21.15.27:0/1705933554 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f9e1803e3d0 0x7f9e18040890 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9e20006fd0 tx=0x7f9e20006e40 comp rx=0 tx=0).stop 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:32.191+0000 7f9e33e00700 1 -- 172.21.15.27:0/1705933554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 msgr2=0x7f9e34192cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:32.191+0000 7f9e33e00700 1 --2- 172.21.15.27:0/1705933554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 0x7f9e34192cf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9e24003f40 tx=0x7f9e24011890 comp rx=0 tx=0).stop 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:32.191+0000 7f9e33e00700 1 -- 172.21.15.27:0/1705933554 shutdown_connections 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:32.191+0000 7f9e33e00700 1 --2- 172.21.15.27:0/1705933554 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f9e1803e3d0 0x7f9e18040890 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:32.191+0000 7f9e33e00700 1 --2- 172.21.15.27:0/1705933554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e34107b80 0x7f9e34192cf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:32.191+0000 7f9e33e00700 1 -- 172.21.15.27:0/1705933554 >> 172.21.15.27:0/1705933554 conn(0x7f9e34103100 msgr2=0x7f9e34103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:32.192+0000 7f9e33e00700 1 -- 172.21.15.27:0/1705933554 shutdown_connections 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:32.192+0000 7f9e33e00700 1 -- 172.21.15.27:0/1705933554 wait complete. 2024-05-10T10:58:32.898 INFO:teuthology.orchestra.run.smithi027.stdout:mgr is available 2024-05-10T10:58:33.048 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:32 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1705933554' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-10T10:58:34.104 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:33 smithi027 ceph-mon[30054]: mgrmap e4: x(active, since 2s) 2024-05-10T10:58:34.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:34.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-05-10T10:58:34.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:58:34.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-10T10:58:34.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-10T10:58:34.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-10T10:58:34.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-10T10:58:34.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-10T10:58:34.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:34.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-10T10:58:34.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-10T10:58:34.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-10T10:58:34.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.550+0000 7f146d200700 1 Processor -- start 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.550+0000 7f146d200700 1 -- start start 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.550+0000 7f146d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 0x7f146810bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.550+0000 7f146d200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f146807ce10 con 0x7f1468109940 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.550+0000 7f1465e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 0x7f146810bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.550+0000 7f1465e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 0x7f146810bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45458/0 (socket says 172.21.15.27:45458) 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.550+0000 7f1465e00700 1 -- 172.21.15.27:0/3872276951 learned_addr learned my addr 172.21.15.27:0/3872276951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:34.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.551+0000 7f1465e00700 1 -- 172.21.15.27:0/3872276951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146810c270 con 0x7f1468109940 2024-05-10T10:58:34.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.551+0000 7f1465e00700 1 --2- 172.21.15.27:0/3872276951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 0x7f146810bd30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f14580098a0 tx=0x7f1458009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60780a254fe588e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:34.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.552+0000 7f1464a00700 1 -- 172.21.15.27:0/3872276951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1458004160 con 0x7f1468109940 2024-05-10T10:58:34.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.552+0000 7f1464a00700 1 -- 172.21.15.27:0/3872276951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1458004300 con 0x7f1468109940 2024-05-10T10:58:34.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.552+0000 7f1464a00700 1 -- 172.21.15.27:0/3872276951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1458011af0 con 0x7f1468109940 2024-05-10T10:58:34.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.552+0000 7f146d200700 1 -- 172.21.15.27:0/3872276951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 msgr2=0x7f146810bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:34.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.552+0000 7f146d200700 1 --2- 172.21.15.27:0/3872276951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 0x7f146810bd30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f14580098a0 tx=0x7f1458009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:34.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.553+0000 7f146d200700 1 -- 172.21.15.27:0/3872276951 shutdown_connections 2024-05-10T10:58:34.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.553+0000 7f146d200700 1 --2- 172.21.15.27:0/3872276951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 0x7f146810bd30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:34.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.553+0000 7f146d200700 1 -- 172.21.15.27:0/3872276951 >> 172.21.15.27:0/3872276951 conn(0x7f14681032b0 msgr2=0x7f1468105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:34.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.553+0000 7f146d200700 1 -- 172.21.15.27:0/3872276951 shutdown_connections 2024-05-10T10:58:34.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.553+0000 7f146d200700 1 -- 172.21.15.27:0/3872276951 wait complete. 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.554+0000 7f146d200700 1 Processor -- start 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.554+0000 7f146d200700 1 -- start start 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.554+0000 7f146d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 0x7f1468109350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.554+0000 7f146d200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f146810d1d0 con 0x7f1468109940 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.555+0000 7f1465e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 0x7f1468109350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.555+0000 7f1465e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 0x7f1468109350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45462/0 (socket says 172.21.15.27:45462) 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.555+0000 7f1465e00700 1 -- 172.21.15.27:0/2504684494 learned_addr learned my addr 172.21.15.27:0/2504684494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.555+0000 7f1465e00700 1 -- 172.21.15.27:0/2504684494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14681078f0 con 0x7f1468109940 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.555+0000 7f1465e00700 1 --2- 172.21.15.27:0/2504684494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 0x7f1468109350 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1458011ac0 tx=0x7f1458011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.556+0000 7f145ea00700 1 -- 172.21.15.27:0/2504684494 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1458003970 con 0x7f1468109940 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.556+0000 7f145ea00700 1 -- 172.21.15.27:0/2504684494 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1458003b10 con 0x7f1468109940 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.556+0000 7f145ea00700 1 -- 172.21.15.27:0/2504684494 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1458024610 con 0x7f1468109940 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.556+0000 7f146d200700 1 -- 172.21.15.27:0/2504684494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1468107b80 con 0x7f1468109940 2024-05-10T10:58:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.556+0000 7f146d200700 1 -- 172.21.15.27:0/2504684494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1468108060 con 0x7f1468109940 2024-05-10T10:58:34.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.557+0000 7f145ea00700 1 -- 172.21.15.27:0/2504684494 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f145801a9e0 con 0x7f1468109940 2024-05-10T10:58:34.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.557+0000 7f146d200700 1 -- 172.21.15.27:0/2504684494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1468064b00 con 0x7f1468109940 2024-05-10T10:58:34.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.558+0000 7f145ea00700 1 --2- 172.21.15.27:0/2504684494 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f144c03e4e0 0x7f144c0409a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:34.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.558+0000 7f145ea00700 1 -- 172.21.15.27:0/2504684494 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1458053840 con 0x7f1468109940 2024-05-10T10:58:34.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.558+0000 7f1465400700 1 --2- 172.21.15.27:0/2504684494 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f144c03e4e0 0x7f144c0409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:34.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.559+0000 7f1465400700 1 --2- 172.21.15.27:0/2504684494 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f144c03e4e0 0x7f144c0409a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1454006fd0 tx=0x7f1454006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:34.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.564+0000 7f145ea00700 1 -- 172.21.15.27:0/2504684494 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f1458024840 con 0x7f1468109940 2024-05-10T10:58:34.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.804+0000 7f146d200700 1 -- 172.21.15.27:0/2504684494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f14681085a0 con 0x7f1468109940 2024-05-10T10:58:34.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.806+0000 7f145ea00700 1 -- 172.21.15.27:0/2504684494 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f1458053260 con 0x7f1468109940 2024-05-10T10:58:34.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.810+0000 7f146d200700 1 -- 172.21.15.27:0/2504684494 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f144c03e4e0 msgr2=0x7f144c0409a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:34.414 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.810+0000 7f146d200700 1 --2- 172.21.15.27:0/2504684494 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f144c03e4e0 0x7f144c0409a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1454006fd0 tx=0x7f1454006e40 comp rx=0 tx=0).stop 2024-05-10T10:58:34.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.810+0000 7f146d200700 1 -- 172.21.15.27:0/2504684494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 msgr2=0x7f1468109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:34.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.810+0000 7f146d200700 1 --2- 172.21.15.27:0/2504684494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 0x7f1468109350 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1458011ac0 tx=0x7f1458011ba0 comp rx=0 tx=0).stop 2024-05-10T10:58:34.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.810+0000 7f146d200700 1 -- 172.21.15.27:0/2504684494 shutdown_connections 2024-05-10T10:58:34.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.810+0000 7f146d200700 1 --2- 172.21.15.27:0/2504684494 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f144c03e4e0 0x7f144c0409a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:34.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.810+0000 7f146d200700 1 --2- 172.21.15.27:0/2504684494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1468109940 0x7f1468109350 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:34.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.810+0000 7f146d200700 1 -- 172.21.15.27:0/2504684494 >> 172.21.15.27:0/2504684494 conn(0x7f14681032b0 msgr2=0x7f1468103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:34.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.810+0000 7f146d200700 1 -- 172.21.15.27:0/2504684494 shutdown_connections 2024-05-10T10:58:34.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:33.810+0000 7f146d200700 1 -- 172.21.15.27:0/2504684494 wait complete. 2024-05-10T10:58:34.415 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling cephadm module... 2024-05-10T10:58:35.070 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:34 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2504684494' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-10T10:58:36.076 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:35 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/675655102' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-10T10:58:36.076 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:35 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: ignoring --setuser ceph since I am not root 2024-05-10T10:58:36.076 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:35 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: ignoring --setgroup ceph since I am not root 2024-05-10T10:58:36.076 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:36 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:36.075+0000 7facfdd12200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-10T10:58:36.553 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.069+0000 7f47dae00700 1 Processor -- start 2024-05-10T10:58:36.554 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.070+0000 7f47dae00700 1 -- start start 2024-05-10T10:58:36.554 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.070+0000 7f47dae00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 0x7f47d410b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:36.554 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.070+0000 7f47dae00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f47d410bad0 con 0x7f47d410b0e0 2024-05-10T10:58:36.554 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.070+0000 7f47d3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 0x7f47d410b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:36.554 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.071+0000 7f47d3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 0x7f47d410b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46330/0 (socket says 172.21.15.27:46330) 2024-05-10T10:58:36.554 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.071+0000 7f47d3e00700 1 -- 172.21.15.27:0/3225224438 learned_addr learned my addr 172.21.15.27:0/3225224438 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:36.554 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.071+0000 7f47d3e00700 1 -- 172.21.15.27:0/3225224438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d410c2a0 con 0x7f47d410b0e0 2024-05-10T10:58:36.554 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.071+0000 7f47d3e00700 1 --2- 172.21.15.27:0/3225224438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 0x7f47d410b500 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f47c40098a0 tx=0x7f47c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9412bd62648ded8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:36.554 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.072+0000 7f47d3400700 1 -- 172.21.15.27:0/3225224438 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47c4004160 con 0x7f47d410b0e0 2024-05-10T10:58:36.554 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.072+0000 7f47d3400700 1 -- 172.21.15.27:0/3225224438 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f47c4004300 con 0x7f47d410b0e0 2024-05-10T10:58:36.554 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.072+0000 7f47d3400700 1 -- 172.21.15.27:0/3225224438 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47c4011af0 con 0x7f47d410b0e0 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.072+0000 7f47dae00700 1 -- 172.21.15.27:0/3225224438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 msgr2=0x7f47d410b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.072+0000 7f47dae00700 1 --2- 172.21.15.27:0/3225224438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 0x7f47d410b500 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f47c40098a0 tx=0x7f47c4009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.073+0000 7f47dae00700 1 -- 172.21.15.27:0/3225224438 shutdown_connections 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.073+0000 7f47dae00700 1 --2- 172.21.15.27:0/3225224438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 0x7f47d410b500 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.073+0000 7f47dae00700 1 -- 172.21.15.27:0/3225224438 >> 172.21.15.27:0/3225224438 conn(0x7f47d407e450 msgr2=0x7f47d407e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.073+0000 7f47dae00700 1 -- 172.21.15.27:0/3225224438 shutdown_connections 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.073+0000 7f47dae00700 1 -- 172.21.15.27:0/3225224438 wait complete. 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.074+0000 7f47dae00700 1 Processor -- start 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.074+0000 7f47dae00700 1 -- start start 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.074+0000 7f47dae00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 0x7f47d419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.074+0000 7f47dae00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f47d410f650 con 0x7f47d410b0e0 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.075+0000 7f47d3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 0x7f47d419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.075+0000 7f47d3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 0x7f47d419efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46336/0 (socket says 172.21.15.27:46336) 2024-05-10T10:58:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.075+0000 7f47d3e00700 1 -- 172.21.15.27:0/675655102 learned_addr learned my addr 172.21.15.27:0/675655102 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.075+0000 7f47d3e00700 1 -- 172.21.15.27:0/675655102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d419f520 con 0x7f47d410b0e0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.075+0000 7f47d3e00700 1 --2- 172.21.15.27:0/675655102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 0x7f47d419efe0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f47c4011ac0 tx=0x7f47c4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.076+0000 7f47d1600700 1 -- 172.21.15.27:0/675655102 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47c4003b20 con 0x7f47d410b0e0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.076+0000 7f47d1600700 1 -- 172.21.15.27:0/675655102 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f47c401a8c0 con 0x7f47d410b0e0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.076+0000 7f47d1600700 1 -- 172.21.15.27:0/675655102 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47c40248d0 con 0x7f47d410b0e0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.076+0000 7f47dae00700 1 -- 172.21.15.27:0/675655102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d419f7b0 con 0x7f47d410b0e0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.076+0000 7f47dae00700 1 -- 172.21.15.27:0/675655102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d41a02d0 con 0x7f47d410b0e0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.077+0000 7f47d1600700 1 -- 172.21.15.27:0/675655102 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f47c4022020 con 0x7f47d410b0e0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.077+0000 7f47dae00700 1 -- 172.21.15.27:0/675655102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d4064b00 con 0x7f47d410b0e0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.078+0000 7f47d1600700 1 --2- 172.21.15.27:0/675655102 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f47b403e530 0x7f47b40409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.078+0000 7f47d1600700 1 -- 172.21.15.27:0/675655102 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f47c4053470 con 0x7f47d410b0e0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.078+0000 7f47cb400700 1 --2- 172.21.15.27:0/675655102 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f47b403e530 0x7f47b40409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.079+0000 7f47cb400700 1 --2- 172.21.15.27:0/675655102 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f47b403e530 0x7f47b40409f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f47bc006fd0 tx=0x7f47bc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.083+0000 7f47d1600700 1 -- 172.21.15.27:0/675655102 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f47c40172b0 con 0x7f47d410b0e0 2024-05-10T10:58:36.556 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.390+0000 7f47dae00700 1 -- 172.21.15.27:0/675655102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f47d41a07f0 con 0x7f47d410b0e0 2024-05-10T10:58:36.557 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.783+0000 7f47d1600700 1 -- 172.21.15.27:0/675655102 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f47c401ab20 con 0x7f47d410b0e0 2024-05-10T10:58:36.557 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.787+0000 7f47d1600700 1 -- 172.21.15.27:0/675655102 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f47c401a430 con 0x7f47d410b0e0 2024-05-10T10:58:36.557 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.788+0000 7f47dae00700 1 -- 172.21.15.27:0/675655102 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f47b403e530 msgr2=0x7f47b40409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:36.557 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.788+0000 7f47dae00700 1 --2- 172.21.15.27:0/675655102 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f47b403e530 0x7f47b40409f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f47bc006fd0 tx=0x7f47bc006e40 comp rx=0 tx=0).stop 2024-05-10T10:58:36.557 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.788+0000 7f47dae00700 1 -- 172.21.15.27:0/675655102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 msgr2=0x7f47d419efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:36.557 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.788+0000 7f47dae00700 1 --2- 172.21.15.27:0/675655102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 0x7f47d419efe0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f47c4011ac0 tx=0x7f47c4011ba0 comp rx=0 tx=0).stop 2024-05-10T10:58:36.557 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.788+0000 7f47dae00700 1 -- 172.21.15.27:0/675655102 shutdown_connections 2024-05-10T10:58:36.557 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.788+0000 7f47dae00700 1 --2- 172.21.15.27:0/675655102 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7f47b403e530 0x7f47b40409f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:36.557 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.788+0000 7f47dae00700 1 --2- 172.21.15.27:0/675655102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47d410b0e0 0x7f47d419efe0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:36.557 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.788+0000 7f47dae00700 1 -- 172.21.15.27:0/675655102 >> 172.21.15.27:0/675655102 conn(0x7f47d407e450 msgr2=0x7f47d4108c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:36.557 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.789+0000 7f47dae00700 1 -- 172.21.15.27:0/675655102 shutdown_connections 2024-05-10T10:58:36.557 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:35.789+0000 7f47dae00700 1 -- 172.21.15.27:0/675655102 wait complete. 2024-05-10T10:58:36.618 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:36 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:36.329+0000 7facfdd12200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-10T10:58:36.618 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:36 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:36.466+0000 7facfdd12200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-10T10:58:37.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:36 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/675655102' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-10T10:58:37.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:36 smithi027 ceph-mon[30054]: mgrmap e5: x(active, since 5s) 2024-05-10T10:58:37.010 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:36 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:36.890+0000 7facfdd12200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-10T10:58:37.368 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:37 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:37.009+0000 7facfdd12200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-10T10:58:38.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:37 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2237302155' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-10T10:58:38.173 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-10T10:58:38.174 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-10T10:58:38.174 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-10T10:58:38.174 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-10T10:58:38.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-10T10:58:38.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-10T10:58:38.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.169+0000 7fafdf200700 1 Processor -- start 2024-05-10T10:58:38.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.169+0000 7fafdf200700 1 -- start start 2024-05-10T10:58:38.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.170+0000 7fafdf200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 0x7fafd810bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:38.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.170+0000 7fafdf200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fafd807ce70 con 0x7fafd81098e0 2024-05-10T10:58:38.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.170+0000 7fafd7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 0x7fafd810bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:38.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.170+0000 7fafd7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 0x7fafd810bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46372/0 (socket says 172.21.15.27:46372) 2024-05-10T10:58:38.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.170+0000 7fafd7e00700 1 -- 172.21.15.27:0/3034762946 learned_addr learned my addr 172.21.15.27:0/3034762946 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:38.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.171+0000 7fafd7e00700 1 -- 172.21.15.27:0/3034762946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd810c210 con 0x7fafd81098e0 2024-05-10T10:58:38.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.171+0000 7fafd7e00700 1 --2- 172.21.15.27:0/3034762946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 0x7fafd810bcd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fafc80098a0 tx=0x7fafc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=823cbbd139ac8e8c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:38.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.171+0000 7fafd6a00700 1 -- 172.21.15.27:0/3034762946 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafc8004160 con 0x7fafd81098e0 2024-05-10T10:58:38.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.172+0000 7fafd6a00700 1 -- 172.21.15.27:0/3034762946 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fafc8004300 con 0x7fafd81098e0 2024-05-10T10:58:38.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.172+0000 7fafd6a00700 1 -- 172.21.15.27:0/3034762946 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafc8011af0 con 0x7fafd81098e0 2024-05-10T10:58:38.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.172+0000 7fafdf200700 1 -- 172.21.15.27:0/3034762946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 msgr2=0x7fafd810bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:38.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.172+0000 7fafdf200700 1 --2- 172.21.15.27:0/3034762946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 0x7fafd810bcd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fafc80098a0 tx=0x7fafc8009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:38.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.172+0000 7fafdf200700 1 -- 172.21.15.27:0/3034762946 shutdown_connections 2024-05-10T10:58:38.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.172+0000 7fafdf200700 1 --2- 172.21.15.27:0/3034762946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 0x7fafd810bcd0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:38.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.172+0000 7fafdf200700 1 -- 172.21.15.27:0/3034762946 >> 172.21.15.27:0/3034762946 conn(0x7fafd81032b0 msgr2=0x7fafd8105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:38.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.173+0000 7fafdf200700 1 -- 172.21.15.27:0/3034762946 shutdown_connections 2024-05-10T10:58:38.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.173+0000 7fafdf200700 1 -- 172.21.15.27:0/3034762946 wait complete. 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.174+0000 7fafdf200700 1 Processor -- start 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.174+0000 7fafdf200700 1 -- start start 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.174+0000 7fafdf200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 0x7fafd819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.174+0000 7fafdf200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fafd810d160 con 0x7fafd81098e0 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.174+0000 7fafd7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 0x7fafd819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.175+0000 7fafd7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 0x7fafd819efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46374/0 (socket says 172.21.15.27:46374) 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.175+0000 7fafd7e00700 1 -- 172.21.15.27:0/2237302155 learned_addr learned my addr 172.21.15.27:0/2237302155 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.175+0000 7fafd7e00700 1 -- 172.21.15.27:0/2237302155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd819f520 con 0x7fafd81098e0 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.175+0000 7fafd7e00700 1 --2- 172.21.15.27:0/2237302155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 0x7fafd819efe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafc8011ac0 tx=0x7fafc8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.176+0000 7fafd4c00700 1 -- 172.21.15.27:0/2237302155 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafc8003970 con 0x7fafd81098e0 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.176+0000 7fafd4c00700 1 -- 172.21.15.27:0/2237302155 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fafc8003b10 con 0x7fafd81098e0 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.176+0000 7fafd4c00700 1 -- 172.21.15.27:0/2237302155 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafc8024610 con 0x7fafd81098e0 2024-05-10T10:58:38.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.176+0000 7fafdf200700 1 -- 172.21.15.27:0/2237302155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd819f810 con 0x7fafd81098e0 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.176+0000 7fafdf200700 1 -- 172.21.15.27:0/2237302155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd81a02d0 con 0x7fafd81098e0 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.177+0000 7fafd4c00700 1 -- 172.21.15.27:0/2237302155 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7fafc801a9e0 con 0x7fafd81098e0 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.177+0000 7fafdf200700 1 -- 172.21.15.27:0/2237302155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd8064b00 con 0x7fafd81098e0 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.178+0000 7fafd4c00700 1 --2- 172.21.15.27:0/2237302155 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fafbc03e580 0x7fafbc040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.178+0000 7fafd4c00700 1 -- 172.21.15.27:0/2237302155 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fafc80537f0 con 0x7fafd81098e0 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.178+0000 7fafd7400700 1 -- 172.21.15.27:0/2237302155 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fafbc03e580 msgr2=0x7fafbc040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2058505174 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.178+0000 7fafd7400700 1 --2- 172.21.15.27:0/2237302155 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fafbc03e580 0x7fafbc040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.188+0000 7fafd4c00700 1 -- 172.21.15.27:0/2237302155 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fafc80247f0 con 0x7fafd81098e0 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.379+0000 7fafd7400700 1 -- 172.21.15.27:0/2237302155 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fafbc03e580 msgr2=0x7fafbc040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2058505174 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.379+0000 7fafd7400700 1 --2- 172.21.15.27:0/2237302155 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fafbc03e580 0x7fafbc040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.491+0000 7fafdf200700 1 -- 172.21.15.27:0/2237302155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fafd81a09b0 con 0x7fafd81098e0 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.492+0000 7fafd4c00700 1 -- 172.21.15.27:0/2237302155 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fafc8053210 con 0x7fafd81098e0 2024-05-10T10:58:38.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.495+0000 7fafdf200700 1 -- 172.21.15.27:0/2237302155 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fafbc03e580 msgr2=0x7fafbc040a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T10:58:38.184 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.495+0000 7fafdf200700 1 --2- 172.21.15.27:0/2237302155 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fafbc03e580 0x7fafbc040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:38.184 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.495+0000 7fafdf200700 1 -- 172.21.15.27:0/2237302155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 msgr2=0x7fafd819efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:38.184 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.495+0000 7fafdf200700 1 --2- 172.21.15.27:0/2237302155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 0x7fafd819efe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafc8011ac0 tx=0x7fafc8011ba0 comp rx=0 tx=0).stop 2024-05-10T10:58:38.184 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.496+0000 7fafdf200700 1 -- 172.21.15.27:0/2237302155 shutdown_connections 2024-05-10T10:58:38.185 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.496+0000 7fafdf200700 1 --2- 172.21.15.27:0/2237302155 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fafbc03e580 0x7fafbc040a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:38.185 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.496+0000 7fafdf200700 1 --2- 172.21.15.27:0/2237302155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafd81098e0 0x7fafd819efe0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:38.185 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.496+0000 7fafdf200700 1 -- 172.21.15.27:0/2237302155 >> 172.21.15.27:0/2237302155 conn(0x7fafd81032b0 msgr2=0x7fafd8103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:38.185 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.496+0000 7fafdf200700 1 -- 172.21.15.27:0/2237302155 shutdown_connections 2024-05-10T10:58:38.185 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:37.496+0000 7fafdf200700 1 -- 172.21.15.27:0/2237302155 wait complete. 2024-05-10T10:58:38.185 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-05-10T10:58:38.185 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 5... 2024-05-10T10:58:38.784 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:38 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:38.661+0000 7facfdd12200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-10T10:58:39.118 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:38 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:38.904+0000 7facfdd12200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-10T10:58:39.395 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:39 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:39.129+0000 7facfdd12200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-10T10:58:39.654 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:39 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:39.525+0000 7facfdd12200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-10T10:58:39.654 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:39 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:39.653+0000 7facfdd12200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-10T10:58:40.178 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:39 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:39.915+0000 7facfdd12200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-10T10:58:40.178 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:40 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:40.177+0000 7facfdd12200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-10T10:58:41.298 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:41 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:41.001+0000 7facfdd12200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-10T10:58:41.298 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:41 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:41.297+0000 7facfdd12200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-10T10:58:42.118 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:41 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:41.872+0000 7facfdd12200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-10T10:58:42.119 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:42 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:42.006+0000 7facfdd12200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-10T10:58:42.869 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:42 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:42.487+0000 7facfdd12200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-10T10:58:43.368 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:42 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:42.960+0000 7facfdd12200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-10T10:58:43.369 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:43 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:43.091+0000 7facfdd12200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-10T10:58:43.753 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:43 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:43.431+0000 7facfdd12200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-10T10:58:43.753 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:43 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:43.618+0000 7facfdd12200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-10T10:58:43.753 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:43 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:43.753+0000 7facfdd12200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-10T10:58:44.618 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:44 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:44.199+0000 7facfdd12200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-10T10:58:44.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:44 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:44.331+0000 7facfdd12200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-10T10:58:45.368 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:45 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:45.131+0000 7facfdd12200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-10T10:58:45.762 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:45 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:58:45.433+0000 7facfdd12200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-10T10:58:45.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:45 smithi027 ceph-mon[30054]: Active manager daemon x restarted 2024-05-10T10:58:45.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:45 smithi027 ceph-mon[30054]: Activating manager daemon x 2024-05-10T10:58:45.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:45 smithi027 ceph-mon[30054]: osdmap e2: 0 total, 0 up, 0 in 2024-05-10T10:58:45.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:45 smithi027 ceph-mon[30054]: mgrmap e6: x(active, starting, since 0.00505388s) 2024-05-10T10:58:45.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:45 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T10:58:45.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:45 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-10T10:58:45.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:45 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-10T10:58:45.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:45 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-10T10:58:45.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:45 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-10T10:58:45.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:45 smithi027 ceph-mon[30054]: Manager daemon x is now available 2024-05-10T10:58:46.826 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:46 smithi027 ceph-mon[30054]: Found migration_current of "None". Setting to last migration. 2024-05-10T10:58:46.826 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:46 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:58:46.826 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:46 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:58:46.826 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:46 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T10:58:46.826 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:46 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T10:58:46.826 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:46 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-10T10:58:46.826 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:46 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-10T10:58:46.827 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:46 smithi027 ceph-mon[30054]: mgrmap e7: x(active, since 1.01669s) 2024-05-10T10:58:47.185 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-10T10:58:47.185 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-10T10:58:47.185 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-10T10:58:47.185 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-10T10:58:47.185 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.784+0000 7fbb2ea00700 1 Processor -- start 2024-05-10T10:58:47.185 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.784+0000 7fbb2ea00700 1 -- start start 2024-05-10T10:58:47.186 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.784+0000 7fbb2ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 0x7fbb2810bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:47.186 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.784+0000 7fbb2ea00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2807ce10 con 0x7fbb28109950 2024-05-10T10:58:47.186 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.785+0000 7fbb27e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 0x7fbb2810bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:47.186 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.785+0000 7fbb27e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 0x7fbb2810bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46386/0 (socket says 172.21.15.27:46386) 2024-05-10T10:58:47.186 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.785+0000 7fbb27e00700 1 -- 172.21.15.27:0/3630378245 learned_addr learned my addr 172.21.15.27:0/3630378245 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:47.186 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.785+0000 7fbb27e00700 1 -- 172.21.15.27:0/3630378245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2810c280 con 0x7fbb28109950 2024-05-10T10:58:47.186 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.786+0000 7fbb27e00700 1 --2- 172.21.15.27:0/3630378245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 0x7fbb2810bd40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbb180098a0 tx=0x7fbb18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0c6c270cf19272 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:47.186 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.786+0000 7fbb26a00700 1 -- 172.21.15.27:0/3630378245 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb18004160 con 0x7fbb28109950 2024-05-10T10:58:47.186 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.786+0000 7fbb26a00700 1 -- 172.21.15.27:0/3630378245 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbb18004300 con 0x7fbb28109950 2024-05-10T10:58:47.186 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.786+0000 7fbb26a00700 1 -- 172.21.15.27:0/3630378245 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb18011af0 con 0x7fbb28109950 2024-05-10T10:58:47.186 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.787+0000 7fbb2ea00700 1 -- 172.21.15.27:0/3630378245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 msgr2=0x7fbb2810bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:47.186 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.787+0000 7fbb2ea00700 1 --2- 172.21.15.27:0/3630378245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 0x7fbb2810bd40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbb180098a0 tx=0x7fbb18009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.787+0000 7fbb2ea00700 1 -- 172.21.15.27:0/3630378245 shutdown_connections 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.787+0000 7fbb2ea00700 1 --2- 172.21.15.27:0/3630378245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 0x7fbb2810bd40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.787+0000 7fbb2ea00700 1 -- 172.21.15.27:0/3630378245 >> 172.21.15.27:0/3630378245 conn(0x7fbb281032c0 msgr2=0x7fbb28105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.787+0000 7fbb2ea00700 1 -- 172.21.15.27:0/3630378245 shutdown_connections 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.787+0000 7fbb2ea00700 1 -- 172.21.15.27:0/3630378245 wait complete. 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.788+0000 7fbb2ea00700 1 Processor -- start 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.789+0000 7fbb2ea00700 1 -- start start 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.789+0000 7fbb2ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 0x7fbb2819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.789+0000 7fbb2ea00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2810d1e0 con 0x7fbb28109950 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.789+0000 7fbb27e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 0x7fbb2819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.789+0000 7fbb27e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 0x7fbb2819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46396/0 (socket says 172.21.15.27:46396) 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.789+0000 7fbb27e00700 1 -- 172.21.15.27:0/2606673681 learned_addr learned my addr 172.21.15.27:0/2606673681 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.790+0000 7fbb27e00700 1 -- 172.21.15.27:0/2606673681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2819f4f0 con 0x7fbb28109950 2024-05-10T10:58:47.187 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.790+0000 7fbb27e00700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 0x7fbb2819efb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb18011ac0 tx=0x7fbb18011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.790+0000 7fbb24c00700 1 -- 172.21.15.27:0/2606673681 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb18003970 con 0x7fbb28109950 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.790+0000 7fbb24c00700 1 -- 172.21.15.27:0/2606673681 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbb18003b10 con 0x7fbb28109950 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.790+0000 7fbb24c00700 1 -- 172.21.15.27:0/2606673681 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb18024610 con 0x7fbb28109950 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.790+0000 7fbb2ea00700 1 -- 172.21.15.27:0/2606673681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb2819f780 con 0x7fbb28109950 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.791+0000 7fbb2ea00700 1 -- 172.21.15.27:0/2606673681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb281a2470 con 0x7fbb28109950 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.792+0000 7fbb24c00700 1 -- 172.21.15.27:0/2606673681 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7fbb1801a9e0 con 0x7fbb28109950 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.792+0000 7fbb24c00700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 0x7fbb0c040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.792+0000 7fbb24c00700 1 -- 172.21.15.27:0/2606673681 --> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbb0c041150 con 0x7fbb0c03e580 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.792+0000 7fbb24c00700 1 -- 172.21.15.27:0/2606673681 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbb18053870 con 0x7fbb28109950 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.792+0000 7fbb27400700 1 -- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 msgr2=0x7fbb0c040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2058505174 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.793+0000 7fbb27400700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 0x7fbb0c040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.994+0000 7fbb27400700 1 -- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 msgr2=0x7fbb0c040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2058505174 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:38.994+0000 7fbb27400700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 0x7fbb0c040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-10T10:58:47.188 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:39.395+0000 7fbb27400700 1 -- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 msgr2=0x7fbb0c040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2058505174 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:39.395+0000 7fbb27400700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 0x7fbb0c040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:40.196+0000 7fbb27400700 1 -- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 msgr2=0x7fbb0c040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2058505174 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:40.196+0000 7fbb27400700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 0x7fbb0c040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:41.798+0000 7fbb27400700 1 -- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 msgr2=0x7fbb0c040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2058505174 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:41.798+0000 7fbb27400700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 0x7fbb0c040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:45.001+0000 7fbb27400700 1 -- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 msgr2=0x7fbb0c040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2058505174 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:45.001+0000 7fbb27400700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 0x7fbb0c040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:45.441+0000 7fbb24c00700 1 -- 172.21.15.27:0/2606673681 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 6) ==== 49989+0+0 (secure 0 0 0) 0x7fbb1801a430 con 0x7fbb28109950 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:45.442+0000 7fbb24c00700 1 -- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 msgr2=0x7fbb0c040a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:45.442+0000 7fbb24c00700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 0x7fbb0c040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.446+0000 7fbb24c00700 1 -- 172.21.15.27:0/2606673681 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7fbb18055560 con 0x7fbb28109950 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.446+0000 7fbb24c00700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7fbb0c041e70 0x7fbb0c044260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.447+0000 7fbb24c00700 1 -- 172.21.15.27:0/2606673681 --> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbb0c041150 con 0x7fbb0c041e70 2024-05-10T10:58:47.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.449+0000 7fbb27400700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7fbb0c041e70 0x7fbb0c044260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.452+0000 7fbb27400700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7fbb0c041e70 0x7fbb0c044260 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbb14003a10 tx=0x7fbb140092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.454+0000 7fbb24c00700 1 -- 172.21.15.27:0/2606673681 <== mgr.14120 v2:172.21.15.27:6800/481366655 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fbb0c041150 con 0x7fbb0c041e70 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.465+0000 7fbb2ea00700 1 -- 172.21.15.27:0/2606673681 --> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fbb2806a5d0 con 0x7fbb0c041e70 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.466+0000 7fbb24c00700 1 -- 172.21.15.27:0/2606673681 <== mgr.14120 v2:172.21.15.27:6800/481366655 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fbb2806a5d0 con 0x7fbb0c041e70 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.466+0000 7fbb2ea00700 1 -- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7fbb0c041e70 msgr2=0x7fbb0c044260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.466+0000 7fbb2ea00700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7fbb0c041e70 0x7fbb0c044260 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbb14003a10 tx=0x7fbb140092b0 comp rx=0 tx=0).stop 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.466+0000 7fbb2ea00700 1 -- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 msgr2=0x7fbb2819efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.466+0000 7fbb2ea00700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 0x7fbb2819efb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb18011ac0 tx=0x7fbb18011ba0 comp rx=0 tx=0).stop 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.466+0000 7fbb2ea00700 1 -- 172.21.15.27:0/2606673681 shutdown_connections 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.466+0000 7fbb2ea00700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7fbb0c041e70 0x7fbb0c044260 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.466+0000 7fbb2ea00700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb28109950 0x7fbb2819efb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.466+0000 7fbb2ea00700 1 --2- 172.21.15.27:0/2606673681 >> [v2:172.21.15.27:6800/2058505174,v1:172.21.15.27:6801/2058505174] conn(0x7fbb0c03e580 0x7fbb0c040a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:47.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.466+0000 7fbb2ea00700 1 -- 172.21.15.27:0/2606673681 >> 172.21.15.27:0/2606673681 conn(0x7fbb281032c0 msgr2=0x7fbb28103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:47.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.467+0000 7fbb2ea00700 1 -- 172.21.15.27:0/2606673681 shutdown_connections 2024-05-10T10:58:47.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:46.467+0000 7fbb2ea00700 1 -- 172.21.15.27:0/2606673681 wait complete. 2024-05-10T10:58:47.191 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 5 is available 2024-05-10T10:58:47.191 INFO:teuthology.orchestra.run.smithi027.stdout:Setting orchestrator backend to cephadm... 2024-05-10T10:58:48.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:47 smithi027 ceph-mon[30054]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-10T10:58:48.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:47 smithi027 ceph-mon[30054]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-10T10:58:48.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:47 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:58:48.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:47 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:58:48.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:47 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T10:58:48.686 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.762+0000 7fb6f5200700 1 Processor -- start 2024-05-10T10:58:48.687 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.762+0000 7fb6f5200700 1 -- start start 2024-05-10T10:58:48.688 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.762+0000 7fb6f5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 0x7fb6f0106df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:48.688 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.762+0000 7fb6f5200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f0107330 con 0x7fb6f01069d0 2024-05-10T10:58:48.688 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.763+0000 7fb6ef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 0x7fb6f0106df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:48.689 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.763+0000 7fb6ef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 0x7fb6f0106df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54776/0 (socket says 172.21.15.27:54776) 2024-05-10T10:58:48.689 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.763+0000 7fb6ef400700 1 -- 172.21.15.27:0/2356620532 learned_addr learned my addr 172.21.15.27:0/2356620532 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:48.689 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.763+0000 7fb6ef400700 1 -- 172.21.15.27:0/2356620532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f01074b0 con 0x7fb6f01069d0 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.764+0000 7fb6ef400700 1 --2- 172.21.15.27:0/2356620532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 0x7fb6f0106df0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb6e00098a0 tx=0x7fb6e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3272869d2ffdf76d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.764+0000 7fb6eea00700 1 -- 172.21.15.27:0/2356620532 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6e0004160 con 0x7fb6f01069d0 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.764+0000 7fb6eea00700 1 -- 172.21.15.27:0/2356620532 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb6e0004300 con 0x7fb6f01069d0 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.764+0000 7fb6eea00700 1 -- 172.21.15.27:0/2356620532 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6e0011af0 con 0x7fb6f01069d0 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.765+0000 7fb6f5200700 1 -- 172.21.15.27:0/2356620532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 msgr2=0x7fb6f0106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.765+0000 7fb6f5200700 1 --2- 172.21.15.27:0/2356620532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 0x7fb6f0106df0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb6e00098a0 tx=0x7fb6e0009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.765+0000 7fb6f5200700 1 -- 172.21.15.27:0/2356620532 shutdown_connections 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.765+0000 7fb6f5200700 1 --2- 172.21.15.27:0/2356620532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 0x7fb6f0106df0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.765+0000 7fb6f5200700 1 -- 172.21.15.27:0/2356620532 >> 172.21.15.27:0/2356620532 conn(0x7fb6f0102470 msgr2=0x7fb6f01048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.765+0000 7fb6f5200700 1 -- 172.21.15.27:0/2356620532 shutdown_connections 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.765+0000 7fb6f5200700 1 -- 172.21.15.27:0/2356620532 wait complete. 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.766+0000 7fb6f5200700 1 Processor -- start 2024-05-10T10:58:48.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.766+0000 7fb6f5200700 1 -- start start 2024-05-10T10:58:48.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.767+0000 7fb6f5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 0x7fb6f0109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:48.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.767+0000 7fb6f5200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f010d000 con 0x7fb6f01069d0 2024-05-10T10:58:48.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.767+0000 7fb6ef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 0x7fb6f0109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:48.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.767+0000 7fb6ef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 0x7fb6f0109ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54786/0 (socket says 172.21.15.27:54786) 2024-05-10T10:58:48.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.767+0000 7fb6ef400700 1 -- 172.21.15.27:0/2376136217 learned_addr learned my addr 172.21.15.27:0/2376136217 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:48.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.767+0000 7fb6ef400700 1 -- 172.21.15.27:0/2376136217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f010a0e0 con 0x7fb6f01069d0 2024-05-10T10:58:48.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.768+0000 7fb6ef400700 1 --2- 172.21.15.27:0/2376136217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 0x7fb6f0109ba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0011ac0 tx=0x7fb6e0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.768+0000 7fb6ecc00700 1 -- 172.21.15.27:0/2376136217 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6e0003b20 con 0x7fb6f01069d0 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.768+0000 7fb6ecc00700 1 -- 172.21.15.27:0/2376136217 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb6e001a8c0 con 0x7fb6f01069d0 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.768+0000 7fb6ecc00700 1 -- 172.21.15.27:0/2376136217 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6e00248d0 con 0x7fb6f01069d0 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.768+0000 7fb6f5200700 1 -- 172.21.15.27:0/2376136217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6f01082d0 con 0x7fb6f01069d0 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.768+0000 7fb6f5200700 1 -- 172.21.15.27:0/2376136217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6f0108810 con 0x7fb6f01069d0 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.769+0000 7fb6f5200700 1 -- 172.21.15.27:0/2376136217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6f006a5d0 con 0x7fb6f01069d0 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.769+0000 7fb6ecc00700 1 -- 172.21.15.27:0/2376136217 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7fb6e0022020 con 0x7fb6f01069d0 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.770+0000 7fb6ecc00700 1 --2- 172.21.15.27:0/2376136217 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7fb6d403e460 0x7fb6d4040920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.770+0000 7fb6ecc00700 1 -- 172.21.15.27:0/2376136217 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb6e00536d0 con 0x7fb6f01069d0 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.770+0000 7fb6e6a00700 1 --2- 172.21.15.27:0/2376136217 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7fb6d403e460 0x7fb6d4040920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.771+0000 7fb6e6a00700 1 --2- 172.21.15.27:0/2376136217 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7fb6d403e460 0x7fb6d4040920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8006fd0 tx=0x7fb6d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.776+0000 7fb6ecc00700 1 -- 172.21.15.27:0/2376136217 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb6e001ab20 con 0x7fb6f01069d0 2024-05-10T10:58:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:47.868+0000 7fb6ecc00700 1 -- 172.21.15.27:0/2376136217 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fb6e001a430 con 0x7fb6f01069d0 2024-05-10T10:58:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:48.024+0000 7fb6f5200700 1 -- 172.21.15.27:0/2376136217 --> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fb6f01035b0 con 0x7fb6d403e460 2024-05-10T10:58:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:48.044+0000 7fb6ecc00700 1 -- 172.21.15.27:0/2376136217 <== mgr.14120 v2:172.21.15.27:6800/481366655 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb6f01035b0 con 0x7fb6d403e460 2024-05-10T10:58:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:48.047+0000 7fb6f5200700 1 -- 172.21.15.27:0/2376136217 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7fb6d403e460 msgr2=0x7fb6d4040920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:48.047+0000 7fb6f5200700 1 --2- 172.21.15.27:0/2376136217 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7fb6d403e460 0x7fb6d4040920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8006fd0 tx=0x7fb6d8006e40 comp rx=0 tx=0).stop 2024-05-10T10:58:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:48.047+0000 7fb6f5200700 1 -- 172.21.15.27:0/2376136217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 msgr2=0x7fb6f0109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:48.047+0000 7fb6f5200700 1 --2- 172.21.15.27:0/2376136217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 0x7fb6f0109ba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0011ac0 tx=0x7fb6e0011ba0 comp rx=0 tx=0).stop 2024-05-10T10:58:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:48.047+0000 7fb6f5200700 1 -- 172.21.15.27:0/2376136217 shutdown_connections 2024-05-10T10:58:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:48.047+0000 7fb6f5200700 1 --2- 172.21.15.27:0/2376136217 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7fb6d403e460 0x7fb6d4040920 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:48.047+0000 7fb6f5200700 1 --2- 172.21.15.27:0/2376136217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6f01069d0 0x7fb6f0109ba0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:48.047+0000 7fb6f5200700 1 -- 172.21.15.27:0/2376136217 >> 172.21.15.27:0/2376136217 conn(0x7fb6f0102470 msgr2=0x7fb6f0102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:48.048+0000 7fb6f5200700 1 -- 172.21.15.27:0/2376136217 shutdown_connections 2024-05-10T10:58:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:48.048+0000 7fb6f5200700 1 -- 172.21.15.27:0/2376136217 wait complete. 2024-05-10T10:58:49.072 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:48 smithi027 ceph-mon[30054]: [10/May/2024:10:58:47] ENGINE Bus STARTING 2024-05-10T10:58:49.073 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:48 smithi027 ceph-mon[30054]: [10/May/2024:10:58:47] ENGINE Serving on https://172.21.15.27:7150 2024-05-10T10:58:49.073 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:48 smithi027 ceph-mon[30054]: [10/May/2024:10:58:47] ENGINE Serving on http://172.21.15.27:8765 2024-05-10T10:58:49.073 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:48 smithi027 ceph-mon[30054]: [10/May/2024:10:58:47] ENGINE Bus STARTED 2024-05-10T10:58:49.073 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:48 smithi027 ceph-mon[30054]: mgrmap e8: x(active, since 2s) 2024-05-10T10:58:49.073 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:48 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:58:49.073 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:48 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T10:58:50.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:49 smithi027 ceph-mon[30054]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T10:58:50.312 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-10T10:58:50.312 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.392+0000 7f224ec00700 1 Processor -- start 2024-05-10T10:58:50.312 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.392+0000 7f224ec00700 1 -- start start 2024-05-10T10:58:50.312 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.393+0000 7f224ec00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 0x7f22480832d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:50.312 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.393+0000 7f224ec00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f22480838a0 con 0x7f2248082eb0 2024-05-10T10:58:50.312 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.393+0000 7f2247e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 0x7f22480832d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.393+0000 7f2247e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 0x7f22480832d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54790/0 (socket says 172.21.15.27:54790) 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.393+0000 7f2247e00700 1 -- 172.21.15.27:0/781820447 learned_addr learned my addr 172.21.15.27:0/781820447 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.394+0000 7f2247e00700 1 -- 172.21.15.27:0/781820447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2248110680 con 0x7f2248082eb0 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.395+0000 7f2247e00700 1 --2- 172.21.15.27:0/781820447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 0x7f22480832d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f22380098a0 tx=0x7f2238009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd8fb911d764a02 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.395+0000 7f2246a00700 1 -- 172.21.15.27:0/781820447 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2238004160 con 0x7f2248082eb0 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.395+0000 7f2246a00700 1 -- 172.21.15.27:0/781820447 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2238004300 con 0x7f2248082eb0 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.395+0000 7f2246a00700 1 -- 172.21.15.27:0/781820447 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2238011af0 con 0x7f2248082eb0 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.396+0000 7f224ec00700 1 -- 172.21.15.27:0/781820447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 msgr2=0x7f22480832d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.396+0000 7f224ec00700 1 --2- 172.21.15.27:0/781820447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 0x7f22480832d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f22380098a0 tx=0x7f2238009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.396+0000 7f224ec00700 1 -- 172.21.15.27:0/781820447 shutdown_connections 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.396+0000 7f224ec00700 1 --2- 172.21.15.27:0/781820447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 0x7f22480832d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.396+0000 7f224ec00700 1 -- 172.21.15.27:0/781820447 >> 172.21.15.27:0/781820447 conn(0x7f224807e450 msgr2=0x7f2248080890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.396+0000 7f224ec00700 1 -- 172.21.15.27:0/781820447 shutdown_connections 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.396+0000 7f224ec00700 1 -- 172.21.15.27:0/781820447 wait complete. 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.397+0000 7f224ec00700 1 Processor -- start 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.397+0000 7f224ec00700 1 -- start start 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.398+0000 7f224ec00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 0x7f22481a3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.398+0000 7f224ec00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2248113a30 con 0x7f2248082eb0 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.398+0000 7f2247e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 0x7f22481a3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.398+0000 7f2247e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 0x7f22481a3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54802/0 (socket says 172.21.15.27:54802) 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.398+0000 7f2247e00700 1 -- 172.21.15.27:0/1999802733 learned_addr learned my addr 172.21.15.27:0/1999802733 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.398+0000 7f2247e00700 1 -- 172.21.15.27:0/1999802733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22481a3860 con 0x7f2248082eb0 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.399+0000 7f2247e00700 1 --2- 172.21.15.27:0/1999802733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 0x7f22481a3320 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2238000c00 tx=0x7f2238011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.399+0000 7f2244c00700 1 -- 172.21.15.27:0/1999802733 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2238003b20 con 0x7f2248082eb0 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.399+0000 7f2244c00700 1 -- 172.21.15.27:0/1999802733 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f223801a8c0 con 0x7f2248082eb0 2024-05-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.399+0000 7f224ec00700 1 -- 172.21.15.27:0/1999802733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22481a3af0 con 0x7f2248082eb0 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.399+0000 7f2244c00700 1 -- 172.21.15.27:0/1999802733 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2238024940 con 0x7f2248082eb0 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.399+0000 7f224ec00700 1 -- 172.21.15.27:0/1999802733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22481a67e0 con 0x7f2248082eb0 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.400+0000 7f224ec00700 1 -- 172.21.15.27:0/1999802733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f224806a5d0 con 0x7f2248082eb0 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.400+0000 7f2244c00700 1 -- 172.21.15.27:0/1999802733 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f2238022020 con 0x7f2248082eb0 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.401+0000 7f2244c00700 1 --2- 172.21.15.27:0/1999802733 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f222c03e530 0x7f222c0409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.401+0000 7f2244c00700 1 -- 172.21.15.27:0/1999802733 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2238053760 con 0x7f2248082eb0 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.401+0000 7f2247400700 1 --2- 172.21.15.27:0/1999802733 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f222c03e530 0x7f222c0409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.402+0000 7f2247400700 1 --2- 172.21.15.27:0/1999802733 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f222c03e530 0x7f222c0409f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2234006fd0 tx=0x7f2234006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.407+0000 7f2244c00700 1 -- 172.21.15.27:0/1999802733 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f2238055ce0 con 0x7f2248082eb0 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.650+0000 7f224ec00700 1 -- 172.21.15.27:0/1999802733 --> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f224807f620 con 0x7f222c03e530 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.651+0000 7f2244c00700 1 -- 172.21.15.27:0/1999802733 <== mgr.14120 v2:172.21.15.27:6800/481366655 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f224807f620 con 0x7f222c03e530 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.654+0000 7f224ec00700 1 -- 172.21.15.27:0/1999802733 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f222c03e530 msgr2=0x7f222c0409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.654+0000 7f224ec00700 1 --2- 172.21.15.27:0/1999802733 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f222c03e530 0x7f222c0409f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2234006fd0 tx=0x7f2234006e40 comp rx=0 tx=0).stop 2024-05-10T10:58:50.316 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.654+0000 7f224ec00700 1 -- 172.21.15.27:0/1999802733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 msgr2=0x7f22481a3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:50.316 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.654+0000 7f224ec00700 1 --2- 172.21.15.27:0/1999802733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 0x7f22481a3320 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2238000c00 tx=0x7f2238011b70 comp rx=0 tx=0).stop 2024-05-10T10:58:50.316 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.655+0000 7f224ec00700 1 -- 172.21.15.27:0/1999802733 shutdown_connections 2024-05-10T10:58:50.316 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.655+0000 7f224ec00700 1 --2- 172.21.15.27:0/1999802733 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f222c03e530 0x7f222c0409f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:50.316 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.655+0000 7f224ec00700 1 --2- 172.21.15.27:0/1999802733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2248082eb0 0x7f22481a3320 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:50.316 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.655+0000 7f224ec00700 1 -- 172.21.15.27:0/1999802733 >> 172.21.15.27:0/1999802733 conn(0x7f224807e450 msgr2=0x7f224807ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:50.316 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.655+0000 7f224ec00700 1 -- 172.21.15.27:0/1999802733 shutdown_connections 2024-05-10T10:58:50.316 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:49.655+0000 7f224ec00700 1 -- 172.21.15.27:0/1999802733 wait complete. 2024-05-10T10:58:50.316 INFO:teuthology.orchestra.run.smithi027.stdout:Generating ssh key... 2024-05-10T10:58:51.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:50 smithi027 ceph-mon[30054]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T10:58:51.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: Generating public/private rsa key pair. 2024-05-10T10:58:51.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: Your identification has been saved in /tmp/tmp_5r8rpk0/key. 2024-05-10T10:58:51.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: Your public key has been saved in /tmp/tmp_5r8rpk0/key.pub. 2024-05-10T10:58:51.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: The key fingerprint is: 2024-05-10T10:58:51.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: SHA256:ttxkmVE1KjQNtOgLYBenJE6oMZ2pQ/QgK9qyM+jShWE ceph-063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:58:51.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: The key's randomart image is: 2024-05-10T10:58:51.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: +---[RSA 3072]----+ 2024-05-10T10:58:51.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: |o+ +o o ..=o..o | 2024-05-10T10:58:51.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: |+oBo o + o +.. . | 2024-05-10T10:58:51.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: |+= .+ o . + . | 2024-05-10T10:58:51.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: |*.E. o . = | 2024-05-10T10:58:51.621 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: |oo.o . S = | 2024-05-10T10:58:51.621 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: |.o. . + * | 2024-05-10T10:58:51.621 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: |=. . + . | 2024-05-10T10:58:51.621 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: |oo. | 2024-05-10T10:58:51.621 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: |.. | 2024-05-10T10:58:51.621 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:58:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: +----[SHA256]-----+ 2024-05-10T10:58:52.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.959+0000 7f4228800700 1 Processor -- start 2024-05-10T10:58:52.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.960+0000 7f4228800700 1 -- start start 2024-05-10T10:58:52.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.960+0000 7f4228800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 0x7f422007fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:52.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.960+0000 7f4228800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4220080230 con 0x7f422007f840 2024-05-10T10:58:52.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.961+0000 7f4225e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 0x7f422007fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:52.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.961+0000 7f4225e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 0x7f422007fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54812/0 (socket says 172.21.15.27:54812) 2024-05-10T10:58:52.050 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.961+0000 7f4225e00700 1 -- 172.21.15.27:0/3274085469 learned_addr learned my addr 172.21.15.27:0/3274085469 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.961+0000 7f4225e00700 1 -- 172.21.15.27:0/3274085469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4220080a40 con 0x7f422007f840 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.962+0000 7f4225e00700 1 --2- 172.21.15.27:0/3274085469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 0x7f422007fc60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f42140098a0 tx=0x7f4214009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d679678620ce82 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.962+0000 7f4224a00700 1 -- 172.21.15.27:0/3274085469 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4214004160 con 0x7f422007f840 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.962+0000 7f4224a00700 1 -- 172.21.15.27:0/3274085469 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4214004300 con 0x7f422007f840 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.962+0000 7f4224a00700 1 -- 172.21.15.27:0/3274085469 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4214011af0 con 0x7f422007f840 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.963+0000 7f4228800700 1 -- 172.21.15.27:0/3274085469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 msgr2=0x7f422007fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.963+0000 7f4228800700 1 --2- 172.21.15.27:0/3274085469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 0x7f422007fc60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f42140098a0 tx=0x7f4214009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.963+0000 7f4228800700 1 -- 172.21.15.27:0/3274085469 shutdown_connections 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.963+0000 7f4228800700 1 --2- 172.21.15.27:0/3274085469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 0x7f422007fc60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.963+0000 7f4228800700 1 -- 172.21.15.27:0/3274085469 >> 172.21.15.27:0/3274085469 conn(0x7f422007e6a0 msgr2=0x7f422007eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.963+0000 7f4228800700 1 -- 172.21.15.27:0/3274085469 shutdown_connections 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.963+0000 7f4228800700 1 -- 172.21.15.27:0/3274085469 wait complete. 2024-05-10T10:58:52.051 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.964+0000 7f4228800700 1 Processor -- start 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.965+0000 7f4228800700 1 -- start start 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.965+0000 7f4228800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 0x7f42201a3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.965+0000 7f4228800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4220081c80 con 0x7f422007f840 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.965+0000 7f4225e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 0x7f42201a3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.965+0000 7f4225e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 0x7f42201a3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54816/0 (socket says 172.21.15.27:54816) 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.965+0000 7f4225e00700 1 -- 172.21.15.27:0/925577279 learned_addr learned my addr 172.21.15.27:0/925577279 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.966+0000 7f4225e00700 1 -- 172.21.15.27:0/925577279 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42201a3950 con 0x7f422007f840 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.966+0000 7f4225e00700 1 --2- 172.21.15.27:0/925577279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 0x7f42201a3410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4214011ac0 tx=0x7f4214011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.966+0000 7f421aa00700 1 -- 172.21.15.27:0/925577279 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4214003970 con 0x7f422007f840 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.966+0000 7f421aa00700 1 -- 172.21.15.27:0/925577279 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4214003b10 con 0x7f422007f840 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.966+0000 7f421aa00700 1 -- 172.21.15.27:0/925577279 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4214024610 con 0x7f422007f840 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.966+0000 7f4228800700 1 -- 172.21.15.27:0/925577279 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42201a3be0 con 0x7f422007f840 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.967+0000 7f4228800700 1 -- 172.21.15.27:0/925577279 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42201a4700 con 0x7f422007f840 2024-05-10T10:58:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.968+0000 7f4228800700 1 -- 172.21.15.27:0/925577279 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4220064b00 con 0x7f422007f840 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.968+0000 7f421aa00700 1 -- 172.21.15.27:0/925577279 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f421401a9e0 con 0x7f422007f840 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.969+0000 7f421aa00700 1 --2- 172.21.15.27:0/925577279 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f420803e580 0x7f4208040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.969+0000 7f421aa00700 1 -- 172.21.15.27:0/925577279 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4214053960 con 0x7f422007f840 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.969+0000 7f4225400700 1 --2- 172.21.15.27:0/925577279 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f420803e580 0x7f4208040a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.970+0000 7f4225400700 1 --2- 172.21.15.27:0/925577279 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f420803e580 0x7f4208040a40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4210006fd0 tx=0x7f4210006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:50.974+0000 7f421aa00700 1 -- 172.21.15.27:0/925577279 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f42140532b0 con 0x7f422007f840 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:51.213+0000 7f4228800700 1 -- 172.21.15.27:0/925577279 --> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f42201a4c40 con 0x7f420803e580 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:51.376+0000 7f421aa00700 1 -- 172.21.15.27:0/925577279 <== mgr.14120 v2:172.21.15.27:6800/481366655 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f42201a4c40 con 0x7f420803e580 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:51.379+0000 7f4228800700 1 -- 172.21.15.27:0/925577279 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f420803e580 msgr2=0x7f4208040a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:51.379+0000 7f4228800700 1 --2- 172.21.15.27:0/925577279 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f420803e580 0x7f4208040a40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4210006fd0 tx=0x7f4210006e40 comp rx=0 tx=0).stop 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:51.379+0000 7f4228800700 1 -- 172.21.15.27:0/925577279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 msgr2=0x7f42201a3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:51.379+0000 7f4228800700 1 --2- 172.21.15.27:0/925577279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 0x7f42201a3410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4214011ac0 tx=0x7f4214011ba0 comp rx=0 tx=0).stop 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:51.379+0000 7f4228800700 1 -- 172.21.15.27:0/925577279 shutdown_connections 2024-05-10T10:58:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:51.379+0000 7f4228800700 1 --2- 172.21.15.27:0/925577279 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f420803e580 0x7f4208040a40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:52.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:51.379+0000 7f4228800700 1 --2- 172.21.15.27:0/925577279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f422007f840 0x7f42201a3410 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:52.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:51.379+0000 7f4228800700 1 -- 172.21.15.27:0/925577279 >> 172.21.15.27:0/925577279 conn(0x7f422007e6a0 msgr2=0x7f4220109730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:52.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:51.379+0000 7f4228800700 1 -- 172.21.15.27:0/925577279 shutdown_connections 2024-05-10T10:58:52.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:51.380+0000 7f4228800700 1 -- 172.21.15.27:0/925577279 wait complete. 2024-05-10T10:58:52.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:52 smithi027 ceph-mon[30054]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T10:58:52.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:52 smithi027 ceph-mon[30054]: Generating ssh key... 2024-05-10T10:58:52.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:52 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:58:52.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:52 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:58:53.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpYPlGk5o+o4rwUpqYnMkbHtINiEkiuWeJ4AzmOPMcl8WQMpL/cPIn+G13OE8bPb4ovQw5YDyPu5u3DQS5MuaQXkfyvLWt3YUgs4t+SP2QkgLccSYaSd+JpF2znmhKThVkvVzjc0RT6+CCPd3qBfx4cHYH2igSveR4+lGUi2bXvagTJVlpSxkcql0Xs3bouCpv5YuWM6FtyVN5c/oBK0Q5HFEyzy8sVrxsWwu33cTv5vJU0a2dyez/eP/mYjreo85nNjoQK3AkJ6+c1ez84IGKFUHe4Hc3Axe49bYWJZY+v4JTwQal7Obb1OrEVWXjli/umzy2tlVUIlD1Oujkpb3zqdpB6J66leVchiYenN/FuI8JtxK5gzlSPq1NWJ1X/UJ7LrshFAHh41QI6K+7LWosUJWAABpcPlKrX0R/UJxzuUVJecy91MgS8ZjPovALvCYeikYdt8gehOROZkPKuKBZAl3T9dk86eVFlg/qbU0IldIIuRXuyonpWhIj7Jd7DqE= ceph-063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:58:53.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.705+0000 7f88f1e00700 1 Processor -- start 2024-05-10T10:58:53.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.705+0000 7f88f1e00700 1 -- start start 2024-05-10T10:58:53.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.706+0000 7f88f1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 0x7f88ec080060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:53.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.706+0000 7f88f1e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec080630 con 0x7f88ec07fc40 2024-05-10T10:58:53.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.706+0000 7f88f0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 0x7f88ec080060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:53.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.706+0000 7f88f0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 0x7f88ec080060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54832/0 (socket says 172.21.15.27:54832) 2024-05-10T10:58:53.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.706+0000 7f88f0a00700 1 -- 172.21.15.27:0/1190695565 learned_addr learned my addr 172.21.15.27:0/1190695565 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:53.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.707+0000 7f88f0a00700 1 -- 172.21.15.27:0/1190695565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ec080e90 con 0x7f88ec07fc40 2024-05-10T10:58:53.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.707+0000 7f88f0a00700 1 --2- 172.21.15.27:0/1190695565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 0x7f88ec080060 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f88e00098a0 tx=0x7f88e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeacba14e128c63 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:53.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.708+0000 7f88eb400700 1 -- 172.21.15.27:0/1190695565 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88e0004160 con 0x7f88ec07fc40 2024-05-10T10:58:53.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.708+0000 7f88eb400700 1 -- 172.21.15.27:0/1190695565 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f88e0004300 con 0x7f88ec07fc40 2024-05-10T10:58:53.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.708+0000 7f88eb400700 1 -- 172.21.15.27:0/1190695565 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88e0011af0 con 0x7f88ec07fc40 2024-05-10T10:58:53.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.708+0000 7f88f1e00700 1 -- 172.21.15.27:0/1190695565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 msgr2=0x7f88ec080060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:53.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.708+0000 7f88f1e00700 1 --2- 172.21.15.27:0/1190695565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 0x7f88ec080060 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f88e00098a0 tx=0x7f88e0009bb0 comp rx=0 tx=0).stop 2024-05-10T10:58:53.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.708+0000 7f88f1e00700 1 -- 172.21.15.27:0/1190695565 shutdown_connections 2024-05-10T10:58:53.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.708+0000 7f88f1e00700 1 --2- 172.21.15.27:0/1190695565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 0x7f88ec080060 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:53.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.708+0000 7f88f1e00700 1 -- 172.21.15.27:0/1190695565 >> 172.21.15.27:0/1190695565 conn(0x7f88ec07e230 msgr2=0x7f88ec07e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:53.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.709+0000 7f88f1e00700 1 -- 172.21.15.27:0/1190695565 shutdown_connections 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.709+0000 7f88f1e00700 1 -- 172.21.15.27:0/1190695565 wait complete. 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.710+0000 7f88f1e00700 1 Processor -- start 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.710+0000 7f88f1e00700 1 -- start start 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.710+0000 7f88f1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 0x7f88ec1a3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.710+0000 7f88f1e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec082250 con 0x7f88ec07fc40 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.711+0000 7f88f0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 0x7f88ec1a3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.711+0000 7f88f0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 0x7f88ec1a3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54842/0 (socket says 172.21.15.27:54842) 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.711+0000 7f88f0a00700 1 -- 172.21.15.27:0/1109844221 learned_addr learned my addr 172.21.15.27:0/1109844221 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.711+0000 7f88f0a00700 1 -- 172.21.15.27:0/1109844221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ec1a3650 con 0x7f88ec07fc40 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.711+0000 7f88f0a00700 1 --2- 172.21.15.27:0/1109844221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 0x7f88ec1a3110 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f88e0003f40 tx=0x7f88e0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.712+0000 7f88e9600700 1 -- 172.21.15.27:0/1109844221 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88e00038b0 con 0x7f88ec07fc40 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.712+0000 7f88e9600700 1 -- 172.21.15.27:0/1109844221 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f88e0003a50 con 0x7f88ec07fc40 2024-05-10T10:58:53.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.712+0000 7f88e9600700 1 -- 172.21.15.27:0/1109844221 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88e001a5c0 con 0x7f88ec07fc40 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.712+0000 7f88f1e00700 1 -- 172.21.15.27:0/1109844221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88ec1a38e0 con 0x7f88ec07fc40 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.712+0000 7f88f1e00700 1 -- 172.21.15.27:0/1109844221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88ec1a65d0 con 0x7f88ec07fc40 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.713+0000 7f88f1e00700 1 -- 172.21.15.27:0/1109844221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88ec06a5d0 con 0x7f88ec07fc40 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.713+0000 7f88e9600700 1 -- 172.21.15.27:0/1109844221 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f88e0024430 con 0x7f88ec07fc40 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.714+0000 7f88e9600700 1 --2- 172.21.15.27:0/1109844221 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f88d403e4e0 0x7f88d40409a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.714+0000 7f88e9600700 1 -- 172.21.15.27:0/1109844221 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f88e0053b20 con 0x7f88ec07fc40 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.714+0000 7f88ebe00700 1 --2- 172.21.15.27:0/1109844221 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f88d403e4e0 0x7f88d40409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.715+0000 7f88ebe00700 1 --2- 172.21.15.27:0/1109844221 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f88d403e4e0 0x7f88d40409a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f88d8006fd0 tx=0x7f88d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.719+0000 7f88e9600700 1 -- 172.21.15.27:0/1109844221 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f88e000b3e0 con 0x7f88ec07fc40 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.958+0000 7f88f1e00700 1 -- 172.21.15.27:0/1109844221 --> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f88ec0008d0 con 0x7f88d403e4e0 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.960+0000 7f88e9600700 1 -- 172.21.15.27:0/1109844221 <== mgr.14120 v2:172.21.15.27:6800/481366655 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f88ec0008d0 con 0x7f88d403e4e0 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.963+0000 7f88f1e00700 1 -- 172.21.15.27:0/1109844221 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f88d403e4e0 msgr2=0x7f88d40409a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:53.549 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.963+0000 7f88f1e00700 1 --2- 172.21.15.27:0/1109844221 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f88d403e4e0 0x7f88d40409a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f88d8006fd0 tx=0x7f88d8006e40 comp rx=0 tx=0).stop 2024-05-10T10:58:53.550 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.963+0000 7f88f1e00700 1 -- 172.21.15.27:0/1109844221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 msgr2=0x7f88ec1a3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:58:53.550 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.963+0000 7f88f1e00700 1 --2- 172.21.15.27:0/1109844221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 0x7f88ec1a3110 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f88e0003f40 tx=0x7f88e0011890 comp rx=0 tx=0).stop 2024-05-10T10:58:53.550 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.963+0000 7f88f1e00700 1 -- 172.21.15.27:0/1109844221 shutdown_connections 2024-05-10T10:58:53.550 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.963+0000 7f88f1e00700 1 --2- 172.21.15.27:0/1109844221 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f88d403e4e0 0x7f88d40409a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:53.550 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.963+0000 7f88f1e00700 1 --2- 172.21.15.27:0/1109844221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec07fc40 0x7f88ec1a3110 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:58:53.550 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.963+0000 7f88f1e00700 1 -- 172.21.15.27:0/1109844221 >> 172.21.15.27:0/1109844221 conn(0x7f88ec07e230 msgr2=0x7f88ec1097e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:58:53.550 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.964+0000 7f88f1e00700 1 -- 172.21.15.27:0/1109844221 shutdown_connections 2024-05-10T10:58:53.550 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:52.964+0000 7f88f1e00700 1 -- 172.21.15.27:0/1109844221 wait complete. 2024-05-10T10:58:53.550 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-10T10:58:53.550 INFO:teuthology.orchestra.run.smithi027.stdout:Adding key to root@localhost authorized_keys... 2024-05-10T10:58:53.551 INFO:teuthology.orchestra.run.smithi027.stdout:Adding host smithi027... 2024-05-10T10:58:54.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:53 smithi027 ceph-mon[30054]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T10:58:55.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:54 smithi027 ceph-mon[30054]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T10:58:56.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:58:55 smithi027 ceph-mon[30054]: Deploying cephadm binary to smithi027 2024-05-10T10:59:01.589 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Added host 'smithi027' with addr '172.21.15.27' 2024-05-10T10:59:01.589 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.165+0000 7f704a400700 1 Processor -- start 2024-05-10T10:59:01.589 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.165+0000 7f704a400700 1 -- start start 2024-05-10T10:59:01.589 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.165+0000 7f704a400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 0x7f704410b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:01.589 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.165+0000 7f704a400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f704410bad0 con 0x7f704410b0e0 2024-05-10T10:59:01.592 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.166+0000 7f7043400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 0x7f704410b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:01.592 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.166+0000 7f7043400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 0x7f704410b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41334/0 (socket says 172.21.15.27:41334) 2024-05-10T10:59:01.592 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.166+0000 7f7043400700 1 -- 172.21.15.27:0/1995702865 learned_addr learned my addr 172.21.15.27:0/1995702865 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:01.592 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.166+0000 7f7043400700 1 -- 172.21.15.27:0/1995702865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704410c2a0 con 0x7f704410b0e0 2024-05-10T10:59:01.592 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.166+0000 7f7043400700 1 --2- 172.21.15.27:0/1995702865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 0x7f704410b500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f70340098a0 tx=0x7f7034009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261ae132450258d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:01.592 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.167+0000 7f7042000700 1 -- 172.21.15.27:0/1995702865 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7034004160 con 0x7f704410b0e0 2024-05-10T10:59:01.593 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.167+0000 7f7042000700 1 -- 172.21.15.27:0/1995702865 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7034004300 con 0x7f704410b0e0 2024-05-10T10:59:01.593 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.167+0000 7f7042000700 1 -- 172.21.15.27:0/1995702865 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7034011af0 con 0x7f704410b0e0 2024-05-10T10:59:01.593 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.168+0000 7f704a400700 1 -- 172.21.15.27:0/1995702865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 msgr2=0x7f704410b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:01.593 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.168+0000 7f704a400700 1 --2- 172.21.15.27:0/1995702865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 0x7f704410b500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f70340098a0 tx=0x7f7034009bb0 comp rx=0 tx=0).stop 2024-05-10T10:59:01.593 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.168+0000 7f704a400700 1 -- 172.21.15.27:0/1995702865 shutdown_connections 2024-05-10T10:59:01.593 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.168+0000 7f704a400700 1 --2- 172.21.15.27:0/1995702865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 0x7f704410b500 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:01.593 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.168+0000 7f704a400700 1 -- 172.21.15.27:0/1995702865 >> 172.21.15.27:0/1995702865 conn(0x7f704407e450 msgr2=0x7f704407e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:01.593 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.168+0000 7f704a400700 1 -- 172.21.15.27:0/1995702865 shutdown_connections 2024-05-10T10:59:01.593 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.168+0000 7f704a400700 1 -- 172.21.15.27:0/1995702865 wait complete. 2024-05-10T10:59:01.593 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.169+0000 7f704a400700 1 Processor -- start 2024-05-10T10:59:01.593 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.169+0000 7f704a400700 1 -- start start 2024-05-10T10:59:01.593 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.170+0000 7f704a400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 0x7f704419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:01.594 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.170+0000 7f704a400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f704410f650 con 0x7f704410b0e0 2024-05-10T10:59:01.594 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.170+0000 7f7043400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 0x7f704419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:01.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.170+0000 7f7043400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 0x7f704419efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41346/0 (socket says 172.21.15.27:41346) 2024-05-10T10:59:01.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.170+0000 7f7043400700 1 -- 172.21.15.27:0/52911101 learned_addr learned my addr 172.21.15.27:0/52911101 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:01.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.170+0000 7f7043400700 1 -- 172.21.15.27:0/52911101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704419f520 con 0x7f704410b0e0 2024-05-10T10:59:01.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.171+0000 7f7043400700 1 --2- 172.21.15.27:0/52911101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 0x7f704419efe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7034011ac0 tx=0x7f7034011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:01.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.171+0000 7f703be00700 1 -- 172.21.15.27:0/52911101 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7034003b20 con 0x7f704410b0e0 2024-05-10T10:59:01.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.171+0000 7f703be00700 1 -- 172.21.15.27:0/52911101 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f703401a8c0 con 0x7f704410b0e0 2024-05-10T10:59:01.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.171+0000 7f703be00700 1 -- 172.21.15.27:0/52911101 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70340248d0 con 0x7f704410b0e0 2024-05-10T10:59:01.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.171+0000 7f704a400700 1 -- 172.21.15.27:0/52911101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f704419f7b0 con 0x7f704410b0e0 2024-05-10T10:59:01.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.171+0000 7f704a400700 1 -- 172.21.15.27:0/52911101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70441a02d0 con 0x7f704410b0e0 2024-05-10T10:59:01.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.172+0000 7f704a400700 1 -- 172.21.15.27:0/52911101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7044064b00 con 0x7f704410b0e0 2024-05-10T10:59:01.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.173+0000 7f703be00700 1 -- 172.21.15.27:0/52911101 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f7034022020 con 0x7f704410b0e0 2024-05-10T10:59:01.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.173+0000 7f703be00700 1 --2- 172.21.15.27:0/52911101 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f702803e580 0x7f7028040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:01.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.173+0000 7f703be00700 1 -- 172.21.15.27:0/52911101 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7034053800 con 0x7f704410b0e0 2024-05-10T10:59:01.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.174+0000 7f7042a00700 1 --2- 172.21.15.27:0/52911101 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f702803e580 0x7f7028040a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:01.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.174+0000 7f7042a00700 1 --2- 172.21.15.27:0/52911101 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f702803e580 0x7f7028040a40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7030006fd0 tx=0x7f7030006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:01.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.179+0000 7f703be00700 1 -- 172.21.15.27:0/52911101 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f703401ab20 con 0x7f704410b0e0 2024-05-10T10:59:01.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:58:54.419+0000 7f704a400700 1 -- 172.21.15.27:0/52911101 --> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}) -- 0x7f7044109360 con 0x7f702803e580 2024-05-10T10:59:01.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:00.853+0000 7f703be00700 1 -- 172.21.15.27:0/52911101 <== mgr.14120 v2:172.21.15.27:6800/481366655 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f7044109360 con 0x7f702803e580 2024-05-10T10:59:01.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:00.857+0000 7f704a400700 1 -- 172.21.15.27:0/52911101 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f702803e580 msgr2=0x7f7028040a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:01.600 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:00.857+0000 7f704a400700 1 --2- 172.21.15.27:0/52911101 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f702803e580 0x7f7028040a40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7030006fd0 tx=0x7f7030006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:01.600 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:00.857+0000 7f704a400700 1 -- 172.21.15.27:0/52911101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 msgr2=0x7f704419efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:01.600 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:00.857+0000 7f704a400700 1 --2- 172.21.15.27:0/52911101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 0x7f704419efe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7034011ac0 tx=0x7f7034011ba0 comp rx=0 tx=0).stop 2024-05-10T10:59:01.600 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:00.858+0000 7f704a400700 1 -- 172.21.15.27:0/52911101 shutdown_connections 2024-05-10T10:59:01.600 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:00.858+0000 7f704a400700 1 --2- 172.21.15.27:0/52911101 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f702803e580 0x7f7028040a40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:01.600 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:00.858+0000 7f704a400700 1 --2- 172.21.15.27:0/52911101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f704410b0e0 0x7f704419efe0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:01.600 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:00.858+0000 7f704a400700 1 -- 172.21.15.27:0/52911101 >> 172.21.15.27:0/52911101 conn(0x7f704407e450 msgr2=0x7f7044108c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:01.600 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:00.858+0000 7f704a400700 1 -- 172.21.15.27:0/52911101 shutdown_connections 2024-05-10T10:59:01.600 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:00.858+0000 7f704a400700 1 -- 172.21.15.27:0/52911101 wait complete. 2024-05-10T10:59:01.600 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mon service... 2024-05-10T10:59:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:01 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:59:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:01 smithi027 ceph-mon[30054]: Added host smithi027 2024-05-10T10:59:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:01 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T10:59:03.655 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-10T10:59:03.656 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.350+0000 7f02d4200700 1 Processor -- start 2024-05-10T10:59:03.656 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.350+0000 7f02d4200700 1 -- start start 2024-05-10T10:59:03.656 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.350+0000 7f02d4200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 0x7f02cc0832d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:03.657 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.350+0000 7f02d4200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc0838a0 con 0x7f02cc082eb0 2024-05-10T10:59:03.657 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.351+0000 7f02d1800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 0x7f02cc0832d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:03.657 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.351+0000 7f02d1800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 0x7f02cc0832d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41358/0 (socket says 172.21.15.27:41358) 2024-05-10T10:59:03.667 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.351+0000 7f02d1800700 1 -- 172.21.15.27:0/4068624093 learned_addr learned my addr 172.21.15.27:0/4068624093 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:03.667 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.352+0000 7f02d1800700 1 -- 172.21.15.27:0/4068624093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02cc110680 con 0x7f02cc082eb0 2024-05-10T10:59:03.667 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.352+0000 7f02d1800700 1 --2- 172.21.15.27:0/4068624093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 0x7f02cc0832d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f02c00098a0 tx=0x7f02c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1571e9177ecf0d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:03.667 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.353+0000 7f02cbe00700 1 -- 172.21.15.27:0/4068624093 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c0004160 con 0x7f02cc082eb0 2024-05-10T10:59:03.667 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.353+0000 7f02cbe00700 1 -- 172.21.15.27:0/4068624093 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f02c0004300 con 0x7f02cc082eb0 2024-05-10T10:59:03.667 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.353+0000 7f02cbe00700 1 -- 172.21.15.27:0/4068624093 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c0011af0 con 0x7f02cc082eb0 2024-05-10T10:59:03.667 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.353+0000 7f02d4200700 1 -- 172.21.15.27:0/4068624093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 msgr2=0x7f02cc0832d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.353+0000 7f02d4200700 1 --2- 172.21.15.27:0/4068624093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 0x7f02cc0832d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f02c00098a0 tx=0x7f02c0009bb0 comp rx=0 tx=0).stop 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.354+0000 7f02d4200700 1 -- 172.21.15.27:0/4068624093 shutdown_connections 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.354+0000 7f02d4200700 1 --2- 172.21.15.27:0/4068624093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 0x7f02cc0832d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.354+0000 7f02d4200700 1 -- 172.21.15.27:0/4068624093 >> 172.21.15.27:0/4068624093 conn(0x7f02cc07e450 msgr2=0x7f02cc080890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.354+0000 7f02d4200700 1 -- 172.21.15.27:0/4068624093 shutdown_connections 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.354+0000 7f02d4200700 1 -- 172.21.15.27:0/4068624093 wait complete. 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.355+0000 7f02d4200700 1 Processor -- start 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.355+0000 7f02d4200700 1 -- start start 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.356+0000 7f02d4200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 0x7f02cc1a3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.356+0000 7f02d4200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc113a30 con 0x7f02cc082eb0 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.356+0000 7f02d1800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 0x7f02cc1a3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.356+0000 7f02d1800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 0x7f02cc1a3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41370/0 (socket says 172.21.15.27:41370) 2024-05-10T10:59:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.356+0000 7f02d1800700 1 -- 172.21.15.27:0/3839739952 learned_addr learned my addr 172.21.15.27:0/3839739952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:03.669 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.356+0000 7f02d1800700 1 -- 172.21.15.27:0/3839739952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02cc1a3960 con 0x7f02cc082eb0 2024-05-10T10:59:03.669 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.357+0000 7f02d1800700 1 --2- 172.21.15.27:0/3839739952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 0x7f02cc1a3420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f02c0000c00 tx=0x7f02c0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:03.669 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.357+0000 7f02ca000700 1 -- 172.21.15.27:0/3839739952 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c0003b20 con 0x7f02cc082eb0 2024-05-10T10:59:03.669 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.357+0000 7f02ca000700 1 -- 172.21.15.27:0/3839739952 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f02c001a8c0 con 0x7f02cc082eb0 2024-05-10T10:59:03.669 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.357+0000 7f02ca000700 1 -- 172.21.15.27:0/3839739952 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c0024940 con 0x7f02cc082eb0 2024-05-10T10:59:03.669 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.357+0000 7f02d4200700 1 -- 172.21.15.27:0/3839739952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02cc1a3bf0 con 0x7f02cc082eb0 2024-05-10T10:59:03.669 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.357+0000 7f02d4200700 1 -- 172.21.15.27:0/3839739952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02cc1a4710 con 0x7f02cc082eb0 2024-05-10T10:59:03.669 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.358+0000 7f02d4200700 1 -- 172.21.15.27:0/3839739952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02cc064b00 con 0x7f02cc082eb0 2024-05-10T10:59:03.669 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.358+0000 7f02ca000700 1 -- 172.21.15.27:0/3839739952 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f02c0022020 con 0x7f02cc082eb0 2024-05-10T10:59:03.669 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.359+0000 7f02ca000700 1 --2- 172.21.15.27:0/3839739952 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f02b403e580 0x7f02b4040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:03.669 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.359+0000 7f02ca000700 1 -- 172.21.15.27:0/3839739952 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f02c0053800 con 0x7f02cc082eb0 2024-05-10T10:59:03.669 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.359+0000 7f02d0e00700 1 --2- 172.21.15.27:0/3839739952 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f02b403e580 0x7f02b4040a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.360+0000 7f02d0e00700 1 --2- 172.21.15.27:0/3839739952 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f02b403e580 0x7f02b4040a40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f02b8006fd0 tx=0x7f02b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.365+0000 7f02ca000700 1 -- 172.21.15.27:0/3839739952 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f02c00212c0 con 0x7f02cc082eb0 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.637+0000 7f02d4200700 1 -- 172.21.15.27:0/3839739952 --> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f02cc07f600 con 0x7f02b403e580 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.645+0000 7f02ca000700 1 -- 172.21.15.27:0/3839739952 <== mgr.14120 v2:172.21.15.27:6800/481366655 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f02cc07f600 con 0x7f02b403e580 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.648+0000 7f02d4200700 1 -- 172.21.15.27:0/3839739952 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f02b403e580 msgr2=0x7f02b4040a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.648+0000 7f02d4200700 1 --2- 172.21.15.27:0/3839739952 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f02b403e580 0x7f02b4040a40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f02b8006fd0 tx=0x7f02b8006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.648+0000 7f02d4200700 1 -- 172.21.15.27:0/3839739952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 msgr2=0x7f02cc1a3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.648+0000 7f02d4200700 1 --2- 172.21.15.27:0/3839739952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 0x7f02cc1a3420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f02c0000c00 tx=0x7f02c0011b70 comp rx=0 tx=0).stop 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.649+0000 7f02d4200700 1 -- 172.21.15.27:0/3839739952 shutdown_connections 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.649+0000 7f02d4200700 1 --2- 172.21.15.27:0/3839739952 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f02b403e580 0x7f02b4040a40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.649+0000 7f02d4200700 1 --2- 172.21.15.27:0/3839739952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02cc082eb0 0x7f02cc1a3420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.649+0000 7f02d4200700 1 -- 172.21.15.27:0/3839739952 >> 172.21.15.27:0/3839739952 conn(0x7f02cc07e450 msgr2=0x7f02cc07eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.649+0000 7f02d4200700 1 -- 172.21.15.27:0/3839739952 shutdown_connections 2024-05-10T10:59:03.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:02.649+0000 7f02d4200700 1 -- 172.21.15.27:0/3839739952 wait complete. 2024-05-10T10:59:03.671 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mgr service... 2024-05-10T10:59:03.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:03 smithi027 ceph-mon[30054]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T10:59:03.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:03 smithi027 ceph-mon[30054]: Saving service mon spec with placement count:5 2024-05-10T10:59:03.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:03 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:59:05.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:05 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:59:05.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:05 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:59:05.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:05 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:59:05.890 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-10T10:59:05.890 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.911+0000 7f637c200700 1 Processor -- start 2024-05-10T10:59:05.890 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.911+0000 7f637c200700 1 -- start start 2024-05-10T10:59:05.891 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.912+0000 7f637c200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 0x7f6374108170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:05.891 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.912+0000 7f637c200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6374108740 con 0x7f6374107d50 2024-05-10T10:59:05.891 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.912+0000 7f6379800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 0x7f6374108170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:05.891 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.912+0000 7f6379800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 0x7f6374108170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41834/0 (socket says 172.21.15.27:41834) 2024-05-10T10:59:05.891 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.912+0000 7f6379800700 1 -- 172.21.15.27:0/1071092578 learned_addr learned my addr 172.21.15.27:0/1071092578 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:05.891 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.913+0000 7f6379800700 1 -- 172.21.15.27:0/1071092578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6374108f50 con 0x7f6374107d50 2024-05-10T10:59:05.891 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.913+0000 7f6379800700 1 --2- 172.21.15.27:0/1071092578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 0x7f6374108170 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f63680098a0 tx=0x7f6368009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e58beecaf5bdee2 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:05.891 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.913+0000 7f6373e00700 1 -- 172.21.15.27:0/1071092578 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6368004160 con 0x7f6374107d50 2024-05-10T10:59:05.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.913+0000 7f6373e00700 1 -- 172.21.15.27:0/1071092578 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6368004300 con 0x7f6374107d50 2024-05-10T10:59:05.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.913+0000 7f6373e00700 1 -- 172.21.15.27:0/1071092578 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6368011a20 con 0x7f6374107d50 2024-05-10T10:59:05.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.914+0000 7f637c200700 1 -- 172.21.15.27:0/1071092578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 msgr2=0x7f6374108170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:05.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.914+0000 7f637c200700 1 --2- 172.21.15.27:0/1071092578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 0x7f6374108170 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f63680098a0 tx=0x7f6368009bb0 comp rx=0 tx=0).stop 2024-05-10T10:59:05.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.914+0000 7f637c200700 1 -- 172.21.15.27:0/1071092578 shutdown_connections 2024-05-10T10:59:05.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.914+0000 7f637c200700 1 --2- 172.21.15.27:0/1071092578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 0x7f6374108170 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:05.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.914+0000 7f637c200700 1 -- 172.21.15.27:0/1071092578 >> 172.21.15.27:0/1071092578 conn(0x7f63741032b0 msgr2=0x7f6374105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:05.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.914+0000 7f637c200700 1 -- 172.21.15.27:0/1071092578 shutdown_connections 2024-05-10T10:59:05.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.914+0000 7f637c200700 1 -- 172.21.15.27:0/1071092578 wait complete. 2024-05-10T10:59:05.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.916+0000 7f637c200700 1 Processor -- start 2024-05-10T10:59:05.892 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.917+0000 7f637c200700 1 -- start start 2024-05-10T10:59:05.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.917+0000 7f637c200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 0x7f637419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:05.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.917+0000 7f637c200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f637407bfb0 con 0x7f6374107d50 2024-05-10T10:59:05.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.917+0000 7f6379800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 0x7f637419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:05.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.917+0000 7f6379800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 0x7f637419f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41848/0 (socket says 172.21.15.27:41848) 2024-05-10T10:59:05.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.917+0000 7f6379800700 1 -- 172.21.15.27:0/501615239 learned_addr learned my addr 172.21.15.27:0/501615239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:05.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.918+0000 7f6379800700 1 -- 172.21.15.27:0/501615239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637419f5e0 con 0x7f6374107d50 2024-05-10T10:59:05.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.918+0000 7f6379800700 1 --2- 172.21.15.27:0/501615239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 0x7f637419f0a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6368000c00 tx=0x7f6368011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:05.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.918+0000 7f6372000700 1 -- 172.21.15.27:0/501615239 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6368003aa0 con 0x7f6374107d50 2024-05-10T10:59:05.893 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.918+0000 7f6372000700 1 -- 172.21.15.27:0/501615239 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f636801b8f0 con 0x7f6374107d50 2024-05-10T10:59:05.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.918+0000 7f637c200700 1 -- 172.21.15.27:0/501615239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637419f870 con 0x7f6374107d50 2024-05-10T10:59:05.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.919+0000 7f6372000700 1 -- 172.21.15.27:0/501615239 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6368023890 con 0x7f6374107d50 2024-05-10T10:59:05.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.919+0000 7f637c200700 1 -- 172.21.15.27:0/501615239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63741a0340 con 0x7f6374107d50 2024-05-10T10:59:05.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.920+0000 7f637c200700 1 -- 172.21.15.27:0/501615239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6374064b00 con 0x7f6374107d50 2024-05-10T10:59:05.894 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.920+0000 7f6372000700 1 -- 172.21.15.27:0/501615239 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f636802c3f0 con 0x7f6374107d50 2024-05-10T10:59:05.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.921+0000 7f6372000700 1 --2- 172.21.15.27:0/501615239 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f635c03e580 0x7f635c040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:05.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.921+0000 7f6372000700 1 -- 172.21.15.27:0/501615239 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6368053e00 con 0x7f6374107d50 2024-05-10T10:59:05.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.922+0000 7f6378e00700 1 --2- 172.21.15.27:0/501615239 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f635c03e580 0x7f635c040a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:05.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.922+0000 7f6378e00700 1 --2- 172.21.15.27:0/501615239 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f635c03e580 0x7f635c040a40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6360006fd0 tx=0x7f6360006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:05.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:04.927+0000 7f6372000700 1 -- 172.21.15.27:0/501615239 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f636801f030 con 0x7f6374107d50 2024-05-10T10:59:05.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:05.186+0000 7f637c200700 1 -- 172.21.15.27:0/501615239 --> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f63741a0860 con 0x7f635c03e580 2024-05-10T10:59:05.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:05.194+0000 7f6372000700 1 -- 172.21.15.27:0/501615239 <== mgr.14120 v2:172.21.15.27:6800/481366655 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f63741a0860 con 0x7f635c03e580 2024-05-10T10:59:05.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:05.197+0000 7f637c200700 1 -- 172.21.15.27:0/501615239 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f635c03e580 msgr2=0x7f635c040a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:05.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:05.197+0000 7f637c200700 1 --2- 172.21.15.27:0/501615239 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f635c03e580 0x7f635c040a40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6360006fd0 tx=0x7f6360006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:05.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:05.197+0000 7f637c200700 1 -- 172.21.15.27:0/501615239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 msgr2=0x7f637419f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:05.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:05.197+0000 7f637c200700 1 --2- 172.21.15.27:0/501615239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 0x7f637419f0a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6368000c00 tx=0x7f6368011aa0 comp rx=0 tx=0).stop 2024-05-10T10:59:05.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:05.197+0000 7f637c200700 1 -- 172.21.15.27:0/501615239 shutdown_connections 2024-05-10T10:59:05.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:05.197+0000 7f637c200700 1 --2- 172.21.15.27:0/501615239 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f635c03e580 0x7f635c040a40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:05.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:05.197+0000 7f637c200700 1 --2- 172.21.15.27:0/501615239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6374107d50 0x7f637419f0a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:05.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:05.197+0000 7f637c200700 1 -- 172.21.15.27:0/501615239 >> 172.21.15.27:0/501615239 conn(0x7f63741032b0 msgr2=0x7f63741955e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:05.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:05.198+0000 7f637c200700 1 -- 172.21.15.27:0/501615239 shutdown_connections 2024-05-10T10:59:05.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:05.198+0000 7f637c200700 1 -- 172.21.15.27:0/501615239 wait complete. 2024-05-10T10:59:06.793 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:06 smithi027 ceph-mon[30054]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T10:59:06.793 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:06 smithi027 ceph-mon[30054]: Saving service mgr spec with placement count:2 2024-05-10T10:59:06.793 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:06 smithi027 ceph-mon[30054]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T10:59:07.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.533+0000 7f195e800700 1 Processor -- start 2024-05-10T10:59:07.513 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.533+0000 7f195e800700 1 -- start start 2024-05-10T10:59:07.513 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.534+0000 7f195e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 0x7f195807d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:07.513 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.534+0000 7f195e800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f195807d8d0 con 0x7f195807eea0 2024-05-10T10:59:07.513 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.534+0000 7f1957400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 0x7f195807d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:07.514 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.535+0000 7f1957400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 0x7f195807d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41862/0 (socket says 172.21.15.27:41862) 2024-05-10T10:59:07.514 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.535+0000 7f1957400700 1 -- 172.21.15.27:0/3238830476 learned_addr learned my addr 172.21.15.27:0/3238830476 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:07.514 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.535+0000 7f1957400700 1 -- 172.21.15.27:0/3238830476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195807da50 con 0x7f195807eea0 2024-05-10T10:59:07.514 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.535+0000 7f1957400700 1 --2- 172.21.15.27:0/3238830476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 0x7f195807d300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f19480098a0 tx=0x7f1948009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb3480129ad0b08 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:07.514 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.536+0000 7f1956000700 1 -- 172.21.15.27:0/3238830476 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1948004160 con 0x7f195807eea0 2024-05-10T10:59:07.514 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.536+0000 7f1956000700 1 -- 172.21.15.27:0/3238830476 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1948004300 con 0x7f195807eea0 2024-05-10T10:59:07.514 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.536+0000 7f1956000700 1 -- 172.21.15.27:0/3238830476 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1948011a20 con 0x7f195807eea0 2024-05-10T10:59:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.537+0000 7f195e800700 1 -- 172.21.15.27:0/3238830476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 msgr2=0x7f195807d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.537+0000 7f195e800700 1 --2- 172.21.15.27:0/3238830476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 0x7f195807d300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f19480098a0 tx=0x7f1948009bb0 comp rx=0 tx=0).stop 2024-05-10T10:59:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.537+0000 7f195e800700 1 -- 172.21.15.27:0/3238830476 shutdown_connections 2024-05-10T10:59:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.537+0000 7f195e800700 1 --2- 172.21.15.27:0/3238830476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 0x7f195807d300 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.537+0000 7f195e800700 1 -- 172.21.15.27:0/3238830476 >> 172.21.15.27:0/3238830476 conn(0x7f1958104400 msgr2=0x7f1958106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.537+0000 7f195e800700 1 -- 172.21.15.27:0/3238830476 shutdown_connections 2024-05-10T10:59:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.537+0000 7f195e800700 1 -- 172.21.15.27:0/3238830476 wait complete. 2024-05-10T10:59:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.538+0000 7f195e800700 1 Processor -- start 2024-05-10T10:59:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.539+0000 7f195e800700 1 -- start start 2024-05-10T10:59:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.539+0000 7f195e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 0x7f195819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.539+0000 7f195e800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f195810f440 con 0x7f195807eea0 2024-05-10T10:59:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.539+0000 7f1957400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 0x7f195819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.539+0000 7f1957400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 0x7f195819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41876/0 (socket says 172.21.15.27:41876) 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.539+0000 7f1957400700 1 -- 172.21.15.27:0/2317836161 learned_addr learned my addr 172.21.15.27:0/2317836161 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.540+0000 7f1957400700 1 -- 172.21.15.27:0/2317836161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195819f550 con 0x7f195807eea0 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.540+0000 7f1957400700 1 --2- 172.21.15.27:0/2317836161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 0x7f195819f010 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1948000c00 tx=0x7f1948011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.540+0000 7f194fe00700 1 -- 172.21.15.27:0/2317836161 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1948003aa0 con 0x7f195807eea0 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.541+0000 7f194fe00700 1 -- 172.21.15.27:0/2317836161 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f194801b8f0 con 0x7f195807eea0 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.541+0000 7f195e800700 1 -- 172.21.15.27:0/2317836161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195819f7e0 con 0x7f195807eea0 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.541+0000 7f194fe00700 1 -- 172.21.15.27:0/2317836161 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1948023890 con 0x7f195807eea0 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.541+0000 7f195e800700 1 -- 172.21.15.27:0/2317836161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19581a0300 con 0x7f195807eea0 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.542+0000 7f194fe00700 1 -- 172.21.15.27:0/2317836161 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f194801b440 con 0x7f195807eea0 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.542+0000 7f195e800700 1 -- 172.21.15.27:0/2317836161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1958064b00 con 0x7f195807eea0 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.543+0000 7f194fe00700 1 --2- 172.21.15.27:0/2317836161 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f193c03e580 0x7f193c040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:07.516 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.543+0000 7f194fe00700 1 -- 172.21.15.27:0/2317836161 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1948053de0 con 0x7f195807eea0 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.544+0000 7f1956a00700 1 --2- 172.21.15.27:0/2317836161 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f193c03e580 0x7f193c040a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.545+0000 7f1956a00700 1 --2- 172.21.15.27:0/2317836161 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f193c03e580 0x7f193c040a40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1944006fd0 tx=0x7f1944006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.551+0000 7f194fe00700 1 -- 172.21.15.27:0/2317836161 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f194801f030 con 0x7f195807eea0 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.792+0000 7f195e800700 1 -- 172.21.15.27:0/2317836161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f19581a07c0 con 0x7f195807eea0 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.798+0000 7f194fe00700 1 -- 172.21.15.27:0/2317836161 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f1948016260 con 0x7f195807eea0 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.801+0000 7f195e800700 1 -- 172.21.15.27:0/2317836161 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f193c03e580 msgr2=0x7f193c040a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.801+0000 7f195e800700 1 --2- 172.21.15.27:0/2317836161 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f193c03e580 0x7f193c040a40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1944006fd0 tx=0x7f1944006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.801+0000 7f195e800700 1 -- 172.21.15.27:0/2317836161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 msgr2=0x7f195819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.801+0000 7f195e800700 1 --2- 172.21.15.27:0/2317836161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 0x7f195819f010 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1948000c00 tx=0x7f1948011aa0 comp rx=0 tx=0).stop 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.801+0000 7f195e800700 1 -- 172.21.15.27:0/2317836161 shutdown_connections 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.801+0000 7f195e800700 1 --2- 172.21.15.27:0/2317836161 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f193c03e580 0x7f193c040a40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.801+0000 7f195e800700 1 --2- 172.21.15.27:0/2317836161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f195807eea0 0x7f195819f010 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.801+0000 7f195e800700 1 -- 172.21.15.27:0/2317836161 >> 172.21.15.27:0/2317836161 conn(0x7f1958104400 msgr2=0x7f1958104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:07.517 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.802+0000 7f195e800700 1 -- 172.21.15.27:0/2317836161 shutdown_connections 2024-05-10T10:59:07.518 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:06.802+0000 7f195e800700 1 -- 172.21.15.27:0/2317836161 wait complete. 2024-05-10T10:59:08.110 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:07 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2317836161' entity='client.admin' 2024-05-10T10:59:09.065 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:08 smithi027 ceph-mon[30054]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T10:59:09.065 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:08 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:59:09.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.557+0000 7f4f70e00700 1 Processor -- start 2024-05-10T10:59:09.483 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.557+0000 7f4f70e00700 1 -- start start 2024-05-10T10:59:09.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.557+0000 7f4f70e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 0x7f4f6c07d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:09.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.557+0000 7f4f70e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6c07d8f0 con 0x7f4f6c07eec0 2024-05-10T10:59:09.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.558+0000 7f4f69e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 0x7f4f6c07d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:09.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.558+0000 7f4f69e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 0x7f4f6c07d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41878/0 (socket says 172.21.15.27:41878) 2024-05-10T10:59:09.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.558+0000 7f4f69e00700 1 -- 172.21.15.27:0/1404772477 learned_addr learned my addr 172.21.15.27:0/1404772477 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:09.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.558+0000 7f4f69e00700 1 -- 172.21.15.27:0/1404772477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6c07da70 con 0x7f4f6c07eec0 2024-05-10T10:59:09.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.558+0000 7f4f69e00700 1 --2- 172.21.15.27:0/1404772477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 0x7f4f6c07d320 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0098a0 tx=0x7f4f5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e94988e41061a3f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:09.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.559+0000 7f4f69400700 1 -- 172.21.15.27:0/1404772477 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f5c004160 con 0x7f4f6c07eec0 2024-05-10T10:59:09.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.559+0000 7f4f69400700 1 -- 172.21.15.27:0/1404772477 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f5c004300 con 0x7f4f6c07eec0 2024-05-10T10:59:09.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.559+0000 7f4f69400700 1 -- 172.21.15.27:0/1404772477 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f5c011a20 con 0x7f4f6c07eec0 2024-05-10T10:59:09.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.559+0000 7f4f70e00700 1 -- 172.21.15.27:0/1404772477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 msgr2=0x7f4f6c07d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:09.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.559+0000 7f4f70e00700 1 --2- 172.21.15.27:0/1404772477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 0x7f4f6c07d320 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0098a0 tx=0x7f4f5c009bb0 comp rx=0 tx=0).stop 2024-05-10T10:59:09.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.560+0000 7f4f70e00700 1 -- 172.21.15.27:0/1404772477 shutdown_connections 2024-05-10T10:59:09.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.560+0000 7f4f70e00700 1 --2- 172.21.15.27:0/1404772477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 0x7f4f6c07d320 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:09.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.560+0000 7f4f70e00700 1 -- 172.21.15.27:0/1404772477 >> 172.21.15.27:0/1404772477 conn(0x7f4f6c104400 msgr2=0x7f4f6c106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:09.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.560+0000 7f4f70e00700 1 -- 172.21.15.27:0/1404772477 shutdown_connections 2024-05-10T10:59:09.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.560+0000 7f4f70e00700 1 -- 172.21.15.27:0/1404772477 wait complete. 2024-05-10T10:59:09.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.561+0000 7f4f70e00700 1 Processor -- start 2024-05-10T10:59:09.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.561+0000 7f4f70e00700 1 -- start start 2024-05-10T10:59:09.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.561+0000 7f4f70e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 0x7f4f6c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.561+0000 7f4f70e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6c10f470 con 0x7f4f6c07eec0 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.562+0000 7f4f69e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 0x7f4f6c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.562+0000 7f4f69e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 0x7f4f6c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41882/0 (socket says 172.21.15.27:41882) 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.562+0000 7f4f69e00700 1 -- 172.21.15.27:0/2080986584 learned_addr learned my addr 172.21.15.27:0/2080986584 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.562+0000 7f4f69e00700 1 -- 172.21.15.27:0/2080986584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6c19f580 con 0x7f4f6c07eec0 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.562+0000 7f4f69e00700 1 --2- 172.21.15.27:0/2080986584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 0x7f4f6c19f040 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009520 tx=0x7f4f5c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.563+0000 7f4f63400700 1 -- 172.21.15.27:0/2080986584 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f5c0038b0 con 0x7f4f6c07eec0 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.563+0000 7f4f63400700 1 -- 172.21.15.27:0/2080986584 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f5c003a50 con 0x7f4f6c07eec0 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.563+0000 7f4f63400700 1 -- 172.21.15.27:0/2080986584 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f5c019690 con 0x7f4f6c07eec0 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.563+0000 7f4f70e00700 1 -- 172.21.15.27:0/2080986584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f6c19f810 con 0x7f4f6c07eec0 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.563+0000 7f4f70e00700 1 -- 172.21.15.27:0/2080986584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f6c1a0330 con 0x7f4f6c07eec0 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.564+0000 7f4f70e00700 1 -- 172.21.15.27:0/2080986584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f6c064b00 con 0x7f4f6c07eec0 2024-05-10T10:59:09.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.564+0000 7f4f63400700 1 -- 172.21.15.27:0/2080986584 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f4f5c01aa90 con 0x7f4f6c07eec0 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.565+0000 7f4f63400700 1 --2- 172.21.15.27:0/2080986584 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f4f5003e580 0x7f4f50040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.565+0000 7f4f63400700 1 -- 172.21.15.27:0/2080986584 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4f5c053bd0 con 0x7f4f6c07eec0 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.565+0000 7f4f61400700 1 --2- 172.21.15.27:0/2080986584 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f4f5003e580 0x7f4f50040a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.566+0000 7f4f61400700 1 --2- 172.21.15.27:0/2080986584 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f4f5003e580 0x7f4f50040a40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4f54006fd0 tx=0x7f4f54006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.571+0000 7f4f63400700 1 -- 172.21.15.27:0/2080986584 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4f5c018350 con 0x7f4f6c07eec0 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.811+0000 7f4f70e00700 1 -- 172.21.15.27:0/2080986584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f4f6c1a0820 con 0x7f4f6c07eec0 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.820+0000 7f4f63400700 1 -- 172.21.15.27:0/2080986584 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f4f5c019c80 con 0x7f4f6c07eec0 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.823+0000 7f4f70e00700 1 -- 172.21.15.27:0/2080986584 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f4f5003e580 msgr2=0x7f4f50040a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.823+0000 7f4f70e00700 1 --2- 172.21.15.27:0/2080986584 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f4f5003e580 0x7f4f50040a40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4f54006fd0 tx=0x7f4f54006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.824+0000 7f4f70e00700 1 -- 172.21.15.27:0/2080986584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 msgr2=0x7f4f6c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.824+0000 7f4f70e00700 1 --2- 172.21.15.27:0/2080986584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 0x7f4f6c19f040 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009520 tx=0x7f4f5c011aa0 comp rx=0 tx=0).stop 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.824+0000 7f4f70e00700 1 -- 172.21.15.27:0/2080986584 shutdown_connections 2024-05-10T10:59:09.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.824+0000 7f4f70e00700 1 --2- 172.21.15.27:0/2080986584 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f4f5003e580 0x7f4f50040a40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:09.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.824+0000 7f4f70e00700 1 --2- 172.21.15.27:0/2080986584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f6c07eec0 0x7f4f6c19f040 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:09.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.824+0000 7f4f70e00700 1 -- 172.21.15.27:0/2080986584 >> 172.21.15.27:0/2080986584 conn(0x7f4f6c104400 msgr2=0x7f4f6c104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:09.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.824+0000 7f4f70e00700 1 -- 172.21.15.27:0/2080986584 shutdown_connections 2024-05-10T10:59:09.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:08.824+0000 7f4f70e00700 1 -- 172.21.15.27:0/2080986584 wait complete. 2024-05-10T10:59:09.488 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling the dashboard module... 2024-05-10T10:59:10.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:09 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2080986584' entity='client.admin' 2024-05-10T10:59:11.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:10 smithi027 ceph-mon[30054]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T10:59:11.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:10 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2508168750' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-10T10:59:11.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:10 smithi027 ceph-mon[30054]: from='mgr.14120 172.21.15.27:0/1756079888' entity='mgr.x' 2024-05-10T10:59:11.119 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:10 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: ignoring --setuser ceph since I am not root 2024-05-10T10:59:11.119 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:10 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: ignoring --setgroup ceph since I am not root 2024-05-10T10:59:11.439 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:11.174+0000 7f47a3a4b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-10T10:59:11.440 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:11.438+0000 7f47a3a4b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-10T10:59:11.570 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.125+0000 7eff83e00700 1 Processor -- start 2024-05-10T10:59:11.570 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.125+0000 7eff83e00700 1 -- start start 2024-05-10T10:59:11.570 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.126+0000 7eff83e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 0x7eff84107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:11.570 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.126+0000 7eff83e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7eff841085c0 con 0x7eff84107bd0 2024-05-10T10:59:11.570 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.127+0000 7eff82a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 0x7eff84107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:11.570 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.127+0000 7eff82a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 0x7eff84107ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41888/0 (socket says 172.21.15.27:41888) 2024-05-10T10:59:11.570 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.127+0000 7eff82a00700 1 -- 172.21.15.27:0/4093001380 learned_addr learned my addr 172.21.15.27:0/4093001380 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:11.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.127+0000 7eff82a00700 1 -- 172.21.15.27:0/4093001380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff84108d80 con 0x7eff84107bd0 2024-05-10T10:59:11.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.128+0000 7eff82a00700 1 --2- 172.21.15.27:0/4093001380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 0x7eff84107ff0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7eff7400bb40 tx=0x7eff7400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af87e9774a0d7b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:11.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.128+0000 7eff81600700 1 -- 172.21.15.27:0/4093001380 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff7400dbe0 con 0x7eff84107bd0 2024-05-10T10:59:11.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.128+0000 7eff81600700 1 -- 172.21.15.27:0/4093001380 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff740044e0 con 0x7eff84107bd0 2024-05-10T10:59:11.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.128+0000 7eff81600700 1 -- 172.21.15.27:0/4093001380 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff74010470 con 0x7eff84107bd0 2024-05-10T10:59:11.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.129+0000 7eff83e00700 1 -- 172.21.15.27:0/4093001380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 msgr2=0x7eff84107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:11.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.129+0000 7eff83e00700 1 --2- 172.21.15.27:0/4093001380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 0x7eff84107ff0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7eff7400bb40 tx=0x7eff7400be50 comp rx=0 tx=0).stop 2024-05-10T10:59:11.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.129+0000 7eff83e00700 1 -- 172.21.15.27:0/4093001380 shutdown_connections 2024-05-10T10:59:11.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.129+0000 7eff83e00700 1 --2- 172.21.15.27:0/4093001380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 0x7eff84107ff0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:11.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.129+0000 7eff83e00700 1 -- 172.21.15.27:0/4093001380 >> 172.21.15.27:0/4093001380 conn(0x7eff841031f0 msgr2=0x7eff84105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:11.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.129+0000 7eff83e00700 1 -- 172.21.15.27:0/4093001380 shutdown_connections 2024-05-10T10:59:11.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.129+0000 7eff83e00700 1 -- 172.21.15.27:0/4093001380 wait complete. 2024-05-10T10:59:11.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.130+0000 7eff83e00700 1 Processor -- start 2024-05-10T10:59:11.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.130+0000 7eff83e00700 1 -- start start 2024-05-10T10:59:11.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.131+0000 7eff83e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 0x7eff8419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:11.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.131+0000 7eff83e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7eff8407c840 con 0x7eff84107bd0 2024-05-10T10:59:11.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.131+0000 7eff82a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 0x7eff8419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:11.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.131+0000 7eff82a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 0x7eff8419a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41898/0 (socket says 172.21.15.27:41898) 2024-05-10T10:59:11.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.131+0000 7eff82a00700 1 -- 172.21.15.27:0/2508168750 learned_addr learned my addr 172.21.15.27:0/2508168750 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:11.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.131+0000 7eff82a00700 1 -- 172.21.15.27:0/2508168750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8419ade0 con 0x7eff84107bd0 2024-05-10T10:59:11.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.132+0000 7eff82a00700 1 --2- 172.21.15.27:0/2508168750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 0x7eff8419a8a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7eff7400d650 tx=0x7eff74003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:11.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.132+0000 7eff7b400700 1 -- 172.21.15.27:0/2508168750 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff74027070 con 0x7eff84107bd0 2024-05-10T10:59:11.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.132+0000 7eff7b400700 1 -- 172.21.15.27:0/2508168750 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff74010c20 con 0x7eff84107bd0 2024-05-10T10:59:11.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.132+0000 7eff83e00700 1 -- 172.21.15.27:0/2508168750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff8419b070 con 0x7eff84107bd0 2024-05-10T10:59:11.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.132+0000 7eff7b400700 1 -- 172.21.15.27:0/2508168750 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff74022b10 con 0x7eff84107bd0 2024-05-10T10:59:11.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.132+0000 7eff83e00700 1 -- 172.21.15.27:0/2508168750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff8419dd60 con 0x7eff84107bd0 2024-05-10T10:59:11.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.134+0000 7eff83e00700 1 -- 172.21.15.27:0/2508168750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff8406a5d0 con 0x7eff84107bd0 2024-05-10T10:59:11.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.134+0000 7eff7b400700 1 -- 172.21.15.27:0/2508168750 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7eff7402a070 con 0x7eff84107bd0 2024-05-10T10:59:11.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.134+0000 7eff7b400700 1 --2- 172.21.15.27:0/2508168750 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7eff6803e4e0 0x7eff680409a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:11.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.135+0000 7eff7b400700 1 -- 172.21.15.27:0/2508168750 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7eff7401f070 con 0x7eff84107bd0 2024-05-10T10:59:11.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.135+0000 7eff82000700 1 --2- 172.21.15.27:0/2508168750 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7eff6803e4e0 0x7eff680409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:11.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.136+0000 7eff82000700 1 --2- 172.21.15.27:0/2508168750 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7eff6803e4e0 0x7eff680409a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7eff70006fd0 tx=0x7eff70006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:11.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.140+0000 7eff7b400700 1 -- 172.21.15.27:0/2508168750 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7eff74015070 con 0x7eff84107bd0 2024-05-10T10:59:11.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.443+0000 7eff83e00700 1 -- 172.21.15.27:0/2508168750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7eff84193e00 con 0x7eff84107bd0 2024-05-10T10:59:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.831+0000 7eff7b400700 1 -- 172.21.15.27:0/2508168750 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7eff84193e00 con 0x7eff84107bd0 2024-05-10T10:59:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.834+0000 7eff7b400700 1 -- 172.21.15.27:0/2508168750 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7eff74022410 con 0x7eff84107bd0 2024-05-10T10:59:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.836+0000 7eff83e00700 1 -- 172.21.15.27:0/2508168750 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7eff6803e4e0 msgr2=0x7eff680409a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.836+0000 7eff83e00700 1 --2- 172.21.15.27:0/2508168750 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7eff6803e4e0 0x7eff680409a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7eff70006fd0 tx=0x7eff70006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.836+0000 7eff83e00700 1 -- 172.21.15.27:0/2508168750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 msgr2=0x7eff8419a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.836+0000 7eff83e00700 1 --2- 172.21.15.27:0/2508168750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 0x7eff8419a8a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7eff7400d650 tx=0x7eff74003ff0 comp rx=0 tx=0).stop 2024-05-10T10:59:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.836+0000 7eff83e00700 1 -- 172.21.15.27:0/2508168750 shutdown_connections 2024-05-10T10:59:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.836+0000 7eff83e00700 1 --2- 172.21.15.27:0/2508168750 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7eff6803e4e0 0x7eff680409a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.836+0000 7eff83e00700 1 --2- 172.21.15.27:0/2508168750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff84107bd0 0x7eff8419a8a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.836+0000 7eff83e00700 1 -- 172.21.15.27:0/2508168750 >> 172.21.15.27:0/2508168750 conn(0x7eff841031f0 msgr2=0x7eff84190ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.836+0000 7eff83e00700 1 -- 172.21.15.27:0/2508168750 shutdown_connections 2024-05-10T10:59:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:10.837+0000 7eff83e00700 1 -- 172.21.15.27:0/2508168750 wait complete. 2024-05-10T10:59:11.720 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:11.577+0000 7f47a3a4b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-10T10:59:12.012 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:11 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2508168750' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-10T10:59:12.012 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:11 smithi027 ceph-mon[30054]: mgrmap e9: x(active, since 25s) 2024-05-10T10:59:12.369 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:12 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:12.011+0000 7f47a3a4b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-10T10:59:12.369 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:12 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:12.132+0000 7f47a3a4b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-10T10:59:13.096 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:12 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3274707685' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-10T10:59:13.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-10T10:59:13.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-10T10:59:13.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-10T10:59:13.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-10T10:59:13.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-10T10:59:13.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-10T10:59:13.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.174+0000 7f0041a00700 1 Processor -- start 2024-05-10T10:59:13.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.174+0000 7f0041a00700 1 -- start start 2024-05-10T10:59:13.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.175+0000 7f0041a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 0x7f003c109280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:13.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.175+0000 7f0041a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f003c1097c0 con 0x7f003c108e60 2024-05-10T10:59:13.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.175+0000 7f003aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 0x7f003c109280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:13.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.175+0000 7f003aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 0x7f003c109280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41920/0 (socket says 172.21.15.27:41920) 2024-05-10T10:59:13.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.175+0000 7f003aa00700 1 -- 172.21.15.27:0/3364889162 learned_addr learned my addr 172.21.15.27:0/3364889162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:13.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.176+0000 7f003aa00700 1 -- 172.21.15.27:0/3364889162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003c109fc0 con 0x7f003c108e60 2024-05-10T10:59:13.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.176+0000 7f003aa00700 1 --2- 172.21.15.27:0/3364889162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 0x7f003c109280 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f002c0098a0 tx=0x7f002c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a264320e373868 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.176+0000 7f0039600700 1 -- 172.21.15.27:0/3364889162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f002c004160 con 0x7f003c108e60 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.176+0000 7f0039600700 1 -- 172.21.15.27:0/3364889162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f002c004300 con 0x7f003c108e60 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.176+0000 7f0039600700 1 -- 172.21.15.27:0/3364889162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f002c011a20 con 0x7f003c108e60 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.177+0000 7f0041a00700 1 -- 172.21.15.27:0/3364889162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 msgr2=0x7f003c109280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.177+0000 7f0041a00700 1 --2- 172.21.15.27:0/3364889162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 0x7f003c109280 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f002c0098a0 tx=0x7f002c009bb0 comp rx=0 tx=0).stop 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.177+0000 7f0041a00700 1 -- 172.21.15.27:0/3364889162 shutdown_connections 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.177+0000 7f0041a00700 1 --2- 172.21.15.27:0/3364889162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 0x7f003c109280 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.177+0000 7f0041a00700 1 -- 172.21.15.27:0/3364889162 >> 172.21.15.27:0/3364889162 conn(0x7f003c104400 msgr2=0x7f003c106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.178+0000 7f0041a00700 1 -- 172.21.15.27:0/3364889162 shutdown_connections 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.178+0000 7f0041a00700 1 -- 172.21.15.27:0/3364889162 wait complete. 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.179+0000 7f0041a00700 1 Processor -- start 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.179+0000 7f0041a00700 1 -- start start 2024-05-10T10:59:13.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.180+0000 7f0041a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 0x7f003c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.180+0000 7f0041a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f003c10d400 con 0x7f003c108e60 2024-05-10T10:59:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.180+0000 7f003aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 0x7f003c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.180+0000 7f003aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 0x7f003c19f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41922/0 (socket says 172.21.15.27:41922) 2024-05-10T10:59:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.180+0000 7f003aa00700 1 -- 172.21.15.27:0/3274707685 learned_addr learned my addr 172.21.15.27:0/3274707685 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.181+0000 7f003aa00700 1 -- 172.21.15.27:0/3274707685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003c19f5a0 con 0x7f003c108e60 2024-05-10T10:59:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.181+0000 7f003aa00700 1 --2- 172.21.15.27:0/3274707685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 0x7f003c19f060 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f002c004660 tx=0x7f002c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.181+0000 7f0033400700 1 -- 172.21.15.27:0/3274707685 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f002c003aa0 con 0x7f003c108e60 2024-05-10T10:59:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.181+0000 7f0033400700 1 -- 172.21.15.27:0/3274707685 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f002c01b8f0 con 0x7f003c108e60 2024-05-10T10:59:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.181+0000 7f0033400700 1 -- 172.21.15.27:0/3274707685 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f002c023900 con 0x7f003c108e60 2024-05-10T10:59:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.181+0000 7f0041a00700 1 -- 172.21.15.27:0/3274707685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f003c19f830 con 0x7f003c108e60 2024-05-10T10:59:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.182+0000 7f0041a00700 1 -- 172.21.15.27:0/3274707685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f003c1a0300 con 0x7f003c108e60 2024-05-10T10:59:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.183+0000 7f0041a00700 1 -- 172.21.15.27:0/3274707685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f003c10d580 con 0x7f003c108e60 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.183+0000 7f0033400700 1 -- 172.21.15.27:0/3274707685 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f002c01baa0 con 0x7f003c108e60 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.184+0000 7f0033400700 1 --2- 172.21.15.27:0/3274707685 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f002003e580 0x7f0020040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.184+0000 7f0033400700 1 -- 172.21.15.27:0/3274707685 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f002c053e90 con 0x7f003c108e60 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.184+0000 7f003a000700 1 -- 172.21.15.27:0/3274707685 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f002003e580 msgr2=0x7f0020040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/481366655 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.184+0000 7f003a000700 1 --2- 172.21.15.27:0/3274707685 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f002003e580 0x7f0020040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.189+0000 7f0033400700 1 -- 172.21.15.27:0/3274707685 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f002c01f030 con 0x7f003c108e60 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.385+0000 7f003a000700 1 -- 172.21.15.27:0/3274707685 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f002003e580 msgr2=0x7f0020040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/481366655 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.385+0000 7f003a000700 1 --2- 172.21.15.27:0/3274707685 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f002003e580 0x7f0020040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.517+0000 7f0041a00700 1 -- 172.21.15.27:0/3274707685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f003c1a07b0 con 0x7f003c108e60 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.517+0000 7f0033400700 1 -- 172.21.15.27:0/3274707685 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f002c0202b0 con 0x7f003c108e60 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.521+0000 7f0041a00700 1 -- 172.21.15.27:0/3274707685 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f002003e580 msgr2=0x7f0020040a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.521+0000 7f0041a00700 1 --2- 172.21.15.27:0/3274707685 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f002003e580 0x7f0020040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.521+0000 7f0041a00700 1 -- 172.21.15.27:0/3274707685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 msgr2=0x7f003c19f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.521+0000 7f0041a00700 1 --2- 172.21.15.27:0/3274707685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 0x7f003c19f060 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f002c004660 tx=0x7f002c011aa0 comp rx=0 tx=0).stop 2024-05-10T10:59:13.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.521+0000 7f0041a00700 1 -- 172.21.15.27:0/3274707685 shutdown_connections 2024-05-10T10:59:13.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.521+0000 7f0041a00700 1 --2- 172.21.15.27:0/3274707685 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f002003e580 0x7f0020040a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:13.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.521+0000 7f0041a00700 1 --2- 172.21.15.27:0/3274707685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f003c108e60 0x7f003c19f060 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:13.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.521+0000 7f0041a00700 1 -- 172.21.15.27:0/3274707685 >> 172.21.15.27:0/3274707685 conn(0x7f003c104400 msgr2=0x7f003c104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:13.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.522+0000 7f0041a00700 1 -- 172.21.15.27:0/3274707685 shutdown_connections 2024-05-10T10:59:13.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:12.522+0000 7f0041a00700 1 -- 172.21.15.27:0/3274707685 wait complete. 2024-05-10T10:59:13.637 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-05-10T10:59:13.637 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 9... 2024-05-10T10:59:13.868 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:13.789+0000 7f47a3a4b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-10T10:59:14.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:14.031+0000 7f47a3a4b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-10T10:59:14.418 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:14.260+0000 7f47a3a4b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-10T10:59:14.743 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:14.662+0000 7f47a3a4b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-10T10:59:15.080 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:14.794+0000 7f47a3a4b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-10T10:59:15.355 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:15 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:15.080+0000 7f47a3a4b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-10T10:59:15.618 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:15 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:15.354+0000 7f47a3a4b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-10T10:59:16.501 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:16 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:16.199+0000 7f47a3a4b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-10T10:59:16.868 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:16 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:16.501+0000 7f47a3a4b200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-10T10:59:17.368 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:17 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:17.083+0000 7f47a3a4b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-10T10:59:17.369 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:17 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:17.218+0000 7f47a3a4b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-10T10:59:18.118 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:17 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:17.701+0000 7f47a3a4b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-10T10:59:18.618 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:18 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:18.176+0000 7f47a3a4b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-10T10:59:18.618 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:18 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:18.306+0000 7f47a3a4b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-10T10:59:18.965 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:18 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:18.644+0000 7f47a3a4b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-10T10:59:18.965 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:18 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:18.832+0000 7f47a3a4b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-10T10:59:19.368 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:18 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:18.965+0000 7f47a3a4b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-10T10:59:19.868 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:19 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:19.381+0000 7f47a3a4b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-10T10:59:19.868 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:19 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:19.513+0000 7f47a3a4b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-10T10:59:20.610 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:20 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:20.310+0000 7f47a3a4b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-10T10:59:20.868 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 10:59:20 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T10:59:20.610+0000 7f47a3a4b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-10T10:59:20.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:20 smithi027 ceph-mon[30054]: Active manager daemon x restarted 2024-05-10T10:59:20.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:20 smithi027 ceph-mon[30054]: Activating manager daemon x 2024-05-10T10:59:20.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:20 smithi027 ceph-mon[30054]: osdmap e3: 0 total, 0 up, 0 in 2024-05-10T10:59:20.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:20 smithi027 ceph-mon[30054]: mgrmap e10: x(active, starting, since 0.00957756s) 2024-05-10T10:59:20.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T10:59:20.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-10T10:59:20.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-10T10:59:20.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-10T10:59:20.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-10T10:59:20.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:20 smithi027 ceph-mon[30054]: Manager daemon x is now available 2024-05-10T10:59:22.013 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T10:59:22.013 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-10T10:59:22.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-10T10:59:22.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:21 smithi027 ceph-mon[30054]: mgrmap e11: x(active, since 1.00981s) 2024-05-10T10:59:22.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-10T10:59:22.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-10T10:59:22.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-10T10:59:22.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-10T10:59:22.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.743+0000 7f3d79600700 1 Processor -- start 2024-05-10T10:59:22.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.743+0000 7f3d79600700 1 -- start start 2024-05-10T10:59:22.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.744+0000 7f3d79600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 0x7f3d7407d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:22.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.744+0000 7f3d79600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7407d8d0 con 0x7f3d7407eea0 2024-05-10T10:59:22.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.744+0000 7f3d72a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 0x7f3d7407d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:22.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.744+0000 7f3d72a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 0x7f3d7407d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57142/0 (socket says 172.21.15.27:57142) 2024-05-10T10:59:22.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.744+0000 7f3d72a00700 1 -- 172.21.15.27:0/2867572445 learned_addr learned my addr 172.21.15.27:0/2867572445 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:22.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.745+0000 7f3d72a00700 1 -- 172.21.15.27:0/2867572445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7407da50 con 0x7f3d7407eea0 2024-05-10T10:59:22.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.745+0000 7f3d72a00700 1 --2- 172.21.15.27:0/2867572445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 0x7f3d7407d300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3d640098a0 tx=0x7f3d64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5fef89e950a6ca server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:22.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.745+0000 7f3d71600700 1 -- 172.21.15.27:0/2867572445 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d64004160 con 0x7f3d7407eea0 2024-05-10T10:59:22.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.745+0000 7f3d71600700 1 -- 172.21.15.27:0/2867572445 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d64004300 con 0x7f3d7407eea0 2024-05-10T10:59:22.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.745+0000 7f3d71600700 1 -- 172.21.15.27:0/2867572445 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d64011a20 con 0x7f3d7407eea0 2024-05-10T10:59:22.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.746+0000 7f3d79600700 1 -- 172.21.15.27:0/2867572445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 msgr2=0x7f3d7407d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:22.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.746+0000 7f3d79600700 1 --2- 172.21.15.27:0/2867572445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 0x7f3d7407d300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3d640098a0 tx=0x7f3d64009bb0 comp rx=0 tx=0).stop 2024-05-10T10:59:22.611 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.746+0000 7f3d79600700 1 -- 172.21.15.27:0/2867572445 shutdown_connections 2024-05-10T10:59:22.611 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.746+0000 7f3d79600700 1 --2- 172.21.15.27:0/2867572445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 0x7f3d7407d300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:22.611 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.746+0000 7f3d79600700 1 -- 172.21.15.27:0/2867572445 >> 172.21.15.27:0/2867572445 conn(0x7f3d74104400 msgr2=0x7f3d74106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:22.611 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.746+0000 7f3d79600700 1 -- 172.21.15.27:0/2867572445 shutdown_connections 2024-05-10T10:59:22.611 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.746+0000 7f3d79600700 1 -- 172.21.15.27:0/2867572445 wait complete. 2024-05-10T10:59:22.611 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.747+0000 7f3d79600700 1 Processor -- start 2024-05-10T10:59:22.612 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.748+0000 7f3d79600700 1 -- start start 2024-05-10T10:59:22.612 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.748+0000 7f3d79600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 0x7f3d7419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:22.612 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.748+0000 7f3d79600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7410f420 con 0x7f3d7407eea0 2024-05-10T10:59:22.613 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.748+0000 7f3d72a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 0x7f3d7419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:22.613 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.748+0000 7f3d72a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 0x7f3d7419ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57144/0 (socket says 172.21.15.27:57144) 2024-05-10T10:59:22.613 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.748+0000 7f3d72a00700 1 -- 172.21.15.27:0/448185670 learned_addr learned my addr 172.21.15.27:0/448185670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:22.613 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.749+0000 7f3d72a00700 1 -- 172.21.15.27:0/448185670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7419f450 con 0x7f3d7407eea0 2024-05-10T10:59:22.614 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.749+0000 7f3d72a00700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 0x7f3d7419ef10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3d64000c00 tx=0x7f3d64011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:22.615 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.749+0000 7f3d6b400700 1 -- 172.21.15.27:0/448185670 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d64003aa0 con 0x7f3d7407eea0 2024-05-10T10:59:22.615 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.749+0000 7f3d6b400700 1 -- 172.21.15.27:0/448185670 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d6401b8f0 con 0x7f3d7407eea0 2024-05-10T10:59:22.615 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.749+0000 7f3d6b400700 1 -- 172.21.15.27:0/448185670 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d64023890 con 0x7f3d7407eea0 2024-05-10T10:59:22.615 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.749+0000 7f3d79600700 1 -- 172.21.15.27:0/448185670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d7419f6e0 con 0x7f3d7407eea0 2024-05-10T10:59:22.615 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.750+0000 7f3d79600700 1 -- 172.21.15.27:0/448185670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d741a23d0 con 0x7f3d7407eea0 2024-05-10T10:59:22.615 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.751+0000 7f3d6b400700 1 -- 172.21.15.27:0/448185670 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f3d6401b440 con 0x7f3d7407eea0 2024-05-10T10:59:22.615 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.751+0000 7f3d6b400700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 0x7f3d58040aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:22.615 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.751+0000 7f3d6b400700 1 -- 172.21.15.27:0/448185670 --> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3d580411b0 con 0x7f3d5803e5e0 2024-05-10T10:59:22.616 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.751+0000 7f3d6b400700 1 -- 172.21.15.27:0/448185670 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3d64053e10 con 0x7f3d7407eea0 2024-05-10T10:59:22.616 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.751+0000 7f3d72000700 1 -- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 msgr2=0x7f3d58040aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/481366655 2024-05-10T10:59:22.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.752+0000 7f3d72000700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 0x7f3d58040aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-10T10:59:22.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.953+0000 7f3d72000700 1 -- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 msgr2=0x7f3d58040aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/481366655 2024-05-10T10:59:22.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:14.953+0000 7f3d72000700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 0x7f3d58040aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-10T10:59:22.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:15.354+0000 7f3d72000700 1 -- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 msgr2=0x7f3d58040aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/481366655 2024-05-10T10:59:22.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:15.354+0000 7f3d72000700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 0x7f3d58040aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-10T10:59:22.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:16.155+0000 7f3d72000700 1 -- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 msgr2=0x7f3d58040aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/481366655 2024-05-10T10:59:22.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:16.156+0000 7f3d72000700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 0x7f3d58040aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-10T10:59:22.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:17.758+0000 7f3d72000700 1 -- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 msgr2=0x7f3d58040aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/481366655 2024-05-10T10:59:22.618 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:17.758+0000 7f3d72000700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 0x7f3d58040aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-10T10:59:22.618 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:20.620+0000 7f3d6b400700 1 -- 172.21.15.27:0/448185670 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 10) ==== 50002+0+0 (secure 0 0 0) 0x7f3d6400b3e0 con 0x7f3d7407eea0 2024-05-10T10:59:22.618 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:20.620+0000 7f3d6b400700 1 -- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 msgr2=0x7f3d58040aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T10:59:22.618 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:20.620+0000 7f3d6b400700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 0x7f3d58040aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:22.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.621+0000 7f3d6b400700 1 -- 172.21.15.27:0/448185670 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 11) ==== 50129+0+0 (secure 0 0 0) 0x7f3d64055b80 con 0x7f3d7407eea0 2024-05-10T10:59:22.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.622+0000 7f3d6b400700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3d58041fc0 0x7f3d580443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:22.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.622+0000 7f3d6b400700 1 -- 172.21.15.27:0/448185670 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3d6400af30 con 0x7f3d58041fc0 2024-05-10T10:59:22.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.623+0000 7f3d72000700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3d58041fc0 0x7f3d580443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:22.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.625+0000 7f3d72000700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3d58041fc0 0x7f3d580443b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3d60003a10 tx=0x7f3d60009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:22.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.627+0000 7f3d6b400700 1 -- 172.21.15.27:0/448185670 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f3d6400af30 con 0x7f3d58041fc0 2024-05-10T10:59:22.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.640+0000 7f3d79600700 1 -- 172.21.15.27:0/448185670 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f3d7406a5d0 con 0x7f3d58041fc0 2024-05-10T10:59:22.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.641+0000 7f3d6b400700 1 -- 172.21.15.27:0/448185670 <== mgr.14152 v2:172.21.15.27:6800/1704535643 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f3d7406a5d0 con 0x7f3d58041fc0 2024-05-10T10:59:22.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.641+0000 7f3d79600700 1 -- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3d58041fc0 msgr2=0x7f3d580443b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:22.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.641+0000 7f3d79600700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3d58041fc0 0x7f3d580443b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3d60003a10 tx=0x7f3d60009270 comp rx=0 tx=0).stop 2024-05-10T10:59:22.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.641+0000 7f3d79600700 1 -- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 msgr2=0x7f3d7419ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:22.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.641+0000 7f3d79600700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 0x7f3d7419ef10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3d64000c00 tx=0x7f3d64011aa0 comp rx=0 tx=0).stop 2024-05-10T10:59:22.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.641+0000 7f3d79600700 1 -- 172.21.15.27:0/448185670 shutdown_connections 2024-05-10T10:59:22.621 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.641+0000 7f3d79600700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3d58041fc0 0x7f3d580443b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:22.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.641+0000 7f3d79600700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7407eea0 0x7f3d7419ef10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:22.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.641+0000 7f3d79600700 1 --2- 172.21.15.27:0/448185670 >> [v2:172.21.15.27:6800/481366655,v1:172.21.15.27:6801/481366655] conn(0x7f3d5803e5e0 0x7f3d58040aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:22.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.641+0000 7f3d79600700 1 -- 172.21.15.27:0/448185670 >> 172.21.15.27:0/448185670 conn(0x7f3d74104400 msgr2=0x7f3d74104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:22.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.642+0000 7f3d79600700 1 -- 172.21.15.27:0/448185670 shutdown_connections 2024-05-10T10:59:22.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:21.642+0000 7f3d79600700 1 -- 172.21.15.27:0/448185670 wait complete. 2024-05-10T10:59:22.622 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 9 is available 2024-05-10T10:59:22.622 INFO:teuthology.orchestra.run.smithi027.stdout:Generating a dashboard self-signed certificate... 2024-05-10T10:59:23.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:22 smithi027 ceph-mon[30054]: [10/May/2024:10:59:21] ENGINE Bus STARTING 2024-05-10T10:59:23.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:22 smithi027 ceph-mon[30054]: [10/May/2024:10:59:21] ENGINE Serving on https://172.21.15.27:7150 2024-05-10T10:59:23.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:22 smithi027 ceph-mon[30054]: [10/May/2024:10:59:21] ENGINE Serving on http://172.21.15.27:8765 2024-05-10T10:59:23.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:22 smithi027 ceph-mon[30054]: [10/May/2024:10:59:21] ENGINE Bus STARTED 2024-05-10T10:59:23.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:22 smithi027 ceph-mon[30054]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-10T10:59:23.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:22 smithi027 ceph-mon[30054]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-10T10:59:24.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:23 smithi027 ceph-mon[30054]: mgrmap e12: x(active, since 2s) 2024-05-10T10:59:24.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:23 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:24.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-10T10:59:24.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.804+0000 7fde90800700 1 Processor -- start 2024-05-10T10:59:24.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.804+0000 7fde90800700 1 -- start start 2024-05-10T10:59:24.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.804+0000 7fde90800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 0x7fde88109330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:24.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.804+0000 7fde90800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fde88109870 con 0x7fde88108f10 2024-05-10T10:59:24.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.805+0000 7fde8de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 0x7fde88109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:24.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.805+0000 7fde8de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 0x7fde88109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57230/0 (socket says 172.21.15.27:57230) 2024-05-10T10:59:24.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.805+0000 7fde8de00700 1 -- 172.21.15.27:0/4247347263 learned_addr learned my addr 172.21.15.27:0/4247347263 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:24.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.806+0000 7fde8de00700 1 -- 172.21.15.27:0/4247347263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8810a070 con 0x7fde88108f10 2024-05-10T10:59:24.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.806+0000 7fde8de00700 1 --2- 172.21.15.27:0/4247347263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 0x7fde88109330 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0098a0 tx=0x7fde7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd870bb9040c71c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:24.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.806+0000 7fde8ca00700 1 -- 172.21.15.27:0/4247347263 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde7c004160 con 0x7fde88108f10 2024-05-10T10:59:24.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.807+0000 7fde8ca00700 1 -- 172.21.15.27:0/4247347263 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde7c004300 con 0x7fde88108f10 2024-05-10T10:59:24.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.807+0000 7fde8ca00700 1 -- 172.21.15.27:0/4247347263 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde7c011a20 con 0x7fde88108f10 2024-05-10T10:59:24.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.807+0000 7fde90800700 1 -- 172.21.15.27:0/4247347263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 msgr2=0x7fde88109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:24.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.807+0000 7fde90800700 1 --2- 172.21.15.27:0/4247347263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 0x7fde88109330 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0098a0 tx=0x7fde7c009bb0 comp rx=0 tx=0).stop 2024-05-10T10:59:24.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.808+0000 7fde90800700 1 -- 172.21.15.27:0/4247347263 shutdown_connections 2024-05-10T10:59:24.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.808+0000 7fde90800700 1 --2- 172.21.15.27:0/4247347263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 0x7fde88109330 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:24.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.808+0000 7fde90800700 1 -- 172.21.15.27:0/4247347263 >> 172.21.15.27:0/4247347263 conn(0x7fde881044d0 msgr2=0x7fde881068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:24.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.808+0000 7fde90800700 1 -- 172.21.15.27:0/4247347263 shutdown_connections 2024-05-10T10:59:24.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.808+0000 7fde90800700 1 -- 172.21.15.27:0/4247347263 wait complete. 2024-05-10T10:59:24.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.809+0000 7fde90800700 1 Processor -- start 2024-05-10T10:59:24.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.810+0000 7fde90800700 1 -- start start 2024-05-10T10:59:24.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.810+0000 7fde90800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 0x7fde8807ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:24.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.810+0000 7fde90800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fde8810d4b0 con 0x7fde88108f10 2024-05-10T10:59:24.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.811+0000 7fde8de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 0x7fde8807ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:24.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.811+0000 7fde8de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 0x7fde8807ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57240/0 (socket says 172.21.15.27:57240) 2024-05-10T10:59:24.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.811+0000 7fde8de00700 1 -- 172.21.15.27:0/1860776471 learned_addr learned my addr 172.21.15.27:0/1860776471 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:24.913 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.811+0000 7fde8de00700 1 -- 172.21.15.27:0/1860776471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8807d280 con 0x7fde88108f10 2024-05-10T10:59:24.913 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.811+0000 7fde8de00700 1 --2- 172.21.15.27:0/1860776471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 0x7fde8807ed30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fde7c004660 tx=0x7fde7c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:24.913 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.812+0000 7fde82a00700 1 -- 172.21.15.27:0/1860776471 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde7c003aa0 con 0x7fde88108f10 2024-05-10T10:59:24.913 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.812+0000 7fde82a00700 1 -- 172.21.15.27:0/1860776471 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde7c01b8f0 con 0x7fde88108f10 2024-05-10T10:59:24.913 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.812+0000 7fde90800700 1 -- 172.21.15.27:0/1860776471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8807d510 con 0x7fde88108f10 2024-05-10T10:59:24.913 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.812+0000 7fde82a00700 1 -- 172.21.15.27:0/1860776471 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde7c023900 con 0x7fde88108f10 2024-05-10T10:59:24.913 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.812+0000 7fde90800700 1 -- 172.21.15.27:0/1860776471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde8807d9f0 con 0x7fde88108f10 2024-05-10T10:59:24.913 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.813+0000 7fde90800700 1 -- 172.21.15.27:0/1860776471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde88064b00 con 0x7fde88108f10 2024-05-10T10:59:24.914 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.814+0000 7fde82a00700 1 -- 172.21.15.27:0/1860776471 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7fde7c01baa0 con 0x7fde88108f10 2024-05-10T10:59:24.914 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.814+0000 7fde82a00700 1 --2- 172.21.15.27:0/1860776471 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fde7003e5e0 0x7fde70040aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:24.914 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.814+0000 7fde82a00700 1 -- 172.21.15.27:0/1860776471 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fde7c054010 con 0x7fde88108f10 2024-05-10T10:59:24.914 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.815+0000 7fde8d400700 1 --2- 172.21.15.27:0/1860776471 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fde7003e5e0 0x7fde70040aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:24.914 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.815+0000 7fde8d400700 1 --2- 172.21.15.27:0/1860776471 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fde7003e5e0 0x7fde70040aa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fde78006fd0 tx=0x7fde78006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:24.914 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:23.820+0000 7fde82a00700 1 -- 172.21.15.27:0/1860776471 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fde7c01f030 con 0x7fde88108f10 2024-05-10T10:59:24.914 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:24.067+0000 7fde90800700 1 -- 172.21.15.27:0/1860776471 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fde8807dfa0 con 0x7fde7003e5e0 2024-05-10T10:59:24.914 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:24.226+0000 7fde82a00700 1 -- 172.21.15.27:0/1860776471 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fde8807dfa0 con 0x7fde7003e5e0 2024-05-10T10:59:24.914 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:24.229+0000 7fde90800700 1 -- 172.21.15.27:0/1860776471 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fde7003e5e0 msgr2=0x7fde70040aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:24.915 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:24.229+0000 7fde90800700 1 --2- 172.21.15.27:0/1860776471 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fde7003e5e0 0x7fde70040aa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fde78006fd0 tx=0x7fde78006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:24.915 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:24.229+0000 7fde90800700 1 -- 172.21.15.27:0/1860776471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 msgr2=0x7fde8807ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:24.915 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:24.229+0000 7fde90800700 1 --2- 172.21.15.27:0/1860776471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 0x7fde8807ed30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fde7c004660 tx=0x7fde7c011aa0 comp rx=0 tx=0).stop 2024-05-10T10:59:24.915 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:24.229+0000 7fde90800700 1 -- 172.21.15.27:0/1860776471 shutdown_connections 2024-05-10T10:59:24.915 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:24.229+0000 7fde90800700 1 --2- 172.21.15.27:0/1860776471 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fde7003e5e0 0x7fde70040aa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:24.915 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:24.229+0000 7fde90800700 1 --2- 172.21.15.27:0/1860776471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde88108f10 0x7fde8807ed30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:24.915 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:24.229+0000 7fde90800700 1 -- 172.21.15.27:0/1860776471 >> 172.21.15.27:0/1860776471 conn(0x7fde881044d0 msgr2=0x7fde88104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:24.915 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:24.229+0000 7fde90800700 1 -- 172.21.15.27:0/1860776471 shutdown_connections 2024-05-10T10:59:24.915 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:24.229+0000 7fde90800700 1 -- 172.21.15.27:0/1860776471 wait complete. 2024-05-10T10:59:24.915 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial admin user... 2024-05-10T10:59:25.512 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:25 smithi027 ceph-mon[30054]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T10:59:25.512 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:25 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:25.513 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:25 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:26.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$kRQWZjUE1CpUm.JuswBGUOwJBokRvtgGRtkmEf2y2BQfKZr71EHNG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715338766, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-10T10:59:26.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.596+0000 7f0ee0e00700 1 Processor -- start 2024-05-10T10:59:26.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.596+0000 7f0ee0e00700 1 -- start start 2024-05-10T10:59:26.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.596+0000 7f0ee0e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 0x7f0edc109300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:26.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.596+0000 7f0ee0e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc109840 con 0x7f0edc108ee0 2024-05-10T10:59:26.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.597+0000 7f0ed9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 0x7f0edc109300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:26.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.597+0000 7f0ed9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 0x7f0edc109300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42560/0 (socket says 172.21.15.27:42560) 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.597+0000 7f0ed9e00700 1 -- 172.21.15.27:0/804808881 learned_addr learned my addr 172.21.15.27:0/804808881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.598+0000 7f0ed9e00700 1 -- 172.21.15.27:0/804808881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0edc109fa0 con 0x7f0edc108ee0 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.598+0000 7f0ed9e00700 1 --2- 172.21.15.27:0/804808881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 0x7f0edc109300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0098a0 tx=0x7f0ecc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881c79986a7686be server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.598+0000 7f0ed8a00700 1 -- 172.21.15.27:0/804808881 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ecc004160 con 0x7f0edc108ee0 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.598+0000 7f0ed8a00700 1 -- 172.21.15.27:0/804808881 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ecc004300 con 0x7f0edc108ee0 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.599+0000 7f0ed8a00700 1 -- 172.21.15.27:0/804808881 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ecc011a20 con 0x7f0edc108ee0 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.599+0000 7f0ee0e00700 1 -- 172.21.15.27:0/804808881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 msgr2=0x7f0edc109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.599+0000 7f0ee0e00700 1 --2- 172.21.15.27:0/804808881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 0x7f0edc109300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0098a0 tx=0x7f0ecc009bb0 comp rx=0 tx=0).stop 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.599+0000 7f0ee0e00700 1 -- 172.21.15.27:0/804808881 shutdown_connections 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.599+0000 7f0ee0e00700 1 --2- 172.21.15.27:0/804808881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 0x7f0edc109300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.600+0000 7f0ee0e00700 1 -- 172.21.15.27:0/804808881 >> 172.21.15.27:0/804808881 conn(0x7f0edc1044a0 msgr2=0x7f0edc1068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.600+0000 7f0ee0e00700 1 -- 172.21.15.27:0/804808881 shutdown_connections 2024-05-10T10:59:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.600+0000 7f0ee0e00700 1 -- 172.21.15.27:0/804808881 wait complete. 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.601+0000 7f0ee0e00700 1 Processor -- start 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.601+0000 7f0ee0e00700 1 -- start start 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.602+0000 7f0ee0e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 0x7f0edc07ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.602+0000 7f0ee0e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc10d3e0 con 0x7f0edc108ee0 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.602+0000 7f0ed9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 0x7f0edc07ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.602+0000 7f0ed9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 0x7f0edc07ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42568/0 (socket says 172.21.15.27:42568) 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.602+0000 7f0ed9e00700 1 -- 172.21.15.27:0/595660294 learned_addr learned my addr 172.21.15.27:0/595660294 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.602+0000 7f0ed9e00700 1 -- 172.21.15.27:0/595660294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0edc07d280 con 0x7f0edc108ee0 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.603+0000 7f0ed9e00700 1 --2- 172.21.15.27:0/595660294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 0x7f0edc07ed30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc004660 tx=0x7f0ecc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.603+0000 7f0ed2a00700 1 -- 172.21.15.27:0/595660294 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ecc003aa0 con 0x7f0edc108ee0 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.603+0000 7f0ed2a00700 1 -- 172.21.15.27:0/595660294 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ecc01b8f0 con 0x7f0edc108ee0 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.603+0000 7f0ed2a00700 1 -- 172.21.15.27:0/595660294 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ecc023900 con 0x7f0edc108ee0 2024-05-10T10:59:26.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.603+0000 7f0ee0e00700 1 -- 172.21.15.27:0/595660294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0edc07d510 con 0x7f0edc108ee0 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.603+0000 7f0ee0e00700 1 -- 172.21.15.27:0/595660294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0edc07d9f0 con 0x7f0edc108ee0 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.604+0000 7f0ee0e00700 1 -- 172.21.15.27:0/595660294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0edc10d560 con 0x7f0edc108ee0 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.605+0000 7f0ed2a00700 1 -- 172.21.15.27:0/595660294 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f0ecc01baa0 con 0x7f0edc108ee0 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.605+0000 7f0ed2a00700 1 --2- 172.21.15.27:0/595660294 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0ec003e5e0 0x7f0ec0040aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.606+0000 7f0ed2a00700 1 -- 172.21.15.27:0/595660294 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0ecc053f10 con 0x7f0edc108ee0 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.606+0000 7f0ed9400700 1 --2- 172.21.15.27:0/595660294 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0ec003e5e0 0x7f0ec0040aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.607+0000 7f0ed9400700 1 --2- 172.21.15.27:0/595660294 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0ec003e5e0 0x7f0ec0040aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8006fd0 tx=0x7f0ec8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.612+0000 7f0ed2a00700 1 -- 172.21.15.27:0/595660294 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f0ecc01f030 con 0x7f0edc108ee0 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:25.885+0000 7f0ee0e00700 1 -- 172.21.15.27:0/595660294 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f0edc07dfa0 con 0x7f0ec003e5e0 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:26.144+0000 7f0ed2a00700 1 -- 172.21.15.27:0/595660294 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f0edc07dfa0 con 0x7f0ec003e5e0 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:26.147+0000 7f0ee0e00700 1 -- 172.21.15.27:0/595660294 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0ec003e5e0 msgr2=0x7f0ec0040aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:26.147+0000 7f0ee0e00700 1 --2- 172.21.15.27:0/595660294 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0ec003e5e0 0x7f0ec0040aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8006fd0 tx=0x7f0ec8006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:26.147+0000 7f0ee0e00700 1 -- 172.21.15.27:0/595660294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 msgr2=0x7f0edc07ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:26.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:26.147+0000 7f0ee0e00700 1 --2- 172.21.15.27:0/595660294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 0x7f0edc07ed30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc004660 tx=0x7f0ecc011aa0 comp rx=0 tx=0).stop 2024-05-10T10:59:26.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:26.148+0000 7f0ee0e00700 1 -- 172.21.15.27:0/595660294 shutdown_connections 2024-05-10T10:59:26.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:26.148+0000 7f0ee0e00700 1 --2- 172.21.15.27:0/595660294 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0ec003e5e0 0x7f0ec0040aa0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:26.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:26.148+0000 7f0ee0e00700 1 --2- 172.21.15.27:0/595660294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0edc108ee0 0x7f0edc07ed30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:26.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:26.148+0000 7f0ee0e00700 1 -- 172.21.15.27:0/595660294 >> 172.21.15.27:0/595660294 conn(0x7f0edc1044a0 msgr2=0x7f0edc104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:26.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:26.148+0000 7f0ee0e00700 1 -- 172.21.15.27:0/595660294 shutdown_connections 2024-05-10T10:59:26.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:26.148+0000 7f0ee0e00700 1 -- 172.21.15.27:0/595660294 wait complete. 2024-05-10T10:59:26.818 INFO:teuthology.orchestra.run.smithi027.stdout:Fetching dashboard port number... 2024-05-10T10:59:27.065 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:26 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:27.065 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:26 smithi027 ceph-mon[30054]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T10:59:27.065 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:26 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:28.239 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:28 smithi027 ceph-mon[30054]: mgrmap e13: x(active, since 6s) 2024-05-10T10:59:28.240 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:28 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/883286159' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-10T10:59:28.934 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 8443 2024-05-10T10:59:28.934 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.511+0000 7fb37a800700 1 Processor -- start 2024-05-10T10:59:28.934 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.511+0000 7fb37a800700 1 -- start start 2024-05-10T10:59:28.934 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.512+0000 7fb37a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 0x7fb37410b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:28.934 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.512+0000 7fb37a800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb37410ba90 con 0x7fb37410b0a0 2024-05-10T10:59:28.934 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.512+0000 7fb373400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 0x7fb37410b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:28.934 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.513+0000 7fb373400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 0x7fb37410b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42580/0 (socket says 172.21.15.27:42580) 2024-05-10T10:59:28.934 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.513+0000 7fb373400700 1 -- 172.21.15.27:0/2984126453 learned_addr learned my addr 172.21.15.27:0/2984126453 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:28.934 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.513+0000 7fb373400700 1 -- 172.21.15.27:0/2984126453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37410c2a0 con 0x7fb37410b0a0 2024-05-10T10:59:28.934 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.513+0000 7fb373400700 1 --2- 172.21.15.27:0/2984126453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 0x7fb37410b4c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3640098a0 tx=0x7fb364009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13aceb11ac463d64 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.514+0000 7fb372000700 1 -- 172.21.15.27:0/2984126453 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb364004160 con 0x7fb37410b0a0 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.514+0000 7fb372000700 1 -- 172.21.15.27:0/2984126453 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb364004300 con 0x7fb37410b0a0 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.514+0000 7fb372000700 1 -- 172.21.15.27:0/2984126453 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb364011a20 con 0x7fb37410b0a0 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.514+0000 7fb37a800700 1 -- 172.21.15.27:0/2984126453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 msgr2=0x7fb37410b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.514+0000 7fb37a800700 1 --2- 172.21.15.27:0/2984126453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 0x7fb37410b4c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3640098a0 tx=0x7fb364009bb0 comp rx=0 tx=0).stop 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.514+0000 7fb37a800700 1 -- 172.21.15.27:0/2984126453 shutdown_connections 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.514+0000 7fb37a800700 1 --2- 172.21.15.27:0/2984126453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 0x7fb37410b4c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.514+0000 7fb37a800700 1 -- 172.21.15.27:0/2984126453 >> 172.21.15.27:0/2984126453 conn(0x7fb37407e4e0 msgr2=0x7fb37407e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.515+0000 7fb37a800700 1 -- 172.21.15.27:0/2984126453 shutdown_connections 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.515+0000 7fb37a800700 1 -- 172.21.15.27:0/2984126453 wait complete. 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.516+0000 7fb37a800700 1 Processor -- start 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.516+0000 7fb37a800700 1 -- start start 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.516+0000 7fb37a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 0x7fb37419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:28.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.516+0000 7fb37a800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb37410f650 con 0x7fb37410b0a0 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.516+0000 7fb373400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 0x7fb37419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.516+0000 7fb373400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 0x7fb37419f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42592/0 (socket says 172.21.15.27:42592) 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.516+0000 7fb373400700 1 -- 172.21.15.27:0/883286159 learned_addr learned my addr 172.21.15.27:0/883286159 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.517+0000 7fb373400700 1 -- 172.21.15.27:0/883286159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37419f550 con 0x7fb37410b0a0 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.517+0000 7fb373400700 1 --2- 172.21.15.27:0/883286159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 0x7fb37419f010 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb364000c00 tx=0x7fb364011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.517+0000 7fb36be00700 1 -- 172.21.15.27:0/883286159 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb364003aa0 con 0x7fb37410b0a0 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.517+0000 7fb36be00700 1 -- 172.21.15.27:0/883286159 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb36401b8f0 con 0x7fb37410b0a0 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.517+0000 7fb37a800700 1 -- 172.21.15.27:0/883286159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb37419f7e0 con 0x7fb37410b0a0 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.518+0000 7fb36be00700 1 -- 172.21.15.27:0/883286159 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb364023890 con 0x7fb37410b0a0 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.518+0000 7fb37a800700 1 -- 172.21.15.27:0/883286159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3741a0300 con 0x7fb37410b0a0 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.519+0000 7fb36be00700 1 -- 172.21.15.27:0/883286159 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fb36401b440 con 0x7fb37410b0a0 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.519+0000 7fb37a800700 1 -- 172.21.15.27:0/883286159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb374064b00 con 0x7fb37410b0a0 2024-05-10T10:59:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.519+0000 7fb36be00700 1 --2- 172.21.15.27:0/883286159 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb35803e680 0x7fb358040b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.520+0000 7fb36be00700 1 -- 172.21.15.27:0/883286159 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb364053fd0 con 0x7fb37410b0a0 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.520+0000 7fb372a00700 1 --2- 172.21.15.27:0/883286159 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb35803e680 0x7fb358040b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.521+0000 7fb372a00700 1 --2- 172.21.15.27:0/883286159 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb35803e680 0x7fb358040b40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb360006fd0 tx=0x7fb360006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.525+0000 7fb36be00700 1 -- 172.21.15.27:0/883286159 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb36401f030 con 0x7fb37410b0a0 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.766+0000 7fb37a800700 1 -- 172.21.15.27:0/883286159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fb3741a07b0 con 0x7fb37410b0a0 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.767+0000 7fb36be00700 1 -- 172.21.15.27:0/883286159 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fb364017e90 con 0x7fb37410b0a0 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.770+0000 7fb37a800700 1 -- 172.21.15.27:0/883286159 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb35803e680 msgr2=0x7fb358040b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.770+0000 7fb37a800700 1 --2- 172.21.15.27:0/883286159 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb35803e680 0x7fb358040b40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb360006fd0 tx=0x7fb360006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.770+0000 7fb37a800700 1 -- 172.21.15.27:0/883286159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 msgr2=0x7fb37419f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.770+0000 7fb37a800700 1 --2- 172.21.15.27:0/883286159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 0x7fb37419f010 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb364000c00 tx=0x7fb364011aa0 comp rx=0 tx=0).stop 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.771+0000 7fb37a800700 1 -- 172.21.15.27:0/883286159 shutdown_connections 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.771+0000 7fb37a800700 1 --2- 172.21.15.27:0/883286159 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb35803e680 0x7fb358040b40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.771+0000 7fb37a800700 1 --2- 172.21.15.27:0/883286159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb37410b0a0 0x7fb37419f010 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:28.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.771+0000 7fb37a800700 1 -- 172.21.15.27:0/883286159 >> 172.21.15.27:0/883286159 conn(0x7fb37407e4e0 msgr2=0x7fb374108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:28.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.771+0000 7fb37a800700 1 -- 172.21.15.27:0/883286159 shutdown_connections 2024-05-10T10:59:28.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:27.771+0000 7fb37a800700 1 -- 172.21.15.27:0/883286159 wait complete. 2024-05-10T10:59:28.961 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-10T10:59:28.961 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-05-10T10:59:28.984 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-10T10:59:28.984 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-05-10T10:59:28.984 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-05-10T10:59:28.984 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-10T10:59:28.989 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph Dashboard is now available at: 2024-05-10T10:59:28.989 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:59:28.989 INFO:teuthology.orchestra.run.smithi027.stdout: URL: https://smithi027.front.sepia.ceph.com:8443/ 2024-05-10T10:59:28.989 INFO:teuthology.orchestra.run.smithi027.stdout: User: admin 2024-05-10T10:59:28.989 INFO:teuthology.orchestra.run.smithi027.stdout: Password: 9bbgam3sky 2024-05-10T10:59:28.989 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:59:28.990 INFO:teuthology.orchestra.run.smithi027.stdout:Saving cluster configuration to /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config directory 2024-05-10T10:59:28.991 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling autotune for osd_memory_target 2024-05-10T10:59:30.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.604+0000 7fb914c00700 1 Processor -- start 2024-05-10T10:59:30.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.604+0000 7fb914c00700 1 -- start start 2024-05-10T10:59:30.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.605+0000 7fb914c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 0x7fb91010b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:30.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.605+0000 7fb914c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb91010b830 con 0x7fb91010aed0 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.606+0000 7fb90f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 0x7fb91010b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.606+0000 7fb90f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 0x7fb91010b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42608/0 (socket says 172.21.15.27:42608) 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.606+0000 7fb90f400700 1 -- 172.21.15.27:0/3800939101 learned_addr learned my addr 172.21.15.27:0/3800939101 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.606+0000 7fb90f400700 1 -- 172.21.15.27:0/3800939101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91010bfe0 con 0x7fb91010aed0 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.606+0000 7fb90f400700 1 --2- 172.21.15.27:0/3800939101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 0x7fb91010b2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb9000098a0 tx=0x7fb900009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75c622ea6f4e5cd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.607+0000 7fb90e000700 1 -- 172.21.15.27:0/3800939101 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb900004160 con 0x7fb91010aed0 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.607+0000 7fb90e000700 1 -- 172.21.15.27:0/3800939101 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb900004300 con 0x7fb91010aed0 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.607+0000 7fb90e000700 1 -- 172.21.15.27:0/3800939101 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb900011a20 con 0x7fb91010aed0 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.607+0000 7fb914c00700 1 -- 172.21.15.27:0/3800939101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 msgr2=0x7fb91010b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.607+0000 7fb914c00700 1 --2- 172.21.15.27:0/3800939101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 0x7fb91010b2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb9000098a0 tx=0x7fb900009bb0 comp rx=0 tx=0).stop 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.608+0000 7fb914c00700 1 -- 172.21.15.27:0/3800939101 shutdown_connections 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.608+0000 7fb914c00700 1 --2- 172.21.15.27:0/3800939101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 0x7fb91010b2f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.608+0000 7fb914c00700 1 -- 172.21.15.27:0/3800939101 >> 172.21.15.27:0/3800939101 conn(0x7fb91007e2e0 msgr2=0x7fb91007e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.608+0000 7fb914c00700 1 -- 172.21.15.27:0/3800939101 shutdown_connections 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.608+0000 7fb914c00700 1 -- 172.21.15.27:0/3800939101 wait complete. 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.609+0000 7fb914c00700 1 Processor -- start 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.609+0000 7fb914c00700 1 -- start start 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.609+0000 7fb914c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 0x7fb91019eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.609+0000 7fb914c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb91010f420 con 0x7fb91010aed0 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.610+0000 7fb90f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 0x7fb91019eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.610+0000 7fb90f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 0x7fb91019eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42610/0 (socket says 172.21.15.27:42610) 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.610+0000 7fb90f400700 1 -- 172.21.15.27:0/3791527749 learned_addr learned my addr 172.21.15.27:0/3791527749 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.610+0000 7fb90f400700 1 -- 172.21.15.27:0/3791527749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91019f1e0 con 0x7fb91010aed0 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.610+0000 7fb90f400700 1 --2- 172.21.15.27:0/3791527749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 0x7fb91019eca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb900000c00 tx=0x7fb900011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.611+0000 7fb907e00700 1 -- 172.21.15.27:0/3791527749 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb900003aa0 con 0x7fb91010aed0 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.611+0000 7fb907e00700 1 -- 172.21.15.27:0/3791527749 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb90001b8f0 con 0x7fb91010aed0 2024-05-10T10:59:30.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.611+0000 7fb907e00700 1 -- 172.21.15.27:0/3791527749 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb900023870 con 0x7fb91010aed0 2024-05-10T10:59:30.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.611+0000 7fb914c00700 1 -- 172.21.15.27:0/3791527749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91019f470 con 0x7fb91010aed0 2024-05-10T10:59:30.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.611+0000 7fb914c00700 1 -- 172.21.15.27:0/3791527749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9101a2160 con 0x7fb91010aed0 2024-05-10T10:59:30.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.612+0000 7fb914c00700 1 -- 172.21.15.27:0/3791527749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb91010f5a0 con 0x7fb91010aed0 2024-05-10T10:59:30.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.612+0000 7fb907e00700 1 -- 172.21.15.27:0/3791527749 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fb90001b440 con 0x7fb91010aed0 2024-05-10T10:59:30.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.613+0000 7fb907e00700 1 --2- 172.21.15.27:0/3791527749 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb8f403e680 0x7fb8f4040b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:30.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.613+0000 7fb907e00700 1 -- 172.21.15.27:0/3791527749 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb900054040 con 0x7fb91010aed0 2024-05-10T10:59:30.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.613+0000 7fb90ea00700 1 --2- 172.21.15.27:0/3791527749 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb8f403e680 0x7fb8f4040b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:30.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.614+0000 7fb90ea00700 1 --2- 172.21.15.27:0/3791527749 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb8f403e680 0x7fb8f4040b40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc006fd0 tx=0x7fb8fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:30.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.619+0000 7fb907e00700 1 -- 172.21.15.27:0/3791527749 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb90001f030 con 0x7fb91010aed0 2024-05-10T10:59:30.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.852+0000 7fb914c00700 1 -- 172.21.15.27:0/3791527749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fb9101a27d0 con 0x7fb91010aed0 2024-05-10T10:59:30.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.854+0000 7fb907e00700 1 -- 172.21.15.27:0/3791527749 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fb900018590 con 0x7fb91010aed0 2024-05-10T10:59:30.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.857+0000 7fb914c00700 1 -- 172.21.15.27:0/3791527749 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb8f403e680 msgr2=0x7fb8f4040b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:30.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.857+0000 7fb914c00700 1 --2- 172.21.15.27:0/3791527749 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb8f403e680 0x7fb8f4040b40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc006fd0 tx=0x7fb8fc006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:30.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.857+0000 7fb914c00700 1 -- 172.21.15.27:0/3791527749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 msgr2=0x7fb91019eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:30.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.857+0000 7fb914c00700 1 --2- 172.21.15.27:0/3791527749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 0x7fb91019eca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb900000c00 tx=0x7fb900011aa0 comp rx=0 tx=0).stop 2024-05-10T10:59:30.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.857+0000 7fb914c00700 1 -- 172.21.15.27:0/3791527749 shutdown_connections 2024-05-10T10:59:30.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.857+0000 7fb914c00700 1 --2- 172.21.15.27:0/3791527749 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb8f403e680 0x7fb8f4040b40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:30.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.857+0000 7fb914c00700 1 --2- 172.21.15.27:0/3791527749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb91010aed0 0x7fb91019eca0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:30.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.857+0000 7fb914c00700 1 -- 172.21.15.27:0/3791527749 >> 172.21.15.27:0/3791527749 conn(0x7fb91007e2e0 msgr2=0x7fb9101083b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:30.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.858+0000 7fb914c00700 1 -- 172.21.15.27:0/3791527749 shutdown_connections 2024-05-10T10:59:30.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:29.858+0000 7fb914c00700 1 -- 172.21.15.27:0/3791527749 wait complete. 2024-05-10T10:59:32.284 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.185+0000 7f9133e00700 1 Processor -- start 2024-05-10T10:59:32.284 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.185+0000 7f9133e00700 1 -- start start 2024-05-10T10:59:32.284 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.186+0000 7f9133e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 0x7f912c10b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.186+0000 7f9133e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f912c10bb80 con 0x7f912c10b190 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.186+0000 7f9131400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 0x7f912c10b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.186+0000 7f9131400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 0x7f912c10b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42624/0 (socket says 172.21.15.27:42624) 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.186+0000 7f9131400700 1 -- 172.21.15.27:0/1790574414 learned_addr learned my addr 172.21.15.27:0/1790574414 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.187+0000 7f9131400700 1 -- 172.21.15.27:0/1790574414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f912c10c240 con 0x7f912c10b190 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.187+0000 7f9131400700 1 --2- 172.21.15.27:0/1790574414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 0x7f912c10b5b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9120009e80 tx=0x7f912000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac82a58fb87f081f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.187+0000 7f912be00700 1 -- 172.21.15.27:0/1790574414 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9120017070 con 0x7f912c10b190 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.188+0000 7f912be00700 1 -- 172.21.15.27:0/1790574414 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f912000b8c0 con 0x7f912c10b190 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.188+0000 7f912be00700 1 -- 172.21.15.27:0/1790574414 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9120004490 con 0x7f912c10b190 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.188+0000 7f9133e00700 1 -- 172.21.15.27:0/1790574414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 msgr2=0x7f912c10b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.188+0000 7f9133e00700 1 --2- 172.21.15.27:0/1790574414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 0x7f912c10b5b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9120009e80 tx=0x7f912000b330 comp rx=0 tx=0).stop 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.188+0000 7f9133e00700 1 -- 172.21.15.27:0/1790574414 shutdown_connections 2024-05-10T10:59:32.285 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.188+0000 7f9133e00700 1 --2- 172.21.15.27:0/1790574414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 0x7f912c10b5b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:32.286 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.188+0000 7f9133e00700 1 -- 172.21.15.27:0/1790574414 >> 172.21.15.27:0/1790574414 conn(0x7f912c07e6a0 msgr2=0x7f912c07ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:32.291 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.189+0000 7f9133e00700 1 -- 172.21.15.27:0/1790574414 shutdown_connections 2024-05-10T10:59:32.291 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.189+0000 7f9133e00700 1 -- 172.21.15.27:0/1790574414 wait complete. 2024-05-10T10:59:32.291 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.190+0000 7f9133e00700 1 Processor -- start 2024-05-10T10:59:32.291 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.190+0000 7f9133e00700 1 -- start start 2024-05-10T10:59:32.291 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.190+0000 7f9133e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 0x7f912c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:32.291 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.191+0000 7f9133e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f912c10f470 con 0x7f912c10b190 2024-05-10T10:59:32.291 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.191+0000 7f9131400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 0x7f912c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:32.291 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.191+0000 7f9131400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 0x7f912c19ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42632/0 (socket says 172.21.15.27:42632) 2024-05-10T10:59:32.291 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.191+0000 7f9131400700 1 -- 172.21.15.27:0/3207142654 learned_addr learned my addr 172.21.15.27:0/3207142654 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:32.291 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.191+0000 7f9131400700 1 -- 172.21.15.27:0/3207142654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f912c19f4c0 con 0x7f912c10b190 2024-05-10T10:59:32.292 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.192+0000 7f9131400700 1 --2- 172.21.15.27:0/3207142654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 0x7f912c19ef80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f912000b7b0 tx=0x7f912000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:32.292 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.192+0000 7f912a000700 1 -- 172.21.15.27:0/3207142654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9120020020 con 0x7f912c10b190 2024-05-10T10:59:32.292 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.192+0000 7f912a000700 1 -- 172.21.15.27:0/3207142654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9120003b50 con 0x7f912c10b190 2024-05-10T10:59:32.292 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.192+0000 7f912a000700 1 -- 172.21.15.27:0/3207142654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9120017040 con 0x7f912c10b190 2024-05-10T10:59:32.292 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.192+0000 7f9133e00700 1 -- 172.21.15.27:0/3207142654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f912c19f750 con 0x7f912c10b190 2024-05-10T10:59:32.292 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.192+0000 7f9133e00700 1 -- 172.21.15.27:0/3207142654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f912c1a0270 con 0x7f912c10b190 2024-05-10T10:59:32.292 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.193+0000 7f9133e00700 1 -- 172.21.15.27:0/3207142654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f912c064b00 con 0x7f912c10b190 2024-05-10T10:59:32.292 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.194+0000 7f912a000700 1 -- 172.21.15.27:0/3207142654 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f9120023020 con 0x7f912c10b190 2024-05-10T10:59:32.292 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.194+0000 7f912a000700 1 --2- 172.21.15.27:0/3207142654 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f911403e670 0x7f9114040b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:32.292 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.194+0000 7f912a000700 1 -- 172.21.15.27:0/3207142654 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f912000f050 con 0x7f912c10b190 2024-05-10T10:59:32.292 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.195+0000 7f9130a00700 1 --2- 172.21.15.27:0/3207142654 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f911403e670 0x7f9114040b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:32.292 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.195+0000 7f9130a00700 1 --2- 172.21.15.27:0/3207142654 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f911403e670 0x7f9114040b30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9118006fd0 tx=0x7f9118006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.199+0000 7f912a000700 1 -- 172.21.15.27:0/3207142654 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f912002a020 con 0x7f912c10b190 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.565+0000 7f9133e00700 1 -- 172.21.15.27:0/3207142654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f912c1a0750 con 0x7f912c10b190 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.571+0000 7f912a000700 1 -- 172.21.15.27:0/3207142654 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f912001bb80 con 0x7f912c10b190 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.575+0000 7f9133e00700 1 -- 172.21.15.27:0/3207142654 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f911403e670 msgr2=0x7f9114040b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.575+0000 7f9133e00700 1 --2- 172.21.15.27:0/3207142654 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f911403e670 0x7f9114040b30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9118006fd0 tx=0x7f9118006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.575+0000 7f9133e00700 1 -- 172.21.15.27:0/3207142654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 msgr2=0x7f912c19ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.575+0000 7f9133e00700 1 --2- 172.21.15.27:0/3207142654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 0x7f912c19ef80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f912000b7b0 tx=0x7f912000bd30 comp rx=0 tx=0).stop 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.575+0000 7f9133e00700 1 -- 172.21.15.27:0/3207142654 shutdown_connections 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.575+0000 7f9133e00700 1 --2- 172.21.15.27:0/3207142654 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f911403e670 0x7f9114040b30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.575+0000 7f9133e00700 1 --2- 172.21.15.27:0/3207142654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912c10b190 0x7f912c19ef80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.575+0000 7f9133e00700 1 -- 172.21.15.27:0/3207142654 >> 172.21.15.27:0/3207142654 conn(0x7f912c07e6a0 msgr2=0x7f912c108d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.575+0000 7f9133e00700 1 -- 172.21.15.27:0/3207142654 shutdown_connections 2024-05-10T10:59:32.293 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-05-10T10:59:31.575+0000 7f9133e00700 1 -- 172.21.15.27:0/3207142654 wait complete. 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout:Or, if you are only running a single cluster on this host: 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: ceph telemetry on 2024-05-10T10:59:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:59:32.295 INFO:teuthology.orchestra.run.smithi027.stdout:For more information see: 2024-05-10T10:59:32.295 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:59:32.295 INFO:teuthology.orchestra.run.smithi027.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-10T10:59:32.295 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:59:32.295 INFO:teuthology.orchestra.run.smithi027.stdout:Bootstrap complete. 2024-05-10T10:59:32.379 INFO:tasks.cephadm:Fetching config... 2024-05-10T10:59:32.380 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T10:59:32.380 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-10T10:59:32.438 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-10T10:59:32.438 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T10:59:32.438 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-10T10:59:32.489 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-10T10:59:32.489 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T10:59:32.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-10T10:59:32.615 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-10T10:59:32.615 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T10:59:32.615 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-10T10:59:32.671 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-10T10:59:32.672 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpYPlGk5o+o4rwUpqYnMkbHtINiEkiuWeJ4AzmOPMcl8WQMpL/cPIn+G13OE8bPb4ovQw5YDyPu5u3DQS5MuaQXkfyvLWt3YUgs4t+SP2QkgLccSYaSd+JpF2znmhKThVkvVzjc0RT6+CCPd3qBfx4cHYH2igSveR4+lGUi2bXvagTJVlpSxkcql0Xs3bouCpv5YuWM6FtyVN5c/oBK0Q5HFEyzy8sVrxsWwu33cTv5vJU0a2dyez/eP/mYjreo85nNjoQK3AkJ6+c1ez84IGKFUHe4Hc3Axe49bYWJZY+v4JTwQal7Obb1OrEVWXjli/umzy2tlVUIlD1Oujkpb3zqdpB6J66leVchiYenN/FuI8JtxK5gzlSPq1NWJ1X/UJ7LrshFAHh41QI6K+7LWosUJWAABpcPlKrX0R/UJxzuUVJecy91MgS8ZjPovALvCYeikYdt8gehOROZkPKuKBZAl3T9dk86eVFlg/qbU0IldIIuRXuyonpWhIj7Jd7DqE= ceph-063c9e18-0ebc-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-10T10:59:32.782 INFO:teuthology.orchestra.run.smithi027.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpYPlGk5o+o4rwUpqYnMkbHtINiEkiuWeJ4AzmOPMcl8WQMpL/cPIn+G13OE8bPb4ovQw5YDyPu5u3DQS5MuaQXkfyvLWt3YUgs4t+SP2QkgLccSYaSd+JpF2znmhKThVkvVzjc0RT6+CCPd3qBfx4cHYH2igSveR4+lGUi2bXvagTJVlpSxkcql0Xs3bouCpv5YuWM6FtyVN5c/oBK0Q5HFEyzy8sVrxsWwu33cTv5vJU0a2dyez/eP/mYjreo85nNjoQK3AkJ6+c1ez84IGKFUHe4Hc3Axe49bYWJZY+v4JTwQal7Obb1OrEVWXjli/umzy2tlVUIlD1Oujkpb3zqdpB6J66leVchiYenN/FuI8JtxK5gzlSPq1NWJ1X/UJ7LrshFAHh41QI6K+7LWosUJWAABpcPlKrX0R/UJxzuUVJecy91MgS8ZjPovALvCYeikYdt8gehOROZkPKuKBZAl3T9dk86eVFlg/qbU0IldIIuRXuyonpWhIj7Jd7DqE= ceph-063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:59:32.816 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpYPlGk5o+o4rwUpqYnMkbHtINiEkiuWeJ4AzmOPMcl8WQMpL/cPIn+G13OE8bPb4ovQw5YDyPu5u3DQS5MuaQXkfyvLWt3YUgs4t+SP2QkgLccSYaSd+JpF2znmhKThVkvVzjc0RT6+CCPd3qBfx4cHYH2igSveR4+lGUi2bXvagTJVlpSxkcql0Xs3bouCpv5YuWM6FtyVN5c/oBK0Q5HFEyzy8sVrxsWwu33cTv5vJU0a2dyez/eP/mYjreo85nNjoQK3AkJ6+c1ez84IGKFUHe4Hc3Axe49bYWJZY+v4JTwQal7Obb1OrEVWXjli/umzy2tlVUIlD1Oujkpb3zqdpB6J66leVchiYenN/FuI8JtxK5gzlSPq1NWJ1X/UJ7LrshFAHh41QI6K+7LWosUJWAABpcPlKrX0R/UJxzuUVJecy91MgS8ZjPovALvCYeikYdt8gehOROZkPKuKBZAl3T9dk86eVFlg/qbU0IldIIuRXuyonpWhIj7Jd7DqE= ceph-063c9e18-0ebc-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-10T10:59:32.853 INFO:teuthology.orchestra.run.smithi076.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpYPlGk5o+o4rwUpqYnMkbHtINiEkiuWeJ4AzmOPMcl8WQMpL/cPIn+G13OE8bPb4ovQw5YDyPu5u3DQS5MuaQXkfyvLWt3YUgs4t+SP2QkgLccSYaSd+JpF2znmhKThVkvVzjc0RT6+CCPd3qBfx4cHYH2igSveR4+lGUi2bXvagTJVlpSxkcql0Xs3bouCpv5YuWM6FtyVN5c/oBK0Q5HFEyzy8sVrxsWwu33cTv5vJU0a2dyez/eP/mYjreo85nNjoQK3AkJ6+c1ez84IGKFUHe4Hc3Axe49bYWJZY+v4JTwQal7Obb1OrEVWXjli/umzy2tlVUIlD1Oujkpb3zqdpB6J66leVchiYenN/FuI8JtxK5gzlSPq1NWJ1X/UJ7LrshFAHh41QI6K+7LWosUJWAABpcPlKrX0R/UJxzuUVJecy91MgS8ZjPovALvCYeikYdt8gehOROZkPKuKBZAl3T9dk86eVFlg/qbU0IldIIuRXuyonpWhIj7Jd7DqE= ceph-063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:59:32.865 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpYPlGk5o+o4rwUpqYnMkbHtINiEkiuWeJ4AzmOPMcl8WQMpL/cPIn+G13OE8bPb4ovQw5YDyPu5u3DQS5MuaQXkfyvLWt3YUgs4t+SP2QkgLccSYaSd+JpF2znmhKThVkvVzjc0RT6+CCPd3qBfx4cHYH2igSveR4+lGUi2bXvagTJVlpSxkcql0Xs3bouCpv5YuWM6FtyVN5c/oBK0Q5HFEyzy8sVrxsWwu33cTv5vJU0a2dyez/eP/mYjreo85nNjoQK3AkJ6+c1ez84IGKFUHe4Hc3Axe49bYWJZY+v4JTwQal7Obb1OrEVWXjli/umzy2tlVUIlD1Oujkpb3zqdpB6J66leVchiYenN/FuI8JtxK5gzlSPq1NWJ1X/UJ7LrshFAHh41QI6K+7LWosUJWAABpcPlKrX0R/UJxzuUVJecy91MgS8ZjPovALvCYeikYdt8gehOROZkPKuKBZAl3T9dk86eVFlg/qbU0IldIIuRXuyonpWhIj7Jd7DqE= ceph-063c9e18-0ebc-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-10T10:59:32.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:32 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3207142654' entity='client.admin' 2024-05-10T10:59:32.903 INFO:teuthology.orchestra.run.smithi189.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpYPlGk5o+o4rwUpqYnMkbHtINiEkiuWeJ4AzmOPMcl8WQMpL/cPIn+G13OE8bPb4ovQw5YDyPu5u3DQS5MuaQXkfyvLWt3YUgs4t+SP2QkgLccSYaSd+JpF2znmhKThVkvVzjc0RT6+CCPd3qBfx4cHYH2igSveR4+lGUi2bXvagTJVlpSxkcql0Xs3bouCpv5YuWM6FtyVN5c/oBK0Q5HFEyzy8sVrxsWwu33cTv5vJU0a2dyez/eP/mYjreo85nNjoQK3AkJ6+c1ez84IGKFUHe4Hc3Axe49bYWJZY+v4JTwQal7Obb1OrEVWXjli/umzy2tlVUIlD1Oujkpb3zqdpB6J66leVchiYenN/FuI8JtxK5gzlSPq1NWJ1X/UJ7LrshFAHh41QI6K+7LWosUJWAABpcPlKrX0R/UJxzuUVJecy91MgS8ZjPovALvCYeikYdt8gehOROZkPKuKBZAl3T9dk86eVFlg/qbU0IldIIuRXuyonpWhIj7Jd7DqE= ceph-063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T10:59:32.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-10T10:59:33.368 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T10:59:35.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.702+0000 7f45b3000700 1 -- 172.21.15.27:0/152532152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac1096b0 msgr2=0x7f45ac109ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:35.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.702+0000 7f45b3000700 1 --2- 172.21.15.27:0/152532152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac1096b0 0x7f45ac109ad0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f459c009910 tx=0x7f459c009c20 comp rx=0 tx=0).stop 2024-05-10T10:59:35.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.702+0000 7f45b3000700 1 -- 172.21.15.27:0/152532152 shutdown_connections 2024-05-10T10:59:35.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.702+0000 7f45b3000700 1 --2- 172.21.15.27:0/152532152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac1096b0 0x7f45ac109ad0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:35.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.702+0000 7f45b3000700 1 -- 172.21.15.27:0/152532152 >> 172.21.15.27:0/152532152 conn(0x7f45ac078590 msgr2=0x7f45ac07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:35.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.702+0000 7f45b3000700 1 -- 172.21.15.27:0/152532152 shutdown_connections 2024-05-10T10:59:35.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.702+0000 7f45b3000700 1 -- 172.21.15.27:0/152532152 wait complete. 2024-05-10T10:59:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.704+0000 7f45b3000700 1 Processor -- start 2024-05-10T10:59:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.704+0000 7f45b3000700 1 -- start start 2024-05-10T10:59:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.704+0000 7f45b3000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac1096b0 0x7f45ac19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.704+0000 7f45b3000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac10f480 con 0x7f45ac1096b0 2024-05-10T10:59:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.705+0000 7f45abe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac1096b0 0x7f45ac19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.705+0000 7f45abe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac1096b0 0x7f45ac19eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60968/0 (socket says 172.21.15.27:60968) 2024-05-10T10:59:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.705+0000 7f45abe00700 1 -- 172.21.15.27:0/2365359746 learned_addr learned my addr 172.21.15.27:0/2365359746 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.705+0000 7f45abe00700 1 -- 172.21.15.27:0/2365359746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac19f0d0 con 0x7f45ac1096b0 2024-05-10T10:59:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.705+0000 7f45abe00700 1 --2- 172.21.15.27:0/2365359746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac1096b0 0x7f45ac19eb90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f459c005950 tx=0x7f459c004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.706+0000 7f45a8c00700 1 -- 172.21.15.27:0/2365359746 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f459c026070 con 0x7f45ac1096b0 2024-05-10T10:59:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.706+0000 7f45b3000700 1 -- 172.21.15.27:0/2365359746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45ac19f360 con 0x7f45ac1096b0 2024-05-10T10:59:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.706+0000 7f45a8c00700 1 -- 172.21.15.27:0/2365359746 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f459c00bc00 con 0x7f45ac1096b0 2024-05-10T10:59:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.706+0000 7f45a8c00700 1 -- 172.21.15.27:0/2365359746 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f459c003a10 con 0x7f45ac1096b0 2024-05-10T10:59:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.706+0000 7f45b3000700 1 -- 172.21.15.27:0/2365359746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45ac1a2000 con 0x7f45ac1096b0 2024-05-10T10:59:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.707+0000 7f45b3000700 1 -- 172.21.15.27:0/2365359746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45ac10fc00 con 0x7f45ac1096b0 2024-05-10T10:59:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.707+0000 7f45a8c00700 1 -- 172.21.15.27:0/2365359746 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f459c029070 con 0x7f45ac1096b0 2024-05-10T10:59:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.708+0000 7f45a8c00700 1 --2- 172.21.15.27:0/2365359746 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f459003e570 0x7f4590040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.708+0000 7f45a8c00700 1 -- 172.21.15.27:0/2365359746 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f459c01e070 con 0x7f45ac1096b0 2024-05-10T10:59:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.708+0000 7f45ab400700 1 --2- 172.21.15.27:0/2365359746 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f459003e570 0x7f4590040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.709+0000 7f45ab400700 1 --2- 172.21.15.27:0/2365359746 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f459003e570 0x7f4590040a30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4598006fd0 tx=0x7f4598006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.714+0000 7f45a8c00700 1 -- 172.21.15.27:0/2365359746 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f459c013070 con 0x7f45ac1096b0 2024-05-10T10:59:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.957+0000 7f45b3000700 1 -- 172.21.15.27:0/2365359746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f45ac02e620 con 0x7f45ac1096b0 2024-05-10T10:59:35.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.962+0000 7f45a8c00700 1 -- 172.21.15.27:0/2365359746 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f459c0300c0 con 0x7f45ac1096b0 2024-05-10T10:59:35.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.965+0000 7f45b3000700 1 -- 172.21.15.27:0/2365359746 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f459003e570 msgr2=0x7f4590040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:35.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.965+0000 7f45b3000700 1 --2- 172.21.15.27:0/2365359746 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f459003e570 0x7f4590040a30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4598006fd0 tx=0x7f4598006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:35.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.966+0000 7f45b3000700 1 -- 172.21.15.27:0/2365359746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac1096b0 msgr2=0x7f45ac19eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.966+0000 7f45b3000700 1 --2- 172.21.15.27:0/2365359746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac1096b0 0x7f45ac19eb90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f459c005950 tx=0x7f459c004fa0 comp rx=0 tx=0).stop 2024-05-10T10:59:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.966+0000 7f45b3000700 1 -- 172.21.15.27:0/2365359746 shutdown_connections 2024-05-10T10:59:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.966+0000 7f45b3000700 1 --2- 172.21.15.27:0/2365359746 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f459003e570 0x7f4590040a30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.966+0000 7f45b3000700 1 --2- 172.21.15.27:0/2365359746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac1096b0 0x7f45ac19eb90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.966+0000 7f45b3000700 1 -- 172.21.15.27:0/2365359746 >> 172.21.15.27:0/2365359746 conn(0x7f45ac078590 msgr2=0x7f45ac0790c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:35.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.966+0000 7f45b3000700 1 -- 172.21.15.27:0/2365359746 shutdown_connections 2024-05-10T10:59:35.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:35.966+0000 7f45b3000700 1 -- 172.21.15.27:0/2365359746 wait complete. 2024-05-10T10:59:37.087 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-10T10:59:37.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-10T10:59:37.284 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:36 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2365359746' entity='client.admin' 2024-05-10T10:59:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T10:59:39.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.934+0000 7f6eb6600700 1 -- 172.21.15.27:0/3862090049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eb0104f00 msgr2=0x7f6eb0105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:39.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.934+0000 7f6eb6600700 1 --2- 172.21.15.27:0/3862090049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eb0104f00 0x7f6eb0105340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009910 tx=0x7f6ea0009c20 comp rx=0 tx=0).stop 2024-05-10T10:59:39.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.935+0000 7f6eb6600700 1 -- 172.21.15.27:0/3862090049 shutdown_connections 2024-05-10T10:59:39.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.935+0000 7f6eb6600700 1 --2- 172.21.15.27:0/3862090049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eb0104f00 0x7f6eb0105340 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:39.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.935+0000 7f6eb6600700 1 -- 172.21.15.27:0/3862090049 >> 172.21.15.27:0/3862090049 conn(0x7f6eb01004f0 msgr2=0x7f6eb0102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:39.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.935+0000 7f6eb6600700 1 -- 172.21.15.27:0/3862090049 shutdown_connections 2024-05-10T10:59:39.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.935+0000 7f6eb6600700 1 -- 172.21.15.27:0/3862090049 wait complete. 2024-05-10T10:59:39.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.936+0000 7f6eb6600700 1 Processor -- start 2024-05-10T10:59:39.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.936+0000 7f6eb6600700 1 -- start start 2024-05-10T10:59:39.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.937+0000 7f6eb6600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eb0104f00 0x7f6eb019a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:39.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.937+0000 7f6eb6600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb010adf0 con 0x7f6eb0104f00 2024-05-10T10:59:39.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.937+0000 7f6eb5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eb0104f00 0x7f6eb019a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:39.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.937+0000 7f6eb5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eb0104f00 0x7f6eb019a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60984/0 (socket says 172.21.15.27:60984) 2024-05-10T10:59:39.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.937+0000 7f6eb5200700 1 -- 172.21.15.27:0/1535637399 learned_addr learned my addr 172.21.15.27:0/1535637399 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:39.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.937+0000 7f6eb5200700 1 -- 172.21.15.27:0/1535637399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb019aae0 con 0x7f6eb0104f00 2024-05-10T10:59:39.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.938+0000 7f6eb5200700 1 --2- 172.21.15.27:0/1535637399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eb0104f00 0x7f6eb019a5a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0005950 tx=0x7f6ea0004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:39.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.938+0000 7f6eae000700 1 -- 172.21.15.27:0/1535637399 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ea0026070 con 0x7f6eb0104f00 2024-05-10T10:59:39.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.938+0000 7f6eae000700 1 -- 172.21.15.27:0/1535637399 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ea000bc00 con 0x7f6eb0104f00 2024-05-10T10:59:39.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.938+0000 7f6eb6600700 1 -- 172.21.15.27:0/1535637399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb019ad70 con 0x7f6eb0104f00 2024-05-10T10:59:39.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.938+0000 7f6eae000700 1 -- 172.21.15.27:0/1535637399 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ea0003a10 con 0x7f6eb0104f00 2024-05-10T10:59:39.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.938+0000 7f6eb6600700 1 -- 172.21.15.27:0/1535637399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb019da60 con 0x7f6eb0104f00 2024-05-10T10:59:39.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.939+0000 7f6eb6600700 1 -- 172.21.15.27:0/1535637399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb010b570 con 0x7f6eb0104f00 2024-05-10T10:59:39.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.939+0000 7f6eae000700 1 -- 172.21.15.27:0/1535637399 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f6ea0029070 con 0x7f6eb0104f00 2024-05-10T10:59:39.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.940+0000 7f6eae000700 1 --2- 172.21.15.27:0/1535637399 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6e9403e510 0x7f6e940409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:39.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.940+0000 7f6eae000700 1 -- 172.21.15.27:0/1535637399 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6ea001e070 con 0x7f6eb0104f00 2024-05-10T10:59:39.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.940+0000 7f6ea7e00700 1 --2- 172.21.15.27:0/1535637399 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6e9403e510 0x7f6e940409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:39.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.941+0000 7f6ea7e00700 1 --2- 172.21.15.27:0/1535637399 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6e9403e510 0x7f6e940409d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6e98006fd0 tx=0x7f6e98006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:39.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:39.946+0000 7f6eae000700 1 -- 172.21.15.27:0/1535637399 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6ea0013070 con 0x7f6eb0104f00 2024-05-10T10:59:40.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:40.187+0000 7f6eb6600700 1 -- 172.21.15.27:0/1535637399 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f6eb002e620 con 0x7f6e9403e510 2024-05-10T10:59:40.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:40.195+0000 7f6eae000700 1 -- 172.21.15.27:0/1535637399 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6eb002e620 con 0x7f6e9403e510 2024-05-10T10:59:40.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:40.198+0000 7f6eb6600700 1 -- 172.21.15.27:0/1535637399 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6e9403e510 msgr2=0x7f6e940409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:40.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:40.198+0000 7f6eb6600700 1 --2- 172.21.15.27:0/1535637399 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6e9403e510 0x7f6e940409d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6e98006fd0 tx=0x7f6e98006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:40.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:40.198+0000 7f6eb6600700 1 -- 172.21.15.27:0/1535637399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eb0104f00 msgr2=0x7f6eb019a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:40.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:40.198+0000 7f6eb6600700 1 --2- 172.21.15.27:0/1535637399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eb0104f00 0x7f6eb019a5a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0005950 tx=0x7f6ea0004fa0 comp rx=0 tx=0).stop 2024-05-10T10:59:40.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:40.198+0000 7f6eb6600700 1 -- 172.21.15.27:0/1535637399 shutdown_connections 2024-05-10T10:59:40.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:40.198+0000 7f6eb6600700 1 --2- 172.21.15.27:0/1535637399 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6e9403e510 0x7f6e940409d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:40.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:40.198+0000 7f6eb6600700 1 --2- 172.21.15.27:0/1535637399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eb0104f00 0x7f6eb019a5a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:40.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:40.198+0000 7f6eb6600700 1 -- 172.21.15.27:0/1535637399 >> 172.21.15.27:0/1535637399 conn(0x7f6eb01004f0 msgr2=0x7f6eb0101080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:40.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:40.199+0000 7f6eb6600700 1 -- 172.21.15.27:0/1535637399 shutdown_connections 2024-05-10T10:59:40.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:40.199+0000 7f6eb6600700 1 -- 172.21.15.27:0/1535637399 wait complete. 2024-05-10T10:59:40.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:40.439 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:40.439 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-05-10T10:59:40.439 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:40.439 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T10:59:40.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:40.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:40.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T10:59:40.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T10:59:40.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T10:59:40.974 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi076 2024-05-10T10:59:40.974 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-05-10T10:59:40.974 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/etc/ceph/ceph.conf 2024-05-10T10:59:40.992 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-05-10T10:59:40.992 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-10T10:59:41.049 INFO:tasks.cephadm:Adding host smithi076 to orchestrator... 2024-05-10T10:59:41.050 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch host add smithi076 2024-05-10T10:59:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T10:59:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:41 smithi027 ceph-mon[30054]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T10:59:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:41 smithi027 ceph-mon[30054]: Updating smithi027:/etc/ceph/ceph.conf 2024-05-10T10:59:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:41 smithi027 ceph-mon[30054]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T10:59:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:41 smithi027 ceph-mon[30054]: Updating smithi027:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T10:59:42.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:42 smithi027 ceph-mon[30054]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-05-10T10:59:43.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.607+0000 7f2c20600700 1 -- 172.21.15.27:0/1298893390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18102010 msgr2=0x7f2c181023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:43.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.607+0000 7f2c20600700 1 --2- 172.21.15.27:0/1298893390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18102010 0x7f2c181023f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009910 tx=0x7f2c0c009c20 comp rx=0 tx=0).stop 2024-05-10T10:59:43.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.607+0000 7f2c20600700 1 -- 172.21.15.27:0/1298893390 shutdown_connections 2024-05-10T10:59:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.607+0000 7f2c20600700 1 --2- 172.21.15.27:0/1298893390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18102010 0x7f2c181023f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.607+0000 7f2c20600700 1 -- 172.21.15.27:0/1298893390 >> 172.21.15.27:0/1298893390 conn(0x7f2c180fd8c0 msgr2=0x7f2c180ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.607+0000 7f2c20600700 1 -- 172.21.15.27:0/1298893390 shutdown_connections 2024-05-10T10:59:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.607+0000 7f2c20600700 1 -- 172.21.15.27:0/1298893390 wait complete. 2024-05-10T10:59:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.609+0000 7f2c20600700 1 Processor -- start 2024-05-10T10:59:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.609+0000 7f2c20600700 1 -- start start 2024-05-10T10:59:43.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.609+0000 7f2c20600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18102010 0x7f2c18103ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:43.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.609+0000 7f2c20600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1810c290 con 0x7f2c18102010 2024-05-10T10:59:43.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.610+0000 7f2c1dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18102010 0x7f2c18103ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:43.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.610+0000 7f2c1dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18102010 0x7f2c18103ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60998/0 (socket says 172.21.15.27:60998) 2024-05-10T10:59:43.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.610+0000 7f2c1dc00700 1 -- 172.21.15.27:0/1915331505 learned_addr learned my addr 172.21.15.27:0/1915331505 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:43.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.610+0000 7f2c1dc00700 1 -- 172.21.15.27:0/1915331505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c181059a0 con 0x7f2c18102010 2024-05-10T10:59:43.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.610+0000 7f2c1dc00700 1 --2- 172.21.15.27:0/1915331505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18102010 0x7f2c18103ab0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c006010 tx=0x7f2c0c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:43.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.611+0000 7f2c16000700 1 -- 172.21.15.27:0/1915331505 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c0c026070 con 0x7f2c18102010 2024-05-10T10:59:43.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.611+0000 7f2c16000700 1 -- 172.21.15.27:0/1915331505 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c0c00bc00 con 0x7f2c18102010 2024-05-10T10:59:43.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.611+0000 7f2c16000700 1 -- 172.21.15.27:0/1915331505 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c0c003910 con 0x7f2c18102010 2024-05-10T10:59:43.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.611+0000 7f2c20600700 1 -- 172.21.15.27:0/1915331505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c18104050 con 0x7f2c18102010 2024-05-10T10:59:43.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.611+0000 7f2c20600700 1 -- 172.21.15.27:0/1915331505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c18104510 con 0x7f2c18102010 2024-05-10T10:59:43.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.612+0000 7f2c20600700 1 -- 172.21.15.27:0/1915331505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c181034f0 con 0x7f2c18102010 2024-05-10T10:59:43.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.612+0000 7f2c16000700 1 -- 172.21.15.27:0/1915331505 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f2c0c029070 con 0x7f2c18102010 2024-05-10T10:59:43.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.613+0000 7f2c16000700 1 --2- 172.21.15.27:0/1915331505 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2c0003e570 0x7f2c00040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:43.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.613+0000 7f2c16000700 1 -- 172.21.15.27:0/1915331505 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2c0c01e070 con 0x7f2c18102010 2024-05-10T10:59:43.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.613+0000 7f2c1d200700 1 --2- 172.21.15.27:0/1915331505 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2c0003e570 0x7f2c00040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:43.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.614+0000 7f2c1d200700 1 --2- 172.21.15.27:0/1915331505 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2c0003e570 0x7f2c00040a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2c04006fd0 tx=0x7f2c04006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:43.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.619+0000 7f2c16000700 1 -- 172.21.15.27:0/1915331505 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f2c0c013070 con 0x7f2c18102010 2024-05-10T10:59:43.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:43.864+0000 7f2c20600700 1 -- 172.21.15.27:0/1915331505 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi076", "target": ["mon-mgr", ""]}) -- 0x7f2c18104fe0 con 0x7f2c0003e570 2024-05-10T10:59:44.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:43 smithi027 ceph-mon[30054]: Updating smithi027:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T10:59:44.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:43 smithi027 ceph-mon[30054]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T10:59:44.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:44.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:44.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:45.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:44 smithi027 ceph-mon[30054]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi076", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T10:59:46.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:45 smithi027 ceph-mon[30054]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T10:59:46.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:45 smithi027 ceph-mon[30054]: Deploying cephadm binary to smithi076 2024-05-10T10:59:48.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:47 smithi027 ceph-mon[30054]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T10:59:48.802 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi076' with addr '172.21.15.76' 2024-05-10T10:59:48.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:48.800+0000 7f2c16000700 1 -- 172.21.15.27:0/1915331505 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f2c18104fe0 con 0x7f2c0003e570 2024-05-10T10:59:48.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:48.804+0000 7f2c20600700 1 -- 172.21.15.27:0/1915331505 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2c0003e570 msgr2=0x7f2c00040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:48.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:48.804+0000 7f2c20600700 1 --2- 172.21.15.27:0/1915331505 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2c0003e570 0x7f2c00040a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2c04006fd0 tx=0x7f2c04006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:48.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:48.804+0000 7f2c20600700 1 -- 172.21.15.27:0/1915331505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18102010 msgr2=0x7f2c18103ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:48.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:48.804+0000 7f2c20600700 1 --2- 172.21.15.27:0/1915331505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18102010 0x7f2c18103ab0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c006010 tx=0x7f2c0c005780 comp rx=0 tx=0).stop 2024-05-10T10:59:48.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:48.805+0000 7f2c20600700 1 -- 172.21.15.27:0/1915331505 shutdown_connections 2024-05-10T10:59:48.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:48.805+0000 7f2c20600700 1 --2- 172.21.15.27:0/1915331505 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2c0003e570 0x7f2c00040a30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:48.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:48.805+0000 7f2c20600700 1 --2- 172.21.15.27:0/1915331505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18102010 0x7f2c18103ab0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:48.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:48.805+0000 7f2c20600700 1 -- 172.21.15.27:0/1915331505 >> 172.21.15.27:0/1915331505 conn(0x7f2c180fd8c0 msgr2=0x7f2c1806b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:48.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:48.805+0000 7f2c20600700 1 -- 172.21.15.27:0/1915331505 shutdown_connections 2024-05-10T10:59:48.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:48.805+0000 7f2c20600700 1 -- 172.21.15.27:0/1915331505 wait complete. 2024-05-10T10:59:49.518 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-10T10:59:49.977 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T10:59:50.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:49 smithi027 ceph-mon[30054]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T10:59:50.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:50.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:49 smithi027 ceph-mon[30054]: Added host smithi076 2024-05-10T10:59:50.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T10:59:50.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:51.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:50 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.851+0000 7f4bbd200700 1 -- 172.21.15.27:0/671943360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb810e4e0 msgr2=0x7f4bb810e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.851+0000 7f4bbd200700 1 --2- 172.21.15.27:0/671943360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb810e4e0 0x7f4bb810e8c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009910 tx=0x7f4ba8009c20 comp rx=0 tx=0).stop 2024-05-10T10:59:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.851+0000 7f4bbd200700 1 -- 172.21.15.27:0/671943360 shutdown_connections 2024-05-10T10:59:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.851+0000 7f4bbd200700 1 --2- 172.21.15.27:0/671943360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb810e4e0 0x7f4bb810e8c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.851+0000 7f4bbd200700 1 -- 172.21.15.27:0/671943360 >> 172.21.15.27:0/671943360 conn(0x7f4bb8078c40 msgr2=0x7f4bb807b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.851+0000 7f4bbd200700 1 -- 172.21.15.27:0/671943360 shutdown_connections 2024-05-10T10:59:51.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.852+0000 7f4bbd200700 1 -- 172.21.15.27:0/671943360 wait complete. 2024-05-10T10:59:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.853+0000 7f4bbd200700 1 Processor -- start 2024-05-10T10:59:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.853+0000 7f4bbd200700 1 -- start start 2024-05-10T10:59:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.853+0000 7f4bbd200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb810e4e0 0x7f4bb819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.854+0000 7f4bbd200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb8113a00 con 0x7f4bb810e4e0 2024-05-10T10:59:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.854+0000 7f4bb5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb810e4e0 0x7f4bb819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.854+0000 7f4bb5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb810e4e0 0x7f4bb819eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54380/0 (socket says 172.21.15.27:54380) 2024-05-10T10:59:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.854+0000 7f4bb5e00700 1 -- 172.21.15.27:0/2187482955 learned_addr learned my addr 172.21.15.27:0/2187482955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.854+0000 7f4bb5e00700 1 -- 172.21.15.27:0/2187482955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb819f430 con 0x7f4bb810e4e0 2024-05-10T10:59:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.855+0000 7f4bb5e00700 1 --2- 172.21.15.27:0/2187482955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb810e4e0 0x7f4bb819eef0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80092e0 tx=0x7f4ba8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.855+0000 7f4baea00700 1 -- 172.21.15.27:0/2187482955 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ba8026070 con 0x7f4bb810e4e0 2024-05-10T10:59:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.855+0000 7f4baea00700 1 -- 172.21.15.27:0/2187482955 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ba800b750 con 0x7f4bb810e4e0 2024-05-10T10:59:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.855+0000 7f4bbd200700 1 -- 172.21.15.27:0/2187482955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb819f6c0 con 0x7f4bb810e4e0 2024-05-10T10:59:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.855+0000 7f4baea00700 1 -- 172.21.15.27:0/2187482955 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ba80038a0 con 0x7f4bb810e4e0 2024-05-10T10:59:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.855+0000 7f4bbd200700 1 -- 172.21.15.27:0/2187482955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb81a3770 con 0x7f4bb810e4e0 2024-05-10T10:59:51.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.856+0000 7f4bbd200700 1 -- 172.21.15.27:0/2187482955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb810abe0 con 0x7f4bb810e4e0 2024-05-10T10:59:51.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.856+0000 7f4baea00700 1 -- 172.21.15.27:0/2187482955 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f4ba8029070 con 0x7f4bb810e4e0 2024-05-10T10:59:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.857+0000 7f4baea00700 1 --2- 172.21.15.27:0/2187482955 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4b9c03e570 0x7f4b9c040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.857+0000 7f4baea00700 1 -- 172.21.15.27:0/2187482955 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4ba801e070 con 0x7f4bb810e4e0 2024-05-10T10:59:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.857+0000 7f4bb5400700 1 --2- 172.21.15.27:0/2187482955 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4b9c03e570 0x7f4b9c040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.858+0000 7f4bb5400700 1 --2- 172.21.15.27:0/2187482955 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4b9c03e570 0x7f4b9c040a30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4006fd0 tx=0x7f4ba4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:51.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:51.863+0000 7f4baea00700 1 -- 172.21.15.27:0/2187482955 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4ba8013070 con 0x7f4bb810e4e0 2024-05-10T10:59:52.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:52.105+0000 7f4bbd200700 1 -- 172.21.15.27:0/2187482955 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4bb81a0200 con 0x7f4b9c03e570 2024-05-10T10:59:52.107 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:51 smithi027 ceph-mon[30054]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T10:59:52.107 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:51 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:52.109+0000 7f4baea00700 1 -- 172.21.15.27:0/2187482955 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f4bb81a0200 con 0x7f4b9c03e570 2024-05-10T10:59:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T10:59:52.110 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.76", "hostname": "smithi076", "labels": [], "status": ""}] 2024-05-10T10:59:52.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:52.112+0000 7f4bbd200700 1 -- 172.21.15.27:0/2187482955 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4b9c03e570 msgr2=0x7f4b9c040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:52.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:52.112+0000 7f4bbd200700 1 --2- 172.21.15.27:0/2187482955 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4b9c03e570 0x7f4b9c040a30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4006fd0 tx=0x7f4ba4006e40 comp rx=0 tx=0).stop 2024-05-10T10:59:52.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:52.112+0000 7f4bbd200700 1 -- 172.21.15.27:0/2187482955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb810e4e0 msgr2=0x7f4bb819eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:52.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:52.112+0000 7f4bbd200700 1 --2- 172.21.15.27:0/2187482955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb810e4e0 0x7f4bb819eef0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80092e0 tx=0x7f4ba8005780 comp rx=0 tx=0).stop 2024-05-10T10:59:52.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:52.113+0000 7f4bbd200700 1 -- 172.21.15.27:0/2187482955 shutdown_connections 2024-05-10T10:59:52.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:52.113+0000 7f4bbd200700 1 --2- 172.21.15.27:0/2187482955 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4b9c03e570 0x7f4b9c040a30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:52.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:52.113+0000 7f4bbd200700 1 --2- 172.21.15.27:0/2187482955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb810e4e0 0x7f4bb819eef0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:52.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:52.113+0000 7f4bbd200700 1 -- 172.21.15.27:0/2187482955 >> 172.21.15.27:0/2187482955 conn(0x7f4bb8078c40 msgr2=0x7f4bb8111040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:52.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:52.113+0000 7f4bbd200700 1 -- 172.21.15.27:0/2187482955 shutdown_connections 2024-05-10T10:59:52.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:52.113+0000 7f4bbd200700 1 -- 172.21.15.27:0/2187482955 wait complete. 2024-05-10T10:59:52.950 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi189 2024-05-10T10:59:52.950 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-10T10:59:52.950 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.conf 2024-05-10T10:59:52.970 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-10T10:59:52.970 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-10T10:59:53.029 INFO:tasks.cephadm:Adding host smithi189 to orchestrator... 2024-05-10T10:59:53.030 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch host add smithi189 2024-05-10T10:59:53.490 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T10:59:54.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:53 smithi027 ceph-mon[30054]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T10:59:54.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:53 smithi027 ceph-mon[30054]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T10:59:55.812 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:55 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:55.812 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:55 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:55.813 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:55 smithi027 ceph-mon[30054]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T10:59:55.813 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:55 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:55.813 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:55 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:55.813 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:55 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-05-10T10:59:55.813 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:55 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T10:59:55.813 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:55 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T10:59:55.813 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:55 smithi027 ceph-mon[30054]: Updating smithi076:/etc/ceph/ceph.conf 2024-05-10T10:59:55.813 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:55 smithi027 ceph-mon[30054]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T10:59:55.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.883+0000 7fa6c6400700 1 -- 172.21.15.27:0/491096930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0105330 msgr2=0x7fa6c0105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T10:59:55.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.883+0000 7fa6c6400700 1 --2- 172.21.15.27:0/491096930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0105330 0x7fa6c0105710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009910 tx=0x7fa6b0009c20 comp rx=0 tx=0).stop 2024-05-10T10:59:55.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.883+0000 7fa6c6400700 1 -- 172.21.15.27:0/491096930 shutdown_connections 2024-05-10T10:59:55.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.883+0000 7fa6c6400700 1 --2- 172.21.15.27:0/491096930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0105330 0x7fa6c0105710 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T10:59:55.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.883+0000 7fa6c6400700 1 -- 172.21.15.27:0/491096930 >> 172.21.15.27:0/491096930 conn(0x7fa6c0100bc0 msgr2=0x7fa6c0102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T10:59:55.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.883+0000 7fa6c6400700 1 -- 172.21.15.27:0/491096930 shutdown_connections 2024-05-10T10:59:55.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.884+0000 7fa6c6400700 1 -- 172.21.15.27:0/491096930 wait complete. 2024-05-10T10:59:55.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.885+0000 7fa6c6400700 1 Processor -- start 2024-05-10T10:59:55.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.885+0000 7fa6c6400700 1 -- start start 2024-05-10T10:59:55.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.886+0000 7fa6c6400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0105330 0x7fa6c01a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:55.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.886+0000 7fa6c6400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c010f5b0 con 0x7fa6c0105330 2024-05-10T10:59:55.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.886+0000 7fa6bf400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0105330 0x7fa6c01a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:55.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.886+0000 7fa6bf400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0105330 0x7fa6c01a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39250/0 (socket says 172.21.15.27:39250) 2024-05-10T10:59:55.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.886+0000 7fa6bf400700 1 -- 172.21.15.27:0/213450725 learned_addr learned my addr 172.21.15.27:0/213450725 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T10:59:55.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.886+0000 7fa6bf400700 1 -- 172.21.15.27:0/213450725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c01a0cd0 con 0x7fa6c0105330 2024-05-10T10:59:55.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.887+0000 7fa6bf400700 1 --2- 172.21.15.27:0/213450725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0105330 0x7fa6c01a0790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa6b00056d0 tx=0x7fa6b00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:55.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.887+0000 7fa6b7e00700 1 -- 172.21.15.27:0/213450725 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6b0026070 con 0x7fa6c0105330 2024-05-10T10:59:55.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.887+0000 7fa6b7e00700 1 -- 172.21.15.27:0/213450725 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6b000bba0 con 0x7fa6c0105330 2024-05-10T10:59:55.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.887+0000 7fa6c6400700 1 -- 172.21.15.27:0/213450725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6c01a0fc0 con 0x7fa6c0105330 2024-05-10T10:59:55.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.887+0000 7fa6b7e00700 1 -- 172.21.15.27:0/213450725 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6b0003920 con 0x7fa6c0105330 2024-05-10T10:59:55.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.887+0000 7fa6c6400700 1 -- 172.21.15.27:0/213450725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6c019a950 con 0x7fa6c0105330 2024-05-10T10:59:55.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.889+0000 7fa6b7e00700 1 -- 172.21.15.27:0/213450725 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fa6b0029070 con 0x7fa6c0105330 2024-05-10T10:59:55.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.889+0000 7fa6c6400700 1 -- 172.21.15.27:0/213450725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6c0106810 con 0x7fa6c0105330 2024-05-10T10:59:55.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.889+0000 7fa6b7e00700 1 --2- 172.21.15.27:0/213450725 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa6a403e570 0x7fa6a4040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T10:59:55.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.889+0000 7fa6b7e00700 1 -- 172.21.15.27:0/213450725 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa6b001e070 con 0x7fa6c0105330 2024-05-10T10:59:55.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.889+0000 7fa6bea00700 1 --2- 172.21.15.27:0/213450725 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa6a403e570 0x7fa6a4040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T10:59:55.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.890+0000 7fa6bea00700 1 --2- 172.21.15.27:0/213450725 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa6a403e570 0x7fa6a4040a30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac006fd0 tx=0x7fa6ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T10:59:55.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:55.895+0000 7fa6b7e00700 1 -- 172.21.15.27:0/213450725 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa6b0013070 con 0x7fa6c0105330 2024-05-10T10:59:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T10:59:56.142+0000 7fa6c6400700 1 -- 172.21.15.27:0/213450725 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}) -- 0x7fa6c019be80 con 0x7fa6a403e570 2024-05-10T10:59:57.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:56 smithi027 ceph-mon[30054]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-05-10T10:59:57.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:56 smithi027 ceph-mon[30054]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T10:59:57.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:57.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:57.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T10:59:58.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:57 smithi027 ceph-mon[30054]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T10:59:58.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:57 smithi027 ceph-mon[30054]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T10:59:58.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:57 smithi027 ceph-mon[30054]: Deploying cephadm binary to smithi189 2024-05-10T11:00:00.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 10:59:59 smithi027 ceph-mon[30054]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:01.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:01.104+0000 7fa6b7e00700 1 -- 172.21.15.27:0/213450725 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fa6c019be80 con 0x7fa6a403e570 2024-05-10T11:00:01.105 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi189' with addr '172.21.15.189' 2024-05-10T11:00:01.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:01.109+0000 7fa6c6400700 1 -- 172.21.15.27:0/213450725 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa6a403e570 msgr2=0x7fa6a4040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:01.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:01.109+0000 7fa6c6400700 1 --2- 172.21.15.27:0/213450725 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa6a403e570 0x7fa6a4040a30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac006fd0 tx=0x7fa6ac006e40 comp rx=0 tx=0).stop 2024-05-10T11:00:01.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:01.109+0000 7fa6c6400700 1 -- 172.21.15.27:0/213450725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0105330 msgr2=0x7fa6c01a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:01.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:01.109+0000 7fa6c6400700 1 --2- 172.21.15.27:0/213450725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0105330 0x7fa6c01a0790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa6b00056d0 tx=0x7fa6b00057b0 comp rx=0 tx=0).stop 2024-05-10T11:00:01.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:01.109+0000 7fa6c6400700 1 -- 172.21.15.27:0/213450725 shutdown_connections 2024-05-10T11:00:01.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:01.109+0000 7fa6c6400700 1 --2- 172.21.15.27:0/213450725 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa6a403e570 0x7fa6a4040a30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:01.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:01.109+0000 7fa6c6400700 1 --2- 172.21.15.27:0/213450725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0105330 0x7fa6c01a0790 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:01.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:01.109+0000 7fa6c6400700 1 -- 172.21.15.27:0/213450725 >> 172.21.15.27:0/213450725 conn(0x7fa6c0100bc0 msgr2=0x7fa6c01080d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:01.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:01.110+0000 7fa6c6400700 1 -- 172.21.15.27:0/213450725 shutdown_connections 2024-05-10T11:00:01.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:01.110+0000 7fa6c6400700 1 -- 172.21.15.27:0/213450725 wait complete. 2024-05-10T11:00:02.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-10T11:00:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:01 smithi027 ceph-mon[30054]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:02.531 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:00:02.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:02 smithi027 ceph-mon[30054]: Added host smithi189 2024-05-10T11:00:02.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:02 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:04.264 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:03 smithi027 ceph-mon[30054]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:04.264 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.919+0000 7f49f1200700 1 -- 172.21.15.27:0/702236787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49ec0757c0 msgr2=0x7f49ec075ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.919+0000 7f49f1200700 1 --2- 172.21.15.27:0/702236787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49ec0757c0 0x7f49ec075ba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009910 tx=0x7f49dc009c20 comp rx=0 tx=0).stop 2024-05-10T11:00:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.920+0000 7f49f1200700 1 -- 172.21.15.27:0/702236787 shutdown_connections 2024-05-10T11:00:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.920+0000 7f49f1200700 1 --2- 172.21.15.27:0/702236787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49ec0757c0 0x7f49ec075ba0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.920+0000 7f49f1200700 1 -- 172.21.15.27:0/702236787 >> 172.21.15.27:0/702236787 conn(0x7f49ec0fe9e0 msgr2=0x7f49ec100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.920+0000 7f49f1200700 1 -- 172.21.15.27:0/702236787 shutdown_connections 2024-05-10T11:00:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.920+0000 7f49f1200700 1 -- 172.21.15.27:0/702236787 wait complete. 2024-05-10T11:00:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.921+0000 7f49f1200700 1 Processor -- start 2024-05-10T11:00:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.921+0000 7f49f1200700 1 -- start start 2024-05-10T11:00:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.921+0000 7f49f1200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49ec0757c0 0x7f49ec1a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.921+0000 7f49f1200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec10f5f0 con 0x7f49ec0757c0 2024-05-10T11:00:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.922+0000 7f49e9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49ec0757c0 0x7f49ec1a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.922+0000 7f49e9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49ec0757c0 0x7f49ec1a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55262/0 (socket says 172.21.15.27:55262) 2024-05-10T11:00:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.922+0000 7f49e9e00700 1 -- 172.21.15.27:0/2977875591 learned_addr learned my addr 172.21.15.27:0/2977875591 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:00:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.922+0000 7f49e9e00700 1 -- 172.21.15.27:0/2977875591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec1a0cf0 con 0x7f49ec0757c0 2024-05-10T11:00:04.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.922+0000 7f49e9e00700 1 --2- 172.21.15.27:0/2977875591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49ec0757c0 0x7f49ec1a07b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f49dc006010 tx=0x7f49dc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:04.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.923+0000 7f49e2a00700 1 -- 172.21.15.27:0/2977875591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49dc026070 con 0x7f49ec0757c0 2024-05-10T11:00:04.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.923+0000 7f49e2a00700 1 -- 172.21.15.27:0/2977875591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49dc00bc00 con 0x7f49ec0757c0 2024-05-10T11:00:04.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.923+0000 7f49f1200700 1 -- 172.21.15.27:0/2977875591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ec1a0f80 con 0x7f49ec0757c0 2024-05-10T11:00:04.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.923+0000 7f49e2a00700 1 -- 172.21.15.27:0/2977875591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49dc003910 con 0x7f49ec0757c0 2024-05-10T11:00:04.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.923+0000 7f49f1200700 1 -- 172.21.15.27:0/2977875591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ec19a970 con 0x7f49ec0757c0 2024-05-10T11:00:04.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.924+0000 7f49f1200700 1 -- 172.21.15.27:0/2977875591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ec076f00 con 0x7f49ec0757c0 2024-05-10T11:00:04.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.924+0000 7f49e2a00700 1 -- 172.21.15.27:0/2977875591 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f49dc029070 con 0x7f49ec0757c0 2024-05-10T11:00:04.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.925+0000 7f49e2a00700 1 --2- 172.21.15.27:0/2977875591 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49d003e570 0x7f49d0040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:04.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.925+0000 7f49e2a00700 1 -- 172.21.15.27:0/2977875591 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f49dc01e070 con 0x7f49ec0757c0 2024-05-10T11:00:04.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.925+0000 7f49e9400700 1 --2- 172.21.15.27:0/2977875591 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49d003e570 0x7f49d0040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:04.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.926+0000 7f49e9400700 1 --2- 172.21.15.27:0/2977875591 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49d003e570 0x7f49d0040a30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f49d8006fd0 tx=0x7f49d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:04.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:04.931+0000 7f49e2a00700 1 -- 172.21.15.27:0/2977875591 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f49dc013070 con 0x7f49ec0757c0 2024-05-10T11:00:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:05.183+0000 7f49f1200700 1 -- 172.21.15.27:0/2977875591 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f49ec19bea0 con 0x7f49d003e570 2024-05-10T11:00:05.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:05.185+0000 7f49e2a00700 1 -- 172.21.15.27:0/2977875591 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f49ec19bea0 con 0x7f49d003e570 2024-05-10T11:00:05.187 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:00:05.187 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.76", "hostname": "smithi076", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}] 2024-05-10T11:00:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:05.189+0000 7f49f1200700 1 -- 172.21.15.27:0/2977875591 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49d003e570 msgr2=0x7f49d0040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:05.189+0000 7f49f1200700 1 --2- 172.21.15.27:0/2977875591 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49d003e570 0x7f49d0040a30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f49d8006fd0 tx=0x7f49d8006e40 comp rx=0 tx=0).stop 2024-05-10T11:00:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:05.189+0000 7f49f1200700 1 -- 172.21.15.27:0/2977875591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49ec0757c0 msgr2=0x7f49ec1a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:05.189+0000 7f49f1200700 1 --2- 172.21.15.27:0/2977875591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49ec0757c0 0x7f49ec1a07b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f49dc006010 tx=0x7f49dc005780 comp rx=0 tx=0).stop 2024-05-10T11:00:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:05.189+0000 7f49f1200700 1 -- 172.21.15.27:0/2977875591 shutdown_connections 2024-05-10T11:00:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:05.189+0000 7f49f1200700 1 --2- 172.21.15.27:0/2977875591 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49d003e570 0x7f49d0040a30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:05.189+0000 7f49f1200700 1 --2- 172.21.15.27:0/2977875591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49ec0757c0 0x7f49ec1a07b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:05.189+0000 7f49f1200700 1 -- 172.21.15.27:0/2977875591 >> 172.21.15.27:0/2977875591 conn(0x7f49ec0fe9e0 msgr2=0x7f49ec1071d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:05.189+0000 7f49f1200700 1 -- 172.21.15.27:0/2977875591 shutdown_connections 2024-05-10T11:00:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:05.189+0000 7f49f1200700 1 -- 172.21.15.27:0/2977875591 wait complete. 2024-05-10T11:00:05.924 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-10T11:00:05.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd crush tunables default 2024-05-10T11:00:06.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:05 smithi027 ceph-mon[30054]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:00:07.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:06 smithi027 ceph-mon[30054]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:00:08.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:07 smithi027 ceph-mon[30054]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:08.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:08.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:08.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:08.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:08.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:00:08.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:08.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.527+0000 7f11db200700 1 -- 172.21.15.27:0/1180008940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11dc105160 msgr2=0x7f11dc105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.527+0000 7f11db200700 1 --2- 172.21.15.27:0/1180008940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11dc105160 0x7f11dc105540 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f11cc009910 tx=0x7f11cc009c20 comp rx=0 tx=0).stop 2024-05-10T11:00:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.527+0000 7f11db200700 1 -- 172.21.15.27:0/1180008940 shutdown_connections 2024-05-10T11:00:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.527+0000 7f11db200700 1 --2- 172.21.15.27:0/1180008940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11dc105160 0x7f11dc105540 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.527+0000 7f11db200700 1 -- 172.21.15.27:0/1180008940 >> 172.21.15.27:0/1180008940 conn(0x7f11dc1009d0 msgr2=0x7f11dc102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.527+0000 7f11db200700 1 -- 172.21.15.27:0/1180008940 shutdown_connections 2024-05-10T11:00:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.527+0000 7f11db200700 1 -- 172.21.15.27:0/1180008940 wait complete. 2024-05-10T11:00:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.528+0000 7f11db200700 1 Processor -- start 2024-05-10T11:00:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.528+0000 7f11db200700 1 -- start start 2024-05-10T11:00:08.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.529+0000 7f11db200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11dc105160 0x7f11dc1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:08.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.529+0000 7f11db200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc10f3e0 con 0x7f11dc105160 2024-05-10T11:00:08.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.529+0000 7f11d9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11dc105160 0x7f11dc1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.529+0000 7f11d9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11dc105160 0x7f11dc1a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55276/0 (socket says 172.21.15.27:55276) 2024-05-10T11:00:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.529+0000 7f11d9e00700 1 -- 172.21.15.27:0/1172771560 learned_addr learned my addr 172.21.15.27:0/1172771560 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:00:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.530+0000 7f11d9e00700 1 -- 172.21.15.27:0/1172771560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11dc1a0ba0 con 0x7f11dc105160 2024-05-10T11:00:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.530+0000 7f11d9e00700 1 --2- 172.21.15.27:0/1172771560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11dc105160 0x7f11dc1a0660 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f11cc006010 tx=0x7f11cc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.530+0000 7f11d2a00700 1 -- 172.21.15.27:0/1172771560 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11cc026070 con 0x7f11dc105160 2024-05-10T11:00:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.530+0000 7f11db200700 1 -- 172.21.15.27:0/1172771560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11dc1a0e90 con 0x7f11dc105160 2024-05-10T11:00:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.530+0000 7f11d2a00700 1 -- 172.21.15.27:0/1172771560 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11cc00bba0 con 0x7f11dc105160 2024-05-10T11:00:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.530+0000 7f11d2a00700 1 -- 172.21.15.27:0/1172771560 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11cc003910 con 0x7f11dc105160 2024-05-10T11:00:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.530+0000 7f11db200700 1 -- 172.21.15.27:0/1172771560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11dc19a7e0 con 0x7f11dc105160 2024-05-10T11:00:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.532+0000 7f11db200700 1 -- 172.21.15.27:0/1172771560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11dc106640 con 0x7f11dc105160 2024-05-10T11:00:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.532+0000 7f11d2a00700 1 -- 172.21.15.27:0/1172771560 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f11cc029070 con 0x7f11dc105160 2024-05-10T11:00:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.532+0000 7f11d2a00700 1 --2- 172.21.15.27:0/1172771560 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f11c003e510 0x7f11c00409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.532+0000 7f11d2a00700 1 -- 172.21.15.27:0/1172771560 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f11cc01e070 con 0x7f11dc105160 2024-05-10T11:00:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.533+0000 7f11d9400700 1 --2- 172.21.15.27:0/1172771560 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f11c003e510 0x7f11c00409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.533+0000 7f11d9400700 1 --2- 172.21.15.27:0/1172771560 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f11c003e510 0x7f11c00409d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11c8006fd0 tx=0x7f11c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.538+0000 7f11d2a00700 1 -- 172.21.15.27:0/1172771560 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f11cc013070 con 0x7f11dc105160 2024-05-10T11:00:08.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:08.781+0000 7f11db200700 1 -- 172.21.15.27:0/1172771560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f11dc19bd10 con 0x7f11dc105160 2024-05-10T11:00:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:08 smithi027 ceph-mon[30054]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-10T11:00:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:08 smithi027 ceph-mon[30054]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:08 smithi027 ceph-mon[30054]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:00:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:08 smithi027 ceph-mon[30054]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:00:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:08 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:08 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:08 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:08 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1172771560' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-10T11:00:09.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:09.284+0000 7f11d2a00700 1 -- 172.21.15.27:0/1172771560 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f11cc02ada0 con 0x7f11dc105160 2024-05-10T11:00:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:09.290+0000 7f11db200700 1 -- 172.21.15.27:0/1172771560 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f11c003e510 msgr2=0x7f11c00409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:09.290+0000 7f11db200700 1 --2- 172.21.15.27:0/1172771560 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f11c003e510 0x7f11c00409d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11c8006fd0 tx=0x7f11c8006e40 comp rx=0 tx=0).stop 2024-05-10T11:00:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:09.290+0000 7f11db200700 1 -- 172.21.15.27:0/1172771560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11dc105160 msgr2=0x7f11dc1a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:09.290+0000 7f11db200700 1 --2- 172.21.15.27:0/1172771560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11dc105160 0x7f11dc1a0660 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f11cc006010 tx=0x7f11cc005780 comp rx=0 tx=0).stop 2024-05-10T11:00:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:09.290+0000 7f11db200700 1 -- 172.21.15.27:0/1172771560 shutdown_connections 2024-05-10T11:00:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:09.290+0000 7f11db200700 1 --2- 172.21.15.27:0/1172771560 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f11c003e510 0x7f11c00409d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:09.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:09.290+0000 7f11db200700 1 --2- 172.21.15.27:0/1172771560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11dc105160 0x7f11dc1a0660 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:09.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:09.290+0000 7f11db200700 1 -- 172.21.15.27:0/1172771560 >> 172.21.15.27:0/1172771560 conn(0x7f11dc1009d0 msgr2=0x7f11dc106b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:09.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:09.290+0000 7f11db200700 1 -- 172.21.15.27:0/1172771560 shutdown_connections 2024-05-10T11:00:09.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:09.290+0000 7f11db200700 1 -- 172.21.15.27:0/1172771560 wait complete. 2024-05-10T11:00:09.293 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-05-10T11:00:09.946 INFO:tasks.cephadm:Adding mon.a on smithi027 2024-05-10T11:00:09.946 INFO:tasks.cephadm:Adding mon.b on smithi076 2024-05-10T11:00:09.947 INFO:tasks.cephadm:Adding mon.c on smithi189 2024-05-10T11:00:09.947 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch apply mon '3;smithi027:172.21.15.27=a;smithi076:172.21.15.76=b;smithi189:172.21.15.189=c' 2024-05-10T11:00:10.185 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:10.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:09 smithi027 ceph-mon[30054]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:10.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:09 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1172771560' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-10T11:00:10.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:09 smithi027 ceph-mon[30054]: osdmap e4: 0 total, 0 up, 0 in 2024-05-10T11:00:11.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.519+0000 7fbea31b6700 1 -- 172.21.15.189:0/1442735518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe9c102020 msgr2=0x7fbe9c102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:11.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.519+0000 7fbea31b6700 1 --2- 172.21.15.189:0/1442735518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe9c102020 0x7fbe9c102400 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009910 tx=0x7fbe90009c20 comp rx=0 tx=0).stop 2024-05-10T11:00:11.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.519+0000 7fbea31b6700 1 -- 172.21.15.189:0/1442735518 shutdown_connections 2024-05-10T11:00:11.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.519+0000 7fbea31b6700 1 --2- 172.21.15.189:0/1442735518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe9c102020 0x7fbe9c102400 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:11.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.519+0000 7fbea31b6700 1 -- 172.21.15.189:0/1442735518 >> 172.21.15.189:0/1442735518 conn(0x7fbe9c0fd8d0 msgr2=0x7fbe9c0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:11.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.519+0000 7fbea31b6700 1 -- 172.21.15.189:0/1442735518 shutdown_connections 2024-05-10T11:00:11.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.519+0000 7fbea31b6700 1 -- 172.21.15.189:0/1442735518 wait complete. 2024-05-10T11:00:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.520+0000 7fbea31b6700 1 Processor -- start 2024-05-10T11:00:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.520+0000 7fbea31b6700 1 -- start start 2024-05-10T11:00:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.520+0000 7fbea31b6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe9c102020 0x7fbe9c199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.520+0000 7fbea31b6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c10c2a0 con 0x7fbe9c102020 2024-05-10T11:00:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.521+0000 7fbea0f52700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe9c102020 0x7fbe9c199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.521+0000 7fbea0f52700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe9c102020 0x7fbe9c199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.189:37978/0 (socket says 172.21.15.189:37978) 2024-05-10T11:00:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.521+0000 7fbea0f52700 1 -- 172.21.15.189:0/2524883426 learned_addr learned my addr 172.21.15.189:0/2524883426 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-10T11:00:11.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.521+0000 7fbea0f52700 1 -- 172.21.15.189:0/2524883426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9c199c90 con 0x7fbe9c102020 2024-05-10T11:00:11.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.521+0000 7fbea0f52700 1 --2- 172.21.15.189:0/2524883426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe9c102020 0x7fbe9c199750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbe900056d0 tx=0x7fbe900057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:11.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.522+0000 7fbe99ffb700 1 -- 172.21.15.189:0/2524883426 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe90026070 con 0x7fbe9c102020 2024-05-10T11:00:11.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.522+0000 7fbea31b6700 1 -- 172.21.15.189:0/2524883426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe9c199f80 con 0x7fbe9c102020 2024-05-10T11:00:11.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.522+0000 7fbea31b6700 1 -- 172.21.15.189:0/2524883426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe9c19a440 con 0x7fbe9c102020 2024-05-10T11:00:11.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.522+0000 7fbe99ffb700 1 -- 172.21.15.189:0/2524883426 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe9000bba0 con 0x7fbe9c102020 2024-05-10T11:00:11.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.522+0000 7fbe99ffb700 1 -- 172.21.15.189:0/2524883426 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe900037f0 con 0x7fbe9c102020 2024-05-10T11:00:11.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.523+0000 7fbea31b6700 1 -- 172.21.15.189:0/2524883426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe80005330 con 0x7fbe9c102020 2024-05-10T11:00:11.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.523+0000 7fbe99ffb700 1 -- 172.21.15.189:0/2524883426 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fbe90029070 con 0x7fbe9c102020 2024-05-10T11:00:11.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.524+0000 7fbe99ffb700 1 --2- 172.21.15.189:0/2524883426 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fbe8c03e510 0x7fbe8c0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:11.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.524+0000 7fbe99ffb700 1 -- 172.21.15.189:0/2524883426 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbe9001e070 con 0x7fbe9c102020 2024-05-10T11:00:11.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.524+0000 7fbe9bfff700 1 --2- 172.21.15.189:0/2524883426 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fbe8c03e510 0x7fbe8c0409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:11.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.525+0000 7fbe9bfff700 1 --2- 172.21.15.189:0/2524883426 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fbe8c03e510 0x7fbe8c0409d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbe88006fd0 tx=0x7fbe88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:11.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.539+0000 7fbe99ffb700 1 -- 172.21.15.189:0/2524883426 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fbe90013070 con 0x7fbe9c102020 2024-05-10T11:00:11.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.752+0000 7fbea31b6700 1 -- 172.21.15.189:0/2524883426 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi076:172.21.15.76=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}) -- 0x7fbe80000c80 con 0x7fbe8c03e510 2024-05-10T11:00:11.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.768+0000 7fbe99ffb700 1 -- 172.21.15.189:0/2524883426 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fbe80000c80 con 0x7fbe8c03e510 2024-05-10T11:00:11.768 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mon update... 2024-05-10T11:00:11.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.770+0000 7fbea31b6700 1 -- 172.21.15.189:0/2524883426 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fbe8c03e510 msgr2=0x7fbe8c0409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:11.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.770+0000 7fbea31b6700 1 --2- 172.21.15.189:0/2524883426 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fbe8c03e510 0x7fbe8c0409d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbe88006fd0 tx=0x7fbe88006e40 comp rx=0 tx=0).stop 2024-05-10T11:00:11.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.770+0000 7fbea31b6700 1 -- 172.21.15.189:0/2524883426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe9c102020 msgr2=0x7fbe9c199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:11.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.770+0000 7fbea31b6700 1 --2- 172.21.15.189:0/2524883426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe9c102020 0x7fbe9c199750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbe900056d0 tx=0x7fbe900057b0 comp rx=0 tx=0).stop 2024-05-10T11:00:11.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.770+0000 7fbea31b6700 1 -- 172.21.15.189:0/2524883426 shutdown_connections 2024-05-10T11:00:11.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.770+0000 7fbea31b6700 1 --2- 172.21.15.189:0/2524883426 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fbe8c03e510 0x7fbe8c0409d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:11.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.770+0000 7fbea31b6700 1 --2- 172.21.15.189:0/2524883426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe9c102020 0x7fbe9c199750 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:11.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.770+0000 7fbea31b6700 1 -- 172.21.15.189:0/2524883426 >> 172.21.15.189:0/2524883426 conn(0x7fbe9c0fd8d0 msgr2=0x7fbe9c1039f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:11.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.770+0000 7fbea31b6700 1 -- 172.21.15.189:0/2524883426 shutdown_connections 2024-05-10T11:00:11.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:11.770+0000 7fbea31b6700 1 -- 172.21.15.189:0/2524883426 wait complete. 2024-05-10T11:00:12.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:11 smithi027 ceph-mon[30054]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:12.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:12.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:12.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:12.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:12.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:12.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:12.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:12.413 DEBUG:teuthology.orchestra.run.smithi076:mon.b> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.b.service 2024-05-10T11:00:12.416 DEBUG:teuthology.orchestra.run.smithi189:mon.c> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.c.service 2024-05-10T11:00:12.418 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-10T11:00:12.418 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-10T11:00:13.323 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:13.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:12 smithi027 ceph-mon[30054]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi076:172.21.15.76=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:13.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:12 smithi027 ceph-mon[30054]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi076:172.21.15.76=b;smithi189:172.21.15.189=c;count:3 2024-05-10T11:00:13.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:12 smithi027 ceph-mon[30054]: Deploying daemon mon.c on smithi189 2024-05-10T11:00:14.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:13 smithi027 ceph-mon[30054]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:16.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.029+0000 7f6e968de700 1 -- 172.21.15.189:0/1679809541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e90103110 msgr2=0x7f6e901034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:16.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.029+0000 7f6e968de700 1 --2- 172.21.15.189:0/1679809541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e90103110 0x7f6e901034f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009910 tx=0x7f6e80009c20 comp rx=0 tx=0).stop 2024-05-10T11:00:16.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.029+0000 7f6e968de700 1 -- 172.21.15.189:0/1679809541 shutdown_connections 2024-05-10T11:00:16.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.029+0000 7f6e968de700 1 --2- 172.21.15.189:0/1679809541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e90103110 0x7f6e901034f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:16.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.029+0000 7f6e968de700 1 -- 172.21.15.189:0/1679809541 >> 172.21.15.189:0/1679809541 conn(0x7f6e900fe9e0 msgr2=0x7f6e90100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:16.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.029+0000 7f6e968de700 1 -- 172.21.15.189:0/1679809541 shutdown_connections 2024-05-10T11:00:16.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.029+0000 7f6e968de700 1 -- 172.21.15.189:0/1679809541 wait complete. 2024-05-10T11:00:16.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.030+0000 7f6e968de700 1 Processor -- start 2024-05-10T11:00:16.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.030+0000 7f6e968de700 1 -- start start 2024-05-10T11:00:16.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.030+0000 7f6e968de700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e90103110 0x7f6e900757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:16.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.030+0000 7f6e968de700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9010d390 con 0x7f6e90103110 2024-05-10T11:00:16.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.031+0000 7f6e8ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e90103110 0x7f6e900757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:16.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.031+0000 7f6e8ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e90103110 0x7f6e900757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.189:33416/0 (socket says 172.21.15.189:33416) 2024-05-10T11:00:16.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.031+0000 7f6e8ffff700 1 -- 172.21.15.189:0/401106806 learned_addr learned my addr 172.21.15.189:0/401106806 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-10T11:00:16.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.032+0000 7f6e8ffff700 1 -- 172.21.15.189:0/401106806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e90077690 con 0x7f6e90103110 2024-05-10T11:00:16.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.032+0000 7f6e8ffff700 1 --2- 172.21.15.189:0/401106806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e90103110 0x7f6e900757a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6e800056d0 tx=0x7f6e800057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:16.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.032+0000 7f6e8d7fa700 1 -- 172.21.15.189:0/401106806 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6e80026070 con 0x7f6e90103110 2024-05-10T11:00:16.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.032+0000 7f6e968de700 1 -- 172.21.15.189:0/401106806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e90075d40 con 0x7f6e90103110 2024-05-10T11:00:16.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.032+0000 7f6e968de700 1 -- 172.21.15.189:0/401106806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e90076200 con 0x7f6e90103110 2024-05-10T11:00:16.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.032+0000 7f6e8d7fa700 1 -- 172.21.15.189:0/401106806 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e8000bba0 con 0x7f6e90103110 2024-05-10T11:00:16.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.032+0000 7f6e8d7fa700 1 -- 172.21.15.189:0/401106806 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6e800037f0 con 0x7f6e90103110 2024-05-10T11:00:16.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.033+0000 7f6e968de700 1 -- 172.21.15.189:0/401106806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e901045f0 con 0x7f6e90103110 2024-05-10T11:00:16.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.034+0000 7f6e8d7fa700 1 -- 172.21.15.189:0/401106806 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f6e80029070 con 0x7f6e90103110 2024-05-10T11:00:16.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.034+0000 7f6e8d7fa700 1 --2- 172.21.15.189:0/401106806 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6e7803e510 0x7f6e780409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:16.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.034+0000 7f6e8d7fa700 1 -- 172.21.15.189:0/401106806 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6e8001e070 con 0x7f6e90103110 2024-05-10T11:00:16.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.035+0000 7f6e8f7fe700 1 --2- 172.21.15.189:0/401106806 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6e7803e510 0x7f6e780409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:16.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.036+0000 7f6e8f7fe700 1 --2- 172.21.15.189:0/401106806 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6e7803e510 0x7f6e780409d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6e84006fd0 tx=0x7f6e84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:16.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.050+0000 7f6e8d7fa700 1 -- 172.21.15.189:0/401106806 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6e80013070 con 0x7f6e90103110 2024-05-10T11:00:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:15 smithi027 ceph-mon[30054]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:16.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.368+0000 7f6e968de700 1 -- 172.21.15.189:0/401106806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6e90076d70 con 0x7f6e90103110 2024-05-10T11:00:16.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.369+0000 7f6e8d7fa700 1 -- 172.21.15.189:0/401106806 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f6e80017440 con 0x7f6e90103110 2024-05-10T11:00:16.370 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T11:00:16.370 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","modified":"2024-05-10T10:58:06.888512Z","created":"2024-05-10T10:58:06.888512Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-10T11:00:16.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.371+0000 7f6e968de700 1 -- 172.21.15.189:0/401106806 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6e7803e510 msgr2=0x7f6e780409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:16.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.371+0000 7f6e968de700 1 --2- 172.21.15.189:0/401106806 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6e7803e510 0x7f6e780409d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6e84006fd0 tx=0x7f6e84006e40 comp rx=0 tx=0).stop 2024-05-10T11:00:16.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.371+0000 7f6e968de700 1 -- 172.21.15.189:0/401106806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e90103110 msgr2=0x7f6e900757a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:16.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.371+0000 7f6e968de700 1 --2- 172.21.15.189:0/401106806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e90103110 0x7f6e900757a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6e800056d0 tx=0x7f6e800057b0 comp rx=0 tx=0).stop 2024-05-10T11:00:16.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.372+0000 7f6e968de700 1 -- 172.21.15.189:0/401106806 shutdown_connections 2024-05-10T11:00:16.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.372+0000 7f6e968de700 1 --2- 172.21.15.189:0/401106806 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6e7803e510 0x7f6e780409d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:16.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.372+0000 7f6e968de700 1 --2- 172.21.15.189:0/401106806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e90103110 0x7f6e900757a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:16.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.372+0000 7f6e968de700 1 -- 172.21.15.189:0/401106806 >> 172.21.15.189:0/401106806 conn(0x7f6e900fe9e0 msgr2=0x7f6e90104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:16.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.372+0000 7f6e968de700 1 -- 172.21.15.189:0/401106806 shutdown_connections 2024-05-10T11:00:16.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:16.372+0000 7f6e968de700 1 -- 172.21.15.189:0/401106806 wait complete. 2024-05-10T11:00:16.373 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-05-10T11:00:16.530 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 podman[34811]: 2024-05-10 11:00:16.506514745 +0000 UTC m=+0.357397790 container start 79bc0699541be7369d64ac8b54589d9c64c089258362090d90c1d9056497555d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-c, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-10T11:00:16.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: set uid:gid to 167:167 (ceph:ceph) 2024-05-10T11:00:16.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: ceph version 19.0.0-3449-gd39fa088 (d39fa08867836ac070c47cb66c782fa559fd480b) squid (dev), process ceph-mon, pid 2 2024-05-10T11:00:16.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pidfile_write: ignore empty --pid-file 2024-05-10T11:00:16.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: load: jerasure load: lrc 2024-05-10T11:00:16.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: RocksDB version: 7.9.2 2024-05-10T11:00:16.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Git sha 0 2024-05-10T11:00:16.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Compile date 2024-05-10 01:38:55 2024-05-10T11:00:16.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: DB SUMMARY 2024-05-10T11:00:16.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: DB Session ID: MRTPI2GB591MSVVEFKJC 2024-05-10T11:00:16.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: CURRENT file: CURRENT 2024-05-10T11:00:16.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: IDENTITY file: IDENTITY 2024-05-10T11:00:16.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-10T11:00:16.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-10T11:00:16.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-10T11:00:16.991 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.error_if_exists: 0 2024-05-10T11:00:16.991 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.create_if_missing: 0 2024-05-10T11:00:16.991 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.paranoid_checks: 1 2024-05-10T11:00:16.991 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-10T11:00:16.991 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-10T11:00:16.992 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-10T11:00:16.992 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.env: 0x55d5af143cc0 2024-05-10T11:00:16.992 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.fs: PosixFileSystem 2024-05-10T11:00:16.992 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.info_log: 0x55d5b11c9da0 2024-05-10T11:00:16.992 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_file_opening_threads: 16 2024-05-10T11:00:16.992 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.statistics: (nil) 2024-05-10T11:00:16.993 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.use_fsync: 0 2024-05-10T11:00:16.993 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_log_file_size: 0 2024-05-10T11:00:16.993 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-10T11:00:16.993 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-10T11:00:16.993 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.keep_log_file_num: 1000 2024-05-10T11:00:16.993 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.recycle_log_file_num: 0 2024-05-10T11:00:16.994 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.allow_fallocate: 1 2024-05-10T11:00:16.994 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.allow_mmap_reads: 0 2024-05-10T11:00:16.994 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.allow_mmap_writes: 0 2024-05-10T11:00:16.994 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.use_direct_reads: 0 2024-05-10T11:00:16.994 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-10T11:00:16.995 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.create_missing_column_families: 0 2024-05-10T11:00:16.995 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.db_log_dir: 2024-05-10T11:00:16.995 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.wal_dir: 2024-05-10T11:00:16.995 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-10T11:00:16.995 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-10T11:00:16.995 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-10T11:00:16.995 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-10T11:00:16.995 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-10T11:00:16.995 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-10T11:00:16.995 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.advise_random_on_open: 1 2024-05-10T11:00:16.996 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.db_write_buffer_size: 0 2024-05-10T11:00:16.996 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.write_buffer_manager: 0x55d5b119b0e0 2024-05-10T11:00:16.996 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-10T11:00:16.996 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-10T11:00:16.996 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-10T11:00:16.996 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.rate_limiter: (nil) 2024-05-10T11:00:16.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-10T11:00:16.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.wal_recovery_mode: 2 2024-05-10T11:00:16.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.enable_thread_tracking: 0 2024-05-10T11:00:16.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.enable_pipelined_write: 0 2024-05-10T11:00:16.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.unordered_write: 0 2024-05-10T11:00:16.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-10T11:00:16.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-10T11:00:16.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-10T11:00:16.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-10T11:00:16.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.row_cache: None 2024-05-10T11:00:16.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.wal_filter: None 2024-05-10T11:00:16.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-10T11:00:16.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.allow_ingest_behind: 0 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.two_write_queues: 0 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.manual_wal_flush: 0 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.wal_compression: 0 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.atomic_flush: 0 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.log_readahead_size: 0 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.best_efforts_recovery: 0 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.allow_data_in_errors: 0 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.db_host_id: __hostname__ 2024-05-10T11:00:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_background_jobs: 2 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_background_compactions: -1 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_subcompactions: 1 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_total_wal_size: 0 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_open_files: -1 2024-05-10T11:00:17.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bytes_per_sync: 0 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_readahead_size: 0 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_background_flushes: -1 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Compression algorithms supported: 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: kZSTD supported: 0 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: kXpressCompression supported: 0 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: kLZ4HCCompression supported: 1 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: kZlibCompression supported: 1 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: kSnappyCompression supported: 1 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: kLZ4Compression supported: 1 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: kBZip2Compression supported: 0 2024-05-10T11:00:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.merge_operator: 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_filter: None 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_filter_factory: None 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.sst_partitioner_factory: None 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d5b11c9f60) 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout: cache_index_and_filter_blocks: 1 2024-05-10T11:00:17.002 INFO:journalctl@ceph.mon.c.smithi189.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: pin_top_level_index_and_filter: 1 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: index_type: 0 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: data_block_index_type: 0 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: index_shortening: 1 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: checksum: 4 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: no_block_cache: 0 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache: 0x55d5b11cd1f0 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache_name: BinnedLRUCache 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache_options: 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: capacity : 536870912 2024-05-10T11:00:17.003 INFO:journalctl@ceph.mon.c.smithi189.stdout: num_shard_bits : 4 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: strict_capacity_limit : 0 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: high_pri_pool_ratio: 0.000 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache_compressed: (nil) 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: persistent_cache: (nil) 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_size: 4096 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_size_deviation: 10 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_restart_interval: 16 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: index_block_restart_interval: 1 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: metadata_block_size: 4096 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: partition_filters: 0 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: use_delta_encoding: 1 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: filter_policy: bloomfilter 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: whole_key_filtering: 1 2024-05-10T11:00:17.004 INFO:journalctl@ceph.mon.c.smithi189.stdout: verify_compression: 0 2024-05-10T11:00:17.005 INFO:journalctl@ceph.mon.c.smithi189.stdout: read_amp_bytes_per_bit: 0 2024-05-10T11:00:17.005 INFO:journalctl@ceph.mon.c.smithi189.stdout: format_version: 5 2024-05-10T11:00:17.005 INFO:journalctl@ceph.mon.c.smithi189.stdout: enable_index_compression: 1 2024-05-10T11:00:17.005 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_align: 0 2024-05-10T11:00:17.005 INFO:journalctl@ceph.mon.c.smithi189.stdout: max_auto_readahead_size: 262144 2024-05-10T11:00:17.005 INFO:journalctl@ceph.mon.c.smithi189.stdout: prepopulate_block_cache: 0 2024-05-10T11:00:17.005 INFO:journalctl@ceph.mon.c.smithi189.stdout: initial_auto_readahead_size: 8192 2024-05-10T11:00:17.005 INFO:journalctl@ceph.mon.c.smithi189.stdout: num_file_reads_for_auto_readahead: 2 2024-05-10T11:00:17.005 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.write_buffer_size: 33554432 2024-05-10T11:00:17.005 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_write_buffer_number: 2 2024-05-10T11:00:17.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compression: NoCompression 2024-05-10T11:00:17.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bottommost_compression: Disabled 2024-05-10T11:00:17.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.prefix_extractor: nullptr 2024-05-10T11:00:17.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-10T11:00:17.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.num_levels: 7 2024-05-10T11:00:17.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-10T11:00:17.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-10T11:00:17.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-10T11:00:17.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-10T11:00:17.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-10T11:00:17.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-10T11:00:17.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-10T11:00:17.007 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-10T11:00:17.008 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-10T11:00:17.008 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-10T11:00:17.008 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-10T11:00:17.008 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-10T11:00:17.008 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-10T11:00:17.008 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compression_opts.level: 32767 2024-05-10T11:00:17.009 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compression_opts.strategy: 0 2024-05-10T11:00:17.009 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-10T11:00:17.009 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-10T11:00:17.009 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-10T11:00:17.009 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-10T11:00:17.009 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compression_opts.enabled: false 2024-05-10T11:00:17.009 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-10T11:00:17.010 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-10T11:00:17.011 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-10T11:00:17.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-10T11:00:17.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.target_file_size_base: 67108864 2024-05-10T11:00:17.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-10T11:00:17.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-10T11:00:17.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-10T11:00:17.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-10T11:00:17.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-10T11:00:17.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-10T11:00:17.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-10T11:00:17.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-10T11:00:17.013 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-10T11:00:17.013 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-10T11:00:17.013 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-10T11:00:17.013 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-10T11:00:17.013 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-10T11:00:17.013 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-10T11:00:17.013 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.arena_block_size: 1048576 2024-05-10T11:00:17.013 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-10T11:00:17.013 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-10T11:00:17.013 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.disable_auto_compactions: 0 2024-05-10T11:00:17.013 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-10T11:00:17.013 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.inplace_update_support: 0 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-10T11:00:17.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-10T11:00:17.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.bloom_locality: 0 2024-05-10T11:00:17.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.max_successive_merges: 0 2024-05-10T11:00:17.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-10T11:00:17.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.paranoid_file_checks: 0 2024-05-10T11:00:17.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.force_consistency_checks: 1 2024-05-10T11:00:17.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.report_bg_io_stats: 0 2024-05-10T11:00:17.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.ttl: 2592000 2024-05-10T11:00:17.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-10T11:00:17.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-10T11:00:17.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-10T11:00:17.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.enable_blob_files: false 2024-05-10T11:00:17.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.min_blob_size: 0 2024-05-10T11:00:17.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.blob_file_size: 268435456 2024-05-10T11:00:17.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-10T11:00:17.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-10T11:00:17.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-10T11:00:17.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-10T11:00:17.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-10T11:00:17.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.blob_file_starting_level: 0 2024-05-10T11:00:17.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-10T11:00:17.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-10T11:00:17.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3c86fbd2-861a-4aec-bd49-05b5921eb030 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715338816536936, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715338816537503, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715338816, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3c86fbd2-861a-4aec-bd49-05b5921eb030", "db_session_id": "MRTPI2GB591MSVVEFKJC", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715338816537607, "job": 1, "event": "recovery_finished"} 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d5b12ea000 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: DB pointer 0x55d5b12d6000 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** DB Stats ** 2024-05-10T11:00:17.017 INFO:journalctl@ceph.mon.c.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** Compaction Stats [default] ** 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-10T11:00:17.018 INFO:journalctl@ceph.mon.c.smithi189.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** Compaction Stats [default] ** 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-10T11:00:17.019 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-10T11:00:17.020 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-10T11:00:17.020 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-10T11:00:17.020 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-10T11:00:17.020 INFO:journalctl@ceph.mon.c.smithi189.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-10T11:00:17.020 INFO:journalctl@ceph.mon.c.smithi189.stdout: Block cache BinnedLRUCache@0x55d5b11cd1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-10T11:00:17.020 INFO:journalctl@ceph.mon.c.smithi189.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-10T11:00:17.020 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-05-10T11:00:17.020 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-10T11:00:17.021 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-10T11:00:17.021 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: using public_addr v2:172.21.15.189:0/0 -> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-05-10T11:00:17.021 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: starting mon.c rank -1 at public addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] at bind addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:17.021 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(???) e0 preinit fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:17.021 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).mds e1 new map 2024-05-10T11:00:17.021 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-10T11:00:17.021 INFO:journalctl@ceph.mon.c.smithi189.stdout: e1 2024-05-10T11:00:17.021 INFO:journalctl@ceph.mon.c.smithi189.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-10T11:00:17.021 INFO:journalctl@ceph.mon.c.smithi189.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-10T11:00:17.021 INFO:journalctl@ceph.mon.c.smithi189.stdout: legacy client fscid: -1 2024-05-10T11:00:17.021 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-05-10T11:00:17.021 INFO:journalctl@ceph.mon.c.smithi189.stdout: No filesystems configured 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi076", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Deploying cephadm binary to smithi076 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Added host smithi076 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.024 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.024 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.024 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:00:17.024 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:17.024 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:17.024 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Updating smithi076:/etc/ceph/ceph.conf 2024-05-10T11:00:17.024 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:17.024 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:00:17.024 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:00:17.024 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.025 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.025 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.025 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:17.025 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.025 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Deploying cephadm binary to smithi189 2024-05-10T11:00:17.025 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.025 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.025 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.025 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:17.025 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.025 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Added host smithi189 2024-05-10T11:00:17.025 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:17.026 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1172771560' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1172771560' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: osdmap e4: 0 total, 0 up, 0 in 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:17.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:17.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:17.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:17.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:17.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:17.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi076:172.21.15.76=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:17.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi076:172.21.15.76=b;smithi189:172.21.15.189=c;count:3 2024-05-10T11:00:17.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: Deploying daemon mon.c on smithi189 2024-05-10T11:00:17.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:17.029 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 ceph-mon[34842]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-10T11:00:17.029 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 bash[34811]: 79bc0699541be7369d64ac8b54589d9c64c089258362090d90c1d9056497555d 2024-05-10T11:00:17.029 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:16 smithi189 systemd[1]: Started Ceph mon.c for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:00:18.167 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-10T11:00:18.168 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-10T11:00:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:00:21.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:00:21.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: mon.a calling monitor election 2024-05-10T11:00:21.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:21.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:21.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:21.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: mon.c calling monitor election 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: monmap epoch 2 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: last_changed 2024-05-10T11:00:16.574214+0000 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: created 2024-05-10T10:58:06.888512+0000 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: min_mon_release 19 (squid) 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: election_strategy: 1 2024-05-10T11:00:21.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-05-10T11:00:21.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-10T11:00:21.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: fsmap 2024-05-10T11:00:21.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: osdmap e4: 0 total, 0 up, 0 in 2024-05-10T11:00:21.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: mgrmap e13: x(active, since 60s) 2024-05-10T11:00:21.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: overall HEALTH_OK 2024-05-10T11:00:21.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:21.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:21.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:21.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:21.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:22.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:00:22.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: mon.a calling monitor election 2024-05-10T11:00:22.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:22.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:22.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:22.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:22.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: mon.c calling monitor election 2024-05-10T11:00:22.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:22.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:22.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:22.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:22.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: monmap epoch 2 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: last_changed 2024-05-10T11:00:16.574214+0000 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: created 2024-05-10T10:58:06.888512+0000 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: min_mon_release 19 (squid) 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: election_strategy: 1 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: fsmap 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: osdmap e4: 0 total, 0 up, 0 in 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: mgrmap e13: x(active, since 60s) 2024-05-10T11:00:22.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: overall HEALTH_OK 2024-05-10T11:00:22.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:22.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:22.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:22.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:22.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:22.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:22 smithi189 ceph-mon[34842]: Deploying daemon mon.b on smithi076 2024-05-10T11:00:22.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:22.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.924+0000 7f2a27d0c700 1 -- 172.21.15.189:0/2766901072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a0c0035c0 msgr2=0x7f2a0c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.924+0000 7f2a27d0c700 1 --2- 172.21.15.189:0/2766901072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a0c0035c0 0x7f2a0c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2a20102ef0 tx=0x7f2a1000feb0 comp rx=0 tx=0).stop 2024-05-10T11:00:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.925+0000 7f2a27d0c700 1 -- 172.21.15.189:0/2766901072 shutdown_connections 2024-05-10T11:00:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.925+0000 7f2a27d0c700 1 --2- 172.21.15.189:0/2766901072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a0c0035c0 0x7f2a0c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.925+0000 7f2a27d0c700 1 -- 172.21.15.189:0/2766901072 >> 172.21.15.189:0/2766901072 conn(0x7f2a200feab0 msgr2=0x7f2a20100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.925+0000 7f2a27d0c700 1 -- 172.21.15.189:0/2766901072 shutdown_connections 2024-05-10T11:00:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.925+0000 7f2a27d0c700 1 -- 172.21.15.189:0/2766901072 wait complete. 2024-05-10T11:00:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.925+0000 7f2a27d0c700 1 Processor -- start 2024-05-10T11:00:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.925+0000 7f2a27d0c700 1 -- start start 2024-05-10T11:00:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.926+0000 7f2a27d0c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c0035c0 0x7f2a201a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.926+0000 7f2a27d0c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a201a0de0 0x7f2a2019aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.926+0000 7f2a27d0c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2010af10 con 0x7f2a201a0de0 2024-05-10T11:00:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.926+0000 7f2a27d0c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2010ad90 con 0x7f2a0c0035c0 2024-05-10T11:00:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.926+0000 7f2a25aa8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c0035c0 0x7f2a201a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.926+0000 7f2a25aa8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c0035c0 0x7f2a201a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57466/0 (socket says 172.21.15.189:57466) 2024-05-10T11:00:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.926+0000 7f2a25aa8700 1 -- 172.21.15.189:0/1505612762 learned_addr learned my addr 172.21.15.189:0/1505612762 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-10T11:00:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.926+0000 7f2a25aa8700 1 -- 172.21.15.189:0/1505612762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c0035c0 msgr2=0x7f2a201a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-10T11:00:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.926+0000 7f2a25aa8700 1 -- 172.21.15.189:0/1505612762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c0035c0 msgr2=0x7f2a201a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-10T11:00:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.926+0000 7f2a25aa8700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c0035c0 0x7f2a201a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-10T11:00:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.926+0000 7f2a25aa8700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c0035c0 0x7f2a201a08a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-10T11:00:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.926+0000 7f2a252a7700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a201a0de0 0x7f2a2019aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:22.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.927+0000 7f2a252a7700 1 -- 172.21.15.189:0/1505612762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c0035c0 msgr2=0x7f2a201a08a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T11:00:22.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.927+0000 7f2a252a7700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c0035c0 0x7f2a201a08a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:22.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.927+0000 7f2a252a7700 1 -- 172.21.15.189:0/1505612762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a201a1420 con 0x7f2a201a0de0 2024-05-10T11:00:22.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.927+0000 7f2a252a7700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a201a0de0 0x7f2a2019aab0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00b4d0 tx=0x7f2a1c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:22.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.927+0000 7f2a16ffd700 1 -- 172.21.15.189:0/1505612762 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2a1c011630 con 0x7f2a201a0de0 2024-05-10T11:00:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.928+0000 7f2a27d0c700 1 -- 172.21.15.189:0/1505612762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a2019aff0 con 0x7f2a201a0de0 2024-05-10T11:00:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.928+0000 7f2a27d0c700 1 -- 172.21.15.189:0/1505612762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a2019b580 con 0x7f2a201a0de0 2024-05-10T11:00:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.928+0000 7f2a16ffd700 1 -- 172.21.15.189:0/1505612762 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a1c011c80 con 0x7f2a201a0de0 2024-05-10T11:00:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.928+0000 7f2a16ffd700 1 -- 172.21.15.189:0/1505612762 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2a1c01adc0 con 0x7f2a201a0de0 2024-05-10T11:00:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.929+0000 7f2a27d0c700 1 -- 172.21.15.189:0/1505612762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a20104700 con 0x7f2a201a0de0 2024-05-10T11:00:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.929+0000 7f2a16ffd700 1 -- 172.21.15.189:0/1505612762 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f2a1c018020 con 0x7f2a201a0de0 2024-05-10T11:00:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.929+0000 7f2a16ffd700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2a0403e2d0 0x7f2a04040790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.929+0000 7f2a16ffd700 1 -- 172.21.15.189:0/1505612762 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2a1c052340 con 0x7f2a201a0de0 2024-05-10T11:00:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.930+0000 7f2a25aa8700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2a0403e2d0 0x7f2a04040790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:22.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.931+0000 7f2a25aa8700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2a0403e2d0 0x7f2a04040790 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2a10001eb0 tx=0x7f2a10005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:22.944+0000 7f2a16ffd700 1 -- 172.21.15.189:0/1505612762 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f2a1c01f020 con 0x7f2a201a0de0 2024-05-10T11:00:23.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:22 smithi027 ceph-mon[30054]: Deploying daemon mon.b on smithi076 2024-05-10T11:00:23.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:23.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.229+0000 7f2a27d0c700 1 -- 172.21.15.189:0/1505612762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f2a2019c030 con 0x7f2a201a0de0 2024-05-10T11:00:23.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.230+0000 7f2a16ffd700 1 -- 172.21.15.189:0/1505612762 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f2a1c022350 con 0x7f2a201a0de0 2024-05-10T11:00:23.230 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T11:00:23.230 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","modified":"2024-05-10T11:00:16.574214Z","created":"2024-05-10T10:58:06.888512Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-10T11:00:23.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.232+0000 7f2a27d0c700 1 -- 172.21.15.189:0/1505612762 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2a0403e2d0 msgr2=0x7f2a04040790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:23.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.232+0000 7f2a27d0c700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2a0403e2d0 0x7f2a04040790 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2a10001eb0 tx=0x7f2a10005960 comp rx=0 tx=0).stop 2024-05-10T11:00:23.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.232+0000 7f2a27d0c700 1 -- 172.21.15.189:0/1505612762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a201a0de0 msgr2=0x7f2a2019aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:23.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.232+0000 7f2a27d0c700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a201a0de0 0x7f2a2019aab0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00b4d0 tx=0x7f2a1c00b890 comp rx=0 tx=0).stop 2024-05-10T11:00:23.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.232+0000 7f2a27d0c700 1 -- 172.21.15.189:0/1505612762 shutdown_connections 2024-05-10T11:00:23.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.232+0000 7f2a27d0c700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2a0403e2d0 0x7f2a04040790 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:23.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.232+0000 7f2a27d0c700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c0035c0 0x7f2a201a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:23.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.232+0000 7f2a27d0c700 1 --2- 172.21.15.189:0/1505612762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a201a0de0 0x7f2a2019aab0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:23.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.232+0000 7f2a27d0c700 1 -- 172.21.15.189:0/1505612762 >> 172.21.15.189:0/1505612762 conn(0x7f2a200feab0 msgr2=0x7f2a20104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:23.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.232+0000 7f2a27d0c700 1 -- 172.21.15.189:0/1505612762 shutdown_connections 2024-05-10T11:00:23.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:23.232+0000 7f2a27d0c700 1 -- 172.21.15.189:0/1505612762 wait complete. 2024-05-10T11:00:23.233 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 2 2024-05-10T11:00:23.793 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:23 smithi189 ceph-mon[34842]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:23.793 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:23 smithi189 ceph-mon[34842]: from='client.? 172.21.15.189:0/1505612762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-10T11:00:23.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:23 smithi027 ceph-mon[30054]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:23.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:23 smithi027 ceph-mon[30054]: from='client.? 172.21.15.189:0/1505612762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-10T11:00:24.794 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-10T11:00:24.795 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-10T11:00:25.038 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:00:25.622 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-10T11:00:25.622 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-10T11:00:25.622 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-10T11:00:25.622 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-10T11:00:25.622 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi076", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Deploying cephadm binary to smithi076 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Added host smithi076 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.623 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:00:25.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Updating smithi076:/etc/ceph/ceph.conf 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Deploying cephadm binary to smithi189 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Added host smithi189 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1172771560' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1172771560' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: osdmap e4: 0 total, 0 up, 0 in 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi076:172.21.15.76=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi076:172.21.15.76=b;smithi189:172.21.15.189=c;count:3 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Deploying daemon mon.c on smithi189 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:00:25.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.a calling monitor election 2024-05-10T11:00:25.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:25.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:25.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:25.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.c calling monitor election 2024-05-10T11:00:25.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: monmap epoch 2 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: last_changed 2024-05-10T11:00:16.574214+0000 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: created 2024-05-10T10:58:06.888512+0000 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: min_mon_release 19 (squid) 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: election_strategy: 1 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: fsmap 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: osdmap e4: 0 total, 0 up, 0 in 2024-05-10T11:00:25.932 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mgrmap e13: x(active, since 60s) 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: overall HEALTH_OK 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: Deploying daemon mon.b on smithi076 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: from='client.? 172.21.15.189:0/1505612762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 ceph-mon[34152]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 bash[34131]: c141b324c977203a29f0ede9ee846bb6232dde9b390e6b312a83b50a0538c13e 2024-05-10T11:00:25.933 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:25 smithi076 systemd[1]: Started Ceph mon.b for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:00:30.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.888+0000 7efdf17a1700 1 -- 172.21.15.189:0/3245976553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd40035c0 msgr2=0x7efdd4005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:30.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.888+0000 7efdf17a1700 1 --2- 172.21.15.189:0/3245976553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd40035c0 0x7efdd4005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efdec06b6f0 tx=0x7efddc00fe60 comp rx=0 tx=0).stop 2024-05-10T11:00:30.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.888+0000 7efdf17a1700 1 -- 172.21.15.189:0/3245976553 shutdown_connections 2024-05-10T11:00:30.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.888+0000 7efdf17a1700 1 --2- 172.21.15.189:0/3245976553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd40035c0 0x7efdd4005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:30.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.888+0000 7efdf17a1700 1 -- 172.21.15.189:0/3245976553 >> 172.21.15.189:0/3245976553 conn(0x7efdec0fd9f0 msgr2=0x7efdec0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:30.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.888+0000 7efdf17a1700 1 -- 172.21.15.189:0/3245976553 shutdown_connections 2024-05-10T11:00:30.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.888+0000 7efdf17a1700 1 -- 172.21.15.189:0/3245976553 wait complete. 2024-05-10T11:00:30.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.889+0000 7efdf17a1700 1 Processor -- start 2024-05-10T11:00:30.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.889+0000 7efdf17a1700 1 -- start start 2024-05-10T11:00:30.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdf17a1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdd40035c0 0x7efdec19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:30.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdf17a1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdec19eda0 0x7efdec198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:30.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdf17a1700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efdec198f60 0x7efdec199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:30.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdf17a1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efdec10aee0 con 0x7efdec19eda0 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdf17a1700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efdec10ad60 con 0x7efdec198f60 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdf17a1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efdec10b060 con 0x7efdd40035c0 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdeaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdd40035c0 0x7efdec19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdeaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdd40035c0 0x7efdec19e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57478/0 (socket says 172.21.15.189:57478) 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdeaffd700 1 -- 172.21.15.189:0/310670147 learned_addr learned my addr 172.21.15.189:0/310670147 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdeaffd700 1 -- 172.21.15.189:0/310670147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdd40035c0 msgr2=0x7efdec19e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdeaffd700 1 -- 172.21.15.189:0/310670147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdd40035c0 msgr2=0x7efdec19e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdeb7fe700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efdec198f60 0x7efdec199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdeaffd700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdd40035c0 0x7efdec19e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdeaffd700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdd40035c0 0x7efdec19e860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdeb7fe700 1 -- 172.21.15.189:0/310670147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efdec198f60 msgr2=0x7efdec199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdeb7fe700 1 -- 172.21.15.189:0/310670147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efdec198f60 msgr2=0x7efdec199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdeb7fe700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efdec198f60 0x7efdec199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-10T11:00:30.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdeb7fe700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efdec198f60 0x7efdec199410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-10T11:00:30.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.890+0000 7efdea7fc700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdec19eda0 0x7efdec198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:30.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.891+0000 7efdea7fc700 1 -- 172.21.15.189:0/310670147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdd40035c0 msgr2=0x7efdec19e860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T11:00:30.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.891+0000 7efdea7fc700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdd40035c0 0x7efdec19e860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:30.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.891+0000 7efdea7fc700 1 -- 172.21.15.189:0/310670147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efdec198f60 msgr2=0x7efdec199410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T11:00:30.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.891+0000 7efdea7fc700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efdec198f60 0x7efdec199410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:30.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.891+0000 7efdea7fc700 1 -- 172.21.15.189:0/310670147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdec1a5730 con 0x7efdec19eda0 2024-05-10T11:00:30.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.892+0000 7efdea7fc700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdec19eda0 0x7efdec198a20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efde000b550 tx=0x7efde000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:30.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.892+0000 7efdd3fff700 1 -- 172.21.15.189:0/310670147 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde0004230 con 0x7efdec19eda0 2024-05-10T11:00:30.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.892+0000 7efdf17a1700 1 -- 172.21.15.189:0/310670147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdec1a5960 con 0x7efdec19eda0 2024-05-10T11:00:30.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.892+0000 7efdf17a1700 1 -- 172.21.15.189:0/310670147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdec1a5ef0 con 0x7efdec19eda0 2024-05-10T11:00:30.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.892+0000 7efdd3fff700 1 -- 172.21.15.189:0/310670147 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efde00043d0 con 0x7efdec19eda0 2024-05-10T11:00:30.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.892+0000 7efdd3fff700 1 -- 172.21.15.189:0/310670147 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde0011a70 con 0x7efdec19eda0 2024-05-10T11:00:30.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.893+0000 7efdf17a1700 1 -- 172.21.15.189:0/310670147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdd4002b60 con 0x7efdec19eda0 2024-05-10T11:00:30.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.894+0000 7efdd3fff700 1 -- 172.21.15.189:0/310670147 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7efde002d7b0 con 0x7efdec19eda0 2024-05-10T11:00:30.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.894+0000 7efdd3fff700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7efdd803e9a0 0x7efdd8040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:30.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.894+0000 7efdd3fff700 1 -- 172.21.15.189:0/310670147 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efde00523a0 con 0x7efdec19eda0 2024-05-10T11:00:30.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.895+0000 7efdeaffd700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7efdd803e9a0 0x7efdd8040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:30.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.896+0000 7efdeaffd700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7efdd803e9a0 0x7efdd8040e60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efddc00fa10 tx=0x7efddc0038d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:30.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:30.911+0000 7efdd3fff700 1 -- 172.21.15.189:0/310670147 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7efde001f070 con 0x7efdec19eda0 2024-05-10T11:00:30.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:00:30.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: mon.c calling monitor election 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: mon.a calling monitor election 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: monmap epoch 3 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:30.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: last_changed 2024-05-10T11:00:25.628028+0000 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: created 2024-05-10T10:58:06.888512+0000 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: min_mon_release 19 (squid) 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: election_strategy: 1 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: 2: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: fsmap 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: osdmap e4: 0 total, 0 up, 0 in 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: mgrmap e13: x(active, since 70s) 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: overall HEALTH_OK 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:30.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:00:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: mon.c calling monitor election 2024-05-10T11:00:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: mon.a calling monitor election 2024-05-10T11:00:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: monmap epoch 3 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: last_changed 2024-05-10T11:00:25.628028+0000 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: created 2024-05-10T10:58:06.888512+0000 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: min_mon_release 19 (squid) 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: election_strategy: 1 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: 2: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: fsmap 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: osdmap e4: 0 total, 0 up, 0 in 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: mgrmap e13: x(active, since 70s) 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: overall HEALTH_OK 2024-05-10T11:00:31.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:31.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:31.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:31.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:31.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.212+0000 7efdf17a1700 1 -- 172.21.15.189:0/310670147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7efdd4002e50 con 0x7efdec19eda0 2024-05-10T11:00:31.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.213+0000 7efdd3fff700 1 -- 172.21.15.189:0/310670147 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7efde000f540 con 0x7efdec19eda0 2024-05-10T11:00:31.213 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-10T11:00:31.213 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":3,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","modified":"2024-05-10T11:00:25.628028Z","created":"2024-05-10T10:58:06.888512Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-10T11:00:31.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.215+0000 7efdf17a1700 1 -- 172.21.15.189:0/310670147 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7efdd803e9a0 msgr2=0x7efdd8040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:31.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.215+0000 7efdf17a1700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7efdd803e9a0 0x7efdd8040e60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efddc00fa10 tx=0x7efddc0038d0 comp rx=0 tx=0).stop 2024-05-10T11:00:31.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.215+0000 7efdf17a1700 1 -- 172.21.15.189:0/310670147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdec19eda0 msgr2=0x7efdec198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:31.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.215+0000 7efdf17a1700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdec19eda0 0x7efdec198a20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efde000b550 tx=0x7efde000b910 comp rx=0 tx=0).stop 2024-05-10T11:00:31.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.215+0000 7efdf17a1700 1 -- 172.21.15.189:0/310670147 shutdown_connections 2024-05-10T11:00:31.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.215+0000 7efdf17a1700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7efdd803e9a0 0x7efdd8040e60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:31.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.215+0000 7efdf17a1700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdd40035c0 0x7efdec19e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:31.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.215+0000 7efdf17a1700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efdec198f60 0x7efdec199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:31.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.215+0000 7efdf17a1700 1 --2- 172.21.15.189:0/310670147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdec19eda0 0x7efdec198a20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:31.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.215+0000 7efdf17a1700 1 -- 172.21.15.189:0/310670147 >> 172.21.15.189:0/310670147 conn(0x7efdec0fd9f0 msgr2=0x7efdec105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:31.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.216+0000 7efdf17a1700 1 -- 172.21.15.189:0/310670147 shutdown_connections 2024-05-10T11:00:31.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:31.216+0000 7efdf17a1700 1 -- 172.21.15.189:0/310670147 wait complete. 2024-05-10T11:00:31.217 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 3 2024-05-10T11:00:31.793 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:31 smithi189 ceph-mon[34842]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:31.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:31.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:31.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:31 smithi189 ceph-mon[34842]: from='client.? 172.21.15.189:0/310670147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-10T11:00:31.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:31.819 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-10T11:00:31.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph config generate-minimal-conf 2024-05-10T11:00:32.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:31 smithi027 ceph-mon[30054]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:31 smithi027 ceph-mon[30054]: from='client.? 172.21.15.189:0/310670147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-10T11:00:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:32.270 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:00:32.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:00:32.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: mon.c calling monitor election 2024-05-10T11:00:32.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: mon.a calling monitor election 2024-05-10T11:00:32.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:32.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:00:32.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:32.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: monmap epoch 3 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: last_changed 2024-05-10T11:00:25.628028+0000 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: created 2024-05-10T10:58:06.888512+0000 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: min_mon_release 19 (squid) 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: election_strategy: 1 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-05-10T11:00:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: 2: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: fsmap 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: osdmap e4: 0 total, 0 up, 0 in 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: mgrmap e13: x(active, since 70s) 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: overall HEALTH_OK 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='client.? 172.21.15.189:0/310670147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-10T11:00:32.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:33.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: mon.b calling monitor election 2024-05-10T11:00:33.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: Updating smithi027:/etc/ceph/ceph.conf 2024-05-10T11:00:33.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: Updating smithi076:/etc/ceph/ceph.conf 2024-05-10T11:00:33.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-10T11:00:33.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: mon.b calling monitor election 2024-05-10T11:00:33.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: mon.c calling monitor election 2024-05-10T11:00:33.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: mon.a calling monitor election 2024-05-10T11:00:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-10T11:00:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: monmap epoch 3 2024-05-10T11:00:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: last_changed 2024-05-10T11:00:25.628028+0000 2024-05-10T11:00:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: created 2024-05-10T10:58:06.888512+0000 2024-05-10T11:00:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: min_mon_release 19 (squid) 2024-05-10T11:00:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: election_strategy: 1 2024-05-10T11:00:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-05-10T11:00:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-10T11:00:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: 2: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2024-05-10T11:00:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: fsmap 2024-05-10T11:00:33.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: osdmap e4: 0 total, 0 up, 0 in 2024-05-10T11:00:33.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: mgrmap e13: x(active, since 72s) 2024-05-10T11:00:33.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: overall HEALTH_OK 2024-05-10T11:00:33.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:33.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:33.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:33.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:33.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:33.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: mon.b calling monitor election 2024-05-10T11:00:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: Updating smithi027:/etc/ceph/ceph.conf 2024-05-10T11:00:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: Updating smithi076:/etc/ceph/ceph.conf 2024-05-10T11:00:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-10T11:00:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: mon.b calling monitor election 2024-05-10T11:00:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: mon.c calling monitor election 2024-05-10T11:00:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: mon.a calling monitor election 2024-05-10T11:00:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: monmap epoch 3 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: last_changed 2024-05-10T11:00:25.628028+0000 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: created 2024-05-10T10:58:06.888512+0000 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: min_mon_release 19 (squid) 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: election_strategy: 1 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: 2: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: fsmap 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: osdmap e4: 0 total, 0 up, 0 in 2024-05-10T11:00:33.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: mgrmap e13: x(active, since 72s) 2024-05-10T11:00:33.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: overall HEALTH_OK 2024-05-10T11:00:33.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:33.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:33.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:33.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:33.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:33.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:33.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: mon.b calling monitor election 2024-05-10T11:00:33.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: Updating smithi027:/etc/ceph/ceph.conf 2024-05-10T11:00:33.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: Updating smithi076:/etc/ceph/ceph.conf 2024-05-10T11:00:33.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-10T11:00:33.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: mon.b calling monitor election 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: mon.c calling monitor election 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: mon.a calling monitor election 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: monmap epoch 3 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: last_changed 2024-05-10T11:00:25.628028+0000 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: created 2024-05-10T10:58:06.888512+0000 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: min_mon_release 19 (squid) 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: election_strategy: 1 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-10T11:00:33.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: 2: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2024-05-10T11:00:33.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: fsmap 2024-05-10T11:00:33.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: osdmap e4: 0 total, 0 up, 0 in 2024-05-10T11:00:33.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: mgrmap e13: x(active, since 72s) 2024-05-10T11:00:33.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: overall HEALTH_OK 2024-05-10T11:00:33.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:33.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:33.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:33.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:33.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:33.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.174+0000 7f955ac00700 1 -- 172.21.15.27:0/47976487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554108f40 msgr2=0x7f9554109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:34.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.174+0000 7f955ac00700 1 --2- 172.21.15.27:0/47976487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554108f40 0x7f9554109320 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9544009910 tx=0x7f9544009c20 comp rx=0 tx=0).stop 2024-05-10T11:00:34.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.174+0000 7f955ac00700 1 -- 172.21.15.27:0/47976487 shutdown_connections 2024-05-10T11:00:34.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.174+0000 7f955ac00700 1 --2- 172.21.15.27:0/47976487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554108f40 0x7f9554109320 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:34.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.174+0000 7f955ac00700 1 -- 172.21.15.27:0/47976487 >> 172.21.15.27:0/47976487 conn(0x7f9554077900 msgr2=0x7f9554077d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:34.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.174+0000 7f955ac00700 1 -- 172.21.15.27:0/47976487 shutdown_connections 2024-05-10T11:00:34.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.174+0000 7f955ac00700 1 -- 172.21.15.27:0/47976487 wait complete. 2024-05-10T11:00:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.175+0000 7f955ac00700 1 Processor -- start 2024-05-10T11:00:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.175+0000 7f955ac00700 1 -- start start 2024-05-10T11:00:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.176+0000 7f955ac00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9554108f40 0x7f955419ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.176+0000 7f955ac00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f955419b270 0x7f955419f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.176+0000 7f955ac00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f955419fb40 0x7f955419fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.176+0000 7f955ac00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f955410d380 con 0x7f955419fb40 2024-05-10T11:00:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.176+0000 7f955ac00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f955410d200 con 0x7f9554108f40 2024-05-10T11:00:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.176+0000 7f955ac00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f955410d500 con 0x7f955419b270 2024-05-10T11:00:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.176+0000 7f9553e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9554108f40 0x7f955419ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.176+0000 7f9558c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f955419fb40 0x7f955419fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:34.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.176+0000 7f9553e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9554108f40 0x7f955419ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60928/0 (socket says 172.21.15.27:60928) 2024-05-10T11:00:34.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.176+0000 7f9553e00700 1 -- 172.21.15.27:0/3019813366 learned_addr learned my addr 172.21.15.27:0/3019813366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:00:34.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.176+0000 7f9553400700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f955419b270 0x7f955419f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.177+0000 7f9558c00700 1 -- 172.21.15.27:0/3019813366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f955419b270 msgr2=0x7f955419f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.177+0000 7f9558c00700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f955419b270 0x7f955419f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.177+0000 7f9558c00700 1 -- 172.21.15.27:0/3019813366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9554108f40 msgr2=0x7f955419ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.177+0000 7f9558c00700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9554108f40 0x7f955419ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.177+0000 7f9558c00700 1 -- 172.21.15.27:0/3019813366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95541a78b0 con 0x7f955419fb40 2024-05-10T11:00:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.177+0000 7f9558c00700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f955419fb40 0x7f955419fff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f954c00bd50 tx=0x7f954c00bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.177+0000 7f9550c00700 1 -- 172.21.15.27:0/3019813366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c019070 con 0x7f955419fb40 2024-05-10T11:00:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.177+0000 7f9550c00700 1 -- 172.21.15.27:0/3019813366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f954c00cbb0 con 0x7f955419fb40 2024-05-10T11:00:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.177+0000 7f955ac00700 1 -- 172.21.15.27:0/3019813366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95541a7ae0 con 0x7f955419fb40 2024-05-10T11:00:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.177+0000 7f9550c00700 1 -- 172.21.15.27:0/3019813366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c0125a0 con 0x7f955419fb40 2024-05-10T11:00:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.178+0000 7f955ac00700 1 -- 172.21.15.27:0/3019813366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95541a8020 con 0x7f955419fb40 2024-05-10T11:00:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.179+0000 7f9550c00700 1 -- 172.21.15.27:0/3019813366 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f954c01c070 con 0x7f955419fb40 2024-05-10T11:00:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.179+0000 7f955ac00700 1 -- 172.21.15.27:0/3019813366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9554064b00 con 0x7f955419fb40 2024-05-10T11:00:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.179+0000 7f9550c00700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f953803e9a0 0x7f9538040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.180+0000 7f9550c00700 1 -- 172.21.15.27:0/3019813366 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f954c051fe0 con 0x7f955419fb40 2024-05-10T11:00:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.180+0000 7f9553e00700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f953803e9a0 0x7f9538040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.181+0000 7f9553e00700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f953803e9a0 0x7f9538040e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9544009750 tx=0x7f954401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:34.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.186+0000 7f9550c00700 1 -- 172.21.15.27:0/3019813366 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f954c01c360 con 0x7f955419fb40 2024-05-10T11:00:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.440+0000 7f955ac00700 1 -- 172.21.15.27:0/3019813366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f955402e6d0 con 0x7f955419fb40 2024-05-10T11:00:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.442+0000 7f9550c00700 1 -- 172.21.15.27:0/3019813366 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f954c01fd60 con 0x7f955419fb40 2024-05-10T11:00:34.441 INFO:teuthology.orchestra.run.smithi027.stdout:# minimal ceph.conf for 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:34.442 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-05-10T11:00:34.442 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:00:34.442 INFO:teuthology.orchestra.run.smithi027.stdout: mon_host = [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-05-10T11:00:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.445+0000 7f955ac00700 1 -- 172.21.15.27:0/3019813366 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f953803e9a0 msgr2=0x7f9538040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.445+0000 7f955ac00700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f953803e9a0 0x7f9538040e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9544009750 tx=0x7f954401a040 comp rx=0 tx=0).stop 2024-05-10T11:00:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.445+0000 7f955ac00700 1 -- 172.21.15.27:0/3019813366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f955419fb40 msgr2=0x7f955419fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.445+0000 7f955ac00700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f955419fb40 0x7f955419fff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f954c00bd50 tx=0x7f954c00bd80 comp rx=0 tx=0).stop 2024-05-10T11:00:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.445+0000 7f955ac00700 1 -- 172.21.15.27:0/3019813366 shutdown_connections 2024-05-10T11:00:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.445+0000 7f955ac00700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f953803e9a0 0x7f9538040e60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.445+0000 7f955ac00700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f955419fb40 0x7f955419fff0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.445+0000 7f955ac00700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9554108f40 0x7f955419ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.445+0000 7f955ac00700 1 --2- 172.21.15.27:0/3019813366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f955419b270 0x7f955419f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.445+0000 7f955ac00700 1 -- 172.21.15.27:0/3019813366 >> 172.21.15.27:0/3019813366 conn(0x7f9554077900 msgr2=0x7f95540ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.445+0000 7f955ac00700 1 -- 172.21.15.27:0/3019813366 shutdown_connections 2024-05-10T11:00:34.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:34.445+0000 7f955ac00700 1 -- 172.21.15.27:0/3019813366 wait complete. 2024-05-10T11:00:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: Updating smithi027:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-10T11:00:34.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:34.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:34 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3019813366' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: Updating smithi027:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-10T11:00:34.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:34.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:34 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3019813366' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:34.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: Updating smithi027:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-10T11:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:34 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3019813366' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:35.133 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-10T11:00:35.134 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T11:00:35.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.conf 2024-05-10T11:00:35.205 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T11:00:35.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:00:35.315 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-05-10T11:00:35.315 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.conf 2024-05-10T11:00:35.345 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-05-10T11:00:35.345 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:00:35.413 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-10T11:00:35.413 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.conf 2024-05-10T11:00:35.440 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-10T11:00:35.440 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:00:35.504 INFO:tasks.cephadm:Adding mgr.x on smithi027 2024-05-10T11:00:35.504 INFO:tasks.cephadm:Adding mgr.y on smithi076 2024-05-10T11:00:35.504 INFO:tasks.cephadm:Adding mgr.z on smithi189 2024-05-10T11:00:35.504 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch apply mgr '3;smithi027=x;smithi076=y;smithi189=z' 2024-05-10T11:00:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:35 smithi189 ceph-mon[34842]: Reconfiguring mon.a (unknown last config time)... 2024-05-10T11:00:35.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:35 smithi189 ceph-mon[34842]: Reconfiguring daemon mon.a on smithi027 2024-05-10T11:00:35.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:35 smithi189 ceph-mon[34842]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:35.777 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:00:35.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:35 smithi027 ceph-mon[30054]: Reconfiguring mon.a (unknown last config time)... 2024-05-10T11:00:35.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:35 smithi027 ceph-mon[30054]: Reconfiguring daemon mon.a on smithi027 2024-05-10T11:00:35.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:35 smithi027 ceph-mon[30054]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:35.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:35 smithi076 ceph-mon[34152]: Reconfiguring mon.a (unknown last config time)... 2024-05-10T11:00:35.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:35 smithi076 ceph-mon[34152]: Reconfiguring daemon mon.a on smithi027 2024-05-10T11:00:35.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:35 smithi076 ceph-mon[34152]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.182+0000 7fe0eeede700 1 -- 172.21.15.189:0/2332079881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0e8101ef0 msgr2=0x7fe0e8102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.182+0000 7fe0eeede700 1 --2- 172.21.15.189:0/2332079881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0e8101ef0 0x7fe0e8102330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc009910 tx=0x7fe0dc009c20 comp rx=0 tx=0).stop 2024-05-10T11:00:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.182+0000 7fe0eeede700 1 -- 172.21.15.189:0/2332079881 shutdown_connections 2024-05-10T11:00:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.182+0000 7fe0eeede700 1 --2- 172.21.15.189:0/2332079881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0e8101ef0 0x7fe0e8102330 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.182+0000 7fe0eeede700 1 -- 172.21.15.189:0/2332079881 >> 172.21.15.189:0/2332079881 conn(0x7fe0e80fd4e0 msgr2=0x7fe0e80ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.182+0000 7fe0eeede700 1 -- 172.21.15.189:0/2332079881 shutdown_connections 2024-05-10T11:00:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.182+0000 7fe0eeede700 1 -- 172.21.15.189:0/2332079881 wait complete. 2024-05-10T11:00:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.183+0000 7fe0eeede700 1 Processor -- start 2024-05-10T11:00:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.183+0000 7fe0eeede700 1 -- start start 2024-05-10T11:00:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.183+0000 7fe0eeede700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8101ef0 0x7fe0e81987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.183+0000 7fe0eeede700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0e8198d30 0x7fe0e819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.183+0000 7fe0eeede700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0e81991b0 0x7fe0e81a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.183+0000 7fe0eeede700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e8107de0 con 0x7fe0e8198d30 2024-05-10T11:00:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.183+0000 7fe0eeede700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e8107c60 con 0x7fe0e81991b0 2024-05-10T11:00:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.183+0000 7fe0eeede700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e8107f60 con 0x7fe0e8101ef0 2024-05-10T11:00:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.183+0000 7fe0ecc7a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8101ef0 0x7fe0e81987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.183+0000 7fe0ecc7a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8101ef0 0x7fe0e81987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42218/0 (socket says 172.21.15.189:42218) 2024-05-10T11:00:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.183+0000 7fe0ecc7a700 1 -- 172.21.15.189:0/717104533 learned_addr learned my addr 172.21.15.189:0/717104533 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-10T11:00:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0ed47b700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0e81991b0 0x7fe0e81a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0e7fff700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0e8198d30 0x7fe0e819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0ecc7a700 1 -- 172.21.15.189:0/717104533 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0e81991b0 msgr2=0x7fe0e81a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0ecc7a700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0e81991b0 0x7fe0e81a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0ecc7a700 1 -- 172.21.15.189:0/717104533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0e8198d30 msgr2=0x7fe0e819dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0ecc7a700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0e8198d30 0x7fe0e819dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0ecc7a700 1 -- 172.21.15.189:0/717104533 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e81a0c00 con 0x7fe0e8101ef0 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0e7fff700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0e8198d30 0x7fe0e819dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0ed47b700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0e81991b0 0x7fe0e81a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0ecc7a700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8101ef0 0x7fe0e81987f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0059d0 tx=0x7fe0dc0050d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0e5ffb700 1 -- 172.21.15.189:0/717104533 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc027070 con 0x7fe0e8101ef0 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0e5ffb700 1 -- 172.21.15.189:0/717104533 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0dc00be10 con 0x7fe0e8101ef0 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0eeede700 1 -- 172.21.15.189:0/717104533 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e81a0e90 con 0x7fe0e8101ef0 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0e5ffb700 1 -- 172.21.15.189:0/717104533 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc022e20 con 0x7fe0e8101ef0 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.184+0000 7fe0eeede700 1 -- 172.21.15.189:0/717104533 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e81a1300 con 0x7fe0e8101ef0 2024-05-10T11:00:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.185+0000 7fe0e5ffb700 1 -- 172.21.15.189:0/717104533 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fe0dc02a070 con 0x7fe0e8101ef0 2024-05-10T11:00:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.185+0000 7fe0eeede700 1 -- 172.21.15.189:0/717104533 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e8064b00 con 0x7fe0e8101ef0 2024-05-10T11:00:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.186+0000 7fe0e5ffb700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe0d803e9a0 0x7fe0d8040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.186+0000 7fe0e5ffb700 1 -- 172.21.15.189:0/717104533 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe0dc053a10 con 0x7fe0e8101ef0 2024-05-10T11:00:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.186+0000 7fe0e7fff700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe0d803e9a0 0x7fe0d8040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.187+0000 7fe0e7fff700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe0d803e9a0 0x7fe0d8040e60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe0d40079b0 tx=0x7fe0d4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:37.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.191+0000 7fe0e5ffb700 1 -- 172.21.15.189:0/717104533 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe0dc013070 con 0x7fe0e8101ef0 2024-05-10T11:00:37.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.408+0000 7fe0eeede700 1 -- 172.21.15.189:0/717104533 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi076=y;smithi189=z", "target": ["mon-mgr", ""]}) -- 0x7fe0e80638e0 con 0x7fe0d803e9a0 2024-05-10T11:00:37.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.422+0000 7fe0e5ffb700 1 -- 172.21.15.189:0/717104533 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe0e80638e0 con 0x7fe0d803e9a0 2024-05-10T11:00:37.424 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mgr update... 2024-05-10T11:00:37.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.424+0000 7fe0eeede700 1 -- 172.21.15.189:0/717104533 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe0d803e9a0 msgr2=0x7fe0d8040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:37.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.424+0000 7fe0eeede700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe0d803e9a0 0x7fe0d8040e60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe0d40079b0 tx=0x7fe0d4008040 comp rx=0 tx=0).stop 2024-05-10T11:00:37.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.424+0000 7fe0eeede700 1 -- 172.21.15.189:0/717104533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8101ef0 msgr2=0x7fe0e81987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:37.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.424+0000 7fe0eeede700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8101ef0 0x7fe0e81987f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0059d0 tx=0x7fe0dc0050d0 comp rx=0 tx=0).stop 2024-05-10T11:00:37.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.425+0000 7fe0eeede700 1 -- 172.21.15.189:0/717104533 shutdown_connections 2024-05-10T11:00:37.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.425+0000 7fe0eeede700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe0d803e9a0 0x7fe0d8040e60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:37.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.425+0000 7fe0eeede700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8101ef0 0x7fe0e81987f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:37.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.425+0000 7fe0eeede700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0e81991b0 0x7fe0e81a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:37.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.425+0000 7fe0eeede700 1 --2- 172.21.15.189:0/717104533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0e8198d30 0x7fe0e819dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:37.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.425+0000 7fe0eeede700 1 -- 172.21.15.189:0/717104533 >> 172.21.15.189:0/717104533 conn(0x7fe0e80fd4e0 msgr2=0x7fe0e818f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:37.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.425+0000 7fe0eeede700 1 -- 172.21.15.189:0/717104533 shutdown_connections 2024-05-10T11:00:37.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:00:37.425+0000 7fe0eeede700 1 -- 172.21.15.189:0/717104533 wait complete. 2024-05-10T11:00:37.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:37.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:37.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:37 smithi076 ceph-mon[34152]: Reconfiguring mon.b (monmap changed)... 2024-05-10T11:00:37.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:37.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-10T11:00:37.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:37.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:37 smithi076 ceph-mon[34152]: Reconfiguring daemon mon.b on smithi076 2024-05-10T11:00:37.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:37 smithi076 ceph-mon[34152]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:37.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:37.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:37.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:37 smithi189 ceph-mon[34842]: Reconfiguring mon.b (monmap changed)... 2024-05-10T11:00:37.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:37.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-10T11:00:37.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:37.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:37 smithi189 ceph-mon[34842]: Reconfiguring daemon mon.b on smithi076 2024-05-10T11:00:37.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:37 smithi189 ceph-mon[34842]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:37.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:37.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:37.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:37 smithi027 ceph-mon[30054]: Reconfiguring mon.b (monmap changed)... 2024-05-10T11:00:37.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:37.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-10T11:00:37.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:37.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:37 smithi027 ceph-mon[30054]: Reconfiguring daemon mon.b on smithi076 2024-05-10T11:00:37.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:37 smithi027 ceph-mon[30054]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:38.116 DEBUG:teuthology.orchestra.run.smithi076:mgr.y> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.y.service 2024-05-10T11:00:38.119 DEBUG:teuthology.orchestra.run.smithi189:mgr.z> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.z.service 2024-05-10T11:00:38.121 INFO:tasks.cephadm:Deploying OSDs... 2024-05-10T11:00:38.121 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T11:00:38.121 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-05-10T11:00:38.181 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-10T11:00:38.182 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-05-10T11:00:38.242 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-10T11:00:38.242 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-10T11:00:38.242 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-05-10T11:00:38.242 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:00:38.242 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-05-10T11:00:38.242 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-05-10 10:59:37.782645443 +0000 2024-05-10T11:00:38.242 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-05-10 10:54:22.903723630 +0000 2024-05-10T11:00:38.243 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-05-10 10:54:22.903723630 +0000 2024-05-10T11:00:38.243 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-05-10 10:54:22.903723630 +0000 2024-05-10T11:00:38.243 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-10T11:00:38.330 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-05-10T11:00:38.330 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-05-10T11:00:38.330 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000442535 s, 1.2 MB/s 2024-05-10T11:00:38.334 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-10T11:00:38.397 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-05-10T11:00:38.452 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-10T11:00:38.453 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-10T11:00:38.453 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-05-10T11:00:38.453 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:00:38.453 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-05-10T11:00:38.453 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-05-10 10:59:37.848643761 +0000 2024-05-10T11:00:38.453 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-05-10 10:54:23.556706733 +0000 2024-05-10T11:00:38.453 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-05-10 10:54:23.556706733 +0000 2024-05-10T11:00:38.453 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-05-10 10:54:23.556706733 +0000 2024-05-10T11:00:38.454 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-10T11:00:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-05-10T11:00:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-05-10T11:00:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000396741 s, 1.3 MB/s 2024-05-10T11:00:38.544 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-10T11:00:38.606 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-05-10T11:00:38.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:38 smithi027 ceph-mon[30054]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi076=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:38.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:38 smithi027 ceph-mon[30054]: Saving service mgr spec with placement smithi027=x;smithi076=y;smithi189=z;count:3 2024-05-10T11:00:38.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:38.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:38.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:38.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:38 smithi027 ceph-mon[30054]: Reconfiguring mon.c (monmap changed)... 2024-05-10T11:00:38.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:38.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-10T11:00:38.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:38.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:38 smithi027 ceph-mon[30054]: Reconfiguring daemon mon.c on smithi189 2024-05-10T11:00:38.660 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-10T11:00:38.660 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-10T11:00:38.660 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-05-10T11:00:38.660 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:00:38.660 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-05-10T11:00:38.660 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-05-10 10:59:37.903642359 +0000 2024-05-10T11:00:38.660 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-05-10 10:54:24.189690355 +0000 2024-05-10T11:00:38.661 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-05-10 10:54:24.189690355 +0000 2024-05-10T11:00:38.661 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-05-10 10:54:24.189690355 +0000 2024-05-10T11:00:38.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-10T11:00:38.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:38 smithi076 ceph-mon[34152]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi076=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:38.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:38 smithi076 ceph-mon[34152]: Saving service mgr spec with placement smithi027=x;smithi076=y;smithi189=z;count:3 2024-05-10T11:00:38.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:38.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:38.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:38.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:38 smithi076 ceph-mon[34152]: Reconfiguring mon.c (monmap changed)... 2024-05-10T11:00:38.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:38.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-10T11:00:38.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:38.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:38 smithi076 ceph-mon[34152]: Reconfiguring daemon mon.c on smithi189 2024-05-10T11:00:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:38 smithi189 ceph-mon[34842]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi076=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:38 smithi189 ceph-mon[34842]: Saving service mgr spec with placement smithi027=x;smithi076=y;smithi189=z;count:3 2024-05-10T11:00:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:38 smithi189 ceph-mon[34842]: Reconfiguring mon.c (monmap changed)... 2024-05-10T11:00:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-10T11:00:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-10T11:00:38.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:38.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:38 smithi189 ceph-mon[34842]: Reconfiguring daemon mon.c on smithi189 2024-05-10T11:00:38.793 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-05-10T11:00:38.793 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-05-10T11:00:38.793 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000424844 s, 1.2 MB/s 2024-05-10T11:00:38.797 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-10T11:00:38.856 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-05-10T11:00:39.056 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-10T11:00:39.056 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-10T11:00:39.056 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-05-10T11:00:39.056 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:00:39.056 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-05-10T11:00:39.056 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-05-10 10:59:37.964640805 +0000 2024-05-10T11:00:39.056 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-05-10 10:54:24.824673924 +0000 2024-05-10T11:00:39.056 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-05-10 10:54:24.824673924 +0000 2024-05-10T11:00:39.056 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-05-10 10:54:24.824673924 +0000 2024-05-10T11:00:39.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-10T11:00:39.149 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-05-10T11:00:39.149 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-05-10T11:00:39.149 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000394032 s, 1.3 MB/s 2024-05-10T11:00:39.154 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-10T11:00:39.217 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-05-10T11:00:39.217 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/scratch_devs of=/dev/stdout 2024-05-10T11:00:39.238 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-10T11:00:39.238 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 2024-05-10T11:00:39.298 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-10T11:00:39.299 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-10T11:00:39.299 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-10T11:00:39.299 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:00:39.299 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-05-10T11:00:39.299 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-05-10 11:00:25.057041565 +0000 2024-05-10T11:00:39.299 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-05-10 10:53:16.894691110 +0000 2024-05-10T11:00:39.300 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-05-10 10:53:16.894691110 +0000 2024-05-10T11:00:39.300 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-05-10 10:53:16.894691110 +0000 2024-05-10T11:00:39.300 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-10T11:00:39.368 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-05-10T11:00:39.368 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-05-10T11:00:39.369 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000322281 s, 1.6 MB/s 2024-05-10T11:00:39.370 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-10T11:00:39.431 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 2024-05-10T11:00:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-10T11:00:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-10T11:00:39.491 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-10T11:00:39.491 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:00:39.492 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-05-10T11:00:39.492 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-05-10 11:00:25.058041536 +0000 2024-05-10T11:00:39.492 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-05-10 10:53:17.259680383 +0000 2024-05-10T11:00:39.492 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-05-10 10:53:17.259680383 +0000 2024-05-10T11:00:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-05-10 10:53:17.259680383 +0000 2024-05-10T11:00:39.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-10T11:00:39.560 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-05-10T11:00:39.560 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-05-10T11:00:39.560 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000284891 s, 1.8 MB/s 2024-05-10T11:00:39.561 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-10T11:00:39.622 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 2024-05-10T11:00:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:39 smithi076 ceph-mon[34152]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:39 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:39 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:39 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:39 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:39.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:39 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:39.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:39 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:39.680 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-10T11:00:39.680 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-10T11:00:39.680 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-10T11:00:39.680 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:00:39.680 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-05-10T11:00:39.680 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-05-10 11:00:25.058041536 +0000 2024-05-10T11:00:39.680 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-05-10 10:53:17.606670186 +0000 2024-05-10T11:00:39.680 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-05-10 10:53:17.606670186 +0000 2024-05-10T11:00:39.681 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-05-10 10:53:17.606670186 +0000 2024-05-10T11:00:39.681 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-10T11:00:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:39 smithi189 ceph-mon[34842]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:39 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:39 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:39 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:39 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:39 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:39 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:39.747 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-05-10T11:00:39.747 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-05-10T11:00:39.747 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000258115 s, 2.0 MB/s 2024-05-10T11:00:39.748 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-10T11:00:39.807 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 2024-05-10T11:00:39.865 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-10T11:00:39.865 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-10T11:00:39.865 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-10T11:00:39.865 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:00:39.866 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-05-10T11:00:39.866 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-05-10 11:00:25.058041536 +0000 2024-05-10T11:00:39.866 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-05-10 10:53:17.956659901 +0000 2024-05-10T11:00:39.866 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-05-10 10:53:17.956659901 +0000 2024-05-10T11:00:39.866 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-05-10 10:53:17.956659901 +0000 2024-05-10T11:00:39.866 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-10T11:00:39.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:39 smithi027 ceph-mon[30054]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:39.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:39 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:39.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:39 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:39.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:39 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:39.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:39 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:39.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:39 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:39.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:39 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:39.932 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-05-10T11:00:39.932 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-05-10T11:00:39.932 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000233611 s, 2.2 MB/s 2024-05-10T11:00:39.933 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-10T11:00:39.991 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-10T11:00:39.992 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-05-10T11:00:40.006 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-10T11:00:40.006 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-05-10T11:00:40.060 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-10T11:00:40.061 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-10T11:00:40.061 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-10T11:00:40.061 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:00:40.061 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-05-10T11:00:40.061 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-05-10 11:00:15.951540221 +0000 2024-05-10T11:00:40.061 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-05-10 10:53:12.661979475 +0000 2024-05-10T11:00:40.061 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-05-10 10:53:12.661979475 +0000 2024-05-10T11:00:40.061 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-05-10 10:53:12.661979475 +0000 2024-05-10T11:00:40.061 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-10T11:00:40.124 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-05-10T11:00:40.124 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-05-10T11:00:40.124 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000215573 s, 2.4 MB/s 2024-05-10T11:00:40.125 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-10T11:00:40.181 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-05-10T11:00:40.237 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-10T11:00:40.238 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-10T11:00:40.238 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-10T11:00:40.238 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:00:40.238 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-05-10T11:00:40.238 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-05-10 11:00:15.951540221 +0000 2024-05-10T11:00:40.238 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-05-10 10:53:13.014972121 +0000 2024-05-10T11:00:40.238 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-05-10 10:53:13.014972121 +0000 2024-05-10T11:00:40.238 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-05-10 10:53:13.014972121 +0000 2024-05-10T11:00:40.238 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-10T11:00:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-05-10T11:00:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-05-10T11:00:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000270032 s, 1.9 MB/s 2024-05-10T11:00:40.302 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-10T11:00:40.358 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-05-10T11:00:40.412 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-10T11:00:40.412 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-10T11:00:40.412 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-10T11:00:40.413 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:00:40.413 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-05-10T11:00:40.413 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-05-10 11:00:15.952540201 +0000 2024-05-10T11:00:40.413 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-05-10 10:53:13.356964997 +0000 2024-05-10T11:00:40.413 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-05-10 10:53:13.356964997 +0000 2024-05-10T11:00:40.413 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-05-10 10:53:13.356964997 +0000 2024-05-10T11:00:40.413 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-10T11:00:40.475 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-05-10T11:00:40.476 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-05-10T11:00:40.476 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000194812 s, 2.6 MB/s 2024-05-10T11:00:40.477 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-10T11:00:40.532 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-05-10T11:00:40.588 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-10T11:00:40.588 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-10T11:00:40.588 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-05-10T11:00:40.588 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:00:40.588 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-05-10T11:00:40.588 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-05-10 11:00:15.952540201 +0000 2024-05-10T11:00:40.588 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-05-10 10:53:13.695957935 +0000 2024-05-10T11:00:40.588 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-05-10 10:53:13.695957935 +0000 2024-05-10T11:00:40.588 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-05-10 10:53:13.695957935 +0000 2024-05-10T11:00:40.589 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-10T11:00:40.650 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-05-10T11:00:40.650 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-05-10T11:00:40.650 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000192627 s, 2.7 MB/s 2024-05-10T11:00:40.651 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-10T11:00:40.707 INFO:tasks.cephadm:Deploying osd.0 on smithi027 with /dev/vg_nvme/lv_4... 2024-05-10T11:00:40.707 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-10T11:00:40.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:40 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-10T11:00:40.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:40 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-10T11:00:40.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:40 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-10T11:00:40.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:40 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:40.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:40 smithi189 ceph-mon[34842]: Deploying daemon mgr.z on smithi189 2024-05-10T11:00:40.791 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-10T11:00:40.791 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-10T11:00:40.792 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-10T11:00:40.792 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:40.792 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:40 smithi027 ceph-mon[30054]: Deploying daemon mgr.z on smithi189 2024-05-10T11:00:40.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:40 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-10T11:00:40.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:40 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-10T11:00:40.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:40 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-10T11:00:40.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:40 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:40.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:40 smithi076 ceph-mon[34152]: Deploying daemon mgr.z on smithi189 2024-05-10T11:00:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:00:41.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:41 smithi027 ceph-mon[30054]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:41.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:41 smithi189 ceph-mon[34842]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:41.673 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:41 smithi189 systemd[1]: Starting Ceph mgr.z for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:00:41.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:41 smithi076 ceph-mon[34152]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:41.966 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:41 smithi189 podman[36696]: 2024-05-10 11:00:41.574924414 +0000 UTC m=+0.017308030 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:00:41.966 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:41 smithi189 podman[36696]: 2024-05-10 11:00:41.697958922 +0000 UTC m=+0.140342554 container create 6f59165a3ad5334ead3f47d105171f522d31254f2f61c9a8d98e91d9140c0a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-10T11:00:41.966 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:41 smithi189 podman[36696]: 2024-05-10 11:00:41.89202249 +0000 UTC m=+0.334406078 container init 6f59165a3ad5334ead3f47d105171f522d31254f2f61c9a8d98e91d9140c0a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z, org.label-schema.build-date=20240507, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-10T11:00:41.966 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:41 smithi189 podman[36696]: 2024-05-10 11:00:41.895235658 +0000 UTC m=+0.337619246 container start 6f59165a3ad5334ead3f47d105171f522d31254f2f61c9a8d98e91d9140c0a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08) 2024-05-10T11:00:41.967 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:41 smithi189 ceph-mgr[36715]: -- 172.21.15.189:0/2151488612 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ab7dcc5a0 con 0x562ab7da0800 2024-05-10T11:00:42.237 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:41 smithi189 bash[36696]: 6f59165a3ad5334ead3f47d105171f522d31254f2f61c9a8d98e91d9140c0a81 2024-05-10T11:00:42.237 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:41 smithi189 systemd[1]: Started Ceph mgr.z for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:00:42.237 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:42.060+0000 7f23e7abc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-10T11:00:42.650 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:42.243+0000 7f23e7abc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-10T11:00:42.650 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:42.343+0000 7f23e7abc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-10T11:00:42.651 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:42.649+0000 7f23e7abc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-10T11:00:42.986 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:42.733+0000 7f23e7abc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-10T11:00:43.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:43.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:43.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:43.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-10T11:00:43.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-10T11:00:43.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-10T11:00:43.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:43.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:43 smithi027 ceph-mon[30054]: Deploying daemon mgr.y on smithi076 2024-05-10T11:00:43.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:43 smithi027 ceph-mon[30054]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:43.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:43.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:43.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-10T11:00:43.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-10T11:00:43.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-10T11:00:43.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:43.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:43 smithi076 ceph-mon[34152]: Deploying daemon mgr.y on smithi076 2024-05-10T11:00:43.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:43 smithi076 ceph-mon[34152]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:43.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:43.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:43.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:43.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-10T11:00:43.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-10T11:00:43.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-10T11:00:43.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:43.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:43 smithi189 ceph-mon[34842]: Deploying daemon mgr.y on smithi076 2024-05-10T11:00:43.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:43 smithi189 ceph-mon[34842]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:43.960 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-10T11:00:43.960 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-10T11:00:43.960 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-05-10T11:00:43.960 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-05-10T11:00:43.960 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0354671 s, 296 MB/s 2024-05-10T11:00:43.960 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-05-10T11:00:44.237 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:43 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:43.931+0000 7f23e7abc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-10T11:00:44.237 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:44.096+0000 7f23e7abc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-10T11:00:44.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:44 smithi076 podman[35527]: 2024-05-10 11:00:44.493697945 +0000 UTC m=+0.353032960 container start c4f46c9ba6decaf63c5b1f9570f8828592f1208d0633b8688a6239a4994d80be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507) 2024-05-10T11:00:44.548 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:44.266+0000 7f23e7abc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-10T11:00:44.549 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:44.548+0000 7f23e7abc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-10T11:00:44.775 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_4 2024-05-10T11:00:44.838 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:44 smithi076 ceph-mgr[35546]: -- 172.21.15.76:0/1753632183 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565101a6a5a0 con 0x565101a3e800 2024-05-10T11:00:44.838 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:44 smithi076 bash[35527]: c4f46c9ba6decaf63c5b1f9570f8828592f1208d0633b8688a6239a4994d80be 2024-05-10T11:00:44.838 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:44 smithi076 systemd[1]: Started Ceph mgr.y for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:00:44.838 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:44 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:44.644+0000 7fbd293e7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-10T11:00:44.838 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:44 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:44.837+0000 7fbd293e7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-10T11:00:44.844 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:44.639+0000 7f23e7abc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-10T11:00:45.175 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:44 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:44.943+0000 7fbd293e7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-10T11:00:45.207 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:00:45.237 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:44.844+0000 7f23e7abc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-10T11:00:45.237 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:45 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:45.039+0000 7f23e7abc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-10T11:00:45.667 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:45 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:45.268+0000 7fbd293e7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-10T11:00:45.667 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:45 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:45.357+0000 7fbd293e7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-10T11:00:45.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:45 smithi027 ceph-mon[30054]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:45.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:45 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:45.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:45 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:45.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:45 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:45.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:45 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:45.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:45 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:45.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:45 smithi076 ceph-mon[34152]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:45.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:45 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:45.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:45 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:45.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:45 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:45.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:45 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:45.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:45 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:45.951 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:45 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:45.677+0000 7f23e7abc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-10T11:00:45.951 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:45 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:45.901+0000 7f23e7abc200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-10T11:00:45.951 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:45 smithi189 ceph-mon[34842]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:45.951 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:45 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:45.951 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:45 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:45.951 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:45 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:45.951 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:45 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:45.951 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:45 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:46.710 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:46 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:46.345+0000 7f23e7abc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-10T11:00:46.710 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:46 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:46.442+0000 7f23e7abc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-10T11:00:46.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:46.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:46.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-10T11:00:46.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-10T11:00:46.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:46.925 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:46 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:46.642+0000 7fbd293e7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-10T11:00:46.925 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:46 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:46.812+0000 7fbd293e7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-10T11:00:46.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:46.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:46.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-10T11:00:46.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-10T11:00:46.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:46.987 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:46 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:46.815+0000 7f23e7abc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-10T11:00:46.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:46.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:46.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:46.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-10T11:00:46.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-10T11:00:46.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.085+0000 7f8e94a00700 1 -- 172.21.15.27:0/3872656655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e90105cf0 msgr2=0x7f8e9010db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.085+0000 7f8e94a00700 1 --2- 172.21.15.27:0/3872656655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e90105cf0 0x7f8e9010db80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009fc0 tx=0x7f8e8800b330 comp rx=0 tx=0).stop 2024-05-10T11:00:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.085+0000 7f8e94a00700 1 -- 172.21.15.27:0/3872656655 shutdown_connections 2024-05-10T11:00:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.085+0000 7f8e94a00700 1 --2- 172.21.15.27:0/3872656655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e90105cf0 0x7f8e9010db80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.085+0000 7f8e94a00700 1 --2- 172.21.15.27:0/3872656655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e90106e10 0x7f8e90107230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.085+0000 7f8e94a00700 1 --2- 172.21.15.27:0/3872656655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e90105350 0x7f8e901057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.085+0000 7f8e94a00700 1 -- 172.21.15.27:0/3872656655 >> 172.21.15.27:0/3872656655 conn(0x7f8e90100cf0 msgr2=0x7f8e90103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:00:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.085+0000 7f8e94a00700 1 -- 172.21.15.27:0/3872656655 shutdown_connections 2024-05-10T11:00:47.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.085+0000 7f8e94a00700 1 -- 172.21.15.27:0/3872656655 wait complete. 2024-05-10T11:00:47.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.086+0000 7f8e94a00700 1 Processor -- start 2024-05-10T11:00:47.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.086+0000 7f8e94a00700 1 -- start start 2024-05-10T11:00:47.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e94a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e90105350 0x7f8e9019b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:47.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e94a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e90105cf0 0x7f8e9019be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:47.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e94a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e90106e10 0x7f8e901a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:47.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e94a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8e90111300 con 0x7f8e90105cf0 2024-05-10T11:00:47.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e94a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8e90111180 con 0x7f8e90106e10 2024-05-10T11:00:47.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e94a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8e90111480 con 0x7f8e90105350 2024-05-10T11:00:47.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e8de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e90105350 0x7f8e9019b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:47.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e8de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e90105350 0x7f8e9019b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:35302/0 (socket says 172.21.15.27:35302) 2024-05-10T11:00:47.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e8de00700 1 -- 172.21.15.27:0/87625917 learned_addr learned my addr 172.21.15.27:0/87625917 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:00:47.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e8d400700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e90105cf0 0x7f8e9019be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:47.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e8e800700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e90106e10 0x7f8e901a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:47.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e8de00700 1 -- 172.21.15.27:0/87625917 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e90106e10 msgr2=0x7f8e901a2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:47.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e8de00700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e90106e10 0x7f8e901a2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:47.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e8de00700 1 -- 172.21.15.27:0/87625917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e90105cf0 msgr2=0x7f8e9019be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:00:47.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e8de00700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e90105cf0 0x7f8e9019be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:00:47.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.087+0000 7f8e8de00700 1 -- 172.21.15.27:0/87625917 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e901a3540 con 0x7f8e90105350 2024-05-10T11:00:47.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.088+0000 7f8e8de00700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e90105350 0x7f8e9019b8e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000e9a0 tx=0x7f8e8000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:47.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.088+0000 7f8e86a00700 1 -- 172.21.15.27:0/87625917 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80004220 con 0x7f8e90105350 2024-05-10T11:00:47.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.088+0000 7f8e86a00700 1 -- 172.21.15.27:0/87625917 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e800043c0 con 0x7f8e90105350 2024-05-10T11:00:47.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.088+0000 7f8e94a00700 1 -- 172.21.15.27:0/87625917 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e901a3830 con 0x7f8e90105350 2024-05-10T11:00:47.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.088+0000 7f8e86a00700 1 -- 172.21.15.27:0/87625917 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80003cc0 con 0x7f8e90105350 2024-05-10T11:00:47.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.088+0000 7f8e94a00700 1 -- 172.21.15.27:0/87625917 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e901a3dc0 con 0x7f8e90105350 2024-05-10T11:00:47.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.089+0000 7f8e86a00700 1 -- 172.21.15.27:0/87625917 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f8e80018430 con 0x7f8e90105350 2024-05-10T11:00:47.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.090+0000 7f8e94a00700 1 -- 172.21.15.27:0/87625917 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e9006a190 con 0x7f8e90105350 2024-05-10T11:00:47.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.090+0000 7f8e86a00700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f8e7403e750 0x7f8e74040c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:00:47.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.090+0000 7f8e86a00700 1 -- 172.21.15.27:0/87625917 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8e80052b50 con 0x7f8e90105350 2024-05-10T11:00:47.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.090+0000 7f8e8d400700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f8e7403e750 0x7f8e74040c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:00:47.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.091+0000 7f8e8d400700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f8e7403e750 0x7f8e74040c10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c009750 tx=0x7f8e7c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:00:47.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.095+0000 7f8e86a00700 1 -- 172.21.15.27:0/87625917 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f8e80054e20 con 0x7f8e90105350 2024-05-10T11:00:47.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:46 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:46.981+0000 7fbd293e7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-10T11:00:47.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:47 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:47.269+0000 7fbd293e7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-10T11:00:47.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:47.346+0000 7f8e94a00700 1 -- 172.21.15.27:0/87625917 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8e900638e0 con 0x7f8e7403e750 2024-05-10T11:00:47.487 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:47 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:47.166+0000 7f23e7abc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-10T11:00:47.487 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:47 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:47.257+0000 7f23e7abc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-10T11:00:47.562 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:47 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:47.361+0000 7fbd293e7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-10T11:00:47.562 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:47 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:47.562+0000 7fbd293e7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-10T11:00:47.925 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:47 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:47.757+0000 7fbd293e7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-10T11:00:47.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:47 smithi076 ceph-mon[34152]: Reconfiguring mgr.x (unknown last config time)... 2024-05-10T11:00:47.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:47 smithi076 ceph-mon[34152]: Reconfiguring daemon mgr.x on smithi027 2024-05-10T11:00:47.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:47 smithi076 ceph-mon[34152]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:47.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:00:47.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:00:47.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:47.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:47 smithi189 ceph-mon[34842]: Reconfiguring mgr.x (unknown last config time)... 2024-05-10T11:00:47.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:47 smithi189 ceph-mon[34842]: Reconfiguring daemon mgr.x on smithi027 2024-05-10T11:00:47.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:47 smithi189 ceph-mon[34842]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:47.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:00:47.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:00:47.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:47.987 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:47 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:47.498+0000 7f23e7abc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-10T11:00:47.987 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:47 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:47.629+0000 7f23e7abc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-10T11:00:47.988 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:47 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:47.724+0000 7f23e7abc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-10T11:00:48.057 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:47 smithi027 ceph-mon[30054]: Reconfiguring mgr.x (unknown last config time)... 2024-05-10T11:00:48.057 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:47 smithi027 ceph-mon[30054]: Reconfiguring daemon mgr.x on smithi027 2024-05-10T11:00:48.057 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:47 smithi027 ceph-mon[30054]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:48.057 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:00:48.057 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:00:48.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:48.487 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:48 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:48.049+0000 7f23e7abc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-10T11:00:48.487 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:48 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:48.140+0000 7f23e7abc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-10T11:00:48.679 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:48 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:48.356+0000 7fbd293e7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-10T11:00:48.679 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:48 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:48.577+0000 7fbd293e7200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-10T11:00:48.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:48 smithi027 ceph-mon[30054]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:48.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:48 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:48.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:48 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:48.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:48 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:48.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:48 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:48.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:48 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:48.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:48 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:48.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:48 smithi189 ceph-mon[34842]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:48.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:48 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:48.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:48 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:48.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:48 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:48.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:48 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:48.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:48 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:48.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:48 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:48.988 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:48 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:48.735+0000 7f23e7abc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-10T11:00:48.988 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:00:48 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:00:48.950+0000 7f23e7abc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-10T11:00:49.003 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:48 smithi076 ceph-mon[34152]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:00:49.003 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:48 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:49.003 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:48 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:49.003 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:48 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:00:49.003 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:48 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:00:49.003 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:48 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:00:49.003 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:48 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:00:49.426 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:49.003+0000 7fbd293e7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-10T11:00:49.426 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:49.099+0000 7fbd293e7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-10T11:00:49.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:49.732+0000 7f8e86a00700 1 -- 172.21.15.27:0/87625917 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 14) ==== 99937+0+0 (secure 0 0 0) 0x7f8e80051e00 con 0x7f8e90105350 2024-05-10T11:00:49.823 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:49.466+0000 7fbd293e7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-10T11:00:49.824 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:49.824+0000 7fbd293e7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-10T11:00:49.824 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:49 smithi076 ceph-mon[34152]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:49.824 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:49 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.189:0/2916270319' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-10T11:00:49.824 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:49 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.189:0/2916270319' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-10T11:00:49.824 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:49 smithi076 ceph-mon[34152]: Standby manager daemon z started 2024-05-10T11:00:49.824 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:49 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.189:0/2916270319' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-10T11:00:49.824 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:49 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.189:0/2916270319' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-10T11:00:49.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:49 smithi027 ceph-mon[30054]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:49.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:49 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.189:0/2916270319' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-10T11:00:49.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:49 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.189:0/2916270319' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-10T11:00:49.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:49 smithi027 ceph-mon[30054]: Standby manager daemon z started 2024-05-10T11:00:49.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:49 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.189:0/2916270319' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-10T11:00:49.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:49 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.189:0/2916270319' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-10T11:00:49.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:49 smithi189 ceph-mon[34842]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:49.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:49 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.189:0/2916270319' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-10T11:00:49.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:49 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.189:0/2916270319' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-10T11:00:49.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:49 smithi189 ceph-mon[34842]: Standby manager daemon z started 2024-05-10T11:00:49.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:49 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.189:0/2916270319' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-10T11:00:49.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:49 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.189:0/2916270319' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-10T11:00:50.162 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:49.917+0000 7fbd293e7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-10T11:00:50.425 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:50 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:50.161+0000 7fbd293e7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-10T11:00:50.425 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:50 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:50.293+0000 7fbd293e7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-10T11:00:50.425 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:50 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:50.388+0000 7fbd293e7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-10T11:00:50.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:50 smithi027 ceph-mon[30054]: mgrmap e14: x(active, since 89s), standbys: z 2024-05-10T11:00:50.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:50 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-10T11:00:51.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:50 smithi076 ceph-mon[34152]: mgrmap e14: x(active, since 89s), standbys: z 2024-05-10T11:00:51.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:50 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-10T11:00:51.176 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:50 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:50.711+0000 7fbd293e7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-10T11:00:51.176 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:50 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:50.806+0000 7fbd293e7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-10T11:00:51.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:50 smithi189 ceph-mon[34842]: mgrmap e14: x(active, since 89s), standbys: z 2024-05-10T11:00:51.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:50 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-10T11:00:51.675 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:51 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:51.405+0000 7fbd293e7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-10T11:00:51.676 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:00:51 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:00:51.624+0000 7fbd293e7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-10T11:00:51.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:00:51.762+0000 7f8e86a00700 1 -- 172.21.15.27:0/87625917 <== mon.1 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f8e80064ff0 con 0x7f8e90105350 2024-05-10T11:00:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:51 smithi027 ceph-mon[30054]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:51 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.76:0/768922026' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-10T11:00:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:51 smithi027 ceph-mon[30054]: Standby manager daemon y started 2024-05-10T11:00:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:51 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.76:0/768922026' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-10T11:00:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:51 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.76:0/768922026' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-10T11:00:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:51 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.76:0/768922026' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-10T11:00:52.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:51 smithi076 ceph-mon[34152]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:52.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:51 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.76:0/768922026' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-10T11:00:52.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:51 smithi076 ceph-mon[34152]: Standby manager daemon y started 2024-05-10T11:00:52.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:51 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.76:0/768922026' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-10T11:00:52.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:51 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.76:0/768922026' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-10T11:00:52.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:51 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.76:0/768922026' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-10T11:00:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:51 smithi189 ceph-mon[34842]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:51 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.76:0/768922026' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-10T11:00:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:51 smithi189 ceph-mon[34842]: Standby manager daemon y started 2024-05-10T11:00:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:51 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.76:0/768922026' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-10T11:00:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:51 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.76:0/768922026' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-10T11:00:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:51 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.76:0/768922026' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-10T11:00:53.098 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:52 smithi189 ceph-mon[34842]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-05-10T11:00:53.098 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:52 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-10T11:00:53.099 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:52 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1196252686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df11dcf5-f3ad-4b7c-99e7-e26f7756278d"}]: dispatch 2024-05-10T11:00:53.099 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:52 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df11dcf5-f3ad-4b7c-99e7-e26f7756278d"}]: dispatch 2024-05-10T11:00:53.099 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:52 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df11dcf5-f3ad-4b7c-99e7-e26f7756278d"}]': finished 2024-05-10T11:00:53.099 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:52 smithi189 ceph-mon[34842]: osdmap e5: 1 total, 0 up, 1 in 2024-05-10T11:00:53.099 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:52 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:00:53.099 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:52 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3883381636' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:00:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:52 smithi027 ceph-mon[30054]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-05-10T11:00:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:52 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-10T11:00:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:52 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1196252686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df11dcf5-f3ad-4b7c-99e7-e26f7756278d"}]: dispatch 2024-05-10T11:00:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:52 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df11dcf5-f3ad-4b7c-99e7-e26f7756278d"}]: dispatch 2024-05-10T11:00:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:52 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df11dcf5-f3ad-4b7c-99e7-e26f7756278d"}]': finished 2024-05-10T11:00:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:52 smithi027 ceph-mon[30054]: osdmap e5: 1 total, 0 up, 1 in 2024-05-10T11:00:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:52 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:00:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:52 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3883381636' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:00:53.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:52 smithi076 ceph-mon[34152]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-05-10T11:00:53.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:52 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-10T11:00:53.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:52 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1196252686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df11dcf5-f3ad-4b7c-99e7-e26f7756278d"}]: dispatch 2024-05-10T11:00:53.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:52 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df11dcf5-f3ad-4b7c-99e7-e26f7756278d"}]: dispatch 2024-05-10T11:00:53.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:52 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df11dcf5-f3ad-4b7c-99e7-e26f7756278d"}]': finished 2024-05-10T11:00:53.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:52 smithi076 ceph-mon[34152]: osdmap e5: 1 total, 0 up, 1 in 2024-05-10T11:00:53.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:52 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:00:53.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:52 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3883381636' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:00:53.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:53 smithi076 ceph-mon[34152]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:54.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:53 smithi027 ceph-mon[30054]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:54.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:53 smithi189 ceph-mon[34842]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:56.056 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:55 smithi027 ceph-mon[30054]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:56.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:55 smithi076 ceph-mon[34152]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:56.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:55 smithi189 ceph-mon[34842]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:58.049 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:57 smithi027 ceph-mon[30054]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:58.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:57 smithi076 ceph-mon[34152]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:00:58.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:57 smithi189 ceph-mon[34842]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:00.063 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:00:59 smithi027 ceph-mon[30054]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:00.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:00:59 smithi076 ceph-mon[34152]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:00.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:00:59 smithi189 ceph-mon[34842]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:01.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:00 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-10T11:01:01.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:00 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:01.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:00 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-10T11:01:01.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:00 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:01.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:00 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-10T11:01:01.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:00 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:02.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:01 smithi027 ceph-mon[30054]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:02.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:01 smithi027 ceph-mon[30054]: Deploying daemon osd.0 on smithi027 2024-05-10T11:01:02.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:01 smithi076 ceph-mon[34152]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:02.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:01 smithi076 ceph-mon[34152]: Deploying daemon osd.0 on smithi027 2024-05-10T11:01:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:01 smithi189 ceph-mon[34842]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:01 smithi189 ceph-mon[34842]: Deploying daemon osd.0 on smithi027 2024-05-10T11:01:04.117 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:03 smithi027 ceph-mon[30054]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:04.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:03 smithi076 ceph-mon[34152]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:04.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:03 smithi189 ceph-mon[34842]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:05.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:05 smithi027 ceph-mon[30054]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:06.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:05 smithi076 ceph-mon[34152]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:06.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:05 smithi189 ceph-mon[34842]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:08.108 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:07 smithi027 ceph-mon[30054]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:08.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:07 smithi076 ceph-mon[34152]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:08.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:07 smithi189 ceph-mon[34842]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:10.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:09 smithi027 ceph-mon[30054]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:10.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:10.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:10.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:01:10.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:09 smithi076 ceph-mon[34152]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:10.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:10.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:10.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:01:10.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:09 smithi189 ceph-mon[34842]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:01:12.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:11 smithi027 ceph-mon[30054]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:12.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:11 smithi076 ceph-mon[34152]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:12.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:11 smithi189 ceph-mon[34842]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:12 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:12 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:12 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:12 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:01:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:12 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:13.247 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:12 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:13.247 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:12 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:13.247 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:12 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:13.247 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:12 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:01:13.247 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:12 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:13.353 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:12 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:13.353 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:12 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:13.353 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:12 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:13.353 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:12 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:01:13.353 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:12 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:14.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.051+0000 7f8e86a00700 1 -- 172.21.15.27:0/87625917 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8e900638e0 con 0x7f8e7403e750 2024-05-10T11:01:14.052 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 0 on host 'smithi027' 2024-05-10T11:01:14.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.056+0000 7f8e94a00700 1 -- 172.21.15.27:0/87625917 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f8e7403e750 msgr2=0x7f8e74040c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:01:14.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.056+0000 7f8e94a00700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f8e7403e750 0x7f8e74040c10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c009750 tx=0x7f8e7c009310 comp rx=0 tx=0).stop 2024-05-10T11:01:14.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.056+0000 7f8e94a00700 1 -- 172.21.15.27:0/87625917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e90105350 msgr2=0x7f8e9019b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:01:14.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.056+0000 7f8e94a00700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e90105350 0x7f8e9019b8e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000e9a0 tx=0x7f8e8000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:01:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.056+0000 7f8e94a00700 1 -- 172.21.15.27:0/87625917 shutdown_connections 2024-05-10T11:01:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.056+0000 7f8e94a00700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f8e7403e750 0x7f8e74040c10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.056+0000 7f8e94a00700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e90105350 0x7f8e9019b8e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.056+0000 7f8e94a00700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e90106e10 0x7f8e901a2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.056+0000 7f8e94a00700 1 --2- 172.21.15.27:0/87625917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e90105cf0 0x7f8e9019be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.056+0000 7f8e94a00700 1 -- 172.21.15.27:0/87625917 >> 172.21.15.27:0/87625917 conn(0x7f8e90100cf0 msgr2=0x7f8e90109bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:01:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.056+0000 7f8e94a00700 1 -- 172.21.15.27:0/87625917 shutdown_connections 2024-05-10T11:01:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:01:14.057+0000 7f8e94a00700 1 -- 172.21.15.27:0/87625917 wait complete. 2024-05-10T11:01:14.277 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:14 smithi076 ceph-mon[34152]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:14.277 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:14 smithi076 ceph-mon[34152]: from='osd.0 [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-10T11:01:14.277 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:14 smithi076 ceph-mon[34152]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-10T11:01:14.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:14 smithi027 ceph-mon[30054]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:14.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:14 smithi027 ceph-mon[30054]: from='osd.0 [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-10T11:01:14.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:14 smithi027 ceph-mon[30054]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-10T11:01:14.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:14 smithi189 ceph-mon[34842]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:14.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:14 smithi189 ceph-mon[34842]: from='osd.0 [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-10T11:01:14.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:14 smithi189 ceph-mon[34842]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-10T11:01:14.839 DEBUG:teuthology.orchestra.run.smithi027:osd.0> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.0.service 2024-05-10T11:01:14.845 INFO:tasks.cephadm:Deploying osd.1 on smithi076 with /dev/vg_nvme/lv_4... 2024-05-10T11:01:14.845 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-10T11:01:15.090 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:15 smithi027 ceph-mon[30054]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-10T11:01:15.090 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:15 smithi027 ceph-mon[30054]: from='osd.0 [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:01:15.091 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:15 smithi027 ceph-mon[30054]: osdmap e6: 1 total, 0 up, 1 in 2024-05-10T11:01:15.091 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:15 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:15.091 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:15 smithi027 ceph-mon[30054]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:01:15.091 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:15 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:01:15.091 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:15 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:15.091 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:15 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:15.095 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.b/config 2024-05-10T11:01:15.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:15 smithi076 ceph-mon[34152]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-10T11:01:15.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:15 smithi076 ceph-mon[34152]: from='osd.0 [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:01:15.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:15 smithi076 ceph-mon[34152]: osdmap e6: 1 total, 0 up, 1 in 2024-05-10T11:01:15.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:15 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:15.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:15 smithi076 ceph-mon[34152]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:01:15.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:15 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:01:15.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:15 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:15.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:15 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:15 smithi189 ceph-mon[34842]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-10T11:01:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:15 smithi189 ceph-mon[34842]: from='osd.0 [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:01:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:15 smithi189 ceph-mon[34842]: osdmap e6: 1 total, 0 up, 1 in 2024-05-10T11:01:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:15 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:15 smithi189 ceph-mon[34842]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:01:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:15 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:01:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:15 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:15 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:16.025 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:01:15 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0[44577]: 2024-05-10T11:01:15.688+0000 7fedffe00700 -1 osd.0 0 waiting for initial osdmap 2024-05-10T11:01:16.025 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:01:15 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0[44577]: 2024-05-10T11:01:15.702+0000 7fedf5c00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-10T11:01:16.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:16 smithi076 ceph-mon[34152]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:16.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:16 smithi076 ceph-mon[34152]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-10T11:01:16.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:16 smithi076 ceph-mon[34152]: osdmap e7: 1 total, 0 up, 1 in 2024-05-10T11:01:16.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:16 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:16.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:16 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:16.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:16 smithi076 ceph-mon[34152]: from='osd.0 ' entity='osd.0' 2024-05-10T11:01:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:16 smithi027 ceph-mon[30054]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:16 smithi027 ceph-mon[30054]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-10T11:01:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:16 smithi027 ceph-mon[30054]: osdmap e7: 1 total, 0 up, 1 in 2024-05-10T11:01:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:16 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:16 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:16 smithi027 ceph-mon[30054]: from='osd.0 ' entity='osd.0' 2024-05-10T11:01:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:16 smithi189 ceph-mon[34842]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:16 smithi189 ceph-mon[34842]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-10T11:01:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:16 smithi189 ceph-mon[34842]: osdmap e7: 1 total, 0 up, 1 in 2024-05-10T11:01:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:16 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:16 smithi189 ceph-mon[34842]: from='osd.0 ' entity='osd.0' 2024-05-10T11:01:16.868 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-10T11:01:16.868 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-10T11:01:16.868 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-05-10T11:01:16.869 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-05-10T11:01:16.869 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114094 s, 919 MB/s 2024-05-10T11:01:16.869 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-05-10T11:01:17.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:17 smithi027 ceph-mon[30054]: purged_snaps scrub starts 2024-05-10T11:01:17.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:17 smithi027 ceph-mon[30054]: purged_snaps scrub ok 2024-05-10T11:01:17.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:17 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:17.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:17 smithi027 ceph-mon[30054]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:17.155 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:17 smithi027 ceph-mon[30054]: osd.0 [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345] boot 2024-05-10T11:01:17.155 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:17 smithi027 ceph-mon[30054]: osdmap e8: 1 total, 1 up, 1 in 2024-05-10T11:01:17.155 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:17 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:17.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:17 smithi076 ceph-mon[34152]: purged_snaps scrub starts 2024-05-10T11:01:17.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:17 smithi076 ceph-mon[34152]: purged_snaps scrub ok 2024-05-10T11:01:17.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:17 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:17.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:17 smithi076 ceph-mon[34152]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:17.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:17 smithi076 ceph-mon[34152]: osd.0 [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345] boot 2024-05-10T11:01:17.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:17 smithi076 ceph-mon[34152]: osdmap e8: 1 total, 1 up, 1 in 2024-05-10T11:01:17.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:17 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:17.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:17 smithi189 ceph-mon[34842]: purged_snaps scrub starts 2024-05-10T11:01:17.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:17 smithi189 ceph-mon[34842]: purged_snaps scrub ok 2024-05-10T11:01:17.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:17 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:17.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:17 smithi189 ceph-mon[34842]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-10T11:01:17.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:17 smithi189 ceph-mon[34842]: osd.0 [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345] boot 2024-05-10T11:01:17.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:17 smithi189 ceph-mon[34842]: osdmap e8: 1 total, 1 up, 1 in 2024-05-10T11:01:17.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:17 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:01:17.524 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi076:vg_nvme/lv_4 2024-05-10T11:01:17.756 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.b/config 2024-05-10T11:01:19.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.228+0000 7fdacf428700 1 -- 172.21.15.76:0/3088174316 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdac8104730 msgr2=0x7fdac8104b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:01:19.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.228+0000 7fdacf428700 1 --2- 172.21.15.76:0/3088174316 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdac8104730 0x7fdac8104b90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdab80099c0 tx=0x7fdab8009cd0 comp rx=0 tx=0).stop 2024-05-10T11:01:19.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.228+0000 7fdacf428700 1 -- 172.21.15.76:0/3088174316 shutdown_connections 2024-05-10T11:01:19.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.228+0000 7fdacf428700 1 --2- 172.21.15.76:0/3088174316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac81050d0 0x7fdac810b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:19.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.228+0000 7fdacf428700 1 --2- 172.21.15.76:0/3088174316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdac8103530 0x7fdac8103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:19.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.228+0000 7fdacf428700 1 --2- 172.21.15.76:0/3088174316 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdac8104730 0x7fdac8104b90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:19.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.228+0000 7fdacf428700 1 -- 172.21.15.76:0/3088174316 >> 172.21.15.76:0/3088174316 conn(0x7fdac80feaf0 msgr2=0x7fdac8100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:01:19.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.228+0000 7fdacf428700 1 -- 172.21.15.76:0/3088174316 shutdown_connections 2024-05-10T11:01:19.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.228+0000 7fdacf428700 1 -- 172.21.15.76:0/3088174316 wait complete. 2024-05-10T11:01:19.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacf428700 1 Processor -- start 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacf428700 1 -- start start 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacf428700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac8103530 0x7fdac80771a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacf428700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdac8104730 0x7fdac8075890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacf428700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdac81050d0 0x7fdac8075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacf428700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdac810f0c0 con 0x7fdac8104730 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacf428700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdac810ef40 con 0x7fdac81050d0 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacf428700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdac810f240 con 0x7fdac8103530 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacd9c5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdac81050d0 0x7fdac8075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacd9c5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdac81050d0 0x7fdac8075dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39018/0 (socket says 172.21.15.76:39018) 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacd9c5700 1 -- 172.21.15.76:0/4116727915 learned_addr learned my addr 172.21.15.76:0/4116727915 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacd1c4700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac8103530 0x7fdac80771a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacd9c5700 1 -- 172.21.15.76:0/4116727915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac8103530 msgr2=0x7fdac80771a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:01:19.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacd9c5700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac8103530 0x7fdac80771a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:19.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacd9c5700 1 -- 172.21.15.76:0/4116727915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdac8104730 msgr2=0x7fdac8075890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:01:19.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacc9c3700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdac8104730 0x7fdac8075890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:01:19.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacd9c5700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdac8104730 0x7fdac8075890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:19.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacd9c5700 1 -- 172.21.15.76:0/4116727915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac80765e0 con 0x7fdac81050d0 2024-05-10T11:01:19.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.229+0000 7fdacc9c3700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdac8104730 0x7fdac8075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:01:19.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.230+0000 7fdacd9c5700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdac81050d0 0x7fdac8075dd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdac000c970 tx=0x7fdac000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:01:19.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.230+0000 7fdabe7fc700 1 -- 172.21.15.76:0/4116727915 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac000ce60 con 0x7fdac81050d0 2024-05-10T11:01:19.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.230+0000 7fdacf428700 1 -- 172.21.15.76:0/4116727915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac81a3830 con 0x7fdac81050d0 2024-05-10T11:01:19.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.230+0000 7fdabe7fc700 1 -- 172.21.15.76:0/4116727915 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdac0004830 con 0x7fdac81050d0 2024-05-10T11:01:19.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.230+0000 7fdabe7fc700 1 -- 172.21.15.76:0/4116727915 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac00054c0 con 0x7fdac81050d0 2024-05-10T11:01:19.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.230+0000 7fdacf428700 1 -- 172.21.15.76:0/4116727915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac81a3d90 con 0x7fdac81050d0 2024-05-10T11:01:19.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.231+0000 7fdacf428700 1 -- 172.21.15.76:0/4116727915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac806a190 con 0x7fdac81050d0 2024-05-10T11:01:19.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.232+0000 7fdabe7fc700 1 -- 172.21.15.76:0/4116727915 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7fdac00049e0 con 0x7fdac81050d0 2024-05-10T11:01:19.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.232+0000 7fdabe7fc700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdab40b2870 0x7fdab40b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:01:19.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.233+0000 7fdabe7fc700 1 -- 172.21.15.76:0/4116727915 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fdac00e0580 con 0x7fdac81050d0 2024-05-10T11:01:19.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.233+0000 7fdacd1c4700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdab40b2870 0x7fdab40b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:01:19.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.234+0000 7fdacd1c4700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdab40b2870 0x7fdab40b4d30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdac8104590 tx=0x7fdac400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:01:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.236+0000 7fdabe7fc700 1 -- 172.21.15.76:0/4116727915 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fdac0014030 con 0x7fdac81050d0 2024-05-10T11:01:19.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:19 smithi027 ceph-mon[30054]: osdmap e9: 1 total, 1 up, 1 in 2024-05-10T11:01:19.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:19 smithi027 ceph-mon[30054]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:19.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:19 smithi076 ceph-mon[34152]: osdmap e9: 1 total, 1 up, 1 in 2024-05-10T11:01:19.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:19 smithi076 ceph-mon[34152]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:19.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:19.448+0000 7fdacf428700 1 -- 172.21.15.76:0/4116727915 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fdac80638e0 con 0x7fdab40b2870 2024-05-10T11:01:19.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:19 smithi189 ceph-mon[34842]: osdmap e9: 1 total, 1 up, 1 in 2024-05-10T11:01:19.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:19 smithi189 ceph-mon[34842]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:20.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:01:20.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:01:20.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:01:20.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:20.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: Detected new or changed devices on smithi027 2024-05-10T11:01:20.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:20.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:20.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:01:20.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi027 to 17089M 2024-05-10T11:01:20.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:20.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:20.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:01:20.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:20.392 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:01:20.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:01:20.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:01:20.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:20.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: Detected new or changed devices on smithi027 2024-05-10T11:01:20.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:20.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:20.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:01:20.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi027 to 17089M 2024-05-10T11:01:20.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:20.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:20.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:01:20.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:20 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:20.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:01:20.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:01:20.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:01:20.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:20.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: Detected new or changed devices on smithi027 2024-05-10T11:01:20.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:20.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:20.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:01:20.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi027 to 17089M 2024-05-10T11:01:20.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:20.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:20.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:01:20.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:20 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:21.248 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:21 smithi076 ceph-mon[34152]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:21.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:21 smithi027 ceph-mon[30054]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:21.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:21 smithi189 ceph-mon[34842]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:22 smithi076 ceph-mon[34152]: from='client.? 172.21.15.76:0/1185186125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "207c4446-366c-45dd-8496-e1234636e6b0"}]: dispatch 2024-05-10T11:01:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:22 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "207c4446-366c-45dd-8496-e1234636e6b0"}]: dispatch 2024-05-10T11:01:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:22 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "207c4446-366c-45dd-8496-e1234636e6b0"}]': finished 2024-05-10T11:01:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:22 smithi076 ceph-mon[34152]: osdmap e10: 2 total, 1 up, 2 in 2024-05-10T11:01:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:22 smithi189 ceph-mon[34842]: from='client.? 172.21.15.76:0/1185186125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "207c4446-366c-45dd-8496-e1234636e6b0"}]: dispatch 2024-05-10T11:01:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:22 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "207c4446-366c-45dd-8496-e1234636e6b0"}]: dispatch 2024-05-10T11:01:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:22 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "207c4446-366c-45dd-8496-e1234636e6b0"}]': finished 2024-05-10T11:01:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:22 smithi189 ceph-mon[34842]: osdmap e10: 2 total, 1 up, 2 in 2024-05-10T11:01:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:22 smithi027 ceph-mon[30054]: from='client.? 172.21.15.76:0/1185186125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "207c4446-366c-45dd-8496-e1234636e6b0"}]: dispatch 2024-05-10T11:01:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:22 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "207c4446-366c-45dd-8496-e1234636e6b0"}]: dispatch 2024-05-10T11:01:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:22 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "207c4446-366c-45dd-8496-e1234636e6b0"}]': finished 2024-05-10T11:01:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:22 smithi027 ceph-mon[30054]: osdmap e10: 2 total, 1 up, 2 in 2024-05-10T11:01:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:23.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:23 smithi076 ceph-mon[34152]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:23.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:23 smithi076 ceph-mon[34152]: from='client.? 172.21.15.76:0/4186497783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:01:23.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:23 smithi189 ceph-mon[34842]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:23.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:23 smithi189 ceph-mon[34842]: from='client.? 172.21.15.76:0/4186497783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:01:23.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:23 smithi027 ceph-mon[30054]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:23.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:23 smithi027 ceph-mon[30054]: from='client.? 172.21.15.76:0/4186497783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:01:25.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:25 smithi189 ceph-mon[34842]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:26.062 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:25 smithi076 ceph-mon[34152]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:26.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:25 smithi027 ceph-mon[30054]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:27.969 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:27 smithi076 ceph-mon[34152]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:27.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:27 smithi189 ceph-mon[34842]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:28.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:27 smithi027 ceph-mon[30054]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:29.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:29 smithi189 ceph-mon[34842]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:29.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-10T11:01:29.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:30.076 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:29 smithi076 ceph-mon[34152]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:30.076 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-10T11:01:30.076 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:30.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:29 smithi027 ceph-mon[30054]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:30.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-10T11:01:30.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:30.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:30 smithi189 ceph-mon[34842]: Deploying daemon osd.1 on smithi076 2024-05-10T11:01:31.079 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:30 smithi076 ceph-mon[34152]: Deploying daemon osd.1 on smithi076 2024-05-10T11:01:31.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:30 smithi027 ceph-mon[30054]: Deploying daemon osd.1 on smithi076 2024-05-10T11:01:31.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:31 smithi189 ceph-mon[34842]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:32.079 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:31 smithi076 ceph-mon[34152]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:32.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:31 smithi027 ceph-mon[30054]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:33.871 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:33 smithi076 ceph-mon[34152]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:33.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:33 smithi189 ceph-mon[34842]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:34.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:33 smithi027 ceph-mon[30054]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:35 smithi189 ceph-mon[34842]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:35 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:35 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:01:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:35 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:36.030 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:35 smithi076 ceph-mon[34152]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:36.030 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:35 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:36.030 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:35 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:01:36.030 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:35 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:36.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:35 smithi027 ceph-mon[30054]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:36.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:35 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:36.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:35 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:01:36.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:35 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:37.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:37 smithi076 ceph-mon[34152]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:37.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:37.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:37.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:37.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:01:37.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:37.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:37 smithi189 ceph-mon[34842]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:37.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:37.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:37.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:37.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:01:37.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:38.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:37 smithi027 ceph-mon[30054]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:38.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:38.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:38.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:38.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:01:38.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:38.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.350+0000 7fdabe7fc700 1 -- 172.21.15.76:0/4116727915 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdac80638e0 con 0x7fdab40b2870 2024-05-10T11:01:38.351 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 1 on host 'smithi076' 2024-05-10T11:01:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.352+0000 7fdacf428700 1 -- 172.21.15.76:0/4116727915 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdab40b2870 msgr2=0x7fdab40b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:01:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.352+0000 7fdacf428700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdab40b2870 0x7fdab40b4d30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdac8104590 tx=0x7fdac400d040 comp rx=0 tx=0).stop 2024-05-10T11:01:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.352+0000 7fdacf428700 1 -- 172.21.15.76:0/4116727915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdac81050d0 msgr2=0x7fdac8075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:01:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.352+0000 7fdacf428700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdac81050d0 0x7fdac8075dd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdac000c970 tx=0x7fdac000cc80 comp rx=0 tx=0).stop 2024-05-10T11:01:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.352+0000 7fdacf428700 1 -- 172.21.15.76:0/4116727915 shutdown_connections 2024-05-10T11:01:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.352+0000 7fdacf428700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdab40b2870 0x7fdab40b4d30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.352+0000 7fdacf428700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac8103530 0x7fdac80771a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.352+0000 7fdacf428700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdac81050d0 0x7fdac8075dd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.352+0000 7fdacf428700 1 --2- 172.21.15.76:0/4116727915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdac8104730 0x7fdac8075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:38.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.352+0000 7fdacf428700 1 -- 172.21.15.76:0/4116727915 >> 172.21.15.76:0/4116727915 conn(0x7fdac80feaf0 msgr2=0x7fdac8107a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:01:38.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.352+0000 7fdacf428700 1 -- 172.21.15.76:0/4116727915 shutdown_connections 2024-05-10T11:01:38.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:01:38.352+0000 7fdacf428700 1 -- 172.21.15.76:0/4116727915 wait complete. 2024-05-10T11:01:38.997 DEBUG:teuthology.orchestra.run.smithi076:osd.1> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.1.service 2024-05-10T11:01:39.000 INFO:tasks.cephadm:Deploying osd.2 on smithi189 with /dev/vg_nvme/lv_4... 2024-05-10T11:01:39.000 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-10T11:01:39.244 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:01:39.574 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:39 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:39.575 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:39 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:39.575 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:39 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:01:39.575 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:39 smithi189 ceph-mon[34842]: from='osd.1 [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-10T11:01:39.575 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:39 smithi189 ceph-mon[34842]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-10T11:01:39.575 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:39 smithi189 ceph-mon[34842]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:39.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:39 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:39.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:39 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:39.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:39 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:01:39.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:39 smithi027 ceph-mon[30054]: from='osd.1 [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-10T11:01:39.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:39 smithi027 ceph-mon[30054]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-10T11:01:39.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:39 smithi027 ceph-mon[30054]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:39 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:39.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:39 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:39.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:39 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:01:39.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:39 smithi076 ceph-mon[34152]: from='osd.1 [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-10T11:01:39.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:39 smithi076 ceph-mon[34152]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-10T11:01:39.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:39 smithi076 ceph-mon[34152]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:40 smithi076 ceph-mon[34152]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-10T11:01:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:40 smithi076 ceph-mon[34152]: from='osd.1 [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:01:40.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:40 smithi076 ceph-mon[34152]: osdmap e11: 2 total, 1 up, 2 in 2024-05-10T11:01:40.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:40 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:40.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:40 smithi076 ceph-mon[34152]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:01:40.426 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:01:40 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1[38778]: 2024-05-10T11:01:40.315+0000 7f2aa11ce700 -1 osd.1 0 waiting for initial osdmap 2024-05-10T11:01:40.426 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:01:40 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1[38778]: 2024-05-10T11:01:40.321+0000 7f2a9cfc1700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-10T11:01:40.429 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:40 smithi189 ceph-mon[34842]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-10T11:01:40.429 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:40 smithi189 ceph-mon[34842]: from='osd.1 [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:01:40.429 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:40 smithi189 ceph-mon[34842]: osdmap e11: 2 total, 1 up, 2 in 2024-05-10T11:01:40.429 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:40 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:40.429 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:40 smithi189 ceph-mon[34842]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:01:40.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:40 smithi027 ceph-mon[30054]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-10T11:01:40.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:40 smithi027 ceph-mon[30054]: from='osd.1 [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:01:40.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:40 smithi027 ceph-mon[30054]: osdmap e11: 2 total, 1 up, 2 in 2024-05-10T11:01:40.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:40.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:40 smithi027 ceph-mon[30054]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:01:41.005 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-10T11:01:41.005 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-10T11:01:41.006 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-05-10T11:01:41.006 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-05-10T11:01:41.006 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114508 s, 916 MB/s 2024-05-10T11:01:41.006 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-05-10T11:01:41.605 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:41 smithi076 ceph-mon[34152]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-05-10T11:01:41.605 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:41 smithi076 ceph-mon[34152]: osdmap e12: 2 total, 1 up, 2 in 2024-05-10T11:01:41.605 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:41.605 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:41.606 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:41 smithi076 ceph-mon[34152]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:41.609 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:41 smithi189 ceph-mon[34842]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-05-10T11:01:41.609 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:41 smithi189 ceph-mon[34842]: osdmap e12: 2 total, 1 up, 2 in 2024-05-10T11:01:41.609 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:41.609 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:41.609 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:41 smithi189 ceph-mon[34842]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:41.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:41 smithi027 ceph-mon[30054]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-05-10T11:01:41.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:41 smithi027 ceph-mon[30054]: osdmap e12: 2 total, 1 up, 2 in 2024-05-10T11:01:41.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:41.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:41.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:41 smithi027 ceph-mon[30054]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-10T11:01:41.636 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_4 2024-05-10T11:01:41.881 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:01:42.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: purged_snaps scrub starts 2024-05-10T11:01:42.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: purged_snaps scrub ok 2024-05-10T11:01:42.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:42.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: osd.1 [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899] boot 2024-05-10T11:01:42.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: osdmap e13: 2 total, 2 up, 2 in 2024-05-10T11:01:42.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:42.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: Detected new or changed devices on smithi076 2024-05-10T11:01:42.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:42.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:42.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:01:42.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:42.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:42.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:01:42.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:42 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:42.631 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: purged_snaps scrub starts 2024-05-10T11:01:42.631 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: purged_snaps scrub ok 2024-05-10T11:01:42.631 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:42.631 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: osd.1 [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899] boot 2024-05-10T11:01:42.631 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: osdmap e13: 2 total, 2 up, 2 in 2024-05-10T11:01:42.631 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:42.631 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: Detected new or changed devices on smithi076 2024-05-10T11:01:42.631 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:42.631 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:42.632 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:01:42.632 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:42.632 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:42.632 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:01:42.632 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:42 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:42.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: purged_snaps scrub starts 2024-05-10T11:01:42.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: purged_snaps scrub ok 2024-05-10T11:01:42.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:42.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: osd.1 [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899] boot 2024-05-10T11:01:42.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: osdmap e13: 2 total, 2 up, 2 in 2024-05-10T11:01:42.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:01:42.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: Detected new or changed devices on smithi076 2024-05-10T11:01:42.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:42.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:42.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:01:42.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:42.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:42.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:01:42.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:42 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:43.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.363+0000 7f55d8d1e700 1 -- 172.21.15.189:0/2227182483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d4103f90 msgr2=0x7f55d4106380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:01:43.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.363+0000 7f55d8d1e700 1 --2- 172.21.15.189:0/2227182483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d4103f90 0x7f55d4106380 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00b0a0 tx=0x7f55cc00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:01:43.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.363+0000 7f55d8d1e700 1 -- 172.21.15.189:0/2227182483 shutdown_connections 2024-05-10T11:01:43.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.363+0000 7f55d8d1e700 1 --2- 172.21.15.189:0/2227182483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d4103f90 0x7f55d4106380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:43.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.363+0000 7f55d8d1e700 1 --2- 172.21.15.189:0/2227182483 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55d41091f0 0x7f55d410b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:43.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.363+0000 7f55d8d1e700 1 --2- 172.21.15.189:0/2227182483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d41068c0 0x7f55d4108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:43.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.363+0000 7f55d8d1e700 1 -- 172.21.15.189:0/2227182483 >> 172.21.15.189:0/2227182483 conn(0x7f55d40fd9a0 msgr2=0x7f55d40ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:01:43.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.363+0000 7f55d8d1e700 1 -- 172.21.15.189:0/2227182483 shutdown_connections 2024-05-10T11:01:43.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.363+0000 7f55d8d1e700 1 -- 172.21.15.189:0/2227182483 wait complete. 2024-05-10T11:01:43.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d8d1e700 1 Processor -- start 2024-05-10T11:01:43.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d8d1e700 1 -- start start 2024-05-10T11:01:43.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d8d1e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d4103f90 0x7f55d41996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:01:43.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d8d1e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d41068c0 0x7f55d4199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:01:43.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d8d1e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55d41091f0 0x7f55d41a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:01:43.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d8d1e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f55d410f070 con 0x7f55d41068c0 2024-05-10T11:01:43.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d8d1e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f55d410eef0 con 0x7f55d41091f0 2024-05-10T11:01:43.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d4103f90 0x7f55d41996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:01:43.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d8d1e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f55d410f1f0 con 0x7f55d4103f90 2024-05-10T11:01:43.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d4103f90 0x7f55d41996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47350/0 (socket says 172.21.15.189:47350) 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d259c700 1 -- 172.21.15.189:0/259238176 learned_addr learned my addr 172.21.15.189:0/259238176 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d2d9d700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55d41091f0 0x7f55d41a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d259c700 1 -- 172.21.15.189:0/259238176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55d41091f0 msgr2=0x7f55d41a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d259c700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55d41091f0 0x7f55d41a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d259c700 1 -- 172.21.15.189:0/259238176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d41068c0 msgr2=0x7f55d4199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d259c700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d41068c0 0x7f55d4199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.364+0000 7f55d259c700 1 -- 172.21.15.189:0/259238176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d41a1340 con 0x7f55d4103f90 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.365+0000 7f55d259c700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d4103f90 0x7f55d41996e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f55cc006070 tx=0x7f55cc00ce50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.365+0000 7f55cb7fe700 1 -- 172.21.15.189:0/259238176 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc021070 con 0x7f55d4103f90 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.365+0000 7f55cb7fe700 1 -- 172.21.15.189:0/259238176 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55cc007d40 con 0x7f55d4103f90 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.365+0000 7f55cb7fe700 1 -- 172.21.15.189:0/259238176 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc004890 con 0x7f55d4103f90 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.365+0000 7f55d8d1e700 1 -- 172.21.15.189:0/259238176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d41a15d0 con 0x7f55d4103f90 2024-05-10T11:01:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.365+0000 7f55d8d1e700 1 -- 172.21.15.189:0/259238176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d41a1be0 con 0x7f55d4103f90 2024-05-10T11:01:43.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.366+0000 7f55d8d1e700 1 -- 172.21.15.189:0/259238176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d406a190 con 0x7f55d4103f90 2024-05-10T11:01:43.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.367+0000 7f55cb7fe700 1 -- 172.21.15.189:0/259238176 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f55cc004020 con 0x7f55d4103f90 2024-05-10T11:01:43.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.367+0000 7f55cb7fe700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f55c00b2870 0x7f55c00b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:01:43.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.368+0000 7f55cb7fe700 1 -- 172.21.15.189:0/259238176 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f55cc00f020 con 0x7f55d4103f90 2024-05-10T11:01:43.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.368+0000 7f55d1d9b700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f55c00b2870 0x7f55c00b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:01:43.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.369+0000 7f55d1d9b700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f55c00b2870 0x7f55c00b4d30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f55bc009790 tx=0x7f55bc006d10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:01:43.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.372+0000 7f55cb7fe700 1 -- 172.21.15.189:0/259238176 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f55cc02a020 con 0x7f55d4103f90 2024-05-10T11:01:43.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:01:43.585+0000 7f55d8d1e700 1 -- 172.21.15.189:0/259238176 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f55d40638e0 con 0x7f55c00b2870 2024-05-10T11:01:43.586 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:43 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi076 to 17088M 2024-05-10T11:01:43.586 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:43 smithi189 ceph-mon[34842]: osdmap e14: 2 total, 2 up, 2 in 2024-05-10T11:01:43.586 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:43 smithi189 ceph-mon[34842]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:43.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:43 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi076 to 17088M 2024-05-10T11:01:43.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:43 smithi027 ceph-mon[30054]: osdmap e14: 2 total, 2 up, 2 in 2024-05-10T11:01:43.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:43 smithi027 ceph-mon[30054]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:43.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:43 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi076 to 17088M 2024-05-10T11:01:43.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:43 smithi076 ceph-mon[34152]: osdmap e14: 2 total, 2 up, 2 in 2024-05-10T11:01:43.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:43 smithi076 ceph-mon[34152]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:44.478 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:44 smithi189 ceph-mon[34842]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:01:44.479 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:01:44.479 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:01:44.479 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:44.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:44 smithi027 ceph-mon[30054]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:01:44.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:01:44.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:01:44.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:44.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:44 smithi076 ceph-mon[34152]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:01:44.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:01:44.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:01:44.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:45.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:45 smithi189 ceph-mon[34842]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:45.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:45 smithi076 ceph-mon[34152]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:45.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:45 smithi027 ceph-mon[30054]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:46.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:46 smithi076 ceph-mon[34152]: from='client.? 172.21.15.189:0/2423326257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9afd53b7-8054-4ff3-888b-de388da33ba2"}]: dispatch 2024-05-10T11:01:46.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:46 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9afd53b7-8054-4ff3-888b-de388da33ba2"}]: dispatch 2024-05-10T11:01:46.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:46 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9afd53b7-8054-4ff3-888b-de388da33ba2"}]': finished 2024-05-10T11:01:46.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:46 smithi076 ceph-mon[34152]: osdmap e15: 3 total, 2 up, 3 in 2024-05-10T11:01:46.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:01:46.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:46 smithi189 ceph-mon[34842]: from='client.? 172.21.15.189:0/2423326257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9afd53b7-8054-4ff3-888b-de388da33ba2"}]: dispatch 2024-05-10T11:01:46.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:46 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9afd53b7-8054-4ff3-888b-de388da33ba2"}]: dispatch 2024-05-10T11:01:46.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:46 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9afd53b7-8054-4ff3-888b-de388da33ba2"}]': finished 2024-05-10T11:01:46.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:46 smithi189 ceph-mon[34842]: osdmap e15: 3 total, 2 up, 3 in 2024-05-10T11:01:46.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:01:46.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:46 smithi027 ceph-mon[30054]: from='client.? 172.21.15.189:0/2423326257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9afd53b7-8054-4ff3-888b-de388da33ba2"}]: dispatch 2024-05-10T11:01:46.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:46 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9afd53b7-8054-4ff3-888b-de388da33ba2"}]: dispatch 2024-05-10T11:01:46.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:46 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9afd53b7-8054-4ff3-888b-de388da33ba2"}]': finished 2024-05-10T11:01:46.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:46 smithi027 ceph-mon[30054]: osdmap e15: 3 total, 2 up, 3 in 2024-05-10T11:01:46.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:01:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:47 smithi076 ceph-mon[34152]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:47 smithi076 ceph-mon[34152]: from='client.? 172.21.15.189:0/1136630246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:01:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:47 smithi189 ceph-mon[34842]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:47.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:47 smithi189 ceph-mon[34842]: from='client.? 172.21.15.189:0/1136630246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:01:47.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:47 smithi027 ceph-mon[30054]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:47.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:47 smithi027 ceph-mon[30054]: from='client.? 172.21.15.189:0/1136630246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:01:49.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:49 smithi189 ceph-mon[34842]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:50.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:49 smithi027 ceph-mon[30054]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:50.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:49 smithi076 ceph-mon[34152]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:51.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:51 smithi189 ceph-mon[34842]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:52.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:51 smithi027 ceph-mon[30054]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:52.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:51 smithi076 ceph-mon[34152]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:53.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:53 smithi189 ceph-mon[34842]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:53.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:53 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-10T11:01:53.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:53 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:54.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:53 smithi027 ceph-mon[30054]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:54.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:53 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-10T11:01:54.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:53 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:54.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:53 smithi076 ceph-mon[34152]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:54.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:53 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-10T11:01:54.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:53 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:01:54.851 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:54 smithi189 ceph-mon[34842]: Deploying daemon osd.2 on smithi189 2024-05-10T11:01:55.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:54 smithi027 ceph-mon[30054]: Deploying daemon osd.2 on smithi189 2024-05-10T11:01:55.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:54 smithi076 ceph-mon[34152]: Deploying daemon osd.2 on smithi189 2024-05-10T11:01:55.856 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:55 smithi189 ceph-mon[34842]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:56.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:55 smithi027 ceph-mon[30054]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:56.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:55 smithi076 ceph-mon[34152]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:57.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:57 smithi189 ceph-mon[34842]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:58.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:57 smithi027 ceph-mon[30054]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:58.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:57 smithi076 ceph-mon[34152]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:59.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:59 smithi027 ceph-mon[30054]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:01:59.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:59 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:59.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:59 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:01:59.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:01:59 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:02:00.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:59 smithi076 ceph-mon[34152]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:02:00.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:59 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:00.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:59 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:00.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:01:59 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:02:00.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:59 smithi189 ceph-mon[34842]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:02:00.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:59 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:00.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:59 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:00.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:01:59 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:02:01.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:01 smithi189 ceph-mon[34842]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:02:01.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:01.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:01.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:01.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:02:01.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:01 smithi027 ceph-mon[30054]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:02:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:02:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:02.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:01 smithi076 ceph-mon[34152]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:02:02.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:02.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:02.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:02.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:02:02.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:02.322 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 2 on host 'smithi189' 2024-05-10T11:02:02.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.322+0000 7f55cb7fe700 1 -- 172.21.15.189:0/259238176 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f55d40638e0 con 0x7f55c00b2870 2024-05-10T11:02:02.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.325+0000 7f55d8d1e700 1 -- 172.21.15.189:0/259238176 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f55c00b2870 msgr2=0x7f55c00b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:02:02.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.325+0000 7f55d8d1e700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f55c00b2870 0x7f55c00b4d30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f55bc009790 tx=0x7f55bc006d10 comp rx=0 tx=0).stop 2024-05-10T11:02:02.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.325+0000 7f55d8d1e700 1 -- 172.21.15.189:0/259238176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d4103f90 msgr2=0x7f55d41996e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:02:02.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.325+0000 7f55d8d1e700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d4103f90 0x7f55d41996e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f55cc006070 tx=0x7f55cc00ce50 comp rx=0 tx=0).stop 2024-05-10T11:02:02.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.325+0000 7f55d8d1e700 1 -- 172.21.15.189:0/259238176 shutdown_connections 2024-05-10T11:02:02.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.325+0000 7f55d8d1e700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f55c00b2870 0x7f55c00b4d30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:02.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.325+0000 7f55d8d1e700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d4103f90 0x7f55d41996e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:02.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.325+0000 7f55d8d1e700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55d41091f0 0x7f55d41a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:02.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.325+0000 7f55d8d1e700 1 --2- 172.21.15.189:0/259238176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d41068c0 0x7f55d4199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:02.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.325+0000 7f55d8d1e700 1 -- 172.21.15.189:0/259238176 >> 172.21.15.189:0/259238176 conn(0x7f55d40fd9a0 msgr2=0x7f55d41049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:02:02.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.325+0000 7f55d8d1e700 1 -- 172.21.15.189:0/259238176 shutdown_connections 2024-05-10T11:02:02.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:02:02.325+0000 7f55d8d1e700 1 -- 172.21.15.189:0/259238176 wait complete. 2024-05-10T11:02:02.964 DEBUG:teuthology.orchestra.run.smithi189:osd.2> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.2.service 2024-05-10T11:02:02.966 INFO:tasks.cephadm:Deploying osd.3 on smithi027 with /dev/vg_nvme/lv_3... 2024-05-10T11:02:02.967 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-10T11:02:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:02:03.601 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:03.601 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:03.601 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:02:03.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:03 smithi027 ceph-mon[30054]: from='osd.2 [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-10T11:02:03.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:03 smithi027 ceph-mon[30054]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-10T11:02:03.602 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:03 smithi027 ceph-mon[30054]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:02:03.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:03.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:02:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:03 smithi076 ceph-mon[34152]: from='osd.2 [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-10T11:02:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:03 smithi076 ceph-mon[34152]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-10T11:02:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:03 smithi076 ceph-mon[34152]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:02:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:02:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:03 smithi189 ceph-mon[34842]: from='osd.2 [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-10T11:02:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:03 smithi189 ceph-mon[34842]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-10T11:02:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:03 smithi189 ceph-mon[34842]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:02:04.444 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:04 smithi189 ceph-mon[34842]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-10T11:02:04.444 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:04 smithi189 ceph-mon[34842]: from='osd.2 [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:02:04.444 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:04 smithi189 ceph-mon[34842]: osdmap e16: 3 total, 2 up, 3 in 2024-05-10T11:02:04.444 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:04 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:04.444 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:04 smithi189 ceph-mon[34842]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:02:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:04 smithi027 ceph-mon[30054]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-10T11:02:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:04 smithi027 ceph-mon[30054]: from='osd.2 [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:02:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:04 smithi027 ceph-mon[30054]: osdmap e16: 3 total, 2 up, 3 in 2024-05-10T11:02:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:04 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:04 smithi027 ceph-mon[30054]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:02:04.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:04 smithi076 ceph-mon[34152]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-10T11:02:04.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:04 smithi076 ceph-mon[34152]: from='osd.2 [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:02:04.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:04 smithi076 ceph-mon[34152]: osdmap e16: 3 total, 2 up, 3 in 2024-05-10T11:02:04.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:04 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:04.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:04 smithi076 ceph-mon[34152]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:02:05.237 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:02:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2[40006]: 2024-05-10T11:02:04.764+0000 7fdefb853700 -1 osd.2 0 waiting for initial osdmap 2024-05-10T11:02:05.237 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:02:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2[40006]: 2024-05-10T11:02:04.769+0000 7fdef363e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-10T11:02:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:05 smithi027 ceph-mon[30054]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-10T11:02:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:05 smithi027 ceph-mon[30054]: osdmap e17: 3 total, 2 up, 3 in 2024-05-10T11:02:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:05 smithi027 ceph-mon[30054]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:02:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:05 smithi027 ceph-mon[30054]: from='osd.2 ' entity='osd.2' 2024-05-10T11:02:05.510 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:05 smithi189 ceph-mon[34842]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-10T11:02:05.510 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:05 smithi189 ceph-mon[34842]: osdmap e17: 3 total, 2 up, 3 in 2024-05-10T11:02:05.511 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:05.511 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:05.511 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:05 smithi189 ceph-mon[34842]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:02:05.511 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:05 smithi189 ceph-mon[34842]: from='osd.2 ' entity='osd.2' 2024-05-10T11:02:05.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:05 smithi076 ceph-mon[34152]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-10T11:02:05.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:05 smithi076 ceph-mon[34152]: osdmap e17: 3 total, 2 up, 3 in 2024-05-10T11:02:05.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:05.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:05.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:05 smithi076 ceph-mon[34152]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-10T11:02:05.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:05 smithi076 ceph-mon[34152]: from='osd.2 ' entity='osd.2' 2024-05-10T11:02:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-10T11:02:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-10T11:02:06.395 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-05-10T11:02:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-05-10T11:02:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0605532 s, 173 MB/s 2024-05-10T11:02:06.396 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 2024-05-10T11:02:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-05-10T11:02:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: purged_snaps scrub starts 2024-05-10T11:02:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: purged_snaps scrub ok 2024-05-10T11:02:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: osd.2 [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247] boot 2024-05-10T11:02:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: osdmap e18: 3 total, 3 up, 3 in 2024-05-10T11:02:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: Detected new or changed devices on smithi189 2024-05-10T11:02:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:02:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi189 to 17088M 2024-05-10T11:02:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:02:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:06.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: purged_snaps scrub starts 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: purged_snaps scrub ok 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: osd.2 [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247] boot 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: osdmap e18: 3 total, 3 up, 3 in 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: Detected new or changed devices on smithi189 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi189 to 17088M 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:02:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:06.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: purged_snaps scrub starts 2024-05-10T11:02:06.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: purged_snaps scrub ok 2024-05-10T11:02:06.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:06.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: osd.2 [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247] boot 2024-05-10T11:02:06.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: osdmap e18: 3 total, 3 up, 3 in 2024-05-10T11:02:06.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:02:06.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: Detected new or changed devices on smithi189 2024-05-10T11:02:06.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:06.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:06.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:02:06.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi189 to 17088M 2024-05-10T11:02:06.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:06.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:06.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:02:06.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:07.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_3 2024-05-10T11:02:07.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:07 smithi027 ceph-mon[30054]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-10T11:02:07.636 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:02:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:07 smithi076 ceph-mon[34152]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-10T11:02:07.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:07 smithi189 ceph-mon[34842]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:07.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-10T11:02:08.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:08 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-10T11:02:08.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:08 smithi027 ceph-mon[30054]: osdmap e19: 3 total, 3 up, 3 in 2024-05-10T11:02:08.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:08 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-10T11:02:08.620 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48912]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-10T11:02:08.620 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48912]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-10T11:02:08.620 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48912]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-10T11:02:08.620 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48912]: pam_unix(sudo:session): session closed for user root 2024-05-10T11:02:08.620 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48918]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-10T11:02:08.620 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48918]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-10T11:02:08.620 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48918]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-10T11:02:08.620 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48918]: pam_unix(sudo:session): session closed for user root 2024-05-10T11:02:08.641 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:08 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-10T11:02:08.641 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:08 smithi076 ceph-mon[34152]: osdmap e19: 3 total, 3 up, 3 in 2024-05-10T11:02:08.641 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:08 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-10T11:02:08.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:08 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-10T11:02:08.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:08 smithi189 ceph-mon[34842]: osdmap e19: 3 total, 3 up, 3 in 2024-05-10T11:02:08.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:08 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-10T11:02:08.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48921]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-10T11:02:08.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48921]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-10T11:02:08.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48921]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-10T11:02:08.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48921]: pam_unix(sudo:session): session closed for user root 2024-05-10T11:02:08.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48924]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-10T11:02:08.925 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:02:08 smithi076 sudo[42630]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-10T11:02:08.925 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:02:08 smithi076 sudo[42630]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-10T11:02:08.925 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:02:08 smithi076 sudo[42630]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-10T11:02:08.925 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:02:08 smithi076 sudo[42630]: pam_unix(sudo:session): session closed for user root 2024-05-10T11:02:08.926 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:02:08 smithi076 sudo[42633]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-10T11:02:08.926 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:02:08 smithi076 sudo[42633]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-10T11:02:08.926 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:02:08 smithi076 sudo[42633]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-10T11:02:08.926 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:02:08 smithi076 sudo[42633]: pam_unix(sudo:session): session closed for user root 2024-05-10T11:02:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43628]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-10T11:02:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43628]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-10T11:02:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43628]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-10T11:02:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43628]: pam_unix(sudo:session): session closed for user root 2024-05-10T11:02:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43631]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-10T11:02:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43631]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-10T11:02:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43631]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-10T11:02:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43631]: pam_unix(sudo:session): session closed for user root 2024-05-10T11:02:09.238 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43622]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-10T11:02:09.238 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43622]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-10T11:02:09.238 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43622]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-10T11:02:09.238 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43622]: pam_unix(sudo:session): session closed for user root 2024-05-10T11:02:09.238 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43625]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-10T11:02:09.238 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43625]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-10T11:02:09.238 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43625]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-10T11:02:09.238 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:02:08 smithi189 sudo[43625]: pam_unix(sudo:session): session closed for user root 2024-05-10T11:02:09.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48924]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-10T11:02:09.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48924]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-10T11:02:09.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:08 smithi027 sudo[48924]: pam_unix(sudo:session): session closed for user root 2024-05-10T11:02:09.370 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:08 smithi076 sudo[42636]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-10T11:02:09.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:08 smithi076 sudo[42636]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-10T11:02:09.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:08 smithi076 sudo[42636]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-10T11:02:09.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 sudo[42636]: pam_unix(sudo:session): session closed for user root 2024-05-10T11:02:09.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 sudo[42639]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-10T11:02:09.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 sudo[42639]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-10T11:02:09.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 sudo[42639]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-10T11:02:09.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 sudo[42639]: pam_unix(sudo:session): session closed for user root 2024-05-10T11:02:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-10T11:02:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: osdmap e20: 3 total, 3 up, 3 in 2024-05-10T11:02:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-10T11:02:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-10T11:02:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:02:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:02:09.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:02:09.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-10T11:02:09.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:02:09.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:02:09.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:02:09.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-10T11:02:09.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-10T11:02:09.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:02:09.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:02:09.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:02:09.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:09 smithi027 ceph-mon[30054]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-10T11:02:09.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-10T11:02:09.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: osdmap e20: 3 total, 3 up, 3 in 2024-05-10T11:02:09.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:09.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:09.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:09.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:09.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-10T11:02:09.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-10T11:02:09.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:02:09.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:02:09.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:02:09.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-10T11:02:09.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:02:09.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:02:09.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:02:09.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-10T11:02:09.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-10T11:02:09.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:02:09.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:02:09.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:02:09.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:09 smithi076 ceph-mon[34152]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-10T11:02:09.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-10T11:02:09.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: osdmap e20: 3 total, 3 up, 3 in 2024-05-10T11:02:09.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:09.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:09.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:09.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:09.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-10T11:02:09.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-10T11:02:09.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:02:09.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:02:09.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:02:09.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-10T11:02:09.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:02:09.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:02:09.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:02:09.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-10T11:02:09.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-10T11:02:09.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:02:09.739 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:02:09.739 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:02:09.739 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:09 smithi189 ceph-mon[34842]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-10T11:02:09.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.847+0000 7f8714c00700 1 -- 172.21.15.27:0/2039984046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8710106940 msgr2=0x7f8710106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:02:09.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.847+0000 7f8714c00700 1 --2- 172.21.15.27:0/2039984046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8710106940 0x7f8710106da0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0099c0 tx=0x7f86fc009cd0 comp rx=0 tx=0).stop 2024-05-10T11:02:09.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.847+0000 7f8714c00700 1 -- 172.21.15.27:0/2039984046 shutdown_connections 2024-05-10T11:02:09.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.847+0000 7f8714c00700 1 --2- 172.21.15.27:0/2039984046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8710105740 0x7f8710105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:09.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.847+0000 7f8714c00700 1 --2- 172.21.15.27:0/2039984046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87101072e0 0x7f871010db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:09.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.847+0000 7f8714c00700 1 --2- 172.21.15.27:0/2039984046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8710106940 0x7f8710106da0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:09.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.847+0000 7f8714c00700 1 -- 172.21.15.27:0/2039984046 >> 172.21.15.27:0/2039984046 conn(0x7f8710100d00 msgr2=0x7f8710103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:02:09.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.847+0000 7f8714c00700 1 -- 172.21.15.27:0/2039984046 shutdown_connections 2024-05-10T11:02:09.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.847+0000 7f8714c00700 1 -- 172.21.15.27:0/2039984046 wait complete. 2024-05-10T11:02:09.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.848+0000 7f8714c00700 1 Processor -- start 2024-05-10T11:02:09.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.848+0000 7f8714c00700 1 -- start start 2024-05-10T11:02:09.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.849+0000 7f8714c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8710105740 0x7f871019b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:02:09.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.849+0000 7f8714c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8710106940 0x7f871019bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:02:09.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.849+0000 7f8714c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87101072e0 0x7f87101a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:02:09.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.849+0000 7f8714c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8710111200 con 0x7f87101072e0 2024-05-10T11:02:09.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.849+0000 7f8714c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8710111080 con 0x7f8710106940 2024-05-10T11:02:09.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.849+0000 7f8714c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8710111380 con 0x7f8710105740 2024-05-10T11:02:09.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.849+0000 7f870e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87101072e0 0x7f87101a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:02:09.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.849+0000 7f870de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8710105740 0x7f871019b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:02:09.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.849+0000 7f870d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8710106940 0x7f871019bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:02:09.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.849+0000 7f870de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8710105740 0x7f871019b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:56082/0 (socket says 172.21.15.27:56082) 2024-05-10T11:02:09.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.849+0000 7f870de00700 1 -- 172.21.15.27:0/2348441338 learned_addr learned my addr 172.21.15.27:0/2348441338 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:02:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.850+0000 7f870de00700 1 -- 172.21.15.27:0/2348441338 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8710106940 msgr2=0x7f871019bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:02:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.850+0000 7f870de00700 1 --2- 172.21.15.27:0/2348441338 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8710106940 0x7f871019bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.850+0000 7f870de00700 1 -- 172.21.15.27:0/2348441338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87101072e0 msgr2=0x7f87101a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:02:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.850+0000 7f870de00700 1 --2- 172.21.15.27:0/2348441338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87101072e0 0x7f87101a2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.850+0000 7f870de00700 1 -- 172.21.15.27:0/2348441338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87101a34e0 con 0x7f8710105740 2024-05-10T11:02:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.850+0000 7f870de00700 1 --2- 172.21.15.27:0/2348441338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8710105740 0x7f871019b880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f870000e9a0 tx=0x7f870000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:02:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.850+0000 7f8706a00700 1 -- 172.21.15.27:0/2348441338 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8700004220 con 0x7f8710105740 2024-05-10T11:02:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.850+0000 7f8706a00700 1 -- 172.21.15.27:0/2348441338 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87000043c0 con 0x7f8710105740 2024-05-10T11:02:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.850+0000 7f8714c00700 1 -- 172.21.15.27:0/2348441338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87101a37d0 con 0x7f8710105740 2024-05-10T11:02:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.850+0000 7f8706a00700 1 -- 172.21.15.27:0/2348441338 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8700003c70 con 0x7f8710105740 2024-05-10T11:02:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.851+0000 7f8714c00700 1 -- 172.21.15.27:0/2348441338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87101a3d60 con 0x7f8710105740 2024-05-10T11:02:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.852+0000 7f8714c00700 1 -- 172.21.15.27:0/2348441338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f871006a190 con 0x7f8710105740 2024-05-10T11:02:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.855+0000 7f8706a00700 1 -- 172.21.15.27:0/2348441338 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f870000c7c0 con 0x7f8710105740 2024-05-10T11:02:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.856+0000 7f8706a00700 1 --2- 172.21.15.27:0/2348441338 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f86f40b2980 0x7f86f40b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:02:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.856+0000 7f8706a00700 1 -- 172.21.15.27:0/2348441338 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f87000e1040 con 0x7f8710105740 2024-05-10T11:02:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.856+0000 7f870d400700 1 --2- 172.21.15.27:0/2348441338 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f86f40b2980 0x7f86f40b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:02:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.857+0000 7f870d400700 1 --2- 172.21.15.27:0/2348441338 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f86f40b2980 0x7f86f40b4e40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f86fc000c00 tx=0x7f86fc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:02:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:09.861+0000 7f8706a00700 1 -- 172.21.15.27:0/2348441338 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f87000aa090 con 0x7f8710105740 2024-05-10T11:02:10.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:10.109+0000 7f8714c00700 1 -- 172.21.15.27:0/2348441338 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f87100638e0 con 0x7f86f40b2980 2024-05-10T11:02:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:10 smithi076 ceph-mon[34152]: osdmap e21: 3 total, 3 up, 3 in 2024-05-10T11:02:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:10 smithi076 ceph-mon[34152]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-10T11:02:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:10 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:02:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:10 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:02:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:10 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:10.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:10 smithi189 ceph-mon[34842]: osdmap e21: 3 total, 3 up, 3 in 2024-05-10T11:02:10.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:10 smithi189 ceph-mon[34842]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-10T11:02:10.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:10 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:02:10.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:10 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:02:10.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:10 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:10.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:10 smithi027 ceph-mon[30054]: osdmap e21: 3 total, 3 up, 3 in 2024-05-10T11:02:10.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:10 smithi027 ceph-mon[30054]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-10T11:02:10.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:10 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:02:10.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:10 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:02:10.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:10 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:11 smithi076 ceph-mon[34152]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:02:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:11 smithi076 ceph-mon[34152]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:11.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:11 smithi189 ceph-mon[34842]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:02:11.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:11 smithi189 ceph-mon[34842]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:11.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:11 smithi027 ceph-mon[30054]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:02:11.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:11 smithi027 ceph-mon[30054]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:13.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:13 smithi189 ceph-mon[34842]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:14.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:13 smithi027 ceph-mon[30054]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:14.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:13 smithi076 ceph-mon[34152]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:14.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:14 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1848539804' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e0125b3-a60e-45ea-9aca-73c16bd0c69e"}]: dispatch 2024-05-10T11:02:14.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:14 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e0125b3-a60e-45ea-9aca-73c16bd0c69e"}]: dispatch 2024-05-10T11:02:14.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:14 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e0125b3-a60e-45ea-9aca-73c16bd0c69e"}]': finished 2024-05-10T11:02:14.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:14 smithi189 ceph-mon[34842]: osdmap e22: 4 total, 3 up, 4 in 2024-05-10T11:02:14.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:14 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:15.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:14 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1848539804' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e0125b3-a60e-45ea-9aca-73c16bd0c69e"}]: dispatch 2024-05-10T11:02:15.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:14 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e0125b3-a60e-45ea-9aca-73c16bd0c69e"}]: dispatch 2024-05-10T11:02:15.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:14 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e0125b3-a60e-45ea-9aca-73c16bd0c69e"}]': finished 2024-05-10T11:02:15.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:14 smithi027 ceph-mon[30054]: osdmap e22: 4 total, 3 up, 4 in 2024-05-10T11:02:15.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:14 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:15.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:14 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1848539804' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e0125b3-a60e-45ea-9aca-73c16bd0c69e"}]: dispatch 2024-05-10T11:02:15.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:14 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e0125b3-a60e-45ea-9aca-73c16bd0c69e"}]: dispatch 2024-05-10T11:02:15.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:14 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e0125b3-a60e-45ea-9aca-73c16bd0c69e"}]': finished 2024-05-10T11:02:15.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:14 smithi076 ceph-mon[34152]: osdmap e22: 4 total, 3 up, 4 in 2024-05-10T11:02:15.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:14 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:15.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:15 smithi189 ceph-mon[34842]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:15.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:15 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/626329534' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:02:16.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:15 smithi027 ceph-mon[30054]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:16.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:15 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/626329534' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:02:16.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:15 smithi076 ceph-mon[34152]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:16.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:15 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/626329534' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:02:17.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:17 smithi189 ceph-mon[34842]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:18.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:17 smithi027 ceph-mon[30054]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:18.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:17 smithi076 ceph-mon[34152]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:19.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:19 smithi189 ceph-mon[34842]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:20.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:19 smithi027 ceph-mon[30054]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:20.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:19 smithi076 ceph-mon[34152]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:22.040 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:21 smithi027 ceph-mon[30054]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:22.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:21 smithi076 ceph-mon[34152]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:22.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:21 smithi189 ceph-mon[34842]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:23.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:23 smithi027 ceph-mon[30054]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:23.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:23 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-10T11:02:23.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:23 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:24.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:23 smithi076 ceph-mon[34152]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:24.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:23 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-10T11:02:24.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:23 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:24.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:23 smithi189 ceph-mon[34842]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:24.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:23 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-10T11:02:24.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:23 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:24.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:24 smithi027 ceph-mon[30054]: Deploying daemon osd.3 on smithi027 2024-05-10T11:02:25.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:24 smithi076 ceph-mon[34152]: Deploying daemon osd.3 on smithi027 2024-05-10T11:02:25.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:24 smithi189 ceph-mon[34842]: Deploying daemon osd.3 on smithi027 2024-05-10T11:02:25.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:25 smithi027 ceph-mon[30054]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:26.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:25 smithi076 ceph-mon[34152]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:26.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:25 smithi189 ceph-mon[34842]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:28.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:27 smithi027 ceph-mon[30054]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:28.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:27 smithi076 ceph-mon[34152]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:28.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:27 smithi189 ceph-mon[34842]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:30.051 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:29 smithi027 ceph-mon[30054]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:30.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:29 smithi076 ceph-mon[34152]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:29 smithi189 ceph-mon[34842]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:31.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:31 smithi027 ceph-mon[30054]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:32.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:31 smithi076 ceph-mon[34152]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:31 smithi189 ceph-mon[34842]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:34.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:33 smithi027 ceph-mon[30054]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:02:34.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:33 smithi076 ceph-mon[34152]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:34.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:34.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:34.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:02:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:33 smithi189 ceph-mon[34842]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:02:36.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:35 smithi027 ceph-mon[30054]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:36.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:35 smithi076 ceph-mon[34152]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:36.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:35 smithi189 ceph-mon[34842]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:37.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:37.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:37.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:37 smithi189 ceph-mon[34842]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:37.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:37 smithi189 ceph-mon[34842]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-10T11:02:37.619 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 3 on host 'smithi027' 2024-05-10T11:02:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.617+0000 7f8706a00700 1 -- 172.21.15.27:0/2348441338 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f87100638e0 con 0x7f86f40b2980 2024-05-10T11:02:37.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.621+0000 7f8714c00700 1 -- 172.21.15.27:0/2348441338 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f86f40b2980 msgr2=0x7f86f40b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:02:37.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.621+0000 7f8714c00700 1 --2- 172.21.15.27:0/2348441338 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f86f40b2980 0x7f86f40b4e40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f86fc000c00 tx=0x7f86fc005fb0 comp rx=0 tx=0).stop 2024-05-10T11:02:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.622+0000 7f8714c00700 1 -- 172.21.15.27:0/2348441338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8710105740 msgr2=0x7f871019b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:02:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.622+0000 7f8714c00700 1 --2- 172.21.15.27:0/2348441338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8710105740 0x7f871019b880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f870000e9a0 tx=0x7f870000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:02:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.622+0000 7f8714c00700 1 -- 172.21.15.27:0/2348441338 shutdown_connections 2024-05-10T11:02:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.622+0000 7f8714c00700 1 --2- 172.21.15.27:0/2348441338 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f86f40b2980 0x7f86f40b4e40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.622+0000 7f8714c00700 1 --2- 172.21.15.27:0/2348441338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8710105740 0x7f871019b880 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.622+0000 7f8714c00700 1 --2- 172.21.15.27:0/2348441338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87101072e0 0x7f87101a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.622+0000 7f8714c00700 1 --2- 172.21.15.27:0/2348441338 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8710106940 0x7f871019bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.622+0000 7f8714c00700 1 -- 172.21.15.27:0/2348441338 >> 172.21.15.27:0/2348441338 conn(0x7f8710100d00 msgr2=0x7f8710109cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:02:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.622+0000 7f8714c00700 1 -- 172.21.15.27:0/2348441338 shutdown_connections 2024-05-10T11:02:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:02:37.622+0000 7f8714c00700 1 -- 172.21.15.27:0/2348441338 wait complete. 2024-05-10T11:02:37.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:37.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:37.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:37 smithi076 ceph-mon[34152]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:37.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:37 smithi076 ceph-mon[34152]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-10T11:02:37.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:37.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:37.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:37 smithi027 ceph-mon[30054]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:37.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:37 smithi027 ceph-mon[30054]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-10T11:02:38.408 DEBUG:teuthology.orchestra.run.smithi027:osd.3> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.3.service 2024-05-10T11:02:38.413 INFO:tasks.cephadm:Deploying osd.4 on smithi076 with /dev/vg_nvme/lv_3... 2024-05-10T11:02:38.413 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-10T11:02:38.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:38 smithi027 ceph-mon[30054]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-10T11:02:38.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:38 smithi027 ceph-mon[30054]: osdmap e23: 4 total, 3 up, 4 in 2024-05-10T11:02:38.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:38 smithi027 ceph-mon[30054]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:02:38.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:38.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:38.622 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:38.653 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.b/config 2024-05-10T11:02:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:38 smithi189 ceph-mon[34842]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-10T11:02:38.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:38 smithi189 ceph-mon[34842]: osdmap e23: 4 total, 3 up, 4 in 2024-05-10T11:02:38.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:38 smithi189 ceph-mon[34842]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:02:38.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:38.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:38.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:38.776 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:38 smithi076 ceph-mon[34152]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-10T11:02:38.776 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:38 smithi076 ceph-mon[34152]: osdmap e23: 4 total, 3 up, 4 in 2024-05-10T11:02:38.776 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:38 smithi076 ceph-mon[34152]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:02:38.776 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:38.776 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:38.776 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:39.370 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:02:39 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3[51229]: 2024-05-10T11:02:39.085+0000 7f5c2da00700 -1 osd.3 0 waiting for initial osdmap 2024-05-10T11:02:39.370 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:02:39 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3[51229]: 2024-05-10T11:02:39.109+0000 7f5c23800700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-10T11:02:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:39 smithi189 ceph-mon[34842]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-10T11:02:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:39 smithi189 ceph-mon[34842]: osdmap e24: 4 total, 3 up, 4 in 2024-05-10T11:02:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:39 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:39 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:39 smithi189 ceph-mon[34842]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:39.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:39 smithi189 ceph-mon[34842]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' 2024-05-10T11:02:39.798 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:39 smithi076 ceph-mon[34152]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-10T11:02:39.798 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:39 smithi076 ceph-mon[34152]: osdmap e24: 4 total, 3 up, 4 in 2024-05-10T11:02:39.799 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:39 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:39.799 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:39 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:39.799 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:39 smithi076 ceph-mon[34152]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:39.799 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:39 smithi076 ceph-mon[34152]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' 2024-05-10T11:02:39.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:39 smithi027 ceph-mon[30054]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-10T11:02:39.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:39 smithi027 ceph-mon[30054]: osdmap e24: 4 total, 3 up, 4 in 2024-05-10T11:02:39.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:39 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:39.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:39 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:39.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:39 smithi027 ceph-mon[30054]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-10T11:02:39.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:39 smithi027 ceph-mon[30054]: from='osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219]' entity='osd.3' 2024-05-10T11:02:40.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:40 smithi076 ceph-mon[34152]: purged_snaps scrub starts 2024-05-10T11:02:40.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:40 smithi076 ceph-mon[34152]: purged_snaps scrub ok 2024-05-10T11:02:40.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:40 smithi076 ceph-mon[34152]: osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219] boot 2024-05-10T11:02:40.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:40 smithi076 ceph-mon[34152]: osdmap e25: 4 total, 4 up, 4 in 2024-05-10T11:02:40.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:40 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:40.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:40 smithi189 ceph-mon[34842]: purged_snaps scrub starts 2024-05-10T11:02:40.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:40 smithi189 ceph-mon[34842]: purged_snaps scrub ok 2024-05-10T11:02:40.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:40 smithi189 ceph-mon[34842]: osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219] boot 2024-05-10T11:02:40.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:40 smithi189 ceph-mon[34842]: osdmap e25: 4 total, 4 up, 4 in 2024-05-10T11:02:40.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:40 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:40.813 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-10T11:02:40.813 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-10T11:02:40.813 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-05-10T11:02:40.813 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-05-10T11:02:40.813 INFO:teuthology.orchestra.run.smithi076.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113672 s, 922 MB/s 2024-05-10T11:02:40.813 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-05-10T11:02:40.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:40 smithi027 ceph-mon[30054]: purged_snaps scrub starts 2024-05-10T11:02:40.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:40 smithi027 ceph-mon[30054]: purged_snaps scrub ok 2024-05-10T11:02:40.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:40 smithi027 ceph-mon[30054]: osd.3 [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219] boot 2024-05-10T11:02:40.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:40 smithi027 ceph-mon[30054]: osdmap e25: 4 total, 4 up, 4 in 2024-05-10T11:02:40.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:02:41.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi076:vg_nvme/lv_3 2024-05-10T11:02:41.743 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.b/config 2024-05-10T11:02:41.787 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:41 smithi076 ceph-mon[34152]: osdmap e26: 4 total, 4 up, 4 in 2024-05-10T11:02:41.787 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:41 smithi076 ceph-mon[34152]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:02:41.787 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:41 smithi076 ceph-mon[34152]: osdmap e27: 4 total, 4 up, 4 in 2024-05-10T11:02:41.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:41 smithi027 ceph-mon[30054]: osdmap e26: 4 total, 4 up, 4 in 2024-05-10T11:02:41.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:41 smithi027 ceph-mon[30054]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:02:41.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:41 smithi027 ceph-mon[30054]: osdmap e27: 4 total, 4 up, 4 in 2024-05-10T11:02:41.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:41 smithi189 ceph-mon[34842]: osdmap e26: 4 total, 4 up, 4 in 2024-05-10T11:02:41.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:41 smithi189 ceph-mon[34842]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:02:41.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:41 smithi189 ceph-mon[34842]: osdmap e27: 4 total, 4 up, 4 in 2024-05-10T11:02:43.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.397+0000 7f72d3fde700 1 -- 172.21.15.76:0/3075487563 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72cc1035f0 msgr2=0x7f72cc103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:02:43.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.397+0000 7f72d3fde700 1 --2- 172.21.15.76:0/3075487563 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72cc1035f0 0x7f72cc103a50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0099c0 tx=0x7f72bc009cd0 comp rx=0 tx=0).stop 2024-05-10T11:02:43.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.397+0000 7f72d3fde700 1 -- 172.21.15.76:0/3075487563 shutdown_connections 2024-05-10T11:02:43.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.397+0000 7f72d3fde700 1 --2- 172.21.15.76:0/3075487563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72cc1023f0 0x7f72cc102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:43.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.397+0000 7f72d3fde700 1 --2- 172.21.15.76:0/3075487563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72cc103f90 0x7f72cc10a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:43.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.397+0000 7f72d3fde700 1 --2- 172.21.15.76:0/3075487563 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72cc1035f0 0x7f72cc103a50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:43.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.397+0000 7f72d3fde700 1 -- 172.21.15.76:0/3075487563 >> 172.21.15.76:0/3075487563 conn(0x7f72cc0fd990 msgr2=0x7f72cc0ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:02:43.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.397+0000 7f72d3fde700 1 -- 172.21.15.76:0/3075487563 shutdown_connections 2024-05-10T11:02:43.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.397+0000 7f72d3fde700 1 -- 172.21.15.76:0/3075487563 wait complete. 2024-05-10T11:02:43.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.398+0000 7f72d3fde700 1 Processor -- start 2024-05-10T11:02:43.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.398+0000 7f72d3fde700 1 -- start start 2024-05-10T11:02:43.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.398+0000 7f72d3fde700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72cc1023f0 0x7f72cc198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:02:43.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.398+0000 7f72d3fde700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72cc1035f0 0x7f72cc198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:02:43.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.398+0000 7f72d3fde700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72cc103f90 0x7f72cc19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:02:43.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.398+0000 7f72d3fde700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc10dfd0 con 0x7f72cc103f90 2024-05-10T11:02:43.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.398+0000 7f72d3fde700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc10de50 con 0x7f72cc1023f0 2024-05-10T11:02:43.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.398+0000 7f72d3fde700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc10e150 con 0x7f72cc1035f0 2024-05-10T11:02:43.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.398+0000 7f72d1d7a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72cc1023f0 0x7f72cc198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:02:43.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.398+0000 7f72d1d7a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72cc1023f0 0x7f72cc198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46658/0 (socket says 172.21.15.76:46658) 2024-05-10T11:02:43.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.398+0000 7f72d1d7a700 1 -- 172.21.15.76:0/1890147863 learned_addr learned my addr 172.21.15.76:0/1890147863 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-05-10T11:02:43.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72d1579700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72cc1035f0 0x7f72cc198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:02:43.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72d1d7a700 1 -- 172.21.15.76:0/1890147863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72cc1035f0 msgr2=0x7f72cc198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:02:43.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72d1d7a700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72cc1035f0 0x7f72cc198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:43.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72d1d7a700 1 -- 172.21.15.76:0/1890147863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72cc103f90 msgr2=0x7f72cc19fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:02:43.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72d1d7a700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72cc103f90 0x7f72cc19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:02:43.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72d1d7a700 1 -- 172.21.15.76:0/1890147863 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72cc1a02d0 con 0x7f72cc1023f0 2024-05-10T11:02:43.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72d1579700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72cc1035f0 0x7f72cc198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:02:43.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72d1d7a700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72cc1023f0 0x7f72cc198670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f72c800b500 tx=0x7f72c800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:02:43.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72c2ffd700 1 -- 172.21.15.76:0/1890147863 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8015020 con 0x7f72cc1023f0 2024-05-10T11:02:43.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72c2ffd700 1 -- 172.21.15.76:0/1890147863 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72c8011ce0 con 0x7f72cc1023f0 2024-05-10T11:02:43.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72d3fde700 1 -- 172.21.15.76:0/1890147863 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72cc1a05c0 con 0x7f72cc1023f0 2024-05-10T11:02:43.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72c2ffd700 1 -- 172.21.15.76:0/1890147863 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8010d50 con 0x7f72cc1023f0 2024-05-10T11:02:43.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.399+0000 7f72d3fde700 1 -- 172.21.15.76:0/1890147863 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72cc1a0b30 con 0x7f72cc1023f0 2024-05-10T11:02:43.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.400+0000 7f72d3fde700 1 -- 172.21.15.76:0/1890147863 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72cc06a190 con 0x7f72cc1023f0 2024-05-10T11:02:43.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.401+0000 7f72c2ffd700 1 -- 172.21.15.76:0/1890147863 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f72c8018020 con 0x7f72cc1023f0 2024-05-10T11:02:43.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.402+0000 7f72c2ffd700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f72b80b2880 0x7f72b80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:02:43.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.402+0000 7f72c2ffd700 1 -- 172.21.15.76:0/1890147863 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3669+0+0 (secure 0 0 0) 0x7f72c80e0dc0 con 0x7f72cc1023f0 2024-05-10T11:02:43.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.402+0000 7f72d1579700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f72b80b2880 0x7f72b80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:02:43.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.403+0000 7f72d1579700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f72b80b2880 0x7f72b80b4d40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0038c0 tx=0x7f72bc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:02:43.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.406+0000 7f72c2ffd700 1 -- 172.21.15.76:0/1890147863 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f72c801f020 con 0x7f72cc1023f0 2024-05-10T11:02:43.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:02:43.618+0000 7f72d3fde700 1 -- 172.21.15.76:0/1890147863 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f72cc0638e0 con 0x7f72b80b2880 2024-05-10T11:02:43.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:43 smithi076 ceph-mon[34152]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-05-10T11:02:43.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:43.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:43.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:02:43.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:02:43.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:43.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:43.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:02:43.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:43.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:02:43.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:02:43.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:43.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:43 smithi189 ceph-mon[34842]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-05-10T11:02:43.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:43.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:43.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:02:43.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:02:43.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:43.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:43.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:02:43.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:43.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:02:43.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:02:43.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:44.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:43 smithi027 ceph-mon[30054]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-05-10T11:02:44.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:44.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:44.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:02:44.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:02:44.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:44.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:44.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:02:44.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:44.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:02:44.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:02:44.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:44.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:44 smithi076 ceph-mon[34152]: Detected new or changed devices on smithi027 2024-05-10T11:02:44.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:44 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi027 to 8544M 2024-05-10T11:02:44.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:44 smithi076 ceph-mon[34152]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:02:44.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:02:44.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:44.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:02:44.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:44 smithi189 ceph-mon[34842]: Detected new or changed devices on smithi027 2024-05-10T11:02:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:44 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi027 to 8544M 2024-05-10T11:02:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:44 smithi189 ceph-mon[34842]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:02:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:02:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:02:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:45.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:44 smithi027 ceph-mon[30054]: Detected new or changed devices on smithi027 2024-05-10T11:02:45.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:44 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi027 to 8544M 2024-05-10T11:02:45.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:44 smithi027 ceph-mon[30054]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:02:45.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:02:45.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:45.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:02:45.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:02:46.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:45 smithi027 ceph-mon[30054]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-10T11:02:46.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:45 smithi076 ceph-mon[34152]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-10T11:02:46.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:45 smithi189 ceph-mon[34842]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-10T11:02:47.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:46 smithi027 ceph-mon[30054]: from='client.? 172.21.15.76:0/4222200667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22993f58-2d0f-4faa-9651-02f3ff04cf15"}]: dispatch 2024-05-10T11:02:47.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:46 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22993f58-2d0f-4faa-9651-02f3ff04cf15"}]: dispatch 2024-05-10T11:02:47.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:46 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22993f58-2d0f-4faa-9651-02f3ff04cf15"}]': finished 2024-05-10T11:02:47.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:46 smithi027 ceph-mon[30054]: osdmap e28: 5 total, 4 up, 5 in 2024-05-10T11:02:47.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:46 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:02:47.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:46 smithi076 ceph-mon[34152]: from='client.? 172.21.15.76:0/4222200667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22993f58-2d0f-4faa-9651-02f3ff04cf15"}]: dispatch 2024-05-10T11:02:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:46 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22993f58-2d0f-4faa-9651-02f3ff04cf15"}]: dispatch 2024-05-10T11:02:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:46 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22993f58-2d0f-4faa-9651-02f3ff04cf15"}]': finished 2024-05-10T11:02:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:46 smithi076 ceph-mon[34152]: osdmap e28: 5 total, 4 up, 5 in 2024-05-10T11:02:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:46 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:02:47.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:46 smithi189 ceph-mon[34842]: from='client.? 172.21.15.76:0/4222200667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22993f58-2d0f-4faa-9651-02f3ff04cf15"}]: dispatch 2024-05-10T11:02:47.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:46 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22993f58-2d0f-4faa-9651-02f3ff04cf15"}]: dispatch 2024-05-10T11:02:47.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:46 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22993f58-2d0f-4faa-9651-02f3ff04cf15"}]': finished 2024-05-10T11:02:47.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:46 smithi189 ceph-mon[34842]: osdmap e28: 5 total, 4 up, 5 in 2024-05-10T11:02:47.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:46 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:02:48.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:47 smithi027 ceph-mon[30054]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-05-10T11:02:48.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:47 smithi027 ceph-mon[30054]: from='client.? 172.21.15.76:0/692107934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:02:48.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:47 smithi076 ceph-mon[34152]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-05-10T11:02:48.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:47 smithi076 ceph-mon[34152]: from='client.? 172.21.15.76:0/692107934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:02:48.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:47 smithi189 ceph-mon[34842]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-05-10T11:02:48.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:47 smithi189 ceph-mon[34842]: from='client.? 172.21.15.76:0/692107934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:02:50.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:49 smithi076 ceph-mon[34152]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-10T11:02:50.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:49 smithi189 ceph-mon[34842]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-10T11:02:50.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:49 smithi027 ceph-mon[30054]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-10T11:02:52.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:51 smithi076 ceph-mon[34152]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-05-10T11:02:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:51 smithi189 ceph-mon[34842]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-05-10T11:02:52.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:51 smithi027 ceph-mon[30054]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-05-10T11:02:54.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:53 smithi076 ceph-mon[34152]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:02:54.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:53 smithi189 ceph-mon[34842]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:02:54.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:53 smithi027 ceph-mon[30054]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:02:55.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:54 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-10T11:02:55.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:54 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:55.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:54 smithi076 ceph-mon[34152]: Deploying daemon osd.4 on smithi076 2024-05-10T11:02:55.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:54 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-10T11:02:55.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:54 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:55.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:54 smithi189 ceph-mon[34842]: Deploying daemon osd.4 on smithi076 2024-05-10T11:02:55.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:54 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-10T11:02:55.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:54 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:02:55.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:54 smithi027 ceph-mon[30054]: Deploying daemon osd.4 on smithi076 2024-05-10T11:02:56.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:55 smithi076 ceph-mon[34152]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:02:56.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:55 smithi027 ceph-mon[30054]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:02:56.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:55 smithi189 ceph-mon[34842]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:02:58.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:57 smithi076 ceph-mon[34152]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:02:58.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:57 smithi189 ceph-mon[34842]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:02:58.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:57 smithi027 ceph-mon[30054]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:00.066 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:59 smithi076 ceph-mon[34152]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:00.067 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:59 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:00.067 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:59 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:00.067 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:02:59 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:03:00.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:59 smithi189 ceph-mon[34842]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:00.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:59 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:00.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:59 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:00.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:02:59 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:03:00.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:59 smithi027 ceph-mon[30054]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:00.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:59 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:00.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:59 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:00.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:02:59 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:03:02.072 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:01 smithi076 ceph-mon[34152]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:01 smithi189 ceph-mon[34842]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:02.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:01 smithi027 ceph-mon[30054]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:03.126 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:03.126 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:03.126 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:03.126 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:03:03.126 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:03.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.204+0000 7f72c2ffd700 1 -- 172.21.15.76:0/1890147863 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f72cc0638e0 con 0x7f72b80b2880 2024-05-10T11:03:03.205 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 4 on host 'smithi076' 2024-05-10T11:03:03.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.206+0000 7f72d3fde700 1 -- 172.21.15.76:0/1890147863 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f72b80b2880 msgr2=0x7f72b80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:03:03.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.206+0000 7f72d3fde700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f72b80b2880 0x7f72b80b4d40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0038c0 tx=0x7f72bc000bc0 comp rx=0 tx=0).stop 2024-05-10T11:03:03.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.206+0000 7f72d3fde700 1 -- 172.21.15.76:0/1890147863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72cc1023f0 msgr2=0x7f72cc198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:03:03.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.206+0000 7f72d3fde700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72cc1023f0 0x7f72cc198670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f72c800b500 tx=0x7f72c800b8c0 comp rx=0 tx=0).stop 2024-05-10T11:03:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.206+0000 7f72d3fde700 1 -- 172.21.15.76:0/1890147863 shutdown_connections 2024-05-10T11:03:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.206+0000 7f72d3fde700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f72b80b2880 0x7f72b80b4d40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.206+0000 7f72d3fde700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72cc103f90 0x7f72cc19fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.206+0000 7f72d3fde700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72cc1023f0 0x7f72cc198670 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.206+0000 7f72d3fde700 1 --2- 172.21.15.76:0/1890147863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72cc1035f0 0x7f72cc198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.206+0000 7f72d3fde700 1 -- 172.21.15.76:0/1890147863 >> 172.21.15.76:0/1890147863 conn(0x7f72cc0fd990 msgr2=0x7f72cc1069c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:03:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.207+0000 7f72d3fde700 1 -- 172.21.15.76:0/1890147863 shutdown_connections 2024-05-10T11:03:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:03:03.207+0000 7f72d3fde700 1 -- 172.21.15.76:0/1890147863 wait complete. 2024-05-10T11:03:03.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:03:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:03:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:03.855 DEBUG:teuthology.orchestra.run.smithi076:osd.4> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.4.service 2024-05-10T11:03:03.857 INFO:tasks.cephadm:Deploying osd.5 on smithi189 with /dev/vg_nvme/lv_3... 2024-05-10T11:03:03.857 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-10T11:03:04.101 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:03:04.166 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:04 smithi076 ceph-mon[34152]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:04.166 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:04 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:04.166 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:04 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:03:04.166 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:04 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:04.166 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:04 smithi076 ceph-mon[34152]: from='osd.4 [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-10T11:03:04.166 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:04 smithi076 ceph-mon[34152]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-10T11:03:04.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:04 smithi189 ceph-mon[34842]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:04.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:04 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:04.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:04 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:03:04.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:04 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:04.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:04 smithi189 ceph-mon[34842]: from='osd.4 [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-10T11:03:04.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:04 smithi189 ceph-mon[34842]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-10T11:03:04.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:04 smithi027 ceph-mon[30054]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:04.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:04 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:04.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:04 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:03:04.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:04 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:04.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:04 smithi027 ceph-mon[30054]: from='osd.4 [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-10T11:03:04.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:04 smithi027 ceph-mon[30054]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-10T11:03:05.352 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:05 smithi076 ceph-mon[34152]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-10T11:03:05.352 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:05 smithi076 ceph-mon[34152]: from='osd.4 [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:03:05.352 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:05 smithi076 ceph-mon[34152]: osdmap e29: 5 total, 4 up, 5 in 2024-05-10T11:03:05.352 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:05.352 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:05 smithi076 ceph-mon[34152]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:03:05.352 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:05 smithi076 ceph-mon[34152]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:05.353 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:03:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4[45477]: 2024-05-10T11:03:05.192+0000 7fca0a36d700 -1 osd.4 0 waiting for initial osdmap 2024-05-10T11:03:05.353 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:03:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4[45477]: 2024-05-10T11:03:05.198+0000 7fca06160700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-10T11:03:05.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:05 smithi189 ceph-mon[34842]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-10T11:03:05.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:05 smithi189 ceph-mon[34842]: from='osd.4 [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:03:05.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:05 smithi189 ceph-mon[34842]: osdmap e29: 5 total, 4 up, 5 in 2024-05-10T11:03:05.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:05.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:05 smithi189 ceph-mon[34842]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:03:05.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:05 smithi189 ceph-mon[34842]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:05 smithi027 ceph-mon[30054]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-10T11:03:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:05 smithi027 ceph-mon[30054]: from='osd.4 [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:03:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:05 smithi027 ceph-mon[30054]: osdmap e29: 5 total, 4 up, 5 in 2024-05-10T11:03:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:05 smithi027 ceph-mon[30054]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:03:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:05 smithi027 ceph-mon[30054]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:06.099 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-10T11:03:06.099 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-10T11:03:06.099 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-05-10T11:03:06.099 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-05-10T11:03:06.099 INFO:teuthology.orchestra.run.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113713 s, 922 MB/s 2024-05-10T11:03:06.100 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-05-10T11:03:06.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:06 smithi189 ceph-mon[34842]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-05-10T11:03:06.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:06 smithi189 ceph-mon[34842]: osdmap e30: 5 total, 4 up, 5 in 2024-05-10T11:03:06.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:06.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:06.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:06.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:06 smithi076 ceph-mon[34152]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-05-10T11:03:06.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:06 smithi076 ceph-mon[34152]: osdmap e30: 5 total, 4 up, 5 in 2024-05-10T11:03:06.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:06.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:06.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:06 smithi027 ceph-mon[30054]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-05-10T11:03:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:06 smithi027 ceph-mon[30054]: osdmap e30: 5 total, 4 up, 5 in 2024-05-10T11:03:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:06.789 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_3 2024-05-10T11:03:07.035 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:03:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: purged_snaps scrub starts 2024-05-10T11:03:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: purged_snaps scrub ok 2024-05-10T11:03:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: osd.4 [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823] boot 2024-05-10T11:03:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: osdmap e31: 5 total, 5 up, 5 in 2024-05-10T11:03:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: Detected new or changed devices on smithi076 2024-05-10T11:03:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:07.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:03:07.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:03:07.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi076 to 8544M 2024-05-10T11:03:07.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:07.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:07.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:03:07.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: purged_snaps scrub starts 2024-05-10T11:03:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: purged_snaps scrub ok 2024-05-10T11:03:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: osd.4 [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823] boot 2024-05-10T11:03:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: osdmap e31: 5 total, 5 up, 5 in 2024-05-10T11:03:07.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:07.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:07.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: Detected new or changed devices on smithi076 2024-05-10T11:03:07.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:07.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:07.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:03:07.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:03:07.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi076 to 8544M 2024-05-10T11:03:07.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:07.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:07.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:03:07.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: purged_snaps scrub starts 2024-05-10T11:03:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: purged_snaps scrub ok 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: osd.4 [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823] boot 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: osdmap e31: 5 total, 5 up, 5 in 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: Detected new or changed devices on smithi076 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi076 to 8544M 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:03:07.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:08.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:08 smithi189 ceph-mon[34842]: osdmap e32: 5 total, 5 up, 5 in 2024-05-10T11:03:08.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:08 smithi027 ceph-mon[30054]: osdmap e32: 5 total, 5 up, 5 in 2024-05-10T11:03:08.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:08 smithi076 ceph-mon[34152]: osdmap e32: 5 total, 5 up, 5 in 2024-05-10T11:03:08.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.711+0000 7f23240fe700 1 -- 172.21.15.189:0/2489512922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f231c102580 msgr2=0x7f231c102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:03:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.711+0000 7f23240fe700 1 --2- 172.21.15.189:0/2489512922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f231c102580 0x7f231c102a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f230c009990 tx=0x7f230c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:03:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.711+0000 7f23240fe700 1 -- 172.21.15.189:0/2489512922 shutdown_connections 2024-05-10T11:03:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.711+0000 7f23240fe700 1 --2- 172.21.15.189:0/2489512922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f231c102f40 0x7f231c1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.711+0000 7f23240fe700 1 --2- 172.21.15.189:0/2489512922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f231c101810 0x7f231c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.711+0000 7f23240fe700 1 --2- 172.21.15.189:0/2489512922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f231c102580 0x7f231c102a00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.711+0000 7f23240fe700 1 -- 172.21.15.189:0/2489512922 >> 172.21.15.189:0/2489512922 conn(0x7f231c0fd170 msgr2=0x7f231c0ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:03:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.711+0000 7f23240fe700 1 -- 172.21.15.189:0/2489512922 shutdown_connections 2024-05-10T11:03:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.711+0000 7f23240fe700 1 -- 172.21.15.189:0/2489512922 wait complete. 2024-05-10T11:03:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.711+0000 7f23240fe700 1 Processor -- start 2024-05-10T11:03:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.711+0000 7f23240fe700 1 -- start start 2024-05-10T11:03:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f23240fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f231c101810 0x7f231c19fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:03:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f23240fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f231c102580 0x7f231c1a0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f23240fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f231c102f40 0x7f231c1a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f23240fe700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f231c115660 con 0x7f231c101810 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f23240fe700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f231c1154e0 con 0x7f231c102580 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f23240fe700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f231c1157e0 con 0x7f231c102f40 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f232269b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f231c102f40 0x7f231c1a71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f232269b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f231c102f40 0x7f231c1a71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44254/0 (socket says 172.21.15.189:44254) 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f232269b700 1 -- 172.21.15.189:0/3234212968 learned_addr learned my addr 172.21.15.189:0/3234212968 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f232269b700 1 -- 172.21.15.189:0/3234212968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f231c102580 msgr2=0x7f231c1a0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f232269b700 1 --2- 172.21.15.189:0/3234212968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f231c102580 0x7f231c1a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f232269b700 1 -- 172.21.15.189:0/3234212968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f231c101810 msgr2=0x7f231c19fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f232269b700 1 --2- 172.21.15.189:0/3234212968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f231c101810 0x7f231c19fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f232269b700 1 -- 172.21.15.189:0/3234212968 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f231c1a78c0 con 0x7f231c102f40 2024-05-10T11:03:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f232269b700 1 --2- 172.21.15.189:0/3234212968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f231c102f40 0x7f231c1a71f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f231400b970 tx=0x7f231400bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:03:08.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f2312ffd700 1 -- 172.21.15.189:0/3234212968 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314019070 con 0x7f231c102f40 2024-05-10T11:03:08.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f2312ffd700 1 -- 172.21.15.189:0/3234212968 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2314014440 con 0x7f231c102f40 2024-05-10T11:03:08.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f23240fe700 1 -- 172.21.15.189:0/3234212968 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f231c1a7bb0 con 0x7f231c102f40 2024-05-10T11:03:08.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f2312ffd700 1 -- 172.21.15.189:0/3234212968 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231400ce60 con 0x7f231c102f40 2024-05-10T11:03:08.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.712+0000 7f23240fe700 1 -- 172.21.15.189:0/3234212968 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f231c1a8170 con 0x7f231c102f40 2024-05-10T11:03:08.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.713+0000 7f23240fe700 1 -- 172.21.15.189:0/3234212968 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f231c06a190 con 0x7f231c102f40 2024-05-10T11:03:08.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.715+0000 7f2312ffd700 1 -- 172.21.15.189:0/3234212968 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f231401c070 con 0x7f231c102f40 2024-05-10T11:03:08.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.715+0000 7f2312ffd700 1 --2- 172.21.15.189:0/3234212968 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f23080b2900 0x7f23080b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:03:08.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.715+0000 7f2312ffd700 1 -- 172.21.15.189:0/3234212968 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4101+0+0 (secure 0 0 0) 0x7f23140e09f0 con 0x7f231c102f40 2024-05-10T11:03:08.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.716+0000 7f2321e9a700 1 --2- 172.21.15.189:0/3234212968 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f23080b2900 0x7f23080b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:03:08.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.717+0000 7f2321e9a700 1 --2- 172.21.15.189:0/3234212968 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f23080b2900 0x7f23080b4dc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2318005950 tx=0x7f23180058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:03:08.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.719+0000 7f2312ffd700 1 -- 172.21.15.189:0/3234212968 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f23140a9780 con 0x7f231c102f40 2024-05-10T11:03:08.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:08.927+0000 7f23240fe700 1 -- 172.21.15.189:0/3234212968 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f231c0638e0 con 0x7f23080b2900 2024-05-10T11:03:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:09 smithi189 ceph-mon[34842]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:09 smithi189 ceph-mon[34842]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:03:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:03:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:03:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:09.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:09 smithi027 ceph-mon[30054]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:09.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:09 smithi027 ceph-mon[30054]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:03:09.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:03:09.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:03:09.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:09 smithi076 ceph-mon[34152]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:09 smithi076 ceph-mon[34152]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:03:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:03:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:03:09.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:12.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:11 smithi027 ceph-mon[30054]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:12.120 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:11 smithi189 ceph-mon[34842]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:12.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:11 smithi076 ceph-mon[34152]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:12 smithi027 ceph-mon[30054]: from='client.? 172.21.15.189:0/3783424163' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5740d97-7270-4869-b523-b6e3c15e5127"}]: dispatch 2024-05-10T11:03:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:12 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5740d97-7270-4869-b523-b6e3c15e5127"}]: dispatch 2024-05-10T11:03:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:12 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5740d97-7270-4869-b523-b6e3c15e5127"}]': finished 2024-05-10T11:03:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:12 smithi027 ceph-mon[30054]: osdmap e33: 6 total, 5 up, 6 in 2024-05-10T11:03:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:12 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:03:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:12 smithi027 ceph-mon[30054]: from='client.? 172.21.15.189:0/732767873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:03:13.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:12 smithi076 ceph-mon[34152]: from='client.? 172.21.15.189:0/3783424163' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5740d97-7270-4869-b523-b6e3c15e5127"}]: dispatch 2024-05-10T11:03:13.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:12 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5740d97-7270-4869-b523-b6e3c15e5127"}]: dispatch 2024-05-10T11:03:13.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:12 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5740d97-7270-4869-b523-b6e3c15e5127"}]': finished 2024-05-10T11:03:13.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:12 smithi076 ceph-mon[34152]: osdmap e33: 6 total, 5 up, 6 in 2024-05-10T11:03:13.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:12 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:03:13.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:12 smithi076 ceph-mon[34152]: from='client.? 172.21.15.189:0/732767873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:03:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:12 smithi189 ceph-mon[34842]: from='client.? 172.21.15.189:0/3783424163' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5740d97-7270-4869-b523-b6e3c15e5127"}]: dispatch 2024-05-10T11:03:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:12 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5740d97-7270-4869-b523-b6e3c15e5127"}]: dispatch 2024-05-10T11:03:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:12 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5740d97-7270-4869-b523-b6e3c15e5127"}]': finished 2024-05-10T11:03:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:12 smithi189 ceph-mon[34842]: osdmap e33: 6 total, 5 up, 6 in 2024-05-10T11:03:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:12 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:03:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:12 smithi189 ceph-mon[34842]: from='client.? 172.21.15.189:0/732767873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:03:14.076 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:13 smithi076 ceph-mon[34152]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:14.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:13 smithi027 ceph-mon[30054]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:14.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:13 smithi189 ceph-mon[34842]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:16.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:15 smithi027 ceph-mon[30054]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:16.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:15 smithi076 ceph-mon[34152]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:16.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:15 smithi189 ceph-mon[34842]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:18.083 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:17 smithi189 ceph-mon[34842]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:18.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:17 smithi027 ceph-mon[30054]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:18.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:17 smithi076 ceph-mon[34152]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:19.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:19 smithi027 ceph-mon[30054]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:19.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:19 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-10T11:03:19.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:19 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:20.101 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:19 smithi189 ceph-mon[34842]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:20.101 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:19 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-10T11:03:20.101 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:19 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:20.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:19 smithi076 ceph-mon[34152]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:20.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:19 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-10T11:03:20.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:19 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:20.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:20 smithi189 ceph-mon[34842]: Deploying daemon osd.5 on smithi189 2024-05-10T11:03:21.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:20 smithi027 ceph-mon[30054]: Deploying daemon osd.5 on smithi189 2024-05-10T11:03:21.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:20 smithi076 ceph-mon[34152]: Deploying daemon osd.5 on smithi189 2024-05-10T11:03:22.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:21 smithi027 ceph-mon[30054]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:22.145 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:21 smithi189 ceph-mon[34842]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:22.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:21 smithi076 ceph-mon[34152]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:23.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:23 smithi189 ceph-mon[34842]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:24.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:23 smithi027 ceph-mon[30054]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:24.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:23 smithi076 ceph-mon[34152]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:26.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:26 smithi189 ceph-mon[34842]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:26.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:26 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:26.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:26 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:26.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:26 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:03:26.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:26 smithi027 ceph-mon[30054]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:26.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:26 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:26.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:26 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:26.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:26 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:03:26.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:26 smithi076 ceph-mon[34152]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:26.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:26 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:26.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:26 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:26.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:26 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:03:28.156 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:28 smithi189 ceph-mon[34842]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:28.156 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:28 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:28.156 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:28 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:28.157 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:28 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:28.157 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:28 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:03:28.157 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:28 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:28 smithi027 ceph-mon[30054]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:28 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:28 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:28 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:28 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:03:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:28 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:28.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:28 smithi076 ceph-mon[34152]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:28.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:28 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:28.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:28 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:28.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:28 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:28.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:28 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:03:28.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:28 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:28.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.968+0000 7f2312ffd700 1 -- 172.21.15.189:0/3234212968 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f231c0638e0 con 0x7f23080b2900 2024-05-10T11:03:28.969 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 5 on host 'smithi189' 2024-05-10T11:03:28.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.971+0000 7f23240fe700 1 -- 172.21.15.189:0/3234212968 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f23080b2900 msgr2=0x7f23080b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:03:28.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.971+0000 7f23240fe700 1 --2- 172.21.15.189:0/3234212968 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f23080b2900 0x7f23080b4dc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2318005950 tx=0x7f23180058e0 comp rx=0 tx=0).stop 2024-05-10T11:03:28.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.971+0000 7f23240fe700 1 -- 172.21.15.189:0/3234212968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f231c102f40 msgr2=0x7f231c1a71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:03:28.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.971+0000 7f23240fe700 1 --2- 172.21.15.189:0/3234212968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f231c102f40 0x7f231c1a71f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f231400b970 tx=0x7f231400bd30 comp rx=0 tx=0).stop 2024-05-10T11:03:28.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.971+0000 7f23240fe700 1 -- 172.21.15.189:0/3234212968 shutdown_connections 2024-05-10T11:03:28.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.971+0000 7f23240fe700 1 --2- 172.21.15.189:0/3234212968 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f23080b2900 0x7f23080b4dc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:28.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.971+0000 7f23240fe700 1 --2- 172.21.15.189:0/3234212968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f231c102f40 0x7f231c1a71f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:28.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.971+0000 7f23240fe700 1 --2- 172.21.15.189:0/3234212968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f231c101810 0x7f231c19fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:28.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.971+0000 7f23240fe700 1 --2- 172.21.15.189:0/3234212968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f231c102580 0x7f231c1a0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:28.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.971+0000 7f23240fe700 1 -- 172.21.15.189:0/3234212968 >> 172.21.15.189:0/3234212968 conn(0x7f231c0fd170 msgr2=0x7f231c1037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:03:28.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.971+0000 7f23240fe700 1 -- 172.21.15.189:0/3234212968 shutdown_connections 2024-05-10T11:03:28.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:03:28.971+0000 7f23240fe700 1 -- 172.21.15.189:0/3234212968 wait complete. 2024-05-10T11:03:29.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:29 smithi189 ceph-mon[34842]: from='osd.5 [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-10T11:03:29.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:29 smithi189 ceph-mon[34842]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-10T11:03:29.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:29.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:29.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:03:29.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:29 smithi027 ceph-mon[30054]: from='osd.5 [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-10T11:03:29.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:29 smithi027 ceph-mon[30054]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-10T11:03:29.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:29.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:29.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:03:29.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:29 smithi076 ceph-mon[34152]: from='osd.5 [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-10T11:03:29.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:29 smithi076 ceph-mon[34152]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-10T11:03:29.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:29.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:29.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:03:29.633 DEBUG:teuthology.orchestra.run.smithi189:osd.5> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.5.service 2024-05-10T11:03:29.636 INFO:tasks.cephadm:Deploying osd.6 on smithi027 with /dev/vg_nvme/lv_2... 2024-05-10T11:03:29.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-10T11:03:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:03:30.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:30 smithi189 ceph-mon[34842]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:30 smithi189 ceph-mon[34842]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-10T11:03:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:30 smithi189 ceph-mon[34842]: from='osd.5 [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:03:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:30 smithi189 ceph-mon[34842]: osdmap e34: 6 total, 5 up, 6 in 2024-05-10T11:03:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:03:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:30 smithi189 ceph-mon[34842]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:03:30.326 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:30 smithi027 ceph-mon[30054]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:30.327 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:30 smithi027 ceph-mon[30054]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-10T11:03:30.327 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:30 smithi027 ceph-mon[30054]: from='osd.5 [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:03:30.327 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:30 smithi027 ceph-mon[30054]: osdmap e34: 6 total, 5 up, 6 in 2024-05-10T11:03:30.327 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:03:30.327 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:30 smithi027 ceph-mon[30054]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:03:30.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:30 smithi076 ceph-mon[34152]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:30.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:30 smithi076 ceph-mon[34152]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-10T11:03:30.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:30 smithi076 ceph-mon[34152]: from='osd.5 [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:03:30.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:30 smithi076 ceph-mon[34152]: osdmap e34: 6 total, 5 up, 6 in 2024-05-10T11:03:30.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:30 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:03:30.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:30 smithi076 ceph-mon[34152]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:03:30.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:03:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:03:30.436+0000 7f1a7b35d700 -1 osd.5 0 waiting for initial osdmap 2024-05-10T11:03:30.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:03:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:03:30.441+0000 7f1a72947700 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-10T11:03:31.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:31 smithi189 ceph-mon[34842]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-10T11:03:31.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:31 smithi189 ceph-mon[34842]: osdmap e35: 6 total, 5 up, 6 in 2024-05-10T11:03:31.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:03:31.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:31 smithi189 ceph-mon[34842]: from='osd.5 ' entity='osd.5' 2024-05-10T11:03:31.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:31 smithi189 ceph-mon[34842]: osd.5 [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682] boot 2024-05-10T11:03:31.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:31 smithi189 ceph-mon[34842]: osdmap e36: 6 total, 6 up, 6 in 2024-05-10T11:03:31.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:03:31.338 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:31 smithi027 ceph-mon[30054]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-10T11:03:31.338 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:31 smithi027 ceph-mon[30054]: osdmap e35: 6 total, 5 up, 6 in 2024-05-10T11:03:31.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:03:31.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:31 smithi027 ceph-mon[30054]: from='osd.5 ' entity='osd.5' 2024-05-10T11:03:31.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:31 smithi027 ceph-mon[30054]: osd.5 [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682] boot 2024-05-10T11:03:31.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:31 smithi027 ceph-mon[30054]: osdmap e36: 6 total, 6 up, 6 in 2024-05-10T11:03:31.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:03:31.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:31 smithi076 ceph-mon[34152]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-10T11:03:31.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:31 smithi076 ceph-mon[34152]: osdmap e35: 6 total, 5 up, 6 in 2024-05-10T11:03:31.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:03:31.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:31 smithi076 ceph-mon[34152]: from='osd.5 ' entity='osd.5' 2024-05-10T11:03:31.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:31 smithi076 ceph-mon[34152]: osd.5 [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682] boot 2024-05-10T11:03:31.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:31 smithi076 ceph-mon[34152]: osdmap e36: 6 total, 6 up, 6 in 2024-05-10T11:03:31.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:03:32.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:32 smithi027 ceph-mon[30054]: purged_snaps scrub starts 2024-05-10T11:03:32.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:32 smithi027 ceph-mon[30054]: purged_snaps scrub ok 2024-05-10T11:03:32.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:32 smithi027 ceph-mon[30054]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:32.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:32 smithi027 ceph-mon[30054]: osdmap e37: 6 total, 6 up, 6 in 2024-05-10T11:03:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:32 smithi189 ceph-mon[34842]: purged_snaps scrub starts 2024-05-10T11:03:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:32 smithi189 ceph-mon[34842]: purged_snaps scrub ok 2024-05-10T11:03:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:32 smithi189 ceph-mon[34842]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:32 smithi189 ceph-mon[34842]: osdmap e37: 6 total, 6 up, 6 in 2024-05-10T11:03:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:32 smithi076 ceph-mon[34152]: purged_snaps scrub starts 2024-05-10T11:03:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:32 smithi076 ceph-mon[34152]: purged_snaps scrub ok 2024-05-10T11:03:32.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:32 smithi076 ceph-mon[34152]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-10T11:03:32.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:32 smithi076 ceph-mon[34152]: osdmap e37: 6 total, 6 up, 6 in 2024-05-10T11:03:33.331 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-10T11:03:33.331 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-10T11:03:33.331 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-05-10T11:03:33.331 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-05-10T11:03:33.331 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0385015 s, 272 MB/s 2024-05-10T11:03:33.331 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-05-10T11:03:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:33 smithi027 ceph-mon[30054]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:33 smithi027 ceph-mon[30054]: Detected new or changed devices on smithi189 2024-05-10T11:03:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:03:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:03:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:33 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi189 to 8544M 2024-05-10T11:03:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:03:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:33 smithi027 ceph-mon[30054]: osdmap e38: 6 total, 6 up, 6 in 2024-05-10T11:03:34.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:33 smithi076 ceph-mon[34152]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:33 smithi076 ceph-mon[34152]: Detected new or changed devices on smithi189 2024-05-10T11:03:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:03:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:03:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:33 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi189 to 8544M 2024-05-10T11:03:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:03:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:33 smithi076 ceph-mon[34152]: osdmap e38: 6 total, 6 up, 6 in 2024-05-10T11:03:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:33 smithi189 ceph-mon[34842]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:33 smithi189 ceph-mon[34842]: Detected new or changed devices on smithi189 2024-05-10T11:03:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:03:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:03:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:33 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi189 to 8544M 2024-05-10T11:03:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:03:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:03:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:33 smithi189 ceph-mon[34842]: osdmap e38: 6 total, 6 up, 6 in 2024-05-10T11:03:34.298 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_2 2024-05-10T11:03:34.767 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:03:36.013 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:35 smithi027 ceph-mon[30054]: pgmap v155: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:36.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:35 smithi076 ceph-mon[34152]: pgmap v155: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:36.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:35 smithi189 ceph-mon[34842]: pgmap v155: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:37.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.075+0000 7f43ef200700 1 -- 172.21.15.27:0/3433323335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43e8105350 msgr2=0x7f43e81057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:03:37.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.075+0000 7f43ef200700 1 --2- 172.21.15.27:0/3433323335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43e8105350 0x7f43e81057b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f43d40099c0 tx=0x7f43d4009cd0 comp rx=0 tx=0).stop 2024-05-10T11:03:37.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.075+0000 7f43ef200700 1 -- 172.21.15.27:0/3433323335 shutdown_connections 2024-05-10T11:03:37.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.075+0000 7f43ef200700 1 --2- 172.21.15.27:0/3433323335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e8105cf0 0x7f43e810db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:37.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.075+0000 7f43ef200700 1 --2- 172.21.15.27:0/3433323335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e8106e10 0x7f43e8107230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:37.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.075+0000 7f43ef200700 1 --2- 172.21.15.27:0/3433323335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43e8105350 0x7f43e81057b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:37.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.075+0000 7f43ef200700 1 -- 172.21.15.27:0/3433323335 >> 172.21.15.27:0/3433323335 conn(0x7f43e8100cf0 msgr2=0x7f43e8103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:03:37.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.075+0000 7f43ef200700 1 -- 172.21.15.27:0/3433323335 shutdown_connections 2024-05-10T11:03:37.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.075+0000 7f43ef200700 1 -- 172.21.15.27:0/3433323335 wait complete. 2024-05-10T11:03:37.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.077+0000 7f43ef200700 1 Processor -- start 2024-05-10T11:03:37.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.077+0000 7f43ef200700 1 -- start start 2024-05-10T11:03:37.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.077+0000 7f43ef200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43e8105350 0x7f43e819b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:03:37.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.077+0000 7f43ef200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e8105cf0 0x7f43e819be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:03:37.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43ef200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e8106e10 0x7f43e81a2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:03:37.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43ef200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f43e81112d0 con 0x7f43e8105cf0 2024-05-10T11:03:37.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43ef200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f43e8111150 con 0x7f43e8106e10 2024-05-10T11:03:37.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43ef200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f43e8111450 con 0x7f43e8105350 2024-05-10T11:03:37.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43e8105350 0x7f43e819b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:03:37.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43e8105350 0x7f43e819b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50410/0 (socket says 172.21.15.27:50410) 2024-05-10T11:03:37.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43ed200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e8106e10 0x7f43e81a2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:03:37.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43e7e00700 1 -- 172.21.15.27:0/3120645392 learned_addr learned my addr 172.21.15.27:0/3120645392 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:03:37.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43e7400700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e8105cf0 0x7f43e819be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:03:37.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43e7e00700 1 -- 172.21.15.27:0/3120645392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e8106e10 msgr2=0x7f43e81a2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:03:37.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43e7e00700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e8106e10 0x7f43e81a2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:37.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43e7e00700 1 -- 172.21.15.27:0/3120645392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e8105cf0 msgr2=0x7f43e819be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:03:37.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43e7e00700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e8105cf0 0x7f43e819be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:03:37.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.078+0000 7f43e7e00700 1 -- 172.21.15.27:0/3120645392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e81a3550 con 0x7f43e8105350 2024-05-10T11:03:37.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.079+0000 7f43e7e00700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43e8105350 0x7f43e819b8f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f43d800e9a0 tx=0x7f43d800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:03:37.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.079+0000 7f43e4c00700 1 -- 172.21.15.27:0/3120645392 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8004220 con 0x7f43e8105350 2024-05-10T11:03:37.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.079+0000 7f43e4c00700 1 -- 172.21.15.27:0/3120645392 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43d80043c0 con 0x7f43e8105350 2024-05-10T11:03:37.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.079+0000 7f43ef200700 1 -- 172.21.15.27:0/3120645392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e81a3840 con 0x7f43e8105350 2024-05-10T11:03:37.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.079+0000 7f43e4c00700 1 -- 172.21.15.27:0/3120645392 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8003cc0 con 0x7f43e8105350 2024-05-10T11:03:37.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.079+0000 7f43ef200700 1 -- 172.21.15.27:0/3120645392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e81a3dd0 con 0x7f43e8105350 2024-05-10T11:03:37.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.080+0000 7f43ef200700 1 -- 172.21.15.27:0/3120645392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43e806a190 con 0x7f43e8105350 2024-05-10T11:03:37.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.082+0000 7f43e4c00700 1 -- 172.21.15.27:0/3120645392 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f43d800c810 con 0x7f43e8105350 2024-05-10T11:03:37.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.084+0000 7f43e4c00700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f43cc0b2980 0x7f43cc0b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:03:37.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.084+0000 7f43e4c00700 1 -- 172.21.15.27:0/3120645392 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4533+0+0 (secure 0 0 0) 0x7f43d80e1540 con 0x7f43e8105350 2024-05-10T11:03:37.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.084+0000 7f43e7400700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f43cc0b2980 0x7f43cc0b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:03:37.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.085+0000 7f43e7400700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f43cc0b2980 0x7f43cc0b4e40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f43d4000c00 tx=0x7f43d4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:03:37.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.086+0000 7f43e4c00700 1 -- 172.21.15.27:0/3120645392 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f43d80aa080 con 0x7f43e8105350 2024-05-10T11:03:37.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:03:37.333+0000 7f43ef200700 1 -- 172.21.15.27:0/3120645392 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f43e80638e0 con 0x7f43cc0b2980 2024-05-10T11:03:38.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:37 smithi027 ceph-mon[30054]: pgmap v156: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:38.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:03:38.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:03:38.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:38.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:37 smithi076 ceph-mon[34152]: pgmap v156: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:38.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:03:38.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:03:38.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:38.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:37 smithi189 ceph-mon[34842]: pgmap v156: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:38.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:03:38.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:03:38.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:39.116 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:38 smithi027 ceph-mon[30054]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:03:39.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:38 smithi076 ceph-mon[34152]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:03:39.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:38 smithi189 ceph-mon[34842]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:03:40.090 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:39 smithi027 ceph-mon[30054]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-05-10T11:03:40.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:39 smithi076 ceph-mon[34152]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-05-10T11:03:40.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:39 smithi189 ceph-mon[34842]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-05-10T11:03:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:41 smithi027 ceph-mon[30054]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-10T11:03:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:41 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3757819778' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa064bd-d7d2-4189-824d-59cc8d0c5e29"}]: dispatch 2024-05-10T11:03:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:41 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3757819778' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa064bd-d7d2-4189-824d-59cc8d0c5e29"}]': finished 2024-05-10T11:03:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:41 smithi027 ceph-mon[30054]: osdmap e39: 7 total, 6 up, 7 in 2024-05-10T11:03:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:03:42.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:41 smithi076 ceph-mon[34152]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-10T11:03:42.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:41 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3757819778' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa064bd-d7d2-4189-824d-59cc8d0c5e29"}]: dispatch 2024-05-10T11:03:42.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:41 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3757819778' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa064bd-d7d2-4189-824d-59cc8d0c5e29"}]': finished 2024-05-10T11:03:42.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:41 smithi076 ceph-mon[34152]: osdmap e39: 7 total, 6 up, 7 in 2024-05-10T11:03:42.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:03:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:41 smithi189 ceph-mon[34842]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-10T11:03:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:41 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3757819778' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa064bd-d7d2-4189-824d-59cc8d0c5e29"}]: dispatch 2024-05-10T11:03:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:41 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3757819778' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa064bd-d7d2-4189-824d-59cc8d0c5e29"}]': finished 2024-05-10T11:03:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:41 smithi189 ceph-mon[34842]: osdmap e39: 7 total, 6 up, 7 in 2024-05-10T11:03:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:03:43.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:42 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/66599455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:03:43.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:42 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/66599455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:03:43.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:42 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/66599455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:03:44.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:43 smithi027 ceph-mon[30054]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-10T11:03:44.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:43 smithi076 ceph-mon[34152]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-10T11:03:44.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:43 smithi189 ceph-mon[34842]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-10T11:03:46.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:45 smithi027 ceph-mon[30054]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:03:46.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:45 smithi076 ceph-mon[34152]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:03:46.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:45 smithi189 ceph-mon[34842]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:03:48.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:47 smithi027 ceph-mon[30054]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:03:48.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:47 smithi076 ceph-mon[34152]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:03:48.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:47 smithi189 ceph-mon[34842]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:03:49.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:49 smithi027 ceph-mon[30054]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:50.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:49 smithi076 ceph-mon[34152]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:50.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:49 smithi189 ceph-mon[34842]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:51 smithi027 ceph-mon[30054]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:51 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-10T11:03:52.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:51 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:52.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:51 smithi076 ceph-mon[34152]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:52.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:51 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-10T11:03:52.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:51 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:52.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:51 smithi189 ceph-mon[34842]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:51 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-10T11:03:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:51 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:03:53.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:52 smithi027 ceph-mon[30054]: Deploying daemon osd.6 on smithi027 2024-05-10T11:03:53.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:52 smithi076 ceph-mon[34152]: Deploying daemon osd.6 on smithi027 2024-05-10T11:03:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:52 smithi189 ceph-mon[34842]: Deploying daemon osd.6 on smithi027 2024-05-10T11:03:54.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:53 smithi076 ceph-mon[34152]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:54.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:53 smithi189 ceph-mon[34842]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:54.328 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:53 smithi027 ceph-mon[30054]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:56.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:55 smithi076 ceph-mon[34152]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:56.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:55 smithi189 ceph-mon[34842]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:56.286 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:55 smithi027 ceph-mon[30054]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:58.101 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:57 smithi027 ceph-mon[30054]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:58.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:57 smithi076 ceph-mon[34152]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:03:58.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:57 smithi189 ceph-mon[34842]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:00.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:03:59 smithi076 ceph-mon[34152]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:00.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:03:59 smithi027 ceph-mon[30054]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:00.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:03:59 smithi189 ceph-mon[34842]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:02.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:02 smithi027 ceph-mon[30054]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:02.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:02 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:02.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:02 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:02.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:02 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:02.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:02 smithi076 ceph-mon[34152]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:02.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:02 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:02.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:02 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:02.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:02 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:02.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:02 smithi189 ceph-mon[34842]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:02.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:02 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:02.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:02 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:02.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:02 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:04.287 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:04 smithi027 ceph-mon[30054]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:04.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:04 smithi076 ceph-mon[34152]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:04.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:04 smithi189 ceph-mon[34842]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:06.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:05 smithi076 ceph-mon[34152]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:06.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:06.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:06.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:05 smithi076 ceph-mon[34152]: from='osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-10T11:04:06.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:05 smithi027 ceph-mon[30054]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:06.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:06.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:06.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:05 smithi027 ceph-mon[30054]: from='osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-10T11:04:06.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.191+0000 7f43e4c00700 1 -- 172.21.15.27:0/3120645392 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f43e80638e0 con 0x7f43cc0b2980 2024-05-10T11:04:06.193 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 6 on host 'smithi027' 2024-05-10T11:04:06.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.194+0000 7f43ef200700 1 -- 172.21.15.27:0/3120645392 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f43cc0b2980 msgr2=0x7f43cc0b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:04:06.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.194+0000 7f43ef200700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f43cc0b2980 0x7f43cc0b4e40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f43d4000c00 tx=0x7f43d4005fb0 comp rx=0 tx=0).stop 2024-05-10T11:04:06.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.194+0000 7f43ef200700 1 -- 172.21.15.27:0/3120645392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43e8105350 msgr2=0x7f43e819b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:04:06.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.194+0000 7f43ef200700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43e8105350 0x7f43e819b8f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f43d800e9a0 tx=0x7f43d800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:04:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.195+0000 7f43ef200700 1 -- 172.21.15.27:0/3120645392 shutdown_connections 2024-05-10T11:04:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.195+0000 7f43ef200700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f43cc0b2980 0x7f43cc0b4e40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.195+0000 7f43ef200700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43e8105350 0x7f43e819b8f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.195+0000 7f43ef200700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43e8106e10 0x7f43e81a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.195+0000 7f43ef200700 1 --2- 172.21.15.27:0/3120645392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e8105cf0 0x7f43e819be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.195+0000 7f43ef200700 1 -- 172.21.15.27:0/3120645392 >> 172.21.15.27:0/3120645392 conn(0x7f43e8100cf0 msgr2=0x7f43e8109c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:04:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.195+0000 7f43ef200700 1 -- 172.21.15.27:0/3120645392 shutdown_connections 2024-05-10T11:04:06.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:04:06.195+0000 7f43ef200700 1 -- 172.21.15.27:0/3120645392 wait complete. 2024-05-10T11:04:06.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:05 smithi189 ceph-mon[34842]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:06.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:06.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:06.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:05 smithi189 ceph-mon[34842]: from='osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-10T11:04:07.129 DEBUG:teuthology.orchestra.run.smithi027:osd.6> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.6.service 2024-05-10T11:04:07.134 INFO:tasks.cephadm:Deploying osd.7 on smithi076 with /dev/vg_nvme/lv_2... 2024-05-10T11:04:07.134 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-10T11:04:07.341 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:07 smithi027 ceph-mon[30054]: from='osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-10T11:04:07.341 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:07 smithi027 ceph-mon[30054]: osdmap e40: 7 total, 6 up, 7 in 2024-05-10T11:04:07.341 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:07 smithi027 ceph-mon[30054]: from='osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:04:07.341 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:04:07.341 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:07.341 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.b/config 2024-05-10T11:04:07.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:07 smithi076 ceph-mon[34152]: from='osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-10T11:04:07.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:07 smithi076 ceph-mon[34152]: osdmap e40: 7 total, 6 up, 7 in 2024-05-10T11:04:07.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:07 smithi076 ceph-mon[34152]: from='osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:04:07.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:04:07.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:07.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:07.442 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:07 smithi189 ceph-mon[34842]: from='osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-10T11:04:07.443 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:07 smithi189 ceph-mon[34842]: osdmap e40: 7 total, 6 up, 7 in 2024-05-10T11:04:07.443 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:07 smithi189 ceph-mon[34842]: from='osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:04:07.443 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:04:07.443 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:07.443 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:08.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:08 smithi027 ceph-mon[30054]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:08.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:08 smithi027 ceph-mon[30054]: from='osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-10T11:04:08.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:08 smithi027 ceph-mon[30054]: osdmap e41: 7 total, 6 up, 7 in 2024-05-10T11:04:08.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:08 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:04:08.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:08 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:04:08.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:08 smithi076 ceph-mon[34152]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:08.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:08 smithi076 ceph-mon[34152]: from='osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-10T11:04:08.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:08 smithi076 ceph-mon[34152]: osdmap e41: 7 total, 6 up, 7 in 2024-05-10T11:04:08.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:08 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:04:08.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:08 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:04:08.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:08 smithi189 ceph-mon[34842]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:08.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:08 smithi189 ceph-mon[34842]: from='osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-10T11:04:08.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:08 smithi189 ceph-mon[34842]: osdmap e41: 7 total, 6 up, 7 in 2024-05-10T11:04:08.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:08 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:04:08.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:08 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:04:09.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:09 smithi076 ceph-mon[34152]: purged_snaps scrub starts 2024-05-10T11:04:09.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:09 smithi076 ceph-mon[34152]: purged_snaps scrub ok 2024-05-10T11:04:09.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:09 smithi076 ceph-mon[34152]: osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800] boot 2024-05-10T11:04:09.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:09 smithi076 ceph-mon[34152]: osdmap e42: 7 total, 7 up, 7 in 2024-05-10T11:04:09.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:04:09.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:09 smithi027 ceph-mon[30054]: purged_snaps scrub starts 2024-05-10T11:04:09.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:09 smithi027 ceph-mon[30054]: purged_snaps scrub ok 2024-05-10T11:04:09.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:09 smithi027 ceph-mon[30054]: osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800] boot 2024-05-10T11:04:09.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:09 smithi027 ceph-mon[30054]: osdmap e42: 7 total, 7 up, 7 in 2024-05-10T11:04:09.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:04:09.418 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-10T11:04:09.419 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-10T11:04:09.419 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-05-10T11:04:09.419 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-05-10T11:04:09.419 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148029 s, 708 MB/s 2024-05-10T11:04:09.419 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-05-10T11:04:09.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:09 smithi189 ceph-mon[34842]: purged_snaps scrub starts 2024-05-10T11:04:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:09 smithi189 ceph-mon[34842]: purged_snaps scrub ok 2024-05-10T11:04:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:09 smithi189 ceph-mon[34842]: osd.6 [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800] boot 2024-05-10T11:04:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:09 smithi189 ceph-mon[34842]: osdmap e42: 7 total, 7 up, 7 in 2024-05-10T11:04:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:04:10.083 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi076:vg_nvme/lv_2 2024-05-10T11:04:10.325 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.b/config 2024-05-10T11:04:10.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:10 smithi027 ceph-mon[30054]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:10.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:10 smithi027 ceph-mon[30054]: osdmap e43: 7 total, 7 up, 7 in 2024-05-10T11:04:10.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:10 smithi076 ceph-mon[34152]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:10.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:10 smithi076 ceph-mon[34152]: osdmap e43: 7 total, 7 up, 7 in 2024-05-10T11:04:10.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:10 smithi189 ceph-mon[34842]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-05-10T11:04:10.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:10 smithi189 ceph-mon[34842]: osdmap e43: 7 total, 7 up, 7 in 2024-05-10T11:04:11.318 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:11 smithi027 ceph-mon[30054]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:11.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:11 smithi076 ceph-mon[34152]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:11.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:11 smithi189 ceph-mon[34842]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:11.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.960+0000 7f7e1e75a700 1 -- 172.21.15.76:0/3017356075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e18101160 msgr2=0x7f7e18101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:04:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.960+0000 7f7e1e75a700 1 --2- 172.21.15.76:0/3017356075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e18101160 0x7f7e18101580 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009990 tx=0x7f7e08009ca0 comp rx=0 tx=0).stop 2024-05-10T11:04:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.960+0000 7f7e1e75a700 1 -- 172.21.15.76:0/3017356075 shutdown_connections 2024-05-10T11:04:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.960+0000 7f7e1e75a700 1 --2- 172.21.15.76:0/3017356075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e181091e0 0x7f7e1810b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.960+0000 7f7e1e75a700 1 --2- 172.21.15.76:0/3017356075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e18101160 0x7f7e18101580 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.960+0000 7f7e1e75a700 1 --2- 172.21.15.76:0/3017356075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e18101b50 0x7f7e18108ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.960+0000 7f7e1e75a700 1 -- 172.21.15.76:0/3017356075 >> 172.21.15.76:0/3017356075 conn(0x7f7e180fccf0 msgr2=0x7f7e180ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:04:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.960+0000 7f7e1e75a700 1 -- 172.21.15.76:0/3017356075 shutdown_connections 2024-05-10T11:04:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.960+0000 7f7e1e75a700 1 -- 172.21.15.76:0/3017356075 wait complete. 2024-05-10T11:04:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.960+0000 7f7e1e75a700 1 Processor -- start 2024-05-10T11:04:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.960+0000 7f7e1e75a700 1 -- start start 2024-05-10T11:04:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e1e75a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e18101160 0x7f7e18199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:04:11.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e1e75a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e18101b50 0x7f7e18199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e1e75a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e181091e0 0x7f7e181a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e1e75a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1810f000 con 0x7f7e18101b50 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e1e75a700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1810ee80 con 0x7f7e18101160 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e1e75a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1810f180 con 0x7f7e181091e0 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e17fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e18101160 0x7f7e18199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e17fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e18101160 0x7f7e18199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47138/0 (socket says 172.21.15.76:47138) 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e17fff700 1 -- 172.21.15.76:0/2231028719 learned_addr learned my addr 172.21.15.76:0/2231028719 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e1ccf7700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e181091e0 0x7f7e181a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e17fff700 1 -- 172.21.15.76:0/2231028719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e181091e0 msgr2=0x7f7e181a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e17fff700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e181091e0 0x7f7e181a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e17fff700 1 -- 172.21.15.76:0/2231028719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e18101b50 msgr2=0x7f7e18199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e17fff700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e18101b50 0x7f7e18199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e17fff700 1 -- 172.21.15.76:0/2231028719 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e181a12c0 con 0x7f7e18101160 2024-05-10T11:04:11.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e1ccf7700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e181091e0 0x7f7e181a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:04:11.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.961+0000 7f7e17fff700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e18101160 0x7f7e18199660 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7e08006010 tx=0x7f7e080047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:04:11.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.962+0000 7f7e157fa700 1 -- 172.21.15.76:0/2231028719 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08027070 con 0x7f7e18101160 2024-05-10T11:04:11.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.962+0000 7f7e157fa700 1 -- 172.21.15.76:0/2231028719 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e08004180 con 0x7f7e18101160 2024-05-10T11:04:11.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.962+0000 7f7e1e75a700 1 -- 172.21.15.76:0/2231028719 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e181a1550 con 0x7f7e18101160 2024-05-10T11:04:11.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.962+0000 7f7e157fa700 1 -- 172.21.15.76:0/2231028719 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0800f460 con 0x7f7e18101160 2024-05-10T11:04:11.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.962+0000 7f7e1e75a700 1 -- 172.21.15.76:0/2231028719 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e181a1a80 con 0x7f7e18101160 2024-05-10T11:04:11.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.963+0000 7f7e1e75a700 1 -- 172.21.15.76:0/2231028719 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e1806a190 con 0x7f7e18101160 2024-05-10T11:04:11.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.964+0000 7f7e157fa700 1 -- 172.21.15.76:0/2231028719 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f7e0802a070 con 0x7f7e18101160 2024-05-10T11:04:11.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.964+0000 7f7e157fa700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7e000b2900 0x7f7e000b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:04:11.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.965+0000 7f7e157fa700 1 -- 172.21.15.76:0/2231028719 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4965+0+0 (secure 0 0 0) 0x7f7e080e2d60 con 0x7f7e18101160 2024-05-10T11:04:11.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.965+0000 7f7e177fe700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7e000b2900 0x7f7e000b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:04:11.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.966+0000 7f7e177fe700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7e000b2900 0x7f7e000b4dc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c005950 tx=0x7f7e0c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:04:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:11.968+0000 7f7e157fa700 1 -- 172.21.15.76:0/2231028719 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7e08031070 con 0x7f7e18101160 2024-05-10T11:04:12.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:12.182+0000 7f7e1e75a700 1 -- 172.21.15.76:0/2231028719 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7e180638e0 con 0x7f7e000b2900 2024-05-10T11:04:12.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:12 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:04:12.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:12 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:04:12.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:12 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:12.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:12 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:04:12.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:12 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:04:12.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:12 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:12.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:12 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:04:12.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:12 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:04:12.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:12 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:13.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: Detected new or changed devices on smithi027 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi027 to 5696M 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:13.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:13.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:13 smithi076 ceph-mon[34152]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:04:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: Detected new or changed devices on smithi027 2024-05-10T11:04:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:13.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:13.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:13.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi027 to 5696M 2024-05-10T11:04:13.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:13.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:13.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:13.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:13.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:13.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:13 smithi027 ceph-mon[30054]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:13.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:04:13.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: Detected new or changed devices on smithi027 2024-05-10T11:04:13.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:13.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:13.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:13.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi027 to 5696M 2024-05-10T11:04:13.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:13.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:13.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:13.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:13.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:13.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:13.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:13 smithi189 ceph-mon[34842]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:16.103 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:15 smithi027 ceph-mon[30054]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:16.103 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:15 smithi027 ceph-mon[30054]: from='client.? 172.21.15.76:0/1856604991' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56a01e95-852d-4072-a5d7-50b414487dfe"}]: dispatch 2024-05-10T11:04:16.103 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:15 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56a01e95-852d-4072-a5d7-50b414487dfe"}]: dispatch 2024-05-10T11:04:16.103 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:15 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56a01e95-852d-4072-a5d7-50b414487dfe"}]': finished 2024-05-10T11:04:16.104 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:15 smithi027 ceph-mon[30054]: osdmap e44: 8 total, 7 up, 8 in 2024-05-10T11:04:16.104 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:15 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:04:16.104 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:15 smithi027 ceph-mon[30054]: from='client.? 172.21.15.76:0/1980215775' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:04:16.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:15 smithi076 ceph-mon[34152]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:16.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:15 smithi076 ceph-mon[34152]: from='client.? 172.21.15.76:0/1856604991' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56a01e95-852d-4072-a5d7-50b414487dfe"}]: dispatch 2024-05-10T11:04:16.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:15 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56a01e95-852d-4072-a5d7-50b414487dfe"}]: dispatch 2024-05-10T11:04:16.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:15 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56a01e95-852d-4072-a5d7-50b414487dfe"}]': finished 2024-05-10T11:04:16.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:15 smithi076 ceph-mon[34152]: osdmap e44: 8 total, 7 up, 8 in 2024-05-10T11:04:16.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:15 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:04:16.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:15 smithi076 ceph-mon[34152]: from='client.? 172.21.15.76:0/1980215775' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:04:16.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:15 smithi189 ceph-mon[34842]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:16.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:15 smithi189 ceph-mon[34842]: from='client.? 172.21.15.76:0/1856604991' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56a01e95-852d-4072-a5d7-50b414487dfe"}]: dispatch 2024-05-10T11:04:16.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:15 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56a01e95-852d-4072-a5d7-50b414487dfe"}]: dispatch 2024-05-10T11:04:16.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:15 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56a01e95-852d-4072-a5d7-50b414487dfe"}]': finished 2024-05-10T11:04:16.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:15 smithi189 ceph-mon[34842]: osdmap e44: 8 total, 7 up, 8 in 2024-05-10T11:04:16.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:15 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:04:16.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:15 smithi189 ceph-mon[34842]: from='client.? 172.21.15.76:0/1980215775' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:04:17.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:17 smithi189 ceph-mon[34842]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:18.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:17 smithi027 ceph-mon[30054]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:18.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:17 smithi076 ceph-mon[34152]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:20.015 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:19 smithi076 ceph-mon[34152]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:20.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:19 smithi027 ceph-mon[30054]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:19 smithi189 ceph-mon[34842]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:21.855 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:21 smithi076 ceph-mon[34152]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:22.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:21 smithi027 ceph-mon[30054]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:22.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:21 smithi189 ceph-mon[34842]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:23.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-10T11:04:23.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:23.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-10T11:04:23.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:23.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-10T11:04:23.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:24.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:23 smithi027 ceph-mon[30054]: Deploying daemon osd.7 on smithi076 2024-05-10T11:04:24.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:23 smithi027 ceph-mon[30054]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:24.158 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:23 smithi076 ceph-mon[34152]: Deploying daemon osd.7 on smithi076 2024-05-10T11:04:24.158 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:23 smithi076 ceph-mon[34152]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:24.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:23 smithi189 ceph-mon[34842]: Deploying daemon osd.7 on smithi076 2024-05-10T11:04:24.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:23 smithi189 ceph-mon[34842]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:26.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:25 smithi027 ceph-mon[30054]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:26.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:25 smithi076 ceph-mon[34152]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:26.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:25 smithi189 ceph-mon[34842]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:28.102 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:27 smithi076 ceph-mon[34152]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:28.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:27 smithi027 ceph-mon[30054]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:28.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:27 smithi189 ceph-mon[34842]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:29.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:29.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:29.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:29.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:29 smithi076 ceph-mon[34152]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:29.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:29.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:29.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:29.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:29 smithi189 ceph-mon[34842]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:29.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:29.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:29.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:29.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:29 smithi027 ceph-mon[30054]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:31.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:31 smithi076 ceph-mon[34152]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:31.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:31.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:31.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:31.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:31.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:32.074 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 7 on host 'smithi076' 2024-05-10T11:04:32.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.074+0000 7f7e157fa700 1 -- 172.21.15.76:0/2231028719 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7e180638e0 con 0x7f7e000b2900 2024-05-10T11:04:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.076+0000 7f7e1e75a700 1 -- 172.21.15.76:0/2231028719 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7e000b2900 msgr2=0x7f7e000b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:04:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.076+0000 7f7e1e75a700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7e000b2900 0x7f7e000b4dc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c005950 tx=0x7f7e0c0058e0 comp rx=0 tx=0).stop 2024-05-10T11:04:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.076+0000 7f7e1e75a700 1 -- 172.21.15.76:0/2231028719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e18101160 msgr2=0x7f7e18199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:04:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.076+0000 7f7e1e75a700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e18101160 0x7f7e18199660 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7e08006010 tx=0x7f7e080047b0 comp rx=0 tx=0).stop 2024-05-10T11:04:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.076+0000 7f7e1e75a700 1 -- 172.21.15.76:0/2231028719 shutdown_connections 2024-05-10T11:04:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.076+0000 7f7e1e75a700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7e000b2900 0x7f7e000b4dc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.076+0000 7f7e1e75a700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e181091e0 0x7f7e181a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.076+0000 7f7e1e75a700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e18101160 0x7f7e18199660 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.076+0000 7f7e1e75a700 1 --2- 172.21.15.76:0/2231028719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e18101b50 0x7f7e18199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.076+0000 7f7e1e75a700 1 -- 172.21.15.76:0/2231028719 >> 172.21.15.76:0/2231028719 conn(0x7f7e180fccf0 msgr2=0x7f7e180fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:04:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.076+0000 7f7e1e75a700 1 -- 172.21.15.76:0/2231028719 shutdown_connections 2024-05-10T11:04:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:04:32.076+0000 7f7e1e75a700 1 -- 172.21.15.76:0/2231028719 wait complete. 2024-05-10T11:04:32.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:31 smithi027 ceph-mon[30054]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:31 smithi189 ceph-mon[34842]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:32.774 DEBUG:teuthology.orchestra.run.smithi076:osd.7> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.7.service 2024-05-10T11:04:32.777 INFO:tasks.cephadm:Deploying osd.8 on smithi189 with /dev/vg_nvme/lv_2... 2024-05-10T11:04:32.777 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-10T11:04:33.020 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:04:33.086 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:32 smithi076 ceph-mon[34152]: from='osd.7 [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-10T11:04:33.087 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:32 smithi076 ceph-mon[34152]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-10T11:04:33.087 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:33.087 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:33.087 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:33.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:32 smithi027 ceph-mon[30054]: from='osd.7 [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-10T11:04:33.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:32 smithi027 ceph-mon[30054]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-10T11:04:33.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:32 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:33.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:32 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:33.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:32 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:33.142 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:32 smithi189 ceph-mon[34842]: from='osd.7 [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-10T11:04:33.142 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:32 smithi189 ceph-mon[34842]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-10T11:04:33.142 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:32 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:33.143 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:32 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:33.143 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:32 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:33.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:33 smithi076 ceph-mon[34152]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:33.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:33 smithi076 ceph-mon[34152]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-10T11:04:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:33 smithi076 ceph-mon[34152]: from='osd.7 [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:04:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:33 smithi076 ceph-mon[34152]: osdmap e45: 8 total, 7 up, 8 in 2024-05-10T11:04:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:04:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:33 smithi076 ceph-mon[34152]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:04:33.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:33 smithi076 ceph-mon[34152]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-05-10T11:04:33.926 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:04:33 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:04:33.775+0000 7f6a5cdab700 -1 osd.7 0 waiting for initial osdmap 2024-05-10T11:04:33.927 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:04:33 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:04:33.783+0000 7f6a58b9e700 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-10T11:04:34.093 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:33 smithi189 ceph-mon[34842]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:34.093 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:33 smithi189 ceph-mon[34842]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-10T11:04:34.093 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:33 smithi189 ceph-mon[34842]: from='osd.7 [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:04:34.094 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:33 smithi189 ceph-mon[34842]: osdmap e45: 8 total, 7 up, 8 in 2024-05-10T11:04:34.094 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:04:34.094 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:33 smithi189 ceph-mon[34842]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:04:34.094 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:33 smithi189 ceph-mon[34842]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-05-10T11:04:34.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:33 smithi027 ceph-mon[30054]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:33 smithi027 ceph-mon[30054]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-10T11:04:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:33 smithi027 ceph-mon[30054]: from='osd.7 [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:04:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:33 smithi027 ceph-mon[30054]: osdmap e45: 8 total, 7 up, 8 in 2024-05-10T11:04:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:04:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:33 smithi027 ceph-mon[30054]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:04:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:33 smithi027 ceph-mon[30054]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-05-10T11:04:34.993 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:34 smithi189 ceph-mon[34842]: osdmap e46: 8 total, 7 up, 8 in 2024-05-10T11:04:34.993 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:04:34.993 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:34 smithi189 ceph-mon[34842]: osd.7 [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194] boot 2024-05-10T11:04:34.993 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:34 smithi189 ceph-mon[34842]: osdmap e47: 8 total, 8 up, 8 in 2024-05-10T11:04:34.993 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:34 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:04:34.994 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-10T11:04:34.994 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-10T11:04:34.994 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-05-10T11:04:34.994 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-05-10T11:04:34.994 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011382 s, 921 MB/s 2024-05-10T11:04:34.994 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-05-10T11:04:35.111 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:34 smithi076 ceph-mon[34152]: osdmap e46: 8 total, 7 up, 8 in 2024-05-10T11:04:35.111 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:04:35.111 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:34 smithi076 ceph-mon[34152]: osd.7 [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194] boot 2024-05-10T11:04:35.112 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:34 smithi076 ceph-mon[34152]: osdmap e47: 8 total, 8 up, 8 in 2024-05-10T11:04:35.112 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:34 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:04:35.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:34 smithi027 ceph-mon[30054]: osdmap e46: 8 total, 7 up, 8 in 2024-05-10T11:04:35.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:04:35.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:34 smithi027 ceph-mon[30054]: osd.7 [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194] boot 2024-05-10T11:04:35.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:34 smithi027 ceph-mon[30054]: osdmap e47: 8 total, 8 up, 8 in 2024-05-10T11:04:35.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:34 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:04:35.714 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_2 2024-05-10T11:04:35.961 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:04:35.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:35 smithi189 ceph-mon[34842]: purged_snaps scrub starts 2024-05-10T11:04:35.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:35 smithi189 ceph-mon[34842]: purged_snaps scrub ok 2024-05-10T11:04:35.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:35 smithi189 ceph-mon[34842]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:35.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:35 smithi189 ceph-mon[34842]: osdmap e48: 8 total, 8 up, 8 in 2024-05-10T11:04:36.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:35 smithi027 ceph-mon[30054]: purged_snaps scrub starts 2024-05-10T11:04:36.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:35 smithi027 ceph-mon[30054]: purged_snaps scrub ok 2024-05-10T11:04:36.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:35 smithi027 ceph-mon[30054]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:36.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:35 smithi027 ceph-mon[30054]: osdmap e48: 8 total, 8 up, 8 in 2024-05-10T11:04:36.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:35 smithi076 ceph-mon[34152]: purged_snaps scrub starts 2024-05-10T11:04:36.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:35 smithi076 ceph-mon[34152]: purged_snaps scrub ok 2024-05-10T11:04:36.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:35 smithi076 ceph-mon[34152]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-10T11:04:36.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:35 smithi076 ceph-mon[34152]: osdmap e48: 8 total, 8 up, 8 in 2024-05-10T11:04:37.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:36 smithi189 ceph-mon[34842]: Detected new or changed devices on smithi076 2024-05-10T11:04:37.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:37.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:37.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:37.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:37.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:37.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:36 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi076 to 5696M 2024-05-10T11:04:37.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:37.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:37.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:37.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:37.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:36 smithi189 ceph-mon[34842]: osdmap e49: 8 total, 8 up, 8 in 2024-05-10T11:04:37.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:36 smithi076 ceph-mon[34152]: Detected new or changed devices on smithi076 2024-05-10T11:04:37.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:37.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:37.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:37.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:37.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:37.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:36 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi076 to 5696M 2024-05-10T11:04:37.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:37.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:37.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:37.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:37.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:36 smithi076 ceph-mon[34152]: osdmap e49: 8 total, 8 up, 8 in 2024-05-10T11:04:37.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:36 smithi027 ceph-mon[30054]: Detected new or changed devices on smithi076 2024-05-10T11:04:37.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:37.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:37.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:37.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:37.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:04:37.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:36 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi076 to 5696M 2024-05-10T11:04:37.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:37.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:37.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:37.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:37.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:36 smithi027 ceph-mon[30054]: osdmap e49: 8 total, 8 up, 8 in 2024-05-10T11:04:37.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.531+0000 7fa9a2676700 1 -- 172.21.15.189:0/2181375653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa99c103fb0 msgr2=0x7fa99c10a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:04:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.531+0000 7fa9a2676700 1 --2- 172.21.15.189:0/2181375653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa99c103fb0 0x7fa99c10a860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa99400b0a0 tx=0x7fa99400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:04:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.531+0000 7fa9a2676700 1 -- 172.21.15.189:0/2181375653 shutdown_connections 2024-05-10T11:04:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.531+0000 7fa9a2676700 1 --2- 172.21.15.189:0/2181375653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa99c103fb0 0x7fa99c10a860 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.531+0000 7fa9a2676700 1 --2- 172.21.15.189:0/2181375653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa99c102410 0x7fa99c102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.531+0000 7fa9a2676700 1 --2- 172.21.15.189:0/2181375653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa99c103610 0x7fa99c103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.531+0000 7fa9a2676700 1 -- 172.21.15.189:0/2181375653 >> 172.21.15.189:0/2181375653 conn(0x7fa99c0fd9d0 msgr2=0x7fa99c0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:04:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.531+0000 7fa9a2676700 1 -- 172.21.15.189:0/2181375653 shutdown_connections 2024-05-10T11:04:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.531+0000 7fa9a2676700 1 -- 172.21.15.189:0/2181375653 wait complete. 2024-05-10T11:04:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.532+0000 7fa9a2676700 1 Processor -- start 2024-05-10T11:04:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.532+0000 7fa9a2676700 1 -- start start 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.532+0000 7fa9a2676700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa99c102410 0x7fa99c198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.532+0000 7fa9a2676700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa99c103610 0x7fa99c198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.532+0000 7fa9a2676700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa99c103fb0 0x7fa99c19fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.532+0000 7fa9a2676700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c10dfa0 con 0x7fa99c103610 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.532+0000 7fa9a2676700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c10de20 con 0x7fa99c103fb0 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.532+0000 7fa9a2676700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c10e120 con 0x7fa99c102410 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa99bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa99c102410 0x7fa99c198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa99bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa99c102410 0x7fa99c198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42082/0 (socket says 172.21.15.189:42082) 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa99bfff700 1 -- 172.21.15.189:0/1293333540 learned_addr learned my addr 172.21.15.189:0/1293333540 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa9a0c13700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa99c103fb0 0x7fa99c19fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa99bfff700 1 -- 172.21.15.189:0/1293333540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa99c103fb0 msgr2=0x7fa99c19fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa99bfff700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa99c103fb0 0x7fa99c19fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:37.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa99bfff700 1 -- 172.21.15.189:0/1293333540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa99c103610 msgr2=0x7fa99c198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:04:37.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa99bfff700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa99c103610 0x7fa99c198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:37.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa99bfff700 1 -- 172.21.15.189:0/1293333540 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99c1a0280 con 0x7fa99c102410 2024-05-10T11:04:37.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa99bfff700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa99c102410 0x7fa99c198620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00d9e0 tx=0x7fa98c00dda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:04:37.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa9997fa700 1 -- 172.21.15.189:0/1293333540 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c01b070 con 0x7fa99c102410 2024-05-10T11:04:37.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa9997fa700 1 -- 172.21.15.189:0/1293333540 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa98c00ecd0 con 0x7fa99c102410 2024-05-10T11:04:37.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa9997fa700 1 -- 172.21.15.189:0/1293333540 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0145e0 con 0x7fa99c102410 2024-05-10T11:04:37.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa9a2676700 1 -- 172.21.15.189:0/1293333540 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa99c06bc90 con 0x7fa99c102410 2024-05-10T11:04:37.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.533+0000 7fa9a2676700 1 -- 172.21.15.189:0/1293333540 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa99c06c250 con 0x7fa99c102410 2024-05-10T11:04:37.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.534+0000 7fa9a2676700 1 -- 172.21.15.189:0/1293333540 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa99c06a190 con 0x7fa99c102410 2024-05-10T11:04:37.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.535+0000 7fa9997fa700 1 -- 172.21.15.189:0/1293333540 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa98c01e070 con 0x7fa99c102410 2024-05-10T11:04:37.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.536+0000 7fa9997fa700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa9840b2880 0x7fa9840b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:04:37.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.536+0000 7fa9997fa700 1 -- 172.21.15.189:0/1293333540 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5397+0+0 (secure 0 0 0) 0x7fa98c0e3200 con 0x7fa99c102410 2024-05-10T11:04:37.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.537+0000 7fa99b7fe700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa9840b2880 0x7fa9840b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:04:37.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.538+0000 7fa99b7fe700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa9840b2880 0x7fa9840b4d40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa9900097b0 tx=0x7fa990006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:04:37.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.540+0000 7fa9997fa700 1 -- 172.21.15.189:0/1293333540 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa98c0ab9e0 con 0x7fa99c102410 2024-05-10T11:04:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:37.751+0000 7fa9a2676700 1 -- 172.21.15.189:0/1293333540 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa99c06c510 con 0x7fa9840b2880 2024-05-10T11:04:38.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:37 smithi076 ceph-mon[34152]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:38.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:04:38.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:04:38.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:38.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:37 smithi189 ceph-mon[34842]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:38.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:04:38.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:04:38.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:38.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:37 smithi027 ceph-mon[30054]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:38.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:04:38.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:04:38.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:39.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:38 smithi189 ceph-mon[34842]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:04:39.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:38 smithi027 ceph-mon[30054]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:04:39.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:38 smithi076 ceph-mon[34152]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:04:40.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:39 smithi189 ceph-mon[34842]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:40.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:39 smithi027 ceph-mon[30054]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:39 smithi076 ceph-mon[34152]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:41.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:40 smithi189 ceph-mon[34842]: from='client.? 172.21.15.189:0/159426299' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28643dd0-d7e0-498b-8c59-9374a9f58239"}]: dispatch 2024-05-10T11:04:41.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:40 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28643dd0-d7e0-498b-8c59-9374a9f58239"}]: dispatch 2024-05-10T11:04:41.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:40 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28643dd0-d7e0-498b-8c59-9374a9f58239"}]': finished 2024-05-10T11:04:41.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:40 smithi189 ceph-mon[34842]: osdmap e50: 9 total, 8 up, 9 in 2024-05-10T11:04:41.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:40 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:04:41.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:40 smithi027 ceph-mon[30054]: from='client.? 172.21.15.189:0/159426299' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28643dd0-d7e0-498b-8c59-9374a9f58239"}]: dispatch 2024-05-10T11:04:41.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:40 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28643dd0-d7e0-498b-8c59-9374a9f58239"}]: dispatch 2024-05-10T11:04:41.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:40 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28643dd0-d7e0-498b-8c59-9374a9f58239"}]': finished 2024-05-10T11:04:41.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:40 smithi027 ceph-mon[30054]: osdmap e50: 9 total, 8 up, 9 in 2024-05-10T11:04:41.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:04:41.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:40 smithi076 ceph-mon[34152]: from='client.? 172.21.15.189:0/159426299' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28643dd0-d7e0-498b-8c59-9374a9f58239"}]: dispatch 2024-05-10T11:04:41.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:40 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28643dd0-d7e0-498b-8c59-9374a9f58239"}]: dispatch 2024-05-10T11:04:41.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:40 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28643dd0-d7e0-498b-8c59-9374a9f58239"}]': finished 2024-05-10T11:04:41.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:40 smithi076 ceph-mon[34152]: osdmap e50: 9 total, 8 up, 9 in 2024-05-10T11:04:41.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:40 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:04:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:41 smithi189 ceph-mon[34842]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-05-10T11:04:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:41 smithi189 ceph-mon[34842]: from='client.? 172.21.15.189:0/810230798' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:04:42.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:41 smithi027 ceph-mon[30054]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-05-10T11:04:42.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:41 smithi027 ceph-mon[30054]: from='client.? 172.21.15.189:0/810230798' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:04:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:41 smithi076 ceph-mon[34152]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-05-10T11:04:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:41 smithi076 ceph-mon[34152]: from='client.? 172.21.15.189:0/810230798' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:04:44.228 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:43 smithi076 ceph-mon[34152]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-05-10T11:04:44.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:43 smithi189 ceph-mon[34842]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-05-10T11:04:44.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:43 smithi027 ceph-mon[30054]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-05-10T11:04:46.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:45 smithi189 ceph-mon[34842]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-10T11:04:46.458 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:45 smithi076 ceph-mon[34152]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-10T11:04:46.458 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:45 smithi027 ceph-mon[30054]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-10T11:04:48.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:47 smithi189 ceph-mon[34842]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:04:48.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-10T11:04:48.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:48.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:47 smithi027 ceph-mon[30054]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:04:48.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-10T11:04:48.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:48.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:47 smithi076 ceph-mon[34152]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:04:48.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-10T11:04:48.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:49.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:48 smithi189 ceph-mon[34842]: Deploying daemon osd.8 on smithi189 2024-05-10T11:04:49.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:48 smithi027 ceph-mon[30054]: Deploying daemon osd.8 on smithi189 2024-05-10T11:04:49.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:48 smithi076 ceph-mon[34152]: Deploying daemon osd.8 on smithi189 2024-05-10T11:04:50.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:49 smithi027 ceph-mon[30054]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:04:50.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:49 smithi189 ceph-mon[34842]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:04:50.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:49 smithi076 ceph-mon[34152]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:04:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:51 smithi189 ceph-mon[34842]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:52.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:51 smithi027 ceph-mon[30054]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:52.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:51 smithi076 ceph-mon[34152]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:53 smithi189 ceph-mon[34842]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:53 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:53 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:53 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:54.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:53 smithi027 ceph-mon[30054]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:54.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:53 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:54.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:53 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:54.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:53 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:54.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:53 smithi076 ceph-mon[34152]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:54.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:53 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:54.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:53 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:54.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:53 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:56.193 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:55 smithi189 ceph-mon[34842]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:56.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:55 smithi027 ceph-mon[30054]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:56.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:55 smithi076 ceph-mon[34152]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:57.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.387+0000 7fa9997fa700 1 -- 172.21.15.189:0/1293333540 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa99c06c510 con 0x7fa9840b2880 2024-05-10T11:04:57.388 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 8 on host 'smithi189' 2024-05-10T11:04:57.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.390+0000 7fa9a2676700 1 -- 172.21.15.189:0/1293333540 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa9840b2880 msgr2=0x7fa9840b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:04:57.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.390+0000 7fa9a2676700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa9840b2880 0x7fa9840b4d40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa9900097b0 tx=0x7fa990006d20 comp rx=0 tx=0).stop 2024-05-10T11:04:57.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.390+0000 7fa9a2676700 1 -- 172.21.15.189:0/1293333540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa99c102410 msgr2=0x7fa99c198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:04:57.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.390+0000 7fa9a2676700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa99c102410 0x7fa99c198620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00d9e0 tx=0x7fa98c00dda0 comp rx=0 tx=0).stop 2024-05-10T11:04:57.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.390+0000 7fa9a2676700 1 -- 172.21.15.189:0/1293333540 shutdown_connections 2024-05-10T11:04:57.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.390+0000 7fa9a2676700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa9840b2880 0x7fa9840b4d40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:57.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.390+0000 7fa9a2676700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa99c102410 0x7fa99c198620 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:57.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.390+0000 7fa9a2676700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa99c103fb0 0x7fa99c19fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:57.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.390+0000 7fa9a2676700 1 --2- 172.21.15.189:0/1293333540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa99c103610 0x7fa99c198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:04:57.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.390+0000 7fa9a2676700 1 -- 172.21.15.189:0/1293333540 >> 172.21.15.189:0/1293333540 conn(0x7fa99c0fd9d0 msgr2=0x7fa99c106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:04:57.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.390+0000 7fa9a2676700 1 -- 172.21.15.189:0/1293333540 shutdown_connections 2024-05-10T11:04:57.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:04:57.390+0000 7fa9a2676700 1 -- 172.21.15.189:0/1293333540 wait complete. 2024-05-10T11:04:57.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:57 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:57 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:57 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:57.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:57 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:57.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:57 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:57 smithi076 ceph-mon[34152]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:57.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:57 smithi076 ceph-mon[34152]: from='osd.8 [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-10T11:04:57.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:57 smithi076 ceph-mon[34152]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-10T11:04:57.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:57 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:57 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:57 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:57 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:57 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:57.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:57 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:57.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:57 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:57 smithi189 ceph-mon[34842]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:57.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:57 smithi189 ceph-mon[34842]: from='osd.8 [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-10T11:04:57.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:57 smithi189 ceph-mon[34842]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-10T11:04:57.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:57 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:57 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:57 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:57 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:57 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:04:57.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:57 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:04:57.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:57 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:57 smithi027 ceph-mon[30054]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:57.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:57 smithi027 ceph-mon[30054]: from='osd.8 [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-10T11:04:57.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:57 smithi027 ceph-mon[30054]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-10T11:04:57.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:57 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:57.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:57 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:04:58.200 DEBUG:teuthology.orchestra.run.smithi189:osd.8> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.8.service 2024-05-10T11:04:58.202 INFO:tasks.cephadm:Deploying osd.9 on smithi027 with /dev/vg_nvme/lv_1... 2024-05-10T11:04:58.202 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-10T11:04:58.437 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:58 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:58.438 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:58 smithi189 ceph-mon[34842]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-10T11:04:58.438 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:58 smithi189 ceph-mon[34842]: from='osd.8 [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:04:58.438 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:58 smithi189 ceph-mon[34842]: osdmap e51: 9 total, 8 up, 9 in 2024-05-10T11:04:58.438 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:58 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:04:58.438 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:58 smithi189 ceph-mon[34842]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:04:58.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:58 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:58.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:58 smithi027 ceph-mon[30054]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-10T11:04:58.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:58 smithi027 ceph-mon[30054]: from='osd.8 [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:04:58.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:58 smithi027 ceph-mon[30054]: osdmap e51: 9 total, 8 up, 9 in 2024-05-10T11:04:58.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:58 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:04:58.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:58 smithi027 ceph-mon[30054]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:04:58.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:58 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:04:58.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:58 smithi076 ceph-mon[34152]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-10T11:04:58.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:58 smithi076 ceph-mon[34152]: from='osd.8 [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:04:58.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:58 smithi076 ceph-mon[34152]: osdmap e51: 9 total, 8 up, 9 in 2024-05-10T11:04:58.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:58 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:04:58.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:58 smithi076 ceph-mon[34152]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:04:58.700 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:04:58.713 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:04:58 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:04:58.437+0000 7f294aef1700 -1 osd.8 0 waiting for initial osdmap 2024-05-10T11:04:58.713 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:04:58 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:04:58.446+0000 7f2946ce4700 -1 osd.8 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-10T11:04:59.657 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:59 smithi189 ceph-mon[34842]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-10T11:04:59.657 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:59 smithi189 ceph-mon[34842]: osdmap e52: 9 total, 8 up, 9 in 2024-05-10T11:04:59.657 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:59 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:04:59.657 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:59 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:04:59.657 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:04:59 smithi189 ceph-mon[34842]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:59.819 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:59 smithi027 ceph-mon[30054]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-10T11:04:59.819 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:59 smithi027 ceph-mon[30054]: osdmap e52: 9 total, 8 up, 9 in 2024-05-10T11:04:59.819 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:59 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:04:59.819 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:59 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:04:59.820 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:04:59 smithi027 ceph-mon[30054]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:04:59.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:59 smithi076 ceph-mon[34152]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-10T11:04:59.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:59 smithi076 ceph-mon[34152]: osdmap e52: 9 total, 8 up, 9 in 2024-05-10T11:04:59.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:59 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:04:59.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:59 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:04:59.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:04:59 smithi076 ceph-mon[34152]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-05-10T11:05:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:00 smithi027 ceph-mon[30054]: purged_snaps scrub starts 2024-05-10T11:05:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:00 smithi027 ceph-mon[30054]: purged_snaps scrub ok 2024-05-10T11:05:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:00 smithi027 ceph-mon[30054]: osd.8 [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386] boot 2024-05-10T11:05:00.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:00 smithi027 ceph-mon[30054]: osdmap e53: 9 total, 9 up, 9 in 2024-05-10T11:05:00.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:00 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:05:00.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:00 smithi189 ceph-mon[34842]: purged_snaps scrub starts 2024-05-10T11:05:00.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:00 smithi189 ceph-mon[34842]: purged_snaps scrub ok 2024-05-10T11:05:00.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:00 smithi189 ceph-mon[34842]: osd.8 [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386] boot 2024-05-10T11:05:00.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:00 smithi189 ceph-mon[34842]: osdmap e53: 9 total, 9 up, 9 in 2024-05-10T11:05:00.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:00 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:05:00.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:00 smithi076 ceph-mon[34152]: purged_snaps scrub starts 2024-05-10T11:05:00.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:00 smithi076 ceph-mon[34152]: purged_snaps scrub ok 2024-05-10T11:05:00.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:00 smithi076 ceph-mon[34152]: osd.8 [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386] boot 2024-05-10T11:05:00.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:00 smithi076 ceph-mon[34152]: osdmap e53: 9 total, 9 up, 9 in 2024-05-10T11:05:00.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:00 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:05:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:01 smithi189 ceph-mon[34842]: osdmap e54: 9 total, 9 up, 9 in 2024-05-10T11:05:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:01 smithi189 ceph-mon[34842]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:01.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:01.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:01.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:01.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:05:01.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:01.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:01 smithi027 ceph-mon[30054]: osdmap e54: 9 total, 9 up, 9 in 2024-05-10T11:05:01.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:01 smithi027 ceph-mon[30054]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:01.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:01.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:01.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:01.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:01.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:01.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:01.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:01.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:05:01.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:01.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:01 smithi076 ceph-mon[34152]: osdmap e54: 9 total, 9 up, 9 in 2024-05-10T11:05:01.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:01 smithi076 ceph-mon[34152]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:01.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:01.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:01.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:01.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:01.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:01.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:01.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:01.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:05:01.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:01.937 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-10T11:05:01.937 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-10T11:05:01.937 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-05-10T11:05:01.937 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-05-10T11:05:01.937 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0375319 s, 279 MB/s 2024-05-10T11:05:01.937 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-05-10T11:05:02.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:02 smithi027 ceph-mon[30054]: Detected new or changed devices on smithi189 2024-05-10T11:05:02.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:02 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi189 to 5696M 2024-05-10T11:05:02.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:02 smithi027 ceph-mon[30054]: osdmap e55: 9 total, 9 up, 9 in 2024-05-10T11:05:02.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:02 smithi189 ceph-mon[34842]: Detected new or changed devices on smithi189 2024-05-10T11:05:02.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:02 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi189 to 5696M 2024-05-10T11:05:02.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:02 smithi189 ceph-mon[34842]: osdmap e55: 9 total, 9 up, 9 in 2024-05-10T11:05:02.893 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_1 2024-05-10T11:05:02.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:02 smithi076 ceph-mon[34152]: Detected new or changed devices on smithi189 2024-05-10T11:05:02.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:02 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi189 to 5696M 2024-05-10T11:05:02.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:02 smithi076 ceph-mon[34152]: osdmap e55: 9 total, 9 up, 9 in 2024-05-10T11:05:03.379 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:05:03.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:03 smithi189 ceph-mon[34842]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:03.811 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:03 smithi027 ceph-mon[30054]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:03.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:03 smithi076 ceph-mon[34152]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.761+0000 7f2a83a00700 1 -- 172.21.15.27:0/1945749325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c075f00 msgr2=0x7f2a7c076380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:05:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.761+0000 7f2a83a00700 1 --2- 172.21.15.27:0/1945749325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c075f00 0x7f2a7c076380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009990 tx=0x7f2a68009ca0 comp rx=0 tx=0).stop 2024-05-10T11:05:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.761+0000 7f2a83a00700 1 -- 172.21.15.27:0/1945749325 shutdown_connections 2024-05-10T11:05:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.761+0000 7f2a83a00700 1 --2- 172.21.15.27:0/1945749325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a7c10fa50 0x7f2a7c111e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.761+0000 7f2a83a00700 1 --2- 172.21.15.27:0/1945749325 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a7c0774d0 0x7f2a7c075930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:05.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.761+0000 7f2a83a00700 1 --2- 172.21.15.27:0/1945749325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c075f00 0x7f2a7c076380 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:05.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.761+0000 7f2a83a00700 1 -- 172.21.15.27:0/1945749325 >> 172.21.15.27:0/1945749325 conn(0x7f2a7c0fea10 msgr2=0x7f2a7c100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:05:05.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.761+0000 7f2a83a00700 1 -- 172.21.15.27:0/1945749325 shutdown_connections 2024-05-10T11:05:05.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.761+0000 7f2a83a00700 1 -- 172.21.15.27:0/1945749325 wait complete. 2024-05-10T11:05:05.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.762+0000 7f2a83a00700 1 Processor -- start 2024-05-10T11:05:05.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.763+0000 7f2a83a00700 1 -- start start 2024-05-10T11:05:05.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.763+0000 7f2a83a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a7c075f00 0x7f2a7c19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:05:05.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.763+0000 7f2a83a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a7c0774d0 0x7f2a7c1a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.763+0000 7f2a83a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c10fa50 0x7f2a7c1a71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.763+0000 7f2a83a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c115550 con 0x7f2a7c075f00 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.763+0000 7f2a83a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c1153d0 con 0x7f2a7c0774d0 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.763+0000 7f2a83a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c1156d0 con 0x7f2a7c10fa50 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.764+0000 7f2a81000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a7c075f00 0x7f2a7c19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.764+0000 7f2a73e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a7c0774d0 0x7f2a7c1a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.764+0000 7f2a73e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a7c0774d0 0x7f2a7c1a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:49150/0 (socket says 172.21.15.27:49150) 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.764+0000 7f2a73e00700 1 -- 172.21.15.27:0/2777369790 learned_addr learned my addr 172.21.15.27:0/2777369790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.764+0000 7f2a81a00700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c10fa50 0x7f2a7c1a71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.764+0000 7f2a73e00700 1 -- 172.21.15.27:0/2777369790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c10fa50 msgr2=0x7f2a7c1a71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.764+0000 7f2a73e00700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c10fa50 0x7f2a7c1a71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.764+0000 7f2a73e00700 1 -- 172.21.15.27:0/2777369790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a7c075f00 msgr2=0x7f2a7c19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.764+0000 7f2a73e00700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a7c075f00 0x7f2a7c19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.764+0000 7f2a73e00700 1 -- 172.21.15.27:0/2777369790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7c1a7870 con 0x7f2a7c0774d0 2024-05-10T11:05:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.764+0000 7f2a81a00700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c10fa50 0x7f2a7c1a71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:05:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.765+0000 7f2a73e00700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a7c0774d0 0x7f2a7c1a01b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2a680098e0 tx=0x7f2a680057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:05:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.765+0000 7f2a7a000700 1 -- 172.21.15.27:0/2777369790 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68027070 con 0x7f2a7c0774d0 2024-05-10T11:05:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.765+0000 7f2a83a00700 1 -- 172.21.15.27:0/2777369790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a7c1a7b00 con 0x7f2a7c0774d0 2024-05-10T11:05:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.765+0000 7f2a7a000700 1 -- 172.21.15.27:0/2777369790 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a68022470 con 0x7f2a7c0774d0 2024-05-10T11:05:05.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.765+0000 7f2a7a000700 1 -- 172.21.15.27:0/2777369790 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68017650 con 0x7f2a7c0774d0 2024-05-10T11:05:05.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.765+0000 7f2a83a00700 1 -- 172.21.15.27:0/2777369790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a7c1a7f50 con 0x7f2a7c0774d0 2024-05-10T11:05:05.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.766+0000 7f2a83a00700 1 -- 172.21.15.27:0/2777369790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a7c06a190 con 0x7f2a7c0774d0 2024-05-10T11:05:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.769+0000 7f2a7a000700 1 -- 172.21.15.27:0/2777369790 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f2a6802a070 con 0x7f2a7c0774d0 2024-05-10T11:05:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.770+0000 7f2a7a000700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2a600b2880 0x7f2a600b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:05:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.770+0000 7f2a7a000700 1 -- 172.21.15.27:0/2777369790 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5829+0+0 (secure 0 0 0) 0x7f2a680e2a30 con 0x7f2a7c0774d0 2024-05-10T11:05:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.771+0000 7f2a81000700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2a600b2880 0x7f2a600b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:05:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.771+0000 7f2a81000700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2a600b2880 0x7f2a600b4d40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c1a1100 tx=0x7f2a6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:05:05.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:05.772+0000 7f2a7a000700 1 -- 172.21.15.27:0/2777369790 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f2a68031080 con 0x7f2a7c0774d0 2024-05-10T11:05:06.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:06.023+0000 7f2a83a00700 1 -- 172.21.15.27:0/2777369790 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2a7c0639a0 con 0x7f2a600b2880 2024-05-10T11:05:06.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:05 smithi027 ceph-mon[30054]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:06.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:05 smithi076 ceph-mon[34152]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:06.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:05 smithi189 ceph-mon[34842]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:07.038 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:06 smithi027 ceph-mon[30054]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:05:07.038 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:05:07.039 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:05:07.039 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:07.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:06 smithi076 ceph-mon[34152]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:05:07.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:05:07.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:05:07.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:07.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:06 smithi189 ceph-mon[34842]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:05:07.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:05:07.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:05:07.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:08.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:07 smithi027 ceph-mon[30054]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-05-10T11:05:08.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:07 smithi076 ceph-mon[34152]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-05-10T11:05:08.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:07 smithi189 ceph-mon[34842]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-05-10T11:05:10.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:09 smithi027 ceph-mon[30054]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-05-10T11:05:10.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:09 smithi076 ceph-mon[34152]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-05-10T11:05:10.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:09 smithi189 ceph-mon[34842]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-05-10T11:05:11.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:10 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3770433015' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f10253f-40eb-43c9-8e6f-f552d3fd736c"}]: dispatch 2024-05-10T11:05:11.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:10 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3770433015' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f10253f-40eb-43c9-8e6f-f552d3fd736c"}]': finished 2024-05-10T11:05:11.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:10 smithi027 ceph-mon[30054]: osdmap e56: 10 total, 9 up, 10 in 2024-05-10T11:05:11.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:10 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:11.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:10 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3770433015' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f10253f-40eb-43c9-8e6f-f552d3fd736c"}]: dispatch 2024-05-10T11:05:11.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:10 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3770433015' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f10253f-40eb-43c9-8e6f-f552d3fd736c"}]': finished 2024-05-10T11:05:11.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:10 smithi076 ceph-mon[34152]: osdmap e56: 10 total, 9 up, 10 in 2024-05-10T11:05:11.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:10 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:11.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:10 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3770433015' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f10253f-40eb-43c9-8e6f-f552d3fd736c"}]: dispatch 2024-05-10T11:05:11.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:10 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3770433015' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f10253f-40eb-43c9-8e6f-f552d3fd736c"}]': finished 2024-05-10T11:05:11.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:10 smithi189 ceph-mon[34842]: osdmap e56: 10 total, 9 up, 10 in 2024-05-10T11:05:11.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:10 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:12.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:11 smithi027 ceph-mon[30054]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-10T11:05:12.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:11 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2872758450' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:05:12.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:11 smithi076 ceph-mon[34152]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-10T11:05:12.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:11 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2872758450' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:05:12.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:11 smithi189 ceph-mon[34842]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-10T11:05:12.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:11 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2872758450' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:05:14.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:13 smithi027 ceph-mon[30054]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:05:14.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:13 smithi076 ceph-mon[34152]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:05:14.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:13 smithi189 ceph-mon[34842]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:05:16.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:15 smithi027 ceph-mon[30054]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:05:16.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:15 smithi076 ceph-mon[34152]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:05:16.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:15 smithi189 ceph-mon[34842]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:05:18.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:17 smithi027 ceph-mon[30054]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:18.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:17 smithi076 ceph-mon[34152]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:18.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:17 smithi189 ceph-mon[34842]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:20.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:19 smithi027 ceph-mon[30054]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:20.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:19 smithi076 ceph-mon[34152]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:19 smithi189 ceph-mon[34842]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:21.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-10T11:05:21.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:20 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:21.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:20 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-10T11:05:21.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:20 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:21.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:20 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-10T11:05:21.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:20 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:21.979 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:21 smithi027 ceph-mon[30054]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:21.980 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:21 smithi027 ceph-mon[30054]: Deploying daemon osd.9 on smithi027 2024-05-10T11:05:22.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:21 smithi076 ceph-mon[34152]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:22.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:21 smithi076 ceph-mon[34152]: Deploying daemon osd.9 on smithi027 2024-05-10T11:05:22.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:21 smithi189 ceph-mon[34842]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:22.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:21 smithi189 ceph-mon[34842]: Deploying daemon osd.9 on smithi027 2024-05-10T11:05:24.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:23 smithi027 ceph-mon[30054]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:24.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:23 smithi076 ceph-mon[34152]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:24.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:23 smithi189 ceph-mon[34842]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:26.076 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:25 smithi027 ceph-mon[30054]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:26.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:25 smithi076 ceph-mon[34152]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:26.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:25 smithi189 ceph-mon[34842]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:28.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:27 smithi027 ceph-mon[30054]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:28.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:27 smithi076 ceph-mon[34152]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:28.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:27 smithi189 ceph-mon[34842]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:30.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:29 smithi027 ceph-mon[30054]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:30.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:30.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:30.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:05:30.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:29 smithi076 ceph-mon[34152]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:30.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:30.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:30.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:05:30.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:29 smithi189 ceph-mon[34842]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:05:32.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:31 smithi027 ceph-mon[30054]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:32.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:31 smithi076 ceph-mon[34152]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:32.240 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:31 smithi189 ceph-mon[34842]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:34.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:33 smithi027 ceph-mon[30054]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:33 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:34.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:33 smithi027 ceph-mon[30054]: from='osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-10T11:05:34.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:33 smithi076 ceph-mon[34152]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:34.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:34.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:33 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:33 smithi076 ceph-mon[34152]: from='osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-10T11:05:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:33 smithi189 ceph-mon[34842]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:33 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:33 smithi189 ceph-mon[34842]: from='osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-10T11:05:34.933 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 9 on host 'smithi027' 2024-05-10T11:05:34.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.932+0000 7f2a7a000700 1 -- 172.21.15.27:0/2777369790 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2a7c0639a0 con 0x7f2a600b2880 2024-05-10T11:05:34.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.935+0000 7f2a83a00700 1 -- 172.21.15.27:0/2777369790 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2a600b2880 msgr2=0x7f2a600b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:05:34.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.935+0000 7f2a83a00700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2a600b2880 0x7f2a600b4d40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c1a1100 tx=0x7f2a6c009260 comp rx=0 tx=0).stop 2024-05-10T11:05:34.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.935+0000 7f2a83a00700 1 -- 172.21.15.27:0/2777369790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a7c0774d0 msgr2=0x7f2a7c1a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:05:34.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.935+0000 7f2a83a00700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a7c0774d0 0x7f2a7c1a01b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2a680098e0 tx=0x7f2a680057c0 comp rx=0 tx=0).stop 2024-05-10T11:05:34.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.936+0000 7f2a83a00700 1 -- 172.21.15.27:0/2777369790 shutdown_connections 2024-05-10T11:05:34.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.936+0000 7f2a83a00700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2a600b2880 0x7f2a600b4d40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:34.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.936+0000 7f2a83a00700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c10fa50 0x7f2a7c1a71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:34.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.936+0000 7f2a83a00700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a7c075f00 0x7f2a7c19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:34.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.936+0000 7f2a83a00700 1 --2- 172.21.15.27:0/2777369790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a7c0774d0 0x7f2a7c1a01b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:34.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.936+0000 7f2a83a00700 1 -- 172.21.15.27:0/2777369790 >> 172.21.15.27:0/2777369790 conn(0x7f2a7c0fea10 msgr2=0x7f2a7c102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:05:34.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.936+0000 7f2a83a00700 1 -- 172.21.15.27:0/2777369790 shutdown_connections 2024-05-10T11:05:34.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:05:34.936+0000 7f2a83a00700 1 -- 172.21.15.27:0/2777369790 wait complete. 2024-05-10T11:05:35.590 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:35 smithi027 ceph-mon[30054]: from='osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-10T11:05:35.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:35 smithi027 ceph-mon[30054]: osdmap e57: 10 total, 9 up, 10 in 2024-05-10T11:05:35.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:35 smithi027 ceph-mon[30054]: from='osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:05:35.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:35 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:35.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:35 smithi027 ceph-mon[30054]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:35.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:35 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:35.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:35 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:05:35.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:35 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:35.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:35 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:35.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:35 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:35.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:35 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:05:35.855 DEBUG:teuthology.orchestra.run.smithi027:osd.9> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.9.service 2024-05-10T11:05:35.861 INFO:tasks.cephadm:Deploying osd.10 on smithi076 with /dev/vg_nvme/lv_1... 2024-05-10T11:05:35.861 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-10T11:05:35.888 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:35 smithi076 ceph-mon[34152]: from='osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-10T11:05:35.888 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:35 smithi076 ceph-mon[34152]: osdmap e57: 10 total, 9 up, 10 in 2024-05-10T11:05:35.888 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:35 smithi076 ceph-mon[34152]: from='osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:05:35.888 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:35 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:35.888 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:35 smithi076 ceph-mon[34152]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:35.888 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:35 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:35.888 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:35 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:05:35.888 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:35 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:35.888 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:35 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:35.888 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:35 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:35.888 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:35 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:05:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:35 smithi189 ceph-mon[34842]: from='osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-10T11:05:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:35 smithi189 ceph-mon[34842]: osdmap e57: 10 total, 9 up, 10 in 2024-05-10T11:05:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:35 smithi189 ceph-mon[34842]: from='osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-10T11:05:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:35 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:35 smithi189 ceph-mon[34842]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:35 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:35 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:05:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:35 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:35 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:35 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:35.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:35 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:05:36.105 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.b/config 2024-05-10T11:05:36.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:36 smithi027 ceph-mon[30054]: from='osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-10T11:05:36.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:36 smithi027 ceph-mon[30054]: osdmap e58: 10 total, 9 up, 10 in 2024-05-10T11:05:36.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:36.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:36.889 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:36 smithi189 ceph-mon[34842]: from='osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-10T11:05:36.889 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:36 smithi189 ceph-mon[34842]: osdmap e58: 10 total, 9 up, 10 in 2024-05-10T11:05:36.889 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:36.889 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:36.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:36 smithi076 ceph-mon[34152]: from='osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-10T11:05:36.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:36 smithi076 ceph-mon[34152]: osdmap e58: 10 total, 9 up, 10 in 2024-05-10T11:05:36.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:36.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:37.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:37 smithi027 ceph-mon[30054]: purged_snaps scrub starts 2024-05-10T11:05:37.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:37 smithi027 ceph-mon[30054]: purged_snaps scrub ok 2024-05-10T11:05:37.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:37 smithi027 ceph-mon[30054]: osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197] boot 2024-05-10T11:05:37.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:37 smithi027 ceph-mon[30054]: osdmap e59: 10 total, 10 up, 10 in 2024-05-10T11:05:37.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:37 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:37.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:37 smithi027 ceph-mon[30054]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:37.866 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:37 smithi076 ceph-mon[34152]: purged_snaps scrub starts 2024-05-10T11:05:37.866 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:37 smithi076 ceph-mon[34152]: purged_snaps scrub ok 2024-05-10T11:05:37.866 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:37 smithi076 ceph-mon[34152]: osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197] boot 2024-05-10T11:05:37.866 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:37 smithi076 ceph-mon[34152]: osdmap e59: 10 total, 10 up, 10 in 2024-05-10T11:05:37.866 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:37 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:37.866 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:37 smithi076 ceph-mon[34152]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:37.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:37 smithi189 ceph-mon[34842]: purged_snaps scrub starts 2024-05-10T11:05:37.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:37 smithi189 ceph-mon[34842]: purged_snaps scrub ok 2024-05-10T11:05:37.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:37 smithi189 ceph-mon[34842]: osd.9 [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197] boot 2024-05-10T11:05:37.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:37 smithi189 ceph-mon[34842]: osdmap e59: 10 total, 10 up, 10 in 2024-05-10T11:05:37.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:37 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:05:37.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:37 smithi189 ceph-mon[34842]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-10T11:05:38.223 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-10T11:05:38.223 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-10T11:05:38.223 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-05-10T11:05:38.223 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-05-10T11:05:38.223 INFO:teuthology.orchestra.run.smithi076.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0118736 s, 883 MB/s 2024-05-10T11:05:38.223 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 2024-05-10T11:05:38.223 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-05-10T11:05:38.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:38 smithi027 ceph-mon[30054]: osdmap e60: 10 total, 10 up, 10 in 2024-05-10T11:05:38.921 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:38 smithi076 ceph-mon[34152]: osdmap e60: 10 total, 10 up, 10 in 2024-05-10T11:05:38.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi076:vg_nvme/lv_1 2024-05-10T11:05:38.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:38 smithi189 ceph-mon[34842]: osdmap e60: 10 total, 10 up, 10 in 2024-05-10T11:05:39.231 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.b/config 2024-05-10T11:05:39.633 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:39 smithi076 ceph-mon[34152]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:39.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:39 smithi027 ceph-mon[30054]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:39.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:39 smithi189 ceph-mon[34842]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:40.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.905+0000 7f023232d700 1 -- 172.21.15.76:0/3984568810 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f022c102e40 msgr2=0x7f022c1032a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:05:40.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.905+0000 7f023232d700 1 --2- 172.21.15.76:0/3984568810 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f022c102e40 0x7f022c1032a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f02200099c0 tx=0x7f0220009cd0 comp rx=0 tx=0).stop 2024-05-10T11:05:40.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.905+0000 7f023232d700 1 -- 172.21.15.76:0/3984568810 shutdown_connections 2024-05-10T11:05:40.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.905+0000 7f023232d700 1 --2- 172.21.15.76:0/3984568810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f022c101c40 0x7f022c102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:40.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.905+0000 7f023232d700 1 --2- 172.21.15.76:0/3984568810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c1037e0 0x7f022c10a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:40.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.905+0000 7f023232d700 1 --2- 172.21.15.76:0/3984568810 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f022c102e40 0x7f022c1032a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:40.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.905+0000 7f023232d700 1 -- 172.21.15.76:0/3984568810 >> 172.21.15.76:0/3984568810 conn(0x7f022c0fd1e0 msgr2=0x7f022c0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.905+0000 7f023232d700 1 -- 172.21.15.76:0/3984568810 shutdown_connections 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.905+0000 7f023232d700 1 -- 172.21.15.76:0/3984568810 wait complete. 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f023232d700 1 Processor -- start 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f023232d700 1 -- start start 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f023232d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f022c101c40 0x7f022c0ff3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f023232d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c102e40 0x7f022c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f023232d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f022c1037e0 0x7f022c1a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f023232d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f022c10d6b0 con 0x7f022c102e40 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f023232d700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f022c10d530 con 0x7f022c1037e0 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f023232d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f022c10d830 con 0x7f022c101c40 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f02308ca700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f022c1037e0 0x7f022c1a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f02308ca700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f022c1037e0 0x7f022c1a0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34902/0 (socket says 172.21.15.76:34902) 2024-05-10T11:05:40.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f02308ca700 1 -- 172.21.15.76:0/2623181710 learned_addr learned my addr 172.21.15.76:0/2623181710 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-05-10T11:05:40.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f022bfff700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f022c101c40 0x7f022c0ff3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:05:40.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f022b7fe700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c102e40 0x7f022c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:05:40.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f02308ca700 1 -- 172.21.15.76:0/2623181710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f022c101c40 msgr2=0x7f022c0ff3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:05:40.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f02308ca700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f022c101c40 0x7f022c0ff3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:40.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f02308ca700 1 -- 172.21.15.76:0/2623181710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c102e40 msgr2=0x7f022c19e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:05:40.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f02308ca700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c102e40 0x7f022c19e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:05:40.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f02308ca700 1 -- 172.21.15.76:0/2623181710 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f022c1a1360 con 0x7f022c1037e0 2024-05-10T11:05:40.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.906+0000 7f022b7fe700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c102e40 0x7f022c19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:05:40.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.907+0000 7f02308ca700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f022c1037e0 0x7f022c1a0a90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f022400ada0 tx=0x7f022400c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:05:40.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.907+0000 7f02297fa700 1 -- 172.21.15.76:0/2623181710 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0224017020 con 0x7f022c1037e0 2024-05-10T11:05:40.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.907+0000 7f023232d700 1 -- 172.21.15.76:0/2623181710 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f022c1a1650 con 0x7f022c1037e0 2024-05-10T11:05:40.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.907+0000 7f02297fa700 1 -- 172.21.15.76:0/2623181710 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f022400ce30 con 0x7f022c1037e0 2024-05-10T11:05:40.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.907+0000 7f02297fa700 1 -- 172.21.15.76:0/2623181710 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02240054a0 con 0x7f022c1037e0 2024-05-10T11:05:40.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.907+0000 7f023232d700 1 -- 172.21.15.76:0/2623181710 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f022c1a1b40 con 0x7f022c1037e0 2024-05-10T11:05:40.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.908+0000 7f023232d700 1 -- 172.21.15.76:0/2623181710 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f022c06a190 con 0x7f022c1037e0 2024-05-10T11:05:40.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.910+0000 7f02297fa700 1 -- 172.21.15.76:0/2623181710 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f022400c980 con 0x7f022c1037e0 2024-05-10T11:05:40.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.910+0000 7f02297fa700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f02140b2880 0x7f02140b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:05:40.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.911+0000 7f02297fa700 1 -- 172.21.15.76:0/2623181710 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6261+0+0 (secure 0 0 0) 0x7f02240e18a0 con 0x7f022c1037e0 2024-05-10T11:05:40.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.911+0000 7f022bfff700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f02140b2880 0x7f02140b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:05:40.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.912+0000 7f022bfff700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f02140b2880 0x7f02140b4d40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f021c005fd0 tx=0x7f021c005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:05:40.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:40.914+0000 7f02297fa700 1 -- 172.21.15.76:0/2623181710 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f0224021020 con 0x7f022c1037e0 2024-05-10T11:05:41.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:05:41.128+0000 7f023232d700 1 -- 172.21.15.76:0/2623181710 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f022c0638e0 con 0x7f02140b2880 2024-05-10T11:05:42.015 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:42.015 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: Detected new or changed devices on smithi027 2024-05-10T11:05:42.015 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:42.015 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:42.015 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:42.015 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:42.015 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:42.015 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:42.016 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi027 to 4272M 2024-05-10T11:05:42.016 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:42.016 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:42.016 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:05:42.016 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:42.016 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:05:42.016 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:05:42.016 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: Detected new or changed devices on smithi027 2024-05-10T11:05:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi027 to 4272M 2024-05-10T11:05:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:05:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:05:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:05:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: Detected new or changed devices on smithi027 2024-05-10T11:05:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:05:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi027 to 4272M 2024-05-10T11:05:42.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:42.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:42.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:05:42.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:42.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:05:42.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:05:42.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:43.108 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:42 smithi076 ceph-mon[34152]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:05:43.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:42 smithi027 ceph-mon[30054]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:05:43.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:42 smithi189 ceph-mon[34842]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:05:44.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:43 smithi027 ceph-mon[30054]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:44.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:43 smithi076 ceph-mon[34152]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:44.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:43 smithi189 ceph-mon[34842]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:45.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:44 smithi027 ceph-mon[30054]: from='client.? 172.21.15.76:0/1078555203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a916c16-6efc-4d99-8494-238b7cbeef63"}]: dispatch 2024-05-10T11:05:45.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:44 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a916c16-6efc-4d99-8494-238b7cbeef63"}]: dispatch 2024-05-10T11:05:45.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:44 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a916c16-6efc-4d99-8494-238b7cbeef63"}]': finished 2024-05-10T11:05:45.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:44 smithi027 ceph-mon[30054]: osdmap e61: 11 total, 10 up, 11 in 2024-05-10T11:05:45.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:05:45.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:44 smithi027 ceph-mon[30054]: from='client.? 172.21.15.76:0/620832935' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:05:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:44 smithi076 ceph-mon[34152]: from='client.? 172.21.15.76:0/1078555203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a916c16-6efc-4d99-8494-238b7cbeef63"}]: dispatch 2024-05-10T11:05:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:44 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a916c16-6efc-4d99-8494-238b7cbeef63"}]: dispatch 2024-05-10T11:05:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:44 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a916c16-6efc-4d99-8494-238b7cbeef63"}]': finished 2024-05-10T11:05:45.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:44 smithi076 ceph-mon[34152]: osdmap e61: 11 total, 10 up, 11 in 2024-05-10T11:05:45.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:05:45.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:44 smithi076 ceph-mon[34152]: from='client.? 172.21.15.76:0/620832935' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:05:45.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:44 smithi189 ceph-mon[34842]: from='client.? 172.21.15.76:0/1078555203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a916c16-6efc-4d99-8494-238b7cbeef63"}]: dispatch 2024-05-10T11:05:45.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:44 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a916c16-6efc-4d99-8494-238b7cbeef63"}]: dispatch 2024-05-10T11:05:45.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:44 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a916c16-6efc-4d99-8494-238b7cbeef63"}]': finished 2024-05-10T11:05:45.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:44 smithi189 ceph-mon[34842]: osdmap e61: 11 total, 10 up, 11 in 2024-05-10T11:05:45.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:05:45.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:44 smithi189 ceph-mon[34842]: from='client.? 172.21.15.76:0/620832935' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:05:46.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:46 smithi027 ceph-mon[30054]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:46.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:46 smithi076 ceph-mon[34152]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:46.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:46 smithi189 ceph-mon[34842]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:48.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:48 smithi027 ceph-mon[30054]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:48 smithi076 ceph-mon[34152]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:48.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:48 smithi189 ceph-mon[34842]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:50.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:50 smithi027 ceph-mon[30054]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:50.297 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:50 smithi076 ceph-mon[34152]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:50.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:50 smithi189 ceph-mon[34842]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:52.281 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:52 smithi076 ceph-mon[34152]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:52.281 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:52 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-10T11:05:52.281 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:52 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:52.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:52 smithi027 ceph-mon[30054]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:52.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:52 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-10T11:05:52.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:52 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:52.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:52 smithi189 ceph-mon[34842]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:52.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:52 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-10T11:05:52.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:52 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:05:53.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:53 smithi027 ceph-mon[30054]: Deploying daemon osd.10 on smithi076 2024-05-10T11:05:53.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:53 smithi076 ceph-mon[34152]: Deploying daemon osd.10 on smithi076 2024-05-10T11:05:53.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:53 smithi189 ceph-mon[34842]: Deploying daemon osd.10 on smithi076 2024-05-10T11:05:54.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:54 smithi027 ceph-mon[30054]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:54.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:54 smithi076 ceph-mon[34152]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:54.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:54 smithi189 ceph-mon[34842]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:56.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:56 smithi027 ceph-mon[30054]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:56.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:56 smithi076 ceph-mon[34152]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:56.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:56 smithi189 ceph-mon[34842]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:58.146 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:58 smithi076 ceph-mon[34152]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:58.146 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:58 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:58.146 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:58 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:05:58.146 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:05:58 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:58.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:58 smithi027 ceph-mon[30054]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:58.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:58 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:58.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:58 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:05:58.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:05:58 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:58.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:58 smithi189 ceph-mon[34842]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:05:58.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:58 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:05:58.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:58 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:05:58.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:05:58 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:00.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:00 smithi076 ceph-mon[34152]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:06:00.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:00 smithi027 ceph-mon[30054]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:06:00.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:00 smithi189 ceph-mon[34842]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:06:01.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.119+0000 7f02297fa700 1 -- 172.21.15.76:0/2623181710 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f022c0638e0 con 0x7f02140b2880 2024-05-10T11:06:01.120 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 10 on host 'smithi076' 2024-05-10T11:06:01.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.122+0000 7f023232d700 1 -- 172.21.15.76:0/2623181710 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f02140b2880 msgr2=0x7f02140b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:01.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.122+0000 7f023232d700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f02140b2880 0x7f02140b4d40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f021c005fd0 tx=0x7f021c005dd0 comp rx=0 tx=0).stop 2024-05-10T11:06:01.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.122+0000 7f023232d700 1 -- 172.21.15.76:0/2623181710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f022c1037e0 msgr2=0x7f022c1a0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:01.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.122+0000 7f023232d700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f022c1037e0 0x7f022c1a0a90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f022400ada0 tx=0x7f022400c330 comp rx=0 tx=0).stop 2024-05-10T11:06:01.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.122+0000 7f023232d700 1 -- 172.21.15.76:0/2623181710 shutdown_connections 2024-05-10T11:06:01.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.122+0000 7f023232d700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f02140b2880 0x7f02140b4d40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:01.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.122+0000 7f023232d700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f022c101c40 0x7f022c0ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:01.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.122+0000 7f023232d700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f022c1037e0 0x7f022c1a0a90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:01.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.122+0000 7f023232d700 1 --2- 172.21.15.76:0/2623181710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c102e40 0x7f022c19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:01.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.122+0000 7f023232d700 1 -- 172.21.15.76:0/2623181710 >> 172.21.15.76:0/2623181710 conn(0x7f022c0fd1e0 msgr2=0x7f022c1065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:01.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.122+0000 7f023232d700 1 -- 172.21.15.76:0/2623181710 shutdown_connections 2024-05-10T11:06:01.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-05-10T11:06:01.122+0000 7f023232d700 1 -- 172.21.15.76:0/2623181710 wait complete. 2024-05-10T11:06:01.887 DEBUG:teuthology.orchestra.run.smithi076:osd.10> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.10.service 2024-05-10T11:06:01.889 INFO:tasks.cephadm:Deploying osd.11 on smithi189 with /dev/vg_nvme/lv_1... 2024-05-10T11:06:01.890 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-10T11:06:02.138 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:06:02.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:02 smithi076 ceph-mon[34152]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:06:02.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:02 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:02.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:02 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:02.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:02 smithi076 ceph-mon[34152]: from='osd.10 [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-10T11:06:02.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:02 smithi076 ceph-mon[34152]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-10T11:06:02.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:02 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:02.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:02 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:02 smithi189 ceph-mon[34842]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:06:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:02 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:02 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:02 smithi189 ceph-mon[34842]: from='osd.10 [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-10T11:06:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:02 smithi189 ceph-mon[34842]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-10T11:06:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:02 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:02 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:02 smithi027 ceph-mon[30054]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:06:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:02 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:02 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:02 smithi027 ceph-mon[30054]: from='osd.10 [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-10T11:06:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:02 smithi027 ceph-mon[30054]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-10T11:06:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:02 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:02 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:03 smithi027 ceph-mon[30054]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-10T11:06:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:03 smithi027 ceph-mon[30054]: from='osd.10 [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:06:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:03 smithi027 ceph-mon[30054]: osdmap e62: 11 total, 10 up, 11 in 2024-05-10T11:06:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:03 smithi027 ceph-mon[30054]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:06:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:03 smithi027 ceph-mon[30054]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:06:03.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:03 smithi076 ceph-mon[34152]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-10T11:06:03.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:03 smithi076 ceph-mon[34152]: from='osd.10 [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:06:03.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:03 smithi076 ceph-mon[34152]: osdmap e62: 11 total, 10 up, 11 in 2024-05-10T11:06:03.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:03.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:03 smithi076 ceph-mon[34152]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:06:03.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:03 smithi076 ceph-mon[34152]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:06:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:03 smithi189 ceph-mon[34842]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-10T11:06:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:03 smithi189 ceph-mon[34842]: from='osd.10 [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:06:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:03 smithi189 ceph-mon[34842]: osdmap e62: 11 total, 10 up, 11 in 2024-05-10T11:06:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:03 smithi189 ceph-mon[34842]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-05-10T11:06:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:03 smithi189 ceph-mon[34842]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:06:03.875 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:06:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:06:03.520+0000 7fc2cd31e700 -1 osd.10 0 waiting for initial osdmap 2024-05-10T11:06:03.875 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:06:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:06:03.532+0000 7fc2c4908700 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-10T11:06:04.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:04 smithi076 ceph-mon[34152]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-05-10T11:06:04.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:04 smithi076 ceph-mon[34152]: osdmap e63: 11 total, 10 up, 11 in 2024-05-10T11:06:04.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:04 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:04.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:04 smithi076 ceph-mon[34152]: osdmap e64: 11 total, 10 up, 11 in 2024-05-10T11:06:04.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:04 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:04.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:04 smithi076 ceph-mon[34152]: from='osd.10 ' entity='osd.10' 2024-05-10T11:06:04.208 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-10T11:06:04.209 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-10T11:06:04.209 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-05-10T11:06:04.209 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-05-10T11:06:04.209 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0172032 s, 610 MB/s 2024-05-10T11:06:04.209 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-05-10T11:06:04.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:04 smithi027 ceph-mon[30054]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-05-10T11:06:04.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:04 smithi027 ceph-mon[30054]: osdmap e63: 11 total, 10 up, 11 in 2024-05-10T11:06:04.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:04 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:04.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:04 smithi027 ceph-mon[30054]: osdmap e64: 11 total, 10 up, 11 in 2024-05-10T11:06:04.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:04 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:04.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:04 smithi027 ceph-mon[30054]: from='osd.10 ' entity='osd.10' 2024-05-10T11:06:04.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:04 smithi189 ceph-mon[34842]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-05-10T11:06:04.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:04 smithi189 ceph-mon[34842]: osdmap e63: 11 total, 10 up, 11 in 2024-05-10T11:06:04.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:04 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:04.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:04 smithi189 ceph-mon[34842]: osdmap e64: 11 total, 10 up, 11 in 2024-05-10T11:06:04.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:04 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:04.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:04 smithi189 ceph-mon[34842]: from='osd.10 ' entity='osd.10' 2024-05-10T11:06:04.941 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_1 2024-05-10T11:06:05.194 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:06:05.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:05 smithi189 ceph-mon[34842]: purged_snaps scrub starts 2024-05-10T11:06:05.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:05 smithi189 ceph-mon[34842]: purged_snaps scrub ok 2024-05-10T11:06:05.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:05.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:05 smithi189 ceph-mon[34842]: osd.10 [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234] boot 2024-05-10T11:06:05.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:05 smithi189 ceph-mon[34842]: osdmap e65: 11 total, 11 up, 11 in 2024-05-10T11:06:05.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:05.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:05 smithi189 ceph-mon[34842]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:06:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:05 smithi027 ceph-mon[30054]: purged_snaps scrub starts 2024-05-10T11:06:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:05 smithi027 ceph-mon[30054]: purged_snaps scrub ok 2024-05-10T11:06:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:05 smithi027 ceph-mon[30054]: osd.10 [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234] boot 2024-05-10T11:06:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:05 smithi027 ceph-mon[30054]: osdmap e65: 11 total, 11 up, 11 in 2024-05-10T11:06:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:05 smithi027 ceph-mon[30054]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:06:05.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:05 smithi076 ceph-mon[34152]: purged_snaps scrub starts 2024-05-10T11:06:05.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:05 smithi076 ceph-mon[34152]: purged_snaps scrub ok 2024-05-10T11:06:05.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:05.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:05 smithi076 ceph-mon[34152]: osd.10 [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234] boot 2024-05-10T11:06:05.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:05 smithi076 ceph-mon[34152]: osdmap e65: 11 total, 11 up, 11 in 2024-05-10T11:06:05.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:06:05.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:05 smithi076 ceph-mon[34152]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-10T11:06:06.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: osdmap e66: 11 total, 11 up, 11 in 2024-05-10T11:06:06.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: Detected new or changed devices on smithi076 2024-05-10T11:06:06.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:06.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:06.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:06.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:06.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi076 to 4272M 2024-05-10T11:06:06.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:06.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:06.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:06:06.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:06.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:06.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: osdmap e66: 11 total, 11 up, 11 in 2024-05-10T11:06:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: Detected new or changed devices on smithi076 2024-05-10T11:06:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi076 to 4272M 2024-05-10T11:06:06.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:06.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:06.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:06:06.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:06.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:06.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: osdmap e66: 11 total, 11 up, 11 in 2024-05-10T11:06:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: Detected new or changed devices on smithi076 2024-05-10T11:06:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:06.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi076 to 4272M 2024-05-10T11:06:06.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:06.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:06.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:06:06.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:06.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:06.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:06.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.875+0000 7f46b9277700 1 -- 172.21.15.189:0/1490994267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b4103620 msgr2=0x7f46b4103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:06.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.875+0000 7f46b9277700 1 --2- 172.21.15.189:0/1490994267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b4103620 0x7f46b4103a80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009990 tx=0x7f46a8009ca0 comp rx=0 tx=0).stop 2024-05-10T11:06:06.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.875+0000 7f46b9277700 1 -- 172.21.15.189:0/1490994267 shutdown_connections 2024-05-10T11:06:06.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.875+0000 7f46b9277700 1 --2- 172.21.15.189:0/1490994267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4103fc0 0x7f46b410a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:06.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.875+0000 7f46b9277700 1 --2- 172.21.15.189:0/1490994267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46b4102420 0x7f46b4102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:06.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.875+0000 7f46b9277700 1 --2- 172.21.15.189:0/1490994267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b4103620 0x7f46b4103a80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:06.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.875+0000 7f46b9277700 1 -- 172.21.15.189:0/1490994267 >> 172.21.15.189:0/1490994267 conn(0x7f46b40fda00 msgr2=0x7f46b40ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:06.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.875+0000 7f46b9277700 1 -- 172.21.15.189:0/1490994267 shutdown_connections 2024-05-10T11:06:06.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.875+0000 7f46b9277700 1 -- 172.21.15.189:0/1490994267 wait complete. 2024-05-10T11:06:06.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.876+0000 7f46b9277700 1 Processor -- start 2024-05-10T11:06:06.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.876+0000 7f46b9277700 1 -- start start 2024-05-10T11:06:06.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.876+0000 7f46b9277700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46b4102420 0x7f46b406c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:06.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.876+0000 7f46b9277700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b4103620 0x7f46b419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:06.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.876+0000 7f46b9277700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4103fc0 0x7f46b41a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.876+0000 7f46b9277700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f46b410df30 con 0x7f46b4103fc0 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.876+0000 7f46b9277700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f46b410ddb0 con 0x7f46b4102420 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.876+0000 7f46b9277700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f46b410e0b0 con 0x7f46b4103620 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.876+0000 7f46b27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b4103620 0x7f46b419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.876+0000 7f46b27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b4103620 0x7f46b419e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35538/0 (socket says 172.21.15.189:35538) 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.876+0000 7f46b27fc700 1 -- 172.21.15.189:0/3432898421 learned_addr learned my addr 172.21.15.189:0/3432898421 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.876+0000 7f46b2ffd700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46b4102420 0x7f46b406c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.877+0000 7f46b27fc700 1 -- 172.21.15.189:0/3432898421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46b4102420 msgr2=0x7f46b406c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.877+0000 7f46b27fc700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46b4102420 0x7f46b406c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.877+0000 7f46b27fc700 1 -- 172.21.15.189:0/3432898421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4103fc0 msgr2=0x7f46b41a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.877+0000 7f46b27fc700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4103fc0 0x7f46b41a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.877+0000 7f46b27fc700 1 -- 172.21.15.189:0/3432898421 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b41a1310 con 0x7f46b4103620 2024-05-10T11:06:06.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.877+0000 7f46b27fc700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b4103620 0x7f46b419e400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f46a800b2e0 tx=0x7f46a800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:06.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.877+0000 7f469bfff700 1 -- 172.21.15.189:0/3432898421 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a8026070 con 0x7f46b4103620 2024-05-10T11:06:06.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.877+0000 7f46b9277700 1 -- 172.21.15.189:0/3432898421 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46b41981d0 con 0x7f46b4103620 2024-05-10T11:06:06.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.877+0000 7f469bfff700 1 -- 172.21.15.189:0/3432898421 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46a800be90 con 0x7f46b4103620 2024-05-10T11:06:06.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.877+0000 7f469bfff700 1 -- 172.21.15.189:0/3432898421 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a8017650 con 0x7f46b4103620 2024-05-10T11:06:06.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.877+0000 7f46b9277700 1 -- 172.21.15.189:0/3432898421 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46b4198730 con 0x7f46b4103620 2024-05-10T11:06:06.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.878+0000 7f46b9277700 1 -- 172.21.15.189:0/3432898421 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46b406a190 con 0x7f46b4103620 2024-05-10T11:06:06.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.880+0000 7f469bfff700 1 -- 172.21.15.189:0/3432898421 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f46a8029070 con 0x7f46b4103620 2024-05-10T11:06:06.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.880+0000 7f469bfff700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f469c0b2880 0x7f469c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:06.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.881+0000 7f469bfff700 1 -- 172.21.15.189:0/3432898421 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6694+0+0 (secure 0 0 0) 0x7f46a80e3330 con 0x7f46b4103620 2024-05-10T11:06:06.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.881+0000 7f46b2ffd700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f469c0b2880 0x7f469c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:06.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.882+0000 7f46b2ffd700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f469c0b2880 0x7f469c0b4d40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f46b4103480 tx=0x7f46a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:06.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:06.886+0000 7f469bfff700 1 -- 172.21.15.189:0/3432898421 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f46a8030070 con 0x7f46b4103620 2024-05-10T11:06:07.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:07.099+0000 7f46b9277700 1 -- 172.21.15.189:0/3432898421 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f46b40638e0 con 0x7f469c0b2880 2024-05-10T11:06:07.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:07 smithi189 ceph-mon[34842]: osdmap e67: 11 total, 11 up, 11 in 2024-05-10T11:06:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:07 smithi189 ceph-mon[34842]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:06:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:06:07.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:07 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:07.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:07 smithi027 ceph-mon[30054]: osdmap e67: 11 total, 11 up, 11 in 2024-05-10T11:06:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:07 smithi027 ceph-mon[30054]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:06:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:06:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:07 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:07 smithi076 ceph-mon[34152]: osdmap e67: 11 total, 11 up, 11 in 2024-05-10T11:06:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:07 smithi076 ceph-mon[34152]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-10T11:06:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-10T11:06:07.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:07 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:08.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:08 smithi189 ceph-mon[34842]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:06:08.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:08 smithi027 ceph-mon[30054]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:06:08.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:08 smithi076 ceph-mon[34152]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:06:09.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:09 smithi189 ceph-mon[34842]: pgmap v261: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:09.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:09 smithi027 ceph-mon[30054]: pgmap v261: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:09 smithi076 ceph-mon[34152]: pgmap v261: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:10.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:10 smithi189 ceph-mon[34842]: from='client.? 172.21.15.189:0/964035830' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab1431f2-89cd-447b-aec8-e2dc786564d0"}]: dispatch 2024-05-10T11:06:10.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:10 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab1431f2-89cd-447b-aec8-e2dc786564d0"}]: dispatch 2024-05-10T11:06:10.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:10 smithi189 ceph-mon[34842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab1431f2-89cd-447b-aec8-e2dc786564d0"}]': finished 2024-05-10T11:06:10.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:10 smithi189 ceph-mon[34842]: osdmap e68: 12 total, 11 up, 12 in 2024-05-10T11:06:10.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:10 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:10.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:10 smithi027 ceph-mon[30054]: from='client.? 172.21.15.189:0/964035830' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab1431f2-89cd-447b-aec8-e2dc786564d0"}]: dispatch 2024-05-10T11:06:10.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:10 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab1431f2-89cd-447b-aec8-e2dc786564d0"}]: dispatch 2024-05-10T11:06:10.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:10 smithi027 ceph-mon[30054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab1431f2-89cd-447b-aec8-e2dc786564d0"}]': finished 2024-05-10T11:06:10.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:10 smithi027 ceph-mon[30054]: osdmap e68: 12 total, 11 up, 12 in 2024-05-10T11:06:10.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:10 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:10 smithi076 ceph-mon[34152]: from='client.? 172.21.15.189:0/964035830' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab1431f2-89cd-447b-aec8-e2dc786564d0"}]: dispatch 2024-05-10T11:06:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:10 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab1431f2-89cd-447b-aec8-e2dc786564d0"}]: dispatch 2024-05-10T11:06:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:10 smithi076 ceph-mon[34152]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab1431f2-89cd-447b-aec8-e2dc786564d0"}]': finished 2024-05-10T11:06:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:10 smithi076 ceph-mon[34152]: osdmap e68: 12 total, 11 up, 12 in 2024-05-10T11:06:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:10 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:11.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:11 smithi027 ceph-mon[30054]: from='client.? 172.21.15.189:0/1168403033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:06:11.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:11 smithi027 ceph-mon[30054]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-10T11:06:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:11 smithi076 ceph-mon[34152]: from='client.? 172.21.15.189:0/1168403033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:06:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:11 smithi076 ceph-mon[34152]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-10T11:06:11.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:11 smithi189 ceph-mon[34842]: from='client.? 172.21.15.189:0/1168403033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-10T11:06:11.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:11 smithi189 ceph-mon[34842]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-10T11:06:14.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:13 smithi027 ceph-mon[30054]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-10T11:06:14.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:13 smithi076 ceph-mon[34152]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-10T11:06:14.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:13 smithi189 ceph-mon[34842]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-10T11:06:16.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:15 smithi027 ceph-mon[30054]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-10T11:06:16.124 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:15 smithi189 ceph-mon[34842]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-10T11:06:16.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:15 smithi076 ceph-mon[34152]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-10T11:06:18.054 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:17 smithi189 ceph-mon[34842]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:06:18.055 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:17 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-10T11:06:18.055 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:17 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:18.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:17 smithi027 ceph-mon[30054]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:06:18.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:17 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-10T11:06:18.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:17 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:18.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:17 smithi076 ceph-mon[34152]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:06:18.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:17 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-10T11:06:18.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:17 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:19.078 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:18 smithi189 ceph-mon[34842]: Deploying daemon osd.11 on smithi189 2024-05-10T11:06:19.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:18 smithi027 ceph-mon[30054]: Deploying daemon osd.11 on smithi189 2024-05-10T11:06:19.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:18 smithi076 ceph-mon[34152]: Deploying daemon osd.11 on smithi189 2024-05-10T11:06:19.911 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:19 smithi189 ceph-mon[34842]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:06:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:19 smithi027 ceph-mon[30054]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:06:20.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:19 smithi076 ceph-mon[34152]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-10T11:06:21.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:21 smithi189 ceph-mon[34842]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:22.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:21 smithi027 ceph-mon[30054]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:22.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:21 smithi076 ceph-mon[34152]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:23.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:23 smithi189 ceph-mon[34842]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:23.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:23 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:23.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:23 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:23.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:23 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:06:24.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:23 smithi027 ceph-mon[30054]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:24.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:23 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:24.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:23 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:24.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:23 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:06:24.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:23 smithi076 ceph-mon[34152]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:24.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:23 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:24.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:23 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:24.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:23 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:06:26.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:25 smithi027 ceph-mon[30054]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:26.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:25 smithi189 ceph-mon[34842]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:26.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:25 smithi076 ceph-mon[34152]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:27.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.339+0000 7f469bfff700 1 -- 172.21.15.189:0/3432898421 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f46b40638e0 con 0x7f469c0b2880 2024-05-10T11:06:27.340 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 11 on host 'smithi189' 2024-05-10T11:06:27.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.341+0000 7f46b9277700 1 -- 172.21.15.189:0/3432898421 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f469c0b2880 msgr2=0x7f469c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:27.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.341+0000 7f46b9277700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f469c0b2880 0x7f469c0b4d40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f46b4103480 tx=0x7f46a4009260 comp rx=0 tx=0).stop 2024-05-10T11:06:27.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.341+0000 7f46b9277700 1 -- 172.21.15.189:0/3432898421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b4103620 msgr2=0x7f46b419e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:27.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.341+0000 7f46b9277700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b4103620 0x7f46b419e400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f46a800b2e0 tx=0x7f46a800f690 comp rx=0 tx=0).stop 2024-05-10T11:06:27.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.341+0000 7f46b9277700 1 -- 172.21.15.189:0/3432898421 shutdown_connections 2024-05-10T11:06:27.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.341+0000 7f46b9277700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f469c0b2880 0x7f469c0b4d40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:27.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.341+0000 7f46b9277700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4103fc0 0x7f46b41a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:27.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.341+0000 7f46b9277700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46b4102420 0x7f46b406c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:27.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.341+0000 7f46b9277700 1 --2- 172.21.15.189:0/3432898421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b4103620 0x7f46b419e400 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:27.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.341+0000 7f46b9277700 1 -- 172.21.15.189:0/3432898421 >> 172.21.15.189:0/3432898421 conn(0x7f46b40fda00 msgr2=0x7f46b4106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:27.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.341+0000 7f46b9277700 1 -- 172.21.15.189:0/3432898421 shutdown_connections 2024-05-10T11:06:27.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:27.341+0000 7f46b9277700 1 -- 172.21.15.189:0/3432898421 wait complete. 2024-05-10T11:06:27.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:27 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:27 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:27 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:27.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:27 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:27.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:27 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:27 smithi189 ceph-mon[34842]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:27.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:27 smithi189 ceph-mon[34842]: from='osd.11 [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-10T11:06:27.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:27 smithi189 ceph-mon[34842]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-10T11:06:27.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:27 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:06:27.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:27 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:27 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:27 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:27 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:27 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:27.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:27 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:27.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:27 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:27 smithi027 ceph-mon[30054]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:27.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:27 smithi027 ceph-mon[30054]: from='osd.11 [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-10T11:06:27.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:27 smithi027 ceph-mon[30054]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-10T11:06:27.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:27 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:06:27.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:27 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:27 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:27 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:27 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:27 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:27.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:27 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:27.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:27 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:27 smithi076 ceph-mon[34152]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:27.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:27 smithi076 ceph-mon[34152]: from='osd.11 [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-10T11:06:27.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:27 smithi076 ceph-mon[34152]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-10T11:06:27.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:27 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:06:27.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:27 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:27.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:27 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:28.049 DEBUG:teuthology.orchestra.run.smithi189:osd.11> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.11.service 2024-05-10T11:06:28.052 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-10T11:06:28.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd stat -f json 2024-05-10T11:06:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:06:28.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:28 smithi027 ceph-mon[30054]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-10T11:06:28.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:28 smithi027 ceph-mon[30054]: from='osd.11 [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:06:28.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:28 smithi027 ceph-mon[30054]: osdmap e69: 12 total, 11 up, 12 in 2024-05-10T11:06:28.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:28 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:28.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:28 smithi027 ceph-mon[30054]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:06:28.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:28 smithi076 ceph-mon[34152]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-10T11:06:28.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:28 smithi076 ceph-mon[34152]: from='osd.11 [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:06:28.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:28 smithi076 ceph-mon[34152]: osdmap e69: 12 total, 11 up, 12 in 2024-05-10T11:06:28.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:28 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:28.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:28 smithi076 ceph-mon[34152]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:06:28.967 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:28 smithi189 ceph-mon[34842]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-10T11:06:28.967 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:28 smithi189 ceph-mon[34842]: from='osd.11 [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:06:28.967 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:28 smithi189 ceph-mon[34842]: osdmap e69: 12 total, 11 up, 12 in 2024-05-10T11:06:28.967 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:28 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:28.967 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:28 smithi189 ceph-mon[34842]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-10T11:06:29.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:06:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:06:28.965+0000 7f11cd22b700 -1 osd.11 0 waiting for initial osdmap 2024-05-10T11:06:29.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:06:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:06:28.978+0000 7f11c5016700 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-10T11:06:29.637 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:29 smithi189 ceph-mon[34842]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-10T11:06:29.637 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:29 smithi189 ceph-mon[34842]: osdmap e70: 12 total, 11 up, 12 in 2024-05-10T11:06:29.637 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:29.638 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:29.638 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:29 smithi189 ceph-mon[34842]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:29.638 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:29 smithi189 ceph-mon[34842]: from='osd.11 ' entity='osd.11' 2024-05-10T11:06:29.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:29 smithi027 ceph-mon[30054]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-10T11:06:29.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:29 smithi027 ceph-mon[30054]: osdmap e70: 12 total, 11 up, 12 in 2024-05-10T11:06:29.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:29.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:29.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:29 smithi027 ceph-mon[30054]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:29.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:29 smithi027 ceph-mon[30054]: from='osd.11 ' entity='osd.11' 2024-05-10T11:06:29.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:29 smithi076 ceph-mon[34152]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-10T11:06:29.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:29 smithi076 ceph-mon[34152]: osdmap e70: 12 total, 11 up, 12 in 2024-05-10T11:06:29.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:29.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:29.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:29 smithi076 ceph-mon[34152]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-10T11:06:29.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:29 smithi076 ceph-mon[34152]: from='osd.11 ' entity='osd.11' 2024-05-10T11:06:30.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:30 smithi189 ceph-mon[34842]: purged_snaps scrub starts 2024-05-10T11:06:30.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:30 smithi189 ceph-mon[34842]: purged_snaps scrub ok 2024-05-10T11:06:30.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:30.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:30 smithi189 ceph-mon[34842]: osd.11 [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096] boot 2024-05-10T11:06:30.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:30 smithi189 ceph-mon[34842]: osdmap e71: 12 total, 12 up, 12 in 2024-05-10T11:06:30.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:30.824 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:30 smithi027 ceph-mon[30054]: purged_snaps scrub starts 2024-05-10T11:06:30.825 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:30 smithi027 ceph-mon[30054]: purged_snaps scrub ok 2024-05-10T11:06:30.825 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:30.825 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:30 smithi027 ceph-mon[30054]: osd.11 [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096] boot 2024-05-10T11:06:30.825 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:30 smithi027 ceph-mon[30054]: osdmap e71: 12 total, 12 up, 12 in 2024-05-10T11:06:30.825 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:30.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.910+0000 7f4572400700 1 -- 172.21.15.27:0/3290672342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f456c106840 msgr2=0x7f456c106ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:30.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.910+0000 7f4572400700 1 --2- 172.21.15.27:0/3290672342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f456c106840 0x7f456c106ca0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f45580099c0 tx=0x7f4558009cd0 comp rx=0 tx=0).stop 2024-05-10T11:06:30.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.910+0000 7f4572400700 1 -- 172.21.15.27:0/3290672342 shutdown_connections 2024-05-10T11:06:30.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.910+0000 7f4572400700 1 --2- 172.21.15.27:0/3290672342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456c105640 0x7f456c105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:30.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.910+0000 7f4572400700 1 --2- 172.21.15.27:0/3290672342 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f456c1071e0 0x7f456c10da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:30.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.910+0000 7f4572400700 1 --2- 172.21.15.27:0/3290672342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f456c106840 0x7f456c106ca0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:30.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.910+0000 7f4572400700 1 -- 172.21.15.27:0/3290672342 >> 172.21.15.27:0/3290672342 conn(0x7f456c100be0 msgr2=0x7f456c103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:30.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.910+0000 7f4572400700 1 -- 172.21.15.27:0/3290672342 shutdown_connections 2024-05-10T11:06:30.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.910+0000 7f4572400700 1 -- 172.21.15.27:0/3290672342 wait complete. 2024-05-10T11:06:30.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.912+0000 7f4572400700 1 Processor -- start 2024-05-10T11:06:30.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.912+0000 7f4572400700 1 -- start start 2024-05-10T11:06:30.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.912+0000 7f4572400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f456c105640 0x7f456c19b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:30.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.912+0000 7f4572400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f456c106840 0x7f456c19bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:30.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.912+0000 7f4572400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456c1071e0 0x7f456c1a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:30.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.912+0000 7f4572400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f456c111100 con 0x7f456c105640 2024-05-10T11:06:30.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.912+0000 7f4572400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f456c110f80 con 0x7f456c106840 2024-05-10T11:06:30.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.912+0000 7f4572400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f456c111280 con 0x7f456c1071e0 2024-05-10T11:06:30.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.913+0000 7f456be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f456c106840 0x7f456c19bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:30.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.913+0000 7f456be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f456c106840 0x7f456c19bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48584/0 (socket says 172.21.15.27:48584) 2024-05-10T11:06:30.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.913+0000 7f456be00700 1 -- 172.21.15.27:0/1940328848 learned_addr learned my addr 172.21.15.27:0/1940328848 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:06:30.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.913+0000 7f4571000700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f456c105640 0x7f456c19b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:30.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.913+0000 7f4571a00700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456c1071e0 0x7f456c1a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:30.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.913+0000 7f456be00700 1 -- 172.21.15.27:0/1940328848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456c1071e0 msgr2=0x7f456c1a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:30.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.913+0000 7f456be00700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456c1071e0 0x7f456c1a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:30.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.913+0000 7f456be00700 1 -- 172.21.15.27:0/1940328848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f456c105640 msgr2=0x7f456c19b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:30.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.913+0000 7f456be00700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f456c105640 0x7f456c19b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:30.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.913+0000 7f456be00700 1 -- 172.21.15.27:0/1940328848 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f456c1a33e0 con 0x7f456c106840 2024-05-10T11:06:30.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.913+0000 7f4571a00700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456c1071e0 0x7f456c1a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:06:30.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.914+0000 7f456be00700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f456c106840 0x7f456c19bd10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f455800fa00 tx=0x7f455800fa30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:30.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.914+0000 7f4569600700 1 -- 172.21.15.27:0/1940328848 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4558027070 con 0x7f456c106840 2024-05-10T11:06:30.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.914+0000 7f4569600700 1 -- 172.21.15.27:0/1940328848 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4558022920 con 0x7f456c106840 2024-05-10T11:06:30.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.914+0000 7f4569600700 1 -- 172.21.15.27:0/1940328848 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4558009e60 con 0x7f456c106840 2024-05-10T11:06:30.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.914+0000 7f4572400700 1 -- 172.21.15.27:0/1940328848 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f456c1a3670 con 0x7f456c106840 2024-05-10T11:06:30.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.914+0000 7f4572400700 1 -- 172.21.15.27:0/1940328848 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f456c1a3b20 con 0x7f456c106840 2024-05-10T11:06:30.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.915+0000 7f4572400700 1 -- 172.21.15.27:0/1940328848 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f456c06a190 con 0x7f456c106840 2024-05-10T11:06:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.918+0000 7f4569600700 1 -- 172.21.15.27:0/1940328848 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f455802a070 con 0x7f456c106840 2024-05-10T11:06:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.919+0000 7f4569600700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f45540b2900 0x7f45540b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.919+0000 7f4569600700 1 -- 172.21.15.27:0/1940328848 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f45580e2970 con 0x7f456c106840 2024-05-10T11:06:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.919+0000 7f4571000700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f45540b2900 0x7f45540b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.920+0000 7f4571000700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f45540b2900 0x7f45540b4dc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f456c1066a0 tx=0x7f456000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:30.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:30.924+0000 7f4569600700 1 -- 172.21.15.27:0/1940328848 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4558013070 con 0x7f456c106840 2024-05-10T11:06:30.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:30 smithi076 ceph-mon[34152]: purged_snaps scrub starts 2024-05-10T11:06:30.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:30 smithi076 ceph-mon[34152]: purged_snaps scrub ok 2024-05-10T11:06:30.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:30 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:30.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:30 smithi076 ceph-mon[34152]: osd.11 [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096] boot 2024-05-10T11:06:30.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:30 smithi076 ceph-mon[34152]: osdmap e71: 12 total, 12 up, 12 in 2024-05-10T11:06:30.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:30 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:06:31.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.166+0000 7f4572400700 1 -- 172.21.15.27:0/1940328848 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f456c0640b0 con 0x7f456c106840 2024-05-10T11:06:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.172+0000 7f4569600700 1 -- 172.21.15.27:0/1940328848 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7f45580aab20 con 0x7f456c106840 2024-05-10T11:06:31.173 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:06:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.176+0000 7f4572400700 1 -- 172.21.15.27:0/1940328848 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f45540b2900 msgr2=0x7f45540b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.176+0000 7f4572400700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f45540b2900 0x7f45540b4dc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f456c1066a0 tx=0x7f456000b410 comp rx=0 tx=0).stop 2024-05-10T11:06:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.176+0000 7f4572400700 1 -- 172.21.15.27:0/1940328848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f456c106840 msgr2=0x7f456c19bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.176+0000 7f4572400700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f456c106840 0x7f456c19bd10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f455800fa00 tx=0x7f455800fa30 comp rx=0 tx=0).stop 2024-05-10T11:06:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.176+0000 7f4572400700 1 -- 172.21.15.27:0/1940328848 shutdown_connections 2024-05-10T11:06:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.176+0000 7f4572400700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f45540b2900 0x7f45540b4dc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.176+0000 7f4572400700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456c1071e0 0x7f456c1a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.176+0000 7f4572400700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f456c105640 0x7f456c19b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.176+0000 7f4572400700 1 --2- 172.21.15.27:0/1940328848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f456c106840 0x7f456c19bd10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.176+0000 7f4572400700 1 -- 172.21.15.27:0/1940328848 >> 172.21.15.27:0/1940328848 conn(0x7f456c100be0 msgr2=0x7f456c109b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.176+0000 7f4572400700 1 -- 172.21.15.27:0/1940328848 shutdown_connections 2024-05-10T11:06:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:31.176+0000 7f4572400700 1 -- 172.21.15.27:0/1940328848 wait complete. 2024-05-10T11:06:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:32.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: osdmap e72: 12 total, 12 up, 12 in 2024-05-10T11:06:32.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1940328848' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-10T11:06:32.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:32.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:32.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:32.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:32.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:32.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:32.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:32.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:32.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:32.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:31 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1715339189,"num_in_osds":12,"osd_in_since":1715339169,"num_remapped_pgs":0} 2024-05-10T11:06:32.140 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:06:32.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: osdmap e72: 12 total, 12 up, 12 in 2024-05-10T11:06:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1940328848' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-10T11:06:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:32.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:31 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: osdmap e72: 12 total, 12 up, 12 in 2024-05-10T11:06:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1940328848' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-10T11:06:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:32.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:32.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:32.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:32.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:32.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:06:32.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:32.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:32.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:32.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:31 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:06:33.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:32 smithi027 ceph-mon[30054]: Detected new or changed devices on smithi189 2024-05-10T11:06:33.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:32 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi189 to 4272M 2024-05-10T11:06:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:32 smithi076 ceph-mon[34152]: Detected new or changed devices on smithi189 2024-05-10T11:06:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:32 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi189 to 4272M 2024-05-10T11:06:33.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:32 smithi189 ceph-mon[34842]: Detected new or changed devices on smithi189 2024-05-10T11:06:33.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:32 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi189 to 4272M 2024-05-10T11:06:34.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:33 smithi027 ceph-mon[30054]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:34.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:33 smithi076 ceph-mon[34152]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:34.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:33 smithi189 ceph-mon[34842]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:35.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.022+0000 7f4defe00700 1 -- 172.21.15.27:0/4155978099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4df0104f10 msgr2=0x7f4df010d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:35.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.022+0000 7f4defe00700 1 --2- 172.21.15.27:0/4155978099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4df0104f10 0x7f4df010d400 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc009990 tx=0x7f4ddc009ca0 comp rx=0 tx=0).stop 2024-05-10T11:06:35.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.022+0000 7f4defe00700 1 -- 172.21.15.27:0/4155978099 shutdown_connections 2024-05-10T11:06:35.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.022+0000 7f4defe00700 1 --2- 172.21.15.27:0/4155978099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4df01045f0 0x7f4df01049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:35.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.022+0000 7f4defe00700 1 --2- 172.21.15.27:0/4155978099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4df010d9d0 0x7f4df010fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:35.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.022+0000 7f4defe00700 1 --2- 172.21.15.27:0/4155978099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4df0104f10 0x7f4df010d400 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:35.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.022+0000 7f4defe00700 1 -- 172.21.15.27:0/4155978099 >> 172.21.15.27:0/4155978099 conn(0x7f4df00fddc0 msgr2=0x7f4df01001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:35.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.022+0000 7f4defe00700 1 -- 172.21.15.27:0/4155978099 shutdown_connections 2024-05-10T11:06:35.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.022+0000 7f4defe00700 1 -- 172.21.15.27:0/4155978099 wait complete. 2024-05-10T11:06:35.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.024+0000 7f4defe00700 1 Processor -- start 2024-05-10T11:06:35.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.024+0000 7f4defe00700 1 -- start start 2024-05-10T11:06:35.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.024+0000 7f4defe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4df01045f0 0x7f4df019bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:35.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.024+0000 7f4defe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4df0104f10 0x7f4df019c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:35.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4defe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4df010d9d0 0x7f4df01a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:35.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4defe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4df01135e0 con 0x7f4df010d9d0 2024-05-10T11:06:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4defe00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4df0113460 con 0x7f4df01045f0 2024-05-10T11:06:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4defe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4df0113760 con 0x7f4df0104f10 2024-05-10T11:06:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4deea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4df01045f0 0x7f4df019bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4dee000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4df0104f10 0x7f4df019c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4deea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4df01045f0 0x7f4df019bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:46556/0 (socket says 172.21.15.27:46556) 2024-05-10T11:06:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4def400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4df010d9d0 0x7f4df01a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4deea00700 1 -- 172.21.15.27:0/1761662567 learned_addr learned my addr 172.21.15.27:0/1761662567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:06:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4deea00700 1 -- 172.21.15.27:0/1761662567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4df0104f10 msgr2=0x7f4df019c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4deea00700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4df0104f10 0x7f4df019c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4deea00700 1 -- 172.21.15.27:0/1761662567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4df010d9d0 msgr2=0x7f4df01a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4deea00700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4df010d9d0 0x7f4df01a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4deea00700 1 -- 172.21.15.27:0/1761662567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df01a0d20 con 0x7f4df01045f0 2024-05-10T11:06:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.025+0000 7f4def400700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4df010d9d0 0x7f4df01a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:06:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.026+0000 7f4deea00700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4df01045f0 0x7f4df019bd10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4de000e9a0 tx=0x7f4de000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.026+0000 7f4de7400700 1 -- 172.21.15.27:0/1761662567 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de000cbd0 con 0x7f4df01045f0 2024-05-10T11:06:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.026+0000 7f4defe00700 1 -- 172.21.15.27:0/1761662567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df01a1010 con 0x7f4df01045f0 2024-05-10T11:06:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.026+0000 7f4de7400700 1 -- 172.21.15.27:0/1761662567 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4de000cd70 con 0x7f4df01045f0 2024-05-10T11:06:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.026+0000 7f4de7400700 1 -- 172.21.15.27:0/1761662567 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0018680 con 0x7f4df01045f0 2024-05-10T11:06:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.026+0000 7f4defe00700 1 -- 172.21.15.27:0/1761662567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df01a8890 con 0x7f4df01045f0 2024-05-10T11:06:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.027+0000 7f4defe00700 1 -- 172.21.15.27:0/1761662567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df006a190 con 0x7f4df01045f0 2024-05-10T11:06:35.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.030+0000 7f4de7400700 1 -- 172.21.15.27:0/1761662567 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f4de0010430 con 0x7f4df01045f0 2024-05-10T11:06:35.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.031+0000 7f4de7400700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4dd40b2880 0x7f4dd40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:35.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.031+0000 7f4de7400700 1 -- 172.21.15.27:0/1761662567 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f4de00e2090 con 0x7f4df01045f0 2024-05-10T11:06:35.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.031+0000 7f4dee000700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4dd40b2880 0x7f4dd40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:35.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.032+0000 7f4dee000700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4dd40b2880 0x7f4dd40b4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4df01102d0 tx=0x7f4ddc00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.035+0000 7f4de7400700 1 -- 172.21.15.27:0/1761662567 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4de00aa1b0 con 0x7f4df01045f0 2024-05-10T11:06:35.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.274+0000 7f4defe00700 1 -- 172.21.15.27:0/1761662567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4df01a8b20 con 0x7f4df01045f0 2024-05-10T11:06:35.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.276+0000 7f4de7400700 1 -- 172.21.15.27:0/1761662567 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18384 (secure 0 0 0) 0x7f4de00a9ab0 con 0x7f4df01045f0 2024-05-10T11:06:35.277 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:06:35.277 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":72,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:06:30.974758+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:06:35.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.281+0000 7f4defe00700 1 -- 172.21.15.27:0/1761662567 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4dd40b2880 msgr2=0x7f4dd40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:35.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.281+0000 7f4defe00700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4dd40b2880 0x7f4dd40b4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4df01102d0 tx=0x7f4ddc00b260 comp rx=0 tx=0).stop 2024-05-10T11:06:35.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.281+0000 7f4defe00700 1 -- 172.21.15.27:0/1761662567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4df01045f0 msgr2=0x7f4df019bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:35.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.281+0000 7f4defe00700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4df01045f0 0x7f4df019bd10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4de000e9a0 tx=0x7f4de000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:06:35.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.281+0000 7f4defe00700 1 -- 172.21.15.27:0/1761662567 shutdown_connections 2024-05-10T11:06:35.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.281+0000 7f4defe00700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4dd40b2880 0x7f4dd40b4d40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:35.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.281+0000 7f4defe00700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4df010d9d0 0x7f4df01a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:35.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.281+0000 7f4defe00700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4df01045f0 0x7f4df019bd10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:35.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.281+0000 7f4defe00700 1 --2- 172.21.15.27:0/1761662567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4df0104f10 0x7f4df019c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:35.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.281+0000 7f4defe00700 1 -- 172.21.15.27:0/1761662567 >> 172.21.15.27:0/1761662567 conn(0x7f4df00fddc0 msgr2=0x7f4df010ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.281+0000 7f4defe00700 1 -- 172.21.15.27:0/1761662567 shutdown_connections 2024-05-10T11:06:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:35.281+0000 7f4defe00700 1 -- 172.21.15.27:0/1761662567 wait complete. 2024-05-10T11:06:36.112 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-10T11:02:06.781475+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-10T11:06:36.113 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-10T11:06:36.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:36 smithi189 ceph-mon[34842]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:36.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:36 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1761662567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:06:36.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:36 smithi027 ceph-mon[30054]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:36.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:36 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1761662567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:06:36.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:36 smithi076 ceph-mon[34152]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:36.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:36 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1761662567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:06:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:06:38.342 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:38 smithi027 ceph-mon[30054]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:38 smithi076 ceph-mon[34152]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:38.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:38 smithi189 ceph-mon[34842]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:38.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.840+0000 7f7ae3e00700 1 -- 172.21.15.27:0/117555612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ae4109460 msgr2=0x7f7ae4109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:38.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.840+0000 7f7ae3e00700 1 --2- 172.21.15.27:0/117555612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ae4109460 0x7f7ae4109840 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400b0a0 tx=0x7f7ad400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:06:38.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.840+0000 7f7ae3e00700 1 -- 172.21.15.27:0/117555612 shutdown_connections 2024-05-10T11:06:38.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.840+0000 7f7ae3e00700 1 --2- 172.21.15.27:0/117555612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ae4103d90 0x7f7ae41102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:38.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.840+0000 7f7ae3e00700 1 --2- 172.21.15.27:0/117555612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ae4109460 0x7f7ae4109840 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:38.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.840+0000 7f7ae3e00700 1 --2- 172.21.15.27:0/117555612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae41033d0 0x7f7ae4103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:38.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.840+0000 7f7ae3e00700 1 -- 172.21.15.27:0/117555612 >> 172.21.15.27:0/117555612 conn(0x7f7ae40fef20 msgr2=0x7f7ae4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:38.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.840+0000 7f7ae3e00700 1 -- 172.21.15.27:0/117555612 shutdown_connections 2024-05-10T11:06:38.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.840+0000 7f7ae3e00700 1 -- 172.21.15.27:0/117555612 wait complete. 2024-05-10T11:06:38.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.841+0000 7f7ae3e00700 1 Processor -- start 2024-05-10T11:06:38.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.842+0000 7f7ae3e00700 1 -- start start 2024-05-10T11:06:38.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.842+0000 7f7ae3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae41033d0 0x7f7ae41a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:38.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.842+0000 7f7ae3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ae4103d90 0x7f7ae41a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:38.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.842+0000 7f7ae3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ae4109460 0x7f7ae419baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:38.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.842+0000 7f7ae3e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae4113590 con 0x7f7ae4103d90 2024-05-10T11:06:38.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.842+0000 7f7ae3e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae4113410 con 0x7f7ae4109460 2024-05-10T11:06:38.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.842+0000 7f7ae3e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae4113710 con 0x7f7ae41033d0 2024-05-10T11:06:38.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.842+0000 7f7ae2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae41033d0 0x7f7ae41a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:38.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.842+0000 7f7ada000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ae4103d90 0x7f7ae41a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:38.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.842+0000 7f7ae2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae41033d0 0x7f7ae41a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37538/0 (socket says 172.21.15.27:37538) 2024-05-10T11:06:38.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.842+0000 7f7ae2a00700 1 -- 172.21.15.27:0/602683600 learned_addr learned my addr 172.21.15.27:0/602683600 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:06:38.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.843+0000 7f7ae3400700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ae4109460 0x7f7ae419baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:38.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.843+0000 7f7ae2a00700 1 -- 172.21.15.27:0/602683600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ae4109460 msgr2=0x7f7ae419baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:38.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.843+0000 7f7ae2a00700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ae4109460 0x7f7ae419baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:38.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.843+0000 7f7ae2a00700 1 -- 172.21.15.27:0/602683600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ae4103d90 msgr2=0x7f7ae41a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:38.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.843+0000 7f7ae2a00700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ae4103d90 0x7f7ae41a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:38.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.843+0000 7f7ae2a00700 1 -- 172.21.15.27:0/602683600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae419c250 con 0x7f7ae41033d0 2024-05-10T11:06:38.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.843+0000 7f7ae3400700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ae4109460 0x7f7ae419baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:06:38.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.843+0000 7f7ae2a00700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae41033d0 0x7f7ae41a18e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400b070 tx=0x7f7ad4004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:38.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.843+0000 7f7adbe00700 1 -- 172.21.15.27:0/602683600 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4021070 con 0x7f7ae41033d0 2024-05-10T11:06:38.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.844+0000 7f7ae3e00700 1 -- 172.21.15.27:0/602683600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae419c4e0 con 0x7f7ae41033d0 2024-05-10T11:06:38.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.844+0000 7f7adbe00700 1 -- 172.21.15.27:0/602683600 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ad4007e30 con 0x7f7ae41033d0 2024-05-10T11:06:38.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.844+0000 7f7adbe00700 1 -- 172.21.15.27:0/602683600 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4004730 con 0x7f7ae41033d0 2024-05-10T11:06:38.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.844+0000 7f7ae3e00700 1 -- 172.21.15.27:0/602683600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae41a87b0 con 0x7f7ae41033d0 2024-05-10T11:06:38.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.845+0000 7f7ae3e00700 1 -- 172.21.15.27:0/602683600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ae406a190 con 0x7f7ae41033d0 2024-05-10T11:06:38.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.847+0000 7f7adbe00700 1 -- 172.21.15.27:0/602683600 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f7ad4007980 con 0x7f7ae41033d0 2024-05-10T11:06:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.849+0000 7f7adbe00700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7ac80b2900 0x7f7ac80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.849+0000 7f7adbe00700 1 -- 172.21.15.27:0/602683600 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f7ad400f020 con 0x7f7ae41033d0 2024-05-10T11:06:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.849+0000 7f7ada000700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7ac80b2900 0x7f7ac80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:38.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.850+0000 7f7ada000700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7ac80b2900 0x7f7ac80b4dc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4110730 tx=0x7f7acc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:38.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:38.853+0000 7f7adbe00700 1 -- 172.21.15.27:0/602683600 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7ad402a020 con 0x7f7ae41033d0 2024-05-10T11:06:39.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.091+0000 7f7ae3e00700 1 -- 172.21.15.27:0/602683600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7ae41a8980 con 0x7f7ae41033d0 2024-05-10T11:06:39.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.092+0000 7f7adbe00700 1 -- 172.21.15.27:0/602683600 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f7ad4004000 con 0x7f7ae41033d0 2024-05-10T11:06:39.093 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-05-10T11:06:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.095+0000 7f7ae3e00700 1 -- 172.21.15.27:0/602683600 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7ac80b2900 msgr2=0x7f7ac80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.095+0000 7f7ae3e00700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7ac80b2900 0x7f7ac80b4dc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4110730 tx=0x7f7acc009260 comp rx=0 tx=0).stop 2024-05-10T11:06:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.095+0000 7f7ae3e00700 1 -- 172.21.15.27:0/602683600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae41033d0 msgr2=0x7f7ae41a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.095+0000 7f7ae3e00700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae41033d0 0x7f7ae41a18e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400b070 tx=0x7f7ad4004d00 comp rx=0 tx=0).stop 2024-05-10T11:06:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.095+0000 7f7ae3e00700 1 -- 172.21.15.27:0/602683600 shutdown_connections 2024-05-10T11:06:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.095+0000 7f7ae3e00700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7ac80b2900 0x7f7ac80b4dc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.095+0000 7f7ae3e00700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae41033d0 0x7f7ae41a18e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.095+0000 7f7ae3e00700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ae4109460 0x7f7ae419baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.095+0000 7f7ae3e00700 1 --2- 172.21.15.27:0/602683600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ae4103d90 0x7f7ae41a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.095+0000 7f7ae3e00700 1 -- 172.21.15.27:0/602683600 >> 172.21.15.27:0/602683600 conn(0x7f7ae40fef20 msgr2=0x7f7ae40ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.096+0000 7f7ae3e00700 1 -- 172.21.15.27:0/602683600 shutdown_connections 2024-05-10T11:06:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:39.096+0000 7f7ae3e00700 1 -- 172.21.15.27:0/602683600 wait complete. 2024-05-10T11:06:39.927 INFO:tasks.cephadm:Adding mds.a on smithi027 2024-05-10T11:06:39.927 INFO:tasks.cephadm:Adding mds.d on smithi027 2024-05-10T11:06:39.927 INFO:tasks.cephadm:Adding mds.g on smithi027 2024-05-10T11:06:39.927 INFO:tasks.cephadm:Adding mds.j on smithi027 2024-05-10T11:06:39.927 INFO:tasks.cephadm:Adding mds.b on smithi076 2024-05-10T11:06:39.927 INFO:tasks.cephadm:Adding mds.e on smithi076 2024-05-10T11:06:39.927 INFO:tasks.cephadm:Adding mds.h on smithi076 2024-05-10T11:06:39.927 INFO:tasks.cephadm:Adding mds.k on smithi076 2024-05-10T11:06:39.927 INFO:tasks.cephadm:Adding mds.c on smithi189 2024-05-10T11:06:39.928 INFO:tasks.cephadm:Adding mds.f on smithi189 2024-05-10T11:06:39.928 INFO:tasks.cephadm:Adding mds.i on smithi189 2024-05-10T11:06:39.928 INFO:tasks.cephadm:Adding mds.l on smithi189 2024-05-10T11:06:39.928 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph orch apply mds all '12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l' 2024-05-10T11:06:40.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:40 smithi027 ceph-mon[30054]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:40.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:40 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/602683600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-10T11:06:40.179 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:06:40.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:40 smithi189 ceph-mon[34842]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:40.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:40 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/602683600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-10T11:06:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:40 smithi076 ceph-mon[34152]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:40 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/602683600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-10T11:06:41.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.888+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/3973099209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9eb01050d0 msgr2=0x7f9eb010b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:41.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.888+0000 7f9eb5bf8700 1 --2- 172.21.15.189:0/3973099209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9eb01050d0 0x7f9eb010b980 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800b0a0 tx=0x7f9ea800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:06:41.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.888+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/3973099209 shutdown_connections 2024-05-10T11:06:41.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.888+0000 7f9eb5bf8700 1 --2- 172.21.15.189:0/3973099209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9eb01050d0 0x7f9eb010b980 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:41.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.888+0000 7f9eb5bf8700 1 --2- 172.21.15.189:0/3973099209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb0103530 0x7f9eb0103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:41.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.888+0000 7f9eb5bf8700 1 --2- 172.21.15.189:0/3973099209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb0104730 0x7f9eb0104b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:41.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.888+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/3973099209 >> 172.21.15.189:0/3973099209 conn(0x7f9eb00feaf0 msgr2=0x7f9eb0100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:41.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.888+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/3973099209 shutdown_connections 2024-05-10T11:06:41.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.888+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/3973099209 wait complete. 2024-05-10T11:06:41.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eb5bf8700 1 Processor -- start 2024-05-10T11:06:41.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eb5bf8700 1 -- start start 2024-05-10T11:06:41.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eb5bf8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb0103530 0x7f9eb01995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:41.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eb5bf8700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb0104730 0x7f9eb0199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:41.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eb5bf8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9eb01050d0 0x7f9eb01a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eb5bf8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb010f050 con 0x7f9eb0103530 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eb5bf8700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb010eed0 con 0x7f9eb0104730 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eb5bf8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb010f1d0 con 0x7f9eb01050d0 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eaffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9eb01050d0 0x7f9eb01a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eaffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9eb01050d0 0x7f9eb01a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38516/0 (socket says 172.21.15.189:38516) 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eaffff700 1 -- 172.21.15.189:0/1803009035 learned_addr learned my addr 172.21.15.189:0/1803009035 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eaeffd700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb0104730 0x7f9eb0199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eaffff700 1 -- 172.21.15.189:0/1803009035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb0104730 msgr2=0x7f9eb0199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eaffff700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb0104730 0x7f9eb0199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eaffff700 1 -- 172.21.15.189:0/1803009035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb0103530 msgr2=0x7f9eb01995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eaffff700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb0103530 0x7f9eb01995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eaffff700 1 -- 172.21.15.189:0/1803009035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb01a1200 con 0x7f9eb01050d0 2024-05-10T11:06:41.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.889+0000 7f9eaeffd700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb0104730 0x7f9eb0199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:06:41.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.890+0000 7f9eaffff700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9eb01050d0 0x7f9eb01a0b60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8006070 tx=0x7f9ea8004ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:41.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.890+0000 7f9eacff9700 1 -- 172.21.15.189:0/1803009035 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea8021070 con 0x7f9eb01050d0 2024-05-10T11:06:41.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.890+0000 7f9eacff9700 1 -- 172.21.15.189:0/1803009035 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ea8004020 con 0x7f9eb01050d0 2024-05-10T11:06:41.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.890+0000 7f9eacff9700 1 -- 172.21.15.189:0/1803009035 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea801d690 con 0x7f9eb01050d0 2024-05-10T11:06:41.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.890+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/1803009035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb01a1490 con 0x7f9eb01050d0 2024-05-10T11:06:41.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.890+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/1803009035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb01a19c0 con 0x7f9eb01050d0 2024-05-10T11:06:41.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.891+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/1803009035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eb006a190 con 0x7f9eb01050d0 2024-05-10T11:06:41.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.892+0000 7f9eacff9700 1 -- 172.21.15.189:0/1803009035 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9ea80044f0 con 0x7f9eb01050d0 2024-05-10T11:06:41.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.893+0000 7f9eacff9700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f9e980b2880 0x7f9e980b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:41.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.893+0000 7f9eacff9700 1 -- 172.21.15.189:0/1803009035 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f9ea800f020 con 0x7f9eb01050d0 2024-05-10T11:06:41.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.893+0000 7f9eaf7fe700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f9e980b2880 0x7f9e980b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:41.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.894+0000 7f9eaf7fe700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f9e980b2880 0x7f9e980b4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0104590 tx=0x7f9ea0006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:41.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:41.896+0000 7f9eacff9700 1 -- 172.21.15.189:0/1803009035 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9ea802a020 con 0x7f9eb01050d0 2024-05-10T11:06:42.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.111+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/1803009035 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}) -- 0x7f9eb0063ba0 con 0x7f9e980b2880 2024-05-10T11:06:42.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.128+0000 7f9eacff9700 1 -- 172.21.15.189:0/1803009035 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f9eb0063ba0 con 0x7f9e980b2880 2024-05-10T11:06:42.129 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mds.all update... 2024-05-10T11:06:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.131+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/1803009035 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f9e980b2880 msgr2=0x7f9e980b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.131+0000 7f9eb5bf8700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f9e980b2880 0x7f9e980b4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0104590 tx=0x7f9ea0006d20 comp rx=0 tx=0).stop 2024-05-10T11:06:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.131+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/1803009035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9eb01050d0 msgr2=0x7f9eb01a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.131+0000 7f9eb5bf8700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9eb01050d0 0x7f9eb01a0b60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8006070 tx=0x7f9ea8004ef0 comp rx=0 tx=0).stop 2024-05-10T11:06:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.131+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/1803009035 shutdown_connections 2024-05-10T11:06:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.131+0000 7f9eb5bf8700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f9e980b2880 0x7f9e980b4d40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.131+0000 7f9eb5bf8700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9eb01050d0 0x7f9eb01a0b60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.131+0000 7f9eb5bf8700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb0103530 0x7f9eb01995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.131+0000 7f9eb5bf8700 1 --2- 172.21.15.189:0/1803009035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb0104730 0x7f9eb0199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.131+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/1803009035 >> 172.21.15.189:0/1803009035 conn(0x7f9eb00feaf0 msgr2=0x7f9eb0107a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:42.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.131+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/1803009035 shutdown_connections 2024-05-10T11:06:42.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-10T11:06:42.131+0000 7f9eb5bf8700 1 -- 172.21.15.189:0/1803009035 wait complete. 2024-05-10T11:06:42.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:42 smithi027 ceph-mon[30054]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:42 smithi076 ceph-mon[34152]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:42.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:42 smithi189 ceph-mon[34842]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:42.952 DEBUG:teuthology.orchestra.run.smithi027:mds.a> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.a.service 2024-05-10T11:06:42.959 DEBUG:teuthology.orchestra.run.smithi027:mds.d> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.d.service 2024-05-10T11:06:43.005 DEBUG:teuthology.orchestra.run.smithi027:mds.g> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.g.service 2024-05-10T11:06:43.051 DEBUG:teuthology.orchestra.run.smithi027:mds.j> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.j.service 2024-05-10T11:06:43.097 DEBUG:teuthology.orchestra.run.smithi076:mds.b> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.b.service 2024-05-10T11:06:43.100 DEBUG:teuthology.orchestra.run.smithi076:mds.e> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.e.service 2024-05-10T11:06:43.142 DEBUG:teuthology.orchestra.run.smithi076:mds.h> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.h.service 2024-05-10T11:06:43.185 DEBUG:teuthology.orchestra.run.smithi076:mds.k> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.k.service 2024-05-10T11:06:43.228 DEBUG:teuthology.orchestra.run.smithi189:mds.c> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.c.service 2024-05-10T11:06:43.231 DEBUG:teuthology.orchestra.run.smithi189:mds.f> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.f.service 2024-05-10T11:06:43.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:43 smithi189 ceph-mon[34842]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:06:43.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:43 smithi189 ceph-mon[34842]: Saving service mds.all spec with placement smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-05-10T11:06:43.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:43.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:06:43.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:43 smithi189 ceph-mon[34842]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:43.237 DEBUG:teuthology.orchestra.run.smithi189:mds.i> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.i.service 2024-05-10T11:06:43.280 DEBUG:teuthology.orchestra.run.smithi189:mds.l> sudo journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.l.service 2024-05-10T11:06:43.324 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-10T11:06:43.324 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:06:43.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:43 smithi027 ceph-mon[30054]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:06:43.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:43 smithi027 ceph-mon[30054]: Saving service mds.all spec with placement smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-05-10T11:06:43.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:43.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:06:43.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:43 smithi027 ceph-mon[30054]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:43 smithi076 ceph-mon[34152]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:06:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:43 smithi076 ceph-mon[34152]: Saving service mds.all spec with placement smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-05-10T11:06:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:06:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:43 smithi076 ceph-mon[34152]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:43.785 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:06:44.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:44.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:44.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:44.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:06:44.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:06:44.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:44 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:44.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:44 smithi027 ceph-mon[30054]: Deploying daemon mds.l on smithi189 2024-05-10T11:06:44.263 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:44.263 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:44.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:44.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:06:44.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:06:44.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:44 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:44.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:44 smithi189 ceph-mon[34842]: Deploying daemon mds.l on smithi189 2024-05-10T11:06:44.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:44.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:06:44.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:44.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:06:44.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:06:44.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:44 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:44.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:44 smithi076 ceph-mon[34152]: Deploying daemon mds.l on smithi189 2024-05-10T11:06:45.291 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:45 smithi027 ceph-mon[30054]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:45.384 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:45 smithi189 ceph-mon[34842]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:45.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:45 smithi076 ceph-mon[34152]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:45.737 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:06:45 smithi189 systemd[1]: Starting Ceph mds.l for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:06:46.082 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:06:45 smithi189 podman[64865]: 2024-05-10 11:06:45.725677007 +0000 UTC m=+0.018193142 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:06:46.082 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:06:45 smithi189 podman[64865]: 2024-05-10 11:06:45.84692991 +0000 UTC m=+0.139446074 container create 25ea3d564c7efc32fd95cf6a37431fb053fd3eaa175d5c187a2c9fa0fe378c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-10T11:06:46.082 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:06:46 smithi189 podman[64865]: 2024-05-10 11:06:46.0829114 +0000 UTC m=+0.375427522 container init 25ea3d564c7efc32fd95cf6a37431fb053fd3eaa175d5c187a2c9fa0fe378c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True) 2024-05-10T11:06:46.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.107+0000 7fa8e2000700 1 -- 172.21.15.27:0/2164802006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8dc109460 msgr2=0x7fa8dc109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:46.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.107+0000 7fa8e2000700 1 --2- 172.21.15.27:0/2164802006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8dc109460 0x7fa8dc109840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009990 tx=0x7fa8cc009ca0 comp rx=0 tx=0).stop 2024-05-10T11:06:46.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.107+0000 7fa8e2000700 1 -- 172.21.15.27:0/2164802006 shutdown_connections 2024-05-10T11:06:46.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.107+0000 7fa8e2000700 1 --2- 172.21.15.27:0/2164802006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8dc103d90 0x7fa8dc1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:46.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.107+0000 7fa8e2000700 1 --2- 172.21.15.27:0/2164802006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8dc109460 0x7fa8dc109840 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:46.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.107+0000 7fa8e2000700 1 --2- 172.21.15.27:0/2164802006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8dc1033d0 0x7fa8dc103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:46.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.107+0000 7fa8e2000700 1 -- 172.21.15.27:0/2164802006 >> 172.21.15.27:0/2164802006 conn(0x7fa8dc0fef20 msgr2=0x7fa8dc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:46.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.107+0000 7fa8e2000700 1 -- 172.21.15.27:0/2164802006 shutdown_connections 2024-05-10T11:06:46.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.107+0000 7fa8e2000700 1 -- 172.21.15.27:0/2164802006 wait complete. 2024-05-10T11:06:46.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.109+0000 7fa8e2000700 1 Processor -- start 2024-05-10T11:06:46.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.109+0000 7fa8e2000700 1 -- start start 2024-05-10T11:06:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.109+0000 7fa8e2000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8dc1033d0 0x7fa8dc19f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.109+0000 7fa8e2000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8dc103d90 0x7fa8dc19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.109+0000 7fa8e2000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8dc109460 0x7fa8dc199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.109+0000 7fa8e2000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc113570 con 0x7fa8dc1033d0 2024-05-10T11:06:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.109+0000 7fa8e2000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc1133f0 con 0x7fa8dc103d90 2024-05-10T11:06:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.109+0000 7fa8e2000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc1136f0 con 0x7fa8dc109460 2024-05-10T11:06:46.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.110+0000 7fa8e1600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8dc109460 0x7fa8dc199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:46.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.110+0000 7fa8e1600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8dc109460 0x7fa8dc199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51214/0 (socket says 172.21.15.27:51214) 2024-05-10T11:06:46.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.110+0000 7fa8e1600700 1 -- 172.21.15.27:0/4172232169 learned_addr learned my addr 172.21.15.27:0/4172232169 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:06:46.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.110+0000 7fa8d3e00700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8dc103d90 0x7fa8dc19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:46.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.110+0000 7fa8e0c00700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8dc1033d0 0x7fa8dc19f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:46.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.110+0000 7fa8e1600700 1 -- 172.21.15.27:0/4172232169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8dc103d90 msgr2=0x7fa8dc19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:46.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.110+0000 7fa8e1600700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8dc103d90 0x7fa8dc19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:46.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.110+0000 7fa8e1600700 1 -- 172.21.15.27:0/4172232169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8dc1033d0 msgr2=0x7fa8dc19f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:46.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.110+0000 7fa8e1600700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8dc1033d0 0x7fa8dc19f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:46.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.110+0000 7fa8e1600700 1 -- 172.21.15.27:0/4172232169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8dc199fd0 con 0x7fa8dc109460 2024-05-10T11:06:46.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.111+0000 7fa8e1600700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8dc109460 0x7fa8dc199870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400ef30 tx=0x7fa8d400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:46.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.111+0000 7fa8da000700 1 -- 172.21.15.27:0/4172232169 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d40092e0 con 0x7fa8dc109460 2024-05-10T11:06:46.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.111+0000 7fa8da000700 1 -- 172.21.15.27:0/4172232169 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8d4004bb0 con 0x7fa8dc109460 2024-05-10T11:06:46.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.111+0000 7fa8e2000700 1 -- 172.21.15.27:0/4172232169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8dc19a260 con 0x7fa8dc109460 2024-05-10T11:06:46.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.111+0000 7fa8da000700 1 -- 172.21.15.27:0/4172232169 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4004020 con 0x7fa8dc109460 2024-05-10T11:06:46.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.111+0000 7fa8e2000700 1 -- 172.21.15.27:0/4172232169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8dc1a6580 con 0x7fa8dc109460 2024-05-10T11:06:46.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.113+0000 7fa8e2000700 1 -- 172.21.15.27:0/4172232169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8dc10b960 con 0x7fa8dc109460 2024-05-10T11:06:46.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.116+0000 7fa8da000700 1 -- 172.21.15.27:0/4172232169 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa8d401b030 con 0x7fa8dc109460 2024-05-10T11:06:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.118+0000 7fa8da000700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa8c00b2900 0x7fa8c00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.118+0000 7fa8e0c00700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa8c00b2900 0x7fa8c00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.118+0000 7fa8da000700 1 -- 172.21.15.27:0/4172232169 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fa8d40e17c0 con 0x7fa8dc109460 2024-05-10T11:06:46.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.119+0000 7fa8e0c00700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa8c00b2900 0x7fa8c00b4dc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00b2e0 tx=0x7fa8cc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:46.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.121+0000 7fa8da000700 1 -- 172.21.15.27:0/4172232169 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa8d4022020 con 0x7fa8dc109460 2024-05-10T11:06:46.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.432+0000 7fa8e2000700 1 -- 172.21.15.27:0/4172232169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8dc06a190 con 0x7fa8dc109460 2024-05-10T11:06:46.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.433+0000 7fa8da000700 1 -- 172.21.15.27:0/4172232169 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fa8d4010040 con 0x7fa8dc109460 2024-05-10T11:06:46.434 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:06:46.434 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-10T11:06:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.436+0000 7fa8e2000700 1 -- 172.21.15.27:0/4172232169 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa8c00b2900 msgr2=0x7fa8c00b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.436+0000 7fa8e2000700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa8c00b2900 0x7fa8c00b4dc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00b2e0 tx=0x7fa8cc01a040 comp rx=0 tx=0).stop 2024-05-10T11:06:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.436+0000 7fa8e2000700 1 -- 172.21.15.27:0/4172232169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8dc109460 msgr2=0x7fa8dc199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.436+0000 7fa8e2000700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8dc109460 0x7fa8dc199870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400ef30 tx=0x7fa8d400c3a0 comp rx=0 tx=0).stop 2024-05-10T11:06:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.437+0000 7fa8e2000700 1 -- 172.21.15.27:0/4172232169 shutdown_connections 2024-05-10T11:06:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.437+0000 7fa8e2000700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa8c00b2900 0x7fa8c00b4dc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.437+0000 7fa8e2000700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8dc109460 0x7fa8dc199870 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.437+0000 7fa8e2000700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8dc1033d0 0x7fa8dc19f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.437+0000 7fa8e2000700 1 --2- 172.21.15.27:0/4172232169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8dc103d90 0x7fa8dc19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.437+0000 7fa8e2000700 1 -- 172.21.15.27:0/4172232169 >> 172.21.15.27:0/4172232169 conn(0x7fa8dc0fef20 msgr2=0x7fa8dc0ff730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.437+0000 7fa8e2000700 1 -- 172.21.15.27:0/4172232169 shutdown_connections 2024-05-10T11:06:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:46.437+0000 7fa8e2000700 1 -- 172.21.15.27:0/4172232169 wait complete. 2024-05-10T11:06:46.439 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 2 2024-05-10T11:06:46.486 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:06:46 smithi189 podman[64865]: 2024-05-10 11:06:46.085459399 +0000 UTC m=+0.377975521 container start 25ea3d564c7efc32fd95cf6a37431fb053fd3eaa175d5c187a2c9fa0fe378c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-10T11:06:46.487 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:06:46 smithi189 ceph-mds[64884]: -- 172.21.15.189:0/2268748273 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562949737a40 con 0x56294970b000 2024-05-10T11:06:46.487 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:06:46 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-l[64880]: starting mds.l at 2024-05-10T11:06:46.487 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:06:46 smithi189 bash[64865]: 25ea3d564c7efc32fd95cf6a37431fb053fd3eaa175d5c187a2c9fa0fe378c94 2024-05-10T11:06:46.487 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:06:46 smithi189 systemd[1]: Started Ceph mds.l for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:06:47.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:47 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.189:6832/2718624698,v1:172.21.15.189:6833/2718624698] up:boot 2024-05-10T11:06:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:47 smithi076 ceph-mon[34152]: fsmap 1 up:standby 2024-05-10T11:06:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-10T11:06:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:06:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:06:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:47 smithi076 ceph-mon[34152]: Deploying daemon mds.k on smithi076 2024-05-10T11:06:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:47 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4172232169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:06:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:47 smithi076 ceph-mon[34152]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:47.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:47 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.189:6832/2718624698,v1:172.21.15.189:6833/2718624698] up:boot 2024-05-10T11:06:47.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:47 smithi027 ceph-mon[30054]: fsmap 1 up:standby 2024-05-10T11:06:47.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-10T11:06:47.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:47.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:47.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:47.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:06:47.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:06:47.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:47.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:47 smithi027 ceph-mon[30054]: Deploying daemon mds.k on smithi076 2024-05-10T11:06:47.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:47 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4172232169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:06:47.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:47 smithi027 ceph-mon[30054]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:47 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.189:6832/2718624698,v1:172.21.15.189:6833/2718624698] up:boot 2024-05-10T11:06:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:47 smithi189 ceph-mon[34842]: fsmap 1 up:standby 2024-05-10T11:06:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-10T11:06:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:06:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:06:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:47.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:47 smithi189 ceph-mon[34842]: Deploying daemon mds.k on smithi076 2024-05-10T11:06:47.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:47 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4172232169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:06:47.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:47 smithi189 ceph-mon[34842]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:48.571 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:06:48 smithi076 systemd[1]: Starting Ceph mds.k for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:06:48.925 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:06:48 smithi076 podman[63786]: 2024-05-10 11:06:48.690591787 +0000 UTC m=+0.017966308 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:06:48.925 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:06:48 smithi076 podman[63786]: 2024-05-10 11:06:48.811448309 +0000 UTC m=+0.138822818 container create f712d006514775982f7838980a5af47efe2267d12b22e7a8032f8a0aed20bc5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-10T11:06:49.193 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:06:49.425 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:06:49 smithi076 podman[63786]: 2024-05-10 11:06:49.056000163 +0000 UTC m=+0.383374671 container init f712d006514775982f7838980a5af47efe2267d12b22e7a8032f8a0aed20bc5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08) 2024-05-10T11:06:49.426 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:06:49 smithi076 podman[63786]: 2024-05-10 11:06:49.058668671 +0000 UTC m=+0.386043190 container start f712d006514775982f7838980a5af47efe2267d12b22e7a8032f8a0aed20bc5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-10T11:06:49.426 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:06:49 smithi076 ceph-mds[63804]: -- 172.21.15.76:0/3683525445 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c4451efa40 con 0x55c4451c3000 2024-05-10T11:06:49.426 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:06:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k[63800]: starting mds.k at 2024-05-10T11:06:49.426 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:06:49 smithi076 bash[63786]: f712d006514775982f7838980a5af47efe2267d12b22e7a8032f8a0aed20bc5a 2024-05-10T11:06:49.426 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:06:49 smithi076 systemd[1]: Started Ceph mds.k for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:06:49.673 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:06:49.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:49 smithi027 ceph-mon[30054]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:49.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:49.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:49.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:49.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:06:49.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:06:49.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:50.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:49 smithi076 ceph-mon[34152]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:50.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:49 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:50.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:49 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:50.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:49 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:50.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:49 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:06:50.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:49 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:06:50.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:49 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:50.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:49 smithi189 ceph-mon[34842]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:50.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:49 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:50.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:49 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:50.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:49 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:50.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:49 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:06:50.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:49 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:06:50.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:49 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:51.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:50 smithi027 ceph-mon[30054]: Deploying daemon mds.j on smithi027 2024-05-10T11:06:51.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:50 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.76:6832/1370063583,v1:172.21.15.76:6833/1370063583] up:boot 2024-05-10T11:06:51.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:50 smithi027 ceph-mon[30054]: fsmap 2 up:standby 2024-05-10T11:06:51.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:50 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-10T11:06:51.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:50 smithi076 ceph-mon[34152]: Deploying daemon mds.j on smithi027 2024-05-10T11:06:51.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:50 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.76:6832/1370063583,v1:172.21.15.76:6833/1370063583] up:boot 2024-05-10T11:06:51.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:50 smithi076 ceph-mon[34152]: fsmap 2 up:standby 2024-05-10T11:06:51.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:50 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-10T11:06:51.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:50 smithi189 ceph-mon[34842]: Deploying daemon mds.j on smithi027 2024-05-10T11:06:51.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:50 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.76:6832/1370063583,v1:172.21.15.76:6833/1370063583] up:boot 2024-05-10T11:06:51.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:50 smithi189 ceph-mon[34842]: fsmap 2 up:standby 2024-05-10T11:06:51.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:50 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-10T11:06:51.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:51 smithi076 ceph-mon[34152]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:52.061 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:51 smithi027 ceph-mon[30054]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:52.211 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:51 smithi189 ceph-mon[34842]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:52.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.134+0000 7f842aa00700 1 -- 172.21.15.27:0/3437023994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84241087a0 msgr2=0x7f8424110350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:52.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.134+0000 7f842aa00700 1 --2- 172.21.15.27:0/3437023994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84241087a0 0x7f8424110350 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f841c00b0a0 tx=0x7f841c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:06:52.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.135+0000 7f842aa00700 1 -- 172.21.15.27:0/3437023994 shutdown_connections 2024-05-10T11:06:52.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.135+0000 7f842aa00700 1 --2- 172.21.15.27:0/3437023994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8424101f30 0x7f8424102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:52.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.135+0000 7f842aa00700 1 --2- 172.21.15.27:0/3437023994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84241087a0 0x7f8424110350 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:52.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.135+0000 7f842aa00700 1 --2- 172.21.15.27:0/3437023994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84241028e0 0x7f84241080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:52.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.135+0000 7f842aa00700 1 -- 172.21.15.27:0/3437023994 >> 172.21.15.27:0/3437023994 conn(0x7f8424077650 msgr2=0x7f8424077a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:52.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.135+0000 7f842aa00700 1 -- 172.21.15.27:0/3437023994 shutdown_connections 2024-05-10T11:06:52.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.135+0000 7f842aa00700 1 -- 172.21.15.27:0/3437023994 wait complete. 2024-05-10T11:06:52.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.136+0000 7f842aa00700 1 Processor -- start 2024-05-10T11:06:52.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.136+0000 7f842aa00700 1 -- start start 2024-05-10T11:06:52.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.137+0000 7f842aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8424101f30 0x7f8424102eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.137+0000 7f842aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84241028e0 0x7f84241033f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.137+0000 7f842aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84241087a0 0x7f8424103930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.137+0000 7f842aa00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8424113620 con 0x7f8424101f30 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.137+0000 7f842aa00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f84241134a0 con 0x7f84241087a0 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.137+0000 7f842aa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f84241137a0 con 0x7f84241028e0 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.137+0000 7f8429600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8424101f30 0x7f8424102eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.137+0000 7f8429600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8424101f30 0x7f8424102eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36784/0 (socket says 172.21.15.27:36784) 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.137+0000 7f8429600700 1 -- 172.21.15.27:0/620074734 learned_addr learned my addr 172.21.15.27:0/620074734 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.137+0000 7f8428c00700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84241028e0 0x7f84241033f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.137+0000 7f842a000700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84241087a0 0x7f8424103930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.138+0000 7f8428c00700 1 -- 172.21.15.27:0/620074734 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84241087a0 msgr2=0x7f8424103930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.138+0000 7f8428c00700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84241087a0 0x7f8424103930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:52.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.138+0000 7f8428c00700 1 -- 172.21.15.27:0/620074734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8424101f30 msgr2=0x7f8424102eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.138+0000 7f8428c00700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8424101f30 0x7f8424102eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.138+0000 7f8428c00700 1 -- 172.21.15.27:0/620074734 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84241a7f40 con 0x7f84241028e0 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.138+0000 7f8428c00700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84241028e0 0x7f84241033f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f841000e9c0 tx=0x7f841000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.138+0000 7f8422000700 1 -- 172.21.15.27:0/620074734 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841000c9c0 con 0x7f84241028e0 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.138+0000 7f842aa00700 1 -- 172.21.15.27:0/620074734 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84241a8230 con 0x7f84241028e0 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.138+0000 7f8422000700 1 -- 172.21.15.27:0/620074734 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f841000cb60 con 0x7f84241028e0 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.138+0000 7f8422000700 1 -- 172.21.15.27:0/620074734 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8410018680 con 0x7f84241028e0 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.138+0000 7f842aa00700 1 -- 172.21.15.27:0/620074734 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84241a8770 con 0x7f84241028e0 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.139+0000 7f842aa00700 1 -- 172.21.15.27:0/620074734 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f842406a190 con 0x7f84241028e0 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.142+0000 7f8422000700 1 -- 172.21.15.27:0/620074734 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8410010430 con 0x7f84241028e0 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.143+0000 7f8422000700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f840c0b2880 0x7f840c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.144+0000 7f8422000700 1 -- 172.21.15.27:0/620074734 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f84100e1f10 con 0x7f84241028e0 2024-05-10T11:06:52.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.144+0000 7f8429600700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f840c0b2880 0x7f840c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:06:52.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.144+0000 7f8429600700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f840c0b2880 0x7f840c0b4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8418009d60 tx=0x7f8418009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:06:52.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.148+0000 7f8422000700 1 -- 172.21.15.27:0/620074734 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f84100aa030 con 0x7f84241028e0 2024-05-10T11:06:52.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.443+0000 7f842aa00700 1 -- 172.21.15.27:0/620074734 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84241a89d0 con 0x7f84241028e0 2024-05-10T11:06:52.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.444+0000 7f8422000700 1 -- 172.21.15.27:0/620074734 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f84100b20b0 con 0x7f84241028e0 2024-05-10T11:06:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:06:52.445 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-10T11:06:52.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.447+0000 7f842aa00700 1 -- 172.21.15.27:0/620074734 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f840c0b2880 msgr2=0x7f840c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:52.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.447+0000 7f842aa00700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f840c0b2880 0x7f840c0b4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8418009d60 tx=0x7f8418009260 comp rx=0 tx=0).stop 2024-05-10T11:06:52.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.447+0000 7f842aa00700 1 -- 172.21.15.27:0/620074734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84241028e0 msgr2=0x7f84241033f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:06:52.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.447+0000 7f842aa00700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84241028e0 0x7f84241033f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f841000e9c0 tx=0x7f841000ecd0 comp rx=0 tx=0).stop 2024-05-10T11:06:52.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.447+0000 7f842aa00700 1 -- 172.21.15.27:0/620074734 shutdown_connections 2024-05-10T11:06:52.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.447+0000 7f842aa00700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f840c0b2880 0x7f840c0b4d40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:52.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.447+0000 7f842aa00700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84241087a0 0x7f8424103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:52.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.447+0000 7f842aa00700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8424101f30 0x7f8424102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:52.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.447+0000 7f842aa00700 1 --2- 172.21.15.27:0/620074734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84241028e0 0x7f84241033f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:06:52.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.447+0000 7f842aa00700 1 -- 172.21.15.27:0/620074734 >> 172.21.15.27:0/620074734 conn(0x7f8424077650 msgr2=0x7f84240ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:06:52.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.448+0000 7f842aa00700 1 -- 172.21.15.27:0/620074734 shutdown_connections 2024-05-10T11:06:52.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:06:52.448+0000 7f842aa00700 1 -- 172.21.15.27:0/620074734 wait complete. 2024-05-10T11:06:52.450 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 3 2024-05-10T11:06:53.117 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:52 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/620074734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:06:53.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:52 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/620074734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:06:53.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:52 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/620074734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:06:54.002 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:53 smithi027 ceph-mon[30054]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:54.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:53 smithi076 ceph-mon[34152]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:54.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:53 smithi189 ceph-mon[34842]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:54.369 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:06:54 smithi027 systemd[1]: Starting Ceph mds.j for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:06:54.780 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:06:54 smithi027 podman[70056]: 2024-05-10 11:06:54.336937172 +0000 UTC m=+0.037119727 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:06:54.780 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:06:54 smithi027 podman[70056]: 2024-05-10 11:06:54.465344364 +0000 UTC m=+0.165526976 container create 955b6fb80c9e16d5572fb8deb1d74dea120cdd7357c4a16245eb4b23ae1f48f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-10T11:06:54.780 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:06:54 smithi027 podman[70056]: 2024-05-10 11:06:54.77859531 +0000 UTC m=+0.478777952 container init 955b6fb80c9e16d5572fb8deb1d74dea120cdd7357c4a16245eb4b23ae1f48f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:06:55.118 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:06:54 smithi027 podman[70056]: 2024-05-10 11:06:54.790922012 +0000 UTC m=+0.491104578 container start 955b6fb80c9e16d5572fb8deb1d74dea120cdd7357c4a16245eb4b23ae1f48f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS) 2024-05-10T11:06:55.119 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:06:54 smithi027 ceph-mds[70076]: -- 172.21.15.27:0/1459063826 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b82235fa40 con 0x55b822332800 2024-05-10T11:06:55.119 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:06:54 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j[70072]: starting mds.j at 2024-05-10T11:06:55.119 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:06:54 smithi027 bash[70056]: 955b6fb80c9e16d5572fb8deb1d74dea120cdd7357c4a16245eb4b23ae1f48f5 2024-05-10T11:06:55.119 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:06:54 smithi027 systemd[1]: Started Ceph mds.j for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:06:55.792 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:06:56.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:56 smithi027 ceph-mon[30054]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:56.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:56.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:56.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:56.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:06:56.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:06:56.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:56.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:56 smithi027 ceph-mon[30054]: Deploying daemon mds.d on smithi027 2024-05-10T11:06:56.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:56 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.27:6834/3274866857,v1:172.21.15.27:6835/3274866857] up:boot 2024-05-10T11:06:56.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:56 smithi027 ceph-mon[30054]: fsmap 3 up:standby 2024-05-10T11:06:56.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-10T11:06:56.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:56 smithi076 ceph-mon[34152]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:56.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:56 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:56.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:56 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:56.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:56 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:56.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:56 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:06:56.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:56 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:06:56.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:56 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:56.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:56 smithi076 ceph-mon[34152]: Deploying daemon mds.d on smithi027 2024-05-10T11:06:56.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:56 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.27:6834/3274866857,v1:172.21.15.27:6835/3274866857] up:boot 2024-05-10T11:06:56.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:56 smithi076 ceph-mon[34152]: fsmap 3 up:standby 2024-05-10T11:06:56.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:56 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-10T11:06:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:56 smithi189 ceph-mon[34842]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:56 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:56 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:56 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:06:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:56 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:06:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:56 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:06:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:56 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:06:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:56 smithi189 ceph-mon[34842]: Deploying daemon mds.d on smithi027 2024-05-10T11:06:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:56 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.27:6834/3274866857,v1:172.21.15.27:6835/3274866857] up:boot 2024-05-10T11:06:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:56 smithi189 ceph-mon[34842]: fsmap 3 up:standby 2024-05-10T11:06:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:56 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-10T11:06:57.447 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:06:58.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:06:58 smithi027 ceph-mon[30054]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:06:58 smithi076 ceph-mon[34152]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:58.436 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:06:58 smithi027 systemd[1]: Starting Ceph mds.d for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:06:58.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:06:58 smithi189 ceph-mon[34842]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:06:59.369 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:06:59 smithi027 podman[70452]: 2024-05-10 11:06:58.97938066 +0000 UTC m=+0.038025324 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:06:59.370 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:06:59 smithi027 podman[70452]: 2024-05-10 11:06:59.137937951 +0000 UTC m=+0.196582646 container create b779ebe9d1e000e0055757c31671f47671642b7ddc6b9dc509f321d53b938344 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:06:59.793 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:06:59 smithi027 podman[70452]: 2024-05-10 11:06:59.408797933 +0000 UTC m=+0.467442634 container init b779ebe9d1e000e0055757c31671f47671642b7ddc6b9dc509f321d53b938344 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-10T11:06:59.793 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:06:59 smithi027 podman[70452]: 2024-05-10 11:06:59.420224441 +0000 UTC m=+0.478869117 container start b779ebe9d1e000e0055757c31671f47671642b7ddc6b9dc509f321d53b938344 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-10T11:06:59.794 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:06:59 smithi027 ceph-mds[70483]: -- 172.21.15.27:0/1968416466 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5583d1077a40 con 0x5583d104a800 2024-05-10T11:06:59.794 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:06:59 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-d[70479]: starting mds.d at 2024-05-10T11:06:59.794 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:06:59 smithi027 bash[70452]: b779ebe9d1e000e0055757c31671f47671642b7ddc6b9dc509f321d53b938344 2024-05-10T11:06:59.794 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:06:59 smithi027 systemd[1]: Started Ceph mds.d for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:07:00.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:00 smithi027 ceph-mon[30054]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:00.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:00 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:00 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:00 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:00 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:00 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:00 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.300+0000 7f4199400700 1 -- 172.21.15.27:0/3405280060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4194103d90 msgr2=0x7f41941102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.300+0000 7f4199400700 1 --2- 172.21.15.27:0/3405280060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4194103d90 0x7f41941102a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f418c00b0a0 tx=0x7f418c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:07:00.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.300+0000 7f4199400700 1 -- 172.21.15.27:0/3405280060 shutdown_connections 2024-05-10T11:07:00.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.300+0000 7f4199400700 1 --2- 172.21.15.27:0/3405280060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4194103d90 0x7f41941102a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:00.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.300+0000 7f4199400700 1 --2- 172.21.15.27:0/3405280060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4194109460 0x7f4194109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:00.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.300+0000 7f4199400700 1 --2- 172.21.15.27:0/3405280060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41941033d0 0x7f4194103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:00.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.300+0000 7f4199400700 1 -- 172.21.15.27:0/3405280060 >> 172.21.15.27:0/3405280060 conn(0x7f41940fef20 msgr2=0x7f4194101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:00.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.300+0000 7f4199400700 1 -- 172.21.15.27:0/3405280060 shutdown_connections 2024-05-10T11:07:00.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.300+0000 7f4199400700 1 -- 172.21.15.27:0/3405280060 wait complete. 2024-05-10T11:07:00.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.302+0000 7f4199400700 1 Processor -- start 2024-05-10T11:07:00.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.302+0000 7f4199400700 1 -- start start 2024-05-10T11:07:00.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.302+0000 7f4199400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41941033d0 0x7f419419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:00.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.302+0000 7f4199400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4194103d90 0x7f419419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:00.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.302+0000 7f4199400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4194109460 0x7f41941998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:00.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.302+0000 7f4199400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41941135c0 con 0x7f4194103d90 2024-05-10T11:07:00.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.302+0000 7f4199400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4194113440 con 0x7f41941033d0 2024-05-10T11:07:00.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.302+0000 7f4199400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4194113740 con 0x7f4194109460 2024-05-10T11:07:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.303+0000 7f4193e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41941033d0 0x7f419419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.303+0000 7f418b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4194103d90 0x7f419419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.303+0000 7f4193e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41941033d0 0x7f419419f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48114/0 (socket says 172.21.15.27:48114) 2024-05-10T11:07:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.303+0000 7f4193e00700 1 -- 172.21.15.27:0/788105353 learned_addr learned my addr 172.21.15.27:0/788105353 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:07:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.303+0000 7f4198a00700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4194109460 0x7f41941998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.303+0000 7f4193e00700 1 -- 172.21.15.27:0/788105353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4194109460 msgr2=0x7f41941998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.303+0000 7f4193e00700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4194109460 0x7f41941998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.303+0000 7f4193e00700 1 -- 172.21.15.27:0/788105353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4194103d90 msgr2=0x7f419419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.303+0000 7f4193e00700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4194103d90 0x7f419419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:00.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.303+0000 7f4193e00700 1 -- 172.21.15.27:0/788105353 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419419a020 con 0x7f41941033d0 2024-05-10T11:07:00.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.303+0000 7f4193e00700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41941033d0 0x7f419419f6b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f418400e9a0 tx=0x7f418400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:00.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.304+0000 7f4191600700 1 -- 172.21.15.27:0/788105353 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418400ca10 con 0x7f41941033d0 2024-05-10T11:07:00.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.304+0000 7f4191600700 1 -- 172.21.15.27:0/788105353 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f418400cbb0 con 0x7f41941033d0 2024-05-10T11:07:00.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.304+0000 7f4199400700 1 -- 172.21.15.27:0/788105353 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419419a310 con 0x7f41941033d0 2024-05-10T11:07:00.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.304+0000 7f4191600700 1 -- 172.21.15.27:0/788105353 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4184018680 con 0x7f41941033d0 2024-05-10T11:07:00.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.304+0000 7f4199400700 1 -- 172.21.15.27:0/788105353 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41941a6580 con 0x7f41941033d0 2024-05-10T11:07:00.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.305+0000 7f4199400700 1 -- 172.21.15.27:0/788105353 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f419406a190 con 0x7f41941033d0 2024-05-10T11:07:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.307+0000 7f4191600700 1 -- 172.21.15.27:0/788105353 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f418404aab0 con 0x7f41941033d0 2024-05-10T11:07:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.309+0000 7f4191600700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f41780b2880 0x7f41780b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.309+0000 7f4191600700 1 -- 172.21.15.27:0/788105353 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f41840e1fe0 con 0x7f41941033d0 2024-05-10T11:07:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.310+0000 7f418b400700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f41780b2880 0x7f41780b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.310+0000 7f418b400700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f41780b2880 0x7f41780b4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4194108a60 tx=0x7f417c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.311+0000 7f4191600700 1 -- 172.21.15.27:0/788105353 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f41840aa120 con 0x7f41941033d0 2024-05-10T11:07:00.418 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:00 smithi076 ceph-mon[34152]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:00.419 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:00 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:00.419 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:00 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:00.419 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:00 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:00.419 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:00 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:00.419 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:00 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:00.420 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:00 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:00.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:00 smithi189 ceph-mon[34842]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:00.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:00 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:00.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:00 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:00.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:00 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:00.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:00 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:00.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:00 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:00.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:00 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:00.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.603+0000 7f4199400700 1 -- 172.21.15.27:0/788105353 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f419402e510 con 0x7f41941033d0 2024-05-10T11:07:00.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.603+0000 7f4191600700 1 -- 172.21.15.27:0/788105353 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f41840a9a20 con 0x7f41941033d0 2024-05-10T11:07:00.608 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:07:00.609 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-05-10T11:07:00.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.606+0000 7f4199400700 1 -- 172.21.15.27:0/788105353 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f41780b2880 msgr2=0x7f41780b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:00.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.606+0000 7f4199400700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f41780b2880 0x7f41780b4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4194108a60 tx=0x7f417c009260 comp rx=0 tx=0).stop 2024-05-10T11:07:00.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.606+0000 7f4199400700 1 -- 172.21.15.27:0/788105353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41941033d0 msgr2=0x7f419419f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:00.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.606+0000 7f4199400700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41941033d0 0x7f419419f6b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f418400e9a0 tx=0x7f418400ecb0 comp rx=0 tx=0).stop 2024-05-10T11:07:00.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.607+0000 7f4199400700 1 -- 172.21.15.27:0/788105353 shutdown_connections 2024-05-10T11:07:00.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.607+0000 7f4199400700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f41780b2880 0x7f41780b4d40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:00.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.607+0000 7f4199400700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4194109460 0x7f41941998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:00.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.607+0000 7f4199400700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41941033d0 0x7f419419f6b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.607+0000 7f4199400700 1 --2- 172.21.15.27:0/788105353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4194103d90 0x7f419419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.607+0000 7f4199400700 1 -- 172.21.15.27:0/788105353 >> 172.21.15.27:0/788105353 conn(0x7f41940fef20 msgr2=0x7f41940ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.607+0000 7f4199400700 1 -- 172.21.15.27:0/788105353 shutdown_connections 2024-05-10T11:07:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:00.607+0000 7f4199400700 1 -- 172.21.15.27:0/788105353 wait complete. 2024-05-10T11:07:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 5 2024-05-10T11:07:01.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:01 smithi027 ceph-mon[30054]: Deploying daemon mds.e on smithi076 2024-05-10T11:07:01.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:01 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.27:6836/1490955215,v1:172.21.15.27:6837/1490955215] up:boot 2024-05-10T11:07:01.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:01 smithi027 ceph-mon[30054]: fsmap 4 up:standby 2024-05-10T11:07:01.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:01 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-10T11:07:01.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:01 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/788105353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:01.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:01 smithi076 ceph-mon[34152]: Deploying daemon mds.e on smithi076 2024-05-10T11:07:01.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:01 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.27:6836/1490955215,v1:172.21.15.27:6837/1490955215] up:boot 2024-05-10T11:07:01.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:01 smithi076 ceph-mon[34152]: fsmap 4 up:standby 2024-05-10T11:07:01.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:01 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-10T11:07:01.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:01 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/788105353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:01 smithi189 ceph-mon[34842]: Deploying daemon mds.e on smithi076 2024-05-10T11:07:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:01 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.27:6836/1490955215,v1:172.21.15.27:6837/1490955215] up:boot 2024-05-10T11:07:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:01 smithi189 ceph-mon[34842]: fsmap 4 up:standby 2024-05-10T11:07:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:01 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-10T11:07:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:01 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/788105353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:02.034 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:07:01 smithi076 systemd[1]: Starting Ceph mds.e for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:07:02.288 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:02 smithi076 ceph-mon[34152]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:02.288 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:07:02 smithi076 podman[64105]: 2024-05-10 11:07:02.02665302 +0000 UTC m=+0.018118256 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:07:02.288 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:07:02 smithi076 podman[64105]: 2024-05-10 11:07:02.15853699 +0000 UTC m=+0.150002225 container create 59efe649ffc03a0f0982af152f601ce2df7eb5ac5ea7f258d91d6b9301d32789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-10T11:07:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:02 smithi027 ceph-mon[30054]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:02.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:02 smithi189 ceph-mon[34842]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:02.675 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:07:02 smithi076 podman[64105]: 2024-05-10 11:07:02.411026558 +0000 UTC m=+0.402491793 container init 59efe649ffc03a0f0982af152f601ce2df7eb5ac5ea7f258d91d6b9301d32789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2) 2024-05-10T11:07:02.676 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:07:02 smithi076 podman[64105]: 2024-05-10 11:07:02.413755173 +0000 UTC m=+0.405220409 container start 59efe649ffc03a0f0982af152f601ce2df7eb5ac5ea7f258d91d6b9301d32789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-10T11:07:02.676 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:07:02 smithi076 ceph-mds[64124]: -- 172.21.15.76:0/3164122972 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55da26c25a40 con 0x55da26bf9000 2024-05-10T11:07:02.676 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:07:02 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-e[64120]: starting mds.e at 2024-05-10T11:07:02.676 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:07:02 smithi076 bash[64105]: 59efe649ffc03a0f0982af152f601ce2df7eb5ac5ea7f258d91d6b9301d32789 2024-05-10T11:07:02.676 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:07:02 smithi076 systemd[1]: Started Ceph mds.e for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:07:03.493 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:07:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:03 smithi027 ceph-mon[30054]: Deploying daemon mds.f on smithi189 2024-05-10T11:07:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:03 smithi027 ceph-mon[30054]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:03 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] up:boot 2024-05-10T11:07:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:03 smithi027 ceph-mon[30054]: fsmap 5 up:standby 2024-05-10T11:07:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-10T11:07:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:03 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-10T11:07:03.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:03.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:03.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:03.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:03.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:03.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:03.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:03 smithi076 ceph-mon[34152]: Deploying daemon mds.f on smithi189 2024-05-10T11:07:03.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:03 smithi076 ceph-mon[34152]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:03.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:03 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] up:boot 2024-05-10T11:07:03.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:03 smithi076 ceph-mon[34152]: fsmap 5 up:standby 2024-05-10T11:07:03.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-10T11:07:03.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:03 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-10T11:07:03.968 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:07:03.980 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:03.980 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:03.980 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:03.980 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:03.981 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:03.981 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:03.981 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:03 smithi189 ceph-mon[34842]: Deploying daemon mds.f on smithi189 2024-05-10T11:07:03.981 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:03 smithi189 ceph-mon[34842]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:03.981 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:03 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] up:boot 2024-05-10T11:07:03.981 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:03 smithi189 ceph-mon[34842]: fsmap 5 up:standby 2024-05-10T11:07:03.981 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-10T11:07:03.981 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:03 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-10T11:07:04.987 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:07:04 smithi189 systemd[1]: Starting Ceph mds.f for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:07:05.312 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:07:05 smithi189 podman[65168]: 2024-05-10 11:07:04.955109859 +0000 UTC m=+0.017789971 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:07:05.312 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:07:05 smithi189 podman[65168]: 2024-05-10 11:07:05.076789135 +0000 UTC m=+0.139469234 container create ce5743ad0fa257c2fda74054a9c8768a57b28bfdc911415428af857b40c58f8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08) 2024-05-10T11:07:05.312 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:07:05 smithi189 podman[65168]: 2024-05-10 11:07:05.312996305 +0000 UTC m=+0.375676408 container init ce5743ad0fa257c2fda74054a9c8768a57b28bfdc911415428af857b40c58f8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-10T11:07:05.737 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:07:05 smithi189 podman[65168]: 2024-05-10 11:07:05.31576111 +0000 UTC m=+0.378441210 container start ce5743ad0fa257c2fda74054a9c8768a57b28bfdc911415428af857b40c58f8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-10T11:07:05.737 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:07:05 smithi189 ceph-mds[65187]: -- 172.21.15.189:0/1560388934 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d26d0ffa40 con 0x55d26d0d3000 2024-05-10T11:07:05.737 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:07:05 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f[65183]: starting mds.f at 2024-05-10T11:07:05.737 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:07:05 smithi189 bash[65168]: ce5743ad0fa257c2fda74054a9c8768a57b28bfdc911415428af857b40c58f8c 2024-05-10T11:07:05.737 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:07:05 smithi189 systemd[1]: Started Ceph mds.f for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:07:06.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:05 smithi027 ceph-mon[30054]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:06.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:06.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:06.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:06.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:06.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:06.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:05 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:06.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:05 smithi076 ceph-mon[34152]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:06.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:06.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:06.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:06.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:06.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:06.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:05 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:06.186 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:05 smithi189 ceph-mon[34842]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:06.186 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:06.186 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:06.186 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:06.186 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:06.186 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:06.186 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:05 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:06.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.320+0000 7fb372000700 1 -- 172.21.15.27:0/1868906668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c109ad0 msgr2=0x7fb36c109f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:06.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.320+0000 7fb372000700 1 --2- 172.21.15.27:0/1868906668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c109ad0 0x7fb36c109f50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb3580098f0 tx=0x7fb358009c00 comp rx=0 tx=0).stop 2024-05-10T11:07:06.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.320+0000 7fb372000700 1 -- 172.21.15.27:0/1868906668 shutdown_connections 2024-05-10T11:07:06.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.320+0000 7fb372000700 1 --2- 172.21.15.27:0/1868906668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb36c10a490 0x7fb36c1169a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:06.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.320+0000 7fb372000700 1 --2- 172.21.15.27:0/1868906668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb36c10fb60 0x7fb36c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:06.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.320+0000 7fb372000700 1 --2- 172.21.15.27:0/1868906668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c109ad0 0x7fb36c109f50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:06.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.320+0000 7fb372000700 1 -- 172.21.15.27:0/1868906668 >> 172.21.15.27:0/1868906668 conn(0x7fb36c076c70 msgr2=0x7fb36c1082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:06.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.320+0000 7fb372000700 1 -- 172.21.15.27:0/1868906668 shutdown_connections 2024-05-10T11:07:06.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.320+0000 7fb372000700 1 -- 172.21.15.27:0/1868906668 wait complete. 2024-05-10T11:07:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.321+0000 7fb372000700 1 Processor -- start 2024-05-10T11:07:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.322+0000 7fb372000700 1 -- start start 2024-05-10T11:07:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.322+0000 7fb372000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb36c109ad0 0x7fb36c1a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.322+0000 7fb372000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb36c10a490 0x7fb36c1a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.322+0000 7fb372000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c10fb60 0x7fb36c1a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.322+0000 7fb372000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c119d20 con 0x7fb36c10a490 2024-05-10T11:07:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.322+0000 7fb372000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c119ba0 con 0x7fb36c109ad0 2024-05-10T11:07:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.322+0000 7fb372000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c119ea0 con 0x7fb36c10fb60 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.323+0000 7fb36be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c10fb60 0x7fb36c1a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.323+0000 7fb36b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb36c109ad0 0x7fb36c1a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.323+0000 7fb36aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb36c10a490 0x7fb36c1a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.323+0000 7fb36be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c10fb60 0x7fb36c1a4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:40234/0 (socket says 172.21.15.27:40234) 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.323+0000 7fb36b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb36c109ad0 0x7fb36c1a0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60962/0 (socket says 172.21.15.27:60962) 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.323+0000 7fb36be00700 1 -- 172.21.15.27:0/3206094444 learned_addr learned my addr 172.21.15.27:0/3206094444 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.323+0000 7fb36be00700 1 -- 172.21.15.27:0/3206094444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb36c109ad0 msgr2=0x7fb36c1a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.323+0000 7fb36be00700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb36c109ad0 0x7fb36c1a0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.323+0000 7fb36be00700 1 -- 172.21.15.27:0/3206094444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb36c10a490 msgr2=0x7fb36c1a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.323+0000 7fb36be00700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb36c10a490 0x7fb36c1a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.323+0000 7fb36be00700 1 -- 172.21.15.27:0/3206094444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c1a5220 con 0x7fb36c10fb60 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.323+0000 7fb36b400700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb36c109ad0 0x7fb36c1a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:07:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.324+0000 7fb36be00700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c10fb60 0x7fb36c1a4ae0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb36400ef30 tx=0x7fb36400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.324+0000 7fb363e00700 1 -- 172.21.15.27:0/3206094444 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3640092e0 con 0x7fb36c10fb60 2024-05-10T11:07:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.324+0000 7fb363e00700 1 -- 172.21.15.27:0/3206094444 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb364004bb0 con 0x7fb36c10fb60 2024-05-10T11:07:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.324+0000 7fb363e00700 1 -- 172.21.15.27:0/3206094444 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364004020 con 0x7fb36c10fb60 2024-05-10T11:07:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.324+0000 7fb372000700 1 -- 172.21.15.27:0/3206094444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36c1a54b0 con 0x7fb36c10fb60 2024-05-10T11:07:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.324+0000 7fb372000700 1 -- 172.21.15.27:0/3206094444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb36c1acd90 con 0x7fb36c10fb60 2024-05-10T11:07:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.326+0000 7fb372000700 1 -- 172.21.15.27:0/3206094444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb36c06a190 con 0x7fb36c10fb60 2024-05-10T11:07:06.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.328+0000 7fb363e00700 1 -- 172.21.15.27:0/3206094444 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fb36401b030 con 0x7fb36c10fb60 2024-05-10T11:07:06.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.330+0000 7fb363e00700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb3500b2900 0x7fb3500b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:06.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.330+0000 7fb363e00700 1 -- 172.21.15.27:0/3206094444 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fb3640e15e0 con 0x7fb36c10fb60 2024-05-10T11:07:06.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.330+0000 7fb36b400700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb3500b2900 0x7fb3500b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.331+0000 7fb36b400700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb3500b2900 0x7fb3500b4dc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb35c005fd0 tx=0x7fb35c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.334+0000 7fb363e00700 1 -- 172.21.15.27:0/3206094444 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb364022020 con 0x7fb36c10fb60 2024-05-10T11:07:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.621+0000 7fb372000700 1 -- 172.21.15.27:0/3206094444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb36c02e510 con 0x7fb36c10fb60 2024-05-10T11:07:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.622+0000 7fb363e00700 1 -- 172.21.15.27:0/3206094444 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7fb364010040 con 0x7fb36c10fb60 2024-05-10T11:07:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:07:06.622 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-10T11:07:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.625+0000 7fb372000700 1 -- 172.21.15.27:0/3206094444 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb3500b2900 msgr2=0x7fb3500b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.625+0000 7fb372000700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb3500b2900 0x7fb3500b4dc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb35c005fd0 tx=0x7fb35c009310 comp rx=0 tx=0).stop 2024-05-10T11:07:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.625+0000 7fb372000700 1 -- 172.21.15.27:0/3206094444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c10fb60 msgr2=0x7fb36c1a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.625+0000 7fb372000700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c10fb60 0x7fb36c1a4ae0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb36400ef30 tx=0x7fb36400c3a0 comp rx=0 tx=0).stop 2024-05-10T11:07:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.625+0000 7fb372000700 1 -- 172.21.15.27:0/3206094444 shutdown_connections 2024-05-10T11:07:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.625+0000 7fb372000700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb3500b2900 0x7fb3500b4dc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.625+0000 7fb372000700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c10fb60 0x7fb36c1a4ae0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.625+0000 7fb372000700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb36c109ad0 0x7fb36c1a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.625+0000 7fb372000700 1 --2- 172.21.15.27:0/3206094444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb36c10a490 0x7fb36c1a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:06.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.625+0000 7fb372000700 1 -- 172.21.15.27:0/3206094444 >> 172.21.15.27:0/3206094444 conn(0x7fb36c076c70 msgr2=0x7fb36c077290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:06.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.626+0000 7fb372000700 1 -- 172.21.15.27:0/3206094444 shutdown_connections 2024-05-10T11:07:06.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:06.626+0000 7fb372000700 1 -- 172.21.15.27:0/3206094444 wait complete. 2024-05-10T11:07:06.627 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-05-10T11:07:07.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:06 smithi027 ceph-mon[30054]: Deploying daemon mds.c on smithi189 2024-05-10T11:07:07.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:06 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.189:6834/1387617890,v1:172.21.15.189:6835/1387617890] up:boot 2024-05-10T11:07:07.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:06 smithi027 ceph-mon[30054]: fsmap 6 up:standby 2024-05-10T11:07:07.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:06 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-10T11:07:07.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:06 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3206094444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:07.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:06 smithi076 ceph-mon[34152]: Deploying daemon mds.c on smithi189 2024-05-10T11:07:07.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:06 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.189:6834/1387617890,v1:172.21.15.189:6835/1387617890] up:boot 2024-05-10T11:07:07.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:06 smithi076 ceph-mon[34152]: fsmap 6 up:standby 2024-05-10T11:07:07.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:06 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-10T11:07:07.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:06 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3206094444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:07.226 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:06 smithi189 ceph-mon[34842]: Deploying daemon mds.c on smithi189 2024-05-10T11:07:07.226 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:06 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.189:6834/1387617890,v1:172.21.15.189:6835/1387617890] up:boot 2024-05-10T11:07:07.226 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:06 smithi189 ceph-mon[34842]: fsmap 6 up:standby 2024-05-10T11:07:07.226 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:06 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-10T11:07:07.227 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:06 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3206094444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:08.091 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:07 smithi189 ceph-mon[34842]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:08.091 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:07:08 smithi189 podman[65485]: 2024-05-10 11:07:07.972030656 +0000 UTC m=+0.017830621 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:07:08.091 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:07:08 smithi189 podman[65485]: 2024-05-10 11:07:08.091396752 +0000 UTC m=+0.137196700 container create 288fb75edf64f6750ee019a23d24e8590b64d0c577d570f80acae2fe2a04204f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-c, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-10T11:07:08.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:07 smithi027 ceph-mon[30054]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:08.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:07 smithi076 ceph-mon[34152]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:08.737 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:07:08 smithi189 podman[65485]: 2024-05-10 11:07:08.344526764 +0000 UTC m=+0.390326711 container init 288fb75edf64f6750ee019a23d24e8590b64d0c577d570f80acae2fe2a04204f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-10T11:07:08.737 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:07:08 smithi189 podman[65485]: 2024-05-10 11:07:08.347485181 +0000 UTC m=+0.393285136 container start 288fb75edf64f6750ee019a23d24e8590b64d0c577d570f80acae2fe2a04204f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-10T11:07:08.737 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:07:08 smithi189 ceph-mds[65504]: -- 172.21.15.189:0/2489955271 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562231aeda40 con 0x562231ac0800 2024-05-10T11:07:08.737 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:07:08 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-c[65500]: starting mds.c at 2024-05-10T11:07:08.737 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:07:08 smithi189 bash[65485]: 288fb75edf64f6750ee019a23d24e8590b64d0c577d570f80acae2fe2a04204f 2024-05-10T11:07:08.737 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:07:08 smithi189 systemd[1]: Started Ceph mds.c for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:07:09.532 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:07:09.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:09.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:09.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:09.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:09.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:09.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:09.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:09 smithi027 ceph-mon[30054]: Deploying daemon mds.h on smithi076 2024-05-10T11:07:09.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:09 smithi027 ceph-mon[30054]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:09.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:09 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] up:boot 2024-05-10T11:07:09.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:09 smithi027 ceph-mon[30054]: fsmap 7 up:standby 2024-05-10T11:07:09.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:09 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-10T11:07:09.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:09.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:09.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:09.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:09.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:09.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:09.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:09 smithi076 ceph-mon[34152]: Deploying daemon mds.h on smithi076 2024-05-10T11:07:09.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:09 smithi076 ceph-mon[34152]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:09.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:09 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] up:boot 2024-05-10T11:07:09.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:09 smithi076 ceph-mon[34152]: fsmap 7 up:standby 2024-05-10T11:07:09.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:09 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-10T11:07:09.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:09 smithi189 ceph-mon[34842]: Deploying daemon mds.h on smithi076 2024-05-10T11:07:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:09 smithi189 ceph-mon[34842]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:09 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] up:boot 2024-05-10T11:07:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:09 smithi189 ceph-mon[34842]: fsmap 7 up:standby 2024-05-10T11:07:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:09 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-10T11:07:10.010 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:07:10.925 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:07:10 smithi076 systemd[1]: Starting Ceph mds.h for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:07:11.340 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:07:11 smithi076 podman[64409]: 2024-05-10 11:07:10.964857983 +0000 UTC m=+0.017507356 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:07:11.340 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:07:11 smithi076 podman[64409]: 2024-05-10 11:07:11.088013503 +0000 UTC m=+0.140662857 container create d0eb84fcde181738e1b7fd706a7ef9cec06d71440ec9b8589b549e54031fa6c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08) 2024-05-10T11:07:11.340 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:07:11 smithi076 podman[64409]: 2024-05-10 11:07:11.341013557 +0000 UTC m=+0.393662911 container init d0eb84fcde181738e1b7fd706a7ef9cec06d71440ec9b8589b549e54031fa6c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-h, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS) 2024-05-10T11:07:11.675 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:07:11 smithi076 podman[64409]: 2024-05-10 11:07:11.3436186 +0000 UTC m=+0.396267954 container start d0eb84fcde181738e1b7fd706a7ef9cec06d71440ec9b8589b549e54031fa6c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-h, org.label-schema.build-date=20240507, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-10T11:07:11.675 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:07:11 smithi076 ceph-mds[64428]: -- 172.21.15.76:0/3308494313 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c221a6fa40 con 0x55c221a42c00 2024-05-10T11:07:11.676 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:07:11 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-h[64424]: starting mds.h at 2024-05-10T11:07:11.676 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:07:11 smithi076 bash[64409]: d0eb84fcde181738e1b7fd706a7ef9cec06d71440ec9b8589b549e54031fa6c0 2024-05-10T11:07:11.676 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:07:11 smithi076 systemd[1]: Started Ceph mds.h for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:07:12.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:11 smithi027 ceph-mon[30054]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:12.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:12.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:12.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:12.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:12.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:12.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:11 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:12.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:11 smithi076 ceph-mon[34152]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:12.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:11 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:12.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:11 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:12.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:11 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:12.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:11 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:12.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:11 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:12.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:11 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:12.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:11 smithi189 ceph-mon[34842]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:12.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:11 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:12.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:11 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:12.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:11 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:12.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:11 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:12.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:11 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:12.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:11 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.409+0000 7ff00ea00700 1 -- 172.21.15.27:0/1740121491 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff008109460 msgr2=0x7ff008109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.409+0000 7ff00ea00700 1 --2- 172.21.15.27:0/1740121491 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff008109460 0x7ff008109840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feffc009f90 tx=0x7feffc00b330 comp rx=0 tx=0).stop 2024-05-10T11:07:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.409+0000 7ff00ea00700 1 -- 172.21.15.27:0/1740121491 shutdown_connections 2024-05-10T11:07:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.409+0000 7ff00ea00700 1 --2- 172.21.15.27:0/1740121491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff008103d90 0x7ff0081102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.409+0000 7ff00ea00700 1 --2- 172.21.15.27:0/1740121491 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff008109460 0x7ff008109840 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:12.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.409+0000 7ff00ea00700 1 --2- 172.21.15.27:0/1740121491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0081033d0 0x7ff008103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:12.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.409+0000 7ff00ea00700 1 -- 172.21.15.27:0/1740121491 >> 172.21.15.27:0/1740121491 conn(0x7ff0080fef20 msgr2=0x7ff008101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:12.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.409+0000 7ff00ea00700 1 -- 172.21.15.27:0/1740121491 shutdown_connections 2024-05-10T11:07:12.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.409+0000 7ff00ea00700 1 -- 172.21.15.27:0/1740121491 wait complete. 2024-05-10T11:07:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.410+0000 7ff00ea00700 1 Processor -- start 2024-05-10T11:07:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.410+0000 7ff00ea00700 1 -- start start 2024-05-10T11:07:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0081033d0 0x7ff00819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff008103d90 0x7ff00819fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff008109460 0x7ff0081998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00ea00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff0081135c0 con 0x7ff0081033d0 2024-05-10T11:07:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00ea00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff008113440 con 0x7ff008109460 2024-05-10T11:07:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00ea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff008113740 con 0x7ff008103d90 2024-05-10T11:07:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0081033d0 0x7ff00819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:12.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0081033d0 0x7ff00819f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36572/0 (socket says 172.21.15.27:36572) 2024-05-10T11:07:12.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00d600700 1 -- 172.21.15.27:0/616504119 learned_addr learned my addr 172.21.15.27:0/616504119 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:07:12.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00cc00700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff008103d90 0x7ff00819fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:12.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00e000700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff008109460 0x7ff0081998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:12.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00cc00700 1 -- 172.21.15.27:0/616504119 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff008109460 msgr2=0x7ff0081998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00cc00700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff008109460 0x7ff0081998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00cc00700 1 -- 172.21.15.27:0/616504119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0081033d0 msgr2=0x7ff00819f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.411+0000 7ff00cc00700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0081033d0 0x7ff00819f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.412+0000 7ff00cc00700 1 -- 172.21.15.27:0/616504119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00819a020 con 0x7ff008103d90 2024-05-10T11:07:12.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.412+0000 7ff00cc00700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff008103d90 0x7ff00819fc40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feff4009c50 tx=0x7feff4009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:12.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.412+0000 7ff006000700 1 -- 172.21.15.27:0/616504119 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff401e070 con 0x7ff008103d90 2024-05-10T11:07:12.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.412+0000 7ff006000700 1 -- 172.21.15.27:0/616504119 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feff4019410 con 0x7ff008103d90 2024-05-10T11:07:12.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.412+0000 7ff00ea00700 1 -- 172.21.15.27:0/616504119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff00819a310 con 0x7ff008103d90 2024-05-10T11:07:12.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.412+0000 7ff006000700 1 -- 172.21.15.27:0/616504119 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4018610 con 0x7ff008103d90 2024-05-10T11:07:12.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.412+0000 7ff00ea00700 1 -- 172.21.15.27:0/616504119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0080756d0 con 0x7ff008103d90 2024-05-10T11:07:12.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.413+0000 7ff00ea00700 1 -- 172.21.15.27:0/616504119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff00806a190 con 0x7ff008103d90 2024-05-10T11:07:12.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.416+0000 7ff006000700 1 -- 172.21.15.27:0/616504119 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7feff4021070 con 0x7ff008103d90 2024-05-10T11:07:12.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.417+0000 7ff006000700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7feff00b2900 0x7feff00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:12.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.417+0000 7ff006000700 1 -- 172.21.15.27:0/616504119 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7feff40e5280 con 0x7ff008103d90 2024-05-10T11:07:12.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.418+0000 7ff00d600700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7feff00b2900 0x7feff00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:12.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.418+0000 7ff00d600700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7feff00b2900 0x7feff00b4dc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7feffc00b6c0 tx=0x7feffc009700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.421+0000 7ff006000700 1 -- 172.21.15.27:0/616504119 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7feff40ad3a0 con 0x7ff008103d90 2024-05-10T11:07:12.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.709+0000 7ff00ea00700 1 -- 172.21.15.27:0/616504119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff00819adf0 con 0x7ff008103d90 2024-05-10T11:07:12.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.710+0000 7ff006000700 1 -- 172.21.15.27:0/616504119 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7feff40b5420 con 0x7ff008103d90 2024-05-10T11:07:12.712 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:07:12.712 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-10T11:07:12.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.714+0000 7ff00ea00700 1 -- 172.21.15.27:0/616504119 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7feff00b2900 msgr2=0x7feff00b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:12.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.714+0000 7ff00ea00700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7feff00b2900 0x7feff00b4dc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7feffc00b6c0 tx=0x7feffc009700 comp rx=0 tx=0).stop 2024-05-10T11:07:12.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.714+0000 7ff00ea00700 1 -- 172.21.15.27:0/616504119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff008103d90 msgr2=0x7ff00819fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:12.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.714+0000 7ff00ea00700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff008103d90 0x7ff00819fc40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feff4009c50 tx=0x7feff4009f60 comp rx=0 tx=0).stop 2024-05-10T11:07:12.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.714+0000 7ff00ea00700 1 -- 172.21.15.27:0/616504119 shutdown_connections 2024-05-10T11:07:12.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.714+0000 7ff00ea00700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7feff00b2900 0x7feff00b4dc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:12.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.714+0000 7ff00ea00700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff008109460 0x7ff0081998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:12.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.714+0000 7ff00ea00700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0081033d0 0x7ff00819f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:12.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.714+0000 7ff00ea00700 1 --2- 172.21.15.27:0/616504119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff008103d90 0x7ff00819fc40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:12.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.714+0000 7ff00ea00700 1 -- 172.21.15.27:0/616504119 >> 172.21.15.27:0/616504119 conn(0x7ff0080fef20 msgr2=0x7ff0080ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:12.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.714+0000 7ff00ea00700 1 -- 172.21.15.27:0/616504119 shutdown_connections 2024-05-10T11:07:12.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:12.714+0000 7ff00ea00700 1 -- 172.21.15.27:0/616504119 wait complete. 2024-05-10T11:07:12.716 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 9 2024-05-10T11:07:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:12 smithi027 ceph-mon[30054]: Deploying daemon mds.i on smithi189 2024-05-10T11:07:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:12 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] up:boot 2024-05-10T11:07:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:12 smithi027 ceph-mon[30054]: fsmap 8 up:standby 2024-05-10T11:07:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:12 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-10T11:07:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:12 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/616504119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:13.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:12 smithi076 ceph-mon[34152]: Deploying daemon mds.i on smithi189 2024-05-10T11:07:13.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:12 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] up:boot 2024-05-10T11:07:13.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:12 smithi076 ceph-mon[34152]: fsmap 8 up:standby 2024-05-10T11:07:13.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:12 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-10T11:07:13.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:12 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/616504119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:13.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:12 smithi189 ceph-mon[34842]: Deploying daemon mds.i on smithi189 2024-05-10T11:07:13.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:12 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] up:boot 2024-05-10T11:07:13.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:12 smithi189 ceph-mon[34842]: fsmap 8 up:standby 2024-05-10T11:07:13.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:12 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-10T11:07:13.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:12 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/616504119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:13.737 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:07:13 smithi189 systemd[1]: Starting Ceph mds.i for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:07:14.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:13 smithi027 ceph-mon[30054]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:14.124 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:13 smithi189 ceph-mon[34842]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:14.125 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:07:13 smithi189 podman[65793]: 2024-05-10 11:07:13.777001625 +0000 UTC m=+0.017993018 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:07:14.125 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:07:13 smithi189 podman[65793]: 2024-05-10 11:07:13.880788252 +0000 UTC m=+0.121779632 container create 1d25da1e43c518bca01553bb5daebba0af240792ed4d198be2233e7161e9797c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-i, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-10T11:07:14.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:13 smithi076 ceph-mon[34152]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:14.487 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:07:14 smithi189 podman[65793]: 2024-05-10 11:07:14.125598451 +0000 UTC m=+0.366589831 container init 1d25da1e43c518bca01553bb5daebba0af240792ed4d198be2233e7161e9797c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD) 2024-05-10T11:07:14.487 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:07:14 smithi189 podman[65793]: 2024-05-10 11:07:14.128900786 +0000 UTC m=+0.369892167 container start 1d25da1e43c518bca01553bb5daebba0af240792ed4d198be2233e7161e9797c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5) 2024-05-10T11:07:14.487 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:07:14 smithi189 ceph-mds[65812]: -- 172.21.15.189:0/2264204154 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5564ca7d5a40 con 0x5564ca7a8800 2024-05-10T11:07:14.487 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:07:14 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-i[65808]: starting mds.i at 2024-05-10T11:07:14.487 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:07:14 smithi189 bash[65793]: 1d25da1e43c518bca01553bb5daebba0af240792ed4d198be2233e7161e9797c 2024-05-10T11:07:14.487 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:07:14 smithi189 systemd[1]: Started Ceph mds.i for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:07:15.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:07:15.595 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:15 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:15.595 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:15 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:15.595 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:15 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:15.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:15 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:15.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:15 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:15.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:15 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:15.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:15 smithi027 ceph-mon[30054]: Deploying daemon mds.g on smithi027 2024-05-10T11:07:15.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:15 smithi027 ceph-mon[30054]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:15.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:15 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] up:boot 2024-05-10T11:07:15.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:15 smithi027 ceph-mon[30054]: fsmap 9 up:standby 2024-05-10T11:07:15.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:15 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-10T11:07:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:15 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:15 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:15 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:15.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:15 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:15.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:15 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:15.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:15 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:15.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:15 smithi076 ceph-mon[34152]: Deploying daemon mds.g on smithi027 2024-05-10T11:07:15.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:15 smithi076 ceph-mon[34152]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:15.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:15 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] up:boot 2024-05-10T11:07:15.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:15 smithi076 ceph-mon[34152]: fsmap 9 up:standby 2024-05-10T11:07:15.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:15 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-10T11:07:15.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:15 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:15.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:15 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:15.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:15 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:15.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:15 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:15.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:15 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:15.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:15 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:15.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:15 smithi189 ceph-mon[34842]: Deploying daemon mds.g on smithi027 2024-05-10T11:07:15.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:15 smithi189 ceph-mon[34842]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:15.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:15 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] up:boot 2024-05-10T11:07:15.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:15 smithi189 ceph-mon[34842]: fsmap 9 up:standby 2024-05-10T11:07:15.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:15 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-10T11:07:16.756 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:07:17.620 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:07:17 smithi027 systemd[1]: Starting Ceph mds.g for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:07:18.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:17 smithi027 ceph-mon[30054]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:18.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:17 smithi076 ceph-mon[34152]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:18.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:17 smithi189 ceph-mon[34842]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:18.417 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:07:18 smithi027 podman[71232]: 2024-05-10 11:07:18.283398783 +0000 UTC m=+0.046313862 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:07:18.417 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:07:18 smithi027 podman[71232]: 2024-05-10 11:07:18.416879455 +0000 UTC m=+0.179794537 container create 090e76e231458a9958d5e15083b7a88530750df2fa75d8f11f33c85dae3abe6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-10T11:07:19.008 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:07:18 smithi027 podman[71232]: 2024-05-10 11:07:18.704975112 +0000 UTC m=+0.467890238 container init 090e76e231458a9958d5e15083b7a88530750df2fa75d8f11f33c85dae3abe6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-g, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-10T11:07:19.008 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:07:18 smithi027 podman[71232]: 2024-05-10 11:07:18.717398785 +0000 UTC m=+0.480313852 container start 090e76e231458a9958d5e15083b7a88530750df2fa75d8f11f33c85dae3abe6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-g, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-10T11:07:19.008 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:07:18 smithi027 ceph-mds[71268]: -- 172.21.15.27:0/1645642297 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bf2a3f1a40 con 0x55bf2a3c4c00 2024-05-10T11:07:19.008 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:07:18 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-g[71262]: starting mds.g at 2024-05-10T11:07:19.264 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:07:19 smithi027 bash[71232]: 090e76e231458a9958d5e15083b7a88530750df2fa75d8f11f33c85dae3abe6f 2024-05-10T11:07:19.265 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:07:19 smithi027 systemd[1]: Started Ceph mds.g for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:07:19.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.426+0000 7f73c8600700 1 -- 172.21.15.27:0/3678159059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73c0103eb0 msgr2=0x7f73c0110390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:19.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.426+0000 7f73c8600700 1 --2- 172.21.15.27:0/3678159059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73c0103eb0 0x7f73c0110390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f73b80098f0 tx=0x7f73b8009c00 comp rx=0 tx=0).stop 2024-05-10T11:07:19.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.426+0000 7f73c8600700 1 -- 172.21.15.27:0/3678159059 shutdown_connections 2024-05-10T11:07:19.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.426+0000 7f73c8600700 1 --2- 172.21.15.27:0/3678159059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73c0109600 0x7f73c01099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:19.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.426+0000 7f73c8600700 1 --2- 172.21.15.27:0/3678159059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73c0103eb0 0x7f73c0110390 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:19.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.426+0000 7f73c8600700 1 --2- 172.21.15.27:0/3678159059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c01034f0 0x7f73c0103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:19.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.426+0000 7f73c8600700 1 -- 172.21.15.27:0/3678159059 >> 172.21.15.27:0/3678159059 conn(0x7f73c007a650 msgr2=0x7f73c007aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:19.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.426+0000 7f73c8600700 1 -- 172.21.15.27:0/3678159059 shutdown_connections 2024-05-10T11:07:19.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.426+0000 7f73c8600700 1 -- 172.21.15.27:0/3678159059 wait complete. 2024-05-10T11:07:19.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.427+0000 7f73c8600700 1 Processor -- start 2024-05-10T11:07:19.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.427+0000 7f73c8600700 1 -- start start 2024-05-10T11:07:19.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.428+0000 7f73c8600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c01034f0 0x7f73c019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:19.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.428+0000 7f73c8600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73c0103eb0 0x7f73c019fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:19.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.428+0000 7f73c8600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73c0109600 0x7f73c0199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:19.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.428+0000 7f73c8600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f73c01136a0 con 0x7f73c01034f0 2024-05-10T11:07:19.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.428+0000 7f73c8600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f73c0113520 con 0x7f73c0103eb0 2024-05-10T11:07:19.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.428+0000 7f73c8600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f73c0113820 con 0x7f73c0109600 2024-05-10T11:07:19.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.428+0000 7f73c5200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73c0103eb0 0x7f73c019fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:19.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.428+0000 7f73c6600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73c0109600 0x7f73c0199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:19.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.428+0000 7f73c6600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73c0109600 0x7f73c0199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:35218/0 (socket says 172.21.15.27:35218) 2024-05-10T11:07:19.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.428+0000 7f73c6600700 1 -- 172.21.15.27:0/2752646097 learned_addr learned my addr 172.21.15.27:0/2752646097 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:07:19.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.429+0000 7f73c5c00700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c01034f0 0x7f73c019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:19.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.429+0000 7f73c6600700 1 -- 172.21.15.27:0/2752646097 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73c0103eb0 msgr2=0x7f73c019fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:19.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.429+0000 7f73c6600700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73c0103eb0 0x7f73c019fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:19.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.429+0000 7f73c6600700 1 -- 172.21.15.27:0/2752646097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c01034f0 msgr2=0x7f73c019f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:19.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.429+0000 7f73c6600700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c01034f0 0x7f73c019f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:19.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.429+0000 7f73c6600700 1 -- 172.21.15.27:0/2752646097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c019a0c0 con 0x7f73c0109600 2024-05-10T11:07:19.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.429+0000 7f73c6600700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73c0109600 0x7f73c0199960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f73b8009810 tx=0x7f73b800fa50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:19.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.430+0000 7f73be000700 1 -- 172.21.15.27:0/2752646097 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b8005140 con 0x7f73c0109600 2024-05-10T11:07:19.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.430+0000 7f73c8600700 1 -- 172.21.15.27:0/2752646097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73c019a350 con 0x7f73c0109600 2024-05-10T11:07:19.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.430+0000 7f73be000700 1 -- 172.21.15.27:0/2752646097 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73b800b560 con 0x7f73c0109600 2024-05-10T11:07:19.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.430+0000 7f73be000700 1 -- 172.21.15.27:0/2752646097 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b8003920 con 0x7f73c0109600 2024-05-10T11:07:19.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.430+0000 7f73c8600700 1 -- 172.21.15.27:0/2752646097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73c01a6670 con 0x7f73c0109600 2024-05-10T11:07:19.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.432+0000 7f73c8600700 1 -- 172.21.15.27:0/2752646097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73c006a190 con 0x7f73c0109600 2024-05-10T11:07:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.434+0000 7f73be000700 1 -- 172.21.15.27:0/2752646097 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f73b800ba10 con 0x7f73c0109600 2024-05-10T11:07:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.435+0000 7f73be000700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f73a80b2900 0x7f73a80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.436+0000 7f73be000700 1 -- 172.21.15.27:0/2752646097 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f73b80e2810 con 0x7f73c0109600 2024-05-10T11:07:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.436+0000 7f73c5c00700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f73a80b2900 0x7f73a80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.437+0000 7f73c5c00700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f73a80b2900 0x7f73a80b4dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f73c019aaa0 tx=0x7f73b400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:19.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.440+0000 7f73be000700 1 -- 172.21.15.27:0/2752646097 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f73b8013030 con 0x7f73c0109600 2024-05-10T11:07:19.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.745+0000 7f73c8600700 1 -- 172.21.15.27:0/2752646097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73c002e510 con 0x7f73c0109600 2024-05-10T11:07:19.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.746+0000 7f73be000700 1 -- 172.21.15.27:0/2752646097 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7f73b80aa9c0 con 0x7f73c0109600 2024-05-10T11:07:19.748 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:07:19.748 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-10T11:07:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.750+0000 7f73c8600700 1 -- 172.21.15.27:0/2752646097 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f73a80b2900 msgr2=0x7f73a80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.750+0000 7f73c8600700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f73a80b2900 0x7f73a80b4dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f73c019aaa0 tx=0x7f73b400b040 comp rx=0 tx=0).stop 2024-05-10T11:07:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.750+0000 7f73c8600700 1 -- 172.21.15.27:0/2752646097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73c0109600 msgr2=0x7f73c0199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.750+0000 7f73c8600700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73c0109600 0x7f73c0199960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f73b8009810 tx=0x7f73b800fa50 comp rx=0 tx=0).stop 2024-05-10T11:07:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.750+0000 7f73c8600700 1 -- 172.21.15.27:0/2752646097 shutdown_connections 2024-05-10T11:07:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.750+0000 7f73c8600700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f73a80b2900 0x7f73a80b4dc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.750+0000 7f73c8600700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73c0109600 0x7f73c0199960 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.750+0000 7f73c8600700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c01034f0 0x7f73c019f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.750+0000 7f73c8600700 1 --2- 172.21.15.27:0/2752646097 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73c0103eb0 0x7f73c019fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.750+0000 7f73c8600700 1 -- 172.21.15.27:0/2752646097 >> 172.21.15.27:0/2752646097 conn(0x7f73c007a650 msgr2=0x7f73c0101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.750+0000 7f73c8600700 1 -- 172.21.15.27:0/2752646097 shutdown_connections 2024-05-10T11:07:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:19.750+0000 7f73c8600700 1 -- 172.21.15.27:0/2752646097 wait complete. 2024-05-10T11:07:19.752 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 11 2024-05-10T11:07:20.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:19 smithi027 ceph-mon[30054]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:19 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.27:6838/3240182674,v1:172.21.15.27:6839/3240182674] up:boot 2024-05-10T11:07:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:19 smithi027 ceph-mon[30054]: fsmap 10 up:standby 2024-05-10T11:07:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:19 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-10T11:07:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:19 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:19 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:19 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:19 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:19 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:19 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:19 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2752646097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:20.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:19 smithi076 ceph-mon[34152]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:20.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:19 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.27:6838/3240182674,v1:172.21.15.27:6839/3240182674] up:boot 2024-05-10T11:07:20.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:19 smithi076 ceph-mon[34152]: fsmap 10 up:standby 2024-05-10T11:07:20.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:19 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-10T11:07:20.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:19 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:20.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:19 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:20.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:19 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:20.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:19 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:20.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:19 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:20.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:19 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:20.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:19 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2752646097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:19 smithi189 ceph-mon[34842]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:19 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.27:6838/3240182674,v1:172.21.15.27:6839/3240182674] up:boot 2024-05-10T11:07:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:19 smithi189 ceph-mon[34842]: fsmap 10 up:standby 2024-05-10T11:07:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:19 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-10T11:07:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:19 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:19 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:19 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:19 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:19 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:19 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:19 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2752646097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:20.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:20 smithi027 ceph-mon[30054]: Deploying daemon mds.a on smithi027 2024-05-10T11:07:21.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:20 smithi076 ceph-mon[34152]: Deploying daemon mds.a on smithi027 2024-05-10T11:07:21.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:20 smithi189 ceph-mon[34842]: Deploying daemon mds.a on smithi027 2024-05-10T11:07:22.089 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:21 smithi027 ceph-mon[30054]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:22.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:21 smithi076 ceph-mon[34152]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:22.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:21 smithi189 ceph-mon[34842]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:22.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:07:23.332 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:07:22 smithi027 systemd[1]: Starting Ceph mds.a for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:07:23.620 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:07:23 smithi027 podman[71628]: 2024-05-10 11:07:23.238014515 +0000 UTC m=+0.036405493 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:07:23.620 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:07:23 smithi027 podman[71628]: 2024-05-10 11:07:23.358241941 +0000 UTC m=+0.156632940 container create 4808f01523b20775d2f362e7c2be200d8eb57b6a27b9dd67f77228759a76540e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-05-10T11:07:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:07:23.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:23 smithi027 ceph-mon[30054]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:23.962 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:07:23 smithi027 podman[71628]: 2024-05-10 11:07:23.620860774 +0000 UTC m=+0.419251794 container init 4808f01523b20775d2f362e7c2be200d8eb57b6a27b9dd67f77228759a76540e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-10T11:07:23.962 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:07:23 smithi027 podman[71628]: 2024-05-10 11:07:23.631366083 +0000 UTC m=+0.429757070 container start 4808f01523b20775d2f362e7c2be200d8eb57b6a27b9dd67f77228759a76540e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-10T11:07:23.962 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:07:23 smithi027 ceph-mds[71673]: -- 172.21.15.27:0/3508657221 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5611fa129a40 con 0x5611fa0fc800 2024-05-10T11:07:23.963 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:07:23 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a[71669]: starting mds.a at 2024-05-10T11:07:23.963 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:07:23 smithi027 bash[71628]: 4808f01523b20775d2f362e7c2be200d8eb57b6a27b9dd67f77228759a76540e 2024-05-10T11:07:23.963 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:07:23 smithi027 systemd[1]: Started Ceph mds.a for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:07:24.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:23 smithi076 ceph-mon[34152]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:24.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:23 smithi189 ceph-mon[34842]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:25.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:24 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.27:6840/3080402649,v1:172.21.15.27:6841/3080402649] up:boot 2024-05-10T11:07:25.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:24 smithi027 ceph-mon[30054]: fsmap 11 up:standby 2024-05-10T11:07:25.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:24 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-10T11:07:25.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:24 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:25.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:24 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:25.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:24 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:25.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:24 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:25.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:24 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:25.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:24 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:25.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:24 smithi027 ceph-mon[30054]: Deploying daemon mds.b on smithi076 2024-05-10T11:07:25.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:24 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.27:6840/3080402649,v1:172.21.15.27:6841/3080402649] up:boot 2024-05-10T11:07:25.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:24 smithi076 ceph-mon[34152]: fsmap 11 up:standby 2024-05-10T11:07:25.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:24 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-10T11:07:25.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:24 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:25.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:24 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:25.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:24 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:25.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:24 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:25.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:24 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:25.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:24 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:25.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:24 smithi076 ceph-mon[34152]: Deploying daemon mds.b on smithi076 2024-05-10T11:07:25.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:24 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.27:6840/3080402649,v1:172.21.15.27:6841/3080402649] up:boot 2024-05-10T11:07:25.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:24 smithi189 ceph-mon[34842]: fsmap 11 up:standby 2024-05-10T11:07:25.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:24 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-10T11:07:25.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:24 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:25.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:24 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:25.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:24 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:25.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:24 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-10T11:07:25.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:24 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-10T11:07:25.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:24 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:25.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:24 smithi189 ceph-mon[34842]: Deploying daemon mds.b on smithi076 2024-05-10T11:07:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.043+0000 7fa052200700 1 -- 172.21.15.27:0/284189525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa04c105690 msgr2=0x7fa04c105b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.043+0000 7fa052200700 1 --2- 172.21.15.27:0/284189525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa04c105690 0x7fa04c105b10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa034009990 tx=0x7fa034009ca0 comp rx=0 tx=0).stop 2024-05-10T11:07:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.043+0000 7fa052200700 1 -- 172.21.15.27:0/284189525 shutdown_connections 2024-05-10T11:07:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.043+0000 7fa052200700 1 --2- 172.21.15.27:0/284189525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa04c106050 0x7fa04c112560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.043+0000 7fa052200700 1 --2- 172.21.15.27:0/284189525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c10b690 0x7fa04c10ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.043+0000 7fa052200700 1 --2- 172.21.15.27:0/284189525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa04c105690 0x7fa04c105b10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.043+0000 7fa052200700 1 -- 172.21.15.27:0/284189525 >> 172.21.15.27:0/284189525 conn(0x7fa04c101180 msgr2=0x7fa04c1035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.043+0000 7fa052200700 1 -- 172.21.15.27:0/284189525 shutdown_connections 2024-05-10T11:07:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.043+0000 7fa052200700 1 -- 172.21.15.27:0/284189525 wait complete. 2024-05-10T11:07:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.044+0000 7fa052200700 1 Processor -- start 2024-05-10T11:07:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.044+0000 7fa052200700 1 -- start start 2024-05-10T11:07:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa052200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa04c105690 0x7fa04c19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa052200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa04c106050 0x7fa04c19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa052200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c10b690 0x7fa04c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa052200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c1158c0 con 0x7fa04c10b690 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa052200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c115740 con 0x7fa04c105690 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa052200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c115a40 con 0x7fa04c106050 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa04b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa04c105690 0x7fa04c19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa04b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa04c105690 0x7fa04c19bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45050/0 (socket says 172.21.15.27:45050) 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa04b400700 1 -- 172.21.15.27:0/2198172732 learned_addr learned my addr 172.21.15.27:0/2198172732 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa04be00700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c10b690 0x7fa04c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa042a00700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa04c106050 0x7fa04c19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa04b400700 1 -- 172.21.15.27:0/2198172732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa04c106050 msgr2=0x7fa04c19c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa04b400700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa04c106050 0x7fa04c19c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa04b400700 1 -- 172.21.15.27:0/2198172732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c10b690 msgr2=0x7fa04c1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa04b400700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c10b690 0x7fa04c1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.045+0000 7fa04b400700 1 -- 172.21.15.27:0/2198172732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04c1a0da0 con 0x7fa04c105690 2024-05-10T11:07:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.046+0000 7fa04b400700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa04c105690 0x7fa04c19bd90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00e9a0 tx=0x7fa03c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.046+0000 7fa048c00700 1 -- 172.21.15.27:0/2198172732 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c00ca10 con 0x7fa04c105690 2024-05-10T11:07:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.046+0000 7fa048c00700 1 -- 172.21.15.27:0/2198172732 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa03c00cbb0 con 0x7fa04c105690 2024-05-10T11:07:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.046+0000 7fa052200700 1 -- 172.21.15.27:0/2198172732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa04c1a1090 con 0x7fa04c105690 2024-05-10T11:07:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.046+0000 7fa048c00700 1 -- 172.21.15.27:0/2198172732 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c018680 con 0x7fa04c105690 2024-05-10T11:07:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.046+0000 7fa052200700 1 -- 172.21.15.27:0/2198172732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa04c1a8910 con 0x7fa04c105690 2024-05-10T11:07:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.047+0000 7fa052200700 1 -- 172.21.15.27:0/2198172732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa04c06a190 con 0x7fa04c105690 2024-05-10T11:07:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.050+0000 7fa048c00700 1 -- 172.21.15.27:0/2198172732 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa03c010430 con 0x7fa04c105690 2024-05-10T11:07:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.051+0000 7fa048c00700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa0300b2900 0x7fa0300b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.052+0000 7fa048c00700 1 -- 172.21.15.27:0/2198172732 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fa03c0e2200 con 0x7fa04c105690 2024-05-10T11:07:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.052+0000 7fa042a00700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa0300b2900 0x7fa0300b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:26.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.052+0000 7fa042a00700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa0300b2900 0x7fa0300b4dc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa034009750 tx=0x7fa03400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:26.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.053+0000 7fa048c00700 1 -- 172.21.15.27:0/2198172732 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa03c0aa320 con 0x7fa04c105690 2024-05-10T11:07:26.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:25 smithi027 ceph-mon[30054]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:26.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:25 smithi076 ceph-mon[34152]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:26.176 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:07:26 smithi076 systemd[1]: Starting Ceph mds.b for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:07:26.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:25 smithi189 ceph-mon[34842]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:26.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.343+0000 7fa052200700 1 -- 172.21.15.27:0/2198172732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa04c02e510 con 0x7fa04c105690 2024-05-10T11:07:26.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.344+0000 7fa048c00700 1 -- 172.21.15.27:0/2198172732 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7fa03c0b23a0 con 0x7fa04c105690 2024-05-10T11:07:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:07:26.345 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-10T11:07:26.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.347+0000 7fa052200700 1 -- 172.21.15.27:0/2198172732 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa0300b2900 msgr2=0x7fa0300b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:26.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.347+0000 7fa052200700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa0300b2900 0x7fa0300b4dc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa034009750 tx=0x7fa03400b260 comp rx=0 tx=0).stop 2024-05-10T11:07:26.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.347+0000 7fa052200700 1 -- 172.21.15.27:0/2198172732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa04c105690 msgr2=0x7fa04c19bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:26.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.347+0000 7fa052200700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa04c105690 0x7fa04c19bd90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00e9a0 tx=0x7fa03c00ecb0 comp rx=0 tx=0).stop 2024-05-10T11:07:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.347+0000 7fa052200700 1 -- 172.21.15.27:0/2198172732 shutdown_connections 2024-05-10T11:07:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.347+0000 7fa052200700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa0300b2900 0x7fa0300b4dc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.347+0000 7fa052200700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c10b690 0x7fa04c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.347+0000 7fa052200700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa04c105690 0x7fa04c19bd90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.347+0000 7fa052200700 1 --2- 172.21.15.27:0/2198172732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa04c106050 0x7fa04c19c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.347+0000 7fa052200700 1 -- 172.21.15.27:0/2198172732 >> 172.21.15.27:0/2198172732 conn(0x7fa04c101180 msgr2=0x7fa04c101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.348+0000 7fa052200700 1 -- 172.21.15.27:0/2198172732 shutdown_connections 2024-05-10T11:07:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:26.348+0000 7fa052200700 1 -- 172.21.15.27:0/2198172732 wait complete. 2024-05-10T11:07:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 12 2024-05-10T11:07:26.505 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:07:26 smithi076 podman[64728]: 2024-05-10 11:07:26.147961129 +0000 UTC m=+0.018976879 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:07:26.505 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:07:26 smithi076 podman[64728]: 2024-05-10 11:07:26.260991398 +0000 UTC m=+0.132007177 container create 46f39929fdc34b5e297c415d7ed8da10de227b508e86027bfe5a2873a2054090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:07:26.505 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:07:26 smithi076 podman[64728]: 2024-05-10 11:07:26.50592787 +0000 UTC m=+0.376943603 container init 46f39929fdc34b5e297c415d7ed8da10de227b508e86027bfe5a2873a2054090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507) 2024-05-10T11:07:26.926 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:07:26 smithi076 podman[64728]: 2024-05-10 11:07:26.508896374 +0000 UTC m=+0.379912115 container start 46f39929fdc34b5e297c415d7ed8da10de227b508e86027bfe5a2873a2054090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-10T11:07:26.926 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:07:26 smithi076 ceph-mds[64747]: -- 172.21.15.76:0/1861551494 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ae310d7a40 con 0x55ae310aac00 2024-05-10T11:07:26.926 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:07:26 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b[64743]: starting mds.b at 2024-05-10T11:07:26.926 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:07:26 smithi076 bash[64728]: 46f39929fdc34b5e297c415d7ed8da10de227b508e86027bfe5a2873a2054090 2024-05-10T11:07:26.926 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:07:26 smithi076 systemd[1]: Started Ceph mds.b for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:07:27.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:26 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2198172732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:27.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:26 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:26 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:26 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:26 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:26 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.239 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:26 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2198172732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:27.239 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:26 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.239 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:26 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.239 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:26 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.240 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:26 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.240 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:26 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:26 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2198172732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:27.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:26 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:26 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:26 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:26 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:27.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:26 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:28.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:27 smithi189 ceph-mon[34842]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:28.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:27 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] up:boot 2024-05-10T11:07:28.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:27 smithi189 ceph-mon[34842]: fsmap 12 up:standby 2024-05-10T11:07:28.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:27 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-10T11:07:28.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:27 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:07:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:27 smithi027 ceph-mon[30054]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:27 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] up:boot 2024-05-10T11:07:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:27 smithi027 ceph-mon[30054]: fsmap 12 up:standby 2024-05-10T11:07:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:27 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-10T11:07:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:27 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:07:28.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:27 smithi076 ceph-mon[34152]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:28.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:27 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] up:boot 2024-05-10T11:07:28.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:27 smithi076 ceph-mon[34152]: fsmap 12 up:standby 2024-05-10T11:07:28.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:27 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-10T11:07:28.393 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:27 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:07:29.294 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:07:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:07:30.076 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:29 smithi027 ceph-mon[30054]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:30.076 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:30.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:30.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:30.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:29 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:30.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:29 smithi076 ceph-mon[34152]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:30.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:30.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:30.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:30.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:29 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:30.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:29 smithi189 ceph-mon[34842]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:30.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:29 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:07:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:07:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:30 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:31.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:30 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:31.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:30 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:31.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:30 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:31.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:30 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:07:31.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:30 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:31.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:30 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:07:31.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:30 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:31.212 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:31.212 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:31.212 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:31.212 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:07:31.212 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:31.212 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:07:31.213 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:30 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:32.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:31 smithi076 ceph-mon[34152]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:32.177 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:31 smithi027 ceph-mon[30054]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.176+0000 7ff226000700 1 -- 172.21.15.27:0/1304743440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22010a3a0 msgr2=0x7ff220116840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.176+0000 7ff226000700 1 --2- 172.21.15.27:0/1304743440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22010a3a0 0x7ff220116840 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff21800b0a0 tx=0x7ff21800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:07:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.176+0000 7ff226000700 1 -- 172.21.15.27:0/1304743440 shutdown_connections 2024-05-10T11:07:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.176+0000 7ff226000700 1 --2- 172.21.15.27:0/1304743440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff22010a3a0 0x7ff220116840 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.176+0000 7ff226000700 1 --2- 172.21.15.27:0/1304743440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff22010e8b0 0x7ff22010ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.176+0000 7ff226000700 1 --2- 172.21.15.27:0/1304743440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2201099e0 0x7ff220109e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.176+0000 7ff226000700 1 -- 172.21.15.27:0/1304743440 >> 172.21.15.27:0/1304743440 conn(0x7ff220079230 msgr2=0x7ff2201082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.176+0000 7ff226000700 1 -- 172.21.15.27:0/1304743440 shutdown_connections 2024-05-10T11:07:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.176+0000 7ff226000700 1 -- 172.21.15.27:0/1304743440 wait complete. 2024-05-10T11:07:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.178+0000 7ff226000700 1 Processor -- start 2024-05-10T11:07:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.178+0000 7ff226000700 1 -- start start 2024-05-10T11:07:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.179+0000 7ff226000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2201099e0 0x7ff2201a0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.179+0000 7ff226000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22010a3a0 0x7ff2201a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.179+0000 7ff226000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff22010e8b0 0x7ff2201a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.179+0000 7ff226000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff220119be0 con 0x7ff2201099e0 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.179+0000 7ff226000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff220119a60 con 0x7ff22010e8b0 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.179+0000 7ff226000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff220119d60 con 0x7ff22010a3a0 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.179+0000 7ff224c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2201099e0 0x7ff2201a0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.179+0000 7ff224c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2201099e0 0x7ff2201a0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49334/0 (socket says 172.21.15.27:49334) 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.179+0000 7ff224c00700 1 -- 172.21.15.27:0/992456692 learned_addr learned my addr 172.21.15.27:0/992456692 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.179+0000 7ff21fe00700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22010a3a0 0x7ff2201a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.179+0000 7ff225600700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff22010e8b0 0x7ff2201a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.180+0000 7ff21fe00700 1 -- 172.21.15.27:0/992456692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff22010e8b0 msgr2=0x7ff2201a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.180+0000 7ff21fe00700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff22010e8b0 0x7ff2201a4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.180+0000 7ff21fe00700 1 -- 172.21.15.27:0/992456692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2201099e0 msgr2=0x7ff2201a0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.180+0000 7ff21fe00700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2201099e0 0x7ff2201a0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.180+0000 7ff21fe00700 1 -- 172.21.15.27:0/992456692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2201a5060 con 0x7ff22010a3a0 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.180+0000 7ff224c00700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2201099e0 0x7ff2201a0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:07:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.180+0000 7ff21fe00700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22010a3a0 0x7ff2201a0590 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00e9f0 tx=0x7ff20c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.180+0000 7ff21d600700 1 -- 172.21.15.27:0/992456692 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c00cb70 con 0x7ff22010a3a0 2024-05-10T11:07:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.180+0000 7ff21d600700 1 -- 172.21.15.27:0/992456692 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff20c00ee90 con 0x7ff22010a3a0 2024-05-10T11:07:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.180+0000 7ff226000700 1 -- 172.21.15.27:0/992456692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2201a5350 con 0x7ff22010a3a0 2024-05-10T11:07:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.180+0000 7ff21d600700 1 -- 172.21.15.27:0/992456692 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c003d40 con 0x7ff22010a3a0 2024-05-10T11:07:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.180+0000 7ff226000700 1 -- 172.21.15.27:0/992456692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2201acbd0 con 0x7ff22010a3a0 2024-05-10T11:07:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.182+0000 7ff226000700 1 -- 172.21.15.27:0/992456692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff22006a190 con 0x7ff22010a3a0 2024-05-10T11:07:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.184+0000 7ff21d600700 1 -- 172.21.15.27:0/992456692 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ff20c003ee0 con 0x7ff22010a3a0 2024-05-10T11:07:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.185+0000 7ff21d600700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff2080b2880 0x7ff2080b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.185+0000 7ff21d600700 1 -- 172.21.15.27:0/992456692 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7ff20c0e1f00 con 0x7ff22010a3a0 2024-05-10T11:07:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.185+0000 7ff224c00700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff2080b2880 0x7ff2080b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.186+0000 7ff224c00700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff2080b2880 0x7ff2080b4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff214011550 tx=0x7ff214009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.189+0000 7ff21d600700 1 -- 172.21.15.27:0/992456692 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ff20c0aa020 con 0x7ff22010a3a0 2024-05-10T11:07:32.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:31 smithi189 ceph-mon[34842]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.486+0000 7ff226000700 1 -- 172.21.15.27:0/992456692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2201a13b0 con 0x7ff22010a3a0 2024-05-10T11:07:32.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.488+0000 7ff21d600700 1 -- 172.21.15.27:0/992456692 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10482 (secure 0 0 0) 0x7ff20c0a9920 con 0x7ff22010a3a0 2024-05-10T11:07:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:07:32.489 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24527,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-10T11:07:32.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.492+0000 7ff226000700 1 -- 172.21.15.27:0/992456692 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff2080b2880 msgr2=0x7ff2080b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:32.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.492+0000 7ff226000700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff2080b2880 0x7ff2080b4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff214011550 tx=0x7ff214009260 comp rx=0 tx=0).stop 2024-05-10T11:07:32.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.492+0000 7ff226000700 1 -- 172.21.15.27:0/992456692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22010a3a0 msgr2=0x7ff2201a0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:32.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.492+0000 7ff226000700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22010a3a0 0x7ff2201a0590 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00e9f0 tx=0x7ff20c00ed00 comp rx=0 tx=0).stop 2024-05-10T11:07:32.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.492+0000 7ff226000700 1 -- 172.21.15.27:0/992456692 shutdown_connections 2024-05-10T11:07:32.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.492+0000 7ff226000700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff2080b2880 0x7ff2080b4d40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:32.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.492+0000 7ff226000700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff22010e8b0 0x7ff2201a4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:32.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.492+0000 7ff226000700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2201099e0 0x7ff2201a0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:32.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.492+0000 7ff226000700 1 --2- 172.21.15.27:0/992456692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22010a3a0 0x7ff2201a0590 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:32.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.492+0000 7ff226000700 1 -- 172.21.15.27:0/992456692 >> 172.21.15.27:0/992456692 conn(0x7ff220079230 msgr2=0x7ff2200797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:32.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.493+0000 7ff226000700 1 -- 172.21.15.27:0/992456692 shutdown_connections 2024-05-10T11:07:32.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:32.493+0000 7ff226000700 1 -- 172.21.15.27:0/992456692 wait complete. 2024-05-10T11:07:32.494 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 13 2024-05-10T11:07:33.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:32 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:33.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:32 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:33.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:32 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:33.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:32 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:33.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:32 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/992456692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:32 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:32 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/992456692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:33.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:32 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:33.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:32 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:33.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:32 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:33.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:32 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:33.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:32 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/992456692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:07:33.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-05-10T11:07:34.014 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:07:34.318 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:34 smithi027 ceph-mon[30054]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:34.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:34 smithi076 ceph-mon[34152]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:34.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:34 smithi189 ceph-mon[34842]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.309+0000 7f3a27800700 1 -- 172.21.15.27:0/4134559769 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a2010b6c0 msgr2=0x7f3a2010baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.309+0000 7f3a27800700 1 --2- 172.21.15.27:0/4134559769 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a2010b6c0 0x7f3a2010baa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009990 tx=0x7f3a10009ca0 comp rx=0 tx=0).stop 2024-05-10T11:07:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.309+0000 7f3a27800700 1 -- 172.21.15.27:0/4134559769 shutdown_connections 2024-05-10T11:07:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.309+0000 7f3a27800700 1 --2- 172.21.15.27:0/4134559769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a20106080 0x7f3a20112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.309+0000 7f3a27800700 1 --2- 172.21.15.27:0/4134559769 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a2010b6c0 0x7f3a2010baa0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.309+0000 7f3a27800700 1 --2- 172.21.15.27:0/4134559769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a201056c0 0x7f3a20105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.309+0000 7f3a27800700 1 -- 172.21.15.27:0/4134559769 >> 172.21.15.27:0/4134559769 conn(0x7f3a201011b0 msgr2=0x7f3a201035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.309+0000 7f3a27800700 1 -- 172.21.15.27:0/4134559769 shutdown_connections 2024-05-10T11:07:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.309+0000 7f3a27800700 1 -- 172.21.15.27:0/4134559769 wait complete. 2024-05-10T11:07:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.310+0000 7f3a27800700 1 Processor -- start 2024-05-10T11:07:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.310+0000 7f3a27800700 1 -- start start 2024-05-10T11:07:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.311+0000 7f3a27800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a201056c0 0x7f3a2019be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.311+0000 7f3a27800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a20106080 0x7f3a2019c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.311+0000 7f3a27800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a2010b6c0 0x7f3a201a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.311+0000 7f3a27800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3a20115910 con 0x7f3a201056c0 2024-05-10T11:07:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.311+0000 7f3a27800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3a20115790 con 0x7f3a20106080 2024-05-10T11:07:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.311+0000 7f3a27800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3a20115a90 con 0x7f3a2010b6c0 2024-05-10T11:07:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.311+0000 7f3a24e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a201056c0 0x7f3a2019be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.311+0000 7f3a24e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a201056c0 0x7f3a2019be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34322/0 (socket says 172.21.15.27:34322) 2024-05-10T11:07:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.311+0000 7f3a24e00700 1 -- 172.21.15.27:0/1401060606 learned_addr learned my addr 172.21.15.27:0/1401060606 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:07:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.311+0000 7f3a25800700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a2010b6c0 0x7f3a201a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:36.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.312+0000 7f3a17e00700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a20106080 0x7f3a2019c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:36.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.312+0000 7f3a24e00700 1 -- 172.21.15.27:0/1401060606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a2010b6c0 msgr2=0x7f3a201a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.312+0000 7f3a24e00700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a2010b6c0 0x7f3a201a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.312+0000 7f3a24e00700 1 -- 172.21.15.27:0/1401060606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a20106080 msgr2=0x7f3a2019c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.312+0000 7f3a24e00700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a20106080 0x7f3a2019c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.312+0000 7f3a24e00700 1 -- 172.21.15.27:0/1401060606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a201a0de0 con 0x7f3a201056c0 2024-05-10T11:07:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.312+0000 7f3a17e00700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a20106080 0x7f3a2019c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:07:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.312+0000 7f3a24e00700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a201056c0 0x7f3a2019be20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3a10005910 tx=0x7f3a1000b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.312+0000 7f3a1e000700 1 -- 172.21.15.27:0/1401060606 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10027070 con 0x7f3a201056c0 2024-05-10T11:07:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.313+0000 7f3a27800700 1 -- 172.21.15.27:0/1401060606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a201a1070 con 0x7f3a201056c0 2024-05-10T11:07:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.313+0000 7f3a1e000700 1 -- 172.21.15.27:0/1401060606 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a10022470 con 0x7f3a201056c0 2024-05-10T11:07:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.313+0000 7f3a1e000700 1 -- 172.21.15.27:0/1401060606 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10017650 con 0x7f3a201056c0 2024-05-10T11:07:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.313+0000 7f3a27800700 1 -- 172.21.15.27:0/1401060606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a201a8950 con 0x7f3a201056c0 2024-05-10T11:07:36.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.314+0000 7f3a27800700 1 -- 172.21.15.27:0/1401060606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a2006a190 con 0x7f3a201056c0 2024-05-10T11:07:36.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:36 smithi027 ceph-mon[30054]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:36.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:36.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:36.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:36.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:07:36.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:36.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.316+0000 7f3a1e000700 1 -- 172.21.15.27:0/1401060606 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f3a1002a070 con 0x7f3a201056c0 2024-05-10T11:07:36.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.318+0000 7f3a1e000700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3a040b2900 0x7f3a040b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:36.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.318+0000 7f3a1e000700 1 -- 172.21.15.27:0/1401060606 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f3a100e2d00 con 0x7f3a201056c0 2024-05-10T11:07:36.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.318+0000 7f3a17e00700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3a040b2900 0x7f3a040b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:36.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.319+0000 7f3a17e00700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3a040b2900 0x7f3a040b4dc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3a2010ad30 tx=0x7f3a0c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:36.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.320+0000 7f3a1e000700 1 -- 172.21.15.27:0/1401060606 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f3a10013070 con 0x7f3a201056c0 2024-05-10T11:07:36.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:36 smithi076 ceph-mon[34152]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:36.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:36.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:36.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:36.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:07:36.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:36.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:36 smithi189 ceph-mon[34842]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:36.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:36.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:36.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:07:36.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:07:36.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:07:36.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.562+0000 7f3a27800700 1 -- 172.21.15.27:0/1401060606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3a2002e510 con 0x7f3a201056c0 2024-05-10T11:07:36.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.563+0000 7f3a1e000700 1 -- 172.21.15.27:0/1401060606 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7f3a100aaeb0 con 0x7f3a201056c0 2024-05-10T11:07:36.564 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:07:36.564 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-05-10T11:07:36.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:07:36.564 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-05-10T11:07:36.564 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-05-10T11:07:36.564 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:07:36.564 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-05-10T11:07:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.566+0000 7f3a27800700 1 -- 172.21.15.27:0/1401060606 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3a040b2900 msgr2=0x7f3a040b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.566+0000 7f3a27800700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3a040b2900 0x7f3a040b4dc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3a2010ad30 tx=0x7f3a0c009260 comp rx=0 tx=0).stop 2024-05-10T11:07:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.566+0000 7f3a27800700 1 -- 172.21.15.27:0/1401060606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a201056c0 msgr2=0x7f3a2019be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.566+0000 7f3a27800700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a201056c0 0x7f3a2019be20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3a10005910 tx=0x7f3a1000b670 comp rx=0 tx=0).stop 2024-05-10T11:07:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.566+0000 7f3a27800700 1 -- 172.21.15.27:0/1401060606 shutdown_connections 2024-05-10T11:07:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.566+0000 7f3a27800700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3a040b2900 0x7f3a040b4dc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:36.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.566+0000 7f3a27800700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a2010b6c0 0x7f3a201a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:36.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.566+0000 7f3a27800700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a201056c0 0x7f3a2019be20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:36.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.566+0000 7f3a27800700 1 --2- 172.21.15.27:0/1401060606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a20106080 0x7f3a2019c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:36.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.566+0000 7f3a27800700 1 -- 172.21.15.27:0/1401060606 >> 172.21.15.27:0/1401060606 conn(0x7f3a201011b0 msgr2=0x7f3a20101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:36.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.567+0000 7f3a27800700 1 -- 172.21.15.27:0/1401060606 shutdown_connections 2024-05-10T11:07:36.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:36.567+0000 7f3a27800700 1 -- 172.21.15.27:0/1401060606 wait complete. 2024-05-10T11:07:37.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:37 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1401060606' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-10T11:07:37.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:37 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1401060606' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-10T11:07:37.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:37 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1401060606' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-10T11:07:37.528 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-10T11:07:37.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-10T11:07:38.001 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:07:38.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:38 smithi027 ceph-mon[30054]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:38 smithi076 ceph-mon[34152]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:38.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:38 smithi189 ceph-mon[34842]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:40.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:40 smithi027 ceph-mon[30054]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:40.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.403+0000 7fd9c0a00700 1 -- 172.21.15.27:0/1932756594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9bc105f90 msgr2=0x7fd9bc1124a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:40.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.403+0000 7fd9c0a00700 1 --2- 172.21.15.27:0/1932756594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9bc105f90 0x7fd9bc1124a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400b0a0 tx=0x7fd9b400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:07:40.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.403+0000 7fd9c0a00700 1 -- 172.21.15.27:0/1932756594 shutdown_connections 2024-05-10T11:07:40.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.403+0000 7fd9c0a00700 1 --2- 172.21.15.27:0/1932756594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9bc10b660 0x7fd9bc10ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:40.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.403+0000 7fd9c0a00700 1 --2- 172.21.15.27:0/1932756594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9bc105f90 0x7fd9bc1124a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:40.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.403+0000 7fd9c0a00700 1 --2- 172.21.15.27:0/1932756594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc1055d0 0x7fd9bc105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:40.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.403+0000 7fd9c0a00700 1 -- 172.21.15.27:0/1932756594 >> 172.21.15.27:0/1932756594 conn(0x7fd9bc1010c0 msgr2=0x7fd9bc1034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:40.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.403+0000 7fd9c0a00700 1 -- 172.21.15.27:0/1932756594 shutdown_connections 2024-05-10T11:07:40.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.403+0000 7fd9c0a00700 1 -- 172.21.15.27:0/1932756594 wait complete. 2024-05-10T11:07:40.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.404+0000 7fd9c0a00700 1 Processor -- start 2024-05-10T11:07:40.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.404+0000 7fd9c0a00700 1 -- start start 2024-05-10T11:07:40.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.405+0000 7fd9c0a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9bc1055d0 0x7fd9bc1a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:40.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.405+0000 7fd9c0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc105f90 0x7fd9bc1a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:40.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.405+0000 7fd9c0a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9bc10b660 0x7fd9bc19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.405+0000 7fd9c0a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc115820 con 0x7fd9bc105f90 2024-05-10T11:07:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.405+0000 7fd9c0a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc1156a0 con 0x7fd9bc1055d0 2024-05-10T11:07:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.405+0000 7fd9c0a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc1159a0 con 0x7fd9bc10b660 2024-05-10T11:07:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.405+0000 7fd9baa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc105f90 0x7fd9bc1a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.405+0000 7fd9baa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc105f90 0x7fd9bc1a1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34328/0 (socket says 172.21.15.27:34328) 2024-05-10T11:07:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.405+0000 7fd9baa00700 1 -- 172.21.15.27:0/811516478 learned_addr learned my addr 172.21.15.27:0/811516478 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:07:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.405+0000 7fd9bb400700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9bc1055d0 0x7fd9bc1a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.405+0000 7fd9bbe00700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9bc10b660 0x7fd9bc19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.406+0000 7fd9baa00700 1 -- 172.21.15.27:0/811516478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9bc10b660 msgr2=0x7fd9bc19ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.406+0000 7fd9baa00700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9bc10b660 0x7fd9bc19ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.406+0000 7fd9baa00700 1 -- 172.21.15.27:0/811516478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9bc1055d0 msgr2=0x7fd9bc1a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.406+0000 7fd9baa00700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9bc1055d0 0x7fd9bc1a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:40.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.406+0000 7fd9baa00700 1 -- 172.21.15.27:0/811516478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9bc19c180 con 0x7fd9bc105f90 2024-05-10T11:07:40.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.406+0000 7fd9bbe00700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9bc10b660 0x7fd9bc19ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:07:40.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.406+0000 7fd9baa00700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc105f90 0x7fd9bc1a1d50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800e9a0 tx=0x7fd9a800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:40.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.406+0000 7fd9b3e00700 1 -- 172.21.15.27:0/811516478 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a800ca10 con 0x7fd9bc105f90 2024-05-10T11:07:40.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.406+0000 7fd9b3e00700 1 -- 172.21.15.27:0/811516478 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9a800cbb0 con 0x7fd9bc105f90 2024-05-10T11:07:40.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.406+0000 7fd9b3e00700 1 -- 172.21.15.27:0/811516478 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8018680 con 0x7fd9bc105f90 2024-05-10T11:07:40.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.406+0000 7fd9c0a00700 1 -- 172.21.15.27:0/811516478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9bc19c470 con 0x7fd9bc105f90 2024-05-10T11:07:40.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.407+0000 7fd9c0a00700 1 -- 172.21.15.27:0/811516478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9bc1a86e0 con 0x7fd9bc105f90 2024-05-10T11:07:40.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.408+0000 7fd9c0a00700 1 -- 172.21.15.27:0/811516478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9bc06a190 con 0x7fd9bc105f90 2024-05-10T11:07:40.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.411+0000 7fd9b3e00700 1 -- 172.21.15.27:0/811516478 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fd9a8010430 con 0x7fd9bc105f90 2024-05-10T11:07:40.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.413+0000 7fd9b3e00700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9a00b2900 0x7fd9a00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:40.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.413+0000 7fd9b3e00700 1 -- 172.21.15.27:0/811516478 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fd9a80e2240 con 0x7fd9bc105f90 2024-05-10T11:07:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.413+0000 7fd9bb400700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9a00b2900 0x7fd9a00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.414+0000 7fd9bb400700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9a00b2900 0x7fd9a00b4dc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc10a720 tx=0x7fd9ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.415+0000 7fd9b3e00700 1 -- 172.21.15.27:0/811516478 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fd9a80aa360 con 0x7fd9bc105f90 2024-05-10T11:07:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:40 smithi076 ceph-mon[34152]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:40.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:40 smithi189 ceph-mon[34842]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:40.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:40.653+0000 7fd9c0a00700 1 -- 172.21.15.27:0/811516478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd9bc02e510 con 0x7fd9bc105f90 2024-05-10T11:07:41.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.108+0000 7fd9b3e00700 1 -- 172.21.15.27:0/811516478 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7fd9a80b23e0 con 0x7fd9bc105f90 2024-05-10T11:07:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.113+0000 7fd9c0a00700 1 -- 172.21.15.27:0/811516478 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9a00b2900 msgr2=0x7fd9a00b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.113+0000 7fd9c0a00700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9a00b2900 0x7fd9a00b4dc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc10a720 tx=0x7fd9ac009260 comp rx=0 tx=0).stop 2024-05-10T11:07:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.113+0000 7fd9c0a00700 1 -- 172.21.15.27:0/811516478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc105f90 msgr2=0x7fd9bc1a1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.113+0000 7fd9c0a00700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc105f90 0x7fd9bc1a1d50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800e9a0 tx=0x7fd9a800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:07:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.114+0000 7fd9c0a00700 1 -- 172.21.15.27:0/811516478 shutdown_connections 2024-05-10T11:07:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.114+0000 7fd9c0a00700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9a00b2900 0x7fd9a00b4dc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.114+0000 7fd9c0a00700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9bc10b660 0x7fd9bc19ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.114+0000 7fd9c0a00700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9bc1055d0 0x7fd9bc1a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.114+0000 7fd9c0a00700 1 --2- 172.21.15.27:0/811516478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc105f90 0x7fd9bc1a1d50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.114+0000 7fd9c0a00700 1 -- 172.21.15.27:0/811516478 >> 172.21.15.27:0/811516478 conn(0x7fd9bc1010c0 msgr2=0x7fd9bc1027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.116+0000 7fd9c0a00700 1 -- 172.21.15.27:0/811516478 shutdown_connections 2024-05-10T11:07:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:41.116+0000 7fd9c0a00700 1 -- 172.21.15.27:0/811516478 wait complete. 2024-05-10T11:07:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-05-10T11:07:41.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:41 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/811516478' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-10T11:07:41.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:41 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/811516478' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-10T11:07:41.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:41 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/811516478' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-10T11:07:41.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-05-10T11:07:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:42 smithi027 ceph-mon[30054]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:42 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/811516478' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-10T11:07:42.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:42 smithi027 ceph-mon[30054]: osdmap e73: 12 total, 12 up, 12 in 2024-05-10T11:07:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:42 smithi076 ceph-mon[34152]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:42 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/811516478' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-10T11:07:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:42 smithi076 ceph-mon[34152]: osdmap e73: 12 total, 12 up, 12 in 2024-05-10T11:07:42.441 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:07:42.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:42 smithi189 ceph-mon[34842]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:42.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:42 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/811516478' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-10T11:07:42.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:42 smithi189 ceph-mon[34842]: osdmap e73: 12 total, 12 up, 12 in 2024-05-10T11:07:43.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:43 smithi027 ceph-mon[30054]: osdmap e74: 12 total, 12 up, 12 in 2024-05-10T11:07:43.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:43 smithi027 ceph-mon[30054]: pgmap v315: 33 pgs: 2 creating+activating, 3 creating+peering, 6 active+clean, 22 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:43 smithi076 ceph-mon[34152]: osdmap e74: 12 total, 12 up, 12 in 2024-05-10T11:07:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:43 smithi076 ceph-mon[34152]: pgmap v315: 33 pgs: 2 creating+activating, 3 creating+peering, 6 active+clean, 22 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:43.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:43 smithi189 ceph-mon[34842]: osdmap e74: 12 total, 12 up, 12 in 2024-05-10T11:07:43.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:43 smithi189 ceph-mon[34842]: pgmap v315: 33 pgs: 2 creating+activating, 3 creating+peering, 6 active+clean, 22 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:44.326 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:44 smithi027 ceph-mon[30054]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-10T11:07:44.326 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:44 smithi027 ceph-mon[30054]: osdmap e75: 12 total, 12 up, 12 in 2024-05-10T11:07:44.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:44 smithi076 ceph-mon[34152]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-10T11:07:44.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:44 smithi076 ceph-mon[34152]: osdmap e75: 12 total, 12 up, 12 in 2024-05-10T11:07:44.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:44 smithi189 ceph-mon[34842]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-10T11:07:44.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:44 smithi189 ceph-mon[34842]: osdmap e75: 12 total, 12 up, 12 in 2024-05-10T11:07:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.891+0000 7f17d0e00700 1 -- 172.21.15.27:0/2963940558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc1055d0 msgr2=0x7f17cc105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.891+0000 7f17d0e00700 1 --2- 172.21.15.27:0/2963940558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc1055d0 0x7f17cc105a50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009990 tx=0x7f17b8009ca0 comp rx=0 tx=0).stop 2024-05-10T11:07:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.891+0000 7f17d0e00700 1 -- 172.21.15.27:0/2963940558 shutdown_connections 2024-05-10T11:07:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.891+0000 7f17d0e00700 1 --2- 172.21.15.27:0/2963940558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17cc105f90 0x7f17cc1124a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.891+0000 7f17d0e00700 1 --2- 172.21.15.27:0/2963940558 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17cc10b660 0x7f17cc10ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.891+0000 7f17d0e00700 1 --2- 172.21.15.27:0/2963940558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc1055d0 0x7f17cc105a50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.891+0000 7f17d0e00700 1 -- 172.21.15.27:0/2963940558 >> 172.21.15.27:0/2963940558 conn(0x7f17cc1010c0 msgr2=0x7f17cc1034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.891+0000 7f17d0e00700 1 -- 172.21.15.27:0/2963940558 shutdown_connections 2024-05-10T11:07:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.891+0000 7f17d0e00700 1 -- 172.21.15.27:0/2963940558 wait complete. 2024-05-10T11:07:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.892+0000 7f17d0e00700 1 Processor -- start 2024-05-10T11:07:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.893+0000 7f17d0e00700 1 -- start start 2024-05-10T11:07:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.893+0000 7f17d0e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17cc1055d0 0x7f17cc1a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.893+0000 7f17d0e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17cc105f90 0x7f17cc1a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.893+0000 7f17d0e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc10b660 0x7f17cc19ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.893+0000 7f17d0e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc115820 con 0x7f17cc10b660 2024-05-10T11:07:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.893+0000 7f17d0e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc1156a0 con 0x7f17cc1055d0 2024-05-10T11:07:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.893+0000 7f17d0e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc1159a0 con 0x7f17cc105f90 2024-05-10T11:07:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.893+0000 7f17cbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc10b660 0x7f17cc19ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.893+0000 7f17caa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17cc105f90 0x7f17cc1a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.894+0000 7f17cbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc10b660 0x7f17cc19ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44570/0 (socket says 172.21.15.27:44570) 2024-05-10T11:07:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.894+0000 7f17cbe00700 1 -- 172.21.15.27:0/1633987326 learned_addr learned my addr 172.21.15.27:0/1633987326 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:07:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.894+0000 7f17cb400700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17cc1055d0 0x7f17cc1a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.894+0000 7f17caa00700 1 -- 172.21.15.27:0/1633987326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17cc1055d0 msgr2=0x7f17cc1a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.894+0000 7f17caa00700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17cc1055d0 0x7f17cc1a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.894+0000 7f17caa00700 1 -- 172.21.15.27:0/1633987326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc10b660 msgr2=0x7f17cc19ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.894+0000 7f17caa00700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc10b660 0x7f17cc19ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.894+0000 7f17caa00700 1 -- 172.21.15.27:0/1633987326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17cc19c1e0 con 0x7f17cc105f90 2024-05-10T11:07:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.894+0000 7f17caa00700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17cc105f90 0x7f17cc1a1db0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f17b80098e0 tx=0x7f17b80057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.895+0000 7f17c3e00700 1 -- 172.21.15.27:0/1633987326 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8027070 con 0x7f17cc105f90 2024-05-10T11:07:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.895+0000 7f17c3e00700 1 -- 172.21.15.27:0/1633987326 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17b8022470 con 0x7f17cc105f90 2024-05-10T11:07:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.895+0000 7f17c3e00700 1 -- 172.21.15.27:0/1633987326 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8017650 con 0x7f17cc105f90 2024-05-10T11:07:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.895+0000 7f17d0e00700 1 -- 172.21.15.27:0/1633987326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17cc19c470 con 0x7f17cc105f90 2024-05-10T11:07:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.895+0000 7f17d0e00700 1 -- 172.21.15.27:0/1633987326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17cc1a8740 con 0x7f17cc105f90 2024-05-10T11:07:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.896+0000 7f17d0e00700 1 -- 172.21.15.27:0/1633987326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17cc106d10 con 0x7f17cc105f90 2024-05-10T11:07:44.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.899+0000 7f17c3e00700 1 -- 172.21.15.27:0/1633987326 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f17b802a070 con 0x7f17cc105f90 2024-05-10T11:07:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.900+0000 7f17c3e00700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f17b00b2900 0x7f17b00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.901+0000 7f17c3e00700 1 -- 172.21.15.27:0/1633987326 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7497+0+0 (secure 0 0 0) 0x7f17b80e2e80 con 0x7f17cc105f90 2024-05-10T11:07:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.901+0000 7f17cb400700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f17b00b2900 0x7f17b00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:44.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.901+0000 7f17cb400700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f17b00b2900 0x7f17b00b4dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f17cc10a720 tx=0x7f17bc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:44.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:44.902+0000 7f17c3e00700 1 -- 172.21.15.27:0/1633987326 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f17b8031080 con 0x7f17cc105f90 2024-05-10T11:07:45.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:45.144+0000 7f17d0e00700 1 -- 172.21.15.27:0/1633987326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f17cc19cf60 con 0x7f17cc105f90 2024-05-10T11:07:45.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:45 smithi027 ceph-mon[30054]: pgmap v317: 33 pgs: 2 creating+activating, 3 creating+peering, 9 active+clean, 19 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:45.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:45 smithi076 ceph-mon[34152]: pgmap v317: 33 pgs: 2 creating+activating, 3 creating+peering, 9 active+clean, 19 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:45.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:45 smithi189 ceph-mon[34842]: pgmap v317: 33 pgs: 2 creating+activating, 3 creating+peering, 9 active+clean, 19 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:46.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.160+0000 7f17c3e00700 1 -- 172.21.15.27:0/1633987326 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v76) ==== 171+0+0 (secure 0 0 0) 0x7f17b8013070 con 0x7f17cc105f90 2024-05-10T11:07:46.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.169+0000 7f17d0e00700 1 -- 172.21.15.27:0/1633987326 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f17b00b2900 msgr2=0x7f17b00b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:46.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.169+0000 7f17d0e00700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f17b00b2900 0x7f17b00b4dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f17cc10a720 tx=0x7f17bc00b410 comp rx=0 tx=0).stop 2024-05-10T11:07:46.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.169+0000 7f17d0e00700 1 -- 172.21.15.27:0/1633987326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17cc105f90 msgr2=0x7f17cc1a1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:46.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.169+0000 7f17d0e00700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17cc105f90 0x7f17cc1a1db0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f17b80098e0 tx=0x7f17b80057c0 comp rx=0 tx=0).stop 2024-05-10T11:07:46.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.169+0000 7f17d0e00700 1 -- 172.21.15.27:0/1633987326 shutdown_connections 2024-05-10T11:07:46.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.169+0000 7f17d0e00700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f17b00b2900 0x7f17b00b4dc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:46.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.169+0000 7f17d0e00700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc10b660 0x7f17cc19ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:46.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.169+0000 7f17d0e00700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17cc1055d0 0x7f17cc1a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:46.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.169+0000 7f17d0e00700 1 --2- 172.21.15.27:0/1633987326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17cc105f90 0x7f17cc1a1db0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:46.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.169+0000 7f17d0e00700 1 -- 172.21.15.27:0/1633987326 >> 172.21.15.27:0/1633987326 conn(0x7f17cc1010c0 msgr2=0x7f17cc102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.172+0000 7f17d0e00700 1 -- 172.21.15.27:0/1633987326 shutdown_connections 2024-05-10T11:07:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:46.172+0000 7f17d0e00700 1 -- 172.21.15.27:0/1633987326 wait complete. 2024-05-10T11:07:46.176 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-05-10T11:07:46.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:46 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1633987326' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-10T11:07:46.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:46 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-10T11:07:46.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:46 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1633987326' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-10T11:07:46.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:46 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-10T11:07:46.612 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:46 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1633987326' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-10T11:07:46.613 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:46 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-10T11:07:47.050 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-10T11:07:47.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:47 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-10T11:07:47.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:47 smithi027 ceph-mon[30054]: osdmap e76: 12 total, 12 up, 12 in 2024-05-10T11:07:47.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:47 smithi027 ceph-mon[30054]: pgmap v319: 97 pgs: 60 unknown, 4 creating+peering, 2 creating+activating, 31 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:47.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:47 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-10T11:07:47.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:47 smithi076 ceph-mon[34152]: osdmap e76: 12 total, 12 up, 12 in 2024-05-10T11:07:47.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:47 smithi076 ceph-mon[34152]: pgmap v319: 97 pgs: 60 unknown, 4 creating+peering, 2 creating+activating, 31 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:47 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-10T11:07:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:47 smithi189 ceph-mon[34842]: osdmap e76: 12 total, 12 up, 12 in 2024-05-10T11:07:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:47 smithi189 ceph-mon[34842]: pgmap v319: 97 pgs: 60 unknown, 4 creating+peering, 2 creating+activating, 31 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:47.550 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:07:48.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:48 smithi027 ceph-mon[30054]: osdmap e77: 12 total, 12 up, 12 in 2024-05-10T11:07:48.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:48 smithi189 ceph-mon[34842]: osdmap e77: 12 total, 12 up, 12 in 2024-05-10T11:07:48.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:48 smithi076 ceph-mon[34152]: osdmap e77: 12 total, 12 up, 12 in 2024-05-10T11:07:49.316 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:49 smithi027 ceph-mon[30054]: osdmap e78: 12 total, 12 up, 12 in 2024-05-10T11:07:49.316 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:49 smithi027 ceph-mon[30054]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-10T11:07:49.316 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:49 smithi027 ceph-mon[30054]: pgmap v322: 97 pgs: 42 unknown, 12 creating+peering, 43 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:49.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:49 smithi189 ceph-mon[34842]: osdmap e78: 12 total, 12 up, 12 in 2024-05-10T11:07:49.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:49 smithi189 ceph-mon[34842]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-10T11:07:49.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:49 smithi189 ceph-mon[34842]: pgmap v322: 97 pgs: 42 unknown, 12 creating+peering, 43 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:49.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:49 smithi076 ceph-mon[34152]: osdmap e78: 12 total, 12 up, 12 in 2024-05-10T11:07:49.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:49 smithi076 ceph-mon[34152]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-10T11:07:49.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:49 smithi076 ceph-mon[34152]: pgmap v322: 97 pgs: 42 unknown, 12 creating+peering, 43 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:07:49.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.933+0000 7fe403e00700 1 -- 172.21.15.27:0/2805358163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe40410b660 msgr2=0x7fe40410ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:49.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.933+0000 7fe403e00700 1 --2- 172.21.15.27:0/2805358163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe40410b660 0x7fe40410ba40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40098f0 tx=0x7fe3f4009c00 comp rx=0 tx=0).stop 2024-05-10T11:07:49.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.933+0000 7fe403e00700 1 -- 172.21.15.27:0/2805358163 shutdown_connections 2024-05-10T11:07:49.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.933+0000 7fe403e00700 1 --2- 172.21.15.27:0/2805358163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe404105f90 0x7fe4041124a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:49.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.933+0000 7fe403e00700 1 --2- 172.21.15.27:0/2805358163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe40410b660 0x7fe40410ba40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:49.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.933+0000 7fe403e00700 1 --2- 172.21.15.27:0/2805358163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4041055d0 0x7fe404105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:49.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.933+0000 7fe403e00700 1 -- 172.21.15.27:0/2805358163 >> 172.21.15.27:0/2805358163 conn(0x7fe4041010c0 msgr2=0x7fe4041034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:49.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.933+0000 7fe403e00700 1 -- 172.21.15.27:0/2805358163 shutdown_connections 2024-05-10T11:07:49.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.933+0000 7fe403e00700 1 -- 172.21.15.27:0/2805358163 wait complete. 2024-05-10T11:07:49.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.934+0000 7fe403e00700 1 Processor -- start 2024-05-10T11:07:49.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.934+0000 7fe403e00700 1 -- start start 2024-05-10T11:07:49.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.935+0000 7fe403e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4041055d0 0x7fe4041a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:49.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.935+0000 7fe403e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe404105f90 0x7fe4041a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:49.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.935+0000 7fe403e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40410b660 0x7fe40419baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:49.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.935+0000 7fe403e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe404115800 con 0x7fe40410b660 2024-05-10T11:07:49.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.935+0000 7fe403e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe404115680 con 0x7fe404105f90 2024-05-10T11:07:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.935+0000 7fe403e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe404115980 con 0x7fe4041055d0 2024-05-10T11:07:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.935+0000 7fe403400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40410b660 0x7fe40419baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.935+0000 7fe402000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe404105f90 0x7fe4041a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.935+0000 7fe402000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe404105f90 0x7fe4041a1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52854/0 (socket says 172.21.15.27:52854) 2024-05-10T11:07:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.935+0000 7fe403400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40410b660 0x7fe40419baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44576/0 (socket says 172.21.15.27:44576) 2024-05-10T11:07:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.935+0000 7fe402000700 1 -- 172.21.15.27:0/1296350528 learned_addr learned my addr 172.21.15.27:0/1296350528 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:07:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.935+0000 7fe402a00700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4041055d0 0x7fe4041a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.936+0000 7fe402000700 1 -- 172.21.15.27:0/1296350528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4041055d0 msgr2=0x7fe4041a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.936+0000 7fe402000700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4041055d0 0x7fe4041a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.936+0000 7fe402000700 1 -- 172.21.15.27:0/1296350528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40410b660 msgr2=0x7fe40419baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.936+0000 7fe402000700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40410b660 0x7fe40419baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.936+0000 7fe402000700 1 -- 172.21.15.27:0/1296350528 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40419c200 con 0x7fe404105f90 2024-05-10T11:07:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.936+0000 7fe402000700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe404105f90 0x7fe4041a1e20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000e9a0 tx=0x7fe3f000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.936+0000 7fe3fb400700 1 -- 172.21.15.27:0/1296350528 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f000ca10 con 0x7fe404105f90 2024-05-10T11:07:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.936+0000 7fe3fb400700 1 -- 172.21.15.27:0/1296350528 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3f000cbb0 con 0x7fe404105f90 2024-05-10T11:07:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.937+0000 7fe403e00700 1 -- 172.21.15.27:0/1296350528 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe40419c4f0 con 0x7fe404105f90 2024-05-10T11:07:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.937+0000 7fe3fb400700 1 -- 172.21.15.27:0/1296350528 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0018680 con 0x7fe404105f90 2024-05-10T11:07:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.937+0000 7fe403e00700 1 -- 172.21.15.27:0/1296350528 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4041a87b0 con 0x7fe404105f90 2024-05-10T11:07:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.938+0000 7fe403e00700 1 -- 172.21.15.27:0/1296350528 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe40406a190 con 0x7fe404105f90 2024-05-10T11:07:49.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.941+0000 7fe3fb400700 1 -- 172.21.15.27:0/1296350528 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe3f0010430 con 0x7fe404105f90 2024-05-10T11:07:49.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.942+0000 7fe3fb400700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe3e80b2900 0x7fe3e80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:49.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.943+0000 7fe3fb400700 1 -- 172.21.15.27:0/1296350528 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7879+0+0 (secure 0 0 0) 0x7fe3f00e2530 con 0x7fe404105f90 2024-05-10T11:07:49.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.943+0000 7fe402a00700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe3e80b2900 0x7fe3e80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:49.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.944+0000 7fe402a00700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe3e80b2900 0x7fe3e80b4dc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009fd0 tx=0x7fe3f4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:49.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:49.944+0000 7fe3fb400700 1 -- 172.21.15.27:0/1296350528 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe3f00aa360 con 0x7fe404105f90 2024-05-10T11:07:50.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.235+0000 7fe403e00700 1 -- 172.21.15.27:0/1296350528 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe40402e510 con 0x7fe404105f90 2024-05-10T11:07:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.298+0000 7fe3fb400700 1 -- 172.21.15.27:0/1296350528 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fe3f00b23e0 con 0x7fe404105f90 2024-05-10T11:07:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.304+0000 7fe403e00700 1 -- 172.21.15.27:0/1296350528 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe3e80b2900 msgr2=0x7fe3e80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.304+0000 7fe403e00700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe3e80b2900 0x7fe3e80b4dc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009fd0 tx=0x7fe3f4019040 comp rx=0 tx=0).stop 2024-05-10T11:07:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.304+0000 7fe403e00700 1 -- 172.21.15.27:0/1296350528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe404105f90 msgr2=0x7fe4041a1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.304+0000 7fe403e00700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe404105f90 0x7fe4041a1e20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000e9a0 tx=0x7fe3f000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:07:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.305+0000 7fe403e00700 1 -- 172.21.15.27:0/1296350528 shutdown_connections 2024-05-10T11:07:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.305+0000 7fe403e00700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe3e80b2900 0x7fe3e80b4dc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.305+0000 7fe403e00700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4041055d0 0x7fe4041a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.305+0000 7fe403e00700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40410b660 0x7fe40419baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.305+0000 7fe403e00700 1 --2- 172.21.15.27:0/1296350528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe404105f90 0x7fe4041a1e20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.305+0000 7fe403e00700 1 -- 172.21.15.27:0/1296350528 >> 172.21.15.27:0/1296350528 conn(0x7fe4041010c0 msgr2=0x7fe404102810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.305+0000 7fe403e00700 1 -- 172.21.15.27:0/1296350528 shutdown_connections 2024-05-10T11:07:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:50.305+0000 7fe403e00700 1 -- 172.21.15.27:0/1296350528 wait complete. 2024-05-10T11:07:50.310 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 2 and data pool 3 2024-05-10T11:07:50.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:50 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1296350528' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-10T11:07:50.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:50 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-10T11:07:50.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:50 smithi027 ceph-mon[30054]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-10T11:07:50.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:50 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1296350528' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-10T11:07:50.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:50 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-10T11:07:50.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:50 smithi076 ceph-mon[34152]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-10T11:07:50.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:50 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1296350528' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-10T11:07:50.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:50 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-10T11:07:50.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:50 smithi189 ceph-mon[34842]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-10T11:07:51.136 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-05-10T11:07:51.137 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-05-10T11:07:51.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:51 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-10T11:07:51.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:51 smithi027 ceph-mon[30054]: osdmap e79: 12 total, 12 up, 12 in 2024-05-10T11:07:51.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:51 smithi027 ceph-mon[30054]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-10T11:07:51.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:51 smithi027 ceph-mon[30054]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-10T11:07:51.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:51 smithi027 ceph-mon[30054]: pgmap v324: 97 pgs: 5 unknown, 12 creating+peering, 80 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 222 B/s wr, 4 op/s 2024-05-10T11:07:51.572 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:07:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:51 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-10T11:07:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:51 smithi076 ceph-mon[34152]: osdmap e79: 12 total, 12 up, 12 in 2024-05-10T11:07:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:51 smithi076 ceph-mon[34152]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-10T11:07:51.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:51 smithi076 ceph-mon[34152]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-10T11:07:51.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:51 smithi076 ceph-mon[34152]: pgmap v324: 97 pgs: 5 unknown, 12 creating+peering, 80 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 222 B/s wr, 4 op/s 2024-05-10T11:07:51.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:51 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-10T11:07:51.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:51 smithi189 ceph-mon[34842]: osdmap e79: 12 total, 12 up, 12 in 2024-05-10T11:07:51.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:51 smithi189 ceph-mon[34842]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-10T11:07:51.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:51 smithi189 ceph-mon[34842]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-10T11:07:51.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:51 smithi189 ceph-mon[34842]: pgmap v324: 97 pgs: 5 unknown, 12 creating+peering, 80 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 222 B/s wr, 4 op/s 2024-05-10T11:07:52.563 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:52 smithi027 ceph-mon[30054]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-10T11:07:52.563 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:52 smithi027 ceph-mon[30054]: Cluster is now healthy 2024-05-10T11:07:52.563 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:52 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] up:active 2024-05-10T11:07:52.563 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:52 smithi027 ceph-mon[30054]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:07:52.563 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:52 smithi027 ceph-mon[30054]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:07:52.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:52 smithi076 ceph-mon[34152]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-10T11:07:52.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:52 smithi076 ceph-mon[34152]: Cluster is now healthy 2024-05-10T11:07:52.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:52 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] up:active 2024-05-10T11:07:52.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:52 smithi076 ceph-mon[34152]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:07:52.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:52 smithi076 ceph-mon[34152]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:07:52.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:52 smithi189 ceph-mon[34842]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-10T11:07:52.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:52 smithi189 ceph-mon[34842]: Cluster is now healthy 2024-05-10T11:07:52.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:52 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] up:active 2024-05-10T11:07:52.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:52 smithi189 ceph-mon[34842]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:07:52.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:52 smithi189 ceph-mon[34842]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:07:53.315 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:07:52 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b[64743]: 2024-05-10T11:07:52.967+0000 7f142e13a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-10T11:07:53.539 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:53 smithi027 ceph-mon[30054]: pgmap v325: 97 pgs: 97 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-10T11:07:53.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:53 smithi076 ceph-mon[34152]: pgmap v325: 97 pgs: 97 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-10T11:07:53.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.724+0000 7fa150600700 1 -- 172.21.15.27:0/3924136778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa1480779d0 msgr2=0x7fa148077db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:53.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.724+0000 7fa150600700 1 --2- 172.21.15.27:0/3924136778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa1480779d0 0x7fa148077db0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009990 tx=0x7fa13c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:07:53.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.724+0000 7fa150600700 1 -- 172.21.15.27:0/3924136778 shutdown_connections 2024-05-10T11:07:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.724+0000 7fa150600700 1 --2- 172.21.15.27:0/3924136778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa148114360 0x7fa148116750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.724+0000 7fa150600700 1 --2- 172.21.15.27:0/3924136778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa1480779d0 0x7fa148077db0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.724+0000 7fa150600700 1 --2- 172.21.15.27:0/3924136778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1480782f0 0x7fa148113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.724+0000 7fa150600700 1 -- 172.21.15.27:0/3924136778 >> 172.21.15.27:0/3924136778 conn(0x7fa1481011b0 msgr2=0x7fa1481035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.724+0000 7fa150600700 1 -- 172.21.15.27:0/3924136778 shutdown_connections 2024-05-10T11:07:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.724+0000 7fa150600700 1 -- 172.21.15.27:0/3924136778 wait complete. 2024-05-10T11:07:53.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.726+0000 7fa150600700 1 Processor -- start 2024-05-10T11:07:53.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.726+0000 7fa150600700 1 -- start start 2024-05-10T11:07:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.727+0000 7fa150600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1480779d0 0x7fa1481a02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.727+0000 7fa150600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa1480782f0 0x7fa1481a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.727+0000 7fa150600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa148114360 0x7fa1481a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.727+0000 7fa150600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa148119d40 con 0x7fa148114360 2024-05-10T11:07:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.727+0000 7fa150600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa148119bc0 con 0x7fa1480782f0 2024-05-10T11:07:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.727+0000 7fa150600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa148119ec0 con 0x7fa1480779d0 2024-05-10T11:07:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.727+0000 7fa14d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa1480782f0 0x7fa1481a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.727+0000 7fa14d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa1480782f0 0x7fa1481a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52876/0 (socket says 172.21.15.27:52876) 2024-05-10T11:07:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.727+0000 7fa14d200700 1 -- 172.21.15.27:0/2601147322 learned_addr learned my addr 172.21.15.27:0/2601147322 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:07:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.727+0000 7fa14e600700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa148114360 0x7fa1481a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.727+0000 7fa14dc00700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1480779d0 0x7fa1481a02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.727+0000 7fa14e600700 1 -- 172.21.15.27:0/2601147322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1480779d0 msgr2=0x7fa1481a02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.728+0000 7fa14e600700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1480779d0 0x7fa1481a02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.728+0000 7fa14e600700 1 -- 172.21.15.27:0/2601147322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa1480782f0 msgr2=0x7fa1481a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.728+0000 7fa14e600700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa1480782f0 0x7fa1481a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.728+0000 7fa14e600700 1 -- 172.21.15.27:0/2601147322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1481a5290 con 0x7fa148114360 2024-05-10T11:07:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.728+0000 7fa14e600700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa148114360 0x7fa1481a4b50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa14807bcc0 tx=0x7fa14000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.728+0000 7fa146000700 1 -- 172.21.15.27:0/2601147322 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1400092e0 con 0x7fa148114360 2024-05-10T11:07:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.728+0000 7fa146000700 1 -- 172.21.15.27:0/2601147322 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa140004bb0 con 0x7fa148114360 2024-05-10T11:07:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.728+0000 7fa146000700 1 -- 172.21.15.27:0/2601147322 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa140004020 con 0x7fa148114360 2024-05-10T11:07:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.728+0000 7fa150600700 1 -- 172.21.15.27:0/2601147322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1481a5520 con 0x7fa148114360 2024-05-10T11:07:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.728+0000 7fa150600700 1 -- 172.21.15.27:0/2601147322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1481ace00 con 0x7fa148114360 2024-05-10T11:07:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.730+0000 7fa150600700 1 -- 172.21.15.27:0/2601147322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa14806a190 con 0x7fa148114360 2024-05-10T11:07:53.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.734+0000 7fa146000700 1 -- 172.21.15.27:0/2601147322 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa14001b030 con 0x7fa148114360 2024-05-10T11:07:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.736+0000 7fa146000700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa1300b2900 0x7fa1300b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.736+0000 7fa146000700 1 -- 172.21.15.27:0/2601147322 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7fa1400e24c0 con 0x7fa148114360 2024-05-10T11:07:53.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.736+0000 7fa14dc00700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa1300b2900 0x7fa1300b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:53.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:53 smithi189 ceph-mon[34842]: pgmap v325: 97 pgs: 97 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-10T11:07:53.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.737+0000 7fa14dc00700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa1300b2900 0x7fa1300b4dc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa13c006010 tx=0x7fa13c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.739+0000 7fa146000700 1 -- 172.21.15.27:0/2601147322 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa140022020 con 0x7fa148114360 2024-05-10T11:07:53.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:53.989+0000 7fa150600700 1 -- 172.21.15.27:0/2601147322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fa14802e510 con 0x7fa148114360 2024-05-10T11:07:54.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.328+0000 7fa146000700 1 -- 172.21.15.27:0/2601147322 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v80) ==== 157+0+0 (secure 0 0 0) 0x7fa140010040 con 0x7fa148114360 2024-05-10T11:07:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.334+0000 7fa150600700 1 -- 172.21.15.27:0/2601147322 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa1300b2900 msgr2=0x7fa1300b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.334+0000 7fa150600700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa1300b2900 0x7fa1300b4dc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa13c006010 tx=0x7fa13c00b260 comp rx=0 tx=0).stop 2024-05-10T11:07:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.334+0000 7fa150600700 1 -- 172.21.15.27:0/2601147322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa148114360 msgr2=0x7fa1481a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.334+0000 7fa150600700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa148114360 0x7fa1481a4b50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa14807bcc0 tx=0x7fa14000ef90 comp rx=0 tx=0).stop 2024-05-10T11:07:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.336+0000 7fa150600700 1 -- 172.21.15.27:0/2601147322 shutdown_connections 2024-05-10T11:07:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.336+0000 7fa150600700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa1300b2900 0x7fa1300b4dc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.336+0000 7fa150600700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1480779d0 0x7fa1481a02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.336+0000 7fa150600700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa148114360 0x7fa1481a4b50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.336+0000 7fa150600700 1 --2- 172.21.15.27:0/2601147322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa1480782f0 0x7fa1481a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.336+0000 7fa150600700 1 -- 172.21.15.27:0/2601147322 >> 172.21.15.27:0/2601147322 conn(0x7fa1481011b0 msgr2=0x7fa1481035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.336+0000 7fa150600700 1 -- 172.21.15.27:0/2601147322 shutdown_connections 2024-05-10T11:07:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:54.336+0000 7fa150600700 1 -- 172.21.15.27:0/2601147322 wait complete. 2024-05-10T11:07:54.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:54 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2601147322' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-10T11:07:54.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:54 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2601147322' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-10T11:07:54.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:54 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2601147322' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-10T11:07:55.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-05-10T11:07:55.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:55 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2601147322' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-10T11:07:55.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:55 smithi027 ceph-mon[30054]: osdmap e80: 12 total, 12 up, 12 in 2024-05-10T11:07:55.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:55 smithi027 ceph-mon[30054]: pgmap v327: 97 pgs: 97 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-10T11:07:55.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:55 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2601147322' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-10T11:07:55.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:55 smithi076 ceph-mon[34152]: osdmap e80: 12 total, 12 up, 12 in 2024-05-10T11:07:55.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:55 smithi076 ceph-mon[34152]: pgmap v327: 97 pgs: 97 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-10T11:07:55.693 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:07:55.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:55 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2601147322' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-10T11:07:55.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:55 smithi189 ceph-mon[34842]: osdmap e80: 12 total, 12 up, 12 in 2024-05-10T11:07:55.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:55 smithi189 ceph-mon[34842]: pgmap v327: 97 pgs: 97 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-10T11:07:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.029+0000 7f6615e00700 1 -- 172.21.15.27:0/1341035107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6610075ff0 msgr2=0x7f661010f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.029+0000 7f6615e00700 1 --2- 172.21.15.27:0/1341035107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6610075ff0 0x7f661010f9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f65f8009990 tx=0x7f65f8009ca0 comp rx=0 tx=0).stop 2024-05-10T11:07:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.029+0000 7f6615e00700 1 -- 172.21.15.27:0/1341035107 shutdown_connections 2024-05-10T11:07:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.029+0000 7f6615e00700 1 --2- 172.21.15.27:0/1341035107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661010ff00 0x7f66101122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.029+0000 7f6615e00700 1 --2- 172.21.15.27:0/1341035107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66100756d0 0x7f6610075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.029+0000 7f6615e00700 1 --2- 172.21.15.27:0/1341035107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6610075ff0 0x7f661010f9c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.029+0000 7f6615e00700 1 -- 172.21.15.27:0/1341035107 >> 172.21.15.27:0/1341035107 conn(0x7f661007a570 msgr2=0x7f661007a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.029+0000 7f6615e00700 1 -- 172.21.15.27:0/1341035107 shutdown_connections 2024-05-10T11:07:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.029+0000 7f6615e00700 1 -- 172.21.15.27:0/1341035107 wait complete. 2024-05-10T11:07:58.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:57 smithi027 ceph-mon[30054]: pgmap v328: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-10T11:07:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.031+0000 7f6615e00700 1 Processor -- start 2024-05-10T11:07:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.031+0000 7f6615e00700 1 -- start start 2024-05-10T11:07:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.031+0000 7f6615e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66100756d0 0x7f66101a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.031+0000 7f6615e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6610075ff0 0x7f66101a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.031+0000 7f6615e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661010ff00 0x7f661019bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.031+0000 7f6615e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6610115860 con 0x7f661010ff00 2024-05-10T11:07:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.031+0000 7f6615e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f66101156e0 con 0x7f66100756d0 2024-05-10T11:07:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.031+0000 7f6615e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66101159e0 con 0x7f6610075ff0 2024-05-10T11:07:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.032+0000 7f660f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661010ff00 0x7f661019bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.032+0000 7f660ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66100756d0 0x7f66101a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.032+0000 7f660f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661010ff00 0x7f661019bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47558/0 (socket says 172.21.15.27:47558) 2024-05-10T11:07:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.032+0000 7f660f400700 1 -- 172.21.15.27:0/3801865174 learned_addr learned my addr 172.21.15.27:0/3801865174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:07:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.032+0000 7f6607e00700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6610075ff0 0x7f66101a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:58.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.032+0000 7f660f400700 1 -- 172.21.15.27:0/3801865174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6610075ff0 msgr2=0x7f66101a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:58.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.032+0000 7f660f400700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6610075ff0 0x7f66101a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:58.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.032+0000 7f660f400700 1 -- 172.21.15.27:0/3801865174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66100756d0 msgr2=0x7f66101a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:58.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.032+0000 7f660f400700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66100756d0 0x7f66101a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:58.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.032+0000 7f660f400700 1 -- 172.21.15.27:0/3801865174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661019c240 con 0x7f661010ff00 2024-05-10T11:07:58.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.032+0000 7f660ea00700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66100756d0 0x7f66101a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:07:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.033+0000 7f660f400700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661010ff00 0x7f661019bae0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f660800c370 tx=0x7f660800c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.033+0000 7f6614a00700 1 -- 172.21.15.27:0/3801865174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66080092e0 con 0x7f661010ff00 2024-05-10T11:07:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.033+0000 7f6614a00700 1 -- 172.21.15.27:0/3801865174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6608003a00 con 0x7f661010ff00 2024-05-10T11:07:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.033+0000 7f6614a00700 1 -- 172.21.15.27:0/3801865174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66080046f0 con 0x7f661010ff00 2024-05-10T11:07:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.033+0000 7f6615e00700 1 -- 172.21.15.27:0/3801865174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f661019c4d0 con 0x7f661010ff00 2024-05-10T11:07:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.033+0000 7f6615e00700 1 -- 172.21.15.27:0/3801865174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66101a87f0 con 0x7f661010ff00 2024-05-10T11:07:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.035+0000 7f6615e00700 1 -- 172.21.15.27:0/3801865174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66100519e0 con 0x7f661010ff00 2024-05-10T11:07:58.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.037+0000 7f6614a00700 1 -- 172.21.15.27:0/3801865174 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f660801a020 con 0x7f661010ff00 2024-05-10T11:07:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.039+0000 7f6614a00700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f65f40b29b0 0x7f65f40b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:07:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.039+0000 7f6614a00700 1 -- 172.21.15.27:0/3801865174 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8277+0+0 (secure 0 0 0) 0x7f66080e20a0 con 0x7f661010ff00 2024-05-10T11:07:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.039+0000 7f660ea00700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f65f40b29b0 0x7f65f40b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:07:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.040+0000 7f660ea00700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f65f40b29b0 0x7f65f40b4e70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6600011490 tx=0x7f6600009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:07:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.041+0000 7f6614a00700 1 -- 172.21.15.27:0/3801865174 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6608010040 con 0x7f661010ff00 2024-05-10T11:07:58.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:57 smithi076 ceph-mon[34152]: pgmap v328: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-10T11:07:58.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:57 smithi189 ceph-mon[34842]: pgmap v328: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-10T11:07:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:58.287+0000 7f6615e00700 1 -- 172.21.15.27:0/3801865174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6610109a20 con 0x7f661010ff00 2024-05-10T11:07:59.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:58 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3801865174' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-10T11:07:59.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:58 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3801865174' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-10T11:07:59.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:58 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3801865174' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-10T11:07:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.848+0000 7f6614a00700 1 -- 172.21.15.27:0/3801865174 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v82) ==== 228+0+0 (secure 0 0 0) 0x7f66080a9d30 con 0x7f661010ff00 2024-05-10T11:07:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.856+0000 7f6615e00700 1 -- 172.21.15.27:0/3801865174 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f65f40b29b0 msgr2=0x7f65f40b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.857+0000 7f6615e00700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f65f40b29b0 0x7f65f40b4e70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6600011490 tx=0x7f6600009260 comp rx=0 tx=0).stop 2024-05-10T11:07:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.857+0000 7f6615e00700 1 -- 172.21.15.27:0/3801865174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661010ff00 msgr2=0x7f661019bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:07:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.857+0000 7f6615e00700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661010ff00 0x7f661019bae0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f660800c370 tx=0x7f660800c680 comp rx=0 tx=0).stop 2024-05-10T11:07:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.857+0000 7f6615e00700 1 -- 172.21.15.27:0/3801865174 shutdown_connections 2024-05-10T11:07:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.857+0000 7f6615e00700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f65f40b29b0 0x7f65f40b4e70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.857+0000 7f6615e00700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661010ff00 0x7f661019bae0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:59.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.857+0000 7f6615e00700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66100756d0 0x7f66101a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:59.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.857+0000 7f6615e00700 1 --2- 172.21.15.27:0/3801865174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6610075ff0 0x7f66101a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:07:59.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.857+0000 7f6615e00700 1 -- 172.21.15.27:0/3801865174 >> 172.21.15.27:0/3801865174 conn(0x7f661007a570 msgr2=0x7f66101006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:07:59.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.858+0000 7f6615e00700 1 -- 172.21.15.27:0/3801865174 shutdown_connections 2024-05-10T11:07:59.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:07:59.859+0000 7f6615e00700 1 -- 172.21.15.27:0/3801865174 wait complete. 2024-05-10T11:07:59.864 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data_ec' created 2024-05-10T11:08:00.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:59 smithi027 ceph-mon[30054]: pgmap v329: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-10T11:08:00.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:07:59 smithi027 ceph-mon[30054]: osdmap e81: 12 total, 12 up, 12 in 2024-05-10T11:08:00.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:59 smithi076 ceph-mon[34152]: pgmap v329: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-10T11:08:00.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:07:59 smithi076 ceph-mon[34152]: osdmap e81: 12 total, 12 up, 12 in 2024-05-10T11:08:00.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:59 smithi189 ceph-mon[34842]: pgmap v329: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-10T11:08:00.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:07:59 smithi189 ceph-mon[34842]: osdmap e81: 12 total, 12 up, 12 in 2024-05-10T11:08:00.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-05-10T11:08:01.170 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:08:01.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:00 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3801865174' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-10T11:08:01.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:00 smithi076 ceph-mon[34152]: osdmap e82: 12 total, 12 up, 12 in 2024-05-10T11:08:01.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:00 smithi076 ceph-mon[34152]: osdmap e83: 12 total, 12 up, 12 in 2024-05-10T11:08:01.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:00 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3801865174' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-10T11:08:01.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:00 smithi189 ceph-mon[34842]: osdmap e82: 12 total, 12 up, 12 in 2024-05-10T11:08:01.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:00 smithi189 ceph-mon[34842]: osdmap e83: 12 total, 12 up, 12 in 2024-05-10T11:08:01.320 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:00 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3801865174' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-10T11:08:01.320 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:00 smithi027 ceph-mon[30054]: osdmap e82: 12 total, 12 up, 12 in 2024-05-10T11:08:01.321 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:00 smithi027 ceph-mon[30054]: osdmap e83: 12 total, 12 up, 12 in 2024-05-10T11:08:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:01 smithi027 ceph-mon[30054]: pgmap v332: 129 pgs: 6 creating+peering, 26 unknown, 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 476 B/s wr, 0 op/s 2024-05-10T11:08:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:01 smithi027 ceph-mon[30054]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-10T11:08:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:01 smithi027 ceph-mon[30054]: osdmap e84: 12 total, 12 up, 12 in 2024-05-10T11:08:02.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:01 smithi076 ceph-mon[34152]: pgmap v332: 129 pgs: 6 creating+peering, 26 unknown, 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 476 B/s wr, 0 op/s 2024-05-10T11:08:02.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:01 smithi076 ceph-mon[34152]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-10T11:08:02.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:01 smithi076 ceph-mon[34152]: osdmap e84: 12 total, 12 up, 12 in 2024-05-10T11:08:02.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:01 smithi189 ceph-mon[34842]: pgmap v332: 129 pgs: 6 creating+peering, 26 unknown, 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 476 B/s wr, 0 op/s 2024-05-10T11:08:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:01 smithi189 ceph-mon[34842]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-10T11:08:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:01 smithi189 ceph-mon[34842]: osdmap e84: 12 total, 12 up, 12 in 2024-05-10T11:08:03.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.665+0000 7f236ea00700 1 -- 172.21.15.27:0/48502082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2368108700 msgr2=0x7f23681102d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:03.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.665+0000 7f236ea00700 1 --2- 172.21.15.27:0/48502082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2368108700 0x7f23681102d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f236000b0d0 tx=0x7f236000b3e0 comp rx=0 tx=0).stop 2024-05-10T11:08:03.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.665+0000 7f236ea00700 1 -- 172.21.15.27:0/48502082 shutdown_connections 2024-05-10T11:08:03.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.665+0000 7f236ea00700 1 --2- 172.21.15.27:0/48502082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2368108700 0x7f23681102d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:03.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.665+0000 7f236ea00700 1 --2- 172.21.15.27:0/48502082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368103620 0x7f2368103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:03.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.665+0000 7f236ea00700 1 --2- 172.21.15.27:0/48502082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2368103fd0 0x7f2368108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:03.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.665+0000 7f236ea00700 1 -- 172.21.15.27:0/48502082 >> 172.21.15.27:0/48502082 conn(0x7f23680feef0 msgr2=0x7f2368101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:03.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.665+0000 7f236ea00700 1 -- 172.21.15.27:0/48502082 shutdown_connections 2024-05-10T11:08:03.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.665+0000 7f236ea00700 1 -- 172.21.15.27:0/48502082 wait complete. 2024-05-10T11:08:03.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.667+0000 7f236ea00700 1 Processor -- start 2024-05-10T11:08:03.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.667+0000 7f236ea00700 1 -- start start 2024-05-10T11:08:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.667+0000 7f236ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368103620 0x7f236819bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.667+0000 7f236ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2368103fd0 0x7f236819c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.667+0000 7f236ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2368108700 0x7f23681a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.667+0000 7f236ea00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2368113610 con 0x7f2368108700 2024-05-10T11:08:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.667+0000 7f236ea00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2368113490 con 0x7f2368103620 2024-05-10T11:08:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.667+0000 7f236ea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2368113790 con 0x7f2368103fd0 2024-05-10T11:08:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.668+0000 7f236d600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368103620 0x7f236819bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.668+0000 7f236cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2368103fd0 0x7f236819c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.668+0000 7f236d600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368103620 0x7f236819bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:49788/0 (socket says 172.21.15.27:49788) 2024-05-10T11:08:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.668+0000 7f236d600700 1 -- 172.21.15.27:0/1638761939 learned_addr learned my addr 172.21.15.27:0/1638761939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.668+0000 7f236e000700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2368108700 0x7f23681a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.668+0000 7f236d600700 1 -- 172.21.15.27:0/1638761939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2368103fd0 msgr2=0x7f236819c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.668+0000 7f236d600700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2368103fd0 0x7f236819c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.668+0000 7f236d600700 1 -- 172.21.15.27:0/1638761939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2368108700 msgr2=0x7f23681a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.668+0000 7f236d600700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2368108700 0x7f23681a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.668+0000 7f236d600700 1 -- 172.21.15.27:0/1638761939 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23681a0d90 con 0x7f2368103620 2024-05-10T11:08:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.669+0000 7f236d600700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368103620 0x7f236819bd80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f235c00e9a0 tx=0x7f235c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.669+0000 7f2366000700 1 -- 172.21.15.27:0/1638761939 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c00ca10 con 0x7f2368103620 2024-05-10T11:08:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.669+0000 7f2366000700 1 -- 172.21.15.27:0/1638761939 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f235c00cbb0 con 0x7f2368103620 2024-05-10T11:08:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.669+0000 7f2366000700 1 -- 172.21.15.27:0/1638761939 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c018680 con 0x7f2368103620 2024-05-10T11:08:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.669+0000 7f236ea00700 1 -- 172.21.15.27:0/1638761939 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23681a1080 con 0x7f2368103620 2024-05-10T11:08:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.669+0000 7f236ea00700 1 -- 172.21.15.27:0/1638761939 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23681a8900 con 0x7f2368103620 2024-05-10T11:08:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.670+0000 7f236ea00700 1 -- 172.21.15.27:0/1638761939 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f236806a190 con 0x7f2368103620 2024-05-10T11:08:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.673+0000 7f2366000700 1 -- 172.21.15.27:0/1638761939 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f235c010430 con 0x7f2368103620 2024-05-10T11:08:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.674+0000 7f2366000700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f23500b2880 0x7f23500b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.675+0000 7f2366000700 1 -- 172.21.15.27:0/1638761939 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8770+0+0 (secure 0 0 0) 0x7f235c0e2940 con 0x7f2368103620 2024-05-10T11:08:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.675+0000 7f236cc00700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f23500b2880 0x7f23500b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.675+0000 7f236cc00700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f23500b2880 0x7f23500b4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f23681018c0 tx=0x7f2354009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.676+0000 7f2366000700 1 -- 172.21.15.27:0/1638761939 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f235c0aa3e0 con 0x7f2368103620 2024-05-10T11:08:03.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:03 smithi027 ceph-mon[30054]: pgmap v335: 129 pgs: 6 creating+peering, 4 unknown, 119 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:03.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:03.923+0000 7f236ea00700 1 -- 172.21.15.27:0/1638761939 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f23681a8ad0 con 0x7f2368103620 2024-05-10T11:08:04.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:03 smithi076 ceph-mon[34152]: pgmap v335: 129 pgs: 6 creating+peering, 4 unknown, 119 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:04.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:03 smithi189 ceph-mon[34842]: pgmap v335: 129 pgs: 6 creating+peering, 4 unknown, 119 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.908+0000 7f2366000700 1 -- 172.21.15.27:0/1638761939 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v85) ==== 169+0+0 (secure 0 0 0) 0x7f235c0a9ce0 con 0x7f2368103620 2024-05-10T11:08:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.912+0000 7f236ea00700 1 -- 172.21.15.27:0/1638761939 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f23500b2880 msgr2=0x7f23500b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.912+0000 7f236ea00700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f23500b2880 0x7f23500b4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f23681018c0 tx=0x7f2354009260 comp rx=0 tx=0).stop 2024-05-10T11:08:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.912+0000 7f236ea00700 1 -- 172.21.15.27:0/1638761939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368103620 msgr2=0x7f236819bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.912+0000 7f236ea00700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368103620 0x7f236819bd80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f235c00e9a0 tx=0x7f235c00ecb0 comp rx=0 tx=0).stop 2024-05-10T11:08:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.913+0000 7f236ea00700 1 -- 172.21.15.27:0/1638761939 shutdown_connections 2024-05-10T11:08:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.913+0000 7f236ea00700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f23500b2880 0x7f23500b4d40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.913+0000 7f236ea00700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2368108700 0x7f23681a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.913+0000 7f236ea00700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368103620 0x7f236819bd80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.913+0000 7f236ea00700 1 --2- 172.21.15.27:0/1638761939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2368103fd0 0x7f236819c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.913+0000 7f236ea00700 1 -- 172.21.15.27:0/1638761939 >> 172.21.15.27:0/1638761939 conn(0x7f23680feef0 msgr2=0x7f236810eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.913+0000 7f236ea00700 1 -- 172.21.15.27:0/1638761939 shutdown_connections 2024-05-10T11:08:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:04.913+0000 7f236ea00700 1 -- 172.21.15.27:0/1638761939 wait complete. 2024-05-10T11:08:04.915 INFO:teuthology.orchestra.run.smithi027.stderr:set pool 4 allow_ec_overwrites to true 2024-05-10T11:08:05.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:04 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1638761939' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-10T11:08:05.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:04 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-10T11:08:05.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:04 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1638761939' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-10T11:08:05.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:04 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-10T11:08:05.330 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:04 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1638761939' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-10T11:08:05.330 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:04 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-10T11:08:05.780 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-05-10T11:08:06.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:05 smithi027 ceph-mon[30054]: pgmap v336: 129 pgs: 6 creating+peering, 1 unknown, 122 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:06.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:05 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-10T11:08:06.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:05 smithi027 ceph-mon[30054]: osdmap e85: 12 total, 12 up, 12 in 2024-05-10T11:08:06.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:05 smithi076 ceph-mon[34152]: pgmap v336: 129 pgs: 6 creating+peering, 1 unknown, 122 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:06.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:05 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-10T11:08:06.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:05 smithi076 ceph-mon[34152]: osdmap e85: 12 total, 12 up, 12 in 2024-05-10T11:08:06.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:05 smithi189 ceph-mon[34842]: pgmap v336: 129 pgs: 6 creating+peering, 1 unknown, 122 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:06.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:05 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-10T11:08:06.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:05 smithi189 ceph-mon[34842]: osdmap e85: 12 total, 12 up, 12 in 2024-05-10T11:08:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:08:08.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:07 smithi027 ceph-mon[30054]: pgmap v338: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:08.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:07 smithi076 ceph-mon[34152]: pgmap v338: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:08.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:07 smithi189 ceph-mon[34842]: pgmap v338: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:08.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.362+0000 7fdd86800700 1 -- 172.21.15.27:0/3546665453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd8006c360 msgr2=0x7fdd8010fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:08.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.362+0000 7fdd86800700 1 --2- 172.21.15.27:0/3546665453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd8006c360 0x7fdd8010fa00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800b0a0 tx=0x7fdd6800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:08:08.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.362+0000 7fdd86800700 1 -- 172.21.15.27:0/3546665453 shutdown_connections 2024-05-10T11:08:08.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.362+0000 7fdd86800700 1 --2- 172.21.15.27:0/3546665453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8010ff40 0x7fdd80112330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:08.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.362+0000 7fdd86800700 1 --2- 172.21.15.27:0/3546665453 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd8006ba40 0x7fdd8006be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:08.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.362+0000 7fdd86800700 1 --2- 172.21.15.27:0/3546665453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd8006c360 0x7fdd8010fa00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:08.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.362+0000 7fdd86800700 1 -- 172.21.15.27:0/3546665453 >> 172.21.15.27:0/3546665453 conn(0x7fdd80079310 msgr2=0x7fdd80079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:08.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.362+0000 7fdd86800700 1 -- 172.21.15.27:0/3546665453 shutdown_connections 2024-05-10T11:08:08.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.362+0000 7fdd86800700 1 -- 172.21.15.27:0/3546665453 wait complete. 2024-05-10T11:08:08.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.363+0000 7fdd86800700 1 Processor -- start 2024-05-10T11:08:08.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.363+0000 7fdd86800700 1 -- start start 2024-05-10T11:08:08.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.364+0000 7fdd86800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd8006ba40 0x7fdd8019bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:08.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.364+0000 7fdd86800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd8006c360 0x7fdd8019c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:08.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.364+0000 7fdd86800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8010ff40 0x7fdd801a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:08.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.364+0000 7fdd86800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdd801158d0 con 0x7fdd8006ba40 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.364+0000 7fdd86800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdd80115750 con 0x7fdd8006c360 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.364+0000 7fdd86800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdd80115a50 con 0x7fdd8010ff40 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.364+0000 7fdd7f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd8006ba40 0x7fdd8019bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.364+0000 7fdd7f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd8006ba40 0x7fdd8019bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53524/0 (socket says 172.21.15.27:53524) 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.364+0000 7fdd7f400700 1 -- 172.21.15.27:0/2010963249 learned_addr learned my addr 172.21.15.27:0/2010963249 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.364+0000 7fdd76a00700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd8006c360 0x7fdd8019c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.364+0000 7fdd7fe00700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8010ff40 0x7fdd801a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.365+0000 7fdd7f400700 1 -- 172.21.15.27:0/2010963249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8010ff40 msgr2=0x7fdd801a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.365+0000 7fdd7f400700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8010ff40 0x7fdd801a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.365+0000 7fdd7f400700 1 -- 172.21.15.27:0/2010963249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd8006c360 msgr2=0x7fdd8019c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.365+0000 7fdd7f400700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd8006c360 0x7fdd8019c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.365+0000 7fdd7f400700 1 -- 172.21.15.27:0/2010963249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd801a0d90 con 0x7fdd8006ba40 2024-05-10T11:08:08.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.365+0000 7fdd76a00700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd8006c360 0x7fdd8019c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:08:08.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.365+0000 7fdd7f400700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd8006ba40 0x7fdd8019bd80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000e9a0 tx=0x7fdd7000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:08.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.365+0000 7fdd7cc00700 1 -- 172.21.15.27:0/2010963249 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7000ca10 con 0x7fdd8006ba40 2024-05-10T11:08:08.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.365+0000 7fdd7cc00700 1 -- 172.21.15.27:0/2010963249 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd7000cbb0 con 0x7fdd8006ba40 2024-05-10T11:08:08.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.365+0000 7fdd86800700 1 -- 172.21.15.27:0/2010963249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd801a1080 con 0x7fdd8006ba40 2024-05-10T11:08:08.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.366+0000 7fdd7cc00700 1 -- 172.21.15.27:0/2010963249 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd70018680 con 0x7fdd8006ba40 2024-05-10T11:08:08.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.366+0000 7fdd86800700 1 -- 172.21.15.27:0/2010963249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd801a8900 con 0x7fdd8006ba40 2024-05-10T11:08:08.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.367+0000 7fdd86800700 1 -- 172.21.15.27:0/2010963249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd8006a190 con 0x7fdd8006ba40 2024-05-10T11:08:08.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.369+0000 7fdd7cc00700 1 -- 172.21.15.27:0/2010963249 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fdd70010430 con 0x7fdd8006ba40 2024-05-10T11:08:08.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.371+0000 7fdd7cc00700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdd640b2900 0x7fdd640b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:08.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.371+0000 7fdd7cc00700 1 -- 172.21.15.27:0/2010963249 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8770+0+0 (secure 0 0 0) 0x7fdd700e28c0 con 0x7fdd8006ba40 2024-05-10T11:08:08.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.371+0000 7fdd76a00700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdd640b2900 0x7fdd640b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:08.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.372+0000 7fdd76a00700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdd640b2900 0x7fdd640b4dc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdd680062a0 tx=0x7fdd68006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.374+0000 7fdd7cc00700 1 -- 172.21.15.27:0/2010963249 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fdd700aa360 con 0x7fdd8006ba40 2024-05-10T11:08:08.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.667+0000 7fdd86800700 1 -- 172.21.15.27:0/2010963249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fdd8002e510 con 0x7fdd8006ba40 2024-05-10T11:08:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.928+0000 7fdd7cc00700 1 -- 172.21.15.27:0/2010963249 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7fdd700b23e0 con 0x7fdd8006ba40 2024-05-10T11:08:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.932+0000 7fdd86800700 1 -- 172.21.15.27:0/2010963249 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdd640b2900 msgr2=0x7fdd640b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.932+0000 7fdd86800700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdd640b2900 0x7fdd640b4dc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdd680062a0 tx=0x7fdd68006210 comp rx=0 tx=0).stop 2024-05-10T11:08:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.932+0000 7fdd86800700 1 -- 172.21.15.27:0/2010963249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd8006ba40 msgr2=0x7fdd8019bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.932+0000 7fdd86800700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd8006ba40 0x7fdd8019bd80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000e9a0 tx=0x7fdd7000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:08:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.933+0000 7fdd86800700 1 -- 172.21.15.27:0/2010963249 shutdown_connections 2024-05-10T11:08:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.933+0000 7fdd86800700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdd640b2900 0x7fdd640b4dc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.933+0000 7fdd86800700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8010ff40 0x7fdd801a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.933+0000 7fdd86800700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd8006ba40 0x7fdd8019bd80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.933+0000 7fdd86800700 1 --2- 172.21.15.27:0/2010963249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd8006c360 0x7fdd8019c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.933+0000 7fdd86800700 1 -- 172.21.15.27:0/2010963249 >> 172.21.15.27:0/2010963249 conn(0x7fdd80079310 msgr2=0x7fdd801113b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.933+0000 7fdd86800700 1 -- 172.21.15.27:0/2010963249 shutdown_connections 2024-05-10T11:08:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:08.933+0000 7fdd86800700 1 -- 172.21.15.27:0/2010963249 wait complete. 2024-05-10T11:08:08.936 INFO:teuthology.orchestra.run.smithi027.stderr:added data pool 4 to fsmap 2024-05-10T11:08:09.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:08 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2010963249' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-10T11:08:09.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:08 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2010963249' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-10T11:08:09.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:08 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2010963249' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-10T11:08:09.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:08:10.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:09 smithi027 ceph-mon[30054]: pgmap v339: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:10.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:09 smithi027 ceph-mon[30054]: osdmap e86: 12 total, 12 up, 12 in 2024-05-10T11:08:10.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:09 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2010963249' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-10T11:08:10.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:09 smithi027 ceph-mon[30054]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:08:10.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:09 smithi189 ceph-mon[34842]: pgmap v339: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:09 smithi189 ceph-mon[34842]: osdmap e86: 12 total, 12 up, 12 in 2024-05-10T11:08:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:09 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2010963249' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-10T11:08:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:09 smithi189 ceph-mon[34842]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:08:10.411 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:08:10.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:09 smithi076 ceph-mon[34152]: pgmap v339: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:10.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:09 smithi076 ceph-mon[34152]: osdmap e86: 12 total, 12 up, 12 in 2024-05-10T11:08:10.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:09 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2010963249' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-10T11:08:10.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:09 smithi076 ceph-mon[34152]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:08:11.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:10 smithi027 ceph-mon[30054]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-10T11:08:11.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:10 smithi027 ceph-mon[30054]: Cluster is now healthy 2024-05-10T11:08:11.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:10 smithi076 ceph-mon[34152]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-10T11:08:11.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:10 smithi076 ceph-mon[34152]: Cluster is now healthy 2024-05-10T11:08:11.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:10 smithi189 ceph-mon[34842]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-10T11:08:11.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:10 smithi189 ceph-mon[34842]: Cluster is now healthy 2024-05-10T11:08:12.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:11 smithi027 ceph-mon[30054]: pgmap v341: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:12.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:11 smithi076 ceph-mon[34152]: pgmap v341: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:12.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:11 smithi189 ceph-mon[34842]: pgmap v341: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:12.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.699+0000 7fb74f200700 1 -- 172.21.15.27:0/979541007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb750104590 msgr2=0x7fb750104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:12.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.699+0000 7fb74f200700 1 --2- 172.21.15.27:0/979541007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb750104590 0x7fb750104970 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb74000b0a0 tx=0x7fb74000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:08:12.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.699+0000 7fb74f200700 1 -- 172.21.15.27:0/979541007 shutdown_connections 2024-05-10T11:08:12.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.699+0000 7fb74f200700 1 --2- 172.21.15.27:0/979541007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb75010d9c0 0x7fb75010fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:12.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.699+0000 7fb74f200700 1 --2- 172.21.15.27:0/979541007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb750104590 0x7fb750104970 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:12.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.699+0000 7fb74f200700 1 --2- 172.21.15.27:0/979541007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb750104eb0 0x7fb75010d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:12.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.699+0000 7fb74f200700 1 -- 172.21.15.27:0/979541007 >> 172.21.15.27:0/979541007 conn(0x7fb7500fddc0 msgr2=0x7fb7501001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:12.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.699+0000 7fb74f200700 1 -- 172.21.15.27:0/979541007 shutdown_connections 2024-05-10T11:08:12.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.699+0000 7fb74f200700 1 -- 172.21.15.27:0/979541007 wait complete. 2024-05-10T11:08:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.700+0000 7fb74f200700 1 Processor -- start 2024-05-10T11:08:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.700+0000 7fb74f200700 1 -- start start 2024-05-10T11:08:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74f200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb750104590 0x7fb75019bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb750104eb0 0x7fb75019c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74f200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb75010d9c0 0x7fb7501a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74f200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb750113620 con 0x7fb75010d9c0 2024-05-10T11:08:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74f200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb7501134a0 con 0x7fb750104590 2024-05-10T11:08:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb7501137a0 con 0x7fb750104eb0 2024-05-10T11:08:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74de00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb750104590 0x7fb75019bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74de00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb750104590 0x7fb75019bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55016/0 (socket says 172.21.15.27:55016) 2024-05-10T11:08:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74de00700 1 -- 172.21.15.27:0/4001626337 learned_addr learned my addr 172.21.15.27:0/4001626337 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74e800700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb75010d9c0 0x7fb7501a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb745400700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb750104eb0 0x7fb75019c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74de00700 1 -- 172.21.15.27:0/4001626337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb750104eb0 msgr2=0x7fb75019c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74de00700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb750104eb0 0x7fb75019c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:12.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74de00700 1 -- 172.21.15.27:0/4001626337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb75010d9c0 msgr2=0x7fb7501a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:12.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74de00700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb75010d9c0 0x7fb7501a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:12.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74de00700 1 -- 172.21.15.27:0/4001626337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7501a0d20 con 0x7fb750104590 2024-05-10T11:08:12.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb745400700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb750104eb0 0x7fb75019c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:08:12.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.701+0000 7fb74e800700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb75010d9c0 0x7fb7501a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:08:12.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.702+0000 7fb74de00700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb750104590 0x7fb75019bd10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb74000d010 tx=0x7fb740004db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:12.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.702+0000 7fb747400700 1 -- 172.21.15.27:0/4001626337 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb740021070 con 0x7fb750104590 2024-05-10T11:08:12.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.702+0000 7fb747400700 1 -- 172.21.15.27:0/4001626337 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb740007cc0 con 0x7fb750104590 2024-05-10T11:08:12.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.702+0000 7fb747400700 1 -- 172.21.15.27:0/4001626337 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7400047b0 con 0x7fb750104590 2024-05-10T11:08:12.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.702+0000 7fb74f200700 1 -- 172.21.15.27:0/4001626337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7501a0fb0 con 0x7fb750104590 2024-05-10T11:08:12.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.702+0000 7fb74f200700 1 -- 172.21.15.27:0/4001626337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7501a8890 con 0x7fb750104590 2024-05-10T11:08:12.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.703+0000 7fb74f200700 1 -- 172.21.15.27:0/4001626337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb75006a190 con 0x7fb750104590 2024-05-10T11:08:12.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.706+0000 7fb747400700 1 -- 172.21.15.27:0/4001626337 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fb740003a00 con 0x7fb750104590 2024-05-10T11:08:12.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.707+0000 7fb747400700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb7340b2900 0x7fb7340b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:12.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.707+0000 7fb747400700 1 -- 172.21.15.27:0/4001626337 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fb74000f020 con 0x7fb750104590 2024-05-10T11:08:12.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.707+0000 7fb745400700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb7340b2900 0x7fb7340b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:12.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.708+0000 7fb745400700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb7340b2900 0x7fb7340b4dc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb75010f2b0 tx=0x7fb738009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:12.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:12.709+0000 7fb747400700 1 -- 172.21.15.27:0/4001626337 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb74002a020 con 0x7fb750104590 2024-05-10T11:08:13.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.009+0000 7fb74f200700 1 -- 172.21.15.27:0/4001626337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7501a8a60 con 0x7fb750104590 2024-05-10T11:08:13.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.010+0000 7fb747400700 1 -- 172.21.15.27:0/4001626337 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11800 (secure 0 0 0) 0x7fb7400ab810 con 0x7fb750104590 2024-05-10T11:08:13.011 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:08:13.012 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:08:08.670053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24527,"qdb_cluster":[24527]},"id":1}]} 2024-05-10T11:08:13.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.014+0000 7fb74f200700 1 -- 172.21.15.27:0/4001626337 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb7340b2900 msgr2=0x7fb7340b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:13.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.014+0000 7fb74f200700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb7340b2900 0x7fb7340b4dc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb75010f2b0 tx=0x7fb738009260 comp rx=0 tx=0).stop 2024-05-10T11:08:13.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.014+0000 7fb74f200700 1 -- 172.21.15.27:0/4001626337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb750104590 msgr2=0x7fb75019bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:13.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.014+0000 7fb74f200700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb750104590 0x7fb75019bd10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb74000d010 tx=0x7fb740004db0 comp rx=0 tx=0).stop 2024-05-10T11:08:13.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.014+0000 7fb74f200700 1 -- 172.21.15.27:0/4001626337 shutdown_connections 2024-05-10T11:08:13.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.014+0000 7fb74f200700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb7340b2900 0x7fb7340b4dc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:13.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.014+0000 7fb74f200700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb75010d9c0 0x7fb7501a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:13.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.014+0000 7fb74f200700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb750104590 0x7fb75019bd10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.014+0000 7fb74f200700 1 --2- 172.21.15.27:0/4001626337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb750104eb0 0x7fb75019c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.014+0000 7fb74f200700 1 -- 172.21.15.27:0/4001626337 >> 172.21.15.27:0/4001626337 conn(0x7fb7500fddc0 msgr2=0x7fb7501001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.014+0000 7fb74f200700 1 -- 172.21.15.27:0/4001626337 shutdown_connections 2024-05-10T11:08:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:13.014+0000 7fb74f200700 1 -- 172.21.15.27:0/4001626337 wait complete. 2024-05-10T11:08:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-05-10T11:08:13.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:08:14.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:13 smithi027 ceph-mon[30054]: pgmap v342: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:14.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:13 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4001626337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:08:14.341 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:08:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:13 smithi076 ceph-mon[34152]: pgmap v342: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:13 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4001626337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:08:14.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:13 smithi189 ceph-mon[34842]: pgmap v342: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:14.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:13 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4001626337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:08:16.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:15 smithi027 ceph-mon[30054]: pgmap v343: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:16.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:15 smithi076 ceph-mon[34152]: pgmap v343: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:16.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:15 smithi189 ceph-mon[34842]: pgmap v343: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.577+0000 7f87f8e00700 1 -- 172.21.15.27:0/2339838835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f4106080 msgr2=0x7f87f4112590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.577+0000 7f87f8e00700 1 --2- 172.21.15.27:0/2339838835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f4106080 0x7f87f4112590 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00b0a0 tx=0x7f87ec00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:08:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.577+0000 7f87f8e00700 1 -- 172.21.15.27:0/2339838835 shutdown_connections 2024-05-10T11:08:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.577+0000 7f87f8e00700 1 --2- 172.21.15.27:0/2339838835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87f410b6c0 0x7f87f410baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.577+0000 7f87f8e00700 1 --2- 172.21.15.27:0/2339838835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f4106080 0x7f87f4112590 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.577+0000 7f87f8e00700 1 --2- 172.21.15.27:0/2339838835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87f41056c0 0x7f87f4105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.577+0000 7f87f8e00700 1 -- 172.21.15.27:0/2339838835 >> 172.21.15.27:0/2339838835 conn(0x7f87f41011b0 msgr2=0x7f87f41035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.578+0000 7f87f8e00700 1 -- 172.21.15.27:0/2339838835 shutdown_connections 2024-05-10T11:08:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.578+0000 7f87f8e00700 1 -- 172.21.15.27:0/2339838835 wait complete. 2024-05-10T11:08:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.579+0000 7f87f8e00700 1 Processor -- start 2024-05-10T11:08:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.579+0000 7f87f8e00700 1 -- start start 2024-05-10T11:08:16.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.579+0000 7f87f8e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87f41056c0 0x7f87f419bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:16.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.579+0000 7f87f8e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f4106080 0x7f87f419c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:16.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.579+0000 7f87f8e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87f410b6c0 0x7f87f41a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.579+0000 7f87f8e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f87f4115950 con 0x7f87f4106080 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.579+0000 7f87f8e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f87f41157d0 con 0x7f87f410b6c0 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.579+0000 7f87f8e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87f4115ad0 con 0x7f87f41056c0 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.580+0000 7f87f1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87f41056c0 0x7f87f419bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.580+0000 7f87f1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f4106080 0x7f87f419c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.580+0000 7f87f1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f4106080 0x7f87f419c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60210/0 (socket says 172.21.15.27:60210) 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.580+0000 7f87f1400700 1 -- 172.21.15.27:0/4114696833 learned_addr learned my addr 172.21.15.27:0/4114696833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.580+0000 7f87f1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87f41056c0 0x7f87f419bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36368/0 (socket says 172.21.15.27:36368) 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.580+0000 7f87f2800700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87f410b6c0 0x7f87f41a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.580+0000 7f87f1400700 1 -- 172.21.15.27:0/4114696833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87f41056c0 msgr2=0x7f87f419bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.580+0000 7f87f1400700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87f41056c0 0x7f87f419bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.580+0000 7f87f1400700 1 -- 172.21.15.27:0/4114696833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87f410b6c0 msgr2=0x7f87f41a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.580+0000 7f87f1400700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87f410b6c0 0x7f87f41a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.580+0000 7f87f1400700 1 -- 172.21.15.27:0/4114696833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f41a0dd0 con 0x7f87f4106080 2024-05-10T11:08:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.581+0000 7f87f1400700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f4106080 0x7f87f419c300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f87e000e9a0 tx=0x7f87e000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.581+0000 7f87eaa00700 1 -- 172.21.15.27:0/4114696833 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e000ca10 con 0x7f87f4106080 2024-05-10T11:08:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.581+0000 7f87eaa00700 1 -- 172.21.15.27:0/4114696833 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87e000cbb0 con 0x7f87f4106080 2024-05-10T11:08:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.581+0000 7f87eaa00700 1 -- 172.21.15.27:0/4114696833 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e0018680 con 0x7f87f4106080 2024-05-10T11:08:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.581+0000 7f87f8e00700 1 -- 172.21.15.27:0/4114696833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87f41a10c0 con 0x7f87f4106080 2024-05-10T11:08:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.581+0000 7f87f8e00700 1 -- 172.21.15.27:0/4114696833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87f41a8940 con 0x7f87f4106080 2024-05-10T11:08:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.582+0000 7f87f8e00700 1 -- 172.21.15.27:0/4114696833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87f406a190 con 0x7f87f4106080 2024-05-10T11:08:16.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.586+0000 7f87eaa00700 1 -- 172.21.15.27:0/4114696833 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f87e0010430 con 0x7f87f4106080 2024-05-10T11:08:16.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.587+0000 7f87eaa00700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f87d80b2900 0x7f87d80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:16.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.587+0000 7f87eaa00700 1 -- 172.21.15.27:0/4114696833 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f87e00e2950 con 0x7f87f4106080 2024-05-10T11:08:16.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.588+0000 7f87f1e00700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f87d80b2900 0x7f87d80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:16.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.588+0000 7f87f1e00700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f87d80b2900 0x7f87d80b4dc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009be0 tx=0x7f87e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:16.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.589+0000 7f87eaa00700 1 -- 172.21.15.27:0/4114696833 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f87e00aa3d0 con 0x7f87f4106080 2024-05-10T11:08:16.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.827+0000 7f87f8e00700 1 -- 172.21.15.27:0/4114696833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87f402e510 con 0x7f87f4106080 2024-05-10T11:08:16.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.834+0000 7f87eaa00700 1 -- 172.21.15.27:0/4114696833 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24056 (secure 0 0 0) 0x7f87e00a9cd0 con 0x7f87f4106080 2024-05-10T11:08:16.836 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:08:16.836 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:08:08.669744+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:08:16.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.838+0000 7f87f8e00700 1 -- 172.21.15.27:0/4114696833 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f87d80b2900 msgr2=0x7f87d80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:16.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.838+0000 7f87f8e00700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f87d80b2900 0x7f87d80b4dc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009be0 tx=0x7f87e4009260 comp rx=0 tx=0).stop 2024-05-10T11:08:16.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.838+0000 7f87f8e00700 1 -- 172.21.15.27:0/4114696833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f4106080 msgr2=0x7f87f419c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:16.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.838+0000 7f87f8e00700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f4106080 0x7f87f419c300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f87e000e9a0 tx=0x7f87e000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:08:16.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.839+0000 7f87f8e00700 1 -- 172.21.15.27:0/4114696833 shutdown_connections 2024-05-10T11:08:16.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.839+0000 7f87f8e00700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f87d80b2900 0x7f87d80b4dc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:16.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.839+0000 7f87f8e00700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87f41056c0 0x7f87f419bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:16.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.839+0000 7f87f8e00700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87f410b6c0 0x7f87f41a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:16.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.839+0000 7f87f8e00700 1 --2- 172.21.15.27:0/4114696833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f4106080 0x7f87f419c300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:16.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.839+0000 7f87f8e00700 1 -- 172.21.15.27:0/4114696833 >> 172.21.15.27:0/4114696833 conn(0x7f87f41011b0 msgr2=0x7f87f41016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:16.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.839+0000 7f87f8e00700 1 -- 172.21.15.27:0/4114696833 shutdown_connections 2024-05-10T11:08:16.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:16.839+0000 7f87f8e00700 1 -- 172.21.15.27:0/4114696833 wait complete. 2024-05-10T11:08:17.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:17 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4114696833' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:08:17.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:17 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4114696833' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:08:17.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:17 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4114696833' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:08:17.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:08:18.146 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:08:18.306 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:18 smithi027 ceph-mon[30054]: pgmap v344: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:18.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:18 smithi076 ceph-mon[34152]: pgmap v344: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:18.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:18 smithi189 ceph-mon[34842]: pgmap v344: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:20 smithi027 ceph-mon[30054]: pgmap v345: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.375+0000 7f1cab200700 1 -- 172.21.15.27:0/1829002417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca40756d0 msgr2=0x7f1ca4075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.375+0000 7f1cab200700 1 --2- 172.21.15.27:0/1829002417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca40756d0 0x7f1ca4075ab0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1c94009990 tx=0x7f1c94009ca0 comp rx=0 tx=0).stop 2024-05-10T11:08:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.375+0000 7f1cab200700 1 -- 172.21.15.27:0/1829002417 shutdown_connections 2024-05-10T11:08:20.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.375+0000 7f1cab200700 1 --2- 172.21.15.27:0/1829002417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca410fea0 0x7f1ca4112290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:20.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.375+0000 7f1cab200700 1 --2- 172.21.15.27:0/1829002417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca40756d0 0x7f1ca4075ab0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:20.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.375+0000 7f1cab200700 1 --2- 172.21.15.27:0/1829002417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca4075ff0 0x7f1ca410f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:20.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.375+0000 7f1cab200700 1 -- 172.21.15.27:0/1829002417 >> 172.21.15.27:0/1829002417 conn(0x7f1ca407a600 msgr2=0x7f1ca407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:20.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.375+0000 7f1cab200700 1 -- 172.21.15.27:0/1829002417 shutdown_connections 2024-05-10T11:08:20.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.375+0000 7f1cab200700 1 -- 172.21.15.27:0/1829002417 wait complete. 2024-05-10T11:08:20.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.377+0000 7f1cab200700 1 Processor -- start 2024-05-10T11:08:20.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.377+0000 7f1cab200700 1 -- start start 2024-05-10T11:08:20.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.377+0000 7f1cab200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca40756d0 0x7f1ca419bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:20.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.377+0000 7f1cab200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca4075ff0 0x7f1ca419c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:20.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.377+0000 7f1cab200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca410fea0 0x7f1ca41a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:20.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.377+0000 7f1cab200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca41158c0 con 0x7f1ca410fea0 2024-05-10T11:08:20.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.377+0000 7f1cab200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca4115740 con 0x7f1ca40756d0 2024-05-10T11:08:20.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.377+0000 7f1cab200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca4115a40 con 0x7f1ca4075ff0 2024-05-10T11:08:20.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.378+0000 7f1ca3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca4075ff0 0x7f1ca419c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.378+0000 7f1ca3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca4075ff0 0x7f1ca419c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36388/0 (socket says 172.21.15.27:36388) 2024-05-10T11:08:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.378+0000 7f1ca3400700 1 -- 172.21.15.27:0/2847485045 learned_addr learned my addr 172.21.15.27:0/2847485045 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.378+0000 7f1ca9200700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca410fea0 0x7f1ca41a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.378+0000 7f1ca3e00700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca40756d0 0x7f1ca419bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.378+0000 7f1ca3e00700 1 -- 172.21.15.27:0/2847485045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca4075ff0 msgr2=0x7f1ca419c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.378+0000 7f1ca3e00700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca4075ff0 0x7f1ca419c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.378+0000 7f1ca3e00700 1 -- 172.21.15.27:0/2847485045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca410fea0 msgr2=0x7f1ca41a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.378+0000 7f1ca3e00700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca410fea0 0x7f1ca41a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.378+0000 7f1ca3e00700 1 -- 172.21.15.27:0/2847485045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca41a0d40 con 0x7f1ca40756d0 2024-05-10T11:08:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.378+0000 7f1ca9200700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca410fea0 0x7f1ca41a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:08:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.379+0000 7f1ca3e00700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca40756d0 0x7f1ca419bd30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c94000c00 tx=0x7f1c940047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.379+0000 7f1ca0c00700 1 -- 172.21.15.27:0/2847485045 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c94026070 con 0x7f1ca40756d0 2024-05-10T11:08:20.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.379+0000 7f1ca0c00700 1 -- 172.21.15.27:0/2847485045 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c94021410 con 0x7f1ca40756d0 2024-05-10T11:08:20.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.379+0000 7f1ca0c00700 1 -- 172.21.15.27:0/2847485045 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c94017650 con 0x7f1ca40756d0 2024-05-10T11:08:20.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.379+0000 7f1cab200700 1 -- 172.21.15.27:0/2847485045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca41a0fd0 con 0x7f1ca40756d0 2024-05-10T11:08:20.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.379+0000 7f1cab200700 1 -- 172.21.15.27:0/2847485045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca41a88b0 con 0x7f1ca40756d0 2024-05-10T11:08:20.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.380+0000 7f1cab200700 1 -- 172.21.15.27:0/2847485045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca4076dc0 con 0x7f1ca40756d0 2024-05-10T11:08:20.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.383+0000 7f1ca0c00700 1 -- 172.21.15.27:0/2847485045 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f1c94029070 con 0x7f1ca40756d0 2024-05-10T11:08:20.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.384+0000 7f1ca0c00700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f1c880b2900 0x7f1c880b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:20.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.384+0000 7f1ca0c00700 1 -- 172.21.15.27:0/2847485045 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f1c940e3290 con 0x7f1ca40756d0 2024-05-10T11:08:20.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.385+0000 7f1ca3400700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f1c880b2900 0x7f1c880b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:20.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.385+0000 7f1ca3400700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f1c880b2900 0x7f1c880b4dc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4111c10 tx=0x7f1c90009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:20.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.386+0000 7f1ca0c00700 1 -- 172.21.15.27:0/2847485045 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f1c94013070 con 0x7f1ca40756d0 2024-05-10T11:08:20.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:20 smithi076 ceph-mon[34152]: pgmap v345: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:20.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:20 smithi189 ceph-mon[34842]: pgmap v345: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:20.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.628+0000 7f1cab200700 1 -- 172.21.15.27:0/2847485045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1ca402e510 con 0x7f1ca40756d0 2024-05-10T11:08:20.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.630+0000 7f1ca0c00700 1 -- 172.21.15.27:0/2847485045 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24056 (secure 0 0 0) 0x7f1c940aada0 con 0x7f1ca40756d0 2024-05-10T11:08:20.631 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:08:20.631 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:08:08.669744+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:08:20.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.633+0000 7f1cab200700 1 -- 172.21.15.27:0/2847485045 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f1c880b2900 msgr2=0x7f1c880b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:20.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.633+0000 7f1cab200700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f1c880b2900 0x7f1c880b4dc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4111c10 tx=0x7f1c90009260 comp rx=0 tx=0).stop 2024-05-10T11:08:20.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.633+0000 7f1cab200700 1 -- 172.21.15.27:0/2847485045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca40756d0 msgr2=0x7f1ca419bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:20.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.633+0000 7f1cab200700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca40756d0 0x7f1ca419bd30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c94000c00 tx=0x7f1c940047b0 comp rx=0 tx=0).stop 2024-05-10T11:08:20.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.634+0000 7f1cab200700 1 -- 172.21.15.27:0/2847485045 shutdown_connections 2024-05-10T11:08:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.634+0000 7f1cab200700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f1c880b2900 0x7f1c880b4dc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.634+0000 7f1cab200700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca410fea0 0x7f1ca41a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.634+0000 7f1cab200700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ca40756d0 0x7f1ca419bd30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.634+0000 7f1cab200700 1 --2- 172.21.15.27:0/2847485045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca4075ff0 0x7f1ca419c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.634+0000 7f1cab200700 1 -- 172.21.15.27:0/2847485045 >> 172.21.15.27:0/2847485045 conn(0x7f1ca407a600 msgr2=0x7f1ca4100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.634+0000 7f1cab200700 1 -- 172.21.15.27:0/2847485045 shutdown_connections 2024-05-10T11:08:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:20.634+0000 7f1cab200700 1 -- 172.21.15.27:0/2847485045 wait complete. 2024-05-10T11:08:21.043 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:21 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2847485045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:08:21.043 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-05-10T11:08:21.044 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-05-10T11:08:21.044 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6, 4, 8]}]: dispatch 2024-05-10T11:08:21.044 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-05-10T11:08:21.044 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]: dispatch 2024-05-10T11:08:21.044 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-05-10T11:08:21.044 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-05-10T11:08:21.044 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-05-10T11:08:21.044 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:21 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-05-10T11:08:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:21 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2847485045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:08:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:21 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-05-10T11:08:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:21 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-05-10T11:08:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:21 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6, 4, 8]}]: dispatch 2024-05-10T11:08:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:21 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-05-10T11:08:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:21 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]: dispatch 2024-05-10T11:08:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:21 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-05-10T11:08:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:21 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-05-10T11:08:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:21 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-05-10T11:08:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:21 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-05-10T11:08:21.498 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:08:21.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:21 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2847485045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:08:21.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-05-10T11:08:21.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-05-10T11:08:21.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6, 4, 8]}]: dispatch 2024-05-10T11:08:21.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-05-10T11:08:21.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]: dispatch 2024-05-10T11:08:21.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-05-10T11:08:21.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-05-10T11:08:21.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-05-10T11:08:21.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:21 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-05-10T11:08:21.929 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:08:22.345 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:22 smithi027 ceph-mon[30054]: pgmap v346: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:22.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-05-10T11:08:22.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-05-10T11:08:22.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6, 4, 8]}]': finished 2024-05-10T11:08:22.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-05-10T11:08:22.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]': finished 2024-05-10T11:08:22.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-05-10T11:08:22.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-05-10T11:08:22.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-05-10T11:08:22.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-05-10T11:08:22.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:22 smithi027 ceph-mon[30054]: osdmap e87: 12 total, 12 up, 12 in 2024-05-10T11:08:22.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:22 smithi027 ceph-mon[30054]: osdmap e88: 12 total, 12 up, 12 in 2024-05-10T11:08:22.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:22 smithi076 ceph-mon[34152]: pgmap v346: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:22.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-05-10T11:08:22.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-05-10T11:08:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6, 4, 8]}]': finished 2024-05-10T11:08:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-05-10T11:08:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]': finished 2024-05-10T11:08:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-05-10T11:08:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-05-10T11:08:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-05-10T11:08:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-05-10T11:08:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:22 smithi076 ceph-mon[34152]: osdmap e87: 12 total, 12 up, 12 in 2024-05-10T11:08:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:22 smithi076 ceph-mon[34152]: osdmap e88: 12 total, 12 up, 12 in 2024-05-10T11:08:22.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:22 smithi189 ceph-mon[34842]: pgmap v346: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-05-10T11:08:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-05-10T11:08:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6, 4, 8]}]': finished 2024-05-10T11:08:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-05-10T11:08:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]': finished 2024-05-10T11:08:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-05-10T11:08:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-05-10T11:08:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-05-10T11:08:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-05-10T11:08:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:22 smithi189 ceph-mon[34842]: osdmap e87: 12 total, 12 up, 12 in 2024-05-10T11:08:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:22 smithi189 ceph-mon[34842]: osdmap e88: 12 total, 12 up, 12 in 2024-05-10T11:08:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.251+0000 7f7595a00700 1 -- 172.21.15.27:0/1216065080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75901055d0 msgr2=0x7f7590105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.251+0000 7f7595a00700 1 --2- 172.21.15.27:0/1216065080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75901055d0 0x7f7590105a50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f757c009990 tx=0x7f757c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:08:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.251+0000 7f7595a00700 1 -- 172.21.15.27:0/1216065080 shutdown_connections 2024-05-10T11:08:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.251+0000 7f7595a00700 1 --2- 172.21.15.27:0/1216065080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f759010b660 0x7f759010ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.251+0000 7f7595a00700 1 --2- 172.21.15.27:0/1216065080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7590105f90 0x7f75901124a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.251+0000 7f7595a00700 1 --2- 172.21.15.27:0/1216065080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75901055d0 0x7f7590105a50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.251+0000 7f7595a00700 1 -- 172.21.15.27:0/1216065080 >> 172.21.15.27:0/1216065080 conn(0x7f75901010c0 msgr2=0x7f75901034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.251+0000 7f7595a00700 1 -- 172.21.15.27:0/1216065080 shutdown_connections 2024-05-10T11:08:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.251+0000 7f7595a00700 1 -- 172.21.15.27:0/1216065080 wait complete. 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.252+0000 7f7595a00700 1 Processor -- start 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.252+0000 7f7595a00700 1 -- start start 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.253+0000 7f7595a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75901055d0 0x7f75901a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.253+0000 7f7595a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7590105f90 0x7f75901a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.253+0000 7f7595a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f759010b660 0x7f759019ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.253+0000 7f7595a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f75901157b0 con 0x7f759010b660 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.253+0000 7f7595a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7590115630 con 0x7f7590105f90 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.253+0000 7f7595a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7590115930 con 0x7f75901055d0 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.253+0000 7f758fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75901055d0 0x7f75901a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.253+0000 7f758f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7590105f90 0x7f75901a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.253+0000 7f758fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75901055d0 0x7f75901a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:32778/0 (socket says 172.21.15.27:32778) 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.253+0000 7f758fe00700 1 -- 172.21.15.27:0/2206487134 learned_addr learned my addr 172.21.15.27:0/2206487134 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.253+0000 7f7595000700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f759010b660 0x7f759019ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:24.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.254+0000 7f758fe00700 1 -- 172.21.15.27:0/2206487134 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7590105f90 msgr2=0x7f75901a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.254+0000 7f758fe00700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7590105f90 0x7f75901a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.254+0000 7f758fe00700 1 -- 172.21.15.27:0/2206487134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f759010b660 msgr2=0x7f759019ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.254+0000 7f758fe00700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f759010b660 0x7f759019ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.254+0000 7f758fe00700 1 -- 172.21.15.27:0/2206487134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759019c1b0 con 0x7f75901055d0 2024-05-10T11:08:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.254+0000 7f758fe00700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75901055d0 0x7f75901a1840 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f758000e970 tx=0x7f758000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.254+0000 7f758cc00700 1 -- 172.21.15.27:0/2206487134 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7580004220 con 0x7f75901055d0 2024-05-10T11:08:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.254+0000 7f7595a00700 1 -- 172.21.15.27:0/2206487134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f759019c4a0 con 0x7f75901055d0 2024-05-10T11:08:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.254+0000 7f758cc00700 1 -- 172.21.15.27:0/2206487134 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75800043c0 con 0x7f75901055d0 2024-05-10T11:08:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.254+0000 7f758cc00700 1 -- 172.21.15.27:0/2206487134 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7580003ce0 con 0x7f75901055d0 2024-05-10T11:08:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.254+0000 7f7595a00700 1 -- 172.21.15.27:0/2206487134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75901a8710 con 0x7f75901055d0 2024-05-10T11:08:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.255+0000 7f7595a00700 1 -- 172.21.15.27:0/2206487134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f759006a190 con 0x7f75901055d0 2024-05-10T11:08:24.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.258+0000 7f758cc00700 1 -- 172.21.15.27:0/2206487134 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f758000c7e0 con 0x7f75901055d0 2024-05-10T11:08:24.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.259+0000 7f758cc00700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f75740b2880 0x7f75740b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:24.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.260+0000 7f758cc00700 1 -- 172.21.15.27:0/2206487134 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7f75800e27e0 con 0x7f75901055d0 2024-05-10T11:08:24.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.260+0000 7f758f400700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f75740b2880 0x7f75740b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.260+0000 7f758f400700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f75740b2880 0x7f75740b4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f757c00b2e0 tx=0x7f757c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.264+0000 7f758cc00700 1 -- 172.21.15.27:0/2206487134 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f75800aa160 con 0x7f75901055d0 2024-05-10T11:08:24.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:24 smithi027 ceph-mon[30054]: pgmap v349: 129 pgs: 3 peering, 126 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:24.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:24 smithi076 ceph-mon[34152]: pgmap v349: 129 pgs: 3 peering, 126 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:24.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:24 smithi189 ceph-mon[34842]: pgmap v349: 129 pgs: 3 peering, 126 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.554+0000 7f7595a00700 1 -- 172.21.15.27:0/2206487134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f759019cf30 con 0x7f75901055d0 2024-05-10T11:08:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.555+0000 7f758cc00700 1 -- 172.21.15.27:0/2206487134 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11800 (secure 0 0 0) 0x7f75800a9a60 con 0x7f75901055d0 2024-05-10T11:08:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:08:24.555 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:08:08.670053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24527,"qdb_cluster":[24527]},"id":1}]} 2024-05-10T11:08:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.558+0000 7f7595a00700 1 -- 172.21.15.27:0/2206487134 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f75740b2880 msgr2=0x7f75740b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.558+0000 7f7595a00700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f75740b2880 0x7f75740b4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f757c00b2e0 tx=0x7f757c005960 comp rx=0 tx=0).stop 2024-05-10T11:08:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.558+0000 7f7595a00700 1 -- 172.21.15.27:0/2206487134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75901055d0 msgr2=0x7f75901a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.558+0000 7f7595a00700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75901055d0 0x7f75901a1840 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f758000e970 tx=0x7f758000ec80 comp rx=0 tx=0).stop 2024-05-10T11:08:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.558+0000 7f7595a00700 1 -- 172.21.15.27:0/2206487134 shutdown_connections 2024-05-10T11:08:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.558+0000 7f7595a00700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f75740b2880 0x7f75740b4d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.558+0000 7f7595a00700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75901055d0 0x7f75901a1840 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.558+0000 7f7595a00700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f759010b660 0x7f759019ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.558+0000 7f7595a00700 1 --2- 172.21.15.27:0/2206487134 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7590105f90 0x7f75901a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.558+0000 7f7595a00700 1 -- 172.21.15.27:0/2206487134 >> 172.21.15.27:0/2206487134 conn(0x7f75901010c0 msgr2=0x7f75901027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.559+0000 7f7595a00700 1 -- 172.21.15.27:0/2206487134 shutdown_connections 2024-05-10T11:08:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:24.559+0000 7f7595a00700 1 -- 172.21.15.27:0/2206487134 wait complete. 2024-05-10T11:08:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-05-10T11:08:25.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:25 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2206487134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:08:25.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:25 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2206487134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:08:25.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:08:25.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:25 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2206487134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:08:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:08:26.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:26 smithi027 ceph-mon[30054]: pgmap v350: 129 pgs: 3 peering, 126 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:26.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:26 smithi076 ceph-mon[34152]: pgmap v350: 129 pgs: 3 peering, 126 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:26.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:26 smithi189 ceph-mon[34842]: pgmap v350: 129 pgs: 3 peering, 126 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.138+0000 7f048fe00700 1 -- 172.21.15.27:0/3740734324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04901099e0 msgr2=0x7f0490109e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.138+0000 7f048fe00700 1 --2- 172.21.15.27:0/3740734324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04901099e0 0x7f0490109e60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f047c009990 tx=0x7f047c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:08:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.138+0000 7f048fe00700 1 -- 172.21.15.27:0/3740734324 shutdown_connections 2024-05-10T11:08:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.138+0000 7f048fe00700 1 --2- 172.21.15.27:0/3740734324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f049010a3a0 0x7f0490116840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.138+0000 7f048fe00700 1 --2- 172.21.15.27:0/3740734324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f049010e8b0 0x7f049010ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.138+0000 7f048fe00700 1 --2- 172.21.15.27:0/3740734324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04901099e0 0x7f0490109e60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.138+0000 7f048fe00700 1 -- 172.21.15.27:0/3740734324 >> 172.21.15.27:0/3740734324 conn(0x7f0490079230 msgr2=0x7f04901082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.138+0000 7f048fe00700 1 -- 172.21.15.27:0/3740734324 shutdown_connections 2024-05-10T11:08:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.138+0000 7f048fe00700 1 -- 172.21.15.27:0/3740734324 wait complete. 2024-05-10T11:08:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.140+0000 7f048fe00700 1 Processor -- start 2024-05-10T11:08:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.140+0000 7f048fe00700 1 -- start start 2024-05-10T11:08:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.140+0000 7f048fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04901099e0 0x7f0490074c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.140+0000 7f048fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f049010a3a0 0x7f049006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.140+0000 7f048fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f049010e8b0 0x7f049006e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.140+0000 7f048fe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0490119be0 con 0x7f04901099e0 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048fe00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0490119a60 con 0x7f049010a3a0 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0490119d60 con 0x7f049010e8b0 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04901099e0 0x7f0490074c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04901099e0 0x7f0490074c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47024/0 (socket says 172.21.15.27:47024) 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048ea00700 1 -- 172.21.15.27:0/4005537729 learned_addr learned my addr 172.21.15.27:0/4005537729 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048e000700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f049010a3a0 0x7f049006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048f400700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f049010e8b0 0x7f049006e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048e000700 1 -- 172.21.15.27:0/4005537729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f049010e8b0 msgr2=0x7f049006e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048e000700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f049010e8b0 0x7f049006e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048e000700 1 -- 172.21.15.27:0/4005537729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04901099e0 msgr2=0x7f0490074c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048e000700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04901099e0 0x7f0490074c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048e000700 1 -- 172.21.15.27:0/4005537729 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049006e880 con 0x7f049010a3a0 2024-05-10T11:08:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.141+0000 7f048ea00700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04901099e0 0x7f0490074c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:08:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.142+0000 7f048e000700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f049010a3a0 0x7f049006dd50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f047c000c00 tx=0x7f047c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.142+0000 7f0487400700 1 -- 172.21.15.27:0/4005537729 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c026070 con 0x7f049010a3a0 2024-05-10T11:08:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.142+0000 7f0487400700 1 -- 172.21.15.27:0/4005537729 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f047c021410 con 0x7f049010a3a0 2024-05-10T11:08:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.142+0000 7f048fe00700 1 -- 172.21.15.27:0/4005537729 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f049006eb10 con 0x7f049010a3a0 2024-05-10T11:08:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.142+0000 7f0487400700 1 -- 172.21.15.27:0/4005537729 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c017650 con 0x7f049010a3a0 2024-05-10T11:08:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.142+0000 7f048fe00700 1 -- 172.21.15.27:0/4005537729 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0490072a20 con 0x7f049010a3a0 2024-05-10T11:08:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.143+0000 7f048fe00700 1 -- 172.21.15.27:0/4005537729 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f049010b0d0 con 0x7f049010a3a0 2024-05-10T11:08:28.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.145+0000 7f0487400700 1 -- 172.21.15.27:0/4005537729 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f047c029070 con 0x7f049010a3a0 2024-05-10T11:08:28.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.147+0000 7f0487400700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f04740b2900 0x7f04740b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:28.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.147+0000 7f0487400700 1 -- 172.21.15.27:0/4005537729 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7f047c0e31d0 con 0x7f049010a3a0 2024-05-10T11:08:28.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.147+0000 7f048ea00700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f04740b2900 0x7f04740b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:28.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.148+0000 7f048ea00700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f04740b2900 0x7f04740b4dc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0480009680 tx=0x7f0480009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:28.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.151+0000 7f0487400700 1 -- 172.21.15.27:0/4005537729 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f047c013070 con 0x7f049010a3a0 2024-05-10T11:08:28.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:28 smithi027 ceph-mon[30054]: pgmap v351: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:28.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.394+0000 7f048fe00700 1 -- 172.21.15.27:0/4005537729 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0490073450 con 0x7f049010a3a0 2024-05-10T11:08:28.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.396+0000 7f0487400700 1 -- 172.21.15.27:0/4005537729 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24497 (secure 0 0 0) 0x7f047c0aabe0 con 0x7f049010a3a0 2024-05-10T11:08:28.396 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:08:28.396 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":88,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:08:22.020361+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6},{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:08:28.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.399+0000 7f048fe00700 1 -- 172.21.15.27:0/4005537729 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f04740b2900 msgr2=0x7f04740b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:28.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.399+0000 7f048fe00700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f04740b2900 0x7f04740b4dc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0480009680 tx=0x7f0480009260 comp rx=0 tx=0).stop 2024-05-10T11:08:28.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.399+0000 7f048fe00700 1 -- 172.21.15.27:0/4005537729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f049010a3a0 msgr2=0x7f049006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:28.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.399+0000 7f048fe00700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f049010a3a0 0x7f049006dd50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f047c000c00 tx=0x7f047c0047b0 comp rx=0 tx=0).stop 2024-05-10T11:08:28.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.399+0000 7f048fe00700 1 -- 172.21.15.27:0/4005537729 shutdown_connections 2024-05-10T11:08:28.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.399+0000 7f048fe00700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f04740b2900 0x7f04740b4dc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:28.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.399+0000 7f048fe00700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f049010e8b0 0x7f049006e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:28.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.399+0000 7f048fe00700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04901099e0 0x7f0490074c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:28.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.399+0000 7f048fe00700 1 --2- 172.21.15.27:0/4005537729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f049010a3a0 0x7f049006dd50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:28.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.399+0000 7f048fe00700 1 -- 172.21.15.27:0/4005537729 >> 172.21.15.27:0/4005537729 conn(0x7f0490079230 msgr2=0x7f0490079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:28.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.400+0000 7f048fe00700 1 -- 172.21.15.27:0/4005537729 shutdown_connections 2024-05-10T11:08:28.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:28.400+0000 7f048fe00700 1 -- 172.21.15.27:0/4005537729 wait complete. 2024-05-10T11:08:28.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:28 smithi076 ceph-mon[34152]: pgmap v351: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:28.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:28 smithi189 ceph-mon[34842]: pgmap v351: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:29.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:29 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4005537729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:08:29.237 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-10T11:07:50.237787+0000', 'modified': '2024-05-10T11:08:08.670053+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24527}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.76:6839/1778204420', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6838', 'nonce': 1778204420}, {'type': 'v1', 'addr': '172.21.15.76:6839', 'nonce': 1778204420}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24527, 'qdb_cluster': [24527]} 2024-05-10T11:08:29.237 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-10T11:08:29.237 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.b status 2024-05-10T11:08:29.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:29 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4005537729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:08:29.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:29 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4005537729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:08:29.704 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:08:30.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:30 smithi027 ceph-mon[30054]: pgmap v352: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:30.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:30 smithi076 ceph-mon[34152]: pgmap v352: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:30.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:30 smithi189 ceph-mon[34842]: pgmap v352: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:32.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.121+0000 7f3080200700 1 -- 172.21.15.27:0/3540851772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3078105ee0 msgr2=0x7f3078106370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:32.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.121+0000 7f3080200700 1 --2- 172.21.15.27:0/3540851772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3078105ee0 0x7f3078106370 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3070009990 tx=0x7f3070009ca0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.121+0000 7f3080200700 1 -- 172.21.15.27:0/3540851772 shutdown_connections 2024-05-10T11:08:32.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.121+0000 7f3080200700 1 --2- 172.21.15.27:0/3540851772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3078105ee0 0x7f3078106370 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.121+0000 7f3080200700 1 --2- 172.21.15.27:0/3540851772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30781083d0 0x7f307806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.121+0000 7f3080200700 1 --2- 172.21.15.27:0/3540851772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f307806b6f0 0x7f307806bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.121+0000 7f3080200700 1 -- 172.21.15.27:0/3540851772 >> 172.21.15.27:0/3540851772 conn(0x7f3078079350 msgr2=0x7f3078079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:32.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.121+0000 7f3080200700 1 -- 172.21.15.27:0/3540851772 shutdown_connections 2024-05-10T11:08:32.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.121+0000 7f3080200700 1 -- 172.21.15.27:0/3540851772 wait complete. 2024-05-10T11:08:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.122+0000 7f3080200700 1 Processor -- start 2024-05-10T11:08:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.122+0000 7f3080200700 1 -- start start 2024-05-10T11:08:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.122+0000 7f3080200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806b6f0 0x7f30781a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f3080200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3078105ee0 0x7f30781a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f3080200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30781083d0 0x7f30781a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f3080200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3078119c60 con 0x7f307806b6f0 2024-05-10T11:08:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f3080200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3078119ae0 con 0x7f30781083d0 2024-05-10T11:08:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f3080200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3078119de0 con 0x7f3078105ee0 2024-05-10T11:08:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f307d800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806b6f0 0x7f30781a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f307ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3078105ee0 0x7f30781a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f307d800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806b6f0 0x7f30781a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47044/0 (socket says 172.21.15.27:47044) 2024-05-10T11:08:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f307d800700 1 -- 172.21.15.27:0/1632298821 learned_addr learned my addr 172.21.15.27:0/1632298821 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f307e200700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30781083d0 0x7f30781a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f307ce00700 1 -- 172.21.15.27:0/1632298821 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30781083d0 msgr2=0x7f30781a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f307ce00700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30781083d0 0x7f30781a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f307ce00700 1 -- 172.21.15.27:0/1632298821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806b6f0 msgr2=0x7f30781a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f307ce00700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806b6f0 0x7f30781a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.123+0000 7f307ce00700 1 -- 172.21.15.27:0/1632298821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30781a51c0 con 0x7f3078105ee0 2024-05-10T11:08:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.124+0000 7f307ce00700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3078105ee0 0x7f30781a06f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f306400e9f0 tx=0x7f306400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.124+0000 7f3076000700 1 -- 172.21.15.27:0/1632298821 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306400c9c0 con 0x7f3078105ee0 2024-05-10T11:08:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.124+0000 7f3076000700 1 -- 172.21.15.27:0/1632298821 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f306400cb60 con 0x7f3078105ee0 2024-05-10T11:08:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.124+0000 7f3076000700 1 -- 172.21.15.27:0/1632298821 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064018680 con 0x7f3078105ee0 2024-05-10T11:08:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.124+0000 7f3080200700 1 -- 172.21.15.27:0/1632298821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30781a54b0 con 0x7f3078105ee0 2024-05-10T11:08:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.124+0000 7f3080200700 1 -- 172.21.15.27:0/1632298821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30781acd30 con 0x7f3078105ee0 2024-05-10T11:08:32.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.128+0000 7f3076000700 1 -- 172.21.15.27:0/1632298821 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f3064010430 con 0x7f3078105ee0 2024-05-10T11:08:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.130+0000 7f308aaffb80 1 Processor -- start 2024-05-10T11:08:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.130+0000 7f308aaffb80 1 -- start start 2024-05-10T11:08:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.130+0000 7f308aaffb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61cdb70 0x55def61cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.130+0000 7f3076000700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30600b2900 0x7f30600b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.131+0000 7f308aaffb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61ce490 0x55def61c8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.131+0000 7f308aaffb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.131+0000 7f308aaffb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55def611a9d0 con 0x55def61c88a0 2024-05-10T11:08:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.131+0000 7f308aaffb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55def60b8710 con 0x55def61ce490 2024-05-10T11:08:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.131+0000 7f308aaffb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55def6116990 con 0x55def61cdb70 2024-05-10T11:08:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.131+0000 7f3076000700 1 -- 172.21.15.27:0/1632298821 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7f30640e2690 con 0x7f3078105ee0 2024-05-10T11:08:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.131+0000 7f307e200700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30600b2900 0x7f30600b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.131+0000 7f307d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61cdb70 0x55def61cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.131+0000 7f307ce00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61ce490 0x55def61c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.131+0000 7f307d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61cdb70 0x55def61cdf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:32822/0 (socket says 172.21.15.27:32822) 2024-05-10T11:08:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.131+0000 7f307d800700 1 -- 172.21.15.27:0/4154613273 learned_addr learned my addr 172.21.15.27:0/4154613273 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.131+0000 7f307e200700 1 --2- 172.21.15.27:0/4154613273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.132+0000 7f307ce00700 1 -- 172.21.15.27:0/4154613273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61cdb70 msgr2=0x55def61cdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:08:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.132+0000 7f307ce00700 1 --2- 172.21.15.27:0/4154613273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61cdb70 0x55def61cdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.132+0000 7f307ce00700 1 -- 172.21.15.27:0/4154613273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 msgr2=0x55def61c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:08:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.132+0000 7f307ce00700 1 --2- 172.21.15.27:0/4154613273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.132+0000 7f307ce00700 1 -- 172.21.15.27:0/4154613273 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55def6067880 con 0x55def61ce490 2024-05-10T11:08:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.132+0000 7f307ce00700 1 --2- 172.21.15.27:0/4154613273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61ce490 0x55def61c8280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f30640a6b80 tx=0x7f30640b20b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e1d63957e76136 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.132+0000 7f307e200700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30600b2900 0x7f30600b4dc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3070005cd0 tx=0x7f30700051a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.132+0000 7f306aa00700 1 -- 172.21.15.27:0/4154613273 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30640aa010 con 0x55def61ce490 2024-05-10T11:08:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.132+0000 7f306aa00700 1 -- 172.21.15.27:0/4154613273 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30640a9910 con 0x55def61ce490 2024-05-10T11:08:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.132+0000 7f306aa00700 1 -- 172.21.15.27:0/4154613273 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30640dff00 con 0x55def61ce490 2024-05-10T11:08:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.133+0000 7f308aaffb80 1 -- 172.21.15.27:0/4154613273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61ce490 msgr2=0x55def61c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.133+0000 7f308aaffb80 1 --2- 172.21.15.27:0/4154613273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61ce490 0x55def61c8280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f30640a6b80 tx=0x7f30640b20b0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.133+0000 7f308aaffb80 1 -- 172.21.15.27:0/4154613273 shutdown_connections 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.133+0000 7f308aaffb80 1 --2- 172.21.15.27:0/4154613273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61cdb70 0x55def61cdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.133+0000 7f308aaffb80 1 --2- 172.21.15.27:0/4154613273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.133+0000 7f308aaffb80 1 --2- 172.21.15.27:0/4154613273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61ce490 0x55def61c8280 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.133+0000 7f308aaffb80 1 -- 172.21.15.27:0/4154613273 >> 172.21.15.27:0/4154613273 conn(0x55def6140680 msgr2=0x55def61c2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.133+0000 7f308aaffb80 1 -- 172.21.15.27:0/4154613273 shutdown_connections 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.133+0000 7f308aaffb80 1 -- 172.21.15.27:0/4154613273 wait complete. 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.134+0000 7f308aaffb80 1 -- start start 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.134+0000 7f308aaffb80 1 Processor -- start 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f308aaffb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61d41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f308aaffb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61cdb70 0x55def61d1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f308aaffb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61ce490 0x55def61d1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f308aaffb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55def61d4e70 con 0x55def61c88a0 2024-05-10T11:08:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f308aaffb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55def6080ad0 con 0x55def61ce490 2024-05-10T11:08:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f308aaffb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55def61d4ff0 con 0x55def61cdb70 2024-05-10T11:08:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f308aaffb80 10 client.0 _ll_register_callbacks cb 0x55def60c1c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:08:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f307d800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61d41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f307d800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61d41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47068/0 (socket says 172.21.15.27:47068) 2024-05-10T11:08:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f307ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61cdb70 0x55def61d1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f307d800700 1 -- 172.21.15.27:0/3117479006 learned_addr learned my addr 172.21.15.27:0/3117479006 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f307e200700 1 --2- 172.21.15.27:0/3117479006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61ce490 0x55def61d1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f307ce00700 1 -- 172.21.15.27:0/3117479006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61ce490 msgr2=0x55def61d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:08:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f307ce00700 1 --2- 172.21.15.27:0/3117479006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61ce490 0x55def61d1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f307ce00700 1 -- 172.21.15.27:0/3117479006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 msgr2=0x55def61d41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:08:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.135+0000 7f307ce00700 1 --2- 172.21.15.27:0/3117479006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61d41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.136+0000 7f307ce00700 1 -- 172.21.15.27:0/3117479006 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55def60e03d0 con 0x55def61cdb70 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.136+0000 7f307ce00700 1 --2- 172.21.15.27:0/3117479006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61cdb70 0x55def61d1540 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f30640a6b80 tx=0x7f30640a51c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e723b40495741607 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.136+0000 7f305f400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.136+0000 7f305f400700 1 -- 172.21.15.27:0/3117479006 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30640d1070 con 0x55def61cdb70 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.136+0000 7f308aaffb80 1 -- 172.21.15.27:0/3117479006 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30640a9910 con 0x55def61cdb70 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.136+0000 7f305f400700 1 -- 172.21.15.27:0/3117479006 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7f30640d1210 con 0x55def61cdb70 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.136+0000 7f305f400700 1 -- 172.21.15.27:0/3117479006 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f30640aa010 con 0x55def61cdb70 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.136+0000 7f308aaffb80 10 client.24607 fetch_fsmap learned FSMap version 17 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.137+0000 7f308aaffb80 1 -- 172.21.15.27:0/3117479006 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55def61d4e70 con 0x55def61cdb70 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.137+0000 7f305f400700 1 -- 172.21.15.27:0/3117479006 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30640a9910 con 0x55def61cdb70 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.137+0000 7f305f400700 1 -- 172.21.15.27:0/3117479006 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f30640c7050 con 0x55def61cdb70 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.137+0000 7f308aaffb80 10 client.24607 fetch_fsmap finished waiting for FSMap version 17 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.137+0000 7f308aaffb80 10 client.24607 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.137+0000 7f308aaffb80 20 client.24607 populate_metadata read hostname 'smithi027' 2024-05-10T11:08:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.137+0000 7f308aaffb80 1 --2- 172.21.15.27:0/3117479006 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55def61e1780 0x55def61e2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.137+0000 7f308aaffb80 4 client.24607 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:08:32.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.137+0000 7f308aaffb80 1 -- 172.21.15.27:0/3117479006 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55def61950f0 con 0x55def61e1780 2024-05-10T11:08:32.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.138+0000 7f307d800700 1 --2- 172.21.15.27:0/3117479006 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55def61e1780 0x55def61e2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.138+0000 7f307d800700 1 --2- 172.21.15.27:0/3117479006 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55def61e1780 0x55def61e2bf0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c13ffabb99f054d server_cookie=947468137aea73f5 in_seq=0 out_seq=0 2024-05-10T11:08:32.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.138+0000 7f305f400700 10 client.24607 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:08:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.139+0000 7f305f400700 1 -- 172.21.15.27:0/3117479006 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55def61950f0 con 0x55def61e1780 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.139+0000 7f305f400700 10 client.24607 handle_command_reply: tid=0 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.139+0000 7f308aaffb80 1 client.24607 shutdown 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.139+0000 7f308aaffb80 1 -- 172.21.15.27:0/3117479006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61cdb70 msgr2=0x55def61d1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.139+0000 7f308aaffb80 1 --2- 172.21.15.27:0/3117479006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61cdb70 0x55def61d1540 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f30640a6b80 tx=0x7f30640a51c0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.140+0000 7f308aaffb80 1 -- 172.21.15.27:0/3117479006 shutdown_connections 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.140+0000 7f308aaffb80 1 --2- 172.21.15.27:0/3117479006 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55def61e1780 0x55def61e2bf0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.140+0000 7f308aaffb80 1 --2- 172.21.15.27:0/3117479006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61ce490 0x55def61d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.140+0000 7f308aaffb80 1 --2- 172.21.15.27:0/3117479006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61d41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.140+0000 7f308aaffb80 1 --2- 172.21.15.27:0/3117479006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61cdb70 0x55def61d1540 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.140+0000 7f305f400700 0 client.24607 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.140+0000 7f308aaffb80 1 -- 172.21.15.27:0/3117479006 >> 172.21.15.27:0/3117479006 conn(0x55def6140680 msgr2=0x55def61c2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.140+0000 7f308aaffb80 1 -- 172.21.15.27:0/3117479006 shutdown_connections 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.140+0000 7f308aaffb80 1 -- 172.21.15.27:0/3117479006 wait complete. 2024-05-10T11:08:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.140+0000 7f308aaffb80 20 client.24607 trim_cache size 0 max 16384 2024-05-10T11:08:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.163+0000 7f308aaffb80 1 Processor -- start 2024-05-10T11:08:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.163+0000 7f308aaffb80 1 -- start start 2024-05-10T11:08:32.165 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:32 smithi027 ceph-mon[30054]: pgmap v353: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.164+0000 7f308aaffb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61c8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.164+0000 7f308aaffb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61cdb70 0x55def61c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.164+0000 7f308aaffb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61ce490 0x55def61ec050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.164+0000 7f308aaffb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55def61e0f30 con 0x55def61c88a0 2024-05-10T11:08:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.164+0000 7f308aaffb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55def61e1600 con 0x55def61cdb70 2024-05-10T11:08:32.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.164+0000 7f308aaffb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55def61e1cd0 con 0x55def61ce490 2024-05-10T11:08:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.164+0000 7f307d800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.164+0000 7f307d800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61c8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47074/0 (socket says 172.21.15.27:47074) 2024-05-10T11:08:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.164+0000 7f307ce00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61cdb70 0x55def61c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.164+0000 7f307d800700 1 -- 172.21.15.27:0/2644089234 learned_addr learned my addr 172.21.15.27:0/2644089234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.164+0000 7f307e200700 1 --2- 172.21.15.27:0/2644089234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61ce490 0x55def61ec050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.165+0000 7f307d800700 1 -- 172.21.15.27:0/2644089234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61ce490 msgr2=0x55def61ec050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:08:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.165+0000 7f307d800700 1 --2- 172.21.15.27:0/2644089234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61ce490 0x55def61ec050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.165+0000 7f307d800700 1 -- 172.21.15.27:0/2644089234 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61cdb70 msgr2=0x55def61c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:08:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.165+0000 7f307d800700 1 --2- 172.21.15.27:0/2644089234 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61cdb70 0x55def61c28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.165+0000 7f307d800700 1 -- 172.21.15.27:0/2644089234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55def61bda60 con 0x55def61c88a0 2024-05-10T11:08:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.165+0000 7f307d800700 1 --2- 172.21.15.27:0/2644089234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61c8280 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x55def61d5100 tx=0x7f306c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b820a6fdeee7097b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.166+0000 7f306a000700 1 -- 172.21.15.27:0/2644089234 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306c004d40 con 0x55def61c88a0 2024-05-10T11:08:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.166+0000 7f306a000700 1 -- 172.21.15.27:0/2644089234 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f306c0047d0 con 0x55def61c88a0 2024-05-10T11:08:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.166+0000 7f306a000700 1 -- 172.21.15.27:0/2644089234 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306c007500 con 0x55def61c88a0 2024-05-10T11:08:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.166+0000 7f308aaffb80 1 -- 172.21.15.27:0/2644089234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 msgr2=0x55def61c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.166+0000 7f308aaffb80 1 --2- 172.21.15.27:0/2644089234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61c8280 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x55def61d5100 tx=0x7f306c00b040 comp rx=0 tx=0).stop 2024-05-10T11:08:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.167+0000 7f308aaffb80 1 -- 172.21.15.27:0/2644089234 shutdown_connections 2024-05-10T11:08:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.167+0000 7f308aaffb80 1 --2- 172.21.15.27:0/2644089234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61ce490 0x55def61ec050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.167+0000 7f308aaffb80 1 --2- 172.21.15.27:0/2644089234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61c88a0 0x55def61c8280 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.167+0000 7f308aaffb80 1 --2- 172.21.15.27:0/2644089234 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61cdb70 0x55def61c28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.167+0000 7f308aaffb80 1 -- 172.21.15.27:0/2644089234 >> 172.21.15.27:0/2644089234 conn(0x55def6140680 msgr2=0x55def61cfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.167+0000 7f308aaffb80 1 -- 172.21.15.27:0/2644089234 shutdown_connections 2024-05-10T11:08:32.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.167+0000 7f308aaffb80 1 -- 172.21.15.27:0/2644089234 wait complete. 2024-05-10T11:08:32.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.167+0000 7f308aaffb80 1 -- start start 2024-05-10T11:08:32.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.167+0000 7f308aaffb80 1 Processor -- start 2024-05-10T11:08:32.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f308aaffb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61cdb70 0x55def61ec070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f308aaffb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61ce490 0x55def61ec5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f308aaffb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61e7020 0x55def61e74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f308aaffb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55def61e2a70 con 0x55def61ce490 2024-05-10T11:08:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f308aaffb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55def61e23a0 con 0x55def61cdb70 2024-05-10T11:08:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f308aaffb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55def61e3140 con 0x55def61e7020 2024-05-10T11:08:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f308aaffb80 10 client.0 _ll_register_callbacks cb 0x55def60c1c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:08:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f307d800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61cdb70 0x55def61ec070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f307e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61e7020 0x55def61e74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f307d800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61cdb70 0x55def61ec070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:49294/0 (socket says 172.21.15.27:49294) 2024-05-10T11:08:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f307ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61ce490 0x55def61ec5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f307d800700 1 -- 172.21.15.27:0/504462747 learned_addr learned my addr 172.21.15.27:0/504462747 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.168+0000 7f307e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61e7020 0x55def61e74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:32840/0 (socket says 172.21.15.27:32840) 2024-05-10T11:08:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.169+0000 7f307e200700 1 -- 172.21.15.27:0/504462747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61cdb70 msgr2=0x55def61ec070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:08:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.169+0000 7f307e200700 1 --2- 172.21.15.27:0/504462747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61cdb70 0x55def61ec070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.169+0000 7f307e200700 1 -- 172.21.15.27:0/504462747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61ce490 msgr2=0x55def61ec5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:08:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.169+0000 7f307e200700 1 --2- 172.21.15.27:0/504462747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61ce490 0x55def61ec5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.169+0000 7f307e200700 1 -- 172.21.15.27:0/504462747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55def61e7ca0 con 0x55def61e7020 2024-05-10T11:08:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.169+0000 7f307ce00700 1 --2- 172.21.15.27:0/504462747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61ce490 0x55def61ec5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:08:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.169+0000 7f307e200700 1 --2- 172.21.15.27:0/504462747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61e7020 0x55def61e74a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3070009750 tx=0x7f3070026000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98be10aae97abaa6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.169+0000 7f305f400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:08:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.169+0000 7f305f400700 1 -- 172.21.15.27:0/504462747 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3070021040 con 0x55def61e7020 2024-05-10T11:08:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.170+0000 7f308aaffb80 1 -- 172.21.15.27:0/504462747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f306c0047d0 con 0x55def61e7020 2024-05-10T11:08:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.170+0000 7f305f400700 1 -- 172.21.15.27:0/504462747 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7f3070003a90 con 0x55def61e7020 2024-05-10T11:08:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.170+0000 7f305f400700 1 -- 172.21.15.27:0/504462747 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f3070042050 con 0x55def61e7020 2024-05-10T11:08:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.170+0000 7f308aaffb80 10 client.24613 fetch_fsmap learned FSMap version 17 2024-05-10T11:08:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.170+0000 7f308aaffb80 1 -- 172.21.15.27:0/504462747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55def624daa0 con 0x55def61e7020 2024-05-10T11:08:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.170+0000 7f305f400700 1 -- 172.21.15.27:0/504462747 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f306c0047d0 con 0x55def61e7020 2024-05-10T11:08:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.170+0000 7f305f400700 1 -- 172.21.15.27:0/504462747 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f3070029430 con 0x55def61e7020 2024-05-10T11:08:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.171+0000 7f308aaffb80 10 client.24613 fetch_fsmap finished waiting for FSMap version 17 2024-05-10T11:08:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.171+0000 7f308aaffb80 10 client.24613 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-10T11:08:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.171+0000 7f308aaffb80 20 client.24613 populate_metadata read hostname 'smithi027' 2024-05-10T11:08:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.171+0000 7f308aaffb80 1 --2- 172.21.15.27:0/504462747 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55def62526a0 0x55def62513b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.171+0000 7f308aaffb80 4 client.24613 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-10T11:08:32.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.171+0000 7f308aaffb80 1 -- 172.21.15.27:0/504462747 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "status"}) -- 0x55def61283b0 con 0x55def62526a0 2024-05-10T11:08:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.171+0000 7f307d800700 1 --2- 172.21.15.27:0/504462747 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55def62526a0 0x55def62513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f307d800700 1 --2- 172.21.15.27:0/504462747 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55def62526a0 0x55def62513b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=640e2ef9ae24ef7f server_cookie=c539046fafc3b185 in_seq=0 out_seq=0 2024-05-10T11:08:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f305f400700 10 client.24613 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:08:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f305f400700 1 -- 172.21.15.27:0/504462747 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55def61283b0 con 0x55def62526a0 2024-05-10T11:08:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f305f400700 10 client.24613 handle_command_reply: tid=0 2024-05-10T11:08:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f308aaffb80 1 client.24613 shutdown 2024-05-10T11:08:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f308aaffb80 1 -- 172.21.15.27:0/504462747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61e7020 msgr2=0x55def61e74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f308aaffb80 1 --2- 172.21.15.27:0/504462747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61e7020 0x55def61e74a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3070009750 tx=0x7f3070026000 comp rx=0 tx=0).stop 2024-05-10T11:08:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f308aaffb80 1 -- 172.21.15.27:0/504462747 shutdown_connections 2024-05-10T11:08:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f308aaffb80 1 --2- 172.21.15.27:0/504462747 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55def62526a0 0x55def62513b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f308aaffb80 1 --2- 172.21.15.27:0/504462747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55def61e7020 0x55def61e74a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f308aaffb80 1 --2- 172.21.15.27:0/504462747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55def61cdb70 0x55def61ec070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f308aaffb80 1 --2- 172.21.15.27:0/504462747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55def61ce490 0x55def61ec5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f308aaffb80 1 -- 172.21.15.27:0/504462747 >> 172.21.15.27:0/504462747 conn(0x55def6140680 msgr2=0x55def61cfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:32.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f305f400700 0 client.24613 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:08:32.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f308aaffb80 1 -- 172.21.15.27:0/504462747 shutdown_connections 2024-05-10T11:08:32.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.172+0000 7f308aaffb80 1 -- 172.21.15.27:0/504462747 wait complete. 2024-05-10T11:08:32.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.173+0000 7f308aaffb80 20 client.24613 trim_cache size 0 max 16384 2024-05-10T11:08:32.182 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:08:32.182 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T11:08:32.183 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-05-10T11:08:32.183 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24527, 2024-05-10T11:08:32.183 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-05-10T11:08:32.183 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-05-10T11:08:32.183 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-05-10T11:08:32.183 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 41.880085002999998, 2024-05-10T11:08:32.183 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 17, 2024-05-10T11:08:32.183 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 88, 2024-05-10T11:08:32.183 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 86, 2024-05-10T11:08:32.183 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 65.638437999999994 2024-05-10T11:08:32.183 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:08:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.173+0000 7f3080200700 1 -- 172.21.15.27:0/1632298821 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30600b2900 msgr2=0x7f30600b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.173+0000 7f3080200700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30600b2900 0x7f30600b4dc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3070005cd0 tx=0x7f30700051a0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.173+0000 7f3080200700 1 -- 172.21.15.27:0/1632298821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3078105ee0 msgr2=0x7f30781a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.173+0000 7f3080200700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3078105ee0 0x7f30781a06f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f306400e9f0 tx=0x7f306400ed00 comp rx=0 tx=0).stop 2024-05-10T11:08:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.173+0000 7f3080200700 1 -- 172.21.15.27:0/1632298821 shutdown_connections 2024-05-10T11:08:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.173+0000 7f3080200700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30600b2900 0x7f30600b4dc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.174+0000 7f3080200700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30781083d0 0x7f30781a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.174+0000 7f3080200700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806b6f0 0x7f30781a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.174+0000 7f3080200700 1 --2- 172.21.15.27:0/1632298821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3078105ee0 0x7f30781a06f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.174+0000 7f3080200700 1 -- 172.21.15.27:0/1632298821 >> 172.21.15.27:0/1632298821 conn(0x7f3078079350 msgr2=0x7f3078104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.174+0000 7f3080200700 1 -- 172.21.15.27:0/1632298821 shutdown_connections 2024-05-10T11:08:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:32.174+0000 7f3080200700 1 -- 172.21.15.27:0/1632298821 wait complete. 2024-05-10T11:08:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:32 smithi076 ceph-mon[34152]: pgmap v353: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:32.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:32 smithi189 ceph-mon[34842]: pgmap v353: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:33.134 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-10T11:08:33.135 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-10T11:08:33.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs ls 2024-05-10T11:08:33.602 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:08:34.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:34 smithi027 ceph-mon[30054]: pgmap v354: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:34.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:34 smithi076 ceph-mon[34152]: pgmap v354: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:34.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:34 smithi189 ceph-mon[34842]: pgmap v354: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.829+0000 7f5314800700 1 -- 172.21.15.27:0/540180711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f530c06b180 msgr2=0x7f530c06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.829+0000 7f5314800700 1 --2- 172.21.15.27:0/540180711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f530c06b180 0x7f530c06b600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009990 tx=0x7f52fc009ca0 comp rx=0 tx=0).stop 2024-05-10T11:08:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.829+0000 7f5314800700 1 -- 172.21.15.27:0/540180711 shutdown_connections 2024-05-10T11:08:35.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.829+0000 7f5314800700 1 --2- 172.21.15.27:0/540180711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f530c06bb40 0x7f530c110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:35.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.829+0000 7f5314800700 1 --2- 172.21.15.27:0/540180711 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f530c109510 0x7f530c1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:35.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.829+0000 7f5314800700 1 --2- 172.21.15.27:0/540180711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f530c06b180 0x7f530c06b600 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:35.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.829+0000 7f5314800700 1 -- 172.21.15.27:0/540180711 >> 172.21.15.27:0/540180711 conn(0x7f530c079350 msgr2=0x7f530c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:35.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.829+0000 7f5314800700 1 -- 172.21.15.27:0/540180711 shutdown_connections 2024-05-10T11:08:35.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.829+0000 7f5314800700 1 -- 172.21.15.27:0/540180711 wait complete. 2024-05-10T11:08:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.830+0000 7f5314800700 1 Processor -- start 2024-05-10T11:08:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.830+0000 7f5314800700 1 -- start start 2024-05-10T11:08:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.831+0000 7f5314800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f530c06b180 0x7f530c10a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.831+0000 7f5314800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f530c06bb40 0x7f530c10a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.831+0000 7f5314800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f530c109510 0x7f530c10ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.831+0000 7f5314800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f530c113670 con 0x7f530c06b180 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.831+0000 7f5314800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f530c1134f0 con 0x7f530c109510 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.831+0000 7f5314800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f530c1137f0 con 0x7f530c06bb40 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.831+0000 7f5311e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f530c06b180 0x7f530c10a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.831+0000 7f5311400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f530c06bb40 0x7f530c10a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.831+0000 7f5311400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f530c06bb40 0x7f530c10a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:38048/0 (socket says 172.21.15.27:38048) 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.831+0000 7f5311400700 1 -- 172.21.15.27:0/1433416322 learned_addr learned my addr 172.21.15.27:0/1433416322 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.831+0000 7f5312800700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f530c109510 0x7f530c10ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.832+0000 7f5311400700 1 -- 172.21.15.27:0/1433416322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f530c109510 msgr2=0x7f530c10ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.832+0000 7f5311400700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f530c109510 0x7f530c10ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.832+0000 7f5311400700 1 -- 172.21.15.27:0/1433416322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f530c06b180 msgr2=0x7f530c10a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.832+0000 7f5311400700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f530c06b180 0x7f530c10a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.832+0000 7f5311400700 1 -- 172.21.15.27:0/1433416322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530c1ac490 con 0x7f530c06bb40 2024-05-10T11:08:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.832+0000 7f5311e00700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f530c06b180 0x7f530c10a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:08:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.832+0000 7f5311400700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f530c06bb40 0x7f530c10a860 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0047b0 tx=0x7f52fc00b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.832+0000 7f5306a00700 1 -- 172.21.15.27:0/1433416322 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc027070 con 0x7f530c06bb40 2024-05-10T11:08:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.833+0000 7f5306a00700 1 -- 172.21.15.27:0/1433416322 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52fc022470 con 0x7f530c06bb40 2024-05-10T11:08:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.833+0000 7f5314800700 1 -- 172.21.15.27:0/1433416322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530c1ac720 con 0x7f530c06bb40 2024-05-10T11:08:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.833+0000 7f5306a00700 1 -- 172.21.15.27:0/1433416322 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc017650 con 0x7f530c06bb40 2024-05-10T11:08:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.833+0000 7f5314800700 1 -- 172.21.15.27:0/1433416322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530c1acb70 con 0x7f530c06bb40 2024-05-10T11:08:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.834+0000 7f5314800700 1 -- 172.21.15.27:0/1433416322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f530c104bd0 con 0x7f530c06bb40 2024-05-10T11:08:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.838+0000 7f5306a00700 1 -- 172.21.15.27:0/1433416322 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f52fc02a070 con 0x7f530c06bb40 2024-05-10T11:08:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.840+0000 7f5306a00700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f52f40b2880 0x7f52f40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.840+0000 7f5306a00700 1 -- 172.21.15.27:0/1433416322 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7f52fc0e34e0 con 0x7f530c06bb40 2024-05-10T11:08:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.840+0000 7f5311e00700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f52f40b2880 0x7f52f40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.840+0000 7f5306a00700 1 -- 172.21.15.27:0/1433416322 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f52fc013070 con 0x7f530c06bb40 2024-05-10T11:08:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:35.841+0000 7f5311e00700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f52f40b2880 0x7f52f40b4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5300009680 tx=0x7f5300009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:36.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.137+0000 7f5314800700 1 -- 172.21.15.27:0/1433416322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f530c1acdd0 con 0x7f530c06bb40 2024-05-10T11:08:36.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.137+0000 7f5306a00700 1 -- 172.21.15.27:0/1433416322 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f52fc0aaef0 con 0x7f530c06bb40 2024-05-10T11:08:36.138 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-10T11:08:36.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.141+0000 7f5314800700 1 -- 172.21.15.27:0/1433416322 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f52f40b2880 msgr2=0x7f52f40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:36.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.141+0000 7f5314800700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f52f40b2880 0x7f52f40b4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5300009680 tx=0x7f5300009260 comp rx=0 tx=0).stop 2024-05-10T11:08:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.141+0000 7f5314800700 1 -- 172.21.15.27:0/1433416322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f530c06bb40 msgr2=0x7f530c10a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.141+0000 7f5314800700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f530c06bb40 0x7f530c10a860 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0047b0 tx=0x7f52fc00b670 comp rx=0 tx=0).stop 2024-05-10T11:08:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.141+0000 7f5314800700 1 -- 172.21.15.27:0/1433416322 shutdown_connections 2024-05-10T11:08:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.141+0000 7f5314800700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f52f40b2880 0x7f52f40b4d40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.141+0000 7f5314800700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f530c109510 0x7f530c10ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.141+0000 7f5314800700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f530c06b180 0x7f530c10a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.141+0000 7f5314800700 1 --2- 172.21.15.27:0/1433416322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f530c06bb40 0x7f530c10a860 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.141+0000 7f5314800700 1 -- 172.21.15.27:0/1433416322 >> 172.21.15.27:0/1433416322 conn(0x7f530c079350 msgr2=0x7f530c10e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:36.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.141+0000 7f5314800700 1 -- 172.21.15.27:0/1433416322 shutdown_connections 2024-05-10T11:08:36.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:36.141+0000 7f5314800700 1 -- 172.21.15.27:0/1433416322 wait complete. 2024-05-10T11:08:36.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:36 smithi027 ceph-mon[30054]: pgmap v355: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:36.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:36 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:08:36.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:36 smithi076 ceph-mon[34152]: pgmap v355: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:36.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:36 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:08:36.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:36 smithi189 ceph-mon[34842]: pgmap v355: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:36.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:36 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:08:37.135 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-10T11:08:37.135 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-10T11:08:37.135 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-05-10T11:08:37.135 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-05-10T11:08:37.135 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-10T11:08:37.135 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-10T11:08:37.135 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-10T11:08:37.135 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-10T11:08:37.135 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-05-10T11:08:37.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:08:37.136 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-05-10T11:08:37.196 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-10T11:08:37.196 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-05-10T11:08:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41975sec preferred_lft 41975sec 2024-05-10T11:08:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-05-10T11:08:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-05-10T11:08:37.198 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-10T11:08:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-05-10T11:08:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-05-10T11:08:37.198 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-10T11:08:37.199 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-10T11:08:37.199 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-05-10T11:08:37.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-05-10T11:08:37.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-05-10T11:08:37.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-05-10T11:08:37.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-10T11:08:37.199 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-05-10T11:08:37.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-10T11:08:37.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:37 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1433416322' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-10T11:08:37.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:37 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1433416322' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-10T11:08:37.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:37 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1433416322' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-10T11:08:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:08:37.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:08:37.671 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-10T11:08:37.744 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-05-10T11:08:37.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:08:37.748 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-05-10T11:08:37.820 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-05-10T11:08:37.820 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-10T11:08:37.820 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-10T11:08:37.820 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-10T11:08:37.821 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-10T11:08:37.821 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-10T11:08:37.821 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-05-10T11:08:37.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-10T11:08:37.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-10T11:08:37.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-10T11:08:37.821 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-05-10T11:08:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-10T11:08:38.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:38 smithi027 ceph-mon[30054]: pgmap v356: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:38.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:38.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:38.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:38.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:38 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:38 smithi076 ceph-mon[34152]: pgmap v356: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:38 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:38.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:38 smithi189 ceph-mon[34842]: pgmap v356: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:38.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:38.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:38.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:38.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:38 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:38.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:08:38.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:08:38.805 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-05-10T11:08:38.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:08:38.850 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-05-10T11:08:38.936 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-10T11:08:38.936 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-05-10T11:08:38.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-10T11:08:38.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-05-10T11:08:38.937 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-05-10T11:08:39.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-10T11:08:39.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:08:39.155 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-05-10T11:08:39.155 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-10T11:08:39.155 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-05-10T11:08:39.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-05-10T11:08:39.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-10T11:08:39.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-05-10T11:08:39.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-05-10T11:08:39.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-05-10T11:08:39.156 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-05-10T11:08:39.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-10T11:08:39.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:08:39.480 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-10T11:08:39.480 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-05-10T11:08:39.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-05-10T11:08:39.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-05-10T11:08:39.481 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-05-10T11:08:39.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-10T11:08:39.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:08:39.675 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-05-10T11:08:39.675 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-10T11:08:39.675 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-05-10T11:08:39.719 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-05-10T11:08:39.719 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-10T11:08:39.719 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-05-10T11:08:39.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-05-10T11:08:39.944 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-05-10T11:08:40.001 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-05-10T11:08:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-05-10T11:08:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-05-10T11:08:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-05-10T11:08:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-05-10T11:08:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-05-10T11:08:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-05-10T11:08:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-05-10T11:08:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-05-10T11:08:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-05-10T11:08:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-05-10T11:08:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-05-10T11:08:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-05-10T11:08:40.003 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/0a7b2365f394cb43753cfa12f8172561ba62c9c0e8560021678ccf3d2fb6b617/merged 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/85f82d129773f9e6defaf9f1029946c013989d77b3ccff9dd60e0328cd96a3c0/merged 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/0 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/d272b0d1604ad474e5a0e9ee3d1528be3dd1dcbb688ec0aca6948ae34dc49c3d/merged 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/cb97bdb7089895f96eae4dcb3e316db89df1d76bd5ace090c6ce8cff537e0c3b/merged 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/b16563ba7f8f53169a6edc4b9d0141384fc91e1aa703572ce60060e66f373e20/merged 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/319661caa2c38429ee888dc52b2c730b471f482be3f96183ffc2800528999c9a/merged 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/d64d184aa9d047464b90a82c1fd3d512c94e077745bd1b17c057880a26b54a9d/merged 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/3c2a74de2c45bf7248267b1826e36fe61d2e0778c42861399e92a9697673ed24/merged 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/4ffddef7b0cd0fd4e6b191d2a35fb676ecdee936a94cbc5746d124bbf9de26bc/merged 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/dc3e84b08775ff8ee90190dd111365ca23e443460f6e296440605aa676d5cd2b/merged 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-10T11:08:40.004 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-10T11:08:40.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:08:40.007 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-05-10T11:08:40.058 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-10T11:08:40.058 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-05-10T11:08:40.063 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-05-10T11:08:40.130 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-05-10T11:08:40.170 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-05-10T11:08:40.171 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-05-10T11:08:40.171 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-05-10T11:08:40.171 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-05-10T11:08:40.171 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-05-10T11:08:40.172 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-05-10T11:08:40.172 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-05-10T11:08:40.172 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-05-10T11:08:40.172 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-05-10T11:08:40.172 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-05-10T11:08:40.173 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-05-10T11:08:40.173 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-05-10T11:08:40.173 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-05-10T11:08:40.173 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-05-10T11:08:40.173 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-05-10T11:08:40.173 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-05-10T11:08:40.174 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-05-10T11:08:40.174 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-05-10T11:08:40.174 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-10T11:08:40.174 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-05-10T11:08:40.175 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-05-10T11:08:40.175 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-10T11:08:40.175 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-05-10T11:08:40.175 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-05-10T11:08:40.175 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-05-10T11:08:40.175 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay 2024-05-10T11:08:40.176 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/0a7b2365f394cb43753cfa12f8172561ba62c9c0e8560021678ccf3d2fb6b617/merged 2024-05-10T11:08:40.176 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/85f82d129773f9e6defaf9f1029946c013989d77b3ccff9dd60e0328cd96a3c0/merged 2024-05-10T11:08:40.176 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/0 2024-05-10T11:08:40.176 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-05-10T11:08:40.176 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/d272b0d1604ad474e5a0e9ee3d1528be3dd1dcbb688ec0aca6948ae34dc49c3d/merged 2024-05-10T11:08:40.177 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/cb97bdb7089895f96eae4dcb3e316db89df1d76bd5ace090c6ce8cff537e0c3b/merged 2024-05-10T11:08:40.177 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/b16563ba7f8f53169a6edc4b9d0141384fc91e1aa703572ce60060e66f373e20/merged 2024-05-10T11:08:40.177 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/319661caa2c38429ee888dc52b2c730b471f482be3f96183ffc2800528999c9a/merged 2024-05-10T11:08:40.177 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/d64d184aa9d047464b90a82c1fd3d512c94e077745bd1b17c057880a26b54a9d/merged 2024-05-10T11:08:40.177 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/3c2a74de2c45bf7248267b1826e36fe61d2e0778c42861399e92a9697673ed24/merged 2024-05-10T11:08:40.177 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/4ffddef7b0cd0fd4e6b191d2a35fb676ecdee936a94cbc5746d124bbf9de26bc/merged 2024-05-10T11:08:40.178 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/dc3e84b08775ff8ee90190dd111365ca23e443460f6e296440605aa676d5cd2b/merged 2024-05-10T11:08:40.178 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-05-10T11:08:40.178 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-10T11:08:40.178 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-10T11:08:40.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:08:40.180 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-05-10T11:08:40.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi027.stderr:2024-05-10T11:08:40.250+0000 7f604bd3d480 -1 init, newargv = 0x5645f6ff58f0 newargc=15 2024-05-10T11:08:40.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi027.stderr:ceph-fuse[75959]: starting ceph client 2024-05-10T11:08:40.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi027.stderr:ceph-fuse[75959]: starting fuse 2024-05-10T11:08:40.273 INFO:teuthology.orchestra.run.smithi027.stdout:114 2024-05-10T11:08:40.273 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-05-10T11:08:40.274 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.admin")) 2024-05-10T11:08:40.275 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-05-10T11:08:40.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:40 smithi027 ceph-mon[30054]: pgmap v357: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:40.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:40.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:40 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:40 smithi076 ceph-mon[34152]: pgmap v357: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:40 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:40 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:40.469 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.admin.75959.asok 2024-05-10T11:08:40.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:08:40.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:40 smithi189 ceph-mon[34842]: pgmap v357: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:40.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:40 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:40.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:40 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:40.489 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75959.asok 2024-05-10T11:08:40.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:08:40.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75959.asok status 2024-05-10T11:08:40.718 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "d39fa08867836ac070c47cb66c782fa559fd480b", 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-3449-gd39fa088 (d39fa08867836ac070c47cb66c782fa559fd480b) squid (dev)", 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "admin", 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "75959", 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 14523, 2024-05-10T11:08:40.719 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 14523 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4102061993 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4102061993 2024-05-10T11:08:40.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:08:40.721 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.14523 172.21.15.27:0/4102061993", 2024-05-10T11:08:40.721 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "172.21.15.27:0/4102061993", 2024-05-10T11:08:40.721 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-05-10T11:08:40.721 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 17, 2024-05-10T11:08:40.721 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 88, 2024-05-10T11:08:40.721 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-05-10T11:08:40.721 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-05-10T11:08:40.721 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-05-10T11:08:40.721 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:08:40.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:08:40.734 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-05-10T11:08:40.734 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-05-10T11:08:40.779 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-05-10T11:08:40.779 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-05-10T11:08:40.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:08:40.780 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-05-10T11:08:40.916 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-05-10T11:08:41.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-10T11:08:41.060 INFO:teuthology.orchestra.run.smithi027.stdout:# file: . 2024-05-10T11:08:41.060 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-05-10T11:08:41.060 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:08:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:08:41.072 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T11:08:41.072 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-10T11:08:41.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd blocklist ls 2024-05-10T11:08:41.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:08:41.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:08:41.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:41.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:41 smithi027 ceph-mon[30054]: pgmap v358: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:41.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:08:41.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:08:41.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:41.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:41 smithi076 ceph-mon[34152]: pgmap v358: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:41.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:08:41.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:08:41.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:08:41.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:41 smithi189 ceph-mon[34842]: pgmap v358: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:41.607 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:08:44.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:43 smithi027 ceph-mon[30054]: pgmap v359: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:44.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:43 smithi076 ceph-mon[34152]: pgmap v359: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.192+0000 7fcadd600700 1 -- 172.21.15.27:0/1693889344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad810fb60 msgr2=0x7fcad810ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.192+0000 7fcadd600700 1 --2- 172.21.15.27:0/1693889344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad810fb60 0x7fcad810ff40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcac8009f90 tx=0x7fcac800b330 comp rx=0 tx=0).stop 2024-05-10T11:08:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.192+0000 7fcadd600700 1 -- 172.21.15.27:0/1693889344 shutdown_connections 2024-05-10T11:08:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.192+0000 7fcadd600700 1 --2- 172.21.15.27:0/1693889344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad810fb60 0x7fcad810ff40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.192+0000 7fcadd600700 1 --2- 172.21.15.27:0/1693889344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcad810a490 0x7fcad81169a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.192+0000 7fcadd600700 1 --2- 172.21.15.27:0/1693889344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad8109ad0 0x7fcad8109f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.192+0000 7fcadd600700 1 -- 172.21.15.27:0/1693889344 >> 172.21.15.27:0/1693889344 conn(0x7fcad8076c70 msgr2=0x7fcad81082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.192+0000 7fcadd600700 1 -- 172.21.15.27:0/1693889344 shutdown_connections 2024-05-10T11:08:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.192+0000 7fcadd600700 1 -- 172.21.15.27:0/1693889344 wait complete. 2024-05-10T11:08:44.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.193+0000 7fcadd600700 1 Processor -- start 2024-05-10T11:08:44.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.193+0000 7fcadd600700 1 -- start start 2024-05-10T11:08:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.194+0000 7fcadd600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad8109ad0 0x7fcad81a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.194+0000 7fcadd600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad810a490 0x7fcad81a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.194+0000 7fcadd600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcad810fb60 0x7fcad81a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.194+0000 7fcadd600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcad8119cc0 con 0x7fcad8109ad0 2024-05-10T11:08:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.194+0000 7fcadd600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcad8119b40 con 0x7fcad810fb60 2024-05-10T11:08:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.194+0000 7fcadd600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcad8119e40 con 0x7fcad810a490 2024-05-10T11:08:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.194+0000 7fcad6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad8109ad0 0x7fcad81a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.194+0000 7fcad6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad8109ad0 0x7fcad81a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57794/0 (socket says 172.21.15.27:57794) 2024-05-10T11:08:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.194+0000 7fcad6a00700 1 -- 172.21.15.27:0/4202881498 learned_addr learned my addr 172.21.15.27:0/4202881498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.194+0000 7fcad7400700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcad810fb60 0x7fcad81a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.194+0000 7fcace000700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad810a490 0x7fcad81a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:44.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.195+0000 7fcad6a00700 1 -- 172.21.15.27:0/4202881498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad810a490 msgr2=0x7fcad81a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:44.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.195+0000 7fcad6a00700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad810a490 0x7fcad81a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:44.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.195+0000 7fcad6a00700 1 -- 172.21.15.27:0/4202881498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcad810fb60 msgr2=0x7fcad81a4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:44.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.195+0000 7fcad6a00700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcad810fb60 0x7fcad81a4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.195+0000 7fcad6a00700 1 -- 172.21.15.27:0/4202881498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad81a5160 con 0x7fcad8109ad0 2024-05-10T11:08:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.195+0000 7fcad7400700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcad810fb60 0x7fcad81a4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:08:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.195+0000 7fcad6a00700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad8109ad0 0x7fcad81a0150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcac8009f60 tx=0x7fcac800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.195+0000 7fcacfe00700 1 -- 172.21.15.27:0/4202881498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8021070 con 0x7fcad8109ad0 2024-05-10T11:08:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.195+0000 7fcadd600700 1 -- 172.21.15.27:0/4202881498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad81a53f0 con 0x7fcad8109ad0 2024-05-10T11:08:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.196+0000 7fcacfe00700 1 -- 172.21.15.27:0/4202881498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcac8007ac0 con 0x7fcad8109ad0 2024-05-10T11:08:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.196+0000 7fcacfe00700 1 -- 172.21.15.27:0/4202881498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8005040 con 0x7fcad8109ad0 2024-05-10T11:08:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.196+0000 7fcadd600700 1 -- 172.21.15.27:0/4202881498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad81accd0 con 0x7fcad8109ad0 2024-05-10T11:08:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.197+0000 7fcadd600700 1 -- 172.21.15.27:0/4202881498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcad8112060 con 0x7fcad8109ad0 2024-05-10T11:08:44.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.199+0000 7fcacfe00700 1 -- 172.21.15.27:0/4202881498 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fcac8024070 con 0x7fcad8109ad0 2024-05-10T11:08:44.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.201+0000 7fcacfe00700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fcabc0b2880 0x7fcabc0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:44.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.201+0000 7fcacfe00700 1 -- 172.21.15.27:0/4202881498 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7fcac800f020 con 0x7fcad8109ad0 2024-05-10T11:08:44.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.201+0000 7fcace000700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fcabc0b2880 0x7fcabc0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:44.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.202+0000 7fcace000700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fcabc0b2880 0x7fcabc0b4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcad8078c00 tx=0x7fcac0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.204+0000 7fcacfe00700 1 -- 172.21.15.27:0/4202881498 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fcac802b020 con 0x7fcad8109ad0 2024-05-10T11:08:44.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:43 smithi189 ceph-mon[34842]: pgmap v359: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:44.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.463+0000 7fcadd600700 1 -- 172.21.15.27:0/4202881498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcad81a14b0 con 0x7fcad8109ad0 2024-05-10T11:08:44.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.464+0000 7fcacfe00700 1 -- 172.21.15.27:0/4202881498 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+590 (secure 0 0 0) 0x7fcac80ab760 con 0x7fcad8109ad0 2024-05-10T11:08:44.464 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2444542866 2024-05-11T10:59:20.613831+0000 2024-05-10T11:08:44.464 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/481366655 2024-05-11T10:59:20.613831+0000 2024-05-10T11:08:44.464 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/481366655 2024-05-11T10:59:20.613831+0000 2024-05-10T11:08:44.465 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2743539323 2024-05-11T10:59:20.613831+0000 2024-05-10T11:08:44.465 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3803325728 2024-05-11T10:58:45.437819+0000 2024-05-10T11:08:44.465 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/2058505174 2024-05-11T10:58:45.437819+0000 2024-05-10T11:08:44.465 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/2058505174 2024-05-11T10:58:45.437819+0000 2024-05-10T11:08:44.465 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1886129738 2024-05-11T10:58:45.437819+0000 2024-05-10T11:08:44.465 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1208957722 2024-05-11T10:59:20.613831+0000 2024-05-10T11:08:44.465 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4158621178 2024-05-11T10:58:45.437819+0000 2024-05-10T11:08:44.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.467+0000 7fcadd600700 1 -- 172.21.15.27:0/4202881498 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fcabc0b2880 msgr2=0x7fcabc0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:44.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.467+0000 7fcadd600700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fcabc0b2880 0x7fcabc0b4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcad8078c00 tx=0x7fcac0009310 comp rx=0 tx=0).stop 2024-05-10T11:08:44.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.467+0000 7fcadd600700 1 -- 172.21.15.27:0/4202881498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad8109ad0 msgr2=0x7fcad81a0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:44.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.467+0000 7fcadd600700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad8109ad0 0x7fcad81a0150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcac8009f60 tx=0x7fcac800ccb0 comp rx=0 tx=0).stop 2024-05-10T11:08:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.467+0000 7fcadd600700 1 -- 172.21.15.27:0/4202881498 shutdown_connections 2024-05-10T11:08:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.467+0000 7fcadd600700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fcabc0b2880 0x7fcabc0b4d40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.467+0000 7fcadd600700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcad810fb60 0x7fcad81a4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.467+0000 7fcadd600700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad8109ad0 0x7fcad81a0150 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.467+0000 7fcadd600700 1 --2- 172.21.15.27:0/4202881498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad810a490 0x7fcad81a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.467+0000 7fcadd600700 1 -- 172.21.15.27:0/4202881498 >> 172.21.15.27:0/4202881498 conn(0x7fcad8076c70 msgr2=0x7fcad8077290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.467+0000 7fcadd600700 1 -- 172.21.15.27:0/4202881498 shutdown_connections 2024-05-10T11:08:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:44.467+0000 7fcadd600700 1 -- 172.21.15.27:0/4202881498 wait complete. 2024-05-10T11:08:44.469 INFO:teuthology.orchestra.run.smithi027.stderr:listed 10 entries 2024-05-10T11:08:45.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:44 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4202881498' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-10T11:08:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:44 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4202881498' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-10T11:08:45.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:44 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4202881498' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-10T11:08:45.351 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T11:08:45.351 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-10T11:08:45.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd blocklist ls 2024-05-10T11:08:45.886 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:08:46.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:45 smithi027 ceph-mon[30054]: pgmap v360: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:46.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:45 smithi076 ceph-mon[34152]: pgmap v360: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:46.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:45 smithi189 ceph-mon[34842]: pgmap v360: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:08:48.116 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:47 smithi027 ceph-mon[30054]: pgmap v361: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-10T11:08:48.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:47 smithi076 ceph-mon[34152]: pgmap v361: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-10T11:08:48.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:47 smithi189 ceph-mon[34842]: pgmap v361: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-10T11:08:48.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.294+0000 7f9227200700 1 -- 172.21.15.27:0/3041839778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f922810b660 msgr2=0x7f922810ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:48.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.294+0000 7f9227200700 1 --2- 172.21.15.27:0/3041839778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f922810b660 0x7f922810ba40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9218009990 tx=0x7f9218009ca0 comp rx=0 tx=0).stop 2024-05-10T11:08:48.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.294+0000 7f9227200700 1 -- 172.21.15.27:0/3041839778 shutdown_connections 2024-05-10T11:08:48.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.294+0000 7f9227200700 1 --2- 172.21.15.27:0/3041839778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9228105f90 0x7f92281124a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:48.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.294+0000 7f9227200700 1 --2- 172.21.15.27:0/3041839778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f922810b660 0x7f922810ba40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:48.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.294+0000 7f9227200700 1 --2- 172.21.15.27:0/3041839778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92281055d0 0x7f9228105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:48.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.294+0000 7f9227200700 1 -- 172.21.15.27:0/3041839778 >> 172.21.15.27:0/3041839778 conn(0x7f92281010c0 msgr2=0x7f92281034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:48.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.294+0000 7f9227200700 1 -- 172.21.15.27:0/3041839778 shutdown_connections 2024-05-10T11:08:48.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.294+0000 7f9227200700 1 -- 172.21.15.27:0/3041839778 wait complete. 2024-05-10T11:08:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.295+0000 7f9227200700 1 Processor -- start 2024-05-10T11:08:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.296+0000 7f9227200700 1 -- start start 2024-05-10T11:08:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.296+0000 7f9227200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92281055d0 0x7f92281a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.296+0000 7f9227200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105f90 0x7f92281a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.296+0000 7f9227200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922810b660 0x7f922819ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.296+0000 7f9227200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9228115800 con 0x7f922810b660 2024-05-10T11:08:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.296+0000 7f9227200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9228115680 con 0x7f92281055d0 2024-05-10T11:08:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.296+0000 7f9227200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9228115980 con 0x7f9228105f90 2024-05-10T11:08:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.296+0000 7f9225e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92281055d0 0x7f92281a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.296+0000 7f9226800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922810b660 0x7f922819ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.296+0000 7f9225e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92281055d0 0x7f92281a18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:39274/0 (socket says 172.21.15.27:39274) 2024-05-10T11:08:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.296+0000 7f9225400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105f90 0x7f92281a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:48.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.296+0000 7f9225e00700 1 -- 172.21.15.27:0/498476245 learned_addr learned my addr 172.21.15.27:0/498476245 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:48.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.297+0000 7f9225e00700 1 -- 172.21.15.27:0/498476245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105f90 msgr2=0x7f92281a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.297+0000 7f9225e00700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105f90 0x7f92281a1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.297+0000 7f9225e00700 1 -- 172.21.15.27:0/498476245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922810b660 msgr2=0x7f922819ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.297+0000 7f9225e00700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922810b660 0x7f922819ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.297+0000 7f9225e00700 1 -- 172.21.15.27:0/498476245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922819c1c0 con 0x7f92281055d0 2024-05-10T11:08:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.297+0000 7f9225400700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105f90 0x7f92281a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:08:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.297+0000 7f9225e00700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92281055d0 0x7f92281a18a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9218000c00 tx=0x7f92180057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.297+0000 7f921ea00700 1 -- 172.21.15.27:0/498476245 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9218027070 con 0x7f92281055d0 2024-05-10T11:08:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.298+0000 7f9227200700 1 -- 172.21.15.27:0/498476245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f922819c450 con 0x7f92281055d0 2024-05-10T11:08:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.298+0000 7f921ea00700 1 -- 172.21.15.27:0/498476245 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9218022470 con 0x7f92281055d0 2024-05-10T11:08:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.298+0000 7f921ea00700 1 -- 172.21.15.27:0/498476245 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9218017650 con 0x7f92281055d0 2024-05-10T11:08:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.298+0000 7f9227200700 1 -- 172.21.15.27:0/498476245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92281a8770 con 0x7f92281055d0 2024-05-10T11:08:48.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.299+0000 7f9227200700 1 -- 172.21.15.27:0/498476245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f922806a190 con 0x7f92281055d0 2024-05-10T11:08:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.301+0000 7f921ea00700 1 -- 172.21.15.27:0/498476245 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f921802a070 con 0x7f92281055d0 2024-05-10T11:08:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.302+0000 7f921ea00700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f920c0b2900 0x7f920c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.303+0000 7f921ea00700 1 -- 172.21.15.27:0/498476245 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7f92180e3200 con 0x7f92281055d0 2024-05-10T11:08:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.303+0000 7f9225400700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f920c0b2900 0x7f920c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.304+0000 7f9225400700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f920c0b2900 0x7f920c0b4dc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f922810ac90 tx=0x7f9214009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:48.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.306+0000 7f921ea00700 1 -- 172.21.15.27:0/498476245 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9218013070 con 0x7f92281055d0 2024-05-10T11:08:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.569+0000 7f9227200700 1 -- 172.21.15.27:0/498476245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f922819cf90 con 0x7f92281055d0 2024-05-10T11:08:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.570+0000 7f921ea00700 1 -- 172.21.15.27:0/498476245 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+590 (secure 0 0 0) 0x7f92180aac10 con 0x7f92281055d0 2024-05-10T11:08:48.571 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2444542866 2024-05-11T10:59:20.613831+0000 2024-05-10T11:08:48.571 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/481366655 2024-05-11T10:59:20.613831+0000 2024-05-10T11:08:48.571 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/481366655 2024-05-11T10:59:20.613831+0000 2024-05-10T11:08:48.571 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2743539323 2024-05-11T10:59:20.613831+0000 2024-05-10T11:08:48.571 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3803325728 2024-05-11T10:58:45.437819+0000 2024-05-10T11:08:48.571 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/2058505174 2024-05-11T10:58:45.437819+0000 2024-05-10T11:08:48.572 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/2058505174 2024-05-11T10:58:45.437819+0000 2024-05-10T11:08:48.572 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1886129738 2024-05-11T10:58:45.437819+0000 2024-05-10T11:08:48.572 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1208957722 2024-05-11T10:59:20.613831+0000 2024-05-10T11:08:48.572 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4158621178 2024-05-11T10:58:45.437819+0000 2024-05-10T11:08:48.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.573+0000 7f9227200700 1 -- 172.21.15.27:0/498476245 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f920c0b2900 msgr2=0x7f920c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:48.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.573+0000 7f9227200700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f920c0b2900 0x7f920c0b4dc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f922810ac90 tx=0x7f9214009260 comp rx=0 tx=0).stop 2024-05-10T11:08:48.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.573+0000 7f9227200700 1 -- 172.21.15.27:0/498476245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92281055d0 msgr2=0x7f92281a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:48.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.573+0000 7f9227200700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92281055d0 0x7f92281a18a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9218000c00 tx=0x7f92180057c0 comp rx=0 tx=0).stop 2024-05-10T11:08:48.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.574+0000 7f9227200700 1 -- 172.21.15.27:0/498476245 shutdown_connections 2024-05-10T11:08:48.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.574+0000 7f9227200700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f920c0b2900 0x7f920c0b4dc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:48.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.574+0000 7f9227200700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922810b660 0x7f922819ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:48.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.574+0000 7f9227200700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92281055d0 0x7f92281a18a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:48.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.574+0000 7f9227200700 1 --2- 172.21.15.27:0/498476245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105f90 0x7f92281a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:48.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.574+0000 7f9227200700 1 -- 172.21.15.27:0/498476245 >> 172.21.15.27:0/498476245 conn(0x7f92281010c0 msgr2=0x7f9228102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:48.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.574+0000 7f9227200700 1 -- 172.21.15.27:0/498476245 shutdown_connections 2024-05-10T11:08:48.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:48.574+0000 7f9227200700 1 -- 172.21.15.27:0/498476245 wait complete. 2024-05-10T11:08:48.576 INFO:teuthology.orchestra.run.smithi027.stderr:listed 10 entries 2024-05-10T11:08:49.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:48 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/498476245' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-10T11:08:49.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:48 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/498476245' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-10T11:08:49.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:48 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/498476245' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-10T11:08:49.538 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-05-10T11:08:49.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:08:49.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-05-10T11:08:49.624 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi027.stderr:ceph-fuse[75959]: fuse finished with error 0 and tester_r 0 2024-05-10T11:08:49.628 INFO:teuthology.orchestra.run:waiting for 300 2024-05-10T11:08:50.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:49 smithi076 ceph-mon[34152]: pgmap v362: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-10T11:08:50.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:49 smithi189 ceph-mon[34842]: pgmap v362: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-10T11:08:50.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:49 smithi027 ceph-mon[30054]: pgmap v362: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-10T11:08:52.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:51 smithi076 ceph-mon[34152]: pgmap v363: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-10T11:08:52.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:51 smithi189 ceph-mon[34842]: pgmap v363: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-10T11:08:52.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:51 smithi027 ceph-mon[30054]: pgmap v363: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-10T11:08:54.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:53 smithi076 ceph-mon[34152]: pgmap v364: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-10T11:08:54.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:53 smithi189 ceph-mon[34842]: pgmap v364: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-10T11:08:54.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:53 smithi027 ceph-mon[30054]: pgmap v364: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-10T11:08:55.730 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-05-10T11:08:55.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:08:55.731 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-05-10T11:08:55.806 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:08:56.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:55 smithi076 ceph-mon[34152]: pgmap v365: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-10T11:08:56.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:55 smithi189 ceph-mon[34842]: pgmap v365: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-10T11:08:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:08:56.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:55 smithi027 ceph-mon[30054]: pgmap v365: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-10T11:08:58.098 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:57 smithi027 ceph-mon[30054]: pgmap v366: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-10T11:08:58.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:57 smithi076 ceph-mon[34152]: pgmap v366: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-10T11:08:58.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:57 smithi189 ceph-mon[34842]: pgmap v366: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-10T11:08:58.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.671+0000 7f5bc2600700 1 -- 172.21.15.27:0/2204141546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bbc102450 msgr2=0x7f5bbc110340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:58.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.671+0000 7f5bc2600700 1 --2- 172.21.15.27:0/2204141546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bbc102450 0x7f5bbc110340 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400b0a0 tx=0x7f5bb400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:08:58.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.671+0000 7f5bc2600700 1 -- 172.21.15.27:0/2204141546 shutdown_connections 2024-05-10T11:08:58.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.671+0000 7f5bc2600700 1 --2- 172.21.15.27:0/2204141546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bbc102450 0x7f5bbc110340 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:58.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.671+0000 7f5bc2600700 1 --2- 172.21.15.27:0/2204141546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bbc109520 0x7f5bbc109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:58.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.671+0000 7f5bc2600700 1 --2- 172.21.15.27:0/2204141546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bbc101a90 0x7f5bbc101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:58.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.671+0000 7f5bc2600700 1 -- 172.21.15.27:0/2204141546 >> 172.21.15.27:0/2204141546 conn(0x7f5bbc078a10 msgr2=0x7f5bbc0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:58.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.671+0000 7f5bc2600700 1 -- 172.21.15.27:0/2204141546 shutdown_connections 2024-05-10T11:08:58.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.671+0000 7f5bc2600700 1 -- 172.21.15.27:0/2204141546 wait complete. 2024-05-10T11:08:58.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.672+0000 7f5bc2600700 1 Processor -- start 2024-05-10T11:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.673+0000 7f5bc2600700 1 -- start start 2024-05-10T11:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.673+0000 7f5bc2600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bbc101a90 0x7f5bbc19bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.673+0000 7f5bc2600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bbc102450 0x7f5bbc19c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.673+0000 7f5bc2600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bbc109520 0x7f5bbc1a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.673+0000 7f5bc2600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5bbc113680 con 0x7f5bbc102450 2024-05-10T11:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.673+0000 7f5bc2600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5bbc113500 con 0x7f5bbc101a90 2024-05-10T11:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.673+0000 7f5bc2600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5bbc113800 con 0x7f5bbc109520 2024-05-10T11:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.674+0000 7f5bbb400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bbc101a90 0x7f5bbc19bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.674+0000 7f5bbbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bbc109520 0x7f5bbc1a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.674+0000 7f5bbb400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bbc101a90 0x7f5bbc19bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33310/0 (socket says 172.21.15.27:33310) 2024-05-10T11:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.674+0000 7f5bbb400700 1 -- 172.21.15.27:0/3258973958 learned_addr learned my addr 172.21.15.27:0/3258973958 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:08:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.674+0000 7f5bb3e00700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bbc102450 0x7f5bbc19c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.674+0000 7f5bbbe00700 1 -- 172.21.15.27:0/3258973958 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bbc101a90 msgr2=0x7f5bbc19bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.674+0000 7f5bbbe00700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bbc101a90 0x7f5bbc19bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.674+0000 7f5bbbe00700 1 -- 172.21.15.27:0/3258973958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bbc102450 msgr2=0x7f5bbc19c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.674+0000 7f5bbbe00700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bbc102450 0x7f5bbc19c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.674+0000 7f5bbbe00700 1 -- 172.21.15.27:0/3258973958 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bbc1a0d60 con 0x7f5bbc109520 2024-05-10T11:08:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.674+0000 7f5bbbe00700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bbc109520 0x7f5bbc1a0620 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400b070 tx=0x7f5bb4004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.675+0000 7f5bb8c00700 1 -- 172.21.15.27:0/3258973958 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4021070 con 0x7f5bbc109520 2024-05-10T11:08:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.675+0000 7f5bc2600700 1 -- 172.21.15.27:0/3258973958 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bbc1a0ff0 con 0x7f5bbc109520 2024-05-10T11:08:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.675+0000 7f5bb8c00700 1 -- 172.21.15.27:0/3258973958 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bb4007e30 con 0x7f5bbc109520 2024-05-10T11:08:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.675+0000 7f5bb8c00700 1 -- 172.21.15.27:0/3258973958 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb40045f0 con 0x7f5bbc109520 2024-05-10T11:08:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.675+0000 7f5bc2600700 1 -- 172.21.15.27:0/3258973958 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bbc1a88d0 con 0x7f5bbc109520 2024-05-10T11:08:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.676+0000 7f5bc2600700 1 -- 172.21.15.27:0/3258973958 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bbc06a190 con 0x7f5bbc109520 2024-05-10T11:08:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.678+0000 7f5bb8c00700 1 -- 172.21.15.27:0/3258973958 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5bb4007980 con 0x7f5bbc109520 2024-05-10T11:08:58.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.679+0000 7f5bb8c00700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5ba00b2900 0x7f5ba00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:08:58.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.679+0000 7f5bb8c00700 1 -- 172.21.15.27:0/3258973958 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7f5bb400f020 con 0x7f5bbc109520 2024-05-10T11:08:58.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.679+0000 7f5bbb400700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5ba00b2900 0x7f5ba00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:08:58.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.681+0000 7f5bbb400700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5ba00b2900 0x7f5ba00b4dc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5bac009680 tx=0x7f5bac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:08:58.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.682+0000 7f5bb8c00700 1 -- 172.21.15.27:0/3258973958 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5bb402a020 con 0x7f5bbc109520 2024-05-10T11:08:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.921+0000 7f5bc2600700 1 -- 172.21.15.27:0/3258973958 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5bbc1a8aa0 con 0x7f5bbc109520 2024-05-10T11:08:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.923+0000 7f5bb8c00700 1 -- 172.21.15.27:0/3258973958 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24497 (secure 0 0 0) 0x7f5bb40ab660 con 0x7f5bbc109520 2024-05-10T11:08:58.924 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:08:58.924 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":88,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:08:22.020361+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6},{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:08:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.926+0000 7f5bc2600700 1 -- 172.21.15.27:0/3258973958 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5ba00b2900 msgr2=0x7f5ba00b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.926+0000 7f5bc2600700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5ba00b2900 0x7f5ba00b4dc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5bac009680 tx=0x7f5bac009260 comp rx=0 tx=0).stop 2024-05-10T11:08:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.926+0000 7f5bc2600700 1 -- 172.21.15.27:0/3258973958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bbc109520 msgr2=0x7f5bbc1a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:08:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.926+0000 7f5bc2600700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bbc109520 0x7f5bbc1a0620 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400b070 tx=0x7f5bb4004c80 comp rx=0 tx=0).stop 2024-05-10T11:08:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.927+0000 7f5bc2600700 1 -- 172.21.15.27:0/3258973958 shutdown_connections 2024-05-10T11:08:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.927+0000 7f5bc2600700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5ba00b2900 0x7f5ba00b4dc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.927+0000 7f5bc2600700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bbc109520 0x7f5bbc1a0620 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.927+0000 7f5bc2600700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bbc101a90 0x7f5bbc19bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.927+0000 7f5bc2600700 1 --2- 172.21.15.27:0/3258973958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bbc102450 0x7f5bbc19c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:08:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.927+0000 7f5bc2600700 1 -- 172.21.15.27:0/3258973958 >> 172.21.15.27:0/3258973958 conn(0x7f5bbc078a10 msgr2=0x7f5bbc1034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:08:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.927+0000 7f5bc2600700 1 -- 172.21.15.27:0/3258973958 shutdown_connections 2024-05-10T11:08:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:08:58.927+0000 7f5bc2600700 1 -- 172.21.15.27:0/3258973958 wait complete. 2024-05-10T11:08:59.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:09:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:59 smithi027 ceph-mon[30054]: pgmap v367: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-10T11:09:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:08:59 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3258973958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:09:00.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:59 smithi076 ceph-mon[34152]: pgmap v367: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-10T11:09:00.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:08:59 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3258973958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:09:00.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:59 smithi189 ceph-mon[34842]: pgmap v367: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-10T11:09:00.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:08:59 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3258973958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:09:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:02.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:01 smithi027 ceph-mon[30054]: pgmap v368: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-10T11:09:02.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:01 smithi076 ceph-mon[34152]: pgmap v368: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-10T11:09:02.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:01 smithi189 ceph-mon[34842]: pgmap v368: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-10T11:09:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.693+0000 7f6ff9c00700 1 -- 172.21.15.27:0/1792999440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ff4103750 msgr2=0x7f6ff4103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.693+0000 7f6ff9c00700 1 --2- 172.21.15.27:0/1792999440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ff4103750 0x7f6ff4103b30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40099c0 tx=0x7f6fe4009cd0 comp rx=0 tx=0).stop 2024-05-10T11:09:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.693+0000 7f6ff9c00700 1 -- 172.21.15.27:0/1792999440 shutdown_connections 2024-05-10T11:09:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.693+0000 7f6ff9c00700 1 --2- 172.21.15.27:0/1792999440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ff4108850 0x7f6ff41103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.693+0000 7f6ff9c00700 1 --2- 172.21.15.27:0/1792999440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ff4103750 0x7f6ff4103b30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.693+0000 7f6ff9c00700 1 --2- 172.21.15.27:0/1792999440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4104100 0x7f6ff4108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.693+0000 7f6ff9c00700 1 -- 172.21.15.27:0/1792999440 >> 172.21.15.27:0/1792999440 conn(0x7f6ff407a600 msgr2=0x7f6ff407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:02.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.694+0000 7f6ff9c00700 1 -- 172.21.15.27:0/1792999440 shutdown_connections 2024-05-10T11:09:02.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.694+0000 7f6ff9c00700 1 -- 172.21.15.27:0/1792999440 wait complete. 2024-05-10T11:09:02.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.695+0000 7f6ff9c00700 1 Processor -- start 2024-05-10T11:09:02.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.695+0000 7f6ff9c00700 1 -- start start 2024-05-10T11:09:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.696+0000 7f6ff9c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ff4103750 0x7f6ff40757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.696+0000 7f6ff9c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4104100 0x7f6ff4075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.696+0000 7f6ff9c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ff4108850 0x7f6ff4076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.696+0000 7f6ff9c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff4113670 con 0x7f6ff4108850 2024-05-10T11:09:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.696+0000 7f6ff9c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41134f0 con 0x7f6ff4103750 2024-05-10T11:09:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.696+0000 7f6ff9c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41137f0 con 0x7f6ff4104100 2024-05-10T11:09:02.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.697+0000 7f6ff3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ff4108850 0x7f6ff4076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:02.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.697+0000 7f6ff2a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ff4103750 0x7f6ff40757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:02.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.697+0000 7f6ff2000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4104100 0x7f6ff4075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:02.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.697+0000 7f6ff2000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4104100 0x7f6ff4075ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41824/0 (socket says 172.21.15.27:41824) 2024-05-10T11:09:02.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.697+0000 7f6ff2a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ff4103750 0x7f6ff40757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33326/0 (socket says 172.21.15.27:33326) 2024-05-10T11:09:02.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.697+0000 7f6ff3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ff4108850 0x7f6ff4076220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36154/0 (socket says 172.21.15.27:36154) 2024-05-10T11:09:02.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.697+0000 7f6ff2000700 1 -- 172.21.15.27:0/49568690 learned_addr learned my addr 172.21.15.27:0/49568690 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:02.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.697+0000 7f6ff2000700 1 -- 172.21.15.27:0/49568690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ff4103750 msgr2=0x7f6ff40757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:02.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.697+0000 7f6ff2000700 1 --2- 172.21.15.27:0/49568690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ff4103750 0x7f6ff40757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:02.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.697+0000 7f6ff2000700 1 -- 172.21.15.27:0/49568690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ff4108850 msgr2=0x7f6ff4076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:02.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.697+0000 7f6ff2000700 1 --2- 172.21.15.27:0/49568690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ff4108850 0x7f6ff4076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:02.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.697+0000 7f6ff2000700 1 -- 172.21.15.27:0/49568690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff41a8070 con 0x7f6ff4104100 2024-05-10T11:09:02.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.698+0000 7f6ff2000700 1 --2- 172.21.15.27:0/49568690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4104100 0x7f6ff4075ce0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6fe000e9a0 tx=0x7f6fe000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:02.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.698+0000 7f6feb400700 1 -- 172.21.15.27:0/49568690 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe000c9c0 con 0x7f6ff4104100 2024-05-10T11:09:02.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.698+0000 7f6ff9c00700 1 -- 172.21.15.27:0/49568690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff41a8360 con 0x7f6ff4104100 2024-05-10T11:09:02.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.698+0000 7f6feb400700 1 -- 172.21.15.27:0/49568690 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fe000cb60 con 0x7f6ff4104100 2024-05-10T11:09:02.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.698+0000 7f6feb400700 1 -- 172.21.15.27:0/49568690 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0018680 con 0x7f6ff4104100 2024-05-10T11:09:02.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.698+0000 7f6ff9c00700 1 -- 172.21.15.27:0/49568690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff41a88a0 con 0x7f6ff4104100 2024-05-10T11:09:02.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.699+0000 7f6ff9c00700 1 -- 172.21.15.27:0/49568690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ff410ba90 con 0x7f6ff4104100 2024-05-10T11:09:02.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.703+0000 7f6feb400700 1 -- 172.21.15.27:0/49568690 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f6fe0010430 con 0x7f6ff4104100 2024-05-10T11:09:02.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.704+0000 7f6feb400700 1 --2- 172.21.15.27:0/49568690 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6fd80b2880 0x7f6fd80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:02.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.704+0000 7f6feb400700 1 -- 172.21.15.27:0/49568690 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7f6fe00e2840 con 0x7f6ff4104100 2024-05-10T11:09:02.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.705+0000 7f6ff2a00700 1 --2- 172.21.15.27:0/49568690 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6fd80b2880 0x7f6fd80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:02.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.705+0000 7f6ff2a00700 1 --2- 172.21.15.27:0/49568690 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6fd80b2880 0x7f6fd80b4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4000c00 tx=0x7f6fe4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:02.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.709+0000 7f6feb400700 1 -- 172.21.15.27:0/49568690 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6fe00aa1c0 con 0x7f6ff4104100 2024-05-10T11:09:02.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.954+0000 7f6ff9c00700 1 -- 172.21.15.27:0/49568690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ff410b470 con 0x7f6ff4104100 2024-05-10T11:09:02.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.956+0000 7f6feb400700 1 -- 172.21.15.27:0/49568690 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24497 (secure 0 0 0) 0x7f6fe00b2240 con 0x7f6ff4104100 2024-05-10T11:09:02.956 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:09:02.957 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":88,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:08:22.020361+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6},{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:09:02.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.960+0000 7f6ff9c00700 1 -- 172.21.15.27:0/49568690 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6fd80b2880 msgr2=0x7f6fd80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.960+0000 7f6ff9c00700 1 --2- 172.21.15.27:0/49568690 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6fd80b2880 0x7f6fd80b4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4000c00 tx=0x7f6fe4005fb0 comp rx=0 tx=0).stop 2024-05-10T11:09:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.960+0000 7f6ff9c00700 1 -- 172.21.15.27:0/49568690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4104100 msgr2=0x7f6ff4075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.960+0000 7f6ff9c00700 1 --2- 172.21.15.27:0/49568690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4104100 0x7f6ff4075ce0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6fe000e9a0 tx=0x7f6fe000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:09:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.960+0000 7f6ff9c00700 1 -- 172.21.15.27:0/49568690 shutdown_connections 2024-05-10T11:09:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.960+0000 7f6ff9c00700 1 --2- 172.21.15.27:0/49568690 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6fd80b2880 0x7f6fd80b4d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.960+0000 7f6ff9c00700 1 --2- 172.21.15.27:0/49568690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ff4108850 0x7f6ff4076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.960+0000 7f6ff9c00700 1 --2- 172.21.15.27:0/49568690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ff4103750 0x7f6ff40757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:02.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.960+0000 7f6ff9c00700 1 --2- 172.21.15.27:0/49568690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4104100 0x7f6ff4075ce0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:02.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.960+0000 7f6ff9c00700 1 -- 172.21.15.27:0/49568690 >> 172.21.15.27:0/49568690 conn(0x7f6ff407a600 msgr2=0x7f6ff41099a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:02.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.960+0000 7f6ff9c00700 1 -- 172.21.15.27:0/49568690 shutdown_connections 2024-05-10T11:09:02.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:02.960+0000 7f6ff9c00700 1 -- 172.21.15.27:0/49568690 wait complete. 2024-05-10T11:09:04.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-05-10T11:09:04.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:03 smithi076 ceph-mon[34152]: pgmap v369: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:04.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:03 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/49568690' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:09:04.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:03 smithi189 ceph-mon[34842]: pgmap v369: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:04.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:03 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/49568690' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:09:04.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:03 smithi027 ceph-mon[30054]: pgmap v369: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:04.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:03 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/49568690' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:09:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:06.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:05 smithi027 ceph-mon[30054]: pgmap v370: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:06.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:05 smithi076 ceph-mon[34152]: pgmap v370: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:06.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:05 smithi189 ceph-mon[34842]: pgmap v370: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:06.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.908+0000 7f33ef200700 1 -- 172.21.15.27:0/2557330786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33e806c3d0 msgr2=0x7f33e8110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.908+0000 7f33ef200700 1 --2- 172.21.15.27:0/2557330786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33e806c3d0 0x7f33e8110320 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009f90 tx=0x7f33e000b330 comp rx=0 tx=0).stop 2024-05-10T11:09:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.908+0000 7f33ef200700 1 -- 172.21.15.27:0/2557330786 shutdown_connections 2024-05-10T11:09:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.908+0000 7f33ef200700 1 --2- 172.21.15.27:0/2557330786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33e806c3d0 0x7f33e8110320 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.908+0000 7f33ef200700 1 --2- 172.21.15.27:0/2557330786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f33e8109500 0x7f33e81098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.908+0000 7f33ef200700 1 --2- 172.21.15.27:0/2557330786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33e806ba10 0x7f33e806be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.908+0000 7f33ef200700 1 -- 172.21.15.27:0/2557330786 >> 172.21.15.27:0/2557330786 conn(0x7f33e8079310 msgr2=0x7f33e8079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.908+0000 7f33ef200700 1 -- 172.21.15.27:0/2557330786 shutdown_connections 2024-05-10T11:09:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.908+0000 7f33ef200700 1 -- 172.21.15.27:0/2557330786 wait complete. 2024-05-10T11:09:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.909+0000 7f33ef200700 1 Processor -- start 2024-05-10T11:09:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.909+0000 7f33ef200700 1 -- start start 2024-05-10T11:09:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33ef200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33e806ba10 0x7f33e81a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33ef200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33e806c3d0 0x7f33e81a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33ef200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f33e8109500 0x7f33e819bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33ef200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33e8113610 con 0x7f33e806c3d0 2024-05-10T11:09:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33ef200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f33e8113490 con 0x7f33e8109500 2024-05-10T11:09:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33ef200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f33e8113790 con 0x7f33e806ba10 2024-05-10T11:09:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33e806ba10 0x7f33e81a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33e806ba10 0x7f33e81a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49488/0 (socket says 172.21.15.27:49488) 2024-05-10T11:09:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33e7e00700 1 -- 172.21.15.27:0/4206392939 learned_addr learned my addr 172.21.15.27:0/4206392939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33e7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33e806c3d0 0x7f33e81a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33ed200700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f33e8109500 0x7f33e819bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33e7e00700 1 -- 172.21.15.27:0/4206392939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f33e8109500 msgr2=0x7f33e819bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:06.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33e7e00700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f33e8109500 0x7f33e819bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:06.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33e7e00700 1 -- 172.21.15.27:0/4206392939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33e806c3d0 msgr2=0x7f33e81a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:06.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33e7e00700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33e806c3d0 0x7f33e81a1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:06.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33e7e00700 1 -- 172.21.15.27:0/4206392939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e819c3f0 con 0x7f33e806ba10 2024-05-10T11:09:06.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.910+0000 7f33ed200700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f33e8109500 0x7f33e819bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:09:06.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.911+0000 7f33e7e00700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33e806ba10 0x7f33e81a1a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33d800e9a0 tx=0x7f33d800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:06.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.911+0000 7f33e4c00700 1 -- 172.21.15.27:0/4206392939 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d800cbd0 con 0x7f33e806ba10 2024-05-10T11:09:06.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.911+0000 7f33e4c00700 1 -- 172.21.15.27:0/4206392939 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33d800cd70 con 0x7f33e806ba10 2024-05-10T11:09:06.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.911+0000 7f33ef200700 1 -- 172.21.15.27:0/4206392939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33e819c6e0 con 0x7f33e806ba10 2024-05-10T11:09:06.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.911+0000 7f33e4c00700 1 -- 172.21.15.27:0/4206392939 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d8018680 con 0x7f33e806ba10 2024-05-10T11:09:06.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.911+0000 7f33ef200700 1 -- 172.21.15.27:0/4206392939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33e81a89c0 con 0x7f33e806ba10 2024-05-10T11:09:06.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.913+0000 7f33ef200700 1 -- 172.21.15.27:0/4206392939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33e8104bc0 con 0x7f33e806ba10 2024-05-10T11:09:06.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.915+0000 7f33e4c00700 1 -- 172.21.15.27:0/4206392939 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f33d8010430 con 0x7f33e806ba10 2024-05-10T11:09:06.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.916+0000 7f33e4c00700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f33cc0b2880 0x7f33cc0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:06.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.916+0000 7f33e4c00700 1 -- 172.21.15.27:0/4206392939 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7f33d80e2760 con 0x7f33e806ba10 2024-05-10T11:09:06.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.916+0000 7f33e7400700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f33cc0b2880 0x7f33cc0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:06.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.917+0000 7f33e7400700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f33cc0b2880 0x7f33cc0b4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f33e819d260 tx=0x7f33d4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:06.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:06.920+0000 7f33e4c00700 1 -- 172.21.15.27:0/4206392939 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f33d80aa0e0 con 0x7f33e806ba10 2024-05-10T11:09:07.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.217+0000 7f33ef200700 1 -- 172.21.15.27:0/4206392939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f33e81a8b90 con 0x7f33e806ba10 2024-05-10T11:09:07.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.927+0000 7f33e4c00700 1 -- 172.21.15.27:0/4206392939 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f33d80b2160 con 0x7f33e806ba10 2024-05-10T11:09:07.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.932+0000 7f33ef200700 1 -- 172.21.15.27:0/4206392939 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f33cc0b2880 msgr2=0x7f33cc0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:07.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.932+0000 7f33ef200700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f33cc0b2880 0x7f33cc0b4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f33e819d260 tx=0x7f33d4009310 comp rx=0 tx=0).stop 2024-05-10T11:09:07.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.932+0000 7f33ef200700 1 -- 172.21.15.27:0/4206392939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33e806ba10 msgr2=0x7f33e81a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:07.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.932+0000 7f33ef200700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33e806ba10 0x7f33e81a1a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33d800e9a0 tx=0x7f33d800ed60 comp rx=0 tx=0).stop 2024-05-10T11:09:07.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.932+0000 7f33ef200700 1 -- 172.21.15.27:0/4206392939 shutdown_connections 2024-05-10T11:09:07.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.932+0000 7f33ef200700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f33cc0b2880 0x7f33cc0b4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:07.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.932+0000 7f33ef200700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33e806ba10 0x7f33e81a1a40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.932+0000 7f33ef200700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f33e8109500 0x7f33e819bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.932+0000 7f33ef200700 1 --2- 172.21.15.27:0/4206392939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33e806c3d0 0x7f33e81a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.932+0000 7f33ef200700 1 -- 172.21.15.27:0/4206392939 >> 172.21.15.27:0/4206392939 conn(0x7f33e8079310 msgr2=0x7f33e810e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.932+0000 7f33ef200700 1 -- 172.21.15.27:0/4206392939 shutdown_connections 2024-05-10T11:09:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:07.932+0000 7f33ef200700 1 -- 172.21.15.27:0/4206392939 wait complete. 2024-05-10T11:09:08.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:07 smithi189 ceph-mon[34842]: pgmap v371: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:08.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:07 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4206392939' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-10T11:09:08.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:07 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-10T11:09:08.311 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:07 smithi027 ceph-mon[30054]: pgmap v371: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:08.312 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:07 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4206392939' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-10T11:09:08.312 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:07 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-10T11:09:08.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:07 smithi076 ceph-mon[34152]: pgmap v371: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:08.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:07 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4206392939' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-10T11:09:08.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:07 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-10T11:09:08.847 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-05-10T11:09:08.847 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-05-10T11:09:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:08 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-10T11:09:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:08 smithi027 ceph-mon[30054]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:09:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:08 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-10T11:09:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:08 smithi189 ceph-mon[34842]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:09:09.310 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:09.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:08 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-10T11:09:09.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:08 smithi076 ceph-mon[34152]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:09:10.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:09 smithi027 ceph-mon[30054]: pgmap v372: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:10.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:09 smithi189 ceph-mon[34842]: pgmap v372: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:10.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:09 smithi076 ceph-mon[34152]: pgmap v372: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:11.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.568+0000 7fa175a00700 1 -- 172.21.15.27:0/1378531071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa170109460 msgr2=0x7fa170109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:11.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.568+0000 7fa175a00700 1 --2- 172.21.15.27:0/1378531071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa170109460 0x7fa170109840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa160009990 tx=0x7fa160009ca0 comp rx=0 tx=0).stop 2024-05-10T11:09:11.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.568+0000 7fa175a00700 1 -- 172.21.15.27:0/1378531071 shutdown_connections 2024-05-10T11:09:11.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.568+0000 7fa175a00700 1 --2- 172.21.15.27:0/1378531071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa170103d90 0x7fa1701102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:11.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.568+0000 7fa175a00700 1 --2- 172.21.15.27:0/1378531071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa170109460 0x7fa170109840 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:11.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.568+0000 7fa175a00700 1 --2- 172.21.15.27:0/1378531071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1701033d0 0x7fa170103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:11.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.568+0000 7fa175a00700 1 -- 172.21.15.27:0/1378531071 >> 172.21.15.27:0/1378531071 conn(0x7fa1700fef20 msgr2=0x7fa170101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:11.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.568+0000 7fa175a00700 1 -- 172.21.15.27:0/1378531071 shutdown_connections 2024-05-10T11:09:11.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.569+0000 7fa175a00700 1 -- 172.21.15.27:0/1378531071 wait complete. 2024-05-10T11:09:11.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.570+0000 7fa175a00700 1 Processor -- start 2024-05-10T11:09:11.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.570+0000 7fa175a00700 1 -- start start 2024-05-10T11:09:11.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.571+0000 7fa175a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1701033d0 0x7fa1701a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.571+0000 7fa175a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa170103d90 0x7fa1701a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.571+0000 7fa175a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa170109460 0x7fa17019ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.571+0000 7fa175a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa170113590 con 0x7fa170109460 2024-05-10T11:09:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.571+0000 7fa175a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa170113410 con 0x7fa170103d90 2024-05-10T11:09:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.571+0000 7fa175a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa170113710 con 0x7fa1701033d0 2024-05-10T11:09:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.571+0000 7fa16f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa170103d90 0x7fa1701a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.571+0000 7fa16fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1701033d0 0x7fa1701a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.571+0000 7fa175000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa170109460 0x7fa17019ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.571+0000 7fa16f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa170103d90 0x7fa1701a1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52400/0 (socket says 172.21.15.27:52400) 2024-05-10T11:09:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.571+0000 7fa16f400700 1 -- 172.21.15.27:0/3751999006 learned_addr learned my addr 172.21.15.27:0/3751999006 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.572+0000 7fa16f400700 1 -- 172.21.15.27:0/3751999006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1701033d0 msgr2=0x7fa1701a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.572+0000 7fa16f400700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1701033d0 0x7fa1701a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.572+0000 7fa16f400700 1 -- 172.21.15.27:0/3751999006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa170109460 msgr2=0x7fa17019ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.572+0000 7fa16f400700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa170109460 0x7fa17019ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.572+0000 7fa16f400700 1 -- 172.21.15.27:0/3751999006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17019c1c0 con 0x7fa170103d90 2024-05-10T11:09:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.572+0000 7fa175000700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa170109460 0x7fa17019ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:09:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.572+0000 7fa16fe00700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1701033d0 0x7fa1701a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:09:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.572+0000 7fa16f400700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa170103d90 0x7fa1701a1de0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa15c00e9a0 tx=0x7fa15c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.573+0000 7fa16cc00700 1 -- 172.21.15.27:0/3751999006 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c00ca10 con 0x7fa170103d90 2024-05-10T11:09:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.573+0000 7fa16cc00700 1 -- 172.21.15.27:0/3751999006 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa15c00cbb0 con 0x7fa170103d90 2024-05-10T11:09:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.573+0000 7fa175a00700 1 -- 172.21.15.27:0/3751999006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa17019c4b0 con 0x7fa170103d90 2024-05-10T11:09:11.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.573+0000 7fa16cc00700 1 -- 172.21.15.27:0/3751999006 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c018680 con 0x7fa170103d90 2024-05-10T11:09:11.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.573+0000 7fa175a00700 1 -- 172.21.15.27:0/3751999006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1701a8770 con 0x7fa170103d90 2024-05-10T11:09:11.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.574+0000 7fa175a00700 1 -- 172.21.15.27:0/3751999006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa17006a190 con 0x7fa170103d90 2024-05-10T11:09:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.577+0000 7fa16cc00700 1 -- 172.21.15.27:0/3751999006 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa15c010430 con 0x7fa170103d90 2024-05-10T11:09:11.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.579+0000 7fa16cc00700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa1540b2900 0x7fa1540b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:11.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.579+0000 7fa16cc00700 1 -- 172.21.15.27:0/3751999006 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7fa15c0e26a0 con 0x7fa170103d90 2024-05-10T11:09:11.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.579+0000 7fa16fe00700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa1540b2900 0x7fa1540b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:11.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.580+0000 7fa16fe00700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa1540b2900 0x7fa1540b4dc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa17019cba0 tx=0x7fa160005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:11.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.583+0000 7fa16cc00700 1 -- 172.21.15.27:0/3751999006 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa15c0aa020 con 0x7fa170103d90 2024-05-10T11:09:11.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.872+0000 7fa175a00700 1 -- 172.21.15.27:0/3751999006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fa1701a89a0 con 0x7fa170103d90 2024-05-10T11:09:11.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.962+0000 7fa16cc00700 1 -- 172.21.15.27:0/3751999006 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fa15c0b20a0 con 0x7fa170103d90 2024-05-10T11:09:11.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.968+0000 7fa175a00700 1 -- 172.21.15.27:0/3751999006 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa1540b2900 msgr2=0x7fa1540b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:11.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.968+0000 7fa175a00700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa1540b2900 0x7fa1540b4dc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa17019cba0 tx=0x7fa160005960 comp rx=0 tx=0).stop 2024-05-10T11:09:11.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.968+0000 7fa175a00700 1 -- 172.21.15.27:0/3751999006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa170103d90 msgr2=0x7fa1701a1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:11.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.968+0000 7fa175a00700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa170103d90 0x7fa1701a1de0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa15c00e9a0 tx=0x7fa15c00ecb0 comp rx=0 tx=0).stop 2024-05-10T11:09:11.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.968+0000 7fa175a00700 1 -- 172.21.15.27:0/3751999006 shutdown_connections 2024-05-10T11:09:11.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.968+0000 7fa175a00700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa1540b2900 0x7fa1540b4dc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:11.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.968+0000 7fa175a00700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1701033d0 0x7fa1701a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:11.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.968+0000 7fa175a00700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa170109460 0x7fa17019ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:11.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.968+0000 7fa175a00700 1 --2- 172.21.15.27:0/3751999006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa170103d90 0x7fa1701a1de0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:11.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.968+0000 7fa175a00700 1 -- 172.21.15.27:0/3751999006 >> 172.21.15.27:0/3751999006 conn(0x7fa1700fef20 msgr2=0x7fa1700ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:11.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.969+0000 7fa175a00700 1 -- 172.21.15.27:0/3751999006 shutdown_connections 2024-05-10T11:09:11.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:11.969+0000 7fa175a00700 1 -- 172.21.15.27:0/3751999006 wait complete. 2024-05-10T11:09:12.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:11 smithi189 ceph-mon[34842]: pgmap v373: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:12.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:11 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3751999006' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-10T11:09:12.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:11 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-10T11:09:12.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:11 smithi027 ceph-mon[30054]: pgmap v373: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:12.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:11 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3751999006' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-10T11:09:12.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:11 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-10T11:09:12.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:11 smithi076 ceph-mon[34152]: pgmap v373: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:09:12.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:11 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3751999006' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-10T11:09:12.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:11 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-10T11:09:12.896 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-05-10T11:09:13.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:12 smithi027 ceph-mon[30054]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-10T11:09:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:12 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-10T11:09:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:12 smithi027 ceph-mon[30054]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-10T11:09:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:12 smithi027 ceph-mon[30054]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:09:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:12 smithi027 ceph-mon[30054]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-10T11:09:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:12 smithi027 ceph-mon[30054]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-10T11:09:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:12 smithi189 ceph-mon[34842]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-10T11:09:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:12 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-10T11:09:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:12 smithi189 ceph-mon[34842]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-10T11:09:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:12 smithi189 ceph-mon[34842]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:09:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:12 smithi189 ceph-mon[34842]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-10T11:09:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:12 smithi189 ceph-mon[34842]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-10T11:09:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:13.425 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:09:12 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b[64743]: 2024-05-10T11:09:12.972+0000 7f142e13a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-05-10T11:09:13.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:12 smithi076 ceph-mon[34152]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-10T11:09:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:12 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-10T11:09:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:12 smithi076 ceph-mon[34152]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-10T11:09:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:12 smithi076 ceph-mon[34152]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-10T11:09:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:12 smithi076 ceph-mon[34152]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-10T11:09:13.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:12 smithi076 ceph-mon[34152]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-10T11:09:14.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:13 smithi189 ceph-mon[34842]: pgmap v374: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-05-10T11:09:14.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:13 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] up:active 2024-05-10T11:09:14.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:13 smithi189 ceph-mon[34842]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-10T11:09:14.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:13 smithi189 ceph-mon[34842]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-10T11:09:14.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:13 smithi189 ceph-mon[34842]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-05-10T11:09:14.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:13 smithi189 ceph-mon[34842]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-05-10T11:09:14.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:13 smithi027 ceph-mon[30054]: pgmap v374: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-05-10T11:09:14.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:13 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] up:active 2024-05-10T11:09:14.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:13 smithi027 ceph-mon[30054]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-10T11:09:14.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:13 smithi027 ceph-mon[30054]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-10T11:09:14.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:13 smithi027 ceph-mon[30054]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-05-10T11:09:14.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:13 smithi027 ceph-mon[30054]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-05-10T11:09:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:13 smithi076 ceph-mon[34152]: pgmap v374: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-05-10T11:09:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:13 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] up:active 2024-05-10T11:09:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:13 smithi076 ceph-mon[34152]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-10T11:09:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:13 smithi076 ceph-mon[34152]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-10T11:09:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:13 smithi076 ceph-mon[34152]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-05-10T11:09:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:13 smithi076 ceph-mon[34152]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-05-10T11:09:15.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:14 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] up:active 2024-05-10T11:09:15.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:14 smithi027 ceph-mon[30054]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-10T11:09:15.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:14 smithi027 ceph-mon[30054]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-05-10T11:09:15.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:14 smithi027 ceph-mon[30054]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-05-10T11:09:15.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:14 smithi027 ceph-mon[30054]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-05-10T11:09:15.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:14 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] up:active 2024-05-10T11:09:15.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:14 smithi076 ceph-mon[34152]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-10T11:09:15.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:14 smithi076 ceph-mon[34152]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-05-10T11:09:15.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:14 smithi076 ceph-mon[34152]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-05-10T11:09:15.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:14 smithi076 ceph-mon[34152]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-05-10T11:09:15.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:14 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] up:active 2024-05-10T11:09:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:14 smithi189 ceph-mon[34842]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-10T11:09:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:14 smithi189 ceph-mon[34842]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-05-10T11:09:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:14 smithi189 ceph-mon[34842]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-05-10T11:09:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:14 smithi189 ceph-mon[34842]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-05-10T11:09:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.609+0000 7ff590a00700 1 -- 172.21.15.27:0/2888742574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff58c109460 msgr2=0x7ff58c109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.609+0000 7ff590a00700 1 --2- 172.21.15.27:0/2888742574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff58c109460 0x7ff58c109840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009990 tx=0x7ff57c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:09:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.609+0000 7ff590a00700 1 -- 172.21.15.27:0/2888742574 shutdown_connections 2024-05-10T11:09:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.609+0000 7ff590a00700 1 --2- 172.21.15.27:0/2888742574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff58c103d90 0x7ff58c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.609+0000 7ff590a00700 1 --2- 172.21.15.27:0/2888742574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff58c109460 0x7ff58c109840 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.609+0000 7ff590a00700 1 --2- 172.21.15.27:0/2888742574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff58c1033d0 0x7ff58c103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.609+0000 7ff590a00700 1 -- 172.21.15.27:0/2888742574 >> 172.21.15.27:0/2888742574 conn(0x7ff58c0fef20 msgr2=0x7ff58c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.610+0000 7ff590a00700 1 -- 172.21.15.27:0/2888742574 shutdown_connections 2024-05-10T11:09:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.610+0000 7ff590a00700 1 -- 172.21.15.27:0/2888742574 wait complete. 2024-05-10T11:09:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.611+0000 7ff590a00700 1 Processor -- start 2024-05-10T11:09:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.611+0000 7ff590a00700 1 -- start start 2024-05-10T11:09:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.611+0000 7ff590a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff58c1033d0 0x7ff58c0757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.611+0000 7ff590a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff58c103d90 0x7ff58c075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.611+0000 7ff590a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff58c109460 0x7ff58c076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.611+0000 7ff590a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff58c1135c0 con 0x7ff58c109460 2024-05-10T11:09:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.611+0000 7ff590a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff58c113440 con 0x7ff58c1033d0 2024-05-10T11:09:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.611+0000 7ff590a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff58c113740 con 0x7ff58c103d90 2024-05-10T11:09:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.612+0000 7ff58a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff58c1033d0 0x7ff58c0757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.612+0000 7ff58a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff58c1033d0 0x7ff58c0757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40088/0 (socket says 172.21.15.27:40088) 2024-05-10T11:09:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.612+0000 7ff58a800700 1 -- 172.21.15.27:0/68074776 learned_addr learned my addr 172.21.15.27:0/68074776 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.612+0000 7ff58b200700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff58c109460 0x7ff58c076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:15.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.612+0000 7ff581e00700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff58c103d90 0x7ff58c075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:15.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.612+0000 7ff58a800700 1 -- 172.21.15.27:0/68074776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff58c103d90 msgr2=0x7ff58c075ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:15.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.612+0000 7ff58a800700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff58c103d90 0x7ff58c075ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:15.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.612+0000 7ff58a800700 1 -- 172.21.15.27:0/68074776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff58c109460 msgr2=0x7ff58c076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:15.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.612+0000 7ff58a800700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff58c109460 0x7ff58c076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:15.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.612+0000 7ff58a800700 1 -- 172.21.15.27:0/68074776 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff58c1a7fe0 con 0x7ff58c1033d0 2024-05-10T11:09:15.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.612+0000 7ff58b200700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff58c109460 0x7ff58c076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:09:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.613+0000 7ff58a800700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff58c1033d0 0x7ff58c0757a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff57c005910 tx=0x7ff57c00b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.613+0000 7ff583e00700 1 -- 172.21.15.27:0/68074776 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c027070 con 0x7ff58c1033d0 2024-05-10T11:09:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.613+0000 7ff583e00700 1 -- 172.21.15.27:0/68074776 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff57c022470 con 0x7ff58c1033d0 2024-05-10T11:09:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.613+0000 7ff590a00700 1 -- 172.21.15.27:0/68074776 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff58c1a8270 con 0x7ff58c1033d0 2024-05-10T11:09:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.613+0000 7ff583e00700 1 -- 172.21.15.27:0/68074776 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c017650 con 0x7ff58c1033d0 2024-05-10T11:09:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.613+0000 7ff590a00700 1 -- 172.21.15.27:0/68074776 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff58c1a86c0 con 0x7ff58c1033d0 2024-05-10T11:09:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.614+0000 7ff590a00700 1 -- 172.21.15.27:0/68074776 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff58c06a190 con 0x7ff58c1033d0 2024-05-10T11:09:15.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.616+0000 7ff583e00700 1 -- 172.21.15.27:0/68074776 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ff57c02a070 con 0x7ff58c1033d0 2024-05-10T11:09:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.618+0000 7ff583e00700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff5700b2880 0x7ff5700b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.618+0000 7ff583e00700 1 -- 172.21.15.27:0/68074776 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7ff57c0e34e0 con 0x7ff58c1033d0 2024-05-10T11:09:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.618+0000 7ff581e00700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff5700b2880 0x7ff5700b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.619+0000 7ff581e00700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff5700b2880 0x7ff5700b4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff58c101560 tx=0x7ff57400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:15.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.620+0000 7ff583e00700 1 -- 172.21.15.27:0/68074776 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ff57c013070 con 0x7ff58c1033d0 2024-05-10T11:09:15.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:15.912+0000 7ff590a00700 1 -- 172.21.15.27:0/68074776 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7ff58c02e510 con 0x7ff58c1033d0 2024-05-10T11:09:16.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.015+0000 7ff583e00700 1 -- 172.21.15.27:0/68074776 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v27) ==== 121+0+0 (secure 0 0 0) 0x7ff57c0aaef0 con 0x7ff58c1033d0 2024-05-10T11:09:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.021+0000 7ff590a00700 1 -- 172.21.15.27:0/68074776 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff5700b2880 msgr2=0x7ff5700b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.021+0000 7ff590a00700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff5700b2880 0x7ff5700b4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff58c101560 tx=0x7ff57400b410 comp rx=0 tx=0).stop 2024-05-10T11:09:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.021+0000 7ff590a00700 1 -- 172.21.15.27:0/68074776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff58c1033d0 msgr2=0x7ff58c0757a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.021+0000 7ff590a00700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff58c1033d0 0x7ff58c0757a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff57c005910 tx=0x7ff57c00b670 comp rx=0 tx=0).stop 2024-05-10T11:09:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.021+0000 7ff590a00700 1 -- 172.21.15.27:0/68074776 shutdown_connections 2024-05-10T11:09:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.021+0000 7ff590a00700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff5700b2880 0x7ff5700b4d40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.021+0000 7ff590a00700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff58c109460 0x7ff58c076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.021+0000 7ff590a00700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff58c1033d0 0x7ff58c0757a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:16.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.021+0000 7ff590a00700 1 --2- 172.21.15.27:0/68074776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff58c103d90 0x7ff58c075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:16.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.021+0000 7ff590a00700 1 -- 172.21.15.27:0/68074776 >> 172.21.15.27:0/68074776 conn(0x7ff58c0fef20 msgr2=0x7ff58c0ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:16.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.021+0000 7ff590a00700 1 -- 172.21.15.27:0/68074776 shutdown_connections 2024-05-10T11:09:16.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:16.022+0000 7ff590a00700 1 -- 172.21.15.27:0/68074776 wait complete. 2024-05-10T11:09:16.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:16 smithi076 ceph-mon[34152]: pgmap v375: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s wr, 1 op/s 2024-05-10T11:09:16.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:16 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] up:active 2024-05-10T11:09:16.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:16 smithi076 ceph-mon[34152]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-10T11:09:16.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:16 smithi076 ceph-mon[34152]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-10T11:09:16.327 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:16 smithi076 ceph-mon[34152]: Cluster is now healthy 2024-05-10T11:09:16.327 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:16 smithi076 ceph-mon[34152]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-05-10T11:09:16.327 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:16 smithi076 ceph-mon[34152]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-10T11:09:16.327 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:16 smithi076 ceph-mon[34152]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-05-10T11:09:16.327 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:16 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/68074776' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-10T11:09:16.327 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:16 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-10T11:09:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:16 smithi027 ceph-mon[30054]: pgmap v375: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s wr, 1 op/s 2024-05-10T11:09:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:16 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] up:active 2024-05-10T11:09:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:16 smithi027 ceph-mon[30054]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-10T11:09:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:16 smithi027 ceph-mon[30054]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-10T11:09:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:16 smithi027 ceph-mon[30054]: Cluster is now healthy 2024-05-10T11:09:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:16 smithi027 ceph-mon[30054]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-05-10T11:09:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:16 smithi027 ceph-mon[30054]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-10T11:09:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:16 smithi027 ceph-mon[30054]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-05-10T11:09:16.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:16 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/68074776' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-10T11:09:16.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:16 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-10T11:09:16.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:16 smithi189 ceph-mon[34842]: pgmap v375: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s wr, 1 op/s 2024-05-10T11:09:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:16 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] up:active 2024-05-10T11:09:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:16 smithi189 ceph-mon[34842]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-10T11:09:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:16 smithi189 ceph-mon[34842]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-10T11:09:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:16 smithi189 ceph-mon[34842]: Cluster is now healthy 2024-05-10T11:09:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:16 smithi189 ceph-mon[34842]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-05-10T11:09:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:16 smithi189 ceph-mon[34842]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-10T11:09:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:16 smithi189 ceph-mon[34842]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-05-10T11:09:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:16 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/68074776' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-10T11:09:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:16 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-10T11:09:16.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-05-10T11:09:17.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:17 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-10T11:09:17.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:17 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] up:active 2024-05-10T11:09:17.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:17 smithi027 ceph-mon[30054]: fsmap cephfs:5 7 up:standby 5 up:active 2024-05-10T11:09:17.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:17 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:09:17.331 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:17 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-10T11:09:17.331 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:17 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] up:active 2024-05-10T11:09:17.331 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:17 smithi189 ceph-mon[34842]: fsmap cephfs:5 7 up:standby 5 up:active 2024-05-10T11:09:17.331 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:17 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:09:17.348 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:17.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:17 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-10T11:09:17.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:17 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] up:active 2024-05-10T11:09:17.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:17 smithi076 ceph-mon[34152]: fsmap cephfs:5 7 up:standby 5 up:active 2024-05-10T11:09:17.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:17 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:09:18.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:18 smithi027 ceph-mon[30054]: pgmap v376: 129 pgs: 129 active+clean; 585 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 2.9 KiB/s wr, 10 op/s 2024-05-10T11:09:18.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:18 smithi076 ceph-mon[34152]: pgmap v376: 129 pgs: 129 active+clean; 585 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 2.9 KiB/s wr, 10 op/s 2024-05-10T11:09:18.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:18 smithi189 ceph-mon[34842]: pgmap v376: 129 pgs: 129 active+clean; 585 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 2.9 KiB/s wr, 10 op/s 2024-05-10T11:09:19.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.671+0000 7f5536400700 1 -- 172.21.15.27:0/853023940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5530108460 msgr2=0x7f5530108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:19.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.671+0000 7f5536400700 1 --2- 172.21.15.27:0/853023940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5530108460 0x7f5530108840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5520009990 tx=0x7f5520009ca0 comp rx=0 tx=0).stop 2024-05-10T11:09:19.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.671+0000 7f5536400700 1 -- 172.21.15.27:0/853023940 shutdown_connections 2024-05-10T11:09:19.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.671+0000 7f5536400700 1 --2- 172.21.15.27:0/853023940 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5530103ef0 0x7f55301103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:19.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.671+0000 7f5536400700 1 --2- 172.21.15.27:0/853023940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5530108460 0x7f5530108840 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:19.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.671+0000 7f5536400700 1 --2- 172.21.15.27:0/853023940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5530103550 0x7f55301039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:19.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.671+0000 7f5536400700 1 -- 172.21.15.27:0/853023940 >> 172.21.15.27:0/853023940 conn(0x7f553007a600 msgr2=0x7f553007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:19.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.671+0000 7f5536400700 1 -- 172.21.15.27:0/853023940 shutdown_connections 2024-05-10T11:09:19.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.671+0000 7f5536400700 1 -- 172.21.15.27:0/853023940 wait complete. 2024-05-10T11:09:19.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.672+0000 7f5536400700 1 Processor -- start 2024-05-10T11:09:19.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.672+0000 7f5536400700 1 -- start start 2024-05-10T11:09:19.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.673+0000 7f5536400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5530103550 0x7f553019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:19.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.673+0000 7f5536400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5530103ef0 0x7f553019fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:19.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.673+0000 7f5536400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5530108460 0x7f55301999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:19.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.673+0000 7f5536400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f55301136a0 con 0x7f5530108460 2024-05-10T11:09:19.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.673+0000 7f5536400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5530113520 con 0x7f5530103550 2024-05-10T11:09:19.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.673+0000 7f5536400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5530113820 con 0x7f5530103ef0 2024-05-10T11:09:19.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.673+0000 7f552f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5530103550 0x7f553019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:19.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.673+0000 7f552f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5530103550 0x7f553019f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40100/0 (socket says 172.21.15.27:40100) 2024-05-10T11:09:19.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.673+0000 7f552f400700 1 -- 172.21.15.27:0/956279157 learned_addr learned my addr 172.21.15.27:0/956279157 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:19.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.673+0000 7f552fe00700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5530108460 0x7f55301999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:19.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.673+0000 7f552ea00700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5530103ef0 0x7f553019fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.674+0000 7f552ea00700 1 -- 172.21.15.27:0/956279157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5530103550 msgr2=0x7f553019f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.674+0000 7f552ea00700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5530103550 0x7f553019f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.674+0000 7f552ea00700 1 -- 172.21.15.27:0/956279157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5530108460 msgr2=0x7f55301999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.674+0000 7f552ea00700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5530108460 0x7f55301999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.674+0000 7f552ea00700 1 -- 172.21.15.27:0/956279157 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553019a140 con 0x7f5530103ef0 2024-05-10T11:09:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.674+0000 7f552f400700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5530103550 0x7f553019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:09:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.674+0000 7f552ea00700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5530103ef0 0x7f553019fd10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f551c00e9f0 tx=0x7f551c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.675+0000 7f5527e00700 1 -- 172.21.15.27:0/956279157 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c004220 con 0x7f5530103ef0 2024-05-10T11:09:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.675+0000 7f5527e00700 1 -- 172.21.15.27:0/956279157 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f551c0043c0 con 0x7f5530103ef0 2024-05-10T11:09:19.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.675+0000 7f5527e00700 1 -- 172.21.15.27:0/956279157 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c005020 con 0x7f5530103ef0 2024-05-10T11:09:19.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.675+0000 7f5536400700 1 -- 172.21.15.27:0/956279157 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f553019a430 con 0x7f5530103ef0 2024-05-10T11:09:19.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.675+0000 7f5536400700 1 -- 172.21.15.27:0/956279157 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55301a66a0 con 0x7f5530103ef0 2024-05-10T11:09:19.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.676+0000 7f5536400700 1 -- 172.21.15.27:0/956279157 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f553006a190 con 0x7f5530103ef0 2024-05-10T11:09:19.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.679+0000 7f5527e00700 1 -- 172.21.15.27:0/956279157 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f551c004750 con 0x7f5530103ef0 2024-05-10T11:09:19.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.681+0000 7f5527e00700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f55140b2900 0x7f55140b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:19.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.681+0000 7f5527e00700 1 -- 172.21.15.27:0/956279157 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9071+0+0 (secure 0 0 0) 0x7f551c0e2450 con 0x7f5530103ef0 2024-05-10T11:09:19.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.681+0000 7f552f400700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f55140b2900 0x7f55140b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:19.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.682+0000 7f5527e00700 1 -- 172.21.15.27:0/956279157 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f551c0a9dd0 con 0x7f5530103ef0 2024-05-10T11:09:19.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.682+0000 7f552f400700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f55140b2900 0x7f55140b4dc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5520006010 tx=0x7f5520005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:19.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:19.983+0000 7f5536400700 1 -- 172.21.15.27:0/956279157 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f553010ba90 con 0x7f5530103ef0 2024-05-10T11:09:20.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.036+0000 7f5527e00700 1 -- 172.21.15.27:0/956279157 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v29) ==== 115+0+0 (secure 0 0 0) 0x7f551c0a96d0 con 0x7f5530103ef0 2024-05-10T11:09:20.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.041+0000 7f5536400700 1 -- 172.21.15.27:0/956279157 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f55140b2900 msgr2=0x7f55140b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:20.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.041+0000 7f5536400700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f55140b2900 0x7f55140b4dc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5520006010 tx=0x7f5520005960 comp rx=0 tx=0).stop 2024-05-10T11:09:20.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.041+0000 7f5536400700 1 -- 172.21.15.27:0/956279157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5530103ef0 msgr2=0x7f553019fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:20.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.041+0000 7f5536400700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5530103ef0 0x7f553019fd10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f551c00e9f0 tx=0x7f551c00ed00 comp rx=0 tx=0).stop 2024-05-10T11:09:20.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.041+0000 7f5536400700 1 -- 172.21.15.27:0/956279157 shutdown_connections 2024-05-10T11:09:20.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.041+0000 7f5536400700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f55140b2900 0x7f55140b4dc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:20.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.041+0000 7f5536400700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5530108460 0x7f55301999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:20.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.041+0000 7f5536400700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5530103550 0x7f553019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:20.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.041+0000 7f5536400700 1 --2- 172.21.15.27:0/956279157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5530103ef0 0x7f553019fd10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:20.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.041+0000 7f5536400700 1 -- 172.21.15.27:0/956279157 >> 172.21.15.27:0/956279157 conn(0x7f553007a600 msgr2=0x7f5530101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:20.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.042+0000 7f5536400700 1 -- 172.21.15.27:0/956279157 shutdown_connections 2024-05-10T11:09:20.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:20.042+0000 7f5536400700 1 -- 172.21.15.27:0/956279157 wait complete. 2024-05-10T11:09:20.125 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:20 smithi027 ceph-mon[30054]: pgmap v377: 129 pgs: 129 active+clean; 586 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3.3 KiB/s wr, 14 op/s 2024-05-10T11:09:20.125 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:20 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/956279157' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-10T11:09:20.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:20 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-10T11:09:20.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:20 smithi076 ceph-mon[34152]: pgmap v377: 129 pgs: 129 active+clean; 586 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3.3 KiB/s wr, 14 op/s 2024-05-10T11:09:20.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:20 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/956279157' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-10T11:09:20.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:20 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-10T11:09:20.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:20 smithi189 ceph-mon[34842]: pgmap v377: 129 pgs: 129 active+clean; 586 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3.3 KiB/s wr, 14 op/s 2024-05-10T11:09:20.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:20 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/956279157' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-10T11:09:20.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:20 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-10T11:09:20.822 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-05-10T11:09:20.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-05-10T11:09:21.299 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:21.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:21 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-10T11:09:21.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:21 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:09:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:21 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-10T11:09:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:21 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:09:21.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:21 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-10T11:09:21.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:21 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:09:22.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:22 smithi027 ceph-mon[30054]: pgmap v378: 129 pgs: 129 active+clean; 587 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-05-10T11:09:22.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-05-10T11:09:22.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-05-10T11:09:22.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-05-10T11:09:22.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-10T11:09:22.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]: dispatch 2024-05-10T11:09:22.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:22 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-05-10T11:09:22.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:22 smithi076 ceph-mon[34152]: pgmap v378: 129 pgs: 129 active+clean; 587 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-05-10T11:09:22.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-05-10T11:09:22.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-05-10T11:09:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-05-10T11:09:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-10T11:09:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]: dispatch 2024-05-10T11:09:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:22 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-05-10T11:09:22.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:22 smithi189 ceph-mon[34842]: pgmap v378: 129 pgs: 129 active+clean; 587 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-05-10T11:09:22.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-05-10T11:09:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-05-10T11:09:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-05-10T11:09:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-10T11:09:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]: dispatch 2024-05-10T11:09:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:22 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-05-10T11:09:23.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:23 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-05-10T11:09:23.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:23 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-05-10T11:09:23.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:23 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-05-10T11:09:23.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:23 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-10T11:09:23.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:23 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]': finished 2024-05-10T11:09:23.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:23 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-05-10T11:09:23.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:23 smithi027 ceph-mon[30054]: osdmap e89: 12 total, 12 up, 12 in 2024-05-10T11:09:23.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:23 smithi027 ceph-mon[30054]: osdmap e90: 12 total, 12 up, 12 in 2024-05-10T11:09:23.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:23 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-05-10T11:09:23.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:23 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-05-10T11:09:23.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:23 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-05-10T11:09:23.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:23 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-10T11:09:23.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:23 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]': finished 2024-05-10T11:09:23.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:23 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-05-10T11:09:23.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:23 smithi076 ceph-mon[34152]: osdmap e89: 12 total, 12 up, 12 in 2024-05-10T11:09:23.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:23 smithi076 ceph-mon[34152]: osdmap e90: 12 total, 12 up, 12 in 2024-05-10T11:09:23.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:23 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-05-10T11:09:23.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:23 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-05-10T11:09:23.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:23 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-05-10T11:09:23.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:23 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-10T11:09:23.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:23 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]': finished 2024-05-10T11:09:23.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:23 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-05-10T11:09:23.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:23 smithi189 ceph-mon[34842]: osdmap e89: 12 total, 12 up, 12 in 2024-05-10T11:09:23.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:23 smithi189 ceph-mon[34842]: osdmap e90: 12 total, 12 up, 12 in 2024-05-10T11:09:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.591+0000 7f8354a00700 1 -- 172.21.15.27:0/1522324455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f835010fd80 msgr2=0x7f8350112170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.591+0000 7f8354a00700 1 --2- 172.21.15.27:0/1522324455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f835010fd80 0x7f8350112170 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8348009f90 tx=0x7f834800b330 comp rx=0 tx=0).stop 2024-05-10T11:09:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.591+0000 7f8354a00700 1 -- 172.21.15.27:0/1522324455 shutdown_connections 2024-05-10T11:09:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.591+0000 7f8354a00700 1 --2- 172.21.15.27:0/1522324455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8350101840 0x7f8350101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.591+0000 7f8354a00700 1 --2- 172.21.15.27:0/1522324455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f835010fd80 0x7f8350112170 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.591+0000 7f8354a00700 1 --2- 172.21.15.27:0/1522324455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8350102160 0x7f835010f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.591+0000 7f8354a00700 1 -- 172.21.15.27:0/1522324455 >> 172.21.15.27:0/1522324455 conn(0x7f8350077140 msgr2=0x7f8350077550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.591+0000 7f8354a00700 1 -- 172.21.15.27:0/1522324455 shutdown_connections 2024-05-10T11:09:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.591+0000 7f8354a00700 1 -- 172.21.15.27:0/1522324455 wait complete. 2024-05-10T11:09:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.592+0000 7f8354a00700 1 Processor -- start 2024-05-10T11:09:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.593+0000 7f8354a00700 1 -- start start 2024-05-10T11:09:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.593+0000 7f8354a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8350101840 0x7f83501a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.593+0000 7f8354a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8350102160 0x7f83501a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.593+0000 7f8354a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f835010fd80 0x7f835019ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.593+0000 7f8354a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f83501157d0 con 0x7f835010fd80 2024-05-10T11:09:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.593+0000 7f8354a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8350115650 con 0x7f8350101840 2024-05-10T11:09:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.593+0000 7f8354a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8350115950 con 0x7f8350102160 2024-05-10T11:09:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.593+0000 7f834f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8350101840 0x7f83501a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.594+0000 7f834fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f835010fd80 0x7f835019ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.594+0000 7f834f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8350101840 0x7f83501a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40128/0 (socket says 172.21.15.27:40128) 2024-05-10T11:09:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.594+0000 7f834f400700 1 -- 172.21.15.27:0/3853536398 learned_addr learned my addr 172.21.15.27:0/3853536398 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.594+0000 7f834ea00700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8350102160 0x7f83501a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.594+0000 7f834f400700 1 -- 172.21.15.27:0/3853536398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8350102160 msgr2=0x7f83501a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:23.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.594+0000 7f834f400700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8350102160 0x7f83501a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:23.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.594+0000 7f834f400700 1 -- 172.21.15.27:0/3853536398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f835010fd80 msgr2=0x7f835019ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:23.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.594+0000 7f834f400700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f835010fd80 0x7f835019ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:23.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.594+0000 7f834f400700 1 -- 172.21.15.27:0/3853536398 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835019c1b0 con 0x7f8350101840 2024-05-10T11:09:23.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.594+0000 7f834fe00700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f835010fd80 0x7f835019ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:09:23.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.594+0000 7f834f400700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8350101840 0x7f83501a1840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f834000e9a0 tx=0x7f834000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:23.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.595+0000 7f8347e00700 1 -- 172.21.15.27:0/3853536398 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834000ca10 con 0x7f8350101840 2024-05-10T11:09:23.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.595+0000 7f8354a00700 1 -- 172.21.15.27:0/3853536398 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f835019c4a0 con 0x7f8350101840 2024-05-10T11:09:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.595+0000 7f8347e00700 1 -- 172.21.15.27:0/3853536398 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f834000cbb0 con 0x7f8350101840 2024-05-10T11:09:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.595+0000 7f8347e00700 1 -- 172.21.15.27:0/3853536398 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340018680 con 0x7f8350101840 2024-05-10T11:09:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.595+0000 7f8354a00700 1 -- 172.21.15.27:0/3853536398 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83501a8710 con 0x7f8350101840 2024-05-10T11:09:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.596+0000 7f8354a00700 1 -- 172.21.15.27:0/3853536398 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f835006a190 con 0x7f8350101840 2024-05-10T11:09:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.598+0000 7f8347e00700 1 -- 172.21.15.27:0/3853536398 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8340010430 con 0x7f8350101840 2024-05-10T11:09:23.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.600+0000 7f8347e00700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f83340b2880 0x7f83340b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:23.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.600+0000 7f8347e00700 1 -- 172.21.15.27:0/3853536398 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f83400e27b0 con 0x7f8350101840 2024-05-10T11:09:23.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.600+0000 7f834ea00700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f83340b2880 0x7f83340b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:23.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.601+0000 7f834ea00700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f83340b2880 0x7f83340b4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f833c009750 tx=0x7f833c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:23.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.604+0000 7f8347e00700 1 -- 172.21.15.27:0/3853536398 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f83400aa080 con 0x7f8350101840 2024-05-10T11:09:23.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.889+0000 7f8354a00700 1 -- 172.21.15.27:0/3853536398 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f83500638e0 con 0x7f83340b2880 2024-05-10T11:09:23.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.922+0000 7f8347e00700 1 -- 172.21.15.27:0/3853536398 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f83500638e0 con 0x7f83340b2880 2024-05-10T11:09:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.925+0000 7f8354a00700 1 -- 172.21.15.27:0/3853536398 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f83340b2880 msgr2=0x7f83340b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.925+0000 7f8354a00700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f83340b2880 0x7f83340b4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f833c009750 tx=0x7f833c009310 comp rx=0 tx=0).stop 2024-05-10T11:09:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.925+0000 7f8354a00700 1 -- 172.21.15.27:0/3853536398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8350101840 msgr2=0x7f83501a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.925+0000 7f8354a00700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8350101840 0x7f83501a1840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f834000e9a0 tx=0x7f834000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:09:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.925+0000 7f8354a00700 1 -- 172.21.15.27:0/3853536398 shutdown_connections 2024-05-10T11:09:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.925+0000 7f8354a00700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f83340b2880 0x7f83340b4d40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.925+0000 7f8354a00700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f835010fd80 0x7f835019ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.925+0000 7f8354a00700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8350101840 0x7f83501a1840 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.925+0000 7f8354a00700 1 --2- 172.21.15.27:0/3853536398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8350102160 0x7f83501a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.925+0000 7f8354a00700 1 -- 172.21.15.27:0/3853536398 >> 172.21.15.27:0/3853536398 conn(0x7f8350077140 msgr2=0x7f8350110340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.925+0000 7f8354a00700 1 -- 172.21.15.27:0/3853536398 shutdown_connections 2024-05-10T11:09:23.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:23.925+0000 7f8354a00700 1 -- 172.21.15.27:0/3853536398 wait complete. 2024-05-10T11:09:24.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:24 smithi027 ceph-mon[30054]: pgmap v380: 129 pgs: 2 peering, 127 active+clean; 587 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 4.2 KiB/s wr, 23 op/s 2024-05-10T11:09:24.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:24 smithi027 ceph-mon[30054]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-05-10T11:09:24.359 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:09:23 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:09:23.898+0000 7f46f8200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:09:24.359 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:09:23 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:09:23.898+0000 7f46f8200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:09:24.359 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:09:23 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:09:23.898+0000 7f46f8200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:09:24.359 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:09:23 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:09:23.898+0000 7f46f8200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:09:24.359 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:09:23 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:09:23.898+0000 7f46f8200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:09:24.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:24 smithi076 ceph-mon[34152]: pgmap v380: 129 pgs: 2 peering, 127 active+clean; 587 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 4.2 KiB/s wr, 23 op/s 2024-05-10T11:09:24.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:24 smithi076 ceph-mon[34152]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-05-10T11:09:24.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:24 smithi189 ceph-mon[34842]: pgmap v380: 129 pgs: 2 peering, 127 active+clean; 587 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 4.2 KiB/s wr, 23 op/s 2024-05-10T11:09:24.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:24 smithi189 ceph-mon[34842]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-05-10T11:09:24.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-10T11:09:25.312 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:25.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:25 smithi027 ceph-mon[30054]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:25.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:25 smithi076 ceph-mon[34152]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:25.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:25 smithi189 ceph-mon[34842]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:26.366 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:26 smithi027 ceph-mon[30054]: pgmap v382: 129 pgs: 2 peering, 127 active+clean; 587 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.6 KiB/s wr, 15 op/s 2024-05-10T11:09:26.366 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:26 smithi027 ceph-mon[30054]: mgrmap e17: x(active, since 10m), standbys: z, y 2024-05-10T11:09:26.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:26 smithi076 ceph-mon[34152]: pgmap v382: 129 pgs: 2 peering, 127 active+clean; 587 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.6 KiB/s wr, 15 op/s 2024-05-10T11:09:26.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:26 smithi076 ceph-mon[34152]: mgrmap e17: x(active, since 10m), standbys: z, y 2024-05-10T11:09:26.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:26 smithi189 ceph-mon[34842]: pgmap v382: 129 pgs: 2 peering, 127 active+clean; 587 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.6 KiB/s wr, 15 op/s 2024-05-10T11:09:26.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:26 smithi189 ceph-mon[34842]: mgrmap e17: x(active, since 10m), standbys: z, y 2024-05-10T11:09:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.599+0000 7f3058600700 1 -- 172.21.15.27:0/16041159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30501056c0 msgr2=0x7f3050105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.599+0000 7f3058600700 1 --2- 172.21.15.27:0/16041159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30501056c0 0x7f3050105b40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f303c0098f0 tx=0x7f303c009c00 comp rx=0 tx=0).stop 2024-05-10T11:09:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.599+0000 7f3058600700 1 -- 172.21.15.27:0/16041159 shutdown_connections 2024-05-10T11:09:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.599+0000 7f3058600700 1 --2- 172.21.15.27:0/16041159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3050106080 0x7f3050112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.599+0000 7f3058600700 1 --2- 172.21.15.27:0/16041159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f305010b6c0 0x7f305010baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.599+0000 7f3058600700 1 --2- 172.21.15.27:0/16041159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30501056c0 0x7f3050105b40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.599+0000 7f3058600700 1 -- 172.21.15.27:0/16041159 >> 172.21.15.27:0/16041159 conn(0x7f30501011b0 msgr2=0x7f30501035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.599+0000 7f3058600700 1 -- 172.21.15.27:0/16041159 shutdown_connections 2024-05-10T11:09:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.599+0000 7f3058600700 1 -- 172.21.15.27:0/16041159 wait complete. 2024-05-10T11:09:27.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.600+0000 7f3058600700 1 Processor -- start 2024-05-10T11:09:27.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.601+0000 7f3058600700 1 -- start start 2024-05-10T11:09:27.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.601+0000 7f3058600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30501056c0 0x7f305019be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.601+0000 7f3058600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3050106080 0x7f305019c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.601+0000 7f3058600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305010b6c0 0x7f30501a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.601+0000 7f3058600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3050115950 con 0x7f305010b6c0 2024-05-10T11:09:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.601+0000 7f3058600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f30501157d0 con 0x7f3050106080 2024-05-10T11:09:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.601+0000 7f3058600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3050115ad0 con 0x7f30501056c0 2024-05-10T11:09:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.602+0000 7f3055c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30501056c0 0x7f305019be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.602+0000 7f3056600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305010b6c0 0x7f30501a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.602+0000 7f3055c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30501056c0 0x7f305019be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:34372/0 (socket says 172.21.15.27:34372) 2024-05-10T11:09:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.602+0000 7f3055c00700 1 -- 172.21.15.27:0/626437234 learned_addr learned my addr 172.21.15.27:0/626437234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.602+0000 7f3055200700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3050106080 0x7f305019c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.602+0000 7f3055200700 1 -- 172.21.15.27:0/626437234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30501056c0 msgr2=0x7f305019be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.602+0000 7f3055200700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30501056c0 0x7f305019be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.602+0000 7f3055200700 1 -- 172.21.15.27:0/626437234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305010b6c0 msgr2=0x7f30501a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.602+0000 7f3055200700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305010b6c0 0x7f30501a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.602+0000 7f3055200700 1 -- 172.21.15.27:0/626437234 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30501a0e10 con 0x7f3050106080 2024-05-10T11:09:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.603+0000 7f3055200700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3050106080 0x7f305019c340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f303c00c010 tx=0x7f303c00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.603+0000 7f304e000700 1 -- 172.21.15.27:0/626437234 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c0050c0 con 0x7f3050106080 2024-05-10T11:09:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.603+0000 7f304e000700 1 -- 172.21.15.27:0/626437234 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f303c00b560 con 0x7f3050106080 2024-05-10T11:09:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.603+0000 7f304e000700 1 -- 172.21.15.27:0/626437234 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c003a00 con 0x7f3050106080 2024-05-10T11:09:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.603+0000 7f3058600700 1 -- 172.21.15.27:0/626437234 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30501a10a0 con 0x7f3050106080 2024-05-10T11:09:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.603+0000 7f3058600700 1 -- 172.21.15.27:0/626437234 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30501a8980 con 0x7f3050106080 2024-05-10T11:09:27.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.604+0000 7f3058600700 1 -- 172.21.15.27:0/626437234 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3050106e00 con 0x7f3050106080 2024-05-10T11:09:27.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.607+0000 7f304e000700 1 -- 172.21.15.27:0/626437234 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f303c00b710 con 0x7f3050106080 2024-05-10T11:09:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.608+0000 7f304e000700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30380b2990 0x7f30380b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.608+0000 7f304e000700 1 -- 172.21.15.27:0/626437234 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f303c0e37f0 con 0x7f3050106080 2024-05-10T11:09:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.608+0000 7f3055c00700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30380b2990 0x7f30380b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.609+0000 7f3055c00700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30380b2990 0x7f30380b4e50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3044009ca0 tx=0x7f3044009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:27.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.611+0000 7f304e000700 1 -- 172.21.15.27:0/626437234 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f303c013030 con 0x7f3050106080 2024-05-10T11:09:27.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.889+0000 7f3058600700 1 -- 172.21.15.27:0/626437234 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f30500638e0 con 0x7f30380b2990 2024-05-10T11:09:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.928+0000 7f304e000700 1 -- 172.21.15.27:0/626437234 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f30500638e0 con 0x7f30380b2990 2024-05-10T11:09:27.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.931+0000 7f3058600700 1 -- 172.21.15.27:0/626437234 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30380b2990 msgr2=0x7f30380b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:27.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.931+0000 7f3058600700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30380b2990 0x7f30380b4e50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3044009ca0 tx=0x7f3044009260 comp rx=0 tx=0).stop 2024-05-10T11:09:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.931+0000 7f3058600700 1 -- 172.21.15.27:0/626437234 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3050106080 msgr2=0x7f305019c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.931+0000 7f3058600700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3050106080 0x7f305019c340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f303c00c010 tx=0x7f303c00fa00 comp rx=0 tx=0).stop 2024-05-10T11:09:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.932+0000 7f3058600700 1 -- 172.21.15.27:0/626437234 shutdown_connections 2024-05-10T11:09:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.932+0000 7f3058600700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30380b2990 0x7f30380b4e50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.932+0000 7f3058600700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30501056c0 0x7f305019be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.932+0000 7f3058600700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305010b6c0 0x7f30501a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.932+0000 7f3058600700 1 --2- 172.21.15.27:0/626437234 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3050106080 0x7f305019c340 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.932+0000 7f3058600700 1 -- 172.21.15.27:0/626437234 >> 172.21.15.27:0/626437234 conn(0x7f30501011b0 msgr2=0x7f30501019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.932+0000 7f3058600700 1 -- 172.21.15.27:0/626437234 shutdown_connections 2024-05-10T11:09:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:27.932+0000 7f3058600700 1 -- 172.21.15.27:0/626437234 wait complete. 2024-05-10T11:09:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:28 smithi027 ceph-mon[30054]: pgmap v383: 129 pgs: 2 peering, 127 active+clean; 593 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 2.1 KiB/s wr, 14 op/s; 88 B/s, 1 keys/s, 1 objects/s recovering 2024-05-10T11:09:28.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:28 smithi076 ceph-mon[34152]: pgmap v383: 129 pgs: 2 peering, 127 active+clean; 593 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 2.1 KiB/s wr, 14 op/s; 88 B/s, 1 keys/s, 1 objects/s recovering 2024-05-10T11:09:28.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:28 smithi189 ceph-mon[34842]: pgmap v383: 129 pgs: 2 peering, 127 active+clean; 593 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 2.1 KiB/s wr, 14 op/s; 88 B/s, 1 keys/s, 1 objects/s recovering 2024-05-10T11:09:28.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-10T11:09:29.344 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:29.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:29 smithi076 ceph-mon[34152]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:29.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:29 smithi189 ceph-mon[34842]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:29.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:29 smithi027 ceph-mon[30054]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:30.392 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:30 smithi027 ceph-mon[30054]: pgmap v384: 129 pgs: 129 active+clean; 593 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 9 op/s; 88 B/s, 1 keys/s, 1 objects/s recovering 2024-05-10T11:09:30.392 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:30 smithi027 ceph-mon[30054]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-05-10T11:09:30.393 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:30 smithi027 ceph-mon[30054]: Cluster is now healthy 2024-05-10T11:09:30.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:30 smithi076 ceph-mon[34152]: pgmap v384: 129 pgs: 129 active+clean; 593 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 9 op/s; 88 B/s, 1 keys/s, 1 objects/s recovering 2024-05-10T11:09:30.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:30 smithi076 ceph-mon[34152]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-05-10T11:09:30.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:30 smithi076 ceph-mon[34152]: Cluster is now healthy 2024-05-10T11:09:30.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:30 smithi189 ceph-mon[34842]: pgmap v384: 129 pgs: 129 active+clean; 593 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 9 op/s; 88 B/s, 1 keys/s, 1 objects/s recovering 2024-05-10T11:09:30.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:30 smithi189 ceph-mon[34842]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-05-10T11:09:30.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:30 smithi189 ceph-mon[34842]: Cluster is now healthy 2024-05-10T11:09:31.323 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:31 smithi027 ceph-mon[30054]: pgmap v385: 129 pgs: 129 active+clean; 607 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.1 KiB/s wr, 11 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-05-10T11:09:31.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:31 smithi076 ceph-mon[34152]: pgmap v385: 129 pgs: 129 active+clean; 607 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.1 KiB/s wr, 11 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-05-10T11:09:31.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:31 smithi189 ceph-mon[34842]: pgmap v385: 129 pgs: 129 active+clean; 607 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.1 KiB/s wr, 11 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-05-10T11:09:31.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.673+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2627639725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6f0108850 msgr2=0x7fd6f01103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.673+0000 7fd6f6c00700 1 --2- 172.21.15.27:0/2627639725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6f0108850 0x7fd6f01103d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800b0d0 tx=0x7fd6e800b3e0 comp rx=0 tx=0).stop 2024-05-10T11:09:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.674+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2627639725 shutdown_connections 2024-05-10T11:09:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.674+0000 7fd6f6c00700 1 --2- 172.21.15.27:0/2627639725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6f0108850 0x7fd6f01103d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.674+0000 7fd6f6c00700 1 --2- 172.21.15.27:0/2627639725 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f0103750 0x7fd6f0103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.674+0000 7fd6f6c00700 1 --2- 172.21.15.27:0/2627639725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f0104100 0x7fd6f0108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.674+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2627639725 >> 172.21.15.27:0/2627639725 conn(0x7fd6f007a600 msgr2=0x7fd6f007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.674+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2627639725 shutdown_connections 2024-05-10T11:09:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.674+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2627639725 wait complete. 2024-05-10T11:09:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.675+0000 7fd6f6c00700 1 Processor -- start 2024-05-10T11:09:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.675+0000 7fd6f6c00700 1 -- start start 2024-05-10T11:09:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.676+0000 7fd6f6c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f0103750 0x7fd6f01a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.676+0000 7fd6f6c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f0104100 0x7fd6f01a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.676+0000 7fd6f6c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6f0108850 0x7fd6f019baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.676+0000 7fd6f6c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f0113670 con 0x7fd6f0108850 2024-05-10T11:09:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.676+0000 7fd6f6c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f01134f0 con 0x7fd6f0103750 2024-05-10T11:09:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.676+0000 7fd6f6c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f01137f0 con 0x7fd6f0104100 2024-05-10T11:09:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.676+0000 7fd6efe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f0103750 0x7fd6f01a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.676+0000 7fd6efe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f0103750 0x7fd6f01a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48524/0 (socket says 172.21.15.27:48524) 2024-05-10T11:09:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.676+0000 7fd6efe00700 1 -- 172.21.15.27:0/2882529580 learned_addr learned my addr 172.21.15.27:0/2882529580 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.676+0000 7fd6ef400700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f0104100 0x7fd6f01a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.676+0000 7fd6f4c00700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6f0108850 0x7fd6f019baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:31.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.677+0000 7fd6efe00700 1 -- 172.21.15.27:0/2882529580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f0104100 msgr2=0x7fd6f01a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:31.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.677+0000 7fd6efe00700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f0104100 0x7fd6f01a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:31.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.677+0000 7fd6efe00700 1 -- 172.21.15.27:0/2882529580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6f0108850 msgr2=0x7fd6f019baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:31.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.677+0000 7fd6efe00700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6f0108850 0x7fd6f019baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:31.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.677+0000 7fd6efe00700 1 -- 172.21.15.27:0/2882529580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f019c250 con 0x7fd6f0103750 2024-05-10T11:09:31.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.677+0000 7fd6efe00700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f0103750 0x7fd6f01a18e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000e9a0 tx=0x7fd6e000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:31.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.677+0000 7fd6ecc00700 1 -- 172.21.15.27:0/2882529580 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e000cbd0 con 0x7fd6f0103750 2024-05-10T11:09:31.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.677+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2882529580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f019c540 con 0x7fd6f0103750 2024-05-10T11:09:31.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.677+0000 7fd6ecc00700 1 -- 172.21.15.27:0/2882529580 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6e000cd70 con 0x7fd6f0103750 2024-05-10T11:09:31.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.677+0000 7fd6ecc00700 1 -- 172.21.15.27:0/2882529580 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0018680 con 0x7fd6f0103750 2024-05-10T11:09:31.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.677+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2882529580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f01a87b0 con 0x7fd6f0103750 2024-05-10T11:09:31.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.679+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2882529580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f006a190 con 0x7fd6f0103750 2024-05-10T11:09:31.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.681+0000 7fd6ecc00700 1 -- 172.21.15.27:0/2882529580 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd6e0010430 con 0x7fd6f0103750 2024-05-10T11:09:31.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.683+0000 7fd6ecc00700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd6d40b2990 0x7fd6d40b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:31.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.683+0000 7fd6ecc00700 1 -- 172.21.15.27:0/2882529580 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fd6e00e29d0 con 0x7fd6f0103750 2024-05-10T11:09:31.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.683+0000 7fd6ef400700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd6d40b2990 0x7fd6d40b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:31.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.683+0000 7fd6ef400700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd6d40b2990 0x7fd6d40b4e50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd6f01031e0 tx=0x7fd6dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.687+0000 7fd6ecc00700 1 -- 172.21.15.27:0/2882529580 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fd6e00aa140 con 0x7fd6f0103750 2024-05-10T11:09:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.966+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2882529580 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fd6f010f730 con 0x7fd6d40b2990 2024-05-10T11:09:31.980 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/qa/sv_0/860ce0a6-10fd-4081-a054-16df9467b4e1 2024-05-10T11:09:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.980+0000 7fd6ecc00700 1 -- 172.21.15.27:0/2882529580 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fd6f010f730 con 0x7fd6d40b2990 2024-05-10T11:09:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.983+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2882529580 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd6d40b2990 msgr2=0x7fd6d40b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.983+0000 7fd6f6c00700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd6d40b2990 0x7fd6d40b4e50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd6f01031e0 tx=0x7fd6dc009260 comp rx=0 tx=0).stop 2024-05-10T11:09:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.983+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2882529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f0103750 msgr2=0x7fd6f01a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.983+0000 7fd6f6c00700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f0103750 0x7fd6f01a18e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000e9a0 tx=0x7fd6e000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:09:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.984+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2882529580 shutdown_connections 2024-05-10T11:09:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.984+0000 7fd6f6c00700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd6d40b2990 0x7fd6d40b4e50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.984+0000 7fd6f6c00700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6f0108850 0x7fd6f019baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.984+0000 7fd6f6c00700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f0103750 0x7fd6f01a18e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.984+0000 7fd6f6c00700 1 --2- 172.21.15.27:0/2882529580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f0104100 0x7fd6f01a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.984+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2882529580 >> 172.21.15.27:0/2882529580 conn(0x7fd6f007a600 msgr2=0x7fd6f0101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.984+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2882529580 shutdown_connections 2024-05-10T11:09:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:31.984+0000 7fd6f6c00700 1 -- 172.21.15.27:0/2882529580 wait complete. 2024-05-10T11:09:32.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:32 smithi027 ceph-mon[30054]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:32 smithi076 ceph-mon[34152]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:32.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:32 smithi189 ceph-mon[34842]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:32.832 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-10T11:09:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:33 smithi076 ceph-mon[34152]: pgmap v386: 129 pgs: 129 active+clean; 607 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.9 KiB/s wr, 11 op/s; 71 B/s, 0 keys/s, 1 objects/s recovering 2024-05-10T11:09:33.472 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:33 smithi027 ceph-mon[30054]: pgmap v386: 129 pgs: 129 active+clean; 607 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.9 KiB/s wr, 11 op/s; 71 B/s, 0 keys/s, 1 objects/s recovering 2024-05-10T11:09:33.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:33 smithi189 ceph-mon[34842]: pgmap v386: 129 pgs: 129 active+clean; 607 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.9 KiB/s wr, 11 op/s; 71 B/s, 0 keys/s, 1 objects/s recovering 2024-05-10T11:09:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.621+0000 7f70f0400700 1 -- 172.21.15.27:0/2604599609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70e8075ff0 msgr2=0x7f70e810f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.621+0000 7f70f0400700 1 --2- 172.21.15.27:0/2604599609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70e8075ff0 0x7f70e810f960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f70d40098f0 tx=0x7f70d4009c00 comp rx=0 tx=0).stop 2024-05-10T11:09:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.621+0000 7f70f0400700 1 -- 172.21.15.27:0/2604599609 shutdown_connections 2024-05-10T11:09:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.621+0000 7f70f0400700 1 --2- 172.21.15.27:0/2604599609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70e810fea0 0x7f70e8112290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.621+0000 7f70f0400700 1 --2- 172.21.15.27:0/2604599609 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70e80756d0 0x7f70e8075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.621+0000 7f70f0400700 1 --2- 172.21.15.27:0/2604599609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70e8075ff0 0x7f70e810f960 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.621+0000 7f70f0400700 1 -- 172.21.15.27:0/2604599609 >> 172.21.15.27:0/2604599609 conn(0x7f70e807a600 msgr2=0x7f70e807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.621+0000 7f70f0400700 1 -- 172.21.15.27:0/2604599609 shutdown_connections 2024-05-10T11:09:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.621+0000 7f70f0400700 1 -- 172.21.15.27:0/2604599609 wait complete. 2024-05-10T11:09:35.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.622+0000 7f70f0400700 1 Processor -- start 2024-05-10T11:09:35.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.622+0000 7f70f0400700 1 -- start start 2024-05-10T11:09:35.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.623+0000 7f70f0400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70e80756d0 0x7f70e819be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:35.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.623+0000 7f70f0400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70e8075ff0 0x7f70e819c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:35.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.623+0000 7f70f0400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70e810fea0 0x7f70e81a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:35.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.623+0000 7f70f0400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f70e81158b0 con 0x7f70e810fea0 2024-05-10T11:09:35.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.623+0000 7f70f0400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f70e8115730 con 0x7f70e80756d0 2024-05-10T11:09:35.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.623+0000 7f70f0400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f70e8115a30 con 0x7f70e8075ff0 2024-05-10T11:09:35.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.623+0000 7f70ee400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70e810fea0 0x7f70e81a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:35.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.623+0000 7f70eda00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70e80756d0 0x7f70e819be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:35.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.623+0000 7f70ee400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70e810fea0 0x7f70e81a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60878/0 (socket says 172.21.15.27:60878) 2024-05-10T11:09:35.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.623+0000 7f70ee400700 1 -- 172.21.15.27:0/2558617987 learned_addr learned my addr 172.21.15.27:0/2558617987 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:35.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.623+0000 7f70ed000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70e8075ff0 0x7f70e819c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.624+0000 7f70eda00700 1 -- 172.21.15.27:0/2558617987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70e8075ff0 msgr2=0x7f70e819c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.624+0000 7f70eda00700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70e8075ff0 0x7f70e819c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.624+0000 7f70eda00700 1 -- 172.21.15.27:0/2558617987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70e810fea0 msgr2=0x7f70e81a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.624+0000 7f70eda00700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70e810fea0 0x7f70e81a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.624+0000 7f70eda00700 1 -- 172.21.15.27:0/2558617987 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e81a0de0 con 0x7f70e80756d0 2024-05-10T11:09:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.624+0000 7f70ed000700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70e8075ff0 0x7f70e819c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:09:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.624+0000 7f70eda00700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70e80756d0 0x7f70e819be20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f70dc00e9c0 tx=0x7f70dc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.625+0000 7f70e6000700 1 -- 172.21.15.27:0/2558617987 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc00c9c0 con 0x7f70e80756d0 2024-05-10T11:09:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.625+0000 7f70f0400700 1 -- 172.21.15.27:0/2558617987 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e81a10d0 con 0x7f70e80756d0 2024-05-10T11:09:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.625+0000 7f70e6000700 1 -- 172.21.15.27:0/2558617987 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70dc00cb60 con 0x7f70e80756d0 2024-05-10T11:09:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.625+0000 7f70e6000700 1 -- 172.21.15.27:0/2558617987 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc018420 con 0x7f70e80756d0 2024-05-10T11:09:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.625+0000 7f70f0400700 1 -- 172.21.15.27:0/2558617987 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e81a8950 con 0x7f70e80756d0 2024-05-10T11:09:35.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.626+0000 7f70f0400700 1 -- 172.21.15.27:0/2558617987 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70e806a190 con 0x7f70e80756d0 2024-05-10T11:09:35.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.628+0000 7f70e6000700 1 -- 172.21.15.27:0/2558617987 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f70dc04aae0 con 0x7f70e80756d0 2024-05-10T11:09:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.630+0000 7f70e6000700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f70d00b2910 0x7f70d00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.630+0000 7f70e6000700 1 -- 172.21.15.27:0/2558617987 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f70dc0e28e0 con 0x7f70e80756d0 2024-05-10T11:09:35.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.630+0000 7f70ed000700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f70d00b2910 0x7f70d00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:35.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.631+0000 7f70ed000700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f70d00b2910 0x7f70d00b4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f70d4009fd0 tx=0x7f70d4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:35.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.631+0000 7f70e6000700 1 -- 172.21.15.27:0/2558617987 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f70dc0aa050 con 0x7f70e80756d0 2024-05-10T11:09:35.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.912+0000 7f70f0400700 1 -- 172.21.15.27:0/2558617987 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f70e80638e0 con 0x7f70d00b2910 2024-05-10T11:09:35.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.950+0000 7f70e6000700 1 -- 172.21.15.27:0/2558617987 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f70e80638e0 con 0x7f70d00b2910 2024-05-10T11:09:35.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.953+0000 7f70f0400700 1 -- 172.21.15.27:0/2558617987 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f70d00b2910 msgr2=0x7f70d00b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:35.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.953+0000 7f70f0400700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f70d00b2910 0x7f70d00b4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f70d4009fd0 tx=0x7f70d4019040 comp rx=0 tx=0).stop 2024-05-10T11:09:35.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.953+0000 7f70f0400700 1 -- 172.21.15.27:0/2558617987 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70e80756d0 msgr2=0x7f70e819be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:35.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.953+0000 7f70f0400700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70e80756d0 0x7f70e819be20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f70dc00e9c0 tx=0x7f70dc00ecd0 comp rx=0 tx=0).stop 2024-05-10T11:09:35.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.954+0000 7f70f0400700 1 -- 172.21.15.27:0/2558617987 shutdown_connections 2024-05-10T11:09:35.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.954+0000 7f70f0400700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f70d00b2910 0x7f70d00b4dd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:35.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.954+0000 7f70f0400700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70e810fea0 0x7f70e81a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:35.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.954+0000 7f70f0400700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f70e80756d0 0x7f70e819be20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:35.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.954+0000 7f70f0400700 1 --2- 172.21.15.27:0/2558617987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70e8075ff0 0x7f70e819c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:35.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.954+0000 7f70f0400700 1 -- 172.21.15.27:0/2558617987 >> 172.21.15.27:0/2558617987 conn(0x7f70e807a600 msgr2=0x7f70e8100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:35.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.954+0000 7f70f0400700 1 -- 172.21.15.27:0/2558617987 shutdown_connections 2024-05-10T11:09:35.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:35.954+0000 7f70f0400700 1 -- 172.21.15.27:0/2558617987 wait complete. 2024-05-10T11:09:36.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:35 smithi076 ceph-mon[34152]: pgmap v387: 129 pgs: 129 active+clean; 607 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s; 60 B/s, 0 keys/s, 1 objects/s recovering 2024-05-10T11:09:36.190 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:35 smithi189 ceph-mon[34842]: pgmap v387: 129 pgs: 129 active+clean; 607 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s; 60 B/s, 0 keys/s, 1 objects/s recovering 2024-05-10T11:09:36.331 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:35 smithi027 ceph-mon[30054]: pgmap v387: 129 pgs: 129 active+clean; 607 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s; 60 B/s, 0 keys/s, 1 objects/s recovering 2024-05-10T11:09:36.729 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-10T11:09:36.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:36 smithi027 ceph-mon[30054]: from='client.24719 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:37.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:36 smithi076 ceph-mon[34152]: from='client.24719 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:37.187 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:37.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:36 smithi189 ceph-mon[34842]: from='client.24719 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:38.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:37 smithi076 ceph-mon[34152]: pgmap v388: 129 pgs: 129 active+clean; 613 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.0 KiB/s wr, 12 op/s; 59 B/s, 0 keys/s, 1 objects/s recovering 2024-05-10T11:09:38.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:37 smithi189 ceph-mon[34842]: pgmap v388: 129 pgs: 129 active+clean; 613 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.0 KiB/s wr, 12 op/s; 59 B/s, 0 keys/s, 1 objects/s recovering 2024-05-10T11:09:38.257 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:37 smithi027 ceph-mon[30054]: pgmap v388: 129 pgs: 129 active+clean; 613 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.0 KiB/s wr, 12 op/s; 59 B/s, 0 keys/s, 1 objects/s recovering 2024-05-10T11:09:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.523+0000 7f7c22800700 1 -- 172.21.15.27:0/4218411703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c109ad0 msgr2=0x7f7c1c109f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.523+0000 7f7c22800700 1 --2- 172.21.15.27:0/4218411703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c109ad0 0x7f7c1c109f50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009990 tx=0x7f7c04009ca0 comp rx=0 tx=0).stop 2024-05-10T11:09:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.523+0000 7f7c22800700 1 -- 172.21.15.27:0/4218411703 shutdown_connections 2024-05-10T11:09:39.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.523+0000 7f7c22800700 1 --2- 172.21.15.27:0/4218411703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c10a490 0x7f7c1c1169a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:39.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.523+0000 7f7c22800700 1 --2- 172.21.15.27:0/4218411703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c10fb60 0x7f7c1c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:39.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.523+0000 7f7c22800700 1 --2- 172.21.15.27:0/4218411703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c109ad0 0x7f7c1c109f50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:39.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.523+0000 7f7c22800700 1 -- 172.21.15.27:0/4218411703 >> 172.21.15.27:0/4218411703 conn(0x7f7c1c076c70 msgr2=0x7f7c1c1082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:39.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.523+0000 7f7c22800700 1 -- 172.21.15.27:0/4218411703 shutdown_connections 2024-05-10T11:09:39.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.523+0000 7f7c22800700 1 -- 172.21.15.27:0/4218411703 wait complete. 2024-05-10T11:09:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.524+0000 7f7c22800700 1 Processor -- start 2024-05-10T11:09:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.525+0000 7f7c22800700 1 -- start start 2024-05-10T11:09:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.525+0000 7f7c22800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c109ad0 0x7f7c1c1a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.525+0000 7f7c22800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c10a490 0x7f7c1c1a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.525+0000 7f7c22800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c10fb60 0x7f7c1c1a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.525+0000 7f7c22800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c119d20 con 0x7f7c1c10fb60 2024-05-10T11:09:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.525+0000 7f7c22800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c119ba0 con 0x7f7c1c10a490 2024-05-10T11:09:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.525+0000 7f7c22800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c119ea0 con 0x7f7c1c109ad0 2024-05-10T11:09:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.525+0000 7f7c1b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c109ad0 0x7f7c1c1a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.525+0000 7f7c1be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c10fb60 0x7f7c1c1a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:39.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.525+0000 7f7c1b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c109ad0 0x7f7c1c1a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:35302/0 (socket says 172.21.15.27:35302) 2024-05-10T11:09:39.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.525+0000 7f7c1b400700 1 -- 172.21.15.27:0/1995763817 learned_addr learned my addr 172.21.15.27:0/1995763817 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:39.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.526+0000 7f7c13e00700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c10a490 0x7f7c1c1a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:39.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.526+0000 7f7c13e00700 1 -- 172.21.15.27:0/1995763817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c109ad0 msgr2=0x7f7c1c1a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:39.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.526+0000 7f7c13e00700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c109ad0 0x7f7c1c1a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:39.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.526+0000 7f7c13e00700 1 -- 172.21.15.27:0/1995763817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c10fb60 msgr2=0x7f7c1c1a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:39.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.526+0000 7f7c13e00700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c10fb60 0x7f7c1c1a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:39.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.526+0000 7f7c13e00700 1 -- 172.21.15.27:0/1995763817 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1c1a51e0 con 0x7f7c1c10a490 2024-05-10T11:09:39.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.526+0000 7f7c13e00700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c10a490 0x7f7c1c1a0710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c04000c00 tx=0x7f7c040047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:39.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.527+0000 7f7c18c00700 1 -- 172.21.15.27:0/1995763817 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04026070 con 0x7f7c1c10a490 2024-05-10T11:09:39.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.527+0000 7f7c22800700 1 -- 172.21.15.27:0/1995763817 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1c1a5470 con 0x7f7c1c10a490 2024-05-10T11:09:39.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.527+0000 7f7c18c00700 1 -- 172.21.15.27:0/1995763817 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c04021410 con 0x7f7c1c10a490 2024-05-10T11:09:39.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.527+0000 7f7c18c00700 1 -- 172.21.15.27:0/1995763817 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04017650 con 0x7f7c1c10a490 2024-05-10T11:09:39.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.527+0000 7f7c22800700 1 -- 172.21.15.27:0/1995763817 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1c1acd50 con 0x7f7c1c10a490 2024-05-10T11:09:39.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.528+0000 7f7c22800700 1 -- 172.21.15.27:0/1995763817 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c1c06a190 con 0x7f7c1c10a490 2024-05-10T11:09:39.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.530+0000 7f7c18c00700 1 -- 172.21.15.27:0/1995763817 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f7c04029070 con 0x7f7c1c10a490 2024-05-10T11:09:39.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.533+0000 7f7c18c00700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7c000b2910 0x7f7c000b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:39.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.533+0000 7f7c18c00700 1 -- 172.21.15.27:0/1995763817 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f7c040e3ca0 con 0x7f7c1c10a490 2024-05-10T11:09:39.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.533+0000 7f7c1b400700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7c000b2910 0x7f7c000b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:39.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.534+0000 7f7c1b400700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7c000b2910 0x7f7c000b4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c007c10 tx=0x7f7c0c007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:39.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.534+0000 7f7c18c00700 1 -- 172.21.15.27:0/1995763817 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7c04030070 con 0x7f7c1c10a490 2024-05-10T11:09:39.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.810+0000 7f7c22800700 1 -- 172.21.15.27:0/1995763817 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f7c1c1a1530 con 0x7f7c000b2910 2024-05-10T11:09:39.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.823+0000 7f7c18c00700 1 -- 172.21.15.27:0/1995763817 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f7c1c1a1530 con 0x7f7c000b2910 2024-05-10T11:09:39.823 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/qa/sv_1/735a009e-987b-4d8e-9de7-031001ea9af4 2024-05-10T11:09:39.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.826+0000 7f7c22800700 1 -- 172.21.15.27:0/1995763817 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7c000b2910 msgr2=0x7f7c000b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:39.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.826+0000 7f7c22800700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7c000b2910 0x7f7c000b4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c007c10 tx=0x7f7c0c007480 comp rx=0 tx=0).stop 2024-05-10T11:09:39.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.826+0000 7f7c22800700 1 -- 172.21.15.27:0/1995763817 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c10a490 msgr2=0x7f7c1c1a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:39.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.826+0000 7f7c22800700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c10a490 0x7f7c1c1a0710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c04000c00 tx=0x7f7c040047b0 comp rx=0 tx=0).stop 2024-05-10T11:09:39.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.827+0000 7f7c22800700 1 -- 172.21.15.27:0/1995763817 shutdown_connections 2024-05-10T11:09:39.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.827+0000 7f7c22800700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7c000b2910 0x7f7c000b4dd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:39.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.827+0000 7f7c22800700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c109ad0 0x7f7c1c1a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:39.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.827+0000 7f7c22800700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c10fb60 0x7f7c1c1a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:39.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.827+0000 7f7c22800700 1 --2- 172.21.15.27:0/1995763817 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c10a490 0x7f7c1c1a0710 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:39.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.827+0000 7f7c22800700 1 -- 172.21.15.27:0/1995763817 >> 172.21.15.27:0/1995763817 conn(0x7f7c1c076c70 msgr2=0x7f7c1c077290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:39.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.827+0000 7f7c22800700 1 -- 172.21.15.27:0/1995763817 shutdown_connections 2024-05-10T11:09:39.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:39.827+0000 7f7c22800700 1 -- 172.21.15.27:0/1995763817 wait complete. 2024-05-10T11:09:40.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:39 smithi027 ceph-mon[30054]: pgmap v389: 129 pgs: 129 active+clean; 613 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.5 KiB/s wr, 11 op/s 2024-05-10T11:09:40.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:39 smithi076 ceph-mon[34152]: pgmap v389: 129 pgs: 129 active+clean; 613 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.5 KiB/s wr, 11 op/s 2024-05-10T11:09:40.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:39 smithi189 ceph-mon[34842]: pgmap v389: 129 pgs: 129 active+clean; 613 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.5 KiB/s wr, 11 op/s 2024-05-10T11:09:40.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:09:41.150 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:41.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:40 smithi076 ceph-mon[34152]: from='client.24725 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:41.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:40 smithi189 ceph-mon[34842]: from='client.24725 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:41.298 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:40 smithi027 ceph-mon[30054]: from='client.24725 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:09:42.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:41 smithi027 ceph-mon[30054]: pgmap v390: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 4.6 KiB/s wr, 13 op/s 2024-05-10T11:09:42.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:09:42.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:42.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:41 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:42.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:41 smithi076 ceph-mon[34152]: pgmap v390: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 4.6 KiB/s wr, 13 op/s 2024-05-10T11:09:42.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:09:42.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:42.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:41 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:42.234 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:41 smithi189 ceph-mon[34842]: pgmap v390: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 4.6 KiB/s wr, 13 op/s 2024-05-10T11:09:42.234 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:09:42.234 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:42.234 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:41 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.350+0000 7f7662200700 1 -- 172.21.15.27:0/472558856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765c06b6f0 msgr2=0x7f765c06bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.350+0000 7f7662200700 1 --2- 172.21.15.27:0/472558856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765c06b6f0 0x7f765c06bb50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7644009990 tx=0x7f7644009ca0 comp rx=0 tx=0).stop 2024-05-10T11:09:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.350+0000 7f7662200700 1 -- 172.21.15.27:0/472558856 shutdown_connections 2024-05-10T11:09:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.350+0000 7f7662200700 1 --2- 172.21.15.27:0/472558856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765c105ee0 0x7f765c106370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.350+0000 7f7662200700 1 --2- 172.21.15.27:0/472558856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c1083d0 0x7f765c06b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.350+0000 7f7662200700 1 --2- 172.21.15.27:0/472558856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765c06b6f0 0x7f765c06bb50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.350+0000 7f7662200700 1 -- 172.21.15.27:0/472558856 >> 172.21.15.27:0/472558856 conn(0x7f765c079350 msgr2=0x7f765c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.350+0000 7f7662200700 1 -- 172.21.15.27:0/472558856 shutdown_connections 2024-05-10T11:09:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.350+0000 7f7662200700 1 -- 172.21.15.27:0/472558856 wait complete. 2024-05-10T11:09:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.351+0000 7f7662200700 1 Processor -- start 2024-05-10T11:09:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.351+0000 7f7662200700 1 -- start start 2024-05-10T11:09:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.352+0000 7f7662200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765c06b6f0 0x7f765c1a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.352+0000 7f7662200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c105ee0 0x7f765c1a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.352+0000 7f7662200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765c1083d0 0x7f765c1a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.352+0000 7f7662200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f765c119c80 con 0x7f765c105ee0 2024-05-10T11:09:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.352+0000 7f7662200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f765c119b00 con 0x7f765c06b6f0 2024-05-10T11:09:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.352+0000 7f7662200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f765c119e00 con 0x7f765c1083d0 2024-05-10T11:09:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.352+0000 7f7653e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c105ee0 0x7f765c1a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.352+0000 7f765b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765c06b6f0 0x7f765c1a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.352+0000 7f7653e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c105ee0 0x7f765c1a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60916/0 (socket says 172.21.15.27:60916) 2024-05-10T11:09:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.352+0000 7f7653e00700 1 -- 172.21.15.27:0/2432638705 learned_addr learned my addr 172.21.15.27:0/2432638705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.352+0000 7f765be00700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765c1083d0 0x7f765c1a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.353+0000 7f7653e00700 1 -- 172.21.15.27:0/2432638705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765c1083d0 msgr2=0x7f765c1a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.353+0000 7f7653e00700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765c1083d0 0x7f765c1a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.353+0000 7f7653e00700 1 -- 172.21.15.27:0/2432638705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765c06b6f0 msgr2=0x7f765c1a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.353+0000 7f7653e00700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765c06b6f0 0x7f765c1a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.353+0000 7f7653e00700 1 -- 172.21.15.27:0/2432638705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765c1a51c0 con 0x7f765c105ee0 2024-05-10T11:09:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.353+0000 7f765be00700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765c1083d0 0x7f765c1a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:09:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.353+0000 7f765b400700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765c06b6f0 0x7f765c1a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:09:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.353+0000 7f7653e00700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c105ee0 0x7f765c1a06f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7644005910 tx=0x7f764400b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.353+0000 7f7658c00700 1 -- 172.21.15.27:0/2432638705 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7644027070 con 0x7f765c105ee0 2024-05-10T11:09:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.353+0000 7f7662200700 1 -- 172.21.15.27:0/2432638705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f765c1a5450 con 0x7f765c105ee0 2024-05-10T11:09:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.354+0000 7f7658c00700 1 -- 172.21.15.27:0/2432638705 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7644022470 con 0x7f765c105ee0 2024-05-10T11:09:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.354+0000 7f7658c00700 1 -- 172.21.15.27:0/2432638705 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7644017650 con 0x7f765c105ee0 2024-05-10T11:09:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.354+0000 7f7662200700 1 -- 172.21.15.27:0/2432638705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f765c1acd30 con 0x7f765c105ee0 2024-05-10T11:09:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.355+0000 7f7662200700 1 -- 172.21.15.27:0/2432638705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f765c06a190 con 0x7f765c105ee0 2024-05-10T11:09:43.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.358+0000 7f7658c00700 1 -- 172.21.15.27:0/2432638705 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f764402a070 con 0x7f765c105ee0 2024-05-10T11:09:43.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.359+0000 7f7658c00700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76400b2990 0x7f76400b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.360+0000 7f7658c00700 1 -- 172.21.15.27:0/2432638705 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f76440e3670 con 0x7f765c105ee0 2024-05-10T11:09:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.360+0000 7f765b400700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76400b2990 0x7f76400b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:43.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.360+0000 7f765b400700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76400b2990 0x7f76400b4e50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f764c009fd0 tx=0x7f764c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.362+0000 7f7658c00700 1 -- 172.21.15.27:0/2432638705 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7644013070 con 0x7f765c105ee0 2024-05-10T11:09:43.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.656+0000 7f7662200700 1 -- 172.21.15.27:0/2432638705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f765c1a1510 con 0x7f765c105ee0 2024-05-10T11:09:43.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.657+0000 7f7658c00700 1 -- 172.21.15.27:0/2432638705 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11936 (secure 0 0 0) 0x7f765c1a1510 con 0x7f765c105ee0 2024-05-10T11:09:43.657 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:09:43.657 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:09:19.986922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:09:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.661+0000 7f7662200700 1 -- 172.21.15.27:0/2432638705 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76400b2990 msgr2=0x7f76400b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.661+0000 7f7662200700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76400b2990 0x7f76400b4e50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f764c009fd0 tx=0x7f764c009260 comp rx=0 tx=0).stop 2024-05-10T11:09:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.661+0000 7f7662200700 1 -- 172.21.15.27:0/2432638705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c105ee0 msgr2=0x7f765c1a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.661+0000 7f7662200700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c105ee0 0x7f765c1a06f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7644005910 tx=0x7f764400b670 comp rx=0 tx=0).stop 2024-05-10T11:09:43.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.661+0000 7f7662200700 1 -- 172.21.15.27:0/2432638705 shutdown_connections 2024-05-10T11:09:43.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.661+0000 7f7662200700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76400b2990 0x7f76400b4e50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:43.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.661+0000 7f7662200700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765c1083d0 0x7f765c1a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:43.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.661+0000 7f7662200700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765c06b6f0 0x7f765c1a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:43.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.661+0000 7f7662200700 1 --2- 172.21.15.27:0/2432638705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c105ee0 0x7f765c1a06f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:43.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.661+0000 7f7662200700 1 -- 172.21.15.27:0/2432638705 >> 172.21.15.27:0/2432638705 conn(0x7f765c079350 msgr2=0x7f765c104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:43.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.661+0000 7f7662200700 1 -- 172.21.15.27:0/2432638705 shutdown_connections 2024-05-10T11:09:43.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:43.661+0000 7f7662200700 1 -- 172.21.15.27:0/2432638705 wait complete. 2024-05-10T11:09:43.662 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-05-10T11:09:43.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:43.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:43.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:43 smithi027 ceph-mon[30054]: pgmap v391: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 12 op/s 2024-05-10T11:09:43.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:43.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:43 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:43.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:43 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2432638705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:09:43.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:43.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:43.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:43 smithi189 ceph-mon[34842]: pgmap v391: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 12 op/s 2024-05-10T11:09:43.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:43.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:43 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:43.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:43 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2432638705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:09:44.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:44.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:44.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:43 smithi076 ceph-mon[34152]: pgmap v391: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 12 op/s 2024-05-10T11:09:44.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:44.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:43 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:44.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:43 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2432638705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:09:44.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:09:45.271 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:46.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:45 smithi027 ceph-mon[30054]: pgmap v392: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-05-10T11:09:46.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:45 smithi076 ceph-mon[34152]: pgmap v392: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-05-10T11:09:46.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:45 smithi189 ceph-mon[34842]: pgmap v392: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-05-10T11:09:47.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:47.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:47 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:47.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:47 smithi027 ceph-mon[30054]: pgmap v393: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.7 KiB/s wr, 13 op/s 2024-05-10T11:09:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.605+0000 7f758b800700 1 -- 172.21.15.27:0/920880476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7584109510 msgr2=0x7f75841098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.605+0000 7f758b800700 1 --2- 172.21.15.27:0/920880476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7584109510 0x7f75841098f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7578009990 tx=0x7f7578009ca0 comp rx=0 tx=0).stop 2024-05-10T11:09:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.605+0000 7f758b800700 1 -- 172.21.15.27:0/920880476 shutdown_connections 2024-05-10T11:09:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.605+0000 7f758b800700 1 --2- 172.21.15.27:0/920880476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f758406bb40 0x7f7584110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.605+0000 7f758b800700 1 --2- 172.21.15.27:0/920880476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7584109510 0x7f75841098f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.605+0000 7f758b800700 1 --2- 172.21.15.27:0/920880476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f758406b180 0x7f758406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.605+0000 7f758b800700 1 -- 172.21.15.27:0/920880476 >> 172.21.15.27:0/920880476 conn(0x7f7584079350 msgr2=0x7f7584079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.605+0000 7f758b800700 1 -- 172.21.15.27:0/920880476 shutdown_connections 2024-05-10T11:09:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.605+0000 7f758b800700 1 -- 172.21.15.27:0/920880476 wait complete. 2024-05-10T11:09:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.607+0000 7f758b800700 1 Processor -- start 2024-05-10T11:09:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.607+0000 7f758b800700 1 -- start start 2024-05-10T11:09:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.607+0000 7f758b800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f758406b180 0x7f7584102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.607+0000 7f758b800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f758406bb40 0x7f75841029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.607+0000 7f758b800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7584109510 0x7f7584102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.607+0000 7f758b800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7584113640 con 0x7f758406bb40 2024-05-10T11:09:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.607+0000 7f758b800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f75841134c0 con 0x7f7584109510 2024-05-10T11:09:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.607+0000 7f758b800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75841137c0 con 0x7f758406b180 2024-05-10T11:09:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.608+0000 7f7583e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f758406bb40 0x7f75841029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.608+0000 7f7588e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f758406b180 0x7f7584102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.608+0000 7f7583e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f758406bb40 0x7f75841029b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44176/0 (socket says 172.21.15.27:44176) 2024-05-10T11:09:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.608+0000 7f7583e00700 1 -- 172.21.15.27:0/2561134217 learned_addr learned my addr 172.21.15.27:0/2561134217 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.608+0000 7f7589800700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7584109510 0x7f7584102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.608+0000 7f7588e00700 1 -- 172.21.15.27:0/2561134217 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7584109510 msgr2=0x7f7584102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.608+0000 7f7588e00700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7584109510 0x7f7584102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.608+0000 7f7588e00700 1 -- 172.21.15.27:0/2561134217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f758406bb40 msgr2=0x7f75841029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.608+0000 7f7588e00700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f758406bb40 0x7f75841029b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.608+0000 7f7588e00700 1 -- 172.21.15.27:0/2561134217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75841a8090 con 0x7f758406b180 2024-05-10T11:09:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.609+0000 7f7588e00700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f758406b180 0x7f7584102470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7578006010 tx=0x7f75780057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.609+0000 7f7581600700 1 -- 172.21.15.27:0/2561134217 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7578027070 con 0x7f758406b180 2024-05-10T11:09:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.609+0000 7f7581600700 1 -- 172.21.15.27:0/2561134217 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7578022470 con 0x7f758406b180 2024-05-10T11:09:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.609+0000 7f758b800700 1 -- 172.21.15.27:0/2561134217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75841a8320 con 0x7f758406b180 2024-05-10T11:09:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.609+0000 7f7581600700 1 -- 172.21.15.27:0/2561134217 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7578017650 con 0x7f758406b180 2024-05-10T11:09:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.609+0000 7f758b800700 1 -- 172.21.15.27:0/2561134217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75841a8770 con 0x7f758406b180 2024-05-10T11:09:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.610+0000 7f758b800700 1 -- 172.21.15.27:0/2561134217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f758410b9f0 con 0x7f758406b180 2024-05-10T11:09:47.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.612+0000 7f7581600700 1 -- 172.21.15.27:0/2561134217 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f757802a070 con 0x7f758406b180 2024-05-10T11:09:47.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.614+0000 7f7581600700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f756c0b2910 0x7f756c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:47.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.614+0000 7f7581600700 1 -- 172.21.15.27:0/2561134217 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f75780e3410 con 0x7f758406b180 2024-05-10T11:09:47.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.614+0000 7f7583e00700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f756c0b2910 0x7f756c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:47.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.615+0000 7f7583e00700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f756c0b2910 0x7f756c0b4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f75840799c0 tx=0x7f7570009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:47.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.616+0000 7f7581600700 1 -- 172.21.15.27:0/2561134217 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7578013070 con 0x7f758406b180 2024-05-10T11:09:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:47 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:47 smithi076 ceph-mon[34152]: pgmap v393: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.7 KiB/s wr, 13 op/s 2024-05-10T11:09:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:47.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:47 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:47.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:47 smithi189 ceph-mon[34842]: pgmap v393: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.7 KiB/s wr, 13 op/s 2024-05-10T11:09:47.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.860+0000 7f758b800700 1 -- 172.21.15.27:0/2561134217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f758410b3d0 con 0x7f758406b180 2024-05-10T11:09:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.863+0000 7f7581600700 1 -- 172.21.15.27:0/2561134217 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24835 (secure 0 0 0) 0x7f75780aac10 con 0x7f758406b180 2024-05-10T11:09:47.863 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:09:47.863 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":90,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:09:23.055559+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6},{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:09:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.866+0000 7f758b800700 1 -- 172.21.15.27:0/2561134217 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f756c0b2910 msgr2=0x7f756c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.866+0000 7f758b800700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f756c0b2910 0x7f756c0b4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f75840799c0 tx=0x7f7570009260 comp rx=0 tx=0).stop 2024-05-10T11:09:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.866+0000 7f758b800700 1 -- 172.21.15.27:0/2561134217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f758406b180 msgr2=0x7f7584102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.866+0000 7f758b800700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f758406b180 0x7f7584102470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7578006010 tx=0x7f75780057c0 comp rx=0 tx=0).stop 2024-05-10T11:09:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.867+0000 7f758b800700 1 -- 172.21.15.27:0/2561134217 shutdown_connections 2024-05-10T11:09:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.867+0000 7f758b800700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f756c0b2910 0x7f756c0b4dd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.867+0000 7f758b800700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f758406b180 0x7f7584102470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.867+0000 7f758b800700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7584109510 0x7f7584102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.867+0000 7f758b800700 1 --2- 172.21.15.27:0/2561134217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f758406bb40 0x7f75841029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.867+0000 7f758b800700 1 -- 172.21.15.27:0/2561134217 >> 172.21.15.27:0/2561134217 conn(0x7f7584079350 msgr2=0x7f75841010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.867+0000 7f758b800700 1 -- 172.21.15.27:0/2561134217 shutdown_connections 2024-05-10T11:09:47.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:47.867+0000 7f758b800700 1 -- 172.21.15.27:0/2561134217 wait complete. 2024-05-10T11:09:48.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:48 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:09:48.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:48 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:09:48.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:48 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:48.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:48 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2561134217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:09:48.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:48 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:09:48.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:48 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:09:48.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:48 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:48.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:48 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2561134217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:09:48.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:48 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:09:48.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:48 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:09:48.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:48 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:09:48.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:48 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2561134217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:09:48.777 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-10T11:09:48.778 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-10T11:09:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:49.406 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:49 smithi027 ceph-mon[30054]: pgmap v394: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-05-10T11:09:49.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:49 smithi076 ceph-mon[34152]: pgmap v394: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-05-10T11:09:49.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:49 smithi189 ceph-mon[34842]: pgmap v394: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-05-10T11:09:51.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.838+0000 7fa2df800700 1 -- 172.21.15.27:0/919031376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2d8109500 msgr2=0x7fa2d81098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:51.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.838+0000 7fa2df800700 1 --2- 172.21.15.27:0/919031376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2d8109500 0x7fa2d81098e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009990 tx=0x7fa2cc009ca0 comp rx=0 tx=0).stop 2024-05-10T11:09:51.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.838+0000 7fa2df800700 1 -- 172.21.15.27:0/919031376 shutdown_connections 2024-05-10T11:09:51.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.838+0000 7fa2df800700 1 --2- 172.21.15.27:0/919031376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d806c3d0 0x7fa2d8110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:51.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.838+0000 7fa2df800700 1 --2- 172.21.15.27:0/919031376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2d8109500 0x7fa2d81098e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:51.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.838+0000 7fa2df800700 1 --2- 172.21.15.27:0/919031376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d806ba10 0x7fa2d806be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:51.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.838+0000 7fa2df800700 1 -- 172.21.15.27:0/919031376 >> 172.21.15.27:0/919031376 conn(0x7fa2d8079310 msgr2=0x7fa2d8079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:51.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.838+0000 7fa2df800700 1 -- 172.21.15.27:0/919031376 shutdown_connections 2024-05-10T11:09:51.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.838+0000 7fa2df800700 1 -- 172.21.15.27:0/919031376 wait complete. 2024-05-10T11:09:51.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.840+0000 7fa2df800700 1 Processor -- start 2024-05-10T11:09:51.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.840+0000 7fa2df800700 1 -- start start 2024-05-10T11:09:51.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.841+0000 7fa2df800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d806ba10 0x7fa2d819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:51.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.841+0000 7fa2df800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d806c3d0 0x7fa2d819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:51.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.841+0000 7fa2df800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2d8109500 0x7fa2d81a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:51.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.841+0000 7fa2df800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8113680 con 0x7fa2d806c3d0 2024-05-10T11:09:51.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.841+0000 7fa2df800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8113500 con 0x7fa2d8109500 2024-05-10T11:09:51.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.841+0000 7fa2df800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8113800 con 0x7fa2d806ba10 2024-05-10T11:09:51.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.841+0000 7fa2dce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d806ba10 0x7fa2d819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:51.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.841+0000 7fa2dce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d806ba10 0x7fa2d819bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:47388/0 (socket says 172.21.15.27:47388) 2024-05-10T11:09:51.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.841+0000 7fa2dce00700 1 -- 172.21.15.27:0/2340802570 learned_addr learned my addr 172.21.15.27:0/2340802570 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:51.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.841+0000 7fa2d7e00700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d806c3d0 0x7fa2d819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:51.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.842+0000 7fa2dd800700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2d8109500 0x7fa2d81a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:51.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.842+0000 7fa2dce00700 1 -- 172.21.15.27:0/2340802570 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2d8109500 msgr2=0x7fa2d81a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:51.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.842+0000 7fa2dce00700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2d8109500 0x7fa2d81a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:51.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.842+0000 7fa2dce00700 1 -- 172.21.15.27:0/2340802570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d806c3d0 msgr2=0x7fa2d819c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:51.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.842+0000 7fa2dce00700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d806c3d0 0x7fa2d819c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:51.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.842+0000 7fa2dce00700 1 -- 172.21.15.27:0/2340802570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d81a0da0 con 0x7fa2d806ba10 2024-05-10T11:09:51.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.842+0000 7fa2dce00700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d806ba10 0x7fa2d819bd90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc005910 tx=0x7fa2cc00b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:51.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.842+0000 7fa2d5600700 1 -- 172.21.15.27:0/2340802570 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc027070 con 0x7fa2d806ba10 2024-05-10T11:09:51.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.842+0000 7fa2d5600700 1 -- 172.21.15.27:0/2340802570 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2cc022470 con 0x7fa2d806ba10 2024-05-10T11:09:51.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.842+0000 7fa2d5600700 1 -- 172.21.15.27:0/2340802570 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc017650 con 0x7fa2d806ba10 2024-05-10T11:09:51.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.842+0000 7fa2df800700 1 -- 172.21.15.27:0/2340802570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2d81a1030 con 0x7fa2d806ba10 2024-05-10T11:09:51.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.843+0000 7fa2df800700 1 -- 172.21.15.27:0/2340802570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2d81a8910 con 0x7fa2d806ba10 2024-05-10T11:09:51.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.844+0000 7fa2df800700 1 -- 172.21.15.27:0/2340802570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2d806a190 con 0x7fa2d806ba10 2024-05-10T11:09:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.846+0000 7fa2d5600700 1 -- 172.21.15.27:0/2340802570 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fa2cc02a070 con 0x7fa2d806ba10 2024-05-10T11:09:51.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.848+0000 7fa2d5600700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa2c00b2990 0x7fa2c00b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:51.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.848+0000 7fa2d5600700 1 -- 172.21.15.27:0/2340802570 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fa2cc0e36b0 con 0x7fa2d806ba10 2024-05-10T11:09:51.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.848+0000 7fa2d7e00700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa2c00b2990 0x7fa2c00b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:51.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.849+0000 7fa2d7e00700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa2c00b2990 0x7fa2c00b4e50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8110760 tx=0x7fa2c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:51.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:51.850+0000 7fa2d5600700 1 -- 172.21.15.27:0/2340802570 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa2cc013070 con 0x7fa2d806ba10 2024-05-10T11:09:52.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.081+0000 7fa2df800700 1 -- 172.21.15.27:0/2340802570 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa2d81015e0 con 0x7fa2c00b2990 2024-05-10T11:09:52.083 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:51 smithi027 ceph-mon[30054]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.3 KiB/s wr, 12 op/s 2024-05-10T11:09:52.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.099+0000 7fa2d5600700 1 -- 172.21.15.27:0/2340802570 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340662 (secure 0 0 0) 0x7fa2d81015e0 con 0x7fa2c00b2990 2024-05-10T11:09:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:09:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.107+0000 7fa2df800700 1 -- 172.21.15.27:0/2340802570 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa2c00b2990 msgr2=0x7fa2c00b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.107+0000 7fa2df800700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa2c00b2990 0x7fa2c00b4e50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8110760 tx=0x7fa2c4009260 comp rx=0 tx=0).stop 2024-05-10T11:09:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.107+0000 7fa2df800700 1 -- 172.21.15.27:0/2340802570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d806ba10 msgr2=0x7fa2d819bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.107+0000 7fa2df800700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d806ba10 0x7fa2d819bd90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc005910 tx=0x7fa2cc00b670 comp rx=0 tx=0).stop 2024-05-10T11:09:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.107+0000 7fa2df800700 1 -- 172.21.15.27:0/2340802570 shutdown_connections 2024-05-10T11:09:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.107+0000 7fa2df800700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa2c00b2990 0x7fa2c00b4e50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.107+0000 7fa2df800700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d806ba10 0x7fa2d819bd90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.107+0000 7fa2df800700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2d8109500 0x7fa2d81a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.107+0000 7fa2df800700 1 --2- 172.21.15.27:0/2340802570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d806c3d0 0x7fa2d819c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.107+0000 7fa2df800700 1 -- 172.21.15.27:0/2340802570 >> 172.21.15.27:0/2340802570 conn(0x7fa2d8079310 msgr2=0x7fa2d8100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.107+0000 7fa2df800700 1 -- 172.21.15.27:0/2340802570 shutdown_connections 2024-05-10T11:09:52.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:52.107+0000 7fa2df800700 1 -- 172.21.15.27:0/2340802570 wait complete. 2024-05-10T11:09:52.110 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-10T11:09:52.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:51 smithi076 ceph-mon[34152]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.3 KiB/s wr, 12 op/s 2024-05-10T11:09:52.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:51 smithi189 ceph-mon[34842]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.3 KiB/s wr, 12 op/s 2024-05-10T11:09:52.907 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-05-10T11:09:50.828744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662060,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":449,"num_read_kb":338,"num_write":375,"num_write_kb":2296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":2952552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361420,"kb_used_data":12256,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580404,"statfs":{"total":1151940427776,"available":1151570333696,"internally_reserved":0,"allocated":12550144,"data_stored":8221077,"data_compressed":497229,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34300,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":134,"num_read_kb":106,"num_write":18,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005299"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066004+0000","last_change":"2024-05-10T11:07:47.212921+0000","last_active":"2024-05-10T11:09:23.066004+0000","last_peered":"2024-05-10T11:09:23.066004+0000","last_clean":"2024-05-10T11:09:23.066004+0000","last_became_active":"2024-05-10T11:07:47.212558+0000","last_became_peered":"2024-05-10T11:07:47.212558+0000","last_unstale":"2024-05-10T11:09:23.066004+0000","last_undegraded":"2024-05-10T11:09:23.066004+0000","last_fullsized":"2024-05-10T11:09:23.066004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:47:20.415883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067896+0000","last_change":"2024-05-10T11:07:47.211186+0000","last_active":"2024-05-10T11:09:23.067896+0000","last_peered":"2024-05-10T11:09:23.067896+0000","last_clean":"2024-05-10T11:09:23.067896+0000","last_became_active":"2024-05-10T11:07:47.203487+0000","last_became_peered":"2024-05-10T11:07:47.203487+0000","last_unstale":"2024-05-10T11:09:23.067896+0000","last_undegraded":"2024-05-10T11:09:23.067896+0000","last_fullsized":"2024-05-10T11:09:23.067896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:51:29.674632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066966+0000","last_change":"2024-05-10T11:07:47.202718+0000","last_active":"2024-05-10T11:09:23.066966+0000","last_peered":"2024-05-10T11:09:23.066966+0000","last_clean":"2024-05-10T11:09:23.066966+0000","last_became_active":"2024-05-10T11:07:47.202048+0000","last_became_peered":"2024-05-10T11:07:47.202048+0000","last_unstale":"2024-05-10T11:09:23.066966+0000","last_undegraded":"2024-05-10T11:09:23.066966+0000","last_fullsized":"2024-05-10T11:09:23.066966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:01:41.156850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062967+0000","last_change":"2024-05-10T11:07:47.193519+0000","last_active":"2024-05-10T11:09:23.062967+0000","last_peered":"2024-05-10T11:09:23.062967+0000","last_clean":"2024-05-10T11:09:23.062967+0000","last_became_active":"2024-05-10T11:07:47.193200+0000","last_became_peered":"2024-05-10T11:07:47.193200+0000","last_unstale":"2024-05-10T11:09:23.062967+0000","last_undegraded":"2024-05-10T11:09:23.062967+0000","last_fullsized":"2024-05-10T11:09:23.062967+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:01.949453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068968+0000","last_change":"2024-05-10T11:07:47.207750+0000","last_active":"2024-05-10T11:09:23.068968+0000","last_peered":"2024-05-10T11:09:23.068968+0000","last_clean":"2024-05-10T11:09:23.068968+0000","last_became_active":"2024-05-10T11:07:47.207384+0000","last_became_peered":"2024-05-10T11:07:47.207384+0000","last_unstale":"2024-05-10T11:09:23.068968+0000","last_undegraded":"2024-05-10T11:09:23.068968+0000","last_fullsized":"2024-05-10T11:09:23.068968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:09.432766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068679+0000","last_change":"2024-05-10T11:08:22.245930+0000","last_active":"2024-05-10T11:09:23.068679+0000","last_peered":"2024-05-10T11:09:23.068679+0000","last_clean":"2024-05-10T11:09:23.068679+0000","last_became_active":"2024-05-10T11:08:22.245553+0000","last_became_peered":"2024-05-10T11:08:22.245553+0000","last_unstale":"2024-05-10T11:09:23.068679+0000","last_undegraded":"2024-05-10T11:09:23.068679+0000","last_fullsized":"2024-05-10T11:09:23.068679+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:47:31.485623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071570+0000","last_change":"2024-05-10T11:07:47.196666+0000","last_active":"2024-05-10T11:09:23.071570+0000","last_peered":"2024-05-10T11:09:23.071570+0000","last_clean":"2024-05-10T11:09:23.071570+0000","last_became_active":"2024-05-10T11:07:47.196300+0000","last_became_peered":"2024-05-10T11:07:47.196300+0000","last_unstale":"2024-05-10T11:09:23.071570+0000","last_undegraded":"2024-05-10T11:09:23.071570+0000","last_fullsized":"2024-05-10T11:09:23.071570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:55:24.343385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076560+0000","last_change":"2024-05-10T11:07:47.213048+0000","last_active":"2024-05-10T11:09:23.076560+0000","last_peered":"2024-05-10T11:09:23.076560+0000","last_clean":"2024-05-10T11:09:23.076560+0000","last_became_active":"2024-05-10T11:07:47.206359+0000","last_became_peered":"2024-05-10T11:07:47.206359+0000","last_unstale":"2024-05-10T11:09:23.076560+0000","last_undegraded":"2024-05-10T11:09:23.076560+0000","last_fullsized":"2024-05-10T11:09:23.076560+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:32:56.307261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.073801+0000","last_change":"2024-05-10T11:07:47.190945+0000","last_active":"2024-05-10T11:09:23.073801+0000","last_peered":"2024-05-10T11:09:23.073801+0000","last_clean":"2024-05-10T11:09:23.073801+0000","last_became_active":"2024-05-10T11:07:47.190631+0000","last_became_peered":"2024-05-10T11:07:47.190631+0000","last_unstale":"2024-05-10T11:09:23.073801+0000","last_undegraded":"2024-05-10T11:09:23.073801+0000","last_fullsized":"2024-05-10T11:09:23.073801+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:34:08.683912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069521+0000","last_change":"2024-05-10T11:07:47.189540+0000","last_active":"2024-05-10T11:09:23.069521+0000","last_peered":"2024-05-10T11:09:23.069521+0000","last_clean":"2024-05-10T11:09:23.069521+0000","last_became_active":"2024-05-10T11:07:47.189185+0000","last_became_peered":"2024-05-10T11:07:47.189185+0000","last_unstale":"2024-05-10T11:09:23.069521+0000","last_undegraded":"2024-05-10T11:09:23.069521+0000","last_fullsized":"2024-05-10T11:09:23.069521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:55:23.892826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071768+0000","last_change":"2024-05-10T11:07:47.195688+0000","last_active":"2024-05-10T11:09:23.071768+0000","last_peered":"2024-05-10T11:09:23.071768+0000","last_clean":"2024-05-10T11:09:23.071768+0000","last_became_active":"2024-05-10T11:07:47.195081+0000","last_became_peered":"2024-05-10T11:07:47.195081+0000","last_unstale":"2024-05-10T11:09:23.071768+0000","last_undegraded":"2024-05-10T11:09:23.071768+0000","last_fullsized":"2024-05-10T11:09:23.071768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:20:55.240827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062915+0000","last_change":"2024-05-10T11:07:47.193736+0000","last_active":"2024-05-10T11:09:23.062915+0000","last_peered":"2024-05-10T11:09:23.062915+0000","last_clean":"2024-05-10T11:09:23.062915+0000","last_became_active":"2024-05-10T11:07:47.193400+0000","last_became_peered":"2024-05-10T11:07:47.193400+0000","last_unstale":"2024-05-10T11:09:23.062915+0000","last_undegraded":"2024-05-10T11:09:23.062915+0000","last_fullsized":"2024-05-10T11:09:23.062915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:46:28.507330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.080862+0000","last_change":"2024-05-10T11:08:22.042849+0000","last_active":"2024-05-10T11:09:23.080862+0000","last_peered":"2024-05-10T11:09:23.080862+0000","last_clean":"2024-05-10T11:09:23.080862+0000","last_became_active":"2024-05-10T11:08:22.042024+0000","last_became_peered":"2024-05-10T11:08:22.042024+0000","last_unstale":"2024-05-10T11:09:23.080862+0000","last_undegraded":"2024-05-10T11:09:23.080862+0000","last_fullsized":"2024-05-10T11:09:23.080862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:29:08.096541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067382+0000","last_change":"2024-05-10T11:07:47.214016+0000","last_active":"2024-05-10T11:09:23.067382+0000","last_peered":"2024-05-10T11:09:23.067382+0000","last_clean":"2024-05-10T11:09:23.067382+0000","last_became_active":"2024-05-10T11:07:47.209658+0000","last_became_peered":"2024-05-10T11:07:47.209658+0000","last_unstale":"2024-05-10T11:09:23.067382+0000","last_undegraded":"2024-05-10T11:09:23.067382+0000","last_fullsized":"2024-05-10T11:09:23.067382+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:44.771221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067118+0000","last_change":"2024-05-10T11:07:47.210153+0000","last_active":"2024-05-10T11:09:23.067118+0000","last_peered":"2024-05-10T11:09:23.067118+0000","last_clean":"2024-05-10T11:09:23.067118+0000","last_became_active":"2024-05-10T11:07:47.208470+0000","last_became_peered":"2024-05-10T11:07:47.208470+0000","last_unstale":"2024-05-10T11:09:23.067118+0000","last_undegraded":"2024-05-10T11:09:23.067118+0000","last_fullsized":"2024-05-10T11:09:23.067118+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:23:03.614670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.073109+0000","last_change":"2024-05-10T11:07:47.210704+0000","last_active":"2024-05-10T11:09:23.073109+0000","last_peered":"2024-05-10T11:09:23.073109+0000","last_clean":"2024-05-10T11:09:23.073109+0000","last_became_active":"2024-05-10T11:07:47.210317+0000","last_became_peered":"2024-05-10T11:07:47.210317+0000","last_unstale":"2024-05-10T11:09:23.073109+0000","last_undegraded":"2024-05-10T11:09:23.073109+0000","last_fullsized":"2024-05-10T11:09:23.073109+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:47:30.848731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065782+0000","last_change":"2024-05-10T11:07:47.213116+0000","last_active":"2024-05-10T11:09:23.065782+0000","last_peered":"2024-05-10T11:09:23.065782+0000","last_clean":"2024-05-10T11:09:23.065782+0000","last_became_active":"2024-05-10T11:07:47.212567+0000","last_became_peered":"2024-05-10T11:07:47.212567+0000","last_unstale":"2024-05-10T11:09:23.065782+0000","last_undegraded":"2024-05-10T11:09:23.065782+0000","last_fullsized":"2024-05-10T11:09:23.065782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:44:32.222012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068569+0000","last_change":"2024-05-10T11:07:47.200973+0000","last_active":"2024-05-10T11:09:23.068569+0000","last_peered":"2024-05-10T11:09:23.068569+0000","last_clean":"2024-05-10T11:09:23.068569+0000","last_became_active":"2024-05-10T11:07:47.200499+0000","last_became_peered":"2024-05-10T11:07:47.200499+0000","last_unstale":"2024-05-10T11:09:23.068569+0000","last_undegraded":"2024-05-10T11:09:23.068569+0000","last_fullsized":"2024-05-10T11:09:23.068569+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:29:24.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069987+0000","last_change":"2024-05-10T11:08:22.042254+0000","last_active":"2024-05-10T11:09:23.069987+0000","last_peered":"2024-05-10T11:09:23.069987+0000","last_clean":"2024-05-10T11:09:23.069987+0000","last_became_active":"2024-05-10T11:08:22.041906+0000","last_became_peered":"2024-05-10T11:08:22.041906+0000","last_unstale":"2024-05-10T11:09:23.069987+0000","last_undegraded":"2024-05-10T11:09:23.069987+0000","last_fullsized":"2024-05-10T11:09:23.069987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:55:27.280478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070194+0000","last_change":"2024-05-10T11:08:00.897746+0000","last_active":"2024-05-10T11:09:23.070194+0000","last_peered":"2024-05-10T11:09:23.070194+0000","last_clean":"2024-05-10T11:09:23.070194+0000","last_became_active":"2024-05-10T11:08:00.897428+0000","last_became_peered":"2024-05-10T11:08:00.897428+0000","last_unstale":"2024-05-10T11:09:23.070194+0000","last_undegraded":"2024-05-10T11:09:23.070194+0000","last_fullsized":"2024-05-10T11:09:23.070194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:38.846373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":91,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:47.107090+0000","last_change":"2024-05-10T11:07:42.139990+0000","last_active":"2024-05-10T11:09:47.107090+0000","last_peered":"2024-05-10T11:09:47.107090+0000","last_clean":"2024-05-10T11:09:47.107090+0000","last_became_active":"2024-05-10T11:07:42.139612+0000","last_became_peered":"2024-05-10T11:07:42.139612+0000","last_unstale":"2024-05-10T11:09:47.107090+0000","last_undegraded":"2024-05-10T11:09:47.107090+0000","last_fullsized":"2024-05-10T11:09:47.107090+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:48:29.590570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077251+0000","last_change":"2024-05-10T11:07:47.214263+0000","last_active":"2024-05-10T11:09:23.077251+0000","last_peered":"2024-05-10T11:09:23.077251+0000","last_clean":"2024-05-10T11:09:23.077251+0000","last_became_active":"2024-05-10T11:07:47.213946+0000","last_became_peered":"2024-05-10T11:07:47.213946+0000","last_unstale":"2024-05-10T11:09:23.077251+0000","last_undegraded":"2024-05-10T11:09:23.077251+0000","last_fullsized":"2024-05-10T11:09:23.077251+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:54:31.015838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062303+0000","last_change":"2024-05-10T11:08:00.890571+0000","last_active":"2024-05-10T11:09:23.062303+0000","last_peered":"2024-05-10T11:09:23.062303+0000","last_clean":"2024-05-10T11:09:23.062303+0000","last_became_active":"2024-05-10T11:08:00.889468+0000","last_became_peered":"2024-05-10T11:08:00.889468+0000","last_unstale":"2024-05-10T11:09:23.062303+0000","last_undegraded":"2024-05-10T11:09:23.062303+0000","last_fullsized":"2024-05-10T11:09:23.062303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:20:54.342656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077801+0000","last_change":"2024-05-10T11:07:42.383396+0000","last_active":"2024-05-10T11:09:23.077801+0000","last_peered":"2024-05-10T11:09:23.077801+0000","last_clean":"2024-05-10T11:09:23.077801+0000","last_became_active":"2024-05-10T11:07:42.382787+0000","last_became_peered":"2024-05-10T11:07:42.382787+0000","last_unstale":"2024-05-10T11:09:23.077801+0000","last_undegraded":"2024-05-10T11:09:23.077801+0000","last_fullsized":"2024-05-10T11:09:23.077801+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:44:34.240119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067761+0000","last_change":"2024-05-10T11:07:47.216770+0000","last_active":"2024-05-10T11:09:23.067761+0000","last_peered":"2024-05-10T11:09:23.067761+0000","last_clean":"2024-05-10T11:09:23.067761+0000","last_became_active":"2024-05-10T11:07:47.216377+0000","last_became_peered":"2024-05-10T11:07:47.216377+0000","last_unstale":"2024-05-10T11:09:23.067761+0000","last_undegraded":"2024-05-10T11:09:23.067761+0000","last_fullsized":"2024-05-10T11:09:23.067761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:59:22.993888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062458+0000","last_change":"2024-05-10T11:08:00.900614+0000","last_active":"2024-05-10T11:09:23.062458+0000","last_peered":"2024-05-10T11:09:23.062458+0000","last_clean":"2024-05-10T11:09:23.062458+0000","last_became_active":"2024-05-10T11:08:00.900162+0000","last_became_peered":"2024-05-10T11:08:00.900162+0000","last_unstale":"2024-05-10T11:09:23.062458+0000","last_undegraded":"2024-05-10T11:09:23.062458+0000","last_fullsized":"2024-05-10T11:09:23.062458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:42:56.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070222+0000","last_change":"2024-05-10T11:07:42.127749+0000","last_active":"2024-05-10T11:09:23.070222+0000","last_peered":"2024-05-10T11:09:23.070222+0000","last_clean":"2024-05-10T11:09:23.070222+0000","last_became_active":"2024-05-10T11:07:42.127296+0000","last_became_peered":"2024-05-10T11:07:42.127296+0000","last_unstale":"2024-05-10T11:09:23.070222+0000","last_undegraded":"2024-05-10T11:09:23.070222+0000","last_fullsized":"2024-05-10T11:09:23.070222+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:37:37.720542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.073147+0000","last_change":"2024-05-10T11:07:47.198971+0000","last_active":"2024-05-10T11:09:23.073147+0000","last_peered":"2024-05-10T11:09:23.073147+0000","last_clean":"2024-05-10T11:09:23.073147+0000","last_became_active":"2024-05-10T11:07:47.198510+0000","last_became_peered":"2024-05-10T11:07:47.198510+0000","last_unstale":"2024-05-10T11:09:23.073147+0000","last_undegraded":"2024-05-10T11:09:23.073147+0000","last_fullsized":"2024-05-10T11:09:23.073147+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:17:24.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062463+0000","last_change":"2024-05-10T11:08:00.878749+0000","last_active":"2024-05-10T11:09:23.062463+0000","last_peered":"2024-05-10T11:09:23.062463+0000","last_clean":"2024-05-10T11:09:23.062463+0000","last_became_active":"2024-05-10T11:08:00.878271+0000","last_became_peered":"2024-05-10T11:08:00.878271+0000","last_unstale":"2024-05-10T11:09:23.062463+0000","last_undegraded":"2024-05-10T11:09:23.062463+0000","last_fullsized":"2024-05-10T11:09:23.062463+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:39:28.558766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069568+0000","last_change":"2024-05-10T11:07:42.129593+0000","last_active":"2024-05-10T11:09:23.069568+0000","last_peered":"2024-05-10T11:09:23.069568+0000","last_clean":"2024-05-10T11:09:23.069568+0000","last_became_active":"2024-05-10T11:07:42.129250+0000","last_became_peered":"2024-05-10T11:07:42.129250+0000","last_unstale":"2024-05-10T11:09:23.069568+0000","last_undegraded":"2024-05-10T11:09:23.069568+0000","last_fullsized":"2024-05-10T11:09:23.069568+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:22:55.541079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070463+0000","last_change":"2024-05-10T11:08:22.244535+0000","last_active":"2024-05-10T11:09:23.070463+0000","last_peered":"2024-05-10T11:09:23.070463+0000","last_clean":"2024-05-10T11:09:23.070463+0000","last_became_active":"2024-05-10T11:08:22.243716+0000","last_became_peered":"2024-05-10T11:08:22.243716+0000","last_unstale":"2024-05-10T11:09:23.070463+0000","last_undegraded":"2024-05-10T11:09:23.070463+0000","last_fullsized":"2024-05-10T11:09:23.070463+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:11.718999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069774+0000","last_change":"2024-05-10T11:08:00.869806+0000","last_active":"2024-05-10T11:09:23.069774+0000","last_peered":"2024-05-10T11:09:23.069774+0000","last_clean":"2024-05-10T11:09:23.069774+0000","last_became_active":"2024-05-10T11:08:00.869355+0000","last_became_peered":"2024-05-10T11:08:00.869355+0000","last_unstale":"2024-05-10T11:09:23.069774+0000","last_undegraded":"2024-05-10T11:09:23.069774+0000","last_fullsized":"2024-05-10T11:09:23.069774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:36.043412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:44.121077+0000","last_change":"2024-05-10T11:09:23.076933+0000","last_active":"2024-05-10T11:09:44.121077+0000","last_peered":"2024-05-10T11:09:44.121077+0000","last_clean":"2024-05-10T11:09:44.121077+0000","last_became_active":"2024-05-10T11:09:23.073274+0000","last_became_peered":"2024-05-10T11:09:23.073274+0000","last_unstale":"2024-05-10T11:09:44.121077+0000","last_undegraded":"2024-05-10T11:09:44.121077+0000","last_fullsized":"2024-05-10T11:09:44.121077+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:26:13.923466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069120+0000","last_change":"2024-05-10T11:07:47.200873+0000","last_active":"2024-05-10T11:09:23.069120+0000","last_peered":"2024-05-10T11:09:23.069120+0000","last_clean":"2024-05-10T11:09:23.069120+0000","last_became_active":"2024-05-10T11:07:47.200436+0000","last_became_peered":"2024-05-10T11:07:47.200436+0000","last_unstale":"2024-05-10T11:09:23.069120+0000","last_undegraded":"2024-05-10T11:09:23.069120+0000","last_fullsized":"2024-05-10T11:09:23.069120+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:38:49.588593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069615+0000","last_change":"2024-05-10T11:08:00.885432+0000","last_active":"2024-05-10T11:09:23.069615+0000","last_peered":"2024-05-10T11:09:23.069615+0000","last_clean":"2024-05-10T11:09:23.069615+0000","last_became_active":"2024-05-10T11:08:00.884987+0000","last_became_peered":"2024-05-10T11:08:00.884987+0000","last_unstale":"2024-05-10T11:09:23.069615+0000","last_undegraded":"2024-05-10T11:09:23.069615+0000","last_fullsized":"2024-05-10T11:09:23.069615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:41:24.082722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.093122+0000","last_change":"2024-05-10T11:09:23.093122+0000","last_active":"2024-05-10T11:09:23.093122+0000","last_peered":"2024-05-10T11:09:23.093122+0000","last_clean":"2024-05-10T11:09:23.093122+0000","last_became_active":"2024-05-10T11:09:23.084485+0000","last_became_peered":"2024-05-10T11:09:23.084485+0000","last_unstale":"2024-05-10T11:09:23.093122+0000","last_undegraded":"2024-05-10T11:09:23.093122+0000","last_fullsized":"2024-05-10T11:09:23.093122+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:12:36.793572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068337+0000","last_change":"2024-05-10T11:08:22.036515+0000","last_active":"2024-05-10T11:09:23.068337+0000","last_peered":"2024-05-10T11:09:23.068337+0000","last_clean":"2024-05-10T11:09:23.068337+0000","last_became_active":"2024-05-10T11:08:22.036072+0000","last_became_peered":"2024-05-10T11:08:22.036072+0000","last_unstale":"2024-05-10T11:09:23.068337+0000","last_undegraded":"2024-05-10T11:09:23.068337+0000","last_fullsized":"2024-05-10T11:09:23.068337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:08.734453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6],"acting":[5,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065660+0000","last_change":"2024-05-10T11:08:00.888673+0000","last_active":"2024-05-10T11:09:23.065660+0000","last_peered":"2024-05-10T11:09:23.065660+0000","last_clean":"2024-05-10T11:09:23.065660+0000","last_became_active":"2024-05-10T11:08:00.888302+0000","last_became_peered":"2024-05-10T11:08:00.888302+0000","last_unstale":"2024-05-10T11:09:23.065660+0000","last_undegraded":"2024-05-10T11:09:23.065660+0000","last_fullsized":"2024-05-10T11:09:23.065660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:15:32.276358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.078139+0000","last_change":"2024-05-10T11:07:42.140993+0000","last_active":"2024-05-10T11:09:23.078139+0000","last_peered":"2024-05-10T11:09:23.078139+0000","last_clean":"2024-05-10T11:09:23.078139+0000","last_became_active":"2024-05-10T11:07:42.140501+0000","last_became_peered":"2024-05-10T11:07:42.140501+0000","last_unstale":"2024-05-10T11:09:23.078139+0000","last_undegraded":"2024-05-10T11:09:23.078139+0000","last_fullsized":"2024-05-10T11:09:23.078139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:01:50.324507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077182+0000","last_change":"2024-05-10T11:07:47.212616+0000","last_active":"2024-05-10T11:09:23.077182+0000","last_peered":"2024-05-10T11:09:23.077182+0000","last_clean":"2024-05-10T11:09:23.077182+0000","last_became_active":"2024-05-10T11:07:47.211081+0000","last_became_peered":"2024-05-10T11:07:47.211081+0000","last_unstale":"2024-05-10T11:09:23.077182+0000","last_undegraded":"2024-05-10T11:09:23.077182+0000","last_fullsized":"2024-05-10T11:09:23.077182+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:03:54.404355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067559+0000","last_change":"2024-05-10T11:08:00.888782+0000","last_active":"2024-05-10T11:09:23.067559+0000","last_peered":"2024-05-10T11:09:23.067559+0000","last_clean":"2024-05-10T11:09:23.067559+0000","last_became_active":"2024-05-10T11:08:00.888366+0000","last_became_peered":"2024-05-10T11:08:00.888366+0000","last_unstale":"2024-05-10T11:09:23.067559+0000","last_undegraded":"2024-05-10T11:09:23.067559+0000","last_fullsized":"2024-05-10T11:09:23.067559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:37.040186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066149+0000","last_change":"2024-05-10T11:07:42.128677+0000","last_active":"2024-05-10T11:09:23.066149+0000","last_peered":"2024-05-10T11:09:23.066149+0000","last_clean":"2024-05-10T11:09:23.066149+0000","last_became_active":"2024-05-10T11:07:42.128330+0000","last_became_peered":"2024-05-10T11:07:42.128330+0000","last_unstale":"2024-05-10T11:09:23.066149+0000","last_undegraded":"2024-05-10T11:09:23.066149+0000","last_fullsized":"2024-05-10T11:09:23.066149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:09:11.738485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069144+0000","last_change":"2024-05-10T11:07:47.215257+0000","last_active":"2024-05-10T11:09:23.069144+0000","last_peered":"2024-05-10T11:09:23.069144+0000","last_clean":"2024-05-10T11:09:23.069144+0000","last_became_active":"2024-05-10T11:07:47.214682+0000","last_became_peered":"2024-05-10T11:07:47.214682+0000","last_unstale":"2024-05-10T11:09:23.069144+0000","last_undegraded":"2024-05-10T11:09:23.069144+0000","last_fullsized":"2024-05-10T11:09:23.069144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:35:45.787737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076006+0000","last_change":"2024-05-10T11:08:00.894446+0000","last_active":"2024-05-10T11:09:23.076006+0000","last_peered":"2024-05-10T11:09:23.076006+0000","last_clean":"2024-05-10T11:09:23.076006+0000","last_became_active":"2024-05-10T11:08:00.886473+0000","last_became_peered":"2024-05-10T11:08:00.886473+0000","last_unstale":"2024-05-10T11:09:23.076006+0000","last_undegraded":"2024-05-10T11:09:23.076006+0000","last_fullsized":"2024-05-10T11:09:23.076006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:40.980328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":118,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:50.085349+0000","last_change":"2024-05-10T11:07:42.138750+0000","last_active":"2024-05-10T11:09:50.085349+0000","last_peered":"2024-05-10T11:09:50.085349+0000","last_clean":"2024-05-10T11:09:50.085349+0000","last_became_active":"2024-05-10T11:07:42.138342+0000","last_became_peered":"2024-05-10T11:07:42.138342+0000","last_unstale":"2024-05-10T11:09:50.085349+0000","last_undegraded":"2024-05-10T11:09:50.085349+0000","last_fullsized":"2024-05-10T11:09:50.085349+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:19:45.791550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.073745+0000","last_change":"2024-05-10T11:07:47.204604+0000","last_active":"2024-05-10T11:09:23.073745+0000","last_peered":"2024-05-10T11:09:23.073745+0000","last_clean":"2024-05-10T11:09:23.073745+0000","last_became_active":"2024-05-10T11:07:47.204261+0000","last_became_peered":"2024-05-10T11:07:47.204261+0000","last_unstale":"2024-05-10T11:09:23.073745+0000","last_undegraded":"2024-05-10T11:09:23.073745+0000","last_fullsized":"2024-05-10T11:09:23.073745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:55:46.409545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068646+0000","last_change":"2024-05-10T11:08:22.038090+0000","last_active":"2024-05-10T11:09:23.068646+0000","last_peered":"2024-05-10T11:09:23.068646+0000","last_clean":"2024-05-10T11:09:23.068646+0000","last_became_active":"2024-05-10T11:08:22.037490+0000","last_became_peered":"2024-05-10T11:08:22.037490+0000","last_unstale":"2024-05-10T11:09:23.068646+0000","last_undegraded":"2024-05-10T11:09:23.068646+0000","last_fullsized":"2024-05-10T11:09:23.068646+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:01:46.319401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071634+0000","last_change":"2024-05-10T11:07:42.128636+0000","last_active":"2024-05-10T11:09:23.071634+0000","last_peered":"2024-05-10T11:09:23.071634+0000","last_clean":"2024-05-10T11:09:23.071634+0000","last_became_active":"2024-05-10T11:07:42.128194+0000","last_became_peered":"2024-05-10T11:07:42.128194+0000","last_unstale":"2024-05-10T11:09:23.071634+0000","last_undegraded":"2024-05-10T11:09:23.071634+0000","last_fullsized":"2024-05-10T11:09:23.071634+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:51:54.279292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068054+0000","last_change":"2024-05-10T11:08:22.037807+0000","last_active":"2024-05-10T11:09:23.068054+0000","last_peered":"2024-05-10T11:09:23.068054+0000","last_clean":"2024-05-10T11:09:23.068054+0000","last_became_active":"2024-05-10T11:08:22.037337+0000","last_became_peered":"2024-05-10T11:08:22.037337+0000","last_unstale":"2024-05-10T11:09:23.068054+0000","last_undegraded":"2024-05-10T11:09:23.068054+0000","last_fullsized":"2024-05-10T11:09:23.068054+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:50:06.481222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062684+0000","last_change":"2024-05-10T11:08:00.893684+0000","last_active":"2024-05-10T11:09:23.062684+0000","last_peered":"2024-05-10T11:09:23.062684+0000","last_clean":"2024-05-10T11:09:23.062684+0000","last_became_active":"2024-05-10T11:08:00.893334+0000","last_became_peered":"2024-05-10T11:08:00.893334+0000","last_unstale":"2024-05-10T11:09:23.062684+0000","last_undegraded":"2024-05-10T11:09:23.062684+0000","last_fullsized":"2024-05-10T11:09:23.062684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:58:04.812308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066307+0000","last_change":"2024-05-10T11:07:42.382086+0000","last_active":"2024-05-10T11:09:23.066307+0000","last_peered":"2024-05-10T11:09:23.066307+0000","last_clean":"2024-05-10T11:09:23.066307+0000","last_became_active":"2024-05-10T11:07:42.381744+0000","last_became_peered":"2024-05-10T11:07:42.381744+0000","last_unstale":"2024-05-10T11:09:23.066307+0000","last_undegraded":"2024-05-10T11:09:23.066307+0000","last_fullsized":"2024-05-10T11:09:23.066307+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:15:10.893834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.073185+0000","last_change":"2024-05-10T11:07:47.194683+0000","last_active":"2024-05-10T11:09:23.073185+0000","last_peered":"2024-05-10T11:09:23.073185+0000","last_clean":"2024-05-10T11:09:23.073185+0000","last_became_active":"2024-05-10T11:07:47.194319+0000","last_became_peered":"2024-05-10T11:07:47.194319+0000","last_unstale":"2024-05-10T11:09:23.073185+0000","last_undegraded":"2024-05-10T11:09:23.073185+0000","last_fullsized":"2024-05-10T11:09:23.073185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:41:48.437093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069563+0000","last_change":"2024-05-10T11:08:00.889616+0000","last_active":"2024-05-10T11:09:23.069563+0000","last_peered":"2024-05-10T11:09:23.069563+0000","last_clean":"2024-05-10T11:09:23.069563+0000","last_became_active":"2024-05-10T11:08:00.882179+0000","last_became_peered":"2024-05-10T11:08:00.882179+0000","last_unstale":"2024-05-10T11:09:23.069563+0000","last_undegraded":"2024-05-10T11:09:23.069563+0000","last_fullsized":"2024-05-10T11:09:23.069563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:23:12.407861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'36","reported_seq":159,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:50.121414+0000","last_change":"2024-05-10T11:07:42.128796+0000","last_active":"2024-05-10T11:09:50.121414+0000","last_peered":"2024-05-10T11:09:50.121414+0000","last_clean":"2024-05-10T11:09:50.121414+0000","last_became_active":"2024-05-10T11:07:42.128402+0000","last_became_peered":"2024-05-10T11:07:42.128402+0000","last_unstale":"2024-05-10T11:09:50.121414+0000","last_undegraded":"2024-05-10T11:09:50.121414+0000","last_fullsized":"2024-05-10T11:09:50.121414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:50:09.205102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64645,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":79,"num_read_kb":103,"num_write":38,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070039+0000","last_change":"2024-05-10T11:07:47.188304+0000","last_active":"2024-05-10T11:09:23.070039+0000","last_peered":"2024-05-10T11:09:23.070039+0000","last_clean":"2024-05-10T11:09:23.070039+0000","last_became_active":"2024-05-10T11:07:47.187798+0000","last_became_peered":"2024-05-10T11:07:47.187798+0000","last_unstale":"2024-05-10T11:09:23.070039+0000","last_undegraded":"2024-05-10T11:09:23.070039+0000","last_fullsized":"2024-05-10T11:09:23.070039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:50:21.059060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066711+0000","last_change":"2024-05-10T11:08:00.886669+0000","last_active":"2024-05-10T11:09:23.066711+0000","last_peered":"2024-05-10T11:09:23.066711+0000","last_clean":"2024-05-10T11:09:23.066711+0000","last_became_active":"2024-05-10T11:08:00.886249+0000","last_became_peered":"2024-05-10T11:08:00.886249+0000","last_unstale":"2024-05-10T11:09:23.066711+0000","last_undegraded":"2024-05-10T11:09:23.066711+0000","last_fullsized":"2024-05-10T11:09:23.066711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:10:07.707462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.095713+0000","last_change":"2024-05-10T11:09:23.095352+0000","last_active":"2024-05-10T11:09:23.095713+0000","last_peered":"2024-05-10T11:09:23.095713+0000","last_clean":"2024-05-10T11:09:23.095713+0000","last_became_active":"2024-05-10T11:09:23.083116+0000","last_became_peered":"2024-05-10T11:09:23.083116+0000","last_unstale":"2024-05-10T11:09:23.095713+0000","last_undegraded":"2024-05-10T11:09:23.095713+0000","last_fullsized":"2024-05-10T11:09:23.095713+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:24:28.945724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.080153+0000","last_change":"2024-05-10T11:07:47.207657+0000","last_active":"2024-05-10T11:09:23.080153+0000","last_peered":"2024-05-10T11:09:23.080153+0000","last_clean":"2024-05-10T11:09:23.080153+0000","last_became_active":"2024-05-10T11:07:47.207317+0000","last_became_peered":"2024-05-10T11:07:47.207317+0000","last_unstale":"2024-05-10T11:09:23.080153+0000","last_undegraded":"2024-05-10T11:09:23.080153+0000","last_fullsized":"2024-05-10T11:09:23.080153+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:07:33.961642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065058+0000","last_change":"2024-05-10T11:08:00.884076+0000","last_active":"2024-05-10T11:09:23.065058+0000","last_peered":"2024-05-10T11:09:23.065058+0000","last_clean":"2024-05-10T11:09:23.065058+0000","last_became_active":"2024-05-10T11:08:00.883670+0000","last_became_peered":"2024-05-10T11:08:00.883670+0000","last_unstale":"2024-05-10T11:09:23.065058+0000","last_undegraded":"2024-05-10T11:09:23.065058+0000","last_fullsized":"2024-05-10T11:09:23.065058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:56.447836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067560+0000","last_change":"2024-05-10T11:07:42.135550+0000","last_active":"2024-05-10T11:09:23.067560+0000","last_peered":"2024-05-10T11:09:23.067560+0000","last_clean":"2024-05-10T11:09:23.067560+0000","last_became_active":"2024-05-10T11:07:42.135192+0000","last_became_peered":"2024-05-10T11:07:42.135192+0000","last_unstale":"2024-05-10T11:09:23.067560+0000","last_undegraded":"2024-05-10T11:09:23.067560+0000","last_fullsized":"2024-05-10T11:09:23.067560+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:17.100873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071503+0000","last_change":"2024-05-10T11:07:47.199529+0000","last_active":"2024-05-10T11:09:23.071503+0000","last_peered":"2024-05-10T11:09:23.071503+0000","last_clean":"2024-05-10T11:09:23.071503+0000","last_became_active":"2024-05-10T11:07:47.199070+0000","last_became_peered":"2024-05-10T11:07:47.199070+0000","last_unstale":"2024-05-10T11:09:23.071503+0000","last_undegraded":"2024-05-10T11:09:23.071503+0000","last_fullsized":"2024-05-10T11:09:23.071503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:24.446639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066958+0000","last_change":"2024-05-10T11:08:00.898607+0000","last_active":"2024-05-10T11:09:23.066958+0000","last_peered":"2024-05-10T11:09:23.066958+0000","last_clean":"2024-05-10T11:09:23.066958+0000","last_became_active":"2024-05-10T11:08:00.897768+0000","last_became_peered":"2024-05-10T11:08:00.897768+0000","last_unstale":"2024-05-10T11:09:23.066958+0000","last_undegraded":"2024-05-10T11:09:23.066958+0000","last_fullsized":"2024-05-10T11:09:23.066958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:24:42.125203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:46.082802+0000","last_change":"2024-05-10T11:07:42.126907+0000","last_active":"2024-05-10T11:09:46.082802+0000","last_peered":"2024-05-10T11:09:46.082802+0000","last_clean":"2024-05-10T11:09:46.082802+0000","last_became_active":"2024-05-10T11:07:42.126462+0000","last_became_peered":"2024-05-10T11:07:42.126462+0000","last_unstale":"2024-05-10T11:09:46.082802+0000","last_undegraded":"2024-05-10T11:09:46.082802+0000","last_fullsized":"2024-05-10T11:09:46.082802+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:18:53.971150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072062+0000","last_change":"2024-05-10T11:07:47.191888+0000","last_active":"2024-05-10T11:09:23.072062+0000","last_peered":"2024-05-10T11:09:23.072062+0000","last_clean":"2024-05-10T11:09:23.072062+0000","last_became_active":"2024-05-10T11:07:47.191557+0000","last_became_peered":"2024-05-10T11:07:47.191557+0000","last_unstale":"2024-05-10T11:09:23.072062+0000","last_undegraded":"2024-05-10T11:09:23.072062+0000","last_fullsized":"2024-05-10T11:09:23.072062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:01:36.529493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079253+0000","last_change":"2024-05-10T11:08:00.894385+0000","last_active":"2024-05-10T11:09:23.079253+0000","last_peered":"2024-05-10T11:09:23.079253+0000","last_clean":"2024-05-10T11:09:23.079253+0000","last_became_active":"2024-05-10T11:08:00.893706+0000","last_became_peered":"2024-05-10T11:08:00.893706+0000","last_unstale":"2024-05-10T11:09:23.079253+0000","last_undegraded":"2024-05-10T11:09:23.079253+0000","last_fullsized":"2024-05-10T11:09:23.079253+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:02.069878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068787+0000","last_change":"2024-05-10T11:07:42.385661+0000","last_active":"2024-05-10T11:09:23.068787+0000","last_peered":"2024-05-10T11:09:23.068787+0000","last_clean":"2024-05-10T11:09:23.068787+0000","last_became_active":"2024-05-10T11:07:42.385308+0000","last_became_peered":"2024-05-10T11:07:42.385308+0000","last_unstale":"2024-05-10T11:09:23.068787+0000","last_undegraded":"2024-05-10T11:09:23.068787+0000","last_fullsized":"2024-05-10T11:09:23.068787+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:57:25.997730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065481+0000","last_change":"2024-05-10T11:07:47.191574+0000","last_active":"2024-05-10T11:09:23.065481+0000","last_peered":"2024-05-10T11:09:23.065481+0000","last_clean":"2024-05-10T11:09:23.065481+0000","last_became_active":"2024-05-10T11:07:47.191210+0000","last_became_peered":"2024-05-10T11:07:47.191210+0000","last_unstale":"2024-05-10T11:09:23.065481+0000","last_undegraded":"2024-05-10T11:09:23.065481+0000","last_fullsized":"2024-05-10T11:09:23.065481+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:11:35.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065575+0000","last_change":"2024-05-10T11:08:00.890076+0000","last_active":"2024-05-10T11:09:23.065575+0000","last_peered":"2024-05-10T11:09:23.065575+0000","last_clean":"2024-05-10T11:09:23.065575+0000","last_became_active":"2024-05-10T11:08:00.889630+0000","last_became_peered":"2024-05-10T11:08:00.889630+0000","last_unstale":"2024-05-10T11:09:23.065575+0000","last_undegraded":"2024-05-10T11:09:23.065575+0000","last_fullsized":"2024-05-10T11:09:23.065575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:53:29.257873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.075061+0000","last_change":"2024-05-10T11:09:23.074550+0000","last_active":"2024-05-10T11:09:23.075061+0000","last_peered":"2024-05-10T11:09:23.075061+0000","last_clean":"2024-05-10T11:09:23.075061+0000","last_became_active":"2024-05-10T11:09:23.067169+0000","last_became_peered":"2024-05-10T11:09:23.067169+0000","last_unstale":"2024-05-10T11:09:23.075061+0000","last_undegraded":"2024-05-10T11:09:23.075061+0000","last_fullsized":"2024-05-10T11:09:23.075061+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:45:06.303003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065275+0000","last_change":"2024-05-10T11:07:47.192042+0000","last_active":"2024-05-10T11:09:23.065275+0000","last_peered":"2024-05-10T11:09:23.065275+0000","last_clean":"2024-05-10T11:09:23.065275+0000","last_became_active":"2024-05-10T11:07:47.191632+0000","last_became_peered":"2024-05-10T11:07:47.191632+0000","last_unstale":"2024-05-10T11:09:23.065275+0000","last_undegraded":"2024-05-10T11:09:23.065275+0000","last_fullsized":"2024-05-10T11:09:23.065275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:55:08.564841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068998+0000","last_change":"2024-05-10T11:08:22.244721+0000","last_active":"2024-05-10T11:09:23.068998+0000","last_peered":"2024-05-10T11:09:23.068998+0000","last_clean":"2024-05-10T11:09:23.068998+0000","last_became_active":"2024-05-10T11:08:22.244274+0000","last_became_peered":"2024-05-10T11:08:22.244274+0000","last_unstale":"2024-05-10T11:09:23.068998+0000","last_undegraded":"2024-05-10T11:09:23.068998+0000","last_fullsized":"2024-05-10T11:09:23.068998+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:26.218071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070759+0000","last_change":"2024-05-10T11:07:42.388358+0000","last_active":"2024-05-10T11:09:23.070759+0000","last_peered":"2024-05-10T11:09:23.070759+0000","last_clean":"2024-05-10T11:09:23.070759+0000","last_became_active":"2024-05-10T11:07:42.387806+0000","last_became_peered":"2024-05-10T11:07:42.387806+0000","last_unstale":"2024-05-10T11:09:23.070759+0000","last_undegraded":"2024-05-10T11:09:23.070759+0000","last_fullsized":"2024-05-10T11:09:23.070759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:27:38.387332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068854+0000","last_change":"2024-05-10T11:07:47.213602+0000","last_active":"2024-05-10T11:09:23.068854+0000","last_peered":"2024-05-10T11:09:23.068854+0000","last_clean":"2024-05-10T11:09:23.068854+0000","last_became_active":"2024-05-10T11:07:47.213184+0000","last_became_peered":"2024-05-10T11:07:47.213184+0000","last_unstale":"2024-05-10T11:09:23.068854+0000","last_undegraded":"2024-05-10T11:09:23.068854+0000","last_fullsized":"2024-05-10T11:09:23.068854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T23:01:52.253279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079537+0000","last_change":"2024-05-10T11:08:00.889850+0000","last_active":"2024-05-10T11:09:23.079537+0000","last_peered":"2024-05-10T11:09:23.079537+0000","last_clean":"2024-05-10T11:09:23.079537+0000","last_became_active":"2024-05-10T11:08:00.888526+0000","last_became_peered":"2024-05-10T11:08:00.888526+0000","last_unstale":"2024-05-10T11:09:23.079537+0000","last_undegraded":"2024-05-10T11:09:23.079537+0000","last_fullsized":"2024-05-10T11:09:23.079537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:11:43.899628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:49.112626+0000","last_change":"2024-05-10T11:07:42.130650+0000","last_active":"2024-05-10T11:09:49.112626+0000","last_peered":"2024-05-10T11:09:49.112626+0000","last_clean":"2024-05-10T11:09:49.112626+0000","last_became_active":"2024-05-10T11:07:42.130259+0000","last_became_peered":"2024-05-10T11:07:42.130259+0000","last_unstale":"2024-05-10T11:09:49.112626+0000","last_undegraded":"2024-05-10T11:09:49.112626+0000","last_fullsized":"2024-05-10T11:09:49.112626+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.870092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068274+0000","last_change":"2024-05-10T11:07:47.204159+0000","last_active":"2024-05-10T11:09:23.068274+0000","last_peered":"2024-05-10T11:09:23.068274+0000","last_clean":"2024-05-10T11:09:23.068274+0000","last_became_active":"2024-05-10T11:07:47.203723+0000","last_became_peered":"2024-05-10T11:07:47.203723+0000","last_unstale":"2024-05-10T11:09:23.068274+0000","last_undegraded":"2024-05-10T11:09:23.068274+0000","last_fullsized":"2024-05-10T11:09:23.068274+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:44:48.232321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.081910+0000","last_change":"2024-05-10T11:08:00.918501+0000","last_active":"2024-05-10T11:09:23.081910+0000","last_peered":"2024-05-10T11:09:23.081910+0000","last_clean":"2024-05-10T11:09:23.081910+0000","last_became_active":"2024-05-10T11:08:00.918071+0000","last_became_peered":"2024-05-10T11:08:00.918071+0000","last_unstale":"2024-05-10T11:09:23.081910+0000","last_undegraded":"2024-05-10T11:09:23.081910+0000","last_fullsized":"2024-05-10T11:09:23.081910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:23:08.390953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067132+0000","last_change":"2024-05-10T11:08:22.036385+0000","last_active":"2024-05-10T11:09:23.067132+0000","last_peered":"2024-05-10T11:09:23.067132+0000","last_clean":"2024-05-10T11:09:23.067132+0000","last_became_active":"2024-05-10T11:08:22.035969+0000","last_became_peered":"2024-05-10T11:08:22.035969+0000","last_unstale":"2024-05-10T11:09:23.067132+0000","last_undegraded":"2024-05-10T11:09:23.067132+0000","last_fullsized":"2024-05-10T11:09:23.067132+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:14:24.069422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:49.086815+0000","last_change":"2024-05-10T11:07:42.381926+0000","last_active":"2024-05-10T11:09:49.086815+0000","last_peered":"2024-05-10T11:09:49.086815+0000","last_clean":"2024-05-10T11:09:49.086815+0000","last_became_active":"2024-05-10T11:07:42.381301+0000","last_became_peered":"2024-05-10T11:07:42.381301+0000","last_unstale":"2024-05-10T11:09:49.086815+0000","last_undegraded":"2024-05-10T11:09:49.086815+0000","last_fullsized":"2024-05-10T11:09:49.086815+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:48:32.291029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071656+0000","last_change":"2024-05-10T11:08:00.889959+0000","last_active":"2024-05-10T11:09:23.071656+0000","last_peered":"2024-05-10T11:09:23.071656+0000","last_clean":"2024-05-10T11:09:23.071656+0000","last_became_active":"2024-05-10T11:08:00.889514+0000","last_became_peered":"2024-05-10T11:08:00.889514+0000","last_unstale":"2024-05-10T11:09:23.071656+0000","last_undegraded":"2024-05-10T11:09:23.071656+0000","last_fullsized":"2024-05-10T11:09:23.071656+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.557175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076563+0000","last_change":"2024-05-10T11:07:47.218025+0000","last_active":"2024-05-10T11:09:23.076563+0000","last_peered":"2024-05-10T11:09:23.076563+0000","last_clean":"2024-05-10T11:09:23.076563+0000","last_became_active":"2024-05-10T11:07:47.207089+0000","last_became_peered":"2024-05-10T11:07:47.207089+0000","last_unstale":"2024-05-10T11:09:23.076563+0000","last_undegraded":"2024-05-10T11:09:23.076563+0000","last_fullsized":"2024-05-10T11:09:23.076563+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:15:50.363129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062645+0000","last_change":"2024-05-10T11:07:42.138182+0000","last_active":"2024-05-10T11:09:23.062645+0000","last_peered":"2024-05-10T11:09:23.062645+0000","last_clean":"2024-05-10T11:09:23.062645+0000","last_became_active":"2024-05-10T11:07:42.137734+0000","last_became_peered":"2024-05-10T11:07:42.137734+0000","last_unstale":"2024-05-10T11:09:23.062645+0000","last_undegraded":"2024-05-10T11:09:23.062645+0000","last_fullsized":"2024-05-10T11:09:23.062645+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:02:30.778878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":119,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071366+0000","last_change":"2024-05-10T11:06:06.321257+0000","last_active":"2024-05-10T11:09:23.071366+0000","last_peered":"2024-05-10T11:09:23.071366+0000","last_clean":"2024-05-10T11:09:23.071366+0000","last_became_active":"2024-05-10T11:06:06.316928+0000","last_became_peered":"2024-05-10T11:06:06.316928+0000","last_unstale":"2024-05-10T11:09:23.071366+0000","last_undegraded":"2024-05-10T11:09:23.071366+0000","last_fullsized":"2024-05-10T11:09:23.071366+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_clean_scrub_stamp":"2024-05-10T11:02:07.346111+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:32:43.670246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065268+0000","last_change":"2024-05-10T11:08:00.918499+0000","last_active":"2024-05-10T11:09:23.065268+0000","last_peered":"2024-05-10T11:09:23.065268+0000","last_clean":"2024-05-10T11:09:23.065268+0000","last_became_active":"2024-05-10T11:08:00.917950+0000","last_became_peered":"2024-05-10T11:08:00.917950+0000","last_unstale":"2024-05-10T11:09:23.065268+0000","last_undegraded":"2024-05-10T11:09:23.065268+0000","last_fullsized":"2024-05-10T11:09:23.065268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:07.591312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077484+0000","last_change":"2024-05-10T11:07:47.194968+0000","last_active":"2024-05-10T11:09:23.077484+0000","last_peered":"2024-05-10T11:09:23.077484+0000","last_clean":"2024-05-10T11:09:23.077484+0000","last_became_active":"2024-05-10T11:07:47.194645+0000","last_became_peered":"2024-05-10T11:07:47.194645+0000","last_unstale":"2024-05-10T11:09:23.077484+0000","last_undegraded":"2024-05-10T11:09:23.077484+0000","last_fullsized":"2024-05-10T11:09:23.077484+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:34.962163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072001+0000","last_change":"2024-05-10T11:07:42.126378+0000","last_active":"2024-05-10T11:09:23.072001+0000","last_peered":"2024-05-10T11:09:23.072001+0000","last_clean":"2024-05-10T11:09:23.072001+0000","last_became_active":"2024-05-10T11:07:42.125984+0000","last_became_peered":"2024-05-10T11:07:42.125984+0000","last_unstale":"2024-05-10T11:09:23.072001+0000","last_undegraded":"2024-05-10T11:09:23.072001+0000","last_fullsized":"2024-05-10T11:09:23.072001+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:15:50.378737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065752+0000","last_change":"2024-05-10T11:08:00.872342+0000","last_active":"2024-05-10T11:09:23.065752+0000","last_peered":"2024-05-10T11:09:23.065752+0000","last_clean":"2024-05-10T11:09:23.065752+0000","last_became_active":"2024-05-10T11:08:00.871926+0000","last_became_peered":"2024-05-10T11:08:00.871926+0000","last_unstale":"2024-05-10T11:09:23.065752+0000","last_undegraded":"2024-05-10T11:09:23.065752+0000","last_fullsized":"2024-05-10T11:09:23.065752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:27:16.519752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070093+0000","last_change":"2024-05-10T11:07:42.384236+0000","last_active":"2024-05-10T11:09:23.070093+0000","last_peered":"2024-05-10T11:09:23.070093+0000","last_clean":"2024-05-10T11:09:23.070093+0000","last_became_active":"2024-05-10T11:07:42.383908+0000","last_became_peered":"2024-05-10T11:07:42.383908+0000","last_unstale":"2024-05-10T11:09:23.070093+0000","last_undegraded":"2024-05-10T11:09:23.070093+0000","last_fullsized":"2024-05-10T11:09:23.070093+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:09:42.302506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.080000+0000","last_change":"2024-05-10T11:07:47.201645+0000","last_active":"2024-05-10T11:09:23.080000+0000","last_peered":"2024-05-10T11:09:23.080000+0000","last_clean":"2024-05-10T11:09:23.080000+0000","last_became_active":"2024-05-10T11:07:47.201300+0000","last_became_peered":"2024-05-10T11:07:47.201300+0000","last_unstale":"2024-05-10T11:09:23.080000+0000","last_undegraded":"2024-05-10T11:09:23.080000+0000","last_fullsized":"2024-05-10T11:09:23.080000+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:37.268530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076907+0000","last_change":"2024-05-10T11:08:00.917988+0000","last_active":"2024-05-10T11:09:23.076907+0000","last_peered":"2024-05-10T11:09:23.076907+0000","last_clean":"2024-05-10T11:09:23.076907+0000","last_became_active":"2024-05-10T11:08:00.917457+0000","last_became_peered":"2024-05-10T11:08:00.917457+0000","last_unstale":"2024-05-10T11:09:23.076907+0000","last_undegraded":"2024-05-10T11:09:23.076907+0000","last_fullsized":"2024-05-10T11:09:23.076907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:16:01.816917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'16","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:42.317110+0000","last_change":"2024-05-10T11:07:42.141005+0000","last_active":"2024-05-10T11:09:42.317110+0000","last_peered":"2024-05-10T11:09:42.317110+0000","last_clean":"2024-05-10T11:09:42.317110+0000","last_became_active":"2024-05-10T11:07:42.140634+0000","last_became_peered":"2024-05-10T11:07:42.140634+0000","last_unstale":"2024-05-10T11:09:42.317110+0000","last_undegraded":"2024-05-10T11:09:42.317110+0000","last_fullsized":"2024-05-10T11:09:42.317110+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:00:32.029374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077678+0000","last_change":"2024-05-10T11:07:47.206864+0000","last_active":"2024-05-10T11:09:23.077678+0000","last_peered":"2024-05-10T11:09:23.077678+0000","last_clean":"2024-05-10T11:09:23.077678+0000","last_became_active":"2024-05-10T11:07:47.206387+0000","last_became_peered":"2024-05-10T11:07:47.206387+0000","last_unstale":"2024-05-10T11:09:23.077678+0000","last_undegraded":"2024-05-10T11:09:23.077678+0000","last_fullsized":"2024-05-10T11:09:23.077678+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:12:35.242270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069056+0000","last_change":"2024-05-10T11:08:00.867872+0000","last_active":"2024-05-10T11:09:23.069056+0000","last_peered":"2024-05-10T11:09:23.069056+0000","last_clean":"2024-05-10T11:09:23.069056+0000","last_became_active":"2024-05-10T11:08:00.867461+0000","last_became_peered":"2024-05-10T11:08:00.867461+0000","last_unstale":"2024-05-10T11:09:23.069056+0000","last_undegraded":"2024-05-10T11:09:23.069056+0000","last_fullsized":"2024-05-10T11:09:23.069056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:29.617335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.063079+0000","last_change":"2024-05-10T11:07:42.599520+0000","last_active":"2024-05-10T11:09:23.063079+0000","last_peered":"2024-05-10T11:09:23.063079+0000","last_clean":"2024-05-10T11:09:23.063079+0000","last_became_active":"2024-05-10T11:07:42.599209+0000","last_became_peered":"2024-05-10T11:07:42.599209+0000","last_unstale":"2024-05-10T11:09:23.063079+0000","last_undegraded":"2024-05-10T11:09:23.063079+0000","last_fullsized":"2024-05-10T11:09:23.063079+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:52.899718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068190+0000","last_change":"2024-05-10T11:07:47.218286+0000","last_active":"2024-05-10T11:09:23.068190+0000","last_peered":"2024-05-10T11:09:23.068190+0000","last_clean":"2024-05-10T11:09:23.068190+0000","last_became_active":"2024-05-10T11:07:47.217960+0000","last_became_peered":"2024-05-10T11:07:47.217960+0000","last_unstale":"2024-05-10T11:09:23.068190+0000","last_undegraded":"2024-05-10T11:09:23.068190+0000","last_fullsized":"2024-05-10T11:09:23.068190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:11:37.208359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062722+0000","last_change":"2024-05-10T11:08:00.919578+0000","last_active":"2024-05-10T11:09:23.062722+0000","last_peered":"2024-05-10T11:09:23.062722+0000","last_clean":"2024-05-10T11:09:23.062722+0000","last_became_active":"2024-05-10T11:08:00.918546+0000","last_became_peered":"2024-05-10T11:08:00.918546+0000","last_unstale":"2024-05-10T11:09:23.062722+0000","last_undegraded":"2024-05-10T11:09:23.062722+0000","last_fullsized":"2024-05-10T11:09:23.062722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:39:05.062210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072455+0000","last_change":"2024-05-10T11:07:42.384577+0000","last_active":"2024-05-10T11:09:23.072455+0000","last_peered":"2024-05-10T11:09:23.072455+0000","last_clean":"2024-05-10T11:09:23.072455+0000","last_became_active":"2024-05-10T11:07:42.384301+0000","last_became_peered":"2024-05-10T11:07:42.384301+0000","last_unstale":"2024-05-10T11:09:23.072455+0000","last_undegraded":"2024-05-10T11:09:23.072455+0000","last_fullsized":"2024-05-10T11:09:23.072455+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:11:03.359490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069029+0000","last_change":"2024-05-10T11:07:47.189708+0000","last_active":"2024-05-10T11:09:23.069029+0000","last_peered":"2024-05-10T11:09:23.069029+0000","last_clean":"2024-05-10T11:09:23.069029+0000","last_became_active":"2024-05-10T11:07:47.189367+0000","last_became_peered":"2024-05-10T11:07:47.189367+0000","last_unstale":"2024-05-10T11:09:23.069029+0000","last_undegraded":"2024-05-10T11:09:23.069029+0000","last_fullsized":"2024-05-10T11:09:23.069029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:25.410477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068571+0000","last_change":"2024-05-10T11:08:00.899359+0000","last_active":"2024-05-10T11:09:23.068571+0000","last_peered":"2024-05-10T11:09:23.068571+0000","last_clean":"2024-05-10T11:09:23.068571+0000","last_became_active":"2024-05-10T11:08:00.898924+0000","last_became_peered":"2024-05-10T11:08:00.898924+0000","last_unstale":"2024-05-10T11:09:23.068571+0000","last_undegraded":"2024-05-10T11:09:23.068571+0000","last_fullsized":"2024-05-10T11:09:23.068571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:14:17.427245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076772+0000","last_change":"2024-05-10T11:07:42.140262+0000","last_active":"2024-05-10T11:09:23.076772+0000","last_peered":"2024-05-10T11:09:23.076772+0000","last_clean":"2024-05-10T11:09:23.076772+0000","last_became_active":"2024-05-10T11:07:42.138853+0000","last_became_peered":"2024-05-10T11:07:42.138853+0000","last_unstale":"2024-05-10T11:09:23.076772+0000","last_undegraded":"2024-05-10T11:09:23.076772+0000","last_fullsized":"2024-05-10T11:09:23.076772+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:05:39.995237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.064865+0000","last_change":"2024-05-10T11:07:47.203637+0000","last_active":"2024-05-10T11:09:23.064865+0000","last_peered":"2024-05-10T11:09:23.064865+0000","last_clean":"2024-05-10T11:09:23.064865+0000","last_became_active":"2024-05-10T11:07:47.203286+0000","last_became_peered":"2024-05-10T11:07:47.203286+0000","last_unstale":"2024-05-10T11:09:23.064865+0000","last_undegraded":"2024-05-10T11:09:23.064865+0000","last_fullsized":"2024-05-10T11:09:23.064865+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:27:46.924625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062709+0000","last_change":"2024-05-10T11:08:00.890082+0000","last_active":"2024-05-10T11:09:23.062709+0000","last_peered":"2024-05-10T11:09:23.062709+0000","last_clean":"2024-05-10T11:09:23.062709+0000","last_became_active":"2024-05-10T11:08:00.889673+0000","last_became_peered":"2024-05-10T11:08:00.889673+0000","last_unstale":"2024-05-10T11:09:23.062709+0000","last_undegraded":"2024-05-10T11:09:23.062709+0000","last_fullsized":"2024-05-10T11:09:23.062709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:21:31.167934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069358+0000","last_change":"2024-05-10T11:07:42.130703+0000","last_active":"2024-05-10T11:09:23.069358+0000","last_peered":"2024-05-10T11:09:23.069358+0000","last_clean":"2024-05-10T11:09:23.069358+0000","last_became_active":"2024-05-10T11:07:42.130098+0000","last_became_peered":"2024-05-10T11:07:42.130098+0000","last_unstale":"2024-05-10T11:09:23.069358+0000","last_undegraded":"2024-05-10T11:09:23.069358+0000","last_fullsized":"2024-05-10T11:09:23.069358+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:59:39.783151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077158+0000","last_change":"2024-05-10T11:07:47.220497+0000","last_active":"2024-05-10T11:09:23.077158+0000","last_peered":"2024-05-10T11:09:23.077158+0000","last_clean":"2024-05-10T11:09:23.077158+0000","last_became_active":"2024-05-10T11:07:47.219916+0000","last_became_peered":"2024-05-10T11:07:47.219916+0000","last_unstale":"2024-05-10T11:09:23.077158+0000","last_undegraded":"2024-05-10T11:09:23.077158+0000","last_fullsized":"2024-05-10T11:09:23.077158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:26:43.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062646+0000","last_change":"2024-05-10T11:08:00.918258+0000","last_active":"2024-05-10T11:09:23.062646+0000","last_peered":"2024-05-10T11:09:23.062646+0000","last_clean":"2024-05-10T11:09:23.062646+0000","last_became_active":"2024-05-10T11:08:00.917808+0000","last_became_peered":"2024-05-10T11:08:00.917808+0000","last_unstale":"2024-05-10T11:09:23.062646+0000","last_undegraded":"2024-05-10T11:09:23.062646+0000","last_fullsized":"2024-05-10T11:09:23.062646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:14:08.867024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068812+0000","last_change":"2024-05-10T11:07:42.127661+0000","last_active":"2024-05-10T11:09:23.068812+0000","last_peered":"2024-05-10T11:09:23.068812+0000","last_clean":"2024-05-10T11:09:23.068812+0000","last_became_active":"2024-05-10T11:07:42.127256+0000","last_became_peered":"2024-05-10T11:07:42.127256+0000","last_unstale":"2024-05-10T11:09:23.068812+0000","last_undegraded":"2024-05-10T11:09:23.068812+0000","last_fullsized":"2024-05-10T11:09:23.068812+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:14:37.821350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071813+0000","last_change":"2024-05-10T11:07:47.199589+0000","last_active":"2024-05-10T11:09:23.071813+0000","last_peered":"2024-05-10T11:09:23.071813+0000","last_clean":"2024-05-10T11:09:23.071813+0000","last_became_active":"2024-05-10T11:07:47.199145+0000","last_became_peered":"2024-05-10T11:07:47.199145+0000","last_unstale":"2024-05-10T11:09:23.071813+0000","last_undegraded":"2024-05-10T11:09:23.071813+0000","last_fullsized":"2024-05-10T11:09:23.071813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:46.577543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.085049+0000","last_change":"2024-05-10T11:09:23.085049+0000","last_active":"2024-05-10T11:09:23.085049+0000","last_peered":"2024-05-10T11:09:23.085049+0000","last_clean":"2024-05-10T11:09:23.085049+0000","last_became_active":"2024-05-10T11:09:23.084420+0000","last_became_peered":"2024-05-10T11:09:23.084420+0000","last_unstale":"2024-05-10T11:09:23.085049+0000","last_undegraded":"2024-05-10T11:09:23.085049+0000","last_fullsized":"2024-05-10T11:09:23.085049+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:32:33.346434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'5","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:45.077464+0000","last_change":"2024-05-10T11:07:42.599395+0000","last_active":"2024-05-10T11:09:45.077464+0000","last_peered":"2024-05-10T11:09:45.077464+0000","last_clean":"2024-05-10T11:09:45.077464+0000","last_became_active":"2024-05-10T11:07:42.598934+0000","last_became_peered":"2024-05-10T11:07:42.598934+0000","last_unstale":"2024-05-10T11:09:45.077464+0000","last_undegraded":"2024-05-10T11:09:45.077464+0000","last_fullsized":"2024-05-10T11:09:45.077464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:49:42.414789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":33,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071892+0000","last_change":"2024-05-10T11:07:47.192087+0000","last_active":"2024-05-10T11:09:23.071892+0000","last_peered":"2024-05-10T11:09:23.071892+0000","last_clean":"2024-05-10T11:09:23.071892+0000","last_became_active":"2024-05-10T11:07:47.191041+0000","last_became_peered":"2024-05-10T11:07:47.191041+0000","last_unstale":"2024-05-10T11:09:23.071892+0000","last_undegraded":"2024-05-10T11:09:23.071892+0000","last_fullsized":"2024-05-10T11:09:23.071892+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:50:14.052863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.080432+0000","last_change":"2024-05-10T11:09:23.080432+0000","last_active":"2024-05-10T11:09:23.080432+0000","last_peered":"2024-05-10T11:09:23.080432+0000","last_clean":"2024-05-10T11:09:23.080432+0000","last_became_active":"2024-05-10T11:09:23.074934+0000","last_became_peered":"2024-05-10T11:09:23.074934+0000","last_unstale":"2024-05-10T11:09:23.080432+0000","last_undegraded":"2024-05-10T11:09:23.080432+0000","last_fullsized":"2024-05-10T11:09:23.080432+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:12:14.613323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066440+0000","last_change":"2024-05-10T11:07:42.130718+0000","last_active":"2024-05-10T11:09:23.066440+0000","last_peered":"2024-05-10T11:09:23.066440+0000","last_clean":"2024-05-10T11:09:23.066440+0000","last_became_active":"2024-05-10T11:07:42.130355+0000","last_became_peered":"2024-05-10T11:07:42.130355+0000","last_unstale":"2024-05-10T11:09:23.066440+0000","last_undegraded":"2024-05-10T11:09:23.066440+0000","last_fullsized":"2024-05-10T11:09:23.066440+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:16:48.093146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071447+0000","last_change":"2024-05-10T11:07:47.213569+0000","last_active":"2024-05-10T11:09:23.071447+0000","last_peered":"2024-05-10T11:09:23.071447+0000","last_clean":"2024-05-10T11:09:23.071447+0000","last_became_active":"2024-05-10T11:07:47.213148+0000","last_became_peered":"2024-05-10T11:07:47.213148+0000","last_unstale":"2024-05-10T11:09:23.071447+0000","last_undegraded":"2024-05-10T11:09:23.071447+0000","last_fullsized":"2024-05-10T11:09:23.071447+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:42:41.161274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065961+0000","last_change":"2024-05-10T11:08:00.870937+0000","last_active":"2024-05-10T11:09:23.065961+0000","last_peered":"2024-05-10T11:09:23.065961+0000","last_clean":"2024-05-10T11:09:23.065961+0000","last_became_active":"2024-05-10T11:08:00.869276+0000","last_became_peered":"2024-05-10T11:08:00.869276+0000","last_unstale":"2024-05-10T11:09:23.065961+0000","last_undegraded":"2024-05-10T11:09:23.065961+0000","last_fullsized":"2024-05-10T11:09:23.065961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:37.058875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079599+0000","last_change":"2024-05-10T11:07:42.130544+0000","last_active":"2024-05-10T11:09:23.079599+0000","last_peered":"2024-05-10T11:09:23.079599+0000","last_clean":"2024-05-10T11:09:23.079599+0000","last_became_active":"2024-05-10T11:07:42.128912+0000","last_became_peered":"2024-05-10T11:07:42.128912+0000","last_unstale":"2024-05-10T11:09:23.079599+0000","last_undegraded":"2024-05-10T11:09:23.079599+0000","last_fullsized":"2024-05-10T11:09:23.079599+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:22:23.521911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072030+0000","last_change":"2024-05-10T11:07:47.194346+0000","last_active":"2024-05-10T11:09:23.072030+0000","last_peered":"2024-05-10T11:09:23.072030+0000","last_clean":"2024-05-10T11:09:23.072030+0000","last_became_active":"2024-05-10T11:07:47.194019+0000","last_became_peered":"2024-05-10T11:07:47.194019+0000","last_unstale":"2024-05-10T11:09:23.072030+0000","last_undegraded":"2024-05-10T11:09:23.072030+0000","last_fullsized":"2024-05-10T11:09:23.072030+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:33:05.262347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.064895+0000","last_change":"2024-05-10T11:07:47.192001+0000","last_active":"2024-05-10T11:09:23.064895+0000","last_peered":"2024-05-10T11:09:23.064895+0000","last_clean":"2024-05-10T11:09:23.064895+0000","last_became_active":"2024-05-10T11:07:47.191668+0000","last_became_peered":"2024-05-10T11:07:47.191668+0000","last_unstale":"2024-05-10T11:09:23.064895+0000","last_undegraded":"2024-05-10T11:09:23.064895+0000","last_fullsized":"2024-05-10T11:09:23.064895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:13:33.680899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070119+0000","last_change":"2024-05-10T11:07:47.192226+0000","last_active":"2024-05-10T11:09:23.070119+0000","last_peered":"2024-05-10T11:09:23.070119+0000","last_clean":"2024-05-10T11:09:23.070119+0000","last_became_active":"2024-05-10T11:07:47.191444+0000","last_became_peered":"2024-05-10T11:07:47.191444+0000","last_unstale":"2024-05-10T11:09:23.070119+0000","last_undegraded":"2024-05-10T11:09:23.070119+0000","last_fullsized":"2024-05-10T11:09:23.070119+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:08.348986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069258+0000","last_change":"2024-05-10T11:07:47.210801+0000","last_active":"2024-05-10T11:09:23.069258+0000","last_peered":"2024-05-10T11:09:23.069258+0000","last_clean":"2024-05-10T11:09:23.069258+0000","last_became_active":"2024-05-10T11:07:47.210496+0000","last_became_peered":"2024-05-10T11:07:47.210496+0000","last_unstale":"2024-05-10T11:09:23.069258+0000","last_undegraded":"2024-05-10T11:09:23.069258+0000","last_fullsized":"2024-05-10T11:09:23.069258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:11:03.661457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068329+0000","last_change":"2024-05-10T11:07:47.211033+0000","last_active":"2024-05-10T11:09:23.068329+0000","last_peered":"2024-05-10T11:09:23.068329+0000","last_clean":"2024-05-10T11:09:23.068329+0000","last_became_active":"2024-05-10T11:07:47.206061+0000","last_became_peered":"2024-05-10T11:07:47.206061+0000","last_unstale":"2024-05-10T11:09:23.068329+0000","last_undegraded":"2024-05-10T11:09:23.068329+0000","last_fullsized":"2024-05-10T11:09:23.068329+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:04:05.878270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079957+0000","last_change":"2024-05-10T11:07:47.214966+0000","last_active":"2024-05-10T11:09:23.079957+0000","last_peered":"2024-05-10T11:09:23.079957+0000","last_clean":"2024-05-10T11:09:23.079957+0000","last_became_active":"2024-05-10T11:07:47.214563+0000","last_became_peered":"2024-05-10T11:07:47.214563+0000","last_unstale":"2024-05-10T11:09:23.079957+0000","last_undegraded":"2024-05-10T11:09:23.079957+0000","last_fullsized":"2024-05-10T11:09:23.079957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:29:01.428893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065626+0000","last_change":"2024-05-10T11:07:47.196740+0000","last_active":"2024-05-10T11:09:23.065626+0000","last_peered":"2024-05-10T11:09:23.065626+0000","last_clean":"2024-05-10T11:09:23.065626+0000","last_became_active":"2024-05-10T11:07:47.196388+0000","last_became_peered":"2024-05-10T11:07:47.196388+0000","last_unstale":"2024-05-10T11:09:23.065626+0000","last_undegraded":"2024-05-10T11:09:23.065626+0000","last_fullsized":"2024-05-10T11:09:23.065626+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:23:09.055338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069342+0000","last_change":"2024-05-10T11:07:47.193568+0000","last_active":"2024-05-10T11:09:23.069342+0000","last_peered":"2024-05-10T11:09:23.069342+0000","last_clean":"2024-05-10T11:09:23.069342+0000","last_became_active":"2024-05-10T11:07:47.193202+0000","last_became_peered":"2024-05-10T11:07:47.193202+0000","last_unstale":"2024-05-10T11:09:23.069342+0000","last_undegraded":"2024-05-10T11:09:23.069342+0000","last_fullsized":"2024-05-10T11:09:23.069342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:36:24.556266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.078184+0000","last_change":"2024-05-10T11:07:47.215344+0000","last_active":"2024-05-10T11:09:23.078184+0000","last_peered":"2024-05-10T11:09:23.078184+0000","last_clean":"2024-05-10T11:09:23.078184+0000","last_became_active":"2024-05-10T11:07:47.214896+0000","last_became_peered":"2024-05-10T11:07:47.214896+0000","last_unstale":"2024-05-10T11:09:23.078184+0000","last_undegraded":"2024-05-10T11:09:23.078184+0000","last_fullsized":"2024-05-10T11:09:23.078184+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:23:07.845212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066258+0000","last_change":"2024-05-10T11:07:47.219047+0000","last_active":"2024-05-10T11:09:23.066258+0000","last_peered":"2024-05-10T11:09:23.066258+0000","last_clean":"2024-05-10T11:09:23.066258+0000","last_became_active":"2024-05-10T11:07:47.218778+0000","last_became_peered":"2024-05-10T11:07:47.218778+0000","last_unstale":"2024-05-10T11:09:23.066258+0000","last_undegraded":"2024-05-10T11:09:23.066258+0000","last_fullsized":"2024-05-10T11:09:23.066258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:02:48.945259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076781+0000","last_change":"2024-05-10T11:07:47.213412+0000","last_active":"2024-05-10T11:09:23.076781+0000","last_peered":"2024-05-10T11:09:23.076781+0000","last_clean":"2024-05-10T11:09:23.076781+0000","last_became_active":"2024-05-10T11:07:47.212963+0000","last_became_peered":"2024-05-10T11:07:47.212963+0000","last_unstale":"2024-05-10T11:09:23.076781+0000","last_undegraded":"2024-05-10T11:09:23.076781+0000","last_fullsized":"2024-05-10T11:09:23.076781+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:33:05.224184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072355+0000","last_change":"2024-05-10T11:07:47.207998+0000","last_active":"2024-05-10T11:09:23.072355+0000","last_peered":"2024-05-10T11:09:23.072355+0000","last_clean":"2024-05-10T11:09:23.072355+0000","last_became_active":"2024-05-10T11:07:47.207298+0000","last_became_peered":"2024-05-10T11:07:47.207298+0000","last_unstale":"2024-05-10T11:09:23.072355+0000","last_undegraded":"2024-05-10T11:09:23.072355+0000","last_fullsized":"2024-05-10T11:09:23.072355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:59:14.001103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.080578+0000","last_change":"2024-05-10T11:07:47.202216+0000","last_active":"2024-05-10T11:09:23.080578+0000","last_peered":"2024-05-10T11:09:23.080578+0000","last_clean":"2024-05-10T11:09:23.080578+0000","last_became_active":"2024-05-10T11:07:47.201867+0000","last_became_peered":"2024-05-10T11:07:47.201867+0000","last_unstale":"2024-05-10T11:09:23.080578+0000","last_undegraded":"2024-05-10T11:09:23.080578+0000","last_fullsized":"2024-05-10T11:09:23.080578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:48:59.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065156+0000","last_change":"2024-05-10T11:07:47.194149+0000","last_active":"2024-05-10T11:09:23.065156+0000","last_peered":"2024-05-10T11:09:23.065156+0000","last_clean":"2024-05-10T11:09:23.065156+0000","last_became_active":"2024-05-10T11:07:47.193838+0000","last_became_peered":"2024-05-10T11:07:47.193838+0000","last_unstale":"2024-05-10T11:09:23.065156+0000","last_undegraded":"2024-05-10T11:09:23.065156+0000","last_fullsized":"2024-05-10T11:09:23.065156+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:46:44.181446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247309,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71458,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":368,"num_read_kb":269,"num_write":250,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":712,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":210109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678058,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":522636,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874288,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1150976,"data_stored":1174398,"data_compressed":68282,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070516,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":526572,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266748,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":522832,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462977,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":1119351,"data_compressed":68282,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626503,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":517980,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":583655,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":519436,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":578920,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182488,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":517688,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986258,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":523067,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822733,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":1114542,"data_compressed":68282,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34167,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34167,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":61662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-10T11:09:52.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-10T11:09:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:52 smithi027 ceph-mon[30054]: from='client.24727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:09:53.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:52 smithi076 ceph-mon[34152]: from='client.24727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:09:53.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:52 smithi189 ceph-mon[34842]: from='client.24727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:09:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:54.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:53 smithi076 ceph-mon[34152]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-05-10T11:09:54.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:53 smithi189 ceph-mon[34842]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-05-10T11:09:54.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:53 smithi027 ceph-mon[30054]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-05-10T11:09:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.622+0000 7f2baca00700 1 -- 172.21.15.27:0/3924025581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba8105f90 msgr2=0x7f2ba81124a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.622+0000 7f2baca00700 1 --2- 172.21.15.27:0/3924025581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba8105f90 0x7f2ba81124a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000b0a0 tx=0x7f2ba000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:09:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.623+0000 7f2baca00700 1 -- 172.21.15.27:0/3924025581 shutdown_connections 2024-05-10T11:09:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.623+0000 7f2baca00700 1 --2- 172.21.15.27:0/3924025581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba8105f90 0x7f2ba81124a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.623+0000 7f2baca00700 1 --2- 172.21.15.27:0/3924025581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba810b660 0x7f2ba810ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.623+0000 7f2baca00700 1 --2- 172.21.15.27:0/3924025581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba81055d0 0x7f2ba8105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.623+0000 7f2baca00700 1 -- 172.21.15.27:0/3924025581 >> 172.21.15.27:0/3924025581 conn(0x7f2ba81010c0 msgr2=0x7f2ba81034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.623+0000 7f2baca00700 1 -- 172.21.15.27:0/3924025581 shutdown_connections 2024-05-10T11:09:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.623+0000 7f2baca00700 1 -- 172.21.15.27:0/3924025581 wait complete. 2024-05-10T11:09:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.624+0000 7f2baca00700 1 Processor -- start 2024-05-10T11:09:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.624+0000 7f2baca00700 1 -- start start 2024-05-10T11:09:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.625+0000 7f2baca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81055d0 0x7f2ba81a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.625+0000 7f2baca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba8105f90 0x7f2ba81a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.625+0000 7f2baca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba810b660 0x7f2ba819b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.625+0000 7f2baca00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba81157b0 con 0x7f2ba81055d0 2024-05-10T11:09:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.625+0000 7f2baca00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba8115630 con 0x7f2ba8105f90 2024-05-10T11:09:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.625+0000 7f2baca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba8115930 con 0x7f2ba810b660 2024-05-10T11:09:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.625+0000 7f2ba7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81055d0 0x7f2ba81a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.625+0000 7f2ba7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81055d0 0x7f2ba81a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39954/0 (socket says 172.21.15.27:39954) 2024-05-10T11:09:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.625+0000 7f2ba7400700 1 -- 172.21.15.27:0/132960087 learned_addr learned my addr 172.21.15.27:0/132960087 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.625+0000 7f2b9ea00700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba8105f90 0x7f2ba81a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:55.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.626+0000 7f2ba7e00700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba810b660 0x7f2ba819b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:55.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.626+0000 7f2ba7400700 1 -- 172.21.15.27:0/132960087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba810b660 msgr2=0x7f2ba819b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:55.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.626+0000 7f2ba7400700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba810b660 0x7f2ba819b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.626+0000 7f2ba7400700 1 -- 172.21.15.27:0/132960087 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba8105f90 msgr2=0x7f2ba81a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.626+0000 7f2ba7400700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba8105f90 0x7f2ba81a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.626+0000 7f2ba7400700 1 -- 172.21.15.27:0/132960087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba819c0d0 con 0x7f2ba81055d0 2024-05-10T11:09:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.626+0000 7f2ba7400700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81055d0 0x7f2ba81a17b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000ea50 tx=0x7f2b9000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.626+0000 7f2ba4c00700 1 -- 172.21.15.27:0/132960087 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9000cbd0 con 0x7f2ba81055d0 2024-05-10T11:09:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.627+0000 7f2baca00700 1 -- 172.21.15.27:0/132960087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba819c3c0 con 0x7f2ba81055d0 2024-05-10T11:09:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.627+0000 7f2ba4c00700 1 -- 172.21.15.27:0/132960087 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b9000cd70 con 0x7f2ba81055d0 2024-05-10T11:09:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.627+0000 7f2ba4c00700 1 -- 172.21.15.27:0/132960087 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b90018680 con 0x7f2ba81055d0 2024-05-10T11:09:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.627+0000 7f2baca00700 1 -- 172.21.15.27:0/132960087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba81a8680 con 0x7f2ba81055d0 2024-05-10T11:09:55.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.628+0000 7f2baca00700 1 -- 172.21.15.27:0/132960087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ba806a190 con 0x7f2ba81055d0 2024-05-10T11:09:55.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.630+0000 7f2ba4c00700 1 -- 172.21.15.27:0/132960087 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f2b9001f020 con 0x7f2ba81055d0 2024-05-10T11:09:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.633+0000 7f2ba4c00700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2b8c0b2910 0x7f2b8c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.633+0000 7f2ba4c00700 1 -- 172.21.15.27:0/132960087 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f2b900e2780 con 0x7f2ba81055d0 2024-05-10T11:09:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.633+0000 7f2b9ea00700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2b8c0b2910 0x7f2b8c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.634+0000 7f2b9ea00700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2b8c0b2910 0x7f2b8c0b4dd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2ba810ac60 tx=0x7f2b98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.638+0000 7f2ba4c00700 1 -- 172.21.15.27:0/132960087 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f2b900a9ef0 con 0x7f2ba81055d0 2024-05-10T11:09:55.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.879+0000 7f2baca00700 1 -- 172.21.15.27:0/132960087 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2ba80638e0 con 0x7f2b8c0b2910 2024-05-10T11:09:55.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.891+0000 7f2ba4c00700 1 -- 172.21.15.27:0/132960087 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340659 (secure 0 0 0) 0x7f2ba80638e0 con 0x7f2b8c0b2910 2024-05-10T11:09:55.893 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:09:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.900+0000 7f2baca00700 1 -- 172.21.15.27:0/132960087 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2b8c0b2910 msgr2=0x7f2b8c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.900+0000 7f2baca00700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2b8c0b2910 0x7f2b8c0b4dd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2ba810ac60 tx=0x7f2b98009260 comp rx=0 tx=0).stop 2024-05-10T11:09:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.900+0000 7f2baca00700 1 -- 172.21.15.27:0/132960087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81055d0 msgr2=0x7f2ba81a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.900+0000 7f2baca00700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81055d0 0x7f2ba81a17b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000ea50 tx=0x7f2b9000ed60 comp rx=0 tx=0).stop 2024-05-10T11:09:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.900+0000 7f2baca00700 1 -- 172.21.15.27:0/132960087 shutdown_connections 2024-05-10T11:09:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.900+0000 7f2baca00700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f2b8c0b2910 0x7f2b8c0b4dd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.900+0000 7f2baca00700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba810b660 0x7f2ba819b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.900+0000 7f2baca00700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81055d0 0x7f2ba81a17b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.900+0000 7f2baca00700 1 --2- 172.21.15.27:0/132960087 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba8105f90 0x7f2ba81a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.900+0000 7f2baca00700 1 -- 172.21.15.27:0/132960087 >> 172.21.15.27:0/132960087 conn(0x7f2ba81010c0 msgr2=0x7f2ba8102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.900+0000 7f2baca00700 1 -- 172.21.15.27:0/132960087 shutdown_connections 2024-05-10T11:09:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:55.900+0000 7f2baca00700 1 -- 172.21.15.27:0/132960087 wait complete. 2024-05-10T11:09:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-10T11:09:56.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:55 smithi076 ceph-mon[34152]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-05-10T11:09:56.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:55 smithi189 ceph-mon[34842]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-05-10T11:09:56.310 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:55 smithi027 ceph-mon[30054]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-05-10T11:09:56.736 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-05-10T11:09:54.830547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662060,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":478,"num_read_kb":347,"num_write":375,"num_write_kb":2296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":2952552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361424,"kb_used_data":12260,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580400,"statfs":{"total":1151940427776,"available":1151570329600,"internally_reserved":0,"allocated":12554240,"data_stored":8225342,"data_compressed":497229,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9668,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":65,"num_write":6,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004934"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066004+0000","last_change":"2024-05-10T11:07:47.212921+0000","last_active":"2024-05-10T11:09:23.066004+0000","last_peered":"2024-05-10T11:09:23.066004+0000","last_clean":"2024-05-10T11:09:23.066004+0000","last_became_active":"2024-05-10T11:07:47.212558+0000","last_became_peered":"2024-05-10T11:07:47.212558+0000","last_unstale":"2024-05-10T11:09:23.066004+0000","last_undegraded":"2024-05-10T11:09:23.066004+0000","last_fullsized":"2024-05-10T11:09:23.066004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:47:20.415883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068038+0000","last_change":"2024-05-10T11:07:47.211186+0000","last_active":"2024-05-10T11:09:23.068038+0000","last_peered":"2024-05-10T11:09:23.068038+0000","last_clean":"2024-05-10T11:09:23.068038+0000","last_became_active":"2024-05-10T11:07:47.203487+0000","last_became_peered":"2024-05-10T11:07:47.203487+0000","last_unstale":"2024-05-10T11:09:23.068038+0000","last_undegraded":"2024-05-10T11:09:23.068038+0000","last_fullsized":"2024-05-10T11:09:23.068038+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:51:29.674632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067110+0000","last_change":"2024-05-10T11:07:47.202718+0000","last_active":"2024-05-10T11:09:23.067110+0000","last_peered":"2024-05-10T11:09:23.067110+0000","last_clean":"2024-05-10T11:09:23.067110+0000","last_became_active":"2024-05-10T11:07:47.202048+0000","last_became_peered":"2024-05-10T11:07:47.202048+0000","last_unstale":"2024-05-10T11:09:23.067110+0000","last_undegraded":"2024-05-10T11:09:23.067110+0000","last_fullsized":"2024-05-10T11:09:23.067110+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:01:41.156850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062967+0000","last_change":"2024-05-10T11:07:47.193519+0000","last_active":"2024-05-10T11:09:23.062967+0000","last_peered":"2024-05-10T11:09:23.062967+0000","last_clean":"2024-05-10T11:09:23.062967+0000","last_became_active":"2024-05-10T11:07:47.193200+0000","last_became_peered":"2024-05-10T11:07:47.193200+0000","last_unstale":"2024-05-10T11:09:23.062967+0000","last_undegraded":"2024-05-10T11:09:23.062967+0000","last_fullsized":"2024-05-10T11:09:23.062967+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:01.949453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068968+0000","last_change":"2024-05-10T11:07:47.207750+0000","last_active":"2024-05-10T11:09:23.068968+0000","last_peered":"2024-05-10T11:09:23.068968+0000","last_clean":"2024-05-10T11:09:23.068968+0000","last_became_active":"2024-05-10T11:07:47.207384+0000","last_became_peered":"2024-05-10T11:07:47.207384+0000","last_unstale":"2024-05-10T11:09:23.068968+0000","last_undegraded":"2024-05-10T11:09:23.068968+0000","last_fullsized":"2024-05-10T11:09:23.068968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:09.432766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068679+0000","last_change":"2024-05-10T11:08:22.245930+0000","last_active":"2024-05-10T11:09:23.068679+0000","last_peered":"2024-05-10T11:09:23.068679+0000","last_clean":"2024-05-10T11:09:23.068679+0000","last_became_active":"2024-05-10T11:08:22.245553+0000","last_became_peered":"2024-05-10T11:08:22.245553+0000","last_unstale":"2024-05-10T11:09:23.068679+0000","last_undegraded":"2024-05-10T11:09:23.068679+0000","last_fullsized":"2024-05-10T11:09:23.068679+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:47:31.485623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071570+0000","last_change":"2024-05-10T11:07:47.196666+0000","last_active":"2024-05-10T11:09:23.071570+0000","last_peered":"2024-05-10T11:09:23.071570+0000","last_clean":"2024-05-10T11:09:23.071570+0000","last_became_active":"2024-05-10T11:07:47.196300+0000","last_became_peered":"2024-05-10T11:07:47.196300+0000","last_unstale":"2024-05-10T11:09:23.071570+0000","last_undegraded":"2024-05-10T11:09:23.071570+0000","last_fullsized":"2024-05-10T11:09:23.071570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:55:24.343385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076560+0000","last_change":"2024-05-10T11:07:47.213048+0000","last_active":"2024-05-10T11:09:23.076560+0000","last_peered":"2024-05-10T11:09:23.076560+0000","last_clean":"2024-05-10T11:09:23.076560+0000","last_became_active":"2024-05-10T11:07:47.206359+0000","last_became_peered":"2024-05-10T11:07:47.206359+0000","last_unstale":"2024-05-10T11:09:23.076560+0000","last_undegraded":"2024-05-10T11:09:23.076560+0000","last_fullsized":"2024-05-10T11:09:23.076560+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:32:56.307261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.073801+0000","last_change":"2024-05-10T11:07:47.190945+0000","last_active":"2024-05-10T11:09:23.073801+0000","last_peered":"2024-05-10T11:09:23.073801+0000","last_clean":"2024-05-10T11:09:23.073801+0000","last_became_active":"2024-05-10T11:07:47.190631+0000","last_became_peered":"2024-05-10T11:07:47.190631+0000","last_unstale":"2024-05-10T11:09:23.073801+0000","last_undegraded":"2024-05-10T11:09:23.073801+0000","last_fullsized":"2024-05-10T11:09:23.073801+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:34:08.683912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069521+0000","last_change":"2024-05-10T11:07:47.189540+0000","last_active":"2024-05-10T11:09:23.069521+0000","last_peered":"2024-05-10T11:09:23.069521+0000","last_clean":"2024-05-10T11:09:23.069521+0000","last_became_active":"2024-05-10T11:07:47.189185+0000","last_became_peered":"2024-05-10T11:07:47.189185+0000","last_unstale":"2024-05-10T11:09:23.069521+0000","last_undegraded":"2024-05-10T11:09:23.069521+0000","last_fullsized":"2024-05-10T11:09:23.069521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:55:23.892826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071768+0000","last_change":"2024-05-10T11:07:47.195688+0000","last_active":"2024-05-10T11:09:23.071768+0000","last_peered":"2024-05-10T11:09:23.071768+0000","last_clean":"2024-05-10T11:09:23.071768+0000","last_became_active":"2024-05-10T11:07:47.195081+0000","last_became_peered":"2024-05-10T11:07:47.195081+0000","last_unstale":"2024-05-10T11:09:23.071768+0000","last_undegraded":"2024-05-10T11:09:23.071768+0000","last_fullsized":"2024-05-10T11:09:23.071768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:20:55.240827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062915+0000","last_change":"2024-05-10T11:07:47.193736+0000","last_active":"2024-05-10T11:09:23.062915+0000","last_peered":"2024-05-10T11:09:23.062915+0000","last_clean":"2024-05-10T11:09:23.062915+0000","last_became_active":"2024-05-10T11:07:47.193400+0000","last_became_peered":"2024-05-10T11:07:47.193400+0000","last_unstale":"2024-05-10T11:09:23.062915+0000","last_undegraded":"2024-05-10T11:09:23.062915+0000","last_fullsized":"2024-05-10T11:09:23.062915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:46:28.507330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.080862+0000","last_change":"2024-05-10T11:08:22.042849+0000","last_active":"2024-05-10T11:09:23.080862+0000","last_peered":"2024-05-10T11:09:23.080862+0000","last_clean":"2024-05-10T11:09:23.080862+0000","last_became_active":"2024-05-10T11:08:22.042024+0000","last_became_peered":"2024-05-10T11:08:22.042024+0000","last_unstale":"2024-05-10T11:09:23.080862+0000","last_undegraded":"2024-05-10T11:09:23.080862+0000","last_fullsized":"2024-05-10T11:09:23.080862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:29:08.096541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067382+0000","last_change":"2024-05-10T11:07:47.214016+0000","last_active":"2024-05-10T11:09:23.067382+0000","last_peered":"2024-05-10T11:09:23.067382+0000","last_clean":"2024-05-10T11:09:23.067382+0000","last_became_active":"2024-05-10T11:07:47.209658+0000","last_became_peered":"2024-05-10T11:07:47.209658+0000","last_unstale":"2024-05-10T11:09:23.067382+0000","last_undegraded":"2024-05-10T11:09:23.067382+0000","last_fullsized":"2024-05-10T11:09:23.067382+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:44.771221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067118+0000","last_change":"2024-05-10T11:07:47.210153+0000","last_active":"2024-05-10T11:09:23.067118+0000","last_peered":"2024-05-10T11:09:23.067118+0000","last_clean":"2024-05-10T11:09:23.067118+0000","last_became_active":"2024-05-10T11:07:47.208470+0000","last_became_peered":"2024-05-10T11:07:47.208470+0000","last_unstale":"2024-05-10T11:09:23.067118+0000","last_undegraded":"2024-05-10T11:09:23.067118+0000","last_fullsized":"2024-05-10T11:09:23.067118+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:23:03.614670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.073109+0000","last_change":"2024-05-10T11:07:47.210704+0000","last_active":"2024-05-10T11:09:23.073109+0000","last_peered":"2024-05-10T11:09:23.073109+0000","last_clean":"2024-05-10T11:09:23.073109+0000","last_became_active":"2024-05-10T11:07:47.210317+0000","last_became_peered":"2024-05-10T11:07:47.210317+0000","last_unstale":"2024-05-10T11:09:23.073109+0000","last_undegraded":"2024-05-10T11:09:23.073109+0000","last_fullsized":"2024-05-10T11:09:23.073109+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:47:30.848731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065782+0000","last_change":"2024-05-10T11:07:47.213116+0000","last_active":"2024-05-10T11:09:23.065782+0000","last_peered":"2024-05-10T11:09:23.065782+0000","last_clean":"2024-05-10T11:09:23.065782+0000","last_became_active":"2024-05-10T11:07:47.212567+0000","last_became_peered":"2024-05-10T11:07:47.212567+0000","last_unstale":"2024-05-10T11:09:23.065782+0000","last_undegraded":"2024-05-10T11:09:23.065782+0000","last_fullsized":"2024-05-10T11:09:23.065782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:44:32.222012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068569+0000","last_change":"2024-05-10T11:07:47.200973+0000","last_active":"2024-05-10T11:09:23.068569+0000","last_peered":"2024-05-10T11:09:23.068569+0000","last_clean":"2024-05-10T11:09:23.068569+0000","last_became_active":"2024-05-10T11:07:47.200499+0000","last_became_peered":"2024-05-10T11:07:47.200499+0000","last_unstale":"2024-05-10T11:09:23.068569+0000","last_undegraded":"2024-05-10T11:09:23.068569+0000","last_fullsized":"2024-05-10T11:09:23.068569+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:29:24.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069987+0000","last_change":"2024-05-10T11:08:22.042254+0000","last_active":"2024-05-10T11:09:23.069987+0000","last_peered":"2024-05-10T11:09:23.069987+0000","last_clean":"2024-05-10T11:09:23.069987+0000","last_became_active":"2024-05-10T11:08:22.041906+0000","last_became_peered":"2024-05-10T11:08:22.041906+0000","last_unstale":"2024-05-10T11:09:23.069987+0000","last_undegraded":"2024-05-10T11:09:23.069987+0000","last_fullsized":"2024-05-10T11:09:23.069987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:55:27.280478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070194+0000","last_change":"2024-05-10T11:08:00.897746+0000","last_active":"2024-05-10T11:09:23.070194+0000","last_peered":"2024-05-10T11:09:23.070194+0000","last_clean":"2024-05-10T11:09:23.070194+0000","last_became_active":"2024-05-10T11:08:00.897428+0000","last_became_peered":"2024-05-10T11:08:00.897428+0000","last_unstale":"2024-05-10T11:09:23.070194+0000","last_undegraded":"2024-05-10T11:09:23.070194+0000","last_fullsized":"2024-05-10T11:09:23.070194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:38.846373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":96,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:51.115971+0000","last_change":"2024-05-10T11:07:42.139990+0000","last_active":"2024-05-10T11:09:51.115971+0000","last_peered":"2024-05-10T11:09:51.115971+0000","last_clean":"2024-05-10T11:09:51.115971+0000","last_became_active":"2024-05-10T11:07:42.139612+0000","last_became_peered":"2024-05-10T11:07:42.139612+0000","last_unstale":"2024-05-10T11:09:51.115971+0000","last_undegraded":"2024-05-10T11:09:51.115971+0000","last_fullsized":"2024-05-10T11:09:51.115971+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:48:29.590570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077251+0000","last_change":"2024-05-10T11:07:47.214263+0000","last_active":"2024-05-10T11:09:23.077251+0000","last_peered":"2024-05-10T11:09:23.077251+0000","last_clean":"2024-05-10T11:09:23.077251+0000","last_became_active":"2024-05-10T11:07:47.213946+0000","last_became_peered":"2024-05-10T11:07:47.213946+0000","last_unstale":"2024-05-10T11:09:23.077251+0000","last_undegraded":"2024-05-10T11:09:23.077251+0000","last_fullsized":"2024-05-10T11:09:23.077251+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:54:31.015838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062303+0000","last_change":"2024-05-10T11:08:00.890571+0000","last_active":"2024-05-10T11:09:23.062303+0000","last_peered":"2024-05-10T11:09:23.062303+0000","last_clean":"2024-05-10T11:09:23.062303+0000","last_became_active":"2024-05-10T11:08:00.889468+0000","last_became_peered":"2024-05-10T11:08:00.889468+0000","last_unstale":"2024-05-10T11:09:23.062303+0000","last_undegraded":"2024-05-10T11:09:23.062303+0000","last_fullsized":"2024-05-10T11:09:23.062303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:20:54.342656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077801+0000","last_change":"2024-05-10T11:07:42.383396+0000","last_active":"2024-05-10T11:09:23.077801+0000","last_peered":"2024-05-10T11:09:23.077801+0000","last_clean":"2024-05-10T11:09:23.077801+0000","last_became_active":"2024-05-10T11:07:42.382787+0000","last_became_peered":"2024-05-10T11:07:42.382787+0000","last_unstale":"2024-05-10T11:09:23.077801+0000","last_undegraded":"2024-05-10T11:09:23.077801+0000","last_fullsized":"2024-05-10T11:09:23.077801+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:44:34.240119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067896+0000","last_change":"2024-05-10T11:07:47.216770+0000","last_active":"2024-05-10T11:09:23.067896+0000","last_peered":"2024-05-10T11:09:23.067896+0000","last_clean":"2024-05-10T11:09:23.067896+0000","last_became_active":"2024-05-10T11:07:47.216377+0000","last_became_peered":"2024-05-10T11:07:47.216377+0000","last_unstale":"2024-05-10T11:09:23.067896+0000","last_undegraded":"2024-05-10T11:09:23.067896+0000","last_fullsized":"2024-05-10T11:09:23.067896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:59:22.993888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062458+0000","last_change":"2024-05-10T11:08:00.900614+0000","last_active":"2024-05-10T11:09:23.062458+0000","last_peered":"2024-05-10T11:09:23.062458+0000","last_clean":"2024-05-10T11:09:23.062458+0000","last_became_active":"2024-05-10T11:08:00.900162+0000","last_became_peered":"2024-05-10T11:08:00.900162+0000","last_unstale":"2024-05-10T11:09:23.062458+0000","last_undegraded":"2024-05-10T11:09:23.062458+0000","last_fullsized":"2024-05-10T11:09:23.062458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:42:56.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070222+0000","last_change":"2024-05-10T11:07:42.127749+0000","last_active":"2024-05-10T11:09:23.070222+0000","last_peered":"2024-05-10T11:09:23.070222+0000","last_clean":"2024-05-10T11:09:23.070222+0000","last_became_active":"2024-05-10T11:07:42.127296+0000","last_became_peered":"2024-05-10T11:07:42.127296+0000","last_unstale":"2024-05-10T11:09:23.070222+0000","last_undegraded":"2024-05-10T11:09:23.070222+0000","last_fullsized":"2024-05-10T11:09:23.070222+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:37:37.720542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.073147+0000","last_change":"2024-05-10T11:07:47.198971+0000","last_active":"2024-05-10T11:09:23.073147+0000","last_peered":"2024-05-10T11:09:23.073147+0000","last_clean":"2024-05-10T11:09:23.073147+0000","last_became_active":"2024-05-10T11:07:47.198510+0000","last_became_peered":"2024-05-10T11:07:47.198510+0000","last_unstale":"2024-05-10T11:09:23.073147+0000","last_undegraded":"2024-05-10T11:09:23.073147+0000","last_fullsized":"2024-05-10T11:09:23.073147+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:17:24.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062463+0000","last_change":"2024-05-10T11:08:00.878749+0000","last_active":"2024-05-10T11:09:23.062463+0000","last_peered":"2024-05-10T11:09:23.062463+0000","last_clean":"2024-05-10T11:09:23.062463+0000","last_became_active":"2024-05-10T11:08:00.878271+0000","last_became_peered":"2024-05-10T11:08:00.878271+0000","last_unstale":"2024-05-10T11:09:23.062463+0000","last_undegraded":"2024-05-10T11:09:23.062463+0000","last_fullsized":"2024-05-10T11:09:23.062463+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:39:28.558766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069568+0000","last_change":"2024-05-10T11:07:42.129593+0000","last_active":"2024-05-10T11:09:23.069568+0000","last_peered":"2024-05-10T11:09:23.069568+0000","last_clean":"2024-05-10T11:09:23.069568+0000","last_became_active":"2024-05-10T11:07:42.129250+0000","last_became_peered":"2024-05-10T11:07:42.129250+0000","last_unstale":"2024-05-10T11:09:23.069568+0000","last_undegraded":"2024-05-10T11:09:23.069568+0000","last_fullsized":"2024-05-10T11:09:23.069568+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:22:55.541079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070463+0000","last_change":"2024-05-10T11:08:22.244535+0000","last_active":"2024-05-10T11:09:23.070463+0000","last_peered":"2024-05-10T11:09:23.070463+0000","last_clean":"2024-05-10T11:09:23.070463+0000","last_became_active":"2024-05-10T11:08:22.243716+0000","last_became_peered":"2024-05-10T11:08:22.243716+0000","last_unstale":"2024-05-10T11:09:23.070463+0000","last_undegraded":"2024-05-10T11:09:23.070463+0000","last_fullsized":"2024-05-10T11:09:23.070463+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:11.718999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069774+0000","last_change":"2024-05-10T11:08:00.869806+0000","last_active":"2024-05-10T11:09:23.069774+0000","last_peered":"2024-05-10T11:09:23.069774+0000","last_clean":"2024-05-10T11:09:23.069774+0000","last_became_active":"2024-05-10T11:08:00.869355+0000","last_became_peered":"2024-05-10T11:08:00.869355+0000","last_unstale":"2024-05-10T11:09:23.069774+0000","last_undegraded":"2024-05-10T11:09:23.069774+0000","last_fullsized":"2024-05-10T11:09:23.069774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:36.043412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":98,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:49.130882+0000","last_change":"2024-05-10T11:09:23.076933+0000","last_active":"2024-05-10T11:09:49.130882+0000","last_peered":"2024-05-10T11:09:49.130882+0000","last_clean":"2024-05-10T11:09:49.130882+0000","last_became_active":"2024-05-10T11:09:23.073274+0000","last_became_peered":"2024-05-10T11:09:23.073274+0000","last_unstale":"2024-05-10T11:09:49.130882+0000","last_undegraded":"2024-05-10T11:09:49.130882+0000","last_fullsized":"2024-05-10T11:09:49.130882+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:26:13.923466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069120+0000","last_change":"2024-05-10T11:07:47.200873+0000","last_active":"2024-05-10T11:09:23.069120+0000","last_peered":"2024-05-10T11:09:23.069120+0000","last_clean":"2024-05-10T11:09:23.069120+0000","last_became_active":"2024-05-10T11:07:47.200436+0000","last_became_peered":"2024-05-10T11:07:47.200436+0000","last_unstale":"2024-05-10T11:09:23.069120+0000","last_undegraded":"2024-05-10T11:09:23.069120+0000","last_fullsized":"2024-05-10T11:09:23.069120+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:38:49.588593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069615+0000","last_change":"2024-05-10T11:08:00.885432+0000","last_active":"2024-05-10T11:09:23.069615+0000","last_peered":"2024-05-10T11:09:23.069615+0000","last_clean":"2024-05-10T11:09:23.069615+0000","last_became_active":"2024-05-10T11:08:00.884987+0000","last_became_peered":"2024-05-10T11:08:00.884987+0000","last_unstale":"2024-05-10T11:09:23.069615+0000","last_undegraded":"2024-05-10T11:09:23.069615+0000","last_fullsized":"2024-05-10T11:09:23.069615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:41:24.082722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.093122+0000","last_change":"2024-05-10T11:09:23.093122+0000","last_active":"2024-05-10T11:09:23.093122+0000","last_peered":"2024-05-10T11:09:23.093122+0000","last_clean":"2024-05-10T11:09:23.093122+0000","last_became_active":"2024-05-10T11:09:23.084485+0000","last_became_peered":"2024-05-10T11:09:23.084485+0000","last_unstale":"2024-05-10T11:09:23.093122+0000","last_undegraded":"2024-05-10T11:09:23.093122+0000","last_fullsized":"2024-05-10T11:09:23.093122+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:12:36.793572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068337+0000","last_change":"2024-05-10T11:08:22.036515+0000","last_active":"2024-05-10T11:09:23.068337+0000","last_peered":"2024-05-10T11:09:23.068337+0000","last_clean":"2024-05-10T11:09:23.068337+0000","last_became_active":"2024-05-10T11:08:22.036072+0000","last_became_peered":"2024-05-10T11:08:22.036072+0000","last_unstale":"2024-05-10T11:09:23.068337+0000","last_undegraded":"2024-05-10T11:09:23.068337+0000","last_fullsized":"2024-05-10T11:09:23.068337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:08.734453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6],"acting":[5,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065660+0000","last_change":"2024-05-10T11:08:00.888673+0000","last_active":"2024-05-10T11:09:23.065660+0000","last_peered":"2024-05-10T11:09:23.065660+0000","last_clean":"2024-05-10T11:09:23.065660+0000","last_became_active":"2024-05-10T11:08:00.888302+0000","last_became_peered":"2024-05-10T11:08:00.888302+0000","last_unstale":"2024-05-10T11:09:23.065660+0000","last_undegraded":"2024-05-10T11:09:23.065660+0000","last_fullsized":"2024-05-10T11:09:23.065660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:15:32.276358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.078139+0000","last_change":"2024-05-10T11:07:42.140993+0000","last_active":"2024-05-10T11:09:23.078139+0000","last_peered":"2024-05-10T11:09:23.078139+0000","last_clean":"2024-05-10T11:09:23.078139+0000","last_became_active":"2024-05-10T11:07:42.140501+0000","last_became_peered":"2024-05-10T11:07:42.140501+0000","last_unstale":"2024-05-10T11:09:23.078139+0000","last_undegraded":"2024-05-10T11:09:23.078139+0000","last_fullsized":"2024-05-10T11:09:23.078139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:01:50.324507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077182+0000","last_change":"2024-05-10T11:07:47.212616+0000","last_active":"2024-05-10T11:09:23.077182+0000","last_peered":"2024-05-10T11:09:23.077182+0000","last_clean":"2024-05-10T11:09:23.077182+0000","last_became_active":"2024-05-10T11:07:47.211081+0000","last_became_peered":"2024-05-10T11:07:47.211081+0000","last_unstale":"2024-05-10T11:09:23.077182+0000","last_undegraded":"2024-05-10T11:09:23.077182+0000","last_fullsized":"2024-05-10T11:09:23.077182+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:03:54.404355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067559+0000","last_change":"2024-05-10T11:08:00.888782+0000","last_active":"2024-05-10T11:09:23.067559+0000","last_peered":"2024-05-10T11:09:23.067559+0000","last_clean":"2024-05-10T11:09:23.067559+0000","last_became_active":"2024-05-10T11:08:00.888366+0000","last_became_peered":"2024-05-10T11:08:00.888366+0000","last_unstale":"2024-05-10T11:09:23.067559+0000","last_undegraded":"2024-05-10T11:09:23.067559+0000","last_fullsized":"2024-05-10T11:09:23.067559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:37.040186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066149+0000","last_change":"2024-05-10T11:07:42.128677+0000","last_active":"2024-05-10T11:09:23.066149+0000","last_peered":"2024-05-10T11:09:23.066149+0000","last_clean":"2024-05-10T11:09:23.066149+0000","last_became_active":"2024-05-10T11:07:42.128330+0000","last_became_peered":"2024-05-10T11:07:42.128330+0000","last_unstale":"2024-05-10T11:09:23.066149+0000","last_undegraded":"2024-05-10T11:09:23.066149+0000","last_fullsized":"2024-05-10T11:09:23.066149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:09:11.738485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069144+0000","last_change":"2024-05-10T11:07:47.215257+0000","last_active":"2024-05-10T11:09:23.069144+0000","last_peered":"2024-05-10T11:09:23.069144+0000","last_clean":"2024-05-10T11:09:23.069144+0000","last_became_active":"2024-05-10T11:07:47.214682+0000","last_became_peered":"2024-05-10T11:07:47.214682+0000","last_unstale":"2024-05-10T11:09:23.069144+0000","last_undegraded":"2024-05-10T11:09:23.069144+0000","last_fullsized":"2024-05-10T11:09:23.069144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:35:45.787737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076006+0000","last_change":"2024-05-10T11:08:00.894446+0000","last_active":"2024-05-10T11:09:23.076006+0000","last_peered":"2024-05-10T11:09:23.076006+0000","last_clean":"2024-05-10T11:09:23.076006+0000","last_became_active":"2024-05-10T11:08:00.886473+0000","last_became_peered":"2024-05-10T11:08:00.886473+0000","last_unstale":"2024-05-10T11:09:23.076006+0000","last_undegraded":"2024-05-10T11:09:23.076006+0000","last_fullsized":"2024-05-10T11:09:23.076006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:40.980328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":118,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:50.085349+0000","last_change":"2024-05-10T11:07:42.138750+0000","last_active":"2024-05-10T11:09:50.085349+0000","last_peered":"2024-05-10T11:09:50.085349+0000","last_clean":"2024-05-10T11:09:50.085349+0000","last_became_active":"2024-05-10T11:07:42.138342+0000","last_became_peered":"2024-05-10T11:07:42.138342+0000","last_unstale":"2024-05-10T11:09:50.085349+0000","last_undegraded":"2024-05-10T11:09:50.085349+0000","last_fullsized":"2024-05-10T11:09:50.085349+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:19:45.791550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.073745+0000","last_change":"2024-05-10T11:07:47.204604+0000","last_active":"2024-05-10T11:09:23.073745+0000","last_peered":"2024-05-10T11:09:23.073745+0000","last_clean":"2024-05-10T11:09:23.073745+0000","last_became_active":"2024-05-10T11:07:47.204261+0000","last_became_peered":"2024-05-10T11:07:47.204261+0000","last_unstale":"2024-05-10T11:09:23.073745+0000","last_undegraded":"2024-05-10T11:09:23.073745+0000","last_fullsized":"2024-05-10T11:09:23.073745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:55:46.409545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068646+0000","last_change":"2024-05-10T11:08:22.038090+0000","last_active":"2024-05-10T11:09:23.068646+0000","last_peered":"2024-05-10T11:09:23.068646+0000","last_clean":"2024-05-10T11:09:23.068646+0000","last_became_active":"2024-05-10T11:08:22.037490+0000","last_became_peered":"2024-05-10T11:08:22.037490+0000","last_unstale":"2024-05-10T11:09:23.068646+0000","last_undegraded":"2024-05-10T11:09:23.068646+0000","last_fullsized":"2024-05-10T11:09:23.068646+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:01:46.319401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071634+0000","last_change":"2024-05-10T11:07:42.128636+0000","last_active":"2024-05-10T11:09:23.071634+0000","last_peered":"2024-05-10T11:09:23.071634+0000","last_clean":"2024-05-10T11:09:23.071634+0000","last_became_active":"2024-05-10T11:07:42.128194+0000","last_became_peered":"2024-05-10T11:07:42.128194+0000","last_unstale":"2024-05-10T11:09:23.071634+0000","last_undegraded":"2024-05-10T11:09:23.071634+0000","last_fullsized":"2024-05-10T11:09:23.071634+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:51:54.279292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068054+0000","last_change":"2024-05-10T11:08:22.037807+0000","last_active":"2024-05-10T11:09:23.068054+0000","last_peered":"2024-05-10T11:09:23.068054+0000","last_clean":"2024-05-10T11:09:23.068054+0000","last_became_active":"2024-05-10T11:08:22.037337+0000","last_became_peered":"2024-05-10T11:08:22.037337+0000","last_unstale":"2024-05-10T11:09:23.068054+0000","last_undegraded":"2024-05-10T11:09:23.068054+0000","last_fullsized":"2024-05-10T11:09:23.068054+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:50:06.481222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062684+0000","last_change":"2024-05-10T11:08:00.893684+0000","last_active":"2024-05-10T11:09:23.062684+0000","last_peered":"2024-05-10T11:09:23.062684+0000","last_clean":"2024-05-10T11:09:23.062684+0000","last_became_active":"2024-05-10T11:08:00.893334+0000","last_became_peered":"2024-05-10T11:08:00.893334+0000","last_unstale":"2024-05-10T11:09:23.062684+0000","last_undegraded":"2024-05-10T11:09:23.062684+0000","last_fullsized":"2024-05-10T11:09:23.062684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:58:04.812308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066307+0000","last_change":"2024-05-10T11:07:42.382086+0000","last_active":"2024-05-10T11:09:23.066307+0000","last_peered":"2024-05-10T11:09:23.066307+0000","last_clean":"2024-05-10T11:09:23.066307+0000","last_became_active":"2024-05-10T11:07:42.381744+0000","last_became_peered":"2024-05-10T11:07:42.381744+0000","last_unstale":"2024-05-10T11:09:23.066307+0000","last_undegraded":"2024-05-10T11:09:23.066307+0000","last_fullsized":"2024-05-10T11:09:23.066307+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:15:10.893834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.073185+0000","last_change":"2024-05-10T11:07:47.194683+0000","last_active":"2024-05-10T11:09:23.073185+0000","last_peered":"2024-05-10T11:09:23.073185+0000","last_clean":"2024-05-10T11:09:23.073185+0000","last_became_active":"2024-05-10T11:07:47.194319+0000","last_became_peered":"2024-05-10T11:07:47.194319+0000","last_unstale":"2024-05-10T11:09:23.073185+0000","last_undegraded":"2024-05-10T11:09:23.073185+0000","last_fullsized":"2024-05-10T11:09:23.073185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:41:48.437093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069563+0000","last_change":"2024-05-10T11:08:00.889616+0000","last_active":"2024-05-10T11:09:23.069563+0000","last_peered":"2024-05-10T11:09:23.069563+0000","last_clean":"2024-05-10T11:09:23.069563+0000","last_became_active":"2024-05-10T11:08:00.882179+0000","last_became_peered":"2024-05-10T11:08:00.882179+0000","last_unstale":"2024-05-10T11:09:23.069563+0000","last_undegraded":"2024-05-10T11:09:23.069563+0000","last_fullsized":"2024-05-10T11:09:23.069563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:23:12.407861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'36","reported_seq":159,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:50.121414+0000","last_change":"2024-05-10T11:07:42.128796+0000","last_active":"2024-05-10T11:09:50.121414+0000","last_peered":"2024-05-10T11:09:50.121414+0000","last_clean":"2024-05-10T11:09:50.121414+0000","last_became_active":"2024-05-10T11:07:42.128402+0000","last_became_peered":"2024-05-10T11:07:42.128402+0000","last_unstale":"2024-05-10T11:09:50.121414+0000","last_undegraded":"2024-05-10T11:09:50.121414+0000","last_fullsized":"2024-05-10T11:09:50.121414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:50:09.205102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64645,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":79,"num_read_kb":103,"num_write":38,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070039+0000","last_change":"2024-05-10T11:07:47.188304+0000","last_active":"2024-05-10T11:09:23.070039+0000","last_peered":"2024-05-10T11:09:23.070039+0000","last_clean":"2024-05-10T11:09:23.070039+0000","last_became_active":"2024-05-10T11:07:47.187798+0000","last_became_peered":"2024-05-10T11:07:47.187798+0000","last_unstale":"2024-05-10T11:09:23.070039+0000","last_undegraded":"2024-05-10T11:09:23.070039+0000","last_fullsized":"2024-05-10T11:09:23.070039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:50:21.059060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066711+0000","last_change":"2024-05-10T11:08:00.886669+0000","last_active":"2024-05-10T11:09:23.066711+0000","last_peered":"2024-05-10T11:09:23.066711+0000","last_clean":"2024-05-10T11:09:23.066711+0000","last_became_active":"2024-05-10T11:08:00.886249+0000","last_became_peered":"2024-05-10T11:08:00.886249+0000","last_unstale":"2024-05-10T11:09:23.066711+0000","last_undegraded":"2024-05-10T11:09:23.066711+0000","last_fullsized":"2024-05-10T11:09:23.066711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:10:07.707462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.095713+0000","last_change":"2024-05-10T11:09:23.095352+0000","last_active":"2024-05-10T11:09:23.095713+0000","last_peered":"2024-05-10T11:09:23.095713+0000","last_clean":"2024-05-10T11:09:23.095713+0000","last_became_active":"2024-05-10T11:09:23.083116+0000","last_became_peered":"2024-05-10T11:09:23.083116+0000","last_unstale":"2024-05-10T11:09:23.095713+0000","last_undegraded":"2024-05-10T11:09:23.095713+0000","last_fullsized":"2024-05-10T11:09:23.095713+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:24:28.945724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.080153+0000","last_change":"2024-05-10T11:07:47.207657+0000","last_active":"2024-05-10T11:09:23.080153+0000","last_peered":"2024-05-10T11:09:23.080153+0000","last_clean":"2024-05-10T11:09:23.080153+0000","last_became_active":"2024-05-10T11:07:47.207317+0000","last_became_peered":"2024-05-10T11:07:47.207317+0000","last_unstale":"2024-05-10T11:09:23.080153+0000","last_undegraded":"2024-05-10T11:09:23.080153+0000","last_fullsized":"2024-05-10T11:09:23.080153+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:07:33.961642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065058+0000","last_change":"2024-05-10T11:08:00.884076+0000","last_active":"2024-05-10T11:09:23.065058+0000","last_peered":"2024-05-10T11:09:23.065058+0000","last_clean":"2024-05-10T11:09:23.065058+0000","last_became_active":"2024-05-10T11:08:00.883670+0000","last_became_peered":"2024-05-10T11:08:00.883670+0000","last_unstale":"2024-05-10T11:09:23.065058+0000","last_undegraded":"2024-05-10T11:09:23.065058+0000","last_fullsized":"2024-05-10T11:09:23.065058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:56.447836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067560+0000","last_change":"2024-05-10T11:07:42.135550+0000","last_active":"2024-05-10T11:09:23.067560+0000","last_peered":"2024-05-10T11:09:23.067560+0000","last_clean":"2024-05-10T11:09:23.067560+0000","last_became_active":"2024-05-10T11:07:42.135192+0000","last_became_peered":"2024-05-10T11:07:42.135192+0000","last_unstale":"2024-05-10T11:09:23.067560+0000","last_undegraded":"2024-05-10T11:09:23.067560+0000","last_fullsized":"2024-05-10T11:09:23.067560+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:17.100873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071503+0000","last_change":"2024-05-10T11:07:47.199529+0000","last_active":"2024-05-10T11:09:23.071503+0000","last_peered":"2024-05-10T11:09:23.071503+0000","last_clean":"2024-05-10T11:09:23.071503+0000","last_became_active":"2024-05-10T11:07:47.199070+0000","last_became_peered":"2024-05-10T11:07:47.199070+0000","last_unstale":"2024-05-10T11:09:23.071503+0000","last_undegraded":"2024-05-10T11:09:23.071503+0000","last_fullsized":"2024-05-10T11:09:23.071503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:24.446639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066958+0000","last_change":"2024-05-10T11:08:00.898607+0000","last_active":"2024-05-10T11:09:23.066958+0000","last_peered":"2024-05-10T11:09:23.066958+0000","last_clean":"2024-05-10T11:09:23.066958+0000","last_became_active":"2024-05-10T11:08:00.897768+0000","last_became_peered":"2024-05-10T11:08:00.897768+0000","last_unstale":"2024-05-10T11:09:23.066958+0000","last_undegraded":"2024-05-10T11:09:23.066958+0000","last_fullsized":"2024-05-10T11:09:23.066958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:24:42.125203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:51.091380+0000","last_change":"2024-05-10T11:07:42.126907+0000","last_active":"2024-05-10T11:09:51.091380+0000","last_peered":"2024-05-10T11:09:51.091380+0000","last_clean":"2024-05-10T11:09:51.091380+0000","last_became_active":"2024-05-10T11:07:42.126462+0000","last_became_peered":"2024-05-10T11:07:42.126462+0000","last_unstale":"2024-05-10T11:09:51.091380+0000","last_undegraded":"2024-05-10T11:09:51.091380+0000","last_fullsized":"2024-05-10T11:09:51.091380+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:18:53.971150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072062+0000","last_change":"2024-05-10T11:07:47.191888+0000","last_active":"2024-05-10T11:09:23.072062+0000","last_peered":"2024-05-10T11:09:23.072062+0000","last_clean":"2024-05-10T11:09:23.072062+0000","last_became_active":"2024-05-10T11:07:47.191557+0000","last_became_peered":"2024-05-10T11:07:47.191557+0000","last_unstale":"2024-05-10T11:09:23.072062+0000","last_undegraded":"2024-05-10T11:09:23.072062+0000","last_fullsized":"2024-05-10T11:09:23.072062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:01:36.529493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079253+0000","last_change":"2024-05-10T11:08:00.894385+0000","last_active":"2024-05-10T11:09:23.079253+0000","last_peered":"2024-05-10T11:09:23.079253+0000","last_clean":"2024-05-10T11:09:23.079253+0000","last_became_active":"2024-05-10T11:08:00.893706+0000","last_became_peered":"2024-05-10T11:08:00.893706+0000","last_unstale":"2024-05-10T11:09:23.079253+0000","last_undegraded":"2024-05-10T11:09:23.079253+0000","last_fullsized":"2024-05-10T11:09:23.079253+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:02.069878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068787+0000","last_change":"2024-05-10T11:07:42.385661+0000","last_active":"2024-05-10T11:09:23.068787+0000","last_peered":"2024-05-10T11:09:23.068787+0000","last_clean":"2024-05-10T11:09:23.068787+0000","last_became_active":"2024-05-10T11:07:42.385308+0000","last_became_peered":"2024-05-10T11:07:42.385308+0000","last_unstale":"2024-05-10T11:09:23.068787+0000","last_undegraded":"2024-05-10T11:09:23.068787+0000","last_fullsized":"2024-05-10T11:09:23.068787+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:57:25.997730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065481+0000","last_change":"2024-05-10T11:07:47.191574+0000","last_active":"2024-05-10T11:09:23.065481+0000","last_peered":"2024-05-10T11:09:23.065481+0000","last_clean":"2024-05-10T11:09:23.065481+0000","last_became_active":"2024-05-10T11:07:47.191210+0000","last_became_peered":"2024-05-10T11:07:47.191210+0000","last_unstale":"2024-05-10T11:09:23.065481+0000","last_undegraded":"2024-05-10T11:09:23.065481+0000","last_fullsized":"2024-05-10T11:09:23.065481+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:11:35.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065575+0000","last_change":"2024-05-10T11:08:00.890076+0000","last_active":"2024-05-10T11:09:23.065575+0000","last_peered":"2024-05-10T11:09:23.065575+0000","last_clean":"2024-05-10T11:09:23.065575+0000","last_became_active":"2024-05-10T11:08:00.889630+0000","last_became_peered":"2024-05-10T11:08:00.889630+0000","last_unstale":"2024-05-10T11:09:23.065575+0000","last_undegraded":"2024-05-10T11:09:23.065575+0000","last_fullsized":"2024-05-10T11:09:23.065575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:53:29.257873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.075061+0000","last_change":"2024-05-10T11:09:23.074550+0000","last_active":"2024-05-10T11:09:23.075061+0000","last_peered":"2024-05-10T11:09:23.075061+0000","last_clean":"2024-05-10T11:09:23.075061+0000","last_became_active":"2024-05-10T11:09:23.067169+0000","last_became_peered":"2024-05-10T11:09:23.067169+0000","last_unstale":"2024-05-10T11:09:23.075061+0000","last_undegraded":"2024-05-10T11:09:23.075061+0000","last_fullsized":"2024-05-10T11:09:23.075061+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:45:06.303003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065275+0000","last_change":"2024-05-10T11:07:47.192042+0000","last_active":"2024-05-10T11:09:23.065275+0000","last_peered":"2024-05-10T11:09:23.065275+0000","last_clean":"2024-05-10T11:09:23.065275+0000","last_became_active":"2024-05-10T11:07:47.191632+0000","last_became_peered":"2024-05-10T11:07:47.191632+0000","last_unstale":"2024-05-10T11:09:23.065275+0000","last_undegraded":"2024-05-10T11:09:23.065275+0000","last_fullsized":"2024-05-10T11:09:23.065275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:55:08.564841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068998+0000","last_change":"2024-05-10T11:08:22.244721+0000","last_active":"2024-05-10T11:09:23.068998+0000","last_peered":"2024-05-10T11:09:23.068998+0000","last_clean":"2024-05-10T11:09:23.068998+0000","last_became_active":"2024-05-10T11:08:22.244274+0000","last_became_peered":"2024-05-10T11:08:22.244274+0000","last_unstale":"2024-05-10T11:09:23.068998+0000","last_undegraded":"2024-05-10T11:09:23.068998+0000","last_fullsized":"2024-05-10T11:09:23.068998+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:26.218071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070759+0000","last_change":"2024-05-10T11:07:42.388358+0000","last_active":"2024-05-10T11:09:23.070759+0000","last_peered":"2024-05-10T11:09:23.070759+0000","last_clean":"2024-05-10T11:09:23.070759+0000","last_became_active":"2024-05-10T11:07:42.387806+0000","last_became_peered":"2024-05-10T11:07:42.387806+0000","last_unstale":"2024-05-10T11:09:23.070759+0000","last_undegraded":"2024-05-10T11:09:23.070759+0000","last_fullsized":"2024-05-10T11:09:23.070759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:27:38.387332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068854+0000","last_change":"2024-05-10T11:07:47.213602+0000","last_active":"2024-05-10T11:09:23.068854+0000","last_peered":"2024-05-10T11:09:23.068854+0000","last_clean":"2024-05-10T11:09:23.068854+0000","last_became_active":"2024-05-10T11:07:47.213184+0000","last_became_peered":"2024-05-10T11:07:47.213184+0000","last_unstale":"2024-05-10T11:09:23.068854+0000","last_undegraded":"2024-05-10T11:09:23.068854+0000","last_fullsized":"2024-05-10T11:09:23.068854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T23:01:52.253279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079537+0000","last_change":"2024-05-10T11:08:00.889850+0000","last_active":"2024-05-10T11:09:23.079537+0000","last_peered":"2024-05-10T11:09:23.079537+0000","last_clean":"2024-05-10T11:09:23.079537+0000","last_became_active":"2024-05-10T11:08:00.888526+0000","last_became_peered":"2024-05-10T11:08:00.888526+0000","last_unstale":"2024-05-10T11:09:23.079537+0000","last_undegraded":"2024-05-10T11:09:23.079537+0000","last_fullsized":"2024-05-10T11:09:23.079537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:11:43.899628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:53.121072+0000","last_change":"2024-05-10T11:07:42.130650+0000","last_active":"2024-05-10T11:09:53.121072+0000","last_peered":"2024-05-10T11:09:53.121072+0000","last_clean":"2024-05-10T11:09:53.121072+0000","last_became_active":"2024-05-10T11:07:42.130259+0000","last_became_peered":"2024-05-10T11:07:42.130259+0000","last_unstale":"2024-05-10T11:09:53.121072+0000","last_undegraded":"2024-05-10T11:09:53.121072+0000","last_fullsized":"2024-05-10T11:09:53.121072+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.870092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068274+0000","last_change":"2024-05-10T11:07:47.204159+0000","last_active":"2024-05-10T11:09:23.068274+0000","last_peered":"2024-05-10T11:09:23.068274+0000","last_clean":"2024-05-10T11:09:23.068274+0000","last_became_active":"2024-05-10T11:07:47.203723+0000","last_became_peered":"2024-05-10T11:07:47.203723+0000","last_unstale":"2024-05-10T11:09:23.068274+0000","last_undegraded":"2024-05-10T11:09:23.068274+0000","last_fullsized":"2024-05-10T11:09:23.068274+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:44:48.232321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.081910+0000","last_change":"2024-05-10T11:08:00.918501+0000","last_active":"2024-05-10T11:09:23.081910+0000","last_peered":"2024-05-10T11:09:23.081910+0000","last_clean":"2024-05-10T11:09:23.081910+0000","last_became_active":"2024-05-10T11:08:00.918071+0000","last_became_peered":"2024-05-10T11:08:00.918071+0000","last_unstale":"2024-05-10T11:09:23.081910+0000","last_undegraded":"2024-05-10T11:09:23.081910+0000","last_fullsized":"2024-05-10T11:09:23.081910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:23:08.390953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067179+0000","last_change":"2024-05-10T11:08:22.036385+0000","last_active":"2024-05-10T11:09:23.067179+0000","last_peered":"2024-05-10T11:09:23.067179+0000","last_clean":"2024-05-10T11:09:23.067179+0000","last_became_active":"2024-05-10T11:08:22.035969+0000","last_became_peered":"2024-05-10T11:08:22.035969+0000","last_unstale":"2024-05-10T11:09:23.067179+0000","last_undegraded":"2024-05-10T11:09:23.067179+0000","last_fullsized":"2024-05-10T11:09:23.067179+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:14:24.069422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:52.092130+0000","last_change":"2024-05-10T11:07:42.381926+0000","last_active":"2024-05-10T11:09:52.092130+0000","last_peered":"2024-05-10T11:09:52.092130+0000","last_clean":"2024-05-10T11:09:52.092130+0000","last_became_active":"2024-05-10T11:07:42.381301+0000","last_became_peered":"2024-05-10T11:07:42.381301+0000","last_unstale":"2024-05-10T11:09:52.092130+0000","last_undegraded":"2024-05-10T11:09:52.092130+0000","last_fullsized":"2024-05-10T11:09:52.092130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:48:32.291029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071656+0000","last_change":"2024-05-10T11:08:00.889959+0000","last_active":"2024-05-10T11:09:23.071656+0000","last_peered":"2024-05-10T11:09:23.071656+0000","last_clean":"2024-05-10T11:09:23.071656+0000","last_became_active":"2024-05-10T11:08:00.889514+0000","last_became_peered":"2024-05-10T11:08:00.889514+0000","last_unstale":"2024-05-10T11:09:23.071656+0000","last_undegraded":"2024-05-10T11:09:23.071656+0000","last_fullsized":"2024-05-10T11:09:23.071656+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.557175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076563+0000","last_change":"2024-05-10T11:07:47.218025+0000","last_active":"2024-05-10T11:09:23.076563+0000","last_peered":"2024-05-10T11:09:23.076563+0000","last_clean":"2024-05-10T11:09:23.076563+0000","last_became_active":"2024-05-10T11:07:47.207089+0000","last_became_peered":"2024-05-10T11:07:47.207089+0000","last_unstale":"2024-05-10T11:09:23.076563+0000","last_undegraded":"2024-05-10T11:09:23.076563+0000","last_fullsized":"2024-05-10T11:09:23.076563+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:15:50.363129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062645+0000","last_change":"2024-05-10T11:07:42.138182+0000","last_active":"2024-05-10T11:09:23.062645+0000","last_peered":"2024-05-10T11:09:23.062645+0000","last_clean":"2024-05-10T11:09:23.062645+0000","last_became_active":"2024-05-10T11:07:42.137734+0000","last_became_peered":"2024-05-10T11:07:42.137734+0000","last_unstale":"2024-05-10T11:09:23.062645+0000","last_undegraded":"2024-05-10T11:09:23.062645+0000","last_fullsized":"2024-05-10T11:09:23.062645+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:02:30.778878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":119,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071366+0000","last_change":"2024-05-10T11:06:06.321257+0000","last_active":"2024-05-10T11:09:23.071366+0000","last_peered":"2024-05-10T11:09:23.071366+0000","last_clean":"2024-05-10T11:09:23.071366+0000","last_became_active":"2024-05-10T11:06:06.316928+0000","last_became_peered":"2024-05-10T11:06:06.316928+0000","last_unstale":"2024-05-10T11:09:23.071366+0000","last_undegraded":"2024-05-10T11:09:23.071366+0000","last_fullsized":"2024-05-10T11:09:23.071366+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_clean_scrub_stamp":"2024-05-10T11:02:07.346111+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:32:43.670246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065268+0000","last_change":"2024-05-10T11:08:00.918499+0000","last_active":"2024-05-10T11:09:23.065268+0000","last_peered":"2024-05-10T11:09:23.065268+0000","last_clean":"2024-05-10T11:09:23.065268+0000","last_became_active":"2024-05-10T11:08:00.917950+0000","last_became_peered":"2024-05-10T11:08:00.917950+0000","last_unstale":"2024-05-10T11:09:23.065268+0000","last_undegraded":"2024-05-10T11:09:23.065268+0000","last_fullsized":"2024-05-10T11:09:23.065268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:07.591312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077638+0000","last_change":"2024-05-10T11:07:47.194968+0000","last_active":"2024-05-10T11:09:23.077638+0000","last_peered":"2024-05-10T11:09:23.077638+0000","last_clean":"2024-05-10T11:09:23.077638+0000","last_became_active":"2024-05-10T11:07:47.194645+0000","last_became_peered":"2024-05-10T11:07:47.194645+0000","last_unstale":"2024-05-10T11:09:23.077638+0000","last_undegraded":"2024-05-10T11:09:23.077638+0000","last_fullsized":"2024-05-10T11:09:23.077638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:34.962163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072001+0000","last_change":"2024-05-10T11:07:42.126378+0000","last_active":"2024-05-10T11:09:23.072001+0000","last_peered":"2024-05-10T11:09:23.072001+0000","last_clean":"2024-05-10T11:09:23.072001+0000","last_became_active":"2024-05-10T11:07:42.125984+0000","last_became_peered":"2024-05-10T11:07:42.125984+0000","last_unstale":"2024-05-10T11:09:23.072001+0000","last_undegraded":"2024-05-10T11:09:23.072001+0000","last_fullsized":"2024-05-10T11:09:23.072001+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:15:50.378737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065752+0000","last_change":"2024-05-10T11:08:00.872342+0000","last_active":"2024-05-10T11:09:23.065752+0000","last_peered":"2024-05-10T11:09:23.065752+0000","last_clean":"2024-05-10T11:09:23.065752+0000","last_became_active":"2024-05-10T11:08:00.871926+0000","last_became_peered":"2024-05-10T11:08:00.871926+0000","last_unstale":"2024-05-10T11:09:23.065752+0000","last_undegraded":"2024-05-10T11:09:23.065752+0000","last_fullsized":"2024-05-10T11:09:23.065752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:27:16.519752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070093+0000","last_change":"2024-05-10T11:07:42.384236+0000","last_active":"2024-05-10T11:09:23.070093+0000","last_peered":"2024-05-10T11:09:23.070093+0000","last_clean":"2024-05-10T11:09:23.070093+0000","last_became_active":"2024-05-10T11:07:42.383908+0000","last_became_peered":"2024-05-10T11:07:42.383908+0000","last_unstale":"2024-05-10T11:09:23.070093+0000","last_undegraded":"2024-05-10T11:09:23.070093+0000","last_fullsized":"2024-05-10T11:09:23.070093+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:09:42.302506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.080149+0000","last_change":"2024-05-10T11:07:47.201645+0000","last_active":"2024-05-10T11:09:23.080149+0000","last_peered":"2024-05-10T11:09:23.080149+0000","last_clean":"2024-05-10T11:09:23.080149+0000","last_became_active":"2024-05-10T11:07:47.201300+0000","last_became_peered":"2024-05-10T11:07:47.201300+0000","last_unstale":"2024-05-10T11:09:23.080149+0000","last_undegraded":"2024-05-10T11:09:23.080149+0000","last_fullsized":"2024-05-10T11:09:23.080149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:37.268530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076907+0000","last_change":"2024-05-10T11:08:00.917988+0000","last_active":"2024-05-10T11:09:23.076907+0000","last_peered":"2024-05-10T11:09:23.076907+0000","last_clean":"2024-05-10T11:09:23.076907+0000","last_became_active":"2024-05-10T11:08:00.917457+0000","last_became_peered":"2024-05-10T11:08:00.917457+0000","last_unstale":"2024-05-10T11:09:23.076907+0000","last_undegraded":"2024-05-10T11:09:23.076907+0000","last_fullsized":"2024-05-10T11:09:23.076907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:16:01.816917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'16","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:45.316098+0000","last_change":"2024-05-10T11:07:42.141005+0000","last_active":"2024-05-10T11:09:45.316098+0000","last_peered":"2024-05-10T11:09:45.316098+0000","last_clean":"2024-05-10T11:09:45.316098+0000","last_became_active":"2024-05-10T11:07:42.140634+0000","last_became_peered":"2024-05-10T11:07:42.140634+0000","last_unstale":"2024-05-10T11:09:45.316098+0000","last_undegraded":"2024-05-10T11:09:45.316098+0000","last_fullsized":"2024-05-10T11:09:45.316098+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:00:32.029374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077678+0000","last_change":"2024-05-10T11:07:47.206864+0000","last_active":"2024-05-10T11:09:23.077678+0000","last_peered":"2024-05-10T11:09:23.077678+0000","last_clean":"2024-05-10T11:09:23.077678+0000","last_became_active":"2024-05-10T11:07:47.206387+0000","last_became_peered":"2024-05-10T11:07:47.206387+0000","last_unstale":"2024-05-10T11:09:23.077678+0000","last_undegraded":"2024-05-10T11:09:23.077678+0000","last_fullsized":"2024-05-10T11:09:23.077678+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:12:35.242270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069056+0000","last_change":"2024-05-10T11:08:00.867872+0000","last_active":"2024-05-10T11:09:23.069056+0000","last_peered":"2024-05-10T11:09:23.069056+0000","last_clean":"2024-05-10T11:09:23.069056+0000","last_became_active":"2024-05-10T11:08:00.867461+0000","last_became_peered":"2024-05-10T11:08:00.867461+0000","last_unstale":"2024-05-10T11:09:23.069056+0000","last_undegraded":"2024-05-10T11:09:23.069056+0000","last_fullsized":"2024-05-10T11:09:23.069056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:29.617335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.063128+0000","last_change":"2024-05-10T11:07:42.599520+0000","last_active":"2024-05-10T11:09:23.063128+0000","last_peered":"2024-05-10T11:09:23.063128+0000","last_clean":"2024-05-10T11:09:23.063128+0000","last_became_active":"2024-05-10T11:07:42.599209+0000","last_became_peered":"2024-05-10T11:07:42.599209+0000","last_unstale":"2024-05-10T11:09:23.063128+0000","last_undegraded":"2024-05-10T11:09:23.063128+0000","last_fullsized":"2024-05-10T11:09:23.063128+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:52.899718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068190+0000","last_change":"2024-05-10T11:07:47.218286+0000","last_active":"2024-05-10T11:09:23.068190+0000","last_peered":"2024-05-10T11:09:23.068190+0000","last_clean":"2024-05-10T11:09:23.068190+0000","last_became_active":"2024-05-10T11:07:47.217960+0000","last_became_peered":"2024-05-10T11:07:47.217960+0000","last_unstale":"2024-05-10T11:09:23.068190+0000","last_undegraded":"2024-05-10T11:09:23.068190+0000","last_fullsized":"2024-05-10T11:09:23.068190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:11:37.208359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062722+0000","last_change":"2024-05-10T11:08:00.919578+0000","last_active":"2024-05-10T11:09:23.062722+0000","last_peered":"2024-05-10T11:09:23.062722+0000","last_clean":"2024-05-10T11:09:23.062722+0000","last_became_active":"2024-05-10T11:08:00.918546+0000","last_became_peered":"2024-05-10T11:08:00.918546+0000","last_unstale":"2024-05-10T11:09:23.062722+0000","last_undegraded":"2024-05-10T11:09:23.062722+0000","last_fullsized":"2024-05-10T11:09:23.062722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:39:05.062210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072522+0000","last_change":"2024-05-10T11:07:42.384577+0000","last_active":"2024-05-10T11:09:23.072522+0000","last_peered":"2024-05-10T11:09:23.072522+0000","last_clean":"2024-05-10T11:09:23.072522+0000","last_became_active":"2024-05-10T11:07:42.384301+0000","last_became_peered":"2024-05-10T11:07:42.384301+0000","last_unstale":"2024-05-10T11:09:23.072522+0000","last_undegraded":"2024-05-10T11:09:23.072522+0000","last_fullsized":"2024-05-10T11:09:23.072522+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:11:03.359490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069029+0000","last_change":"2024-05-10T11:07:47.189708+0000","last_active":"2024-05-10T11:09:23.069029+0000","last_peered":"2024-05-10T11:09:23.069029+0000","last_clean":"2024-05-10T11:09:23.069029+0000","last_became_active":"2024-05-10T11:07:47.189367+0000","last_became_peered":"2024-05-10T11:07:47.189367+0000","last_unstale":"2024-05-10T11:09:23.069029+0000","last_undegraded":"2024-05-10T11:09:23.069029+0000","last_fullsized":"2024-05-10T11:09:23.069029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:25.410477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068571+0000","last_change":"2024-05-10T11:08:00.899359+0000","last_active":"2024-05-10T11:09:23.068571+0000","last_peered":"2024-05-10T11:09:23.068571+0000","last_clean":"2024-05-10T11:09:23.068571+0000","last_became_active":"2024-05-10T11:08:00.898924+0000","last_became_peered":"2024-05-10T11:08:00.898924+0000","last_unstale":"2024-05-10T11:09:23.068571+0000","last_undegraded":"2024-05-10T11:09:23.068571+0000","last_fullsized":"2024-05-10T11:09:23.068571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:14:17.427245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076772+0000","last_change":"2024-05-10T11:07:42.140262+0000","last_active":"2024-05-10T11:09:23.076772+0000","last_peered":"2024-05-10T11:09:23.076772+0000","last_clean":"2024-05-10T11:09:23.076772+0000","last_became_active":"2024-05-10T11:07:42.138853+0000","last_became_peered":"2024-05-10T11:07:42.138853+0000","last_unstale":"2024-05-10T11:09:23.076772+0000","last_undegraded":"2024-05-10T11:09:23.076772+0000","last_fullsized":"2024-05-10T11:09:23.076772+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:05:39.995237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.064865+0000","last_change":"2024-05-10T11:07:47.203637+0000","last_active":"2024-05-10T11:09:23.064865+0000","last_peered":"2024-05-10T11:09:23.064865+0000","last_clean":"2024-05-10T11:09:23.064865+0000","last_became_active":"2024-05-10T11:07:47.203286+0000","last_became_peered":"2024-05-10T11:07:47.203286+0000","last_unstale":"2024-05-10T11:09:23.064865+0000","last_undegraded":"2024-05-10T11:09:23.064865+0000","last_fullsized":"2024-05-10T11:09:23.064865+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:27:46.924625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062709+0000","last_change":"2024-05-10T11:08:00.890082+0000","last_active":"2024-05-10T11:09:23.062709+0000","last_peered":"2024-05-10T11:09:23.062709+0000","last_clean":"2024-05-10T11:09:23.062709+0000","last_became_active":"2024-05-10T11:08:00.889673+0000","last_became_peered":"2024-05-10T11:08:00.889673+0000","last_unstale":"2024-05-10T11:09:23.062709+0000","last_undegraded":"2024-05-10T11:09:23.062709+0000","last_fullsized":"2024-05-10T11:09:23.062709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:21:31.167934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069404+0000","last_change":"2024-05-10T11:07:42.130703+0000","last_active":"2024-05-10T11:09:23.069404+0000","last_peered":"2024-05-10T11:09:23.069404+0000","last_clean":"2024-05-10T11:09:23.069404+0000","last_became_active":"2024-05-10T11:07:42.130098+0000","last_became_peered":"2024-05-10T11:07:42.130098+0000","last_unstale":"2024-05-10T11:09:23.069404+0000","last_undegraded":"2024-05-10T11:09:23.069404+0000","last_fullsized":"2024-05-10T11:09:23.069404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:59:39.783151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077158+0000","last_change":"2024-05-10T11:07:47.220497+0000","last_active":"2024-05-10T11:09:23.077158+0000","last_peered":"2024-05-10T11:09:23.077158+0000","last_clean":"2024-05-10T11:09:23.077158+0000","last_became_active":"2024-05-10T11:07:47.219916+0000","last_became_peered":"2024-05-10T11:07:47.219916+0000","last_unstale":"2024-05-10T11:09:23.077158+0000","last_undegraded":"2024-05-10T11:09:23.077158+0000","last_fullsized":"2024-05-10T11:09:23.077158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:26:43.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062646+0000","last_change":"2024-05-10T11:08:00.918258+0000","last_active":"2024-05-10T11:09:23.062646+0000","last_peered":"2024-05-10T11:09:23.062646+0000","last_clean":"2024-05-10T11:09:23.062646+0000","last_became_active":"2024-05-10T11:08:00.917808+0000","last_became_peered":"2024-05-10T11:08:00.917808+0000","last_unstale":"2024-05-10T11:09:23.062646+0000","last_undegraded":"2024-05-10T11:09:23.062646+0000","last_fullsized":"2024-05-10T11:09:23.062646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:14:08.867024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068812+0000","last_change":"2024-05-10T11:07:42.127661+0000","last_active":"2024-05-10T11:09:23.068812+0000","last_peered":"2024-05-10T11:09:23.068812+0000","last_clean":"2024-05-10T11:09:23.068812+0000","last_became_active":"2024-05-10T11:07:42.127256+0000","last_became_peered":"2024-05-10T11:07:42.127256+0000","last_unstale":"2024-05-10T11:09:23.068812+0000","last_undegraded":"2024-05-10T11:09:23.068812+0000","last_fullsized":"2024-05-10T11:09:23.068812+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:14:37.821350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071813+0000","last_change":"2024-05-10T11:07:47.199589+0000","last_active":"2024-05-10T11:09:23.071813+0000","last_peered":"2024-05-10T11:09:23.071813+0000","last_clean":"2024-05-10T11:09:23.071813+0000","last_became_active":"2024-05-10T11:07:47.199145+0000","last_became_peered":"2024-05-10T11:07:47.199145+0000","last_unstale":"2024-05-10T11:09:23.071813+0000","last_undegraded":"2024-05-10T11:09:23.071813+0000","last_fullsized":"2024-05-10T11:09:23.071813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:46.577543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.085049+0000","last_change":"2024-05-10T11:09:23.085049+0000","last_active":"2024-05-10T11:09:23.085049+0000","last_peered":"2024-05-10T11:09:23.085049+0000","last_clean":"2024-05-10T11:09:23.085049+0000","last_became_active":"2024-05-10T11:09:23.084420+0000","last_became_peered":"2024-05-10T11:09:23.084420+0000","last_unstale":"2024-05-10T11:09:23.085049+0000","last_undegraded":"2024-05-10T11:09:23.085049+0000","last_fullsized":"2024-05-10T11:09:23.085049+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:32:33.346434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'5","reported_seq":87,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:50.085871+0000","last_change":"2024-05-10T11:07:42.599395+0000","last_active":"2024-05-10T11:09:50.085871+0000","last_peered":"2024-05-10T11:09:50.085871+0000","last_clean":"2024-05-10T11:09:50.085871+0000","last_became_active":"2024-05-10T11:07:42.598934+0000","last_became_peered":"2024-05-10T11:07:42.598934+0000","last_unstale":"2024-05-10T11:09:50.085871+0000","last_undegraded":"2024-05-10T11:09:50.085871+0000","last_fullsized":"2024-05-10T11:09:50.085871+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:49:42.414789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071892+0000","last_change":"2024-05-10T11:07:47.192087+0000","last_active":"2024-05-10T11:09:23.071892+0000","last_peered":"2024-05-10T11:09:23.071892+0000","last_clean":"2024-05-10T11:09:23.071892+0000","last_became_active":"2024-05-10T11:07:47.191041+0000","last_became_peered":"2024-05-10T11:07:47.191041+0000","last_unstale":"2024-05-10T11:09:23.071892+0000","last_undegraded":"2024-05-10T11:09:23.071892+0000","last_fullsized":"2024-05-10T11:09:23.071892+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:50:14.052863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.080432+0000","last_change":"2024-05-10T11:09:23.080432+0000","last_active":"2024-05-10T11:09:23.080432+0000","last_peered":"2024-05-10T11:09:23.080432+0000","last_clean":"2024-05-10T11:09:23.080432+0000","last_became_active":"2024-05-10T11:09:23.074934+0000","last_became_peered":"2024-05-10T11:09:23.074934+0000","last_unstale":"2024-05-10T11:09:23.080432+0000","last_undegraded":"2024-05-10T11:09:23.080432+0000","last_fullsized":"2024-05-10T11:09:23.080432+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:12:14.613323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066440+0000","last_change":"2024-05-10T11:07:42.130718+0000","last_active":"2024-05-10T11:09:23.066440+0000","last_peered":"2024-05-10T11:09:23.066440+0000","last_clean":"2024-05-10T11:09:23.066440+0000","last_became_active":"2024-05-10T11:07:42.130355+0000","last_became_peered":"2024-05-10T11:07:42.130355+0000","last_unstale":"2024-05-10T11:09:23.066440+0000","last_undegraded":"2024-05-10T11:09:23.066440+0000","last_fullsized":"2024-05-10T11:09:23.066440+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:16:48.093146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071447+0000","last_change":"2024-05-10T11:07:47.213569+0000","last_active":"2024-05-10T11:09:23.071447+0000","last_peered":"2024-05-10T11:09:23.071447+0000","last_clean":"2024-05-10T11:09:23.071447+0000","last_became_active":"2024-05-10T11:07:47.213148+0000","last_became_peered":"2024-05-10T11:07:47.213148+0000","last_unstale":"2024-05-10T11:09:23.071447+0000","last_undegraded":"2024-05-10T11:09:23.071447+0000","last_fullsized":"2024-05-10T11:09:23.071447+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:42:41.161274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065961+0000","last_change":"2024-05-10T11:08:00.870937+0000","last_active":"2024-05-10T11:09:23.065961+0000","last_peered":"2024-05-10T11:09:23.065961+0000","last_clean":"2024-05-10T11:09:23.065961+0000","last_became_active":"2024-05-10T11:08:00.869276+0000","last_became_peered":"2024-05-10T11:08:00.869276+0000","last_unstale":"2024-05-10T11:09:23.065961+0000","last_undegraded":"2024-05-10T11:09:23.065961+0000","last_fullsized":"2024-05-10T11:09:23.065961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:37.058875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079599+0000","last_change":"2024-05-10T11:07:42.130544+0000","last_active":"2024-05-10T11:09:23.079599+0000","last_peered":"2024-05-10T11:09:23.079599+0000","last_clean":"2024-05-10T11:09:23.079599+0000","last_became_active":"2024-05-10T11:07:42.128912+0000","last_became_peered":"2024-05-10T11:07:42.128912+0000","last_unstale":"2024-05-10T11:09:23.079599+0000","last_undegraded":"2024-05-10T11:09:23.079599+0000","last_fullsized":"2024-05-10T11:09:23.079599+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:22:23.521911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072030+0000","last_change":"2024-05-10T11:07:47.194346+0000","last_active":"2024-05-10T11:09:23.072030+0000","last_peered":"2024-05-10T11:09:23.072030+0000","last_clean":"2024-05-10T11:09:23.072030+0000","last_became_active":"2024-05-10T11:07:47.194019+0000","last_became_peered":"2024-05-10T11:07:47.194019+0000","last_unstale":"2024-05-10T11:09:23.072030+0000","last_undegraded":"2024-05-10T11:09:23.072030+0000","last_fullsized":"2024-05-10T11:09:23.072030+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:33:05.262347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.064895+0000","last_change":"2024-05-10T11:07:47.192001+0000","last_active":"2024-05-10T11:09:23.064895+0000","last_peered":"2024-05-10T11:09:23.064895+0000","last_clean":"2024-05-10T11:09:23.064895+0000","last_became_active":"2024-05-10T11:07:47.191668+0000","last_became_peered":"2024-05-10T11:07:47.191668+0000","last_unstale":"2024-05-10T11:09:23.064895+0000","last_undegraded":"2024-05-10T11:09:23.064895+0000","last_fullsized":"2024-05-10T11:09:23.064895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:13:33.680899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070163+0000","last_change":"2024-05-10T11:07:47.192226+0000","last_active":"2024-05-10T11:09:23.070163+0000","last_peered":"2024-05-10T11:09:23.070163+0000","last_clean":"2024-05-10T11:09:23.070163+0000","last_became_active":"2024-05-10T11:07:47.191444+0000","last_became_peered":"2024-05-10T11:07:47.191444+0000","last_unstale":"2024-05-10T11:09:23.070163+0000","last_undegraded":"2024-05-10T11:09:23.070163+0000","last_fullsized":"2024-05-10T11:09:23.070163+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:08.348986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069306+0000","last_change":"2024-05-10T11:07:47.210801+0000","last_active":"2024-05-10T11:09:23.069306+0000","last_peered":"2024-05-10T11:09:23.069306+0000","last_clean":"2024-05-10T11:09:23.069306+0000","last_became_active":"2024-05-10T11:07:47.210496+0000","last_became_peered":"2024-05-10T11:07:47.210496+0000","last_unstale":"2024-05-10T11:09:23.069306+0000","last_undegraded":"2024-05-10T11:09:23.069306+0000","last_fullsized":"2024-05-10T11:09:23.069306+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:11:03.661457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068462+0000","last_change":"2024-05-10T11:07:47.211033+0000","last_active":"2024-05-10T11:09:23.068462+0000","last_peered":"2024-05-10T11:09:23.068462+0000","last_clean":"2024-05-10T11:09:23.068462+0000","last_became_active":"2024-05-10T11:07:47.206061+0000","last_became_peered":"2024-05-10T11:07:47.206061+0000","last_unstale":"2024-05-10T11:09:23.068462+0000","last_undegraded":"2024-05-10T11:09:23.068462+0000","last_fullsized":"2024-05-10T11:09:23.068462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:04:05.878270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079957+0000","last_change":"2024-05-10T11:07:47.214966+0000","last_active":"2024-05-10T11:09:23.079957+0000","last_peered":"2024-05-10T11:09:23.079957+0000","last_clean":"2024-05-10T11:09:23.079957+0000","last_became_active":"2024-05-10T11:07:47.214563+0000","last_became_peered":"2024-05-10T11:07:47.214563+0000","last_unstale":"2024-05-10T11:09:23.079957+0000","last_undegraded":"2024-05-10T11:09:23.079957+0000","last_fullsized":"2024-05-10T11:09:23.079957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:29:01.428893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065626+0000","last_change":"2024-05-10T11:07:47.196740+0000","last_active":"2024-05-10T11:09:23.065626+0000","last_peered":"2024-05-10T11:09:23.065626+0000","last_clean":"2024-05-10T11:09:23.065626+0000","last_became_active":"2024-05-10T11:07:47.196388+0000","last_became_peered":"2024-05-10T11:07:47.196388+0000","last_unstale":"2024-05-10T11:09:23.065626+0000","last_undegraded":"2024-05-10T11:09:23.065626+0000","last_fullsized":"2024-05-10T11:09:23.065626+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:23:09.055338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069342+0000","last_change":"2024-05-10T11:07:47.193568+0000","last_active":"2024-05-10T11:09:23.069342+0000","last_peered":"2024-05-10T11:09:23.069342+0000","last_clean":"2024-05-10T11:09:23.069342+0000","last_became_active":"2024-05-10T11:07:47.193202+0000","last_became_peered":"2024-05-10T11:07:47.193202+0000","last_unstale":"2024-05-10T11:09:23.069342+0000","last_undegraded":"2024-05-10T11:09:23.069342+0000","last_fullsized":"2024-05-10T11:09:23.069342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:36:24.556266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.078184+0000","last_change":"2024-05-10T11:07:47.215344+0000","last_active":"2024-05-10T11:09:23.078184+0000","last_peered":"2024-05-10T11:09:23.078184+0000","last_clean":"2024-05-10T11:09:23.078184+0000","last_became_active":"2024-05-10T11:07:47.214896+0000","last_became_peered":"2024-05-10T11:07:47.214896+0000","last_unstale":"2024-05-10T11:09:23.078184+0000","last_undegraded":"2024-05-10T11:09:23.078184+0000","last_fullsized":"2024-05-10T11:09:23.078184+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:23:07.845212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066258+0000","last_change":"2024-05-10T11:07:47.219047+0000","last_active":"2024-05-10T11:09:23.066258+0000","last_peered":"2024-05-10T11:09:23.066258+0000","last_clean":"2024-05-10T11:09:23.066258+0000","last_became_active":"2024-05-10T11:07:47.218778+0000","last_became_peered":"2024-05-10T11:07:47.218778+0000","last_unstale":"2024-05-10T11:09:23.066258+0000","last_undegraded":"2024-05-10T11:09:23.066258+0000","last_fullsized":"2024-05-10T11:09:23.066258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:02:48.945259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076781+0000","last_change":"2024-05-10T11:07:47.213412+0000","last_active":"2024-05-10T11:09:23.076781+0000","last_peered":"2024-05-10T11:09:23.076781+0000","last_clean":"2024-05-10T11:09:23.076781+0000","last_became_active":"2024-05-10T11:07:47.212963+0000","last_became_peered":"2024-05-10T11:07:47.212963+0000","last_unstale":"2024-05-10T11:09:23.076781+0000","last_undegraded":"2024-05-10T11:09:23.076781+0000","last_fullsized":"2024-05-10T11:09:23.076781+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:33:05.224184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072355+0000","last_change":"2024-05-10T11:07:47.207998+0000","last_active":"2024-05-10T11:09:23.072355+0000","last_peered":"2024-05-10T11:09:23.072355+0000","last_clean":"2024-05-10T11:09:23.072355+0000","last_became_active":"2024-05-10T11:07:47.207298+0000","last_became_peered":"2024-05-10T11:07:47.207298+0000","last_unstale":"2024-05-10T11:09:23.072355+0000","last_undegraded":"2024-05-10T11:09:23.072355+0000","last_fullsized":"2024-05-10T11:09:23.072355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:59:14.001103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.080578+0000","last_change":"2024-05-10T11:07:47.202216+0000","last_active":"2024-05-10T11:09:23.080578+0000","last_peered":"2024-05-10T11:09:23.080578+0000","last_clean":"2024-05-10T11:09:23.080578+0000","last_became_active":"2024-05-10T11:07:47.201867+0000","last_became_peered":"2024-05-10T11:07:47.201867+0000","last_unstale":"2024-05-10T11:09:23.080578+0000","last_undegraded":"2024-05-10T11:09:23.080578+0000","last_fullsized":"2024-05-10T11:09:23.080578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:48:59.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065156+0000","last_change":"2024-05-10T11:07:47.194149+0000","last_active":"2024-05-10T11:09:23.065156+0000","last_peered":"2024-05-10T11:09:23.065156+0000","last_clean":"2024-05-10T11:09:23.065156+0000","last_became_active":"2024-05-10T11:07:47.193838+0000","last_became_peered":"2024-05-10T11:07:47.193838+0000","last_unstale":"2024-05-10T11:09:23.065156+0000","last_undegraded":"2024-05-10T11:09:23.065156+0000","last_fullsized":"2024-05-10T11:09:23.065156+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:46:44.181446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247309,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71458,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":397,"num_read_kb":278,"num_write":250,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":712,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678058,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":522636,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874288,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1150976,"data_stored":1174398,"data_compressed":68282,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070517,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":526572,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266748,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":522832,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462978,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":1119351,"data_compressed":68282,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626504,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":517980,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":583655,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":519436,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":583185,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182488,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":517688,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986259,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":523067,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822734,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":1114542,"data_compressed":68282,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34167,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34167,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-10T11:09:56.742 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-10T11:09:56.742 INFO:tasks.cephadm:Setting up client nodes... 2024-05-10T11:09:56.742 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-10T11:09:57.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:56 smithi076 ceph-mon[34152]: from='client.14631 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:09:57.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:56 smithi189 ceph-mon[34842]: from='client.14631 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:09:57.264 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:09:57.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:56 smithi027 ceph-mon[30054]: from='client.14631 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:09:58.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:57 smithi027 ceph-mon[30054]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-05-10T11:09:58.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:57 smithi076 ceph-mon[34152]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-05-10T11:09:58.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:57 smithi189 ceph-mon[34842]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-05-10T11:09:59.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.486+0000 7f0baf000700 1 -- 172.21.15.27:0/3639518949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ba81047c0 msgr2=0x7f0ba8104c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:59.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.486+0000 7f0baf000700 1 --2- 172.21.15.27:0/3639518949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ba81047c0 0x7f0ba8104c20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0b940098f0 tx=0x7f0b94009c00 comp rx=0 tx=0).stop 2024-05-10T11:09:59.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.486+0000 7f0baf000700 1 -- 172.21.15.27:0/3639518949 shutdown_connections 2024-05-10T11:09:59.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.486+0000 7f0baf000700 1 --2- 172.21.15.27:0/3639518949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ba8105160 0x7f0ba810ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:59.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.486+0000 7f0baf000700 1 --2- 172.21.15.27:0/3639518949 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ba81017e0 0x7f0ba8101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:59.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.486+0000 7f0baf000700 1 --2- 172.21.15.27:0/3639518949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ba81047c0 0x7f0ba8104c20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:59.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.486+0000 7f0baf000700 1 -- 172.21.15.27:0/3639518949 >> 172.21.15.27:0/3639518949 conn(0x7f0ba80fd140 msgr2=0x7f0ba80ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:59.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.486+0000 7f0baf000700 1 -- 172.21.15.27:0/3639518949 shutdown_connections 2024-05-10T11:09:59.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.486+0000 7f0baf000700 1 -- 172.21.15.27:0/3639518949 wait complete. 2024-05-10T11:09:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.487+0000 7f0baf000700 1 Processor -- start 2024-05-10T11:09:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.487+0000 7f0baf000700 1 -- start start 2024-05-10T11:09:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.488+0000 7f0baf000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ba81017e0 0x7f0ba819b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.488+0000 7f0baf000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ba81047c0 0x7f0ba819bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.488+0000 7f0baf000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ba8105160 0x7f0ba81a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.488+0000 7f0baf000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba810efd0 con 0x7f0ba81047c0 2024-05-10T11:09:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.488+0000 7f0baf000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba810ee50 con 0x7f0ba81017e0 2024-05-10T11:09:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.488+0000 7f0baf000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba810f150 con 0x7f0ba8105160 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.488+0000 7f0ba7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ba81017e0 0x7f0ba819b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.488+0000 7f0ba7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ba81017e0 0x7f0ba819b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50254/0 (socket says 172.21.15.27:50254) 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.488+0000 7f0ba7e00700 1 -- 172.21.15.27:0/105723041 learned_addr learned my addr 172.21.15.27:0/105723041 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.488+0000 7f0bad000700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ba8105160 0x7f0ba81a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.489+0000 7f0ba7400700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ba81047c0 0x7f0ba819bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.489+0000 7f0ba7e00700 1 -- 172.21.15.27:0/105723041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ba8105160 msgr2=0x7f0ba81a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.489+0000 7f0ba7e00700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ba8105160 0x7f0ba81a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.489+0000 7f0ba7e00700 1 -- 172.21.15.27:0/105723041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ba81047c0 msgr2=0x7f0ba819bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.489+0000 7f0ba7e00700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ba81047c0 0x7f0ba819bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.489+0000 7f0ba7e00700 1 -- 172.21.15.27:0/105723041 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba81a3470 con 0x7f0ba81017e0 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.489+0000 7f0ba7e00700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ba81017e0 0x7f0ba819b860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800e9f0 tx=0x7f0b9800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.489+0000 7f0ba4c00700 1 -- 172.21.15.27:0/105723041 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98004220 con 0x7f0ba81017e0 2024-05-10T11:09:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.489+0000 7f0ba4c00700 1 -- 172.21.15.27:0/105723041 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b980043c0 con 0x7f0ba81017e0 2024-05-10T11:09:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.489+0000 7f0baf000700 1 -- 172.21.15.27:0/105723041 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba81a3760 con 0x7f0ba81017e0 2024-05-10T11:09:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.490+0000 7f0ba4c00700 1 -- 172.21.15.27:0/105723041 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98003cb0 con 0x7f0ba81017e0 2024-05-10T11:09:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.490+0000 7f0baf000700 1 -- 172.21.15.27:0/105723041 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba81a3cf0 con 0x7f0ba81017e0 2024-05-10T11:09:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.491+0000 7f0baf000700 1 -- 172.21.15.27:0/105723041 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ba80519e0 con 0x7f0ba81017e0 2024-05-10T11:09:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.494+0000 7f0ba4c00700 1 -- 172.21.15.27:0/105723041 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0b9800c7c0 con 0x7f0ba81017e0 2024-05-10T11:09:59.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.495+0000 7f0ba4c00700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0b8c0b2a30 0x7f0b8c0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:09:59.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.495+0000 7f0ba4c00700 1 -- 172.21.15.27:0/105723041 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f0b980e2c10 con 0x7f0ba81017e0 2024-05-10T11:09:59.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.495+0000 7f0ba7400700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0b8c0b2a30 0x7f0b8c0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:09:59.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.496+0000 7f0ba7400700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0b8c0b2a30 0x7f0b8c0b4ef0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400faf0 tx=0x7f0b94019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:09:59.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.497+0000 7f0ba4c00700 1 -- 172.21.15.27:0/105723041 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f0b980aa410 con 0x7f0ba81017e0 2024-05-10T11:09:59.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.832+0000 7f0baf000700 1 -- 172.21.15.27:0/105723041 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f0ba81a3fb0 con 0x7f0ba81017e0 2024-05-10T11:09:59.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.840+0000 7f0ba4c00700 1 -- 172.21.15.27:0/105723041 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f0b980a9d10 con 0x7f0ba81017e0 2024-05-10T11:09:59.841 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-05-10T11:09:59.841 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCHAD5m7yvEMRAAjJxqxo+0e9tu8ze9Dk2oow== 2024-05-10T11:09:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.843+0000 7f0baf000700 1 -- 172.21.15.27:0/105723041 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0b8c0b2a30 msgr2=0x7f0b8c0b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.843+0000 7f0baf000700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0b8c0b2a30 0x7f0b8c0b4ef0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400faf0 tx=0x7f0b94019040 comp rx=0 tx=0).stop 2024-05-10T11:09:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.843+0000 7f0baf000700 1 -- 172.21.15.27:0/105723041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ba81017e0 msgr2=0x7f0ba819b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:09:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.843+0000 7f0baf000700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ba81017e0 0x7f0ba819b860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800e9f0 tx=0x7f0b9800ed00 comp rx=0 tx=0).stop 2024-05-10T11:09:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.843+0000 7f0baf000700 1 -- 172.21.15.27:0/105723041 shutdown_connections 2024-05-10T11:09:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.843+0000 7f0baf000700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0b8c0b2a30 0x7f0b8c0b4ef0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.843+0000 7f0baf000700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ba8105160 0x7f0ba81a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.843+0000 7f0baf000700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ba81017e0 0x7f0ba819b860 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.843+0000 7f0baf000700 1 --2- 172.21.15.27:0/105723041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ba81047c0 0x7f0ba819bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:09:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.843+0000 7f0baf000700 1 -- 172.21.15.27:0/105723041 >> 172.21.15.27:0/105723041 conn(0x7f0ba80fd140 msgr2=0x7f0ba81025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:09:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.844+0000 7f0baf000700 1 -- 172.21.15.27:0/105723041 shutdown_connections 2024-05-10T11:09:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:09:59.844+0000 7f0baf000700 1 -- 172.21.15.27:0/105723041 wait complete. 2024-05-10T11:10:00.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:59 smithi027 ceph-mon[30054]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-10T11:10:00.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:59 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/105723041' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-10T11:10:00.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:59 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-10T11:10:00.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:09:59 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-10T11:10:00.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:59 smithi076 ceph-mon[34152]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-10T11:10:00.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:59 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/105723041' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-10T11:10:00.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:59 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-10T11:10:00.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:09:59 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-10T11:10:00.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:59 smithi189 ceph-mon[34842]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-10T11:10:00.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:59 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/105723041' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-10T11:10:00.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:59 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-10T11:10:00.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:09:59 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-10T11:10:00.801 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T11:10:00.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-10T11:10:00.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-10T11:10:00.959 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-10T11:10:00.959 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-10T11:10:00.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph mgr dump --format=json 2024-05-10T11:10:01.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:00 smithi027 ceph-mon[30054]: overall HEALTH_OK 2024-05-10T11:10:01.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:00 smithi189 ceph-mon[34842]: overall HEALTH_OK 2024-05-10T11:10:01.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:00 smithi076 ceph-mon[34152]: overall HEALTH_OK 2024-05-10T11:10:01.470 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:02.105 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:01 smithi027 ceph-mon[30054]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-10T11:10:02.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:01 smithi189 ceph-mon[34842]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-10T11:10:02.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:01 smithi076 ceph-mon[34152]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-10T11:10:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.904+0000 7f957fe00700 1 -- 172.21.15.27:0/505355774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9580108700 msgr2=0x7f95801102d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.904+0000 7f957fe00700 1 --2- 172.21.15.27:0/505355774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9580108700 0x7f95801102d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9578009f90 tx=0x7f957800b330 comp rx=0 tx=0).stop 2024-05-10T11:10:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.904+0000 7f957fe00700 1 -- 172.21.15.27:0/505355774 shutdown_connections 2024-05-10T11:10:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.904+0000 7f957fe00700 1 --2- 172.21.15.27:0/505355774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9580103620 0x7f9580103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.904+0000 7f957fe00700 1 --2- 172.21.15.27:0/505355774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9580108700 0x7f95801102d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.904+0000 7f957fe00700 1 --2- 172.21.15.27:0/505355774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9580103fd0 0x7f9580108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:03.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.904+0000 7f957fe00700 1 -- 172.21.15.27:0/505355774 >> 172.21.15.27:0/505355774 conn(0x7f95800feeb0 msgr2=0x7f95801012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:03.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.904+0000 7f957fe00700 1 -- 172.21.15.27:0/505355774 shutdown_connections 2024-05-10T11:10:03.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.904+0000 7f957fe00700 1 -- 172.21.15.27:0/505355774 wait complete. 2024-05-10T11:10:03.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.905+0000 7f957fe00700 1 Processor -- start 2024-05-10T11:10:03.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.905+0000 7f957fe00700 1 -- start start 2024-05-10T11:10:03.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.906+0000 7f957fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9580103620 0x7f95800757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:03.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.906+0000 7f957fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9580103fd0 0x7f9580075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:03.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.906+0000 7f957fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9580108700 0x7f9580076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.906+0000 7f957fe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f95801135a0 con 0x7f9580108700 2024-05-10T11:10:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.906+0000 7f957fe00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9580113420 con 0x7f9580103fd0 2024-05-10T11:10:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.906+0000 7f957fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9580113720 con 0x7f9580103620 2024-05-10T11:10:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.906+0000 7f957f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9580108700 0x7f9580076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.906+0000 7f957ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9580103620 0x7f95800757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.906+0000 7f957f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9580108700 0x7f9580076220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40010/0 (socket says 172.21.15.27:40010) 2024-05-10T11:10:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.906+0000 7f957f400700 1 -- 172.21.15.27:0/4086142690 learned_addr learned my addr 172.21.15.27:0/4086142690 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.906+0000 7f957e000700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9580103fd0 0x7f9580075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.907+0000 7f957ea00700 1 -- 172.21.15.27:0/4086142690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9580103fd0 msgr2=0x7f9580075ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.907+0000 7f957ea00700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9580103fd0 0x7f9580075ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.907+0000 7f957ea00700 1 -- 172.21.15.27:0/4086142690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9580108700 msgr2=0x7f9580076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.907+0000 7f957ea00700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9580108700 0x7f9580076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.907+0000 7f957ea00700 1 -- 172.21.15.27:0/4086142690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95801a7f70 con 0x7f9580103620 2024-05-10T11:10:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.907+0000 7f957f400700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9580108700 0x7f9580076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:10:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.907+0000 7f957ea00700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9580103620 0x7f95800757a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f957000e9c0 tx=0x7f957000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.907+0000 7f9577400700 1 -- 172.21.15.27:0/4086142690 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957000c9c0 con 0x7f9580103620 2024-05-10T11:10:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.908+0000 7f9577400700 1 -- 172.21.15.27:0/4086142690 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f957000cb60 con 0x7f9580103620 2024-05-10T11:10:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.908+0000 7f9577400700 1 -- 172.21.15.27:0/4086142690 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9570018420 con 0x7f9580103620 2024-05-10T11:10:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.908+0000 7f957fe00700 1 -- 172.21.15.27:0/4086142690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95801a8260 con 0x7f9580103620 2024-05-10T11:10:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.908+0000 7f957fe00700 1 -- 172.21.15.27:0/4086142690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95801a87a0 con 0x7f9580103620 2024-05-10T11:10:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.909+0000 7f957fe00700 1 -- 172.21.15.27:0/4086142690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f958006a190 con 0x7f9580103620 2024-05-10T11:10:03.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.912+0000 7f9577400700 1 -- 172.21.15.27:0/4086142690 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9570010430 con 0x7f9580103620 2024-05-10T11:10:03.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.913+0000 7f9577400700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f95640b2910 0x7f95640b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:03.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.913+0000 7f9577400700 1 -- 172.21.15.27:0/4086142690 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f95700e2dc0 con 0x7f9580103620 2024-05-10T11:10:03.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.914+0000 7f957e000700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f95640b2910 0x7f95640b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:03.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.914+0000 7f957e000700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f95640b2910 0x7f95640b4dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f956c009680 tx=0x7f956c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:03.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:03.915+0000 7f9577400700 1 -- 172.21.15.27:0/4086142690 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f95700aa530 con 0x7f9580103620 2024-05-10T11:10:04.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:03 smithi027 ceph-mon[30054]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-05-10T11:10:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.224+0000 7f957fe00700 1 -- 172.21.15.27:0/4086142690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f958002e510 con 0x7f9580103620 2024-05-10T11:10:04.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.230+0000 7f9577400700 1 -- 172.21.15.27:0/4086142690 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+288647 (secure 0 0 0) 0x7f95700a9e30 con 0x7f9580103620 2024-05-10T11:10:04.236 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:10:04.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:03 smithi189 ceph-mon[34842]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-05-10T11:10:04.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.236+0000 7f957fe00700 1 -- 172.21.15.27:0/4086142690 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f95640b2910 msgr2=0x7f95640b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:04.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.236+0000 7f957fe00700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f95640b2910 0x7f95640b4dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f956c009680 tx=0x7f956c009260 comp rx=0 tx=0).stop 2024-05-10T11:10:04.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.236+0000 7f957fe00700 1 -- 172.21.15.27:0/4086142690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9580103620 msgr2=0x7f95800757a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:04.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.236+0000 7f957fe00700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9580103620 0x7f95800757a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f957000e9c0 tx=0x7f957000ecd0 comp rx=0 tx=0).stop 2024-05-10T11:10:04.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.237+0000 7f957fe00700 1 -- 172.21.15.27:0/4086142690 shutdown_connections 2024-05-10T11:10:04.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.237+0000 7f957fe00700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f95640b2910 0x7f95640b4dd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:04.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.237+0000 7f957fe00700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9580103620 0x7f95800757a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:04.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.237+0000 7f957fe00700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9580108700 0x7f9580076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:04.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.237+0000 7f957fe00700 1 --2- 172.21.15.27:0/4086142690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9580103fd0 0x7f9580075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:04.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.237+0000 7f957fe00700 1 -- 172.21.15.27:0/4086142690 >> 172.21.15.27:0/4086142690 conn(0x7f95800feeb0 msgr2=0x7f95801011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:04.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.237+0000 7f957fe00700 1 -- 172.21.15.27:0/4086142690 shutdown_connections 2024-05-10T11:10:04.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:04.237+0000 7f957fe00700 1 -- 172.21.15.27:0/4086142690 wait complete. 2024-05-10T11:10:04.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:03 smithi076 ceph-mon[34152]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-05-10T11:10:05.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:04 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4086142690' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-10T11:10:05.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:04 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4086142690' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-10T11:10:05.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:04 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4086142690' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-10T11:10:05.449 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1704535643},{"type":"v1","addr":"172.21.15.27:6801","nonce":1704535643}]},"active_addr":"172.21.15.27:6801/1704535643","active_change":"2024-05-10T10:59:20.614174+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.27:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1410017298}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2991615845}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1021663132}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1450791252}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3561831925}]}]} 2024-05-10T11:10:05.455 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-10T11:10:05.455 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-10T11:10:05.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:10:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:06.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:05 smithi189 ceph-mon[34842]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-05-10T11:10:06.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:05 smithi027 ceph-mon[30054]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-05-10T11:10:06.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:05 smithi076 ceph-mon[34152]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-05-10T11:10:08.040 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:07 smithi027 ceph-mon[30054]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-05-10T11:10:08.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:07 smithi189 ceph-mon[34842]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-05-10T11:10:08.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.325+0000 7fa65e200700 1 -- 172.21.15.27:0/3205132534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6580779d0 msgr2=0x7fa658077db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:08.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.325+0000 7fa65e200700 1 --2- 172.21.15.27:0/3205132534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6580779d0 0x7fa658077db0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa64c009990 tx=0x7fa64c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:10:08.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.325+0000 7fa65e200700 1 -- 172.21.15.27:0/3205132534 shutdown_connections 2024-05-10T11:10:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.325+0000 7fa65e200700 1 --2- 172.21.15.27:0/3205132534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6580779d0 0x7fa658077db0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.325+0000 7fa65e200700 1 --2- 172.21.15.27:0/3205132534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa658114250 0x7fa658116640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.325+0000 7fa65e200700 1 --2- 172.21.15.27:0/3205132534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6580782f0 0x7fa658113d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.325+0000 7fa65e200700 1 -- 172.21.15.27:0/3205132534 >> 172.21.15.27:0/3205132534 conn(0x7fa6581010e0 msgr2=0x7fa658103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.325+0000 7fa65e200700 1 -- 172.21.15.27:0/3205132534 shutdown_connections 2024-05-10T11:10:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.325+0000 7fa65e200700 1 -- 172.21.15.27:0/3205132534 wait complete. 2024-05-10T11:10:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.327+0000 7fa65e200700 1 Processor -- start 2024-05-10T11:10:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.327+0000 7fa65e200700 1 -- start start 2024-05-10T11:10:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6580779d0 0x7fa6581a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65e200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6580782f0 0x7fa6581a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65e200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa658114250 0x7fa6581a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65e200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa658119bd0 con 0x7fa6580782f0 2024-05-10T11:10:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65e200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa658119a50 con 0x7fa658114250 2024-05-10T11:10:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa658119d50 con 0x7fa6580779d0 2024-05-10T11:10:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6580779d0 0x7fa6581a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6580779d0 0x7fa6581a0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:43166/0 (socket says 172.21.15.27:43166) 2024-05-10T11:10:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65ce00700 1 -- 172.21.15.27:0/1640757671 learned_addr learned my addr 172.21.15.27:0/1640757671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa657e00700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6580782f0 0x7fa6581a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65d800700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa658114250 0x7fa6581a49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65ce00700 1 -- 172.21.15.27:0/1640757671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa658114250 msgr2=0x7fa6581a49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65ce00700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa658114250 0x7fa6581a49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65ce00700 1 -- 172.21.15.27:0/1640757671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6580782f0 msgr2=0x7fa6581a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65ce00700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6580782f0 0x7fa6581a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.328+0000 7fa65ce00700 1 -- 172.21.15.27:0/1640757671 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6581a50e0 con 0x7fa6580779d0 2024-05-10T11:10:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.329+0000 7fa65ce00700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6580779d0 0x7fa6581a0120 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000c00 tx=0x7fa64c00bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.329+0000 7fa655600700 1 -- 172.21.15.27:0/1640757671 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c027070 con 0x7fa6580779d0 2024-05-10T11:10:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.329+0000 7fa655600700 1 -- 172.21.15.27:0/1640757671 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa64c022470 con 0x7fa6580779d0 2024-05-10T11:10:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.329+0000 7fa655600700 1 -- 172.21.15.27:0/1640757671 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c017650 con 0x7fa6580779d0 2024-05-10T11:10:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.329+0000 7fa65e200700 1 -- 172.21.15.27:0/1640757671 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6581a5370 con 0x7fa6580779d0 2024-05-10T11:10:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.329+0000 7fa65e200700 1 -- 172.21.15.27:0/1640757671 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6581acc50 con 0x7fa6580779d0 2024-05-10T11:10:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.330+0000 7fa65e200700 1 -- 172.21.15.27:0/1640757671 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa658079110 con 0x7fa6580779d0 2024-05-10T11:10:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.333+0000 7fa655600700 1 -- 172.21.15.27:0/1640757671 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fa64c02a070 con 0x7fa6580779d0 2024-05-10T11:10:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.334+0000 7fa655600700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa6400b2990 0x7fa6400b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.335+0000 7fa655600700 1 -- 172.21.15.27:0/1640757671 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fa64c0e3a10 con 0x7fa6580779d0 2024-05-10T11:10:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.335+0000 7fa657e00700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa6400b2990 0x7fa6400b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.335+0000 7fa657e00700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa6400b2990 0x7fa6400b4e50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa65807a6b0 tx=0x7fa644009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.340+0000 7fa655600700 1 -- 172.21.15.27:0/1640757671 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa64c013070 con 0x7fa6580779d0 2024-05-10T11:10:08.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:07 smithi076 ceph-mon[34152]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-05-10T11:10:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.570+0000 7fa65e200700 1 -- 172.21.15.27:0/1640757671 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa65802e510 con 0x7fa6580779d0 2024-05-10T11:10:08.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.572+0000 7fa655600700 1 -- 172.21.15.27:0/1640757671 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24835 (secure 0 0 0) 0x7fa64c0b3020 con 0x7fa6580779d0 2024-05-10T11:10:08.573 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:10:08.573 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":90,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:09:23.055559+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6},{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:10:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.575+0000 7fa65e200700 1 -- 172.21.15.27:0/1640757671 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa6400b2990 msgr2=0x7fa6400b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.575+0000 7fa65e200700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa6400b2990 0x7fa6400b4e50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa65807a6b0 tx=0x7fa644009260 comp rx=0 tx=0).stop 2024-05-10T11:10:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.575+0000 7fa65e200700 1 -- 172.21.15.27:0/1640757671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6580779d0 msgr2=0x7fa6581a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.575+0000 7fa65e200700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6580779d0 0x7fa6581a0120 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000c00 tx=0x7fa64c00bfd0 comp rx=0 tx=0).stop 2024-05-10T11:10:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.575+0000 7fa65e200700 1 -- 172.21.15.27:0/1640757671 shutdown_connections 2024-05-10T11:10:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.575+0000 7fa65e200700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa6400b2990 0x7fa6400b4e50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.575+0000 7fa65e200700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6580779d0 0x7fa6581a0120 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.575+0000 7fa65e200700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa658114250 0x7fa6581a49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.575+0000 7fa65e200700 1 --2- 172.21.15.27:0/1640757671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6580782f0 0x7fa6581a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.575+0000 7fa65e200700 1 -- 172.21.15.27:0/1640757671 >> 172.21.15.27:0/1640757671 conn(0x7fa6581010e0 msgr2=0x7fa6581034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.575+0000 7fa65e200700 1 -- 172.21.15.27:0/1640757671 shutdown_connections 2024-05-10T11:10:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:08.575+0000 7fa65e200700 1 -- 172.21.15.27:0/1640757671 wait complete. 2024-05-10T11:10:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:08 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1640757671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:10:09.335 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:08 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1640757671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:10:09.405 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-10T11:10:09.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:10:09.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:08 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1640757671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:10:09.899 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:10.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:09 smithi189 ceph-mon[34842]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-05-10T11:10:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:09 smithi027 ceph-mon[30054]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-05-10T11:10:10.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:09 smithi076 ceph-mon[34152]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-05-10T11:10:12.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.006+0000 7fa402200700 1 -- 172.21.15.27:0/774038310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc075ff0 msgr2=0x7fa3fc10f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:12.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.006+0000 7fa402200700 1 --2- 172.21.15.27:0/774038310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc075ff0 0x7fa3fc10f9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8009990 tx=0x7fa3e8009ca0 comp rx=0 tx=0).stop 2024-05-10T11:10:12.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.007+0000 7fa402200700 1 -- 172.21.15.27:0/774038310 shutdown_connections 2024-05-10T11:10:12.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.007+0000 7fa402200700 1 --2- 172.21.15.27:0/774038310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3fc0756d0 0x7fa3fc075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:12.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.007+0000 7fa402200700 1 --2- 172.21.15.27:0/774038310 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3fc10ff00 0x7fa3fc1122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:12.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.007+0000 7fa402200700 1 --2- 172.21.15.27:0/774038310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc075ff0 0x7fa3fc10f9c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:12.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.007+0000 7fa402200700 1 -- 172.21.15.27:0/774038310 >> 172.21.15.27:0/774038310 conn(0x7fa3fc07a600 msgr2=0x7fa3fc07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:12.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.007+0000 7fa402200700 1 -- 172.21.15.27:0/774038310 shutdown_connections 2024-05-10T11:10:12.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.007+0000 7fa402200700 1 -- 172.21.15.27:0/774038310 wait complete. 2024-05-10T11:10:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.008+0000 7fa402200700 1 Processor -- start 2024-05-10T11:10:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.008+0000 7fa402200700 1 -- start start 2024-05-10T11:10:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.008+0000 7fa402200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc0756d0 0x7fa3fc19bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.008+0000 7fa402200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3fc075ff0 0x7fa3fc19c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.008+0000 7fa402200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3fc10ff00 0x7fa3fc1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.008+0000 7fa402200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1158f0 con 0x7fa3fc0756d0 2024-05-10T11:10:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.008+0000 7fa402200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc115770 con 0x7fa3fc10ff00 2024-05-10T11:10:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.008+0000 7fa402200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc115a70 con 0x7fa3fc075ff0 2024-05-10T11:10:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.009+0000 7fa3fb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc0756d0 0x7fa3fc19bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.009+0000 7fa3fb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc0756d0 0x7fa3fc19bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53594/0 (socket says 172.21.15.27:53594) 2024-05-10T11:10:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.009+0000 7fa3fb400700 1 -- 172.21.15.27:0/755985209 learned_addr learned my addr 172.21.15.27:0/755985209 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.009+0000 7fa3faa00700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3fc075ff0 0x7fa3fc19c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.009+0000 7fa3fbe00700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3fc10ff00 0x7fa3fc1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.009+0000 7fa3fb400700 1 -- 172.21.15.27:0/755985209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3fc075ff0 msgr2=0x7fa3fc19c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.009+0000 7fa3fb400700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3fc075ff0 0x7fa3fc19c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.009+0000 7fa3fb400700 1 -- 172.21.15.27:0/755985209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3fc10ff00 msgr2=0x7fa3fc1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.009+0000 7fa3fb400700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3fc10ff00 0x7fa3fc1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.009+0000 7fa3fb400700 1 -- 172.21.15.27:0/755985209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3fc1a0d70 con 0x7fa3fc0756d0 2024-05-10T11:10:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.010+0000 7fa3fb400700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc0756d0 0x7fa3fc19bd60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00e9a0 tx=0x7fa3ec00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.010+0000 7fa3f3e00700 1 -- 172.21.15.27:0/755985209 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec00ca10 con 0x7fa3fc0756d0 2024-05-10T11:10:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.010+0000 7fa402200700 1 -- 172.21.15.27:0/755985209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3fc1a1060 con 0x7fa3fc0756d0 2024-05-10T11:10:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.010+0000 7fa3f3e00700 1 -- 172.21.15.27:0/755985209 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3ec00cbb0 con 0x7fa3fc0756d0 2024-05-10T11:10:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.010+0000 7fa3f3e00700 1 -- 172.21.15.27:0/755985209 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec018680 con 0x7fa3fc0756d0 2024-05-10T11:10:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.010+0000 7fa402200700 1 -- 172.21.15.27:0/755985209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3fc1a88e0 con 0x7fa3fc0756d0 2024-05-10T11:10:12.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.011+0000 7fa402200700 1 -- 172.21.15.27:0/755985209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3fc06a190 con 0x7fa3fc0756d0 2024-05-10T11:10:12.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.014+0000 7fa3f3e00700 1 -- 172.21.15.27:0/755985209 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fa3ec010430 con 0x7fa3fc0756d0 2024-05-10T11:10:12.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.015+0000 7fa3f3e00700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa3e00b2990 0x7fa3e00b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:12.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.015+0000 7fa3f3e00700 1 -- 172.21.15.27:0/755985209 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fa3ec0e28e0 con 0x7fa3fc0756d0 2024-05-10T11:10:12.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.016+0000 7fa3faa00700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa3e00b2990 0x7fa3e00b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:12.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.016+0000 7fa3faa00700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa3e00b2990 0x7fa3e00b4e50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc111b90 tx=0x7fa3e8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:12.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.020+0000 7fa3f3e00700 1 -- 172.21.15.27:0/755985209 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa3ec0aa050 con 0x7fa3fc0756d0 2024-05-10T11:10:12.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:11 smithi027 ceph-mon[30054]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-10T11:10:12.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.263+0000 7fa402200700 1 -- 172.21.15.27:0/755985209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa3fc02e510 con 0x7fa3fc0756d0 2024-05-10T11:10:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.268+0000 7fa3f3e00700 1 -- 172.21.15.27:0/755985209 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24835 (secure 0 0 0) 0x7fa3ec0b20d0 con 0x7fa3fc0756d0 2024-05-10T11:10:12.269 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:10:12.270 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":90,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:09:23.055559+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6},{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:10:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.271+0000 7fa402200700 1 -- 172.21.15.27:0/755985209 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa3e00b2990 msgr2=0x7fa3e00b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.271+0000 7fa402200700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa3e00b2990 0x7fa3e00b4e50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc111b90 tx=0x7fa3e8005fd0 comp rx=0 tx=0).stop 2024-05-10T11:10:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.271+0000 7fa402200700 1 -- 172.21.15.27:0/755985209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc0756d0 msgr2=0x7fa3fc19bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.271+0000 7fa402200700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc0756d0 0x7fa3fc19bd60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00e9a0 tx=0x7fa3ec00ecb0 comp rx=0 tx=0).stop 2024-05-10T11:10:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.271+0000 7fa402200700 1 -- 172.21.15.27:0/755985209 shutdown_connections 2024-05-10T11:10:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.271+0000 7fa402200700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa3e00b2990 0x7fa3e00b4e50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.271+0000 7fa402200700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3fc10ff00 0x7fa3fc1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.271+0000 7fa402200700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc0756d0 0x7fa3fc19bd60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.271+0000 7fa402200700 1 --2- 172.21.15.27:0/755985209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3fc075ff0 0x7fa3fc19c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.271+0000 7fa402200700 1 -- 172.21.15.27:0/755985209 >> 172.21.15.27:0/755985209 conn(0x7fa3fc07a600 msgr2=0x7fa3fc100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.272+0000 7fa402200700 1 -- 172.21.15.27:0/755985209 shutdown_connections 2024-05-10T11:10:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:12.272+0000 7fa402200700 1 -- 172.21.15.27:0/755985209 wait complete. 2024-05-10T11:10:12.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:11 smithi076 ceph-mon[34152]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-10T11:10:12.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:11 smithi189 ceph-mon[34842]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-10T11:10:13.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-10T11:10:13.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-10T11:10:13.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-10T11:10:13.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-10T11:10:13.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-10T11:10:13.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-10T11:10:13.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-10T11:10:13.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-10T11:10:13.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-10T11:10:13.104 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-10T11:10:13.104 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-10T11:10:13.104 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-10T11:10:13.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:13 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/755985209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:10:13.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:13 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/755985209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:10:13.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:13 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/755985209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:10:14.206 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:14.211 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:14.215 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:14.258 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:14.281 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:14.282 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:14.282 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:14.288 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:14.316 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:14.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:14 smithi027 ceph-mon[30054]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-10T11:10:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:14 smithi076 ceph-mon[34152]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-10T11:10:14.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:14 smithi189 ceph-mon[34842]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-10T11:10:16.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:16 smithi027 ceph-mon[30054]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-05-10T11:10:16.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:16 smithi076 ceph-mon[34152]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-05-10T11:10:16.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:16 smithi189 ceph-mon[34842]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-05-10T11:10:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:18.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:18 smithi189 ceph-mon[34842]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-10T11:10:18.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:18 smithi027 ceph-mon[30054]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-10T11:10:18.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:18 smithi076 ceph-mon[34152]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-10T11:10:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:20 smithi027 ceph-mon[30054]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:20.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:20 smithi076 ceph-mon[34152]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:20.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:20 smithi189 ceph-mon[34842]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:22.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:22 smithi027 ceph-mon[30054]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:10:22.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:22 smithi076 ceph-mon[34152]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:10:22.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:22 smithi189 ceph-mon[34842]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:10:24.166 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:24 smithi027 ceph-mon[30054]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:24.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:24 smithi076 ceph-mon[34152]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:24.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:24 smithi189 ceph-mon[34842]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.467+0000 7f22c8200700 1 -- 172.21.15.27:0/4245337633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22c010b6c0 msgr2=0x7f22c010baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.467+0000 7f22c8200700 1 --2- 172.21.15.27:0/4245337633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22c010b6c0 0x7f22c010baa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f22b4009990 tx=0x7f22b4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:10:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.467+0000 7f22c8200700 1 -- 172.21.15.27:0/4245337633 shutdown_connections 2024-05-10T11:10:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.467+0000 7f22c8200700 1 --2- 172.21.15.27:0/4245337633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c0106080 0x7f22c0112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.467+0000 7f22c8200700 1 --2- 172.21.15.27:0/4245337633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22c010b6c0 0x7f22c010baa0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.467+0000 7f22c8200700 1 --2- 172.21.15.27:0/4245337633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c01056c0 0x7f22c0105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.467+0000 7f22c8200700 1 -- 172.21.15.27:0/4245337633 >> 172.21.15.27:0/4245337633 conn(0x7f22c01011b0 msgr2=0x7f22c01035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.467+0000 7f22c8200700 1 -- 172.21.15.27:0/4245337633 shutdown_connections 2024-05-10T11:10:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.467+0000 7f22c8200700 1 -- 172.21.15.27:0/4245337633 wait complete. 2024-05-10T11:10:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.468+0000 7f22c8200700 1 Processor -- start 2024-05-10T11:10:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.469+0000 7f22c8200700 1 -- start start 2024-05-10T11:10:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.469+0000 7f22c8200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c01056c0 0x7f22c019bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.469+0000 7f22c8200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22c0106080 0x7f22c019c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.469+0000 7f22c8200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c010b6c0 0x7f22c01a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.469+0000 7f22c8200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f22c0115900 con 0x7f22c01056c0 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.469+0000 7f22c8200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f22c0115780 con 0x7f22c0106080 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.469+0000 7f22c8200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f22c0115a80 con 0x7f22c010b6c0 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.470+0000 7f22c5800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c01056c0 0x7f22c019bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.470+0000 7f22c5800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c01056c0 0x7f22c019bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53852/0 (socket says 172.21.15.27:53852) 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.470+0000 7f22c5800700 1 -- 172.21.15.27:0/2120089136 learned_addr learned my addr 172.21.15.27:0/2120089136 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.470+0000 7f22c4e00700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22c0106080 0x7f22c019c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.470+0000 7f22c6200700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c010b6c0 0x7f22c01a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.470+0000 7f22c5800700 1 -- 172.21.15.27:0/2120089136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c010b6c0 msgr2=0x7f22c01a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.470+0000 7f22c5800700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c010b6c0 0x7f22c01a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.470+0000 7f22c5800700 1 -- 172.21.15.27:0/2120089136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22c0106080 msgr2=0x7f22c019c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.470+0000 7f22c5800700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22c0106080 0x7f22c019c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.470+0000 7f22c5800700 1 -- 172.21.15.27:0/2120089136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c01a0da0 con 0x7f22c01056c0 2024-05-10T11:10:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.470+0000 7f22c4e00700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22c0106080 0x7f22c019c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:10:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.471+0000 7f22c5800700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c01056c0 0x7f22c019bd90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f22b4005910 tx=0x7f22b400b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.471+0000 7f22be000700 1 -- 172.21.15.27:0/2120089136 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b4027070 con 0x7f22c01056c0 2024-05-10T11:10:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.471+0000 7f22be000700 1 -- 172.21.15.27:0/2120089136 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22b4022470 con 0x7f22c01056c0 2024-05-10T11:10:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.471+0000 7f22be000700 1 -- 172.21.15.27:0/2120089136 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b4017630 con 0x7f22c01056c0 2024-05-10T11:10:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.471+0000 7f22c8200700 1 -- 172.21.15.27:0/2120089136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22c01a1030 con 0x7f22c01056c0 2024-05-10T11:10:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.471+0000 7f22c8200700 1 -- 172.21.15.27:0/2120089136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22c01a8910 con 0x7f22c01056c0 2024-05-10T11:10:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.472+0000 7f22c8200700 1 -- 172.21.15.27:0/2120089136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f22c0063c30 con 0x7f22c01056c0 2024-05-10T11:10:25.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.476+0000 7f22be000700 1 -- 172.21.15.27:0/2120089136 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f22b4009d10 con 0x7f22c01056c0 2024-05-10T11:10:25.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.477+0000 7f22be000700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f22a80b2990 0x7f22a80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:25.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.478+0000 7f22c4e00700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f22a80b2990 0x7f22a80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:25.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.478+0000 7f22be000700 1 -- 172.21.15.27:0/2120089136 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f22b40e3bd0 con 0x7f22c01056c0 2024-05-10T11:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.478+0000 7f22be000700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682] conn(0x7f22a80bdc40 0x7f22a80c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.478+0000 7f22be000700 1 -- 172.21.15.27:0/2120089136 --> [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22b4017e50 con 0x7f22a80bdc40 2024-05-10T11:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.478+0000 7f22be000700 1 -- 172.21.15.27:0/2120089136 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f22b40e3f80 con 0x7f22c01056c0 2024-05-10T11:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.478+0000 7f22c6200700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682] conn(0x7f22a80bdc40 0x7f22a80c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.478+0000 7f22c4e00700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f22a80b2990 0x7f22a80b4e50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f22c0103850 tx=0x7f22ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.479+0000 7f22c6200700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682] conn(0x7f22a80bdc40 0x7f22a80c00e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.480+0000 7f22be000700 1 -- 172.21.15.27:0/2120089136 <== osd.5 v2:172.21.15.189:6808/1669864682 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22b4017e50 con 0x7f22a80bdc40 2024-05-10T11:10:25.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.499+0000 7f22c8200700 1 -- 172.21.15.27:0/2120089136 --> [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f22c0064030 con 0x7f22a80bdc40 2024-05-10T11:10:25.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.499+0000 7f22be000700 1 -- 172.21.15.27:0/2120089136 <== osd.5 v2:172.21.15.189:6808/1669864682 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f22c0064030 con 0x7f22a80bdc40 2024-05-10T11:10:25.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.500+0000 7f22c8200700 1 -- 172.21.15.27:0/2120089136 >> [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682] conn(0x7f22a80bdc40 msgr2=0x7f22a80c00e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:25.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.500+0000 7f22c8200700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682] conn(0x7f22a80bdc40 0x7f22a80c00e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:25.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.500+0000 7f22c8200700 1 -- 172.21.15.27:0/2120089136 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f22a80b2990 msgr2=0x7f22a80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:25.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.500+0000 7f22c8200700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f22a80b2990 0x7f22a80b4e50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f22c0103850 tx=0x7f22ac009260 comp rx=0 tx=0).stop 2024-05-10T11:10:25.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.500+0000 7f22c8200700 1 -- 172.21.15.27:0/2120089136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c01056c0 msgr2=0x7f22c019bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:25.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.500+0000 7f22c8200700 1 --2- 172.21.15.27:0/2120089136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c01056c0 0x7f22c019bd90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f22b4005910 tx=0x7f22b400b670 comp rx=0 tx=0).stop 2024-05-10T11:10:25.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.500+0000 7f22c6200700 1 -- 172.21.15.27:0/2120089136 reap_dead start 2024-05-10T11:10:25.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.500+0000 7f22c8200700 1 -- 172.21.15.27:0/2120089136 shutdown_connections 2024-05-10T11:10:25.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.500+0000 7f22c8200700 1 -- 172.21.15.27:0/2120089136 >> 172.21.15.27:0/2120089136 conn(0x7f22c01011b0 msgr2=0x7f22c0101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:25.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.501+0000 7f22c8200700 1 -- 172.21.15.27:0/2120089136 shutdown_connections 2024-05-10T11:10:25.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:25.501+0000 7f22c8200700 1 -- 172.21.15.27:0/2120089136 wait complete. 2024-05-10T11:10:26.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:26 smithi027 ceph-mon[30054]: pgmap v412: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:26.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:26 smithi076 ceph-mon[34152]: pgmap v412: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:26.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:26 smithi189 ceph-mon[34842]: pgmap v412: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:27.636 INFO:teuthology.orchestra.run.smithi027.stdout:154618822741 2024-05-10T11:10:27.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-10T11:10:28.065 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:28 smithi027 ceph-mon[30054]: pgmap v413: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-10T11:10:28.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.114+0000 7f1b4b200700 1 -- 172.21.15.27:0/498406030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b4c0761f0 msgr2=0x7f1b4c10f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:28.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.114+0000 7f1b4b200700 1 --2- 172.21.15.27:0/498406030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b4c0761f0 0x7f1b4c10f6d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009990 tx=0x7f1b38009ca0 comp rx=0 tx=0).stop 2024-05-10T11:10:28.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.114+0000 7f1b4b200700 1 -- 172.21.15.27:0/498406030 shutdown_connections 2024-05-10T11:10:28.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.114+0000 7f1b4b200700 1 --2- 172.21.15.27:0/498406030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b4c10fc10 0x7f1b4c112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:28.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.114+0000 7f1b4b200700 1 --2- 172.21.15.27:0/498406030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b4c0758d0 0x7f1b4c075cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:28.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.114+0000 7f1b4b200700 1 --2- 172.21.15.27:0/498406030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b4c0761f0 0x7f1b4c10f6d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:28.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.114+0000 7f1b4b200700 1 -- 172.21.15.27:0/498406030 >> 172.21.15.27:0/498406030 conn(0x7f1b4c0fef20 msgr2=0x7f1b4c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:28.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.115+0000 7f1b4b200700 1 -- 172.21.15.27:0/498406030 shutdown_connections 2024-05-10T11:10:28.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.115+0000 7f1b4b200700 1 -- 172.21.15.27:0/498406030 wait complete. 2024-05-10T11:10:28.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.116+0000 7f1b4b200700 1 Processor -- start 2024-05-10T11:10:28.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.116+0000 7f1b4b200700 1 -- start start 2024-05-10T11:10:28.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.116+0000 7f1b4b200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b4c0758d0 0x7f1b4c1a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:28.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.116+0000 7f1b4b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b4c0761f0 0x7f1b4c1a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:28.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.117+0000 7f1b4b200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b4c10fc10 0x7f1b4c19bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:28.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.117+0000 7f1b4b200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4c1157f0 con 0x7f1b4c0758d0 2024-05-10T11:10:28.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.117+0000 7f1b4b200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4c115670 con 0x7f1b4c10fc10 2024-05-10T11:10:28.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.117+0000 7f1b4b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4c115970 con 0x7f1b4c0761f0 2024-05-10T11:10:28.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.117+0000 7f1b49400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b4c0761f0 0x7f1b4c1a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:28.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.117+0000 7f1b49400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b4c0761f0 0x7f1b4c1a1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41542/0 (socket says 172.21.15.27:41542) 2024-05-10T11:10:28.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.117+0000 7f1b49400700 1 -- 172.21.15.27:0/2943956886 learned_addr learned my addr 172.21.15.27:0/2943956886 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:28.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.117+0000 7f1b49e00700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b4c0758d0 0x7f1b4c1a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:28.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.117+0000 7f1b4a800700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b4c10fc10 0x7f1b4c19bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:28.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.118+0000 7f1b49e00700 1 -- 172.21.15.27:0/2943956886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b4c0761f0 msgr2=0x7f1b4c1a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:28.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.118+0000 7f1b49e00700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b4c0761f0 0x7f1b4c1a1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:28.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.118+0000 7f1b49e00700 1 -- 172.21.15.27:0/2943956886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b4c10fc10 msgr2=0x7f1b4c19bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:28.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.118+0000 7f1b49e00700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b4c10fc10 0x7f1b4c19bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:28.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.118+0000 7f1b49e00700 1 -- 172.21.15.27:0/2943956886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4c19c450 con 0x7f1b4c0758d0 2024-05-10T11:10:28.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.119+0000 7f1b49e00700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b4c0758d0 0x7f1b4c1a1a50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00e9f0 tx=0x7f1b3c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:28.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.119+0000 7f1b42a00700 1 -- 172.21.15.27:0/2943956886 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c00cb70 con 0x7f1b4c0758d0 2024-05-10T11:10:28.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.119+0000 7f1b4b200700 1 -- 172.21.15.27:0/2943956886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b4c19c740 con 0x7f1b4c0758d0 2024-05-10T11:10:28.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.119+0000 7f1b42a00700 1 -- 172.21.15.27:0/2943956886 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b3c00ee90 con 0x7f1b4c0758d0 2024-05-10T11:10:28.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.119+0000 7f1b42a00700 1 -- 172.21.15.27:0/2943956886 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c003d40 con 0x7f1b4c0758d0 2024-05-10T11:10:28.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.119+0000 7f1b4b200700 1 -- 172.21.15.27:0/2943956886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b4c1a89d0 con 0x7f1b4c0758d0 2024-05-10T11:10:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.120+0000 7f1b4b200700 1 -- 172.21.15.27:0/2943956886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1b4c063bf0 con 0x7f1b4c0758d0 2024-05-10T11:10:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.123+0000 7f1b42a00700 1 -- 172.21.15.27:0/2943956886 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f1b3c003ee0 con 0x7f1b4c0758d0 2024-05-10T11:10:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.125+0000 7f1b42a00700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f1b300b2990 0x7f1b300b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.125+0000 7f1b42a00700 1 -- 172.21.15.27:0/2943956886 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f1b3c0e2e70 con 0x7f1b4c0758d0 2024-05-10T11:10:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.125+0000 7f1b49400700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f1b300b2990 0x7f1b300b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.125+0000 7f1b42a00700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386] conn(0x7f1b300bdd70 0x7f1b300c0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.125+0000 7f1b42a00700 1 -- 172.21.15.27:0/2943956886 --> [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b300c08e0 con 0x7f1b300bdd70 2024-05-10T11:10:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.125+0000 7f1b42a00700 1 -- 172.21.15.27:0/2943956886 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f1b3c0e3220 con 0x7f1b4c0758d0 2024-05-10T11:10:28.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.126+0000 7f1b4a800700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386] conn(0x7f1b300bdd70 0x7f1b300c0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:28.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.126+0000 7f1b49400700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f1b300b2990 0x7f1b300b4e50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c19d270 tx=0x7f1b3800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:28.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.127+0000 7f1b4a800700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386] conn(0x7f1b300bdd70 0x7f1b300c0210 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:28.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.128+0000 7f1b42a00700 1 -- 172.21.15.27:0/2943956886 <== osd.8 v2:172.21.15.189:6816/2550733386 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1b300c08e0 con 0x7f1b300bdd70 2024-05-10T11:10:28.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.147+0000 7f1b4b200700 1 -- 172.21.15.27:0/2943956886 --> [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1b4c063ff0 con 0x7f1b300bdd70 2024-05-10T11:10:28.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.147+0000 7f1b42a00700 1 -- 172.21.15.27:0/2943956886 <== osd.8 v2:172.21.15.189:6816/2550733386 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1b4c063ff0 con 0x7f1b300bdd70 2024-05-10T11:10:28.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.148+0000 7f1b4b200700 1 -- 172.21.15.27:0/2943956886 >> [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386] conn(0x7f1b300bdd70 msgr2=0x7f1b300c0210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:28.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.148+0000 7f1b4b200700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386] conn(0x7f1b300bdd70 0x7f1b300c0210 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:28.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.148+0000 7f1b4b200700 1 -- 172.21.15.27:0/2943956886 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f1b300b2990 msgr2=0x7f1b300b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:28.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.148+0000 7f1b4b200700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f1b300b2990 0x7f1b300b4e50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c19d270 tx=0x7f1b3800b280 comp rx=0 tx=0).stop 2024-05-10T11:10:28.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.148+0000 7f1b4b200700 1 -- 172.21.15.27:0/2943956886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b4c0758d0 msgr2=0x7f1b4c1a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:28.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.148+0000 7f1b4b200700 1 --2- 172.21.15.27:0/2943956886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b4c0758d0 0x7f1b4c1a1a50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00e9f0 tx=0x7f1b3c00ed00 comp rx=0 tx=0).stop 2024-05-10T11:10:28.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.148+0000 7f1b4a800700 1 -- 172.21.15.27:0/2943956886 reap_dead start 2024-05-10T11:10:28.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.148+0000 7f1b4b200700 1 -- 172.21.15.27:0/2943956886 shutdown_connections 2024-05-10T11:10:28.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.148+0000 7f1b4b200700 1 -- 172.21.15.27:0/2943956886 >> 172.21.15.27:0/2943956886 conn(0x7f1b4c0fef20 msgr2=0x7f1b4c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:28.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.149+0000 7f1b4b200700 1 -- 172.21.15.27:0/2943956886 shutdown_connections 2024-05-10T11:10:28.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:28.149+0000 7f1b4b200700 1 -- 172.21.15.27:0/2943956886 wait complete. 2024-05-10T11:10:28.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:28 smithi076 ceph-mon[34152]: pgmap v413: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-10T11:10:28.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:28 smithi189 ceph-mon[34842]: pgmap v413: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-10T11:10:30.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:30 smithi027 ceph-mon[30054]: pgmap v414: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-10T11:10:30.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:30 smithi076 ceph-mon[34152]: pgmap v414: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-10T11:10:30.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:30 smithi189 ceph-mon[34842]: pgmap v414: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-10T11:10:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.510+0000 7f782b400700 1 -- 172.21.15.27:0/1693877539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78241056c0 msgr2=0x7f7824105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.510+0000 7f782b400700 1 --2- 172.21.15.27:0/1693877539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78241056c0 0x7f7824105b40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7810009990 tx=0x7f7810009ca0 comp rx=0 tx=0).stop 2024-05-10T11:10:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.510+0000 7f782b400700 1 -- 172.21.15.27:0/1693877539 shutdown_connections 2024-05-10T11:10:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.510+0000 7f782b400700 1 --2- 172.21.15.27:0/1693877539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f782410b6c0 0x7f782410baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.510+0000 7f782b400700 1 --2- 172.21.15.27:0/1693877539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7824106080 0x7f7824112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.510+0000 7f782b400700 1 --2- 172.21.15.27:0/1693877539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78241056c0 0x7f7824105b40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.510+0000 7f782b400700 1 -- 172.21.15.27:0/1693877539 >> 172.21.15.27:0/1693877539 conn(0x7f78241011d0 msgr2=0x7f78241035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.511+0000 7f782b400700 1 -- 172.21.15.27:0/1693877539 shutdown_connections 2024-05-10T11:10:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.511+0000 7f782b400700 1 -- 172.21.15.27:0/1693877539 wait complete. 2024-05-10T11:10:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.512+0000 7f782b400700 1 Processor -- start 2024-05-10T11:10:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.512+0000 7f782b400700 1 -- start start 2024-05-10T11:10:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.512+0000 7f782b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78241056c0 0x7f782419be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.512+0000 7f782b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7824106080 0x7f782419c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.512+0000 7f782b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f782410b6c0 0x7f78241a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.512+0000 7f782b400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7824115910 con 0x7f78241056c0 2024-05-10T11:10:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.512+0000 7f782b400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7824115790 con 0x7f7824106080 2024-05-10T11:10:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.512+0000 7f782b400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7824115a90 con 0x7f782410b6c0 2024-05-10T11:10:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.513+0000 7f7828a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78241056c0 0x7f782419be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.513+0000 7f7828a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78241056c0 0x7f782419be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53894/0 (socket says 172.21.15.27:53894) 2024-05-10T11:10:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.513+0000 7f7828a00700 1 -- 172.21.15.27:0/174790197 learned_addr learned my addr 172.21.15.27:0/174790197 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.513+0000 7f7823e00700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7824106080 0x7f782419c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.513+0000 7f7829400700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f782410b6c0 0x7f78241a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.513+0000 7f7828a00700 1 -- 172.21.15.27:0/174790197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f782410b6c0 msgr2=0x7f78241a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.513+0000 7f7828a00700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f782410b6c0 0x7f78241a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.513+0000 7f7828a00700 1 -- 172.21.15.27:0/174790197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7824106080 msgr2=0x7f782419c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.513+0000 7f7828a00700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7824106080 0x7f782419c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.513+0000 7f7828a00700 1 -- 172.21.15.27:0/174790197 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78241a0e20 con 0x7f78241056c0 2024-05-10T11:10:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.514+0000 7f7828a00700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78241056c0 0x7f782419be60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f781800e9c0 tx=0x7f781800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.514+0000 7f7821600700 1 -- 172.21.15.27:0/174790197 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781800cbd0 con 0x7f78241056c0 2024-05-10T11:10:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.514+0000 7f7821600700 1 -- 172.21.15.27:0/174790197 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f781800ee50 con 0x7f78241056c0 2024-05-10T11:10:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.514+0000 7f782b400700 1 -- 172.21.15.27:0/174790197 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78241a1110 con 0x7f78241056c0 2024-05-10T11:10:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.514+0000 7f7821600700 1 -- 172.21.15.27:0/174790197 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818005550 con 0x7f78241056c0 2024-05-10T11:10:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.514+0000 7f782b400700 1 -- 172.21.15.27:0/174790197 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78241a8990 con 0x7f78241056c0 2024-05-10T11:10:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.515+0000 7f782b400700 1 -- 172.21.15.27:0/174790197 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7824063c30 con 0x7f78241056c0 2024-05-10T11:10:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.518+0000 7f7821600700 1 -- 172.21.15.27:0/174790197 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f7818004750 con 0x7f78241056c0 2024-05-10T11:10:30.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.519+0000 7f7821600700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f780c0b2910 0x7f780c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:30.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.519+0000 7f7821600700 1 -- 172.21.15.27:0/174790197 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f78180e29c0 con 0x7f78241056c0 2024-05-10T11:10:30.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.519+0000 7f7823e00700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f780c0b2910 0x7f780c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:30.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.520+0000 7f7821600700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823] conn(0x7f780c0bdcf0 0x7f780c0c0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:30.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.520+0000 7f7821600700 1 -- 172.21.15.27:0/174790197 --> [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f781801ee50 con 0x7f780c0bdcf0 2024-05-10T11:10:30.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.520+0000 7f7821600700 1 -- 172.21.15.27:0/174790197 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f78180e72a0 con 0x7f78241056c0 2024-05-10T11:10:30.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.520+0000 7f7829400700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823] conn(0x7f780c0bdcf0 0x7f780c0c0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:30.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.520+0000 7f7823e00700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f780c0b2910 0x7f780c0b4dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7824103890 tx=0x7f781000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:30.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.521+0000 7f7829400700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823] conn(0x7f780c0bdcf0 0x7f780c0c0190 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:30.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.522+0000 7f7821600700 1 -- 172.21.15.27:0/174790197 <== osd.4 v2:172.21.15.76:6808/1681276823 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f781801ee50 con 0x7f780c0bdcf0 2024-05-10T11:10:30.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.540+0000 7f782b400700 1 -- 172.21.15.27:0/174790197 --> [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f78240640b0 con 0x7f780c0bdcf0 2024-05-10T11:10:30.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.541+0000 7f7821600700 1 -- 172.21.15.27:0/174790197 <== osd.4 v2:172.21.15.76:6808/1681276823 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f78240640b0 con 0x7f780c0bdcf0 2024-05-10T11:10:30.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.542+0000 7f782b400700 1 -- 172.21.15.27:0/174790197 >> [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823] conn(0x7f780c0bdcf0 msgr2=0x7f780c0c0190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:30.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.542+0000 7f782b400700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823] conn(0x7f780c0bdcf0 0x7f780c0c0190 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:30.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.542+0000 7f782b400700 1 -- 172.21.15.27:0/174790197 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f780c0b2910 msgr2=0x7f780c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:30.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.542+0000 7f782b400700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f780c0b2910 0x7f780c0b4dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7824103890 tx=0x7f781000b260 comp rx=0 tx=0).stop 2024-05-10T11:10:30.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.542+0000 7f782b400700 1 -- 172.21.15.27:0/174790197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78241056c0 msgr2=0x7f782419be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:30.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.542+0000 7f782b400700 1 --2- 172.21.15.27:0/174790197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78241056c0 0x7f782419be60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f781800e9c0 tx=0x7f781800ecd0 comp rx=0 tx=0).stop 2024-05-10T11:10:30.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.542+0000 7f7829400700 1 -- 172.21.15.27:0/174790197 reap_dead start 2024-05-10T11:10:30.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.542+0000 7f782b400700 1 -- 172.21.15.27:0/174790197 shutdown_connections 2024-05-10T11:10:30.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.542+0000 7f782b400700 1 -- 172.21.15.27:0/174790197 >> 172.21.15.27:0/174790197 conn(0x7f78241011d0 msgr2=0x7f7824101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:30.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.542+0000 7f782b400700 1 -- 172.21.15.27:0/174790197 shutdown_connections 2024-05-10T11:10:30.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:30.542+0000 7f782b400700 1 -- 172.21.15.27:0/174790197 wait complete. 2024-05-10T11:10:31.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.937+0000 7f49c7e00700 1 -- 172.21.15.27:0/4229807913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c006c590 msgr2=0x7f49c0110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:31.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.937+0000 7f49c7e00700 1 --2- 172.21.15.27:0/4229807913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c006c590 0x7f49c0110330 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f49b800b0a0 tx=0x7f49b800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:10:31.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.937+0000 7f49c7e00700 1 -- 172.21.15.27:0/4229807913 shutdown_connections 2024-05-10T11:10:31.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.937+0000 7f49c7e00700 1 --2- 172.21.15.27:0/4229807913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c0109510 0x7f49c01098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:31.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.937+0000 7f49c7e00700 1 --2- 172.21.15.27:0/4229807913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c006c590 0x7f49c0110330 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:31.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.937+0000 7f49c7e00700 1 --2- 172.21.15.27:0/4229807913 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49c006bbd0 0x7f49c006c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:31.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.937+0000 7f49c7e00700 1 -- 172.21.15.27:0/4229807913 >> 172.21.15.27:0/4229807913 conn(0x7f49c0079350 msgr2=0x7f49c0079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:31.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.937+0000 7f49c7e00700 1 -- 172.21.15.27:0/4229807913 shutdown_connections 2024-05-10T11:10:31.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.937+0000 7f49c7e00700 1 -- 172.21.15.27:0/4229807913 wait complete. 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.938+0000 7f49c7e00700 1 Processor -- start 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.939+0000 7f49c7e00700 1 -- start start 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.939+0000 7f49c7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49c006bbd0 0x7f49c01024a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.939+0000 7f49c7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c006c590 0x7f49c01029e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.939+0000 7f49c7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c0109510 0x7f49c0102f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.939+0000 7f49c7e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f49c0113670 con 0x7f49c0109510 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.939+0000 7f49c7e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f49c01134f0 con 0x7f49c006bbd0 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.939+0000 7f49c7e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f49c01137f0 con 0x7f49c006c590 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.939+0000 7f49c5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c0109510 0x7f49c0102f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.939+0000 7f49c4a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c006c590 0x7f49c01029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.939+0000 7f49c5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c0109510 0x7f49c0102f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53864/0 (socket says 172.21.15.27:53864) 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.939+0000 7f49c5e00700 1 -- 172.21.15.27:0/1939273914 learned_addr learned my addr 172.21.15.27:0/1939273914 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.940+0000 7f49c5400700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49c006bbd0 0x7f49c01024a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:31.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.940+0000 7f49c5e00700 1 -- 172.21.15.27:0/1939273914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c006c590 msgr2=0x7f49c01029e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:31.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.940+0000 7f49c5e00700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c006c590 0x7f49c01029e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:31.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.940+0000 7f49c5e00700 1 -- 172.21.15.27:0/1939273914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49c006bbd0 msgr2=0x7f49c01024a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:31.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.940+0000 7f49c5e00700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49c006bbd0 0x7f49c01024a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:31.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.940+0000 7f49c5e00700 1 -- 172.21.15.27:0/1939273914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c01a8070 con 0x7f49c0109510 2024-05-10T11:10:31.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.940+0000 7f49c4a00700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c006c590 0x7f49c01029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:10:31.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.940+0000 7f49c5e00700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c0109510 0x7f49c0102f20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f49b800b070 tx=0x7f49b8009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:31.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.940+0000 7f49be000700 1 -- 172.21.15.27:0/1939273914 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b8021070 con 0x7f49c0109510 2024-05-10T11:10:31.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.941+0000 7f49be000700 1 -- 172.21.15.27:0/1939273914 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49b8004e40 con 0x7f49c0109510 2024-05-10T11:10:31.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.941+0000 7f49c7e00700 1 -- 172.21.15.27:0/1939273914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49c01a8300 con 0x7f49c0109510 2024-05-10T11:10:31.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.941+0000 7f49be000700 1 -- 172.21.15.27:0/1939273914 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b8004630 con 0x7f49c0109510 2024-05-10T11:10:31.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.941+0000 7f49c7e00700 1 -- 172.21.15.27:0/1939273914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49c01a8800 con 0x7f49c0109510 2024-05-10T11:10:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.942+0000 7f49c7e00700 1 -- 172.21.15.27:0/1939273914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f49c0063bb0 con 0x7f49c0109510 2024-05-10T11:10:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.944+0000 7f49be000700 1 -- 172.21.15.27:0/1939273914 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f49b8007980 con 0x7f49c0109510 2024-05-10T11:10:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.946+0000 7f49be000700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49a80b2990 0x7f49a80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.946+0000 7f49be000700 1 -- 172.21.15.27:0/1939273914 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f49b800f020 con 0x7f49c0109510 2024-05-10T11:10:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.946+0000 7f49c5400700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49a80b2990 0x7f49a80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.946+0000 7f49be000700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197] conn(0x7f49a80bdd70 0x7f49a80c0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.946+0000 7f49be000700 1 -- 172.21.15.27:0/1939273914 --> [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49b8019ec0 con 0x7f49a80bdd70 2024-05-10T11:10:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.946+0000 7f49be000700 1 -- 172.21.15.27:0/1939273914 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f49b80e4050 con 0x7f49c0109510 2024-05-10T11:10:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.947+0000 7f49c4a00700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197] conn(0x7f49a80bdd70 0x7f49a80c0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.947+0000 7f49c5400700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49a80b2990 0x7f49a80b4e50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f49b4009be0 tx=0x7f49b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.947+0000 7f49c4a00700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197] conn(0x7f49a80bdd70 0x7f49a80c0210 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.948+0000 7f49be000700 1 -- 172.21.15.27:0/1939273914 <== osd.9 v2:172.21.15.27:6826/2571330197 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f49b8019ec0 con 0x7f49a80bdd70 2024-05-10T11:10:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.968+0000 7f49c7e00700 1 -- 172.21.15.27:0/1939273914 --> [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f49c0064030 con 0x7f49a80bdd70 2024-05-10T11:10:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.969+0000 7f49be000700 1 -- 172.21.15.27:0/1939273914 <== osd.9 v2:172.21.15.27:6826/2571330197 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f49c0064030 con 0x7f49a80bdd70 2024-05-10T11:10:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.970+0000 7f49c7e00700 1 -- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197] conn(0x7f49a80bdd70 msgr2=0x7f49a80c0210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.970+0000 7f49c7e00700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197] conn(0x7f49a80bdd70 0x7f49a80c0210 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.970+0000 7f49c7e00700 1 -- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49a80b2990 msgr2=0x7f49a80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.970+0000 7f49c7e00700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49a80b2990 0x7f49a80b4e50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f49b4009be0 tx=0x7f49b4009260 comp rx=0 tx=0).stop 2024-05-10T11:10:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.970+0000 7f49c7e00700 1 -- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c0109510 msgr2=0x7f49c0102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.970+0000 7f49c7e00700 1 --2- 172.21.15.27:0/1939273914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c0109510 0x7f49c0102f20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f49b800b070 tx=0x7f49b8009c90 comp rx=0 tx=0).stop 2024-05-10T11:10:31.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.970+0000 7f49c5e00700 1 -- 172.21.15.27:0/1939273914 reap_dead start 2024-05-10T11:10:31.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.970+0000 7f49c7e00700 1 -- 172.21.15.27:0/1939273914 shutdown_connections 2024-05-10T11:10:31.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.970+0000 7f49c7e00700 1 -- 172.21.15.27:0/1939273914 >> 172.21.15.27:0/1939273914 conn(0x7f49c0079350 msgr2=0x7f49c0100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:31.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.971+0000 7f49c7e00700 1 -- 172.21.15.27:0/1939273914 shutdown_connections 2024-05-10T11:10:31.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:26.971+0000 7f49c7e00700 1 -- 172.21.15.27:0/1939273914 wait complete. 2024-05-10T11:10:32.308 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:32 smithi027 ceph-mon[30054]: pgmap v415: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-10T11:10:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:32 smithi076 ceph-mon[34152]: pgmap v415: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-10T11:10:32.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.470+0000 7f3c59e00700 1 -- 172.21.15.27:0/4281711044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c541056c0 msgr2=0x7f3c54105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:32.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.470+0000 7f3c59e00700 1 --2- 172.21.15.27:0/4281711044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c541056c0 0x7f3c54105b40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3c40009990 tx=0x7f3c40009ca0 comp rx=0 tx=0).stop 2024-05-10T11:10:32.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.470+0000 7f3c59e00700 1 -- 172.21.15.27:0/4281711044 shutdown_connections 2024-05-10T11:10:32.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.470+0000 7f3c59e00700 1 --2- 172.21.15.27:0/4281711044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c5410b6c0 0x7f3c5410baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:32.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.470+0000 7f3c59e00700 1 --2- 172.21.15.27:0/4281711044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c54106080 0x7f3c54112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:32.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.470+0000 7f3c59e00700 1 --2- 172.21.15.27:0/4281711044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c541056c0 0x7f3c54105b40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:32.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.470+0000 7f3c59e00700 1 -- 172.21.15.27:0/4281711044 >> 172.21.15.27:0/4281711044 conn(0x7f3c541011b0 msgr2=0x7f3c541035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.470+0000 7f3c59e00700 1 -- 172.21.15.27:0/4281711044 shutdown_connections 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.471+0000 7f3c59e00700 1 -- 172.21.15.27:0/4281711044 wait complete. 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.472+0000 7f3c59e00700 1 Processor -- start 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.472+0000 7f3c59e00700 1 -- start start 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.473+0000 7f3c59e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c541056c0 0x7f3c5419bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.473+0000 7f3c59e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c54106080 0x7f3c5419c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.473+0000 7f3c59e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c5410b6c0 0x7f3c541a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.473+0000 7f3c59e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3c54115910 con 0x7f3c541056c0 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.473+0000 7f3c59e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3c54115790 con 0x7f3c54106080 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.473+0000 7f3c59e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3c54115a90 con 0x7f3c5410b6c0 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.473+0000 7f3c52a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c541056c0 0x7f3c5419bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.473+0000 7f3c52a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c541056c0 0x7f3c5419bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53900/0 (socket says 172.21.15.27:53900) 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.473+0000 7f3c52a00700 1 -- 172.21.15.27:0/3561418750 learned_addr learned my addr 172.21.15.27:0/3561418750 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:32.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.473+0000 7f3c53400700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c5410b6c0 0x7f3c541a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:32.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.473+0000 7f3c52000700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c54106080 0x7f3c5419c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.474+0000 7f3c53400700 1 -- 172.21.15.27:0/3561418750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c54106080 msgr2=0x7f3c5419c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.474+0000 7f3c53400700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c54106080 0x7f3c5419c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.474+0000 7f3c53400700 1 -- 172.21.15.27:0/3561418750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c541056c0 msgr2=0x7f3c5419bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.474+0000 7f3c53400700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c541056c0 0x7f3c5419bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.474+0000 7f3c53400700 1 -- 172.21.15.27:0/3561418750 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c541a0da0 con 0x7f3c5410b6c0 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.474+0000 7f3c52a00700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c541056c0 0x7f3c5419bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.474+0000 7f3c53400700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c5410b6c0 0x7f3c541a0660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00efc0 tx=0x7f3c4c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.474+0000 7f3c58a00700 1 -- 172.21.15.27:0/3561418750 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c0092e0 con 0x7f3c5410b6c0 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.474+0000 7f3c59e00700 1 -- 172.21.15.27:0/3561418750 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c541a1030 con 0x7f3c5410b6c0 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.474+0000 7f3c58a00700 1 -- 172.21.15.27:0/3561418750 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c4c004bb0 con 0x7f3c5410b6c0 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.475+0000 7f3c58a00700 1 -- 172.21.15.27:0/3561418750 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c004020 con 0x7f3c5410b6c0 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.475+0000 7f3c59e00700 1 -- 172.21.15.27:0/3561418750 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c541a8910 con 0x7f3c5410b6c0 2024-05-10T11:10:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.475+0000 7f3c59e00700 1 -- 172.21.15.27:0/3561418750 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3c54063c30 con 0x7f3c5410b6c0 2024-05-10T11:10:32.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.478+0000 7f3c58a00700 1 -- 172.21.15.27:0/3561418750 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f3c4c01b030 con 0x7f3c5410b6c0 2024-05-10T11:10:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.480+0000 7f3c58a00700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3c380b2910 0x7f3c380b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.480+0000 7f3c58a00700 1 -- 172.21.15.27:0/3561418750 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f3c4c0e2c40 con 0x7f3c5410b6c0 2024-05-10T11:10:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.480+0000 7f3c52a00700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3c380b2910 0x7f3c380b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.480+0000 7f3c58a00700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194] conn(0x7f3c380bdcf0 0x7f3c380c0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.480+0000 7f3c58a00700 1 -- 172.21.15.27:0/3561418750 --> [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c4c00be50 con 0x7f3c380bdcf0 2024-05-10T11:10:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.480+0000 7f3c58a00700 1 -- 172.21.15.27:0/3561418750 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f3c4c0e2ff0 con 0x7f3c5410b6c0 2024-05-10T11:10:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.481+0000 7f3c52000700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194] conn(0x7f3c380bdcf0 0x7f3c380c0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.481+0000 7f3c52a00700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3c380b2910 0x7f3c380b4dd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3c44011550 tx=0x7f3c44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.481+0000 7f3c52000700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194] conn(0x7f3c380bdcf0 0x7f3c380c0190 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.482+0000 7f3c58a00700 1 -- 172.21.15.27:0/3561418750 <== osd.7 v2:172.21.15.76:6816/1478795194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c4c00be50 con 0x7f3c380bdcf0 2024-05-10T11:10:32.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:32 smithi189 ceph-mon[34842]: pgmap v415: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-10T11:10:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.501+0000 7f3c59e00700 1 -- 172.21.15.27:0/3561418750 --> [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3c540640b0 con 0x7f3c380bdcf0 2024-05-10T11:10:32.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.502+0000 7f3c58a00700 1 -- 172.21.15.27:0/3561418750 <== osd.7 v2:172.21.15.76:6816/1478795194 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3c540640b0 con 0x7f3c380bdcf0 2024-05-10T11:10:32.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.502+0000 7f3c59e00700 1 -- 172.21.15.27:0/3561418750 >> [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194] conn(0x7f3c380bdcf0 msgr2=0x7f3c380c0190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:32.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.502+0000 7f3c59e00700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194] conn(0x7f3c380bdcf0 0x7f3c380c0190 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:32.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.503+0000 7f3c59e00700 1 -- 172.21.15.27:0/3561418750 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3c380b2910 msgr2=0x7f3c380b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:32.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.503+0000 7f3c59e00700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f3c380b2910 0x7f3c380b4dd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3c44011550 tx=0x7f3c44009260 comp rx=0 tx=0).stop 2024-05-10T11:10:32.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.503+0000 7f3c59e00700 1 -- 172.21.15.27:0/3561418750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c5410b6c0 msgr2=0x7f3c541a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:32.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.503+0000 7f3c59e00700 1 --2- 172.21.15.27:0/3561418750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c5410b6c0 0x7f3c541a0660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00efc0 tx=0x7f3c4c00c3a0 comp rx=0 tx=0).stop 2024-05-10T11:10:32.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.503+0000 7f3c53400700 1 -- 172.21.15.27:0/3561418750 reap_dead start 2024-05-10T11:10:32.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.503+0000 7f3c59e00700 1 -- 172.21.15.27:0/3561418750 shutdown_connections 2024-05-10T11:10:32.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.503+0000 7f3c59e00700 1 -- 172.21.15.27:0/3561418750 >> 172.21.15.27:0/3561418750 conn(0x7f3c541011b0 msgr2=0x7f3c54101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:32.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.503+0000 7f3c59e00700 1 -- 172.21.15.27:0/3561418750 shutdown_connections 2024-05-10T11:10:32.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:32.503+0000 7f3c59e00700 1 -- 172.21.15.27:0/3561418750 wait complete. 2024-05-10T11:10:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.179+0000 7f654be00700 1 -- 172.21.15.27:0/548505015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f654c109460 msgr2=0x7f654c109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.179+0000 7f654be00700 1 --2- 172.21.15.27:0/548505015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f654c109460 0x7f654c109840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f653c009990 tx=0x7f653c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:10:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.180+0000 7f654be00700 1 -- 172.21.15.27:0/548505015 shutdown_connections 2024-05-10T11:10:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.180+0000 7f654be00700 1 --2- 172.21.15.27:0/548505015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f654c103d90 0x7f654c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.180+0000 7f654be00700 1 --2- 172.21.15.27:0/548505015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f654c109460 0x7f654c109840 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.180+0000 7f654be00700 1 --2- 172.21.15.27:0/548505015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f654c1033d0 0x7f654c103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.180+0000 7f654be00700 1 -- 172.21.15.27:0/548505015 >> 172.21.15.27:0/548505015 conn(0x7f654c0fef20 msgr2=0x7f654c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.180+0000 7f654be00700 1 -- 172.21.15.27:0/548505015 shutdown_connections 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.180+0000 7f654be00700 1 -- 172.21.15.27:0/548505015 wait complete. 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.181+0000 7f654be00700 1 Processor -- start 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.181+0000 7f654be00700 1 -- start start 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.182+0000 7f654be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f654c1033d0 0x7f654c19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.182+0000 7f654be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f654c103d90 0x7f654c19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.182+0000 7f654be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f654c109460 0x7f654c199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.182+0000 7f654be00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f654c113570 con 0x7f654c103d90 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.182+0000 7f654be00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f654c1133f0 con 0x7f654c109460 2024-05-10T11:10:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.182+0000 7f654be00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f654c1136f0 con 0x7f654c1033d0 2024-05-10T11:10:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.182+0000 7f654a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f654c103d90 0x7f654c19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.182+0000 7f654b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f654c109460 0x7f654c199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.182+0000 7f654a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f654c103d90 0x7f654c19fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53924/0 (socket says 172.21.15.27:53924) 2024-05-10T11:10:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.182+0000 7f654a000700 1 -- 172.21.15.27:0/2160780920 learned_addr learned my addr 172.21.15.27:0/2160780920 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.182+0000 7f654aa00700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f654c1033d0 0x7f654c19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.183+0000 7f654b400700 1 -- 172.21.15.27:0/2160780920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f654c1033d0 msgr2=0x7f654c19f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.183+0000 7f654b400700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f654c1033d0 0x7f654c19f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.183+0000 7f654b400700 1 -- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f654c103d90 msgr2=0x7f654c19fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.183+0000 7f654b400700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f654c103d90 0x7f654c19fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.183+0000 7f654b400700 1 -- 172.21.15.27:0/2160780920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654c199ff0 con 0x7f654c109460 2024-05-10T11:10:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.183+0000 7f654a000700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f654c103d90 0x7f654c19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:10:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.183+0000 7f654b400700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f654c109460 0x7f654c199890 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6544007f10 tx=0x7f654400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.184+0000 7f6543400700 1 -- 172.21.15.27:0/2160780920 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6544004db0 con 0x7f654c109460 2024-05-10T11:10:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.184+0000 7f654be00700 1 -- 172.21.15.27:0/2160780920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f654c19a2e0 con 0x7f654c109460 2024-05-10T11:10:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.184+0000 7f654be00700 1 -- 172.21.15.27:0/2160780920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f654c1a65a0 con 0x7f654c109460 2024-05-10T11:10:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.184+0000 7f6543400700 1 -- 172.21.15.27:0/2160780920 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6544003a00 con 0x7f654c109460 2024-05-10T11:10:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.184+0000 7f6543400700 1 -- 172.21.15.27:0/2160780920 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6544019020 con 0x7f654c109460 2024-05-10T11:10:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.185+0000 7f654be00700 1 -- 172.21.15.27:0/2160780920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f652c001030 con 0x7f654c109460 2024-05-10T11:10:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.187+0000 7f6543400700 1 -- 172.21.15.27:0/2160780920 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f654401c020 con 0x7f654c109460 2024-05-10T11:10:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.189+0000 7f6543400700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f65300b2910 0x7f65300b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.189+0000 7f6543400700 1 -- 172.21.15.27:0/2160780920 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f65440e27e0 con 0x7f654c109460 2024-05-10T11:10:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.189+0000 7f654aa00700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f65300b2910 0x7f65300b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.190+0000 7f6543400700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345] conn(0x7f65300bdcf0 0x7f65300c0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.190+0000 7f6543400700 1 -- 172.21.15.27:0/2160780920 --> [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f654401ce50 con 0x7f65300bdcf0 2024-05-10T11:10:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.190+0000 7f6543400700 1 -- 172.21.15.27:0/2160780920 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f65440e2b90 con 0x7f654c109460 2024-05-10T11:10:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.190+0000 7f654a000700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345] conn(0x7f65300bdcf0 0x7f65300c0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.191+0000 7f654aa00700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f65300b2910 0x7f65300b4dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f654c19a9d0 tx=0x7f653c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.191+0000 7f654a000700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345] conn(0x7f65300bdcf0 0x7f65300c0190 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.192+0000 7f6543400700 1 -- 172.21.15.27:0/2160780920 <== osd.0 v2:172.21.15.27:6802/677446345 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f654401ce50 con 0x7f65300bdcf0 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.211+0000 7f654be00700 1 -- 172.21.15.27:0/2160780920 --> [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f652c002e60 con 0x7f65300bdcf0 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.212+0000 7f6543400700 1 -- 172.21.15.27:0/2160780920 <== osd.0 v2:172.21.15.27:6802/677446345 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f652c002e60 con 0x7f65300bdcf0 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.213+0000 7f654be00700 1 -- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345] conn(0x7f65300bdcf0 msgr2=0x7f65300c0190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.213+0000 7f654be00700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345] conn(0x7f65300bdcf0 0x7f65300c0190 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.213+0000 7f654be00700 1 -- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f65300b2910 msgr2=0x7f65300b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.213+0000 7f654be00700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f65300b2910 0x7f65300b4dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f654c19a9d0 tx=0x7f653c005960 comp rx=0 tx=0).stop 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.213+0000 7f654be00700 1 -- 172.21.15.27:0/2160780920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f654c109460 msgr2=0x7f654c199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.213+0000 7f654be00700 1 --2- 172.21.15.27:0/2160780920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f654c109460 0x7f654c199890 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6544007f10 tx=0x7f654400d3b0 comp rx=0 tx=0).stop 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.213+0000 7f654b400700 1 -- 172.21.15.27:0/2160780920 reap_dead start 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.213+0000 7f654be00700 1 -- 172.21.15.27:0/2160780920 shutdown_connections 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.213+0000 7f654be00700 1 -- 172.21.15.27:0/2160780920 >> 172.21.15.27:0/2160780920 conn(0x7f654c0fef20 msgr2=0x7f654c0ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.213+0000 7f654be00700 1 -- 172.21.15.27:0/2160780920 shutdown_connections 2024-05-10T11:10:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:33.213+0000 7f654be00700 1 -- 172.21.15.27:0/2160780920 wait complete. 2024-05-10T11:10:34.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.050+0000 7fd5dc800700 1 -- 172.21.15.27:0/860223855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5d410fb60 msgr2=0x7fd5d410ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.050+0000 7fd5dc800700 1 --2- 172.21.15.27:0/860223855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5d410fb60 0x7fd5d410ff40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd5c80098f0 tx=0x7fd5c8009c00 comp rx=0 tx=0).stop 2024-05-10T11:10:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.050+0000 7fd5dc800700 1 -- 172.21.15.27:0/860223855 shutdown_connections 2024-05-10T11:10:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.050+0000 7fd5dc800700 1 --2- 172.21.15.27:0/860223855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5d410fb60 0x7fd5d410ff40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.050+0000 7fd5dc800700 1 --2- 172.21.15.27:0/860223855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5d410a490 0x7fd5d41169a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.050+0000 7fd5dc800700 1 --2- 172.21.15.27:0/860223855 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5d4109ad0 0x7fd5d4109f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.050+0000 7fd5dc800700 1 -- 172.21.15.27:0/860223855 >> 172.21.15.27:0/860223855 conn(0x7fd5d4076c70 msgr2=0x7fd5d4079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.050+0000 7fd5dc800700 1 -- 172.21.15.27:0/860223855 shutdown_connections 2024-05-10T11:10:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.050+0000 7fd5dc800700 1 -- 172.21.15.27:0/860223855 wait complete. 2024-05-10T11:10:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.051+0000 7fd5dc800700 1 Processor -- start 2024-05-10T11:10:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.051+0000 7fd5dc800700 1 -- start start 2024-05-10T11:10:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.052+0000 7fd5dc800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5d4109ad0 0x7fd5d41a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.052+0000 7fd5dc800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5d410a490 0x7fd5d41a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.052+0000 7fd5dc800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5d410fb60 0x7fd5d41a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.052+0000 7fd5dc800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4119cd0 con 0x7fd5d4109ad0 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.052+0000 7fd5dc800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4119b50 con 0x7fd5d410fb60 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.052+0000 7fd5dc800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4119e50 con 0x7fd5d410a490 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.052+0000 7fd5d9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5d4109ad0 0x7fd5d41a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.052+0000 7fd5d9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5d4109ad0 0x7fd5d41a01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60948/0 (socket says 172.21.15.27:60948) 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.052+0000 7fd5d9e00700 1 -- 172.21.15.27:0/2923848862 learned_addr learned my addr 172.21.15.27:0/2923848862 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.052+0000 7fd5d9400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5d410a490 0x7fd5d41a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.052+0000 7fd5da800700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5d410fb60 0x7fd5d41a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.053+0000 7fd5d9e00700 1 -- 172.21.15.27:0/2923848862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5d410a490 msgr2=0x7fd5d41a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.053+0000 7fd5d9e00700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5d410a490 0x7fd5d41a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.053+0000 7fd5d9e00700 1 -- 172.21.15.27:0/2923848862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5d410fb60 msgr2=0x7fd5d41a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.053+0000 7fd5d9e00700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5d410fb60 0x7fd5d41a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.053+0000 7fd5d9e00700 1 -- 172.21.15.27:0/2923848862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d41a51f0 con 0x7fd5d4109ad0 2024-05-10T11:10:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.053+0000 7fd5d9400700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5d410a490 0x7fd5d41a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:10:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.053+0000 7fd5d9e00700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5d4109ad0 0x7fd5d41a01e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800f990 tx=0x7fd5c800f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.054+0000 7fd5cea00700 1 -- 172.21.15.27:0/2923848862 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8027070 con 0x7fd5d4109ad0 2024-05-10T11:10:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.054+0000 7fd5dc800700 1 -- 172.21.15.27:0/2923848862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d41a5480 con 0x7fd5d4109ad0 2024-05-10T11:10:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.054+0000 7fd5dc800700 1 -- 172.21.15.27:0/2923848862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d41acd60 con 0x7fd5d4109ad0 2024-05-10T11:10:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.055+0000 7fd5dc800700 1 -- 172.21.15.27:0/2923848862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd5d4063bf0 con 0x7fd5d4109ad0 2024-05-10T11:10:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.057+0000 7fd5cea00700 1 -- 172.21.15.27:0/2923848862 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5d4063bf0 con 0x7fd5d4109ad0 2024-05-10T11:10:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.058+0000 7fd5cea00700 1 -- 172.21.15.27:0/2923848862 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8004d80 con 0x7fd5d4109ad0 2024-05-10T11:10:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.061+0000 7fd5cea00700 1 -- 172.21.15.27:0/2923848862 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd5c802a070 con 0x7fd5d4109ad0 2024-05-10T11:10:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.062+0000 7fd5cea00700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd5bc0b2990 0x7fd5bc0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.063+0000 7fd5cea00700 1 -- 172.21.15.27:0/2923848862 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fd5c80e32e0 con 0x7fd5d4109ad0 2024-05-10T11:10:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.063+0000 7fd5d9400700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd5bc0b2990 0x7fd5bc0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.063+0000 7fd5cea00700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234] conn(0x7fd5bc0bdc40 0x7fd5bc0c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:34.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.063+0000 7fd5cea00700 1 -- 172.21.15.27:0/2923848862 --> [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5c8015e50 con 0x7fd5bc0bdc40 2024-05-10T11:10:34.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.063+0000 7fd5cea00700 1 -- 172.21.15.27:0/2923848862 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fd5c80e3690 con 0x7fd5d4109ad0 2024-05-10T11:10:34.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.063+0000 7fd5d9400700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd5bc0b2990 0x7fd5bc0b4e50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd5d40793e0 tx=0x7fd5c4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:34.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.063+0000 7fd5da800700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234] conn(0x7fd5bc0bdc40 0x7fd5bc0c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:34.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.064+0000 7fd5da800700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234] conn(0x7fd5bc0bdc40 0x7fd5bc0c00e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:34.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.065+0000 7fd5cea00700 1 -- 172.21.15.27:0/2923848862 <== osd.10 v2:172.21.15.76:6824/2809936234 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd5c8015e50 con 0x7fd5bc0bdc40 2024-05-10T11:10:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.084+0000 7fd5dc800700 1 -- 172.21.15.27:0/2923848862 --> [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd5d4064020 con 0x7fd5bc0bdc40 2024-05-10T11:10:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.084+0000 7fd5cea00700 1 -- 172.21.15.27:0/2923848862 <== osd.10 v2:172.21.15.76:6824/2809936234 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd5d4064020 con 0x7fd5bc0bdc40 2024-05-10T11:10:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.085+0000 7fd5dc800700 1 -- 172.21.15.27:0/2923848862 >> [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234] conn(0x7fd5bc0bdc40 msgr2=0x7fd5bc0c00e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.085+0000 7fd5dc800700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234] conn(0x7fd5bc0bdc40 0x7fd5bc0c00e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.085+0000 7fd5dc800700 1 -- 172.21.15.27:0/2923848862 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd5bc0b2990 msgr2=0x7fd5bc0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.085+0000 7fd5dc800700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd5bc0b2990 0x7fd5bc0b4e50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd5d40793e0 tx=0x7fd5c4009310 comp rx=0 tx=0).stop 2024-05-10T11:10:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.085+0000 7fd5dc800700 1 -- 172.21.15.27:0/2923848862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5d4109ad0 msgr2=0x7fd5d41a01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.085+0000 7fd5dc800700 1 --2- 172.21.15.27:0/2923848862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5d4109ad0 0x7fd5d41a01e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800f990 tx=0x7fd5c800f9c0 comp rx=0 tx=0).stop 2024-05-10T11:10:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.085+0000 7fd5da800700 1 -- 172.21.15.27:0/2923848862 reap_dead start 2024-05-10T11:10:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.085+0000 7fd5dc800700 1 -- 172.21.15.27:0/2923848862 shutdown_connections 2024-05-10T11:10:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.085+0000 7fd5dc800700 1 -- 172.21.15.27:0/2923848862 >> 172.21.15.27:0/2923848862 conn(0x7fd5d4076c70 msgr2=0x7fd5d4077510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.086+0000 7fd5dc800700 1 -- 172.21.15.27:0/2923848862 shutdown_connections 2024-05-10T11:10:34.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:34.086+0000 7fd5dc800700 1 -- 172.21.15.27:0/2923848862 wait complete. 2024-05-10T11:10:34.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:34 smithi027 ceph-mon[30054]: pgmap v416: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-05-10T11:10:34.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:34 smithi076 ceph-mon[34152]: pgmap v416: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-05-10T11:10:34.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:34 smithi189 ceph-mon[34842]: pgmap v416: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-05-10T11:10:35.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.468+0000 7f0b61e00700 1 -- 172.21.15.27:0/314315928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c103750 msgr2=0x7f0b5c103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:35.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.468+0000 7f0b61e00700 1 --2- 172.21.15.27:0/314315928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c103750 0x7f0b5c103b30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009990 tx=0x7f0b4c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:10:35.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.468+0000 7f0b61e00700 1 -- 172.21.15.27:0/314315928 shutdown_connections 2024-05-10T11:10:35.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.468+0000 7f0b61e00700 1 --2- 172.21.15.27:0/314315928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c103750 0x7f0b5c103b30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:35.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.468+0000 7f0b61e00700 1 --2- 172.21.15.27:0/314315928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b5c108850 0x7f0b5c1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:35.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.468+0000 7f0b61e00700 1 --2- 172.21.15.27:0/314315928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b5c104100 0x7f0b5c108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:35.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.468+0000 7f0b61e00700 1 -- 172.21.15.27:0/314315928 >> 172.21.15.27:0/314315928 conn(0x7f0b5c07a600 msgr2=0x7f0b5c07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:35.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.468+0000 7f0b61e00700 1 -- 172.21.15.27:0/314315928 shutdown_connections 2024-05-10T11:10:35.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.468+0000 7f0b61e00700 1 -- 172.21.15.27:0/314315928 wait complete. 2024-05-10T11:10:35.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.469+0000 7f0b61e00700 1 Processor -- start 2024-05-10T11:10:35.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.469+0000 7f0b61e00700 1 -- start start 2024-05-10T11:10:35.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.470+0000 7f0b61e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b5c103750 0x7f0b5c19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:35.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.470+0000 7f0b61e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c104100 0x7f0b5c19c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:35.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.470+0000 7f0b61e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b5c108850 0x7f0b5c1a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:35.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.470+0000 7f0b61e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5c113670 con 0x7f0b5c108850 2024-05-10T11:10:35.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.470+0000 7f0b61e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5c1134f0 con 0x7f0b5c103750 2024-05-10T11:10:35.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.470+0000 7f0b61e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5c1137f0 con 0x7f0b5c104100 2024-05-10T11:10:35.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.470+0000 7f0b5b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b5c108850 0x7f0b5c1a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:35.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.470+0000 7f0b5aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b5c103750 0x7f0b5c19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:35.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.470+0000 7f0b5aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b5c103750 0x7f0b5c19bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:38284/0 (socket says 172.21.15.27:38284) 2024-05-10T11:10:35.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.470+0000 7f0b5aa00700 1 -- 172.21.15.27:0/2774260140 learned_addr learned my addr 172.21.15.27:0/2774260140 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:35.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.470+0000 7f0b5a000700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c104100 0x7f0b5c19c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:35.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.471+0000 7f0b5aa00700 1 -- 172.21.15.27:0/2774260140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c104100 msgr2=0x7f0b5c19c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.471+0000 7f0b5aa00700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c104100 0x7f0b5c19c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.471+0000 7f0b5aa00700 1 -- 172.21.15.27:0/2774260140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b5c108850 msgr2=0x7f0b5c1a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.471+0000 7f0b5aa00700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b5c108850 0x7f0b5c1a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.471+0000 7f0b5aa00700 1 -- 172.21.15.27:0/2774260140 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b5c1a0d40 con 0x7f0b5c103750 2024-05-10T11:10:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.471+0000 7f0b5aa00700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b5c103750 0x7f0b5c19bd30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c005910 tx=0x7f0b4c00bfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.471+0000 7f0b60a00700 1 -- 172.21.15.27:0/2774260140 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c027070 con 0x7f0b5c103750 2024-05-10T11:10:35.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.471+0000 7f0b61e00700 1 -- 172.21.15.27:0/2774260140 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b5c1a0fd0 con 0x7f0b5c103750 2024-05-10T11:10:35.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.471+0000 7f0b60a00700 1 -- 172.21.15.27:0/2774260140 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b4c00b610 con 0x7f0b5c103750 2024-05-10T11:10:35.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.471+0000 7f0b60a00700 1 -- 172.21.15.27:0/2774260140 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c0173f0 con 0x7f0b5c103750 2024-05-10T11:10:35.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.472+0000 7f0b61e00700 1 -- 172.21.15.27:0/2774260140 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b5c1a88b0 con 0x7f0b5c103750 2024-05-10T11:10:35.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.472+0000 7f0b61e00700 1 -- 172.21.15.27:0/2774260140 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0b5c101f80 con 0x7f0b5c103750 2024-05-10T11:10:35.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.475+0000 7f0b60a00700 1 -- 172.21.15.27:0/2774260140 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0b4c02a070 con 0x7f0b5c103750 2024-05-10T11:10:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.477+0000 7f0b60a00700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0b400b2910 0x7f0b400b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.477+0000 7f0b60a00700 1 -- 172.21.15.27:0/2774260140 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f0b4c0e3d10 con 0x7f0b5c103750 2024-05-10T11:10:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.477+0000 7f0b5a000700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0b400b2910 0x7f0b400b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.477+0000 7f0b60a00700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096] conn(0x7f0b400bdcf0 0x7f0b400c0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.477+0000 7f0b60a00700 1 -- 172.21.15.27:0/2774260140 --> [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0b4c007e80 con 0x7f0b400bdcf0 2024-05-10T11:10:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.477+0000 7f0b60a00700 1 -- 172.21.15.27:0/2774260140 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f0b4c0e40c0 con 0x7f0b5c103750 2024-05-10T11:10:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.478+0000 7f0b5b400700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096] conn(0x7f0b400bdcf0 0x7f0b400c0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.478+0000 7f0b5a000700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0b400b2910 0x7f0b400b4dd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009680 tx=0x7f0b48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.478+0000 7f0b5b400700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096] conn(0x7f0b400bdcf0 0x7f0b400c0190 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.479+0000 7f0b60a00700 1 -- 172.21.15.27:0/2774260140 <== osd.11 v2:172.21.15.189:6824/49546096 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0b4c007e80 con 0x7f0b400bdcf0 2024-05-10T11:10:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.498+0000 7f0b61e00700 1 -- 172.21.15.27:0/2774260140 --> [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0b5c104be0 con 0x7f0b400bdcf0 2024-05-10T11:10:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.499+0000 7f0b60a00700 1 -- 172.21.15.27:0/2774260140 <== osd.11 v2:172.21.15.189:6824/49546096 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0b5c104be0 con 0x7f0b400bdcf0 2024-05-10T11:10:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.499+0000 7f0b61e00700 1 -- 172.21.15.27:0/2774260140 >> [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096] conn(0x7f0b400bdcf0 msgr2=0x7f0b400c0190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.499+0000 7f0b61e00700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096] conn(0x7f0b400bdcf0 0x7f0b400c0190 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.499+0000 7f0b61e00700 1 -- 172.21.15.27:0/2774260140 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0b400b2910 msgr2=0x7f0b400b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.499+0000 7f0b61e00700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f0b400b2910 0x7f0b400b4dd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009680 tx=0x7f0b48009260 comp rx=0 tx=0).stop 2024-05-10T11:10:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.499+0000 7f0b61e00700 1 -- 172.21.15.27:0/2774260140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b5c103750 msgr2=0x7f0b5c19bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.499+0000 7f0b61e00700 1 --2- 172.21.15.27:0/2774260140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b5c103750 0x7f0b5c19bd30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c005910 tx=0x7f0b4c00bfc0 comp rx=0 tx=0).stop 2024-05-10T11:10:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.500+0000 7f0b5b400700 1 -- 172.21.15.27:0/2774260140 reap_dead start 2024-05-10T11:10:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.500+0000 7f0b61e00700 1 -- 172.21.15.27:0/2774260140 shutdown_connections 2024-05-10T11:10:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.500+0000 7f0b61e00700 1 -- 172.21.15.27:0/2774260140 >> 172.21.15.27:0/2774260140 conn(0x7f0b5c07a600 msgr2=0x7f0b5c101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.500+0000 7f0b61e00700 1 -- 172.21.15.27:0/2774260140 shutdown_connections 2024-05-10T11:10:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:35.500+0000 7f0b61e00700 1 -- 172.21.15.27:0/2774260140 wait complete. 2024-05-10T11:10:36.240 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:36 smithi027 ceph-mon[30054]: pgmap v417: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-05-10T11:10:36.388 INFO:teuthology.orchestra.run.smithi027.stdout:201863462986 2024-05-10T11:10:36.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-10T11:10:36.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:36 smithi076 ceph-mon[34152]: pgmap v417: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-05-10T11:10:36.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:36 smithi189 ceph-mon[34842]: pgmap v417: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-05-10T11:10:37.847 INFO:teuthology.orchestra.run.smithi027.stdout:133143986267 2024-05-10T11:10:37.847 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-10T11:10:37.962 INFO:teuthology.orchestra.run.smithi027.stdout:279172874297 2024-05-10T11:10:37.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-10T11:10:38.312 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:38 smithi027 ceph-mon[30054]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-10T11:10:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:38 smithi076 ceph-mon[34152]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-10T11:10:38.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:38 smithi189 ceph-mon[34842]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-10T11:10:38.908 INFO:teuthology.orchestra.run.smithi027.stdout:253403070524 2024-05-10T11:10:38.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-10T11:10:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.396+0000 7fb52fa00700 1 -- 172.21.15.27:0/2338306160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb528104100 msgr2=0x7fb528108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.396+0000 7fb52fa00700 1 --2- 172.21.15.27:0/2338306160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb528104100 0x7fb528108120 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb5140099c0 tx=0x7fb514009cd0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.397+0000 7fb52fa00700 1 -- 172.21.15.27:0/2338306160 shutdown_connections 2024-05-10T11:10:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.397+0000 7fb52fa00700 1 --2- 172.21.15.27:0/2338306160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb528103750 0x7fb528103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.397+0000 7fb52fa00700 1 --2- 172.21.15.27:0/2338306160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb528108850 0x7fb5281103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.397+0000 7fb52fa00700 1 --2- 172.21.15.27:0/2338306160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb528104100 0x7fb528108120 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.397+0000 7fb52fa00700 1 -- 172.21.15.27:0/2338306160 >> 172.21.15.27:0/2338306160 conn(0x7fb52807a600 msgr2=0x7fb52807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.397+0000 7fb52fa00700 1 -- 172.21.15.27:0/2338306160 shutdown_connections 2024-05-10T11:10:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.397+0000 7fb52fa00700 1 -- 172.21.15.27:0/2338306160 wait complete. 2024-05-10T11:10:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.398+0000 7fb52fa00700 1 Processor -- start 2024-05-10T11:10:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.398+0000 7fb52fa00700 1 -- start start 2024-05-10T11:10:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.399+0000 7fb52fa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb528103750 0x7fb52819bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.399+0000 7fb52fa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb528104100 0x7fb52819c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.399+0000 7fb52fa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb528108850 0x7fb5281a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.399+0000 7fb52fa00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb528113660 con 0x7fb528108850 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.399+0000 7fb52fa00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb5281134e0 con 0x7fb528103750 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.399+0000 7fb52fa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb5281137e0 con 0x7fb528104100 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.399+0000 7fb52da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb528108850 0x7fb5281a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.399+0000 7fb52d000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb528103750 0x7fb52819bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.399+0000 7fb52da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb528108850 0x7fb5281a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60978/0 (socket says 172.21.15.27:60978) 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.399+0000 7fb52d000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb528103750 0x7fb52819bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:38308/0 (socket says 172.21.15.27:38308) 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.399+0000 7fb52da00700 1 -- 172.21.15.27:0/178995103 learned_addr learned my addr 172.21.15.27:0/178995103 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.399+0000 7fb527e00700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb528104100 0x7fb52819c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.400+0000 7fb52da00700 1 -- 172.21.15.27:0/178995103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb528104100 msgr2=0x7fb52819c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.400+0000 7fb52da00700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb528104100 0x7fb52819c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.400+0000 7fb52da00700 1 -- 172.21.15.27:0/178995103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb528103750 msgr2=0x7fb52819bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.400+0000 7fb52da00700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb528103750 0x7fb52819bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.400+0000 7fb52da00700 1 -- 172.21.15.27:0/178995103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5281a0d30 con 0x7fb528108850 2024-05-10T11:10:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.400+0000 7fb52da00700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb528108850 0x7fb5281a05f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb52000c370 tx=0x7fb52000c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.400+0000 7fb525600700 1 -- 172.21.15.27:0/178995103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52000cb60 con 0x7fb528108850 2024-05-10T11:10:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.400+0000 7fb525600700 1 -- 172.21.15.27:0/178995103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb520018070 con 0x7fb528108850 2024-05-10T11:10:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.400+0000 7fb52fa00700 1 -- 172.21.15.27:0/178995103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5281a0fc0 con 0x7fb528108850 2024-05-10T11:10:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.400+0000 7fb525600700 1 -- 172.21.15.27:0/178995103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520013660 con 0x7fb528108850 2024-05-10T11:10:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.401+0000 7fb52fa00700 1 -- 172.21.15.27:0/178995103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5281a88a0 con 0x7fb528108850 2024-05-10T11:10:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.402+0000 7fb52fa00700 1 -- 172.21.15.27:0/178995103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb528102020 con 0x7fb528108850 2024-05-10T11:10:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.404+0000 7fb525600700 1 -- 172.21.15.27:0/178995103 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fb520012070 con 0x7fb528108850 2024-05-10T11:10:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.406+0000 7fb525600700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb5100b2990 0x7fb5100b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.406+0000 7fb525600700 1 -- 172.21.15.27:0/178995103 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fb5200e2400 con 0x7fb528108850 2024-05-10T11:10:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.406+0000 7fb52d000700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb5100b2990 0x7fb5100b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.406+0000 7fb525600700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219] conn(0x7fb5100bdd70 0x7fb5100c0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.406+0000 7fb525600700 1 -- 172.21.15.27:0/178995103 --> [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb520009c20 con 0x7fb5100bdd70 2024-05-10T11:10:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.406+0000 7fb525600700 1 -- 172.21.15.27:0/178995103 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb5200e9050 con 0x7fb528108850 2024-05-10T11:10:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.407+0000 7fb527e00700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219] conn(0x7fb5100bdd70 0x7fb5100c0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.407+0000 7fb52d000700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb5100b2990 0x7fb5100b4e50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb51c005950 tx=0x7fb51c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.407+0000 7fb527e00700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219] conn(0x7fb5100bdd70 0x7fb5100c0210 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.408+0000 7fb525600700 1 -- 172.21.15.27:0/178995103 <== osd.3 v2:172.21.15.27:6810/2407454219 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb520009c20 con 0x7fb5100bdd70 2024-05-10T11:10:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.427+0000 7fb52fa00700 1 -- 172.21.15.27:0/178995103 --> [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb528104be0 con 0x7fb5100bdd70 2024-05-10T11:10:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.427+0000 7fb525600700 1 -- 172.21.15.27:0/178995103 <== osd.3 v2:172.21.15.27:6810/2407454219 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb528104be0 con 0x7fb5100bdd70 2024-05-10T11:10:39.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.428+0000 7fb52fa00700 1 -- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219] conn(0x7fb5100bdd70 msgr2=0x7fb5100c0210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:39.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.428+0000 7fb52fa00700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219] conn(0x7fb5100bdd70 0x7fb5100c0210 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.428+0000 7fb52fa00700 1 -- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb5100b2990 msgr2=0x7fb5100b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:39.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.428+0000 7fb52fa00700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb5100b2990 0x7fb5100b4e50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb51c005950 tx=0x7fb51c00b410 comp rx=0 tx=0).stop 2024-05-10T11:10:39.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.428+0000 7fb52fa00700 1 -- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb528108850 msgr2=0x7fb5281a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:39.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.428+0000 7fb52fa00700 1 --2- 172.21.15.27:0/178995103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb528108850 0x7fb5281a05f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb52000c370 tx=0x7fb52000c680 comp rx=0 tx=0).stop 2024-05-10T11:10:39.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.428+0000 7fb52da00700 1 -- 172.21.15.27:0/178995103 reap_dead start 2024-05-10T11:10:39.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.428+0000 7fb52fa00700 1 -- 172.21.15.27:0/178995103 shutdown_connections 2024-05-10T11:10:39.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.428+0000 7fb52fa00700 1 -- 172.21.15.27:0/178995103 >> 172.21.15.27:0/178995103 conn(0x7fb52807a600 msgr2=0x7fb5281015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:39.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.429+0000 7fb52fa00700 1 -- 172.21.15.27:0/178995103 shutdown_connections 2024-05-10T11:10:39.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.429+0000 7fb52fa00700 1 -- 172.21.15.27:0/178995103 wait complete. 2024-05-10T11:10:39.366 INFO:teuthology.orchestra.run.smithi027.stdout:227633266756 2024-05-10T11:10:39.366 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-10T11:10:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.538+0000 7fe343e00700 1 -- 172.21.15.27:0/3225849801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe33c06b180 msgr2=0x7fe33c06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.538+0000 7fe343e00700 1 --2- 172.21.15.27:0/3225849801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe33c06b180 0x7fe33c06b600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe328009990 tx=0x7fe328009ca0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.538+0000 7fe343e00700 1 -- 172.21.15.27:0/3225849801 shutdown_connections 2024-05-10T11:10:39.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.538+0000 7fe343e00700 1 --2- 172.21.15.27:0/3225849801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe33c109510 0x7fe33c1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.538+0000 7fe343e00700 1 --2- 172.21.15.27:0/3225849801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe33c06bb40 0x7fe33c110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.538+0000 7fe343e00700 1 --2- 172.21.15.27:0/3225849801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe33c06b180 0x7fe33c06b600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.538+0000 7fe343e00700 1 -- 172.21.15.27:0/3225849801 >> 172.21.15.27:0/3225849801 conn(0x7fe33c079350 msgr2=0x7fe33c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:39.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.538+0000 7fe343e00700 1 -- 172.21.15.27:0/3225849801 shutdown_connections 2024-05-10T11:10:39.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.539+0000 7fe343e00700 1 -- 172.21.15.27:0/3225849801 wait complete. 2024-05-10T11:10:39.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.540+0000 7fe343e00700 1 Processor -- start 2024-05-10T11:10:39.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.540+0000 7fe343e00700 1 -- start start 2024-05-10T11:10:39.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.540+0000 7fe343e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe33c06b180 0x7fe33c19be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:39.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.540+0000 7fe343e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe33c06bb40 0x7fe33c19c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:39.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.540+0000 7fe343e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe33c109510 0x7fe33c1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:39.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.540+0000 7fe343e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe33c113670 con 0x7fe33c109510 2024-05-10T11:10:39.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.540+0000 7fe343e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe33c1134f0 con 0x7fe33c06bb40 2024-05-10T11:10:39.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.540+0000 7fe343e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe33c1137f0 con 0x7fe33c06b180 2024-05-10T11:10:39.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.540+0000 7fe341e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe33c109510 0x7fe33c1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:39.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.541+0000 7fe341400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe33c06b180 0x7fe33c19be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:39.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.541+0000 7fe341e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe33c109510 0x7fe33c1a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32768/0 (socket says 172.21.15.27:32768) 2024-05-10T11:10:39.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.541+0000 7fe341e00700 1 -- 172.21.15.27:0/701420802 learned_addr learned my addr 172.21.15.27:0/701420802 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:39.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.541+0000 7fe340a00700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe33c06bb40 0x7fe33c19c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:39.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.541+0000 7fe341400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe33c06b180 0x7fe33c19be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:52304/0 (socket says 172.21.15.27:52304) 2024-05-10T11:10:39.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.541+0000 7fe341400700 1 -- 172.21.15.27:0/701420802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe33c06bb40 msgr2=0x7fe33c19c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:39.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.541+0000 7fe341400700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe33c06bb40 0x7fe33c19c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.541+0000 7fe341400700 1 -- 172.21.15.27:0/701420802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe33c109510 msgr2=0x7fe33c1a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:39.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.541+0000 7fe341400700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe33c109510 0x7fe33c1a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.541+0000 7fe341400700 1 -- 172.21.15.27:0/701420802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33c1a0dc0 con 0x7fe33c06b180 2024-05-10T11:10:39.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.542+0000 7fe341400700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe33c06b180 0x7fe33c19be00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00e9f0 tx=0x7fe32c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:39.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.542+0000 7fe33a000700 1 -- 172.21.15.27:0/701420802 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c00c9c0 con 0x7fe33c06b180 2024-05-10T11:10:39.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.542+0000 7fe33a000700 1 -- 172.21.15.27:0/701420802 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe32c00cb60 con 0x7fe33c06b180 2024-05-10T11:10:39.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.542+0000 7fe343e00700 1 -- 172.21.15.27:0/701420802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe33c1a10b0 con 0x7fe33c06b180 2024-05-10T11:10:39.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.542+0000 7fe33a000700 1 -- 172.21.15.27:0/701420802 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c018680 con 0x7fe33c06b180 2024-05-10T11:10:39.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.542+0000 7fe343e00700 1 -- 172.21.15.27:0/701420802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe33c1a8930 con 0x7fe33c06b180 2024-05-10T11:10:39.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.543+0000 7fe343e00700 1 -- 172.21.15.27:0/701420802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe33c101880 con 0x7fe33c06b180 2024-05-10T11:10:39.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.547+0000 7fe33a000700 1 -- 172.21.15.27:0/701420802 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fe32c010430 con 0x7fe33c06b180 2024-05-10T11:10:39.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.549+0000 7fe33a000700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe3200b2990 0x7fe3200b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:39.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.549+0000 7fe33a000700 1 -- 172.21.15.27:0/701420802 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fe32c0e2dc0 con 0x7fe33c06b180 2024-05-10T11:10:39.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.549+0000 7fe340a00700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe3200b2990 0x7fe3200b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:39.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.549+0000 7fe33a000700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247] conn(0x7fe3200bdd70 0x7fe3200c0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:39.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.549+0000 7fe33a000700 1 -- 172.21.15.27:0/701420802 --> [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3200c08e0 con 0x7fe3200bdd70 2024-05-10T11:10:39.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.549+0000 7fe33a000700 1 -- 172.21.15.27:0/701420802 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fe32c0e3210 con 0x7fe33c06b180 2024-05-10T11:10:39.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.550+0000 7fe341e00700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247] conn(0x7fe3200bdd70 0x7fe3200c0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:39.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.550+0000 7fe340a00700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe3200b2990 0x7fe3200b4e50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe33c104490 tx=0x7fe32800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:39.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.550+0000 7fe341e00700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247] conn(0x7fe3200bdd70 0x7fe3200c0210 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:39.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.551+0000 7fe33a000700 1 -- 172.21.15.27:0/701420802 <== osd.2 v2:172.21.15.189:6800/1136376247 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3200c08e0 con 0x7fe3200bdd70 2024-05-10T11:10:39.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.570+0000 7fe343e00700 1 -- 172.21.15.27:0/701420802 --> [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe33c104bd0 con 0x7fe3200bdd70 2024-05-10T11:10:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.571+0000 7fe33a000700 1 -- 172.21.15.27:0/701420802 <== osd.2 v2:172.21.15.189:6800/1136376247 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe33c104bd0 con 0x7fe3200bdd70 2024-05-10T11:10:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.571+0000 7fe343e00700 1 -- 172.21.15.27:0/701420802 >> [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247] conn(0x7fe3200bdd70 msgr2=0x7fe3200c0210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.571+0000 7fe343e00700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247] conn(0x7fe3200bdd70 0x7fe3200c0210 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.571+0000 7fe343e00700 1 -- 172.21.15.27:0/701420802 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe3200b2990 msgr2=0x7fe3200b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.571+0000 7fe343e00700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe3200b2990 0x7fe3200b4e50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe33c104490 tx=0x7fe32800b260 comp rx=0 tx=0).stop 2024-05-10T11:10:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.571+0000 7fe343e00700 1 -- 172.21.15.27:0/701420802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe33c06b180 msgr2=0x7fe33c19be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.571+0000 7fe343e00700 1 --2- 172.21.15.27:0/701420802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe33c06b180 0x7fe33c19be00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00e9f0 tx=0x7fe32c00ed00 comp rx=0 tx=0).stop 2024-05-10T11:10:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.571+0000 7fe341e00700 1 -- 172.21.15.27:0/701420802 reap_dead start 2024-05-10T11:10:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.571+0000 7fe343e00700 1 -- 172.21.15.27:0/701420802 shutdown_connections 2024-05-10T11:10:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.571+0000 7fe343e00700 1 -- 172.21.15.27:0/701420802 >> 172.21.15.27:0/701420802 conn(0x7fe33c079350 msgr2=0x7fe33c100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.572+0000 7fe343e00700 1 -- 172.21.15.27:0/701420802 shutdown_connections 2024-05-10T11:10:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:38.572+0000 7fe343e00700 1 -- 172.21.15.27:0/701420802 wait complete. 2024-05-10T11:10:40.330 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:40 smithi027 ceph-mon[30054]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:40.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.379+0000 7febb4800700 1 -- 172.21.15.27:0/2147377890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac106080 msgr2=0x7febac112590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:40.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.379+0000 7febb4800700 1 --2- 172.21.15.27:0/2147377890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac106080 0x7febac112590 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7feba800b0a0 tx=0x7feba800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:10:40.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.379+0000 7febb4800700 1 -- 172.21.15.27:0/2147377890 shutdown_connections 2024-05-10T11:10:40.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.379+0000 7febb4800700 1 --2- 172.21.15.27:0/2147377890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac106080 0x7febac112590 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:40.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.379+0000 7febb4800700 1 --2- 172.21.15.27:0/2147377890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febac10b6c0 0x7febac10baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:40.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.379+0000 7febb4800700 1 --2- 172.21.15.27:0/2147377890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac1056c0 0x7febac105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:40.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.379+0000 7febb4800700 1 -- 172.21.15.27:0/2147377890 >> 172.21.15.27:0/2147377890 conn(0x7febac1011b0 msgr2=0x7febac1035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:40.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.380+0000 7febb4800700 1 -- 172.21.15.27:0/2147377890 shutdown_connections 2024-05-10T11:10:40.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.380+0000 7febb4800700 1 -- 172.21.15.27:0/2147377890 wait complete. 2024-05-10T11:10:40.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.381+0000 7febb4800700 1 Processor -- start 2024-05-10T11:10:40.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.381+0000 7febb4800700 1 -- start start 2024-05-10T11:10:40.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.381+0000 7febb4800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febac1056c0 0x7febac1a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:40.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.381+0000 7febb4800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac106080 0x7febac1a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:40.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.381+0000 7febb4800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac10b6c0 0x7febac19bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:40.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb4800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7febac115900 con 0x7febac10b6c0 2024-05-10T11:10:40.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb4800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7febac115780 con 0x7febac1056c0 2024-05-10T11:10:40.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb4800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7febac115a80 con 0x7febac106080 2024-05-10T11:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb1e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febac1056c0 0x7febac1a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb1e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febac1056c0 0x7febac1a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:38348/0 (socket says 172.21.15.27:38348) 2024-05-10T11:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb1e00700 1 -- 172.21.15.27:0/2440147179 learned_addr learned my addr 172.21.15.27:0/2440147179 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb2800700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac10b6c0 0x7febac19bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb1400700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac106080 0x7febac1a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb1e00700 1 -- 172.21.15.27:0/2440147179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac106080 msgr2=0x7febac1a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb1e00700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac106080 0x7febac1a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb1e00700 1 -- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac10b6c0 msgr2=0x7febac19bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb1e00700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac10b6c0 0x7febac19bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.382+0000 7febb1e00700 1 -- 172.21.15.27:0/2440147179 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febac19c2a0 con 0x7febac1056c0 2024-05-10T11:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.383+0000 7febb1e00700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febac1056c0 0x7febac1a1930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feba000e9a0 tx=0x7feba000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.383+0000 7feba6a00700 1 -- 172.21.15.27:0/2440147179 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba000ca10 con 0x7febac1056c0 2024-05-10T11:10:40.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.383+0000 7feba6a00700 1 -- 172.21.15.27:0/2440147179 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feba000cbb0 con 0x7febac1056c0 2024-05-10T11:10:40.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.383+0000 7feba6a00700 1 -- 172.21.15.27:0/2440147179 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba0018680 con 0x7febac1056c0 2024-05-10T11:10:40.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.383+0000 7febb4800700 1 -- 172.21.15.27:0/2440147179 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febac19c590 con 0x7febac1056c0 2024-05-10T11:10:40.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.383+0000 7febb4800700 1 -- 172.21.15.27:0/2440147179 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febac1a8800 con 0x7febac1056c0 2024-05-10T11:10:40.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.384+0000 7febb4800700 1 -- 172.21.15.27:0/2440147179 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7febac063bf0 con 0x7febac1056c0 2024-05-10T11:10:40.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.387+0000 7feba6a00700 1 -- 172.21.15.27:0/2440147179 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7feba0010430 con 0x7febac1056c0 2024-05-10T11:10:40.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.389+0000 7feba6a00700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7feb940b2910 0x7feb940b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:40.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.389+0000 7feba6a00700 1 -- 172.21.15.27:0/2440147179 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7feba00e2ec0 con 0x7febac1056c0 2024-05-10T11:10:40.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.389+0000 7febb1400700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7feb940b2910 0x7feb940b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:40.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.389+0000 7feba6a00700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800] conn(0x7feb940bdcf0 0x7feb940c0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:40.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.389+0000 7feba6a00700 1 -- 172.21.15.27:0/2440147179 --> [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feba0024e70 con 0x7feb940bdcf0 2024-05-10T11:10:40.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.389+0000 7feba6a00700 1 -- 172.21.15.27:0/2440147179 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7feba00e3270 con 0x7febac1056c0 2024-05-10T11:10:40.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.389+0000 7febb2800700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800] conn(0x7feb940bdcf0 0x7feb940c0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:40.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.390+0000 7febb1400700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7feb940b2910 0x7feb940b4dd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7febac10acc0 tx=0x7feb9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:40.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.390+0000 7febb2800700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800] conn(0x7feb940bdcf0 0x7feb940c0190 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:40.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.391+0000 7feba6a00700 1 -- 172.21.15.27:0/2440147179 <== osd.6 v2:172.21.15.27:6818/3624966800 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feba0024e70 con 0x7feb940bdcf0 2024-05-10T11:10:40.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.410+0000 7febb4800700 1 -- 172.21.15.27:0/2440147179 --> [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7febac064070 con 0x7feb940bdcf0 2024-05-10T11:10:40.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.411+0000 7feba6a00700 1 -- 172.21.15.27:0/2440147179 <== osd.6 v2:172.21.15.27:6818/3624966800 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7febac064070 con 0x7feb940bdcf0 2024-05-10T11:10:40.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.411+0000 7febb4800700 1 -- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800] conn(0x7feb940bdcf0 msgr2=0x7feb940c0190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:40.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.411+0000 7febb4800700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800] conn(0x7feb940bdcf0 0x7feb940c0190 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:40.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.412+0000 7febb4800700 1 -- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7feb940b2910 msgr2=0x7feb940b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:40.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.412+0000 7febb4800700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7feb940b2910 0x7feb940b4dd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7febac10acc0 tx=0x7feb9c009260 comp rx=0 tx=0).stop 2024-05-10T11:10:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.412+0000 7febb4800700 1 -- 172.21.15.27:0/2440147179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febac1056c0 msgr2=0x7febac1a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.412+0000 7febb4800700 1 --2- 172.21.15.27:0/2440147179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febac1056c0 0x7febac1a1930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feba000e9a0 tx=0x7feba000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:10:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.412+0000 7febb2800700 1 -- 172.21.15.27:0/2440147179 reap_dead start 2024-05-10T11:10:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.412+0000 7febb4800700 1 -- 172.21.15.27:0/2440147179 shutdown_connections 2024-05-10T11:10:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.412+0000 7febb4800700 1 -- 172.21.15.27:0/2440147179 >> 172.21.15.27:0/2440147179 conn(0x7febac1011b0 msgr2=0x7febac1017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.412+0000 7febb4800700 1 -- 172.21.15.27:0/2440147179 shutdown_connections 2024-05-10T11:10:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:40.412+0000 7febb4800700 1 -- 172.21.15.27:0/2440147179 wait complete. 2024-05-10T11:10:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:40 smithi076 ceph-mon[34152]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:40.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:40 smithi189 ceph-mon[34842]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:40.848 INFO:teuthology.orchestra.run.smithi027.stdout:304942678067 2024-05-10T11:10:40.849 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-10T11:10:41.469 INFO:teuthology.orchestra.run.smithi027.stdout:77309411432 2024-05-10T11:10:41.469 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-10T11:10:41.565 INFO:teuthology.orchestra.run.smithi027.stdout:34359738483 2024-05-10T11:10:41.565 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-10T11:10:42.328 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:42 smithi027 ceph-mon[30054]: pgmap v420: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-05-10T11:10:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:42 smithi076 ceph-mon[34152]: pgmap v420: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-05-10T11:10:42.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:42 smithi189 ceph-mon[34842]: pgmap v420: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-05-10T11:10:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.957+0000 7fd9f5a00700 1 -- 172.21.15.27:0/812982471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f0103fd0 msgr2=0x7fd9f0108020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.957+0000 7fd9f5a00700 1 --2- 172.21.15.27:0/812982471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f0103fd0 0x7fd9f0108020 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0099c0 tx=0x7fd9dc009cd0 comp rx=0 tx=0).stop 2024-05-10T11:10:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.957+0000 7fd9f5a00700 1 -- 172.21.15.27:0/812982471 shutdown_connections 2024-05-10T11:10:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.957+0000 7fd9f5a00700 1 --2- 172.21.15.27:0/812982471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f0103620 0x7fd9f0103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.957+0000 7fd9f5a00700 1 --2- 172.21.15.27:0/812982471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f0108700 0x7fd9f01102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:42.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.957+0000 7fd9f5a00700 1 --2- 172.21.15.27:0/812982471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f0103fd0 0x7fd9f0108020 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:42.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.957+0000 7fd9f5a00700 1 -- 172.21.15.27:0/812982471 >> 172.21.15.27:0/812982471 conn(0x7fd9f00feeb0 msgr2=0x7fd9f01012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:42.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.957+0000 7fd9f5a00700 1 -- 172.21.15.27:0/812982471 shutdown_connections 2024-05-10T11:10:42.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.957+0000 7fd9f5a00700 1 -- 172.21.15.27:0/812982471 wait complete. 2024-05-10T11:10:42.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.958+0000 7fd9f5a00700 1 Processor -- start 2024-05-10T11:10:42.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.958+0000 7fd9f5a00700 1 -- start start 2024-05-10T11:10:42.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.959+0000 7fd9f5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f0103620 0x7fd9f019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:42.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.959+0000 7fd9f5a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f0103fd0 0x7fd9f019fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:42.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.959+0000 7fd9f5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f0108700 0x7fd9f0199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:42.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.959+0000 7fd9f5a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f01135c0 con 0x7fd9f0103620 2024-05-10T11:10:42.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.959+0000 7fd9f5a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f0113440 con 0x7fd9f0103fd0 2024-05-10T11:10:42.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.959+0000 7fd9f5a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f0113740 con 0x7fd9f0108700 2024-05-10T11:10:42.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.959+0000 7fd9f5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f0108700 0x7fd9f0199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:42.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.959+0000 7fd9efe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f0103620 0x7fd9f019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:42.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.959+0000 7fd9f5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f0108700 0x7fd9f0199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:52350/0 (socket says 172.21.15.27:52350) 2024-05-10T11:10:42.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.959+0000 7fd9f5000700 1 -- 172.21.15.27:0/918389888 learned_addr learned my addr 172.21.15.27:0/918389888 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:42.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.959+0000 7fd9ef400700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f0103fd0 0x7fd9f019fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.960+0000 7fd9f5000700 1 -- 172.21.15.27:0/918389888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f0103fd0 msgr2=0x7fd9f019fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.960+0000 7fd9f5000700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f0103fd0 0x7fd9f019fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.960+0000 7fd9f5000700 1 -- 172.21.15.27:0/918389888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f0103620 msgr2=0x7fd9f019f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.960+0000 7fd9f5000700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f0103620 0x7fd9f019f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.960+0000 7fd9f5000700 1 -- 172.21.15.27:0/918389888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f019a080 con 0x7fd9f0108700 2024-05-10T11:10:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.960+0000 7fd9efe00700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f0103620 0x7fd9f019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:10:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.960+0000 7fd9f5000700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f0108700 0x7fd9f0199920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800ef30 tx=0x7fd9e800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.960+0000 7fd9ecc00700 1 -- 172.21.15.27:0/918389888 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e80092e0 con 0x7fd9f0108700 2024-05-10T11:10:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.960+0000 7fd9f5a00700 1 -- 172.21.15.27:0/918389888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f019a310 con 0x7fd9f0108700 2024-05-10T11:10:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.961+0000 7fd9ecc00700 1 -- 172.21.15.27:0/918389888 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9e8004bb0 con 0x7fd9f0108700 2024-05-10T11:10:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.961+0000 7fd9ecc00700 1 -- 172.21.15.27:0/918389888 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8004280 con 0x7fd9f0108700 2024-05-10T11:10:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.961+0000 7fd9f5a00700 1 -- 172.21.15.27:0/918389888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f01a65e0 con 0x7fd9f0108700 2024-05-10T11:10:42.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.961+0000 7fd9f5a00700 1 -- 172.21.15.27:0/918389888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd9f0063bb0 con 0x7fd9f0108700 2024-05-10T11:10:42.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.964+0000 7fd9ecc00700 1 -- 172.21.15.27:0/918389888 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd9e801b030 con 0x7fd9f0108700 2024-05-10T11:10:42.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.966+0000 7fd9ecc00700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9d40b2990 0x7fd9d40b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:42.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.966+0000 7fd9ecc00700 1 -- 172.21.15.27:0/918389888 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fd9e80e2c50 con 0x7fd9f0108700 2024-05-10T11:10:42.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.966+0000 7fd9efe00700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9d40b2990 0x7fd9d40b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:42.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.967+0000 7fd9ecc00700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899] conn(0x7fd9d40bdcf0 0x7fd9d40c0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:42.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.967+0000 7fd9ecc00700 1 -- 172.21.15.27:0/918389888 --> [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9e800be50 con 0x7fd9d40bdcf0 2024-05-10T11:10:42.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.967+0000 7fd9ecc00700 1 -- 172.21.15.27:0/918389888 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fd9e80e3000 con 0x7fd9f0108700 2024-05-10T11:10:42.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.967+0000 7fd9ef400700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899] conn(0x7fd9d40bdcf0 0x7fd9d40c0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:42.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.967+0000 7fd9efe00700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9d40b2990 0x7fd9d40b4e50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0108530 tx=0x7fd9e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:42.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.968+0000 7fd9ef400700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899] conn(0x7fd9d40bdcf0 0x7fd9d40c0190 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:42.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.969+0000 7fd9ecc00700 1 -- 172.21.15.27:0/918389888 <== osd.1 v2:172.21.15.76:6800/98361899 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9e800be50 con 0x7fd9d40bdcf0 2024-05-10T11:10:42.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.987+0000 7fd9f5a00700 1 -- 172.21.15.27:0/918389888 --> [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd9f0064030 con 0x7fd9d40bdcf0 2024-05-10T11:10:42.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.988+0000 7fd9ecc00700 1 -- 172.21.15.27:0/918389888 <== osd.1 v2:172.21.15.76:6800/98361899 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd9f0064030 con 0x7fd9d40bdcf0 2024-05-10T11:10:42.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.988+0000 7fd9f5a00700 1 -- 172.21.15.27:0/918389888 >> [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899] conn(0x7fd9d40bdcf0 msgr2=0x7fd9d40c0190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:42.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.988+0000 7fd9f5a00700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899] conn(0x7fd9d40bdcf0 0x7fd9d40c0190 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:42.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.988+0000 7fd9f5a00700 1 -- 172.21.15.27:0/918389888 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9d40b2990 msgr2=0x7fd9d40b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:42.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.988+0000 7fd9f5a00700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9d40b2990 0x7fd9d40b4e50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0108530 tx=0x7fd9e0009260 comp rx=0 tx=0).stop 2024-05-10T11:10:42.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.988+0000 7fd9f5a00700 1 -- 172.21.15.27:0/918389888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f0108700 msgr2=0x7fd9f0199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:42.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.988+0000 7fd9f5a00700 1 --2- 172.21.15.27:0/918389888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f0108700 0x7fd9f0199920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800ef30 tx=0x7fd9e800c3a0 comp rx=0 tx=0).stop 2024-05-10T11:10:42.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.988+0000 7fd9f5000700 1 -- 172.21.15.27:0/918389888 reap_dead start 2024-05-10T11:10:42.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.989+0000 7fd9f5a00700 1 -- 172.21.15.27:0/918389888 shutdown_connections 2024-05-10T11:10:42.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.989+0000 7fd9f5a00700 1 -- 172.21.15.27:0/918389888 >> 172.21.15.27:0/918389888 conn(0x7fd9f00feeb0 msgr2=0x7fd9f0101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:42.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.989+0000 7fd9f5a00700 1 -- 172.21.15.27:0/918389888 shutdown_connections 2024-05-10T11:10:42.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:42.989+0000 7fd9f5a00700 1 -- 172.21.15.27:0/918389888 wait complete. 2024-05-10T11:10:43.876 INFO:teuthology.orchestra.run.smithi027.stdout:107374182498 2024-05-10T11:10:43.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-10T11:10:44.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:44 smithi027 ceph-mon[30054]: pgmap v421: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-10T11:10:44.409 INFO:teuthology.orchestra.run.smithi027.stdout:55834574958 2024-05-10T11:10:44.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-10T11:10:44.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:44 smithi076 ceph-mon[34152]: pgmap v421: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-10T11:10:44.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:44 smithi189 ceph-mon[34842]: pgmap v421: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-10T11:10:44.527 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:44.542 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:44.561 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:44.561 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:44.591 INFO:teuthology.orchestra.run.smithi027.stdout:180388626514 2024-05-10T11:10:44.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-10T11:10:45.978 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:45.984 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:46.366 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:46 smithi027 ceph-mon[30054]: pgmap v422: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-05-10T11:10:46.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:46 smithi076 ceph-mon[34152]: pgmap v422: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-05-10T11:10:46.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:46 smithi189 ceph-mon[34842]: pgmap v422: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-05-10T11:10:47.024 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:47.025 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:48.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:48 smithi027 ceph-mon[30054]: pgmap v423: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-05-10T11:10:48.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:48 smithi076 ceph-mon[34152]: pgmap v423: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-05-10T11:10:48.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:48 smithi189 ceph-mon[34842]: pgmap v423: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-05-10T11:10:49.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:10:49.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:49.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:49.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:49.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:49 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:49.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:49 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:10:49.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:49 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:49.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:49 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:49.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:49 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:49.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:49 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:49.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:49 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:10:49.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:49 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:49.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:49 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:49.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:49 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:49.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:49 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:50.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:50 smithi027 ceph-mon[30054]: pgmap v424: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:50.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:50 smithi076 ceph-mon[34152]: pgmap v424: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:50.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:50 smithi189 ceph-mon[34842]: pgmap v424: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:10:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.542+0000 7f2655600700 1 -- 172.21.15.27:0/73889198 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2650104590 msgr2=0x7f2650104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.542+0000 7f2655600700 1 --2- 172.21.15.27:0/73889198 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2650104590 0x7f2650104970 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2640009990 tx=0x7f2640009ca0 comp rx=0 tx=0).stop 2024-05-10T11:10:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.542+0000 7f2655600700 1 -- 172.21.15.27:0/73889198 shutdown_connections 2024-05-10T11:10:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.542+0000 7f2655600700 1 --2- 172.21.15.27:0/73889198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f265010d9c0 0x7f265010fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.542+0000 7f2655600700 1 --2- 172.21.15.27:0/73889198 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2650104590 0x7f2650104970 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.542+0000 7f2655600700 1 --2- 172.21.15.27:0/73889198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2650104eb0 0x7f265010d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.542+0000 7f2655600700 1 -- 172.21.15.27:0/73889198 >> 172.21.15.27:0/73889198 conn(0x7f26500fddc0 msgr2=0x7f26501001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.542+0000 7f2655600700 1 -- 172.21.15.27:0/73889198 shutdown_connections 2024-05-10T11:10:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.542+0000 7f2655600700 1 -- 172.21.15.27:0/73889198 wait complete. 2024-05-10T11:10:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.543+0000 7f2655600700 1 Processor -- start 2024-05-10T11:10:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.544+0000 7f2655600700 1 -- start start 2024-05-10T11:10:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.544+0000 7f2655600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2650104590 0x7f26501a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.544+0000 7f2655600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2650104eb0 0x7f26501a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.544+0000 7f2655600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f265010d9c0 0x7f265019baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.544+0000 7f2655600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f26501135d0 con 0x7f2650104eb0 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.544+0000 7f2655600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2650113450 con 0x7f265010d9c0 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.544+0000 7f2655600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2650113750 con 0x7f2650104590 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.544+0000 7f264fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2650104590 0x7f26501a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.544+0000 7f264f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2650104eb0 0x7f26501a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.544+0000 7f264fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2650104590 0x7f26501a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:52908/0 (socket says 172.21.15.27:52908) 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.544+0000 7f264fe00700 1 -- 172.21.15.27:0/3185783389 learned_addr learned my addr 172.21.15.27:0/3185783389 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.545+0000 7f2654c00700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f265010d9c0 0x7f265019baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.545+0000 7f264fe00700 1 -- 172.21.15.27:0/3185783389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f265010d9c0 msgr2=0x7f265019baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.545+0000 7f264fe00700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f265010d9c0 0x7f265019baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.545+0000 7f264fe00700 1 -- 172.21.15.27:0/3185783389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2650104eb0 msgr2=0x7f26501a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.545+0000 7f264fe00700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2650104eb0 0x7f26501a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.545+0000 7f264fe00700 1 -- 172.21.15.27:0/3185783389 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265019c250 con 0x7f2650104590 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.545+0000 7f264fe00700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2650104590 0x7f26501a1930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2640000c00 tx=0x7f264000bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.546+0000 7f264cc00700 1 -- 172.21.15.27:0/3185783389 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640027070 con 0x7f2650104590 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.546+0000 7f2655600700 1 -- 172.21.15.27:0/3185783389 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f265019c4e0 con 0x7f2650104590 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.546+0000 7f264cc00700 1 -- 172.21.15.27:0/3185783389 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2640022470 con 0x7f2650104590 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.546+0000 7f264cc00700 1 -- 172.21.15.27:0/3185783389 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640017650 con 0x7f2650104590 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.546+0000 7f2655600700 1 -- 172.21.15.27:0/3185783389 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26501a8800 con 0x7f2650104590 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.547+0000 7f2655600700 1 -- 172.21.15.27:0/3185783389 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f265006a190 con 0x7f2650104590 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.549+0000 7f264cc00700 1 -- 172.21.15.27:0/3185783389 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f264002a070 con 0x7f2650104590 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.551+0000 7f264cc00700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f26340b2900 0x7f26340b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.551+0000 7f264cc00700 1 -- 172.21.15.27:0/3185783389 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f26400e3230 con 0x7f2650104590 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.551+0000 7f264f400700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f26340b2900 0x7f26340b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.552+0000 7f264f400700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f26340b2900 0x7f26340b4dc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f263c009680 tx=0x7f263c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.555+0000 7f264cc00700 1 -- 172.21.15.27:0/3185783389 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f2640013070 con 0x7f2650104590 2024-05-10T11:10:51.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.796+0000 7f2655600700 1 -- 172.21.15.27:0/3185783389 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f265002e510 con 0x7f2650104590 2024-05-10T11:10:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.797+0000 7f264cc00700 1 -- 172.21.15.27:0/3185783389 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f26400aab90 con 0x7f2650104590 2024-05-10T11:10:51.798 INFO:teuthology.orchestra.run.smithi027.stdout:201863462990 2024-05-10T11:10:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.800+0000 7f2655600700 1 -- 172.21.15.27:0/3185783389 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f26340b2900 msgr2=0x7f26340b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.800+0000 7f2655600700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f26340b2900 0x7f26340b4dc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f263c009680 tx=0x7f263c009260 comp rx=0 tx=0).stop 2024-05-10T11:10:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.800+0000 7f2655600700 1 -- 172.21.15.27:0/3185783389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2650104590 msgr2=0x7f26501a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.800+0000 7f2655600700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2650104590 0x7f26501a1930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2640000c00 tx=0x7f264000bfd0 comp rx=0 tx=0).stop 2024-05-10T11:10:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.801+0000 7f2655600700 1 -- 172.21.15.27:0/3185783389 shutdown_connections 2024-05-10T11:10:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.801+0000 7f2655600700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f26340b2900 0x7f26340b4dc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.801+0000 7f2655600700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2650104590 0x7f26501a1930 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.801+0000 7f2655600700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f265010d9c0 0x7f265019baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.801+0000 7f2655600700 1 --2- 172.21.15.27:0/3185783389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2650104eb0 0x7f26501a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.801+0000 7f2655600700 1 -- 172.21.15.27:0/3185783389 >> 172.21.15.27:0/3185783389 conn(0x7f26500fddc0 msgr2=0x7f26500fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.801+0000 7f2655600700 1 -- 172.21.15.27:0/3185783389 shutdown_connections 2024-05-10T11:10:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:51.801+0000 7f2655600700 1 -- 172.21.15.27:0/3185783389 wait complete. 2024-05-10T11:10:52.333 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:52 smithi027 ceph-mon[30054]: pgmap v425: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:10:52.333 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:52 smithi027 ceph-mon[30054]: mgrmap e18: x(active, since 11m), standbys: z, y 2024-05-10T11:10:52.333 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:52 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3185783389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-10T11:10:52.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:52 smithi076 ceph-mon[34152]: pgmap v425: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:10:52.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:52 smithi076 ceph-mon[34152]: mgrmap e18: x(active, since 11m), standbys: z, y 2024-05-10T11:10:52.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:52 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3185783389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-10T11:10:52.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:52 smithi189 ceph-mon[34842]: pgmap v425: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:10:52.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:52 smithi189 ceph-mon[34842]: mgrmap e18: x(active, since 11m), standbys: z, y 2024-05-10T11:10:52.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:52 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3185783389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-10T11:10:54.321 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:54 smithi027 ceph-mon[30054]: pgmap v426: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-10T11:10:54.321 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:54 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:10:54.321 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:54 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:10:54.321 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:54 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:54.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:54 smithi076 ceph-mon[34152]: pgmap v426: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-10T11:10:54.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:54 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:10:54.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:54 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:10:54.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:54 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:54.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:54 smithi189 ceph-mon[34842]: pgmap v426: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-10T11:10:54.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:54 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:10:54.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:54 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:10:54.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:54 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:10:55.335 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462986 got 201863462990 for osd.7 2024-05-10T11:10:55.336 DEBUG:teuthology.parallel:result is None 2024-05-10T11:10:55.923 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:10:56.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:56 smithi027 ceph-mon[30054]: pgmap v427: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 9 op/s 2024-05-10T11:10:56.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:56 smithi076 ceph-mon[34152]: pgmap v427: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 9 op/s 2024-05-10T11:10:56.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:56 smithi189 ceph-mon[34842]: pgmap v427: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 9 op/s 2024-05-10T11:10:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.275+0000 7f69a4800700 1 -- 172.21.15.27:0/3004421794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c06b180 msgr2=0x7f699c06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.275+0000 7f69a4800700 1 --2- 172.21.15.27:0/3004421794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c06b180 0x7f699c06b600 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f698c009990 tx=0x7f698c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.276+0000 7f69a4800700 1 -- 172.21.15.27:0/3004421794 shutdown_connections 2024-05-10T11:10:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.276+0000 7f69a4800700 1 --2- 172.21.15.27:0/3004421794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f699c06bb40 0x7f699c110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.276+0000 7f69a4800700 1 --2- 172.21.15.27:0/3004421794 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f699c109510 0x7f699c1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.276+0000 7f69a4800700 1 --2- 172.21.15.27:0/3004421794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c06b180 0x7f699c06b600 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.276+0000 7f69a4800700 1 -- 172.21.15.27:0/3004421794 >> 172.21.15.27:0/3004421794 conn(0x7f699c079350 msgr2=0x7f699c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.276+0000 7f69a4800700 1 -- 172.21.15.27:0/3004421794 shutdown_connections 2024-05-10T11:10:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.276+0000 7f69a4800700 1 -- 172.21.15.27:0/3004421794 wait complete. 2024-05-10T11:10:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.277+0000 7f69a4800700 1 Processor -- start 2024-05-10T11:10:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.277+0000 7f69a4800700 1 -- start start 2024-05-10T11:10:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.277+0000 7f69a4800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c06b180 0x7f699c19bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.277+0000 7f69a4800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f699c06bb40 0x7f699c19c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.277+0000 7f69a4800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f699c109510 0x7f699c1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.277+0000 7f69a4800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f699c113690 con 0x7f699c06b180 2024-05-10T11:10:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.277+0000 7f69a4800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f699c113510 con 0x7f699c109510 2024-05-10T11:10:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.277+0000 7f69a4800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f699c113810 con 0x7f699c06bb40 2024-05-10T11:10:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.278+0000 7f69a1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c06b180 0x7f699c19bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.278+0000 7f69a2800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f699c109510 0x7f699c1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.278+0000 7f69a1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c06b180 0x7f699c19bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41108/0 (socket says 172.21.15.27:41108) 2024-05-10T11:10:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.278+0000 7f69a2800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f699c109510 0x7f699c1a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:59080/0 (socket says 172.21.15.27:59080) 2024-05-10T11:10:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.278+0000 7f69a1e00700 1 -- 172.21.15.27:0/2381766664 learned_addr learned my addr 172.21.15.27:0/2381766664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.278+0000 7f69a1400700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f699c06bb40 0x7f699c19c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.278+0000 7f69a2800700 1 -- 172.21.15.27:0/2381766664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f699c06bb40 msgr2=0x7f699c19c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.278+0000 7f69a2800700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f699c06bb40 0x7f699c19c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.278+0000 7f69a2800700 1 -- 172.21.15.27:0/2381766664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c06b180 msgr2=0x7f699c19bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.278+0000 7f69a2800700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c06b180 0x7f699c19bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.278+0000 7f69a2800700 1 -- 172.21.15.27:0/2381766664 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699c1a0db0 con 0x7f699c109510 2024-05-10T11:10:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.278+0000 7f69a1400700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f699c06bb40 0x7f699c19c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:10:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.279+0000 7f69a2800700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f699c109510 0x7f699c1a0670 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f699800efc0 tx=0x7f699800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.279+0000 7f6996a00700 1 -- 172.21.15.27:0/2381766664 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69980092e0 con 0x7f699c109510 2024-05-10T11:10:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.279+0000 7f6996a00700 1 -- 172.21.15.27:0/2381766664 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6998004bb0 con 0x7f699c109510 2024-05-10T11:10:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.279+0000 7f6996a00700 1 -- 172.21.15.27:0/2381766664 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6998004020 con 0x7f699c109510 2024-05-10T11:10:57.155 INFO:teuthology.orchestra.run.smithi027.stdout:154618822747 2024-05-10T11:10:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.279+0000 7f69a4800700 1 -- 172.21.15.27:0/2381766664 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f699c1a1040 con 0x7f699c109510 2024-05-10T11:10:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.279+0000 7f69a4800700 1 -- 172.21.15.27:0/2381766664 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f699c1a8920 con 0x7f699c109510 2024-05-10T11:10:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.281+0000 7f69a4800700 1 -- 172.21.15.27:0/2381766664 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f699c06a190 con 0x7f699c109510 2024-05-10T11:10:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.283+0000 7f6996a00700 1 -- 172.21.15.27:0/2381766664 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f699801b030 con 0x7f699c109510 2024-05-10T11:10:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.284+0000 7f6996a00700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f69840b2900 0x7f69840b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.284+0000 7f6996a00700 1 -- 172.21.15.27:0/2381766664 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f69980e20b0 con 0x7f699c109510 2024-05-10T11:10:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.285+0000 7f69a1e00700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f69840b2900 0x7f69840b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.285+0000 7f69a1e00700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f69840b2900 0x7f69840b4dc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6990009ca0 tx=0x7f6990009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.290+0000 7f6996a00700 1 -- 172.21.15.27:0/2381766664 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6998022020 con 0x7f699c109510 2024-05-10T11:10:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.534+0000 7f69a4800700 1 -- 172.21.15.27:0/2381766664 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f699c1a8af0 con 0x7f699c109510 2024-05-10T11:10:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.536+0000 7f6996a00700 1 -- 172.21.15.27:0/2381766664 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6998010040 con 0x7f699c109510 2024-05-10T11:10:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.538+0000 7f69a4800700 1 -- 172.21.15.27:0/2381766664 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f69840b2900 msgr2=0x7f69840b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.538+0000 7f69a4800700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f69840b2900 0x7f69840b4dc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6990009ca0 tx=0x7f6990009260 comp rx=0 tx=0).stop 2024-05-10T11:10:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.538+0000 7f69a4800700 1 -- 172.21.15.27:0/2381766664 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f699c109510 msgr2=0x7f699c1a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.538+0000 7f69a4800700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f699c109510 0x7f699c1a0670 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f699800efc0 tx=0x7f699800c3a0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.538+0000 7f69a4800700 1 -- 172.21.15.27:0/2381766664 shutdown_connections 2024-05-10T11:10:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.538+0000 7f69a4800700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f69840b2900 0x7f69840b4dc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.538+0000 7f69a4800700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f699c109510 0x7f699c1a0670 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.538+0000 7f69a4800700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c06b180 0x7f699c19bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.538+0000 7f69a4800700 1 --2- 172.21.15.27:0/2381766664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f699c06bb40 0x7f699c19c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.538+0000 7f69a4800700 1 -- 172.21.15.27:0/2381766664 >> 172.21.15.27:0/2381766664 conn(0x7f699c079350 msgr2=0x7f699c10e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.538+0000 7f69a4800700 1 -- 172.21.15.27:0/2381766664 shutdown_connections 2024-05-10T11:10:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:56.538+0000 7f69a4800700 1 -- 172.21.15.27:0/2381766664 wait complete. 2024-05-10T11:10:57.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:57 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2381766664' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-10T11:10:57.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:57 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2381766664' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-10T11:10:57.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:57 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2381766664' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-10T11:10:57.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.801+0000 7fcb05600700 1 -- 172.21.15.27:0/3808027334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb00103620 msgr2=0x7fcb00103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:57.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.801+0000 7fcb05600700 1 --2- 172.21.15.27:0/3808027334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb00103620 0x7fcb00103a00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcaf00099c0 tx=0x7fcaf0009cd0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.801+0000 7fcb05600700 1 -- 172.21.15.27:0/3808027334 shutdown_connections 2024-05-10T11:10:57.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.801+0000 7fcb05600700 1 --2- 172.21.15.27:0/3808027334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb00108700 0x7fcb001102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.801+0000 7fcb05600700 1 --2- 172.21.15.27:0/3808027334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb00103620 0x7fcb00103a00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.801+0000 7fcb05600700 1 --2- 172.21.15.27:0/3808027334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcb00103fd0 0x7fcb00108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.801+0000 7fcb05600700 1 -- 172.21.15.27:0/3808027334 >> 172.21.15.27:0/3808027334 conn(0x7fcb000feef0 msgr2=0x7fcb00101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:57.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.801+0000 7fcb05600700 1 -- 172.21.15.27:0/3808027334 shutdown_connections 2024-05-10T11:10:57.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.801+0000 7fcb05600700 1 -- 172.21.15.27:0/3808027334 wait complete. 2024-05-10T11:10:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.802+0000 7fcb05600700 1 Processor -- start 2024-05-10T11:10:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.803+0000 7fcb05600700 1 -- start start 2024-05-10T11:10:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.803+0000 7fcb05600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcb00103620 0x7fcb001a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.803+0000 7fcb05600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb00103fd0 0x7fcb001a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.803+0000 7fcb05600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb00108700 0x7fcb0019bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.803+0000 7fcb05600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcb001135e0 con 0x7fcb00103fd0 2024-05-10T11:10:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.803+0000 7fcb05600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcb00113460 con 0x7fcb00103620 2024-05-10T11:10:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.803+0000 7fcb05600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcb00113760 con 0x7fcb00108700 2024-05-10T11:10:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcaff400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb00103fd0 0x7fcb001a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcaffe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcb00103620 0x7fcb001a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcaff400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb00103fd0 0x7fcb001a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41124/0 (socket says 172.21.15.27:41124) 2024-05-10T11:10:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcaffe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcb00103620 0x7fcb001a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:59096/0 (socket says 172.21.15.27:59096) 2024-05-10T11:10:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcaff400700 1 -- 172.21.15.27:0/789731624 learned_addr learned my addr 172.21.15.27:0/789731624 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcb04c00700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb00108700 0x7fcb0019bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcaffe00700 1 -- 172.21.15.27:0/789731624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb00108700 msgr2=0x7fcb0019bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcaffe00700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb00108700 0x7fcb0019bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcaffe00700 1 -- 172.21.15.27:0/789731624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb00103fd0 msgr2=0x7fcb001a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcaffe00700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb00103fd0 0x7fcb001a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcaffe00700 1 -- 172.21.15.27:0/789731624 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0019c210 con 0x7fcb00103620 2024-05-10T11:10:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcb04c00700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb00108700 0x7fcb0019bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:10:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.804+0000 7fcaffe00700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcb00103620 0x7fcb001a18f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcaf00098e0 tx=0x7fcaf0005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.805+0000 7fcafcc00700 1 -- 172.21.15.27:0/789731624 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0026070 con 0x7fcb00103620 2024-05-10T11:10:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.805+0000 7fcb05600700 1 -- 172.21.15.27:0/789731624 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0019c4a0 con 0x7fcb00103620 2024-05-10T11:10:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.805+0000 7fcafcc00700 1 -- 172.21.15.27:0/789731624 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcaf00056c0 con 0x7fcb00103620 2024-05-10T11:10:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.805+0000 7fcafcc00700 1 -- 172.21.15.27:0/789731624 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf000f990 con 0x7fcb00103620 2024-05-10T11:10:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.805+0000 7fcb05600700 1 -- 172.21.15.27:0/789731624 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb001a87c0 con 0x7fcb00103620 2024-05-10T11:10:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.806+0000 7fcb05600700 1 -- 172.21.15.27:0/789731624 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb0006a190 con 0x7fcb00103620 2024-05-10T11:10:57.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.808+0000 7fcafcc00700 1 -- 172.21.15.27:0/789731624 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fcaf0029070 con 0x7fcb00103620 2024-05-10T11:10:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.810+0000 7fcafcc00700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fcae40b2900 0x7fcae40b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.810+0000 7fcafcc00700 1 -- 172.21.15.27:0/789731624 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fcaf00e31c0 con 0x7fcb00103620 2024-05-10T11:10:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.810+0000 7fcaff400700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fcae40b2900 0x7fcae40b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.811+0000 7fcaff400700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fcae40b2900 0x7fcae40b4dc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcb0010a060 tx=0x7fcae8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:57.812+0000 7fcafcc00700 1 -- 172.21.15.27:0/789731624 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fcaf0030020 con 0x7fcb00103620 2024-05-10T11:10:58.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.057+0000 7fcb05600700 1 -- 172.21.15.27:0/789731624 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fcb001a8990 con 0x7fcb00103620 2024-05-10T11:10:58.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.058+0000 7fcafcc00700 1 -- 172.21.15.27:0/789731624 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fcaf0013070 con 0x7fcb00103620 2024-05-10T11:10:58.058 INFO:teuthology.orchestra.run.smithi027.stdout:304942678072 2024-05-10T11:10:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.061+0000 7fcb05600700 1 -- 172.21.15.27:0/789731624 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fcae40b2900 msgr2=0x7fcae40b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.061+0000 7fcb05600700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fcae40b2900 0x7fcae40b4dc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcb0010a060 tx=0x7fcae8009260 comp rx=0 tx=0).stop 2024-05-10T11:10:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.061+0000 7fcb05600700 1 -- 172.21.15.27:0/789731624 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcb00103620 msgr2=0x7fcb001a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.061+0000 7fcb05600700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcb00103620 0x7fcb001a18f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcaf00098e0 tx=0x7fcaf0005280 comp rx=0 tx=0).stop 2024-05-10T11:10:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.061+0000 7fcb05600700 1 -- 172.21.15.27:0/789731624 shutdown_connections 2024-05-10T11:10:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.061+0000 7fcb05600700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fcae40b2900 0x7fcae40b4dc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.061+0000 7fcb05600700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb00108700 0x7fcb0019bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.061+0000 7fcb05600700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcb00103620 0x7fcb001a18f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.061+0000 7fcb05600700 1 --2- 172.21.15.27:0/789731624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb00103fd0 0x7fcb001a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.061+0000 7fcb05600700 1 -- 172.21.15.27:0/789731624 >> 172.21.15.27:0/789731624 conn(0x7fcb000feef0 msgr2=0x7fcb001011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:58.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.063+0000 7fcb05600700 1 -- 172.21.15.27:0/789731624 shutdown_connections 2024-05-10T11:10:58.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.063+0000 7fcb05600700 1 -- 172.21.15.27:0/789731624 wait complete. 2024-05-10T11:10:58.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.180+0000 7f5c1e000700 1 -- 172.21.15.27:0/1511889225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c18076160 msgr2=0x7f5c18113e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:58.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.180+0000 7f5c1e000700 1 --2- 172.21.15.27:0/1511889225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c18076160 0x7f5c18113e80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009990 tx=0x7f5c08009ca0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.180+0000 7f5c1e000700 1 -- 172.21.15.27:0/1511889225 shutdown_connections 2024-05-10T11:10:58.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.180+0000 7f5c1e000700 1 --2- 172.21.15.27:0/1511889225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c181143c0 0x7f5c181167b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.180+0000 7f5c1e000700 1 --2- 172.21.15.27:0/1511889225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c18075840 0x7f5c18075c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.180+0000 7f5c1e000700 1 --2- 172.21.15.27:0/1511889225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c18076160 0x7f5c18113e80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.180+0000 7f5c1e000700 1 -- 172.21.15.27:0/1511889225 >> 172.21.15.27:0/1511889225 conn(0x7f5c180feec0 msgr2=0x7f5c181012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:58.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.180+0000 7f5c1e000700 1 -- 172.21.15.27:0/1511889225 shutdown_connections 2024-05-10T11:10:58.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.181+0000 7f5c1e000700 1 -- 172.21.15.27:0/1511889225 wait complete. 2024-05-10T11:10:58.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.181+0000 7f5c1e000700 1 Processor -- start 2024-05-10T11:10:58.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c1e000700 1 -- start start 2024-05-10T11:10:58.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c1e000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c18075840 0x7f5c181a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:58.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c1e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c18076160 0x7f5c181a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:58.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c1e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c181143c0 0x7f5c181a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:58.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c1e000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5c18119c00 con 0x7f5c181143c0 2024-05-10T11:10:58.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c1e000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5c18119a80 con 0x7f5c18075840 2024-05-10T11:10:58.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c1e000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5c18119d80 con 0x7f5c18076160 2024-05-10T11:10:58.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c17e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c18076160 0x7f5c181a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:58.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c1cc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c18075840 0x7f5c181a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:58.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c17e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c18076160 0x7f5c181a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50492/0 (socket says 172.21.15.27:50492) 2024-05-10T11:10:58.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c1cc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c18075840 0x7f5c181a00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:59120/0 (socket says 172.21.15.27:59120) 2024-05-10T11:10:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c17e00700 1 -- 172.21.15.27:0/2593747971 learned_addr learned my addr 172.21.15.27:0/2593747971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.182+0000 7f5c1d600700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c181143c0 0x7f5c181a49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.183+0000 7f5c17e00700 1 -- 172.21.15.27:0/2593747971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c18075840 msgr2=0x7f5c181a00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.183+0000 7f5c17e00700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c18075840 0x7f5c181a00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.183+0000 7f5c17e00700 1 -- 172.21.15.27:0/2593747971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c181143c0 msgr2=0x7f5c181a49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.183+0000 7f5c17e00700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c181143c0 0x7f5c181a49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.183+0000 7f5c17e00700 1 -- 172.21.15.27:0/2593747971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c181a50e0 con 0x7f5c18076160 2024-05-10T11:10:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.183+0000 7f5c17e00700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c18076160 0x7f5c181a0610 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5c08005910 tx=0x7f5c0800b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.183+0000 7f5c15600700 1 -- 172.21.15.27:0/2593747971 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08027070 con 0x7f5c18076160 2024-05-10T11:10:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.184+0000 7f5c15600700 1 -- 172.21.15.27:0/2593747971 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c08022470 con 0x7f5c18076160 2024-05-10T11:10:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.184+0000 7f5c15600700 1 -- 172.21.15.27:0/2593747971 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08017650 con 0x7f5c18076160 2024-05-10T11:10:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.184+0000 7f5c1e000700 1 -- 172.21.15.27:0/2593747971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c181a5370 con 0x7f5c18076160 2024-05-10T11:10:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.184+0000 7f5c1e000700 1 -- 172.21.15.27:0/2593747971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c181acc50 con 0x7f5c18076160 2024-05-10T11:10:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.185+0000 7f5c1e000700 1 -- 172.21.15.27:0/2593747971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c1806a190 con 0x7f5c18076160 2024-05-10T11:10:58.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.187+0000 7f5c15600700 1 -- 172.21.15.27:0/2593747971 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f5c0802a070 con 0x7f5c18076160 2024-05-10T11:10:58.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.189+0000 7f5c15600700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5c000b2900 0x7f5c000b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:58.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.189+0000 7f5c15600700 1 -- 172.21.15.27:0/2593747971 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f5c080e3280 con 0x7f5c18076160 2024-05-10T11:10:58.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.189+0000 7f5c1cc00700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5c000b2900 0x7f5c000b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:58.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.190+0000 7f5c1cc00700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5c000b2900 0x7f5c000b4dc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009c60 tx=0x7f5c0c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:58.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.193+0000 7f5c15600700 1 -- 172.21.15.27:0/2593747971 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5c08013070 con 0x7f5c18076160 2024-05-10T11:10:58.314 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:58 smithi027 ceph-mon[30054]: pgmap v428: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-10T11:10:58.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:58 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/789731624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-10T11:10:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:58 smithi076 ceph-mon[34152]: pgmap v428: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-10T11:10:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:58 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/789731624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-10T11:10:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.443+0000 7f5c1e000700 1 -- 172.21.15.27:0/2593747971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f5c1802e510 con 0x7f5c18076160 2024-05-10T11:10:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.444+0000 7f5c15600700 1 -- 172.21.15.27:0/2593747971 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5c080aabe0 con 0x7f5c18076160 2024-05-10T11:10:58.444 INFO:teuthology.orchestra.run.smithi027.stdout:227633266762 2024-05-10T11:10:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.447+0000 7f5c1e000700 1 -- 172.21.15.27:0/2593747971 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5c000b2900 msgr2=0x7f5c000b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.447+0000 7f5c1e000700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5c000b2900 0x7f5c000b4dc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009c60 tx=0x7f5c0c009260 comp rx=0 tx=0).stop 2024-05-10T11:10:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.447+0000 7f5c1e000700 1 -- 172.21.15.27:0/2593747971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c18076160 msgr2=0x7f5c181a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.447+0000 7f5c1e000700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c18076160 0x7f5c181a0610 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5c08005910 tx=0x7f5c0800b670 comp rx=0 tx=0).stop 2024-05-10T11:10:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.447+0000 7f5c1e000700 1 -- 172.21.15.27:0/2593747971 shutdown_connections 2024-05-10T11:10:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.447+0000 7f5c1e000700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5c000b2900 0x7f5c000b4dc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.447+0000 7f5c1e000700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c181143c0 0x7f5c181a49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.447+0000 7f5c1e000700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c18075840 0x7f5c181a00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.447+0000 7f5c1e000700 1 --2- 172.21.15.27:0/2593747971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c18076160 0x7f5c181a0610 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.447+0000 7f5c1e000700 1 -- 172.21.15.27:0/2593747971 >> 172.21.15.27:0/2593747971 conn(0x7f5c180feec0 msgr2=0x7f5c181006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.448+0000 7f5c1e000700 1 -- 172.21.15.27:0/2593747971 shutdown_connections 2024-05-10T11:10:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.448+0000 7f5c1e000700 1 -- 172.21.15.27:0/2593747971 wait complete. 2024-05-10T11:10:58.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:58 smithi189 ceph-mon[34842]: pgmap v428: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-10T11:10:58.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:58 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/789731624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-10T11:10:58.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.775+0000 7f6a4ba00700 1 -- 172.21.15.27:0/212920813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a44109510 msgr2=0x7f6a441098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:58.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.775+0000 7f6a4ba00700 1 --2- 172.21.15.27:0/212920813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a44109510 0x7f6a441098f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6a340098f0 tx=0x7f6a34009c00 comp rx=0 tx=0).stop 2024-05-10T11:10:58.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.775+0000 7f6a4ba00700 1 -- 172.21.15.27:0/212920813 shutdown_connections 2024-05-10T11:10:58.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.775+0000 7f6a4ba00700 1 --2- 172.21.15.27:0/212920813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4406bb40 0x7f6a44110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.775+0000 7f6a4ba00700 1 --2- 172.21.15.27:0/212920813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a44109510 0x7f6a441098f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.775+0000 7f6a4ba00700 1 --2- 172.21.15.27:0/212920813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a4406b180 0x7f6a4406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.775+0000 7f6a4ba00700 1 -- 172.21.15.27:0/212920813 >> 172.21.15.27:0/212920813 conn(0x7f6a44079350 msgr2=0x7f6a44079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:58.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.776+0000 7f6a4ba00700 1 -- 172.21.15.27:0/212920813 shutdown_connections 2024-05-10T11:10:58.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.776+0000 7f6a4ba00700 1 -- 172.21.15.27:0/212920813 wait complete. 2024-05-10T11:10:58.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.777+0000 7f6a4ba00700 1 Processor -- start 2024-05-10T11:10:58.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.777+0000 7f6a4ba00700 1 -- start start 2024-05-10T11:10:58.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.777+0000 7f6a4ba00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4406b180 0x7f6a441a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:58.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.777+0000 7f6a4ba00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a4406bb40 0x7f6a441a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:58.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.777+0000 7f6a4ba00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a44109510 0x7f6a4410f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:58.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.777+0000 7f6a4ba00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6a44113670 con 0x7f6a4406b180 2024-05-10T11:10:58.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.777+0000 7f6a4ba00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441134f0 con 0x7f6a4406bb40 2024-05-10T11:10:58.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.777+0000 7f6a4ba00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441137f0 con 0x7f6a44109510 2024-05-10T11:10:58.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.778+0000 7f6a49000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4406b180 0x7f6a441a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:58.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.778+0000 7f6a49000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4406b180 0x7f6a441a4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41158/0 (socket says 172.21.15.27:41158) 2024-05-10T11:10:58.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.778+0000 7f6a49000700 1 -- 172.21.15.27:0/1870075510 learned_addr learned my addr 172.21.15.27:0/1870075510 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:10:58.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.778+0000 7f6a3be00700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a4406bb40 0x7f6a441a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:58.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.778+0000 7f6a49a00700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a44109510 0x7f6a4410f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:58.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.778+0000 7f6a49000700 1 -- 172.21.15.27:0/1870075510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a44109510 msgr2=0x7f6a4410f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:58.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.778+0000 7f6a49000700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a44109510 0x7f6a4410f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.778+0000 7f6a49000700 1 -- 172.21.15.27:0/1870075510 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a4406bb40 msgr2=0x7f6a441a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:58.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.778+0000 7f6a49000700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a4406bb40 0x7f6a441a5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:58.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.778+0000 7f6a49000700 1 -- 172.21.15.27:0/1870075510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4410fd70 con 0x7f6a4406b180 2024-05-10T11:10:58.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.779+0000 7f6a49000700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4406b180 0x7f6a441a4e50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6a34009230 tx=0x7f6a3400fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:58.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.779+0000 7f6a42000700 1 -- 172.21.15.27:0/1870075510 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a340050c0 con 0x7f6a4406b180 2024-05-10T11:10:58.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.779+0000 7f6a42000700 1 -- 172.21.15.27:0/1870075510 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a3400b560 con 0x7f6a4406b180 2024-05-10T11:10:58.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.779+0000 7f6a4ba00700 1 -- 172.21.15.27:0/1870075510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a44110000 con 0x7f6a4406b180 2024-05-10T11:10:58.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.779+0000 7f6a42000700 1 -- 172.21.15.27:0/1870075510 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34003a00 con 0x7f6a4406b180 2024-05-10T11:10:58.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.779+0000 7f6a4ba00700 1 -- 172.21.15.27:0/1870075510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a441acdf0 con 0x7f6a4406b180 2024-05-10T11:10:58.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.780+0000 7f6a4ba00700 1 -- 172.21.15.27:0/1870075510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a4406a190 con 0x7f6a4406b180 2024-05-10T11:10:58.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.783+0000 7f6a42000700 1 -- 172.21.15.27:0/1870075510 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f6a3400b710 con 0x7f6a4406b180 2024-05-10T11:10:58.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.784+0000 7f6a42000700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6a280b2900 0x7f6a280b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:10:58.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.784+0000 7f6a42000700 1 -- 172.21.15.27:0/1870075510 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f6a340e36e0 con 0x7f6a4406b180 2024-05-10T11:10:58.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.785+0000 7f6a3be00700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6a280b2900 0x7f6a280b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:10:58.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.785+0000 7f6a3be00700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6a280b2900 0x7f6a280b4dc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009680 tx=0x7f6a30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:10:58.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:58.787+0000 7f6a42000700 1 -- 172.21.15.27:0/1870075510 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6a34013030 con 0x7f6a4406b180 2024-05-10T11:10:59.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.039+0000 7f6a4ba00700 1 -- 172.21.15.27:0/1870075510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f6a4402e510 con 0x7f6a4406b180 2024-05-10T11:10:59.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.040+0000 7f6a42000700 1 -- 172.21.15.27:0/1870075510 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6a340ab040 con 0x7f6a4406b180 2024-05-10T11:10:59.039 INFO:teuthology.orchestra.run.smithi027.stdout:107374182502 2024-05-10T11:10:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.043+0000 7f6a4ba00700 1 -- 172.21.15.27:0/1870075510 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6a280b2900 msgr2=0x7f6a280b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.043+0000 7f6a4ba00700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6a280b2900 0x7f6a280b4dc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009680 tx=0x7f6a30009260 comp rx=0 tx=0).stop 2024-05-10T11:10:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.043+0000 7f6a4ba00700 1 -- 172.21.15.27:0/1870075510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4406b180 msgr2=0x7f6a441a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:10:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.043+0000 7f6a4ba00700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4406b180 0x7f6a441a4e50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6a34009230 tx=0x7f6a3400fa70 comp rx=0 tx=0).stop 2024-05-10T11:10:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.043+0000 7f6a4ba00700 1 -- 172.21.15.27:0/1870075510 shutdown_connections 2024-05-10T11:10:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.043+0000 7f6a4ba00700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f6a280b2900 0x7f6a280b4dc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.043+0000 7f6a4ba00700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a44109510 0x7f6a4410f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.043+0000 7f6a4ba00700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4406b180 0x7f6a441a4e50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.043+0000 7f6a4ba00700 1 --2- 172.21.15.27:0/1870075510 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a4406bb40 0x7f6a441a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:10:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.043+0000 7f6a4ba00700 1 -- 172.21.15.27:0/1870075510 >> 172.21.15.27:0/1870075510 conn(0x7f6a44079350 msgr2=0x7f6a44100ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:10:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.044+0000 7f6a4ba00700 1 -- 172.21.15.27:0/1870075510 shutdown_connections 2024-05-10T11:10:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.044+0000 7f6a4ba00700 1 -- 172.21.15.27:0/1870075510 wait complete. 2024-05-10T11:10:59.421 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:59 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2593747971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-10T11:10:59.421 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:10:59 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1870075510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-10T11:10:59.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:59 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2593747971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-10T11:10:59.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:10:59 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1870075510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-10T11:10:59.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:59 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2593747971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-10T11:10:59.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:10:59 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1870075510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-10T11:11:00.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:00 smithi027 ceph-mon[30054]: pgmap v429: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-10T11:11:00.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:00 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4197377201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-10T11:11:00.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:00 smithi076 ceph-mon[34152]: pgmap v429: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-10T11:11:00.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:00 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4197377201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-10T11:11:00.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:00 smithi189 ceph-mon[34842]: pgmap v429: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-10T11:11:00.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:00 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4197377201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-10T11:11:00.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.832+0000 7f4d5c800700 1 -- 172.21.15.27:0/1687207428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d5406bb40 msgr2=0x7f4d54110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:00.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.832+0000 7f4d5c800700 1 --2- 172.21.15.27:0/1687207428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d5406bb40 0x7f4d54110330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000b0a0 tx=0x7f4d5000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:11:00.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.833+0000 7f4d5c800700 1 -- 172.21.15.27:0/1687207428 shutdown_connections 2024-05-10T11:11:00.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.833+0000 7f4d5c800700 1 --2- 172.21.15.27:0/1687207428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d5406bb40 0x7f4d54110330 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:00.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.833+0000 7f4d5c800700 1 --2- 172.21.15.27:0/1687207428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d54109510 0x7f4d541098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:00.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.833+0000 7f4d5c800700 1 --2- 172.21.15.27:0/1687207428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d5406b180 0x7f4d5406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:00.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.833+0000 7f4d5c800700 1 -- 172.21.15.27:0/1687207428 >> 172.21.15.27:0/1687207428 conn(0x7f4d54079350 msgr2=0x7f4d54079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:00.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.833+0000 7f4d5c800700 1 -- 172.21.15.27:0/1687207428 shutdown_connections 2024-05-10T11:11:00.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.833+0000 7f4d5c800700 1 -- 172.21.15.27:0/1687207428 wait complete. 2024-05-10T11:11:00.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.834+0000 7f4d5c800700 1 Processor -- start 2024-05-10T11:11:00.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.835+0000 7f4d5c800700 1 -- start start 2024-05-10T11:11:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.835+0000 7f4d5c800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d5406b180 0x7f4d5419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.835+0000 7f4d5c800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d5406bb40 0x7f4d5419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.835+0000 7f4d5c800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d54109510 0x7f4d54199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.835+0000 7f4d5c800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4d54113690 con 0x7f4d54109510 2024-05-10T11:11:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.835+0000 7f4d5c800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4d54113510 con 0x7f4d5406b180 2024-05-10T11:11:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.835+0000 7f4d5c800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4d54113810 con 0x7f4d5406bb40 2024-05-10T11:11:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.835+0000 7f4d5a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d54109510 0x7f4d54199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.835+0000 7f4d5a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d54109510 0x7f4d54199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41178/0 (socket says 172.21.15.27:41178) 2024-05-10T11:11:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.835+0000 7f4d5a800700 1 -- 172.21.15.27:0/4197377201 learned_addr learned my addr 172.21.15.27:0/4197377201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.835+0000 7f4d59400700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d5406bb40 0x7f4d5419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.836+0000 7f4d59e00700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d5406b180 0x7f4d5419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.836+0000 7f4d5a800700 1 -- 172.21.15.27:0/4197377201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d5406bb40 msgr2=0x7f4d5419fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:00.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.836+0000 7f4d5a800700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d5406bb40 0x7f4d5419fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:00.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.836+0000 7f4d5a800700 1 -- 172.21.15.27:0/4197377201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d5406b180 msgr2=0x7f4d5419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:00.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.836+0000 7f4d5a800700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d5406b180 0x7f4d5419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:00.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.836+0000 7f4d5a800700 1 -- 172.21.15.27:0/4197377201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5419a0d0 con 0x7f4d54109510 2024-05-10T11:11:00.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.836+0000 7f4d59400700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d5406bb40 0x7f4d5419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:00.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.836+0000 7f4d5a800700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d54109510 0x7f4d54199970 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000b070 tx=0x7f4d50009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:00.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.837+0000 7f4d4ea00700 1 -- 172.21.15.27:0/4197377201 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d50021070 con 0x7f4d54109510 2024-05-10T11:11:00.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.837+0000 7f4d5c800700 1 -- 172.21.15.27:0/4197377201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5419a360 con 0x7f4d54109510 2024-05-10T11:11:00.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.837+0000 7f4d4ea00700 1 -- 172.21.15.27:0/4197377201 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d50004e40 con 0x7f4d54109510 2024-05-10T11:11:00.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.837+0000 7f4d4ea00700 1 -- 172.21.15.27:0/4197377201 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d50004630 con 0x7f4d54109510 2024-05-10T11:11:00.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.837+0000 7f4d5c800700 1 -- 172.21.15.27:0/4197377201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d541a6680 con 0x7f4d54109510 2024-05-10T11:11:00.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.839+0000 7f4d5c800700 1 -- 172.21.15.27:0/4197377201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d5406a190 con 0x7f4d54109510 2024-05-10T11:11:00.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.841+0000 7f4d4ea00700 1 -- 172.21.15.27:0/4197377201 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f4d50007980 con 0x7f4d54109510 2024-05-10T11:11:00.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.842+0000 7f4d4ea00700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4d3c0b2880 0x7f4d3c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:00.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.842+0000 7f4d4ea00700 1 -- 172.21.15.27:0/4197377201 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f4d5000f020 con 0x7f4d54109510 2024-05-10T11:11:00.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.843+0000 7f4d59e00700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4d3c0b2880 0x7f4d3c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:00.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.843+0000 7f4d59e00700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4d3c0b2880 0x7f4d3c0b4d40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009680 tx=0x7f4d48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:00.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.851+0000 7f4d4ea00700 1 -- 172.21.15.27:0/4197377201 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4d5002a020 con 0x7f4d54109510 2024-05-10T11:11:00.897 INFO:teuthology.orchestra.run.smithi027.stdout:253403070531 2024-05-10T11:11:00.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.098+0000 7f4d5c800700 1 -- 172.21.15.27:0/4197377201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f4d5402e510 con 0x7f4d54109510 2024-05-10T11:11:00.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.099+0000 7f4d4ea00700 1 -- 172.21.15.27:0/4197377201 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4d500074e0 con 0x7f4d54109510 2024-05-10T11:11:00.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.102+0000 7f4d5c800700 1 -- 172.21.15.27:0/4197377201 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4d3c0b2880 msgr2=0x7f4d3c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:00.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.102+0000 7f4d5c800700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4d3c0b2880 0x7f4d3c0b4d40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009680 tx=0x7f4d48009260 comp rx=0 tx=0).stop 2024-05-10T11:11:00.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.102+0000 7f4d5c800700 1 -- 172.21.15.27:0/4197377201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d54109510 msgr2=0x7f4d54199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:00.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.102+0000 7f4d5c800700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d54109510 0x7f4d54199970 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000b070 tx=0x7f4d50009c90 comp rx=0 tx=0).stop 2024-05-10T11:11:00.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.102+0000 7f4d5c800700 1 -- 172.21.15.27:0/4197377201 shutdown_connections 2024-05-10T11:11:00.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.102+0000 7f4d5c800700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4d3c0b2880 0x7f4d3c0b4d40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.102+0000 7f4d5c800700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d54109510 0x7f4d54199970 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.102+0000 7f4d5c800700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d5406b180 0x7f4d5419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.102+0000 7f4d5c800700 1 --2- 172.21.15.27:0/4197377201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d5406bb40 0x7f4d5419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.102+0000 7f4d5c800700 1 -- 172.21.15.27:0/4197377201 >> 172.21.15.27:0/4197377201 conn(0x7f4d54079350 msgr2=0x7f4d54101040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.103+0000 7f4d5c800700 1 -- 172.21.15.27:0/4197377201 shutdown_connections 2024-05-10T11:11:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.103+0000 7f4d5c800700 1 -- 172.21.15.27:0/4197377201 wait complete. 2024-05-10T11:11:01.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.959+0000 7fd2cfe00700 1 -- 172.21.15.27:0/1099705660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d0104f00 msgr2=0x7fd2d010d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:01.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.959+0000 7fd2cfe00700 1 --2- 172.21.15.27:0/1099705660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d0104f00 0x7fd2d010d3f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009990 tx=0x7fd2bc009ca0 comp rx=0 tx=0).stop 2024-05-10T11:11:01.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.960+0000 7fd2cfe00700 1 -- 172.21.15.27:0/1099705660 shutdown_connections 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.960+0000 7fd2cfe00700 1 --2- 172.21.15.27:0/1099705660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d01045e0 0x7fd2d01049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.960+0000 7fd2cfe00700 1 --2- 172.21.15.27:0/1099705660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d010da10 0x7fd2d010fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.960+0000 7fd2cfe00700 1 --2- 172.21.15.27:0/1099705660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d0104f00 0x7fd2d010d3f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.960+0000 7fd2cfe00700 1 -- 172.21.15.27:0/1099705660 >> 172.21.15.27:0/1099705660 conn(0x7fd2d00fde30 msgr2=0x7fd2d0100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.960+0000 7fd2cfe00700 1 -- 172.21.15.27:0/1099705660 shutdown_connections 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.960+0000 7fd2cfe00700 1 -- 172.21.15.27:0/1099705660 wait complete. 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.961+0000 7fd2cfe00700 1 Processor -- start 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.961+0000 7fd2cfe00700 1 -- start start 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.961+0000 7fd2cfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d01045e0 0x7fd2d019bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.961+0000 7fd2cfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d0104f00 0x7fd2d019c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2cfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d010da10 0x7fd2d01a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2cfe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d0113630 con 0x7fd2d010da10 2024-05-10T11:11:01.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2cfe00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d01134b0 con 0x7fd2d0104f00 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2cfe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d01137b0 con 0x7fd2d01045e0 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2ce000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d0104f00 0x7fd2d019c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2ce000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d0104f00 0x7fd2d019c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:59176/0 (socket says 172.21.15.27:59176) 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2ce000700 1 -- 172.21.15.27:0/4220937430 learned_addr learned my addr 172.21.15.27:0/4220937430 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2cf400700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d010da10 0x7fd2d01a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2cea00700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d01045e0 0x7fd2d019bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2ce000700 1 -- 172.21.15.27:0/4220937430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d01045e0 msgr2=0x7fd2d019bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2ce000700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d01045e0 0x7fd2d019bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2ce000700 1 -- 172.21.15.27:0/4220937430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d010da10 msgr2=0x7fd2d01a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2ce000700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d010da10 0x7fd2d01a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.962+0000 7fd2ce000700 1 -- 172.21.15.27:0/4220937430 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d01a0dd0 con 0x7fd2d0104f00 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.963+0000 7fd2ce000700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d0104f00 0x7fd2d019c300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc000c00 tx=0x7fd2bc0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:01.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.963+0000 7fd2d4a00700 1 -- 172.21.15.27:0/4220937430 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc026070 con 0x7fd2d0104f00 2024-05-10T11:11:01.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.963+0000 7fd2d4a00700 1 -- 172.21.15.27:0/4220937430 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2bc021410 con 0x7fd2d0104f00 2024-05-10T11:11:01.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.963+0000 7fd2d4a00700 1 -- 172.21.15.27:0/4220937430 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc017650 con 0x7fd2d0104f00 2024-05-10T11:11:01.264 INFO:teuthology.orchestra.run.smithi027.stdout:77309411436 2024-05-10T11:11:01.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.963+0000 7fd2cfe00700 1 -- 172.21.15.27:0/4220937430 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d01a1060 con 0x7fd2d0104f00 2024-05-10T11:11:01.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.963+0000 7fd2cfe00700 1 -- 172.21.15.27:0/4220937430 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d01a8940 con 0x7fd2d0104f00 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.964+0000 7fd2cfe00700 1 -- 172.21.15.27:0/4220937430 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2d006c2f0 con 0x7fd2d0104f00 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.967+0000 7fd2d4a00700 1 -- 172.21.15.27:0/4220937430 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fd2bc029070 con 0x7fd2d0104f00 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.969+0000 7fd2d4a00700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd2b40b2900 0x7fd2b40b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.969+0000 7fd2d4a00700 1 -- 172.21.15.27:0/4220937430 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fd2bc0e3490 con 0x7fd2d0104f00 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.969+0000 7fd2cea00700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd2b40b2900 0x7fd2b40b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.970+0000 7fd2cea00700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd2b40b2900 0x7fd2b40b4dc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009c60 tx=0x7fd2c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:10:59.970+0000 7fd2d4a00700 1 -- 172.21.15.27:0/4220937430 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fd2bc013070 con 0x7fd2d0104f00 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.224+0000 7fd2cfe00700 1 -- 172.21.15.27:0/4220937430 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd2d019d150 con 0x7fd2d0104f00 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.224+0000 7fd2d4a00700 1 -- 172.21.15.27:0/4220937430 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd2d019d150 con 0x7fd2d0104f00 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.228+0000 7fd2cfe00700 1 -- 172.21.15.27:0/4220937430 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd2b40b2900 msgr2=0x7fd2b40b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.228+0000 7fd2cfe00700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd2b40b2900 0x7fd2b40b4dc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009c60 tx=0x7fd2c0009260 comp rx=0 tx=0).stop 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.228+0000 7fd2cfe00700 1 -- 172.21.15.27:0/4220937430 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d0104f00 msgr2=0x7fd2d019c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.228+0000 7fd2cfe00700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d0104f00 0x7fd2d019c300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc000c00 tx=0x7fd2bc0047b0 comp rx=0 tx=0).stop 2024-05-10T11:11:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.228+0000 7fd2cfe00700 1 -- 172.21.15.27:0/4220937430 shutdown_connections 2024-05-10T11:11:01.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.228+0000 7fd2cfe00700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd2b40b2900 0x7fd2b40b4dc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:01.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.228+0000 7fd2cfe00700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d01045e0 0x7fd2d019bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:01.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.228+0000 7fd2cfe00700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d010da10 0x7fd2d01a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:01.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.228+0000 7fd2cfe00700 1 --2- 172.21.15.27:0/4220937430 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d0104f00 0x7fd2d019c300 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:01.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.228+0000 7fd2cfe00700 1 -- 172.21.15.27:0/4220937430 >> 172.21.15.27:0/4220937430 conn(0x7fd2d00fde30 msgr2=0x7fd2d0100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:01.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.228+0000 7fd2cfe00700 1 -- 172.21.15.27:0/4220937430 shutdown_connections 2024-05-10T11:11:01.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:00.228+0000 7fd2cfe00700 1 -- 172.21.15.27:0/4220937430 wait complete. 2024-05-10T11:11:01.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:01 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4220937430' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-10T11:11:01.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:01 smithi076 ceph-mon[34152]: pgmap v430: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-10T11:11:01.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:01 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4220937430' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-10T11:11:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:01 smithi189 ceph-mon[34842]: pgmap v430: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-10T11:11:01.547 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:01 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4220937430' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-10T11:11:01.547 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:01 smithi027 ceph-mon[30054]: pgmap v430: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-10T11:11:02.165 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182498 got 107374182502 for osd.3 2024-05-10T11:11:02.166 DEBUG:teuthology.parallel:result is None 2024-05-10T11:11:03.315 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678067 got 304942678072 for osd.11 2024-05-10T11:11:03.316 DEBUG:teuthology.parallel:result is None 2024-05-10T11:11:03.489 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:04.025 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:04.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:03 smithi027 ceph-mon[30054]: pgmap v431: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-10T11:11:04.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:03 smithi076 ceph-mon[34152]: pgmap v431: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-10T11:11:04.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:03 smithi189 ceph-mon[34842]: pgmap v431: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-10T11:11:04.314 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822741 got 154618822747 for osd.5 2024-05-10T11:11:04.315 DEBUG:teuthology.parallel:result is None 2024-05-10T11:11:04.716 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411432 got 77309411436 for osd.2 2024-05-10T11:11:04.716 DEBUG:teuthology.parallel:result is None 2024-05-10T11:11:04.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.015+0000 7f58b1800700 1 -- 172.21.15.27:0/2869868396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac1033d0 msgr2=0x7f58ac103850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:04.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.015+0000 7f58b1800700 1 --2- 172.21.15.27:0/2869868396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac1033d0 0x7f58ac103850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f58980098f0 tx=0x7f5898009c00 comp rx=0 tx=0).stop 2024-05-10T11:11:04.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.015+0000 7f58b1800700 1 -- 172.21.15.27:0/2869868396 shutdown_connections 2024-05-10T11:11:04.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.015+0000 7f58b1800700 1 --2- 172.21.15.27:0/2869868396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ac103d90 0x7f58ac1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.015+0000 7f58b1800700 1 --2- 172.21.15.27:0/2869868396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58ac109460 0x7f58ac109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.015+0000 7f58b1800700 1 --2- 172.21.15.27:0/2869868396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac1033d0 0x7f58ac103850 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.015+0000 7f58b1800700 1 -- 172.21.15.27:0/2869868396 >> 172.21.15.27:0/2869868396 conn(0x7f58ac0fef20 msgr2=0x7f58ac101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.015+0000 7f58b1800700 1 -- 172.21.15.27:0/2869868396 shutdown_connections 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.016+0000 7f58b1800700 1 -- 172.21.15.27:0/2869868396 wait complete. 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.017+0000 7f58b1800700 1 Processor -- start 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.017+0000 7f58b1800700 1 -- start start 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.018+0000 7f58b1800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac1033d0 0x7f58ac0757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.018+0000 7f58b1800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58ac103d90 0x7f58ac075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.018+0000 7f58b1800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ac109460 0x7f58ac076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.018+0000 7f58b1800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f58ac1135e0 con 0x7f58ac109460 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.018+0000 7f58b1800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f58ac113460 con 0x7f58ac103d90 2024-05-10T11:11:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.018+0000 7f58b1800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f58ac113760 con 0x7f58ac1033d0 2024-05-10T11:11:04.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.018+0000 7f58abe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac1033d0 0x7f58ac0757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:04.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.018+0000 7f58abe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac1033d0 0x7f58ac0757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:53692/0 (socket says 172.21.15.27:53692) 2024-05-10T11:11:04.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.018+0000 7f58ab400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58ac103d90 0x7f58ac075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:04.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.018+0000 7f58abe00700 1 -- 172.21.15.27:0/1037860122 learned_addr learned my addr 172.21.15.27:0/1037860122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:04.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.018+0000 7f58b0e00700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ac109460 0x7f58ac076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:04.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.019+0000 7f58ab400700 1 -- 172.21.15.27:0/1037860122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac1033d0 msgr2=0x7f58ac0757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:04.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.019+0000 7f58ab400700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac1033d0 0x7f58ac0757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:04.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.019+0000 7f58ab400700 1 -- 172.21.15.27:0/1037860122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ac109460 msgr2=0x7f58ac076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:04.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.019+0000 7f58ab400700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ac109460 0x7f58ac076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:04.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.019+0000 7f58ab400700 1 -- 172.21.15.27:0/1037860122 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58ac1a80e0 con 0x7f58ac103d90 2024-05-10T11:11:04.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.019+0000 7f58ab400700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58ac103d90 0x7f58ac075ce0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5898009810 tx=0x7f5898005340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:04.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.019+0000 7f58a8c00700 1 -- 172.21.15.27:0/1037860122 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589800b760 con 0x7f58ac103d90 2024-05-10T11:11:04.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.020+0000 7f58a8c00700 1 -- 172.21.15.27:0/1037860122 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5898003680 con 0x7f58ac103d90 2024-05-10T11:11:04.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.020+0000 7f58b1800700 1 -- 172.21.15.27:0/1037860122 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58ac1a8370 con 0x7f58ac103d90 2024-05-10T11:11:04.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.020+0000 7f58a8c00700 1 -- 172.21.15.27:0/1037860122 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898017730 con 0x7f58ac103d90 2024-05-10T11:11:04.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.020+0000 7f58b1800700 1 -- 172.21.15.27:0/1037860122 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58ac1a8820 con 0x7f58ac103d90 2024-05-10T11:11:04.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.021+0000 7f58b1800700 1 -- 172.21.15.27:0/1037860122 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58ac06a190 con 0x7f58ac103d90 2024-05-10T11:11:04.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.025+0000 7f58a8c00700 1 -- 172.21.15.27:0/1037860122 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f5898003ac0 con 0x7f58ac103d90 2024-05-10T11:11:04.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.026+0000 7f58a8c00700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f58900b2900 0x7f58900b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:04.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.026+0000 7f58a8c00700 1 -- 172.21.15.27:0/1037860122 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f58980e37a0 con 0x7f58ac103d90 2024-05-10T11:11:04.864 INFO:teuthology.orchestra.run.smithi027.stdout:279172874303 2024-05-10T11:11:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.027+0000 7f58abe00700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f58900b2900 0x7f58900b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.027+0000 7f58abe00700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f58900b2900 0x7f58900b4dc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f589c009ce0 tx=0x7f589c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.031+0000 7f58a8c00700 1 -- 172.21.15.27:0/1037860122 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5898013030 con 0x7f58ac103d90 2024-05-10T11:11:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.276+0000 7f58b1800700 1 -- 172.21.15.27:0/1037860122 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f58ac02e510 con 0x7f58ac103d90 2024-05-10T11:11:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.277+0000 7f58a8c00700 1 -- 172.21.15.27:0/1037860122 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f58980ab100 con 0x7f58ac103d90 2024-05-10T11:11:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.280+0000 7f58b1800700 1 -- 172.21.15.27:0/1037860122 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f58900b2900 msgr2=0x7f58900b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.280+0000 7f58b1800700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f58900b2900 0x7f58900b4dc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f589c009ce0 tx=0x7f589c009260 comp rx=0 tx=0).stop 2024-05-10T11:11:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.280+0000 7f58b1800700 1 -- 172.21.15.27:0/1037860122 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58ac103d90 msgr2=0x7f58ac075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.280+0000 7f58b1800700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58ac103d90 0x7f58ac075ce0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5898009810 tx=0x7f5898005340 comp rx=0 tx=0).stop 2024-05-10T11:11:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.280+0000 7f58b1800700 1 -- 172.21.15.27:0/1037860122 shutdown_connections 2024-05-10T11:11:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.280+0000 7f58b1800700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f58900b2900 0x7f58900b4dc0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.280+0000 7f58b1800700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac1033d0 0x7f58ac0757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.280+0000 7f58b1800700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ac109460 0x7f58ac076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.280+0000 7f58b1800700 1 --2- 172.21.15.27:0/1037860122 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58ac103d90 0x7f58ac075ce0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.280+0000 7f58b1800700 1 -- 172.21.15.27:0/1037860122 >> 172.21.15.27:0/1037860122 conn(0x7f58ac0fef20 msgr2=0x7f58ac0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.281+0000 7f58b1800700 1 -- 172.21.15.27:0/1037860122 shutdown_connections 2024-05-10T11:11:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:04.281+0000 7f58b1800700 1 -- 172.21.15.27:0/1037860122 wait complete. 2024-05-10T11:11:05.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:04 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1037860122' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-10T11:11:05.229 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:04 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1037860122' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-10T11:11:05.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:04 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1037860122' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-10T11:11:06.163 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070524 got 253403070531 for osd.9 2024-05-10T11:11:06.163 DEBUG:teuthology.parallel:result is None 2024-05-10T11:11:06.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:05 smithi027 ceph-mon[30054]: pgmap v432: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-05-10T11:11:06.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:05 smithi189 ceph-mon[34842]: pgmap v432: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-05-10T11:11:06.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:05 smithi076 ceph-mon[34152]: pgmap v432: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-05-10T11:11:06.478 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874297 got 279172874303 for osd.10 2024-05-10T11:11:06.478 DEBUG:teuthology.parallel:result is None 2024-05-10T11:11:07.071 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266756 got 227633266762 for osd.8 2024-05-10T11:11:07.071 DEBUG:teuthology.parallel:result is None 2024-05-10T11:11:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.309+0000 7f713be00700 1 -- 172.21.15.27:0/1698027421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f713c1033d0 msgr2=0x7f713c103850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.309+0000 7f713be00700 1 --2- 172.21.15.27:0/1698027421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f713c1033d0 0x7f713c103850 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7128009990 tx=0x7f7128009ca0 comp rx=0 tx=0).stop 2024-05-10T11:11:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.310+0000 7f713be00700 1 -- 172.21.15.27:0/1698027421 shutdown_connections 2024-05-10T11:11:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.310+0000 7f713be00700 1 --2- 172.21.15.27:0/1698027421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f713c103d90 0x7f713c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.310+0000 7f713be00700 1 --2- 172.21.15.27:0/1698027421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713c109460 0x7f713c109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.310+0000 7f713be00700 1 --2- 172.21.15.27:0/1698027421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f713c1033d0 0x7f713c103850 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.310+0000 7f713be00700 1 -- 172.21.15.27:0/1698027421 >> 172.21.15.27:0/1698027421 conn(0x7f713c0fef20 msgr2=0x7f713c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.310+0000 7f713be00700 1 -- 172.21.15.27:0/1698027421 shutdown_connections 2024-05-10T11:11:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.310+0000 7f713be00700 1 -- 172.21.15.27:0/1698027421 wait complete. 2024-05-10T11:11:07.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.311+0000 7f713be00700 1 Processor -- start 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.311+0000 7f713be00700 1 -- start start 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.311+0000 7f713be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713c1033d0 0x7f713c1a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.311+0000 7f713be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f713c103d90 0x7f713c1a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.311+0000 7f713be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f713c109460 0x7f713c19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.311+0000 7f713be00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f713c1135c0 con 0x7f713c103d90 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.311+0000 7f713be00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f713c113440 con 0x7f713c1033d0 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.311+0000 7f713be00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f713c113740 con 0x7f713c109460 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.312+0000 7f713aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713c1033d0 0x7f713c1a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.312+0000 7f713aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713c1033d0 0x7f713c1a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35288/0 (socket says 172.21.15.27:35288) 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.312+0000 7f713aa00700 1 -- 172.21.15.27:0/443727024 learned_addr learned my addr 172.21.15.27:0/443727024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.312+0000 7f713a000700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f713c103d90 0x7f713c1a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.312+0000 7f713b400700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f713c109460 0x7f713c19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.312+0000 7f713aa00700 1 -- 172.21.15.27:0/443727024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f713c109460 msgr2=0x7f713c19bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.312+0000 7f713aa00700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f713c109460 0x7f713c19bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:07.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.312+0000 7f713aa00700 1 -- 172.21.15.27:0/443727024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f713c103d90 msgr2=0x7f713c1a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:07.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.312+0000 7f713aa00700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f713c103d90 0x7f713c1a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:07.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.312+0000 7f713aa00700 1 -- 172.21.15.27:0/443727024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713c19c230 con 0x7f713c1033d0 2024-05-10T11:11:07.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.313+0000 7f713aa00700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713c1033d0 0x7f713c1a1910 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f712c00e9a0 tx=0x7f712c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:07.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.313+0000 7f7140a00700 1 -- 172.21.15.27:0/443727024 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c00cbd0 con 0x7f713c1033d0 2024-05-10T11:11:07.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.313+0000 7f7140a00700 1 -- 172.21.15.27:0/443727024 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f712c00cd70 con 0x7f713c1033d0 2024-05-10T11:11:07.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.313+0000 7f713be00700 1 -- 172.21.15.27:0/443727024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f713c19c520 con 0x7f713c1033d0 2024-05-10T11:11:07.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.313+0000 7f7140a00700 1 -- 172.21.15.27:0/443727024 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c018680 con 0x7f713c1033d0 2024-05-10T11:11:07.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.313+0000 7f713be00700 1 -- 172.21.15.27:0/443727024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f713c1a87e0 con 0x7f713c1033d0 2024-05-10T11:11:07.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.314+0000 7f713be00700 1 -- 172.21.15.27:0/443727024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f713c06a190 con 0x7f713c1033d0 2024-05-10T11:11:07.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.317+0000 7f7140a00700 1 -- 172.21.15.27:0/443727024 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f712c010430 con 0x7f713c1033d0 2024-05-10T11:11:07.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.318+0000 7f7140a00700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f71200b2880 0x7f71200b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:07.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.319+0000 7f7140a00700 1 -- 172.21.15.27:0/443727024 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f712c0e2d30 con 0x7f713c1033d0 2024-05-10T11:11:07.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.319+0000 7f713a000700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f71200b2880 0x7f71200b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:07.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.319+0000 7f713a000700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f71200b2880 0x7f71200b4d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f713c110770 tx=0x7f7128005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:07.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.320+0000 7f7140a00700 1 -- 172.21.15.27:0/443727024 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f712c0aa600 con 0x7f713c1033d0 2024-05-10T11:11:07.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.569+0000 7f713be00700 1 -- 172.21.15.27:0/443727024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f713c1a8a10 con 0x7f713c1033d0 2024-05-10T11:11:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.569+0000 7f7140a00700 1 -- 172.21.15.27:0/443727024 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f712c0cc090 con 0x7f713c1033d0 2024-05-10T11:11:07.570 INFO:teuthology.orchestra.run.smithi027.stdout:34359738489 2024-05-10T11:11:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.573+0000 7f713be00700 1 -- 172.21.15.27:0/443727024 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f71200b2880 msgr2=0x7f71200b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.573+0000 7f713be00700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f71200b2880 0x7f71200b4d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f713c110770 tx=0x7f7128005fd0 comp rx=0 tx=0).stop 2024-05-10T11:11:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.573+0000 7f713be00700 1 -- 172.21.15.27:0/443727024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713c1033d0 msgr2=0x7f713c1a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.573+0000 7f713be00700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713c1033d0 0x7f713c1a1910 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f712c00e9a0 tx=0x7f712c00ecb0 comp rx=0 tx=0).stop 2024-05-10T11:11:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.573+0000 7f713be00700 1 -- 172.21.15.27:0/443727024 shutdown_connections 2024-05-10T11:11:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.573+0000 7f713be00700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f71200b2880 0x7f71200b4d40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.573+0000 7f713be00700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f713c109460 0x7f713c19bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.573+0000 7f713be00700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713c1033d0 0x7f713c1a1910 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.573+0000 7f713be00700 1 --2- 172.21.15.27:0/443727024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f713c103d90 0x7f713c1a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.573+0000 7f713be00700 1 -- 172.21.15.27:0/443727024 >> 172.21.15.27:0/443727024 conn(0x7f713c0fef20 msgr2=0x7f713c0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.573+0000 7f713be00700 1 -- 172.21.15.27:0/443727024 shutdown_connections 2024-05-10T11:11:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:07.573+0000 7f713be00700 1 -- 172.21.15.27:0/443727024 wait complete. 2024-05-10T11:11:08.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:07 smithi189 ceph-mon[34842]: pgmap v433: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-10T11:11:08.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:07 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/443727024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-10T11:11:08.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:07 smithi027 ceph-mon[30054]: pgmap v433: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-10T11:11:08.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:07 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/443727024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-10T11:11:08.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:07 smithi076 ceph-mon[34152]: pgmap v433: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-10T11:11:08.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:07 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/443727024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-10T11:11:08.759 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738483 got 34359738489 for osd.0 2024-05-10T11:11:08.760 DEBUG:teuthology.parallel:result is None 2024-05-10T11:11:08.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.990+0000 7f4c24a00700 1 -- 172.21.15.27:0/36314410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c2010b6c0 msgr2=0x7f4c2010baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.990+0000 7f4c24a00700 1 --2- 172.21.15.27:0/36314410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c2010b6c0 0x7f4c2010baa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009990 tx=0x7f4c10009ca0 comp rx=0 tx=0).stop 2024-05-10T11:11:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.990+0000 7f4c24a00700 1 -- 172.21.15.27:0/36314410 shutdown_connections 2024-05-10T11:11:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.990+0000 7f4c24a00700 1 --2- 172.21.15.27:0/36314410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c2010b6c0 0x7f4c2010baa0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.990+0000 7f4c24a00700 1 --2- 172.21.15.27:0/36314410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c20106080 0x7f4c20112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.990+0000 7f4c24a00700 1 --2- 172.21.15.27:0/36314410 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c201056c0 0x7f4c20105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.990+0000 7f4c24a00700 1 -- 172.21.15.27:0/36314410 >> 172.21.15.27:0/36314410 conn(0x7f4c201011b0 msgr2=0x7f4c201035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.991+0000 7f4c24a00700 1 -- 172.21.15.27:0/36314410 shutdown_connections 2024-05-10T11:11:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.991+0000 7f4c24a00700 1 -- 172.21.15.27:0/36314410 wait complete. 2024-05-10T11:11:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.992+0000 7f4c24a00700 1 Processor -- start 2024-05-10T11:11:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.992+0000 7f4c24a00700 1 -- start start 2024-05-10T11:11:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.992+0000 7f4c24a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c201056c0 0x7f4c2019bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.993+0000 7f4c24a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c20106080 0x7f4c2019c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.993+0000 7f4c24a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c2010b6c0 0x7f4c201a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.993+0000 7f4c24a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4c201158a0 con 0x7f4c201056c0 2024-05-10T11:11:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.993+0000 7f4c24a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4c20115720 con 0x7f4c20106080 2024-05-10T11:11:08.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.993+0000 7f4c24a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4c20115a20 con 0x7f4c2010b6c0 2024-05-10T11:11:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.993+0000 7f4c1de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c201056c0 0x7f4c2019bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.993+0000 7f4c1d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c20106080 0x7f4c2019c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.993+0000 7f4c1d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c20106080 0x7f4c2019c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35314/0 (socket says 172.21.15.27:35314) 2024-05-10T11:11:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.993+0000 7f4c1de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c201056c0 0x7f4c2019bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60812/0 (socket says 172.21.15.27:60812) 2024-05-10T11:11:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.993+0000 7f4c1d400700 1 -- 172.21.15.27:0/1386207899 learned_addr learned my addr 172.21.15.27:0/1386207899 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.993+0000 7f4c1e800700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c2010b6c0 0x7f4c201a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.994+0000 7f4c1e800700 1 -- 172.21.15.27:0/1386207899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c20106080 msgr2=0x7f4c2019c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.994+0000 7f4c1e800700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c20106080 0x7f4c2019c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.994+0000 7f4c1e800700 1 -- 172.21.15.27:0/1386207899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c201056c0 msgr2=0x7f4c2019bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.994+0000 7f4c1e800700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c201056c0 0x7f4c2019bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.994+0000 7f4c1e800700 1 -- 172.21.15.27:0/1386207899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c201a0d20 con 0x7f4c2010b6c0 2024-05-10T11:11:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.994+0000 7f4c1d400700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c20106080 0x7f4c2019c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.994+0000 7f4c1de00700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c201056c0 0x7f4c2019bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.994+0000 7f4c1e800700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c2010b6c0 0x7f4c201a05e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800efc0 tx=0x7f4c1800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.994+0000 7f4c16a00700 1 -- 172.21.15.27:0/1386207899 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c180092e0 con 0x7f4c2010b6c0 2024-05-10T11:11:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.994+0000 7f4c16a00700 1 -- 172.21.15.27:0/1386207899 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c18004bb0 con 0x7f4c2010b6c0 2024-05-10T11:11:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.994+0000 7f4c16a00700 1 -- 172.21.15.27:0/1386207899 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c18004020 con 0x7f4c2010b6c0 2024-05-10T11:11:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.995+0000 7f4c24a00700 1 -- 172.21.15.27:0/1386207899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c201a0fb0 con 0x7f4c2010b6c0 2024-05-10T11:11:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.995+0000 7f4c24a00700 1 -- 172.21.15.27:0/1386207899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c201a8890 con 0x7f4c2010b6c0 2024-05-10T11:11:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.996+0000 7f4c24a00700 1 -- 172.21.15.27:0/1386207899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c2010dc50 con 0x7f4c2010b6c0 2024-05-10T11:11:08.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:08.998+0000 7f4c16a00700 1 -- 172.21.15.27:0/1386207899 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f4c1801b030 con 0x7f4c2010b6c0 2024-05-10T11:11:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.000+0000 7f4c16a00700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4c040b2900 0x7f4c040b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.000+0000 7f4c16a00700 1 -- 172.21.15.27:0/1386207899 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f4c180e2070 con 0x7f4c2010b6c0 2024-05-10T11:11:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.000+0000 7f4c1de00700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4c040b2900 0x7f4c040b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:09.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.001+0000 7f4c1de00700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4c040b2900 0x7f4c040b4dc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4c100098e0 tx=0x7f4c10005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:09.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.004+0000 7f4c16a00700 1 -- 172.21.15.27:0/1386207899 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4c18022020 con 0x7f4c2010b6c0 2024-05-10T11:11:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.252+0000 7f4c24a00700 1 -- 172.21.15.27:0/1386207899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f4c2010d630 con 0x7f4c2010b6c0 2024-05-10T11:11:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.253+0000 7f4c16a00700 1 -- 172.21.15.27:0/1386207899 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4c18010040 con 0x7f4c2010b6c0 2024-05-10T11:11:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:133143986275 2024-05-10T11:11:09.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.256+0000 7f4c24a00700 1 -- 172.21.15.27:0/1386207899 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4c040b2900 msgr2=0x7f4c040b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:09.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.256+0000 7f4c24a00700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4c040b2900 0x7f4c040b4dc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4c100098e0 tx=0x7f4c10005960 comp rx=0 tx=0).stop 2024-05-10T11:11:09.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.256+0000 7f4c24a00700 1 -- 172.21.15.27:0/1386207899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c2010b6c0 msgr2=0x7f4c201a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:09.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.256+0000 7f4c24a00700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c2010b6c0 0x7f4c201a05e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800efc0 tx=0x7f4c1800c3a0 comp rx=0 tx=0).stop 2024-05-10T11:11:09.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.256+0000 7f4c24a00700 1 -- 172.21.15.27:0/1386207899 shutdown_connections 2024-05-10T11:11:09.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.256+0000 7f4c24a00700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f4c040b2900 0x7f4c040b4dc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:09.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.256+0000 7f4c24a00700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c2010b6c0 0x7f4c201a05e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:09.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.256+0000 7f4c24a00700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c201056c0 0x7f4c2019bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:09.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.256+0000 7f4c24a00700 1 --2- 172.21.15.27:0/1386207899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c20106080 0x7f4c2019c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:09.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.256+0000 7f4c24a00700 1 -- 172.21.15.27:0/1386207899 >> 172.21.15.27:0/1386207899 conn(0x7f4c201011b0 msgr2=0x7f4c20101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:09.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.256+0000 7f4c24a00700 1 -- 172.21.15.27:0/1386207899 shutdown_connections 2024-05-10T11:11:09.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:09.256+0000 7f4c24a00700 1 -- 172.21.15.27:0/1386207899 wait complete. 2024-05-10T11:11:10.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:09 smithi189 ceph-mon[34842]: pgmap v434: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:09 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1386207899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-10T11:11:10.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:09 smithi027 ceph-mon[30054]: pgmap v434: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:10.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:09 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1386207899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-10T11:11:10.421 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986267 got 133143986275 for osd.4 2024-05-10T11:11:10.421 DEBUG:teuthology.parallel:result is None 2024-05-10T11:11:10.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:09 smithi076 ceph-mon[34152]: pgmap v434: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:10.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:09 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1386207899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-10T11:11:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.094+0000 7f9182000700 1 -- 172.21.15.27:0/1272525619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f917c0756d0 msgr2=0x7f917c075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.094+0000 7f9182000700 1 --2- 172.21.15.27:0/1272525619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f917c0756d0 0x7f917c075ab0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f916c009f90 tx=0x7f916c00b330 comp rx=0 tx=0).stop 2024-05-10T11:11:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.095+0000 7f9182000700 1 -- 172.21.15.27:0/1272525619 shutdown_connections 2024-05-10T11:11:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.095+0000 7f9182000700 1 --2- 172.21.15.27:0/1272525619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917c114670 0x7f917c116a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.095+0000 7f9182000700 1 --2- 172.21.15.27:0/1272525619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f917c0756d0 0x7f917c075ab0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.095+0000 7f9182000700 1 --2- 172.21.15.27:0/1272525619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f917c075ff0 0x7f917c114130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.095+0000 7f9182000700 1 -- 172.21.15.27:0/1272525619 >> 172.21.15.27:0/1272525619 conn(0x7f917c07a600 msgr2=0x7f917c07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.095+0000 7f9182000700 1 -- 172.21.15.27:0/1272525619 shutdown_connections 2024-05-10T11:11:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.095+0000 7f9182000700 1 -- 172.21.15.27:0/1272525619 wait complete. 2024-05-10T11:11:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.097+0000 7f9182000700 1 Processor -- start 2024-05-10T11:11:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.097+0000 7f9182000700 1 -- start start 2024-05-10T11:11:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.097+0000 7f9182000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f917c0756d0 0x7f917c1a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.097+0000 7f9182000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917c075ff0 0x7f917c1a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.097+0000 7f9182000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f917c114670 0x7f917c1a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.097+0000 7f9182000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f917c119cd0 con 0x7f917c075ff0 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.097+0000 7f9182000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f917c119b50 con 0x7f917c114670 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.097+0000 7f9182000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f917c119e50 con 0x7f917c0756d0 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.098+0000 7f9172a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917c075ff0 0x7f917c1a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.098+0000 7f9172a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917c075ff0 0x7f917c1a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60840/0 (socket says 172.21.15.27:60840) 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.098+0000 7f9172a00700 1 -- 172.21.15.27:0/2269228450 learned_addr learned my addr 172.21.15.27:0/2269228450 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.098+0000 7f917be00700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f917c114670 0x7f917c1a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.098+0000 7f917b400700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f917c0756d0 0x7f917c1a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.098+0000 7f9172a00700 1 -- 172.21.15.27:0/2269228450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f917c0756d0 msgr2=0x7f917c1a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.098+0000 7f9172a00700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f917c0756d0 0x7f917c1a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.098+0000 7f9172a00700 1 -- 172.21.15.27:0/2269228450 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f917c114670 msgr2=0x7f917c1a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.098+0000 7f9172a00700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f917c114670 0x7f917c1a4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.098+0000 7f9172a00700 1 -- 172.21.15.27:0/2269228450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c1a5250 con 0x7f917c075ff0 2024-05-10T11:11:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.099+0000 7f9172a00700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917c075ff0 0x7f917c1a0780 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f916400e9f0 tx=0x7f916400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.099+0000 7f9178c00700 1 -- 172.21.15.27:0/2269228450 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164004a30 con 0x7f917c075ff0 2024-05-10T11:11:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.099+0000 7f9178c00700 1 -- 172.21.15.27:0/2269228450 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f916400cc70 con 0x7f917c075ff0 2024-05-10T11:11:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.099+0000 7f9182000700 1 -- 172.21.15.27:0/2269228450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f917c1a5540 con 0x7f917c075ff0 2024-05-10T11:11:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.099+0000 7f9178c00700 1 -- 172.21.15.27:0/2269228450 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164005540 con 0x7f917c075ff0 2024-05-10T11:11:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.099+0000 7f9182000700 1 -- 172.21.15.27:0/2269228450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f917c1acdc0 con 0x7f917c075ff0 2024-05-10T11:11:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.100+0000 7f9182000700 1 -- 172.21.15.27:0/2269228450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f917c06a190 con 0x7f917c075ff0 2024-05-10T11:11:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.105+0000 7f9178c00700 1 -- 172.21.15.27:0/2269228450 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f9164004bd0 con 0x7f917c075ff0 2024-05-10T11:11:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.107+0000 7f9178c00700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f91600b2900 0x7f91600b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.107+0000 7f9178c00700 1 -- 172.21.15.27:0/2269228450 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f91640e2890 con 0x7f917c075ff0 2024-05-10T11:11:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.107+0000 7f917b400700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f91600b2900 0x7f91600b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:12.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.107+0000 7f9178c00700 1 -- 172.21.15.27:0/2269228450 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9164054d70 con 0x7f917c075ff0 2024-05-10T11:11:12.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.108+0000 7f917b400700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f91600b2900 0x7f91600b4dc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f916c009da0 tx=0x7f916c0096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:12.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:11 smithi027 ceph-mon[30054]: pgmap v435: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:12.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:11 smithi189 ceph-mon[34842]: pgmap v435: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:12.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.356+0000 7f9182000700 1 -- 172.21.15.27:0/2269228450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f917c1ad050 con 0x7f917c075ff0 2024-05-10T11:11:12.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.357+0000 7f9178c00700 1 -- 172.21.15.27:0/2269228450 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f916400ce20 con 0x7f917c075ff0 2024-05-10T11:11:12.357 INFO:teuthology.orchestra.run.smithi027.stdout:180388626520 2024-05-10T11:11:12.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.360+0000 7f9182000700 1 -- 172.21.15.27:0/2269228450 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f91600b2900 msgr2=0x7f91600b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:12.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.360+0000 7f9182000700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f91600b2900 0x7f91600b4dc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f916c009da0 tx=0x7f916c0096c0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.360+0000 7f9182000700 1 -- 172.21.15.27:0/2269228450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917c075ff0 msgr2=0x7f917c1a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:12.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.360+0000 7f9182000700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917c075ff0 0x7f917c1a0780 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f916400e9f0 tx=0x7f916400ed00 comp rx=0 tx=0).stop 2024-05-10T11:11:12.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.360+0000 7f9182000700 1 -- 172.21.15.27:0/2269228450 shutdown_connections 2024-05-10T11:11:12.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.360+0000 7f9182000700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f91600b2900 0x7f91600b4dc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.360+0000 7f9182000700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f917c0756d0 0x7f917c1a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.360+0000 7f9182000700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f917c114670 0x7f917c1a4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.360+0000 7f9182000700 1 --2- 172.21.15.27:0/2269228450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917c075ff0 0x7f917c1a0780 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.360+0000 7f9182000700 1 -- 172.21.15.27:0/2269228450 >> 172.21.15.27:0/2269228450 conn(0x7f917c07a600 msgr2=0x7f917c1134a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:12.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.361+0000 7f9182000700 1 -- 172.21.15.27:0/2269228450 shutdown_connections 2024-05-10T11:11:12.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.361+0000 7f9182000700 1 -- 172.21.15.27:0/2269228450 wait complete. 2024-05-10T11:11:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.391+0000 7f9571a00700 1 -- 172.21.15.27:0/1673787052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c06bad0 msgr2=0x7f956c10fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.391+0000 7f9571a00700 1 --2- 172.21.15.27:0/1673787052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c06bad0 0x7f956c10fa10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9558009990 tx=0x7f9558009ca0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.391+0000 7f9571a00700 1 -- 172.21.15.27:0/1673787052 shutdown_connections 2024-05-10T11:11:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.391+0000 7f9571a00700 1 --2- 172.21.15.27:0/1673787052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f956c10ff50 0x7f956c112340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.391+0000 7f9571a00700 1 --2- 172.21.15.27:0/1673787052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f956c06b1b0 0x7f956c06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.391+0000 7f9571a00700 1 --2- 172.21.15.27:0/1673787052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c06bad0 0x7f956c10fa10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.391+0000 7f9571a00700 1 -- 172.21.15.27:0/1673787052 >> 172.21.15.27:0/1673787052 conn(0x7f956c079350 msgr2=0x7f956c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.391+0000 7f9571a00700 1 -- 172.21.15.27:0/1673787052 shutdown_connections 2024-05-10T11:11:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.391+0000 7f9571a00700 1 -- 172.21.15.27:0/1673787052 wait complete. 2024-05-10T11:11:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.392+0000 7f9571a00700 1 Processor -- start 2024-05-10T11:11:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.393+0000 7f9571a00700 1 -- start start 2024-05-10T11:11:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.393+0000 7f9571a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f956c06b1b0 0x7f956c104680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.393+0000 7f9571a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c06bad0 0x7f956c104bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.393+0000 7f9571a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f956c10ff50 0x7f956c105100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.393+0000 7f9571a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1158b0 con 0x7f956c06b1b0 2024-05-10T11:11:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.393+0000 7f9571a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f956c115730 con 0x7f956c10ff50 2024-05-10T11:11:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.393+0000 7f9571a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f956c115a30 con 0x7f956c06bad0 2024-05-10T11:11:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.393+0000 7f956aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f956c06b1b0 0x7f956c104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.393+0000 7f956aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f956c06b1b0 0x7f956c104680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60854/0 (socket says 172.21.15.27:60854) 2024-05-10T11:11:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.393+0000 7f956aa00700 1 -- 172.21.15.27:0/2006411638 learned_addr learned my addr 172.21.15.27:0/2006411638 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:12.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.393+0000 7f956a000700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c06bad0 0x7f956c104bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.394+0000 7f956b400700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f956c10ff50 0x7f956c105100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.394+0000 7f956a000700 1 -- 172.21.15.27:0/2006411638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f956c10ff50 msgr2=0x7f956c105100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.394+0000 7f956a000700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f956c10ff50 0x7f956c105100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.394+0000 7f956a000700 1 -- 172.21.15.27:0/2006411638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f956c06b1b0 msgr2=0x7f956c104680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.394+0000 7f956a000700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f956c06b1b0 0x7f956c104680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.394+0000 7f956a000700 1 -- 172.21.15.27:0/2006411638 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956c1ac4d0 con 0x7f956c06bad0 2024-05-10T11:11:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.394+0000 7f956aa00700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f956c06b1b0 0x7f956c104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:11:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.394+0000 7f956a000700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c06bad0 0x7f956c104bc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f95580098e0 tx=0x7f95580047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.395+0000 7f9563400700 1 -- 172.21.15.27:0/2006411638 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955800fad0 con 0x7f956c06bad0 2024-05-10T11:11:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.395+0000 7f9571a00700 1 -- 172.21.15.27:0/2006411638 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956c1ac760 con 0x7f956c06bad0 2024-05-10T11:11:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.395+0000 7f9571a00700 1 -- 172.21.15.27:0/2006411638 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f956c1acbb0 con 0x7f956c06bad0 2024-05-10T11:11:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.395+0000 7f9563400700 1 -- 172.21.15.27:0/2006411638 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9558004ce0 con 0x7f956c06bad0 2024-05-10T11:11:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.395+0000 7f9563400700 1 -- 172.21.15.27:0/2006411638 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9558005520 con 0x7f956c06bad0 2024-05-10T11:11:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.397+0000 7f9571a00700 1 -- 172.21.15.27:0/2006411638 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f954c005330 con 0x7f956c06bad0 2024-05-10T11:11:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.399+0000 7f9563400700 1 -- 172.21.15.27:0/2006411638 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f9558009d10 con 0x7f956c06bad0 2024-05-10T11:11:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.400+0000 7f9563400700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f95500b2930 0x7f95500b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.401+0000 7f9563400700 1 -- 172.21.15.27:0/2006411638 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f95580e3060 con 0x7f956c06bad0 2024-05-10T11:11:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.401+0000 7f956aa00700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f95500b2930 0x7f95500b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.402+0000 7f956aa00700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f95500b2930 0x7f95500b4df0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f955c009d80 tx=0x7f955c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:12.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.405+0000 7f9563400700 1 -- 172.21.15.27:0/2006411638 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9558013070 con 0x7f956c06bad0 2024-05-10T11:11:12.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:11 smithi076 ceph-mon[34152]: pgmap v435: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:12.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.659+0000 7f9571a00700 1 -- 172.21.15.27:0/2006411638 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f954c005620 con 0x7f956c06bad0 2024-05-10T11:11:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.660+0000 7f9563400700 1 -- 172.21.15.27:0/2006411638 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f95580aa9c0 con 0x7f956c06bad0 2024-05-10T11:11:12.661 INFO:teuthology.orchestra.run.smithi027.stdout:55834574963 2024-05-10T11:11:12.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.664+0000 7f9571a00700 1 -- 172.21.15.27:0/2006411638 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f95500b2930 msgr2=0x7f95500b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:12.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.664+0000 7f9571a00700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f95500b2930 0x7f95500b4df0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f955c009d80 tx=0x7f955c009260 comp rx=0 tx=0).stop 2024-05-10T11:11:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.664+0000 7f9571a00700 1 -- 172.21.15.27:0/2006411638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c06bad0 msgr2=0x7f956c104bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.664+0000 7f9571a00700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c06bad0 0x7f956c104bc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f95580098e0 tx=0x7f95580047b0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.664+0000 7f9571a00700 1 -- 172.21.15.27:0/2006411638 shutdown_connections 2024-05-10T11:11:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.664+0000 7f9571a00700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f95500b2930 0x7f95500b4df0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.664+0000 7f9571a00700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f956c10ff50 0x7f956c105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.664+0000 7f9571a00700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f956c06b1b0 0x7f956c104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.664+0000 7f9571a00700 1 --2- 172.21.15.27:0/2006411638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c06bad0 0x7f956c104bc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.664+0000 7f9571a00700 1 -- 172.21.15.27:0/2006411638 >> 172.21.15.27:0/2006411638 conn(0x7f956c079350 msgr2=0x7f956c1114a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.664+0000 7f9571a00700 1 -- 172.21.15.27:0/2006411638 shutdown_connections 2024-05-10T11:11:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:12.664+0000 7f9571a00700 1 -- 172.21.15.27:0/2006411638 wait complete. 2024-05-10T11:11:13.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:12 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2269228450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-10T11:11:13.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:12 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2006411638' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-10T11:11:13.328 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:12 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2269228450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-10T11:11:13.328 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:12 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2006411638' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-10T11:11:13.401 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626514 got 180388626520 for osd.6 2024-05-10T11:11:13.402 DEBUG:teuthology.parallel:result is None 2024-05-10T11:11:13.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:12 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2269228450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-10T11:11:13.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:12 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2006411638' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-10T11:11:13.905 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574958 got 55834574963 for osd.1 2024-05-10T11:11:13.905 DEBUG:teuthology.parallel:result is None 2024-05-10T11:11:13.905 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-10T11:11:13.905 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-10T11:11:14.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:13 smithi027 ceph-mon[30054]: pgmap v436: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:14.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:13 smithi189 ceph-mon[34842]: pgmap v436: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:14.388 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:13 smithi076 ceph-mon[34152]: pgmap v436: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:16.090 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:15 smithi027 ceph-mon[30054]: pgmap v437: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:16.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:15 smithi189 ceph-mon[34842]: pgmap v437: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:16.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:15 smithi076 ceph-mon[34152]: pgmap v437: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.650+0000 7fd3fc000700 1 -- 172.21.15.27:0/781202848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4108460 msgr2=0x7fd3f4108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.650+0000 7fd3fc000700 1 --2- 172.21.15.27:0/781202848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4108460 0x7fd3f4108840 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd3e80098f0 tx=0x7fd3e8009c00 comp rx=0 tx=0).stop 2024-05-10T11:11:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.650+0000 7fd3fc000700 1 -- 172.21.15.27:0/781202848 shutdown_connections 2024-05-10T11:11:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.650+0000 7fd3fc000700 1 --2- 172.21.15.27:0/781202848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4108460 0x7fd3f4108840 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.650+0000 7fd3fc000700 1 --2- 172.21.15.27:0/781202848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3f4103ef0 0x7fd3f41103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.650+0000 7fd3fc000700 1 --2- 172.21.15.27:0/781202848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f4103550 0x7fd3f41039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.650+0000 7fd3fc000700 1 -- 172.21.15.27:0/781202848 >> 172.21.15.27:0/781202848 conn(0x7fd3f407a600 msgr2=0x7fd3f407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.650+0000 7fd3fc000700 1 -- 172.21.15.27:0/781202848 shutdown_connections 2024-05-10T11:11:16.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.650+0000 7fd3fc000700 1 -- 172.21.15.27:0/781202848 wait complete. 2024-05-10T11:11:16.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.651+0000 7fd3fc000700 1 Processor -- start 2024-05-10T11:11:16.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.651+0000 7fd3fc000700 1 -- start start 2024-05-10T11:11:16.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.652+0000 7fd3fc000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3f4103550 0x7fd3f419f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:16.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.652+0000 7fd3fc000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4103ef0 0x7fd3f419fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:16.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.652+0000 7fd3fc000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f4108460 0x7fd3f41999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:16.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.652+0000 7fd3fc000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f41136f0 con 0x7fd3f4108460 2024-05-10T11:11:16.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.652+0000 7fd3fc000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f4113570 con 0x7fd3f4103550 2024-05-10T11:11:16.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.652+0000 7fd3fc000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f4113870 con 0x7fd3f4103ef0 2024-05-10T11:11:16.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.652+0000 7fd3fa000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f4108460 0x7fd3f41999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:16.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.653+0000 7fd3f8c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4103ef0 0x7fd3f419fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:16.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.653+0000 7fd3f9600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3f4103550 0x7fd3f419f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:16.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.653+0000 7fd3fa000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f4108460 0x7fd3f41999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46074/0 (socket says 172.21.15.27:46074) 2024-05-10T11:11:16.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.653+0000 7fd3fa000700 1 -- 172.21.15.27:0/325358648 learned_addr learned my addr 172.21.15.27:0/325358648 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.653+0000 7fd3f9600700 1 -- 172.21.15.27:0/325358648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4103ef0 msgr2=0x7fd3f419fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.653+0000 7fd3f9600700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4103ef0 0x7fd3f419fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.653+0000 7fd3f9600700 1 -- 172.21.15.27:0/325358648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f4108460 msgr2=0x7fd3f41999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.653+0000 7fd3f9600700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f4108460 0x7fd3f41999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.653+0000 7fd3f9600700 1 -- 172.21.15.27:0/325358648 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f419a150 con 0x7fd3f4103550 2024-05-10T11:11:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.653+0000 7fd3f8c00700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4103ef0 0x7fd3f419fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.653+0000 7fd3fa000700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f4108460 0x7fd3f41999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.653+0000 7fd3f9600700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3f4103550 0x7fd3f419f830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8000c00 tx=0x7fd3e800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.654+0000 7fd3f2000700 1 -- 172.21.15.27:0/325358648 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e80050c0 con 0x7fd3f4103550 2024-05-10T11:11:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.654+0000 7fd3f2000700 1 -- 172.21.15.27:0/325358648 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3e800b560 con 0x7fd3f4103550 2024-05-10T11:11:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.654+0000 7fd3fc000700 1 -- 172.21.15.27:0/325358648 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3f419a3e0 con 0x7fd3f4103550 2024-05-10T11:11:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.654+0000 7fd3f2000700 1 -- 172.21.15.27:0/325358648 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8003a00 con 0x7fd3f4103550 2024-05-10T11:11:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.654+0000 7fd3fc000700 1 -- 172.21.15.27:0/325358648 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3f41a6700 con 0x7fd3f4103550 2024-05-10T11:11:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.655+0000 7fd3fc000700 1 -- 172.21.15.27:0/325358648 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3f410ba90 con 0x7fd3f4103550 2024-05-10T11:11:16.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.658+0000 7fd3f2000700 1 -- 172.21.15.27:0/325358648 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fd3e800b9c0 con 0x7fd3f4103550 2024-05-10T11:11:16.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.659+0000 7fd3f2000700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd3dc0b2880 0x7fd3dc0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.660+0000 7fd3f2000700 1 -- 172.21.15.27:0/325358648 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fd3e80e3a10 con 0x7fd3f4103550 2024-05-10T11:11:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.660+0000 7fd3f8c00700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd3dc0b2880 0x7fd3dc0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.661+0000 7fd3f8c00700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd3dc0b2880 0x7fd3dc0b4d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0009750 tx=0x7fd3e0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.661+0000 7fd3f2000700 1 -- 172.21.15.27:0/325358648 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fd3e8013030 con 0x7fd3f4103550 2024-05-10T11:11:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.894+0000 7fd3fc000700 1 -- 172.21.15.27:0/325358648 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd3f419af20 con 0x7fd3dc0b2880 2024-05-10T11:11:16.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.912+0000 7fd3f2000700 1 -- 172.21.15.27:0/325358648 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340782 (secure 0 0 0) 0x7fd3f419af20 con 0x7fd3dc0b2880 2024-05-10T11:11:16.919 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:11:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.919+0000 7fd3fc000700 1 -- 172.21.15.27:0/325358648 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd3dc0b2880 msgr2=0x7fd3dc0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.919+0000 7fd3fc000700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd3dc0b2880 0x7fd3dc0b4d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0009750 tx=0x7fd3e0009310 comp rx=0 tx=0).stop 2024-05-10T11:11:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.919+0000 7fd3fc000700 1 -- 172.21.15.27:0/325358648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3f4103550 msgr2=0x7fd3f419f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.919+0000 7fd3fc000700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3f4103550 0x7fd3f419f830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8000c00 tx=0x7fd3e800fa00 comp rx=0 tx=0).stop 2024-05-10T11:11:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.919+0000 7fd3fc000700 1 -- 172.21.15.27:0/325358648 shutdown_connections 2024-05-10T11:11:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.919+0000 7fd3fc000700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd3dc0b2880 0x7fd3dc0b4d40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:16.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.919+0000 7fd3fc000700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f4108460 0x7fd3f41999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:16.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.919+0000 7fd3fc000700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3f4103550 0x7fd3f419f830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:16.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.919+0000 7fd3fc000700 1 --2- 172.21.15.27:0/325358648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4103ef0 0x7fd3f419fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:16.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.919+0000 7fd3fc000700 1 -- 172.21.15.27:0/325358648 >> 172.21.15.27:0/325358648 conn(0x7fd3f407a600 msgr2=0x7fd3f41016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:16.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.920+0000 7fd3fc000700 1 -- 172.21.15.27:0/325358648 shutdown_connections 2024-05-10T11:11:16.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:16.920+0000 7fd3fc000700 1 -- 172.21.15.27:0/325358648 wait complete. 2024-05-10T11:11:16.923 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-10T11:11:17.795 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-05-10T11:11:16.870804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662450,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1305,"num_read_kb":761,"num_write":382,"num_write_kb":2301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":2952552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":815,"ondisk_log_size":815,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361412,"kb_used_data":12260,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580412,"statfs":{"total":1151940427776,"available":1151570341888,"internally_reserved":0,"allocated":12554240,"data_stored":8226512,"data_compressed":497229,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":129,"num_read_kb":65,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006180"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.768848+0000","last_change":"2024-05-10T11:07:47.212921+0000","last_active":"2024-05-10T11:10:50.768848+0000","last_peered":"2024-05-10T11:10:50.768848+0000","last_clean":"2024-05-10T11:10:50.768848+0000","last_became_active":"2024-05-10T11:07:47.212558+0000","last_became_peered":"2024-05-10T11:07:47.212558+0000","last_unstale":"2024-05-10T11:10:50.768848+0000","last_undegraded":"2024-05-10T11:10:50.768848+0000","last_fullsized":"2024-05-10T11:10:50.768848+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:47:20.415883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.785263+0000","last_change":"2024-05-10T11:07:47.211186+0000","last_active":"2024-05-10T11:10:50.785263+0000","last_peered":"2024-05-10T11:10:50.785263+0000","last_clean":"2024-05-10T11:10:50.785263+0000","last_became_active":"2024-05-10T11:07:47.203487+0000","last_became_peered":"2024-05-10T11:07:47.203487+0000","last_unstale":"2024-05-10T11:10:50.785263+0000","last_undegraded":"2024-05-10T11:10:50.785263+0000","last_fullsized":"2024-05-10T11:10:50.785263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:51:29.674632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.801345+0000","last_change":"2024-05-10T11:07:47.202718+0000","last_active":"2024-05-10T11:10:50.801345+0000","last_peered":"2024-05-10T11:10:50.801345+0000","last_clean":"2024-05-10T11:10:50.801345+0000","last_became_active":"2024-05-10T11:07:47.202048+0000","last_became_peered":"2024-05-10T11:07:47.202048+0000","last_unstale":"2024-05-10T11:10:50.801345+0000","last_undegraded":"2024-05-10T11:10:50.801345+0000","last_fullsized":"2024-05-10T11:10:50.801345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:01:41.156850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.745401+0000","last_change":"2024-05-10T11:07:47.193519+0000","last_active":"2024-05-10T11:10:50.745401+0000","last_peered":"2024-05-10T11:10:50.745401+0000","last_clean":"2024-05-10T11:10:50.745401+0000","last_became_active":"2024-05-10T11:07:47.193200+0000","last_became_peered":"2024-05-10T11:07:47.193200+0000","last_unstale":"2024-05-10T11:10:50.745401+0000","last_undegraded":"2024-05-10T11:10:50.745401+0000","last_fullsized":"2024-05-10T11:10:50.745401+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:01.949453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.785839+0000","last_change":"2024-05-10T11:07:47.207750+0000","last_active":"2024-05-10T11:10:50.785839+0000","last_peered":"2024-05-10T11:10:50.785839+0000","last_clean":"2024-05-10T11:10:50.785839+0000","last_became_active":"2024-05-10T11:07:47.207384+0000","last_became_peered":"2024-05-10T11:07:47.207384+0000","last_unstale":"2024-05-10T11:10:50.785839+0000","last_undegraded":"2024-05-10T11:10:50.785839+0000","last_fullsized":"2024-05-10T11:10:50.785839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:09.432766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.762081+0000","last_change":"2024-05-10T11:08:22.245930+0000","last_active":"2024-05-10T11:10:50.762081+0000","last_peered":"2024-05-10T11:10:50.762081+0000","last_clean":"2024-05-10T11:10:50.762081+0000","last_became_active":"2024-05-10T11:08:22.245553+0000","last_became_peered":"2024-05-10T11:08:22.245553+0000","last_unstale":"2024-05-10T11:10:50.762081+0000","last_undegraded":"2024-05-10T11:10:50.762081+0000","last_fullsized":"2024-05-10T11:10:50.762081+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:47:31.485623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.763185+0000","last_change":"2024-05-10T11:07:47.196666+0000","last_active":"2024-05-10T11:10:50.763185+0000","last_peered":"2024-05-10T11:10:50.763185+0000","last_clean":"2024-05-10T11:10:50.763185+0000","last_became_active":"2024-05-10T11:07:47.196300+0000","last_became_peered":"2024-05-10T11:07:47.196300+0000","last_unstale":"2024-05-10T11:10:50.763185+0000","last_undegraded":"2024-05-10T11:10:50.763185+0000","last_fullsized":"2024-05-10T11:10:50.763185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:55:24.343385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.788582+0000","last_change":"2024-05-10T11:07:47.213048+0000","last_active":"2024-05-10T11:10:50.788582+0000","last_peered":"2024-05-10T11:10:50.788582+0000","last_clean":"2024-05-10T11:10:50.788582+0000","last_became_active":"2024-05-10T11:07:47.206359+0000","last_became_peered":"2024-05-10T11:07:47.206359+0000","last_unstale":"2024-05-10T11:10:50.788582+0000","last_undegraded":"2024-05-10T11:10:50.788582+0000","last_fullsized":"2024-05-10T11:10:50.788582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:32:56.307261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.765728+0000","last_change":"2024-05-10T11:07:47.190945+0000","last_active":"2024-05-10T11:10:50.765728+0000","last_peered":"2024-05-10T11:10:50.765728+0000","last_clean":"2024-05-10T11:10:50.765728+0000","last_became_active":"2024-05-10T11:07:47.190631+0000","last_became_peered":"2024-05-10T11:07:47.190631+0000","last_unstale":"2024-05-10T11:10:50.765728+0000","last_undegraded":"2024-05-10T11:10:50.765728+0000","last_fullsized":"2024-05-10T11:10:50.765728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:34:08.683912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.751222+0000","last_change":"2024-05-10T11:07:47.189540+0000","last_active":"2024-05-10T11:10:50.751222+0000","last_peered":"2024-05-10T11:10:50.751222+0000","last_clean":"2024-05-10T11:10:50.751222+0000","last_became_active":"2024-05-10T11:07:47.189185+0000","last_became_peered":"2024-05-10T11:07:47.189185+0000","last_unstale":"2024-05-10T11:10:50.751222+0000","last_undegraded":"2024-05-10T11:10:50.751222+0000","last_fullsized":"2024-05-10T11:10:50.751222+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:55:23.892826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.769207+0000","last_change":"2024-05-10T11:07:47.195688+0000","last_active":"2024-05-10T11:10:50.769207+0000","last_peered":"2024-05-10T11:10:50.769207+0000","last_clean":"2024-05-10T11:10:50.769207+0000","last_became_active":"2024-05-10T11:07:47.195081+0000","last_became_peered":"2024-05-10T11:07:47.195081+0000","last_unstale":"2024-05-10T11:10:50.769207+0000","last_undegraded":"2024-05-10T11:10:50.769207+0000","last_fullsized":"2024-05-10T11:10:50.769207+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:20:55.240827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.747771+0000","last_change":"2024-05-10T11:07:47.193736+0000","last_active":"2024-05-10T11:10:50.747771+0000","last_peered":"2024-05-10T11:10:50.747771+0000","last_clean":"2024-05-10T11:10:50.747771+0000","last_became_active":"2024-05-10T11:07:47.193400+0000","last_became_peered":"2024-05-10T11:07:47.193400+0000","last_unstale":"2024-05-10T11:10:50.747771+0000","last_undegraded":"2024-05-10T11:10:50.747771+0000","last_fullsized":"2024-05-10T11:10:50.747771+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:46:28.507330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.791543+0000","last_change":"2024-05-10T11:08:22.042849+0000","last_active":"2024-05-10T11:10:50.791543+0000","last_peered":"2024-05-10T11:10:50.791543+0000","last_clean":"2024-05-10T11:10:50.791543+0000","last_became_active":"2024-05-10T11:08:22.042024+0000","last_became_peered":"2024-05-10T11:08:22.042024+0000","last_unstale":"2024-05-10T11:10:50.791543+0000","last_undegraded":"2024-05-10T11:10:50.791543+0000","last_fullsized":"2024-05-10T11:10:50.791543+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:29:08.096541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.801249+0000","last_change":"2024-05-10T11:07:47.214016+0000","last_active":"2024-05-10T11:10:50.801249+0000","last_peered":"2024-05-10T11:10:50.801249+0000","last_clean":"2024-05-10T11:10:50.801249+0000","last_became_active":"2024-05-10T11:07:47.209658+0000","last_became_peered":"2024-05-10T11:07:47.209658+0000","last_unstale":"2024-05-10T11:10:50.801249+0000","last_undegraded":"2024-05-10T11:10:50.801249+0000","last_fullsized":"2024-05-10T11:10:50.801249+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:44.771221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.800323+0000","last_change":"2024-05-10T11:07:47.210153+0000","last_active":"2024-05-10T11:10:50.800323+0000","last_peered":"2024-05-10T11:10:50.800323+0000","last_clean":"2024-05-10T11:10:50.800323+0000","last_became_active":"2024-05-10T11:07:47.208470+0000","last_became_peered":"2024-05-10T11:07:47.208470+0000","last_unstale":"2024-05-10T11:10:50.800323+0000","last_undegraded":"2024-05-10T11:10:50.800323+0000","last_fullsized":"2024-05-10T11:10:50.800323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:23:03.614670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.768324+0000","last_change":"2024-05-10T11:07:47.210704+0000","last_active":"2024-05-10T11:10:50.768324+0000","last_peered":"2024-05-10T11:10:50.768324+0000","last_clean":"2024-05-10T11:10:50.768324+0000","last_became_active":"2024-05-10T11:07:47.210317+0000","last_became_peered":"2024-05-10T11:07:47.210317+0000","last_unstale":"2024-05-10T11:10:50.768324+0000","last_undegraded":"2024-05-10T11:10:50.768324+0000","last_fullsized":"2024-05-10T11:10:50.768324+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:47:30.848731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.771107+0000","last_change":"2024-05-10T11:07:47.213116+0000","last_active":"2024-05-10T11:10:50.771107+0000","last_peered":"2024-05-10T11:10:50.771107+0000","last_clean":"2024-05-10T11:10:50.771107+0000","last_became_active":"2024-05-10T11:07:47.212567+0000","last_became_peered":"2024-05-10T11:07:47.212567+0000","last_unstale":"2024-05-10T11:10:50.771107+0000","last_undegraded":"2024-05-10T11:10:50.771107+0000","last_fullsized":"2024-05-10T11:10:50.771107+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:44:32.222012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.773409+0000","last_change":"2024-05-10T11:07:47.200973+0000","last_active":"2024-05-10T11:10:50.773409+0000","last_peered":"2024-05-10T11:10:50.773409+0000","last_clean":"2024-05-10T11:10:50.773409+0000","last_became_active":"2024-05-10T11:07:47.200499+0000","last_became_peered":"2024-05-10T11:07:47.200499+0000","last_unstale":"2024-05-10T11:10:50.773409+0000","last_undegraded":"2024-05-10T11:10:50.773409+0000","last_fullsized":"2024-05-10T11:10:50.773409+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:29:24.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.786245+0000","last_change":"2024-05-10T11:08:22.042254+0000","last_active":"2024-05-10T11:10:50.786245+0000","last_peered":"2024-05-10T11:10:50.786245+0000","last_clean":"2024-05-10T11:10:50.786245+0000","last_became_active":"2024-05-10T11:08:22.041906+0000","last_became_peered":"2024-05-10T11:08:22.041906+0000","last_unstale":"2024-05-10T11:10:50.786245+0000","last_undegraded":"2024-05-10T11:10:50.786245+0000","last_fullsized":"2024-05-10T11:10:50.786245+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:55:27.280478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070243+0000","last_change":"2024-05-10T11:08:00.897746+0000","last_active":"2024-05-10T11:09:23.070243+0000","last_peered":"2024-05-10T11:09:23.070243+0000","last_clean":"2024-05-10T11:09:23.070243+0000","last_became_active":"2024-05-10T11:08:00.897428+0000","last_became_peered":"2024-05-10T11:08:00.897428+0000","last_unstale":"2024-05-10T11:09:23.070243+0000","last_undegraded":"2024-05-10T11:09:23.070243+0000","last_fullsized":"2024-05-10T11:09:23.070243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:38.846373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":176,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:11.303636+0000","last_change":"2024-05-10T11:07:42.139990+0000","last_active":"2024-05-10T11:11:11.303636+0000","last_peered":"2024-05-10T11:11:11.303636+0000","last_clean":"2024-05-10T11:11:11.303636+0000","last_became_active":"2024-05-10T11:07:42.139612+0000","last_became_peered":"2024-05-10T11:07:42.139612+0000","last_unstale":"2024-05-10T11:11:11.303636+0000","last_undegraded":"2024-05-10T11:11:11.303636+0000","last_fullsized":"2024-05-10T11:11:11.303636+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:48:29.590570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.759313+0000","last_change":"2024-05-10T11:07:47.214263+0000","last_active":"2024-05-10T11:10:50.759313+0000","last_peered":"2024-05-10T11:10:50.759313+0000","last_clean":"2024-05-10T11:10:50.759313+0000","last_became_active":"2024-05-10T11:07:47.213946+0000","last_became_peered":"2024-05-10T11:07:47.213946+0000","last_unstale":"2024-05-10T11:10:50.759313+0000","last_undegraded":"2024-05-10T11:10:50.759313+0000","last_fullsized":"2024-05-10T11:10:50.759313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:54:31.015838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062366+0000","last_change":"2024-05-10T11:08:00.890571+0000","last_active":"2024-05-10T11:09:23.062366+0000","last_peered":"2024-05-10T11:09:23.062366+0000","last_clean":"2024-05-10T11:09:23.062366+0000","last_became_active":"2024-05-10T11:08:00.889468+0000","last_became_peered":"2024-05-10T11:08:00.889468+0000","last_unstale":"2024-05-10T11:09:23.062366+0000","last_undegraded":"2024-05-10T11:09:23.062366+0000","last_fullsized":"2024-05-10T11:09:23.062366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:20:54.342656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077801+0000","last_change":"2024-05-10T11:07:42.383396+0000","last_active":"2024-05-10T11:09:23.077801+0000","last_peered":"2024-05-10T11:09:23.077801+0000","last_clean":"2024-05-10T11:09:23.077801+0000","last_became_active":"2024-05-10T11:07:42.382787+0000","last_became_peered":"2024-05-10T11:07:42.382787+0000","last_unstale":"2024-05-10T11:09:23.077801+0000","last_undegraded":"2024-05-10T11:09:23.077801+0000","last_fullsized":"2024-05-10T11:09:23.077801+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:44:34.240119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.802653+0000","last_change":"2024-05-10T11:07:47.216770+0000","last_active":"2024-05-10T11:10:50.802653+0000","last_peered":"2024-05-10T11:10:50.802653+0000","last_clean":"2024-05-10T11:10:50.802653+0000","last_became_active":"2024-05-10T11:07:47.216377+0000","last_became_peered":"2024-05-10T11:07:47.216377+0000","last_unstale":"2024-05-10T11:10:50.802653+0000","last_undegraded":"2024-05-10T11:10:50.802653+0000","last_fullsized":"2024-05-10T11:10:50.802653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:59:22.993888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062516+0000","last_change":"2024-05-10T11:08:00.900614+0000","last_active":"2024-05-10T11:09:23.062516+0000","last_peered":"2024-05-10T11:09:23.062516+0000","last_clean":"2024-05-10T11:09:23.062516+0000","last_became_active":"2024-05-10T11:08:00.900162+0000","last_became_peered":"2024-05-10T11:08:00.900162+0000","last_unstale":"2024-05-10T11:09:23.062516+0000","last_undegraded":"2024-05-10T11:09:23.062516+0000","last_fullsized":"2024-05-10T11:09:23.062516+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:42:56.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070291+0000","last_change":"2024-05-10T11:07:42.127749+0000","last_active":"2024-05-10T11:09:23.070291+0000","last_peered":"2024-05-10T11:09:23.070291+0000","last_clean":"2024-05-10T11:09:23.070291+0000","last_became_active":"2024-05-10T11:07:42.127296+0000","last_became_peered":"2024-05-10T11:07:42.127296+0000","last_unstale":"2024-05-10T11:09:23.070291+0000","last_undegraded":"2024-05-10T11:09:23.070291+0000","last_fullsized":"2024-05-10T11:09:23.070291+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:37:37.720542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.781187+0000","last_change":"2024-05-10T11:07:47.198971+0000","last_active":"2024-05-10T11:10:50.781187+0000","last_peered":"2024-05-10T11:10:50.781187+0000","last_clean":"2024-05-10T11:10:50.781187+0000","last_became_active":"2024-05-10T11:07:47.198510+0000","last_became_peered":"2024-05-10T11:07:47.198510+0000","last_unstale":"2024-05-10T11:10:50.781187+0000","last_undegraded":"2024-05-10T11:10:50.781187+0000","last_fullsized":"2024-05-10T11:10:50.781187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:17:24.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062534+0000","last_change":"2024-05-10T11:08:00.878749+0000","last_active":"2024-05-10T11:09:23.062534+0000","last_peered":"2024-05-10T11:09:23.062534+0000","last_clean":"2024-05-10T11:09:23.062534+0000","last_became_active":"2024-05-10T11:08:00.878271+0000","last_became_peered":"2024-05-10T11:08:00.878271+0000","last_unstale":"2024-05-10T11:09:23.062534+0000","last_undegraded":"2024-05-10T11:09:23.062534+0000","last_fullsized":"2024-05-10T11:09:23.062534+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:39:28.558766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069628+0000","last_change":"2024-05-10T11:07:42.129593+0000","last_active":"2024-05-10T11:09:23.069628+0000","last_peered":"2024-05-10T11:09:23.069628+0000","last_clean":"2024-05-10T11:09:23.069628+0000","last_became_active":"2024-05-10T11:07:42.129250+0000","last_became_peered":"2024-05-10T11:07:42.129250+0000","last_unstale":"2024-05-10T11:09:23.069628+0000","last_undegraded":"2024-05-10T11:09:23.069628+0000","last_fullsized":"2024-05-10T11:09:23.069628+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:22:55.541079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.782571+0000","last_change":"2024-05-10T11:08:22.244535+0000","last_active":"2024-05-10T11:10:50.782571+0000","last_peered":"2024-05-10T11:10:50.782571+0000","last_clean":"2024-05-10T11:10:50.782571+0000","last_became_active":"2024-05-10T11:08:22.243716+0000","last_became_peered":"2024-05-10T11:08:22.243716+0000","last_unstale":"2024-05-10T11:10:50.782571+0000","last_undegraded":"2024-05-10T11:10:50.782571+0000","last_fullsized":"2024-05-10T11:10:50.782571+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:11.718999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069816+0000","last_change":"2024-05-10T11:08:00.869806+0000","last_active":"2024-05-10T11:09:23.069816+0000","last_peered":"2024-05-10T11:09:23.069816+0000","last_clean":"2024-05-10T11:09:23.069816+0000","last_became_active":"2024-05-10T11:08:00.869355+0000","last_became_peered":"2024-05-10T11:08:00.869355+0000","last_unstale":"2024-05-10T11:09:23.069816+0000","last_undegraded":"2024-05-10T11:09:23.069816+0000","last_fullsized":"2024-05-10T11:09:23.069816+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:36.043412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":184,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:14.317725+0000","last_change":"2024-05-10T11:09:23.076933+0000","last_active":"2024-05-10T11:11:14.317725+0000","last_peered":"2024-05-10T11:11:14.317725+0000","last_clean":"2024-05-10T11:11:14.317725+0000","last_became_active":"2024-05-10T11:09:23.073274+0000","last_became_peered":"2024-05-10T11:09:23.073274+0000","last_unstale":"2024-05-10T11:11:14.317725+0000","last_undegraded":"2024-05-10T11:11:14.317725+0000","last_fullsized":"2024-05-10T11:11:14.317725+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:26:13.923466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.762350+0000","last_change":"2024-05-10T11:07:47.200873+0000","last_active":"2024-05-10T11:10:50.762350+0000","last_peered":"2024-05-10T11:10:50.762350+0000","last_clean":"2024-05-10T11:10:50.762350+0000","last_became_active":"2024-05-10T11:07:47.200436+0000","last_became_peered":"2024-05-10T11:07:47.200436+0000","last_unstale":"2024-05-10T11:10:50.762350+0000","last_undegraded":"2024-05-10T11:10:50.762350+0000","last_fullsized":"2024-05-10T11:10:50.762350+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:38:49.588593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069663+0000","last_change":"2024-05-10T11:08:00.885432+0000","last_active":"2024-05-10T11:09:23.069663+0000","last_peered":"2024-05-10T11:09:23.069663+0000","last_clean":"2024-05-10T11:09:23.069663+0000","last_became_active":"2024-05-10T11:08:00.884987+0000","last_became_peered":"2024-05-10T11:08:00.884987+0000","last_unstale":"2024-05-10T11:09:23.069663+0000","last_undegraded":"2024-05-10T11:09:23.069663+0000","last_fullsized":"2024-05-10T11:09:23.069663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:41:24.082722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.093561+0000","last_change":"2024-05-10T11:09:23.093122+0000","last_active":"2024-05-10T11:09:23.093561+0000","last_peered":"2024-05-10T11:09:23.093561+0000","last_clean":"2024-05-10T11:09:23.093561+0000","last_became_active":"2024-05-10T11:09:23.084485+0000","last_became_peered":"2024-05-10T11:09:23.084485+0000","last_unstale":"2024-05-10T11:09:23.093561+0000","last_undegraded":"2024-05-10T11:09:23.093561+0000","last_fullsized":"2024-05-10T11:09:23.093561+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:12:36.793572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.785927+0000","last_change":"2024-05-10T11:08:22.036515+0000","last_active":"2024-05-10T11:10:50.785927+0000","last_peered":"2024-05-10T11:10:50.785927+0000","last_clean":"2024-05-10T11:10:50.785927+0000","last_became_active":"2024-05-10T11:08:22.036072+0000","last_became_peered":"2024-05-10T11:08:22.036072+0000","last_unstale":"2024-05-10T11:10:50.785927+0000","last_undegraded":"2024-05-10T11:10:50.785927+0000","last_fullsized":"2024-05-10T11:10:50.785927+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:08.734453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6],"acting":[5,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065713+0000","last_change":"2024-05-10T11:08:00.888673+0000","last_active":"2024-05-10T11:09:23.065713+0000","last_peered":"2024-05-10T11:09:23.065713+0000","last_clean":"2024-05-10T11:09:23.065713+0000","last_became_active":"2024-05-10T11:08:00.888302+0000","last_became_peered":"2024-05-10T11:08:00.888302+0000","last_unstale":"2024-05-10T11:09:23.065713+0000","last_undegraded":"2024-05-10T11:09:23.065713+0000","last_fullsized":"2024-05-10T11:09:23.065713+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:15:32.276358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.078317+0000","last_change":"2024-05-10T11:07:42.140993+0000","last_active":"2024-05-10T11:09:23.078317+0000","last_peered":"2024-05-10T11:09:23.078317+0000","last_clean":"2024-05-10T11:09:23.078317+0000","last_became_active":"2024-05-10T11:07:42.140501+0000","last_became_peered":"2024-05-10T11:07:42.140501+0000","last_unstale":"2024-05-10T11:09:23.078317+0000","last_undegraded":"2024-05-10T11:09:23.078317+0000","last_fullsized":"2024-05-10T11:09:23.078317+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:01:50.324507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.790621+0000","last_change":"2024-05-10T11:07:47.212616+0000","last_active":"2024-05-10T11:10:50.790621+0000","last_peered":"2024-05-10T11:10:50.790621+0000","last_clean":"2024-05-10T11:10:50.790621+0000","last_became_active":"2024-05-10T11:07:47.211081+0000","last_became_peered":"2024-05-10T11:07:47.211081+0000","last_unstale":"2024-05-10T11:10:50.790621+0000","last_undegraded":"2024-05-10T11:10:50.790621+0000","last_fullsized":"2024-05-10T11:10:50.790621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:03:54.404355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067703+0000","last_change":"2024-05-10T11:08:00.888782+0000","last_active":"2024-05-10T11:09:23.067703+0000","last_peered":"2024-05-10T11:09:23.067703+0000","last_clean":"2024-05-10T11:09:23.067703+0000","last_became_active":"2024-05-10T11:08:00.888366+0000","last_became_peered":"2024-05-10T11:08:00.888366+0000","last_unstale":"2024-05-10T11:09:23.067703+0000","last_undegraded":"2024-05-10T11:09:23.067703+0000","last_fullsized":"2024-05-10T11:09:23.067703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:37.040186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066199+0000","last_change":"2024-05-10T11:07:42.128677+0000","last_active":"2024-05-10T11:09:23.066199+0000","last_peered":"2024-05-10T11:09:23.066199+0000","last_clean":"2024-05-10T11:09:23.066199+0000","last_became_active":"2024-05-10T11:07:42.128330+0000","last_became_peered":"2024-05-10T11:07:42.128330+0000","last_unstale":"2024-05-10T11:09:23.066199+0000","last_undegraded":"2024-05-10T11:09:23.066199+0000","last_fullsized":"2024-05-10T11:09:23.066199+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:09:11.738485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.783986+0000","last_change":"2024-05-10T11:07:47.215257+0000","last_active":"2024-05-10T11:10:50.783986+0000","last_peered":"2024-05-10T11:10:50.783986+0000","last_clean":"2024-05-10T11:10:50.783986+0000","last_became_active":"2024-05-10T11:07:47.214682+0000","last_became_peered":"2024-05-10T11:07:47.214682+0000","last_unstale":"2024-05-10T11:10:50.783986+0000","last_undegraded":"2024-05-10T11:10:50.783986+0000","last_fullsized":"2024-05-10T11:10:50.783986+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:35:45.787737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076185+0000","last_change":"2024-05-10T11:08:00.894446+0000","last_active":"2024-05-10T11:09:23.076185+0000","last_peered":"2024-05-10T11:09:23.076185+0000","last_clean":"2024-05-10T11:09:23.076185+0000","last_became_active":"2024-05-10T11:08:00.886473+0000","last_became_peered":"2024-05-10T11:08:00.886473+0000","last_unstale":"2024-05-10T11:09:23.076185+0000","last_undegraded":"2024-05-10T11:09:23.076185+0000","last_fullsized":"2024-05-10T11:09:23.076185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:40.980328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":287,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:14.221369+0000","last_change":"2024-05-10T11:07:42.138750+0000","last_active":"2024-05-10T11:11:14.221369+0000","last_peered":"2024-05-10T11:11:14.221369+0000","last_clean":"2024-05-10T11:11:14.221369+0000","last_became_active":"2024-05-10T11:07:42.138342+0000","last_became_peered":"2024-05-10T11:07:42.138342+0000","last_unstale":"2024-05-10T11:11:14.221369+0000","last_undegraded":"2024-05-10T11:11:14.221369+0000","last_fullsized":"2024-05-10T11:11:14.221369+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:19:45.791550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":238,"num_read_kb":238,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.774389+0000","last_change":"2024-05-10T11:07:47.204604+0000","last_active":"2024-05-10T11:10:50.774389+0000","last_peered":"2024-05-10T11:10:50.774389+0000","last_clean":"2024-05-10T11:10:50.774389+0000","last_became_active":"2024-05-10T11:07:47.204261+0000","last_became_peered":"2024-05-10T11:07:47.204261+0000","last_unstale":"2024-05-10T11:10:50.774389+0000","last_undegraded":"2024-05-10T11:10:50.774389+0000","last_fullsized":"2024-05-10T11:10:50.774389+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:55:46.409545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068712+0000","last_change":"2024-05-10T11:08:22.038090+0000","last_active":"2024-05-10T11:09:23.068712+0000","last_peered":"2024-05-10T11:09:23.068712+0000","last_clean":"2024-05-10T11:09:23.068712+0000","last_became_active":"2024-05-10T11:08:22.037490+0000","last_became_peered":"2024-05-10T11:08:22.037490+0000","last_unstale":"2024-05-10T11:09:23.068712+0000","last_undegraded":"2024-05-10T11:09:23.068712+0000","last_fullsized":"2024-05-10T11:09:23.068712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:01:46.319401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071747+0000","last_change":"2024-05-10T11:07:42.128636+0000","last_active":"2024-05-10T11:09:23.071747+0000","last_peered":"2024-05-10T11:09:23.071747+0000","last_clean":"2024-05-10T11:09:23.071747+0000","last_became_active":"2024-05-10T11:07:42.128194+0000","last_became_peered":"2024-05-10T11:07:42.128194+0000","last_unstale":"2024-05-10T11:09:23.071747+0000","last_undegraded":"2024-05-10T11:09:23.071747+0000","last_fullsized":"2024-05-10T11:09:23.071747+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:51:54.279292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.802905+0000","last_change":"2024-05-10T11:08:22.037807+0000","last_active":"2024-05-10T11:10:50.802905+0000","last_peered":"2024-05-10T11:10:50.802905+0000","last_clean":"2024-05-10T11:10:50.802905+0000","last_became_active":"2024-05-10T11:08:22.037337+0000","last_became_peered":"2024-05-10T11:08:22.037337+0000","last_unstale":"2024-05-10T11:10:50.802905+0000","last_undegraded":"2024-05-10T11:10:50.802905+0000","last_fullsized":"2024-05-10T11:10:50.802905+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:50:06.481222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062740+0000","last_change":"2024-05-10T11:08:00.893684+0000","last_active":"2024-05-10T11:09:23.062740+0000","last_peered":"2024-05-10T11:09:23.062740+0000","last_clean":"2024-05-10T11:09:23.062740+0000","last_became_active":"2024-05-10T11:08:00.893334+0000","last_became_peered":"2024-05-10T11:08:00.893334+0000","last_unstale":"2024-05-10T11:09:23.062740+0000","last_undegraded":"2024-05-10T11:09:23.062740+0000","last_fullsized":"2024-05-10T11:09:23.062740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:58:04.812308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066370+0000","last_change":"2024-05-10T11:07:42.382086+0000","last_active":"2024-05-10T11:09:23.066370+0000","last_peered":"2024-05-10T11:09:23.066370+0000","last_clean":"2024-05-10T11:09:23.066370+0000","last_became_active":"2024-05-10T11:07:42.381744+0000","last_became_peered":"2024-05-10T11:07:42.381744+0000","last_unstale":"2024-05-10T11:09:23.066370+0000","last_undegraded":"2024-05-10T11:09:23.066370+0000","last_fullsized":"2024-05-10T11:09:23.066370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:15:10.893834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.762106+0000","last_change":"2024-05-10T11:07:47.194683+0000","last_active":"2024-05-10T11:10:50.762106+0000","last_peered":"2024-05-10T11:10:50.762106+0000","last_clean":"2024-05-10T11:10:50.762106+0000","last_became_active":"2024-05-10T11:07:47.194319+0000","last_became_peered":"2024-05-10T11:07:47.194319+0000","last_unstale":"2024-05-10T11:10:50.762106+0000","last_undegraded":"2024-05-10T11:10:50.762106+0000","last_fullsized":"2024-05-10T11:10:50.762106+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:41:48.437093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069730+0000","last_change":"2024-05-10T11:08:00.889616+0000","last_active":"2024-05-10T11:09:23.069730+0000","last_peered":"2024-05-10T11:09:23.069730+0000","last_clean":"2024-05-10T11:09:23.069730+0000","last_became_active":"2024-05-10T11:08:00.882179+0000","last_became_peered":"2024-05-10T11:08:00.882179+0000","last_unstale":"2024-05-10T11:09:23.069730+0000","last_undegraded":"2024-05-10T11:09:23.069730+0000","last_fullsized":"2024-05-10T11:09:23.069730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:23:12.407861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'39","reported_seq":332,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:14.250598+0000","last_change":"2024-05-10T11:07:42.128796+0000","last_active":"2024-05-10T11:11:14.250598+0000","last_peered":"2024-05-10T11:11:14.250598+0000","last_clean":"2024-05-10T11:11:14.250598+0000","last_became_active":"2024-05-10T11:07:42.128402+0000","last_became_peered":"2024-05-10T11:07:42.128402+0000","last_unstale":"2024-05-10T11:11:14.250598+0000","last_undegraded":"2024-05-10T11:11:14.250598+0000","last_fullsized":"2024-05-10T11:11:14.250598+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:50:09.205102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64894,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":248,"num_read_kb":188,"num_write":41,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.762334+0000","last_change":"2024-05-10T11:07:47.188304+0000","last_active":"2024-05-10T11:10:50.762334+0000","last_peered":"2024-05-10T11:10:50.762334+0000","last_clean":"2024-05-10T11:10:50.762334+0000","last_became_active":"2024-05-10T11:07:47.187798+0000","last_became_peered":"2024-05-10T11:07:47.187798+0000","last_unstale":"2024-05-10T11:10:50.762334+0000","last_undegraded":"2024-05-10T11:10:50.762334+0000","last_fullsized":"2024-05-10T11:10:50.762334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:50:21.059060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066761+0000","last_change":"2024-05-10T11:08:00.886669+0000","last_active":"2024-05-10T11:09:23.066761+0000","last_peered":"2024-05-10T11:09:23.066761+0000","last_clean":"2024-05-10T11:09:23.066761+0000","last_became_active":"2024-05-10T11:08:00.886249+0000","last_became_peered":"2024-05-10T11:08:00.886249+0000","last_unstale":"2024-05-10T11:09:23.066761+0000","last_undegraded":"2024-05-10T11:09:23.066761+0000","last_fullsized":"2024-05-10T11:09:23.066761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:10:07.707462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.095713+0000","last_change":"2024-05-10T11:09:23.095352+0000","last_active":"2024-05-10T11:09:23.095713+0000","last_peered":"2024-05-10T11:09:23.095713+0000","last_clean":"2024-05-10T11:09:23.095713+0000","last_became_active":"2024-05-10T11:09:23.083116+0000","last_became_peered":"2024-05-10T11:09:23.083116+0000","last_unstale":"2024-05-10T11:09:23.095713+0000","last_undegraded":"2024-05-10T11:09:23.095713+0000","last_fullsized":"2024-05-10T11:09:23.095713+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:24:28.945724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.790060+0000","last_change":"2024-05-10T11:07:47.207657+0000","last_active":"2024-05-10T11:10:50.790060+0000","last_peered":"2024-05-10T11:10:50.790060+0000","last_clean":"2024-05-10T11:10:50.790060+0000","last_became_active":"2024-05-10T11:07:47.207317+0000","last_became_peered":"2024-05-10T11:07:47.207317+0000","last_unstale":"2024-05-10T11:10:50.790060+0000","last_undegraded":"2024-05-10T11:10:50.790060+0000","last_fullsized":"2024-05-10T11:10:50.790060+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:07:33.961642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065113+0000","last_change":"2024-05-10T11:08:00.884076+0000","last_active":"2024-05-10T11:09:23.065113+0000","last_peered":"2024-05-10T11:09:23.065113+0000","last_clean":"2024-05-10T11:09:23.065113+0000","last_became_active":"2024-05-10T11:08:00.883670+0000","last_became_peered":"2024-05-10T11:08:00.883670+0000","last_unstale":"2024-05-10T11:09:23.065113+0000","last_undegraded":"2024-05-10T11:09:23.065113+0000","last_fullsized":"2024-05-10T11:09:23.065113+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:56.447836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067621+0000","last_change":"2024-05-10T11:07:42.135550+0000","last_active":"2024-05-10T11:09:23.067621+0000","last_peered":"2024-05-10T11:09:23.067621+0000","last_clean":"2024-05-10T11:09:23.067621+0000","last_became_active":"2024-05-10T11:07:42.135192+0000","last_became_peered":"2024-05-10T11:07:42.135192+0000","last_unstale":"2024-05-10T11:09:23.067621+0000","last_undegraded":"2024-05-10T11:09:23.067621+0000","last_fullsized":"2024-05-10T11:09:23.067621+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:17.100873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.787215+0000","last_change":"2024-05-10T11:07:47.199529+0000","last_active":"2024-05-10T11:10:50.787215+0000","last_peered":"2024-05-10T11:10:50.787215+0000","last_clean":"2024-05-10T11:10:50.787215+0000","last_became_active":"2024-05-10T11:07:47.199070+0000","last_became_peered":"2024-05-10T11:07:47.199070+0000","last_unstale":"2024-05-10T11:10:50.787215+0000","last_undegraded":"2024-05-10T11:10:50.787215+0000","last_fullsized":"2024-05-10T11:10:50.787215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:24.446639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068411+0000","last_change":"2024-05-10T11:08:00.898607+0000","last_active":"2024-05-10T11:09:23.068411+0000","last_peered":"2024-05-10T11:09:23.068411+0000","last_clean":"2024-05-10T11:09:23.068411+0000","last_became_active":"2024-05-10T11:08:00.897768+0000","last_became_peered":"2024-05-10T11:08:00.897768+0000","last_unstale":"2024-05-10T11:09:23.068411+0000","last_undegraded":"2024-05-10T11:09:23.068411+0000","last_fullsized":"2024-05-10T11:09:23.068411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:24:42.125203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":172,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:10.250255+0000","last_change":"2024-05-10T11:07:42.126907+0000","last_active":"2024-05-10T11:11:10.250255+0000","last_peered":"2024-05-10T11:11:10.250255+0000","last_clean":"2024-05-10T11:11:10.250255+0000","last_became_active":"2024-05-10T11:07:42.126462+0000","last_became_peered":"2024-05-10T11:07:42.126462+0000","last_unstale":"2024-05-10T11:11:10.250255+0000","last_undegraded":"2024-05-10T11:11:10.250255+0000","last_fullsized":"2024-05-10T11:11:10.250255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:18:53.971150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.755058+0000","last_change":"2024-05-10T11:07:47.191888+0000","last_active":"2024-05-10T11:10:50.755058+0000","last_peered":"2024-05-10T11:10:50.755058+0000","last_clean":"2024-05-10T11:10:50.755058+0000","last_became_active":"2024-05-10T11:07:47.191557+0000","last_became_peered":"2024-05-10T11:07:47.191557+0000","last_unstale":"2024-05-10T11:10:50.755058+0000","last_undegraded":"2024-05-10T11:10:50.755058+0000","last_fullsized":"2024-05-10T11:10:50.755058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:01:36.529493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079432+0000","last_change":"2024-05-10T11:08:00.894385+0000","last_active":"2024-05-10T11:09:23.079432+0000","last_peered":"2024-05-10T11:09:23.079432+0000","last_clean":"2024-05-10T11:09:23.079432+0000","last_became_active":"2024-05-10T11:08:00.893706+0000","last_became_peered":"2024-05-10T11:08:00.893706+0000","last_unstale":"2024-05-10T11:09:23.079432+0000","last_undegraded":"2024-05-10T11:09:23.079432+0000","last_fullsized":"2024-05-10T11:09:23.079432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:02.069878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068853+0000","last_change":"2024-05-10T11:07:42.385661+0000","last_active":"2024-05-10T11:09:23.068853+0000","last_peered":"2024-05-10T11:09:23.068853+0000","last_clean":"2024-05-10T11:09:23.068853+0000","last_became_active":"2024-05-10T11:07:42.385308+0000","last_became_peered":"2024-05-10T11:07:42.385308+0000","last_unstale":"2024-05-10T11:09:23.068853+0000","last_undegraded":"2024-05-10T11:09:23.068853+0000","last_fullsized":"2024-05-10T11:09:23.068853+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:57:25.997730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.739312+0000","last_change":"2024-05-10T11:07:47.191574+0000","last_active":"2024-05-10T11:10:50.739312+0000","last_peered":"2024-05-10T11:10:50.739312+0000","last_clean":"2024-05-10T11:10:50.739312+0000","last_became_active":"2024-05-10T11:07:47.191210+0000","last_became_peered":"2024-05-10T11:07:47.191210+0000","last_unstale":"2024-05-10T11:10:50.739312+0000","last_undegraded":"2024-05-10T11:10:50.739312+0000","last_fullsized":"2024-05-10T11:10:50.739312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:11:35.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065620+0000","last_change":"2024-05-10T11:08:00.890076+0000","last_active":"2024-05-10T11:09:23.065620+0000","last_peered":"2024-05-10T11:09:23.065620+0000","last_clean":"2024-05-10T11:09:23.065620+0000","last_became_active":"2024-05-10T11:08:00.889630+0000","last_became_peered":"2024-05-10T11:08:00.889630+0000","last_unstale":"2024-05-10T11:09:23.065620+0000","last_undegraded":"2024-05-10T11:09:23.065620+0000","last_fullsized":"2024-05-10T11:09:23.065620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:53:29.257873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.075061+0000","last_change":"2024-05-10T11:09:23.074550+0000","last_active":"2024-05-10T11:09:23.075061+0000","last_peered":"2024-05-10T11:09:23.075061+0000","last_clean":"2024-05-10T11:09:23.075061+0000","last_became_active":"2024-05-10T11:09:23.067169+0000","last_became_peered":"2024-05-10T11:09:23.067169+0000","last_unstale":"2024-05-10T11:09:23.075061+0000","last_undegraded":"2024-05-10T11:09:23.075061+0000","last_fullsized":"2024-05-10T11:09:23.075061+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:45:06.303003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.755360+0000","last_change":"2024-05-10T11:07:47.192042+0000","last_active":"2024-05-10T11:10:50.755360+0000","last_peered":"2024-05-10T11:10:50.755360+0000","last_clean":"2024-05-10T11:10:50.755360+0000","last_became_active":"2024-05-10T11:07:47.191632+0000","last_became_peered":"2024-05-10T11:07:47.191632+0000","last_unstale":"2024-05-10T11:10:50.755360+0000","last_undegraded":"2024-05-10T11:10:50.755360+0000","last_fullsized":"2024-05-10T11:10:50.755360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:55:08.564841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069056+0000","last_change":"2024-05-10T11:08:22.244721+0000","last_active":"2024-05-10T11:09:23.069056+0000","last_peered":"2024-05-10T11:09:23.069056+0000","last_clean":"2024-05-10T11:09:23.069056+0000","last_became_active":"2024-05-10T11:08:22.244274+0000","last_became_peered":"2024-05-10T11:08:22.244274+0000","last_unstale":"2024-05-10T11:09:23.069056+0000","last_undegraded":"2024-05-10T11:09:23.069056+0000","last_fullsized":"2024-05-10T11:09:23.069056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:26.218071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070759+0000","last_change":"2024-05-10T11:07:42.388358+0000","last_active":"2024-05-10T11:09:23.070759+0000","last_peered":"2024-05-10T11:09:23.070759+0000","last_clean":"2024-05-10T11:09:23.070759+0000","last_became_active":"2024-05-10T11:07:42.387806+0000","last_became_peered":"2024-05-10T11:07:42.387806+0000","last_unstale":"2024-05-10T11:09:23.070759+0000","last_undegraded":"2024-05-10T11:09:23.070759+0000","last_fullsized":"2024-05-10T11:09:23.070759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:27:38.387332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.783965+0000","last_change":"2024-05-10T11:07:47.213602+0000","last_active":"2024-05-10T11:10:50.783965+0000","last_peered":"2024-05-10T11:10:50.783965+0000","last_clean":"2024-05-10T11:10:50.783965+0000","last_became_active":"2024-05-10T11:07:47.213184+0000","last_became_peered":"2024-05-10T11:07:47.213184+0000","last_unstale":"2024-05-10T11:10:50.783965+0000","last_undegraded":"2024-05-10T11:10:50.783965+0000","last_fullsized":"2024-05-10T11:10:50.783965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T23:01:52.253279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079684+0000","last_change":"2024-05-10T11:08:00.889850+0000","last_active":"2024-05-10T11:09:23.079684+0000","last_peered":"2024-05-10T11:09:23.079684+0000","last_clean":"2024-05-10T11:09:23.079684+0000","last_became_active":"2024-05-10T11:08:00.888526+0000","last_became_peered":"2024-05-10T11:08:00.888526+0000","last_unstale":"2024-05-10T11:09:23.079684+0000","last_undegraded":"2024-05-10T11:09:23.079684+0000","last_fullsized":"2024-05-10T11:09:23.079684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:11:43.899628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:13.309811+0000","last_change":"2024-05-10T11:07:42.130650+0000","last_active":"2024-05-10T11:11:13.309811+0000","last_peered":"2024-05-10T11:11:13.309811+0000","last_clean":"2024-05-10T11:11:13.309811+0000","last_became_active":"2024-05-10T11:07:42.130259+0000","last_became_peered":"2024-05-10T11:07:42.130259+0000","last_unstale":"2024-05-10T11:11:13.309811+0000","last_undegraded":"2024-05-10T11:11:13.309811+0000","last_fullsized":"2024-05-10T11:11:13.309811+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.870092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.763336+0000","last_change":"2024-05-10T11:07:47.204159+0000","last_active":"2024-05-10T11:10:50.763336+0000","last_peered":"2024-05-10T11:10:50.763336+0000","last_clean":"2024-05-10T11:10:50.763336+0000","last_became_active":"2024-05-10T11:07:47.203723+0000","last_became_peered":"2024-05-10T11:07:47.203723+0000","last_unstale":"2024-05-10T11:10:50.763336+0000","last_undegraded":"2024-05-10T11:10:50.763336+0000","last_fullsized":"2024-05-10T11:10:50.763336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:44:48.232321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.082000+0000","last_change":"2024-05-10T11:08:00.918501+0000","last_active":"2024-05-10T11:09:23.082000+0000","last_peered":"2024-05-10T11:09:23.082000+0000","last_clean":"2024-05-10T11:09:23.082000+0000","last_became_active":"2024-05-10T11:08:00.918071+0000","last_became_peered":"2024-05-10T11:08:00.918071+0000","last_unstale":"2024-05-10T11:09:23.082000+0000","last_undegraded":"2024-05-10T11:09:23.082000+0000","last_fullsized":"2024-05-10T11:09:23.082000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:23:08.390953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.785780+0000","last_change":"2024-05-10T11:08:22.036385+0000","last_active":"2024-05-10T11:10:50.785780+0000","last_peered":"2024-05-10T11:10:50.785780+0000","last_clean":"2024-05-10T11:10:50.785780+0000","last_became_active":"2024-05-10T11:08:22.035969+0000","last_became_peered":"2024-05-10T11:08:22.035969+0000","last_unstale":"2024-05-10T11:10:50.785780+0000","last_undegraded":"2024-05-10T11:10:50.785780+0000","last_fullsized":"2024-05-10T11:10:50.785780+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:14:24.069422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:12.253472+0000","last_change":"2024-05-10T11:07:42.381926+0000","last_active":"2024-05-10T11:11:12.253472+0000","last_peered":"2024-05-10T11:11:12.253472+0000","last_clean":"2024-05-10T11:11:12.253472+0000","last_became_active":"2024-05-10T11:07:42.381301+0000","last_became_peered":"2024-05-10T11:07:42.381301+0000","last_unstale":"2024-05-10T11:11:12.253472+0000","last_undegraded":"2024-05-10T11:11:12.253472+0000","last_fullsized":"2024-05-10T11:11:12.253472+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:48:32.291029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:27.923059+0000","last_change":"2024-05-10T11:08:00.889959+0000","last_active":"2024-05-10T11:09:27.923059+0000","last_peered":"2024-05-10T11:09:27.923059+0000","last_clean":"2024-05-10T11:09:27.923059+0000","last_became_active":"2024-05-10T11:08:00.889514+0000","last_became_peered":"2024-05-10T11:08:00.889514+0000","last_unstale":"2024-05-10T11:09:27.923059+0000","last_undegraded":"2024-05-10T11:09:27.923059+0000","last_fullsized":"2024-05-10T11:09:27.923059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.557175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.779373+0000","last_change":"2024-05-10T11:07:47.218025+0000","last_active":"2024-05-10T11:10:50.779373+0000","last_peered":"2024-05-10T11:10:50.779373+0000","last_clean":"2024-05-10T11:10:50.779373+0000","last_became_active":"2024-05-10T11:07:47.207089+0000","last_became_peered":"2024-05-10T11:07:47.207089+0000","last_unstale":"2024-05-10T11:10:50.779373+0000","last_undegraded":"2024-05-10T11:10:50.779373+0000","last_fullsized":"2024-05-10T11:10:50.779373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:15:50.363129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062645+0000","last_change":"2024-05-10T11:07:42.138182+0000","last_active":"2024-05-10T11:09:23.062645+0000","last_peered":"2024-05-10T11:09:23.062645+0000","last_clean":"2024-05-10T11:09:23.062645+0000","last_became_active":"2024-05-10T11:07:42.137734+0000","last_became_peered":"2024-05-10T11:07:42.137734+0000","last_unstale":"2024-05-10T11:09:23.062645+0000","last_undegraded":"2024-05-10T11:09:23.062645+0000","last_fullsized":"2024-05-10T11:09:23.062645+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:02:30.778878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071459+0000","last_change":"2024-05-10T11:06:06.321257+0000","last_active":"2024-05-10T11:09:23.071459+0000","last_peered":"2024-05-10T11:09:23.071459+0000","last_clean":"2024-05-10T11:09:23.071459+0000","last_became_active":"2024-05-10T11:06:06.316928+0000","last_became_peered":"2024-05-10T11:06:06.316928+0000","last_unstale":"2024-05-10T11:09:23.071459+0000","last_undegraded":"2024-05-10T11:09:23.071459+0000","last_fullsized":"2024-05-10T11:09:23.071459+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_clean_scrub_stamp":"2024-05-10T11:02:07.346111+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:32:43.670246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065321+0000","last_change":"2024-05-10T11:08:00.918499+0000","last_active":"2024-05-10T11:09:23.065321+0000","last_peered":"2024-05-10T11:09:23.065321+0000","last_clean":"2024-05-10T11:09:23.065321+0000","last_became_active":"2024-05-10T11:08:00.917950+0000","last_became_peered":"2024-05-10T11:08:00.917950+0000","last_unstale":"2024-05-10T11:09:23.065321+0000","last_undegraded":"2024-05-10T11:09:23.065321+0000","last_fullsized":"2024-05-10T11:09:23.065321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:07.591312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.758046+0000","last_change":"2024-05-10T11:07:47.194968+0000","last_active":"2024-05-10T11:10:50.758046+0000","last_peered":"2024-05-10T11:10:50.758046+0000","last_clean":"2024-05-10T11:10:50.758046+0000","last_became_active":"2024-05-10T11:07:47.194645+0000","last_became_peered":"2024-05-10T11:07:47.194645+0000","last_unstale":"2024-05-10T11:10:50.758046+0000","last_undegraded":"2024-05-10T11:10:50.758046+0000","last_fullsized":"2024-05-10T11:10:50.758046+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:34.962163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072001+0000","last_change":"2024-05-10T11:07:42.126378+0000","last_active":"2024-05-10T11:09:23.072001+0000","last_peered":"2024-05-10T11:09:23.072001+0000","last_clean":"2024-05-10T11:09:23.072001+0000","last_became_active":"2024-05-10T11:07:42.125984+0000","last_became_peered":"2024-05-10T11:07:42.125984+0000","last_unstale":"2024-05-10T11:09:23.072001+0000","last_undegraded":"2024-05-10T11:09:23.072001+0000","last_fullsized":"2024-05-10T11:09:23.072001+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:15:50.378737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065803+0000","last_change":"2024-05-10T11:08:00.872342+0000","last_active":"2024-05-10T11:09:23.065803+0000","last_peered":"2024-05-10T11:09:23.065803+0000","last_clean":"2024-05-10T11:09:23.065803+0000","last_became_active":"2024-05-10T11:08:00.871926+0000","last_became_peered":"2024-05-10T11:08:00.871926+0000","last_unstale":"2024-05-10T11:09:23.065803+0000","last_undegraded":"2024-05-10T11:09:23.065803+0000","last_fullsized":"2024-05-10T11:09:23.065803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:27:16.519752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070093+0000","last_change":"2024-05-10T11:07:42.384236+0000","last_active":"2024-05-10T11:09:23.070093+0000","last_peered":"2024-05-10T11:09:23.070093+0000","last_clean":"2024-05-10T11:09:23.070093+0000","last_became_active":"2024-05-10T11:07:42.383908+0000","last_became_peered":"2024-05-10T11:07:42.383908+0000","last_unstale":"2024-05-10T11:09:23.070093+0000","last_undegraded":"2024-05-10T11:09:23.070093+0000","last_fullsized":"2024-05-10T11:09:23.070093+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:09:42.302506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.770400+0000","last_change":"2024-05-10T11:07:47.201645+0000","last_active":"2024-05-10T11:10:50.770400+0000","last_peered":"2024-05-10T11:10:50.770400+0000","last_clean":"2024-05-10T11:10:50.770400+0000","last_became_active":"2024-05-10T11:07:47.201300+0000","last_became_peered":"2024-05-10T11:07:47.201300+0000","last_unstale":"2024-05-10T11:10:50.770400+0000","last_undegraded":"2024-05-10T11:10:50.770400+0000","last_fullsized":"2024-05-10T11:10:50.770400+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:37.268530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077040+0000","last_change":"2024-05-10T11:08:00.917988+0000","last_active":"2024-05-10T11:09:23.077040+0000","last_peered":"2024-05-10T11:09:23.077040+0000","last_clean":"2024-05-10T11:09:23.077040+0000","last_became_active":"2024-05-10T11:08:00.917457+0000","last_became_peered":"2024-05-10T11:08:00.917457+0000","last_unstale":"2024-05-10T11:09:23.077040+0000","last_undegraded":"2024-05-10T11:09:23.077040+0000","last_fullsized":"2024-05-10T11:09:23.077040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:16:01.816917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'17","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:52.582808+0000","last_change":"2024-05-10T11:07:42.141005+0000","last_active":"2024-05-10T11:09:52.582808+0000","last_peered":"2024-05-10T11:09:52.582808+0000","last_clean":"2024-05-10T11:09:52.582808+0000","last_became_active":"2024-05-10T11:07:42.140634+0000","last_became_peered":"2024-05-10T11:07:42.140634+0000","last_unstale":"2024-05-10T11:09:52.582808+0000","last_undegraded":"2024-05-10T11:09:52.582808+0000","last_fullsized":"2024-05-10T11:09:52.582808+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:00:32.029374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.788308+0000","last_change":"2024-05-10T11:07:47.206864+0000","last_active":"2024-05-10T11:10:50.788308+0000","last_peered":"2024-05-10T11:10:50.788308+0000","last_clean":"2024-05-10T11:10:50.788308+0000","last_became_active":"2024-05-10T11:07:47.206387+0000","last_became_peered":"2024-05-10T11:07:47.206387+0000","last_unstale":"2024-05-10T11:10:50.788308+0000","last_undegraded":"2024-05-10T11:10:50.788308+0000","last_fullsized":"2024-05-10T11:10:50.788308+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:12:35.242270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069114+0000","last_change":"2024-05-10T11:08:00.867872+0000","last_active":"2024-05-10T11:09:23.069114+0000","last_peered":"2024-05-10T11:09:23.069114+0000","last_clean":"2024-05-10T11:09:23.069114+0000","last_became_active":"2024-05-10T11:08:00.867461+0000","last_became_peered":"2024-05-10T11:08:00.867461+0000","last_unstale":"2024-05-10T11:09:23.069114+0000","last_undegraded":"2024-05-10T11:09:23.069114+0000","last_fullsized":"2024-05-10T11:09:23.069114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:29.617335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.063128+0000","last_change":"2024-05-10T11:07:42.599520+0000","last_active":"2024-05-10T11:09:23.063128+0000","last_peered":"2024-05-10T11:09:23.063128+0000","last_clean":"2024-05-10T11:09:23.063128+0000","last_became_active":"2024-05-10T11:07:42.599209+0000","last_became_peered":"2024-05-10T11:07:42.599209+0000","last_unstale":"2024-05-10T11:09:23.063128+0000","last_undegraded":"2024-05-10T11:09:23.063128+0000","last_fullsized":"2024-05-10T11:09:23.063128+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:52.899718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.799440+0000","last_change":"2024-05-10T11:07:47.218286+0000","last_active":"2024-05-10T11:10:50.799440+0000","last_peered":"2024-05-10T11:10:50.799440+0000","last_clean":"2024-05-10T11:10:50.799440+0000","last_became_active":"2024-05-10T11:07:47.217960+0000","last_became_peered":"2024-05-10T11:07:47.217960+0000","last_unstale":"2024-05-10T11:10:50.799440+0000","last_undegraded":"2024-05-10T11:10:50.799440+0000","last_fullsized":"2024-05-10T11:10:50.799440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:11:37.208359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062781+0000","last_change":"2024-05-10T11:08:00.919578+0000","last_active":"2024-05-10T11:09:23.062781+0000","last_peered":"2024-05-10T11:09:23.062781+0000","last_clean":"2024-05-10T11:09:23.062781+0000","last_became_active":"2024-05-10T11:08:00.918546+0000","last_became_peered":"2024-05-10T11:08:00.918546+0000","last_unstale":"2024-05-10T11:09:23.062781+0000","last_undegraded":"2024-05-10T11:09:23.062781+0000","last_fullsized":"2024-05-10T11:09:23.062781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:39:05.062210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072522+0000","last_change":"2024-05-10T11:07:42.384577+0000","last_active":"2024-05-10T11:09:23.072522+0000","last_peered":"2024-05-10T11:09:23.072522+0000","last_clean":"2024-05-10T11:09:23.072522+0000","last_became_active":"2024-05-10T11:07:42.384301+0000","last_became_peered":"2024-05-10T11:07:42.384301+0000","last_unstale":"2024-05-10T11:09:23.072522+0000","last_undegraded":"2024-05-10T11:09:23.072522+0000","last_fullsized":"2024-05-10T11:09:23.072522+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:11:03.359490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.723954+0000","last_change":"2024-05-10T11:07:47.189708+0000","last_active":"2024-05-10T11:10:50.723954+0000","last_peered":"2024-05-10T11:10:50.723954+0000","last_clean":"2024-05-10T11:10:50.723954+0000","last_became_active":"2024-05-10T11:07:47.189367+0000","last_became_peered":"2024-05-10T11:07:47.189367+0000","last_unstale":"2024-05-10T11:10:50.723954+0000","last_undegraded":"2024-05-10T11:10:50.723954+0000","last_fullsized":"2024-05-10T11:10:50.723954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:25.410477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068671+0000","last_change":"2024-05-10T11:08:00.899359+0000","last_active":"2024-05-10T11:09:23.068671+0000","last_peered":"2024-05-10T11:09:23.068671+0000","last_clean":"2024-05-10T11:09:23.068671+0000","last_became_active":"2024-05-10T11:08:00.898924+0000","last_became_peered":"2024-05-10T11:08:00.898924+0000","last_unstale":"2024-05-10T11:09:23.068671+0000","last_undegraded":"2024-05-10T11:09:23.068671+0000","last_fullsized":"2024-05-10T11:09:23.068671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:14:17.427245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076950+0000","last_change":"2024-05-10T11:07:42.140262+0000","last_active":"2024-05-10T11:09:23.076950+0000","last_peered":"2024-05-10T11:09:23.076950+0000","last_clean":"2024-05-10T11:09:23.076950+0000","last_became_active":"2024-05-10T11:07:42.138853+0000","last_became_peered":"2024-05-10T11:07:42.138853+0000","last_unstale":"2024-05-10T11:09:23.076950+0000","last_undegraded":"2024-05-10T11:09:23.076950+0000","last_fullsized":"2024-05-10T11:09:23.076950+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:05:39.995237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.764312+0000","last_change":"2024-05-10T11:07:47.203637+0000","last_active":"2024-05-10T11:10:50.764312+0000","last_peered":"2024-05-10T11:10:50.764312+0000","last_clean":"2024-05-10T11:10:50.764312+0000","last_became_active":"2024-05-10T11:07:47.203286+0000","last_became_peered":"2024-05-10T11:07:47.203286+0000","last_unstale":"2024-05-10T11:10:50.764312+0000","last_undegraded":"2024-05-10T11:10:50.764312+0000","last_fullsized":"2024-05-10T11:10:50.764312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:27:46.924625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:35.945269+0000","last_change":"2024-05-10T11:08:00.890082+0000","last_active":"2024-05-10T11:09:35.945269+0000","last_peered":"2024-05-10T11:09:35.945269+0000","last_clean":"2024-05-10T11:09:35.945269+0000","last_became_active":"2024-05-10T11:08:00.889673+0000","last_became_peered":"2024-05-10T11:08:00.889673+0000","last_unstale":"2024-05-10T11:09:35.945269+0000","last_undegraded":"2024-05-10T11:09:35.945269+0000","last_fullsized":"2024-05-10T11:09:35.945269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:21:31.167934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069404+0000","last_change":"2024-05-10T11:07:42.130703+0000","last_active":"2024-05-10T11:09:23.069404+0000","last_peered":"2024-05-10T11:09:23.069404+0000","last_clean":"2024-05-10T11:09:23.069404+0000","last_became_active":"2024-05-10T11:07:42.130098+0000","last_became_peered":"2024-05-10T11:07:42.130098+0000","last_unstale":"2024-05-10T11:09:23.069404+0000","last_undegraded":"2024-05-10T11:09:23.069404+0000","last_fullsized":"2024-05-10T11:09:23.069404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:59:39.783151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.806618+0000","last_change":"2024-05-10T11:07:47.220497+0000","last_active":"2024-05-10T11:10:50.806618+0000","last_peered":"2024-05-10T11:10:50.806618+0000","last_clean":"2024-05-10T11:10:50.806618+0000","last_became_active":"2024-05-10T11:07:47.219916+0000","last_became_peered":"2024-05-10T11:07:47.219916+0000","last_unstale":"2024-05-10T11:10:50.806618+0000","last_undegraded":"2024-05-10T11:10:50.806618+0000","last_fullsized":"2024-05-10T11:10:50.806618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:26:43.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062702+0000","last_change":"2024-05-10T11:08:00.918258+0000","last_active":"2024-05-10T11:09:23.062702+0000","last_peered":"2024-05-10T11:09:23.062702+0000","last_clean":"2024-05-10T11:09:23.062702+0000","last_became_active":"2024-05-10T11:08:00.917808+0000","last_became_peered":"2024-05-10T11:08:00.917808+0000","last_unstale":"2024-05-10T11:09:23.062702+0000","last_undegraded":"2024-05-10T11:09:23.062702+0000","last_fullsized":"2024-05-10T11:09:23.062702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:14:08.867024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068897+0000","last_change":"2024-05-10T11:07:42.127661+0000","last_active":"2024-05-10T11:09:23.068897+0000","last_peered":"2024-05-10T11:09:23.068897+0000","last_clean":"2024-05-10T11:09:23.068897+0000","last_became_active":"2024-05-10T11:07:42.127256+0000","last_became_peered":"2024-05-10T11:07:42.127256+0000","last_unstale":"2024-05-10T11:09:23.068897+0000","last_undegraded":"2024-05-10T11:09:23.068897+0000","last_fullsized":"2024-05-10T11:09:23.068897+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:14:37.821350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.786018+0000","last_change":"2024-05-10T11:07:47.199589+0000","last_active":"2024-05-10T11:10:50.786018+0000","last_peered":"2024-05-10T11:10:50.786018+0000","last_clean":"2024-05-10T11:10:50.786018+0000","last_became_active":"2024-05-10T11:07:47.199145+0000","last_became_peered":"2024-05-10T11:07:47.199145+0000","last_unstale":"2024-05-10T11:10:50.786018+0000","last_undegraded":"2024-05-10T11:10:50.786018+0000","last_fullsized":"2024-05-10T11:10:50.786018+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:46.577543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.085808+0000","last_change":"2024-05-10T11:09:23.085049+0000","last_active":"2024-05-10T11:09:23.085808+0000","last_peered":"2024-05-10T11:09:23.085808+0000","last_clean":"2024-05-10T11:09:23.085808+0000","last_became_active":"2024-05-10T11:09:23.084420+0000","last_became_peered":"2024-05-10T11:09:23.084420+0000","last_unstale":"2024-05-10T11:09:23.085808+0000","last_undegraded":"2024-05-10T11:09:23.085808+0000","last_fullsized":"2024-05-10T11:09:23.085808+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:32:33.346434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'5","reported_seq":173,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:15.223310+0000","last_change":"2024-05-10T11:07:42.599395+0000","last_active":"2024-05-10T11:11:15.223310+0000","last_peered":"2024-05-10T11:11:15.223310+0000","last_clean":"2024-05-10T11:11:15.223310+0000","last_became_active":"2024-05-10T11:07:42.598934+0000","last_became_peered":"2024-05-10T11:07:42.598934+0000","last_unstale":"2024-05-10T11:11:15.223310+0000","last_undegraded":"2024-05-10T11:11:15.223310+0000","last_fullsized":"2024-05-10T11:11:15.223310+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:49:42.414789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.769182+0000","last_change":"2024-05-10T11:07:47.192087+0000","last_active":"2024-05-10T11:10:50.769182+0000","last_peered":"2024-05-10T11:10:50.769182+0000","last_clean":"2024-05-10T11:10:50.769182+0000","last_became_active":"2024-05-10T11:07:47.191041+0000","last_became_peered":"2024-05-10T11:07:47.191041+0000","last_unstale":"2024-05-10T11:10:50.769182+0000","last_undegraded":"2024-05-10T11:10:50.769182+0000","last_fullsized":"2024-05-10T11:10:50.769182+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:50:14.052863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.081786+0000","last_change":"2024-05-10T11:09:23.080432+0000","last_active":"2024-05-10T11:09:23.081786+0000","last_peered":"2024-05-10T11:09:23.081786+0000","last_clean":"2024-05-10T11:09:23.081786+0000","last_became_active":"2024-05-10T11:09:23.074934+0000","last_became_peered":"2024-05-10T11:09:23.074934+0000","last_unstale":"2024-05-10T11:09:23.081786+0000","last_undegraded":"2024-05-10T11:09:23.081786+0000","last_fullsized":"2024-05-10T11:09:23.081786+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:12:14.613323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066498+0000","last_change":"2024-05-10T11:07:42.130718+0000","last_active":"2024-05-10T11:09:23.066498+0000","last_peered":"2024-05-10T11:09:23.066498+0000","last_clean":"2024-05-10T11:09:23.066498+0000","last_became_active":"2024-05-10T11:07:42.130355+0000","last_became_peered":"2024-05-10T11:07:42.130355+0000","last_unstale":"2024-05-10T11:09:23.066498+0000","last_undegraded":"2024-05-10T11:09:23.066498+0000","last_fullsized":"2024-05-10T11:09:23.066498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:16:48.093146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.784346+0000","last_change":"2024-05-10T11:07:47.213569+0000","last_active":"2024-05-10T11:10:50.784346+0000","last_peered":"2024-05-10T11:10:50.784346+0000","last_clean":"2024-05-10T11:10:50.784346+0000","last_became_active":"2024-05-10T11:07:47.213148+0000","last_became_peered":"2024-05-10T11:07:47.213148+0000","last_unstale":"2024-05-10T11:10:50.784346+0000","last_undegraded":"2024-05-10T11:10:50.784346+0000","last_fullsized":"2024-05-10T11:10:50.784346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:42:41.161274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066016+0000","last_change":"2024-05-10T11:08:00.870937+0000","last_active":"2024-05-10T11:09:23.066016+0000","last_peered":"2024-05-10T11:09:23.066016+0000","last_clean":"2024-05-10T11:09:23.066016+0000","last_became_active":"2024-05-10T11:08:00.869276+0000","last_became_peered":"2024-05-10T11:08:00.869276+0000","last_unstale":"2024-05-10T11:09:23.066016+0000","last_undegraded":"2024-05-10T11:09:23.066016+0000","last_fullsized":"2024-05-10T11:09:23.066016+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:37.058875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079741+0000","last_change":"2024-05-10T11:07:42.130544+0000","last_active":"2024-05-10T11:09:23.079741+0000","last_peered":"2024-05-10T11:09:23.079741+0000","last_clean":"2024-05-10T11:09:23.079741+0000","last_became_active":"2024-05-10T11:07:42.128912+0000","last_became_peered":"2024-05-10T11:07:42.128912+0000","last_unstale":"2024-05-10T11:09:23.079741+0000","last_undegraded":"2024-05-10T11:09:23.079741+0000","last_fullsized":"2024-05-10T11:09:23.079741+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:22:23.521911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.763514+0000","last_change":"2024-05-10T11:07:47.194346+0000","last_active":"2024-05-10T11:10:50.763514+0000","last_peered":"2024-05-10T11:10:50.763514+0000","last_clean":"2024-05-10T11:10:50.763514+0000","last_became_active":"2024-05-10T11:07:47.194019+0000","last_became_peered":"2024-05-10T11:07:47.194019+0000","last_unstale":"2024-05-10T11:10:50.763514+0000","last_undegraded":"2024-05-10T11:10:50.763514+0000","last_fullsized":"2024-05-10T11:10:50.763514+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:33:05.262347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.759134+0000","last_change":"2024-05-10T11:07:47.192001+0000","last_active":"2024-05-10T11:10:50.759134+0000","last_peered":"2024-05-10T11:10:50.759134+0000","last_clean":"2024-05-10T11:10:50.759134+0000","last_became_active":"2024-05-10T11:07:47.191668+0000","last_became_peered":"2024-05-10T11:07:47.191668+0000","last_unstale":"2024-05-10T11:10:50.759134+0000","last_undegraded":"2024-05-10T11:10:50.759134+0000","last_fullsized":"2024-05-10T11:10:50.759134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:13:33.680899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.749397+0000","last_change":"2024-05-10T11:07:47.192226+0000","last_active":"2024-05-10T11:10:50.749397+0000","last_peered":"2024-05-10T11:10:50.749397+0000","last_clean":"2024-05-10T11:10:50.749397+0000","last_became_active":"2024-05-10T11:07:47.191444+0000","last_became_peered":"2024-05-10T11:07:47.191444+0000","last_unstale":"2024-05-10T11:10:50.749397+0000","last_undegraded":"2024-05-10T11:10:50.749397+0000","last_fullsized":"2024-05-10T11:10:50.749397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:08.348986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.792269+0000","last_change":"2024-05-10T11:07:47.210801+0000","last_active":"2024-05-10T11:10:50.792269+0000","last_peered":"2024-05-10T11:10:50.792269+0000","last_clean":"2024-05-10T11:10:50.792269+0000","last_became_active":"2024-05-10T11:07:47.210496+0000","last_became_peered":"2024-05-10T11:07:47.210496+0000","last_unstale":"2024-05-10T11:10:50.792269+0000","last_undegraded":"2024-05-10T11:10:50.792269+0000","last_fullsized":"2024-05-10T11:10:50.792269+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:11:03.661457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.802348+0000","last_change":"2024-05-10T11:07:47.211033+0000","last_active":"2024-05-10T11:10:50.802348+0000","last_peered":"2024-05-10T11:10:50.802348+0000","last_clean":"2024-05-10T11:10:50.802348+0000","last_became_active":"2024-05-10T11:07:47.206061+0000","last_became_peered":"2024-05-10T11:07:47.206061+0000","last_unstale":"2024-05-10T11:10:50.802348+0000","last_undegraded":"2024-05-10T11:10:50.802348+0000","last_fullsized":"2024-05-10T11:10:50.802348+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:04:05.878270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.805488+0000","last_change":"2024-05-10T11:07:47.214966+0000","last_active":"2024-05-10T11:10:50.805488+0000","last_peered":"2024-05-10T11:10:50.805488+0000","last_clean":"2024-05-10T11:10:50.805488+0000","last_became_active":"2024-05-10T11:07:47.214563+0000","last_became_peered":"2024-05-10T11:07:47.214563+0000","last_unstale":"2024-05-10T11:10:50.805488+0000","last_undegraded":"2024-05-10T11:10:50.805488+0000","last_fullsized":"2024-05-10T11:10:50.805488+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:29:01.428893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.758172+0000","last_change":"2024-05-10T11:07:47.196740+0000","last_active":"2024-05-10T11:10:50.758172+0000","last_peered":"2024-05-10T11:10:50.758172+0000","last_clean":"2024-05-10T11:10:50.758172+0000","last_became_active":"2024-05-10T11:07:47.196388+0000","last_became_peered":"2024-05-10T11:07:47.196388+0000","last_unstale":"2024-05-10T11:10:50.758172+0000","last_undegraded":"2024-05-10T11:10:50.758172+0000","last_fullsized":"2024-05-10T11:10:50.758172+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:23:09.055338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.748036+0000","last_change":"2024-05-10T11:07:47.193568+0000","last_active":"2024-05-10T11:10:50.748036+0000","last_peered":"2024-05-10T11:10:50.748036+0000","last_clean":"2024-05-10T11:10:50.748036+0000","last_became_active":"2024-05-10T11:07:47.193202+0000","last_became_peered":"2024-05-10T11:07:47.193202+0000","last_unstale":"2024-05-10T11:10:50.748036+0000","last_undegraded":"2024-05-10T11:10:50.748036+0000","last_fullsized":"2024-05-10T11:10:50.748036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:36:24.556266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.761674+0000","last_change":"2024-05-10T11:07:47.215344+0000","last_active":"2024-05-10T11:10:50.761674+0000","last_peered":"2024-05-10T11:10:50.761674+0000","last_clean":"2024-05-10T11:10:50.761674+0000","last_became_active":"2024-05-10T11:07:47.214896+0000","last_became_peered":"2024-05-10T11:07:47.214896+0000","last_unstale":"2024-05-10T11:10:50.761674+0000","last_undegraded":"2024-05-10T11:10:50.761674+0000","last_fullsized":"2024-05-10T11:10:50.761674+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:23:07.845212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.762357+0000","last_change":"2024-05-10T11:07:47.219047+0000","last_active":"2024-05-10T11:10:50.762357+0000","last_peered":"2024-05-10T11:10:50.762357+0000","last_clean":"2024-05-10T11:10:50.762357+0000","last_became_active":"2024-05-10T11:07:47.218778+0000","last_became_peered":"2024-05-10T11:07:47.218778+0000","last_unstale":"2024-05-10T11:10:50.762357+0000","last_undegraded":"2024-05-10T11:10:50.762357+0000","last_fullsized":"2024-05-10T11:10:50.762357+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:02:48.945259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.787652+0000","last_change":"2024-05-10T11:07:47.213412+0000","last_active":"2024-05-10T11:10:50.787652+0000","last_peered":"2024-05-10T11:10:50.787652+0000","last_clean":"2024-05-10T11:10:50.787652+0000","last_became_active":"2024-05-10T11:07:47.212963+0000","last_became_peered":"2024-05-10T11:07:47.212963+0000","last_unstale":"2024-05-10T11:10:50.787652+0000","last_undegraded":"2024-05-10T11:10:50.787652+0000","last_fullsized":"2024-05-10T11:10:50.787652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:33:05.224184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.778780+0000","last_change":"2024-05-10T11:07:47.207998+0000","last_active":"2024-05-10T11:10:50.778780+0000","last_peered":"2024-05-10T11:10:50.778780+0000","last_clean":"2024-05-10T11:10:50.778780+0000","last_became_active":"2024-05-10T11:07:47.207298+0000","last_became_peered":"2024-05-10T11:07:47.207298+0000","last_unstale":"2024-05-10T11:10:50.778780+0000","last_undegraded":"2024-05-10T11:10:50.778780+0000","last_fullsized":"2024-05-10T11:10:50.778780+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:59:14.001103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.806604+0000","last_change":"2024-05-10T11:07:47.202216+0000","last_active":"2024-05-10T11:10:50.806604+0000","last_peered":"2024-05-10T11:10:50.806604+0000","last_clean":"2024-05-10T11:10:50.806604+0000","last_became_active":"2024-05-10T11:07:47.201867+0000","last_became_peered":"2024-05-10T11:07:47.201867+0000","last_unstale":"2024-05-10T11:10:50.806604+0000","last_undegraded":"2024-05-10T11:10:50.806604+0000","last_fullsized":"2024-05-10T11:10:50.806604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:48:59.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.757654+0000","last_change":"2024-05-10T11:07:47.194149+0000","last_active":"2024-05-10T11:10:50.757654+0000","last_peered":"2024-05-10T11:10:50.757654+0000","last_clean":"2024-05-10T11:10:50.757654+0000","last_became_active":"2024-05-10T11:07:47.193838+0000","last_became_peered":"2024-05-10T11:07:47.193838+0000","last_unstale":"2024-05-10T11:10:50.757654+0000","last_undegraded":"2024-05-10T11:10:50.757654+0000","last_fullsized":"2024-05-10T11:10:50.757654+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:46:44.181446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247309,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71848,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1224,"num_read_kb":692,"num_write":257,"num_write_kb":149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":712,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":215544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678076,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":522636,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874306,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1150976,"data_stored":1174788,"data_compressed":68282,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070534,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":526572,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266766,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":522832,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462995,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":1119492,"data_compressed":68282,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626521,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":517980,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":583904,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738491,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":519436,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":583434,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182506,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":517688,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986276,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":523208,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822752,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":1114542,"data_compressed":68282,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34167,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34167,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":66176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":67162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-10T11:11:17.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-10T11:11:18.227 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:17 smithi076 ceph-mon[34152]: pgmap v438: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:18.228 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:17 smithi076 ceph-mon[34152]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:11:18.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:17 smithi189 ceph-mon[34842]: pgmap v438: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:18.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:17 smithi189 ceph-mon[34842]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:11:18.332 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:18.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:17 smithi027 ceph-mon[30054]: pgmap v438: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:18.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:17 smithi027 ceph-mon[30054]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:11:20.105 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:19 smithi027 ceph-mon[30054]: pgmap v439: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:20.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:19 smithi189 ceph-mon[34842]: pgmap v439: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:20.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:19 smithi076 ceph-mon[34152]: pgmap v439: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.663+0000 7f4959a00700 1 -- 172.21.15.27:0/1619067320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4954103750 msgr2=0x7f4954103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.663+0000 7f4959a00700 1 --2- 172.21.15.27:0/1619067320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4954103750 0x7f4954103b30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4944009920 tx=0x7f4944009c30 comp rx=0 tx=0).stop 2024-05-10T11:11:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.663+0000 7f4959a00700 1 -- 172.21.15.27:0/1619067320 shutdown_connections 2024-05-10T11:11:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.663+0000 7f4959a00700 1 --2- 172.21.15.27:0/1619067320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4954108850 0x7f49541103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.663+0000 7f4959a00700 1 --2- 172.21.15.27:0/1619067320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4954103750 0x7f4954103b30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.663+0000 7f4959a00700 1 --2- 172.21.15.27:0/1619067320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4954104100 0x7f4954108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.663+0000 7f4959a00700 1 -- 172.21.15.27:0/1619067320 >> 172.21.15.27:0/1619067320 conn(0x7f495407a600 msgr2=0x7f495407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.663+0000 7f4959a00700 1 -- 172.21.15.27:0/1619067320 shutdown_connections 2024-05-10T11:11:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.663+0000 7f4959a00700 1 -- 172.21.15.27:0/1619067320 wait complete. 2024-05-10T11:11:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.664+0000 7f4959a00700 1 Processor -- start 2024-05-10T11:11:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.665+0000 7f4959a00700 1 -- start start 2024-05-10T11:11:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.665+0000 7f4959a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4954103750 0x7f495419f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.665+0000 7f4959a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4954104100 0x7f495419fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.665+0000 7f4959a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4954108850 0x7f49541999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.665+0000 7f4959a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4954113670 con 0x7f4954108850 2024-05-10T11:11:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.665+0000 7f4959a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f49541134f0 con 0x7f4954103750 2024-05-10T11:11:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.665+0000 7f4959a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f49541137f0 con 0x7f4954104100 2024-05-10T11:11:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.666+0000 7f4952a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4954103750 0x7f495419f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.666+0000 7f4952a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4954103750 0x7f495419f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50088/0 (socket says 172.21.15.27:50088) 2024-05-10T11:11:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.666+0000 7f4952a00700 1 -- 172.21.15.27:0/578136959 learned_addr learned my addr 172.21.15.27:0/578136959 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.666+0000 7f4952000700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4954104100 0x7f495419fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.666+0000 7f4952a00700 1 -- 172.21.15.27:0/578136959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4954104100 msgr2=0x7f495419fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.666+0000 7f4952a00700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4954104100 0x7f495419fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.666+0000 7f4952a00700 1 -- 172.21.15.27:0/578136959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4954108850 msgr2=0x7f49541999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.666+0000 7f4952a00700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4954108850 0x7f49541999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.666+0000 7f4952a00700 1 -- 172.21.15.27:0/578136959 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495419a130 con 0x7f4954103750 2024-05-10T11:11:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.667+0000 7f4952a00700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4954103750 0x7f495419f810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4944000c00 tx=0x7f494400f990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.667+0000 7f494b400700 1 -- 172.21.15.27:0/578136959 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4944027070 con 0x7f4954103750 2024-05-10T11:11:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.667+0000 7f494b400700 1 -- 172.21.15.27:0/578136959 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4944005570 con 0x7f4954103750 2024-05-10T11:11:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.667+0000 7f4959a00700 1 -- 172.21.15.27:0/578136959 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f495419a3c0 con 0x7f4954103750 2024-05-10T11:11:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.667+0000 7f494b400700 1 -- 172.21.15.27:0/578136959 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4944004930 con 0x7f4954103750 2024-05-10T11:11:20.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.667+0000 7f4959a00700 1 -- 172.21.15.27:0/578136959 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49541a66e0 con 0x7f4954103750 2024-05-10T11:11:20.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.668+0000 7f4959a00700 1 -- 172.21.15.27:0/578136959 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4954104be0 con 0x7f4954103750 2024-05-10T11:11:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.671+0000 7f494b400700 1 -- 172.21.15.27:0/578136959 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f494402a070 con 0x7f4954103750 2024-05-10T11:11:20.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.673+0000 7f494b400700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49380b2880 0x7f49380b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:20.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.673+0000 7f494b400700 1 -- 172.21.15.27:0/578136959 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f49440e3450 con 0x7f4954103750 2024-05-10T11:11:20.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.673+0000 7f4952000700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49380b2880 0x7f49380b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:20.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.674+0000 7f4952000700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49380b2880 0x7f49380b4d40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4954103140 tx=0x7f4940009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:20.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.674+0000 7f494b400700 1 -- 172.21.15.27:0/578136959 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4944031020 con 0x7f4954103750 2024-05-10T11:11:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.913+0000 7f4959a00700 1 -- 172.21.15.27:0/578136959 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f49540638e0 con 0x7f49380b2880 2024-05-10T11:11:20.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.935+0000 7f494b400700 1 -- 172.21.15.27:0/578136959 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340782 (secure 0 0 0) 0x7f49540638e0 con 0x7f49380b2880 2024-05-10T11:11:20.945 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:11:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.944+0000 7f4959a00700 1 -- 172.21.15.27:0/578136959 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49380b2880 msgr2=0x7f49380b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.944+0000 7f4959a00700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49380b2880 0x7f49380b4d40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4954103140 tx=0x7f4940009260 comp rx=0 tx=0).stop 2024-05-10T11:11:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.944+0000 7f4959a00700 1 -- 172.21.15.27:0/578136959 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4954103750 msgr2=0x7f495419f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.944+0000 7f4959a00700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4954103750 0x7f495419f810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4944000c00 tx=0x7f494400f990 comp rx=0 tx=0).stop 2024-05-10T11:11:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.944+0000 7f4959a00700 1 -- 172.21.15.27:0/578136959 shutdown_connections 2024-05-10T11:11:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.944+0000 7f4959a00700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f49380b2880 0x7f49380b4d40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.944+0000 7f4959a00700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4954108850 0x7f49541999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.944+0000 7f4959a00700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4954103750 0x7f495419f810 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.944+0000 7f4959a00700 1 --2- 172.21.15.27:0/578136959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4954104100 0x7f495419fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.944+0000 7f4959a00700 1 -- 172.21.15.27:0/578136959 >> 172.21.15.27:0/578136959 conn(0x7f495407a600 msgr2=0x7f4954101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.944+0000 7f4959a00700 1 -- 172.21.15.27:0/578136959 shutdown_connections 2024-05-10T11:11:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:20.944+0000 7f4959a00700 1 -- 172.21.15.27:0/578136959 wait complete. 2024-05-10T11:11:20.949 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-10T11:11:21.794 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-05-10T11:11:20.872472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662450,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1345,"num_read_kb":786,"num_write":382,"num_write_kb":2301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":2952552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":815,"ondisk_log_size":815,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361412,"kb_used_data":12260,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580412,"statfs":{"total":1151940427776,"available":1151570341888,"internally_reserved":0,"allocated":12554240,"data_stored":8226512,"data_compressed":497229,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":129,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005674"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.768848+0000","last_change":"2024-05-10T11:07:47.212921+0000","last_active":"2024-05-10T11:10:50.768848+0000","last_peered":"2024-05-10T11:10:50.768848+0000","last_clean":"2024-05-10T11:10:50.768848+0000","last_became_active":"2024-05-10T11:07:47.212558+0000","last_became_peered":"2024-05-10T11:07:47.212558+0000","last_unstale":"2024-05-10T11:10:50.768848+0000","last_undegraded":"2024-05-10T11:10:50.768848+0000","last_fullsized":"2024-05-10T11:10:50.768848+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:47:20.415883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.785263+0000","last_change":"2024-05-10T11:07:47.211186+0000","last_active":"2024-05-10T11:10:50.785263+0000","last_peered":"2024-05-10T11:10:50.785263+0000","last_clean":"2024-05-10T11:10:50.785263+0000","last_became_active":"2024-05-10T11:07:47.203487+0000","last_became_peered":"2024-05-10T11:07:47.203487+0000","last_unstale":"2024-05-10T11:10:50.785263+0000","last_undegraded":"2024-05-10T11:10:50.785263+0000","last_fullsized":"2024-05-10T11:10:50.785263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:51:29.674632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.801345+0000","last_change":"2024-05-10T11:07:47.202718+0000","last_active":"2024-05-10T11:10:50.801345+0000","last_peered":"2024-05-10T11:10:50.801345+0000","last_clean":"2024-05-10T11:10:50.801345+0000","last_became_active":"2024-05-10T11:07:47.202048+0000","last_became_peered":"2024-05-10T11:07:47.202048+0000","last_unstale":"2024-05-10T11:10:50.801345+0000","last_undegraded":"2024-05-10T11:10:50.801345+0000","last_fullsized":"2024-05-10T11:10:50.801345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:01:41.156850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.745401+0000","last_change":"2024-05-10T11:07:47.193519+0000","last_active":"2024-05-10T11:10:50.745401+0000","last_peered":"2024-05-10T11:10:50.745401+0000","last_clean":"2024-05-10T11:10:50.745401+0000","last_became_active":"2024-05-10T11:07:47.193200+0000","last_became_peered":"2024-05-10T11:07:47.193200+0000","last_unstale":"2024-05-10T11:10:50.745401+0000","last_undegraded":"2024-05-10T11:10:50.745401+0000","last_fullsized":"2024-05-10T11:10:50.745401+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:01.949453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.785839+0000","last_change":"2024-05-10T11:07:47.207750+0000","last_active":"2024-05-10T11:10:50.785839+0000","last_peered":"2024-05-10T11:10:50.785839+0000","last_clean":"2024-05-10T11:10:50.785839+0000","last_became_active":"2024-05-10T11:07:47.207384+0000","last_became_peered":"2024-05-10T11:07:47.207384+0000","last_unstale":"2024-05-10T11:10:50.785839+0000","last_undegraded":"2024-05-10T11:10:50.785839+0000","last_fullsized":"2024-05-10T11:10:50.785839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:09.432766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.762081+0000","last_change":"2024-05-10T11:08:22.245930+0000","last_active":"2024-05-10T11:10:50.762081+0000","last_peered":"2024-05-10T11:10:50.762081+0000","last_clean":"2024-05-10T11:10:50.762081+0000","last_became_active":"2024-05-10T11:08:22.245553+0000","last_became_peered":"2024-05-10T11:08:22.245553+0000","last_unstale":"2024-05-10T11:10:50.762081+0000","last_undegraded":"2024-05-10T11:10:50.762081+0000","last_fullsized":"2024-05-10T11:10:50.762081+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:47:31.485623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.763185+0000","last_change":"2024-05-10T11:07:47.196666+0000","last_active":"2024-05-10T11:10:50.763185+0000","last_peered":"2024-05-10T11:10:50.763185+0000","last_clean":"2024-05-10T11:10:50.763185+0000","last_became_active":"2024-05-10T11:07:47.196300+0000","last_became_peered":"2024-05-10T11:07:47.196300+0000","last_unstale":"2024-05-10T11:10:50.763185+0000","last_undegraded":"2024-05-10T11:10:50.763185+0000","last_fullsized":"2024-05-10T11:10:50.763185+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:55:24.343385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.788582+0000","last_change":"2024-05-10T11:07:47.213048+0000","last_active":"2024-05-10T11:10:50.788582+0000","last_peered":"2024-05-10T11:10:50.788582+0000","last_clean":"2024-05-10T11:10:50.788582+0000","last_became_active":"2024-05-10T11:07:47.206359+0000","last_became_peered":"2024-05-10T11:07:47.206359+0000","last_unstale":"2024-05-10T11:10:50.788582+0000","last_undegraded":"2024-05-10T11:10:50.788582+0000","last_fullsized":"2024-05-10T11:10:50.788582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:32:56.307261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.765728+0000","last_change":"2024-05-10T11:07:47.190945+0000","last_active":"2024-05-10T11:10:50.765728+0000","last_peered":"2024-05-10T11:10:50.765728+0000","last_clean":"2024-05-10T11:10:50.765728+0000","last_became_active":"2024-05-10T11:07:47.190631+0000","last_became_peered":"2024-05-10T11:07:47.190631+0000","last_unstale":"2024-05-10T11:10:50.765728+0000","last_undegraded":"2024-05-10T11:10:50.765728+0000","last_fullsized":"2024-05-10T11:10:50.765728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:34:08.683912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.751222+0000","last_change":"2024-05-10T11:07:47.189540+0000","last_active":"2024-05-10T11:10:50.751222+0000","last_peered":"2024-05-10T11:10:50.751222+0000","last_clean":"2024-05-10T11:10:50.751222+0000","last_became_active":"2024-05-10T11:07:47.189185+0000","last_became_peered":"2024-05-10T11:07:47.189185+0000","last_unstale":"2024-05-10T11:10:50.751222+0000","last_undegraded":"2024-05-10T11:10:50.751222+0000","last_fullsized":"2024-05-10T11:10:50.751222+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:55:23.892826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.769207+0000","last_change":"2024-05-10T11:07:47.195688+0000","last_active":"2024-05-10T11:10:50.769207+0000","last_peered":"2024-05-10T11:10:50.769207+0000","last_clean":"2024-05-10T11:10:50.769207+0000","last_became_active":"2024-05-10T11:07:47.195081+0000","last_became_peered":"2024-05-10T11:07:47.195081+0000","last_unstale":"2024-05-10T11:10:50.769207+0000","last_undegraded":"2024-05-10T11:10:50.769207+0000","last_fullsized":"2024-05-10T11:10:50.769207+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:20:55.240827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.747771+0000","last_change":"2024-05-10T11:07:47.193736+0000","last_active":"2024-05-10T11:10:50.747771+0000","last_peered":"2024-05-10T11:10:50.747771+0000","last_clean":"2024-05-10T11:10:50.747771+0000","last_became_active":"2024-05-10T11:07:47.193400+0000","last_became_peered":"2024-05-10T11:07:47.193400+0000","last_unstale":"2024-05-10T11:10:50.747771+0000","last_undegraded":"2024-05-10T11:10:50.747771+0000","last_fullsized":"2024-05-10T11:10:50.747771+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:46:28.507330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.791543+0000","last_change":"2024-05-10T11:08:22.042849+0000","last_active":"2024-05-10T11:10:50.791543+0000","last_peered":"2024-05-10T11:10:50.791543+0000","last_clean":"2024-05-10T11:10:50.791543+0000","last_became_active":"2024-05-10T11:08:22.042024+0000","last_became_peered":"2024-05-10T11:08:22.042024+0000","last_unstale":"2024-05-10T11:10:50.791543+0000","last_undegraded":"2024-05-10T11:10:50.791543+0000","last_fullsized":"2024-05-10T11:10:50.791543+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:29:08.096541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.801249+0000","last_change":"2024-05-10T11:07:47.214016+0000","last_active":"2024-05-10T11:10:50.801249+0000","last_peered":"2024-05-10T11:10:50.801249+0000","last_clean":"2024-05-10T11:10:50.801249+0000","last_became_active":"2024-05-10T11:07:47.209658+0000","last_became_peered":"2024-05-10T11:07:47.209658+0000","last_unstale":"2024-05-10T11:10:50.801249+0000","last_undegraded":"2024-05-10T11:10:50.801249+0000","last_fullsized":"2024-05-10T11:10:50.801249+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:44.771221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.800323+0000","last_change":"2024-05-10T11:07:47.210153+0000","last_active":"2024-05-10T11:10:50.800323+0000","last_peered":"2024-05-10T11:10:50.800323+0000","last_clean":"2024-05-10T11:10:50.800323+0000","last_became_active":"2024-05-10T11:07:47.208470+0000","last_became_peered":"2024-05-10T11:07:47.208470+0000","last_unstale":"2024-05-10T11:10:50.800323+0000","last_undegraded":"2024-05-10T11:10:50.800323+0000","last_fullsized":"2024-05-10T11:10:50.800323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:23:03.614670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.768324+0000","last_change":"2024-05-10T11:07:47.210704+0000","last_active":"2024-05-10T11:10:50.768324+0000","last_peered":"2024-05-10T11:10:50.768324+0000","last_clean":"2024-05-10T11:10:50.768324+0000","last_became_active":"2024-05-10T11:07:47.210317+0000","last_became_peered":"2024-05-10T11:07:47.210317+0000","last_unstale":"2024-05-10T11:10:50.768324+0000","last_undegraded":"2024-05-10T11:10:50.768324+0000","last_fullsized":"2024-05-10T11:10:50.768324+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:47:30.848731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.771107+0000","last_change":"2024-05-10T11:07:47.213116+0000","last_active":"2024-05-10T11:10:50.771107+0000","last_peered":"2024-05-10T11:10:50.771107+0000","last_clean":"2024-05-10T11:10:50.771107+0000","last_became_active":"2024-05-10T11:07:47.212567+0000","last_became_peered":"2024-05-10T11:07:47.212567+0000","last_unstale":"2024-05-10T11:10:50.771107+0000","last_undegraded":"2024-05-10T11:10:50.771107+0000","last_fullsized":"2024-05-10T11:10:50.771107+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:44:32.222012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.773409+0000","last_change":"2024-05-10T11:07:47.200973+0000","last_active":"2024-05-10T11:10:50.773409+0000","last_peered":"2024-05-10T11:10:50.773409+0000","last_clean":"2024-05-10T11:10:50.773409+0000","last_became_active":"2024-05-10T11:07:47.200499+0000","last_became_peered":"2024-05-10T11:07:47.200499+0000","last_unstale":"2024-05-10T11:10:50.773409+0000","last_undegraded":"2024-05-10T11:10:50.773409+0000","last_fullsized":"2024-05-10T11:10:50.773409+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:29:24.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.786245+0000","last_change":"2024-05-10T11:08:22.042254+0000","last_active":"2024-05-10T11:10:50.786245+0000","last_peered":"2024-05-10T11:10:50.786245+0000","last_clean":"2024-05-10T11:10:50.786245+0000","last_became_active":"2024-05-10T11:08:22.041906+0000","last_became_peered":"2024-05-10T11:08:22.041906+0000","last_unstale":"2024-05-10T11:10:50.786245+0000","last_undegraded":"2024-05-10T11:10:50.786245+0000","last_fullsized":"2024-05-10T11:10:50.786245+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:55:27.280478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070243+0000","last_change":"2024-05-10T11:08:00.897746+0000","last_active":"2024-05-10T11:09:23.070243+0000","last_peered":"2024-05-10T11:09:23.070243+0000","last_clean":"2024-05-10T11:09:23.070243+0000","last_became_active":"2024-05-10T11:08:00.897428+0000","last_became_peered":"2024-05-10T11:08:00.897428+0000","last_unstale":"2024-05-10T11:09:23.070243+0000","last_undegraded":"2024-05-10T11:09:23.070243+0000","last_fullsized":"2024-05-10T11:09:23.070243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:38.846373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":181,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:16.315539+0000","last_change":"2024-05-10T11:07:42.139990+0000","last_active":"2024-05-10T11:11:16.315539+0000","last_peered":"2024-05-10T11:11:16.315539+0000","last_clean":"2024-05-10T11:11:16.315539+0000","last_became_active":"2024-05-10T11:07:42.139612+0000","last_became_peered":"2024-05-10T11:07:42.139612+0000","last_unstale":"2024-05-10T11:11:16.315539+0000","last_undegraded":"2024-05-10T11:11:16.315539+0000","last_fullsized":"2024-05-10T11:11:16.315539+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:48:29.590570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.759313+0000","last_change":"2024-05-10T11:07:47.214263+0000","last_active":"2024-05-10T11:10:50.759313+0000","last_peered":"2024-05-10T11:10:50.759313+0000","last_clean":"2024-05-10T11:10:50.759313+0000","last_became_active":"2024-05-10T11:07:47.213946+0000","last_became_peered":"2024-05-10T11:07:47.213946+0000","last_unstale":"2024-05-10T11:10:50.759313+0000","last_undegraded":"2024-05-10T11:10:50.759313+0000","last_fullsized":"2024-05-10T11:10:50.759313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:54:31.015838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062366+0000","last_change":"2024-05-10T11:08:00.890571+0000","last_active":"2024-05-10T11:09:23.062366+0000","last_peered":"2024-05-10T11:09:23.062366+0000","last_clean":"2024-05-10T11:09:23.062366+0000","last_became_active":"2024-05-10T11:08:00.889468+0000","last_became_peered":"2024-05-10T11:08:00.889468+0000","last_unstale":"2024-05-10T11:09:23.062366+0000","last_undegraded":"2024-05-10T11:09:23.062366+0000","last_fullsized":"2024-05-10T11:09:23.062366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:20:54.342656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077801+0000","last_change":"2024-05-10T11:07:42.383396+0000","last_active":"2024-05-10T11:09:23.077801+0000","last_peered":"2024-05-10T11:09:23.077801+0000","last_clean":"2024-05-10T11:09:23.077801+0000","last_became_active":"2024-05-10T11:07:42.382787+0000","last_became_peered":"2024-05-10T11:07:42.382787+0000","last_unstale":"2024-05-10T11:09:23.077801+0000","last_undegraded":"2024-05-10T11:09:23.077801+0000","last_fullsized":"2024-05-10T11:09:23.077801+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:44:34.240119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.802653+0000","last_change":"2024-05-10T11:07:47.216770+0000","last_active":"2024-05-10T11:10:50.802653+0000","last_peered":"2024-05-10T11:10:50.802653+0000","last_clean":"2024-05-10T11:10:50.802653+0000","last_became_active":"2024-05-10T11:07:47.216377+0000","last_became_peered":"2024-05-10T11:07:47.216377+0000","last_unstale":"2024-05-10T11:10:50.802653+0000","last_undegraded":"2024-05-10T11:10:50.802653+0000","last_fullsized":"2024-05-10T11:10:50.802653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:59:22.993888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062516+0000","last_change":"2024-05-10T11:08:00.900614+0000","last_active":"2024-05-10T11:09:23.062516+0000","last_peered":"2024-05-10T11:09:23.062516+0000","last_clean":"2024-05-10T11:09:23.062516+0000","last_became_active":"2024-05-10T11:08:00.900162+0000","last_became_peered":"2024-05-10T11:08:00.900162+0000","last_unstale":"2024-05-10T11:09:23.062516+0000","last_undegraded":"2024-05-10T11:09:23.062516+0000","last_fullsized":"2024-05-10T11:09:23.062516+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:42:56.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070291+0000","last_change":"2024-05-10T11:07:42.127749+0000","last_active":"2024-05-10T11:09:23.070291+0000","last_peered":"2024-05-10T11:09:23.070291+0000","last_clean":"2024-05-10T11:09:23.070291+0000","last_became_active":"2024-05-10T11:07:42.127296+0000","last_became_peered":"2024-05-10T11:07:42.127296+0000","last_unstale":"2024-05-10T11:09:23.070291+0000","last_undegraded":"2024-05-10T11:09:23.070291+0000","last_fullsized":"2024-05-10T11:09:23.070291+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:37:37.720542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.781187+0000","last_change":"2024-05-10T11:07:47.198971+0000","last_active":"2024-05-10T11:10:50.781187+0000","last_peered":"2024-05-10T11:10:50.781187+0000","last_clean":"2024-05-10T11:10:50.781187+0000","last_became_active":"2024-05-10T11:07:47.198510+0000","last_became_peered":"2024-05-10T11:07:47.198510+0000","last_unstale":"2024-05-10T11:10:50.781187+0000","last_undegraded":"2024-05-10T11:10:50.781187+0000","last_fullsized":"2024-05-10T11:10:50.781187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:17:24.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062534+0000","last_change":"2024-05-10T11:08:00.878749+0000","last_active":"2024-05-10T11:09:23.062534+0000","last_peered":"2024-05-10T11:09:23.062534+0000","last_clean":"2024-05-10T11:09:23.062534+0000","last_became_active":"2024-05-10T11:08:00.878271+0000","last_became_peered":"2024-05-10T11:08:00.878271+0000","last_unstale":"2024-05-10T11:09:23.062534+0000","last_undegraded":"2024-05-10T11:09:23.062534+0000","last_fullsized":"2024-05-10T11:09:23.062534+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:39:28.558766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069628+0000","last_change":"2024-05-10T11:07:42.129593+0000","last_active":"2024-05-10T11:09:23.069628+0000","last_peered":"2024-05-10T11:09:23.069628+0000","last_clean":"2024-05-10T11:09:23.069628+0000","last_became_active":"2024-05-10T11:07:42.129250+0000","last_became_peered":"2024-05-10T11:07:42.129250+0000","last_unstale":"2024-05-10T11:09:23.069628+0000","last_undegraded":"2024-05-10T11:09:23.069628+0000","last_fullsized":"2024-05-10T11:09:23.069628+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:22:55.541079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.782571+0000","last_change":"2024-05-10T11:08:22.244535+0000","last_active":"2024-05-10T11:10:50.782571+0000","last_peered":"2024-05-10T11:10:50.782571+0000","last_clean":"2024-05-10T11:10:50.782571+0000","last_became_active":"2024-05-10T11:08:22.243716+0000","last_became_peered":"2024-05-10T11:08:22.243716+0000","last_unstale":"2024-05-10T11:10:50.782571+0000","last_undegraded":"2024-05-10T11:10:50.782571+0000","last_fullsized":"2024-05-10T11:10:50.782571+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:11.718999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069816+0000","last_change":"2024-05-10T11:08:00.869806+0000","last_active":"2024-05-10T11:09:23.069816+0000","last_peered":"2024-05-10T11:09:23.069816+0000","last_clean":"2024-05-10T11:09:23.069816+0000","last_became_active":"2024-05-10T11:08:00.869355+0000","last_became_peered":"2024-05-10T11:08:00.869355+0000","last_unstale":"2024-05-10T11:09:23.069816+0000","last_undegraded":"2024-05-10T11:09:23.069816+0000","last_fullsized":"2024-05-10T11:09:23.069816+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:36.043412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":184,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:14.317725+0000","last_change":"2024-05-10T11:09:23.076933+0000","last_active":"2024-05-10T11:11:14.317725+0000","last_peered":"2024-05-10T11:11:14.317725+0000","last_clean":"2024-05-10T11:11:14.317725+0000","last_became_active":"2024-05-10T11:09:23.073274+0000","last_became_peered":"2024-05-10T11:09:23.073274+0000","last_unstale":"2024-05-10T11:11:14.317725+0000","last_undegraded":"2024-05-10T11:11:14.317725+0000","last_fullsized":"2024-05-10T11:11:14.317725+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:26:13.923466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.762350+0000","last_change":"2024-05-10T11:07:47.200873+0000","last_active":"2024-05-10T11:10:50.762350+0000","last_peered":"2024-05-10T11:10:50.762350+0000","last_clean":"2024-05-10T11:10:50.762350+0000","last_became_active":"2024-05-10T11:07:47.200436+0000","last_became_peered":"2024-05-10T11:07:47.200436+0000","last_unstale":"2024-05-10T11:10:50.762350+0000","last_undegraded":"2024-05-10T11:10:50.762350+0000","last_fullsized":"2024-05-10T11:10:50.762350+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:38:49.588593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069663+0000","last_change":"2024-05-10T11:08:00.885432+0000","last_active":"2024-05-10T11:09:23.069663+0000","last_peered":"2024-05-10T11:09:23.069663+0000","last_clean":"2024-05-10T11:09:23.069663+0000","last_became_active":"2024-05-10T11:08:00.884987+0000","last_became_peered":"2024-05-10T11:08:00.884987+0000","last_unstale":"2024-05-10T11:09:23.069663+0000","last_undegraded":"2024-05-10T11:09:23.069663+0000","last_fullsized":"2024-05-10T11:09:23.069663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:41:24.082722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.093561+0000","last_change":"2024-05-10T11:09:23.093122+0000","last_active":"2024-05-10T11:09:23.093561+0000","last_peered":"2024-05-10T11:09:23.093561+0000","last_clean":"2024-05-10T11:09:23.093561+0000","last_became_active":"2024-05-10T11:09:23.084485+0000","last_became_peered":"2024-05-10T11:09:23.084485+0000","last_unstale":"2024-05-10T11:09:23.093561+0000","last_undegraded":"2024-05-10T11:09:23.093561+0000","last_fullsized":"2024-05-10T11:09:23.093561+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:12:36.793572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.785927+0000","last_change":"2024-05-10T11:08:22.036515+0000","last_active":"2024-05-10T11:10:50.785927+0000","last_peered":"2024-05-10T11:10:50.785927+0000","last_clean":"2024-05-10T11:10:50.785927+0000","last_became_active":"2024-05-10T11:08:22.036072+0000","last_became_peered":"2024-05-10T11:08:22.036072+0000","last_unstale":"2024-05-10T11:10:50.785927+0000","last_undegraded":"2024-05-10T11:10:50.785927+0000","last_fullsized":"2024-05-10T11:10:50.785927+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:08.734453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6],"acting":[5,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065713+0000","last_change":"2024-05-10T11:08:00.888673+0000","last_active":"2024-05-10T11:09:23.065713+0000","last_peered":"2024-05-10T11:09:23.065713+0000","last_clean":"2024-05-10T11:09:23.065713+0000","last_became_active":"2024-05-10T11:08:00.888302+0000","last_became_peered":"2024-05-10T11:08:00.888302+0000","last_unstale":"2024-05-10T11:09:23.065713+0000","last_undegraded":"2024-05-10T11:09:23.065713+0000","last_fullsized":"2024-05-10T11:09:23.065713+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:15:32.276358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.078317+0000","last_change":"2024-05-10T11:07:42.140993+0000","last_active":"2024-05-10T11:09:23.078317+0000","last_peered":"2024-05-10T11:09:23.078317+0000","last_clean":"2024-05-10T11:09:23.078317+0000","last_became_active":"2024-05-10T11:07:42.140501+0000","last_became_peered":"2024-05-10T11:07:42.140501+0000","last_unstale":"2024-05-10T11:09:23.078317+0000","last_undegraded":"2024-05-10T11:09:23.078317+0000","last_fullsized":"2024-05-10T11:09:23.078317+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:01:50.324507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.790621+0000","last_change":"2024-05-10T11:07:47.212616+0000","last_active":"2024-05-10T11:10:50.790621+0000","last_peered":"2024-05-10T11:10:50.790621+0000","last_clean":"2024-05-10T11:10:50.790621+0000","last_became_active":"2024-05-10T11:07:47.211081+0000","last_became_peered":"2024-05-10T11:07:47.211081+0000","last_unstale":"2024-05-10T11:10:50.790621+0000","last_undegraded":"2024-05-10T11:10:50.790621+0000","last_fullsized":"2024-05-10T11:10:50.790621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:03:54.404355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067703+0000","last_change":"2024-05-10T11:08:00.888782+0000","last_active":"2024-05-10T11:09:23.067703+0000","last_peered":"2024-05-10T11:09:23.067703+0000","last_clean":"2024-05-10T11:09:23.067703+0000","last_became_active":"2024-05-10T11:08:00.888366+0000","last_became_peered":"2024-05-10T11:08:00.888366+0000","last_unstale":"2024-05-10T11:09:23.067703+0000","last_undegraded":"2024-05-10T11:09:23.067703+0000","last_fullsized":"2024-05-10T11:09:23.067703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:37.040186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066199+0000","last_change":"2024-05-10T11:07:42.128677+0000","last_active":"2024-05-10T11:09:23.066199+0000","last_peered":"2024-05-10T11:09:23.066199+0000","last_clean":"2024-05-10T11:09:23.066199+0000","last_became_active":"2024-05-10T11:07:42.128330+0000","last_became_peered":"2024-05-10T11:07:42.128330+0000","last_unstale":"2024-05-10T11:09:23.066199+0000","last_undegraded":"2024-05-10T11:09:23.066199+0000","last_fullsized":"2024-05-10T11:09:23.066199+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:09:11.738485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.783986+0000","last_change":"2024-05-10T11:07:47.215257+0000","last_active":"2024-05-10T11:10:50.783986+0000","last_peered":"2024-05-10T11:10:50.783986+0000","last_clean":"2024-05-10T11:10:50.783986+0000","last_became_active":"2024-05-10T11:07:47.214682+0000","last_became_peered":"2024-05-10T11:07:47.214682+0000","last_unstale":"2024-05-10T11:10:50.783986+0000","last_undegraded":"2024-05-10T11:10:50.783986+0000","last_fullsized":"2024-05-10T11:10:50.783986+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:35:45.787737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076185+0000","last_change":"2024-05-10T11:08:00.894446+0000","last_active":"2024-05-10T11:09:23.076185+0000","last_peered":"2024-05-10T11:09:23.076185+0000","last_clean":"2024-05-10T11:09:23.076185+0000","last_became_active":"2024-05-10T11:08:00.886473+0000","last_became_peered":"2024-05-10T11:08:00.886473+0000","last_unstale":"2024-05-10T11:09:23.076185+0000","last_undegraded":"2024-05-10T11:09:23.076185+0000","last_fullsized":"2024-05-10T11:09:23.076185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:40.980328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":297,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:19.228696+0000","last_change":"2024-05-10T11:07:42.138750+0000","last_active":"2024-05-10T11:11:19.228696+0000","last_peered":"2024-05-10T11:11:19.228696+0000","last_clean":"2024-05-10T11:11:19.228696+0000","last_became_active":"2024-05-10T11:07:42.138342+0000","last_became_peered":"2024-05-10T11:07:42.138342+0000","last_unstale":"2024-05-10T11:11:19.228696+0000","last_undegraded":"2024-05-10T11:11:19.228696+0000","last_fullsized":"2024-05-10T11:11:19.228696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:19:45.791550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":248,"num_read_kb":248,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.774389+0000","last_change":"2024-05-10T11:07:47.204604+0000","last_active":"2024-05-10T11:10:50.774389+0000","last_peered":"2024-05-10T11:10:50.774389+0000","last_clean":"2024-05-10T11:10:50.774389+0000","last_became_active":"2024-05-10T11:07:47.204261+0000","last_became_peered":"2024-05-10T11:07:47.204261+0000","last_unstale":"2024-05-10T11:10:50.774389+0000","last_undegraded":"2024-05-10T11:10:50.774389+0000","last_fullsized":"2024-05-10T11:10:50.774389+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:55:46.409545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068712+0000","last_change":"2024-05-10T11:08:22.038090+0000","last_active":"2024-05-10T11:09:23.068712+0000","last_peered":"2024-05-10T11:09:23.068712+0000","last_clean":"2024-05-10T11:09:23.068712+0000","last_became_active":"2024-05-10T11:08:22.037490+0000","last_became_peered":"2024-05-10T11:08:22.037490+0000","last_unstale":"2024-05-10T11:09:23.068712+0000","last_undegraded":"2024-05-10T11:09:23.068712+0000","last_fullsized":"2024-05-10T11:09:23.068712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:01:46.319401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071747+0000","last_change":"2024-05-10T11:07:42.128636+0000","last_active":"2024-05-10T11:09:23.071747+0000","last_peered":"2024-05-10T11:09:23.071747+0000","last_clean":"2024-05-10T11:09:23.071747+0000","last_became_active":"2024-05-10T11:07:42.128194+0000","last_became_peered":"2024-05-10T11:07:42.128194+0000","last_unstale":"2024-05-10T11:09:23.071747+0000","last_undegraded":"2024-05-10T11:09:23.071747+0000","last_fullsized":"2024-05-10T11:09:23.071747+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:51:54.279292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.802905+0000","last_change":"2024-05-10T11:08:22.037807+0000","last_active":"2024-05-10T11:10:50.802905+0000","last_peered":"2024-05-10T11:10:50.802905+0000","last_clean":"2024-05-10T11:10:50.802905+0000","last_became_active":"2024-05-10T11:08:22.037337+0000","last_became_peered":"2024-05-10T11:08:22.037337+0000","last_unstale":"2024-05-10T11:10:50.802905+0000","last_undegraded":"2024-05-10T11:10:50.802905+0000","last_fullsized":"2024-05-10T11:10:50.802905+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:50:06.481222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062740+0000","last_change":"2024-05-10T11:08:00.893684+0000","last_active":"2024-05-10T11:09:23.062740+0000","last_peered":"2024-05-10T11:09:23.062740+0000","last_clean":"2024-05-10T11:09:23.062740+0000","last_became_active":"2024-05-10T11:08:00.893334+0000","last_became_peered":"2024-05-10T11:08:00.893334+0000","last_unstale":"2024-05-10T11:09:23.062740+0000","last_undegraded":"2024-05-10T11:09:23.062740+0000","last_fullsized":"2024-05-10T11:09:23.062740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:58:04.812308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066370+0000","last_change":"2024-05-10T11:07:42.382086+0000","last_active":"2024-05-10T11:09:23.066370+0000","last_peered":"2024-05-10T11:09:23.066370+0000","last_clean":"2024-05-10T11:09:23.066370+0000","last_became_active":"2024-05-10T11:07:42.381744+0000","last_became_peered":"2024-05-10T11:07:42.381744+0000","last_unstale":"2024-05-10T11:09:23.066370+0000","last_undegraded":"2024-05-10T11:09:23.066370+0000","last_fullsized":"2024-05-10T11:09:23.066370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:15:10.893834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.762106+0000","last_change":"2024-05-10T11:07:47.194683+0000","last_active":"2024-05-10T11:10:50.762106+0000","last_peered":"2024-05-10T11:10:50.762106+0000","last_clean":"2024-05-10T11:10:50.762106+0000","last_became_active":"2024-05-10T11:07:47.194319+0000","last_became_peered":"2024-05-10T11:07:47.194319+0000","last_unstale":"2024-05-10T11:10:50.762106+0000","last_undegraded":"2024-05-10T11:10:50.762106+0000","last_fullsized":"2024-05-10T11:10:50.762106+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:41:48.437093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069730+0000","last_change":"2024-05-10T11:08:00.889616+0000","last_active":"2024-05-10T11:09:23.069730+0000","last_peered":"2024-05-10T11:09:23.069730+0000","last_clean":"2024-05-10T11:09:23.069730+0000","last_became_active":"2024-05-10T11:08:00.882179+0000","last_became_peered":"2024-05-10T11:08:00.882179+0000","last_unstale":"2024-05-10T11:09:23.069730+0000","last_undegraded":"2024-05-10T11:09:23.069730+0000","last_fullsized":"2024-05-10T11:09:23.069730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:23:12.407861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'39","reported_seq":342,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:19.257919+0000","last_change":"2024-05-10T11:07:42.128796+0000","last_active":"2024-05-10T11:11:19.257919+0000","last_peered":"2024-05-10T11:11:19.257919+0000","last_clean":"2024-05-10T11:11:19.257919+0000","last_became_active":"2024-05-10T11:07:42.128402+0000","last_became_peered":"2024-05-10T11:07:42.128402+0000","last_unstale":"2024-05-10T11:11:19.257919+0000","last_undegraded":"2024-05-10T11:11:19.257919+0000","last_fullsized":"2024-05-10T11:11:19.257919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:50:09.205102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64894,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":258,"num_read_kb":193,"num_write":41,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.762334+0000","last_change":"2024-05-10T11:07:47.188304+0000","last_active":"2024-05-10T11:10:50.762334+0000","last_peered":"2024-05-10T11:10:50.762334+0000","last_clean":"2024-05-10T11:10:50.762334+0000","last_became_active":"2024-05-10T11:07:47.187798+0000","last_became_peered":"2024-05-10T11:07:47.187798+0000","last_unstale":"2024-05-10T11:10:50.762334+0000","last_undegraded":"2024-05-10T11:10:50.762334+0000","last_fullsized":"2024-05-10T11:10:50.762334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:50:21.059060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066761+0000","last_change":"2024-05-10T11:08:00.886669+0000","last_active":"2024-05-10T11:09:23.066761+0000","last_peered":"2024-05-10T11:09:23.066761+0000","last_clean":"2024-05-10T11:09:23.066761+0000","last_became_active":"2024-05-10T11:08:00.886249+0000","last_became_peered":"2024-05-10T11:08:00.886249+0000","last_unstale":"2024-05-10T11:09:23.066761+0000","last_undegraded":"2024-05-10T11:09:23.066761+0000","last_fullsized":"2024-05-10T11:09:23.066761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:10:07.707462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.095713+0000","last_change":"2024-05-10T11:09:23.095352+0000","last_active":"2024-05-10T11:09:23.095713+0000","last_peered":"2024-05-10T11:09:23.095713+0000","last_clean":"2024-05-10T11:09:23.095713+0000","last_became_active":"2024-05-10T11:09:23.083116+0000","last_became_peered":"2024-05-10T11:09:23.083116+0000","last_unstale":"2024-05-10T11:09:23.095713+0000","last_undegraded":"2024-05-10T11:09:23.095713+0000","last_fullsized":"2024-05-10T11:09:23.095713+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:24:28.945724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.790060+0000","last_change":"2024-05-10T11:07:47.207657+0000","last_active":"2024-05-10T11:10:50.790060+0000","last_peered":"2024-05-10T11:10:50.790060+0000","last_clean":"2024-05-10T11:10:50.790060+0000","last_became_active":"2024-05-10T11:07:47.207317+0000","last_became_peered":"2024-05-10T11:07:47.207317+0000","last_unstale":"2024-05-10T11:10:50.790060+0000","last_undegraded":"2024-05-10T11:10:50.790060+0000","last_fullsized":"2024-05-10T11:10:50.790060+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:07:33.961642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065113+0000","last_change":"2024-05-10T11:08:00.884076+0000","last_active":"2024-05-10T11:09:23.065113+0000","last_peered":"2024-05-10T11:09:23.065113+0000","last_clean":"2024-05-10T11:09:23.065113+0000","last_became_active":"2024-05-10T11:08:00.883670+0000","last_became_peered":"2024-05-10T11:08:00.883670+0000","last_unstale":"2024-05-10T11:09:23.065113+0000","last_undegraded":"2024-05-10T11:09:23.065113+0000","last_fullsized":"2024-05-10T11:09:23.065113+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:56.447836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.067621+0000","last_change":"2024-05-10T11:07:42.135550+0000","last_active":"2024-05-10T11:09:23.067621+0000","last_peered":"2024-05-10T11:09:23.067621+0000","last_clean":"2024-05-10T11:09:23.067621+0000","last_became_active":"2024-05-10T11:07:42.135192+0000","last_became_peered":"2024-05-10T11:07:42.135192+0000","last_unstale":"2024-05-10T11:09:23.067621+0000","last_undegraded":"2024-05-10T11:09:23.067621+0000","last_fullsized":"2024-05-10T11:09:23.067621+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:17.100873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.787215+0000","last_change":"2024-05-10T11:07:47.199529+0000","last_active":"2024-05-10T11:10:50.787215+0000","last_peered":"2024-05-10T11:10:50.787215+0000","last_clean":"2024-05-10T11:10:50.787215+0000","last_became_active":"2024-05-10T11:07:47.199070+0000","last_became_peered":"2024-05-10T11:07:47.199070+0000","last_unstale":"2024-05-10T11:10:50.787215+0000","last_undegraded":"2024-05-10T11:10:50.787215+0000","last_fullsized":"2024-05-10T11:10:50.787215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:24.446639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068411+0000","last_change":"2024-05-10T11:08:00.898607+0000","last_active":"2024-05-10T11:09:23.068411+0000","last_peered":"2024-05-10T11:09:23.068411+0000","last_clean":"2024-05-10T11:09:23.068411+0000","last_became_active":"2024-05-10T11:08:00.897768+0000","last_became_peered":"2024-05-10T11:08:00.897768+0000","last_unstale":"2024-05-10T11:09:23.068411+0000","last_undegraded":"2024-05-10T11:09:23.068411+0000","last_fullsized":"2024-05-10T11:09:23.068411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:24:42.125203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":177,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:15.260382+0000","last_change":"2024-05-10T11:07:42.126907+0000","last_active":"2024-05-10T11:11:15.260382+0000","last_peered":"2024-05-10T11:11:15.260382+0000","last_clean":"2024-05-10T11:11:15.260382+0000","last_became_active":"2024-05-10T11:07:42.126462+0000","last_became_peered":"2024-05-10T11:07:42.126462+0000","last_unstale":"2024-05-10T11:11:15.260382+0000","last_undegraded":"2024-05-10T11:11:15.260382+0000","last_fullsized":"2024-05-10T11:11:15.260382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:18:53.971150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.755058+0000","last_change":"2024-05-10T11:07:47.191888+0000","last_active":"2024-05-10T11:10:50.755058+0000","last_peered":"2024-05-10T11:10:50.755058+0000","last_clean":"2024-05-10T11:10:50.755058+0000","last_became_active":"2024-05-10T11:07:47.191557+0000","last_became_peered":"2024-05-10T11:07:47.191557+0000","last_unstale":"2024-05-10T11:10:50.755058+0000","last_undegraded":"2024-05-10T11:10:50.755058+0000","last_fullsized":"2024-05-10T11:10:50.755058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:01:36.529493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079432+0000","last_change":"2024-05-10T11:08:00.894385+0000","last_active":"2024-05-10T11:09:23.079432+0000","last_peered":"2024-05-10T11:09:23.079432+0000","last_clean":"2024-05-10T11:09:23.079432+0000","last_became_active":"2024-05-10T11:08:00.893706+0000","last_became_peered":"2024-05-10T11:08:00.893706+0000","last_unstale":"2024-05-10T11:09:23.079432+0000","last_undegraded":"2024-05-10T11:09:23.079432+0000","last_fullsized":"2024-05-10T11:09:23.079432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:02.069878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068853+0000","last_change":"2024-05-10T11:07:42.385661+0000","last_active":"2024-05-10T11:09:23.068853+0000","last_peered":"2024-05-10T11:09:23.068853+0000","last_clean":"2024-05-10T11:09:23.068853+0000","last_became_active":"2024-05-10T11:07:42.385308+0000","last_became_peered":"2024-05-10T11:07:42.385308+0000","last_unstale":"2024-05-10T11:09:23.068853+0000","last_undegraded":"2024-05-10T11:09:23.068853+0000","last_fullsized":"2024-05-10T11:09:23.068853+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:57:25.997730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.739312+0000","last_change":"2024-05-10T11:07:47.191574+0000","last_active":"2024-05-10T11:10:50.739312+0000","last_peered":"2024-05-10T11:10:50.739312+0000","last_clean":"2024-05-10T11:10:50.739312+0000","last_became_active":"2024-05-10T11:07:47.191210+0000","last_became_peered":"2024-05-10T11:07:47.191210+0000","last_unstale":"2024-05-10T11:10:50.739312+0000","last_undegraded":"2024-05-10T11:10:50.739312+0000","last_fullsized":"2024-05-10T11:10:50.739312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:11:35.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065620+0000","last_change":"2024-05-10T11:08:00.890076+0000","last_active":"2024-05-10T11:09:23.065620+0000","last_peered":"2024-05-10T11:09:23.065620+0000","last_clean":"2024-05-10T11:09:23.065620+0000","last_became_active":"2024-05-10T11:08:00.889630+0000","last_became_peered":"2024-05-10T11:08:00.889630+0000","last_unstale":"2024-05-10T11:09:23.065620+0000","last_undegraded":"2024-05-10T11:09:23.065620+0000","last_fullsized":"2024-05-10T11:09:23.065620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:53:29.257873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.075061+0000","last_change":"2024-05-10T11:09:23.074550+0000","last_active":"2024-05-10T11:09:23.075061+0000","last_peered":"2024-05-10T11:09:23.075061+0000","last_clean":"2024-05-10T11:09:23.075061+0000","last_became_active":"2024-05-10T11:09:23.067169+0000","last_became_peered":"2024-05-10T11:09:23.067169+0000","last_unstale":"2024-05-10T11:09:23.075061+0000","last_undegraded":"2024-05-10T11:09:23.075061+0000","last_fullsized":"2024-05-10T11:09:23.075061+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:45:06.303003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.755360+0000","last_change":"2024-05-10T11:07:47.192042+0000","last_active":"2024-05-10T11:10:50.755360+0000","last_peered":"2024-05-10T11:10:50.755360+0000","last_clean":"2024-05-10T11:10:50.755360+0000","last_became_active":"2024-05-10T11:07:47.191632+0000","last_became_peered":"2024-05-10T11:07:47.191632+0000","last_unstale":"2024-05-10T11:10:50.755360+0000","last_undegraded":"2024-05-10T11:10:50.755360+0000","last_fullsized":"2024-05-10T11:10:50.755360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:55:08.564841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069056+0000","last_change":"2024-05-10T11:08:22.244721+0000","last_active":"2024-05-10T11:09:23.069056+0000","last_peered":"2024-05-10T11:09:23.069056+0000","last_clean":"2024-05-10T11:09:23.069056+0000","last_became_active":"2024-05-10T11:08:22.244274+0000","last_became_peered":"2024-05-10T11:08:22.244274+0000","last_unstale":"2024-05-10T11:09:23.069056+0000","last_undegraded":"2024-05-10T11:09:23.069056+0000","last_fullsized":"2024-05-10T11:09:23.069056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:26.218071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070759+0000","last_change":"2024-05-10T11:07:42.388358+0000","last_active":"2024-05-10T11:09:23.070759+0000","last_peered":"2024-05-10T11:09:23.070759+0000","last_clean":"2024-05-10T11:09:23.070759+0000","last_became_active":"2024-05-10T11:07:42.387806+0000","last_became_peered":"2024-05-10T11:07:42.387806+0000","last_unstale":"2024-05-10T11:09:23.070759+0000","last_undegraded":"2024-05-10T11:09:23.070759+0000","last_fullsized":"2024-05-10T11:09:23.070759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:27:38.387332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.783965+0000","last_change":"2024-05-10T11:07:47.213602+0000","last_active":"2024-05-10T11:10:50.783965+0000","last_peered":"2024-05-10T11:10:50.783965+0000","last_clean":"2024-05-10T11:10:50.783965+0000","last_became_active":"2024-05-10T11:07:47.213184+0000","last_became_peered":"2024-05-10T11:07:47.213184+0000","last_unstale":"2024-05-10T11:10:50.783965+0000","last_undegraded":"2024-05-10T11:10:50.783965+0000","last_fullsized":"2024-05-10T11:10:50.783965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T23:01:52.253279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079684+0000","last_change":"2024-05-10T11:08:00.889850+0000","last_active":"2024-05-10T11:09:23.079684+0000","last_peered":"2024-05-10T11:09:23.079684+0000","last_clean":"2024-05-10T11:09:23.079684+0000","last_became_active":"2024-05-10T11:08:00.888526+0000","last_became_peered":"2024-05-10T11:08:00.888526+0000","last_unstale":"2024-05-10T11:09:23.079684+0000","last_undegraded":"2024-05-10T11:09:23.079684+0000","last_fullsized":"2024-05-10T11:09:23.079684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:11:43.899628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:18.320605+0000","last_change":"2024-05-10T11:07:42.130650+0000","last_active":"2024-05-10T11:11:18.320605+0000","last_peered":"2024-05-10T11:11:18.320605+0000","last_clean":"2024-05-10T11:11:18.320605+0000","last_became_active":"2024-05-10T11:07:42.130259+0000","last_became_peered":"2024-05-10T11:07:42.130259+0000","last_unstale":"2024-05-10T11:11:18.320605+0000","last_undegraded":"2024-05-10T11:11:18.320605+0000","last_fullsized":"2024-05-10T11:11:18.320605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.870092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.763336+0000","last_change":"2024-05-10T11:07:47.204159+0000","last_active":"2024-05-10T11:10:50.763336+0000","last_peered":"2024-05-10T11:10:50.763336+0000","last_clean":"2024-05-10T11:10:50.763336+0000","last_became_active":"2024-05-10T11:07:47.203723+0000","last_became_peered":"2024-05-10T11:07:47.203723+0000","last_unstale":"2024-05-10T11:10:50.763336+0000","last_undegraded":"2024-05-10T11:10:50.763336+0000","last_fullsized":"2024-05-10T11:10:50.763336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:44:48.232321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.082000+0000","last_change":"2024-05-10T11:08:00.918501+0000","last_active":"2024-05-10T11:09:23.082000+0000","last_peered":"2024-05-10T11:09:23.082000+0000","last_clean":"2024-05-10T11:09:23.082000+0000","last_became_active":"2024-05-10T11:08:00.918071+0000","last_became_peered":"2024-05-10T11:08:00.918071+0000","last_unstale":"2024-05-10T11:09:23.082000+0000","last_undegraded":"2024-05-10T11:09:23.082000+0000","last_fullsized":"2024-05-10T11:09:23.082000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:23:08.390953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.785780+0000","last_change":"2024-05-10T11:08:22.036385+0000","last_active":"2024-05-10T11:10:50.785780+0000","last_peered":"2024-05-10T11:10:50.785780+0000","last_clean":"2024-05-10T11:10:50.785780+0000","last_became_active":"2024-05-10T11:08:22.035969+0000","last_became_peered":"2024-05-10T11:08:22.035969+0000","last_unstale":"2024-05-10T11:10:50.785780+0000","last_undegraded":"2024-05-10T11:10:50.785780+0000","last_fullsized":"2024-05-10T11:10:50.785780+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:14:24.069422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:17.264928+0000","last_change":"2024-05-10T11:07:42.381926+0000","last_active":"2024-05-10T11:11:17.264928+0000","last_peered":"2024-05-10T11:11:17.264928+0000","last_clean":"2024-05-10T11:11:17.264928+0000","last_became_active":"2024-05-10T11:07:42.381301+0000","last_became_peered":"2024-05-10T11:07:42.381301+0000","last_unstale":"2024-05-10T11:11:17.264928+0000","last_undegraded":"2024-05-10T11:11:17.264928+0000","last_fullsized":"2024-05-10T11:11:17.264928+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:48:32.291029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":124,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:27.923059+0000","last_change":"2024-05-10T11:08:00.889959+0000","last_active":"2024-05-10T11:09:27.923059+0000","last_peered":"2024-05-10T11:09:27.923059+0000","last_clean":"2024-05-10T11:09:27.923059+0000","last_became_active":"2024-05-10T11:08:00.889514+0000","last_became_peered":"2024-05-10T11:08:00.889514+0000","last_unstale":"2024-05-10T11:09:27.923059+0000","last_undegraded":"2024-05-10T11:09:27.923059+0000","last_fullsized":"2024-05-10T11:09:27.923059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.557175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.779373+0000","last_change":"2024-05-10T11:07:47.218025+0000","last_active":"2024-05-10T11:10:50.779373+0000","last_peered":"2024-05-10T11:10:50.779373+0000","last_clean":"2024-05-10T11:10:50.779373+0000","last_became_active":"2024-05-10T11:07:47.207089+0000","last_became_peered":"2024-05-10T11:07:47.207089+0000","last_unstale":"2024-05-10T11:10:50.779373+0000","last_undegraded":"2024-05-10T11:10:50.779373+0000","last_fullsized":"2024-05-10T11:10:50.779373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:15:50.363129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062645+0000","last_change":"2024-05-10T11:07:42.138182+0000","last_active":"2024-05-10T11:09:23.062645+0000","last_peered":"2024-05-10T11:09:23.062645+0000","last_clean":"2024-05-10T11:09:23.062645+0000","last_became_active":"2024-05-10T11:07:42.137734+0000","last_became_peered":"2024-05-10T11:07:42.137734+0000","last_unstale":"2024-05-10T11:09:23.062645+0000","last_undegraded":"2024-05-10T11:09:23.062645+0000","last_fullsized":"2024-05-10T11:09:23.062645+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:02:30.778878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.071459+0000","last_change":"2024-05-10T11:06:06.321257+0000","last_active":"2024-05-10T11:09:23.071459+0000","last_peered":"2024-05-10T11:09:23.071459+0000","last_clean":"2024-05-10T11:09:23.071459+0000","last_became_active":"2024-05-10T11:06:06.316928+0000","last_became_peered":"2024-05-10T11:06:06.316928+0000","last_unstale":"2024-05-10T11:09:23.071459+0000","last_undegraded":"2024-05-10T11:09:23.071459+0000","last_fullsized":"2024-05-10T11:09:23.071459+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_clean_scrub_stamp":"2024-05-10T11:02:07.346111+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:32:43.670246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065321+0000","last_change":"2024-05-10T11:08:00.918499+0000","last_active":"2024-05-10T11:09:23.065321+0000","last_peered":"2024-05-10T11:09:23.065321+0000","last_clean":"2024-05-10T11:09:23.065321+0000","last_became_active":"2024-05-10T11:08:00.917950+0000","last_became_peered":"2024-05-10T11:08:00.917950+0000","last_unstale":"2024-05-10T11:09:23.065321+0000","last_undegraded":"2024-05-10T11:09:23.065321+0000","last_fullsized":"2024-05-10T11:09:23.065321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:07.591312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.758046+0000","last_change":"2024-05-10T11:07:47.194968+0000","last_active":"2024-05-10T11:10:50.758046+0000","last_peered":"2024-05-10T11:10:50.758046+0000","last_clean":"2024-05-10T11:10:50.758046+0000","last_became_active":"2024-05-10T11:07:47.194645+0000","last_became_peered":"2024-05-10T11:07:47.194645+0000","last_unstale":"2024-05-10T11:10:50.758046+0000","last_undegraded":"2024-05-10T11:10:50.758046+0000","last_fullsized":"2024-05-10T11:10:50.758046+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:34.962163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072001+0000","last_change":"2024-05-10T11:07:42.126378+0000","last_active":"2024-05-10T11:09:23.072001+0000","last_peered":"2024-05-10T11:09:23.072001+0000","last_clean":"2024-05-10T11:09:23.072001+0000","last_became_active":"2024-05-10T11:07:42.125984+0000","last_became_peered":"2024-05-10T11:07:42.125984+0000","last_unstale":"2024-05-10T11:09:23.072001+0000","last_undegraded":"2024-05-10T11:09:23.072001+0000","last_fullsized":"2024-05-10T11:09:23.072001+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:15:50.378737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.065803+0000","last_change":"2024-05-10T11:08:00.872342+0000","last_active":"2024-05-10T11:09:23.065803+0000","last_peered":"2024-05-10T11:09:23.065803+0000","last_clean":"2024-05-10T11:09:23.065803+0000","last_became_active":"2024-05-10T11:08:00.871926+0000","last_became_peered":"2024-05-10T11:08:00.871926+0000","last_unstale":"2024-05-10T11:09:23.065803+0000","last_undegraded":"2024-05-10T11:09:23.065803+0000","last_fullsized":"2024-05-10T11:09:23.065803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:27:16.519752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.070093+0000","last_change":"2024-05-10T11:07:42.384236+0000","last_active":"2024-05-10T11:09:23.070093+0000","last_peered":"2024-05-10T11:09:23.070093+0000","last_clean":"2024-05-10T11:09:23.070093+0000","last_became_active":"2024-05-10T11:07:42.383908+0000","last_became_peered":"2024-05-10T11:07:42.383908+0000","last_unstale":"2024-05-10T11:09:23.070093+0000","last_undegraded":"2024-05-10T11:09:23.070093+0000","last_fullsized":"2024-05-10T11:09:23.070093+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:09:42.302506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.770400+0000","last_change":"2024-05-10T11:07:47.201645+0000","last_active":"2024-05-10T11:10:50.770400+0000","last_peered":"2024-05-10T11:10:50.770400+0000","last_clean":"2024-05-10T11:10:50.770400+0000","last_became_active":"2024-05-10T11:07:47.201300+0000","last_became_peered":"2024-05-10T11:07:47.201300+0000","last_unstale":"2024-05-10T11:10:50.770400+0000","last_undegraded":"2024-05-10T11:10:50.770400+0000","last_fullsized":"2024-05-10T11:10:50.770400+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:37.268530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.077040+0000","last_change":"2024-05-10T11:08:00.917988+0000","last_active":"2024-05-10T11:09:23.077040+0000","last_peered":"2024-05-10T11:09:23.077040+0000","last_clean":"2024-05-10T11:09:23.077040+0000","last_became_active":"2024-05-10T11:08:00.917457+0000","last_became_peered":"2024-05-10T11:08:00.917457+0000","last_unstale":"2024-05-10T11:09:23.077040+0000","last_undegraded":"2024-05-10T11:09:23.077040+0000","last_fullsized":"2024-05-10T11:09:23.077040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:16:01.816917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'17","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:52.582808+0000","last_change":"2024-05-10T11:07:42.141005+0000","last_active":"2024-05-10T11:09:52.582808+0000","last_peered":"2024-05-10T11:09:52.582808+0000","last_clean":"2024-05-10T11:09:52.582808+0000","last_became_active":"2024-05-10T11:07:42.140634+0000","last_became_peered":"2024-05-10T11:07:42.140634+0000","last_unstale":"2024-05-10T11:09:52.582808+0000","last_undegraded":"2024-05-10T11:09:52.582808+0000","last_fullsized":"2024-05-10T11:09:52.582808+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:00:32.029374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.788308+0000","last_change":"2024-05-10T11:07:47.206864+0000","last_active":"2024-05-10T11:10:50.788308+0000","last_peered":"2024-05-10T11:10:50.788308+0000","last_clean":"2024-05-10T11:10:50.788308+0000","last_became_active":"2024-05-10T11:07:47.206387+0000","last_became_peered":"2024-05-10T11:07:47.206387+0000","last_unstale":"2024-05-10T11:10:50.788308+0000","last_undegraded":"2024-05-10T11:10:50.788308+0000","last_fullsized":"2024-05-10T11:10:50.788308+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:12:35.242270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069114+0000","last_change":"2024-05-10T11:08:00.867872+0000","last_active":"2024-05-10T11:09:23.069114+0000","last_peered":"2024-05-10T11:09:23.069114+0000","last_clean":"2024-05-10T11:09:23.069114+0000","last_became_active":"2024-05-10T11:08:00.867461+0000","last_became_peered":"2024-05-10T11:08:00.867461+0000","last_unstale":"2024-05-10T11:09:23.069114+0000","last_undegraded":"2024-05-10T11:09:23.069114+0000","last_fullsized":"2024-05-10T11:09:23.069114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:29.617335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.063128+0000","last_change":"2024-05-10T11:07:42.599520+0000","last_active":"2024-05-10T11:09:23.063128+0000","last_peered":"2024-05-10T11:09:23.063128+0000","last_clean":"2024-05-10T11:09:23.063128+0000","last_became_active":"2024-05-10T11:07:42.599209+0000","last_became_peered":"2024-05-10T11:07:42.599209+0000","last_unstale":"2024-05-10T11:09:23.063128+0000","last_undegraded":"2024-05-10T11:09:23.063128+0000","last_fullsized":"2024-05-10T11:09:23.063128+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:52.899718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.799440+0000","last_change":"2024-05-10T11:07:47.218286+0000","last_active":"2024-05-10T11:10:50.799440+0000","last_peered":"2024-05-10T11:10:50.799440+0000","last_clean":"2024-05-10T11:10:50.799440+0000","last_became_active":"2024-05-10T11:07:47.217960+0000","last_became_peered":"2024-05-10T11:07:47.217960+0000","last_unstale":"2024-05-10T11:10:50.799440+0000","last_undegraded":"2024-05-10T11:10:50.799440+0000","last_fullsized":"2024-05-10T11:10:50.799440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:11:37.208359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062781+0000","last_change":"2024-05-10T11:08:00.919578+0000","last_active":"2024-05-10T11:09:23.062781+0000","last_peered":"2024-05-10T11:09:23.062781+0000","last_clean":"2024-05-10T11:09:23.062781+0000","last_became_active":"2024-05-10T11:08:00.918546+0000","last_became_peered":"2024-05-10T11:08:00.918546+0000","last_unstale":"2024-05-10T11:09:23.062781+0000","last_undegraded":"2024-05-10T11:09:23.062781+0000","last_fullsized":"2024-05-10T11:09:23.062781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:39:05.062210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.072522+0000","last_change":"2024-05-10T11:07:42.384577+0000","last_active":"2024-05-10T11:09:23.072522+0000","last_peered":"2024-05-10T11:09:23.072522+0000","last_clean":"2024-05-10T11:09:23.072522+0000","last_became_active":"2024-05-10T11:07:42.384301+0000","last_became_peered":"2024-05-10T11:07:42.384301+0000","last_unstale":"2024-05-10T11:09:23.072522+0000","last_undegraded":"2024-05-10T11:09:23.072522+0000","last_fullsized":"2024-05-10T11:09:23.072522+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:11:03.359490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.723954+0000","last_change":"2024-05-10T11:07:47.189708+0000","last_active":"2024-05-10T11:10:50.723954+0000","last_peered":"2024-05-10T11:10:50.723954+0000","last_clean":"2024-05-10T11:10:50.723954+0000","last_became_active":"2024-05-10T11:07:47.189367+0000","last_became_peered":"2024-05-10T11:07:47.189367+0000","last_unstale":"2024-05-10T11:10:50.723954+0000","last_undegraded":"2024-05-10T11:10:50.723954+0000","last_fullsized":"2024-05-10T11:10:50.723954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:25.410477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068671+0000","last_change":"2024-05-10T11:08:00.899359+0000","last_active":"2024-05-10T11:09:23.068671+0000","last_peered":"2024-05-10T11:09:23.068671+0000","last_clean":"2024-05-10T11:09:23.068671+0000","last_became_active":"2024-05-10T11:08:00.898924+0000","last_became_peered":"2024-05-10T11:08:00.898924+0000","last_unstale":"2024-05-10T11:09:23.068671+0000","last_undegraded":"2024-05-10T11:09:23.068671+0000","last_fullsized":"2024-05-10T11:09:23.068671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:14:17.427245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.076950+0000","last_change":"2024-05-10T11:07:42.140262+0000","last_active":"2024-05-10T11:09:23.076950+0000","last_peered":"2024-05-10T11:09:23.076950+0000","last_clean":"2024-05-10T11:09:23.076950+0000","last_became_active":"2024-05-10T11:07:42.138853+0000","last_became_peered":"2024-05-10T11:07:42.138853+0000","last_unstale":"2024-05-10T11:09:23.076950+0000","last_undegraded":"2024-05-10T11:09:23.076950+0000","last_fullsized":"2024-05-10T11:09:23.076950+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:05:39.995237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.764312+0000","last_change":"2024-05-10T11:07:47.203637+0000","last_active":"2024-05-10T11:10:50.764312+0000","last_peered":"2024-05-10T11:10:50.764312+0000","last_clean":"2024-05-10T11:10:50.764312+0000","last_became_active":"2024-05-10T11:07:47.203286+0000","last_became_peered":"2024-05-10T11:07:47.203286+0000","last_unstale":"2024-05-10T11:10:50.764312+0000","last_undegraded":"2024-05-10T11:10:50.764312+0000","last_fullsized":"2024-05-10T11:10:50.764312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:27:46.924625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:35.945269+0000","last_change":"2024-05-10T11:08:00.890082+0000","last_active":"2024-05-10T11:09:35.945269+0000","last_peered":"2024-05-10T11:09:35.945269+0000","last_clean":"2024-05-10T11:09:35.945269+0000","last_became_active":"2024-05-10T11:08:00.889673+0000","last_became_peered":"2024-05-10T11:08:00.889673+0000","last_unstale":"2024-05-10T11:09:35.945269+0000","last_undegraded":"2024-05-10T11:09:35.945269+0000","last_fullsized":"2024-05-10T11:09:35.945269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:21:31.167934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.069404+0000","last_change":"2024-05-10T11:07:42.130703+0000","last_active":"2024-05-10T11:09:23.069404+0000","last_peered":"2024-05-10T11:09:23.069404+0000","last_clean":"2024-05-10T11:09:23.069404+0000","last_became_active":"2024-05-10T11:07:42.130098+0000","last_became_peered":"2024-05-10T11:07:42.130098+0000","last_unstale":"2024-05-10T11:09:23.069404+0000","last_undegraded":"2024-05-10T11:09:23.069404+0000","last_fullsized":"2024-05-10T11:09:23.069404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:59:39.783151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.806618+0000","last_change":"2024-05-10T11:07:47.220497+0000","last_active":"2024-05-10T11:10:50.806618+0000","last_peered":"2024-05-10T11:10:50.806618+0000","last_clean":"2024-05-10T11:10:50.806618+0000","last_became_active":"2024-05-10T11:07:47.219916+0000","last_became_peered":"2024-05-10T11:07:47.219916+0000","last_unstale":"2024-05-10T11:10:50.806618+0000","last_undegraded":"2024-05-10T11:10:50.806618+0000","last_fullsized":"2024-05-10T11:10:50.806618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:26:43.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.062702+0000","last_change":"2024-05-10T11:08:00.918258+0000","last_active":"2024-05-10T11:09:23.062702+0000","last_peered":"2024-05-10T11:09:23.062702+0000","last_clean":"2024-05-10T11:09:23.062702+0000","last_became_active":"2024-05-10T11:08:00.917808+0000","last_became_peered":"2024-05-10T11:08:00.917808+0000","last_unstale":"2024-05-10T11:09:23.062702+0000","last_undegraded":"2024-05-10T11:09:23.062702+0000","last_fullsized":"2024-05-10T11:09:23.062702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:14:08.867024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.068897+0000","last_change":"2024-05-10T11:07:42.127661+0000","last_active":"2024-05-10T11:09:23.068897+0000","last_peered":"2024-05-10T11:09:23.068897+0000","last_clean":"2024-05-10T11:09:23.068897+0000","last_became_active":"2024-05-10T11:07:42.127256+0000","last_became_peered":"2024-05-10T11:07:42.127256+0000","last_unstale":"2024-05-10T11:09:23.068897+0000","last_undegraded":"2024-05-10T11:09:23.068897+0000","last_fullsized":"2024-05-10T11:09:23.068897+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:14:37.821350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.786018+0000","last_change":"2024-05-10T11:07:47.199589+0000","last_active":"2024-05-10T11:10:50.786018+0000","last_peered":"2024-05-10T11:10:50.786018+0000","last_clean":"2024-05-10T11:10:50.786018+0000","last_became_active":"2024-05-10T11:07:47.199145+0000","last_became_peered":"2024-05-10T11:07:47.199145+0000","last_unstale":"2024-05-10T11:10:50.786018+0000","last_undegraded":"2024-05-10T11:10:50.786018+0000","last_fullsized":"2024-05-10T11:10:50.786018+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:46.577543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.085808+0000","last_change":"2024-05-10T11:09:23.085049+0000","last_active":"2024-05-10T11:09:23.085808+0000","last_peered":"2024-05-10T11:09:23.085808+0000","last_clean":"2024-05-10T11:09:23.085808+0000","last_became_active":"2024-05-10T11:09:23.084420+0000","last_became_peered":"2024-05-10T11:09:23.084420+0000","last_unstale":"2024-05-10T11:09:23.085808+0000","last_undegraded":"2024-05-10T11:09:23.085808+0000","last_fullsized":"2024-05-10T11:09:23.085808+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:32:33.346434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'5","reported_seq":173,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:11:15.223310+0000","last_change":"2024-05-10T11:07:42.599395+0000","last_active":"2024-05-10T11:11:15.223310+0000","last_peered":"2024-05-10T11:11:15.223310+0000","last_clean":"2024-05-10T11:11:15.223310+0000","last_became_active":"2024-05-10T11:07:42.598934+0000","last_became_peered":"2024-05-10T11:07:42.598934+0000","last_unstale":"2024-05-10T11:11:15.223310+0000","last_undegraded":"2024-05-10T11:11:15.223310+0000","last_fullsized":"2024-05-10T11:11:15.223310+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:49:42.414789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.769182+0000","last_change":"2024-05-10T11:07:47.192087+0000","last_active":"2024-05-10T11:10:50.769182+0000","last_peered":"2024-05-10T11:10:50.769182+0000","last_clean":"2024-05-10T11:10:50.769182+0000","last_became_active":"2024-05-10T11:07:47.191041+0000","last_became_peered":"2024-05-10T11:07:47.191041+0000","last_unstale":"2024-05-10T11:10:50.769182+0000","last_undegraded":"2024-05-10T11:10:50.769182+0000","last_fullsized":"2024-05-10T11:10:50.769182+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:50:14.052863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.081786+0000","last_change":"2024-05-10T11:09:23.080432+0000","last_active":"2024-05-10T11:09:23.081786+0000","last_peered":"2024-05-10T11:09:23.081786+0000","last_clean":"2024-05-10T11:09:23.081786+0000","last_became_active":"2024-05-10T11:09:23.074934+0000","last_became_peered":"2024-05-10T11:09:23.074934+0000","last_unstale":"2024-05-10T11:09:23.081786+0000","last_undegraded":"2024-05-10T11:09:23.081786+0000","last_fullsized":"2024-05-10T11:09:23.081786+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:12:14.613323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066498+0000","last_change":"2024-05-10T11:07:42.130718+0000","last_active":"2024-05-10T11:09:23.066498+0000","last_peered":"2024-05-10T11:09:23.066498+0000","last_clean":"2024-05-10T11:09:23.066498+0000","last_became_active":"2024-05-10T11:07:42.130355+0000","last_became_peered":"2024-05-10T11:07:42.130355+0000","last_unstale":"2024-05-10T11:09:23.066498+0000","last_undegraded":"2024-05-10T11:09:23.066498+0000","last_fullsized":"2024-05-10T11:09:23.066498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:16:48.093146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.784346+0000","last_change":"2024-05-10T11:07:47.213569+0000","last_active":"2024-05-10T11:10:50.784346+0000","last_peered":"2024-05-10T11:10:50.784346+0000","last_clean":"2024-05-10T11:10:50.784346+0000","last_became_active":"2024-05-10T11:07:47.213148+0000","last_became_peered":"2024-05-10T11:07:47.213148+0000","last_unstale":"2024-05-10T11:10:50.784346+0000","last_undegraded":"2024-05-10T11:10:50.784346+0000","last_fullsized":"2024-05-10T11:10:50.784346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:42:41.161274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.066016+0000","last_change":"2024-05-10T11:08:00.870937+0000","last_active":"2024-05-10T11:09:23.066016+0000","last_peered":"2024-05-10T11:09:23.066016+0000","last_clean":"2024-05-10T11:09:23.066016+0000","last_became_active":"2024-05-10T11:08:00.869276+0000","last_became_peered":"2024-05-10T11:08:00.869276+0000","last_unstale":"2024-05-10T11:09:23.066016+0000","last_undegraded":"2024-05-10T11:09:23.066016+0000","last_fullsized":"2024-05-10T11:09:23.066016+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:37.058875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:09:23.079741+0000","last_change":"2024-05-10T11:07:42.130544+0000","last_active":"2024-05-10T11:09:23.079741+0000","last_peered":"2024-05-10T11:09:23.079741+0000","last_clean":"2024-05-10T11:09:23.079741+0000","last_became_active":"2024-05-10T11:07:42.128912+0000","last_became_peered":"2024-05-10T11:07:42.128912+0000","last_unstale":"2024-05-10T11:09:23.079741+0000","last_undegraded":"2024-05-10T11:09:23.079741+0000","last_fullsized":"2024-05-10T11:09:23.079741+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:22:23.521911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.763514+0000","last_change":"2024-05-10T11:07:47.194346+0000","last_active":"2024-05-10T11:10:50.763514+0000","last_peered":"2024-05-10T11:10:50.763514+0000","last_clean":"2024-05-10T11:10:50.763514+0000","last_became_active":"2024-05-10T11:07:47.194019+0000","last_became_peered":"2024-05-10T11:07:47.194019+0000","last_unstale":"2024-05-10T11:10:50.763514+0000","last_undegraded":"2024-05-10T11:10:50.763514+0000","last_fullsized":"2024-05-10T11:10:50.763514+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:33:05.262347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.759134+0000","last_change":"2024-05-10T11:07:47.192001+0000","last_active":"2024-05-10T11:10:50.759134+0000","last_peered":"2024-05-10T11:10:50.759134+0000","last_clean":"2024-05-10T11:10:50.759134+0000","last_became_active":"2024-05-10T11:07:47.191668+0000","last_became_peered":"2024-05-10T11:07:47.191668+0000","last_unstale":"2024-05-10T11:10:50.759134+0000","last_undegraded":"2024-05-10T11:10:50.759134+0000","last_fullsized":"2024-05-10T11:10:50.759134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:13:33.680899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.749397+0000","last_change":"2024-05-10T11:07:47.192226+0000","last_active":"2024-05-10T11:10:50.749397+0000","last_peered":"2024-05-10T11:10:50.749397+0000","last_clean":"2024-05-10T11:10:50.749397+0000","last_became_active":"2024-05-10T11:07:47.191444+0000","last_became_peered":"2024-05-10T11:07:47.191444+0000","last_unstale":"2024-05-10T11:10:50.749397+0000","last_undegraded":"2024-05-10T11:10:50.749397+0000","last_fullsized":"2024-05-10T11:10:50.749397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:08.348986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.792269+0000","last_change":"2024-05-10T11:07:47.210801+0000","last_active":"2024-05-10T11:10:50.792269+0000","last_peered":"2024-05-10T11:10:50.792269+0000","last_clean":"2024-05-10T11:10:50.792269+0000","last_became_active":"2024-05-10T11:07:47.210496+0000","last_became_peered":"2024-05-10T11:07:47.210496+0000","last_unstale":"2024-05-10T11:10:50.792269+0000","last_undegraded":"2024-05-10T11:10:50.792269+0000","last_fullsized":"2024-05-10T11:10:50.792269+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:11:03.661457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.802348+0000","last_change":"2024-05-10T11:07:47.211033+0000","last_active":"2024-05-10T11:10:50.802348+0000","last_peered":"2024-05-10T11:10:50.802348+0000","last_clean":"2024-05-10T11:10:50.802348+0000","last_became_active":"2024-05-10T11:07:47.206061+0000","last_became_peered":"2024-05-10T11:07:47.206061+0000","last_unstale":"2024-05-10T11:10:50.802348+0000","last_undegraded":"2024-05-10T11:10:50.802348+0000","last_fullsized":"2024-05-10T11:10:50.802348+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:04:05.878270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.805488+0000","last_change":"2024-05-10T11:07:47.214966+0000","last_active":"2024-05-10T11:10:50.805488+0000","last_peered":"2024-05-10T11:10:50.805488+0000","last_clean":"2024-05-10T11:10:50.805488+0000","last_became_active":"2024-05-10T11:07:47.214563+0000","last_became_peered":"2024-05-10T11:07:47.214563+0000","last_unstale":"2024-05-10T11:10:50.805488+0000","last_undegraded":"2024-05-10T11:10:50.805488+0000","last_fullsized":"2024-05-10T11:10:50.805488+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:29:01.428893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.758172+0000","last_change":"2024-05-10T11:07:47.196740+0000","last_active":"2024-05-10T11:10:50.758172+0000","last_peered":"2024-05-10T11:10:50.758172+0000","last_clean":"2024-05-10T11:10:50.758172+0000","last_became_active":"2024-05-10T11:07:47.196388+0000","last_became_peered":"2024-05-10T11:07:47.196388+0000","last_unstale":"2024-05-10T11:10:50.758172+0000","last_undegraded":"2024-05-10T11:10:50.758172+0000","last_fullsized":"2024-05-10T11:10:50.758172+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:23:09.055338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.748036+0000","last_change":"2024-05-10T11:07:47.193568+0000","last_active":"2024-05-10T11:10:50.748036+0000","last_peered":"2024-05-10T11:10:50.748036+0000","last_clean":"2024-05-10T11:10:50.748036+0000","last_became_active":"2024-05-10T11:07:47.193202+0000","last_became_peered":"2024-05-10T11:07:47.193202+0000","last_unstale":"2024-05-10T11:10:50.748036+0000","last_undegraded":"2024-05-10T11:10:50.748036+0000","last_fullsized":"2024-05-10T11:10:50.748036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:36:24.556266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.761674+0000","last_change":"2024-05-10T11:07:47.215344+0000","last_active":"2024-05-10T11:10:50.761674+0000","last_peered":"2024-05-10T11:10:50.761674+0000","last_clean":"2024-05-10T11:10:50.761674+0000","last_became_active":"2024-05-10T11:07:47.214896+0000","last_became_peered":"2024-05-10T11:07:47.214896+0000","last_unstale":"2024-05-10T11:10:50.761674+0000","last_undegraded":"2024-05-10T11:10:50.761674+0000","last_fullsized":"2024-05-10T11:10:50.761674+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:23:07.845212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.762357+0000","last_change":"2024-05-10T11:07:47.219047+0000","last_active":"2024-05-10T11:10:50.762357+0000","last_peered":"2024-05-10T11:10:50.762357+0000","last_clean":"2024-05-10T11:10:50.762357+0000","last_became_active":"2024-05-10T11:07:47.218778+0000","last_became_peered":"2024-05-10T11:07:47.218778+0000","last_unstale":"2024-05-10T11:10:50.762357+0000","last_undegraded":"2024-05-10T11:10:50.762357+0000","last_fullsized":"2024-05-10T11:10:50.762357+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:02:48.945259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.787652+0000","last_change":"2024-05-10T11:07:47.213412+0000","last_active":"2024-05-10T11:10:50.787652+0000","last_peered":"2024-05-10T11:10:50.787652+0000","last_clean":"2024-05-10T11:10:50.787652+0000","last_became_active":"2024-05-10T11:07:47.212963+0000","last_became_peered":"2024-05-10T11:07:47.212963+0000","last_unstale":"2024-05-10T11:10:50.787652+0000","last_undegraded":"2024-05-10T11:10:50.787652+0000","last_fullsized":"2024-05-10T11:10:50.787652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:33:05.224184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.778780+0000","last_change":"2024-05-10T11:07:47.207998+0000","last_active":"2024-05-10T11:10:50.778780+0000","last_peered":"2024-05-10T11:10:50.778780+0000","last_clean":"2024-05-10T11:10:50.778780+0000","last_became_active":"2024-05-10T11:07:47.207298+0000","last_became_peered":"2024-05-10T11:07:47.207298+0000","last_unstale":"2024-05-10T11:10:50.778780+0000","last_undegraded":"2024-05-10T11:10:50.778780+0000","last_fullsized":"2024-05-10T11:10:50.778780+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:59:14.001103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.806604+0000","last_change":"2024-05-10T11:07:47.202216+0000","last_active":"2024-05-10T11:10:50.806604+0000","last_peered":"2024-05-10T11:10:50.806604+0000","last_clean":"2024-05-10T11:10:50.806604+0000","last_became_active":"2024-05-10T11:07:47.201867+0000","last_became_peered":"2024-05-10T11:07:47.201867+0000","last_unstale":"2024-05-10T11:10:50.806604+0000","last_undegraded":"2024-05-10T11:10:50.806604+0000","last_fullsized":"2024-05-10T11:10:50.806604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:48:59.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-10T11:10:50.757654+0000","last_change":"2024-05-10T11:07:47.194149+0000","last_active":"2024-05-10T11:10:50.757654+0000","last_peered":"2024-05-10T11:10:50.757654+0000","last_clean":"2024-05-10T11:10:50.757654+0000","last_became_active":"2024-05-10T11:07:47.193838+0000","last_became_peered":"2024-05-10T11:07:47.193838+0000","last_unstale":"2024-05-10T11:10:50.757654+0000","last_undegraded":"2024-05-10T11:10:50.757654+0000","last_fullsized":"2024-05-10T11:10:50.757654+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:46:44.181446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247309,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71848,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1264,"num_read_kb":717,"num_write":257,"num_write_kb":149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":712,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":215544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678077,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":522636,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874307,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1150976,"data_stored":1174788,"data_compressed":68282,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070535,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":526572,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266767,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":522832,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462996,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":1119492,"data_compressed":68282,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626522,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":517980,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":583904,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738492,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":519436,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":583434,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182507,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":517688,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986277,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":523208,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822752,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":1114542,"data_compressed":68282,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34167,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34167,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":66176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":67162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-10T11:11:21.800 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-10T11:11:21.800 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-10T11:11:21.801 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-10T11:11:21.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph health --format=json 2024-05-10T11:11:22.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:21 smithi189 ceph-mon[34842]: pgmap v440: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:22.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:21 smithi189 ceph-mon[34842]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:11:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:22.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:21 smithi027 ceph-mon[30054]: pgmap v440: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:22.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:21 smithi027 ceph-mon[30054]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:11:22.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:21 smithi076 ceph-mon[34152]: pgmap v440: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:22.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:21 smithi076 ceph-mon[34152]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:11:24.256 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:23 smithi027 ceph-mon[30054]: pgmap v441: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:24.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:23 smithi076 ceph-mon[34152]: pgmap v441: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:24.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:23 smithi189 ceph-mon[34842]: pgmap v441: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.549+0000 7f765a200700 1 -- 172.21.15.27:0/279990692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7654106080 msgr2=0x7f7654112590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.549+0000 7f765a200700 1 --2- 172.21.15.27:0/279990692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7654106080 0x7f7654112590 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f764c00b0a0 tx=0x7f764c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:11:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.550+0000 7f765a200700 1 -- 172.21.15.27:0/279990692 shutdown_connections 2024-05-10T11:11:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.550+0000 7f765a200700 1 --2- 172.21.15.27:0/279990692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765410b6c0 0x7f765410baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.550+0000 7f765a200700 1 --2- 172.21.15.27:0/279990692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7654106080 0x7f7654112590 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.550+0000 7f765a200700 1 --2- 172.21.15.27:0/279990692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76541056c0 0x7f7654105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.550+0000 7f765a200700 1 -- 172.21.15.27:0/279990692 >> 172.21.15.27:0/279990692 conn(0x7f76541011b0 msgr2=0x7f76541035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.550+0000 7f765a200700 1 -- 172.21.15.27:0/279990692 shutdown_connections 2024-05-10T11:11:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.550+0000 7f765a200700 1 -- 172.21.15.27:0/279990692 wait complete. 2024-05-10T11:11:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.551+0000 7f765a200700 1 Processor -- start 2024-05-10T11:11:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.551+0000 7f765a200700 1 -- start start 2024-05-10T11:11:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.552+0000 7f765a200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76541056c0 0x7f765419bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.552+0000 7f765a200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7654106080 0x7f765419c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.552+0000 7f765a200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765410b6c0 0x7f76541a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.552+0000 7f765a200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7654115910 con 0x7f76541056c0 2024-05-10T11:11:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.552+0000 7f765a200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7654115790 con 0x7f765410b6c0 2024-05-10T11:11:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.552+0000 7f765a200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7654115a90 con 0x7f7654106080 2024-05-10T11:11:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.552+0000 7f7653400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76541056c0 0x7f765419bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.552+0000 7f7653400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76541056c0 0x7f765419bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35338/0 (socket says 172.21.15.27:35338) 2024-05-10T11:11:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.552+0000 7f7653400700 1 -- 172.21.15.27:0/604211983 learned_addr learned my addr 172.21.15.27:0/604211983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.552+0000 7f7652a00700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7654106080 0x7f765419c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.552+0000 7f7653e00700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765410b6c0 0x7f76541a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.553+0000 7f7653400700 1 -- 172.21.15.27:0/604211983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7654106080 msgr2=0x7f765419c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.553+0000 7f7653400700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7654106080 0x7f765419c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.553+0000 7f7653400700 1 -- 172.21.15.27:0/604211983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765410b6c0 msgr2=0x7f76541a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.553+0000 7f7653400700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765410b6c0 0x7f76541a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.553+0000 7f7653400700 1 -- 172.21.15.27:0/604211983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76541a0d60 con 0x7f76541056c0 2024-05-10T11:11:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.553+0000 7f7653e00700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765410b6c0 0x7f76541a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:11:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.553+0000 7f7653400700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76541056c0 0x7f765419bda0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f764400e9c0 tx=0x7f764400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.553+0000 7f764be00700 1 -- 172.21.15.27:0/604211983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764400cbd0 con 0x7f76541056c0 2024-05-10T11:11:24.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.553+0000 7f764be00700 1 -- 172.21.15.27:0/604211983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f764400cd70 con 0x7f76541056c0 2024-05-10T11:11:24.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.553+0000 7f765a200700 1 -- 172.21.15.27:0/604211983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76541a1050 con 0x7f76541056c0 2024-05-10T11:11:24.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.553+0000 7f764be00700 1 -- 172.21.15.27:0/604211983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7644005610 con 0x7f76541056c0 2024-05-10T11:11:24.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.554+0000 7f765a200700 1 -- 172.21.15.27:0/604211983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76541a88d0 con 0x7f76541056c0 2024-05-10T11:11:24.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.555+0000 7f765a200700 1 -- 172.21.15.27:0/604211983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f765406a190 con 0x7f76541056c0 2024-05-10T11:11:24.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.557+0000 7f764be00700 1 -- 172.21.15.27:0/604211983 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f7644004750 con 0x7f76541056c0 2024-05-10T11:11:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.559+0000 7f764be00700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76380b2900 0x7f76380b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.559+0000 7f764be00700 1 -- 172.21.15.27:0/604211983 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f76440e2a90 con 0x7f76541056c0 2024-05-10T11:11:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.559+0000 7f7652a00700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76380b2900 0x7f76380b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.560+0000 7f7652a00700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76380b2900 0x7f76380b4dc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f76541037a0 tx=0x7f7640009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.561+0000 7f764be00700 1 -- 172.21.15.27:0/604211983 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7644054d70 con 0x7f76541056c0 2024-05-10T11:11:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.852+0000 7f765a200700 1 -- 172.21.15.27:0/604211983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f765402e510 con 0x7f76541056c0 2024-05-10T11:11:24.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.853+0000 7f764be00700 1 -- 172.21.15.27:0/604211983 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f76440aa360 con 0x7f76541056c0 2024-05-10T11:11:24.853 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:11:24.853 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-10T11:11:24.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.857+0000 7f765a200700 1 -- 172.21.15.27:0/604211983 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76380b2900 msgr2=0x7f76380b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:24.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.857+0000 7f765a200700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76380b2900 0x7f76380b4dc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f76541037a0 tx=0x7f7640009260 comp rx=0 tx=0).stop 2024-05-10T11:11:24.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.857+0000 7f765a200700 1 -- 172.21.15.27:0/604211983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76541056c0 msgr2=0x7f765419bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:24.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.857+0000 7f765a200700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76541056c0 0x7f765419bda0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f764400e9c0 tx=0x7f764400ecd0 comp rx=0 tx=0).stop 2024-05-10T11:11:24.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.857+0000 7f765a200700 1 -- 172.21.15.27:0/604211983 shutdown_connections 2024-05-10T11:11:24.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.857+0000 7f765a200700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76380b2900 0x7f76380b4dc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:24.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.857+0000 7f765a200700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f765410b6c0 0x7f76541a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:24.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.857+0000 7f765a200700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76541056c0 0x7f765419bda0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:24.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.857+0000 7f765a200700 1 --2- 172.21.15.27:0/604211983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7654106080 0x7f765419c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:24.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.857+0000 7f765a200700 1 -- 172.21.15.27:0/604211983 >> 172.21.15.27:0/604211983 conn(0x7f76541011b0 msgr2=0x7f7654101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:24.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.858+0000 7f765a200700 1 -- 172.21.15.27:0/604211983 shutdown_connections 2024-05-10T11:11:24.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:24.858+0000 7f765a200700 1 -- 172.21.15.27:0/604211983 wait complete. 2024-05-10T11:11:25.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:24 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/604211983' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-10T11:11:25.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:24 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/604211983' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-10T11:11:25.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:24 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/604211983' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-10T11:11:25.702 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-10T11:11:25.702 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:11:26.177 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:26.326 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:25 smithi027 ceph-mon[30054]: pgmap v442: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:26.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:26 smithi076 ceph-mon[34152]: pgmap v442: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:26.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:25 smithi189 ceph-mon[34842]: pgmap v442: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:28.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:28 smithi027 ceph-mon[30054]: pgmap v443: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:28.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:28 smithi076 ceph-mon[34152]: pgmap v443: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:28.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.450+0000 7fdb20a00700 1 -- 172.21.15.27:0/2930126363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb1c075ff0 msgr2=0x7fdb1c10f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:28.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.450+0000 7fdb20a00700 1 --2- 172.21.15.27:0/2930126363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb1c075ff0 0x7fdb1c10f960 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009990 tx=0x7fdb08009ca0 comp rx=0 tx=0).stop 2024-05-10T11:11:28.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.451+0000 7fdb20a00700 1 -- 172.21.15.27:0/2930126363 shutdown_connections 2024-05-10T11:11:28.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.451+0000 7fdb20a00700 1 --2- 172.21.15.27:0/2930126363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb1c0756d0 0x7fdb1c075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:28.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.451+0000 7fdb20a00700 1 --2- 172.21.15.27:0/2930126363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb1c10fea0 0x7fdb1c112290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:28.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.451+0000 7fdb20a00700 1 --2- 172.21.15.27:0/2930126363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb1c075ff0 0x7fdb1c10f960 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:28.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.451+0000 7fdb20a00700 1 -- 172.21.15.27:0/2930126363 >> 172.21.15.27:0/2930126363 conn(0x7fdb1c07a600 msgr2=0x7fdb1c07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:28.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.451+0000 7fdb20a00700 1 -- 172.21.15.27:0/2930126363 shutdown_connections 2024-05-10T11:11:28.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.451+0000 7fdb20a00700 1 -- 172.21.15.27:0/2930126363 wait complete. 2024-05-10T11:11:28.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.452+0000 7fdb20a00700 1 Processor -- start 2024-05-10T11:11:28.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.452+0000 7fdb20a00700 1 -- start start 2024-05-10T11:11:28.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.453+0000 7fdb20a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb1c0756d0 0x7fdb1c19bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:28.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.453+0000 7fdb20a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb1c075ff0 0x7fdb1c19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:28.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.453+0000 7fdb20a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb1c10fea0 0x7fdb1c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:28.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.453+0000 7fdb20a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdb1c115890 con 0x7fdb1c10fea0 2024-05-10T11:11:28.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.453+0000 7fdb20a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdb1c115710 con 0x7fdb1c075ff0 2024-05-10T11:11:28.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.453+0000 7fdb20a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdb1c115a10 con 0x7fdb1c0756d0 2024-05-10T11:11:28.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.453+0000 7fdb19e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb1c0756d0 0x7fdb1c19bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:28.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.453+0000 7fdb1a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb1c10fea0 0x7fdb1c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:28.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.453+0000 7fdb19e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb1c0756d0 0x7fdb1c19bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41888/0 (socket says 172.21.15.27:41888) 2024-05-10T11:11:28.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.453+0000 7fdb19400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb1c075ff0 0x7fdb1c19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:28.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.453+0000 7fdb19e00700 1 -- 172.21.15.27:0/638959358 learned_addr learned my addr 172.21.15.27:0/638959358 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:28.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.454+0000 7fdb19e00700 1 -- 172.21.15.27:0/638959358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb1c075ff0 msgr2=0x7fdb1c19c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:28.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.454+0000 7fdb19e00700 1 --2- 172.21.15.27:0/638959358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb1c075ff0 0x7fdb1c19c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:28.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.454+0000 7fdb19e00700 1 -- 172.21.15.27:0/638959358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb1c10fea0 msgr2=0x7fdb1c1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:28.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.454+0000 7fdb19e00700 1 --2- 172.21.15.27:0/638959358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb1c10fea0 0x7fdb1c1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:28.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.454+0000 7fdb19e00700 1 -- 172.21.15.27:0/638959358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1c1a0d90 con 0x7fdb1c0756d0 2024-05-10T11:11:28.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.454+0000 7fdb19e00700 1 --2- 172.21.15.27:0/638959358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb1c0756d0 0x7fdb1c19bd80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c00e9a0 tx=0x7fdb0c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:28.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.454+0000 7fdb12a00700 1 -- 172.21.15.27:0/638959358 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c00ca10 con 0x7fdb1c0756d0 2024-05-10T11:11:28.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.454+0000 7fdb12a00700 1 -- 172.21.15.27:0/638959358 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb0c00cbb0 con 0x7fdb1c0756d0 2024-05-10T11:11:28.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.454+0000 7fdb12a00700 1 -- 172.21.15.27:0/638959358 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c018680 con 0x7fdb1c0756d0 2024-05-10T11:11:28.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.454+0000 7fdb20a00700 1 -- 172.21.15.27:0/638959358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb1c1a1080 con 0x7fdb1c0756d0 2024-05-10T11:11:28.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.455+0000 7fdb20a00700 1 -- 172.21.15.27:0/638959358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb1c1a8900 con 0x7fdb1c0756d0 2024-05-10T11:11:28.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.456+0000 7fdb20a00700 1 -- 172.21.15.27:0/638959358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb1c06a190 con 0x7fdb1c0756d0 2024-05-10T11:11:28.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.458+0000 7fdb12a00700 1 -- 172.21.15.27:0/638959358 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fdb0c010430 con 0x7fdb1c0756d0 2024-05-10T11:11:28.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.459+0000 7fdb12a00700 1 --2- 172.21.15.27:0/638959358 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdb000b2880 0x7fdb000b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:28.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.460+0000 7fdb12a00700 1 -- 172.21.15.27:0/638959358 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fdb0c0e2b10 con 0x7fdb1c0756d0 2024-05-10T11:11:28.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.460+0000 7fdb19400700 1 --2- 172.21.15.27:0/638959358 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdb000b2880 0x7fdb000b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:28.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.461+0000 7fdb19400700 1 --2- 172.21.15.27:0/638959358 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdb000b2880 0x7fdb000b4d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c103000 tx=0x7fdb0800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:28.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.461+0000 7fdb12a00700 1 -- 172.21.15.27:0/638959358 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fdb0c0aa3e0 con 0x7fdb1c0756d0 2024-05-10T11:11:28.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:28 smithi189 ceph-mon[34842]: pgmap v443: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.751+0000 7fdb20a00700 1 -- 172.21.15.27:0/638959358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb1c02e510 con 0x7fdb1c0756d0 2024-05-10T11:11:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.752+0000 7fdb12a00700 1 -- 172.21.15.27:0/638959358 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11936 (secure 0 0 0) 0x7fdb0c0a9ce0 con 0x7fdb1c0756d0 2024-05-10T11:11:28.753 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:11:28.753 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:09:19.986922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:11:28.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.755+0000 7fdb20a00700 1 -- 172.21.15.27:0/638959358 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdb000b2880 msgr2=0x7fdb000b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.755+0000 7fdb20a00700 1 --2- 172.21.15.27:0/638959358 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdb000b2880 0x7fdb000b4d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c103000 tx=0x7fdb0800b260 comp rx=0 tx=0).stop 2024-05-10T11:11:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.755+0000 7fdb20a00700 1 -- 172.21.15.27:0/638959358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb1c0756d0 msgr2=0x7fdb1c19bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.755+0000 7fdb20a00700 1 --2- 172.21.15.27:0/638959358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb1c0756d0 0x7fdb1c19bd80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c00e9a0 tx=0x7fdb0c00ecb0 comp rx=0 tx=0).stop 2024-05-10T11:11:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.755+0000 7fdb20a00700 1 -- 172.21.15.27:0/638959358 shutdown_connections 2024-05-10T11:11:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.755+0000 7fdb20a00700 1 --2- 172.21.15.27:0/638959358 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fdb000b2880 0x7fdb000b4d40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.755+0000 7fdb20a00700 1 --2- 172.21.15.27:0/638959358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb1c0756d0 0x7fdb1c19bd80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.755+0000 7fdb20a00700 1 --2- 172.21.15.27:0/638959358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb1c10fea0 0x7fdb1c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.755+0000 7fdb20a00700 1 --2- 172.21.15.27:0/638959358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb1c075ff0 0x7fdb1c19c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.755+0000 7fdb20a00700 1 -- 172.21.15.27:0/638959358 >> 172.21.15.27:0/638959358 conn(0x7fdb1c07a600 msgr2=0x7fdb1c1006b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.756+0000 7fdb20a00700 1 -- 172.21.15.27:0/638959358 shutdown_connections 2024-05-10T11:11:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:28.756+0000 7fdb20a00700 1 -- 172.21.15.27:0/638959358 wait complete. 2024-05-10T11:11:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-05-10T11:11:29.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:29 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/638959358' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:11:29.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:29 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/638959358' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:11:29.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:29 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/638959358' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:11:29.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:11:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:30.334 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:30 smithi027 ceph-mon[30054]: pgmap v444: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:30.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:30 smithi076 ceph-mon[34152]: pgmap v444: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:30.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:30 smithi189 ceph-mon[34842]: pgmap v444: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:32.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:32 smithi027 ceph-mon[30054]: pgmap v445: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:32 smithi076 ceph-mon[34152]: pgmap v445: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:32.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:32 smithi189 ceph-mon[34842]: pgmap v445: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.605+0000 7fefa7200700 1 -- 172.21.15.27:0/3470821328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8109460 msgr2=0x7fefa8109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.605+0000 7fefa7200700 1 --2- 172.21.15.27:0/3470821328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8109460 0x7fefa8109840 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fef98009990 tx=0x7fef98009ca0 comp rx=0 tx=0).stop 2024-05-10T11:11:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.605+0000 7fefa7200700 1 -- 172.21.15.27:0/3470821328 shutdown_connections 2024-05-10T11:11:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.605+0000 7fefa7200700 1 --2- 172.21.15.27:0/3470821328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8109460 0x7fefa8109840 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.605+0000 7fefa7200700 1 --2- 172.21.15.27:0/3470821328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa8103d90 0x7fefa81102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.605+0000 7fefa7200700 1 --2- 172.21.15.27:0/3470821328 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefa81033d0 0x7fefa8103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.605+0000 7fefa7200700 1 -- 172.21.15.27:0/3470821328 >> 172.21.15.27:0/3470821328 conn(0x7fefa80fef20 msgr2=0x7fefa8101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.605+0000 7fefa7200700 1 -- 172.21.15.27:0/3470821328 shutdown_connections 2024-05-10T11:11:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.605+0000 7fefa7200700 1 -- 172.21.15.27:0/3470821328 wait complete. 2024-05-10T11:11:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.607+0000 7fefa7200700 1 Processor -- start 2024-05-10T11:11:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.608+0000 7fefa7200700 1 -- start start 2024-05-10T11:11:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.608+0000 7fefa7200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa81033d0 0x7fefa80757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.608+0000 7fefa7200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefa8103d90 0x7fefa8075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.608+0000 7fefa7200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8109460 0x7fefa8076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.608+0000 7fefa7200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fefa81135c0 con 0x7fefa81033d0 2024-05-10T11:11:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.608+0000 7fefa7200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fefa8113440 con 0x7fefa8103d90 2024-05-10T11:11:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.608+0000 7fefa7200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fefa8113740 con 0x7fefa8109460 2024-05-10T11:11:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.609+0000 7fefa5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa81033d0 0x7fefa80757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.609+0000 7fefa5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefa8103d90 0x7fefa8075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.609+0000 7fefa5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa81033d0 0x7fefa80757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35390/0 (socket says 172.21.15.27:35390) 2024-05-10T11:11:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.609+0000 7fefa5e00700 1 -- 172.21.15.27:0/1150183334 learned_addr learned my addr 172.21.15.27:0/1150183334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.609+0000 7fefa6800700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8109460 0x7fefa8076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.609+0000 7fefa5400700 1 -- 172.21.15.27:0/1150183334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8109460 msgr2=0x7fefa8076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.609+0000 7fefa5400700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8109460 0x7fefa8076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.609+0000 7fefa5400700 1 -- 172.21.15.27:0/1150183334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa81033d0 msgr2=0x7fefa80757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.609+0000 7fefa5400700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa81033d0 0x7fefa80757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.609+0000 7fefa5400700 1 -- 172.21.15.27:0/1150183334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa81a7fd0 con 0x7fefa8103d90 2024-05-10T11:11:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.609+0000 7fefa5e00700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa81033d0 0x7fefa80757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:11:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.610+0000 7fefa5400700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefa8103d90 0x7fefa8075ce0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fef9400e9f0 tx=0x7fef9400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.610+0000 7fef9ea00700 1 -- 172.21.15.27:0/1150183334 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9400cbd0 con 0x7fefa8103d90 2024-05-10T11:11:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.610+0000 7fef9ea00700 1 -- 172.21.15.27:0/1150183334 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef9400cd70 con 0x7fefa8103d90 2024-05-10T11:11:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.610+0000 7fefa7200700 1 -- 172.21.15.27:0/1150183334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa81a82c0 con 0x7fefa8103d90 2024-05-10T11:11:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.610+0000 7fef9ea00700 1 -- 172.21.15.27:0/1150183334 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef94018680 con 0x7fefa8103d90 2024-05-10T11:11:32.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.610+0000 7fefa7200700 1 -- 172.21.15.27:0/1150183334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa81a8850 con 0x7fefa8103d90 2024-05-10T11:11:32.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.612+0000 7fefa7200700 1 -- 172.21.15.27:0/1150183334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefa810b960 con 0x7fefa8103d90 2024-05-10T11:11:32.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.615+0000 7fef9ea00700 1 -- 172.21.15.27:0/1150183334 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fef94010430 con 0x7fefa8103d90 2024-05-10T11:11:32.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.617+0000 7fef9ea00700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fef8c0b2900 0x7fef8c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:32.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.618+0000 7fef9ea00700 1 -- 172.21.15.27:0/1150183334 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fef940e2780 con 0x7fefa8103d90 2024-05-10T11:11:32.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.618+0000 7fefa5e00700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fef8c0b2900 0x7fef8c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:32.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.618+0000 7fefa5e00700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fef8c0b2900 0x7fef8c0b4dc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fef98006010 tx=0x7fef9801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:32.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.620+0000 7fef9ea00700 1 -- 172.21.15.27:0/1150183334 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fef940aa050 con 0x7fefa8103d90 2024-05-10T11:11:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.904+0000 7fefa7200700 1 -- 172.21.15.27:0/1150183334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefa8076c80 con 0x7fefa8103d90 2024-05-10T11:11:32.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.905+0000 7fef9ea00700 1 -- 172.21.15.27:0/1150183334 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11936 (secure 0 0 0) 0x7fef940b20d0 con 0x7fefa8103d90 2024-05-10T11:11:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:11:32.907 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:09:19.986922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:11:32.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.909+0000 7fefa7200700 1 -- 172.21.15.27:0/1150183334 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fef8c0b2900 msgr2=0x7fef8c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:32.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.909+0000 7fefa7200700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fef8c0b2900 0x7fef8c0b4dc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fef98006010 tx=0x7fef9801a040 comp rx=0 tx=0).stop 2024-05-10T11:11:32.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.909+0000 7fefa7200700 1 -- 172.21.15.27:0/1150183334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefa8103d90 msgr2=0x7fefa8075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:32.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.909+0000 7fefa7200700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefa8103d90 0x7fefa8075ce0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fef9400e9f0 tx=0x7fef9400ed00 comp rx=0 tx=0).stop 2024-05-10T11:11:32.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.909+0000 7fefa7200700 1 -- 172.21.15.27:0/1150183334 shutdown_connections 2024-05-10T11:11:32.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.909+0000 7fefa7200700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fef8c0b2900 0x7fef8c0b4dc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:32.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.909+0000 7fefa7200700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8109460 0x7fefa8076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:32.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.909+0000 7fefa7200700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa81033d0 0x7fefa80757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:32.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.909+0000 7fefa7200700 1 --2- 172.21.15.27:0/1150183334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefa8103d90 0x7fefa8075ce0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:32.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.909+0000 7fefa7200700 1 -- 172.21.15.27:0/1150183334 >> 172.21.15.27:0/1150183334 conn(0x7fefa80fef20 msgr2=0x7fefa80ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:32.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.909+0000 7fefa7200700 1 -- 172.21.15.27:0/1150183334 shutdown_connections 2024-05-10T11:11:32.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:32.909+0000 7fefa7200700 1 -- 172.21.15.27:0/1150183334 wait complete. 2024-05-10T11:11:32.911 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-05-10T11:11:33.329 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:33 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1150183334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:11:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:33 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1150183334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:11:33.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:33 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1150183334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:11:33.895 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:11:34.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:34 smithi027 ceph-mon[30054]: pgmap v446: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:34.380 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:34.440 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:34 smithi076 ceph-mon[34152]: pgmap v446: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:34.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:34 smithi189 ceph-mon[34842]: pgmap v446: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:36.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:36 smithi027 ceph-mon[30054]: pgmap v447: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:36.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:36 smithi076 ceph-mon[34152]: pgmap v447: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:36.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:36 smithi189 ceph-mon[34842]: pgmap v447: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:36.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.721+0000 7f7c22a00700 1 -- 172.21.15.27:0/2452298989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c103550 msgr2=0x7f7c1c1039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:36.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.721+0000 7f7c22a00700 1 --2- 172.21.15.27:0/2452298989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c103550 0x7f7c1c1039b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009990 tx=0x7f7c08009ca0 comp rx=0 tx=0).stop 2024-05-10T11:11:36.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.721+0000 7f7c22a00700 1 -- 172.21.15.27:0/2452298989 shutdown_connections 2024-05-10T11:11:36.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.721+0000 7f7c22a00700 1 --2- 172.21.15.27:0/2452298989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c103ef0 0x7f7c1c1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:36.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.721+0000 7f7c22a00700 1 --2- 172.21.15.27:0/2452298989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c108460 0x7f7c1c108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:36.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.721+0000 7f7c22a00700 1 --2- 172.21.15.27:0/2452298989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c103550 0x7f7c1c1039b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:36.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.721+0000 7f7c22a00700 1 -- 172.21.15.27:0/2452298989 >> 172.21.15.27:0/2452298989 conn(0x7f7c1c07a600 msgr2=0x7f7c1c07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:36.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.721+0000 7f7c22a00700 1 -- 172.21.15.27:0/2452298989 shutdown_connections 2024-05-10T11:11:36.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.721+0000 7f7c22a00700 1 -- 172.21.15.27:0/2452298989 wait complete. 2024-05-10T11:11:36.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.722+0000 7f7c22a00700 1 Processor -- start 2024-05-10T11:11:36.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.723+0000 7f7c22a00700 1 -- start start 2024-05-10T11:11:36.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.723+0000 7f7c22a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c103550 0x7f7c1c19bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:36.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.723+0000 7f7c22a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c103ef0 0x7f7c1c19c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:36.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.723+0000 7f7c22a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c108460 0x7f7c1c1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:36.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.723+0000 7f7c22a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1136e0 con 0x7f7c1c103550 2024-05-10T11:11:36.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.723+0000 7f7c22a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c113560 con 0x7f7c1c103ef0 2024-05-10T11:11:36.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.723+0000 7f7c22a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c113860 con 0x7f7c1c108460 2024-05-10T11:11:36.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.723+0000 7f7c1be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c103550 0x7f7c1c19bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:36.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.724+0000 7f7c1be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c103550 0x7f7c1c19bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55808/0 (socket says 172.21.15.27:55808) 2024-05-10T11:11:36.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.724+0000 7f7c1be00700 1 -- 172.21.15.27:0/934334238 learned_addr learned my addr 172.21.15.27:0/934334238 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:36.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.724+0000 7f7c20a00700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c108460 0x7f7c1c1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:36.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.724+0000 7f7c1b400700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c103ef0 0x7f7c1c19c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:36.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.724+0000 7f7c1be00700 1 -- 172.21.15.27:0/934334238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c108460 msgr2=0x7f7c1c1a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:36.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.724+0000 7f7c1be00700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c108460 0x7f7c1c1a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:36.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.724+0000 7f7c1be00700 1 -- 172.21.15.27:0/934334238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c103ef0 msgr2=0x7f7c1c19c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:36.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.724+0000 7f7c1be00700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c103ef0 0x7f7c1c19c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:36.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.724+0000 7f7c1be00700 1 -- 172.21.15.27:0/934334238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1c1a0de0 con 0x7f7c1c103550 2024-05-10T11:11:36.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.724+0000 7f7c20a00700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c108460 0x7f7c1c1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:36.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.725+0000 7f7c1be00700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c103550 0x7f7c1c19bdd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00e970 tx=0x7f7c0c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:36.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.725+0000 7f7c18c00700 1 -- 172.21.15.27:0/934334238 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c004220 con 0x7f7c1c103550 2024-05-10T11:11:36.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.725+0000 7f7c18c00700 1 -- 172.21.15.27:0/934334238 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c0c0043c0 con 0x7f7c1c103550 2024-05-10T11:11:36.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.725+0000 7f7c18c00700 1 -- 172.21.15.27:0/934334238 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c003ce0 con 0x7f7c1c103550 2024-05-10T11:11:36.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.725+0000 7f7c22a00700 1 -- 172.21.15.27:0/934334238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1c1a10d0 con 0x7f7c1c103550 2024-05-10T11:11:36.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.725+0000 7f7c22a00700 1 -- 172.21.15.27:0/934334238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1c1a8950 con 0x7f7c1c103550 2024-05-10T11:11:36.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.726+0000 7f7c22a00700 1 -- 172.21.15.27:0/934334238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c1c06a190 con 0x7f7c1c103550 2024-05-10T11:11:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.729+0000 7f7c18c00700 1 -- 172.21.15.27:0/934334238 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f7c0c00c7e0 con 0x7f7c1c103550 2024-05-10T11:11:36.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.732+0000 7f7c18c00700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7c000b2880 0x7f7c000b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:36.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.732+0000 7f7c18c00700 1 -- 172.21.15.27:0/934334238 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f7c0c0e2990 con 0x7f7c1c103550 2024-05-10T11:11:36.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.732+0000 7f7c1b400700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7c000b2880 0x7f7c000b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:36.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.733+0000 7f7c1b400700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7c000b2880 0x7f7c000b4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c103390 tx=0x7f7c08005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:36.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.735+0000 7f7c18c00700 1 -- 172.21.15.27:0/934334238 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7c0c0aa260 con 0x7f7c1c103550 2024-05-10T11:11:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.988+0000 7f7c22a00700 1 -- 172.21.15.27:0/934334238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c1c19d130 con 0x7f7c1c103550 2024-05-10T11:11:36.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.991+0000 7f7c18c00700 1 -- 172.21.15.27:0/934334238 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24835 (secure 0 0 0) 0x7f7c0c0a9b60 con 0x7f7c1c103550 2024-05-10T11:11:36.991 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:11:36.991 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":90,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:09:23.055559+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6},{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:11:36.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.995+0000 7f7c22a00700 1 -- 172.21.15.27:0/934334238 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7c000b2880 msgr2=0x7f7c000b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:36.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.995+0000 7f7c22a00700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7c000b2880 0x7f7c000b4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c103390 tx=0x7f7c08005fd0 comp rx=0 tx=0).stop 2024-05-10T11:11:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.995+0000 7f7c22a00700 1 -- 172.21.15.27:0/934334238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c103550 msgr2=0x7f7c1c19bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.995+0000 7f7c22a00700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c103550 0x7f7c1c19bdd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00e970 tx=0x7f7c0c00ec80 comp rx=0 tx=0).stop 2024-05-10T11:11:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.995+0000 7f7c22a00700 1 -- 172.21.15.27:0/934334238 shutdown_connections 2024-05-10T11:11:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.995+0000 7f7c22a00700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f7c000b2880 0x7f7c000b4d40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.995+0000 7f7c22a00700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c1c108460 0x7f7c1c1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.995+0000 7f7c22a00700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c1c103550 0x7f7c1c19bdd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.995+0000 7f7c22a00700 1 --2- 172.21.15.27:0/934334238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c1c103ef0 0x7f7c1c19c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.995+0000 7f7c22a00700 1 -- 172.21.15.27:0/934334238 >> 172.21.15.27:0/934334238 conn(0x7f7c1c07a600 msgr2=0x7f7c1c1013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:36.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.996+0000 7f7c22a00700 1 -- 172.21.15.27:0/934334238 shutdown_connections 2024-05-10T11:11:36.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:36.996+0000 7f7c22a00700 1 -- 172.21.15.27:0/934334238 wait complete. 2024-05-10T11:11:37.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:37 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/934334238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:11:37.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:37 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/934334238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:11:37.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:37 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/934334238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:11:37.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:11:38.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:38 smithi027 ceph-mon[30054]: pgmap v448: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:38.371 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:38 smithi076 ceph-mon[34152]: pgmap v448: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:38.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:38 smithi189 ceph-mon[34842]: pgmap v448: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:40.268 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:40 smithi027 ceph-mon[30054]: pgmap v449: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:40 smithi076 ceph-mon[34152]: pgmap v449: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:40.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:40 smithi189 ceph-mon[34842]: pgmap v449: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.827+0000 7ff74ee00700 1 -- 172.21.15.27:0/1212788998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff748105ee0 msgr2=0x7ff748106370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.827+0000 7ff74ee00700 1 --2- 172.21.15.27:0/1212788998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff748105ee0 0x7ff748106370 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff74000b0a0 tx=0x7ff74000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:11:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.827+0000 7ff74ee00700 1 -- 172.21.15.27:0/1212788998 shutdown_connections 2024-05-10T11:11:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.827+0000 7ff74ee00700 1 --2- 172.21.15.27:0/1212788998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff748105ee0 0x7ff748106370 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.827+0000 7ff74ee00700 1 --2- 172.21.15.27:0/1212788998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7481083d0 0x7ff74806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.827+0000 7ff74ee00700 1 --2- 172.21.15.27:0/1212788998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff74806b6f0 0x7ff74806bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.827+0000 7ff74ee00700 1 -- 172.21.15.27:0/1212788998 >> 172.21.15.27:0/1212788998 conn(0x7ff748079350 msgr2=0x7ff748079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.827+0000 7ff74ee00700 1 -- 172.21.15.27:0/1212788998 shutdown_connections 2024-05-10T11:11:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.827+0000 7ff74ee00700 1 -- 172.21.15.27:0/1212788998 wait complete. 2024-05-10T11:11:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.828+0000 7ff74ee00700 1 Processor -- start 2024-05-10T11:11:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.829+0000 7ff74ee00700 1 -- start start 2024-05-10T11:11:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.829+0000 7ff74ee00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff74806b6f0 0x7ff7481a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.829+0000 7ff74ee00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff748105ee0 0x7ff7481a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.829+0000 7ff74ee00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff7481083d0 0x7ff7481a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.829+0000 7ff74ee00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff748119c80 con 0x7ff748105ee0 2024-05-10T11:11:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.829+0000 7ff74ee00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff748119b00 con 0x7ff7481083d0 2024-05-10T11:11:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.829+0000 7ff74ee00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff748119e00 con 0x7ff74806b6f0 2024-05-10T11:11:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.829+0000 7ff747e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff74806b6f0 0x7ff7481a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.830+0000 7ff73f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff748105ee0 0x7ff7481a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.830+0000 7ff747e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff74806b6f0 0x7ff7481a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:47046/0 (socket says 172.21.15.27:47046) 2024-05-10T11:11:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.830+0000 7ff747e00700 1 -- 172.21.15.27:0/594258046 learned_addr learned my addr 172.21.15.27:0/594258046 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.830+0000 7ff74ce00700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff7481083d0 0x7ff7481a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.830+0000 7ff747e00700 1 -- 172.21.15.27:0/594258046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff7481083d0 msgr2=0x7ff7481a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.830+0000 7ff747e00700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff7481083d0 0x7ff7481a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:40.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.830+0000 7ff747e00700 1 -- 172.21.15.27:0/594258046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff748105ee0 msgr2=0x7ff7481a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:40.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.830+0000 7ff747e00700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff748105ee0 0x7ff7481a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:40.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.830+0000 7ff747e00700 1 -- 172.21.15.27:0/594258046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7481a51e0 con 0x7ff74806b6f0 2024-05-10T11:11:40.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.830+0000 7ff74ce00700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff7481083d0 0x7ff7481a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:11:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.830+0000 7ff747e00700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff74806b6f0 0x7ff7481a01d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff73800e9a0 tx=0x7ff73800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.831+0000 7ff745600700 1 -- 172.21.15.27:0/594258046 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73800cbd0 con 0x7ff74806b6f0 2024-05-10T11:11:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.831+0000 7ff745600700 1 -- 172.21.15.27:0/594258046 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff73800cd70 con 0x7ff74806b6f0 2024-05-10T11:11:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.831+0000 7ff74ee00700 1 -- 172.21.15.27:0/594258046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7481a54d0 con 0x7ff74806b6f0 2024-05-10T11:11:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.831+0000 7ff745600700 1 -- 172.21.15.27:0/594258046 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff738018680 con 0x7ff74806b6f0 2024-05-10T11:11:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.831+0000 7ff74ee00700 1 -- 172.21.15.27:0/594258046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7481acd50 con 0x7ff74806b6f0 2024-05-10T11:11:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.832+0000 7ff74ee00700 1 -- 172.21.15.27:0/594258046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff74806a190 con 0x7ff74806b6f0 2024-05-10T11:11:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.835+0000 7ff745600700 1 -- 172.21.15.27:0/594258046 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7ff738010430 con 0x7ff74806b6f0 2024-05-10T11:11:40.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.836+0000 7ff745600700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff72c0b2900 0x7ff72c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:40.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.836+0000 7ff745600700 1 -- 172.21.15.27:0/594258046 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7ff7380e2ca0 con 0x7ff74806b6f0 2024-05-10T11:11:40.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.836+0000 7ff73f400700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff72c0b2900 0x7ff72c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:40.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.837+0000 7ff73f400700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff72c0b2900 0x7ff72c0b4dc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff74806b4c0 tx=0x7ff730009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:40.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:40.838+0000 7ff745600700 1 -- 172.21.15.27:0/594258046 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ff7380aa570 con 0x7ff74806b6f0 2024-05-10T11:11:41.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.129+0000 7ff74ee00700 1 -- 172.21.15.27:0/594258046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7481acf80 con 0x7ff74806b6f0 2024-05-10T11:11:41.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.130+0000 7ff745600700 1 -- 172.21.15.27:0/594258046 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11936 (secure 0 0 0) 0x7ff7380a9e70 con 0x7ff74806b6f0 2024-05-10T11:11:41.130 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:11:41.130 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:09:19.986922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:11:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.134+0000 7ff74ee00700 1 -- 172.21.15.27:0/594258046 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff72c0b2900 msgr2=0x7ff72c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.134+0000 7ff74ee00700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff72c0b2900 0x7ff72c0b4dc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff74806b4c0 tx=0x7ff730009260 comp rx=0 tx=0).stop 2024-05-10T11:11:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.134+0000 7ff74ee00700 1 -- 172.21.15.27:0/594258046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff74806b6f0 msgr2=0x7ff7481a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.134+0000 7ff74ee00700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff74806b6f0 0x7ff7481a01d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff73800e9a0 tx=0x7ff73800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:11:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.134+0000 7ff74ee00700 1 -- 172.21.15.27:0/594258046 shutdown_connections 2024-05-10T11:11:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.134+0000 7ff74ee00700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff72c0b2900 0x7ff72c0b4dc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.134+0000 7ff74ee00700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff74806b6f0 0x7ff7481a01d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.134+0000 7ff74ee00700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff7481083d0 0x7ff7481a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.134+0000 7ff74ee00700 1 --2- 172.21.15.27:0/594258046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff748105ee0 0x7ff7481a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.134+0000 7ff74ee00700 1 -- 172.21.15.27:0/594258046 >> 172.21.15.27:0/594258046 conn(0x7ff748079350 msgr2=0x7ff748104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.134+0000 7ff74ee00700 1 -- 172.21.15.27:0/594258046 shutdown_connections 2024-05-10T11:11:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:41.134+0000 7ff74ee00700 1 -- 172.21.15.27:0/594258046 wait complete. 2024-05-10T11:11:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-05-10T11:11:42.009 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-10T11:07:50.237787+0000', 'modified': '2024-05-10T11:09:19.986922+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24527, 'mds_1': 24488, 'mds_2': 24484, 'mds_3': 24472, 'mds_4': 24467}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14403': {'gid': 14403, 'name': 'a', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6841/3080402649', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6840', 'nonce': 3080402649}, {'type': 'v1', 'addr': '172.21.15.27:6841', 'nonce': 3080402649}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24418': {'gid': 24418, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6833/2718624698', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6832', 'nonce': 2718624698}, {'type': 'v1', 'addr': '172.21.15.189:6833', 'nonce': 2718624698}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24437': {'gid': 24437, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.76:6833/1370063583', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 1370063583}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 1370063583}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24449': {'gid': 24449, 'name': 'j', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6835/3274866857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6834', 'nonce': 3274866857}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 3274866857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6835/1387617890', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 1387617890}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 1387617890}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'e', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.76:6835/3618511593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6834', 'nonce': 3618511593}, {'type': 'v1', 'addr': '172.21.15.76:6835', 'nonce': 3618511593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 33, 'addr': '172.21.15.189:6837/3185830081', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 3185830081}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 3185830081}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.189:6839/2071082194', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 2071082194}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 2071082194}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 32, 'addr': '172.21.15.76:6837/569330029', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6836', 'nonce': 569330029}, {'type': 'v1', 'addr': '172.21.15.76:6837', 'nonce': 569330029}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.76:6839/1778204420', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6838', 'nonce': 1778204420}, {'type': 'v1', 'addr': '172.21.15.76:6839', 'nonce': 1778204420}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24527, 'qdb_cluster': [24467, 24488, 24472, 24484, 24527]} 2024-05-10T11:11:42.010 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-10T11:11:42.010 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.e status 2024-05-10T11:11:42.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:42 smithi027 ceph-mon[30054]: pgmap v450: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:42.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:42 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/594258046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:11:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:42 smithi076 ceph-mon[34152]: pgmap v450: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:42 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/594258046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:11:42.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:42 smithi189 ceph-mon[34842]: pgmap v450: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:42.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:42 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/594258046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:11:42.494 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:44.267 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:44 smithi027 ceph-mon[30054]: pgmap v451: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:44.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:44 smithi076 ceph-mon[34152]: pgmap v451: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:44.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:44 smithi189 ceph-mon[34842]: pgmap v451: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:44.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.849+0000 7f762a600700 1 -- 172.21.15.27:0/3959798298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762406b180 msgr2=0x7f762406b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:44.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.849+0000 7f762a600700 1 --2- 172.21.15.27:0/3959798298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762406b180 0x7f762406b600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f760c009990 tx=0x7f760c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.849+0000 7f762a600700 1 -- 172.21.15.27:0/3959798298 shutdown_connections 2024-05-10T11:11:44.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.849+0000 7f762a600700 1 --2- 172.21.15.27:0/3959798298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7624109510 0x7f76241098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.849+0000 7f762a600700 1 --2- 172.21.15.27:0/3959798298 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f762406bb40 0x7f7624110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.849+0000 7f762a600700 1 --2- 172.21.15.27:0/3959798298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762406b180 0x7f762406b600 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.849+0000 7f762a600700 1 -- 172.21.15.27:0/3959798298 >> 172.21.15.27:0/3959798298 conn(0x7f7624079350 msgr2=0x7f7624079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:44.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.849+0000 7f762a600700 1 -- 172.21.15.27:0/3959798298 shutdown_connections 2024-05-10T11:11:44.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.849+0000 7f762a600700 1 -- 172.21.15.27:0/3959798298 wait complete. 2024-05-10T11:11:44.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.850+0000 7f762a600700 1 Processor -- start 2024-05-10T11:11:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.850+0000 7f762a600700 1 -- start start 2024-05-10T11:11:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.851+0000 7f762a600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762406b180 0x7f762419bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.851+0000 7f762a600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f762406bb40 0x7f762419c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.851+0000 7f762a600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7624109510 0x7f76241a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.851+0000 7f762a600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7624113670 con 0x7f762406b180 2024-05-10T11:11:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.851+0000 7f762a600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f76241134f0 con 0x7f7624109510 2024-05-10T11:11:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.851+0000 7f762a600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f76241137f0 con 0x7f762406bb40 2024-05-10T11:11:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.851+0000 7f7623400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762406b180 0x7f762419bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.851+0000 7f7623400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762406b180 0x7f762419bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55926/0 (socket says 172.21.15.27:55926) 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.851+0000 7f7623400700 1 -- 172.21.15.27:0/1546983514 learned_addr learned my addr 172.21.15.27:0/1546983514 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.851+0000 7f7623e00700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7624109510 0x7f76241a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.851+0000 7f761aa00700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f762406bb40 0x7f762419c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.852+0000 7f7623400700 1 -- 172.21.15.27:0/1546983514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f762406bb40 msgr2=0x7f762419c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.852+0000 7f7623400700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f762406bb40 0x7f762419c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.852+0000 7f7623400700 1 -- 172.21.15.27:0/1546983514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7624109510 msgr2=0x7f76241a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.852+0000 7f7623400700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7624109510 0x7f76241a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.852+0000 7f7623400700 1 -- 172.21.15.27:0/1546983514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76241a0dd0 con 0x7f762406b180 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.852+0000 7f7623400700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762406b180 0x7f762419bdc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f761400e9a0 tx=0x7f761400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.852+0000 7f7620c00700 1 -- 172.21.15.27:0/1546983514 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761400ca10 con 0x7f762406b180 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.852+0000 7f7620c00700 1 -- 172.21.15.27:0/1546983514 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f761400cbb0 con 0x7f762406b180 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.852+0000 7f7620c00700 1 -- 172.21.15.27:0/1546983514 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7614018680 con 0x7f762406b180 2024-05-10T11:11:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.853+0000 7f762a600700 1 -- 172.21.15.27:0/1546983514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76241a10c0 con 0x7f762406b180 2024-05-10T11:11:44.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.853+0000 7f762a600700 1 -- 172.21.15.27:0/1546983514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76241a8940 con 0x7f762406b180 2024-05-10T11:11:44.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.856+0000 7f7620c00700 1 -- 172.21.15.27:0/1546983514 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f7614010430 con 0x7f762406b180 2024-05-10T11:11:44.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.858+0000 7f7634fcfb80 1 Processor -- start 2024-05-10T11:11:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.858+0000 7f7634fcfb80 1 -- start start 2024-05-10T11:11:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.858+0000 7f7620c00700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76080b2900 0x7f76080b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.858+0000 7f7634fcfb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 0x561f63fde3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.858+0000 7f7634fcfb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fe2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.858+0000 7f7634fcfb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fe2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.858+0000 7f7634fcfb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561f63f309d0 con 0x561f63fde930 2024-05-10T11:11:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.858+0000 7f7634fcfb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561f63ece710 con 0x561f63fde010 2024-05-10T11:11:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.858+0000 7f7634fcfb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561f63f2c990 con 0x561f63fe2980 2024-05-10T11:11:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.858+0000 7f7620c00700 1 -- 172.21.15.27:0/1546983514 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f76140e26e0 con 0x7f762406b180 2024-05-10T11:11:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.859+0000 7f761aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fe2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.859+0000 7f7623400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 0x561f63fde3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.859+0000 7f7623e00700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76080b2900 0x7f76080b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.859+0000 7f7623400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 0x561f63fde3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35684/0 (socket says 172.21.15.27:35684) 2024-05-10T11:11:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.859+0000 7f7623400700 1 -- 172.21.15.27:0/1937708787 learned_addr learned my addr 172.21.15.27:0/1937708787 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.859+0000 7f7623e00700 1 --2- 172.21.15.27:0/1937708787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fe2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.859+0000 7f7623400700 1 -- 172.21.15.27:0/1937708787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 msgr2=0x561f63fe2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.859+0000 7f7623400700 1 --2- 172.21.15.27:0/1937708787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fe2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.859+0000 7f7623400700 1 -- 172.21.15.27:0/1937708787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 msgr2=0x561f63fe2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.859+0000 7f7623400700 1 --2- 172.21.15.27:0/1937708787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fe2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.859+0000 7f7623400700 1 -- 172.21.15.27:0/1937708787 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f63e7d880 con 0x561f63fde010 2024-05-10T11:11:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.860+0000 7f7623400700 1 --2- 172.21.15.27:0/1937708787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 0x561f63fde3f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7614009230 tx=0x7f76140a90a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c74f3a09a79d002d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.860+0000 7f7623e00700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76080b2900 0x7f76080b4dc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f761c009ec0 tx=0x7f761c009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.860+0000 7f7618c00700 1 -- 172.21.15.27:0/1937708787 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76140a9fb0 con 0x561f63fde010 2024-05-10T11:11:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.860+0000 7f7618c00700 1 -- 172.21.15.27:0/1937708787 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76140a98b0 con 0x561f63fde010 2024-05-10T11:11:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.860+0000 7f7618c00700 1 -- 172.21.15.27:0/1937708787 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76140a9b10 con 0x561f63fde010 2024-05-10T11:11:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.861+0000 7f7634fcfb80 1 -- 172.21.15.27:0/1937708787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 msgr2=0x561f63fde3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.861+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/1937708787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 0x561f63fde3f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7614009230 tx=0x7f76140a90a0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.861+0000 7f7634fcfb80 1 -- 172.21.15.27:0/1937708787 shutdown_connections 2024-05-10T11:11:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.861+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/1937708787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fe2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.861+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/1937708787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 0x561f63fde3f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.861+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/1937708787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fe2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.861+0000 7f7634fcfb80 1 -- 172.21.15.27:0/1937708787 >> 172.21.15.27:0/1937708787 conn(0x561f63f0cb10 msgr2=0x561f63fd8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.862+0000 7f7634fcfb80 1 -- 172.21.15.27:0/1937708787 shutdown_connections 2024-05-10T11:11:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.862+0000 7f7634fcfb80 1 -- 172.21.15.27:0/1937708787 wait complete. 2024-05-10T11:11:44.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.862+0000 7f7634fcfb80 1 -- start start 2024-05-10T11:11:44.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.862+0000 7f7634fcfb80 1 Processor -- start 2024-05-10T11:11:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.863+0000 7f7634fcfb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fde010 0x561f63fe9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.863+0000 7f7634fcfb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fe7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.863+0000 7f7634fcfb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fe2980 0x561f63fe7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.863+0000 7f7634fcfb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561f63feab90 con 0x561f63fde930 2024-05-10T11:11:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.863+0000 7f7634fcfb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561f63e96ad0 con 0x561f63fe2980 2024-05-10T11:11:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.863+0000 7f7634fcfb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561f63fead10 con 0x561f63fde010 2024-05-10T11:11:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.863+0000 7f7634fcfb80 10 client.0 _ll_register_callbacks cb 0x561f63ed7c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:11:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.863+0000 7f7623400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fde010 0x561f63fe9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.863+0000 7f7623400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fde010 0x561f63fe9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:46346/0 (socket says 172.21.15.27:46346) 2024-05-10T11:11:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.863+0000 7f7623400700 1 -- 172.21.15.27:0/68815592 learned_addr learned my addr 172.21.15.27:0/68815592 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.863+0000 7f761aa00700 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fe7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.863+0000 7f7623e00700 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fe2980 0x561f63fe7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.864+0000 7f7623400700 1 -- 172.21.15.27:0/68815592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fe2980 msgr2=0x561f63fe7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.864+0000 7f7623400700 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fe2980 0x561f63fe7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.864+0000 7f7623400700 1 -- 172.21.15.27:0/68815592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 msgr2=0x561f63fe7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.864+0000 7f7623400700 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fe7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.864+0000 7f7623400700 1 -- 172.21.15.27:0/68815592 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f63ef63d0 con 0x561f63fde010 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.864+0000 7f7623e00700 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fe2980 0x561f63fe7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.864+0000 7f7623400700 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fde010 0x561f63fe9e10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f76140a5930 tx=0x7f76140adc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9eec7b2df36693db server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.864+0000 7f7611600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.864+0000 7f7611600700 1 -- 172.21.15.27:0/68815592 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76140a9b10 con 0x561f63fde010 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.864+0000 7f7611600700 1 -- 172.21.15.27:0/68815592 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f76140a9cb0 con 0x561f63fde010 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.865+0000 7f7634fcfb80 1 -- 172.21.15.27:0/68815592 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76140a98b0 con 0x561f63fde010 2024-05-10T11:11:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.865+0000 7f7611600700 1 -- 172.21.15.27:0/68815592 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f76140b2280 con 0x561f63fde010 2024-05-10T11:11:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.865+0000 7f7634fcfb80 10 client.24955 fetch_fsmap learned FSMap version 29 2024-05-10T11:11:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.865+0000 7f7634fcfb80 1 -- 172.21.15.27:0/68815592 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561f63feab90 con 0x561f63fde010 2024-05-10T11:11:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.865+0000 7f7611600700 1 -- 172.21.15.27:0/68815592 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76140a98b0 con 0x561f63fde010 2024-05-10T11:11:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.865+0000 7f7611600700 1 -- 172.21.15.27:0/68815592 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f76140e2030 con 0x561f63fde010 2024-05-10T11:11:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.865+0000 7f7634fcfb80 10 client.24955 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:11:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.865+0000 7f7634fcfb80 10 client.24955 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-10T11:11:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.865+0000 7f7634fcfb80 20 client.24955 populate_metadata read hostname 'smithi027' 2024-05-10T11:11:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.865+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x561f63ff7450 0x561f63ff88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.865+0000 7f7634fcfb80 4 client.24955 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:11:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.865+0000 7f7634fcfb80 1 -- 172.21.15.27:0/68815592 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f63fab050 con 0x561f63ff7450 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.866+0000 7f761aa00700 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x561f63ff7450 0x561f63ff88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.866+0000 7f761aa00700 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x561f63ff7450 0x561f63ff88c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1343f209d67f6a4e server_cookie=add62434d47055d3 in_seq=0 out_seq=0 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.866+0000 7f7611600700 10 client.24955 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.867+0000 7f7611600700 1 -- 172.21.15.27:0/68815592 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561f63fab050 con 0x561f63ff7450 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.867+0000 7f7611600700 10 client.24955 handle_command_reply: tid=0 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.867+0000 7f7634fcfb80 1 client.24955 shutdown 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.868+0000 7f7634fcfb80 1 -- 172.21.15.27:0/68815592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fde010 msgr2=0x561f63fe9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.868+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fde010 0x561f63fe9e10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f76140a5930 tx=0x7f76140adc60 comp rx=0 tx=0).stop 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.868+0000 7f7634fcfb80 1 -- 172.21.15.27:0/68815592 shutdown_connections 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.868+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x561f63ff7450 0x561f63ff88c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.868+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fde010 0x561f63fe9e10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.868+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fe2980 0x561f63fe7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.868+0000 7f7611600700 0 client.24955 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:11:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.868+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/68815592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fe7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.868+0000 7f7634fcfb80 1 -- 172.21.15.27:0/68815592 >> 172.21.15.27:0/68815592 conn(0x561f63f0cb10 msgr2=0x561f63fd3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.868+0000 7f7634fcfb80 1 -- 172.21.15.27:0/68815592 shutdown_connections 2024-05-10T11:11:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.868+0000 7f7634fcfb80 1 -- 172.21.15.27:0/68815592 wait complete. 2024-05-10T11:11:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.868+0000 7f7634fcfb80 20 client.24955 trim_cache size 0 max 16384 2024-05-10T11:11:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.892+0000 7f7634fcfb80 1 Processor -- start 2024-05-10T11:11:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.893+0000 7f7634fcfb80 1 -- start start 2024-05-10T11:11:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.893+0000 7f7634fcfb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 0x561f63fd29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.893+0000 7f7634fcfb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fd2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.893+0000 7f7634fcfb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fd3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.893+0000 7f7634fcfb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561f63ff6c00 con 0x561f63fde930 2024-05-10T11:11:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.893+0000 7f7634fcfb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561f63ff72d0 con 0x561f63fde010 2024-05-10T11:11:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.893+0000 7f7634fcfb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561f63ff79a0 con 0x561f63fe2980 2024-05-10T11:11:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.893+0000 7f7623e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fd3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.893+0000 7f7623e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fd3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:46360/0 (socket says 172.21.15.27:46360) 2024-05-10T11:11:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.893+0000 7f7623e00700 1 -- 172.21.15.27:0/1738595524 learned_addr learned my addr 172.21.15.27:0/1738595524 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.894+0000 7f7623400700 1 --2- 172.21.15.27:0/1738595524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 0x561f63fd29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.894+0000 7f761aa00700 1 --2- 172.21.15.27:0/1738595524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fd2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.894+0000 7f7623400700 1 -- 172.21.15.27:0/1738595524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 msgr2=0x561f63fd3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.894+0000 7f7623400700 1 --2- 172.21.15.27:0/1738595524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fd3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.894+0000 7f7623400700 1 -- 172.21.15.27:0/1738595524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 msgr2=0x561f63fd2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.894+0000 7f7623400700 1 --2- 172.21.15.27:0/1738595524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fd2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.894+0000 7f7623400700 1 -- 172.21.15.27:0/1738595524 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f63fd95c0 con 0x561f63fde010 2024-05-10T11:11:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.894+0000 7f761aa00700 1 --2- 172.21.15.27:0/1738595524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fd2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:11:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.895+0000 7f7623400700 1 --2- 172.21.15.27:0/1738595524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 0x561f63fd29d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f761400e970 tx=0x7f7614009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=799da354522738eb server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.895+0000 7f7613e00700 1 -- 172.21.15.27:0/1738595524 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76140b2710 con 0x561f63fde010 2024-05-10T11:11:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.895+0000 7f7613e00700 1 -- 172.21.15.27:0/1738595524 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76140b2420 con 0x561f63fde010 2024-05-10T11:11:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.895+0000 7f7613e00700 1 -- 172.21.15.27:0/1738595524 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76140a9c00 con 0x561f63fde010 2024-05-10T11:11:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.896+0000 7f7634fcfb80 1 -- 172.21.15.27:0/1738595524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 msgr2=0x561f63fd29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.896+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/1738595524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 0x561f63fd29d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f761400e970 tx=0x7f7614009820 comp rx=0 tx=0).stop 2024-05-10T11:11:44.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.896+0000 7f7634fcfb80 1 -- 172.21.15.27:0/1738595524 shutdown_connections 2024-05-10T11:11:44.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.896+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/1738595524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fd3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.896+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/1738595524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde010 0x561f63fd29d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.896+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/1738595524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde930 0x561f63fd2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.896+0000 7f7634fcfb80 1 -- 172.21.15.27:0/1738595524 >> 172.21.15.27:0/1738595524 conn(0x561f63f0cb10 msgr2=0x561f63fd2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:44.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.896+0000 7f7634fcfb80 1 -- 172.21.15.27:0/1738595524 shutdown_connections 2024-05-10T11:11:44.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.896+0000 7f7634fcfb80 1 -- 172.21.15.27:0/1738595524 wait complete. 2024-05-10T11:11:44.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.897+0000 7f7634fcfb80 1 -- start start 2024-05-10T11:11:44.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.897+0000 7f7634fcfb80 1 Processor -- start 2024-05-10T11:11:44.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.898+0000 7f7634fcfb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde010 0x561f63fe3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.898+0000 7f7634fcfb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde930 0x561f63fe44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.898+0000 7f7634fcfb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fe4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.898+0000 7f7634fcfb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561f63ff8740 con 0x561f63fde010 2024-05-10T11:11:44.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.898+0000 7f7634fcfb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561f63ff8070 con 0x561f63fde930 2024-05-10T11:11:44.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.898+0000 7f7634fcfb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561f63ff8e10 con 0x561f63fe2980 2024-05-10T11:11:44.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.898+0000 7f7634fcfb80 10 client.0 _ll_register_callbacks cb 0x561f63ed7c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:11:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.898+0000 7f7623400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde010 0x561f63fe3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.898+0000 7f7623e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fe4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.898+0000 7f7623400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde010 0x561f63fe3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55958/0 (socket says 172.21.15.27:55958) 2024-05-10T11:11:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.898+0000 7f7623400700 1 -- 172.21.15.27:0/3891340579 learned_addr learned my addr 172.21.15.27:0/3891340579 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.898+0000 7f761aa00700 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde930 0x561f63fe44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.899+0000 7f7623400700 1 -- 172.21.15.27:0/3891340579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 msgr2=0x561f63fe4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.899+0000 7f7623400700 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fe4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.899+0000 7f7623400700 1 -- 172.21.15.27:0/3891340579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde930 msgr2=0x561f63fe44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.899+0000 7f7623400700 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde930 0x561f63fe44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.899+0000 7f7623400700 1 -- 172.21.15.27:0/3891340579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f63fffff0 con 0x561f63fde010 2024-05-10T11:11:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.899+0000 7f7623e00700 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fe4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.899+0000 7f7623400700 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde010 0x561f63fe3f80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f761400c2d0 tx=0x7f7614066a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e88a692c0bec70f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:44.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.899+0000 7f7611600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:11:44.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.899+0000 7f7611600700 1 -- 172.21.15.27:0/3891340579 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76140b2710 con 0x561f63fde010 2024-05-10T11:11:44.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.900+0000 7f7611600700 1 -- 172.21.15.27:0/3891340579 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f76140a5ea0 con 0x561f63fde010 2024-05-10T11:11:44.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.900+0000 7f7634fcfb80 1 -- 172.21.15.27:0/3891340579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76140b2420 con 0x561f63fde010 2024-05-10T11:11:44.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.900+0000 7f7611600700 1 -- 172.21.15.27:0/3891340579 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7614010980 con 0x561f63fde010 2024-05-10T11:11:44.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.900+0000 7f7634fcfb80 10 client.14853 fetch_fsmap learned FSMap version 29 2024-05-10T11:11:44.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.900+0000 7f7634fcfb80 1 -- 172.21.15.27:0/3891340579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561f63fffff0 con 0x561f63fde010 2024-05-10T11:11:44.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.901+0000 7f7611600700 1 -- 172.21.15.27:0/3891340579 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76140b2420 con 0x561f63fde010 2024-05-10T11:11:44.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.901+0000 7f7611600700 1 -- 172.21.15.27:0/3891340579 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f76140e2030 con 0x561f63fde010 2024-05-10T11:11:44.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.901+0000 7f7634fcfb80 10 client.14853 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.901+0000 7f7634fcfb80 10 client.14853 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.901+0000 7f7634fcfb80 20 client.14853 populate_metadata read hostname 'smithi027' 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.901+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x561f64066ea0 0x561f64065bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.901+0000 7f7634fcfb80 4 client.14853 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.901+0000 7f7634fcfb80 1 -- 172.21.15.27:0/3891340579 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "status"}) -- 0x561f63f3e3b0 con 0x561f64066ea0 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.901+0000 7f761aa00700 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x561f64066ea0 0x561f64065bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.902+0000 7f761aa00700 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x561f64066ea0 0x561f64065bb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3f475394ca642c47 server_cookie=42f40e60bde8d4ab in_seq=0 out_seq=0 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.902+0000 7f7611600700 10 client.14853 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.902+0000 7f7611600700 1 -- 172.21.15.27:0/3891340579 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561f63f3e3b0 con 0x561f64066ea0 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.902+0000 7f7611600700 10 client.14853 handle_command_reply: tid=0 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.902+0000 7f7634fcfb80 1 client.14853 shutdown 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.903+0000 7f7634fcfb80 1 -- 172.21.15.27:0/3891340579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde010 msgr2=0x561f63fe3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.903+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde010 0x561f63fe3f80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f761400c2d0 tx=0x7f7614066a60 comp rx=0 tx=0).stop 2024-05-10T11:11:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.903+0000 7f7634fcfb80 1 -- 172.21.15.27:0/3891340579 shutdown_connections 2024-05-10T11:11:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.903+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x561f64066ea0 0x561f64065bb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.903+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f63fe2980 0x561f63fe4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.903+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561f63fde010 0x561f63fe3f80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.903+0000 7f7634fcfb80 1 --2- 172.21.15.27:0/3891340579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561f63fde930 0x561f63fe44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.903+0000 7f7634fcfb80 1 -- 172.21.15.27:0/3891340579 >> 172.21.15.27:0/3891340579 conn(0x561f63f0cb10 msgr2=0x561f63fd29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.903+0000 7f7611600700 0 client.14853 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:11:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.903+0000 7f7634fcfb80 1 -- 172.21.15.27:0/3891340579 shutdown_connections 2024-05-10T11:11:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.903+0000 7f7634fcfb80 1 -- 172.21.15.27:0/3891340579 wait complete. 2024-05-10T11:11:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.903+0000 7f7634fcfb80 20 client.14853 trim_cache size 0 max 16384 2024-05-10T11:11:44.905 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:11:44.905 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T11:11:44.905 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 4, 2024-05-10T11:11:44.905 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24467, 2024-05-10T11:11:44.905 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-05-10T11:11:44.905 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-05-10T11:11:44.905 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-05-10T11:11:44.905 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 149.89391138400001, 2024-05-10T11:11:44.905 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 29, 2024-05-10T11:11:44.905 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 90, 2024-05-10T11:11:44.905 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 88, 2024-05-10T11:11:44.906 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 282.46362248299999 2024-05-10T11:11:44.906 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:11:44.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.904+0000 7f762a600700 1 -- 172.21.15.27:0/1546983514 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76080b2900 msgr2=0x7f76080b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:44.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.904+0000 7f762a600700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76080b2900 0x7f76080b4dc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f761c009ec0 tx=0x7f761c009920 comp rx=0 tx=0).stop 2024-05-10T11:11:44.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.904+0000 7f762a600700 1 -- 172.21.15.27:0/1546983514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762406b180 msgr2=0x7f762419bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:44.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.904+0000 7f762a600700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762406b180 0x7f762419bdc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f761400e9a0 tx=0x7f761400ecb0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.904+0000 7f762a600700 1 -- 172.21.15.27:0/1546983514 shutdown_connections 2024-05-10T11:11:44.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.904+0000 7f762a600700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76080b2900 0x7f76080b4dc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.904+0000 7f762a600700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7624109510 0x7f76241a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.904+0000 7f762a600700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762406b180 0x7f762419bdc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.904+0000 7f762a600700 1 --2- 172.21.15.27:0/1546983514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f762406bb40 0x7f762419c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:44.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.904+0000 7f762a600700 1 -- 172.21.15.27:0/1546983514 >> 172.21.15.27:0/1546983514 conn(0x7f7624079350 msgr2=0x7f7624100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:44.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.905+0000 7f762a600700 1 -- 172.21.15.27:0/1546983514 shutdown_connections 2024-05-10T11:11:44.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:44.905+0000 7f762a600700 1 -- 172.21.15.27:0/1546983514 wait complete. 2024-05-10T11:11:45.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.c status 2024-05-10T11:11:46.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:46 smithi027 ceph-mon[30054]: pgmap v452: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:46.390 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:46.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:46 smithi076 ceph-mon[34152]: pgmap v452: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:46.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:46 smithi189 ceph-mon[34842]: pgmap v452: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:48.322 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:48 smithi027 ceph-mon[30054]: pgmap v453: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:48.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:48 smithi076 ceph-mon[34152]: pgmap v453: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:48.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:48 smithi189 ceph-mon[34842]: pgmap v453: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.863+0000 7fe715200700 1 -- 172.21.15.27:0/631813394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe710108850 msgr2=0x7fe7101103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.863+0000 7fe715200700 1 --2- 172.21.15.27:0/631813394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe710108850 0x7fe7101103d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe70800b0a0 tx=0x7fe70800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.863+0000 7fe715200700 1 -- 172.21.15.27:0/631813394 shutdown_connections 2024-05-10T11:11:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.863+0000 7fe715200700 1 --2- 172.21.15.27:0/631813394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe710108850 0x7fe7101103d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.863+0000 7fe715200700 1 --2- 172.21.15.27:0/631813394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe710103750 0x7fe710103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.863+0000 7fe715200700 1 --2- 172.21.15.27:0/631813394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe710104100 0x7fe710108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.863+0000 7fe715200700 1 -- 172.21.15.27:0/631813394 >> 172.21.15.27:0/631813394 conn(0x7fe71007a600 msgr2=0x7fe71007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.864+0000 7fe715200700 1 -- 172.21.15.27:0/631813394 shutdown_connections 2024-05-10T11:11:48.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.864+0000 7fe715200700 1 -- 172.21.15.27:0/631813394 wait complete. 2024-05-10T11:11:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.865+0000 7fe715200700 1 Processor -- start 2024-05-10T11:11:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.865+0000 7fe715200700 1 -- start start 2024-05-10T11:11:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.865+0000 7fe715200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe710103750 0x7fe7101a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.865+0000 7fe715200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe710104100 0x7fe7101a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.865+0000 7fe715200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe710108850 0x7fe71019bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.865+0000 7fe715200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe710113670 con 0x7fe710104100 2024-05-10T11:11:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.865+0000 7fe715200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe7101134f0 con 0x7fe710108850 2024-05-10T11:11:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.865+0000 7fe715200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe7101137f0 con 0x7fe710103750 2024-05-10T11:11:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.866+0000 7fe70de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe710103750 0x7fe7101a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.866+0000 7fe70de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe710103750 0x7fe7101a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:46400/0 (socket says 172.21.15.27:46400) 2024-05-10T11:11:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.866+0000 7fe70de00700 1 -- 172.21.15.27:0/2477582042 learned_addr learned my addr 172.21.15.27:0/2477582042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.866+0000 7fe70d400700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe710104100 0x7fe7101a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.866+0000 7fe70e800700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe710108850 0x7fe71019bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.866+0000 7fe70de00700 1 -- 172.21.15.27:0/2477582042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe710108850 msgr2=0x7fe71019bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.866+0000 7fe70de00700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe710108850 0x7fe71019bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.866+0000 7fe70de00700 1 -- 172.21.15.27:0/2477582042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe710104100 msgr2=0x7fe7101a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.866+0000 7fe70de00700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe710104100 0x7fe7101a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.866+0000 7fe70de00700 1 -- 172.21.15.27:0/2477582042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe71019c220 con 0x7fe710103750 2024-05-10T11:11:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.867+0000 7fe70de00700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe710103750 0x7fe7101a1900 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe70000e9a0 tx=0x7fe70000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.867+0000 7fe706a00700 1 -- 172.21.15.27:0/2477582042 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70000ca10 con 0x7fe710103750 2024-05-10T11:11:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.867+0000 7fe706a00700 1 -- 172.21.15.27:0/2477582042 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe70000cbb0 con 0x7fe710103750 2024-05-10T11:11:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.867+0000 7fe706a00700 1 -- 172.21.15.27:0/2477582042 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe700018680 con 0x7fe710103750 2024-05-10T11:11:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.867+0000 7fe715200700 1 -- 172.21.15.27:0/2477582042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe71019c510 con 0x7fe710103750 2024-05-10T11:11:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.867+0000 7fe715200700 1 -- 172.21.15.27:0/2477582042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7101a87d0 con 0x7fe710103750 2024-05-10T11:11:48.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.871+0000 7fe706a00700 1 -- 172.21.15.27:0/2477582042 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fe700010430 con 0x7fe710103750 2024-05-10T11:11:48.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.872+0000 7fe71fad6b80 1 Processor -- start 2024-05-10T11:11:48.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.872+0000 7fe71fad6b80 1 -- start start 2024-05-10T11:11:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.872+0000 7fe71fad6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f26ab70 0x562a5f26af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.872+0000 7fe71fad6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26b490 0x562a5f265280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.873+0000 7fe71fad6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f2658a0 0x562a5f265d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.873+0000 7fe71fad6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562a5f1b79d0 con 0x562a5f2658a0 2024-05-10T11:11:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.873+0000 7fe71fad6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562a5f155710 con 0x562a5f26b490 2024-05-10T11:11:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.873+0000 7fe71fad6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562a5f1b3990 con 0x562a5f26ab70 2024-05-10T11:11:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.873+0000 7fe706a00700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe6f40b2900 0x7fe6f40b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.873+0000 7fe70e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f26ab70 0x562a5f26af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.873+0000 7fe70e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f26ab70 0x562a5f26af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:46402/0 (socket says 172.21.15.27:46402) 2024-05-10T11:11:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.873+0000 7fe70de00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26b490 0x562a5f265280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.873+0000 7fe70d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f2658a0 0x562a5f265d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.873+0000 7fe70e800700 1 -- 172.21.15.27:0/854774711 learned_addr learned my addr 172.21.15.27:0/854774711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.873+0000 7fe706a00700 1 -- 172.21.15.27:0/2477582042 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fe7000e2750 con 0x7fe710103750 2024-05-10T11:11:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.874+0000 7fe70e800700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe6f40b2900 0x7fe6f40b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.874+0000 7fe70d400700 1 -- 172.21.15.27:0/854774711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f26ab70 msgr2=0x562a5f26af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.874+0000 7fe70d400700 1 --2- 172.21.15.27:0/854774711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f26ab70 0x562a5f26af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.874+0000 7fe70d400700 1 -- 172.21.15.27:0/854774711 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26b490 msgr2=0x562a5f265280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.874+0000 7fe70d400700 1 --2- 172.21.15.27:0/854774711 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26b490 0x562a5f265280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.874+0000 7fe70d400700 1 -- 172.21.15.27:0/854774711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a5f104880 con 0x562a5f2658a0 2024-05-10T11:11:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.874+0000 7fe70d400700 1 --2- 172.21.15.27:0/854774711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f2658a0 0x562a5f265d20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc005950 tx=0x7fe6fc00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5885329bc371cd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.874+0000 7fe70e800700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe6f40b2900 0x7fe6f40b4dc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe708009e60 tx=0x7fe70800b3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.874+0000 7fe6fb400700 1 -- 172.21.15.27:0/854774711 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc010c70 con 0x562a5f2658a0 2024-05-10T11:11:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.875+0000 7fe6fb400700 1 -- 172.21.15.27:0/854774711 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6fc0044e0 con 0x562a5f2658a0 2024-05-10T11:11:48.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.875+0000 7fe6fb400700 1 -- 172.21.15.27:0/854774711 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc005280 con 0x562a5f2658a0 2024-05-10T11:11:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.875+0000 7fe71fad6b80 1 -- 172.21.15.27:0/854774711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f2658a0 msgr2=0x562a5f265d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.875+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/854774711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f2658a0 0x562a5f265d20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc005950 tx=0x7fe6fc00bf60 comp rx=0 tx=0).stop 2024-05-10T11:11:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.875+0000 7fe71fad6b80 1 -- 172.21.15.27:0/854774711 shutdown_connections 2024-05-10T11:11:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.875+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/854774711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f26ab70 0x562a5f26af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.875+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/854774711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f2658a0 0x562a5f265d20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.875+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/854774711 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26b490 0x562a5f265280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.875+0000 7fe71fad6b80 1 -- 172.21.15.27:0/854774711 >> 172.21.15.27:0/854774711 conn(0x562a5f1dd680 msgr2=0x562a5f25fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.876+0000 7fe71fad6b80 1 -- 172.21.15.27:0/854774711 shutdown_connections 2024-05-10T11:11:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.876+0000 7fe71fad6b80 1 -- 172.21.15.27:0/854774711 wait complete. 2024-05-10T11:11:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.876+0000 7fe71fad6b80 1 -- start start 2024-05-10T11:11:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.876+0000 7fe71fad6b80 1 Processor -- start 2024-05-10T11:11:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.877+0000 7fe71fad6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f2658a0 0x562a5f271200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.877+0000 7fe71fad6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 0x562a5f26e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.877+0000 7fe71fad6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26b490 0x562a5f26eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.877+0000 7fe71fad6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562a5f271ed0 con 0x562a5f26ab70 2024-05-10T11:11:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.877+0000 7fe71fad6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562a5f11dad0 con 0x562a5f26b490 2024-05-10T11:11:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.877+0000 7fe71fad6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562a5f272050 con 0x562a5f2658a0 2024-05-10T11:11:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.877+0000 7fe71fad6b80 10 client.0 _ll_register_callbacks cb 0x562a5f15ec20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:11:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.877+0000 7fe70d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 0x562a5f26e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.877+0000 7fe70d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 0x562a5f26e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55984/0 (socket says 172.21.15.27:55984) 2024-05-10T11:11:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.877+0000 7fe70d400700 1 -- 172.21.15.27:0/2279732051 learned_addr learned my addr 172.21.15.27:0/2279732051 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.877+0000 7fe70e800700 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26b490 0x562a5f26eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.877+0000 7fe70de00700 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f2658a0 0x562a5f271200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.878+0000 7fe70d400700 1 -- 172.21.15.27:0/2279732051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f2658a0 msgr2=0x562a5f271200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.878+0000 7fe70d400700 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f2658a0 0x562a5f271200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.878+0000 7fe70d400700 1 -- 172.21.15.27:0/2279732051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26b490 msgr2=0x562a5f26eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.878+0000 7fe70d400700 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26b490 0x562a5f26eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.878+0000 7fe70d400700 1 -- 172.21.15.27:0/2279732051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a5f17d3d0 con 0x562a5f26ab70 2024-05-10T11:11:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.878+0000 7fe70d400700 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 0x562a5f26e5a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00c010 tx=0x7fe6fc004a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1271d2399770ff server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.878+0000 7fe6efe00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:11:48.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.878+0000 7fe6efe00700 1 -- 172.21.15.27:0/2279732051 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc02a060 con 0x562a5f26ab70 2024-05-10T11:11:48.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.879+0000 7fe71fad6b80 1 -- 172.21.15.27:0/2279732051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6fc0044e0 con 0x562a5f26ab70 2024-05-10T11:11:48.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.879+0000 7fe6efe00700 1 -- 172.21.15.27:0/2279732051 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fe6fc010c80 con 0x562a5f26ab70 2024-05-10T11:11:48.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.879+0000 7fe6efe00700 1 -- 172.21.15.27:0/2279732051 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe6fc031050 con 0x562a5f26ab70 2024-05-10T11:11:48.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.879+0000 7fe71fad6b80 10 client.14865 fetch_fsmap learned FSMap version 29 2024-05-10T11:11:48.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.879+0000 7fe71fad6b80 1 -- 172.21.15.27:0/2279732051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562a5f27c7d0 con 0x562a5f26ab70 2024-05-10T11:11:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.879+0000 7fe6efe00700 1 -- 172.21.15.27:0/2279732051 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6fc0044e0 con 0x562a5f26ab70 2024-05-10T11:11:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.880+0000 7fe6efe00700 1 -- 172.21.15.27:0/2279732051 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe6fc011440 con 0x562a5f26ab70 2024-05-10T11:11:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.880+0000 7fe71fad6b80 10 client.14865 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:11:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.880+0000 7fe71fad6b80 10 client.14865 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-10T11:11:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.880+0000 7fe71fad6b80 20 client.14865 populate_metadata read hostname 'smithi027' 2024-05-10T11:11:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.880+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a5f27ea30 0x562a5f27fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.880+0000 7fe71fad6b80 4 client.14865 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:11:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.880+0000 7fe71fad6b80 1 -- 172.21.15.27:0/2279732051 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a5f2320f0 con 0x562a5f27ea30 2024-05-10T11:11:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.880+0000 7fe70de00700 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a5f27ea30 0x562a5f27fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.881+0000 7fe70de00700 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a5f27ea30 0x562a5f27fea0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9d62353cbfa4079d server_cookie=32ff11d4c3dbde2e in_seq=0 out_seq=0 2024-05-10T11:11:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.881+0000 7fe6efe00700 10 client.14865 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:11:48.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe6efe00700 1 -- 172.21.15.27:0/2279732051 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562a5f2320f0 con 0x562a5f27ea30 2024-05-10T11:11:48.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe6efe00700 10 client.14865 handle_command_reply: tid=0 2024-05-10T11:11:48.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe71fad6b80 1 client.14865 shutdown 2024-05-10T11:11:48.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe71fad6b80 1 -- 172.21.15.27:0/2279732051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 msgr2=0x562a5f26e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:48.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 0x562a5f26e5a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00c010 tx=0x7fe6fc004a10 comp rx=0 tx=0).stop 2024-05-10T11:11:48.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe71fad6b80 1 -- 172.21.15.27:0/2279732051 shutdown_connections 2024-05-10T11:11:48.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a5f27ea30 0x562a5f27fea0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f2658a0 0x562a5f271200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26b490 0x562a5f26eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/2279732051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 0x562a5f26e5a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe6efe00700 0 client.14865 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:11:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe71fad6b80 1 -- 172.21.15.27:0/2279732051 >> 172.21.15.27:0/2279732051 conn(0x562a5f1dd680 msgr2=0x562a5f25f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe71fad6b80 1 -- 172.21.15.27:0/2279732051 shutdown_connections 2024-05-10T11:11:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe71fad6b80 1 -- 172.21.15.27:0/2279732051 wait complete. 2024-05-10T11:11:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.882+0000 7fe71fad6b80 20 client.14865 trim_cache size 0 max 16384 2024-05-10T11:11:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.906+0000 7fe71fad6b80 1 Processor -- start 2024-05-10T11:11:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.906+0000 7fe71fad6b80 1 -- start start 2024-05-10T11:11:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.906+0000 7fe71fad6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f2658a0 0x562a5f265280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.906+0000 7fe71fad6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26ab70 0x562a5f2616d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.906+0000 7fe71fad6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26b490 0x562a5f26bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.906+0000 7fe71fad6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562a5f27e1e0 con 0x562a5f26b490 2024-05-10T11:11:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.906+0000 7fe71fad6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562a5f27e8b0 con 0x562a5f26ab70 2024-05-10T11:11:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.906+0000 7fe71fad6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562a5f27ef80 con 0x562a5f2658a0 2024-05-10T11:11:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.907+0000 7fe70de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f2658a0 0x562a5f265280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.907+0000 7fe70d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26ab70 0x562a5f2616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.907+0000 7fe70de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f2658a0 0x562a5f265280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:46420/0 (socket says 172.21.15.27:46420) 2024-05-10T11:11:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.907+0000 7fe70de00700 1 -- 172.21.15.27:0/1405839536 learned_addr learned my addr 172.21.15.27:0/1405839536 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.907+0000 7fe70e800700 1 --2- 172.21.15.27:0/1405839536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26b490 0x562a5f26bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.907+0000 7fe70d400700 1 -- 172.21.15.27:0/1405839536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f2658a0 msgr2=0x562a5f265280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.907+0000 7fe70d400700 1 --2- 172.21.15.27:0/1405839536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f2658a0 0x562a5f265280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.907+0000 7fe70d400700 1 -- 172.21.15.27:0/1405839536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26b490 msgr2=0x562a5f26bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.907+0000 7fe70d400700 1 --2- 172.21.15.27:0/1405839536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26b490 0x562a5f26bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.907+0000 7fe70d400700 1 -- 172.21.15.27:0/1405839536 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a5f26b1a0 con 0x562a5f26ab70 2024-05-10T11:11:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.907+0000 7fe70de00700 1 --2- 172.21.15.27:0/1405839536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f2658a0 0x562a5f265280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.908+0000 7fe70d400700 1 --2- 172.21.15.27:0/1405839536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26ab70 0x562a5f2616d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00bfa0 tx=0x7fe6fc00b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=952c4dee4ab1f700 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.908+0000 7fe6faa00700 1 -- 172.21.15.27:0/1405839536 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc02a040 con 0x562a5f26ab70 2024-05-10T11:11:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.908+0000 7fe6faa00700 1 -- 172.21.15.27:0/1405839536 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6fc0106b0 con 0x562a5f26ab70 2024-05-10T11:11:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.908+0000 7fe6faa00700 1 -- 172.21.15.27:0/1405839536 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc003680 con 0x562a5f26ab70 2024-05-10T11:11:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.909+0000 7fe71fad6b80 1 -- 172.21.15.27:0/1405839536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26ab70 msgr2=0x562a5f2616d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.909+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/1405839536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26ab70 0x562a5f2616d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00bfa0 tx=0x7fe6fc00b7f0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.909+0000 7fe71fad6b80 1 -- 172.21.15.27:0/1405839536 shutdown_connections 2024-05-10T11:11:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.909+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/1405839536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f2658a0 0x562a5f265280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.909+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/1405839536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26b490 0x562a5f26bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.909+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/1405839536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f26ab70 0x562a5f2616d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.909+0000 7fe71fad6b80 1 -- 172.21.15.27:0/1405839536 >> 172.21.15.27:0/1405839536 conn(0x562a5f1dd680 msgr2=0x562a5f26b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.909+0000 7fe71fad6b80 1 -- 172.21.15.27:0/1405839536 shutdown_connections 2024-05-10T11:11:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.909+0000 7fe71fad6b80 1 -- 172.21.15.27:0/1405839536 wait complete. 2024-05-10T11:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.909+0000 7fe71fad6b80 1 -- start start 2024-05-10T11:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.910+0000 7fe71fad6b80 1 Processor -- start 2024-05-10T11:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.910+0000 7fe71fad6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f2658a0 0x562a5f25edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.910+0000 7fe71fad6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 0x562a5f25f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.910+0000 7fe71fad6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f26b490 0x562a5f25f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.910+0000 7fe71fad6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562a5f27fd20 con 0x562a5f26ab70 2024-05-10T11:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.910+0000 7fe71fad6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562a5f27f650 con 0x562a5f2658a0 2024-05-10T11:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.910+0000 7fe71fad6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562a5f2803f0 con 0x562a5f26b490 2024-05-10T11:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.910+0000 7fe71fad6b80 10 client.0 _ll_register_callbacks cb 0x562a5f15ec20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.911+0000 7fe70d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 0x562a5f25f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.911+0000 7fe70d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 0x562a5f25f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56012/0 (socket says 172.21.15.27:56012) 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.911+0000 7fe70d400700 1 -- 172.21.15.27:0/568135618 learned_addr learned my addr 172.21.15.27:0/568135618 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.911+0000 7fe70de00700 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f2658a0 0x562a5f25edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.911+0000 7fe70e800700 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f26b490 0x562a5f25f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.911+0000 7fe70d400700 1 -- 172.21.15.27:0/568135618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f26b490 msgr2=0x562a5f25f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.911+0000 7fe70d400700 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f26b490 0x562a5f25f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.911+0000 7fe70d400700 1 -- 172.21.15.27:0/568135618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f2658a0 msgr2=0x562a5f25edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.911+0000 7fe70d400700 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f2658a0 0x562a5f25edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.911+0000 7fe70d400700 1 -- 172.21.15.27:0/568135618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a5f2875d0 con 0x562a5f26ab70 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.911+0000 7fe70d400700 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 0x562a5f25f330 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00b750 tx=0x7fe6fc00b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f152a7220d6e7f80 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.911+0000 7fe6efe00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.912+0000 7fe6efe00700 1 -- 172.21.15.27:0/568135618 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc02a040 con 0x562a5f26ab70 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.912+0000 7fe71fad6b80 1 -- 172.21.15.27:0/568135618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6fc0106b0 con 0x562a5f26ab70 2024-05-10T11:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.912+0000 7fe6efe00700 1 -- 172.21.15.27:0/568135618 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fe6fc004040 con 0x562a5f26ab70 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.912+0000 7fe6efe00700 1 -- 172.21.15.27:0/568135618 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe6fc045050 con 0x562a5f26ab70 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.912+0000 7fe71fad6b80 10 client.14871 fetch_fsmap learned FSMap version 29 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.912+0000 7fe71fad6b80 1 -- 172.21.15.27:0/568135618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562a5f2ea120 con 0x562a5f26ab70 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.913+0000 7fe6efe00700 1 -- 172.21.15.27:0/568135618 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6fc0106b0 con 0x562a5f26ab70 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.913+0000 7fe6efe00700 1 -- 172.21.15.27:0/568135618 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe6fc03b030 con 0x562a5f26ab70 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.913+0000 7fe71fad6b80 10 client.14871 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.913+0000 7fe71fad6b80 10 client.14871 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.913+0000 7fe71fad6b80 20 client.14871 populate_metadata read hostname 'smithi027' 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.913+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a5f2ee880 0x562a5f2ed590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.913+0000 7fe71fad6b80 4 client.14871 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.913+0000 7fe71fad6b80 1 -- 172.21.15.27:0/568135618 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "status"}) -- 0x562a5f1c53b0 con 0x562a5f2ee880 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.913+0000 7fe70de00700 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a5f2ee880 0x562a5f2ed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe70de00700 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a5f2ee880 0x562a5f2ed590 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5a9d45b42c22e739 server_cookie=80b4614b01efd03e in_seq=0 out_seq=0 2024-05-10T11:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe6efe00700 10 client.14871 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe6efe00700 1 -- 172.21.15.27:0/568135618 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x562a5f1c53b0 con 0x562a5f2ee880 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe6efe00700 10 client.14871 handle_command_reply: tid=0 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe71fad6b80 1 client.14871 shutdown 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe71fad6b80 1 -- 172.21.15.27:0/568135618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 msgr2=0x562a5f25f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 0x562a5f25f330 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00b750 tx=0x7fe6fc00b780 comp rx=0 tx=0).stop 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe71fad6b80 1 -- 172.21.15.27:0/568135618 shutdown_connections 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a5f2ee880 0x562a5f2ed590 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a5f26b490 0x562a5f25f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a5f2658a0 0x562a5f25edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe71fad6b80 1 --2- 172.21.15.27:0/568135618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a5f26ab70 0x562a5f25f330 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe6efe00700 0 client.14871 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.914+0000 7fe71fad6b80 1 -- 172.21.15.27:0/568135618 >> 172.21.15.27:0/568135618 conn(0x562a5f1dd680 msgr2=0x562a5f26c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.915+0000 7fe71fad6b80 1 -- 172.21.15.27:0/568135618 shutdown_connections 2024-05-10T11:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.915+0000 7fe71fad6b80 1 -- 172.21.15.27:0/568135618 wait complete. 2024-05-10T11:11:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.915+0000 7fe71fad6b80 20 client.14871 trim_cache size 0 max 16384 2024-05-10T11:11:48.921 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:11:48.921 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T11:11:48.921 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 3, 2024-05-10T11:11:48.921 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24472, 2024-05-10T11:11:48.921 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-05-10T11:11:48.921 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-05-10T11:11:48.921 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-05-10T11:11:48.921 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 154.91832588899999, 2024-05-10T11:11:48.921 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 29, 2024-05-10T11:11:48.921 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 90, 2024-05-10T11:11:48.922 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 88, 2024-05-10T11:11:48.922 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 280.54268517200001 2024-05-10T11:11:48.922 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:11:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.915+0000 7fe715200700 1 -- 172.21.15.27:0/2477582042 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe6f40b2900 msgr2=0x7fe6f40b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.915+0000 7fe715200700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe6f40b2900 0x7fe6f40b4dc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe708009e60 tx=0x7fe70800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.915+0000 7fe715200700 1 -- 172.21.15.27:0/2477582042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe710103750 msgr2=0x7fe7101a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.915+0000 7fe715200700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe710103750 0x7fe7101a1900 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe70000e9a0 tx=0x7fe70000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.916+0000 7fe715200700 1 -- 172.21.15.27:0/2477582042 shutdown_connections 2024-05-10T11:11:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.916+0000 7fe715200700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fe6f40b2900 0x7fe6f40b4dc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.916+0000 7fe715200700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe710103750 0x7fe7101a1900 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.916+0000 7fe715200700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe710108850 0x7fe71019bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.916+0000 7fe715200700 1 --2- 172.21.15.27:0/2477582042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe710104100 0x7fe7101a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.916+0000 7fe715200700 1 -- 172.21.15.27:0/2477582042 >> 172.21.15.27:0/2477582042 conn(0x7fe71007a600 msgr2=0x7fe7101017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.916+0000 7fe715200700 1 -- 172.21.15.27:0/2477582042 shutdown_connections 2024-05-10T11:11:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:48.916+0000 7fe715200700 1 -- 172.21.15.27:0/2477582042 wait complete. 2024-05-10T11:11:49.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.i status 2024-05-10T11:11:50.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:50 smithi027 ceph-mon[30054]: pgmap v454: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:50.403 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:50.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:50 smithi076 ceph-mon[34152]: pgmap v454: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:50.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:50 smithi189 ceph-mon[34842]: pgmap v454: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:52.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:52 smithi027 ceph-mon[30054]: pgmap v455: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:52.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:52 smithi076 ceph-mon[34152]: pgmap v455: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:52.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:52 smithi189 ceph-mon[34842]: pgmap v455: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:11:52.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.840+0000 7fd9fde00700 1 -- 172.21.15.27:0/1987532027 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f810ff50 msgr2=0x7fd9f8112340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:52.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.840+0000 7fd9fde00700 1 --2- 172.21.15.27:0/1987532027 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f810ff50 0x7fd9f8112340 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000b0a0 tx=0x7fd9f000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.840+0000 7fd9fde00700 1 -- 172.21.15.27:0/1987532027 shutdown_connections 2024-05-10T11:11:52.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.840+0000 7fd9fde00700 1 --2- 172.21.15.27:0/1987532027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f806b1b0 0x7fd9f806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.840+0000 7fd9fde00700 1 --2- 172.21.15.27:0/1987532027 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f810ff50 0x7fd9f8112340 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.840+0000 7fd9fde00700 1 --2- 172.21.15.27:0/1987532027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f806bad0 0x7fd9f810fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.840+0000 7fd9fde00700 1 -- 172.21.15.27:0/1987532027 >> 172.21.15.27:0/1987532027 conn(0x7fd9f8079350 msgr2=0x7fd9f8079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:52.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.840+0000 7fd9fde00700 1 -- 172.21.15.27:0/1987532027 shutdown_connections 2024-05-10T11:11:52.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.840+0000 7fd9fde00700 1 -- 172.21.15.27:0/1987532027 wait complete. 2024-05-10T11:11:52.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.841+0000 7fd9fde00700 1 Processor -- start 2024-05-10T11:11:52.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.842+0000 7fd9fde00700 1 -- start start 2024-05-10T11:11:52.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.842+0000 7fd9fde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f806b1b0 0x7fd9f819bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.842+0000 7fd9fde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f806bad0 0x7fd9f819c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.842+0000 7fd9fde00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f810ff50 0x7fd9f81a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.842+0000 7fd9fde00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f81158c0 con 0x7fd9f806b1b0 2024-05-10T11:11:52.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.842+0000 7fd9fde00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f8115740 con 0x7fd9f810ff50 2024-05-10T11:11:52.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.842+0000 7fd9fde00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f8115a40 con 0x7fd9f806bad0 2024-05-10T11:11:52.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.842+0000 7fd9f6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f806b1b0 0x7fd9f819bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.843+0000 7fd9f6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f806b1b0 0x7fd9f819bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56040/0 (socket says 172.21.15.27:56040) 2024-05-10T11:11:52.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.843+0000 7fd9f6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f806bad0 0x7fd9f819c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.843+0000 7fd9f6a00700 1 -- 172.21.15.27:0/3357866129 learned_addr learned my addr 172.21.15.27:0/3357866129 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:52.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.843+0000 7fd9f7400700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f810ff50 0x7fd9f81a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.843+0000 7fd9f6a00700 1 -- 172.21.15.27:0/3357866129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f806bad0 msgr2=0x7fd9f819c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:52.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.843+0000 7fd9f6a00700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f806bad0 0x7fd9f819c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.843+0000 7fd9f6a00700 1 -- 172.21.15.27:0/3357866129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f810ff50 msgr2=0x7fd9f81a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:52.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.843+0000 7fd9f6a00700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f810ff50 0x7fd9f81a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.843+0000 7fd9f6a00700 1 -- 172.21.15.27:0/3357866129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f81a0e00 con 0x7fd9f806b1b0 2024-05-10T11:11:52.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.843+0000 7fd9f6a00700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f806b1b0 0x7fd9f819bdf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800e9c0 tx=0x7fd9e800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:52.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.844+0000 7fd9fca00700 1 -- 172.21.15.27:0/3357866129 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e800cbd0 con 0x7fd9f806b1b0 2024-05-10T11:11:52.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.844+0000 7fd9fde00700 1 -- 172.21.15.27:0/3357866129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f81a10f0 con 0x7fd9f806b1b0 2024-05-10T11:11:52.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.844+0000 7fd9fca00700 1 -- 172.21.15.27:0/3357866129 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9e800ee60 con 0x7fd9f806b1b0 2024-05-10T11:11:52.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.844+0000 7fd9fca00700 1 -- 172.21.15.27:0/3357866129 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e80055c0 con 0x7fd9f806b1b0 2024-05-10T11:11:52.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.844+0000 7fd9fde00700 1 -- 172.21.15.27:0/3357866129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f81a8970 con 0x7fd9f806b1b0 2024-05-10T11:11:52.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.848+0000 7fd9fca00700 1 -- 172.21.15.27:0/3357866129 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fd9e8004750 con 0x7fd9f806b1b0 2024-05-10T11:11:52.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.849+0000 7fda087f1b80 1 Processor -- start 2024-05-10T11:11:52.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.849+0000 7fda087f1b80 1 -- start start 2024-05-10T11:11:52.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.849+0000 7fda087f1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f010 0x55cdd9b1f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.849+0000 7fda087f1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f930 0x55cdd9b23440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.849+0000 7fda087f1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b23980 0x55cdd9b23de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.849+0000 7fda087f1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cdd9a719d0 con 0x55cdd9b23980 2024-05-10T11:11:52.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.849+0000 7fda087f1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cdd9a0f710 con 0x55cdd9b1f010 2024-05-10T11:11:52.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.849+0000 7fda087f1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cdd9a6d990 con 0x55cdd9b1f930 2024-05-10T11:11:52.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.849+0000 7fd9fca00700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9dc0b2880 0x7fd9dc0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.850+0000 7fd9f7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f010 0x55cdd9b1f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.850+0000 7fd9f6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f930 0x55cdd9b23440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.850+0000 7fd9f7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f010 0x55cdd9b1f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35834/0 (socket says 172.21.15.27:35834) 2024-05-10T11:11:52.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.850+0000 7fd9f7400700 1 -- 172.21.15.27:0/1109559568 learned_addr learned my addr 172.21.15.27:0/1109559568 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:52.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.850+0000 7fd9f6000700 1 --2- 172.21.15.27:0/1109559568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b23980 0x55cdd9b23de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.850+0000 7fd9f7400700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9dc0b2880 0x7fd9dc0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.850+0000 7fd9fca00700 1 -- 172.21.15.27:0/3357866129 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fd9e80e28e0 con 0x7fd9f806b1b0 2024-05-10T11:11:52.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.850+0000 7fd9f6a00700 1 -- 172.21.15.27:0/1109559568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f010 msgr2=0x55cdd9b1f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:52.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.850+0000 7fd9f6a00700 1 --2- 172.21.15.27:0/1109559568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f010 0x55cdd9b1f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.850+0000 7fd9f6a00700 1 -- 172.21.15.27:0/1109559568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b23980 msgr2=0x55cdd9b23de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:52.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.850+0000 7fd9f6a00700 1 --2- 172.21.15.27:0/1109559568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b23980 0x55cdd9b23de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.850+0000 7fd9f6a00700 1 -- 172.21.15.27:0/1109559568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdd99be880 con 0x55cdd9b1f930 2024-05-10T11:11:52.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.851+0000 7fd9f6a00700 1 --2- 172.21.15.27:0/1109559568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f930 0x55cdd9b23440 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8073370 tx=0x7fd9e80776d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2430d876f4e41fd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:52.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.851+0000 7fd9ecc00700 1 -- 172.21.15.27:0/1109559568 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e80a9400 con 0x55cdd9b1f930 2024-05-10T11:11:52.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.851+0000 7fd9f7400700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9dc0b2880 0x7fd9dc0b4d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000b780 tx=0x7fd9f000b3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:52.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.851+0000 7fd9ecc00700 1 -- 172.21.15.27:0/1109559568 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9e80b2250 con 0x55cdd9b1f930 2024-05-10T11:11:52.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.851+0000 7fd9ecc00700 1 -- 172.21.15.27:0/1109559568 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e80a9220 con 0x55cdd9b1f930 2024-05-10T11:11:52.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.852+0000 7fda087f1b80 1 -- 172.21.15.27:0/1109559568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f930 msgr2=0x55cdd9b23440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:52.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.852+0000 7fda087f1b80 1 --2- 172.21.15.27:0/1109559568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f930 0x55cdd9b23440 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8073370 tx=0x7fd9e80776d0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.852+0000 7fda087f1b80 1 -- 172.21.15.27:0/1109559568 shutdown_connections 2024-05-10T11:11:52.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.852+0000 7fda087f1b80 1 --2- 172.21.15.27:0/1109559568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b23980 0x55cdd9b23de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.852+0000 7fda087f1b80 1 --2- 172.21.15.27:0/1109559568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f010 0x55cdd9b1f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.852+0000 7fda087f1b80 1 --2- 172.21.15.27:0/1109559568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f930 0x55cdd9b23440 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.852+0000 7fda087f1b80 1 -- 172.21.15.27:0/1109559568 >> 172.21.15.27:0/1109559568 conn(0x55cdd9a4db10 msgr2=0x55cdd9b19750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:52.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.852+0000 7fda087f1b80 1 -- 172.21.15.27:0/1109559568 shutdown_connections 2024-05-10T11:11:52.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.852+0000 7fda087f1b80 1 -- 172.21.15.27:0/1109559568 wait complete. 2024-05-10T11:11:52.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.853+0000 7fda087f1b80 1 -- start start 2024-05-10T11:11:52.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.853+0000 7fda087f1b80 1 Processor -- start 2024-05-10T11:11:52.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fda087f1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b2aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fda087f1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b1f930 0x55cdd9b282c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fda087f1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b23980 0x55cdd9b28800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fda087f1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cdd9b2bad0 con 0x55cdd9b1f930 2024-05-10T11:11:52.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fda087f1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cdd99d7ad0 con 0x55cdd9b23980 2024-05-10T11:11:52.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fda087f1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cdd9b2bc50 con 0x55cdd9b1f010 2024-05-10T11:11:52.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fda087f1b80 10 client.0 _ll_register_callbacks cb 0x55cdd9a18c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:11:52.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fd9f6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b1f930 0x55cdd9b282c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fd9f6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b2aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fd9f6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b2aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:46468/0 (socket says 172.21.15.27:46468) 2024-05-10T11:11:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fd9f6a00700 1 -- 172.21.15.27:0/632965157 learned_addr learned my addr 172.21.15.27:0/632965157 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fd9f7400700 1 --2- 172.21.15.27:0/632965157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b23980 0x55cdd9b28800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fd9f6a00700 1 -- 172.21.15.27:0/632965157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b23980 msgr2=0x55cdd9b28800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fd9f6a00700 1 --2- 172.21.15.27:0/632965157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b23980 0x55cdd9b28800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fd9f6a00700 1 -- 172.21.15.27:0/632965157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b1f930 msgr2=0x55cdd9b282c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.854+0000 7fd9f6a00700 1 --2- 172.21.15.27:0/632965157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b1f930 0x55cdd9b282c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.855+0000 7fd9f6a00700 1 -- 172.21.15.27:0/632965157 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdd9a373d0 con 0x55cdd9b1f010 2024-05-10T11:11:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.855+0000 7fd9f6a00700 1 --2- 172.21.15.27:0/632965157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b2aeb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8005060 tx=0x7fd9e80a9310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d93bed3975ba6e70 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.855+0000 7fd9e1600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:11:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.855+0000 7fd9e1600700 1 -- 172.21.15.27:0/632965157 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e80a9ab0 con 0x55cdd9b1f010 2024-05-10T11:11:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.855+0000 7fd9e1600700 1 -- 172.21.15.27:0/632965157 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fd9e8073170 con 0x55cdd9b1f010 2024-05-10T11:11:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.855+0000 7fda087f1b80 1 -- 172.21.15.27:0/632965157 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cdd9aec050 con 0x55cdd9b1f010 2024-05-10T11:11:52.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.855+0000 7fd9e1600700 1 -- 172.21.15.27:0/632965157 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd9e80aa1b0 con 0x55cdd9b1f010 2024-05-10T11:11:52.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.855+0000 7fda087f1b80 10 client.24991 fetch_fsmap learned FSMap version 29 2024-05-10T11:11:52.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.855+0000 7fda087f1b80 1 -- 172.21.15.27:0/632965157 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cdd9b2bad0 con 0x55cdd9b1f010 2024-05-10T11:11:52.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.856+0000 7fd9e1600700 1 -- 172.21.15.27:0/632965157 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55cdd9aec050 con 0x55cdd9b1f010 2024-05-10T11:11:52.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.856+0000 7fd9e1600700 1 -- 172.21.15.27:0/632965157 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd9e80a9d10 con 0x55cdd9b1f010 2024-05-10T11:11:52.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.856+0000 7fda087f1b80 10 client.24991 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:11:52.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.856+0000 7fda087f1b80 10 client.24991 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-10T11:11:52.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.856+0000 7fda087f1b80 20 client.24991 populate_metadata read hostname 'smithi027' 2024-05-10T11:11:52.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.856+0000 7fda087f1b80 1 --2- 172.21.15.27:0/632965157 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55cdd9b384c0 0x55cdd9b39930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.856+0000 7fda087f1b80 4 client.24991 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:11:52.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.856+0000 7fda087f1b80 1 -- 172.21.15.27:0/632965157 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdd9a7f3b0 con 0x55cdd9b384c0 2024-05-10T11:11:52.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.857+0000 7fd9f6000700 1 --2- 172.21.15.27:0/632965157 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55cdd9b384c0 0x55cdd9b39930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.857+0000 7fd9f6000700 1 --2- 172.21.15.27:0/632965157 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55cdd9b384c0 0x55cdd9b39930 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=954340208f438f8b server_cookie=8af801e594ab56ee in_seq=0 out_seq=0 2024-05-10T11:11:52.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.857+0000 7fd9e1600700 10 client.24991 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:11:52.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.858+0000 7fd9e1600700 1 -- 172.21.15.27:0/632965157 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cdd9a7f3b0 con 0x55cdd9b384c0 2024-05-10T11:11:52.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.858+0000 7fd9e1600700 10 client.24991 handle_command_reply: tid=0 2024-05-10T11:11:52.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fda087f1b80 1 client.24991 shutdown 2024-05-10T11:11:52.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fda087f1b80 1 -- 172.21.15.27:0/632965157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 msgr2=0x55cdd9b2aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:52.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fda087f1b80 1 --2- 172.21.15.27:0/632965157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b2aeb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8005060 tx=0x7fd9e80a9310 comp rx=0 tx=0).stop 2024-05-10T11:11:52.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fda087f1b80 1 -- 172.21.15.27:0/632965157 shutdown_connections 2024-05-10T11:11:52.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fda087f1b80 1 --2- 172.21.15.27:0/632965157 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55cdd9b384c0 0x55cdd9b39930 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fda087f1b80 1 --2- 172.21.15.27:0/632965157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b2aeb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fda087f1b80 1 --2- 172.21.15.27:0/632965157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b23980 0x55cdd9b28800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fda087f1b80 1 --2- 172.21.15.27:0/632965157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b1f930 0x55cdd9b282c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fda087f1b80 1 -- 172.21.15.27:0/632965157 >> 172.21.15.27:0/632965157 conn(0x55cdd9a4db10 msgr2=0x55cdd9b19710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:52.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fd9e1600700 0 client.24991 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:11:52.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fda087f1b80 1 -- 172.21.15.27:0/632965157 shutdown_connections 2024-05-10T11:11:52.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fda087f1b80 1 -- 172.21.15.27:0/632965157 wait complete. 2024-05-10T11:11:52.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.859+0000 7fda087f1b80 20 client.24991 trim_cache size 0 max 16384 2024-05-10T11:11:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.883+0000 7fda087f1b80 1 Processor -- start 2024-05-10T11:11:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.883+0000 7fda087f1b80 1 -- start start 2024-05-10T11:11:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.883+0000 7fda087f1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b27240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.883+0000 7fda087f1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f930 0x55cdd9b28850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.883+0000 7fda087f1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b23980 0x55cdd9b42d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.884+0000 7fda087f1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cdd9b37c70 con 0x55cdd9b23980 2024-05-10T11:11:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.884+0000 7fda087f1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cdd9b38340 con 0x55cdd9b1f930 2024-05-10T11:11:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.884+0000 7fda087f1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cdd9b38a10 con 0x55cdd9b1f010 2024-05-10T11:11:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.884+0000 7fd9f7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b23980 0x55cdd9b42d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.884+0000 7fd9f6000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f930 0x55cdd9b28850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.884+0000 7fd9f6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b27240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.884+0000 7fd9f7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b23980 0x55cdd9b42d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56068/0 (socket says 172.21.15.27:56068) 2024-05-10T11:11:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.884+0000 7fd9f6000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f930 0x55cdd9b28850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35852/0 (socket says 172.21.15.27:35852) 2024-05-10T11:11:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.884+0000 7fd9f7400700 1 -- 172.21.15.27:0/3166997771 learned_addr learned my addr 172.21.15.27:0/3166997771 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.884+0000 7fd9f6000700 1 -- 172.21.15.27:0/3166997771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 msgr2=0x55cdd9b27240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.885+0000 7fd9f6000700 1 --2- 172.21.15.27:0/3166997771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b27240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.885+0000 7fd9f6000700 1 -- 172.21.15.27:0/3166997771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b23980 msgr2=0x55cdd9b42d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:52.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.885+0000 7fd9f6000700 1 --2- 172.21.15.27:0/3166997771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b23980 0x55cdd9b42d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.885+0000 7fd9f6000700 1 -- 172.21.15.27:0/3166997771 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdd9b163a0 con 0x55cdd9b1f930 2024-05-10T11:11:52.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.885+0000 7fd9f6000700 1 --2- 172.21.15.27:0/3166997771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f930 0x55cdd9b28850 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x55cdd9b3a190 tx=0x7fd9e4009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32af786d71598f71 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:52.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.885+0000 7fd9e3e00700 1 -- 172.21.15.27:0/3166997771 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4026070 con 0x55cdd9b1f930 2024-05-10T11:11:52.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.885+0000 7fd9e3e00700 1 -- 172.21.15.27:0/3166997771 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9e4003eb0 con 0x55cdd9b1f930 2024-05-10T11:11:52.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.885+0000 7fd9e3e00700 1 -- 172.21.15.27:0/3166997771 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4010430 con 0x55cdd9b1f930 2024-05-10T11:11:52.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.886+0000 7fda087f1b80 1 -- 172.21.15.27:0/3166997771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f930 msgr2=0x55cdd9b28850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:52.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.886+0000 7fda087f1b80 1 --2- 172.21.15.27:0/3166997771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f930 0x55cdd9b28850 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x55cdd9b3a190 tx=0x7fd9e4009b60 comp rx=0 tx=0).stop 2024-05-10T11:11:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.886+0000 7fda087f1b80 1 -- 172.21.15.27:0/3166997771 shutdown_connections 2024-05-10T11:11:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.886+0000 7fda087f1b80 1 --2- 172.21.15.27:0/3166997771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b27240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.886+0000 7fda087f1b80 1 --2- 172.21.15.27:0/3166997771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b23980 0x55cdd9b42d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.886+0000 7fda087f1b80 1 --2- 172.21.15.27:0/3166997771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b1f930 0x55cdd9b28850 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.886+0000 7fda087f1b80 1 -- 172.21.15.27:0/3166997771 >> 172.21.15.27:0/3166997771 conn(0x55cdd9a4db10 msgr2=0x55cdd9b14260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.886+0000 7fda087f1b80 1 -- 172.21.15.27:0/3166997771 shutdown_connections 2024-05-10T11:11:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.887+0000 7fda087f1b80 1 -- 172.21.15.27:0/3166997771 wait complete. 2024-05-10T11:11:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.887+0000 7fda087f1b80 1 -- start start 2024-05-10T11:11:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.887+0000 7fda087f1b80 1 Processor -- start 2024-05-10T11:11:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.888+0000 7fda087f1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b3ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.888+0000 7fda087f1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b1f930 0x55cdd9b3f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.888+0000 7fda087f1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b23980 0x55cdd9b3f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.888+0000 7fda087f1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cdd9b397b0 con 0x55cdd9b1f930 2024-05-10T11:11:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.888+0000 7fda087f1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cdd9b390e0 con 0x55cdd9b23980 2024-05-10T11:11:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.888+0000 7fda087f1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cdd9b39e80 con 0x55cdd9b1f010 2024-05-10T11:11:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.888+0000 7fda087f1b80 10 client.0 _ll_register_callbacks cb 0x55cdd9a18c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:11:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.888+0000 7fd9f6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b1f930 0x55cdd9b3f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.888+0000 7fd9f6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b3ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.888+0000 7fd9f7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b23980 0x55cdd9b3f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.888+0000 7fd9f6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b1f930 0x55cdd9b3f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56074/0 (socket says 172.21.15.27:56074) 2024-05-10T11:11:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.888+0000 7fd9f6000700 1 -- 172.21.15.27:0/2271440500 learned_addr learned my addr 172.21.15.27:0/2271440500 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.889+0000 7fd9f7400700 1 -- 172.21.15.27:0/2271440500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 msgr2=0x55cdd9b3ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.889+0000 7fd9f7400700 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b3ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.889+0000 7fd9f7400700 1 -- 172.21.15.27:0/2271440500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b1f930 msgr2=0x55cdd9b3f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.889+0000 7fd9f7400700 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b1f930 0x55cdd9b3f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.889+0000 7fd9f7400700 1 -- 172.21.15.27:0/2271440500 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdd9b1f640 con 0x55cdd9b23980 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.889+0000 7fd9f6a00700 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b3ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.889+0000 7fd9f6000700 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b1f930 0x55cdd9b3f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.889+0000 7fd9f7400700 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b23980 0x55cdd9b3f6a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x55cdd9b1bab0 tx=0x7fd9f00099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21a5f7c927c620ea server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.889+0000 7fd9e1600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.890+0000 7fd9e1600700 1 -- 172.21.15.27:0/2271440500 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0031070 con 0x55cdd9b23980 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.890+0000 7fda087f1b80 1 -- 172.21.15.27:0/2271440500 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9e4003eb0 con 0x55cdd9b23980 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.890+0000 7fd9e1600700 1 -- 172.21.15.27:0/2271440500 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fd9f0007a50 con 0x55cdd9b23980 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.890+0000 7fd9e1600700 1 -- 172.21.15.27:0/2271440500 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd9f003b050 con 0x55cdd9b23980 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.890+0000 7fda087f1b80 10 client.25013 fetch_fsmap learned FSMap version 29 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.890+0000 7fda087f1b80 1 -- 172.21.15.27:0/2271440500 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cdd9ba3da0 con 0x55cdd9b23980 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.890+0000 7fd9e1600700 1 -- 172.21.15.27:0/2271440500 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9e4003eb0 con 0x55cdd9b23980 2024-05-10T11:11:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.891+0000 7fd9e1600700 1 -- 172.21.15.27:0/2271440500 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd9f003e020 con 0x55cdd9b23980 2024-05-10T11:11:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.891+0000 7fda087f1b80 10 client.25013 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:11:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.891+0000 7fda087f1b80 10 client.25013 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-10T11:11:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.891+0000 7fda087f1b80 20 client.25013 populate_metadata read hostname 'smithi027' 2024-05-10T11:11:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.891+0000 7fda087f1b80 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55cdd9ba8a00 0x55cdd9ba7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.891+0000 7fda087f1b80 4 client.25013 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-10T11:11:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.891+0000 7fda087f1b80 1 -- 172.21.15.27:0/2271440500 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "status"}) -- 0x55cdd9a79450 con 0x55cdd9ba8a00 2024-05-10T11:11:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.891+0000 7fd9f6a00700 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55cdd9ba8a00 0x55cdd9ba7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.892+0000 7fd9f6a00700 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55cdd9ba8a00 0x55cdd9ba7710 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e0220c35b61aa001 server_cookie=66f1427fb6c62931 in_seq=0 out_seq=0 2024-05-10T11:11:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.892+0000 7fd9e1600700 10 client.25013 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:11:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.892+0000 7fd9e1600700 1 -- 172.21.15.27:0/2271440500 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55cdd9a79450 con 0x55cdd9ba8a00 2024-05-10T11:11:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.892+0000 7fd9e1600700 10 client.25013 handle_command_reply: tid=0 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.892+0000 7fda087f1b80 1 client.25013 shutdown 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.893+0000 7fda087f1b80 1 -- 172.21.15.27:0/2271440500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b23980 msgr2=0x55cdd9b3f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.893+0000 7fda087f1b80 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b23980 0x55cdd9b3f6a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x55cdd9b1bab0 tx=0x7fd9f00099e0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.893+0000 7fda087f1b80 1 -- 172.21.15.27:0/2271440500 shutdown_connections 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.893+0000 7fda087f1b80 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55cdd9ba8a00 0x55cdd9ba7710 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.893+0000 7fda087f1b80 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdd9b1f010 0x55cdd9b3ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.893+0000 7fda087f1b80 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cdd9b23980 0x55cdd9b3f6a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.893+0000 7fda087f1b80 1 --2- 172.21.15.27:0/2271440500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdd9b1f930 0x55cdd9b3f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.893+0000 7fd9e1600700 0 client.25013 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.893+0000 7fda087f1b80 1 -- 172.21.15.27:0/2271440500 >> 172.21.15.27:0/2271440500 conn(0x55cdd9a4db10 msgr2=0x55cdd9b1de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.893+0000 7fda087f1b80 1 -- 172.21.15.27:0/2271440500 shutdown_connections 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.893+0000 7fda087f1b80 1 -- 172.21.15.27:0/2271440500 wait complete. 2024-05-10T11:11:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.893+0000 7fda087f1b80 20 client.25013 trim_cache size 0 max 16384 2024-05-10T11:11:52.897 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:11:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T11:11:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 2, 2024-05-10T11:11:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24484, 2024-05-10T11:11:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-05-10T11:11:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-05-10T11:11:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-05-10T11:11:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 159.90963991300001, 2024-05-10T11:11:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 29, 2024-05-10T11:11:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 90, 2024-05-10T11:11:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 88, 2024-05-10T11:11:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 278.73951525400003 2024-05-10T11:11:52.898 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:11:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.894+0000 7fd9fde00700 1 -- 172.21.15.27:0/3357866129 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9dc0b2880 msgr2=0x7fd9dc0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.894+0000 7fd9fde00700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9dc0b2880 0x7fd9dc0b4d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000b780 tx=0x7fd9f000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.894+0000 7fd9fde00700 1 -- 172.21.15.27:0/3357866129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f806b1b0 msgr2=0x7fd9f819bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.894+0000 7fd9fde00700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f806b1b0 0x7fd9f819bdf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800e9c0 tx=0x7fd9e800ecd0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.895+0000 7fd9fde00700 1 -- 172.21.15.27:0/3357866129 shutdown_connections 2024-05-10T11:11:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.895+0000 7fd9fde00700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fd9dc0b2880 0x7fd9dc0b4d40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.895+0000 7fd9fde00700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9f810ff50 0x7fd9f81a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.895+0000 7fd9fde00700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9f806b1b0 0x7fd9f819bdf0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.895+0000 7fd9fde00700 1 --2- 172.21.15.27:0/3357866129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f806bad0 0x7fd9f819c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.895+0000 7fd9fde00700 1 -- 172.21.15.27:0/3357866129 >> 172.21.15.27:0/3357866129 conn(0x7fd9f8079350 msgr2=0x7fd9f8111460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.896+0000 7fd9fde00700 1 -- 172.21.15.27:0/3357866129 shutdown_connections 2024-05-10T11:11:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:52.896+0000 7fd9fde00700 1 -- 172.21.15.27:0/3357866129 wait complete. 2024-05-10T11:11:53.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.h status 2024-05-10T11:11:54.333 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:54.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:54 smithi027 ceph-mon[30054]: pgmap v456: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:54.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:54 smithi076 ceph-mon[34152]: pgmap v456: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:54.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:54 smithi189 ceph-mon[34842]: pgmap v456: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:55.360 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:55 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:11:55.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:55 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:11:55.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:55 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:11:56.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:56 smithi027 ceph-mon[30054]: pgmap v457: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:56.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:11:56.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:11:56.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:11:56.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:56 smithi076 ceph-mon[34152]: pgmap v457: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:56.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:56 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:11:56.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:56 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:11:56.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:56 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:11:56.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:56 smithi189 ceph-mon[34842]: pgmap v457: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:11:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:56 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:11:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:56 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:11:56.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:56 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:11:56.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.570+0000 7f8fd8400700 1 -- 172.21.15.27:0/3709668182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0108850 msgr2=0x7f8fd01103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:56.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.570+0000 7f8fd8400700 1 --2- 172.21.15.27:0/3709668182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0108850 0x7f8fd01103d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800b0d0 tx=0x7f8fc800b3e0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.570+0000 7f8fd8400700 1 -- 172.21.15.27:0/3709668182 shutdown_connections 2024-05-10T11:11:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.570+0000 7f8fd8400700 1 --2- 172.21.15.27:0/3709668182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd0103750 0x7f8fd0103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.570+0000 7f8fd8400700 1 --2- 172.21.15.27:0/3709668182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0108850 0x7f8fd01103d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.570+0000 7f8fd8400700 1 --2- 172.21.15.27:0/3709668182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fd0104100 0x7f8fd0108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.570+0000 7f8fd8400700 1 -- 172.21.15.27:0/3709668182 >> 172.21.15.27:0/3709668182 conn(0x7f8fd007a600 msgr2=0x7f8fd007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.571+0000 7f8fd8400700 1 -- 172.21.15.27:0/3709668182 shutdown_connections 2024-05-10T11:11:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.571+0000 7f8fd8400700 1 -- 172.21.15.27:0/3709668182 wait complete. 2024-05-10T11:11:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.572+0000 7f8fd8400700 1 Processor -- start 2024-05-10T11:11:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.572+0000 7f8fd8400700 1 -- start start 2024-05-10T11:11:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.572+0000 7f8fd8400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fd0103750 0x7f8fd019bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.572+0000 7f8fd8400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0104100 0x7f8fd019c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.572+0000 7f8fd8400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd0108850 0x7f8fd01a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.572+0000 7f8fd8400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd0113670 con 0x7f8fd0104100 2024-05-10T11:11:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.572+0000 7f8fd8400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd01134f0 con 0x7f8fd0103750 2024-05-10T11:11:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.572+0000 7f8fd8400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd01137f0 con 0x7f8fd0108850 2024-05-10T11:11:56.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.573+0000 7f8fd5000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0104100 0x7f8fd019c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.573+0000 7f8fd5000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0104100 0x7f8fd019c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35676/0 (socket says 172.21.15.27:35676) 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.573+0000 7f8fd5000700 1 -- 172.21.15.27:0/2879174160 learned_addr learned my addr 172.21.15.27:0/2879174160 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.573+0000 7f8fd5a00700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fd0103750 0x7f8fd019bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.573+0000 7f8fd6400700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd0108850 0x7f8fd01a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.573+0000 7f8fd5000700 1 -- 172.21.15.27:0/2879174160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd0108850 msgr2=0x7f8fd01a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.573+0000 7f8fd5000700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd0108850 0x7f8fd01a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.573+0000 7f8fd5000700 1 -- 172.21.15.27:0/2879174160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fd0103750 msgr2=0x7f8fd019bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.573+0000 7f8fd5000700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fd0103750 0x7f8fd019bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.573+0000 7f8fd5000700 1 -- 172.21.15.27:0/2879174160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd01a0d50 con 0x7f8fd0104100 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.573+0000 7f8fd5a00700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fd0103750 0x7f8fd019bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.574+0000 7f8fd5000700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0104100 0x7f8fd019c2d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00eb30 tx=0x7f8fbc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.574+0000 7f8fce000700 1 -- 172.21.15.27:0/2879174160 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc00cbd0 con 0x7f8fd0104100 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.574+0000 7f8fd8400700 1 -- 172.21.15.27:0/2879174160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd01a1040 con 0x7f8fd0104100 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.574+0000 7f8fce000700 1 -- 172.21.15.27:0/2879174160 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fbc0044d0 con 0x7f8fd0104100 2024-05-10T11:11:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.574+0000 7f8fce000700 1 -- 172.21.15.27:0/2879174160 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc003e30 con 0x7f8fd0104100 2024-05-10T11:11:56.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.574+0000 7f8fd8400700 1 -- 172.21.15.27:0/2879174160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd01a88c0 con 0x7f8fd0104100 2024-05-10T11:11:56.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.580+0000 7f8fce000700 1 -- 172.21.15.27:0/2879174160 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f8fbc004000 con 0x7f8fd0104100 2024-05-10T11:11:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.581+0000 7f8fe2d08b80 1 Processor -- start 2024-05-10T11:11:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.581+0000 7f8fe2d08b80 1 -- start start 2024-05-10T11:11:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.581+0000 7f8fe2d08b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457010 0x55a32c4573f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.581+0000 7f8fe2d08b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c457930 0x55a32c45b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.581+0000 7f8fe2d08b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c45b980 0x55a32c45bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.581+0000 7f8fe2d08b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a32c3a99d0 con 0x55a32c457930 2024-05-10T11:11:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.581+0000 7f8fe2d08b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a32c347710 con 0x55a32c45b980 2024-05-10T11:11:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.581+0000 7f8fe2d08b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a32c3a5990 con 0x55a32c457010 2024-05-10T11:11:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.582+0000 7f8fd5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c457930 0x55a32c45b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.582+0000 7f8fd5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c457930 0x55a32c45b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35686/0 (socket says 172.21.15.27:35686) 2024-05-10T11:11:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.582+0000 7f8fd5a00700 1 -- 172.21.15.27:0/2671614462 learned_addr learned my addr 172.21.15.27:0/2671614462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.582+0000 7f8fce000700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f8fb80b2880 0x7f8fb80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.582+0000 7f8fd6400700 1 --2- 172.21.15.27:0/2671614462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457010 0x55a32c4573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.582+0000 7f8fd5000700 1 --2- 172.21.15.27:0/2671614462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c45b980 0x55a32c45bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.582+0000 7f8fd5a00700 1 -- 172.21.15.27:0/2671614462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457010 msgr2=0x55a32c4573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.582+0000 7f8fd5a00700 1 --2- 172.21.15.27:0/2671614462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457010 0x55a32c4573f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.582+0000 7f8fd5a00700 1 -- 172.21.15.27:0/2671614462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c45b980 msgr2=0x55a32c45bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.582+0000 7f8fd5a00700 1 --2- 172.21.15.27:0/2671614462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c45b980 0x55a32c45bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.582+0000 7f8fd5a00700 1 -- 172.21.15.27:0/2671614462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a32c2f6880 con 0x55a32c457930 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.582+0000 7f8fce000700 1 -- 172.21.15.27:0/2879174160 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f8fbc0e2970 con 0x7f8fd0104100 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.583+0000 7f8fd6400700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f8fb80b2880 0x7f8fb80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.583+0000 7f8fd5a00700 1 --2- 172.21.15.27:0/2671614462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c457930 0x55a32c45b440 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40115f0 tx=0x7f8fc4011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1500a0486433c5a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.583+0000 7f8fc2a00700 1 -- 172.21.15.27:0/2671614462 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc401e070 con 0x55a32c457930 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.583+0000 7f8fc2a00700 1 -- 172.21.15.27:0/2671614462 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc400fb30 con 0x55a32c457930 2024-05-10T11:11:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.583+0000 7f8fc2a00700 1 -- 172.21.15.27:0/2671614462 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4019bc0 con 0x55a32c457930 2024-05-10T11:11:56.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.584+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2671614462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c457930 msgr2=0x55a32c45b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:56.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.584+0000 7f8fd6400700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f8fb80b2880 0x7f8fb80b4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009e60 tx=0x7f8fc8005040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:56.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.584+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/2671614462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c457930 0x55a32c45b440 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40115f0 tx=0x7f8fc4011900 comp rx=0 tx=0).stop 2024-05-10T11:11:56.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.584+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2671614462 shutdown_connections 2024-05-10T11:11:56.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.584+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/2671614462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457010 0x55a32c4573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.584+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/2671614462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c45b980 0x55a32c45bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.584+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/2671614462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c457930 0x55a32c45b440 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.584+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2671614462 >> 172.21.15.27:0/2671614462 conn(0x55a32c385b10 msgr2=0x55a32c451750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:56.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.584+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2671614462 shutdown_connections 2024-05-10T11:11:56.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.584+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2671614462 wait complete. 2024-05-10T11:11:56.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.585+0000 7f8fe2d08b80 1 -- start start 2024-05-10T11:11:56.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.585+0000 7f8fe2d08b80 1 Processor -- start 2024-05-10T11:11:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.586+0000 7f8fe2d08b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457010 0x55a32c462e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.586+0000 7f8fe2d08b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457930 0x55a32c4601c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.586+0000 7f8fe2d08b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c460700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.586+0000 7f8fe2d08b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a32c463b90 con 0x55a32c45b980 2024-05-10T11:11:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.586+0000 7f8fe2d08b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a32c30fad0 con 0x55a32c457930 2024-05-10T11:11:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.586+0000 7f8fe2d08b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a32c463d10 con 0x55a32c457010 2024-05-10T11:11:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.586+0000 7f8fe2d08b80 10 client.0 _ll_register_callbacks cb 0x55a32c350c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:11:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.586+0000 7f8fd6400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c460700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.586+0000 7f8fd5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457010 0x55a32c462e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.586+0000 7f8fd5000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457930 0x55a32c4601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.586+0000 7f8fd6400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c460700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35694/0 (socket says 172.21.15.27:35694) 2024-05-10T11:11:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.586+0000 7f8fd6400700 1 -- 172.21.15.27:0/538449741 learned_addr learned my addr 172.21.15.27:0/538449741 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.587+0000 7f8fd5a00700 1 -- 172.21.15.27:0/538449741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457930 msgr2=0x55a32c4601c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.587+0000 7f8fd5a00700 1 --2- 172.21.15.27:0/538449741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457930 0x55a32c4601c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.587+0000 7f8fd5a00700 1 -- 172.21.15.27:0/538449741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 msgr2=0x55a32c460700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.587+0000 7f8fd5a00700 1 --2- 172.21.15.27:0/538449741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c460700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.587+0000 7f8fd5a00700 1 -- 172.21.15.27:0/538449741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a32c36f3d0 con 0x55a32c457010 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.587+0000 7f8fd5a00700 1 --2- 172.21.15.27:0/538449741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457010 0x55a32c462e20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4000c00 tx=0x7f8fc4019660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1d099dae50af203 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.587+0000 7f8fb7400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.587+0000 7f8fb7400700 1 -- 172.21.15.27:0/538449741 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4027030 con 0x55a32c457010 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.587+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/538449741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fc400fb30 con 0x55a32c457010 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.587+0000 7f8fb7400700 1 -- 172.21.15.27:0/538449741 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f8fc40043d0 con 0x55a32c457010 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.588+0000 7f8fb7400700 1 -- 172.21.15.27:0/538449741 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8fc4031050 con 0x55a32c457010 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.588+0000 7f8fe2d08b80 10 client.25009 fetch_fsmap learned FSMap version 29 2024-05-10T11:11:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.588+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/538449741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a32c46e410 con 0x55a32c457010 2024-05-10T11:11:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.588+0000 7f8fb7400700 1 -- 172.21.15.27:0/538449741 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc400fb30 con 0x55a32c457010 2024-05-10T11:11:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.588+0000 7f8fb7400700 1 -- 172.21.15.27:0/538449741 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8fc4019840 con 0x55a32c457010 2024-05-10T11:11:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.588+0000 7f8fe2d08b80 10 client.25009 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:11:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.588+0000 7f8fe2d08b80 10 client.25009 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-10T11:11:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.588+0000 7f8fe2d08b80 20 client.25009 populate_metadata read hostname 'smithi027' 2024-05-10T11:11:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.588+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/538449741 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a32c470850 0x55a32c471cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.588+0000 7f8fe2d08b80 4 client.25009 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:11:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.588+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/538449741 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a32c424050 con 0x55a32c470850 2024-05-10T11:11:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.589+0000 7f8fd5000700 1 --2- 172.21.15.27:0/538449741 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a32c470850 0x55a32c471cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.589+0000 7f8fd5000700 1 --2- 172.21.15.27:0/538449741 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a32c470850 0x55a32c471cc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f59ca4192744084c server_cookie=c3311150080e49d5 in_seq=0 out_seq=0 2024-05-10T11:11:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.589+0000 7f8fb7400700 10 client.25009 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:11:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.590+0000 7f8fb7400700 1 -- 172.21.15.27:0/538449741 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a32c424050 con 0x55a32c470850 2024-05-10T11:11:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.590+0000 7f8fb7400700 10 client.25009 handle_command_reply: tid=0 2024-05-10T11:11:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.590+0000 7f8fe2d08b80 1 client.25009 shutdown 2024-05-10T11:11:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.591+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/538449741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457010 msgr2=0x55a32c462e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.591+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/538449741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457010 0x55a32c462e20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4000c00 tx=0x7f8fc4019660 comp rx=0 tx=0).stop 2024-05-10T11:11:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.591+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/538449741 shutdown_connections 2024-05-10T11:11:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.591+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/538449741 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a32c470850 0x55a32c471cc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.591+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/538449741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457010 0x55a32c462e20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.591+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/538449741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c460700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.591+0000 7f8fb7400700 0 client.25009 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:11:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.591+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/538449741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457930 0x55a32c4601c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.591+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/538449741 >> 172.21.15.27:0/538449741 conn(0x55a32c385b10 msgr2=0x55a32c44c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.591+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/538449741 shutdown_connections 2024-05-10T11:11:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.591+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/538449741 wait complete. 2024-05-10T11:11:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.591+0000 7f8fe2d08b80 20 client.25009 trim_cache size 0 max 16384 2024-05-10T11:11:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.614+0000 7f8fe2d08b80 1 Processor -- start 2024-05-10T11:11:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.614+0000 7f8fe2d08b80 1 -- start start 2024-05-10T11:11:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.614+0000 7f8fe2d08b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 0x55a32c45de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.614+0000 7f8fe2d08b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 0x55a32c460730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.614+0000 7f8fe2d08b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c44b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.614+0000 7f8fe2d08b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a32c470000 con 0x55a32c45b980 2024-05-10T11:11:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.615+0000 7f8fe2d08b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a32c4706d0 con 0x55a32c457010 2024-05-10T11:11:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.615+0000 7f8fe2d08b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a32c470da0 con 0x55a32c457930 2024-05-10T11:11:56.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.615+0000 7f8fd5a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 0x55a32c45de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.615+0000 7f8fd5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 0x55a32c460730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.615+0000 7f8fd5a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 0x55a32c45de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51984/0 (socket says 172.21.15.27:51984) 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.615+0000 7f8fd5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 0x55a32c460730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:57476/0 (socket says 172.21.15.27:57476) 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.615+0000 7f8fd5a00700 1 -- 172.21.15.27:0/3146279995 learned_addr learned my addr 172.21.15.27:0/3146279995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.615+0000 7f8fd6400700 1 --2- 172.21.15.27:0/3146279995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c44b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.616+0000 7f8fd5000700 1 -- 172.21.15.27:0/3146279995 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 msgr2=0x55a32c45de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.616+0000 7f8fd5000700 1 --2- 172.21.15.27:0/3146279995 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 0x55a32c45de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.616+0000 7f8fd5000700 1 -- 172.21.15.27:0/3146279995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 msgr2=0x55a32c44b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.616+0000 7f8fd5000700 1 --2- 172.21.15.27:0/3146279995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c44b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.616+0000 7f8fd5000700 1 -- 172.21.15.27:0/3146279995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a32c45e730 con 0x55a32c457930 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.616+0000 7f8fd6400700 1 --2- 172.21.15.27:0/3146279995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c44b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.616+0000 7f8fd5a00700 1 --2- 172.21.15.27:0/3146279995 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 0x55a32c45de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:56.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.616+0000 7f8fd5000700 1 --2- 172.21.15.27:0/3146279995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 0x55a32c460730 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc06f050 tx=0x7f8fbc0764c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6835246fd10349dd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:56.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.616+0000 7f8fc2000700 1 -- 172.21.15.27:0/3146279995 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc0a9490 con 0x55a32c457930 2024-05-10T11:11:56.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.616+0000 7f8fc2000700 1 -- 172.21.15.27:0/3146279995 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fbc0b22e0 con 0x55a32c457930 2024-05-10T11:11:56.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.616+0000 7f8fc2000700 1 -- 172.21.15.27:0/3146279995 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc0a52d0 con 0x55a32c457930 2024-05-10T11:11:56.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.617+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/3146279995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 msgr2=0x55a32c460730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.617+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/3146279995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 0x55a32c460730 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc06f050 tx=0x7f8fbc0764c0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.617+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/3146279995 shutdown_connections 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.617+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/3146279995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c44b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.617+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/3146279995 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 0x55a32c45de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.617+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/3146279995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 0x55a32c460730 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.617+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/3146279995 >> 172.21.15.27:0/3146279995 conn(0x55a32c385b10 msgr2=0x55a32c45eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.617+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/3146279995 shutdown_connections 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.617+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/3146279995 wait complete. 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.618+0000 7f8fe2d08b80 1 -- start start 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.618+0000 7f8fe2d08b80 1 Processor -- start 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fe2d08b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 0x55a32c45fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fe2d08b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 0x55a32c460260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fe2d08b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c4797c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fe2d08b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a32c471b40 con 0x55a32c45b980 2024-05-10T11:11:56.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fe2d08b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a32c471470 con 0x55a32c457010 2024-05-10T11:11:56.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fe2d08b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a32c472210 con 0x55a32c457930 2024-05-10T11:11:56.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fe2d08b80 10 client.0 _ll_register_callbacks cb 0x55a32c350c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:11:56.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fd6400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c4797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fd5a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 0x55a32c45fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fd5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 0x55a32c460260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fd6400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c4797c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35712/0 (socket says 172.21.15.27:35712) 2024-05-10T11:11:56.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fd6400700 1 -- 172.21.15.27:0/2776524467 learned_addr learned my addr 172.21.15.27:0/2776524467 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:11:56.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.619+0000 7f8fd5a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 0x55a32c45fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51990/0 (socket says 172.21.15.27:51990) 2024-05-10T11:11:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.620+0000 7f8fd5a00700 1 -- 172.21.15.27:0/2776524467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 msgr2=0x55a32c460260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.620+0000 7f8fd5a00700 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 0x55a32c460260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.620+0000 7f8fd5a00700 1 -- 172.21.15.27:0/2776524467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 msgr2=0x55a32c4797c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:11:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.620+0000 7f8fd5a00700 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c4797c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.620+0000 7f8fd5a00700 1 -- 172.21.15.27:0/2776524467 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a32c460b30 con 0x55a32c457010 2024-05-10T11:11:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.620+0000 7f8fd5000700 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 0x55a32c460260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.620+0000 7f8fd6400700 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c4797c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:11:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.620+0000 7f8fd5a00700 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 0x55a32c45fd20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4019c90 tx=0x7f8fc4005df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58ef41e0a3e08fdd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:11:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.620+0000 7f8fb7400700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.620+0000 7f8fb7400700 1 -- 172.21.15.27:0/2776524467 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4027040 con 0x55a32c457010 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.621+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2776524467 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a32c3b73b0 con 0x55a32c457010 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.621+0000 7f8fb7400700 1 -- 172.21.15.27:0/2776524467 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f8fc400f480 con 0x55a32c457010 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.621+0000 7f8fb7400700 1 -- 172.21.15.27:0/2776524467 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8fc4043050 con 0x55a32c457010 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.621+0000 7f8fe2d08b80 10 client.25031 fetch_fsmap learned FSMap version 29 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.621+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2776524467 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a32c4dc300 con 0x55a32c457010 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.621+0000 7f8fb7400700 1 -- 172.21.15.27:0/2776524467 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a32c3b73b0 con 0x55a32c457010 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.621+0000 7f8fb7400700 1 -- 172.21.15.27:0/2776524467 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8fc4003920 con 0x55a32c457010 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.622+0000 7f8fe2d08b80 10 client.25031 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.622+0000 7f8fe2d08b80 10 client.25031 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.622+0000 7f8fe2d08b80 20 client.25031 populate_metadata read hostname 'smithi027' 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.622+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a32c4e0db0 0x55a32c4dfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.622+0000 7f8fe2d08b80 4 client.25031 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-10T11:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.622+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2776524467 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "status"}) -- 0x55a32c3b1450 con 0x55a32c4e0db0 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.622+0000 7f8fd5000700 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a32c4e0db0 0x55a32c4dfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fd5000700 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a32c4e0db0 0x55a32c4dfac0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=940ac19f4c32fa01 server_cookie=37cb1c951058454 in_seq=0 out_seq=0 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fb7400700 10 client.25031 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fb7400700 1 -- 172.21.15.27:0/2776524467 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a32c3b1450 con 0x55a32c4e0db0 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fb7400700 10 client.25031 handle_command_reply: tid=0 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fe2d08b80 1 client.25031 shutdown 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2776524467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 msgr2=0x55a32c45fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 0x55a32c45fd20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4019c90 tx=0x7f8fc4005df0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2776524467 shutdown_connections 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a32c45b980 0x55a32c4797c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a32c457010 0x55a32c45fd20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a32c4e0db0 0x55a32c4dfac0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fe2d08b80 1 --2- 172.21.15.27:0/2776524467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a32c457930 0x55a32c460260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2776524467 >> 172.21.15.27:0/2776524467 conn(0x55a32c385b10 msgr2=0x55a32c44b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fb7400700 0 client.25031 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:11:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2776524467 shutdown_connections 2024-05-10T11:11:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.623+0000 7f8fe2d08b80 1 -- 172.21.15.27:0/2776524467 wait complete. 2024-05-10T11:11:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.624+0000 7f8fe2d08b80 20 client.25031 trim_cache size 0 max 16384 2024-05-10T11:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T11:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 1, 2024-05-10T11:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24488, 2024-05-10T11:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-05-10T11:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-05-10T11:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-05-10T11:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 164.65380155700001, 2024-05-10T11:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 29, 2024-05-10T11:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 90, 2024-05-10T11:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 88, 2024-05-10T11:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 285.25477424500002 2024-05-10T11:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:11:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.624+0000 7f8fd8400700 1 -- 172.21.15.27:0/2879174160 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f8fb80b2880 msgr2=0x7f8fb80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.624+0000 7f8fd8400700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f8fb80b2880 0x7f8fb80b4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009e60 tx=0x7f8fc8005040 comp rx=0 tx=0).stop 2024-05-10T11:11:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.624+0000 7f8fd8400700 1 -- 172.21.15.27:0/2879174160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0104100 msgr2=0x7f8fd019c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:11:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.624+0000 7f8fd8400700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0104100 0x7f8fd019c2d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00eb30 tx=0x7f8fbc00ee40 comp rx=0 tx=0).stop 2024-05-10T11:11:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.624+0000 7f8fd8400700 1 -- 172.21.15.27:0/2879174160 shutdown_connections 2024-05-10T11:11:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.625+0000 7f8fd8400700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f8fb80b2880 0x7f8fb80b4d40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.625+0000 7f8fd8400700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd0108850 0x7f8fd01a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.625+0000 7f8fd8400700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fd0103750 0x7f8fd019bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.625+0000 7f8fd8400700 1 --2- 172.21.15.27:0/2879174160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0104100 0x7f8fd019c2d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:11:56.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.625+0000 7f8fd8400700 1 -- 172.21.15.27:0/2879174160 >> 172.21.15.27:0/2879174160 conn(0x7f8fd007a600 msgr2=0x7f8fd0101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:11:56.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.625+0000 7f8fd8400700 1 -- 172.21.15.27:0/2879174160 shutdown_connections 2024-05-10T11:11:56.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:11:56.625+0000 7f8fd8400700 1 -- 172.21.15.27:0/2879174160 wait complete. 2024-05-10T11:11:57.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.b status 2024-05-10T11:11:58.182 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:11:58.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:11:58 smithi027 ceph-mon[30054]: pgmap v458: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:11:58 smithi076 ceph-mon[34152]: pgmap v458: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:11:58.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:11:58 smithi189 ceph-mon[34842]: pgmap v458: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:00.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:00 smithi027 ceph-mon[30054]: pgmap v459: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:00.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:00 smithi076 ceph-mon[34152]: pgmap v459: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:00.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:00 smithi189 ceph-mon[34842]: pgmap v459: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.579+0000 7ff291200700 1 -- 172.21.15.27:0/2078246016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff28c075ff0 msgr2=0x7ff28c10f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.579+0000 7ff291200700 1 --2- 172.21.15.27:0/2078246016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff28c075ff0 0x7ff28c10f960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff278009990 tx=0x7ff278009ca0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.579+0000 7ff291200700 1 -- 172.21.15.27:0/2078246016 shutdown_connections 2024-05-10T11:12:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.579+0000 7ff291200700 1 --2- 172.21.15.27:0/2078246016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10fea0 0x7ff28c112290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.579+0000 7ff291200700 1 --2- 172.21.15.27:0/2078246016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28c0756d0 0x7ff28c075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.579+0000 7ff291200700 1 --2- 172.21.15.27:0/2078246016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff28c075ff0 0x7ff28c10f960 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.579+0000 7ff291200700 1 -- 172.21.15.27:0/2078246016 >> 172.21.15.27:0/2078246016 conn(0x7ff28c07a600 msgr2=0x7ff28c07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.579+0000 7ff291200700 1 -- 172.21.15.27:0/2078246016 shutdown_connections 2024-05-10T11:12:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.579+0000 7ff291200700 1 -- 172.21.15.27:0/2078246016 wait complete. 2024-05-10T11:12:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.581+0000 7ff291200700 1 Processor -- start 2024-05-10T11:12:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.581+0000 7ff291200700 1 -- start start 2024-05-10T11:12:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.581+0000 7ff291200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28c0756d0 0x7ff28c19bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.581+0000 7ff291200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff28c075ff0 0x7ff28c19c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.581+0000 7ff291200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10fea0 0x7ff28c1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.581+0000 7ff291200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1158b0 con 0x7ff28c0756d0 2024-05-10T11:12:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff291200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c115730 con 0x7ff28c075ff0 2024-05-10T11:12:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff291200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c115a30 con 0x7ff28c10fea0 2024-05-10T11:12:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff289e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28c0756d0 0x7ff28c19bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff289400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff28c075ff0 0x7ff28c19c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff289e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28c0756d0 0x7ff28c19bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35732/0 (socket says 172.21.15.27:35732) 2024-05-10T11:12:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff289e00700 1 -- 172.21.15.27:0/4218025018 learned_addr learned my addr 172.21.15.27:0/4218025018 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff28a800700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10fea0 0x7ff28c1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff289400700 1 -- 172.21.15.27:0/4218025018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10fea0 msgr2=0x7ff28c1a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:00.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff289400700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10fea0 0x7ff28c1a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff289400700 1 -- 172.21.15.27:0/4218025018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28c0756d0 msgr2=0x7ff28c19bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff289400700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28c0756d0 0x7ff28c19bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff289400700 1 -- 172.21.15.27:0/4218025018 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28c1a0dd0 con 0x7ff28c075ff0 2024-05-10T11:12:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.582+0000 7ff289e00700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28c0756d0 0x7ff28c19bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.583+0000 7ff289400700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff28c075ff0 0x7ff28c19c300 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff278000c00 tx=0x7ff2780047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.583+0000 7ff282a00700 1 -- 172.21.15.27:0/4218025018 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278026070 con 0x7ff28c075ff0 2024-05-10T11:12:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.583+0000 7ff282a00700 1 -- 172.21.15.27:0/4218025018 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff278021410 con 0x7ff28c075ff0 2024-05-10T11:12:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.583+0000 7ff282a00700 1 -- 172.21.15.27:0/4218025018 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278017650 con 0x7ff28c075ff0 2024-05-10T11:12:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.583+0000 7ff291200700 1 -- 172.21.15.27:0/4218025018 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff28c1a1060 con 0x7ff28c075ff0 2024-05-10T11:12:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.583+0000 7ff291200700 1 -- 172.21.15.27:0/4218025018 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff28c1a8940 con 0x7ff28c075ff0 2024-05-10T11:12:00.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.587+0000 7ff282a00700 1 -- 172.21.15.27:0/4218025018 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7ff278029070 con 0x7ff28c075ff0 2024-05-10T11:12:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.588+0000 7ff29bbb1b80 1 Processor -- start 2024-05-10T11:12:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.588+0000 7ff29bbb1b80 1 -- start start 2024-05-10T11:12:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff29bbb1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622a010 0x5625b622a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff282a00700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff2700b2900 0x7ff2700b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff29bbb1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a930 0x5625b622e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff29bbb1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622e980 0x5625b622ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff29bbb1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5625b617c9d0 con 0x5625b622e980 2024-05-10T11:12:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff29bbb1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5625b611a710 con 0x5625b622a010 2024-05-10T11:12:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff29bbb1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5625b6178990 con 0x5625b622a930 2024-05-10T11:12:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff282a00700 1 -- 172.21.15.27:0/4218025018 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7ff2780e3230 con 0x7ff28c075ff0 2024-05-10T11:12:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff289e00700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff2700b2900 0x7ff2700b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff28a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622a010 0x5625b622a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff28a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622a010 0x5625b622a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52032/0 (socket says 172.21.15.27:52032) 2024-05-10T11:12:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff28a800700 1 -- 172.21.15.27:0/3543932291 learned_addr learned my addr 172.21.15.27:0/3543932291 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff28a800700 1 --2- 172.21.15.27:0/3543932291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622e980 0x5625b622ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.589+0000 7ff289400700 1 --2- 172.21.15.27:0/3543932291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a930 0x5625b622e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.590+0000 7ff289400700 1 -- 172.21.15.27:0/3543932291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622a010 msgr2=0x5625b622a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.590+0000 7ff289400700 1 --2- 172.21.15.27:0/3543932291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622a010 0x5625b622a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.590+0000 7ff289400700 1 -- 172.21.15.27:0/3543932291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622e980 msgr2=0x5625b622ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.590+0000 7ff289400700 1 --2- 172.21.15.27:0/3543932291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622e980 0x5625b622ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.590+0000 7ff289400700 1 -- 172.21.15.27:0/3543932291 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625b60c9880 con 0x5625b622a930 2024-05-10T11:12:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.590+0000 7ff289400700 1 --2- 172.21.15.27:0/3543932291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a930 0x5625b622e440 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff27800b2e0 tx=0x7ff278009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a59693a3eb26ba0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.590+0000 7ff289e00700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff2700b2900 0x7ff2700b4dc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff27c011630 tx=0x7ff27c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.590+0000 7ff277400700 1 -- 172.21.15.27:0/3543932291 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2780c4030 con 0x5625b622a930 2024-05-10T11:12:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.591+0000 7ff277400700 1 -- 172.21.15.27:0/3543932291 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2780a9fa0 con 0x5625b622a930 2024-05-10T11:12:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.591+0000 7ff277400700 1 -- 172.21.15.27:0/3543932291 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2780ae840 con 0x5625b622a930 2024-05-10T11:12:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.591+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/3543932291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a930 msgr2=0x5625b622e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.591+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/3543932291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a930 0x5625b622e440 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff27800b2e0 tx=0x7ff278009310 comp rx=0 tx=0).stop 2024-05-10T11:12:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.591+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/3543932291 shutdown_connections 2024-05-10T11:12:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.591+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/3543932291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622e980 0x5625b622ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.591+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/3543932291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622a010 0x5625b622a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.591+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/3543932291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a930 0x5625b622e440 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.591+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/3543932291 >> 172.21.15.27:0/3543932291 conn(0x5625b6158b10 msgr2=0x5625b6224750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.592+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/3543932291 shutdown_connections 2024-05-10T11:12:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.592+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/3543932291 wait complete. 2024-05-10T11:12:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.592+0000 7ff29bbb1b80 1 -- start start 2024-05-10T11:12:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.592+0000 7ff29bbb1b80 1 Processor -- start 2024-05-10T11:12:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.593+0000 7ff29bbb1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 0x5625b6235e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.593+0000 7ff29bbb1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a930 0x5625b62331f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.593+0000 7ff29bbb1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622e980 0x5625b6233730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.593+0000 7ff29bbb1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5625b6236950 con 0x5625b622a010 2024-05-10T11:12:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.593+0000 7ff29bbb1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5625b60e2ad0 con 0x5625b622e980 2024-05-10T11:12:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.593+0000 7ff29bbb1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5625b6236ad0 con 0x5625b622a930 2024-05-10T11:12:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.593+0000 7ff29bbb1b80 10 client.0 _ll_register_callbacks cb 0x5625b6123c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:12:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.593+0000 7ff289e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 0x5625b6235e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.593+0000 7ff289400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a930 0x5625b62331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.593+0000 7ff289e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 0x5625b6235e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35738/0 (socket says 172.21.15.27:35738) 2024-05-10T11:12:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.593+0000 7ff289e00700 1 -- 172.21.15.27:0/572902930 learned_addr learned my addr 172.21.15.27:0/572902930 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.594+0000 7ff28a800700 1 --2- 172.21.15.27:0/572902930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622e980 0x5625b6233730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.594+0000 7ff289e00700 1 -- 172.21.15.27:0/572902930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a930 msgr2=0x5625b62331f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.594+0000 7ff289e00700 1 --2- 172.21.15.27:0/572902930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a930 0x5625b62331f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.594+0000 7ff289e00700 1 -- 172.21.15.27:0/572902930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622e980 msgr2=0x5625b6233730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.594+0000 7ff289e00700 1 --2- 172.21.15.27:0/572902930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622e980 0x5625b6233730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.594+0000 7ff289e00700 1 -- 172.21.15.27:0/572902930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625b61423d0 con 0x5625b622a010 2024-05-10T11:12:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.594+0000 7ff289e00700 1 --2- 172.21.15.27:0/572902930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 0x5625b6235e50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff27c01da70 tx=0x7ff27c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129ab7240c70b124 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.594+0000 7ff26be00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:12:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.595+0000 7ff26be00700 1 -- 172.21.15.27:0/572902930 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c01d250 con 0x5625b622a010 2024-05-10T11:12:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.595+0000 7ff26be00700 1 -- 172.21.15.27:0/572902930 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7ff27c01b7b0 con 0x5625b622a010 2024-05-10T11:12:00.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.595+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/572902930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2780a9fa0 con 0x5625b622a010 2024-05-10T11:12:00.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.595+0000 7ff26be00700 1 -- 172.21.15.27:0/572902930 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff27c01b9b0 con 0x5625b622a010 2024-05-10T11:12:00.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.595+0000 7ff29bbb1b80 10 client.14919 fetch_fsmap learned FSMap version 29 2024-05-10T11:12:00.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.595+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/572902930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625b62411a0 con 0x5625b622a010 2024-05-10T11:12:00.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.596+0000 7ff26be00700 1 -- 172.21.15.27:0/572902930 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2780a9fa0 con 0x5625b622a010 2024-05-10T11:12:00.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.596+0000 7ff26be00700 1 -- 172.21.15.27:0/572902930 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff27c003c20 con 0x5625b622a010 2024-05-10T11:12:00.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.596+0000 7ff29bbb1b80 10 client.14919 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:12:00.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.596+0000 7ff29bbb1b80 10 client.14919 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-10T11:12:00.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.596+0000 7ff29bbb1b80 20 client.14919 populate_metadata read hostname 'smithi027' 2024-05-10T11:12:00.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.596+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/572902930 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5625b62437e0 0x5625b6244c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.596+0000 7ff29bbb1b80 4 client.14919 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:12:00.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.596+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/572902930 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625b61f7050 con 0x5625b62437e0 2024-05-10T11:12:00.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.596+0000 7ff289400700 1 --2- 172.21.15.27:0/572902930 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5625b62437e0 0x5625b6244c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.597+0000 7ff289400700 1 --2- 172.21.15.27:0/572902930 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5625b62437e0 0x5625b6244c50 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d126d923d65b976f server_cookie=82fffdd9dce7ff49 in_seq=0 out_seq=0 2024-05-10T11:12:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.597+0000 7ff26be00700 10 client.14919 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:12:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.598+0000 7ff26be00700 1 -- 172.21.15.27:0/572902930 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5625b61f7050 con 0x5625b62437e0 2024-05-10T11:12:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.598+0000 7ff26be00700 10 client.14919 handle_command_reply: tid=0 2024-05-10T11:12:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.598+0000 7ff29bbb1b80 1 client.14919 shutdown 2024-05-10T11:12:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.598+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/572902930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 msgr2=0x5625b6235e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:00.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.598+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/572902930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 0x5625b6235e50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff27c01da70 tx=0x7ff27c0047c0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.598+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/572902930 shutdown_connections 2024-05-10T11:12:00.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.598+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/572902930 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5625b62437e0 0x5625b6244c50 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.598+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/572902930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622e980 0x5625b6233730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.598+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/572902930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 0x5625b6235e50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.598+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/572902930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a930 0x5625b62331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.598+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/572902930 >> 172.21.15.27:0/572902930 conn(0x5625b6158b10 msgr2=0x5625b621f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:00.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.598+0000 7ff26be00700 0 client.14919 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:12:00.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.599+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/572902930 shutdown_connections 2024-05-10T11:12:00.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.599+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/572902930 wait complete. 2024-05-10T11:12:00.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.599+0000 7ff29bbb1b80 20 client.14919 trim_cache size 0 max 16384 2024-05-10T11:12:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.622+0000 7ff29bbb1b80 1 Processor -- start 2024-05-10T11:12:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.622+0000 7ff29bbb1b80 1 -- start start 2024-05-10T11:12:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.622+0000 7ff29bbb1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a010 0x5625b6243110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.622+0000 7ff29bbb1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a930 0x5625b621f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.622+0000 7ff29bbb1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622e980 0x5625b624e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.622+0000 7ff29bbb1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5625b6242f90 con 0x5625b622a930 2024-05-10T11:12:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.622+0000 7ff29bbb1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5625b6243660 con 0x5625b622e980 2024-05-10T11:12:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.622+0000 7ff29bbb1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5625b6243d30 con 0x5625b622a010 2024-05-10T11:12:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.623+0000 7ff28a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622e980 0x5625b624e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.623+0000 7ff289e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a010 0x5625b6243110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.623+0000 7ff289400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a930 0x5625b621f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.623+0000 7ff28a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622e980 0x5625b624e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52054/0 (socket says 172.21.15.27:52054) 2024-05-10T11:12:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.623+0000 7ff289e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a010 0x5625b6243110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:57526/0 (socket says 172.21.15.27:57526) 2024-05-10T11:12:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.623+0000 7ff28a800700 1 -- 172.21.15.27:0/924822546 learned_addr learned my addr 172.21.15.27:0/924822546 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.623+0000 7ff28a800700 1 -- 172.21.15.27:0/924822546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a010 msgr2=0x5625b6243110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.623+0000 7ff28a800700 1 --2- 172.21.15.27:0/924822546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a010 0x5625b6243110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.623+0000 7ff28a800700 1 -- 172.21.15.27:0/924822546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a930 msgr2=0x5625b621f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.623+0000 7ff28a800700 1 --2- 172.21.15.27:0/924822546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a930 0x5625b621f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.623+0000 7ff28a800700 1 -- 172.21.15.27:0/924822546 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625b62213a0 con 0x5625b622e980 2024-05-10T11:12:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.624+0000 7ff28a800700 1 --2- 172.21.15.27:0/924822546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622e980 0x5625b624e0b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff284008440 tx=0x7ff284004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a9b72b8fca59a7c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.624+0000 7ff276a00700 1 -- 172.21.15.27:0/924822546 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff284003cd0 con 0x5625b622e980 2024-05-10T11:12:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.624+0000 7ff276a00700 1 -- 172.21.15.27:0/924822546 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff28401f030 con 0x5625b622e980 2024-05-10T11:12:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.624+0000 7ff276a00700 1 -- 172.21.15.27:0/924822546 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff284007760 con 0x5625b622e980 2024-05-10T11:12:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.625+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/924822546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622e980 msgr2=0x5625b624e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.625+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/924822546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622e980 0x5625b624e0b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff284008440 tx=0x7ff284004b40 comp rx=0 tx=0).stop 2024-05-10T11:12:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.625+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/924822546 shutdown_connections 2024-05-10T11:12:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.625+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/924822546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622a010 0x5625b6243110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.625+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/924822546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622e980 0x5625b624e0b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.625+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/924822546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a930 0x5625b621f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.625+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/924822546 >> 172.21.15.27:0/924822546 conn(0x5625b6158b10 msgr2=0x5625b621e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.625+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/924822546 shutdown_connections 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.625+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/924822546 wait complete. 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.625+0000 7ff29bbb1b80 1 -- start start 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.626+0000 7ff29bbb1b80 1 Processor -- start 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.626+0000 7ff29bbb1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 0x5625b624e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.626+0000 7ff29bbb1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622a930 0x5625b6249840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.626+0000 7ff29bbb1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622e980 0x5625b6249d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.626+0000 7ff29bbb1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5625b6244ad0 con 0x5625b622a010 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.626+0000 7ff29bbb1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5625b6244400 con 0x5625b622a930 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.626+0000 7ff29bbb1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5625b62451a0 con 0x5625b622e980 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.626+0000 7ff29bbb1b80 10 client.0 _ll_register_callbacks cb 0x5625b6123c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff289e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 0x5625b624e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff289e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 0x5625b624e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35754/0 (socket says 172.21.15.27:35754) 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff289e00700 1 -- 172.21.15.27:0/2861200482 learned_addr learned my addr 172.21.15.27:0/2861200482 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff289400700 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622a930 0x5625b6249840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff28a800700 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622e980 0x5625b6249d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff289400700 1 -- 172.21.15.27:0/2861200482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622e980 msgr2=0x5625b6249d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff289400700 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622e980 0x5625b6249d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff289400700 1 -- 172.21.15.27:0/2861200482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 msgr2=0x5625b624e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff289400700 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 0x5625b624e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff289400700 1 -- 172.21.15.27:0/2861200482 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625b624a480 con 0x5625b622a930 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff289e00700 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 0x5625b624e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff289400700 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622a930 0x5625b6249840 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff2780aa190 tx=0x7ff27800b310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc9d48a021a4409f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.627+0000 7ff26be00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.628+0000 7ff26be00700 1 -- 172.21.15.27:0/2861200482 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2780c4040 con 0x5625b622a930 2024-05-10T11:12:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.628+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/2861200482 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff28401f030 con 0x5625b622a930 2024-05-10T11:12:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.628+0000 7ff26be00700 1 -- 172.21.15.27:0/2861200482 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7ff2780218c0 con 0x5625b622a930 2024-05-10T11:12:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.628+0000 7ff26be00700 1 -- 172.21.15.27:0/2861200482 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2780a9fa0 con 0x5625b622a930 2024-05-10T11:12:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.628+0000 7ff29bbb1b80 10 client.25049 fetch_fsmap learned FSMap version 29 2024-05-10T11:12:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.628+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/2861200482 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625b62aeb60 con 0x5625b622a930 2024-05-10T11:12:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.628+0000 7ff26be00700 1 -- 172.21.15.27:0/2861200482 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff28401f030 con 0x5625b622a930 2024-05-10T11:12:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.628+0000 7ff26be00700 1 -- 172.21.15.27:0/2861200482 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff2780c48c0 con 0x5625b622a930 2024-05-10T11:12:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.629+0000 7ff29bbb1b80 10 client.25049 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:12:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.629+0000 7ff29bbb1b80 10 client.25049 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-10T11:12:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.629+0000 7ff29bbb1b80 20 client.25049 populate_metadata read hostname 'smithi027' 2024-05-10T11:12:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.629+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5625b62b35f0 0x5625b62b2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.629+0000 7ff29bbb1b80 4 client.25049 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-10T11:12:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.629+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/2861200482 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "status"}) -- 0x5625b618a3b0 con 0x5625b62b35f0 2024-05-10T11:12:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.629+0000 7ff289e00700 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5625b62b35f0 0x5625b62b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff26be00700 10 client.25049 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:12:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff289e00700 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5625b62b35f0 0x5625b62b2300 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4234dfb931c5a0fc server_cookie=c585a597fc620109 in_seq=0 out_seq=0 2024-05-10T11:12:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff26be00700 1 -- 172.21.15.27:0/2861200482 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5625b618a3b0 con 0x5625b62b35f0 2024-05-10T11:12:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff26be00700 10 client.25049 handle_command_reply: tid=0 2024-05-10T11:12:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff29bbb1b80 1 client.25049 shutdown 2024-05-10T11:12:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/2861200482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622a930 msgr2=0x5625b6249840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622a930 0x5625b6249840 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff2780aa190 tx=0x7ff27800b310 comp rx=0 tx=0).stop 2024-05-10T11:12:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/2861200482 shutdown_connections 2024-05-10T11:12:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5625b62b35f0 0x5625b62b2300 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625b622e980 0x5625b6249d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625b622a010 0x5625b624e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff29bbb1b80 1 --2- 172.21.15.27:0/2861200482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5625b622a930 0x5625b6249840 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/2861200482 >> 172.21.15.27:0/2861200482 conn(0x5625b6158b10 msgr2=0x5625b6228e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:00.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff26be00700 0 client.25049 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:12:00.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/2861200482 shutdown_connections 2024-05-10T11:12:00.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff29bbb1b80 1 -- 172.21.15.27:0/2861200482 wait complete. 2024-05-10T11:12:00.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.630+0000 7ff29bbb1b80 20 client.25049 trim_cache size 0 max 16384 2024-05-10T11:12:00.636 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:12:00.636 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T11:12:00.636 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-05-10T11:12:00.636 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24527, 2024-05-10T11:12:00.637 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-05-10T11:12:00.637 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-05-10T11:12:00.637 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-05-10T11:12:00.637 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 250.338050993, 2024-05-10T11:12:00.637 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 29, 2024-05-10T11:12:00.637 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 90, 2024-05-10T11:12:00.637 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 88, 2024-05-10T11:12:00.637 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 274.09640391699998 2024-05-10T11:12:00.637 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:12:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.631+0000 7ff291200700 1 -- 172.21.15.27:0/4218025018 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff2700b2900 msgr2=0x7ff2700b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.631+0000 7ff291200700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff2700b2900 0x7ff2700b4dc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff27c011630 tx=0x7ff27c009260 comp rx=0 tx=0).stop 2024-05-10T11:12:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.631+0000 7ff291200700 1 -- 172.21.15.27:0/4218025018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff28c075ff0 msgr2=0x7ff28c19c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.631+0000 7ff291200700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff28c075ff0 0x7ff28c19c300 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff278000c00 tx=0x7ff2780047b0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.631+0000 7ff291200700 1 -- 172.21.15.27:0/4218025018 shutdown_connections 2024-05-10T11:12:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.631+0000 7ff291200700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7ff2700b2900 0x7ff2700b4dc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.631+0000 7ff291200700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10fea0 0x7ff28c1a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.631+0000 7ff291200700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28c0756d0 0x7ff28c19bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.631+0000 7ff291200700 1 --2- 172.21.15.27:0/4218025018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff28c075ff0 0x7ff28c19c300 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.631+0000 7ff291200700 1 -- 172.21.15.27:0/4218025018 >> 172.21.15.27:0/4218025018 conn(0x7ff28c07a600 msgr2=0x7ff28c100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.631+0000 7ff291200700 1 -- 172.21.15.27:0/4218025018 shutdown_connections 2024-05-10T11:12:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:00.632+0000 7ff291200700 1 -- 172.21.15.27:0/4218025018 wait complete. 2024-05-10T11:12:01.782 INFO:tasks.cephadm:Setup complete, yielding 2024-05-10T11:12:01.783 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-10T11:12:01.795 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-05-10T11:12:01.796 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- bash -c 'ceph orch status' 2024-05-10T11:12:02.266 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:02.417 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:02 smithi027 ceph-mon[30054]: pgmap v460: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:02.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:02 smithi076 ceph-mon[34152]: pgmap v460: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:02.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:02 smithi189 ceph-mon[34842]: pgmap v460: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:04.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:04 smithi027 ceph-mon[30054]: pgmap v461: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:04.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:04 smithi076 ceph-mon[34152]: pgmap v461: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:04.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:04 smithi189 ceph-mon[34842]: pgmap v461: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.612+0000 7f17fec00700 1 -- 172.21.15.27:0/4199947484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8105640 msgr2=0x7f17f8105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.612+0000 7f17fec00700 1 --2- 172.21.15.27:0/4199947484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8105640 0x7f17f8105a60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0099c0 tx=0x7f17ec009cd0 comp rx=0 tx=0).stop 2024-05-10T11:12:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.613+0000 7f17fec00700 1 -- 172.21.15.27:0/4199947484 shutdown_connections 2024-05-10T11:12:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.613+0000 7f17fec00700 1 --2- 172.21.15.27:0/4199947484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f81071e0 0x7f17f810da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.613+0000 7f17fec00700 1 --2- 172.21.15.27:0/4199947484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8105640 0x7f17f8105a60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.613+0000 7f17fec00700 1 --2- 172.21.15.27:0/4199947484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17f8106840 0x7f17f8106ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.613+0000 7f17fec00700 1 -- 172.21.15.27:0/4199947484 >> 172.21.15.27:0/4199947484 conn(0x7f17f8100be0 msgr2=0x7f17f8103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.613+0000 7f17fec00700 1 -- 172.21.15.27:0/4199947484 shutdown_connections 2024-05-10T11:12:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.613+0000 7f17fec00700 1 -- 172.21.15.27:0/4199947484 wait complete. 2024-05-10T11:12:04.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.614+0000 7f17fec00700 1 Processor -- start 2024-05-10T11:12:04.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.615+0000 7f17fec00700 1 -- start start 2024-05-10T11:12:04.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.615+0000 7f17fec00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f8105640 0x7f17f819b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:04.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.615+0000 7f17fec00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8106840 0x7f17f819bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:04.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.615+0000 7f17fec00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17f81071e0 0x7f17f81a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:04.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.615+0000 7f17fec00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f17f8111130 con 0x7f17f8106840 2024-05-10T11:12:04.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.615+0000 7f17fec00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f17f8110fb0 con 0x7f17f81071e0 2024-05-10T11:12:04.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.615+0000 7f17fec00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81112b0 con 0x7f17f8105640 2024-05-10T11:12:04.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.615+0000 7f17fce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8106840 0x7f17f819bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:04.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.615+0000 7f17fd800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f8105640 0x7f17f819b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:04.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.615+0000 7f17fce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8106840 0x7f17f819bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50498/0 (socket says 172.21.15.27:50498) 2024-05-10T11:12:04.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.615+0000 7f17fd800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f8105640 0x7f17f819b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:33222/0 (socket says 172.21.15.27:33222) 2024-05-10T11:12:04.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.615+0000 7f17fce00700 1 -- 172.21.15.27:0/1156163574 learned_addr learned my addr 172.21.15.27:0/1156163574 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:04.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.616+0000 7f17fe200700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17f81071e0 0x7f17f81a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:04.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.616+0000 7f17fd800700 1 -- 172.21.15.27:0/1156163574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17f81071e0 msgr2=0x7f17f81a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:04.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.616+0000 7f17fd800700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17f81071e0 0x7f17f81a2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:04.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.616+0000 7f17fd800700 1 -- 172.21.15.27:0/1156163574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8106840 msgr2=0x7f17f819bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:04.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.616+0000 7f17fd800700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8106840 0x7f17f819bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:04.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.616+0000 7f17fd800700 1 -- 172.21.15.27:0/1156163574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f81a3400 con 0x7f17f8105640 2024-05-10T11:12:04.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.616+0000 7f17fe200700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17f81071e0 0x7f17f81a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:12:04.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.616+0000 7f17fce00700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8106840 0x7f17f819bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:04.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.616+0000 7f17fd800700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f8105640 0x7f17f819b7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0099c0 tx=0x7f17ec003af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:04.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.617+0000 7f17f6000700 1 -- 172.21.15.27:0/1156163574 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ec0050c0 con 0x7f17f8105640 2024-05-10T11:12:04.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.617+0000 7f17f6000700 1 -- 172.21.15.27:0/1156163574 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17ec00f460 con 0x7f17f8105640 2024-05-10T11:12:04.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.617+0000 7f17fec00700 1 -- 172.21.15.27:0/1156163574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f81a3690 con 0x7f17f8105640 2024-05-10T11:12:04.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.617+0000 7f17f6000700 1 -- 172.21.15.27:0/1156163574 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ec017670 con 0x7f17f8105640 2024-05-10T11:12:04.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.617+0000 7f17fec00700 1 -- 172.21.15.27:0/1156163574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f81a3ae0 con 0x7f17f8105640 2024-05-10T11:12:04.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.618+0000 7f17fec00700 1 -- 172.21.15.27:0/1156163574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f806a190 con 0x7f17f8105640 2024-05-10T11:12:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.620+0000 7f17f6000700 1 -- 172.21.15.27:0/1156163574 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f17ec00f610 con 0x7f17f8105640 2024-05-10T11:12:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.622+0000 7f17f6000700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f17e00b2880 0x7f17e00b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.622+0000 7f17f6000700 1 -- 172.21.15.27:0/1156163574 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f17ec0e3430 con 0x7f17f8105640 2024-05-10T11:12:04.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.622+0000 7f17fce00700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f17e00b2880 0x7f17e00b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:04.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.623+0000 7f17fce00700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f17e00b2880 0x7f17e00b4d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f17e4011550 tx=0x7f17e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:04.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.624+0000 7f17f6000700 1 -- 172.21.15.27:0/1156163574 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f17ec013070 con 0x7f17f8105640 2024-05-10T11:12:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.884+0000 7f17fec00700 1 -- 172.21.15.27:0/1156163574 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f17f80638e0 con 0x7f17e00b2880 2024-05-10T11:12:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.886+0000 7f17f6000700 1 -- 172.21.15.27:0/1156163574 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f17f80638e0 con 0x7f17e00b2880 2024-05-10T11:12:04.887 INFO:teuthology.orchestra.run.smithi027.stdout:Backend: cephadm 2024-05-10T11:12:04.887 INFO:teuthology.orchestra.run.smithi027.stdout:Available: Yes 2024-05-10T11:12:04.887 INFO:teuthology.orchestra.run.smithi027.stdout:Paused: No 2024-05-10T11:12:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.890+0000 7f17fec00700 1 -- 172.21.15.27:0/1156163574 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f17e00b2880 msgr2=0x7f17e00b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.890+0000 7f17fec00700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f17e00b2880 0x7f17e00b4d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f17e4011550 tx=0x7f17e4009260 comp rx=0 tx=0).stop 2024-05-10T11:12:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.890+0000 7f17fec00700 1 -- 172.21.15.27:0/1156163574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f8105640 msgr2=0x7f17f819b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.890+0000 7f17fec00700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f8105640 0x7f17f819b7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0099c0 tx=0x7f17ec003af0 comp rx=0 tx=0).stop 2024-05-10T11:12:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.890+0000 7f17fec00700 1 -- 172.21.15.27:0/1156163574 shutdown_connections 2024-05-10T11:12:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.890+0000 7f17fec00700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f17e00b2880 0x7f17e00b4d40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.890+0000 7f17fec00700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f8105640 0x7f17f819b7a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.890+0000 7f17fec00700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17f81071e0 0x7f17f81a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.890+0000 7f17fec00700 1 --2- 172.21.15.27:0/1156163574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8106840 0x7f17f819bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.890+0000 7f17fec00700 1 -- 172.21.15.27:0/1156163574 >> 172.21.15.27:0/1156163574 conn(0x7f17f8100be0 msgr2=0x7f17f8109bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.890+0000 7f17fec00700 1 -- 172.21.15.27:0/1156163574 shutdown_connections 2024-05-10T11:12:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:04.890+0000 7f17fec00700 1 -- 172.21.15.27:0/1156163574 wait complete. 2024-05-10T11:12:05.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-10T11:12:06.291 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:06.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:06 smithi027 ceph-mon[30054]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:06.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:06 smithi027 ceph-mon[30054]: pgmap v462: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:06.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:06 smithi076 ceph-mon[34152]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:06.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:06 smithi076 ceph-mon[34152]: pgmap v462: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:06.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:06 smithi189 ceph-mon[34842]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:06.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:06 smithi189 ceph-mon[34842]: pgmap v462: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:08.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:08 smithi027 ceph-mon[30054]: pgmap v463: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:08.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:08 smithi076 ceph-mon[34152]: pgmap v463: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:08.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:08 smithi189 ceph-mon[34842]: pgmap v463: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.787+0000 7f255e800700 1 -- 172.21.15.27:0/1503953365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25580774d0 msgr2=0x7f2558075930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.787+0000 7f255e800700 1 --2- 172.21.15.27:0/1503953365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25580774d0 0x7f2558075930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f254c00b0a0 tx=0x7f254c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:12:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.787+0000 7f255e800700 1 -- 172.21.15.27:0/1503953365 shutdown_connections 2024-05-10T11:12:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.787+0000 7f255e800700 1 --2- 172.21.15.27:0/1503953365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f255810b2f0 0x7f255810d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.787+0000 7f255e800700 1 --2- 172.21.15.27:0/1503953365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25580774d0 0x7f2558075930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.787+0000 7f255e800700 1 --2- 172.21.15.27:0/1503953365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2558075f00 0x7f2558076380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.787+0000 7f255e800700 1 -- 172.21.15.27:0/1503953365 >> 172.21.15.27:0/1503953365 conn(0x7f25580fe9e0 msgr2=0x7f2558100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.787+0000 7f255e800700 1 -- 172.21.15.27:0/1503953365 shutdown_connections 2024-05-10T11:12:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.787+0000 7f255e800700 1 -- 172.21.15.27:0/1503953365 wait complete. 2024-05-10T11:12:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.788+0000 7f255e800700 1 Processor -- start 2024-05-10T11:12:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.788+0000 7f255e800700 1 -- start start 2024-05-10T11:12:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.788+0000 7f255e800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2558075f00 0x7f255819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.788+0000 7f255e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25580774d0 0x7f255819bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f255810b2f0 0x7f25581a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255e800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f25581110e0 con 0x7f255810b2f0 2024-05-10T11:12:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255e800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2558110f60 con 0x7f2558075f00 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255e800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2558111260 con 0x7f25580774d0 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2558075f00 0x7f255819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f255810b2f0 0x7f25581a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f255810b2f0 0x7f25581a2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50518/0 (socket says 172.21.15.27:50518) 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2558075f00 0x7f255819b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33328/0 (socket says 172.21.15.27:33328) 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255de00700 1 -- 172.21.15.27:0/1585734421 learned_addr learned my addr 172.21.15.27:0/1585734421 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255ca00700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25580774d0 0x7f255819bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255d400700 1 -- 172.21.15.27:0/1585734421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25580774d0 msgr2=0x7f255819bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255d400700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25580774d0 0x7f255819bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255d400700 1 -- 172.21.15.27:0/1585734421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f255810b2f0 msgr2=0x7f25581a2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255d400700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f255810b2f0 0x7f25581a2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.789+0000 7f255d400700 1 -- 172.21.15.27:0/1585734421 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25581a3410 con 0x7f2558075f00 2024-05-10T11:12:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.790+0000 7f255d400700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2558075f00 0x7f255819b800 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f254c00d010 tx=0x7f254c004d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.790+0000 7f2556000700 1 -- 172.21.15.27:0/1585734421 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c021070 con 0x7f2558075f00 2024-05-10T11:12:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.790+0000 7f255e800700 1 -- 172.21.15.27:0/1585734421 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25581a36a0 con 0x7f2558075f00 2024-05-10T11:12:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.790+0000 7f2556000700 1 -- 172.21.15.27:0/1585734421 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f254c007c40 con 0x7f2558075f00 2024-05-10T11:12:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.790+0000 7f2556000700 1 -- 172.21.15.27:0/1585734421 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c0052a0 con 0x7f2558075f00 2024-05-10T11:12:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.790+0000 7f255e800700 1 -- 172.21.15.27:0/1585734421 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25581a3be0 con 0x7f2558075f00 2024-05-10T11:12:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.791+0000 7f255e800700 1 -- 172.21.15.27:0/1585734421 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f255806a190 con 0x7f2558075f00 2024-05-10T11:12:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.794+0000 7f2556000700 1 -- 172.21.15.27:0/1585734421 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f254c003a00 con 0x7f2558075f00 2024-05-10T11:12:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.795+0000 7f2556000700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f25400b2880 0x7f25400b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.795+0000 7f2556000700 1 -- 172.21.15.27:0/1585734421 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f254c00f020 con 0x7f2558075f00 2024-05-10T11:12:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.796+0000 7f255ca00700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f25400b2880 0x7f25400b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.796+0000 7f255ca00700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f25400b2880 0x7f25400b4d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2558077200 tx=0x7f254400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:08.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:08.797+0000 7f2556000700 1 -- 172.21.15.27:0/1585734421 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f254c02a020 con 0x7f2558075f00 2024-05-10T11:12:09.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.043+0000 7f255e800700 1 -- 172.21.15.27:0/1585734421 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f25580638e0 con 0x7f25400b2880 2024-05-10T11:12:09.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.094+0000 7f2556000700 1 -- 172.21.15.27:0/1585734421 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f25580638e0 con 0x7f25400b2880 2024-05-10T11:12:09.095 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-10T11:12:09.095 INFO:teuthology.orchestra.run.smithi027.stdout:mds.a smithi027 running (4m) 2m ago 4m 14.7M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 4808f01523b2 2024-05-10T11:12:09.095 INFO:teuthology.orchestra.run.smithi027.stdout:mds.b smithi076 running (4m) 2m ago 4m 25.6M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 46f39929fdc3 2024-05-10T11:12:09.095 INFO:teuthology.orchestra.run.smithi027.stdout:mds.c smithi189 running (5m) 2m ago 5m 16.6M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 288fb75edf64 2024-05-10T11:12:09.095 INFO:teuthology.orchestra.run.smithi027.stdout:mds.d smithi027 running (5m) 2m ago 5m 13.1M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 b779ebe9d1e0 2024-05-10T11:12:09.095 INFO:teuthology.orchestra.run.smithi027.stdout:mds.e smithi076 running (5m) 2m ago 5m 19.9M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 59efe649ffc0 2024-05-10T11:12:09.095 INFO:teuthology.orchestra.run.smithi027.stdout:mds.f smithi189 running (5m) 2m ago 5m 16.5M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 ce5743ad0fa2 2024-05-10T11:12:09.095 INFO:teuthology.orchestra.run.smithi027.stdout:mds.g smithi027 running (4m) 2m ago 4m 13.0M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 090e76e23145 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.h smithi076 running (4m) 2m ago 4m 17.4M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 d0eb84fcde18 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.i smithi189 running (4m) 2m ago 4m 19.8M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 1d25da1e43c5 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.j smithi027 running (5m) 2m ago 5m 14.9M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 955b6fb80c9e 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.k smithi076 running (5m) 2m ago 5m 18.8M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 f712d0065147 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.l smithi189 running (5m) 2m ago 5m 16.5M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 25ea3d564c7e 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.x smithi027 *:9283,8765 running (13m) 2m ago 13m 531M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 6a7db7f10344 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.y smithi076 *:8443,8765 running (11m) 2m ago 11m 467M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 c4f46c9ba6de 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.z smithi189 *:8443,8765 running (11m) 2m ago 11m 470M - 19.0.0-3449-gd39fa088 ee5e4f21ef58 6f59165a3ad5 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:mon.a smithi027 running (13m) 2m ago 13m 50.1M 2048M 19.0.0-3449-gd39fa088 ee5e4f21ef58 36f5ec0cc8e0 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:mon.b smithi076 running (11m) 2m ago 11m 44.2M 2048M 19.0.0-3449-gd39fa088 ee5e4f21ef58 c141b324c977 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:mon.c smithi189 running (11m) 2m ago 11m 44.4M 2048M 19.0.0-3449-gd39fa088 ee5e4f21ef58 79bc0699541b 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (11m) 2m ago 11m 57.2M 4272M 19.0.0-3449-gd39fa088 ee5e4f21ef58 d5188f7b0a67 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi076 running (10m) 2m ago 10m 45.8M 4272M 19.0.0-3449-gd39fa088 ee5e4f21ef58 4ba0385fa0a9 2024-05-10T11:12:09.096 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi189 running (10m) 2m ago 10m 59.6M 4272M 19.0.0-3449-gd39fa088 ee5e4f21ef58 8877a5d1aae9 2024-05-10T11:12:09.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi027 running (9m) 2m ago 9m 54.3M 4272M 19.0.0-3449-gd39fa088 ee5e4f21ef58 257e59e73faf 2024-05-10T11:12:09.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi076 running (9m) 2m ago 9m 42.3M 4272M 19.0.0-3449-gd39fa088 ee5e4f21ef58 44ca8387f64c 2024-05-10T11:12:09.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi189 running (8m) 2m ago 8m 56.1M 4272M 19.0.0-3449-gd39fa088 ee5e4f21ef58 c11f46b41064 2024-05-10T11:12:09.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 smithi027 running (8m) 2m ago 8m 41.7M 4272M 19.0.0-3449-gd39fa088 ee5e4f21ef58 b9c4302f8adf 2024-05-10T11:12:09.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7 smithi076 running (7m) 2m ago 7m 44.7M 4272M 19.0.0-3449-gd39fa088 ee5e4f21ef58 99ee9eb90d3a 2024-05-10T11:12:09.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8 smithi189 running (7m) 2m ago 7m 46.0M 4272M 19.0.0-3449-gd39fa088 ee5e4f21ef58 ba987b32ed9f 2024-05-10T11:12:09.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 smithi027 running (6m) 2m ago 6m 41.9M 4272M 19.0.0-3449-gd39fa088 ee5e4f21ef58 c19611320898 2024-05-10T11:12:09.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10 smithi076 running (6m) 2m ago 6m 57.6M 4272M 19.0.0-3449-gd39fa088 ee5e4f21ef58 e668e2629b34 2024-05-10T11:12:09.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11 smithi189 running (5m) 2m ago 5m 53.5M 4272M 19.0.0-3449-gd39fa088 ee5e4f21ef58 c24dbd23f761 2024-05-10T11:12:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.097+0000 7f255e800700 1 -- 172.21.15.27:0/1585734421 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f25400b2880 msgr2=0x7f25400b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.097+0000 7f255e800700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f25400b2880 0x7f25400b4d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2558077200 tx=0x7f254400b410 comp rx=0 tx=0).stop 2024-05-10T11:12:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.097+0000 7f255e800700 1 -- 172.21.15.27:0/1585734421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2558075f00 msgr2=0x7f255819b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.097+0000 7f255e800700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2558075f00 0x7f255819b800 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f254c00d010 tx=0x7f254c004d70 comp rx=0 tx=0).stop 2024-05-10T11:12:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.097+0000 7f255e800700 1 -- 172.21.15.27:0/1585734421 shutdown_connections 2024-05-10T11:12:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.097+0000 7f255e800700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f25400b2880 0x7f25400b4d40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.097+0000 7f255e800700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f255810b2f0 0x7f25581a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.097+0000 7f255e800700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2558075f00 0x7f255819b800 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.097+0000 7f255e800700 1 --2- 172.21.15.27:0/1585734421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25580774d0 0x7f255819bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.097+0000 7f255e800700 1 -- 172.21.15.27:0/1585734421 >> 172.21.15.27:0/1585734421 conn(0x7f25580fe9e0 msgr2=0x7f25581071f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.097+0000 7f255e800700 1 -- 172.21.15.27:0/1585734421 shutdown_connections 2024-05-10T11:12:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:09.097+0000 7f255e800700 1 -- 172.21.15.27:0/1585734421 wait complete. 2024-05-10T11:12:09.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-10T11:12:10.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:10 smithi027 ceph-mon[30054]: pgmap v464: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:10.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:10 smithi027 ceph-mon[30054]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:10.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:10 smithi076 ceph-mon[34152]: pgmap v464: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:10.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:10 smithi076 ceph-mon[34152]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:10.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:10 smithi189 ceph-mon[34842]: pgmap v464: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:10.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:10 smithi189 ceph-mon[34842]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:12.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:12 smithi027 ceph-mon[30054]: pgmap v465: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:12.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:12 smithi076 ceph-mon[34152]: pgmap v465: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:12.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:12 smithi189 ceph-mon[34842]: pgmap v465: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.843+0000 7f74e7800700 1 -- 172.21.15.27:0/3083424818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e0105ce0 msgr2=0x7f74e010db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.843+0000 7f74e7800700 1 --2- 172.21.15.27:0/3083424818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e0105ce0 0x7f74e010db70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f74d800b0a0 tx=0x7f74d800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:12:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.843+0000 7f74e7800700 1 -- 172.21.15.27:0/3083424818 shutdown_connections 2024-05-10T11:12:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.843+0000 7f74e7800700 1 --2- 172.21.15.27:0/3083424818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e0105ce0 0x7f74e010db70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.843+0000 7f74e7800700 1 --2- 172.21.15.27:0/3083424818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74e0106e00 0x7f74e0107220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.843+0000 7f74e7800700 1 --2- 172.21.15.27:0/3083424818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e0105340 0x7f74e01057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.843+0000 7f74e7800700 1 -- 172.21.15.27:0/3083424818 >> 172.21.15.27:0/3083424818 conn(0x7f74e0100c80 msgr2=0x7f74e0103100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.843+0000 7f74e7800700 1 -- 172.21.15.27:0/3083424818 shutdown_connections 2024-05-10T11:12:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.843+0000 7f74e7800700 1 -- 172.21.15.27:0/3083424818 wait complete. 2024-05-10T11:12:12.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.844+0000 7f74e7800700 1 Processor -- start 2024-05-10T11:12:12.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.845+0000 7f74e7800700 1 -- start start 2024-05-10T11:12:12.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.845+0000 7f74e7800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e0105340 0x7f74e019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.845+0000 7f74e7800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74e0105ce0 0x7f74e019be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.845+0000 7f74e7800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e0106e00 0x7f74e01a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.845+0000 7f74e7800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f74e01112c0 con 0x7f74e0106e00 2024-05-10T11:12:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.845+0000 7f74e7800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f74e0111140 con 0x7f74e0105ce0 2024-05-10T11:12:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.845+0000 7f74e7800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74e0111440 con 0x7f74e0105340 2024-05-10T11:12:12.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.845+0000 7f74e4e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e0105340 0x7f74e019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:12.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.845+0000 7f74e5800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e0106e00 0x7f74e01a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:12.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.845+0000 7f74e4e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e0105340 0x7f74e019b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:33258/0 (socket says 172.21.15.27:33258) 2024-05-10T11:12:12.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.845+0000 7f74e4e00700 1 -- 172.21.15.27:0/2103173631 learned_addr learned my addr 172.21.15.27:0/2103173631 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:12.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.846+0000 7f74dfe00700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74e0105ce0 0x7f74e019be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:12.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.846+0000 7f74e4e00700 1 -- 172.21.15.27:0/2103173631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74e0105ce0 msgr2=0x7f74e019be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:12.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.846+0000 7f74e4e00700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74e0105ce0 0x7f74e019be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:12.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.846+0000 7f74e4e00700 1 -- 172.21.15.27:0/2103173631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e0106e00 msgr2=0x7f74e01a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:12.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.846+0000 7f74e4e00700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e0106e00 0x7f74e01a2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:12.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.846+0000 7f74e4e00700 1 -- 172.21.15.27:0/2103173631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e01a3530 con 0x7f74e0105340 2024-05-10T11:12:12.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.846+0000 7f74e4e00700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e0105340 0x7f74e019b8d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f74d400e9a0 tx=0x7f74d400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:12.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.847+0000 7f74dd600700 1 -- 172.21.15.27:0/2103173631 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4004220 con 0x7f74e0105340 2024-05-10T11:12:12.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.847+0000 7f74e7800700 1 -- 172.21.15.27:0/2103173631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e01a3820 con 0x7f74e0105340 2024-05-10T11:12:12.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.847+0000 7f74dd600700 1 -- 172.21.15.27:0/2103173631 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74d40043c0 con 0x7f74e0105340 2024-05-10T11:12:12.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.847+0000 7f74dd600700 1 -- 172.21.15.27:0/2103173631 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4003cc0 con 0x7f74e0105340 2024-05-10T11:12:12.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.847+0000 7f74e7800700 1 -- 172.21.15.27:0/2103173631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e01a3db0 con 0x7f74e0105340 2024-05-10T11:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.848+0000 7f74e7800700 1 -- 172.21.15.27:0/2103173631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e006a190 con 0x7f74e0105340 2024-05-10T11:12:12.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.851+0000 7f74dd600700 1 -- 172.21.15.27:0/2103173631 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f74d400c810 con 0x7f74e0105340 2024-05-10T11:12:12.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.853+0000 7f74dd600700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f74c80b2900 0x7f74c80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:12.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.853+0000 7f74dd600700 1 -- 172.21.15.27:0/2103173631 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f74d40e2870 con 0x7f74e0105340 2024-05-10T11:12:12.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.853+0000 7f74dfe00700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f74c80b2900 0x7f74c80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:12.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.854+0000 7f74dfe00700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f74c80b2900 0x7f74c80b4dc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009c60 tx=0x7f74cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:12.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:12.854+0000 7f74dd600700 1 -- 172.21.15.27:0/2103173631 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f74d40aa1d0 con 0x7f74e0105340 2024-05-10T11:12:13.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.101+0000 7f74e7800700 1 -- 172.21.15.27:0/2103173631 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f74e00638e0 con 0x7f74c80b2900 2024-05-10T11:12:13.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.120+0000 7f74dd600700 1 -- 172.21.15.27:0/2103173631 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f74e00638e0 con 0x7f74c80b2900 2024-05-10T11:12:13.120 INFO:teuthology.orchestra.run.smithi027.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-10T11:12:13.120 INFO:teuthology.orchestra.run.smithi027.stdout:mds.all 12/12 2m ago 5m smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-05-10T11:12:13.121 INFO:teuthology.orchestra.run.smithi027.stdout:mgr 3/3 2m ago 11m smithi027=x;smithi076=y;smithi189=z;count:3 2024-05-10T11:12:13.121 INFO:teuthology.orchestra.run.smithi027.stdout:mon 3/3 2m ago 12m smithi027:172.21.15.27=a;smithi076:172.21.15.76=b;smithi189:172.21.15.189=c;count:3 2024-05-10T11:12:13.121 INFO:teuthology.orchestra.run.smithi027.stdout:osd 12 2m ago - 2024-05-10T11:12:13.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.123+0000 7f74e7800700 1 -- 172.21.15.27:0/2103173631 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f74c80b2900 msgr2=0x7f74c80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:13.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.123+0000 7f74e7800700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f74c80b2900 0x7f74c80b4dc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009c60 tx=0x7f74cc009260 comp rx=0 tx=0).stop 2024-05-10T11:12:13.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.123+0000 7f74e7800700 1 -- 172.21.15.27:0/2103173631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e0105340 msgr2=0x7f74e019b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:13.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.123+0000 7f74e7800700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e0105340 0x7f74e019b8d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f74d400e9a0 tx=0x7f74d400ecb0 comp rx=0 tx=0).stop 2024-05-10T11:12:13.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.123+0000 7f74e7800700 1 -- 172.21.15.27:0/2103173631 shutdown_connections 2024-05-10T11:12:13.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.123+0000 7f74e7800700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f74c80b2900 0x7f74c80b4dc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:13.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.123+0000 7f74e7800700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e0105340 0x7f74e019b8d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:13.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.123+0000 7f74e7800700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e0106e00 0x7f74e01a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:13.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.123+0000 7f74e7800700 1 --2- 172.21.15.27:0/2103173631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74e0105ce0 0x7f74e019be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:13.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.123+0000 7f74e7800700 1 -- 172.21.15.27:0/2103173631 >> 172.21.15.27:0/2103173631 conn(0x7f74e0100c80 msgr2=0x7f74e0109c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:13.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.123+0000 7f74e7800700 1 -- 172.21.15.27:0/2103173631 shutdown_connections 2024-05-10T11:12:13.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:13.123+0000 7f74e7800700 1 -- 172.21.15.27:0/2103173631 wait complete. 2024-05-10T11:12:14.201 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-10T11:12:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:14 smithi076 ceph-mon[34152]: pgmap v466: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:14 smithi076 ceph-mon[34152]: from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:14.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:14 smithi189 ceph-mon[34842]: pgmap v466: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:14.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:14 smithi189 ceph-mon[34842]: from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:14.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:14 smithi027 ceph-mon[30054]: pgmap v466: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:14.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:14 smithi027 ceph-mon[30054]: from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:15.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:15 smithi076 ceph-mon[34152]: pgmap v467: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:15.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:15 smithi189 ceph-mon[34842]: pgmap v467: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:15.617 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:15 smithi027 ceph-mon[30054]: pgmap v467: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:16.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:16.998+0000 7f872cc00700 1 -- 172.21.15.27:0/397267130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728106fe0 msgr2=0x7f8728107460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:16.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:16.998+0000 7f872cc00700 1 --2- 172.21.15.27:0/397267130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728106fe0 0x7f8728107460 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f872000b0a0 tx=0x7f872000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:12:16.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:16.998+0000 7f872cc00700 1 -- 172.21.15.27:0/397267130 shutdown_connections 2024-05-10T11:12:16.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:16.998+0000 7f872cc00700 1 --2- 172.21.15.27:0/397267130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728106fe0 0x7f8728107460 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:16.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:16.998+0000 7f872cc00700 1 --2- 172.21.15.27:0/397267130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8728101ca0 0x7f87281040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:16.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:16.998+0000 7f872cc00700 1 --2- 172.21.15.27:0/397267130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8728104610 0x7f8728106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:17.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:16.998+0000 7f872cc00700 1 -- 172.21.15.27:0/397267130 >> 172.21.15.27:0/397267130 conn(0x7f87280fd8c0 msgr2=0x7f87280ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:17.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:16.998+0000 7f872cc00700 1 -- 172.21.15.27:0/397267130 shutdown_connections 2024-05-10T11:12:17.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:16.999+0000 7f872cc00700 1 -- 172.21.15.27:0/397267130 wait complete. 2024-05-10T11:12:17.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.000+0000 7f872cc00700 1 Processor -- start 2024-05-10T11:12:17.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.000+0000 7f872cc00700 1 -- start start 2024-05-10T11:12:17.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.000+0000 7f872cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8728101ca0 0x7f872819fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:17.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.000+0000 7f872cc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8728104610 0x7f872819ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:17.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.000+0000 7f872cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728106fe0 0x7f87281a7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:17.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.000+0000 7f872cc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8728115460 con 0x7f8728106fe0 2024-05-10T11:12:17.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.000+0000 7f872cc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f87281152e0 con 0x7f8728104610 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.000+0000 7f872cc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87281155e0 con 0x7f8728101ca0 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.001+0000 7f8727e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728106fe0 0x7f87281a7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.001+0000 7f8727e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728106fe0 0x7f87281a7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53646/0 (socket says 172.21.15.27:53646) 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.001+0000 7f8727e00700 1 -- 172.21.15.27:0/2159382500 learned_addr learned my addr 172.21.15.27:0/2159382500 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.001+0000 7f8727400700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8728101ca0 0x7f872819fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.001+0000 7f8726a00700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8728104610 0x7f872819ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.001+0000 7f8727e00700 1 -- 172.21.15.27:0/2159382500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8728101ca0 msgr2=0x7f872819fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.001+0000 7f8727e00700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8728101ca0 0x7f872819fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.001+0000 7f8727e00700 1 -- 172.21.15.27:0/2159382500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8728104610 msgr2=0x7f872819ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.001+0000 7f8727e00700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8728104610 0x7f872819ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.001+0000 7f8727e00700 1 -- 172.21.15.27:0/2159382500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87281a76b0 con 0x7f8728106fe0 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.002+0000 7f8727e00700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728106fe0 0x7f87281a7010 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8720009fd0 tx=0x7f8720004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.002+0000 7f871fe00700 1 -- 172.21.15.27:0/2159382500 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8720021070 con 0x7f8728106fe0 2024-05-10T11:12:17.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.002+0000 7f872cc00700 1 -- 172.21.15.27:0/2159382500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87281a7940 con 0x7f8728106fe0 2024-05-10T11:12:17.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.002+0000 7f871fe00700 1 -- 172.21.15.27:0/2159382500 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8720007b80 con 0x7f8728106fe0 2024-05-10T11:12:17.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.002+0000 7f871fe00700 1 -- 172.21.15.27:0/2159382500 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87200052a0 con 0x7f8728106fe0 2024-05-10T11:12:17.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.002+0000 7f872cc00700 1 -- 172.21.15.27:0/2159382500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87281a7ed0 con 0x7f8728106fe0 2024-05-10T11:12:17.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.003+0000 7f872cc00700 1 -- 172.21.15.27:0/2159382500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f872806a190 con 0x7f8728106fe0 2024-05-10T11:12:17.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.006+0000 7f871fe00700 1 -- 172.21.15.27:0/2159382500 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f8720024070 con 0x7f8728106fe0 2024-05-10T11:12:17.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.007+0000 7f871fe00700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f870c0b2880 0x7f870c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:17.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.007+0000 7f871fe00700 1 -- 172.21.15.27:0/2159382500 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f872000f020 con 0x7f8728106fe0 2024-05-10T11:12:17.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.007+0000 7f8727400700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f870c0b2880 0x7f870c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:17.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.008+0000 7f8727400700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f870c0b2880 0x7f870c0b4d40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8718009c60 tx=0x7f8718009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:17.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.010+0000 7f871fe00700 1 -- 172.21.15.27:0/2159382500 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f872002b020 con 0x7f8728106fe0 2024-05-10T11:12:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.260+0000 7f872cc00700 1 -- 172.21.15.27:0/2159382500 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f87280638e0 con 0x7f870c0b2880 2024-05-10T11:12:17.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.262+0000 7f871fe00700 1 -- 172.21.15.27:0/2159382500 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f87280638e0 con 0x7f870c0b2880 2024-05-10T11:12:17.262 INFO:teuthology.orchestra.run.smithi027.stdout:HOST ADDR LABELS STATUS 2024-05-10T11:12:17.262 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 172.21.15.27 2024-05-10T11:12:17.262 INFO:teuthology.orchestra.run.smithi027.stdout:smithi076 172.21.15.76 2024-05-10T11:12:17.262 INFO:teuthology.orchestra.run.smithi027.stdout:smithi189 172.21.15.189 2024-05-10T11:12:17.262 INFO:teuthology.orchestra.run.smithi027.stdout:3 hosts in cluster 2024-05-10T11:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.265+0000 7f872cc00700 1 -- 172.21.15.27:0/2159382500 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f870c0b2880 msgr2=0x7f870c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.265+0000 7f872cc00700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f870c0b2880 0x7f870c0b4d40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8718009c60 tx=0x7f8718009260 comp rx=0 tx=0).stop 2024-05-10T11:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.265+0000 7f872cc00700 1 -- 172.21.15.27:0/2159382500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728106fe0 msgr2=0x7f87281a7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.265+0000 7f872cc00700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728106fe0 0x7f87281a7010 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8720009fd0 tx=0x7f8720004dc0 comp rx=0 tx=0).stop 2024-05-10T11:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.266+0000 7f872cc00700 1 -- 172.21.15.27:0/2159382500 shutdown_connections 2024-05-10T11:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.266+0000 7f872cc00700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f870c0b2880 0x7f870c0b4d40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.266+0000 7f872cc00700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8728101ca0 0x7f872819fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.266+0000 7f872cc00700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728106fe0 0x7f87281a7010 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.266+0000 7f872cc00700 1 --2- 172.21.15.27:0/2159382500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8728104610 0x7f872819ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.266+0000 7f872cc00700 1 -- 172.21.15.27:0/2159382500 >> 172.21.15.27:0/2159382500 conn(0x7f87280fd8c0 msgr2=0x7f8728102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.266+0000 7f872cc00700 1 -- 172.21.15.27:0/2159382500 shutdown_connections 2024-05-10T11:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:17.266+0000 7f872cc00700 1 -- 172.21.15.27:0/2159382500 wait complete. 2024-05-10T11:12:18.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:17 smithi189 ceph-mon[34842]: pgmap v468: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:18.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:17 smithi027 ceph-mon[30054]: pgmap v468: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:18.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-10T11:12:18.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:17 smithi076 ceph-mon[34152]: pgmap v468: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:18.827 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:19.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:18 smithi189 ceph-mon[34842]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:19.239 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:18 smithi027 ceph-mon[30054]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:19.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:18 smithi076 ceph-mon[34152]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:20.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:19 smithi189 ceph-mon[34842]: pgmap v469: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:20.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:19 smithi027 ceph-mon[30054]: pgmap v469: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:20.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:19 smithi076 ceph-mon[34152]: pgmap v469: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:21.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.238+0000 7fb802800700 1 -- 172.21.15.27:0/1414794007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7fc104620 msgr2=0x7fb7fc104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.238+0000 7fb802800700 1 --2- 172.21.15.27:0/1414794007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7fc104620 0x7fb7fc104a80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0099c0 tx=0x7fb7ec009cd0 comp rx=0 tx=0).stop 2024-05-10T11:12:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.238+0000 7fb802800700 1 -- 172.21.15.27:0/1414794007 shutdown_connections 2024-05-10T11:12:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.238+0000 7fb802800700 1 --2- 172.21.15.27:0/1414794007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7fc103420 0x7fb7fc103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.238+0000 7fb802800700 1 --2- 172.21.15.27:0/1414794007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7fc104fc0 0x7fb7fc10b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.238+0000 7fb802800700 1 --2- 172.21.15.27:0/1414794007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7fc104620 0x7fb7fc104a80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.238+0000 7fb802800700 1 -- 172.21.15.27:0/1414794007 >> 172.21.15.27:0/1414794007 conn(0x7fb7fc0fe9e0 msgr2=0x7fb7fc100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.239+0000 7fb802800700 1 -- 172.21.15.27:0/1414794007 shutdown_connections 2024-05-10T11:12:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.239+0000 7fb802800700 1 -- 172.21.15.27:0/1414794007 wait complete. 2024-05-10T11:12:21.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.240+0000 7fb802800700 1 Processor -- start 2024-05-10T11:12:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.240+0000 7fb802800700 1 -- start start 2024-05-10T11:12:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.240+0000 7fb802800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7fc103420 0x7fb7fc077240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.240+0000 7fb802800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7fc104620 0x7fb7fc075890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb802800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7fc104fc0 0x7fb7fc075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb802800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc10eea0 con 0x7fb7fc104fc0 2024-05-10T11:12:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb802800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc10ed20 con 0x7fb7fc103420 2024-05-10T11:12:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb802800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc10f020 con 0x7fb7fc104620 2024-05-10T11:12:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb801400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7fc103420 0x7fb7fc077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb801400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7fc103420 0x7fb7fc077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58450/0 (socket says 172.21.15.27:58450) 2024-05-10T11:12:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb801400700 1 -- 172.21.15.27:0/4220245952 learned_addr learned my addr 172.21.15.27:0/4220245952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb801e00700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7fc104fc0 0x7fb7fc075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb800a00700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7fc104620 0x7fb7fc075890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:21.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb801400700 1 -- 172.21.15.27:0/4220245952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7fc104620 msgr2=0x7fb7fc075890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:21.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb801400700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7fc104620 0x7fb7fc075890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:21.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb801400700 1 -- 172.21.15.27:0/4220245952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7fc104fc0 msgr2=0x7fb7fc075dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:21.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb801400700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7fc104fc0 0x7fb7fc075dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:21.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.241+0000 7fb801400700 1 -- 172.21.15.27:0/4220245952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7fc076680 con 0x7fb7fc103420 2024-05-10T11:12:21.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.242+0000 7fb801400700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7fc103420 0x7fb7fc077240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000e970 tx=0x7fb7f000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:21.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.242+0000 7fb7fa000700 1 -- 172.21.15.27:0/4220245952 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0004220 con 0x7fb7fc103420 2024-05-10T11:12:21.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.242+0000 7fb7fa000700 1 -- 172.21.15.27:0/4220245952 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7f00043c0 con 0x7fb7fc103420 2024-05-10T11:12:21.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.242+0000 7fb7fa000700 1 -- 172.21.15.27:0/4220245952 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0003ce0 con 0x7fb7fc103420 2024-05-10T11:12:21.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.242+0000 7fb802800700 1 -- 172.21.15.27:0/4220245952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7fc1a36a0 con 0x7fb7fc103420 2024-05-10T11:12:21.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.242+0000 7fb802800700 1 -- 172.21.15.27:0/4220245952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7fc1a3bb0 con 0x7fb7fc103420 2024-05-10T11:12:21.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.243+0000 7fb802800700 1 -- 172.21.15.27:0/4220245952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7fc06a190 con 0x7fb7fc103420 2024-05-10T11:12:21.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.246+0000 7fb7fa000700 1 -- 172.21.15.27:0/4220245952 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fb7f000cd00 con 0x7fb7fc103420 2024-05-10T11:12:21.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.247+0000 7fb7fa000700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb7e40b2880 0x7fb7e40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:21.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.248+0000 7fb7fa000700 1 -- 172.21.15.27:0/4220245952 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fb7f00e28d0 con 0x7fb7fc103420 2024-05-10T11:12:21.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.248+0000 7fb800a00700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb7e40b2880 0x7fb7e40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:21.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.248+0000 7fb800a00700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb7e40b2880 0x7fb7e40b4d40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec000c00 tx=0x7fb7ec005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:21.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.249+0000 7fb7fa000700 1 -- 172.21.15.27:0/4220245952 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb7f00aa230 con 0x7fb7fc103420 2024-05-10T11:12:21.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.494+0000 7fb802800700 1 -- 172.21.15.27:0/4220245952 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fb7fc0638e0 con 0x7fb7e40b2880 2024-05-10T11:12:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.498+0000 7fb7fa000700 1 -- 172.21.15.27:0/4220245952 <== mgr.14152 v2:172.21.15.27:6800/1704535643 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fb7fc0638e0 con 0x7fb7e40b2880 2024-05-10T11:12:21.498 INFO:teuthology.orchestra.run.smithi027.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-10T11:12:21.498 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001C400BGN 372G No 6m ago Has a FileSystem, LVM detected 2024-05-10T11:12:21.498 INFO:teuthology.orchestra.run.smithi027.stdout:smithi076 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400U4400BGN 372G No 6m ago Has a FileSystem, LVM detected 2024-05-10T11:12:21.498 INFO:teuthology.orchestra.run.smithi027.stdout:smithi189 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232004L400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-05-10T11:12:21.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.501+0000 7fb802800700 1 -- 172.21.15.27:0/4220245952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb7e40b2880 msgr2=0x7fb7e40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:21.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.501+0000 7fb802800700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb7e40b2880 0x7fb7e40b4d40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec000c00 tx=0x7fb7ec005fb0 comp rx=0 tx=0).stop 2024-05-10T11:12:21.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.501+0000 7fb802800700 1 -- 172.21.15.27:0/4220245952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7fc103420 msgr2=0x7fb7fc077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:21.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.501+0000 7fb802800700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7fc103420 0x7fb7fc077240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000e970 tx=0x7fb7f000ec80 comp rx=0 tx=0).stop 2024-05-10T11:12:21.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.501+0000 7fb802800700 1 -- 172.21.15.27:0/4220245952 shutdown_connections 2024-05-10T11:12:21.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.501+0000 7fb802800700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fb7e40b2880 0x7fb7e40b4d40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:21.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.501+0000 7fb802800700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7fc104fc0 0x7fb7fc075dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:21.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.501+0000 7fb802800700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7fc103420 0x7fb7fc077240 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:21.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.501+0000 7fb802800700 1 --2- 172.21.15.27:0/4220245952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7fc104620 0x7fb7fc075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:21.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.501+0000 7fb802800700 1 -- 172.21.15.27:0/4220245952 >> 172.21.15.27:0/4220245952 conn(0x7fb7fc0fe9e0 msgr2=0x7fb7fc1079f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:21.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.502+0000 7fb802800700 1 -- 172.21.15.27:0/4220245952 shutdown_connections 2024-05-10T11:12:21.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:21.502+0000 7fb802800700 1 -- 172.21.15.27:0/4220245952 wait complete. 2024-05-10T11:12:22.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:21 smithi189 ceph-mon[34842]: pgmap v470: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:22.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:21 smithi027 ceph-mon[30054]: pgmap v470: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:22.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:21 smithi076 ceph-mon[34152]: pgmap v470: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:22.470 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-10T11:12:22.480 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-05-10T11:12:22.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-10T11:12:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:23.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:22 smithi189 ceph-mon[34842]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:23.289 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:22 smithi027 ceph-mon[30054]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:23.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:22 smithi076 ceph-mon[34152]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:12:24.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:23 smithi189 ceph-mon[34842]: pgmap v471: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:24.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:23 smithi027 ceph-mon[30054]: pgmap v471: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:24.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:23 smithi076 ceph-mon[34152]: pgmap v471: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:25.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.342+0000 7f42d8200700 1 -- 172.21.15.27:0/2357802338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42d0103f90 msgr2=0x7f42d0106380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:25.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.342+0000 7f42d8200700 1 --2- 172.21.15.27:0/2357802338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42d0103f90 0x7f42d0106380 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009990 tx=0x7f42c4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:12:25.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.342+0000 7f42d8200700 1 -- 172.21.15.27:0/2357802338 shutdown_connections 2024-05-10T11:12:25.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.342+0000 7f42d8200700 1 --2- 172.21.15.27:0/2357802338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42d0103f90 0x7f42d0106380 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:25.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.342+0000 7f42d8200700 1 --2- 172.21.15.27:0/2357802338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42d01091f0 0x7f42d010b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:25.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.342+0000 7f42d8200700 1 --2- 172.21.15.27:0/2357802338 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42d01068c0 0x7f42d0108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:25.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.342+0000 7f42d8200700 1 -- 172.21.15.27:0/2357802338 >> 172.21.15.27:0/2357802338 conn(0x7f42d00fd9a0 msgr2=0x7f42d00ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:25.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.342+0000 7f42d8200700 1 -- 172.21.15.27:0/2357802338 shutdown_connections 2024-05-10T11:12:25.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.342+0000 7f42d8200700 1 -- 172.21.15.27:0/2357802338 wait complete. 2024-05-10T11:12:25.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.343+0000 7f42d8200700 1 Processor -- start 2024-05-10T11:12:25.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.343+0000 7f42d8200700 1 -- start start 2024-05-10T11:12:25.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.344+0000 7f42d8200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42d0103f90 0x7f42d01038f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:25.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.344+0000 7f42d8200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42d01068c0 0x7f42d0101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:25.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.344+0000 7f42d8200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42d01091f0 0x7f42d0102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:25.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.344+0000 7f42d8200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f42d010f010 con 0x7f42d01091f0 2024-05-10T11:12:25.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.344+0000 7f42d8200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f42d010ee90 con 0x7f42d0103f90 2024-05-10T11:12:25.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.344+0000 7f42d8200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f42d010f190 con 0x7f42d01068c0 2024-05-10T11:12:25.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.344+0000 7f42d5800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42d0103f90 0x7f42d01038f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:25.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.344+0000 7f42d5800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42d0103f90 0x7f42d01038f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58874/0 (socket says 172.21.15.27:58874) 2024-05-10T11:12:25.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.344+0000 7f42d4e00700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42d01068c0 0x7f42d0101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:25.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.344+0000 7f42d5800700 1 -- 172.21.15.27:0/3646895662 learned_addr learned my addr 172.21.15.27:0/3646895662 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:25.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.344+0000 7f42d6200700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42d01091f0 0x7f42d0102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:25.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.345+0000 7f42d4e00700 1 -- 172.21.15.27:0/3646895662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42d0103f90 msgr2=0x7f42d01038f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:25.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.345+0000 7f42d4e00700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42d0103f90 0x7f42d01038f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:25.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.345+0000 7f42d4e00700 1 -- 172.21.15.27:0/3646895662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42d01091f0 msgr2=0x7f42d0102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:25.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.345+0000 7f42d4e00700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42d01091f0 0x7f42d0102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:25.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.345+0000 7f42d4e00700 1 -- 172.21.15.27:0/3646895662 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d0102d30 con 0x7f42d01068c0 2024-05-10T11:12:25.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.345+0000 7f42d4e00700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42d01068c0 0x7f42d0101f40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00e9f0 tx=0x7f42bc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:25.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.345+0000 7f42ce000700 1 -- 172.21.15.27:0/3646895662 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc004220 con 0x7f42d01068c0 2024-05-10T11:12:25.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.346+0000 7f42ce000700 1 -- 172.21.15.27:0/3646895662 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42bc0043c0 con 0x7f42d01068c0 2024-05-10T11:12:25.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.346+0000 7f42d8200700 1 -- 172.21.15.27:0/3646895662 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d01a3860 con 0x7f42d01068c0 2024-05-10T11:12:25.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.346+0000 7f42ce000700 1 -- 172.21.15.27:0/3646895662 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc005020 con 0x7f42d01068c0 2024-05-10T11:12:25.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.346+0000 7f42d8200700 1 -- 172.21.15.27:0/3646895662 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d01a3d70 con 0x7f42d01068c0 2024-05-10T11:12:25.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.347+0000 7f42d8200700 1 -- 172.21.15.27:0/3646895662 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42d006a190 con 0x7f42d01068c0 2024-05-10T11:12:25.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.349+0000 7f42ce000700 1 -- 172.21.15.27:0/3646895662 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f42bc004750 con 0x7f42d01068c0 2024-05-10T11:12:25.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.351+0000 7f42ce000700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f42b80b2900 0x7f42b80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:25.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.351+0000 7f42ce000700 1 -- 172.21.15.27:0/3646895662 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f42bc0e2650 con 0x7f42d01068c0 2024-05-10T11:12:25.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.351+0000 7f42d5800700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f42b80b2900 0x7f42b80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:25.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.352+0000 7f42d5800700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f42b80b2900 0x7f42b80b4dc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f42c400b2e0 tx=0x7f42c4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:25.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.353+0000 7f42ce000700 1 -- 172.21.15.27:0/3646895662 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f42bc0a9fb0 con 0x7f42d01068c0 2024-05-10T11:12:25.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.642+0000 7f42d8200700 1 -- 172.21.15.27:0/3646895662 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f42d002e510 con 0x7f42d01068c0 2024-05-10T11:12:25.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.643+0000 7f42ce000700 1 -- 172.21.15.27:0/3646895662 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) ==== 76+0+3094 (secure 0 0 0) 0x7f42bc0a98b0 con 0x7f42d01068c0 2024-05-10T11:12:25.785 INFO:teuthology.orchestra.run.smithi027.stdout:e29 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 29 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-05-10T11:07:50.237787+0000 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-05-10T11:09:19.986922+0000 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 300 2024-05-10T11:12:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 0 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 5 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:in 0,1,2,3,4 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=24527,1=24488,2=24484,3=24472,4=24467} 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3,4] 2024-05-10T11:12:25.787 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 0 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 24527 members: 24467,24488,24472,24484,24527 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b{0:24527} state up:active seq 7 addr [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] compat {c=[1],r=[1],i=[1fff]}] 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.f{0:24460} state up:standby-replay seq 1 addr [v2:172.21.15.189:6834/1387617890,v1:172.21.15.189:6835/1387617890] compat {c=[1],r=[1],i=[1fff]}] 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.h{1:24488} state up:active seq 32 addr [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] compat {c=[1],r=[1],i=[1fff]}] 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.j{1:24449} state up:standby-replay seq 1 addr [v2:172.21.15.27:6834/3274866857,v1:172.21.15.27:6835/3274866857] compat {c=[1],r=[1],i=[1fff]}] 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.i{2:24484} state up:active seq 31 addr [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] compat {c=[1],r=[1],i=[1fff]}] 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.k{2:24437} state up:standby-replay seq 1 addr [v2:172.21.15.76:6832/1370063583,v1:172.21.15.76:6833/1370063583] compat {c=[1],r=[1],i=[1fff]}] 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c{3:24472} state up:active seq 33 addr [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] compat {c=[1],r=[1],i=[1fff]}] 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.l{3:24418} state up:standby-replay seq 1 addr [v2:172.21.15.189:6832/2718624698,v1:172.21.15.189:6833/2718624698] compat {c=[1],r=[1],i=[1fff]}] 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.e{4:24467} state up:active seq 35 addr [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] compat {c=[1],r=[1],i=[1fff]}] 2024-05-10T11:12:25.788 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a{4:14403} state up:standby-replay seq 1 addr [v2:172.21.15.27:6840/3080402649,v1:172.21.15.27:6841/3080402649] compat {c=[1],r=[1],i=[1fff]}] 2024-05-10T11:12:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:12:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:12:25.789 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-05-10T11:12:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:12:25.789 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d{-1:14367} state up:standby seq 1 addr [v2:172.21.15.27:6836/1490955215,v1:172.21.15.27:6837/1490955215] compat {c=[1],r=[1],i=[1fff]}] 2024-05-10T11:12:25.789 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.g{-1:14391} state up:standby seq 1 addr [v2:172.21.15.27:6838/3240182674,v1:172.21.15.27:6839/3240182674] compat {c=[1],r=[1],i=[1fff]}] 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.646+0000 7f42d8200700 1 -- 172.21.15.27:0/3646895662 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f42b80b2900 msgr2=0x7f42b80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.646+0000 7f42d8200700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f42b80b2900 0x7f42b80b4dc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f42c400b2e0 tx=0x7f42c4005fb0 comp rx=0 tx=0).stop 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.646+0000 7f42d8200700 1 -- 172.21.15.27:0/3646895662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42d01068c0 msgr2=0x7f42d0101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.646+0000 7f42d8200700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42d01068c0 0x7f42d0101f40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00e9f0 tx=0x7f42bc00ed00 comp rx=0 tx=0).stop 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.646+0000 7f42d8200700 1 -- 172.21.15.27:0/3646895662 shutdown_connections 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.646+0000 7f42d8200700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f42b80b2900 0x7f42b80b4dc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.647+0000 7f42d8200700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42d01091f0 0x7f42d0102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.647+0000 7f42d8200700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42d0103f90 0x7f42d01038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.647+0000 7f42d8200700 1 --2- 172.21.15.27:0/3646895662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42d01068c0 0x7f42d0101f40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.647+0000 7f42d8200700 1 -- 172.21.15.27:0/3646895662 >> 172.21.15.27:0/3646895662 conn(0x7f42d00fd9a0 msgr2=0x7f42d0104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.647+0000 7f42d8200700 1 -- 172.21.15.27:0/3646895662 shutdown_connections 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:25.647+0000 7f42d8200700 1 -- 172.21.15.27:0/3646895662 wait complete. 2024-05-10T11:12:25.790 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-05-10T11:12:26.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:25 smithi027 ceph-mon[30054]: pgmap v472: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:26.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:25 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3646895662' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-10T11:12:26.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:25 smithi076 ceph-mon[34152]: pgmap v472: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:26.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:25 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3646895662' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-10T11:12:26.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:25 smithi189 ceph-mon[34842]: pgmap v472: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:26.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:25 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3646895662' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-10T11:12:26.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-10T11:12:27.183 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:28.258 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:27 smithi027 ceph-mon[30054]: pgmap v473: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:28.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:27 smithi076 ceph-mon[34152]: pgmap v473: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:28.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:27 smithi189 ceph-mon[34842]: pgmap v473: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.498+0000 7f52f1600700 1 -- 172.21.15.27:0/3884777156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52ec077470 msgr2=0x7f52ec0758d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.498+0000 7f52f1600700 1 --2- 172.21.15.27:0/3884777156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52ec077470 0x7f52ec0758d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009990 tx=0x7f52dc009ca0 comp rx=0 tx=0).stop 2024-05-10T11:12:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.499+0000 7f52f1600700 1 -- 172.21.15.27:0/3884777156 shutdown_connections 2024-05-10T11:12:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.499+0000 7f52f1600700 1 --2- 172.21.15.27:0/3884777156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52ec10b3e0 0x7f52ec10d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.499+0000 7f52f1600700 1 --2- 172.21.15.27:0/3884777156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52ec077470 0x7f52ec0758d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.499+0000 7f52f1600700 1 --2- 172.21.15.27:0/3884777156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f52ec075ea0 0x7f52ec076320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.499+0000 7f52f1600700 1 -- 172.21.15.27:0/3884777156 >> 172.21.15.27:0/3884777156 conn(0x7f52ec0feaa0 msgr2=0x7f52ec100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.499+0000 7f52f1600700 1 -- 172.21.15.27:0/3884777156 shutdown_connections 2024-05-10T11:12:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.499+0000 7f52f1600700 1 -- 172.21.15.27:0/3884777156 wait complete. 2024-05-10T11:12:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.500+0000 7f52f1600700 1 Processor -- start 2024-05-10T11:12:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.500+0000 7f52f1600700 1 -- start start 2024-05-10T11:12:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.500+0000 7f52f1600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52ec075ea0 0x7f52ec19b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.500+0000 7f52f1600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f52ec077470 0x7f52ec19be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.500+0000 7f52f1600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52ec10b3e0 0x7f52ec1a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.500+0000 7f52f1600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec1111a0 con 0x7f52ec075ea0 2024-05-10T11:12:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.500+0000 7f52f1600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec111020 con 0x7f52ec077470 2024-05-10T11:12:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.500+0000 7f52f1600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec111320 con 0x7f52ec10b3e0 2024-05-10T11:12:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.501+0000 7f52ea000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f52ec077470 0x7f52ec19be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.501+0000 7f52eaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52ec075ea0 0x7f52ec19b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.501+0000 7f52eb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52ec10b3e0 0x7f52ec1a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.501+0000 7f52ea000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f52ec077470 0x7f52ec19be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58888/0 (socket says 172.21.15.27:58888) 2024-05-10T11:12:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.501+0000 7f52ea000700 1 -- 172.21.15.27:0/2751366845 learned_addr learned my addr 172.21.15.27:0/2751366845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.501+0000 7f52ea000700 1 -- 172.21.15.27:0/2751366845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52ec10b3e0 msgr2=0x7f52ec1a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.501+0000 7f52ea000700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52ec10b3e0 0x7f52ec1a2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.501+0000 7f52ea000700 1 -- 172.21.15.27:0/2751366845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52ec075ea0 msgr2=0x7f52ec19b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.501+0000 7f52ea000700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52ec075ea0 0x7f52ec19b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.501+0000 7f52ea000700 1 -- 172.21.15.27:0/2751366845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ec1a34d0 con 0x7f52ec077470 2024-05-10T11:12:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.501+0000 7f52eb400700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52ec10b3e0 0x7f52ec1a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.502+0000 7f52ea000700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f52ec077470 0x7f52ec19be00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f52d800e9a0 tx=0x7f52d800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.502+0000 7f52e3400700 1 -- 172.21.15.27:0/2751366845 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d800ca10 con 0x7f52ec077470 2024-05-10T11:12:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.502+0000 7f52e3400700 1 -- 172.21.15.27:0/2751366845 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52d800cbb0 con 0x7f52ec077470 2024-05-10T11:12:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.502+0000 7f52f1600700 1 -- 172.21.15.27:0/2751366845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52ec1a37c0 con 0x7f52ec077470 2024-05-10T11:12:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.502+0000 7f52e3400700 1 -- 172.21.15.27:0/2751366845 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8018680 con 0x7f52ec077470 2024-05-10T11:12:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.502+0000 7f52f1600700 1 -- 172.21.15.27:0/2751366845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52ec1a3d00 con 0x7f52ec077470 2024-05-10T11:12:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.503+0000 7f52f1600700 1 -- 172.21.15.27:0/2751366845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52ec06a190 con 0x7f52ec077470 2024-05-10T11:12:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.507+0000 7f52e3400700 1 -- 172.21.15.27:0/2751366845 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f52d8010430 con 0x7f52ec077470 2024-05-10T11:12:29.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.508+0000 7f52e3400700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f52d00b2880 0x7f52d00b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.508+0000 7f52e3400700 1 -- 172.21.15.27:0/2751366845 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f52d80e2a10 con 0x7f52ec077470 2024-05-10T11:12:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.509+0000 7f52eaa00700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f52d00b2880 0x7f52d00b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:29.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.509+0000 7f52e3400700 1 -- 172.21.15.27:0/2751366845 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f52d80aa2e0 con 0x7f52ec077470 2024-05-10T11:12:29.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.509+0000 7f52eaa00700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f52d00b2880 0x7f52d00b4d40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00b2e0 tx=0x7f52dc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:29.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.750+0000 7f52f1600700 1 -- 172.21.15.27:0/2751366845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f52ec02e510 con 0x7f52ec077470 2024-05-10T11:12:29.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.751+0000 7f52e3400700 1 -- 172.21.15.27:0/2751366845 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+5711 (secure 0 0 0) 0x7f52d80a9be0 con 0x7f52ec077470 2024-05-10T11:12:29.753 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 90 2024-05-10T11:12:29.753 INFO:teuthology.orchestra.run.smithi027.stdout:fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:12:29.753 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-05-10T10:58:11.299955+0000 2024-05-10T11:12:29.753 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-05-10T11:09:23.055559+0000 2024-05-10T11:12:29.753 INFO:teuthology.orchestra.run.smithi027.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-10T11:12:29.754 INFO:teuthology.orchestra.run.smithi027.stdout:crush_version 27 2024-05-10T11:12:29.754 INFO:teuthology.orchestra.run.smithi027.stdout:full_ratio 0.95 2024-05-10T11:12:29.754 INFO:teuthology.orchestra.run.smithi027.stdout:backfillfull_ratio 0.9 2024-05-10T11:12:29.754 INFO:teuthology.orchestra.run.smithi027.stdout:nearfull_ratio 0.85 2024-05-10T11:12:29.754 INFO:teuthology.orchestra.run.smithi027.stdout:require_min_compat_client luminous 2024-05-10T11:12:29.758 INFO:teuthology.orchestra.run.smithi027.stdout:min_compat_client luminous 2024-05-10T11:12:29.758 INFO:teuthology.orchestra.run.smithi027.stdout:require_osd_release squid 2024-05-10T11:12:29.758 INFO:teuthology.orchestra.run.smithi027.stdout:stretch_mode_enabled false 2024-05-10T11:12:29.758 INFO:teuthology.orchestra.run.smithi027.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-10T11:12:29.758 INFO:teuthology.orchestra.run.smithi027.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-05-10T11:12:29.758 INFO:teuthology.orchestra.run.smithi027.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-05-10T11:12:29.758 INFO:teuthology.orchestra.run.smithi027.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-05-10T11:12:29.758 INFO:teuthology.orchestra.run.smithi027.stdout:max_osd 12 2024-05-10T11:12:29.758 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 up in weight 1 up_from 8 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6802/677446345,v1:172.21.15.27:6803/677446345] [v2:172.21.15.27:6804/677446345,v1:172.21.15.27:6805/677446345] exists,up df11dcf5-f3ad-4b7c-99e7-e26f7756278d 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 up in weight 1 up_from 13 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6800/98361899,v1:172.21.15.76:6801/98361899] [v2:172.21.15.76:6802/98361899,v1:172.21.15.76:6803/98361899] exists,up 207c4446-366c-45dd-8496-e1234636e6b0 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6800/1136376247,v1:172.21.15.189:6801/1136376247] [v2:172.21.15.189:6802/1136376247,v1:172.21.15.189:6803/1136376247] exists,up 9afd53b7-8054-4ff3-888b-de388da33ba2 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 up in weight 1 up_from 25 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6810/2407454219,v1:172.21.15.27:6811/2407454219] [v2:172.21.15.27:6812/2407454219,v1:172.21.15.27:6813/2407454219] exists,up 6e0125b3-a60e-45ea-9aca-73c16bd0c69e 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 up in weight 1 up_from 31 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6808/1681276823,v1:172.21.15.76:6809/1681276823] [v2:172.21.15.76:6810/1681276823,v1:172.21.15.76:6811/1681276823] exists,up 22993f58-2d0f-4faa-9651-02f3ff04cf15 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 up in weight 1 up_from 36 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6808/1669864682,v1:172.21.15.189:6809/1669864682] [v2:172.21.15.189:6810/1669864682,v1:172.21.15.189:6811/1669864682] exists,up d5740d97-7270-4869-b523-b6e3c15e5127 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 up in weight 1 up_from 42 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6818/3624966800,v1:172.21.15.27:6819/3624966800] [v2:172.21.15.27:6820/3624966800,v1:172.21.15.27:6821/3624966800] exists,up 2aa064bd-d7d2-4189-824d-59cc8d0c5e29 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7 up in weight 1 up_from 47 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6816/1478795194,v1:172.21.15.76:6817/1478795194] [v2:172.21.15.76:6818/1478795194,v1:172.21.15.76:6819/1478795194] exists,up 56a01e95-852d-4072-a5d7-50b414487dfe 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8 up in weight 1 up_from 53 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6816/2550733386,v1:172.21.15.189:6817/2550733386] [v2:172.21.15.189:6818/2550733386,v1:172.21.15.189:6819/2550733386] exists,up 28643dd0-d7e0-498b-8c59-9374a9f58239 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 up in weight 1 up_from 59 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6826/2571330197,v1:172.21.15.27:6827/2571330197] [v2:172.21.15.27:6828/2571330197,v1:172.21.15.27:6829/2571330197] exists,up 9f10253f-40eb-43c9-8e6f-f552d3fd736c 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10 up in weight 1 up_from 65 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6824/2809936234,v1:172.21.15.76:6825/2809936234] [v2:172.21.15.76:6826/2809936234,v1:172.21.15.76:6827/2809936234] exists,up 4a916c16-6efc-4d99-8494-238b7cbeef63 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11 up in weight 1 up_from 71 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6824/49546096,v1:172.21.15.189:6825/49546096] [v2:172.21.15.189:6826/49546096,v1:172.21.15.189:6827/49546096] exists,up ab1431f2-89cd-447b-aec8-e2dc786564d0 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.f [8,11] 2024-05-10T11:12:29.759 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.13 [2,11] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.1a [8,1] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.1b [2,4] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.0 [4,6] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.17 [4,8] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.1b [3,6,4,8] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.1d [5,9] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.2d [3,9] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.33 [5,9] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.3a [4,7] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 4.2 [9,1] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 4.a [5,10] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 4.b [9,8] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 4.10 [9,8] 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/2444542866 expires 2024-05-11T10:59:20.613831+0000 2024-05-10T11:12:29.760 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6801/481366655 expires 2024-05-11T10:59:20.613831+0000 2024-05-10T11:12:29.761 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6800/481366655 expires 2024-05-11T10:59:20.613831+0000 2024-05-10T11:12:29.761 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/2743539323 expires 2024-05-11T10:59:20.613831+0000 2024-05-10T11:12:29.761 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/3803325728 expires 2024-05-11T10:58:45.437819+0000 2024-05-10T11:12:29.761 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6800/2058505174 expires 2024-05-11T10:58:45.437819+0000 2024-05-10T11:12:29.761 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6801/2058505174 expires 2024-05-11T10:58:45.437819+0000 2024-05-10T11:12:29.761 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/1886129738 expires 2024-05-11T10:58:45.437819+0000 2024-05-10T11:12:29.761 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/1208957722 expires 2024-05-11T10:59:20.613831+0000 2024-05-10T11:12:29.761 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/4158621178 expires 2024-05-11T10:58:45.437819+0000 2024-05-10T11:12:29.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.754+0000 7f52f1600700 1 -- 172.21.15.27:0/2751366845 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f52d00b2880 msgr2=0x7f52d00b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:29.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.754+0000 7f52f1600700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f52d00b2880 0x7f52d00b4d40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00b2e0 tx=0x7f52dc005fb0 comp rx=0 tx=0).stop 2024-05-10T11:12:29.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.755+0000 7f52f1600700 1 -- 172.21.15.27:0/2751366845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f52ec077470 msgr2=0x7f52ec19be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:29.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.755+0000 7f52f1600700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f52ec077470 0x7f52ec19be00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f52d800e9a0 tx=0x7f52d800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:12:29.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.755+0000 7f52f1600700 1 -- 172.21.15.27:0/2751366845 shutdown_connections 2024-05-10T11:12:29.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.755+0000 7f52f1600700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f52d00b2880 0x7f52d00b4d40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:29.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.755+0000 7f52f1600700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52ec10b3e0 0x7f52ec1a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:29.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.755+0000 7f52f1600700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52ec075ea0 0x7f52ec19b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:29.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.755+0000 7f52f1600700 1 --2- 172.21.15.27:0/2751366845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f52ec077470 0x7f52ec19be00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:29.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.755+0000 7f52f1600700 1 -- 172.21.15.27:0/2751366845 >> 172.21.15.27:0/2751366845 conn(0x7f52ec0feaa0 msgr2=0x7f52ec107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:29.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.755+0000 7f52f1600700 1 -- 172.21.15.27:0/2751366845 shutdown_connections 2024-05-10T11:12:29.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:29.755+0000 7f52f1600700 1 -- 172.21.15.27:0/2751366845 wait complete. 2024-05-10T11:12:30.360 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:30 smithi027 ceph-mon[30054]: pgmap v474: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:30.360 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:30 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2751366845' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-10T11:12:30.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:30 smithi076 ceph-mon[34152]: pgmap v474: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:30.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:30 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2751366845' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-10T11:12:30.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:30 smithi189 ceph-mon[34842]: pgmap v474: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:30.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:30 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2751366845' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-10T11:12:30.663 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-10T11:12:30.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:12:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:32.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:32 smithi027 ceph-mon[30054]: pgmap v475: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:32 smithi076 ceph-mon[34152]: pgmap v475: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:32.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:32 smithi189 ceph-mon[34842]: pgmap v475: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:33.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.416+0000 7fba74c00700 1 -- 172.21.15.27:0/2348303635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba7006b6f0 msgr2=0x7fba7006bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:33.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.416+0000 7fba74c00700 1 --2- 172.21.15.27:0/2348303635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba7006b6f0 0x7fba7006bb50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fba58009990 tx=0x7fba58009ca0 comp rx=0 tx=0).stop 2024-05-10T11:12:33.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.416+0000 7fba74c00700 1 -- 172.21.15.27:0/2348303635 shutdown_connections 2024-05-10T11:12:33.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.416+0000 7fba74c00700 1 --2- 172.21.15.27:0/2348303635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba70105ee0 0x7fba70106370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:33.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.416+0000 7fba74c00700 1 --2- 172.21.15.27:0/2348303635 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba701083d0 0x7fba7006b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:33.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.416+0000 7fba74c00700 1 --2- 172.21.15.27:0/2348303635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba7006b6f0 0x7fba7006bb50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:33.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.416+0000 7fba74c00700 1 -- 172.21.15.27:0/2348303635 >> 172.21.15.27:0/2348303635 conn(0x7fba70079350 msgr2=0x7fba70079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:33.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.416+0000 7fba74c00700 1 -- 172.21.15.27:0/2348303635 shutdown_connections 2024-05-10T11:12:33.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.416+0000 7fba74c00700 1 -- 172.21.15.27:0/2348303635 wait complete. 2024-05-10T11:12:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.417+0000 7fba74c00700 1 Processor -- start 2024-05-10T11:12:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.417+0000 7fba74c00700 1 -- start start 2024-05-10T11:12:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.418+0000 7fba74c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7006b6f0 0x7fba701a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.418+0000 7fba74c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba70105ee0 0x7fba701a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.418+0000 7fba74c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba701083d0 0x7fba701a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.418+0000 7fba74c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fba70119c20 con 0x7fba70105ee0 2024-05-10T11:12:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.418+0000 7fba74c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fba70119aa0 con 0x7fba701083d0 2024-05-10T11:12:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.418+0000 7fba74c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba70119da0 con 0x7fba7006b6f0 2024-05-10T11:12:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.418+0000 7fba67e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba70105ee0 0x7fba701a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.418+0000 7fba6de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7006b6f0 0x7fba701a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.418+0000 7fba67e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba70105ee0 0x7fba701a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54380/0 (socket says 172.21.15.27:54380) 2024-05-10T11:12:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.418+0000 7fba67e00700 1 -- 172.21.15.27:0/2630092086 learned_addr learned my addr 172.21.15.27:0/2630092086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.418+0000 7fba6e800700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba701083d0 0x7fba701a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.419+0000 7fba67e00700 1 -- 172.21.15.27:0/2630092086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7006b6f0 msgr2=0x7fba701a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.419+0000 7fba67e00700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7006b6f0 0x7fba701a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.419+0000 7fba67e00700 1 -- 172.21.15.27:0/2630092086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba701083d0 msgr2=0x7fba701a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.419+0000 7fba67e00700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba701083d0 0x7fba701a49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.419+0000 7fba67e00700 1 -- 172.21.15.27:0/2630092086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba701a5120 con 0x7fba70105ee0 2024-05-10T11:12:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.419+0000 7fba6e800700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba701083d0 0x7fba701a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.419+0000 7fba67e00700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba70105ee0 0x7fba701a0650 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fba580057f0 tx=0x7fba5800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.420+0000 7fba66a00700 1 -- 172.21.15.27:0/2630092086 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58026070 con 0x7fba70105ee0 2024-05-10T11:12:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.420+0000 7fba74c00700 1 -- 172.21.15.27:0/2630092086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba701a53b0 con 0x7fba70105ee0 2024-05-10T11:12:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.420+0000 7fba66a00700 1 -- 172.21.15.27:0/2630092086 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba58021410 con 0x7fba70105ee0 2024-05-10T11:12:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.420+0000 7fba66a00700 1 -- 172.21.15.27:0/2630092086 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58017650 con 0x7fba70105ee0 2024-05-10T11:12:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.420+0000 7fba74c00700 1 -- 172.21.15.27:0/2630092086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba701acc90 con 0x7fba70105ee0 2024-05-10T11:12:33.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.421+0000 7fba74c00700 1 -- 172.21.15.27:0/2630092086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba7006a190 con 0x7fba70105ee0 2024-05-10T11:12:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.423+0000 7fba66a00700 1 -- 172.21.15.27:0/2630092086 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fba58029070 con 0x7fba70105ee0 2024-05-10T11:12:33.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.425+0000 7fba66a00700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fba540b2880 0x7fba540b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:33.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.425+0000 7fba66a00700 1 -- 172.21.15.27:0/2630092086 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fba580e2fb0 con 0x7fba70105ee0 2024-05-10T11:12:33.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.425+0000 7fba6de00700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fba540b2880 0x7fba540b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:33.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.426+0000 7fba6de00700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fba540b2880 0x7fba540b4d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fba60009be0 tx=0x7fba60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:33.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.430+0000 7fba66a00700 1 -- 172.21.15.27:0/2630092086 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fba58013070 con 0x7fba70105ee0 2024-05-10T11:12:33.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.718+0000 7fba74c00700 1 -- 172.21.15.27:0/2630092086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba701a1470 con 0x7fba70105ee0 2024-05-10T11:12:33.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.721+0000 7fba66a00700 1 -- 172.21.15.27:0/2630092086 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11936 (secure 0 0 0) 0x7fba580aaa00 con 0x7fba70105ee0 2024-05-10T11:12:33.722 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:12:33.723 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:09:19.986922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:12:33.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.725+0000 7fba74c00700 1 -- 172.21.15.27:0/2630092086 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fba540b2880 msgr2=0x7fba540b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:33.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.725+0000 7fba74c00700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fba540b2880 0x7fba540b4d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fba60009be0 tx=0x7fba60009260 comp rx=0 tx=0).stop 2024-05-10T11:12:33.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.725+0000 7fba74c00700 1 -- 172.21.15.27:0/2630092086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba70105ee0 msgr2=0x7fba701a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:33.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.725+0000 7fba74c00700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba70105ee0 0x7fba701a0650 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fba580057f0 tx=0x7fba5800bfd0 comp rx=0 tx=0).stop 2024-05-10T11:12:33.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.725+0000 7fba74c00700 1 -- 172.21.15.27:0/2630092086 shutdown_connections 2024-05-10T11:12:33.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.725+0000 7fba74c00700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fba540b2880 0x7fba540b4d40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:33.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.725+0000 7fba74c00700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7006b6f0 0x7fba701a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:33.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.725+0000 7fba74c00700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba701083d0 0x7fba701a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:33.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.725+0000 7fba74c00700 1 --2- 172.21.15.27:0/2630092086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba70105ee0 0x7fba701a0650 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:33.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.725+0000 7fba74c00700 1 -- 172.21.15.27:0/2630092086 >> 172.21.15.27:0/2630092086 conn(0x7fba70079350 msgr2=0x7fba70104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:33.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.725+0000 7fba74c00700 1 -- 172.21.15.27:0/2630092086 shutdown_connections 2024-05-10T11:12:33.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:33.726+0000 7fba74c00700 1 -- 172.21.15.27:0/2630092086 wait complete. 2024-05-10T11:12:33.727 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-05-10T11:12:34.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:34 smithi027 ceph-mon[30054]: pgmap v476: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:34.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:34 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2630092086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:12:34.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:34 smithi076 ceph-mon[34152]: pgmap v476: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:34.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:34 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2630092086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:12:34.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:34 smithi189 ceph-mon[34842]: pgmap v476: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:34.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:34 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2630092086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:12:34.729 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:12:35.208 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:36.289 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:36 smithi189 ceph-mon[34842]: pgmap v477: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:36.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:36 smithi027 ceph-mon[30054]: pgmap v477: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:36.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:36 smithi076 ceph-mon[34152]: pgmap v477: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:37.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.649+0000 7f3539800700 1 -- 172.21.15.27:0/2316661338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3534075ff0 msgr2=0x7f353410f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:37.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.649+0000 7f3539800700 1 --2- 172.21.15.27:0/2316661338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3534075ff0 0x7f353410f960 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3520009990 tx=0x7f3520009ca0 comp rx=0 tx=0).stop 2024-05-10T11:12:37.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.649+0000 7f3539800700 1 -- 172.21.15.27:0/2316661338 shutdown_connections 2024-05-10T11:12:37.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.649+0000 7f3539800700 1 --2- 172.21.15.27:0/2316661338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f353410fea0 0x7f3534112290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:37.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.649+0000 7f3539800700 1 --2- 172.21.15.27:0/2316661338 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f35340756d0 0x7f3534075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:37.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.649+0000 7f3539800700 1 --2- 172.21.15.27:0/2316661338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3534075ff0 0x7f353410f960 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:37.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.649+0000 7f3539800700 1 -- 172.21.15.27:0/2316661338 >> 172.21.15.27:0/2316661338 conn(0x7f353407a600 msgr2=0x7f353407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:37.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.649+0000 7f3539800700 1 -- 172.21.15.27:0/2316661338 shutdown_connections 2024-05-10T11:12:37.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.650+0000 7f3539800700 1 -- 172.21.15.27:0/2316661338 wait complete. 2024-05-10T11:12:37.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.651+0000 7f3539800700 1 Processor -- start 2024-05-10T11:12:37.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.651+0000 7f3539800700 1 -- start start 2024-05-10T11:12:37.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.652+0000 7f3539800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340756d0 0x7f353419bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:37.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.652+0000 7f3539800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3534075ff0 0x7f353419c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:37.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.652+0000 7f3539800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f353410fea0 0x7f35341a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:37.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.652+0000 7f3539800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f35341158d0 con 0x7f35340756d0 2024-05-10T11:12:37.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.652+0000 7f3539800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3534115750 con 0x7f3534075ff0 2024-05-10T11:12:37.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.652+0000 7f3539800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3534115a50 con 0x7f353410fea0 2024-05-10T11:12:37.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.652+0000 7f3532a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340756d0 0x7f353419bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:37.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.652+0000 7f3532000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3534075ff0 0x7f353419c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:37.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.652+0000 7f3532a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340756d0 0x7f353419bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50256/0 (socket says 172.21.15.27:50256) 2024-05-10T11:12:37.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.652+0000 7f3532a00700 1 -- 172.21.15.27:0/2968223867 learned_addr learned my addr 172.21.15.27:0/2968223867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:37.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.652+0000 7f3533400700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f353410fea0 0x7f35341a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:37.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.653+0000 7f3532000700 1 -- 172.21.15.27:0/2968223867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f353410fea0 msgr2=0x7f35341a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.653+0000 7f3532000700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f353410fea0 0x7f35341a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.653+0000 7f3532000700 1 -- 172.21.15.27:0/2968223867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340756d0 msgr2=0x7f353419bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.653+0000 7f3532000700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340756d0 0x7f353419bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.653+0000 7f3532000700 1 -- 172.21.15.27:0/2968223867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35341a0d30 con 0x7f3534075ff0 2024-05-10T11:12:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.653+0000 7f3532a00700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340756d0 0x7f353419bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.653+0000 7f3532000700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3534075ff0 0x7f353419c260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f35200098e0 tx=0x7f35200057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.654+0000 7f352b400700 1 -- 172.21.15.27:0/2968223867 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3520027070 con 0x7f3534075ff0 2024-05-10T11:12:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.654+0000 7f352b400700 1 -- 172.21.15.27:0/2968223867 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3520022470 con 0x7f3534075ff0 2024-05-10T11:12:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.654+0000 7f3539800700 1 -- 172.21.15.27:0/2968223867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35341a0fc0 con 0x7f3534075ff0 2024-05-10T11:12:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.654+0000 7f352b400700 1 -- 172.21.15.27:0/2968223867 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3520017650 con 0x7f3534075ff0 2024-05-10T11:12:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.654+0000 7f3539800700 1 -- 172.21.15.27:0/2968223867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35341a88a0 con 0x7f3534075ff0 2024-05-10T11:12:37.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.655+0000 7f3539800700 1 -- 172.21.15.27:0/2968223867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3534076dc0 con 0x7f3534075ff0 2024-05-10T11:12:37.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.658+0000 7f352b400700 1 -- 172.21.15.27:0/2968223867 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f352002a070 con 0x7f3534075ff0 2024-05-10T11:12:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.659+0000 7f352b400700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f35180b2880 0x7f35180b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.659+0000 7f352b400700 1 -- 172.21.15.27:0/2968223867 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f35200e3450 con 0x7f3534075ff0 2024-05-10T11:12:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.660+0000 7f3532a00700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f35180b2880 0x7f35180b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.660+0000 7f3532a00700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f35180b2880 0x7f35180b4d40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3524011550 tx=0x7f3524009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.661+0000 7f352b400700 1 -- 172.21.15.27:0/2968223867 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f3520013070 con 0x7f3534075ff0 2024-05-10T11:12:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.953+0000 7f3539800700 1 -- 172.21.15.27:0/2968223867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f353419d080 con 0x7f3534075ff0 2024-05-10T11:12:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.955+0000 7f352b400700 1 -- 172.21.15.27:0/2968223867 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11936 (secure 0 0 0) 0x7f35200aae10 con 0x7f3534075ff0 2024-05-10T11:12:37.956 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:12:37.956 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:09:19.986922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:12:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.959+0000 7f3539800700 1 -- 172.21.15.27:0/2968223867 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f35180b2880 msgr2=0x7f35180b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.959+0000 7f3539800700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f35180b2880 0x7f35180b4d40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3524011550 tx=0x7f3524009260 comp rx=0 tx=0).stop 2024-05-10T11:12:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.959+0000 7f3539800700 1 -- 172.21.15.27:0/2968223867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3534075ff0 msgr2=0x7f353419c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.959+0000 7f3539800700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3534075ff0 0x7f353419c260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f35200098e0 tx=0x7f35200057c0 comp rx=0 tx=0).stop 2024-05-10T11:12:37.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.960+0000 7f3539800700 1 -- 172.21.15.27:0/2968223867 shutdown_connections 2024-05-10T11:12:37.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.960+0000 7f3539800700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f35180b2880 0x7f35180b4d40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:37.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.960+0000 7f3539800700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f353410fea0 0x7f35341a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:37.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.960+0000 7f3539800700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340756d0 0x7f353419bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:37.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.960+0000 7f3539800700 1 --2- 172.21.15.27:0/2968223867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3534075ff0 0x7f353419c260 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:37.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.960+0000 7f3539800700 1 -- 172.21.15.27:0/2968223867 >> 172.21.15.27:0/2968223867 conn(0x7f353407a600 msgr2=0x7f3534100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:37.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.960+0000 7f3539800700 1 -- 172.21.15.27:0/2968223867 shutdown_connections 2024-05-10T11:12:37.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:37.960+0000 7f3539800700 1 -- 172.21.15.27:0/2968223867 wait complete. 2024-05-10T11:12:37.962 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-05-10T11:12:38.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:38 smithi027 ceph-mon[30054]: pgmap v478: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:38.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:38 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2968223867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:12:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:38 smithi076 ceph-mon[34152]: pgmap v478: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:38 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2968223867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:12:38.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:38 smithi189 ceph-mon[34842]: pgmap v478: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:38.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:38 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2968223867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:12:38.913 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:12:39.376 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:40.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:40 smithi027 ceph-mon[30054]: pgmap v479: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:40 smithi076 ceph-mon[34152]: pgmap v479: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:40.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:40 smithi189 ceph-mon[34842]: pgmap v479: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:41.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.570+0000 7f389cc00700 1 -- 172.21.15.27:0/3708414257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3898103ef0 msgr2=0x7f38981103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:41.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.570+0000 7f389cc00700 1 --2- 172.21.15.27:0/3708414257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3898103ef0 0x7f38981103d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3890009990 tx=0x7f3890009ca0 comp rx=0 tx=0).stop 2024-05-10T11:12:41.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.570+0000 7f389cc00700 1 -- 172.21.15.27:0/3708414257 shutdown_connections 2024-05-10T11:12:41.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.570+0000 7f389cc00700 1 --2- 172.21.15.27:0/3708414257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3898103ef0 0x7f38981103d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:41.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.570+0000 7f389cc00700 1 --2- 172.21.15.27:0/3708414257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3898108460 0x7f3898108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:41.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.570+0000 7f389cc00700 1 --2- 172.21.15.27:0/3708414257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3898103550 0x7f38981039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:41.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.570+0000 7f389cc00700 1 -- 172.21.15.27:0/3708414257 >> 172.21.15.27:0/3708414257 conn(0x7f389807a600 msgr2=0x7f389807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:41.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.570+0000 7f389cc00700 1 -- 172.21.15.27:0/3708414257 shutdown_connections 2024-05-10T11:12:41.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.570+0000 7f389cc00700 1 -- 172.21.15.27:0/3708414257 wait complete. 2024-05-10T11:12:41.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.572+0000 7f389cc00700 1 Processor -- start 2024-05-10T11:12:41.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.572+0000 7f389cc00700 1 -- start start 2024-05-10T11:12:41.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.572+0000 7f389cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3898103550 0x7f389819bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:41.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.572+0000 7f389cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3898103ef0 0x7f389819c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:41.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.572+0000 7f389cc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3898108460 0x7f38981a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:41.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.573+0000 7f389cc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f38981136f0 con 0x7f3898103550 2024-05-10T11:12:41.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.573+0000 7f389cc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3898113570 con 0x7f3898108460 2024-05-10T11:12:41.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.573+0000 7f389cc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3898113870 con 0x7f3898103ef0 2024-05-10T11:12:41.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.573+0000 7f3895e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3898103550 0x7f389819bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:41.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.573+0000 7f3895e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3898103550 0x7f389819bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50278/0 (socket says 172.21.15.27:50278) 2024-05-10T11:12:41.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.573+0000 7f3895e00700 1 -- 172.21.15.27:0/2633510291 learned_addr learned my addr 172.21.15.27:0/2633510291 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:41.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.573+0000 7f3896800700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3898108460 0x7f38981a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:41.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.573+0000 7f3895400700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3898103ef0 0x7f389819c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:41.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.573+0000 7f3896800700 1 -- 172.21.15.27:0/2633510291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3898103ef0 msgr2=0x7f389819c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:41.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.573+0000 7f3896800700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3898103ef0 0x7f389819c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:41.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.573+0000 7f3896800700 1 -- 172.21.15.27:0/2633510291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3898103550 msgr2=0x7f389819bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:41.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.573+0000 7f3896800700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3898103550 0x7f389819bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:41.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.574+0000 7f3896800700 1 -- 172.21.15.27:0/2633510291 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38981a0dd0 con 0x7f3898108460 2024-05-10T11:12:41.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.574+0000 7f3895e00700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3898103550 0x7f389819bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:41.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.574+0000 7f3896800700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3898108460 0x7f38981a0690 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3890006010 tx=0x7f38900057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:41.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.574+0000 7f388ea00700 1 -- 172.21.15.27:0/2633510291 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389000be90 con 0x7f3898108460 2024-05-10T11:12:41.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.574+0000 7f389cc00700 1 -- 172.21.15.27:0/2633510291 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38981a1060 con 0x7f3898108460 2024-05-10T11:12:41.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.575+0000 7f388ea00700 1 -- 172.21.15.27:0/2633510291 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3890022470 con 0x7f3898108460 2024-05-10T11:12:41.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.575+0000 7f389cc00700 1 -- 172.21.15.27:0/2633510291 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38981a8940 con 0x7f3898108460 2024-05-10T11:12:41.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.575+0000 7f388ea00700 1 -- 172.21.15.27:0/2633510291 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3890017630 con 0x7f3898108460 2024-05-10T11:12:41.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.576+0000 7f389cc00700 1 -- 172.21.15.27:0/2633510291 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3898104c10 con 0x7f3898108460 2024-05-10T11:12:41.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.578+0000 7f388ea00700 1 -- 172.21.15.27:0/2633510291 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f3890009d10 con 0x7f3898108460 2024-05-10T11:12:41.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.580+0000 7f388ea00700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f387c0b2a20 0x7f387c0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:41.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.580+0000 7f388ea00700 1 -- 172.21.15.27:0/2633510291 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f38900e33e0 con 0x7f3898108460 2024-05-10T11:12:41.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.580+0000 7f3895e00700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f387c0b2a20 0x7f387c0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:41.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.581+0000 7f3895e00700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f387c0b2a20 0x7f387c0b4ee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3888003ef0 tx=0x7f388800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:41.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.582+0000 7f388ea00700 1 -- 172.21.15.27:0/2633510291 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f3890013070 con 0x7f3898108460 2024-05-10T11:12:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.821+0000 7f389cc00700 1 -- 172.21.15.27:0/2633510291 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38981a8b10 con 0x7f3898108460 2024-05-10T11:12:41.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.823+0000 7f388ea00700 1 -- 172.21.15.27:0/2633510291 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24835 (secure 0 0 0) 0x7f38900aad40 con 0x7f3898108460 2024-05-10T11:12:41.824 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:12:41.825 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":90,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:09:23.055559+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6},{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:12:41.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.826+0000 7f389cc00700 1 -- 172.21.15.27:0/2633510291 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f387c0b2a20 msgr2=0x7f387c0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:41.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.826+0000 7f389cc00700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f387c0b2a20 0x7f387c0b4ee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3888003ef0 tx=0x7f388800b040 comp rx=0 tx=0).stop 2024-05-10T11:12:41.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.827+0000 7f389cc00700 1 -- 172.21.15.27:0/2633510291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3898108460 msgr2=0x7f38981a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:41.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.827+0000 7f389cc00700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3898108460 0x7f38981a0690 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3890006010 tx=0x7f38900057c0 comp rx=0 tx=0).stop 2024-05-10T11:12:41.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.827+0000 7f389cc00700 1 -- 172.21.15.27:0/2633510291 shutdown_connections 2024-05-10T11:12:41.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.827+0000 7f389cc00700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f387c0b2a20 0x7f387c0b4ee0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:41.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.827+0000 7f389cc00700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3898108460 0x7f38981a0690 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:41.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.827+0000 7f389cc00700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3898103550 0x7f389819bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:41.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.827+0000 7f389cc00700 1 --2- 172.21.15.27:0/2633510291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3898103ef0 0x7f389819c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:41.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.827+0000 7f389cc00700 1 -- 172.21.15.27:0/2633510291 >> 172.21.15.27:0/2633510291 conn(0x7f389807a600 msgr2=0x7f38981013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:41.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.827+0000 7f389cc00700 1 -- 172.21.15.27:0/2633510291 shutdown_connections 2024-05-10T11:12:41.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:41.827+0000 7f389cc00700 1 -- 172.21.15.27:0/2633510291 wait complete. 2024-05-10T11:12:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:42 smithi027 ceph-mon[30054]: pgmap v480: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:42.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:42 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2633510291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:12:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:42 smithi076 ceph-mon[34152]: pgmap v480: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:42 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2633510291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:12:42.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:42 smithi189 ceph-mon[34842]: pgmap v480: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:42.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:42 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2633510291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:12:42.838 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-10T11:07:50.237787+0000', 'modified': '2024-05-10T11:09:19.986922+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24527, 'mds_1': 24488, 'mds_2': 24484, 'mds_3': 24472, 'mds_4': 24467}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14403': {'gid': 14403, 'name': 'a', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6841/3080402649', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6840', 'nonce': 3080402649}, {'type': 'v1', 'addr': '172.21.15.27:6841', 'nonce': 3080402649}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24418': {'gid': 24418, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6833/2718624698', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6832', 'nonce': 2718624698}, {'type': 'v1', 'addr': '172.21.15.189:6833', 'nonce': 2718624698}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24437': {'gid': 24437, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.76:6833/1370063583', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 1370063583}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 1370063583}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24449': {'gid': 24449, 'name': 'j', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6835/3274866857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6834', 'nonce': 3274866857}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 3274866857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6835/1387617890', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 1387617890}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 1387617890}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'e', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.76:6835/3618511593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6834', 'nonce': 3618511593}, {'type': 'v1', 'addr': '172.21.15.76:6835', 'nonce': 3618511593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 33, 'addr': '172.21.15.189:6837/3185830081', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 3185830081}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 3185830081}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.189:6839/2071082194', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 2071082194}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 2071082194}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 32, 'addr': '172.21.15.76:6837/569330029', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6836', 'nonce': 569330029}, {'type': 'v1', 'addr': '172.21.15.76:6837', 'nonce': 569330029}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.76:6839/1778204420', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6838', 'nonce': 1778204420}, {'type': 'v1', 'addr': '172.21.15.76:6839', 'nonce': 1778204420}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24527, 'qdb_cluster': [24467, 24488, 24472, 24484, 24527]} 2024-05-10T11:12:42.838 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-10T11:12:42.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.e status 2024-05-10T11:12:43.315 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:44.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:44 smithi027 ceph-mon[30054]: pgmap v481: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:44.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:44 smithi076 ceph-mon[34152]: pgmap v481: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:44.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:44 smithi189 ceph-mon[34842]: pgmap v481: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:45.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.691+0000 7f30ba600700 1 -- 172.21.15.27:0/2182361386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30b410ff00 msgr2=0x7f30b41122f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.691+0000 7f30ba600700 1 --2- 172.21.15.27:0/2182361386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30b410ff00 0x7f30b41122f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f30ac00b0a0 tx=0x7f30ac00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.692+0000 7f30ba600700 1 -- 172.21.15.27:0/2182361386 shutdown_connections 2024-05-10T11:12:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.692+0000 7f30ba600700 1 --2- 172.21.15.27:0/2182361386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b40756d0 0x7f30b4075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.692+0000 7f30ba600700 1 --2- 172.21.15.27:0/2182361386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30b410ff00 0x7f30b41122f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.692+0000 7f30ba600700 1 --2- 172.21.15.27:0/2182361386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b4075ff0 0x7f30b410f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.692+0000 7f30ba600700 1 -- 172.21.15.27:0/2182361386 >> 172.21.15.27:0/2182361386 conn(0x7f30b407a600 msgr2=0x7f30b407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.692+0000 7f30ba600700 1 -- 172.21.15.27:0/2182361386 shutdown_connections 2024-05-10T11:12:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.692+0000 7f30ba600700 1 -- 172.21.15.27:0/2182361386 wait complete. 2024-05-10T11:12:45.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.693+0000 7f30ba600700 1 Processor -- start 2024-05-10T11:12:45.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.693+0000 7f30ba600700 1 -- start start 2024-05-10T11:12:45.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30ba600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40756d0 0x7f30b41a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30ba600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b4075ff0 0x7f30b41a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30ba600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30b410ff00 0x7f30b419bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30ba600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f30b41158e0 con 0x7f30b40756d0 2024-05-10T11:12:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30ba600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f30b4115760 con 0x7f30b410ff00 2024-05-10T11:12:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30ba600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f30b4115a60 con 0x7f30b4075ff0 2024-05-10T11:12:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30b3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40756d0 0x7f30b41a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30b3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40756d0 0x7f30b41a1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58392/0 (socket says 172.21.15.27:58392) 2024-05-10T11:12:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30b3400700 1 -- 172.21.15.27:0/1365917213 learned_addr learned my addr 172.21.15.27:0/1365917213 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30b2a00700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b4075ff0 0x7f30b41a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30b3e00700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30b410ff00 0x7f30b419bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30b2a00700 1 -- 172.21.15.27:0/1365917213 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30b410ff00 msgr2=0x7f30b419bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30b2a00700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30b410ff00 0x7f30b419bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30b2a00700 1 -- 172.21.15.27:0/1365917213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40756d0 msgr2=0x7f30b41a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30b2a00700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40756d0 0x7f30b41a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.694+0000 7f30b2a00700 1 -- 172.21.15.27:0/1365917213 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b419c400 con 0x7f30b4075ff0 2024-05-10T11:12:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.695+0000 7f30b3400700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40756d0 0x7f30b41a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.695+0000 7f30b2a00700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b4075ff0 0x7f30b41a1f90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f30a000e9c0 tx=0x7f30a000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.695+0000 7f30abe00700 1 -- 172.21.15.27:0/1365917213 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a000cb70 con 0x7f30b4075ff0 2024-05-10T11:12:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.695+0000 7f30abe00700 1 -- 172.21.15.27:0/1365917213 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30a000cd10 con 0x7f30b4075ff0 2024-05-10T11:12:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.695+0000 7f30ba600700 1 -- 172.21.15.27:0/1365917213 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30b419c6f0 con 0x7f30b4075ff0 2024-05-10T11:12:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.695+0000 7f30abe00700 1 -- 172.21.15.27:0/1365917213 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a0018680 con 0x7f30b4075ff0 2024-05-10T11:12:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.695+0000 7f30ba600700 1 -- 172.21.15.27:0/1365917213 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30b41a89d0 con 0x7f30b4075ff0 2024-05-10T11:12:45.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.700+0000 7f30abe00700 1 -- 172.21.15.27:0/1365917213 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f30a001f020 con 0x7f30b4075ff0 2024-05-10T11:12:45.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.700+0000 7f30c501eb80 1 Processor -- start 2024-05-10T11:12:45.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.700+0000 7f30c501eb80 1 -- start start 2024-05-10T11:12:45.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.701+0000 7f30c501eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc79b70 0x55fb5dc79f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.701+0000 7f30c501eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc7a490 0x55fb5dc74280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.701+0000 7f30c501eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc748a0 0x55fb5dc74d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.701+0000 7f30c501eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fb5dbc69d0 con 0x55fb5dc79b70 2024-05-10T11:12:45.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.701+0000 7f30c501eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55fb5db64710 con 0x55fb5dc7a490 2024-05-10T11:12:45.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.701+0000 7f30c501eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fb5dbc2990 con 0x55fb5dc748a0 2024-05-10T11:12:45.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.701+0000 7f30b3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc79b70 0x55fb5dc79f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.701+0000 7f30b3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc79b70 0x55fb5dc79f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58400/0 (socket says 172.21.15.27:58400) 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.701+0000 7f30b3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc7a490 0x55fb5dc74280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.701+0000 7f30b3e00700 1 -- 172.21.15.27:0/1099892551 learned_addr learned my addr 172.21.15.27:0/1099892551 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.701+0000 7f30b2a00700 1 --2- 172.21.15.27:0/1099892551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc748a0 0x55fb5dc74d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.702+0000 7f30b3e00700 1 -- 172.21.15.27:0/1099892551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc748a0 msgr2=0x55fb5dc74d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.702+0000 7f30b3e00700 1 --2- 172.21.15.27:0/1099892551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc748a0 0x55fb5dc74d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.702+0000 7f30b3e00700 1 -- 172.21.15.27:0/1099892551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc7a490 msgr2=0x55fb5dc74280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.702+0000 7f30b3e00700 1 --2- 172.21.15.27:0/1099892551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc7a490 0x55fb5dc74280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.702+0000 7f30abe00700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30980b2880 0x7f30980b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.702+0000 7f30b3e00700 1 -- 172.21.15.27:0/1099892551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb5db13880 con 0x55fb5dc79b70 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.702+0000 7f30b2a00700 1 --2- 172.21.15.27:0/1099892551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc748a0 0x55fb5dc74d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.702+0000 7f30abe00700 1 -- 172.21.15.27:0/1365917213 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f30a00e2480 con 0x7f30b4075ff0 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.702+0000 7f30b3e00700 1 --2- 172.21.15.27:0/1099892551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc79b70 0x55fb5dc79f50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f30ac00b0a0 tx=0x7f30ac0092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49ae08ef5b1e360 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.702+0000 7f30b2a00700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30980b2880 0x7f30980b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.702+0000 7f30a8c00700 1 -- 172.21.15.27:0/1099892551 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac00bcb0 con 0x55fb5dc79b70 2024-05-10T11:12:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.703+0000 7f30a8c00700 1 -- 172.21.15.27:0/1099892551 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30ac0074e0 con 0x55fb5dc79b70 2024-05-10T11:12:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.703+0000 7f30a8c00700 1 -- 172.21.15.27:0/1099892551 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac0052a0 con 0x55fb5dc79b70 2024-05-10T11:12:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.703+0000 7f30c501eb80 1 -- 172.21.15.27:0/1099892551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc79b70 msgr2=0x55fb5dc79f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.703+0000 7f30c501eb80 1 --2- 172.21.15.27:0/1099892551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc79b70 0x55fb5dc79f50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f30ac00b0a0 tx=0x7f30ac0092e0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.703+0000 7f30b2a00700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30980b2880 0x7f30980b4d40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f30a00dff00 tx=0x7f30a0010c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.704+0000 7f30c501eb80 1 -- 172.21.15.27:0/1099892551 shutdown_connections 2024-05-10T11:12:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.704+0000 7f30c501eb80 1 --2- 172.21.15.27:0/1099892551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc748a0 0x55fb5dc74d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.704+0000 7f30c501eb80 1 --2- 172.21.15.27:0/1099892551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc79b70 0x55fb5dc79f50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.704+0000 7f30c501eb80 1 --2- 172.21.15.27:0/1099892551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc7a490 0x55fb5dc74280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.704+0000 7f30c501eb80 1 -- 172.21.15.27:0/1099892551 >> 172.21.15.27:0/1099892551 conn(0x55fb5dbec680 msgr2=0x55fb5dc6eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:45.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.704+0000 7f30c501eb80 1 -- 172.21.15.27:0/1099892551 shutdown_connections 2024-05-10T11:12:45.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.704+0000 7f30c501eb80 1 -- 172.21.15.27:0/1099892551 wait complete. 2024-05-10T11:12:45.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.704+0000 7f30c501eb80 1 -- start start 2024-05-10T11:12:45.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.705+0000 7f30c501eb80 1 Processor -- start 2024-05-10T11:12:45.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.705+0000 7f30c501eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc801c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.705+0000 7f30c501eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc7d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.705+0000 7f30c501eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 0x55fb5dc7daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.705+0000 7f30c501eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fb5dc80e70 con 0x55fb5dc748a0 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.705+0000 7f30c501eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55fb5db2cad0 con 0x55fb5dc79b70 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.705+0000 7f30c501eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fb5dc80ff0 con 0x55fb5dc7a490 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.705+0000 7f30c501eb80 10 client.0 _ll_register_callbacks cb 0x55fb5db6dc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.705+0000 7f30b3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc801c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.706+0000 7f30b3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc801c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58404/0 (socket says 172.21.15.27:58404) 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.706+0000 7f30b3e00700 1 -- 172.21.15.27:0/239350044 learned_addr learned my addr 172.21.15.27:0/239350044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.706+0000 7f30b3400700 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc7d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.706+0000 7f30b3e00700 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 0x55fb5dc7daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.706+0000 7f30b3400700 1 -- 172.21.15.27:0/239350044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 msgr2=0x55fb5dc7daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:45.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.706+0000 7f30b3400700 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 0x55fb5dc7daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.706+0000 7f30b3400700 1 -- 172.21.15.27:0/239350044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 msgr2=0x55fb5dc801c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.706+0000 7f30b3400700 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc801c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.706+0000 7f30b3400700 1 -- 172.21.15.27:0/239350044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb5db8c3d0 con 0x55fb5dc79b70 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.706+0000 7f30b3400700 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc7d560 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f30a40096b0 tx=0x7f30a4011db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdd8566a14c550ce server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.706+0000 7f309d600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.707+0000 7f309d600700 1 -- 172.21.15.27:0/239350044 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a4004260 con 0x55fb5dc79b70 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.707+0000 7f309d600700 1 -- 172.21.15.27:0/239350044 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f30a4004400 con 0x55fb5dc79b70 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.707+0000 7f30c501eb80 1 -- 172.21.15.27:0/239350044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30ac0074e0 con 0x55fb5dc79b70 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.707+0000 7f309d600700 1 -- 172.21.15.27:0/239350044 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f30a4025050 con 0x55fb5dc79b70 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.707+0000 7f30c501eb80 10 client.25121 fetch_fsmap learned FSMap version 29 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.707+0000 7f30c501eb80 1 -- 172.21.15.27:0/239350044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fb5dc8b720 con 0x55fb5dc79b70 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.707+0000 7f309d600700 1 -- 172.21.15.27:0/239350044 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30ac0074e0 con 0x55fb5dc79b70 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.708+0000 7f309d600700 1 -- 172.21.15.27:0/239350044 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f30a4003b60 con 0x55fb5dc79b70 2024-05-10T11:12:45.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.708+0000 7f30c501eb80 10 client.25121 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:12:45.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.708+0000 7f30c501eb80 10 client.25121 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-10T11:12:45.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.708+0000 7f30c501eb80 20 client.25121 populate_metadata read hostname 'smithi027' 2024-05-10T11:12:45.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.708+0000 7f30c501eb80 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55fb5dc8ddc0 0x55fb5dc8f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.708+0000 7f30c501eb80 4 client.25121 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:12:45.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.708+0000 7f30c501eb80 1 -- 172.21.15.27:0/239350044 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb5dc410f0 con 0x55fb5dc8ddc0 2024-05-10T11:12:45.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.708+0000 7f30b3e00700 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55fb5dc8ddc0 0x55fb5dc8f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.709+0000 7f30b3e00700 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55fb5dc8ddc0 0x55fb5dc8f230 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=69e251966b2d4ddc server_cookie=258afd9d79f591d6 in_seq=0 out_seq=0 2024-05-10T11:12:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.709+0000 7f309d600700 10 client.25121 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:12:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f309d600700 1 -- 172.21.15.27:0/239350044 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fb5dc410f0 con 0x55fb5dc8ddc0 2024-05-10T11:12:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f309d600700 10 client.25121 handle_command_reply: tid=0 2024-05-10T11:12:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f30c501eb80 1 client.25121 shutdown 2024-05-10T11:12:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f30c501eb80 1 -- 172.21.15.27:0/239350044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 msgr2=0x55fb5dc7d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f30c501eb80 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc7d560 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f30a40096b0 tx=0x7f30a4011db0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f30c501eb80 1 -- 172.21.15.27:0/239350044 shutdown_connections 2024-05-10T11:12:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f30c501eb80 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55fb5dc8ddc0 0x55fb5dc8f230 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f30c501eb80 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 0x55fb5dc7daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f30c501eb80 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc801c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f30c501eb80 1 --2- 172.21.15.27:0/239350044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc7d560 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f309d600700 0 client.25121 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:12:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f30c501eb80 1 -- 172.21.15.27:0/239350044 >> 172.21.15.27:0/239350044 conn(0x55fb5dbec680 msgr2=0x55fb5dc6e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f30c501eb80 1 -- 172.21.15.27:0/239350044 shutdown_connections 2024-05-10T11:12:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f30c501eb80 1 -- 172.21.15.27:0/239350044 wait complete. 2024-05-10T11:12:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.710+0000 7f30c501eb80 20 client.25121 trim_cache size 0 max 16384 2024-05-10T11:12:45.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.734+0000 7f30c501eb80 1 Processor -- start 2024-05-10T11:12:45.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.734+0000 7f30c501eb80 1 -- start start 2024-05-10T11:12:45.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.735+0000 7f30c501eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc74280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.735+0000 7f30c501eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc6e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.735+0000 7f30c501eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 0x55fb5dc7a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.735+0000 7f30c501eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fb5dc8d570 con 0x55fb5dc748a0 2024-05-10T11:12:45.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.735+0000 7f30c501eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55fb5dc8dc40 con 0x55fb5dc79b70 2024-05-10T11:12:45.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.735+0000 7f30c501eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fb5dc8e310 con 0x55fb5dc7a490 2024-05-10T11:12:45.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.735+0000 7f30b3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc74280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.735+0000 7f30b3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc6e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.735+0000 7f30b3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc74280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58408/0 (socket says 172.21.15.27:58408) 2024-05-10T11:12:45.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.735+0000 7f30b3e00700 1 -- 172.21.15.27:0/769679746 learned_addr learned my addr 172.21.15.27:0/769679746 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:45.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.735+0000 7f30b3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc6e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42066/0 (socket says 172.21.15.27:42066) 2024-05-10T11:12:45.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.735+0000 7f30b3e00700 1 --2- 172.21.15.27:0/769679746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 0x55fb5dc7a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.736+0000 7f30b3400700 1 -- 172.21.15.27:0/769679746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 msgr2=0x55fb5dc7a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:45.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.736+0000 7f30b3400700 1 --2- 172.21.15.27:0/769679746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 0x55fb5dc7a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.736+0000 7f30b3400700 1 -- 172.21.15.27:0/769679746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 msgr2=0x55fb5dc74280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.736+0000 7f30b3400700 1 --2- 172.21.15.27:0/769679746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc74280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.736+0000 7f30b3400700 1 -- 172.21.15.27:0/769679746 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb5dc70b20 con 0x55fb5dc79b70 2024-05-10T11:12:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.736+0000 7f30b3e00700 1 --2- 172.21.15.27:0/769679746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc74280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.736+0000 7f30b3400700 1 --2- 172.21.15.27:0/769679746 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc6e8c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f30a4004020 tx=0x7f30a401a830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88d3d26971e926e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.736+0000 7f309fe00700 1 -- 172.21.15.27:0/769679746 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a40052e0 con 0x55fb5dc79b70 2024-05-10T11:12:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.736+0000 7f309fe00700 1 -- 172.21.15.27:0/769679746 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30a4005480 con 0x55fb5dc79b70 2024-05-10T11:12:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.737+0000 7f309fe00700 1 -- 172.21.15.27:0/769679746 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a400fd00 con 0x55fb5dc79b70 2024-05-10T11:12:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.737+0000 7f30c501eb80 1 -- 172.21.15.27:0/769679746 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 msgr2=0x55fb5dc6e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.737+0000 7f30c501eb80 1 --2- 172.21.15.27:0/769679746 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc6e8c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f30a4004020 tx=0x7f30a401a830 comp rx=0 tx=0).stop 2024-05-10T11:12:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.737+0000 7f30c501eb80 1 -- 172.21.15.27:0/769679746 shutdown_connections 2024-05-10T11:12:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.737+0000 7f30c501eb80 1 --2- 172.21.15.27:0/769679746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 0x55fb5dc7a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.737+0000 7f30c501eb80 1 --2- 172.21.15.27:0/769679746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc74280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.737+0000 7f30c501eb80 1 --2- 172.21.15.27:0/769679746 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc6e8c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.737+0000 7f30c501eb80 1 -- 172.21.15.27:0/769679746 >> 172.21.15.27:0/769679746 conn(0x55fb5dbec680 msgr2=0x55fb5dc7be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.737+0000 7f30c501eb80 1 -- 172.21.15.27:0/769679746 shutdown_connections 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.738+0000 7f30c501eb80 1 -- 172.21.15.27:0/769679746 wait complete. 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.738+0000 7f30c501eb80 1 -- start start 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.738+0000 7f30c501eb80 1 Processor -- start 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30c501eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc6de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30c501eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc6e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30c501eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 0x55fb5dc6e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30c501eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fb5dc8f0b0 con 0x55fb5dc748a0 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30c501eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55fb5dc8e9e0 con 0x55fb5dc79b70 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30c501eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fb5dc8f780 con 0x55fb5dc7a490 2024-05-10T11:12:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30c501eb80 10 client.0 _ll_register_callbacks cb 0x55fb5db6dc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:12:45.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30b3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc6de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30b3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc6e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30b3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc6de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58420/0 (socket says 172.21.15.27:58420) 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30b3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc6e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42068/0 (socket says 172.21.15.27:42068) 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30b3e00700 1 -- 172.21.15.27:0/3570993529 learned_addr learned my addr 172.21.15.27:0/3570993529 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.739+0000 7f30b3e00700 1 --2- 172.21.15.27:0/3570993529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 0x55fb5dc6e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.740+0000 7f30b3400700 1 -- 172.21.15.27:0/3570993529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 msgr2=0x55fb5dc6e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.740+0000 7f30b3400700 1 --2- 172.21.15.27:0/3570993529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 0x55fb5dc6e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.740+0000 7f30b3400700 1 -- 172.21.15.27:0/3570993529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 msgr2=0x55fb5dc6de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.740+0000 7f30b3400700 1 --2- 172.21.15.27:0/3570993529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc6de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.740+0000 7f30b3400700 1 -- 172.21.15.27:0/3570993529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb5dc96cf0 con 0x55fb5dc79b70 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.740+0000 7f30b3400700 1 --2- 172.21.15.27:0/3570993529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc6e3d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f30a4006010 tx=0x7f30a400f8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff30e0f4c7423db8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.740+0000 7f309d600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.740+0000 7f309d600700 1 -- 172.21.15.27:0/3570993529 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a401abc0 con 0x55fb5dc79b70 2024-05-10T11:12:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.741+0000 7f30c501eb80 1 -- 172.21.15.27:0/3570993529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30a4005480 con 0x55fb5dc79b70 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.741+0000 7f309d600700 1 -- 172.21.15.27:0/3570993529 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f30a401ad60 con 0x55fb5dc79b70 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.741+0000 7f309d600700 1 -- 172.21.15.27:0/3570993529 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f30a4041050 con 0x55fb5dc79b70 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.741+0000 7f30c501eb80 10 client.25127 fetch_fsmap learned FSMap version 29 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.741+0000 7f30c501eb80 1 -- 172.21.15.27:0/3570993529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fb5dcf9640 con 0x55fb5dc79b70 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.741+0000 7f309d600700 1 -- 172.21.15.27:0/3570993529 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30a4005480 con 0x55fb5dc79b70 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.742+0000 7f309d600700 1 -- 172.21.15.27:0/3570993529 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f30a40039a0 con 0x55fb5dc79b70 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.742+0000 7f30c501eb80 10 client.25127 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.742+0000 7f30c501eb80 10 client.25127 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.742+0000 7f30c501eb80 20 client.25127 populate_metadata read hostname 'smithi027' 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.742+0000 7f30c501eb80 1 --2- 172.21.15.27:0/3570993529 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55fb5dcfde40 0x55fb5dcfcb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.742+0000 7f30c501eb80 4 client.25127 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.742+0000 7f30c501eb80 1 -- 172.21.15.27:0/3570993529 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "status"}) -- 0x55fb5dbd43b0 con 0x55fb5dcfde40 2024-05-10T11:12:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.742+0000 7f30b3e00700 1 --2- 172.21.15.27:0/3570993529 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55fb5dcfde40 0x55fb5dcfcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f30b3e00700 1 --2- 172.21.15.27:0/3570993529 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55fb5dcfde40 0x55fb5dcfcb50 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6e38c9596b74b9ad server_cookie=de316fa3866e229 in_seq=0 out_seq=0 2024-05-10T11:12:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f309d600700 10 client.25127 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:12:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f309d600700 1 -- 172.21.15.27:0/3570993529 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fb5dbd43b0 con 0x55fb5dcfde40 2024-05-10T11:12:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f309d600700 10 client.25127 handle_command_reply: tid=0 2024-05-10T11:12:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f30c501eb80 1 client.25127 shutdown 2024-05-10T11:12:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f30c501eb80 1 -- 172.21.15.27:0/3570993529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 msgr2=0x55fb5dc6e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f30c501eb80 1 --2- 172.21.15.27:0/3570993529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc6e3d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f30a4006010 tx=0x7f30a400f8a0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f30c501eb80 1 -- 172.21.15.27:0/3570993529 shutdown_connections 2024-05-10T11:12:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f30c501eb80 1 --2- 172.21.15.27:0/3570993529 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55fb5dcfde40 0x55fb5dcfcb50 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f30c501eb80 1 --2- 172.21.15.27:0/3570993529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb5dc7a490 0x55fb5dc6e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f30c501eb80 1 --2- 172.21.15.27:0/3570993529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb5dc748a0 0x55fb5dc6de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f30c501eb80 1 --2- 172.21.15.27:0/3570993529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb5dc79b70 0x55fb5dc6e3d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f309d600700 0 client.25127 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:12:45.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.743+0000 7f30c501eb80 1 -- 172.21.15.27:0/3570993529 >> 172.21.15.27:0/3570993529 conn(0x55fb5dbec680 msgr2=0x55fb5dc7a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:45.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.744+0000 7f30c501eb80 1 -- 172.21.15.27:0/3570993529 shutdown_connections 2024-05-10T11:12:45.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.744+0000 7f30c501eb80 1 -- 172.21.15.27:0/3570993529 wait complete. 2024-05-10T11:12:45.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.744+0000 7f30c501eb80 20 client.25127 trim_cache size 0 max 16384 2024-05-10T11:12:45.747 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:12:45.747 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T11:12:45.747 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 4, 2024-05-10T11:12:45.747 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24467, 2024-05-10T11:12:45.747 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-05-10T11:12:45.747 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-05-10T11:12:45.748 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-05-10T11:12:45.748 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 210.73578619400001, 2024-05-10T11:12:45.748 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 29, 2024-05-10T11:12:45.748 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 90, 2024-05-10T11:12:45.748 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 88, 2024-05-10T11:12:45.748 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 343.30549670099998 2024-05-10T11:12:45.748 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:12:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.745+0000 7f30ba600700 1 -- 172.21.15.27:0/1365917213 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30980b2880 msgr2=0x7f30980b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.745+0000 7f30ba600700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30980b2880 0x7f30980b4d40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f30a00dff00 tx=0x7f30a0010c80 comp rx=0 tx=0).stop 2024-05-10T11:12:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.745+0000 7f30ba600700 1 -- 172.21.15.27:0/1365917213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b4075ff0 msgr2=0x7f30b41a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.745+0000 7f30ba600700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b4075ff0 0x7f30b41a1f90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f30a000e9c0 tx=0x7f30a000ecd0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.745+0000 7f30ba600700 1 -- 172.21.15.27:0/1365917213 shutdown_connections 2024-05-10T11:12:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.745+0000 7f30ba600700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f30980b2880 0x7f30980b4d40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.745+0000 7f30ba600700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30b410ff00 0x7f30b419bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.745+0000 7f30ba600700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40756d0 0x7f30b41a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.745+0000 7f30ba600700 1 --2- 172.21.15.27:0/1365917213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b4075ff0 0x7f30b41a1f90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.745+0000 7f30ba600700 1 -- 172.21.15.27:0/1365917213 >> 172.21.15.27:0/1365917213 conn(0x7f30b407a600 msgr2=0x7f30b4101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:45.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.745+0000 7f30ba600700 1 -- 172.21.15.27:0/1365917213 shutdown_connections 2024-05-10T11:12:45.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:45.745+0000 7f30ba600700 1 -- 172.21.15.27:0/1365917213 wait complete. 2024-05-10T11:12:46.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:46 smithi027 ceph-mon[30054]: pgmap v482: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:46.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:46 smithi076 ceph-mon[34152]: pgmap v482: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:46.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:46 smithi189 ceph-mon[34842]: pgmap v482: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:46.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.c status 2024-05-10T11:12:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:48.302 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:48 smithi027 ceph-mon[30054]: pgmap v483: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:48.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:48 smithi076 ceph-mon[34152]: pgmap v483: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:48.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:48 smithi189 ceph-mon[34842]: pgmap v483: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:49.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.658+0000 7f917f200700 1 -- 172.21.15.27:0/2225964377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917810b6e0 msgr2=0x7f917810bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:49.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.658+0000 7f917f200700 1 --2- 172.21.15.27:0/2225964377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917810b6e0 0x7f917810bac0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9168009990 tx=0x7f9168009ca0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.658+0000 7f917f200700 1 -- 172.21.15.27:0/2225964377 shutdown_connections 2024-05-10T11:12:49.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.658+0000 7f917f200700 1 --2- 172.21.15.27:0/2225964377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91781060a0 0x7f91781125b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.658+0000 7f917f200700 1 --2- 172.21.15.27:0/2225964377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917810b6e0 0x7f917810bac0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.658+0000 7f917f200700 1 --2- 172.21.15.27:0/2225964377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91781056e0 0x7f9178105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.658+0000 7f917f200700 1 -- 172.21.15.27:0/2225964377 >> 172.21.15.27:0/2225964377 conn(0x7f91781011d0 msgr2=0x7f91781035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:49.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.658+0000 7f917f200700 1 -- 172.21.15.27:0/2225964377 shutdown_connections 2024-05-10T11:12:49.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.658+0000 7f917f200700 1 -- 172.21.15.27:0/2225964377 wait complete. 2024-05-10T11:12:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.660+0000 7f917f200700 1 Processor -- start 2024-05-10T11:12:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.660+0000 7f917f200700 1 -- start start 2024-05-10T11:12:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.660+0000 7f917f200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91781056e0 0x7f917819bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.660+0000 7f917f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91781060a0 0x7f917819c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.660+0000 7f917f200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917810b6e0 0x7f91781a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.660+0000 7f917f200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f91781158e0 con 0x7f917810b6e0 2024-05-10T11:12:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.660+0000 7f917f200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9178115760 con 0x7f91781056e0 2024-05-10T11:12:49.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.660+0000 7f917f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9178115a60 con 0x7f91781060a0 2024-05-10T11:12:49.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.661+0000 7f9177400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91781060a0 0x7f917819c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.661+0000 7f9177400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91781060a0 0x7f917819c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41266/0 (socket says 172.21.15.27:41266) 2024-05-10T11:12:49.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.661+0000 7f9177400700 1 -- 172.21.15.27:0/1501685712 learned_addr learned my addr 172.21.15.27:0/1501685712 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:49.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.661+0000 7f917d200700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917810b6e0 0x7f91781a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.661+0000 7f9177e00700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91781056e0 0x7f917819bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.661+0000 7f9177400700 1 -- 172.21.15.27:0/1501685712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91781056e0 msgr2=0x7f917819bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:49.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.661+0000 7f9177400700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91781056e0 0x7f917819bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.661+0000 7f9177400700 1 -- 172.21.15.27:0/1501685712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917810b6e0 msgr2=0x7f91781a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:49.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.661+0000 7f9177400700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917810b6e0 0x7f91781a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.661+0000 7f9177400700 1 -- 172.21.15.27:0/1501685712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91781a0d80 con 0x7f91781060a0 2024-05-10T11:12:49.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.662+0000 7f9177400700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91781060a0 0x7f917819c2b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f916400d950 tx=0x7f916400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:49.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.662+0000 7f9174c00700 1 -- 172.21.15.27:0/1501685712 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164016070 con 0x7f91781060a0 2024-05-10T11:12:49.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.662+0000 7f9174c00700 1 -- 172.21.15.27:0/1501685712 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f916400fd40 con 0x7f91781060a0 2024-05-10T11:12:49.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.662+0000 7f9174c00700 1 -- 172.21.15.27:0/1501685712 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916400e3b0 con 0x7f91781060a0 2024-05-10T11:12:49.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.662+0000 7f917f200700 1 -- 172.21.15.27:0/1501685712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91781a1070 con 0x7f91781060a0 2024-05-10T11:12:49.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.662+0000 7f917f200700 1 -- 172.21.15.27:0/1501685712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91781a88f0 con 0x7f91781060a0 2024-05-10T11:12:49.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.667+0000 7f9189b03b80 1 Processor -- start 2024-05-10T11:12:49.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.667+0000 7f9189b03b80 1 -- start start 2024-05-10T11:12:49.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.668+0000 7f9189b03b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 0x56158bbbc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.668+0000 7f9189b03b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc930 0x56158bbc0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.668+0000 7f9189b03b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbc0980 0x56158bbc0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.668+0000 7f9189b03b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56158bb0e9d0 con 0x56158bbc0980 2024-05-10T11:12:49.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.668+0000 7f9189b03b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56158baac710 con 0x56158bbbc010 2024-05-10T11:12:49.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.668+0000 7f9189b03b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56158bb0a990 con 0x56158bbbc930 2024-05-10T11:12:49.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.668+0000 7f917d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 0x56158bbbc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.668+0000 7f917d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 0x56158bbbc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42080/0 (socket says 172.21.15.27:42080) 2024-05-10T11:12:49.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.668+0000 7f917d200700 1 -- 172.21.15.27:0/2889597056 learned_addr learned my addr 172.21.15.27:0/2889597056 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:49.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.668+0000 7f9177e00700 1 --2- 172.21.15.27:0/2889597056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc930 0x56158bbc0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.669+0000 7f917d200700 1 -- 172.21.15.27:0/2889597056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc930 msgr2=0x56158bbc0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:49.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.669+0000 7f917d200700 1 --2- 172.21.15.27:0/2889597056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc930 0x56158bbc0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.669+0000 7f917d200700 1 -- 172.21.15.27:0/2889597056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbc0980 msgr2=0x56158bbc0de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:12:49.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.669+0000 7f917d200700 1 --2- 172.21.15.27:0/2889597056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbc0980 0x56158bbc0de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.669+0000 7f917d200700 1 -- 172.21.15.27:0/2889597056 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56158ba5b880 con 0x56158bbbc010 2024-05-10T11:12:49.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.669+0000 7f9174c00700 1 -- 172.21.15.27:0/1501685712 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f9164019070 con 0x7f91781060a0 2024-05-10T11:12:49.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.669+0000 7f917d200700 1 --2- 172.21.15.27:0/2889597056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 0x56158bbbc3f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9170011680 tx=0x7f9170009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62f3beca55183222 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:49.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.669+0000 7f916d600700 1 -- 172.21.15.27:0/2889597056 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170019030 con 0x56158bbbc010 2024-05-10T11:12:49.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.669+0000 7f916d600700 1 -- 172.21.15.27:0/2889597056 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91700092e0 con 0x56158bbbc010 2024-05-10T11:12:49.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.669+0000 7f916d600700 1 -- 172.21.15.27:0/2889597056 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170007500 con 0x56158bbbc010 2024-05-10T11:12:49.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.670+0000 7f9189b03b80 1 -- 172.21.15.27:0/2889597056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 msgr2=0x56158bbbc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:49.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.670+0000 7f9189b03b80 1 --2- 172.21.15.27:0/2889597056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 0x56158bbbc3f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9170011680 tx=0x7f9170009920 comp rx=0 tx=0).stop 2024-05-10T11:12:49.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.670+0000 7f9189b03b80 1 -- 172.21.15.27:0/2889597056 shutdown_connections 2024-05-10T11:12:49.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.670+0000 7f9189b03b80 1 --2- 172.21.15.27:0/2889597056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbc0980 0x56158bbc0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.670+0000 7f9189b03b80 1 --2- 172.21.15.27:0/2889597056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 0x56158bbbc3f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.670+0000 7f9189b03b80 1 --2- 172.21.15.27:0/2889597056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc930 0x56158bbc0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.670+0000 7f9189b03b80 1 -- 172.21.15.27:0/2889597056 >> 172.21.15.27:0/2889597056 conn(0x56158baeab10 msgr2=0x56158bbb6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:49.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.671+0000 7f9189b03b80 1 -- 172.21.15.27:0/2889597056 shutdown_connections 2024-05-10T11:12:49.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.671+0000 7f9189b03b80 1 -- 172.21.15.27:0/2889597056 wait complete. 2024-05-10T11:12:49.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.671+0000 7f9174c00700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f915c0b2810 0x7f915c0b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.671+0000 7f9189b03b80 1 -- start start 2024-05-10T11:12:49.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.672+0000 7f9174c00700 1 -- 172.21.15.27:0/1501685712 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f91640e20d0 con 0x7f91781060a0 2024-05-10T11:12:49.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.672+0000 7f9189b03b80 1 Processor -- start 2024-05-10T11:12:49.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.672+0000 7f917d200700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f915c0b2810 0x7f915c0b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.673+0000 7f917d200700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f915c0b2810 0x7f915c0b4cd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9170011300 tx=0x7f91700044b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:49.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.673+0000 7f9189b03b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 0x56158bbc7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.673+0000 7f9189b03b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.673+0000 7f9189b03b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbc0980 0x56158bbc56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.673+0000 7f9189b03b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56158bbc8b90 con 0x56158bbbc930 2024-05-10T11:12:49.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.673+0000 7f9189b03b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56158ba74ad0 con 0x56158bbc0980 2024-05-10T11:12:49.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.673+0000 7f9189b03b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56158bbc8d10 con 0x56158bbbc010 2024-05-10T11:12:49.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.673+0000 7f9189b03b80 10 client.0 _ll_register_callbacks cb 0x56158bab5c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:12:49.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.673+0000 7f9177e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 0x56158bbc7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.673+0000 7f9177e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 0x56158bbc7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41286/0 (socket says 172.21.15.27:41286) 2024-05-10T11:12:49.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.673+0000 7f9177e00700 1 -- 172.21.15.27:0/800786191 learned_addr learned my addr 172.21.15.27:0/800786191 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:49.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.674+0000 7f917d200700 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbc0980 0x56158bbc56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.674+0000 7f9177400700 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.674+0000 7f9177e00700 1 -- 172.21.15.27:0/800786191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbc0980 msgr2=0x56158bbc56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:49.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.674+0000 7f9177e00700 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbc0980 0x56158bbc56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.674+0000 7f9177e00700 1 -- 172.21.15.27:0/800786191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 msgr2=0x56158bbc51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:49.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.674+0000 7f9177e00700 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.674+0000 7f9177e00700 1 -- 172.21.15.27:0/800786191 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56158bad43d0 con 0x56158bbbc010 2024-05-10T11:12:49.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.674+0000 7f9177400700 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:12:49.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.674+0000 7f9177e00700 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 0x56158bbc7d80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9168000c00 tx=0x7f9168005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69d94bf8cab464b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:49.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.674+0000 7f9162000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:12:49.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.674+0000 7f9162000700 1 -- 172.21.15.27:0/800786191 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9168029060 con 0x56158bbbc010 2024-05-10T11:12:49.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.675+0000 7f9162000700 1 -- 172.21.15.27:0/800786191 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f9168003680 con 0x56158bbbc010 2024-05-10T11:12:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.675+0000 7f9189b03b80 1 -- 172.21.15.27:0/800786191 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91700092e0 con 0x56158bbbc010 2024-05-10T11:12:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.675+0000 7f9162000700 1 -- 172.21.15.27:0/800786191 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9168030050 con 0x56158bbbc010 2024-05-10T11:12:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.675+0000 7f9189b03b80 10 client.25123 fetch_fsmap learned FSMap version 29 2024-05-10T11:12:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.675+0000 7f9189b03b80 1 -- 172.21.15.27:0/800786191 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56158bbd33b0 con 0x56158bbbc010 2024-05-10T11:12:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.675+0000 7f9162000700 1 -- 172.21.15.27:0/800786191 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91700092e0 con 0x56158bbbc010 2024-05-10T11:12:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.675+0000 7f9162000700 1 -- 172.21.15.27:0/800786191 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f916800f7e0 con 0x56158bbbc010 2024-05-10T11:12:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.675+0000 7f9189b03b80 10 client.25123 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:12:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.676+0000 7f9189b03b80 10 client.25123 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-10T11:12:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.676+0000 7f9189b03b80 20 client.25123 populate_metadata read hostname 'smithi027' 2024-05-10T11:12:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.676+0000 7f9189b03b80 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56158bbd5990 0x56158bbd6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.676+0000 7f9189b03b80 4 client.25123 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:12:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.676+0000 7f9189b03b80 1 -- 172.21.15.27:0/800786191 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56158bb89050 con 0x56158bbd5990 2024-05-10T11:12:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.676+0000 7f9177400700 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56158bbd5990 0x56158bbd6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.677+0000 7f9177400700 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56158bbd5990 0x56158bbd6e00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6d5445a1b9ff7d92 server_cookie=6aa4b2329acd0fbb in_seq=0 out_seq=0 2024-05-10T11:12:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.677+0000 7f9162000700 10 client.25123 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:12:49.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.677+0000 7f9162000700 1 -- 172.21.15.27:0/800786191 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56158bb89050 con 0x56158bbd5990 2024-05-10T11:12:49.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.677+0000 7f9162000700 10 client.25123 handle_command_reply: tid=0 2024-05-10T11:12:49.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9189b03b80 1 client.25123 shutdown 2024-05-10T11:12:49.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9189b03b80 1 -- 172.21.15.27:0/800786191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 msgr2=0x56158bbc7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:49.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9189b03b80 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 0x56158bbc7d80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9168000c00 tx=0x7f9168005c00 comp rx=0 tx=0).stop 2024-05-10T11:12:49.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9189b03b80 1 -- 172.21.15.27:0/800786191 shutdown_connections 2024-05-10T11:12:49.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9189b03b80 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56158bbd5990 0x56158bbd6e00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9189b03b80 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 0x56158bbc7d80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9189b03b80 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbc0980 0x56158bbc56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9162000700 0 client.25123 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:12:49.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9189b03b80 1 --2- 172.21.15.27:0/800786191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9189b03b80 1 -- 172.21.15.27:0/800786191 >> 172.21.15.27:0/800786191 conn(0x56158baeab10 msgr2=0x56158bbb1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:49.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9189b03b80 1 -- 172.21.15.27:0/800786191 shutdown_connections 2024-05-10T11:12:49.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9189b03b80 1 -- 172.21.15.27:0/800786191 wait complete. 2024-05-10T11:12:49.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.678+0000 7f9189b03b80 20 client.25123 trim_cache size 0 max 16384 2024-05-10T11:12:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.702+0000 7f9189b03b80 1 Processor -- start 2024-05-10T11:12:49.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.702+0000 7f9189b03b80 1 -- start start 2024-05-10T11:12:49.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.702+0000 7f9189b03b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 0x56158bbc4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.702+0000 7f9189b03b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.702+0000 7f9189b03b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbc0980 0x56158bbc1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.702+0000 7f9189b03b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56158bbd5140 con 0x56158bbbc930 2024-05-10T11:12:49.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.702+0000 7f9189b03b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56158bbd5810 con 0x56158bbc0980 2024-05-10T11:12:49.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.702+0000 7f9189b03b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56158bbd5ee0 con 0x56158bbbc010 2024-05-10T11:12:49.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.703+0000 7f9177400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.703+0000 7f9177e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 0x56158bbc4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.703+0000 7f917d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbc0980 0x56158bbc1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.703+0000 7f9177e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 0x56158bbc4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41292/0 (socket says 172.21.15.27:41292) 2024-05-10T11:12:49.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.703+0000 7f9177e00700 1 -- 172.21.15.27:0/3642439083 learned_addr learned my addr 172.21.15.27:0/3642439083 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:49.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.703+0000 7f9177e00700 1 -- 172.21.15.27:0/3642439083 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbc0980 msgr2=0x56158bbc1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:49.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.703+0000 7f9177e00700 1 --2- 172.21.15.27:0/3642439083 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbc0980 0x56158bbc1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.703+0000 7f9177e00700 1 -- 172.21.15.27:0/3642439083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 msgr2=0x56158bbc2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:49.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.703+0000 7f9177e00700 1 --2- 172.21.15.27:0/3642439083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.703+0000 7f9177e00700 1 -- 172.21.15.27:0/3642439083 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56158bbb33a0 con 0x56158bbbc010 2024-05-10T11:12:49.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.703+0000 7f917d200700 1 --2- 172.21.15.27:0/3642439083 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbc0980 0x56158bbc1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:49.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.704+0000 7f9177e00700 1 --2- 172.21.15.27:0/3642439083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 0x56158bbc4100 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9168009ce0 tx=0x7f916800ba80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aed20cdc054f0c1b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:49.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.704+0000 7f916cc00700 1 -- 172.21.15.27:0/3642439083 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9168029070 con 0x56158bbbc010 2024-05-10T11:12:49.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.704+0000 7f916cc00700 1 -- 172.21.15.27:0/3642439083 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f916800f910 con 0x56158bbbc010 2024-05-10T11:12:49.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.704+0000 7f916cc00700 1 -- 172.21.15.27:0/3642439083 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9168003e20 con 0x56158bbbc010 2024-05-10T11:12:49.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.705+0000 7f9189b03b80 1 -- 172.21.15.27:0/3642439083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 msgr2=0x56158bbc4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:49.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.705+0000 7f9189b03b80 1 --2- 172.21.15.27:0/3642439083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 0x56158bbc4100 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9168009ce0 tx=0x7f916800ba80 comp rx=0 tx=0).stop 2024-05-10T11:12:49.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.705+0000 7f9189b03b80 1 -- 172.21.15.27:0/3642439083 shutdown_connections 2024-05-10T11:12:49.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.705+0000 7f9189b03b80 1 --2- 172.21.15.27:0/3642439083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbbc010 0x56158bbc4100 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.705+0000 7f9189b03b80 1 --2- 172.21.15.27:0/3642439083 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbc0980 0x56158bbc1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.705+0000 7f9189b03b80 1 --2- 172.21.15.27:0/3642439083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.705+0000 7f9189b03b80 1 -- 172.21.15.27:0/3642439083 >> 172.21.15.27:0/3642439083 conn(0x56158baeab10 msgr2=0x56158bbb09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:49.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.705+0000 7f9189b03b80 1 -- 172.21.15.27:0/3642439083 shutdown_connections 2024-05-10T11:12:49.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.705+0000 7f9189b03b80 1 -- 172.21.15.27:0/3642439083 wait complete. 2024-05-10T11:12:49.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.706+0000 7f9189b03b80 1 -- start start 2024-05-10T11:12:49.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.706+0000 7f9189b03b80 1 Processor -- start 2024-05-10T11:12:49.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.706+0000 7f9189b03b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 0x56158bbc6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.706+0000 7f9189b03b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.706+0000 7f9189b03b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbc0980 0x56158bbc70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.706+0000 7f9189b03b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56158bbd6c80 con 0x56158bbbc930 2024-05-10T11:12:49.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.706+0000 7f9189b03b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56158bbd65b0 con 0x56158bbbc010 2024-05-10T11:12:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.706+0000 7f9189b03b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56158bbd7350 con 0x56158bbc0980 2024-05-10T11:12:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.707+0000 7f9189b03b80 10 client.0 _ll_register_callbacks cb 0x56158bab5c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:12:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.707+0000 7f9177400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.707+0000 7f9177400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58468/0 (socket says 172.21.15.27:58468) 2024-05-10T11:12:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.707+0000 7f9177400700 1 -- 172.21.15.27:0/2904180538 learned_addr learned my addr 172.21.15.27:0/2904180538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.707+0000 7f9177e00700 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 0x56158bbc6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.707+0000 7f917d200700 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbc0980 0x56158bbc70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.707+0000 7f9177400700 1 -- 172.21.15.27:0/2904180538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbc0980 msgr2=0x56158bbc70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.707+0000 7f9177400700 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbc0980 0x56158bbc70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.707+0000 7f9177400700 1 -- 172.21.15.27:0/2904180538 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 msgr2=0x56158bbc6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.707+0000 7f9177400700 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 0x56158bbc6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.708+0000 7f9177400700 1 -- 172.21.15.27:0/2904180538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56158bbb75c0 con 0x56158bbbc930 2024-05-10T11:12:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.708+0000 7f9177e00700 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 0x56158bbc6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.708+0000 7f9177400700 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc6b70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f91640a9230 tx=0x7f9164000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c7ab0c9a413937 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.708+0000 7f9162000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:12:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.708+0000 7f9162000700 1 -- 172.21.15.27:0/2904180538 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164014070 con 0x56158bbbc930 2024-05-10T11:12:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.708+0000 7f9162000700 1 -- 172.21.15.27:0/2904180538 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f91640a9500 con 0x56158bbbc930 2024-05-10T11:12:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.708+0000 7f9189b03b80 1 -- 172.21.15.27:0/2904180538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f916800f910 con 0x56158bbbc930 2024-05-10T11:12:49.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.708+0000 7f9162000700 1 -- 172.21.15.27:0/2904180538 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9164020040 con 0x56158bbbc930 2024-05-10T11:12:49.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.709+0000 7f9189b03b80 10 client.15015 fetch_fsmap learned FSMap version 29 2024-05-10T11:12:49.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.709+0000 7f9189b03b80 1 -- 172.21.15.27:0/2904180538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56158bc40e60 con 0x56158bbbc930 2024-05-10T11:12:49.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.709+0000 7f9162000700 1 -- 172.21.15.27:0/2904180538 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f916800f910 con 0x56158bbbc930 2024-05-10T11:12:49.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.709+0000 7f9162000700 1 -- 172.21.15.27:0/2904180538 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f91640b1c70 con 0x56158bbbc930 2024-05-10T11:12:49.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.709+0000 7f9189b03b80 10 client.15015 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:12:49.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.709+0000 7f9189b03b80 10 client.15015 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-10T11:12:49.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.709+0000 7f9189b03b80 20 client.15015 populate_metadata read hostname 'smithi027' 2024-05-10T11:12:49.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.709+0000 7f9189b03b80 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56158bc459d0 0x56158bc446e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:49.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.709+0000 7f9189b03b80 4 client.15015 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-10T11:12:49.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.709+0000 7f9189b03b80 1 -- 172.21.15.27:0/2904180538 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "status"}) -- 0x56158bb1c3b0 con 0x56158bc459d0 2024-05-10T11:12:49.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.710+0000 7f9177e00700 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56158bc459d0 0x56158bc446e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:49.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.710+0000 7f9177e00700 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56158bc459d0 0x56158bc446e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=26981e777f909fee server_cookie=21ce380d4c4b80bb in_seq=0 out_seq=0 2024-05-10T11:12:49.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.710+0000 7f9162000700 10 client.15015 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:12:49.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9162000700 1 -- 172.21.15.27:0/2904180538 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56158bb1c3b0 con 0x56158bc459d0 2024-05-10T11:12:49.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9162000700 10 client.15015 handle_command_reply: tid=0 2024-05-10T11:12:49.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9189b03b80 1 client.15015 shutdown 2024-05-10T11:12:49.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9189b03b80 1 -- 172.21.15.27:0/2904180538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 msgr2=0x56158bbc6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:49.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9189b03b80 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc6b70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f91640a9230 tx=0x7f9164000bc0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9189b03b80 1 -- 172.21.15.27:0/2904180538 shutdown_connections 2024-05-10T11:12:49.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9189b03b80 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56158bc459d0 0x56158bc446e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9189b03b80 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56158bbc0980 0x56158bbc70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9189b03b80 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56158bbbc010 0x56158bbc6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9189b03b80 1 --2- 172.21.15.27:0/2904180538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56158bbbc930 0x56158bbc6b70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9189b03b80 1 -- 172.21.15.27:0/2904180538 >> 172.21.15.27:0/2904180538 conn(0x56158baeab10 msgr2=0x56158bbb1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:49.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9162000700 0 client.15015 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:12:49.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9189b03b80 1 -- 172.21.15.27:0/2904180538 shutdown_connections 2024-05-10T11:12:49.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9189b03b80 1 -- 172.21.15.27:0/2904180538 wait complete. 2024-05-10T11:12:49.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.711+0000 7f9189b03b80 20 client.15015 trim_cache size 0 max 16384 2024-05-10T11:12:49.720 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:12:49.720 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T11:12:49.721 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 3, 2024-05-10T11:12:49.721 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24472, 2024-05-10T11:12:49.721 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-05-10T11:12:49.721 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-05-10T11:12:49.721 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-05-10T11:12:49.721 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 215.71489593000001, 2024-05-10T11:12:49.721 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 29, 2024-05-10T11:12:49.721 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 90, 2024-05-10T11:12:49.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 88, 2024-05-10T11:12:49.722 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 341.33925868400001 2024-05-10T11:12:49.722 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:12:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.712+0000 7f917f200700 1 -- 172.21.15.27:0/1501685712 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f915c0b2810 msgr2=0x7f915c0b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.712+0000 7f917f200700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f915c0b2810 0x7f915c0b4cd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9170011300 tx=0x7f91700044b0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.712+0000 7f917f200700 1 -- 172.21.15.27:0/1501685712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91781060a0 msgr2=0x7f917819c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.712+0000 7f917f200700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91781060a0 0x7f917819c2b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f916400d950 tx=0x7f916400dc60 comp rx=0 tx=0).stop 2024-05-10T11:12:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.712+0000 7f917f200700 1 -- 172.21.15.27:0/1501685712 shutdown_connections 2024-05-10T11:12:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.712+0000 7f917f200700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f915c0b2810 0x7f915c0b4cd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.712+0000 7f917f200700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917810b6e0 0x7f91781a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.712+0000 7f917f200700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f91781056e0 0x7f917819bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.712+0000 7f917f200700 1 --2- 172.21.15.27:0/1501685712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91781060a0 0x7f917819c2b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.712+0000 7f917f200700 1 -- 172.21.15.27:0/1501685712 >> 172.21.15.27:0/1501685712 conn(0x7f91781011d0 msgr2=0x7f9178101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.713+0000 7f917f200700 1 -- 172.21.15.27:0/1501685712 shutdown_connections 2024-05-10T11:12:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:49.713+0000 7f917f200700 1 -- 172.21.15.27:0/1501685712 wait complete. 2024-05-10T11:12:50.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:50 smithi027 ceph-mon[30054]: pgmap v484: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:50.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:50 smithi076 ceph-mon[34152]: pgmap v484: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:50.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:50 smithi189 ceph-mon[34842]: pgmap v484: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:50.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.i status 2024-05-10T11:12:51.122 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:52.308 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:52 smithi027 ceph-mon[30054]: pgmap v485: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:52.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:52 smithi076 ceph-mon[34152]: pgmap v485: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:52.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:52 smithi189 ceph-mon[34842]: pgmap v485: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:12:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.599+0000 7fa60ea00700 1 -- 172.21.15.27:0/1337884818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa608108460 msgr2=0x7fa608108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.599+0000 7fa60ea00700 1 --2- 172.21.15.27:0/1337884818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa608108460 0x7fa608108840 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009990 tx=0x7fa5f8009ca0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.599+0000 7fa60ea00700 1 -- 172.21.15.27:0/1337884818 shutdown_connections 2024-05-10T11:12:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.599+0000 7fa60ea00700 1 --2- 172.21.15.27:0/1337884818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa608108460 0x7fa608108840 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.599+0000 7fa60ea00700 1 --2- 172.21.15.27:0/1337884818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa608103ef0 0x7fa6081103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.599+0000 7fa60ea00700 1 --2- 172.21.15.27:0/1337884818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa608103550 0x7fa6081039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.599+0000 7fa60ea00700 1 -- 172.21.15.27:0/1337884818 >> 172.21.15.27:0/1337884818 conn(0x7fa60807a600 msgr2=0x7fa60807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.599+0000 7fa60ea00700 1 -- 172.21.15.27:0/1337884818 shutdown_connections 2024-05-10T11:12:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.599+0000 7fa60ea00700 1 -- 172.21.15.27:0/1337884818 wait complete. 2024-05-10T11:12:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.600+0000 7fa60ea00700 1 Processor -- start 2024-05-10T11:12:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.600+0000 7fa60ea00700 1 -- start start 2024-05-10T11:12:53.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.601+0000 7fa60ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa608103550 0x7fa6080757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.601+0000 7fa60ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa608103ef0 0x7fa608075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.601+0000 7fa60ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa608108460 0x7fa608076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.601+0000 7fa60ea00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa6081136f0 con 0x7fa608108460 2024-05-10T11:12:53.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.601+0000 7fa60ea00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa608113570 con 0x7fa608103ef0 2024-05-10T11:12:53.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.601+0000 7fa60ea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa608113870 con 0x7fa608103550 2024-05-10T11:12:53.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.601+0000 7fa5ff400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa608103ef0 0x7fa608075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.601+0000 7fa607e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa608103550 0x7fa6080757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.601+0000 7fa60ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa608108460 0x7fa608076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.601+0000 7fa5ff400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa608103ef0 0x7fa608075ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42128/0 (socket says 172.21.15.27:42128) 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.601+0000 7fa607e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa608103550 0x7fa6080757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41318/0 (socket says 172.21.15.27:41318) 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.601+0000 7fa5ff400700 1 -- 172.21.15.27:0/3121416332 learned_addr learned my addr 172.21.15.27:0/3121416332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.602+0000 7fa5ff400700 1 -- 172.21.15.27:0/3121416332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa608103550 msgr2=0x7fa6080757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.602+0000 7fa5ff400700 1 --2- 172.21.15.27:0/3121416332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa608103550 0x7fa6080757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.602+0000 7fa5ff400700 1 -- 172.21.15.27:0/3121416332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa608108460 msgr2=0x7fa608076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.602+0000 7fa5ff400700 1 --2- 172.21.15.27:0/3121416332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa608108460 0x7fa608076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.602+0000 7fa5ff400700 1 -- 172.21.15.27:0/3121416332 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6081a8130 con 0x7fa608103ef0 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.602+0000 7fa5ff400700 1 --2- 172.21.15.27:0/3121416332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa608103ef0 0x7fa608075ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000e9c0 tx=0x7fa5f000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.603+0000 7fa605600700 1 -- 172.21.15.27:0/3121416332 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f000cbd0 con 0x7fa608103ef0 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.603+0000 7fa60ea00700 1 -- 172.21.15.27:0/3121416332 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6081a8420 con 0x7fa608103ef0 2024-05-10T11:12:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.603+0000 7fa60ea00700 1 -- 172.21.15.27:0/3121416332 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6081a8960 con 0x7fa608103ef0 2024-05-10T11:12:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.603+0000 7fa605600700 1 -- 172.21.15.27:0/3121416332 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5f000ee60 con 0x7fa608103ef0 2024-05-10T11:12:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.603+0000 7fa605600700 1 -- 172.21.15.27:0/3121416332 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0005550 con 0x7fa608103ef0 2024-05-10T11:12:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.606+0000 7fa605600700 1 -- 172.21.15.27:0/3121416332 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fa5f0004750 con 0x7fa608103ef0 2024-05-10T11:12:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.608+0000 7fa619319b80 1 Processor -- start 2024-05-10T11:12:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.608+0000 7fa619319b80 1 -- start start 2024-05-10T11:12:53.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.609+0000 7fa619319b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784010 0x55ccd17843f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.609+0000 7fa619319b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 0x55ccd1788440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.609+0000 7fa619319b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1788980 0x55ccd1788de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.609+0000 7fa619319b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ccd16d69d0 con 0x55ccd1784930 2024-05-10T11:12:53.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.609+0000 7fa619319b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ccd1674710 con 0x55ccd1788980 2024-05-10T11:12:53.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.609+0000 7fa619319b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ccd16d2990 con 0x55ccd1784010 2024-05-10T11:12:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.609+0000 7fa607e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 0x55ccd1788440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.609+0000 7fa60ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784010 0x55ccd17843f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.609+0000 7fa607e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 0x55ccd1788440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58492/0 (socket says 172.21.15.27:58492) 2024-05-10T11:12:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.609+0000 7fa607e00700 1 -- 172.21.15.27:0/3039336790 learned_addr learned my addr 172.21.15.27:0/3039336790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.609+0000 7fa605600700 1 --2- 172.21.15.27:0/3121416332 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa5ec0b2880 0x7fa5ec0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.609+0000 7fa5ff400700 1 --2- 172.21.15.27:0/3039336790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1788980 0x55ccd1788de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.610+0000 7fa605600700 1 -- 172.21.15.27:0/3121416332 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fa5f00e2900 con 0x7fa608103ef0 2024-05-10T11:12:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.610+0000 7fa60ca00700 1 --2- 172.21.15.27:0/3121416332 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa5ec0b2880 0x7fa5ec0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.610+0000 7fa607e00700 1 -- 172.21.15.27:0/3039336790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784010 msgr2=0x55ccd17843f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.610+0000 7fa607e00700 1 --2- 172.21.15.27:0/3039336790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784010 0x55ccd17843f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.610+0000 7fa607e00700 1 -- 172.21.15.27:0/3039336790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1788980 msgr2=0x55ccd1788de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.610+0000 7fa607e00700 1 --2- 172.21.15.27:0/3039336790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1788980 0x55ccd1788de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.610+0000 7fa607e00700 1 -- 172.21.15.27:0/3039336790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccd1623880 con 0x55ccd1784930 2024-05-10T11:12:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.610+0000 7fa607e00700 1 --2- 172.21.15.27:0/3039336790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 0x55ccd1788440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8015000 tx=0x7fa5f80096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da1c80bda225784 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:53.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.611+0000 7fa60ca00700 1 --2- 172.21.15.27:0/3121416332 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa5ec0b2880 0x7fa5ec0b4d40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa600009fd0 tx=0x7fa600004b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:53.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.611+0000 7fa5fd600700 1 -- 172.21.15.27:0/3039336790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8027070 con 0x55ccd1784930 2024-05-10T11:12:53.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.611+0000 7fa5fd600700 1 -- 172.21.15.27:0/3039336790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5f80056d0 con 0x55ccd1784930 2024-05-10T11:12:53.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.611+0000 7fa5fd600700 1 -- 172.21.15.27:0/3039336790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8003a80 con 0x55ccd1784930 2024-05-10T11:12:53.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.611+0000 7fa619319b80 1 -- 172.21.15.27:0/3039336790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 msgr2=0x55ccd1788440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.611+0000 7fa619319b80 1 --2- 172.21.15.27:0/3039336790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 0x55ccd1788440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8015000 tx=0x7fa5f80096a0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.612+0000 7fa619319b80 1 -- 172.21.15.27:0/3039336790 shutdown_connections 2024-05-10T11:12:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.612+0000 7fa619319b80 1 --2- 172.21.15.27:0/3039336790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784010 0x55ccd17843f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.612+0000 7fa619319b80 1 --2- 172.21.15.27:0/3039336790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1788980 0x55ccd1788de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.612+0000 7fa619319b80 1 --2- 172.21.15.27:0/3039336790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 0x55ccd1788440 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.612+0000 7fa619319b80 1 -- 172.21.15.27:0/3039336790 >> 172.21.15.27:0/3039336790 conn(0x55ccd16b2b10 msgr2=0x55ccd177e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.612+0000 7fa619319b80 1 -- 172.21.15.27:0/3039336790 shutdown_connections 2024-05-10T11:12:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.612+0000 7fa619319b80 1 -- 172.21.15.27:0/3039336790 wait complete. 2024-05-10T11:12:53.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.612+0000 7fa619319b80 1 -- start start 2024-05-10T11:12:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.613+0000 7fa619319b80 1 Processor -- start 2024-05-10T11:12:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.613+0000 7fa619319b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784010 0x55ccd178fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.613+0000 7fa619319b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784930 0x55ccd178d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.613+0000 7fa619319b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1788980 0x55ccd178d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.613+0000 7fa619319b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ccd1790b30 con 0x55ccd1784010 2024-05-10T11:12:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.613+0000 7fa619319b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ccd163cad0 con 0x55ccd1788980 2024-05-10T11:12:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.613+0000 7fa619319b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ccd1790cb0 con 0x55ccd1784930 2024-05-10T11:12:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.613+0000 7fa619319b80 10 client.0 _ll_register_callbacks cb 0x55ccd167dc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:12:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.614+0000 7fa607e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784010 0x55ccd178fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.614+0000 7fa5ff400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784930 0x55ccd178d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.614+0000 7fa607e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784010 0x55ccd178fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58506/0 (socket says 172.21.15.27:58506) 2024-05-10T11:12:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.614+0000 7fa607e00700 1 -- 172.21.15.27:0/2004753531 learned_addr learned my addr 172.21.15.27:0/2004753531 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.614+0000 7fa60ca00700 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1788980 0x55ccd178d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.614+0000 7fa607e00700 1 -- 172.21.15.27:0/2004753531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784930 msgr2=0x55ccd178d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.614+0000 7fa607e00700 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784930 0x55ccd178d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.614+0000 7fa607e00700 1 -- 172.21.15.27:0/2004753531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1788980 msgr2=0x55ccd178d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.614+0000 7fa607e00700 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1788980 0x55ccd178d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.614+0000 7fa607e00700 1 -- 172.21.15.27:0/2004753531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccd169c3d0 con 0x55ccd1784010 2024-05-10T11:12:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.614+0000 7fa5ff400700 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784930 0x55ccd178d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.615+0000 7fa607e00700 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784010 0x55ccd178fdf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009750 tx=0x7fa5f800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf86e8bc661a8da3 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.615+0000 7fa5f6000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.615+0000 7fa5f6000700 1 -- 172.21.15.27:0/2004753531 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8027070 con 0x55ccd1784010 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.615+0000 7fa5f6000700 1 -- 172.21.15.27:0/2004753531 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fa5f8003930 con 0x55ccd1784010 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.615+0000 7fa619319b80 1 -- 172.21.15.27:0/2004753531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5f80056d0 con 0x55ccd1784010 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.615+0000 7fa5f6000700 1 -- 172.21.15.27:0/2004753531 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa5f8003b30 con 0x55ccd1784010 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.615+0000 7fa619319b80 10 client.15027 fetch_fsmap learned FSMap version 29 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.615+0000 7fa619319b80 1 -- 172.21.15.27:0/2004753531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ccd179b3c0 con 0x55ccd1784010 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.616+0000 7fa5f6000700 1 -- 172.21.15.27:0/2004753531 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5f80056d0 con 0x55ccd1784010 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.616+0000 7fa5f6000700 1 -- 172.21.15.27:0/2004753531 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa5f8021070 con 0x55ccd1784010 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.616+0000 7fa619319b80 10 client.15027 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.616+0000 7fa619319b80 10 client.15027 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.616+0000 7fa619319b80 20 client.15027 populate_metadata read hostname 'smithi027' 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.616+0000 7fa619319b80 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55ccd179c710 0x55ccd179e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.616+0000 7fa619319b80 4 client.15027 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:12:53.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.616+0000 7fa619319b80 1 -- 172.21.15.27:0/2004753531 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccd1751050 con 0x55ccd179c710 2024-05-10T11:12:53.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.617+0000 7fa5ff400700 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55ccd179c710 0x55ccd179e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.617+0000 7fa5ff400700 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55ccd179c710 0x55ccd179e8b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e10c75b66dd9e196 server_cookie=73e76200680a54c7 in_seq=0 out_seq=0 2024-05-10T11:12:53.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.617+0000 7fa5f6000700 10 client.15027 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.618+0000 7fa5f6000700 1 -- 172.21.15.27:0/2004753531 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ccd1751050 con 0x55ccd179c710 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.618+0000 7fa5f6000700 10 client.15027 handle_command_reply: tid=0 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.618+0000 7fa619319b80 1 client.15027 shutdown 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.618+0000 7fa619319b80 1 -- 172.21.15.27:0/2004753531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784010 msgr2=0x55ccd178fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.618+0000 7fa619319b80 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784010 0x55ccd178fdf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009750 tx=0x7fa5f800be20 comp rx=0 tx=0).stop 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.619+0000 7fa619319b80 1 -- 172.21.15.27:0/2004753531 shutdown_connections 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.619+0000 7fa619319b80 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55ccd179c710 0x55ccd179e8b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.619+0000 7fa619319b80 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1788980 0x55ccd178d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.619+0000 7fa619319b80 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784010 0x55ccd178fdf0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.619+0000 7fa619319b80 1 --2- 172.21.15.27:0/2004753531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784930 0x55ccd178d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.619+0000 7fa5f6000700 0 client.15027 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.619+0000 7fa619319b80 1 -- 172.21.15.27:0/2004753531 >> 172.21.15.27:0/2004753531 conn(0x55ccd16b2b10 msgr2=0x55ccd1779260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.619+0000 7fa619319b80 1 -- 172.21.15.27:0/2004753531 shutdown_connections 2024-05-10T11:12:53.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.619+0000 7fa619319b80 1 -- 172.21.15.27:0/2004753531 wait complete. 2024-05-10T11:12:53.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.619+0000 7fa619319b80 20 client.15027 trim_cache size 0 max 16384 2024-05-10T11:12:53.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.642+0000 7fa619319b80 1 Processor -- start 2024-05-10T11:12:53.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.643+0000 7fa619319b80 1 -- start start 2024-05-10T11:12:53.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.643+0000 7fa619319b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 0x55ccd17789d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.643+0000 7fa619319b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784930 0x55ccd1778f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.643+0000 7fa619319b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1788980 0x55ccd1779450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.643+0000 7fa619319b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ccd179cbf0 con 0x55ccd1788980 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.643+0000 7fa619319b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ccd179d2c0 con 0x55ccd1784010 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.643+0000 7fa619319b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ccd179d990 con 0x55ccd1784930 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.643+0000 7fa607e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 0x55ccd17789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.643+0000 7fa607e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 0x55ccd17789d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42160/0 (socket says 172.21.15.27:42160) 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.643+0000 7fa607e00700 1 -- 172.21.15.27:0/2872532275 learned_addr learned my addr 172.21.15.27:0/2872532275 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.643+0000 7fa60ca00700 1 --2- 172.21.15.27:0/2872532275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1788980 0x55ccd1779450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.643+0000 7fa5ff400700 1 --2- 172.21.15.27:0/2872532275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784930 0x55ccd1778f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.644+0000 7fa607e00700 1 -- 172.21.15.27:0/2872532275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784930 msgr2=0x55ccd1778f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.644+0000 7fa607e00700 1 --2- 172.21.15.27:0/2872532275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784930 0x55ccd1778f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.644+0000 7fa607e00700 1 -- 172.21.15.27:0/2872532275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1788980 msgr2=0x55ccd1779450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.644+0000 7fa607e00700 1 --2- 172.21.15.27:0/2872532275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1788980 0x55ccd1779450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.644+0000 7fa607e00700 1 -- 172.21.15.27:0/2872532275 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccd177f970 con 0x55ccd1784010 2024-05-10T11:12:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.644+0000 7fa607e00700 1 --2- 172.21.15.27:0/2872532275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 0x55ccd17789d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009720 tx=0x7fa5f8009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=300c87fccd367ab8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.644+0000 7fa5fcc00700 1 -- 172.21.15.27:0/2872532275 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8027070 con 0x55ccd1784010 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.644+0000 7fa5fcc00700 1 -- 172.21.15.27:0/2872532275 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5f8031070 con 0x55ccd1784010 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.644+0000 7fa5fcc00700 1 -- 172.21.15.27:0/2872532275 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f800f8c0 con 0x55ccd1784010 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.645+0000 7fa619319b80 1 -- 172.21.15.27:0/2872532275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 msgr2=0x55ccd17789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.645+0000 7fa619319b80 1 --2- 172.21.15.27:0/2872532275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 0x55ccd17789d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009720 tx=0x7fa5f8009290 comp rx=0 tx=0).stop 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.645+0000 7fa619319b80 1 -- 172.21.15.27:0/2872532275 shutdown_connections 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.645+0000 7fa619319b80 1 --2- 172.21.15.27:0/2872532275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1788980 0x55ccd1779450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.645+0000 7fa619319b80 1 --2- 172.21.15.27:0/2872532275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 0x55ccd17789d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.645+0000 7fa619319b80 1 --2- 172.21.15.27:0/2872532275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1784930 0x55ccd1778f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.645+0000 7fa619319b80 1 -- 172.21.15.27:0/2872532275 >> 172.21.15.27:0/2872532275 conn(0x55ccd16b2b10 msgr2=0x55ccd178ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.645+0000 7fa619319b80 1 -- 172.21.15.27:0/2872532275 shutdown_connections 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.646+0000 7fa619319b80 1 -- 172.21.15.27:0/2872532275 wait complete. 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.646+0000 7fa619319b80 1 -- start start 2024-05-10T11:12:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.646+0000 7fa619319b80 1 Processor -- start 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa619319b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 0x55ccd17a7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa619319b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 0x55ccd17a8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa619319b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1788980 0x55ccd17a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa619319b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ccd179e730 con 0x55ccd1784930 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa619319b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ccd179e060 con 0x55ccd1784010 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa619319b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ccd179ee00 con 0x55ccd1788980 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa619319b80 10 client.0 _ll_register_callbacks cb 0x55ccd167dc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa60ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1788980 0x55ccd17a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa607e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 0x55ccd17a7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa60ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1788980 0x55ccd17a2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41342/0 (socket says 172.21.15.27:41342) 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa607e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 0x55ccd17a7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42168/0 (socket says 172.21.15.27:42168) 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa60ca00700 1 -- 172.21.15.27:0/3984330025 learned_addr learned my addr 172.21.15.27:0/3984330025 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:53.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.647+0000 7fa5ff400700 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 0x55ccd17a8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.648+0000 7fa607e00700 1 -- 172.21.15.27:0/3984330025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1788980 msgr2=0x55ccd17a2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:53.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.648+0000 7fa607e00700 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1788980 0x55ccd17a2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.648+0000 7fa607e00700 1 -- 172.21.15.27:0/3984330025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 msgr2=0x55ccd17a8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:53.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.648+0000 7fa607e00700 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 0x55ccd17a8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.648+0000 7fa607e00700 1 -- 172.21.15.27:0/3984330025 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccd17a3320 con 0x55ccd1784010 2024-05-10T11:12:53.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.648+0000 7fa60ca00700 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1788980 0x55ccd17a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:53.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.648+0000 7fa5ff400700 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 0x55ccd17a8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:53.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.648+0000 7fa607e00700 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 0x55ccd17a7d30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009750 tx=0x7fa5f80098c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b80b4188f5ccd693 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:53.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.648+0000 7fa5f6000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:12:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.648+0000 7fa5f6000700 1 -- 172.21.15.27:0/3984330025 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8027070 con 0x55ccd1784010 2024-05-10T11:12:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.648+0000 7fa619319b80 1 -- 172.21.15.27:0/3984330025 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5f8031070 con 0x55ccd1784010 2024-05-10T11:12:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.648+0000 7fa5f6000700 1 -- 172.21.15.27:0/3984330025 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7fa5f800f660 con 0x55ccd1784010 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.649+0000 7fa5f6000700 1 -- 172.21.15.27:0/3984330025 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa5f800f860 con 0x55ccd1784010 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.649+0000 7fa619319b80 10 client.25163 fetch_fsmap learned FSMap version 29 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.649+0000 7fa619319b80 1 -- 172.21.15.27:0/3984330025 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ccd18091f0 con 0x55ccd1784010 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.649+0000 7fa5f6000700 1 -- 172.21.15.27:0/3984330025 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5f8031070 con 0x55ccd1784010 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.649+0000 7fa5f6000700 1 -- 172.21.15.27:0/3984330025 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa5f8021040 con 0x55ccd1784010 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.649+0000 7fa619319b80 10 client.25163 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.649+0000 7fa619319b80 10 client.25163 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.649+0000 7fa619319b80 20 client.25163 populate_metadata read hostname 'smithi027' 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.649+0000 7fa619319b80 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55ccd180d9c0 0x55ccd180c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.649+0000 7fa619319b80 4 client.25163 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.649+0000 7fa619319b80 1 -- 172.21.15.27:0/3984330025 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "status"}) -- 0x55ccd16e43b0 con 0x55ccd180d9c0 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.650+0000 7fa5ff400700 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55ccd180d9c0 0x55ccd180c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.650+0000 7fa5ff400700 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55ccd180d9c0 0x55ccd180c6d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2100f87f097176dd server_cookie=ee2db5563d07823 in_seq=0 out_seq=0 2024-05-10T11:12:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.650+0000 7fa5f6000700 10 client.25163 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa5f6000700 1 -- 172.21.15.27:0/3984330025 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ccd16e43b0 con 0x55ccd180d9c0 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa5f6000700 10 client.25163 handle_command_reply: tid=0 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa619319b80 1 client.25163 shutdown 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa619319b80 1 -- 172.21.15.27:0/3984330025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 msgr2=0x55ccd17a7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa619319b80 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 0x55ccd17a7d30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009750 tx=0x7fa5f80098c0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa619319b80 1 -- 172.21.15.27:0/3984330025 shutdown_connections 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa619319b80 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55ccd180d9c0 0x55ccd180c6d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa619319b80 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccd1788980 0x55ccd17a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa619319b80 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ccd1784010 0x55ccd17a7d30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa619319b80 1 --2- 172.21.15.27:0/3984330025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ccd1784930 0x55ccd17a8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa5f6000700 0 client.25163 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa619319b80 1 -- 172.21.15.27:0/3984330025 >> 172.21.15.27:0/3984330025 conn(0x55ccd16b2b10 msgr2=0x55ccd17789d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa619319b80 1 -- 172.21.15.27:0/3984330025 shutdown_connections 2024-05-10T11:12:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa619319b80 1 -- 172.21.15.27:0/3984330025 wait complete. 2024-05-10T11:12:53.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.651+0000 7fa619319b80 20 client.25163 trim_cache size 0 max 16384 2024-05-10T11:12:53.657 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:12:53.657 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T11:12:53.657 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 2, 2024-05-10T11:12:53.657 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24484, 2024-05-10T11:12:53.657 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-05-10T11:12:53.657 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-05-10T11:12:53.657 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-05-10T11:12:53.657 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 220.667956635, 2024-05-10T11:12:53.657 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 29, 2024-05-10T11:12:53.658 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 90, 2024-05-10T11:12:53.658 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 88, 2024-05-10T11:12:53.658 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 339.49783151299999 2024-05-10T11:12:53.658 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:12:53.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.652+0000 7fa60ea00700 1 -- 172.21.15.27:0/3121416332 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa5ec0b2880 msgr2=0x7fa5ec0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:53.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.652+0000 7fa60ea00700 1 --2- 172.21.15.27:0/3121416332 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa5ec0b2880 0x7fa5ec0b4d40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa600009fd0 tx=0x7fa600004b40 comp rx=0 tx=0).stop 2024-05-10T11:12:53.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.652+0000 7fa60ea00700 1 -- 172.21.15.27:0/3121416332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa608103ef0 msgr2=0x7fa608075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:53.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.652+0000 7fa60ea00700 1 --2- 172.21.15.27:0/3121416332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa608103ef0 0x7fa608075ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000e9c0 tx=0x7fa5f000ecd0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.652+0000 7fa60ea00700 1 -- 172.21.15.27:0/3121416332 shutdown_connections 2024-05-10T11:12:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.652+0000 7fa60ea00700 1 --2- 172.21.15.27:0/3121416332 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7fa5ec0b2880 0x7fa5ec0b4d40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.652+0000 7fa60ea00700 1 --2- 172.21.15.27:0/3121416332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa608103550 0x7fa6080757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.652+0000 7fa60ea00700 1 --2- 172.21.15.27:0/3121416332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa608108460 0x7fa608076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.652+0000 7fa60ea00700 1 --2- 172.21.15.27:0/3121416332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa608103ef0 0x7fa608075ce0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.652+0000 7fa60ea00700 1 -- 172.21.15.27:0/3121416332 >> 172.21.15.27:0/3121416332 conn(0x7fa60807a600 msgr2=0x7fa608101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.652+0000 7fa60ea00700 1 -- 172.21.15.27:0/3121416332 shutdown_connections 2024-05-10T11:12:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:53.652+0000 7fa60ea00700 1 -- 172.21.15.27:0/3121416332 wait complete. 2024-05-10T11:12:54.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:54 smithi027 ceph-mon[30054]: pgmap v486: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:54.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:54 smithi076 ceph-mon[34152]: pgmap v486: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:54.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:54 smithi189 ceph-mon[34842]: pgmap v486: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:54.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.h status 2024-05-10T11:12:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:12:56.342 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:56 smithi027 ceph-mon[30054]: pgmap v487: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:56.343 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:56 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:12:56.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:56 smithi076 ceph-mon[34152]: pgmap v487: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:56.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:56 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:12:56.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:56 smithi189 ceph-mon[34842]: pgmap v487: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:12:56.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:56 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:12:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.414+0000 7f8306e00700 1 -- 172.21.15.27:0/1600626796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f830010da10 msgr2=0x7f830010fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.414+0000 7f8306e00700 1 --2- 172.21.15.27:0/1600626796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f830010da10 0x7f830010fed0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f82f800b0a0 tx=0x7f82f800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.414+0000 7f8306e00700 1 -- 172.21.15.27:0/1600626796 shutdown_connections 2024-05-10T11:12:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.414+0000 7f8306e00700 1 --2- 172.21.15.27:0/1600626796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f830010da10 0x7f830010fed0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.414+0000 7f8306e00700 1 --2- 172.21.15.27:0/1600626796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83001045e0 0x7f83001049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.414+0000 7f8306e00700 1 --2- 172.21.15.27:0/1600626796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8300104f00 0x7f830010d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.414+0000 7f8306e00700 1 -- 172.21.15.27:0/1600626796 >> 172.21.15.27:0/1600626796 conn(0x7f83000fde30 msgr2=0x7f8300100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.414+0000 7f8306e00700 1 -- 172.21.15.27:0/1600626796 shutdown_connections 2024-05-10T11:12:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.414+0000 7f8306e00700 1 -- 172.21.15.27:0/1600626796 wait complete. 2024-05-10T11:12:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.415+0000 7f8306e00700 1 Processor -- start 2024-05-10T11:12:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.415+0000 7f8306e00700 1 -- start start 2024-05-10T11:12:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.416+0000 7f8306e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83001045e0 0x7f830019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.416+0000 7f8306e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8300104f00 0x7f830019fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.416+0000 7f8306e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f830010da10 0x7f83001998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.416+0000 7f8306e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8300113610 con 0x7f8300104f00 2024-05-10T11:12:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.416+0000 7f8306e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8300113490 con 0x7f830010da10 2024-05-10T11:12:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.416+0000 7f8306e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8300113790 con 0x7f83001045e0 2024-05-10T11:12:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.416+0000 7f8305000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8300104f00 0x7f830019fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.416+0000 7f8306400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f830010da10 0x7f83001998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.416+0000 7f8305a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83001045e0 0x7f830019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.416+0000 7f8305000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8300104f00 0x7f830019fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42674/0 (socket says 172.21.15.27:42674) 2024-05-10T11:12:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.416+0000 7f8306400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f830010da10 0x7f83001998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:39294/0 (socket says 172.21.15.27:39294) 2024-05-10T11:12:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.416+0000 7f8305000700 1 -- 172.21.15.27:0/1275444612 learned_addr learned my addr 172.21.15.27:0/1275444612 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.417+0000 7f8306400700 1 -- 172.21.15.27:0/1275444612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83001045e0 msgr2=0x7f830019f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.417+0000 7f8306400700 1 --2- 172.21.15.27:0/1275444612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83001045e0 0x7f830019f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.417+0000 7f8306400700 1 -- 172.21.15.27:0/1275444612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8300104f00 msgr2=0x7f830019fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.417+0000 7f8306400700 1 --2- 172.21.15.27:0/1275444612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8300104f00 0x7f830019fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.417+0000 7f8306400700 1 -- 172.21.15.27:0/1275444612 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f830019a030 con 0x7f830010da10 2024-05-10T11:12:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.417+0000 7f8306400700 1 --2- 172.21.15.27:0/1275444612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f830010da10 0x7f83001998d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f82f800b070 tx=0x7f82f8004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:57.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.417+0000 7f82fe000700 1 -- 172.21.15.27:0/1275444612 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f8021070 con 0x7f830010da10 2024-05-10T11:12:57.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.417+0000 7f8306e00700 1 -- 172.21.15.27:0/1275444612 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f830019a2c0 con 0x7f830010da10 2024-05-10T11:12:57.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.417+0000 7f82fe000700 1 -- 172.21.15.27:0/1275444612 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82f8007e30 con 0x7f830010da10 2024-05-10T11:12:57.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.417+0000 7f82fe000700 1 -- 172.21.15.27:0/1275444612 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f80045d0 con 0x7f830010da10 2024-05-10T11:12:57.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.417+0000 7f8306e00700 1 -- 172.21.15.27:0/1275444612 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83001a65e0 con 0x7f830010da10 2024-05-10T11:12:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.421+0000 7f82fe000700 1 -- 172.21.15.27:0/1275444612 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f82f8003c00 con 0x7f830010da10 2024-05-10T11:12:57.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.422+0000 7f8312e68b80 1 Processor -- start 2024-05-10T11:12:57.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.422+0000 7f8312e68b80 1 -- start start 2024-05-10T11:12:57.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f8312e68b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce99782990 0x55ce99782d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f8312e68b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce9977cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f8312e68b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9977d5b0 0x55ce9977da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f8312e68b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ce996cf9d0 con 0x55ce99782990 2024-05-10T11:12:57.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f8312e68b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ce9966d710 con 0x55ce997832b0 2024-05-10T11:12:57.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f8312e68b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ce996cb990 con 0x55ce9977d5b0 2024-05-10T11:12:57.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f82fe000700 1 --2- 172.21.15.27:0/1275444612 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f82e80b2880 0x7f82e80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f8305000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce99782990 0x55ce99782d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f8305000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce99782990 0x55ce99782d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42676/0 (socket says 172.21.15.27:42676) 2024-05-10T11:12:57.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f8305000700 1 -- 172.21.15.27:0/3543715146 learned_addr learned my addr 172.21.15.27:0/3543715146 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:57.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f8305a00700 1 --2- 172.21.15.27:0/3543715146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce9977cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f82fe000700 1 -- 172.21.15.27:0/1275444612 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f82f800f020 con 0x7f830010da10 2024-05-10T11:12:57.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f8305000700 1 --2- 172.21.15.27:0/3543715146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9977d5b0 0x55ce9977da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.423+0000 7f8306400700 1 --2- 172.21.15.27:0/1275444612 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f82e80b2880 0x7f82e80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.424+0000 7f8305a00700 1 -- 172.21.15.27:0/3543715146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9977d5b0 msgr2=0x55ce9977da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:57.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.424+0000 7f8305a00700 1 --2- 172.21.15.27:0/3543715146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9977d5b0 0x55ce9977da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.424+0000 7f8305a00700 1 -- 172.21.15.27:0/3543715146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce99782990 msgr2=0x55ce99782d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:57.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.424+0000 7f8305a00700 1 --2- 172.21.15.27:0/3543715146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce99782990 0x55ce99782d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.424+0000 7f8305a00700 1 -- 172.21.15.27:0/3543715146 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce9961c880 con 0x55ce997832b0 2024-05-10T11:12:57.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.424+0000 7f8305000700 1 --2- 172.21.15.27:0/3543715146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce99782990 0x55ce99782d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:12:57.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.424+0000 7f8306400700 1 --2- 172.21.15.27:0/1275444612 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f82e80b2880 0x7f82e80b4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f82f80c4620 tx=0x7f82f8009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:57.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.425+0000 7f8305a00700 1 --2- 172.21.15.27:0/3543715146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce9977cf90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009c80 tx=0x7f82f4009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f874ae9907169437 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:57.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.425+0000 7f82f2a00700 1 -- 172.21.15.27:0/3543715146 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f4011cb0 con 0x55ce997832b0 2024-05-10T11:12:57.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.425+0000 7f82f2a00700 1 -- 172.21.15.27:0/3543715146 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82f4011e50 con 0x55ce997832b0 2024-05-10T11:12:57.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.425+0000 7f82f2a00700 1 -- 172.21.15.27:0/3543715146 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f40193e0 con 0x55ce997832b0 2024-05-10T11:12:57.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.426+0000 7f8312e68b80 1 -- 172.21.15.27:0/3543715146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 msgr2=0x55ce9977cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:57.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.426+0000 7f8312e68b80 1 --2- 172.21.15.27:0/3543715146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce9977cf90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009c80 tx=0x7f82f4009f90 comp rx=0 tx=0).stop 2024-05-10T11:12:57.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.426+0000 7f8312e68b80 1 -- 172.21.15.27:0/3543715146 shutdown_connections 2024-05-10T11:12:57.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.426+0000 7f8312e68b80 1 --2- 172.21.15.27:0/3543715146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9977d5b0 0x55ce9977da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.426+0000 7f8312e68b80 1 --2- 172.21.15.27:0/3543715146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce99782990 0x55ce99782d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.426+0000 7f8312e68b80 1 --2- 172.21.15.27:0/3543715146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce9977cf90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.426+0000 7f8312e68b80 1 -- 172.21.15.27:0/3543715146 >> 172.21.15.27:0/3543715146 conn(0x55ce996abb10 msgr2=0x55ce99777960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.426+0000 7f8312e68b80 1 -- 172.21.15.27:0/3543715146 shutdown_connections 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.426+0000 7f8312e68b80 1 -- 172.21.15.27:0/3543715146 wait complete. 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.426+0000 7f8312e68b80 1 -- start start 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.427+0000 7f8312e68b80 1 Processor -- start 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.427+0000 7f8312e68b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 0x55ce99788e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.427+0000 7f8312e68b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce99789390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.427+0000 7f8312e68b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce997863f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.427+0000 7f8312e68b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ce99789d60 con 0x55ce9977d5b0 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.427+0000 7f8312e68b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ce99635ad0 con 0x55ce997832b0 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.427+0000 7f8312e68b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ce99789ee0 con 0x55ce99782990 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.427+0000 7f8312e68b80 10 client.0 _ll_register_callbacks cb 0x55ce99676c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.428+0000 7f8305000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 0x55ce99788e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.428+0000 7f8305000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 0x55ce99788e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42686/0 (socket says 172.21.15.27:42686) 2024-05-10T11:12:57.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.428+0000 7f8305000700 1 -- 172.21.15.27:0/2993448866 learned_addr learned my addr 172.21.15.27:0/2993448866 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:57.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.428+0000 7f8305a00700 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce99789390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.428+0000 7f8305000700 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce997863f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.428+0000 7f8305000700 1 -- 172.21.15.27:0/2993448866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 msgr2=0x55ce99789390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:57.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.428+0000 7f8305000700 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce99789390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.428+0000 7f8305000700 1 -- 172.21.15.27:0/2993448866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 msgr2=0x55ce997863f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.428+0000 7f8305000700 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce997863f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.428+0000 7f8305000700 1 -- 172.21.15.27:0/2993448866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce996953d0 con 0x55ce9977d5b0 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.429+0000 7f8305000700 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 0x55ce99788e50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f82ec011750 tx=0x7f82ec009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76cba0c40a2286c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.429+0000 7f82e7400700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.429+0000 7f82e7400700 1 -- 172.21.15.27:0/2993448866 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec01c040 con 0x55ce9977d5b0 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.429+0000 7f82e7400700 1 -- 172.21.15.27:0/2993448866 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f82ec004020 con 0x55ce9977d5b0 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.429+0000 7f8312e68b80 1 -- 172.21.15.27:0/2993448866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82f4011e50 con 0x55ce9977d5b0 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.430+0000 7f82e7400700 1 -- 172.21.15.27:0/2993448866 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f82ec004220 con 0x55ce9977d5b0 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.430+0000 7f8312e68b80 10 client.15045 fetch_fsmap learned FSMap version 29 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.430+0000 7f8312e68b80 1 -- 172.21.15.27:0/2993448866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ce997944d0 con 0x55ce9977d5b0 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.430+0000 7f82e7400700 1 -- 172.21.15.27:0/2993448866 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82f4011e50 con 0x55ce9977d5b0 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.430+0000 7f82e7400700 1 -- 172.21.15.27:0/2993448866 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f82ec0050f0 con 0x55ce9977d5b0 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.430+0000 7f8312e68b80 10 client.15045 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.430+0000 7f8312e68b80 10 client.15045 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-10T11:12:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.430+0000 7f8312e68b80 20 client.15045 populate_metadata read hostname 'smithi027' 2024-05-10T11:12:57.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.430+0000 7f8312e68b80 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce99796b00 0x55ce99797f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.430+0000 7f8312e68b80 4 client.15045 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:12:57.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.430+0000 7f8312e68b80 1 -- 172.21.15.27:0/2993448866 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce9974a050 con 0x55ce99796b00 2024-05-10T11:12:57.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.431+0000 7f8305a00700 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce99796b00 0x55ce99797f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.431+0000 7f8305a00700 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce99796b00 0x55ce99797f70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9bf939251e4b20e0 server_cookie=37d66c005d403638 in_seq=0 out_seq=0 2024-05-10T11:12:57.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.431+0000 7f82e7400700 10 client.15045 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:12:57.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.432+0000 7f82e7400700 1 -- 172.21.15.27:0/2993448866 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ce9974a050 con 0x55ce99796b00 2024-05-10T11:12:57.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.432+0000 7f82e7400700 10 client.15045 handle_command_reply: tid=0 2024-05-10T11:12:57.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.432+0000 7f8312e68b80 1 client.15045 shutdown 2024-05-10T11:12:57.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.433+0000 7f8312e68b80 1 -- 172.21.15.27:0/2993448866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 msgr2=0x55ce99788e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:57.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.433+0000 7f8312e68b80 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 0x55ce99788e50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f82ec011750 tx=0x7f82ec009260 comp rx=0 tx=0).stop 2024-05-10T11:12:57.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.433+0000 7f8312e68b80 1 -- 172.21.15.27:0/2993448866 shutdown_connections 2024-05-10T11:12:57.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.433+0000 7f8312e68b80 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce997863f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.433+0000 7f8312e68b80 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 0x55ce99788e50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.433+0000 7f8312e68b80 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce99796b00 0x55ce99797f70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.433+0000 7f8312e68b80 1 --2- 172.21.15.27:0/2993448866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce99789390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.433+0000 7f8312e68b80 1 -- 172.21.15.27:0/2993448866 >> 172.21.15.27:0/2993448866 conn(0x55ce996abb10 msgr2=0x55ce99772410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:57.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.433+0000 7f82e7400700 0 client.15045 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:12:57.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.433+0000 7f8312e68b80 1 -- 172.21.15.27:0/2993448866 shutdown_connections 2024-05-10T11:12:57.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.433+0000 7f8312e68b80 1 -- 172.21.15.27:0/2993448866 wait complete. 2024-05-10T11:12:57.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.433+0000 7f8312e68b80 20 client.15045 trim_cache size 0 max 16384 2024-05-10T11:12:57.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.456+0000 7f8312e68b80 1 Processor -- start 2024-05-10T11:12:57.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.456+0000 7f8312e68b80 1 -- start start 2024-05-10T11:12:57.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.457+0000 7f8312e68b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 0x55ce9977cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.457+0000 7f8312e68b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce99788660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.457+0000 7f8312e68b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce99788e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.457+0000 7f8312e68b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ce997962b0 con 0x55ce9977d5b0 2024-05-10T11:12:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.457+0000 7f8312e68b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ce99796980 con 0x55ce997832b0 2024-05-10T11:12:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.457+0000 7f8312e68b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ce99797050 con 0x55ce99782990 2024-05-10T11:12:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.457+0000 7f8305a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce99788660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.457+0000 7f8305000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 0x55ce9977cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.457+0000 7f8305a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce99788660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:52288/0 (socket says 172.21.15.27:52288) 2024-05-10T11:12:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.457+0000 7f8305a00700 1 -- 172.21.15.27:0/2769850909 learned_addr learned my addr 172.21.15.27:0/2769850909 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.457+0000 7f8305000700 1 --2- 172.21.15.27:0/2769850909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce99788e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.458+0000 7f8305a00700 1 -- 172.21.15.27:0/2769850909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 msgr2=0x55ce99788e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.458+0000 7f8305a00700 1 --2- 172.21.15.27:0/2769850909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce99788e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.458+0000 7f8305a00700 1 -- 172.21.15.27:0/2769850909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 msgr2=0x55ce9977cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.458+0000 7f8305a00700 1 --2- 172.21.15.27:0/2769850909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 0x55ce9977cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.458+0000 7f8305a00700 1 -- 172.21.15.27:0/2769850909 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce99782d70 con 0x55ce99782990 2024-05-10T11:12:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.458+0000 7f8305000700 1 --2- 172.21.15.27:0/2769850909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 0x55ce9977cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:12:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.458+0000 7f8305a00700 1 --2- 172.21.15.27:0/2769850909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce99788660 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f82f401a000 tx=0x7f82f4009640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5d8b21cb3cbbf3a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:57.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.458+0000 7f82f2000700 1 -- 172.21.15.27:0/2769850909 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f4011ab0 con 0x55ce99782990 2024-05-10T11:12:57.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.458+0000 7f82f2000700 1 -- 172.21.15.27:0/2769850909 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82f4005580 con 0x55ce99782990 2024-05-10T11:12:57.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.458+0000 7f82f2000700 1 -- 172.21.15.27:0/2769850909 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f402b580 con 0x55ce99782990 2024-05-10T11:12:57.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.459+0000 7f8312e68b80 1 -- 172.21.15.27:0/2769850909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 msgr2=0x55ce99788660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:57.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.459+0000 7f8312e68b80 1 --2- 172.21.15.27:0/2769850909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce99788660 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f82f401a000 tx=0x7f82f4009640 comp rx=0 tx=0).stop 2024-05-10T11:12:57.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.459+0000 7f8312e68b80 1 -- 172.21.15.27:0/2769850909 shutdown_connections 2024-05-10T11:12:57.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.459+0000 7f8312e68b80 1 --2- 172.21.15.27:0/2769850909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce997832b0 0x55ce99788e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.459+0000 7f8312e68b80 1 --2- 172.21.15.27:0/2769850909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9977d5b0 0x55ce9977cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.459+0000 7f8312e68b80 1 --2- 172.21.15.27:0/2769850909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce99788660 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.459+0000 7f8312e68b80 1 -- 172.21.15.27:0/2769850909 >> 172.21.15.27:0/2769850909 conn(0x55ce996abb10 msgr2=0x55ce99777710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:57.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.460+0000 7f8312e68b80 1 -- 172.21.15.27:0/2769850909 shutdown_connections 2024-05-10T11:12:57.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.460+0000 7f8312e68b80 1 -- 172.21.15.27:0/2769850909 wait complete. 2024-05-10T11:12:57.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.460+0000 7f8312e68b80 1 -- start start 2024-05-10T11:12:57.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.460+0000 7f8312e68b80 1 Processor -- start 2024-05-10T11:12:57.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.461+0000 7f8312e68b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9977d5b0 0x55ce997a1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.461+0000 7f8312e68b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce9979c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.461+0000 7f8312e68b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce997832b0 0x55ce9979c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.461+0000 7f8312e68b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ce99797df0 con 0x55ce997832b0 2024-05-10T11:12:57.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.461+0000 7f8312e68b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ce99797720 con 0x55ce9977d5b0 2024-05-10T11:12:57.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.461+0000 7f8312e68b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ce997984c0 con 0x55ce99782990 2024-05-10T11:12:57.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.461+0000 7f8312e68b80 10 client.0 _ll_register_callbacks cb 0x55ce99676c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:12:57.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.461+0000 7f8305a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce9979c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.461+0000 7f8305a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce9979c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:52298/0 (socket says 172.21.15.27:52298) 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.461+0000 7f8305a00700 1 -- 172.21.15.27:0/4184750166 learned_addr learned my addr 172.21.15.27:0/4184750166 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.461+0000 7f8305000700 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9977d5b0 0x55ce997a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.461+0000 7f8305000700 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce997832b0 0x55ce9979c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.462+0000 7f8305a00700 1 -- 172.21.15.27:0/4184750166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9977d5b0 msgr2=0x55ce997a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.462+0000 7f8305a00700 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9977d5b0 0x55ce997a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.462+0000 7f8305a00700 1 -- 172.21.15.27:0/4184750166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce997832b0 msgr2=0x55ce9979c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.462+0000 7f8305a00700 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce997832b0 0x55ce9979c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.462+0000 7f8305a00700 1 -- 172.21.15.27:0/4184750166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce9979cf60 con 0x55ce99782990 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.462+0000 7f8305a00700 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce9979c3a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009fd0 tx=0x7f82f4005730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c9af93032514f85 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.462+0000 7f82e7400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.462+0000 7f82e7400700 1 -- 172.21.15.27:0/4184750166 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f402f070 con 0x55ce99782990 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.462+0000 7f82e7400700 1 -- 172.21.15.27:0/4184750166 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f82f4005250 con 0x55ce99782990 2024-05-10T11:12:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.462+0000 7f8312e68b80 1 -- 172.21.15.27:0/4184750166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82f4005580 con 0x55ce99782990 2024-05-10T11:12:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.463+0000 7f82e7400700 1 -- 172.21.15.27:0/4184750166 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f82f403b050 con 0x55ce99782990 2024-05-10T11:12:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.463+0000 7f8312e68b80 10 client.25165 fetch_fsmap learned FSMap version 29 2024-05-10T11:12:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.463+0000 7f8312e68b80 1 -- 172.21.15.27:0/4184750166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ce99802e20 con 0x55ce99782990 2024-05-10T11:12:57.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.463+0000 7f82e7400700 1 -- 172.21.15.27:0/4184750166 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82f4005580 con 0x55ce99782990 2024-05-10T11:12:57.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.463+0000 7f82e7400700 1 -- 172.21.15.27:0/4184750166 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f82f4018070 con 0x55ce99782990 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.463+0000 7f8312e68b80 10 client.25165 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.463+0000 7f8312e68b80 10 client.25165 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.463+0000 7f8312e68b80 20 client.25165 populate_metadata read hostname 'smithi027' 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.463+0000 7f8312e68b80 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce99807240 0x55ce99805f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.463+0000 7f8312e68b80 4 client.25165 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.463+0000 7f8312e68b80 1 -- 172.21.15.27:0/4184750166 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "status"}) -- 0x55ce996dd3b0 con 0x55ce99807240 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.464+0000 7f8305000700 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce99807240 0x55ce99805f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.464+0000 7f8305000700 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce99807240 0x55ce99805f50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7a49565f60679bc9 server_cookie=54960039357ada7c in_seq=0 out_seq=0 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.464+0000 7f82e7400700 10 client.25165 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f82e7400700 1 -- 172.21.15.27:0/4184750166 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ce996dd3b0 con 0x55ce99807240 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f82e7400700 10 client.25165 handle_command_reply: tid=0 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f8312e68b80 1 client.25165 shutdown 2024-05-10T11:12:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f8312e68b80 1 -- 172.21.15.27:0/4184750166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 msgr2=0x55ce9979c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f8312e68b80 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce9979c3a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009fd0 tx=0x7f82f4005730 comp rx=0 tx=0).stop 2024-05-10T11:12:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f8312e68b80 1 -- 172.21.15.27:0/4184750166 shutdown_connections 2024-05-10T11:12:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f8312e68b80 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce997832b0 0x55ce9979c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f8312e68b80 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9977d5b0 0x55ce997a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f8312e68b80 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce99807240 0x55ce99805f50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f8312e68b80 1 --2- 172.21.15.27:0/4184750166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce99782990 0x55ce9979c3a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f8312e68b80 1 -- 172.21.15.27:0/4184750166 >> 172.21.15.27:0/4184750166 conn(0x55ce996abb10 msgr2=0x55ce99788e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f82e7400700 0 client.25165 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:12:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f8312e68b80 1 -- 172.21.15.27:0/4184750166 shutdown_connections 2024-05-10T11:12:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f8312e68b80 1 -- 172.21.15.27:0/4184750166 wait complete. 2024-05-10T11:12:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.465+0000 7f8312e68b80 20 client.25165 trim_cache size 0 max 16384 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 1, 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24488, 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 225.494553612, 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 29, 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 90, 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 88, 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 346.09552447200002 2024-05-10T11:12:57.468 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:12:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.466+0000 7f8306e00700 1 -- 172.21.15.27:0/1275444612 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f82e80b2880 msgr2=0x7f82e80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.466+0000 7f8306e00700 1 --2- 172.21.15.27:0/1275444612 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f82e80b2880 0x7f82e80b4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f82f80c4620 tx=0x7f82f8009040 comp rx=0 tx=0).stop 2024-05-10T11:12:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.466+0000 7f8306e00700 1 -- 172.21.15.27:0/1275444612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f830010da10 msgr2=0x7f83001998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:12:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.466+0000 7f8306e00700 1 --2- 172.21.15.27:0/1275444612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f830010da10 0x7f83001998d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f82f800b070 tx=0x7f82f8004d00 comp rx=0 tx=0).stop 2024-05-10T11:12:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.466+0000 7f8306e00700 1 -- 172.21.15.27:0/1275444612 shutdown_connections 2024-05-10T11:12:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.466+0000 7f8306e00700 1 --2- 172.21.15.27:0/1275444612 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f82e80b2880 0x7f82e80b4d40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.466+0000 7f8306e00700 1 --2- 172.21.15.27:0/1275444612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83001045e0 0x7f830019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.466+0000 7f8306e00700 1 --2- 172.21.15.27:0/1275444612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f830010da10 0x7f83001998d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.466+0000 7f8306e00700 1 --2- 172.21.15.27:0/1275444612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8300104f00 0x7f830019fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:12:57.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.466+0000 7f8306e00700 1 -- 172.21.15.27:0/1275444612 >> 172.21.15.27:0/1275444612 conn(0x7f83000fde30 msgr2=0x7f83000feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:12:57.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.467+0000 7f8306e00700 1 -- 172.21.15.27:0/1275444612 shutdown_connections 2024-05-10T11:12:57.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:12:57.467+0000 7f8306e00700 1 -- 172.21.15.27:0/1275444612 wait complete. 2024-05-10T11:12:58.350 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:58 smithi027 ceph-mon[30054]: pgmap v488: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:58.350 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:58 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:12:58.350 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:58 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:12:58.350 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:12:58 smithi027 ceph-mon[30054]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:12:58.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.b status 2024-05-10T11:12:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:58 smithi076 ceph-mon[34152]: pgmap v488: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:58 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:12:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:58 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:12:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:12:58 smithi076 ceph-mon[34152]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:12:58.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:58 smithi189 ceph-mon[34842]: pgmap v488: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-10T11:12:58.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:58 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:12:58.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:58 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:12:58.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:12:58 smithi189 ceph-mon[34842]: from='mgr.14152 172.21.15.27:0/1707889537' entity='mgr.x' 2024-05-10T11:12:58.903 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:13:00.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:00 smithi027 ceph-mon[30054]: pgmap v489: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:13:00.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:00 smithi076 ceph-mon[34152]: pgmap v489: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:13:00.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:00 smithi189 ceph-mon[34842]: pgmap v489: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-10T11:13:01.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.402+0000 7efef5600700 1 -- 172.21.15.27:0/1743878173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef0110250 msgr2=0x7efef0112640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:01.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.402+0000 7efef5600700 1 --2- 172.21.15.27:0/1743878173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef0110250 0x7efef0112640 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efee800b0a0 tx=0x7efee800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.402+0000 7efef5600700 1 -- 172.21.15.27:0/1743878173 shutdown_connections 2024-05-10T11:13:01.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.402+0000 7efef5600700 1 --2- 172.21.15.27:0/1743878173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef0110250 0x7efef0112640 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.402+0000 7efef5600700 1 --2- 172.21.15.27:0/1743878173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efef006bc50 0x7efef006c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.402+0000 7efef5600700 1 --2- 172.21.15.27:0/1743878173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef006c570 0x7efef010fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.402+0000 7efef5600700 1 -- 172.21.15.27:0/1743878173 >> 172.21.15.27:0/1743878173 conn(0x7efef00791a0 msgr2=0x7efef00795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:01.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.402+0000 7efef5600700 1 -- 172.21.15.27:0/1743878173 shutdown_connections 2024-05-10T11:13:01.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.402+0000 7efef5600700 1 -- 172.21.15.27:0/1743878173 wait complete. 2024-05-10T11:13:01.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.403+0000 7efef5600700 1 Processor -- start 2024-05-10T11:13:01.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.404+0000 7efef5600700 1 -- start start 2024-05-10T11:13:01.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.404+0000 7efef5600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef006bc50 0x7efef019be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.404+0000 7efef5600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef006c570 0x7efef019c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.404+0000 7efef5600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efef0110250 0x7efef01a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.404+0000 7efef5600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efef01159a0 con 0x7efef006c570 2024-05-10T11:13:01.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.404+0000 7efef5600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efef0115820 con 0x7efef0110250 2024-05-10T11:13:01.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.404+0000 7efef5600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efef0115b20 con 0x7efef006bc50 2024-05-10T11:13:01.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.404+0000 7efeeea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef006bc50 0x7efef019be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.404+0000 7efeee000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef006c570 0x7efef019c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.404+0000 7efeeea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef006bc50 0x7efef019be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:52304/0 (socket says 172.21.15.27:52304) 2024-05-10T11:13:01.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.404+0000 7efeeea00700 1 -- 172.21.15.27:0/1098942398 learned_addr learned my addr 172.21.15.27:0/1098942398 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:01.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.405+0000 7efeef400700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efef0110250 0x7efef01a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.405+0000 7efeeea00700 1 -- 172.21.15.27:0/1098942398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efef0110250 msgr2=0x7efef01a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:01.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.405+0000 7efeeea00700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efef0110250 0x7efef01a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.405+0000 7efeeea00700 1 -- 172.21.15.27:0/1098942398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef006c570 msgr2=0x7efef019c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:01.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.405+0000 7efeeea00700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef006c570 0x7efef019c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.405+0000 7efeeea00700 1 -- 172.21.15.27:0/1098942398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef01a0e10 con 0x7efef006bc50 2024-05-10T11:13:01.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.405+0000 7efeee000700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef006c570 0x7efef019c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:01.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.405+0000 7efeeea00700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef006bc50 0x7efef019be50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efee000ea50 tx=0x7efee000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:01.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.405+0000 7efee7400700 1 -- 172.21.15.27:0/1098942398 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee000cbd0 con 0x7efef006bc50 2024-05-10T11:13:01.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.406+0000 7efef5600700 1 -- 172.21.15.27:0/1098942398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef01a1100 con 0x7efef006bc50 2024-05-10T11:13:01.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.406+0000 7efee7400700 1 -- 172.21.15.27:0/1098942398 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee000cd70 con 0x7efef006bc50 2024-05-10T11:13:01.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.406+0000 7efee7400700 1 -- 172.21.15.27:0/1098942398 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0018680 con 0x7efef006bc50 2024-05-10T11:13:01.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.406+0000 7efef5600700 1 -- 172.21.15.27:0/1098942398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef01a8980 con 0x7efef006bc50 2024-05-10T11:13:01.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.409+0000 7efee7400700 1 -- 172.21.15.27:0/1098942398 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7efee001f020 con 0x7efef006bc50 2024-05-10T11:13:01.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.410+0000 7eff0000ab80 1 Processor -- start 2024-05-10T11:13:01.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.411+0000 7eff0000ab80 1 -- start start 2024-05-10T11:13:01.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.411+0000 7eff0000ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad603f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.411+0000 7eff0000ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60930 0x55c10ad64440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.411+0000 7eff0000ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad64980 0x55c10ad64de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.411+0000 7eff0000ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c10acb29d0 con 0x55c10ad60010 2024-05-10T11:13:01.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.411+0000 7eff0000ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c10ac50710 con 0x55c10ad64980 2024-05-10T11:13:01.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.411+0000 7eff0000ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c10acae990 con 0x55c10ad60930 2024-05-10T11:13:01.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.412+0000 7efeef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.412+0000 7efeef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad603f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42748/0 (socket says 172.21.15.27:42748) 2024-05-10T11:13:01.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.412+0000 7efeef400700 1 -- 172.21.15.27:0/1973594113 learned_addr learned my addr 172.21.15.27:0/1973594113 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.412+0000 7efeee000700 1 --2- 172.21.15.27:0/1973594113 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad64980 0x55c10ad64de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.412+0000 7efeeea00700 1 --2- 172.21.15.27:0/1973594113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60930 0x55c10ad64440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.412+0000 7efeef400700 1 -- 172.21.15.27:0/1973594113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60930 msgr2=0x55c10ad64440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.412+0000 7efeef400700 1 --2- 172.21.15.27:0/1973594113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60930 0x55c10ad64440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.412+0000 7efeef400700 1 -- 172.21.15.27:0/1973594113 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad64980 msgr2=0x55c10ad64de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.412+0000 7efeef400700 1 --2- 172.21.15.27:0/1973594113 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad64980 0x55c10ad64de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.412+0000 7efeef400700 1 -- 172.21.15.27:0/1973594113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c10abff880 con 0x55c10ad60010 2024-05-10T11:13:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.412+0000 7efee7400700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7efed40b2880 0x7efed40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.413+0000 7efeef400700 1 --2- 172.21.15.27:0/1973594113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad603f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efee800b0a0 tx=0x7efee8009470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427d460b530f2afa server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:01.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.413+0000 7efeeea00700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7efed40b2880 0x7efed40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.413+0000 7efee7400700 1 -- 172.21.15.27:0/1098942398 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7efee00e25f0 con 0x7efef006bc50 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.413+0000 7efedbe00700 1 -- 172.21.15.27:0/1973594113 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee800bcb0 con 0x55c10ad60010 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.413+0000 7efedbe00700 1 -- 172.21.15.27:0/1973594113 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee80074e0 con 0x55c10ad60010 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.413+0000 7efedbe00700 1 -- 172.21.15.27:0/1973594113 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8005260 con 0x55c10ad60010 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.414+0000 7eff0000ab80 1 -- 172.21.15.27:0/1973594113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 msgr2=0x55c10ad603f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.414+0000 7efeeea00700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7efed40b2880 0x7efed40b4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efee00a6a30 tx=0x7efee0010c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.414+0000 7eff0000ab80 1 --2- 172.21.15.27:0/1973594113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad603f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efee800b0a0 tx=0x7efee8009470 comp rx=0 tx=0).stop 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.414+0000 7eff0000ab80 1 -- 172.21.15.27:0/1973594113 shutdown_connections 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.414+0000 7eff0000ab80 1 --2- 172.21.15.27:0/1973594113 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad64980 0x55c10ad64de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.414+0000 7eff0000ab80 1 --2- 172.21.15.27:0/1973594113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad603f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.414+0000 7eff0000ab80 1 --2- 172.21.15.27:0/1973594113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60930 0x55c10ad64440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.414+0000 7eff0000ab80 1 -- 172.21.15.27:0/1973594113 >> 172.21.15.27:0/1973594113 conn(0x55c10ac8eb10 msgr2=0x55c10ad5a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:01.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.414+0000 7eff0000ab80 1 -- 172.21.15.27:0/1973594113 shutdown_connections 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.414+0000 7eff0000ab80 1 -- 172.21.15.27:0/1973594113 wait complete. 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.415+0000 7eff0000ab80 1 -- start start 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.415+0000 7eff0000ab80 1 Processor -- start 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.416+0000 7eff0000ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60010 0x55c10ad6bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.416+0000 7eff0000ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 0x55c10ad69170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.416+0000 7eff0000ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad64980 0x55c10ad696b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.416+0000 7eff0000ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c10ad6cb30 con 0x55c10ad64980 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.416+0000 7eff0000ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c10ac18ad0 con 0x55c10ad60930 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.416+0000 7eff0000ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c10ad6ccb0 con 0x55c10ad60010 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.416+0000 7eff0000ab80 10 client.0 _ll_register_callbacks cb 0x55c10ac59c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.416+0000 7efeee000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 0x55c10ad69170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.416+0000 7efeee000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 0x55c10ad69170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:39368/0 (socket says 172.21.15.27:39368) 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.416+0000 7efeee000700 1 -- 172.21.15.27:0/267093646 learned_addr learned my addr 172.21.15.27:0/267093646 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.416+0000 7efeef400700 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad64980 0x55c10ad696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.416+0000 7efeef400700 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60010 0x55c10ad6bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.417+0000 7efeee000700 1 -- 172.21.15.27:0/267093646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60010 msgr2=0x55c10ad6bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:01.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.417+0000 7efeee000700 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60010 0x55c10ad6bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.417+0000 7efeee000700 1 -- 172.21.15.27:0/267093646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad64980 msgr2=0x55c10ad696b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:01.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.417+0000 7efeee000700 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad64980 0x55c10ad696b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.417+0000 7efeee000700 1 -- 172.21.15.27:0/267093646 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c10ac783d0 con 0x55c10ad60930 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.417+0000 7efeef400700 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad64980 0x55c10ad696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.417+0000 7efeee000700 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 0x55c10ad69170 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efedc012d50 tx=0x7efedc00f6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89b3fbdf67ec85e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.417+0000 7efed8c00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.417+0000 7efed8c00700 1 -- 172.21.15.27:0/267093646 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc009e00 con 0x55c10ad60930 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.417+0000 7efed8c00700 1 -- 172.21.15.27:0/267093646 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7efedc00fc30 con 0x55c10ad60930 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.417+0000 7eff0000ab80 1 -- 172.21.15.27:0/267093646 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efee80074e0 con 0x55c10ad60930 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.418+0000 7efed8c00700 1 -- 172.21.15.27:0/267093646 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efedc010430 con 0x55c10ad60930 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.418+0000 7eff0000ab80 10 client.25193 fetch_fsmap learned FSMap version 29 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.418+0000 7eff0000ab80 1 -- 172.21.15.27:0/267093646 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c10ad77410 con 0x55c10ad60930 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.418+0000 7efed8c00700 1 -- 172.21.15.27:0/267093646 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee80074e0 con 0x55c10ad60930 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.418+0000 7efed8c00700 1 -- 172.21.15.27:0/267093646 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efedc01c420 con 0x55c10ad60930 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.418+0000 7eff0000ab80 10 client.25193 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:13:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.418+0000 7eff0000ab80 10 client.25193 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.418+0000 7eff0000ab80 20 client.25193 populate_metadata read hostname 'smithi027' 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.418+0000 7eff0000ab80 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c10ad79a90 0x55c10ad7af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.418+0000 7eff0000ab80 4 client.25193 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.418+0000 7eff0000ab80 1 -- 172.21.15.27:0/267093646 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c10ad2d050 con 0x55c10ad79a90 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.419+0000 7efeef400700 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c10ad79a90 0x55c10ad7af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.419+0000 7efeef400700 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c10ad79a90 0x55c10ad7af00 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acc97a6d0be5e8ef server_cookie=52098a16986cc8c0 in_seq=0 out_seq=0 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.419+0000 7efed8c00700 10 client.25193 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.420+0000 7efed8c00700 1 -- 172.21.15.27:0/267093646 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c10ad2d050 con 0x55c10ad79a90 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.420+0000 7efed8c00700 10 client.25193 handle_command_reply: tid=0 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.420+0000 7eff0000ab80 1 client.25193 shutdown 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.420+0000 7eff0000ab80 1 -- 172.21.15.27:0/267093646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 msgr2=0x55c10ad69170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.420+0000 7eff0000ab80 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 0x55c10ad69170 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efedc012d50 tx=0x7efedc00f6a0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.420+0000 7eff0000ab80 1 -- 172.21.15.27:0/267093646 shutdown_connections 2024-05-10T11:13:01.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.420+0000 7eff0000ab80 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c10ad79a90 0x55c10ad7af00 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.420+0000 7eff0000ab80 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60010 0x55c10ad6bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.420+0000 7eff0000ab80 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad64980 0x55c10ad696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.420+0000 7eff0000ab80 1 --2- 172.21.15.27:0/267093646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 0x55c10ad69170 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.420+0000 7eff0000ab80 1 -- 172.21.15.27:0/267093646 >> 172.21.15.27:0/267093646 conn(0x55c10ac8eb10 msgr2=0x55c10ad5a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:01.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.420+0000 7efed8c00700 0 client.25193 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:13:01.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.421+0000 7eff0000ab80 1 -- 172.21.15.27:0/267093646 shutdown_connections 2024-05-10T11:13:01.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.421+0000 7eff0000ab80 1 -- 172.21.15.27:0/267093646 wait complete. 2024-05-10T11:13:01.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.421+0000 7eff0000ab80 20 client.25193 trim_cache size 0 max 16384 2024-05-10T11:13:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.444+0000 7eff0000ab80 1 Processor -- start 2024-05-10T11:13:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.444+0000 7eff0000ab80 1 -- start start 2024-05-10T11:13:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.444+0000 7eff0000ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad696e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.444+0000 7eff0000ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60930 0x55c10ad84360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.444+0000 7eff0000ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad64980 0x55c10ad848a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7eff0000ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c10ad79240 con 0x55c10ad60010 2024-05-10T11:13:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7eff0000ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c10ad79910 con 0x55c10ad64980 2024-05-10T11:13:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7eff0000ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c10ad79fe0 con 0x55c10ad60930 2024-05-10T11:13:01.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7efeef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7efeef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad696e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42766/0 (socket says 172.21.15.27:42766) 2024-05-10T11:13:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7efeef400700 1 -- 172.21.15.27:0/3621668527 learned_addr learned my addr 172.21.15.27:0/3621668527 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7efeee000700 1 --2- 172.21.15.27:0/3621668527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60930 0x55c10ad84360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7efeef400700 1 --2- 172.21.15.27:0/3621668527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad64980 0x55c10ad848a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7efeef400700 1 -- 172.21.15.27:0/3621668527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60930 msgr2=0x55c10ad84360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7efeef400700 1 --2- 172.21.15.27:0/3621668527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60930 0x55c10ad84360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7efeef400700 1 -- 172.21.15.27:0/3621668527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad64980 msgr2=0x55c10ad848a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7efeef400700 1 --2- 172.21.15.27:0/3621668527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad64980 0x55c10ad848a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.445+0000 7efeef400700 1 -- 172.21.15.27:0/3621668527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c10ad5b5c0 con 0x55c10ad60010 2024-05-10T11:13:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.446+0000 7efeef400700 1 --2- 172.21.15.27:0/3621668527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad696e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efee8007fd0 tx=0x7efee8006210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702735d47be30836 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.446+0000 7efedb400700 1 -- 172.21.15.27:0/3621668527 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8034070 con 0x55c10ad60010 2024-05-10T11:13:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.446+0000 7efedb400700 1 -- 172.21.15.27:0/3621668527 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee800be20 con 0x55c10ad60010 2024-05-10T11:13:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.446+0000 7efedb400700 1 -- 172.21.15.27:0/3621668527 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8034610 con 0x55c10ad60010 2024-05-10T11:13:01.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.447+0000 7eff0000ab80 1 -- 172.21.15.27:0/3621668527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 msgr2=0x55c10ad696e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:01.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.447+0000 7eff0000ab80 1 --2- 172.21.15.27:0/3621668527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad696e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efee8007fd0 tx=0x7efee8006210 comp rx=0 tx=0).stop 2024-05-10T11:13:01.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.447+0000 7eff0000ab80 1 -- 172.21.15.27:0/3621668527 shutdown_connections 2024-05-10T11:13:01.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.447+0000 7eff0000ab80 1 --2- 172.21.15.27:0/3621668527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad64980 0x55c10ad848a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.447+0000 7eff0000ab80 1 --2- 172.21.15.27:0/3621668527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad696e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.447+0000 7eff0000ab80 1 --2- 172.21.15.27:0/3621668527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad60930 0x55c10ad84360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.447+0000 7eff0000ab80 1 -- 172.21.15.27:0/3621668527 >> 172.21.15.27:0/3621668527 conn(0x55c10ac8eb10 msgr2=0x55c10ad5a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.447+0000 7eff0000ab80 1 -- 172.21.15.27:0/3621668527 shutdown_connections 2024-05-10T11:13:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.447+0000 7eff0000ab80 1 -- 172.21.15.27:0/3621668527 wait complete. 2024-05-10T11:13:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.448+0000 7eff0000ab80 1 -- start start 2024-05-10T11:13:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.448+0000 7eff0000ab80 1 Processor -- start 2024-05-10T11:13:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.449+0000 7eff0000ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad844a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.449+0000 7eff0000ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 0x55c10ad849e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.449+0000 7eff0000ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad64980 0x55c10ad7fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.449+0000 7eff0000ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c10ad7ad80 con 0x55c10ad60010 2024-05-10T11:13:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.449+0000 7eff0000ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c10ad7a6b0 con 0x55c10ad60930 2024-05-10T11:13:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.449+0000 7eff0000ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c10ad7b450 con 0x55c10ad64980 2024-05-10T11:13:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.449+0000 7eff0000ab80 10 client.0 _ll_register_callbacks cb 0x55c10ac59c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:13:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.449+0000 7efeef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad844a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.449+0000 7efeee000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 0x55c10ad849e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.449+0000 7efeee000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 0x55c10ad849e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:39380/0 (socket says 172.21.15.27:39380) 2024-05-10T11:13:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.449+0000 7efeef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad844a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42774/0 (socket says 172.21.15.27:42774) 2024-05-10T11:13:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.449+0000 7efeee000700 1 -- 172.21.15.27:0/2539622361 learned_addr learned my addr 172.21.15.27:0/2539622361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7efeee000700 1 -- 172.21.15.27:0/2539622361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad64980 msgr2=0x55c10ad7fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7efeee000700 1 --2- 172.21.15.27:0/2539622361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad64980 0x55c10ad7fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7efeee000700 1 -- 172.21.15.27:0/2539622361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 msgr2=0x55c10ad844a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7efeee000700 1 --2- 172.21.15.27:0/2539622361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad844a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7efeee000700 1 -- 172.21.15.27:0/2539622361 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c10ad573a0 con 0x55c10ad60930 2024-05-10T11:13:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7efeef400700 1 --2- 172.21.15.27:0/2539622361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad844a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7efeee000700 1 --2- 172.21.15.27:0/2539622361 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 0x55c10ad849e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x55c10ad6a6f0 tx=0x7efedc00fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adbd8c48b79665dc server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7efed8c00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:13:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7efed8c00700 1 -- 172.21.15.27:0/2539622361 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc030060 con 0x55c10ad60930 2024-05-10T11:13:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7efed8c00700 1 -- 172.21.15.27:0/2539622361 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7efedc01c630 con 0x55c10ad60930 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7eff0000ab80 1 -- 172.21.15.27:0/2539622361 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efee800be20 con 0x55c10ad60930 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7efed8c00700 1 -- 172.21.15.27:0/2539622361 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efedc01cab0 con 0x55c10ad60930 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.450+0000 7eff0000ab80 10 client.25199 fetch_fsmap learned FSMap version 29 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.451+0000 7eff0000ab80 1 -- 172.21.15.27:0/2539622361 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c10ade4a10 con 0x55c10ad60930 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.451+0000 7efed8c00700 1 -- 172.21.15.27:0/2539622361 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee800be20 con 0x55c10ad60930 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.451+0000 7efed8c00700 1 -- 172.21.15.27:0/2539622361 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efedc0037a0 con 0x55c10ad60930 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.451+0000 7eff0000ab80 10 client.25199 fetch_fsmap finished waiting for FSMap version 29 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.451+0000 7eff0000ab80 10 client.25199 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.451+0000 7eff0000ab80 20 client.25199 populate_metadata read hostname 'smithi027' 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.451+0000 7eff0000ab80 1 --2- 172.21.15.27:0/2539622361 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c10ade96a0 0x55c10ade83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.451+0000 7eff0000ab80 4 client.25199 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.451+0000 7eff0000ab80 1 -- 172.21.15.27:0/2539622361 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "status"}) -- 0x55c10acc03b0 con 0x55c10ade96a0 2024-05-10T11:13:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.452+0000 7efeef400700 1 --2- 172.21.15.27:0/2539622361 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c10ade96a0 0x55c10ade83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.452+0000 7efeef400700 1 --2- 172.21.15.27:0/2539622361 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c10ade96a0 0x55c10ade83b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f032b7b3e2adfe8d server_cookie=d85ee4bc1a7dc0c6 in_seq=0 out_seq=0 2024-05-10T11:13:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.452+0000 7efed8c00700 10 client.25199 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:13:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.452+0000 7efed8c00700 1 -- 172.21.15.27:0/2539622361 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c10acc03b0 con 0x55c10ade96a0 2024-05-10T11:13:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.452+0000 7efed8c00700 10 client.25199 handle_command_reply: tid=0 2024-05-10T11:13:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.452+0000 7eff0000ab80 1 client.25199 shutdown 2024-05-10T11:13:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.453+0000 7eff0000ab80 1 -- 172.21.15.27:0/2539622361 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 msgr2=0x55c10ad849e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.453+0000 7eff0000ab80 1 --2- 172.21.15.27:0/2539622361 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 0x55c10ad849e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x55c10ad6a6f0 tx=0x7efedc00fb20 comp rx=0 tx=0).stop 2024-05-10T11:13:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.453+0000 7eff0000ab80 1 -- 172.21.15.27:0/2539622361 shutdown_connections 2024-05-10T11:13:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.453+0000 7eff0000ab80 1 --2- 172.21.15.27:0/2539622361 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c10ade96a0 0x55c10ade83b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.453+0000 7eff0000ab80 1 --2- 172.21.15.27:0/2539622361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c10ad64980 0x55c10ad7fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.453+0000 7eff0000ab80 1 --2- 172.21.15.27:0/2539622361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c10ad60010 0x55c10ad844a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.453+0000 7eff0000ab80 1 --2- 172.21.15.27:0/2539622361 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c10ad60930 0x55c10ad849e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.453+0000 7eff0000ab80 1 -- 172.21.15.27:0/2539622361 >> 172.21.15.27:0/2539622361 conn(0x55c10ac8eb10 msgr2=0x55c10ad55260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.453+0000 7efed8c00700 0 client.25199 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:13:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.453+0000 7eff0000ab80 1 -- 172.21.15.27:0/2539622361 shutdown_connections 2024-05-10T11:13:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.453+0000 7eff0000ab80 1 -- 172.21.15.27:0/2539622361 wait complete. 2024-05-10T11:13:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.453+0000 7eff0000ab80 20 client.25199 trim_cache size 0 max 16384 2024-05-10T11:13:01.458 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:13:01.458 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "063c9e18-0ebc-11ef-bc98-c7b262605968", 2024-05-10T11:13:01.458 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-05-10T11:13:01.458 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24527, 2024-05-10T11:13:01.458 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-05-10T11:13:01.458 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-05-10T11:13:01.459 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-05-10T11:13:01.459 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 311.15972232299998, 2024-05-10T11:13:01.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 29, 2024-05-10T11:13:01.459 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 90, 2024-05-10T11:13:01.459 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 88, 2024-05-10T11:13:01.459 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 334.918076966 2024-05-10T11:13:01.459 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:13:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.454+0000 7efef5600700 1 -- 172.21.15.27:0/1098942398 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7efed40b2880 msgr2=0x7efed40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.454+0000 7efef5600700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7efed40b2880 0x7efed40b4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efee00a6a30 tx=0x7efee0010c80 comp rx=0 tx=0).stop 2024-05-10T11:13:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.454+0000 7efef5600700 1 -- 172.21.15.27:0/1098942398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef006bc50 msgr2=0x7efef019be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.454+0000 7efef5600700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef006bc50 0x7efef019be50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efee000ea50 tx=0x7efee000ed60 comp rx=0 tx=0).stop 2024-05-10T11:13:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.454+0000 7efef5600700 1 -- 172.21.15.27:0/1098942398 shutdown_connections 2024-05-10T11:13:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.454+0000 7efef5600700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7efed40b2880 0x7efed40b4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.454+0000 7efef5600700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef006bc50 0x7efef019be50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.454+0000 7efef5600700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efef0110250 0x7efef01a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.454+0000 7efef5600700 1 --2- 172.21.15.27:0/1098942398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef006c570 0x7efef019c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.454+0000 7efef5600700 1 -- 172.21.15.27:0/1098942398 >> 172.21.15.27:0/1098942398 conn(0x7efef00791a0 msgr2=0x7efef0111640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.454+0000 7efef5600700 1 -- 172.21.15.27:0/1098942398 shutdown_connections 2024-05-10T11:13:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:01.454+0000 7efef5600700 1 -- 172.21.15.27:0/1098942398 wait complete. 2024-05-10T11:13:02.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:02 smithi027 ceph-mon[30054]: pgmap v490: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:13:02.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:02 smithi076 ceph-mon[34152]: pgmap v490: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:13:02.481 INFO:teuthology.run_tasks:Running task sequential... 2024-05-10T11:13:02.492 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-10T11:13:02.492 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-10T11:13:02.493 INFO:teuthology.task.print:Enabling mgr modules 2024-05-10T11:13:02.493 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-05-10T11:13:02.494 INFO:teuthology.task.exec:Executing custom commands... 2024-05-10T11:13:02.494 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-05-10T11:13:02.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-05-10T11:13:02.495 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:02 smithi189 ceph-mon[34842]: pgmap v490: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-10T11:13:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.713+0000 7f7712e00640 1 -- 172.21.15.27:0/2504636214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770c104a70 msgr2=0x7f770c104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.713+0000 7f7712e00640 1 --2- 172.21.15.27:0/2504636214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770c104a70 0x7f770c104e90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7700009900 tx=0x7f770002a3f0 comp rx=0 tx=0).stop 2024-05-10T11:13:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.713+0000 7f7712e00640 1 -- 172.21.15.27:0/2504636214 shutdown_connections 2024-05-10T11:13:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.713+0000 7f7712e00640 1 --2- 172.21.15.27:0/2504636214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f770c106620 0x7f770c10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.713+0000 7f7712e00640 1 --2- 172.21.15.27:0/2504636214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f770c105c60 0x7f770c1060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.713+0000 7f7712e00640 1 --2- 172.21.15.27:0/2504636214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770c104a70 0x7f770c104e90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:02.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.713+0000 7f7712e00640 1 -- 172.21.15.27:0/2504636214 >> 172.21.15.27:0/2504636214 conn(0x7f770c100020 msgr2=0x7f770c102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:02.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.713+0000 7f7712e00640 1 -- 172.21.15.27:0/2504636214 shutdown_connections 2024-05-10T11:13:02.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.713+0000 7f7712e00640 1 -- 172.21.15.27:0/2504636214 wait complete. 2024-05-10T11:13:02.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.714+0000 7f7712e00640 1 Processor -- start 2024-05-10T11:13:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.715+0000 7f7712e00640 1 -- start start 2024-05-10T11:13:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.715+0000 7f7712e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f770c104a70 0x7f770c19ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.715+0000 7f7712e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f770c105c60 0x7f770c19b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.715+0000 7f7712e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770c106620 0x7f770c1a2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.715+0000 7f7712e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1105f0 con 0x7f770c106620 2024-05-10T11:13:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.715+0000 7f7712e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f770c110470 con 0x7f770c105c60 2024-05-10T11:13:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.715+0000 7f7712e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f770c110770 con 0x7f770c104a70 2024-05-10T11:13:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.715+0000 7f7712400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770c106620 0x7f770c1a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.715+0000 7f7711a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f770c104a70 0x7f770c19ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.715+0000 7f7712400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770c106620 0x7f770c1a2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42786/0 (socket says 172.21.15.27:42786) 2024-05-10T11:13:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.715+0000 7f7711a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f770c104a70 0x7f770c19ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:52364/0 (socket says 172.21.15.27:52364) 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.716+0000 7f7712400640 1 -- 172.21.15.27:0/1687726549 learned_addr learned my addr 172.21.15.27:0/1687726549 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.716+0000 7f7711000640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f770c105c60 0x7f770c19b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.716+0000 7f7711a00640 1 -- 172.21.15.27:0/1687726549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f770c105c60 msgr2=0x7f770c19b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.716+0000 7f7711a00640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f770c105c60 0x7f770c19b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.716+0000 7f7711a00640 1 -- 172.21.15.27:0/1687726549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770c106620 msgr2=0x7f770c1a2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.716+0000 7f7711a00640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770c106620 0x7f770c1a2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.716+0000 7f7711a00640 1 -- 172.21.15.27:0/1687726549 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770c1a28b0 con 0x7f770c104a70 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.716+0000 7f7712400640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770c106620 0x7f770c1a2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.716+0000 7f7711a00640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f770c104a70 0x7f770c19ac70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f770002a910 tx=0x7f7700003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.717+0000 7f770a000640 1 -- 172.21.15.27:0/1687726549 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770003f070 con 0x7f770c104a70 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.717+0000 7f7712e00640 1 -- 172.21.15.27:0/1687726549 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770c1a2b40 con 0x7f770c104a70 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.717+0000 7f770a000640 1 -- 172.21.15.27:0/1687726549 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7700031c50 con 0x7f770c104a70 2024-05-10T11:13:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.717+0000 7f770a000640 1 -- 172.21.15.27:0/1687726549 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700035030 con 0x7f770c104a70 2024-05-10T11:13:02.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.717+0000 7f7712e00640 1 -- 172.21.15.27:0/1687726549 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f770c1a3050 con 0x7f770c104a70 2024-05-10T11:13:02.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.718+0000 7f7712e00640 1 -- 172.21.15.27:0/1687726549 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f770c10be10 con 0x7f770c104a70 2024-05-10T11:13:02.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.721+0000 7f770a000640 1 -- 172.21.15.27:0/1687726549 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f7700042070 con 0x7f770c104a70 2024-05-10T11:13:02.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.722+0000 7f770a000640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76f40b2860 0x7f76f40b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:02.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.722+0000 7f770a000640 1 -- 172.21.15.27:0/1687726549 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f7700103650 con 0x7f770c104a70 2024-05-10T11:13:02.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.723+0000 7f7711000640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76f40b2860 0x7f76f40b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:02.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.723+0000 7f7711000640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76f40b2860 0x7f76f40b4d20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f76f80095f0 tx=0x7f76f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:02.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.725+0000 7f770a000640 1 -- 172.21.15.27:0/1687726549 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f77000ca8d0 con 0x7f770c104a70 2024-05-10T11:13:02.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:02.991+0000 7f7712e00640 1 -- 172.21.15.27:0/1687726549 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f770c02db50 con 0x7f770c104a70 2024-05-10T11:13:03.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.109+0000 7f770a000640 1 -- 172.21.15.27:0/1687726549 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f77000c6b60 con 0x7f770c104a70 2024-05-10T11:13:03.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.110+0000 7f770a000640 1 -- 172.21.15.27:0/1687726549 <== mon.1 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f77001045d0 con 0x7f770c104a70 2024-05-10T11:13:03.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.114+0000 7f7712e00640 1 -- 172.21.15.27:0/1687726549 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76f40b2860 msgr2=0x7f76f40b4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:03.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.114+0000 7f7712e00640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76f40b2860 0x7f76f40b4d20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f76f80095f0 tx=0x7f76f8005f90 comp rx=0 tx=0).stop 2024-05-10T11:13:03.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.114+0000 7f7712e00640 1 -- 172.21.15.27:0/1687726549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f770c104a70 msgr2=0x7f770c19ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:03.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.114+0000 7f7712e00640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f770c104a70 0x7f770c19ac70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f770002a910 tx=0x7f7700003c10 comp rx=0 tx=0).stop 2024-05-10T11:13:03.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.115+0000 7f7712e00640 1 -- 172.21.15.27:0/1687726549 shutdown_connections 2024-05-10T11:13:03.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.115+0000 7f7712e00640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f76f40b2860 0x7f76f40b4d20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:03.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.115+0000 7f7712e00640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770c106620 0x7f770c1a2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:03.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.115+0000 7f7712e00640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f770c105c60 0x7f770c19b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:03.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.115+0000 7f7712e00640 1 --2- 172.21.15.27:0/1687726549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f770c104a70 0x7f770c19ac70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:03.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.115+0000 7f7712e00640 1 -- 172.21.15.27:0/1687726549 >> 172.21.15.27:0/1687726549 conn(0x7f770c100020 msgr2=0x7f770c108fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:03.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.115+0000 7f7712e00640 1 -- 172.21.15.27:0/1687726549 shutdown_connections 2024-05-10T11:13:03.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:03.115+0000 7f7712e00640 1 -- 172.21.15.27:0/1687726549 wait complete. 2024-05-10T11:13:03.133 INFO:teuthology.run_tasks:Running task kclient... 2024-05-10T11:13:03.147 INFO:tasks.kclient:Mounting kernel clients... 2024-05-10T11:13:03.147 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-10T11:13:03.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:13:03.147 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-05-10T11:13:03.203 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-05-10T11:13:03.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:13:03.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-10T11:13:03.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:13:03.276 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-05-10T11:13:03.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:03 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1687726549' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-10T11:13:03.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:03 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-10T11:13:03.368 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:03 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: ignoring --setuser ceph since I am not root 2024-05-10T11:13:03.368 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:03 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: ignoring --setgroup ceph since I am not root 2024-05-10T11:13:03.377 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-05-10T11:13:03.378 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-10T11:13:03.378 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-10T11:13:03.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs ls 2024-05-10T11:13:03.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:03 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1687726549' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-10T11:13:03.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:03 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-10T11:13:03.425 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: ignoring --setuser ceph since I am not root 2024-05-10T11:13:03.425 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: ignoring --setgroup ceph since I am not root 2024-05-10T11:13:03.425 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:03 smithi076 ceph-mgr[35546]: -- 172.21.15.76:0/434439495 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5635e71385a0 con 0x5635e710d000 2024-05-10T11:13:03.425 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:03.281+0000 7f767bbe0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-10T11:13:03.460 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:03 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1687726549' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-10T11:13:03.460 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:03 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-10T11:13:03.461 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: ignoring --setuser ceph since I am not root 2024-05-10T11:13:03.461 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: ignoring --setgroup ceph since I am not root 2024-05-10T11:13:03.461 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:03 smithi189 ceph-mgr[36715]: -- 172.21.15.189:0/2237475037 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559a8f0f45a0 con 0x559a8f0c9000 2024-05-10T11:13:03.461 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:03.275+0000 7f060f8bb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-10T11:13:03.461 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:03.460+0000 7f060f8bb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-10T11:13:03.618 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:03 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:03.432+0000 7fa2c21ef200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-10T11:13:03.736 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:03.563+0000 7f060f8bb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-10T11:13:03.810 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:13:03.880 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:03.468+0000 7f767bbe0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-10T11:13:03.881 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:03.570+0000 7f767bbe0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-10T11:13:03.954 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:03 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:03.688+0000 7fa2c21ef200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-10T11:13:03.954 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:03 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:03.841+0000 7fa2c21ef200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-10T11:13:04.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:04 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-10T11:13:04.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:04 smithi076 ceph-mon[34152]: mgrmap e19: x(active, since 13m), standbys: z, y 2024-05-10T11:13:04.175 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:03.880+0000 7f767bbe0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-10T11:13:04.175 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:03.966+0000 7f767bbe0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-10T11:13:04.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:04 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-10T11:13:04.207 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:04 smithi027 ceph-mon[30054]: mgrmap e19: x(active, since 13m), standbys: z, y 2024-05-10T11:13:04.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:04 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-10T11:13:04.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:04 smithi189 ceph-mon[34842]: mgrmap e19: x(active, since 13m), standbys: z, y 2024-05-10T11:13:04.237 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:03.874+0000 7f060f8bb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-10T11:13:04.237 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:03.960+0000 7f060f8bb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-10T11:13:04.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:04 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:04.269+0000 7fa2c21ef200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-10T11:13:04.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:04 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:04.396+0000 7fa2c21ef200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-10T11:13:05.425 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:05.147+0000 7f767bbe0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-10T11:13:05.425 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:05.317+0000 7f767bbe0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-10T11:13:05.486 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:05 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:05.155+0000 7f060f8bb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-10T11:13:05.486 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:05 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:05.327+0000 7f060f8bb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-10T11:13:05.775 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:05.487+0000 7f767bbe0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-10T11:13:05.789 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:05 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:05.500+0000 7f060f8bb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-10T11:13:06.068 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:05.775+0000 7f767bbe0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-10T11:13:06.068 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:05.867+0000 7f767bbe0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-10T11:13:06.086 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:05 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:05.789+0000 7f060f8bb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-10T11:13:06.087 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:05 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:05.880+0000 7f060f8bb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-10T11:13:06.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.134+0000 7f5d30a00700 1 -- 172.21.15.27:0/4227300419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c104610 msgr2=0x7f5d2c1049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:06.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.134+0000 7f5d30a00700 1 --2- 172.21.15.27:0/4227300419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c104610 0x7f5d2c1049f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009990 tx=0x7f5d1c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:13:06.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.135+0000 7f5d30a00700 1 -- 172.21.15.27:0/4227300419 shutdown_connections 2024-05-10T11:13:06.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.135+0000 7f5d30a00700 1 --2- 172.21.15.27:0/4227300419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c104610 0x7f5d2c1049f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:06.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.135+0000 7f5d30a00700 1 --2- 172.21.15.27:0/4227300419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c10d9f0 0x7f5d2c10fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:06.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.135+0000 7f5d30a00700 1 --2- 172.21.15.27:0/4227300419 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d2c104f30 0x7f5d2c10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:06.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.135+0000 7f5d30a00700 1 -- 172.21.15.27:0/4227300419 >> 172.21.15.27:0/4227300419 conn(0x7f5d2c0fddc0 msgr2=0x7f5d2c1001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:06.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.135+0000 7f5d30a00700 1 -- 172.21.15.27:0/4227300419 shutdown_connections 2024-05-10T11:13:06.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.135+0000 7f5d30a00700 1 -- 172.21.15.27:0/4227300419 wait complete. 2024-05-10T11:13:06.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.136+0000 7f5d30a00700 1 Processor -- start 2024-05-10T11:13:06.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.136+0000 7f5d30a00700 1 -- start start 2024-05-10T11:13:06.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.136+0000 7f5d30a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d2c104610 0x7f5d2c19bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:06.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.136+0000 7f5d30a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c104f30 0x7f5d2c19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:06.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.136+0000 7f5d30a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c10d9f0 0x7f5d2c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:06.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.136+0000 7f5d30a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c113630 con 0x7f5d2c10d9f0 2024-05-10T11:13:06.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.136+0000 7f5d30a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c1134b0 con 0x7f5d2c104610 2024-05-10T11:13:06.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.136+0000 7f5d30a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c1137b0 con 0x7f5d2c104f30 2024-05-10T11:13:06.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.137+0000 7f5d29e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c104f30 0x7f5d2c19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:06.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.137+0000 7f5d2a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d2c104610 0x7f5d2c19bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:06.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.137+0000 7f5d29e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c104f30 0x7f5d2c19c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:45450/0 (socket says 172.21.15.27:45450) 2024-05-10T11:13:06.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.137+0000 7f5d29e00700 1 -- 172.21.15.27:0/1722357952 learned_addr learned my addr 172.21.15.27:0/1722357952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:06.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.137+0000 7f5d2b200700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c10d9f0 0x7f5d2c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:06.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.137+0000 7f5d29e00700 1 -- 172.21.15.27:0/1722357952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d2c104610 msgr2=0x7f5d2c19bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:06.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.137+0000 7f5d29e00700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d2c104610 0x7f5d2c19bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:06.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.137+0000 7f5d29e00700 1 -- 172.21.15.27:0/1722357952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c10d9f0 msgr2=0x7f5d2c1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:06.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.137+0000 7f5d29e00700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c10d9f0 0x7f5d2c1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:06.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.137+0000 7f5d29e00700 1 -- 172.21.15.27:0/1722357952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2c1a0d90 con 0x7f5d2c104f30 2024-05-10T11:13:06.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.138+0000 7f5d29e00700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c104f30 0x7f5d2c19c2c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5d1800e9f0 tx=0x7f5d1800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:06.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.138+0000 7f5d23400700 1 -- 172.21.15.27:0/1722357952 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d18004a30 con 0x7f5d2c104f30 2024-05-10T11:13:06.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.138+0000 7f5d23400700 1 -- 172.21.15.27:0/1722357952 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d1800cc70 con 0x7f5d2c104f30 2024-05-10T11:13:06.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.138+0000 7f5d23400700 1 -- 172.21.15.27:0/1722357952 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d18005530 con 0x7f5d2c104f30 2024-05-10T11:13:06.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.138+0000 7f5d30a00700 1 -- 172.21.15.27:0/1722357952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d2c1a1080 con 0x7f5d2c104f30 2024-05-10T11:13:06.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.138+0000 7f5d30a00700 1 -- 172.21.15.27:0/1722357952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d2c1a8900 con 0x7f5d2c104f30 2024-05-10T11:13:06.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.139+0000 7f5d30a00700 1 -- 172.21.15.27:0/1722357952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d2c06a190 con 0x7f5d2c104f30 2024-05-10T11:13:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.142+0000 7f5d23400700 1 -- 172.21.15.27:0/1722357952 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f5d18004bd0 con 0x7f5d2c104f30 2024-05-10T11:13:06.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.143+0000 7f5d23400700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5d100b2950 0x7f5d100b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:06.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.144+0000 7f5d2a800700 1 -- 172.21.15.27:0/1722357952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5d100b2950 msgr2=0x7f5d100b4e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1704535643 2024-05-10T11:13:06.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.144+0000 7f5d23400700 1 -- 172.21.15.27:0/1722357952 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f5d180e2830 con 0x7f5d2c104f30 2024-05-10T11:13:06.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.144+0000 7f5d2a800700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5d100b2950 0x7f5d100b4e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-10T11:13:06.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.147+0000 7f5d23400700 1 -- 172.21.15.27:0/1722357952 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5d1800ce20 con 0x7f5d2c104f30 2024-05-10T11:13:06.305 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:06 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:06.062+0000 7fa2c21ef200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-10T11:13:06.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.345+0000 7f5d2a800700 1 -- 172.21.15.27:0/1722357952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5d100b2950 msgr2=0x7f5d100b4e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1704535643 2024-05-10T11:13:06.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.345+0000 7f5d2a800700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5d100b2950 0x7f5d100b4e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-10T11:13:06.425 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:06 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:06.067+0000 7f767bbe0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-10T11:13:06.425 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:06 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:06.263+0000 7f767bbe0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-10T11:13:06.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.432+0000 7f5d30a00700 1 -- 172.21.15.27:0/1722357952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5d2c1a8b30 con 0x7f5d2c104f30 2024-05-10T11:13:06.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.433+0000 7f5d23400700 1 -- 172.21.15.27:0/1722357952 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v29) ==== 53+0+88 (secure 0 0 0) 0x7f5d180aa100 con 0x7f5d2c104f30 2024-05-10T11:13:06.434 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-10T11:13:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.436+0000 7f5d30a00700 1 -- 172.21.15.27:0/1722357952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5d100b2950 msgr2=0x7f5d100b4e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T11:13:06.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.436+0000 7f5d30a00700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5d100b2950 0x7f5d100b4e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:06.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.436+0000 7f5d30a00700 1 -- 172.21.15.27:0/1722357952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c104f30 msgr2=0x7f5d2c19c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:06.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.436+0000 7f5d30a00700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c104f30 0x7f5d2c19c2c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5d1800e9f0 tx=0x7f5d1800ed00 comp rx=0 tx=0).stop 2024-05-10T11:13:06.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.436+0000 7f5d30a00700 1 -- 172.21.15.27:0/1722357952 shutdown_connections 2024-05-10T11:13:06.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.436+0000 7f5d30a00700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f5d100b2950 0x7f5d100b4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:06.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.436+0000 7f5d30a00700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c10d9f0 0x7f5d2c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:06.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.436+0000 7f5d30a00700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d2c104610 0x7f5d2c19bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:06.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.436+0000 7f5d30a00700 1 --2- 172.21.15.27:0/1722357952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c104f30 0x7f5d2c19c2c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:06.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.436+0000 7f5d30a00700 1 -- 172.21.15.27:0/1722357952 >> 172.21.15.27:0/1722357952 conn(0x7f5d2c0fddc0 msgr2=0x7f5d2c1001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:06.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.436+0000 7f5d30a00700 1 -- 172.21.15.27:0/1722357952 shutdown_connections 2024-05-10T11:13:06.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:06.436+0000 7f5d30a00700 1 -- 172.21.15.27:0/1722357952 wait complete. 2024-05-10T11:13:06.486 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:06 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:06.086+0000 7f060f8bb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-10T11:13:06.486 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:06 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:06.281+0000 7f060f8bb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-10T11:13:06.618 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:06 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:06.303+0000 7fa2c21ef200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-10T11:13:06.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:06 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:06.537+0000 7fa2c21ef200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-10T11:13:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:06 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1722357952' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-10T11:13:06.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:06 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1722357952' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-10T11:13:06.860 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:06 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1722357952' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-10T11:13:07.078 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:06 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:06.945+0000 7fa2c21ef200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-10T11:13:07.175 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:06 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:06.860+0000 7f767bbe0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-10T11:13:07.175 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:07 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:07.081+0000 7f767bbe0200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-10T11:13:07.236 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:06 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:06.890+0000 7f060f8bb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-10T11:13:07.236 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:07 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:07.113+0000 7f060f8bb200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-10T11:13:07.286 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-10T11:13:07.286 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-10T11:13:07.286 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-05-10T11:13:07.286 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-10T11:13:07.286 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-10T11:13:07.286 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-10T11:13:07.287 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-10T11:13:07.287 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-10T11:13:07.287 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-10T11:13:07.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:13:07.287 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-05-10T11:13:07.338 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-10T11:13:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-10T11:13:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-05-10T11:13:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-05-10T11:13:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-05-10T11:13:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-05-10T11:13:07.338 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-10T11:13:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41705sec preferred_lft 41705sec 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-05-10T11:13:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-05-10T11:13:07.340 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-10T11:13:07.341 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-10T11:13:07.341 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-05-10T11:13:07.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-05-10T11:13:07.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-05-10T11:13:07.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-05-10T11:13:07.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-10T11:13:07.341 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-05-10T11:13:07.359 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:07 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:07.077+0000 7fa2c21ef200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-10T11:13:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-10T11:13:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:13:07.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:13:07.582 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-10T11:13:07.618 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:07 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:07.359+0000 7fa2c21ef200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-10T11:13:07.687 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-05-10T11:13:07.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:13:07.691 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-05-10T11:13:07.740 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-05-10T11:13:07.740 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-10T11:13:07.740 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-10T11:13:07.740 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-10T11:13:07.740 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-10T11:13:07.741 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-10T11:13:07.741 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-05-10T11:13:07.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-10T11:13:07.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-10T11:13:07.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-10T11:13:07.741 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-05-10T11:13:07.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-10T11:13:07.896 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:07 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:07.643+0000 7fa2c21ef200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-10T11:13:07.925 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:07 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:07.510+0000 7f767bbe0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-10T11:13:07.925 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:07 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:07.607+0000 7f767bbe0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-10T11:13:07.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:13:07.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:13:07.936 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-05-10T11:13:07.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:13:07.980 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-05-10T11:13:07.986 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:07 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:07.539+0000 7f060f8bb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-10T11:13:07.986 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:07 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:07.636+0000 7f060f8bb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-10T11:13:08.070 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-10T11:13:08.070 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-05-10T11:13:08.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-10T11:13:08.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-10T11:13:08.070 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-05-10T11:13:08.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-10T11:13:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:13:08.286 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-10T11:13:08.286 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-10T11:13:08.286 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-05-10T11:13:08.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-10T11:13:08.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-10T11:13:08.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-10T11:13:08.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-10T11:13:08.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-10T11:13:08.287 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-05-10T11:13:08.329 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:07 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:07.975+0000 7f767bbe0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-10T11:13:08.370 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:08 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:08.012+0000 7f060f8bb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-10T11:13:08.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-10T11:13:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:13:08.546 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-10T11:13:08.546 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-05-10T11:13:08.547 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-05-10T11:13:08.547 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-05-10T11:13:08.547 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-05-10T11:13:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-10T11:13:08.662 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:08 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:08.329+0000 7f767bbe0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-10T11:13:08.662 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:08 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:08.421+0000 7f767bbe0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-10T11:13:08.706 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:08 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:08.369+0000 7f060f8bb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-10T11:13:08.706 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:08 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:08.464+0000 7f060f8bb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-10T11:13:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:13:08.736 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-10T11:13:08.736 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-10T11:13:08.774 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-10T11:13:08.774 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-10T11:13:08.775 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-10T11:13:08.832 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:08 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:08.517+0000 7fa2c21ef200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-10T11:13:08.861 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-05-10T11:13:08.861 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-05-10T11:13:08.862 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/735a009e-987b-4d8e-9de7-031001ea9af4 2024-05-10T11:13:08.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:13:08.862 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/735a009e-987b-4d8e-9de7-031001ea9af4 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-05-10T11:13:08.925 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:08 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:08.661+0000 7f767bbe0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-10T11:13:08.925 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:08 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:08.794+0000 7f767bbe0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-10T11:13:08.925 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:08 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:08.888+0000 7f767bbe0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-10T11:13:08.986 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:08 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:08.706+0000 7f060f8bb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-10T11:13:08.986 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:08 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:08.840+0000 7f060f8bb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-10T11:13:08.987 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:08 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:08.936+0000 7f060f8bb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-10T11:13:09.021 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-05-10T11:13:09.021 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-05-10T11:13:09.021 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-05-10T11:13:09.118 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:08 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:08.831+0000 7fa2c21ef200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-10T11:13:09.675 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:09 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:09.212+0000 7f767bbe0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-10T11:13:09.675 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:09 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:09.303+0000 7f767bbe0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-10T11:13:09.736 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:09.269+0000 7f060f8bb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-10T11:13:09.736 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:09.361+0000 7f060f8bb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-10T11:13:09.771 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:09 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:09.448+0000 7fa2c21ef200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-10T11:13:09.771 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:09 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:09.589+0000 7fa2c21ef200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-10T11:13:09.784 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-05-10T11:13:09.785 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-05-10T11:13:09.785 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-05-10T11:13:09.785 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.76:6789,172.21.15.189:6789" 2024-05-10T11:13:09.785 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.76:6789,172.21.15.189:6789:/volumes/qa/sv_1/735a009e-987b-4d8e-9de7-031001ea9af4 2024-05-10T11:13:09.785 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=063c9e18-0ebc-11ef-bc98-c7b262605968" will pass to kernel 2024-05-10T11:13:09.787 INFO:teuthology.orchestra.run.smithi027.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-10T11:13:09.787 INFO:teuthology.orchestra.run.smithi027.stdout: You just mounted a file system that supports labels which does not 2024-05-10T11:13:09.787 INFO:teuthology.orchestra.run.smithi027.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-10T11:13:09.788 INFO:teuthology.orchestra.run.smithi027.stdout: applications will generate AVC messages and not be allowed access to 2024-05-10T11:13:09.788 INFO:teuthology.orchestra.run.smithi027.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-10T11:13:09.791 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-10T11:13:09.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:13:09.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-10T11:13:09.871 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-10T11:13:09.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-10T11:13:09.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:13:09.980 INFO:teuthology.orchestra.run.smithi027.stdout:client25210 2024-05-10T11:13:09.981 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T11:13:09.981 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-10T11:13:10.028 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-10T11:13:10.175 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:09 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:09.906+0000 7f767bbe0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-10T11:13:10.175 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:13:10 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y[35542]: 2024-05-10T11:13:10.121+0000 7f767bbe0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-10T11:13:10.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-10T11:13:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:13:10.198 INFO:teuthology.orchestra.run.smithi027.stdout:client25210 2024-05-10T11:13:10.199 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-10T11:13:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:10 smithi189 ceph-mon[34842]: Standby manager daemon y restarted 2024-05-10T11:13:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:10 smithi189 ceph-mon[34842]: Standby manager daemon y started 2024-05-10T11:13:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:10 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.76:0/557578445' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-10T11:13:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:10 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.76:0/557578445' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-10T11:13:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:10 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.76:0/557578445' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-10T11:13:10.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:10 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.76:0/557578445' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-10T11:13:10.237 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:09.964+0000 7f060f8bb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-10T11:13:10.237 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:13:10 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z[36711]: 2024-05-10T11:13:10.185+0000 7f060f8bb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-10T11:13:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-10T11:13:10.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:13:10.303 INFO:teuthology.orchestra.run.smithi027.stdout:client25210 2024-05-10T11:13:10.303 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968.client25210/status') 2024-05-10T11:13:10.368 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:10 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:10.087+0000 7fa2c21ef200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-10T11:13:10.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:10 smithi027 ceph-mon[30054]: Standby manager daemon y restarted 2024-05-10T11:13:10.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:10 smithi027 ceph-mon[30054]: Standby manager daemon y started 2024-05-10T11:13:10.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:10 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.76:0/557578445' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-10T11:13:10.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:10 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.76:0/557578445' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-10T11:13:10.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:10 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.76:0/557578445' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-10T11:13:10.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:10 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.76:0/557578445' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-10T11:13:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-10T11:13:10.427 INFO:teuthology.orchestra.run.smithi027.stdout:instance: client.25210 (3)172.21.15.27:0/1153028249 2024-05-10T11:13:10.428 INFO:teuthology.orchestra.run.smithi027.stdout:blocklisted: false 2024-05-10T11:13:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:0+1 records in 2024-05-10T11:13:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:0+1 records out 2024-05-10T11:13:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:71 bytes copied, 8.0731e-05 s, 879 kB/s 2024-05-10T11:13:10.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-10T11:13:10.444 INFO:teuthology.run_tasks:Running task exec... 2024-05-10T11:13:10.454 INFO:teuthology.task.exec:Executing custom commands... 2024-05-10T11:13:10.454 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-05-10T11:13:10.454 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-05-10T11:13:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.646+0000 7f805aa00640 1 -- 172.21.15.27:0/3000586182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8054101300 msgr2=0x7f8054101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.646+0000 7f805aa00640 1 --2- 172.21.15.27:0/3000586182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8054101300 0x7f8054101720 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8044009860 tx=0x7f804402a3d0 comp rx=0 tx=0).stop 2024-05-10T11:13:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.646+0000 7f805aa00640 1 -- 172.21.15.27:0/3000586182 shutdown_connections 2024-05-10T11:13:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.646+0000 7f805aa00640 1 --2- 172.21.15.27:0/3000586182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054102600 0x7f8054102a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.646+0000 7f805aa00640 1 --2- 172.21.15.27:0/3000586182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8054101c60 0x7f80541020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:10.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.646+0000 7f805aa00640 1 --2- 172.21.15.27:0/3000586182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8054101300 0x7f8054101720 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:10.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.646+0000 7f805aa00640 1 -- 172.21.15.27:0/3000586182 >> 172.21.15.27:0/3000586182 conn(0x7f80540fce80 msgr2=0x7f80540ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:10.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.647+0000 7f805aa00640 1 -- 172.21.15.27:0/3000586182 shutdown_connections 2024-05-10T11:13:10.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.647+0000 7f805aa00640 1 -- 172.21.15.27:0/3000586182 wait complete. 2024-05-10T11:13:10.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.648+0000 7f805aa00640 1 Processor -- start 2024-05-10T11:13:10.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.648+0000 7f805aa00640 1 -- start start 2024-05-10T11:13:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.648+0000 7f805aa00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8054101300 0x7f805419ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f805aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054101c60 0x7f805419b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f805aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8054102600 0x7f80541a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f805aa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8054110790 con 0x7f8054101c60 2024-05-10T11:13:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f805aa00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8054110610 con 0x7f8054101300 2024-05-10T11:13:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f805aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8054110910 con 0x7f8054102600 2024-05-10T11:13:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f8053e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8054101300 0x7f805419ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f804be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054101c60 0x7f805419b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f8053e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8054101300 0x7f805419ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41480/0 (socket says 172.21.15.27:41480) 2024-05-10T11:13:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f8053e00640 1 -- 172.21.15.27:0/1196703834 learned_addr learned my addr 172.21.15.27:0/1196703834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f8058a00640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8054102600 0x7f80541a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f804be00640 1 -- 172.21.15.27:0/1196703834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8054102600 msgr2=0x7f80541a23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f804be00640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8054102600 0x7f80541a23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f804be00640 1 -- 172.21.15.27:0/1196703834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8054101300 msgr2=0x7f805419ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f804be00640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8054101300 0x7f805419ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.649+0000 7f804be00640 1 -- 172.21.15.27:0/1196703834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80541a2a60 con 0x7f8054101c60 2024-05-10T11:13:10.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.650+0000 7f804be00640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054101c60 0x7f805419b380 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f803c00e9f0 tx=0x7f803c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:10.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.650+0000 7f8051600640 1 -- 172.21.15.27:0/1196703834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c00ccc0 con 0x7f8054101c60 2024-05-10T11:13:10.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.650+0000 7f8051600640 1 -- 172.21.15.27:0/1196703834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f803c005020 con 0x7f8054101c60 2024-05-10T11:13:10.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.650+0000 7f805aa00640 1 -- 172.21.15.27:0/1196703834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80541a2d50 con 0x7f8054101c60 2024-05-10T11:13:10.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.650+0000 7f8051600640 1 -- 172.21.15.27:0/1196703834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c004a00 con 0x7f8054101c60 2024-05-10T11:13:10.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.650+0000 7f805aa00640 1 -- 172.21.15.27:0/1196703834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80541a31f0 con 0x7f8054101c60 2024-05-10T11:13:10.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.651+0000 7f805aa00640 1 -- 172.21.15.27:0/1196703834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f805410c050 con 0x7f8054101c60 2024-05-10T11:13:10.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.654+0000 7f8051600640 1 -- 172.21.15.27:0/1196703834 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 20) ==== 149669+0+0 (secure 0 0 0) 0x7f803c004ba0 con 0x7f8054101c60 2024-05-10T11:13:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.656+0000 7f8051600640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f80380b28f0 0x7f80380b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.656+0000 7f8053e00640 1 -- 172.21.15.27:0/1196703834 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f80380b28f0 msgr2=0x7f80380b4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1704535643 2024-05-10T11:13:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.656+0000 7f8051600640 1 -- 172.21.15.27:0/1196703834 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f803c0e2670 con 0x7f8054101c60 2024-05-10T11:13:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.656+0000 7f8053e00640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f80380b28f0 0x7f80380b4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-10T11:13:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.663+0000 7f8051600640 1 -- 172.21.15.27:0/1196703834 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f803c0a9fd0 con 0x7f8054101c60 2024-05-10T11:13:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:10 smithi076 ceph-mon[34152]: Standby manager daemon y restarted 2024-05-10T11:13:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:10 smithi076 ceph-mon[34152]: Standby manager daemon y started 2024-05-10T11:13:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:10 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.76:0/557578445' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-10T11:13:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:10 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.76:0/557578445' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-10T11:13:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:10 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.76:0/557578445' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-10T11:13:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:10 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.76:0/557578445' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-10T11:13:10.740 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:10 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:10.599+0000 7fa2c21ef200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-10T11:13:10.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.857+0000 7f8053e00640 1 -- 172.21.15.27:0/1196703834 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f80380b28f0 msgr2=0x7f80380b4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1704535643 2024-05-10T11:13:10.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.857+0000 7f8053e00640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f80380b28f0 0x7f80380b4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-10T11:13:10.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:10.914+0000 7f805aa00640 1 -- 172.21.15.27:0/1196703834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f805402db50 con 0x7f8054101c60 2024-05-10T11:13:11.084 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:10 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:10.739+0000 7fa2c21ef200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-10T11:13:11.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.198+0000 7f8051600640 1 -- 172.21.15.27:0/1196703834 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 21) ==== 149669+0+0 (secure 0 0 0) 0x7f803c0a5b60 con 0x7f8054101c60 2024-05-10T11:13:11.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.211+0000 7f8051600640 1 -- 172.21.15.27:0/1196703834 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v30) ==== 118+0+0 (secure 0 0 0) 0x7f803c0dcd80 con 0x7f8054101c60 2024-05-10T11:13:11.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.215+0000 7f805aa00640 1 -- 172.21.15.27:0/1196703834 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f80380b28f0 msgr2=0x7f80380b4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T11:13:11.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.215+0000 7f805aa00640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f80380b28f0 0x7f80380b4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:11.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.215+0000 7f805aa00640 1 -- 172.21.15.27:0/1196703834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054101c60 msgr2=0x7f805419b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:11.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.215+0000 7f805aa00640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054101c60 0x7f805419b380 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f803c00e9f0 tx=0x7f803c00eeb0 comp rx=0 tx=0).stop 2024-05-10T11:13:11.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.215+0000 7f805aa00640 1 -- 172.21.15.27:0/1196703834 shutdown_connections 2024-05-10T11:13:11.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.215+0000 7f805aa00640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f80380b28f0 0x7f80380b4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:11.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.215+0000 7f805aa00640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8054102600 0x7f80541a23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:11.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.215+0000 7f805aa00640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054101c60 0x7f805419b380 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:11.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.215+0000 7f805aa00640 1 --2- 172.21.15.27:0/1196703834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8054101300 0x7f805419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:11.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.215+0000 7f805aa00640 1 -- 172.21.15.27:0/1196703834 >> 172.21.15.27:0/1196703834 conn(0x7f80540fce80 msgr2=0x7f80540fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:11.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.215+0000 7f805aa00640 1 -- 172.21.15.27:0/1196703834 shutdown_connections 2024-05-10T11:13:11.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.215+0000 7f805aa00640 1 -- 172.21.15.27:0/1196703834 wait complete. 2024-05-10T11:13:11.234 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-05-10T11:13:11.368 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:11.082+0000 7fa2c21ef200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-10T11:13:11.369 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:11.284+0000 7fa2c21ef200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-10T11:13:11.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:11 smithi027 ceph-mon[30054]: mgrmap e20: x(active, since 13m), standbys: z, y 2024-05-10T11:13:11.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:11 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.189:0/3827760306' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-10T11:13:11.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:11 smithi027 ceph-mon[30054]: Standby manager daemon z restarted 2024-05-10T11:13:11.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:11 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.189:0/3827760306' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-10T11:13:11.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:11 smithi027 ceph-mon[30054]: Standby manager daemon z started 2024-05-10T11:13:11.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:11 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.189:0/3827760306' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-10T11:13:11.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:11 smithi027 ceph-mon[30054]: from='mgr.? 172.21.15.189:0/3827760306' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-10T11:13:11.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:11 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1196703834' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-10T11:13:11.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.424+0000 7f946d600640 1 -- 172.21.15.27:0/3696912882 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94600318a0 con 0x7f9470102a10 2024-05-10T11:13:11.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.425+0000 7f9475e00640 1 -- 172.21.15.27:0/3696912882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9470102a10 msgr2=0x7f9470102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:11.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.425+0000 7f9475e00640 1 --2- 172.21.15.27:0/3696912882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9470102a10 0x7f9470102e30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9460009900 tx=0x7f946002a400 comp rx=0 tx=0).stop 2024-05-10T11:13:11.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.425+0000 7f9475e00640 1 -- 172.21.15.27:0/3696912882 shutdown_connections 2024-05-10T11:13:11.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.425+0000 7f9475e00640 1 --2- 172.21.15.27:0/3696912882 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94701045c0 0x7f947010ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:11.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.425+0000 7f9475e00640 1 --2- 172.21.15.27:0/3696912882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9470103c00 0x7f9470104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:11.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.425+0000 7f9475e00640 1 --2- 172.21.15.27:0/3696912882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9470102a10 0x7f9470102e30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:11.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.425+0000 7f9475e00640 1 -- 172.21.15.27:0/3696912882 >> 172.21.15.27:0/3696912882 conn(0x7f94700fdf80 msgr2=0x7f94701003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:11.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.425+0000 7f9475e00640 1 -- 172.21.15.27:0/3696912882 shutdown_connections 2024-05-10T11:13:11.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.425+0000 7f9475e00640 1 -- 172.21.15.27:0/3696912882 wait complete. 2024-05-10T11:13:11.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.426+0000 7f9475e00640 1 Processor -- start 2024-05-10T11:13:11.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.426+0000 7f9475e00640 1 -- start start 2024-05-10T11:13:11.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.426+0000 7f9475e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9470102a10 0x7f947019add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:11.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f9475e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9470103c00 0x7f947019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f9475e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94701045c0 0x7f94701a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f9475e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9470075770 con 0x7f9470102a10 2024-05-10T11:13:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f9475e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f94700755f0 con 0x7f9470103c00 2024-05-10T11:13:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f9475e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f94700758f0 con 0x7f94701045c0 2024-05-10T11:13:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f946ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9470102a10 0x7f947019add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f946e000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9470103c00 0x7f947019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f946ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9470102a10 0x7f947019add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33034/0 (socket says 172.21.15.27:33034) 2024-05-10T11:13:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f946ea00640 1 -- 172.21.15.27:0/2236264060 learned_addr learned my addr 172.21.15.27:0/2236264060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f946f400640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94701045c0 0x7f94701a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f946e000640 1 -- 172.21.15.27:0/2236264060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94701045c0 msgr2=0x7f94701a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f946e000640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94701045c0 0x7f94701a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:11.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f946e000640 1 -- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9470102a10 msgr2=0x7f947019add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:11.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f946e000640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9470102a10 0x7f947019add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:11.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f946e000640 1 -- 172.21.15.27:0/2236264060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94701a29f0 con 0x7f9470103c00 2024-05-10T11:13:11.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.427+0000 7f946ea00640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9470102a10 0x7f947019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:11.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.428+0000 7f946e000640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9470103c00 0x7f947019b310 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f945c00e9f0 tx=0x7f945c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:11.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.428+0000 7f9474a00640 1 -- 172.21.15.27:0/2236264060 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c00ccc0 con 0x7f9470103c00 2024-05-10T11:13:11.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.428+0000 7f9475e00640 1 -- 172.21.15.27:0/2236264060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94701a2ce0 con 0x7f9470103c00 2024-05-10T11:13:11.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.428+0000 7f9474a00640 1 -- 172.21.15.27:0/2236264060 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f945c005020 con 0x7f9470103c00 2024-05-10T11:13:11.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.428+0000 7f9474a00640 1 -- 172.21.15.27:0/2236264060 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c00ce60 con 0x7f9470103c00 2024-05-10T11:13:11.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.428+0000 7f9475e00640 1 -- 172.21.15.27:0/2236264060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94701a3180 con 0x7f9470103c00 2024-05-10T11:13:11.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.429+0000 7f9475e00640 1 -- 172.21.15.27:0/2236264060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94701a3410 con 0x7f9470103c00 2024-05-10T11:13:11.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.431+0000 7f9474a00640 1 -- 172.21.15.27:0/2236264060 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 21) ==== 149669+0+0 (secure 0 0 0) 0x7f945c0051d0 con 0x7f9470103c00 2024-05-10T11:13:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.433+0000 7f9474a00640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f94540b28f0 0x7f94540b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.433+0000 7f946ea00640 1 -- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f94540b28f0 msgr2=0x7f94540b4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1704535643 2024-05-10T11:13:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.433+0000 7f9474a00640 1 -- 172.21.15.27:0/2236264060 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9245+0+0 (secure 0 0 0) 0x7f945c0e25f0 con 0x7f9470103c00 2024-05-10T11:13:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.433+0000 7f946ea00640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f94540b28f0 0x7f94540b4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-10T11:13:11.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.437+0000 7f9474a00640 1 -- 172.21.15.27:0/2236264060 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f945c0a9ec0 con 0x7f9470103c00 2024-05-10T11:13:11.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:11 smithi189 ceph-mon[34842]: mgrmap e20: x(active, since 13m), standbys: z, y 2024-05-10T11:13:11.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:11 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.189:0/3827760306' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-10T11:13:11.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:11 smithi189 ceph-mon[34842]: Standby manager daemon z restarted 2024-05-10T11:13:11.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:11 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.189:0/3827760306' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-10T11:13:11.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:11 smithi189 ceph-mon[34842]: Standby manager daemon z started 2024-05-10T11:13:11.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:11 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.189:0/3827760306' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-10T11:13:11.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:11 smithi189 ceph-mon[34842]: from='mgr.? 172.21.15.189:0/3827760306' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-10T11:13:11.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:11 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1196703834' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-10T11:13:11.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.634+0000 7f946ea00640 1 -- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f94540b28f0 msgr2=0x7f94540b4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1704535643 2024-05-10T11:13:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.634+0000 7f946ea00640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f94540b28f0 0x7f94540b4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-10T11:13:11.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:11.674+0000 7f9475e00640 1 -- 172.21.15.27:0/2236264060 --> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f94700ff180 con 0x7f94540b28f0 2024-05-10T11:13:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:11 smithi076 ceph-mon[34152]: mgrmap e20: x(active, since 13m), standbys: z, y 2024-05-10T11:13:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:11 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.189:0/3827760306' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-10T11:13:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:11 smithi076 ceph-mon[34152]: Standby manager daemon z restarted 2024-05-10T11:13:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:11 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.189:0/3827760306' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-10T11:13:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:11 smithi076 ceph-mon[34152]: Standby manager daemon z started 2024-05-10T11:13:11.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:11 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.189:0/3827760306' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-10T11:13:11.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:11 smithi076 ceph-mon[34152]: from='mgr.? 172.21.15.189:0/3827760306' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-10T11:13:11.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:11 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1196703834' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-10T11:13:11.855 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:11.425+0000 7fa2c21ef200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-10T11:13:12.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:12.035+0000 7f946ea00640 1 -- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f94540b28f0 msgr2=0x7f94540b4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1704535643 2024-05-10T11:13:12.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:12.035+0000 7f946ea00640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f94540b28f0 0x7f94540b4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-10T11:13:12.119 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:11.853+0000 7fa2c21ef200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-10T11:13:12.119 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:11.987+0000 7fa2c21ef200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-10T11:13:12.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:12 smithi189 ceph-mon[34842]: mgrmap e21: x(active, since 13m), standbys: z, y 2024-05-10T11:13:12.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:12 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1196703834' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-10T11:13:12.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:12 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:13:12.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:12 smithi027 ceph-mon[30054]: mgrmap e21: x(active, since 13m), standbys: z, y 2024-05-10T11:13:12.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:12 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1196703834' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-10T11:13:12.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:12 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:13:12.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:12 smithi076 ceph-mon[34152]: mgrmap e21: x(active, since 13m), standbys: z, y 2024-05-10T11:13:12.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:12 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1196703834' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-10T11:13:12.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:12 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:13:12.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:12.836+0000 7f946ea00640 1 -- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f94540b28f0 msgr2=0x7f94540b4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1704535643 2024-05-10T11:13:12.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:12.836+0000 7f946ea00640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f94540b28f0 0x7f94540b4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-10T11:13:13.113 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:12 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:12.798+0000 7fa2c21ef200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-10T11:13:13.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:13.136+0000 7f9474a00640 1 -- 172.21.15.27:0/2236264060 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 22) ==== 149337+0+0 (secure 0 0 0) 0x7f945c0a5a50 con 0x7f9470103c00 2024-05-10T11:13:13.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:13.136+0000 7f9474a00640 1 -- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f94540b28f0 msgr2=0x7f94540b4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T11:13:13.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:13.136+0000 7f9474a00640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1704535643,v1:172.21.15.27:6801/1704535643] conn(0x7f94540b28f0 0x7f94540b4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:13.370 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:13.112+0000 7fa2c21ef200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-10T11:13:13.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: Active manager daemon x restarted 2024-05-10T11:13:13.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: Activating manager daemon x 2024-05-10T11:13:13.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: osdmap e91: 12 total, 12 up, 12 in 2024-05-10T11:13:13.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: mgrmap e22: x(active, starting, since 0.023862s), standbys: z, y 2024-05-10T11:13:13.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:13:13.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:13:13.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:13:13.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-10T11:13:13.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-10T11:13:13.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-10T11:13:13.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:13:13.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:13:13.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:13:13.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:13:13.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:13:13.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:13:13.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:13:13.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:13:13.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-10T11:13:13.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-10T11:13:13.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-10T11:13:13.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:13 smithi027 ceph-mon[30054]: Manager daemon x is now available 2024-05-10T11:13:13.461 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: Active manager daemon x restarted 2024-05-10T11:13:13.461 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: Activating manager daemon x 2024-05-10T11:13:13.461 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: osdmap e91: 12 total, 12 up, 12 in 2024-05-10T11:13:13.461 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: mgrmap e22: x(active, starting, since 0.023862s), standbys: z, y 2024-05-10T11:13:13.461 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:13:13.462 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:13:13.462 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:13:13.462 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-10T11:13:13.462 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-10T11:13:13.462 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-10T11:13:13.462 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-10T11:13:13.462 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-10T11:13:13.462 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-10T11:13:13.462 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:13:13.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:13:13.464 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:13:13.464 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:13:13.464 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:13:13.464 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:13:13.464 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:13:13.464 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-10T11:13:13.464 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-10T11:13:13.464 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-10T11:13:13.464 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:13 smithi189 ceph-mon[34842]: Manager daemon x is now available 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: Active manager daemon x restarted 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: Activating manager daemon x 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: osdmap e91: 12 total, 12 up, 12 in 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: mgrmap e22: x(active, starting, since 0.023862s), standbys: z, y 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-10T11:13:13.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-10T11:13:13.466 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-10T11:13:13.467 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-10T11:13:13.467 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-10T11:13:13.467 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-10T11:13:13.467 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-10T11:13:13.467 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-10T11:13:13.467 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-10T11:13:13.467 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-10T11:13:13.467 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-10T11:13:13.467 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-10T11:13:13.467 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:13 smithi076 ceph-mon[34152]: Manager daemon x is now available 2024-05-10T11:13:13.681 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:13.460+0000 7fa213600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:13.682 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:13.460+0000 7fa213600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:13.682 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:13.460+0000 7fa213600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:13.682 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:13.460+0000 7fa213600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:13.682 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:13.460+0000 7fa213600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:13.682 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:13.460+0000 7fa215e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:13.682 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:13.460+0000 7fa215e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:13.682 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:13.460+0000 7fa215e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:13.682 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:13.460+0000 7fa215e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:13.682 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:13.460+0000 7fa215e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.151+0000 7f9474a00640 1 -- 172.21.15.27:0/2236264060 <== mon.2 v2:172.21.15.76:3300/0 8 ==== mgrmap(e 23) ==== 149464+0+0 (secure 0 0 0) 0x7f945c0dfd20 con 0x7f9470103c00 2024-05-10T11:13:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.151+0000 7f9474a00640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9454074ae0 0x7f9454074ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.152+0000 7f9474a00640 1 -- 172.21.15.27:0/2236264060 --> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f945c0aeae0 con 0x7f9454074ae0 2024-05-10T11:13:14.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.165+0000 7f946ea00640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9454074ae0 0x7f9454074ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:14.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.167+0000 7f946ea00640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9454074ae0 0x7f9454074ec0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f94600096c0 tx=0x7f94600314a0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.191+0000 7f9474a00640 1 -- 172.21.15.27:0/2236264060 <== mgr.25220 v2:172.21.15.27:6800/1167199542 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f945c0aeae0 con 0x7f9454074ae0 2024-05-10T11:13:14.192 INFO:teuthology.orchestra.run.smithi027.stdout:{} 2024-05-10T11:13:14.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.194+0000 7f9475e00640 1 -- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9454074ae0 msgr2=0x7f9454074ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:14.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.195+0000 7f9475e00640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9454074ae0 0x7f9454074ec0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f94600096c0 tx=0x7f94600314a0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.195+0000 7f9475e00640 1 -- 172.21.15.27:0/2236264060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9470103c00 msgr2=0x7f947019b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:14.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.195+0000 7f9475e00640 1 --2- 172.21.15.27:0/2236264060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9470103c00 0x7f947019b310 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f945c00e9f0 tx=0x7f945c00edf0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.195+0000 7f946f400640 1 -- 172.21.15.27:0/2236264060 reap_dead start 2024-05-10T11:13:14.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.195+0000 7f9475e00640 1 -- 172.21.15.27:0/2236264060 shutdown_connections 2024-05-10T11:13:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.195+0000 7f9475e00640 1 -- 172.21.15.27:0/2236264060 >> 172.21.15.27:0/2236264060 conn(0x7f94700fdf80 msgr2=0x7f94700fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.195+0000 7f9475e00640 1 -- 172.21.15.27:0/2236264060 shutdown_connections 2024-05-10T11:13:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.195+0000 7f9475e00640 1 -- 172.21.15.27:0/2236264060 wait complete. 2024-05-10T11:13:14.220 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-10T11:13:14.231 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-10T11:13:14.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-10T11:13:14.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.421+0000 7f325ee00640 1 -- 172.21.15.27:0/3912464102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3258101a40 msgr2=0x7f3258101ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:14.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.421+0000 7f325ee00640 1 --2- 172.21.15.27:0/3912464102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3258101a40 0x7f3258101ea0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3244009860 tx=0x7f324402a340 comp rx=0 tx=0).stop 2024-05-10T11:13:14.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.421+0000 7f325ee00640 1 -- 172.21.15.27:0/3912464102 shutdown_connections 2024-05-10T11:13:14.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.421+0000 7f325ee00640 1 --2- 172.21.15.27:0/3912464102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32581023e0 0x7f3258102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.421+0000 7f325ee00640 1 --2- 172.21.15.27:0/3912464102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3258101a40 0x7f3258101ea0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.421+0000 7f325ee00640 1 --2- 172.21.15.27:0/3912464102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f32581010e0 0x7f3258101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.421+0000 7f325ee00640 1 -- 172.21.15.27:0/3912464102 >> 172.21.15.27:0/3912464102 conn(0x7f32580fcc80 msgr2=0x7f32580ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:14.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.421+0000 7f325ee00640 1 -- 172.21.15.27:0/3912464102 shutdown_connections 2024-05-10T11:13:14.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.421+0000 7f325ee00640 1 -- 172.21.15.27:0/3912464102 wait complete. 2024-05-10T11:13:14.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.423+0000 7f325ee00640 1 Processor -- start 2024-05-10T11:13:14.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.423+0000 7f325ee00640 1 -- start start 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.423+0000 7f325ee00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f32581010e0 0x7f325819acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.423+0000 7f325ee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3258101a40 0x7f325819b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.423+0000 7f325ee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32581023e0 0x7f32581a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.423+0000 7f325ee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f32581104f0 con 0x7f32581023e0 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.423+0000 7f325ee00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3258110370 con 0x7f32581010e0 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.423+0000 7f325ee00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3258110670 con 0x7f3258101a40 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.423+0000 7f325d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3258101a40 0x7f325819b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.424+0000 7f325da00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f32581010e0 0x7f325819acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.424+0000 7f325e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32581023e0 0x7f32581a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.424+0000 7f325d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3258101a40 0x7f325819b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42196/0 (socket says 172.21.15.27:42196) 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.424+0000 7f325d000640 1 -- 172.21.15.27:0/2373645672 learned_addr learned my addr 172.21.15.27:0/2373645672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.424+0000 7f325da00640 1 -- 172.21.15.27:0/2373645672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3258101a40 msgr2=0x7f325819b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.424+0000 7f325da00640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3258101a40 0x7f325819b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.424+0000 7f325da00640 1 -- 172.21.15.27:0/2373645672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32581023e0 msgr2=0x7f32581a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.424+0000 7f325da00640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32581023e0 0x7f32581a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.424+0000 7f325da00640 1 -- 172.21.15.27:0/2373645672 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32581a2900 con 0x7f32581010e0 2024-05-10T11:13:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.424+0000 7f325d000640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3258101a40 0x7f325819b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.424+0000 7f325da00640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f32581010e0 0x7f325819acf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f324c009fd0 tx=0x7f324c00ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.425+0000 7f3256000640 1 -- 172.21.15.27:0/2373645672 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c00ccc0 con 0x7f32581010e0 2024-05-10T11:13:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.425+0000 7f325ee00640 1 -- 172.21.15.27:0/2373645672 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32581a2bf0 con 0x7f32581010e0 2024-05-10T11:13:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.425+0000 7f3256000640 1 -- 172.21.15.27:0/2373645672 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f324c0044d0 con 0x7f32581010e0 2024-05-10T11:13:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.425+0000 7f3256000640 1 -- 172.21.15.27:0/2373645672 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c01c070 con 0x7f32581010e0 2024-05-10T11:13:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.425+0000 7f325ee00640 1 -- 172.21.15.27:0/2373645672 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32581a3140 con 0x7f32581010e0 2024-05-10T11:13:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.426+0000 7f325ee00640 1 -- 172.21.15.27:0/2373645672 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f325810be30 con 0x7f32581010e0 2024-05-10T11:13:14.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.428+0000 7f3256000640 1 -- 172.21.15.27:0/2373645672 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 23) ==== 149464+0+0 (secure 0 0 0) 0x7f324c0047c0 con 0x7f32581010e0 2024-05-10T11:13:14.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.430+0000 7f3256000640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f32400b25e0 0x7f32400b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:14.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.430+0000 7f3256000640 1 -- 172.21.15.27:0/2373645672 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f324c0e2300 con 0x7f32581010e0 2024-05-10T11:13:14.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.430+0000 7f325d000640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f32400b25e0 0x7f32400b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:14.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.431+0000 7f325d000640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f32400b25e0 0x7f32400b4aa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f32440095f0 tx=0x7f3244030470 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:14.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.434+0000 7f3256000640 1 -- 172.21.15.27:0/2373645672 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f324c0a9de0 con 0x7f32581010e0 2024-05-10T11:13:14.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:14 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:14.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:14 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-10T11:13:14.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:14 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-10T11:13:14.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:14 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-10T11:13:14.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:14 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-10T11:13:14.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:14 smithi189 ceph-mon[34842]: mgrmap e23: x(active, since 1.03924s), standbys: z, y 2024-05-10T11:13:14.519 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:14 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:14.520 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:14 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-10T11:13:14.520 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:14 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-10T11:13:14.520 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:14 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-10T11:13:14.520 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:14 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-10T11:13:14.520 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:14 smithi076 ceph-mon[34152]: mgrmap e23: x(active, since 1.03924s), standbys: z, y 2024-05-10T11:13:14.618 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:14.179+0000 7fa20fa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:14.618 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:14.179+0000 7fa20fa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:14.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:14.179+0000 7fa20fa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:14.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:14.179+0000 7fa20fa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:14.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:13:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:13:14.179+0000 7fa20fa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:13:14.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:14 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:14.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:14 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-10T11:13:14.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:14 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-10T11:13:14.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:14 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-10T11:13:14.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:14 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-10T11:13:14.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:14 smithi027 ceph-mon[30054]: mgrmap e23: x(active, since 1.03924s), standbys: z, y 2024-05-10T11:13:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.680+0000 7f325ee00640 1 -- 172.21.15.27:0/2373645672 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32580635b0 con 0x7f32581010e0 2024-05-10T11:13:14.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.682+0000 7f3256000640 1 -- 172.21.15.27:0/2373645672 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25213 (secure 0 0 0) 0x7f324c0a96e0 con 0x7f32581010e0 2024-05-10T11:13:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:13:14.683 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":91,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:13:13.118756+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6},{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1021663132":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6801/1704535643":"2024-05-11T11:13:13.118713+0000","172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/1450791252":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1410017298":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/1704535643":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/2991615845":"2024-05-11T11:13:13.118713+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:13:14.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.685+0000 7f325ee00640 1 -- 172.21.15.27:0/2373645672 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f32400b25e0 msgr2=0x7f32400b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.685+0000 7f325ee00640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f32400b25e0 0x7f32400b4aa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f32440095f0 tx=0x7f3244030470 comp rx=0 tx=0).stop 2024-05-10T11:13:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.685+0000 7f325ee00640 1 -- 172.21.15.27:0/2373645672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f32581010e0 msgr2=0x7f325819acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.685+0000 7f325ee00640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f32581010e0 0x7f325819acf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f324c009fd0 tx=0x7f324c00ec20 comp rx=0 tx=0).stop 2024-05-10T11:13:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.685+0000 7f325ee00640 1 -- 172.21.15.27:0/2373645672 shutdown_connections 2024-05-10T11:13:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.685+0000 7f325ee00640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f32400b25e0 0x7f32400b4aa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.685+0000 7f325ee00640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32581023e0 0x7f32581a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.685+0000 7f325ee00640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3258101a40 0x7f325819b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.685+0000 7f325ee00640 1 --2- 172.21.15.27:0/2373645672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f32581010e0 0x7f325819acf0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.685+0000 7f325ee00640 1 -- 172.21.15.27:0/2373645672 >> 172.21.15.27:0/2373645672 conn(0x7f32580fcc80 msgr2=0x7f3258100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.685+0000 7f325ee00640 1 -- 172.21.15.27:0/2373645672 shutdown_connections 2024-05-10T11:13:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.685+0000 7f325ee00640 1 -- 172.21.15.27:0/2373645672 wait complete. 2024-05-10T11:13:14.703 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-10T11:02:06.781475+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-10T11:07:40.667104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-10T11:07:45.158839+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-10T11:07:58.869099+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-10T11:13:14.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-10T11:13:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.892+0000 7ff7a2000640 1 -- 172.21.15.27:0/1742091478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff79c103350 msgr2=0x7ff79c105740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.892+0000 7ff7a2000640 1 --2- 172.21.15.27:0/1742091478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff79c103350 0x7ff79c105740 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff790009860 tx=0x7ff79002a340 comp rx=0 tx=0).stop 2024-05-10T11:13:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.892+0000 7ff7a2000640 1 -- 172.21.15.27:0/1742091478 shutdown_connections 2024-05-10T11:13:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.892+0000 7ff7a2000640 1 --2- 172.21.15.27:0/1742091478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79c1085b0 0x7ff79c10aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.892+0000 7ff7a2000640 1 --2- 172.21.15.27:0/1742091478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79c105c80 0x7ff79c108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.892+0000 7ff7a2000640 1 --2- 172.21.15.27:0/1742091478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff79c103350 0x7ff79c105740 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.892+0000 7ff7a2000640 1 -- 172.21.15.27:0/1742091478 >> 172.21.15.27:0/1742091478 conn(0x7ff79c0fcd60 msgr2=0x7ff79c0ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:14.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.892+0000 7ff7a2000640 1 -- 172.21.15.27:0/1742091478 shutdown_connections 2024-05-10T11:13:14.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.892+0000 7ff7a2000640 1 -- 172.21.15.27:0/1742091478 wait complete. 2024-05-10T11:13:14.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.893+0000 7ff7a2000640 1 Processor -- start 2024-05-10T11:13:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.893+0000 7ff7a2000640 1 -- start start 2024-05-10T11:13:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a2000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff79c103350 0x7ff79c19adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79c105c80 0x7ff79c19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79c1085b0 0x7ff79c1a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a2000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c10e400 con 0x7ff79c1085b0 2024-05-10T11:13:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a2000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c10e280 con 0x7ff79c103350 2024-05-10T11:13:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a2000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c10e580 con 0x7ff79c105c80 2024-05-10T11:13:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79c1085b0 0x7ff79c1a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a0c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff79c103350 0x7ff79c19adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a0c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff79c103350 0x7ff79c19adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37970/0 (socket says 172.21.15.27:37970) 2024-05-10T11:13:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a0c00640 1 -- 172.21.15.27:0/511475600 learned_addr learned my addr 172.21.15.27:0/511475600 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff79be00640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79c105c80 0x7ff79c19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a0c00640 1 -- 172.21.15.27:0/511475600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79c105c80 msgr2=0x7ff79c19b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a0c00640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79c105c80 0x7ff79c19b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a0c00640 1 -- 172.21.15.27:0/511475600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79c1085b0 msgr2=0x7ff79c1a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.894+0000 7ff7a0c00640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79c1085b0 0x7ff79c1a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.895+0000 7ff7a0c00640 1 -- 172.21.15.27:0/511475600 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79c1a29c0 con 0x7ff79c103350 2024-05-10T11:13:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.895+0000 7ff7a1600640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79c1085b0 0x7ff79c1a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.895+0000 7ff7a0c00640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff79c103350 0x7ff79c19adb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff7900057c0 tx=0x7ff790032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.895+0000 7ff799600640 1 -- 172.21.15.27:0/511475600 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790037040 con 0x7ff79c103350 2024-05-10T11:13:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.895+0000 7ff7a2000640 1 -- 172.21.15.27:0/511475600 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff79c1a2c50 con 0x7ff79c103350 2024-05-10T11:13:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.895+0000 7ff799600640 1 -- 172.21.15.27:0/511475600 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff790030bd0 con 0x7ff79c103350 2024-05-10T11:13:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.895+0000 7ff799600640 1 -- 172.21.15.27:0/511475600 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790005280 con 0x7ff79c103350 2024-05-10T11:13:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.895+0000 7ff7a2000640 1 -- 172.21.15.27:0/511475600 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff79c1a3130 con 0x7ff79c103350 2024-05-10T11:13:14.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.896+0000 7ff7a2000640 1 -- 172.21.15.27:0/511475600 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff79c10a100 con 0x7ff79c103350 2024-05-10T11:13:14.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.899+0000 7ff799600640 1 -- 172.21.15.27:0/511475600 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 23) ==== 149464+0+0 (secure 0 0 0) 0x7ff790043020 con 0x7ff79c103350 2024-05-10T11:13:14.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.900+0000 7ff799600640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff7840b2660 0x7ff7840b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:14.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.900+0000 7ff799600640 1 -- 172.21.15.27:0/511475600 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7ff790101970 con 0x7ff79c103350 2024-05-10T11:13:14.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.900+0000 7ff79be00640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff7840b2660 0x7ff7840b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:14.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.901+0000 7ff79be00640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff7840b2660 0x7ff7840b4b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff79c06ba30 tx=0x7ff78c005f90 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:14.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:14.905+0000 7ff799600640 1 -- 172.21.15.27:0/511475600 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ff7900c9450 con 0x7ff79c103350 2024-05-10T11:13:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.107+0000 7ff7a2000640 1 -- 172.21.15.27:0/511475600 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff79c0635b0 con 0x7ff79c103350 2024-05-10T11:13:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.108+0000 7ff799600640 1 -- 172.21.15.27:0/511475600 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7ff7900c8d50 con 0x7ff79c103350 2024-05-10T11:13:15.109 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-05-10T11:13:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.111+0000 7ff7a2000640 1 -- 172.21.15.27:0/511475600 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff7840b2660 msgr2=0x7ff7840b4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.111+0000 7ff7a2000640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff7840b2660 0x7ff7840b4b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff79c06ba30 tx=0x7ff78c005f90 comp rx=0 tx=0).stop 2024-05-10T11:13:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.111+0000 7ff7a2000640 1 -- 172.21.15.27:0/511475600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff79c103350 msgr2=0x7ff79c19adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.111+0000 7ff7a2000640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff79c103350 0x7ff79c19adb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff7900057c0 tx=0x7ff790032da0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.111+0000 7ff7a2000640 1 -- 172.21.15.27:0/511475600 shutdown_connections 2024-05-10T11:13:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.111+0000 7ff7a2000640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff7840b2660 0x7ff7840b4b20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.111+0000 7ff7a2000640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79c1085b0 0x7ff79c1a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.111+0000 7ff7a2000640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79c105c80 0x7ff79c19b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.111+0000 7ff7a2000640 1 --2- 172.21.15.27:0/511475600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff79c103350 0x7ff79c19adb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.111+0000 7ff7a2000640 1 -- 172.21.15.27:0/511475600 >> 172.21.15.27:0/511475600 conn(0x7ff79c0fcd60 msgr2=0x7ff79c102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.112+0000 7ff7a2000640 1 -- 172.21.15.27:0/511475600 shutdown_connections 2024-05-10T11:13:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.112+0000 7ff7a2000640 1 -- 172.21.15.27:0/511475600 wait complete. 2024-05-10T11:13:15.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-10T11:13:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.353+0000 7fca1f000640 1 -- 172.21.15.27:0/1476799057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca18103bf0 msgr2=0x7fca18104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.353+0000 7fca1f000640 1 --2- 172.21.15.27:0/1476799057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca18103bf0 0x7fca18104070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fca04009900 tx=0x7fca0402a3f0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.353+0000 7fca1f000640 1 -- 172.21.15.27:0/1476799057 shutdown_connections 2024-05-10T11:13:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.353+0000 7fca1f000640 1 --2- 172.21.15.27:0/1476799057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca181045b0 0x7fca1810ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.353+0000 7fca1f000640 1 --2- 172.21.15.27:0/1476799057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca18103bf0 0x7fca18104070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.353+0000 7fca1f000640 1 --2- 172.21.15.27:0/1476799057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca18102a00 0x7fca18102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.353+0000 7fca1f000640 1 -- 172.21.15.27:0/1476799057 >> 172.21.15.27:0/1476799057 conn(0x7fca180fdf90 msgr2=0x7fca181003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.353+0000 7fca1f000640 1 -- 172.21.15.27:0/1476799057 shutdown_connections 2024-05-10T11:13:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.353+0000 7fca1f000640 1 -- 172.21.15.27:0/1476799057 wait complete. 2024-05-10T11:13:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.355+0000 7fca1f000640 1 Processor -- start 2024-05-10T11:13:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.355+0000 7fca1f000640 1 -- start start 2024-05-10T11:13:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.355+0000 7fca1f000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca18102a00 0x7fca1819ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca1f000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca18103bf0 0x7fca1819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca1f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca181045b0 0x7fca181a2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca1f000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fca1810e580 con 0x7fca18102a00 2024-05-10T11:13:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca1f000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fca1810e400 con 0x7fca18103bf0 2024-05-10T11:13:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca1f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fca1810e700 con 0x7fca181045b0 2024-05-10T11:13:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca17400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca18103bf0 0x7fca1819b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca17400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca18103bf0 0x7fca1819b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37994/0 (socket says 172.21.15.27:37994) 2024-05-10T11:13:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca17400640 1 -- 172.21.15.27:0/3657877048 learned_addr learned my addr 172.21.15.27:0/3657877048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca17e00640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca18102a00 0x7fca1819ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca1d000640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca181045b0 0x7fca181a2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca17400640 1 -- 172.21.15.27:0/3657877048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca181045b0 msgr2=0x7fca181a2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca17400640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca181045b0 0x7fca181a2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca17400640 1 -- 172.21.15.27:0/3657877048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca18102a00 msgr2=0x7fca1819ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca17400640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca18102a00 0x7fca1819ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca17400640 1 -- 172.21.15.27:0/3657877048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca181a2a90 con 0x7fca18103bf0 2024-05-10T11:13:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca17e00640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca18102a00 0x7fca1819ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:13:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.356+0000 7fca1d000640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca181045b0 0x7fca181a2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:13:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.357+0000 7fca17400640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca18103bf0 0x7fca1819b3b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fca04005ab0 tx=0x7fca04003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.357+0000 7fca14c00640 1 -- 172.21.15.27:0/3657877048 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0403f070 con 0x7fca18103bf0 2024-05-10T11:13:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.357+0000 7fca14c00640 1 -- 172.21.15.27:0/3657877048 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca04031c50 con 0x7fca18103bf0 2024-05-10T11:13:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.357+0000 7fca1f000640 1 -- 172.21.15.27:0/3657877048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca181a2d20 con 0x7fca18103bf0 2024-05-10T11:13:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.357+0000 7fca14c00640 1 -- 172.21.15.27:0/3657877048 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04035030 con 0x7fca18103bf0 2024-05-10T11:13:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.357+0000 7fca1f000640 1 -- 172.21.15.27:0/3657877048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca181a31c0 con 0x7fca18103bf0 2024-05-10T11:13:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.359+0000 7fca1f000640 1 -- 172.21.15.27:0/3657877048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca181a33f0 con 0x7fca18103bf0 2024-05-10T11:13:15.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.361+0000 7fca14c00640 1 -- 172.21.15.27:0/3657877048 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 24) ==== 149823+0+0 (secure 0 0 0) 0x7fca04042070 con 0x7fca18103bf0 2024-05-10T11:13:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.363+0000 7fca14c00640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9fc0b29f0 0x7fc9fc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.364+0000 7fca14c00640 1 -- 172.21.15.27:0/3657877048 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fca041034c0 con 0x7fca18103bf0 2024-05-10T11:13:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.364+0000 7fca17e00640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9fc0b29f0 0x7fc9fc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:15.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.365+0000 7fca17e00640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9fc0b29f0 0x7fc9fc0b4eb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fca18103a50 tx=0x7fca08005f90 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.367+0000 7fca14c00640 1 -- 172.21.15.27:0/3657877048 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fca040ca900 con 0x7fca18103bf0 2024-05-10T11:13:15.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:15 smithi189 ceph-mon[34842]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:15.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:15 smithi189 ceph-mon[34842]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:13:15.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:15 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2373645672' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:13:15.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:15 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/511475600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-10T11:13:15.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.588+0000 7fca1f000640 1 -- 172.21.15.27:0/3657877048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fca181a3780 con 0x7fca18103bf0 2024-05-10T11:13:15.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.588+0000 7fca14c00640 1 -- 172.21.15.27:0/3657877048 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fca040ca200 con 0x7fca18103bf0 2024-05-10T11:13:15.589 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-05-10T11:13:15.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.591+0000 7fca1f000640 1 -- 172.21.15.27:0/3657877048 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9fc0b29f0 msgr2=0x7fc9fc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:15.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.591+0000 7fca1f000640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9fc0b29f0 0x7fc9fc0b4eb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fca18103a50 tx=0x7fca08005f90 comp rx=0 tx=0).stop 2024-05-10T11:13:15.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.591+0000 7fca1f000640 1 -- 172.21.15.27:0/3657877048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca18103bf0 msgr2=0x7fca1819b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:15.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.591+0000 7fca1f000640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca18103bf0 0x7fca1819b3b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fca04005ab0 tx=0x7fca04003b90 comp rx=0 tx=0).stop 2024-05-10T11:13:15.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.591+0000 7fca1f000640 1 -- 172.21.15.27:0/3657877048 shutdown_connections 2024-05-10T11:13:15.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.591+0000 7fca1f000640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9fc0b29f0 0x7fc9fc0b4eb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.591+0000 7fca1f000640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca181045b0 0x7fca181a2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.591+0000 7fca1f000640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca18103bf0 0x7fca1819b3b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.591+0000 7fca1f000640 1 --2- 172.21.15.27:0/3657877048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca18102a00 0x7fca1819ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.591+0000 7fca1f000640 1 -- 172.21.15.27:0/3657877048 >> 172.21.15.27:0/3657877048 conn(0x7fca180fdf90 msgr2=0x7fca18074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:15.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.591+0000 7fca1f000640 1 -- 172.21.15.27:0/3657877048 shutdown_connections 2024-05-10T11:13:15.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.591+0000 7fca1f000640 1 -- 172.21.15.27:0/3657877048 wait complete. 2024-05-10T11:13:15.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:15 smithi027 ceph-mon[30054]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:15.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:15 smithi027 ceph-mon[30054]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:13:15.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:15 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2373645672' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:13:15.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:15 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/511475600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-10T11:13:15.610 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-10T11:13:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:15 smithi076 ceph-mon[34152]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:15 smithi076 ceph-mon[34152]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:13:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:15 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2373645672' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:13:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:15 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/511475600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-10T11:13:15.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.807+0000 7f30eca00640 1 -- 172.21.15.27:0/985418386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104af0 msgr2=0x7f30e8104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:15.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.807+0000 7f30eca00640 1 --2- 172.21.15.27:0/985418386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104af0 0x7f30e8104f10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009900 tx=0x7f30d802a410 comp rx=0 tx=0).stop 2024-05-10T11:13:15.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.807+0000 7f30eca00640 1 -- 172.21.15.27:0/985418386 shutdown_connections 2024-05-10T11:13:15.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.807+0000 7f30eca00640 1 --2- 172.21.15.27:0/985418386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e81066a0 0x7f30e810cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.807+0000 7f30eca00640 1 --2- 172.21.15.27:0/985418386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e8105ce0 0x7f30e8106160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.807+0000 7f30eca00640 1 --2- 172.21.15.27:0/985418386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e8104af0 0x7f30e8104f10 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.807+0000 7f30eca00640 1 -- 172.21.15.27:0/985418386 >> 172.21.15.27:0/985418386 conn(0x7f30e8100060 msgr2=0x7f30e81024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:15.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.807+0000 7f30eca00640 1 -- 172.21.15.27:0/985418386 shutdown_connections 2024-05-10T11:13:15.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.808+0000 7f30eca00640 1 -- 172.21.15.27:0/985418386 wait complete. 2024-05-10T11:13:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.808+0000 7f30eca00640 1 Processor -- start 2024-05-10T11:13:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.809+0000 7f30eca00640 1 -- start start 2024-05-10T11:13:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.809+0000 7f30eca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e8104af0 0x7f30e819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.809+0000 7f30eca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e8105ce0 0x7f30e819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.809+0000 7f30eca00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e81066a0 0x7f30e81a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.809+0000 7f30eca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f30e8110660 con 0x7f30e8104af0 2024-05-10T11:13:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.809+0000 7f30eca00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f30e81104e0 con 0x7f30e81066a0 2024-05-10T11:13:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.809+0000 7f30eca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f30e81107e0 con 0x7f30e8105ce0 2024-05-10T11:13:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.809+0000 7f30e6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e8105ce0 0x7f30e819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.809+0000 7f30e6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e8105ce0 0x7f30e819b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42256/0 (socket says 172.21.15.27:42256) 2024-05-10T11:13:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.809+0000 7f30e6a00640 1 -- 172.21.15.27:0/4030195963 learned_addr learned my addr 172.21.15.27:0/4030195963 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.809+0000 7f30e7e00640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e81066a0 0x7f30e81a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.809+0000 7f30e7400640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e8104af0 0x7f30e819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.810+0000 7f30e6a00640 1 -- 172.21.15.27:0/4030195963 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e81066a0 msgr2=0x7f30e81a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.810+0000 7f30e6a00640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e81066a0 0x7f30e81a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.810+0000 7f30e6a00640 1 -- 172.21.15.27:0/4030195963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e8104af0 msgr2=0x7f30e819ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.810+0000 7f30e6a00640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e8104af0 0x7f30e819ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.810+0000 7f30e6a00640 1 -- 172.21.15.27:0/4030195963 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e81a2960 con 0x7f30e8105ce0 2024-05-10T11:13:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.810+0000 7f30e6a00640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e8105ce0 0x7f30e819b290 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f30d400e9f0 tx=0x7f30d400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.810+0000 7f30dfe00640 1 -- 172.21.15.27:0/4030195963 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d400cd80 con 0x7f30e8105ce0 2024-05-10T11:13:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.810+0000 7f30eca00640 1 -- 172.21.15.27:0/4030195963 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e81a2c50 con 0x7f30e8105ce0 2024-05-10T11:13:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.810+0000 7f30dfe00640 1 -- 172.21.15.27:0/4030195963 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30d40054d0 con 0x7f30e8105ce0 2024-05-10T11:13:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.810+0000 7f30dfe00640 1 -- 172.21.15.27:0/4030195963 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4010690 con 0x7f30e8105ce0 2024-05-10T11:13:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.810+0000 7f30eca00640 1 -- 172.21.15.27:0/4030195963 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e81a31a0 con 0x7f30e8105ce0 2024-05-10T11:13:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.811+0000 7f30eca00640 1 -- 172.21.15.27:0/4030195963 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e810be60 con 0x7f30e8105ce0 2024-05-10T11:13:15.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.814+0000 7f30dfe00640 1 -- 172.21.15.27:0/4030195963 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 24) ==== 149823+0+0 (secure 0 0 0) 0x7f30d4004e90 con 0x7f30e8105ce0 2024-05-10T11:13:15.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.816+0000 7f30dfe00640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f30cc0b29f0 0x7f30cc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:15.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.816+0000 7f30dfe00640 1 -- 172.21.15.27:0/4030195963 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f30d40e30e0 con 0x7f30e8105ce0 2024-05-10T11:13:15.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.816+0000 7f30e7400640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f30cc0b29f0 0x7f30cc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:15.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.817+0000 7f30e7400640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f30cc0b29f0 0x7f30cc0b4eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f30e819c220 tx=0x7f30d80314a0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:15.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:15.820+0000 7f30dfe00640 1 -- 172.21.15.27:0/4030195963 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f30d40aa580 con 0x7f30e8105ce0 2024-05-10T11:13:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.075+0000 7f30eca00640 1 -- 172.21.15.27:0/4030195963 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f30e802dd50 con 0x7f30e8105ce0 2024-05-10T11:13:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.075+0000 7f30dfe00640 1 -- 172.21.15.27:0/4030195963 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f30d40a9e80 con 0x7f30e8105ce0 2024-05-10T11:13:16.076 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-05-10T11:13:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.078+0000 7f30eca00640 1 -- 172.21.15.27:0/4030195963 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f30cc0b29f0 msgr2=0x7f30cc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.078+0000 7f30eca00640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f30cc0b29f0 0x7f30cc0b4eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f30e819c220 tx=0x7f30d80314a0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.078+0000 7f30eca00640 1 -- 172.21.15.27:0/4030195963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e8105ce0 msgr2=0x7f30e819b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.078+0000 7f30eca00640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e8105ce0 0x7f30e819b290 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f30d400e9f0 tx=0x7f30d400edf0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.078+0000 7f30eca00640 1 -- 172.21.15.27:0/4030195963 shutdown_connections 2024-05-10T11:13:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.078+0000 7f30eca00640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f30cc0b29f0 0x7f30cc0b4eb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.078+0000 7f30eca00640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30e81066a0 0x7f30e81a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.078+0000 7f30eca00640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e8105ce0 0x7f30e819b290 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.078+0000 7f30eca00640 1 --2- 172.21.15.27:0/4030195963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e8104af0 0x7f30e819ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.078+0000 7f30eca00640 1 -- 172.21.15.27:0/4030195963 >> 172.21.15.27:0/4030195963 conn(0x7f30e8100060 msgr2=0x7f30e8109030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.078+0000 7f30eca00640 1 -- 172.21.15.27:0/4030195963 shutdown_connections 2024-05-10T11:13:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.078+0000 7f30eca00640 1 -- 172.21.15.27:0/4030195963 wait complete. 2024-05-10T11:13:16.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-10T11:13:16.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.288+0000 7fc716800640 1 -- 172.21.15.27:0/2715137166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7101028e0 msgr2=0x7fc710102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:16.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.288+0000 7fc716800640 1 --2- 172.21.15.27:0/2715137166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7101028e0 0x7fc710102d00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc704009900 tx=0x7fc70402a3f0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.288+0000 7fc716800640 1 -- 172.21.15.27:0/2715137166 shutdown_connections 2024-05-10T11:13:16.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.288+0000 7fc716800640 1 --2- 172.21.15.27:0/2715137166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc710104490 0x7fc71010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.288+0000 7fc716800640 1 --2- 172.21.15.27:0/2715137166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710103ad0 0x7fc710103f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.288+0000 7fc716800640 1 --2- 172.21.15.27:0/2715137166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7101028e0 0x7fc710102d00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.288+0000 7fc716800640 1 -- 172.21.15.27:0/2715137166 >> 172.21.15.27:0/2715137166 conn(0x7fc7100fde50 msgr2=0x7fc710100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.288+0000 7fc716800640 1 -- 172.21.15.27:0/2715137166 shutdown_connections 2024-05-10T11:13:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.288+0000 7fc716800640 1 -- 172.21.15.27:0/2715137166 wait complete. 2024-05-10T11:13:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.290+0000 7fc716800640 1 Processor -- start 2024-05-10T11:13:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.290+0000 7fc716800640 1 -- start start 2024-05-10T11:13:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.290+0000 7fc716800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7101028e0 0x7fc710076760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc716800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc710103ad0 0x7fc710074da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc716800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710104490 0x7fc7100752e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc716800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc71010e460 con 0x7fc710104490 2024-05-10T11:13:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc716800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc71010e2e0 con 0x7fc7101028e0 2024-05-10T11:13:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc716800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc71010e5e0 con 0x7fc710103ad0 2024-05-10T11:13:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc715400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7101028e0 0x7fc710076760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc715400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7101028e0 0x7fc710076760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:38026/0 (socket says 172.21.15.27:38026) 2024-05-10T11:13:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc715400640 1 -- 172.21.15.27:0/1685902621 learned_addr learned my addr 172.21.15.27:0/1685902621 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc714a00640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc710103ad0 0x7fc710074da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc715e00640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710104490 0x7fc7100752e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc714a00640 1 -- 172.21.15.27:0/1685902621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7101028e0 msgr2=0x7fc710076760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc714a00640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7101028e0 0x7fc710076760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc714a00640 1 -- 172.21.15.27:0/1685902621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710104490 msgr2=0x7fc7100752e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc714a00640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710104490 0x7fc7100752e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.291+0000 7fc714a00640 1 -- 172.21.15.27:0/1685902621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc710075b70 con 0x7fc710103ad0 2024-05-10T11:13:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.292+0000 7fc714a00640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc710103ad0 0x7fc710074da0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00e9f0 tx=0x7fc6fc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:16.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.292+0000 7fc70e000640 1 -- 172.21.15.27:0/1685902621 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc00ccc0 con 0x7fc710103ad0 2024-05-10T11:13:16.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.292+0000 7fc70e000640 1 -- 172.21.15.27:0/1685902621 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6fc0108e0 con 0x7fc710103ad0 2024-05-10T11:13:16.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.292+0000 7fc70e000640 1 -- 172.21.15.27:0/1685902621 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc018880 con 0x7fc710103ad0 2024-05-10T11:13:16.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.292+0000 7fc716800640 1 -- 172.21.15.27:0/1685902621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7101a2c50 con 0x7fc710103ad0 2024-05-10T11:13:16.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.292+0000 7fc716800640 1 -- 172.21.15.27:0/1685902621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7101a3150 con 0x7fc710103ad0 2024-05-10T11:13:16.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.293+0000 7fc716800640 1 -- 172.21.15.27:0/1685902621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc710109c20 con 0x7fc710103ad0 2024-05-10T11:13:16.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.295+0000 7fc70e000640 1 -- 172.21.15.27:0/1685902621 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 24) ==== 149823+0+0 (secure 0 0 0) 0x7fc6fc010430 con 0x7fc710103ad0 2024-05-10T11:13:16.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.297+0000 7fc70e000640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc6f80b29f0 0x7fc6f80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:16.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.297+0000 7fc70e000640 1 -- 172.21.15.27:0/1685902621 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fc6fc0e2f00 con 0x7fc710103ad0 2024-05-10T11:13:16.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.297+0000 7fc715400640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc6f80b29f0 0x7fc6f80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:16.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.298+0000 7fc715400640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc6f80b29f0 0x7fc6f80b4eb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc704000c00 tx=0x7fc704005a20 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:16.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.301+0000 7fc70e000640 1 -- 172.21.15.27:0/1685902621 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fc6fc0a9cf0 con 0x7fc710103ad0 2024-05-10T11:13:16.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:16 smithi027 ceph-mon[30054]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:13:16.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:16 smithi027 ceph-mon[30054]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-05-10T11:13:16.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:16 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:16.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:16 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:16.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:16 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:16 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:16 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3657877048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-10T11:13:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:16 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4030195963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-10T11:13:16.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:16 smithi189 ceph-mon[34842]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:13:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:16 smithi189 ceph-mon[34842]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-05-10T11:13:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:16 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:16 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:16 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:16 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:16 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3657877048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-10T11:13:16.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:16 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4030195963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-10T11:13:16.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.564+0000 7fc716800640 1 -- 172.21.15.27:0/1685902621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc71002dda0 con 0x7fc710103ad0 2024-05-10T11:13:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.565+0000 7fc70e000640 1 -- 172.21.15.27:0/1685902621 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7fc6fc0adff0 con 0x7fc710103ad0 2024-05-10T11:13:16.565 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-05-10T11:13:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.567+0000 7fc716800640 1 -- 172.21.15.27:0/1685902621 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc6f80b29f0 msgr2=0x7fc6f80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.567+0000 7fc716800640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc6f80b29f0 0x7fc6f80b4eb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc704000c00 tx=0x7fc704005a20 comp rx=0 tx=0).stop 2024-05-10T11:13:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.567+0000 7fc716800640 1 -- 172.21.15.27:0/1685902621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc710103ad0 msgr2=0x7fc710074da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.567+0000 7fc716800640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc710103ad0 0x7fc710074da0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00e9f0 tx=0x7fc6fc00eeb0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.567+0000 7fc716800640 1 -- 172.21.15.27:0/1685902621 shutdown_connections 2024-05-10T11:13:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.567+0000 7fc716800640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc6f80b29f0 0x7fc6f80b4eb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.567+0000 7fc716800640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710104490 0x7fc7100752e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.567+0000 7fc716800640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc710103ad0 0x7fc710074da0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.567+0000 7fc716800640 1 --2- 172.21.15.27:0/1685902621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7101028e0 0x7fc710076760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.567+0000 7fc716800640 1 -- 172.21.15.27:0/1685902621 >> 172.21.15.27:0/1685902621 conn(0x7fc7100fde50 msgr2=0x7fc710106df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.567+0000 7fc716800640 1 -- 172.21.15.27:0/1685902621 shutdown_connections 2024-05-10T11:13:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:16.567+0000 7fc716800640 1 -- 172.21.15.27:0/1685902621 wait complete. 2024-05-10T11:13:16.585 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-10T11:13:16.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:13:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:16 smithi076 ceph-mon[34152]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:13:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:16 smithi076 ceph-mon[34152]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-05-10T11:13:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:16 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:16 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:16 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:16 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:16 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3657877048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-10T11:13:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:16 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4030195963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-10T11:13:17.021 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:13:17.490 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: [10/May/2024:11:13:15] ENGINE Bus STARTING 2024-05-10T11:13:17.490 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: [10/May/2024:11:13:15] ENGINE Serving on http://172.21.15.27:8765 2024-05-10T11:13:17.490 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: [10/May/2024:11:13:15] ENGINE Serving on https://172.21.15.27:7150 2024-05-10T11:13:17.490 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: [10/May/2024:11:13:15] ENGINE Bus STARTED 2024-05-10T11:13:17.490 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.490 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.490 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.490 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1685902621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-10T11:13:17.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:17 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: [10/May/2024:11:13:15] ENGINE Bus STARTING 2024-05-10T11:13:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: [10/May/2024:11:13:15] ENGINE Serving on http://172.21.15.27:8765 2024-05-10T11:13:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: [10/May/2024:11:13:15] ENGINE Serving on https://172.21.15.27:7150 2024-05-10T11:13:17.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: [10/May/2024:11:13:15] ENGINE Bus STARTED 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.504 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1685902621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-10T11:13:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:17 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: [10/May/2024:11:13:15] ENGINE Bus STARTING 2024-05-10T11:13:17.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: [10/May/2024:11:13:15] ENGINE Serving on http://172.21.15.27:8765 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: [10/May/2024:11:13:15] ENGINE Serving on https://172.21.15.27:7150 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: [10/May/2024:11:13:15] ENGINE Bus STARTED 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:17.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1685902621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-10T11:13:17.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:17.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:17 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:18.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:18 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-05-10T11:13:18.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:18 smithi027 ceph-mon[30054]: Unable to set osd_memory_target on smithi189 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-10T11:13:18.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:18 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-05-10T11:13:18.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:18 smithi027 ceph-mon[30054]: Unable to set osd_memory_target on smithi076 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-05-10T11:13:18.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:18 smithi027 ceph-mon[30054]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:13:18.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:18 smithi027 ceph-mon[30054]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-05-10T11:13:18.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:18 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-05-10T11:13:18.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:18 smithi189 ceph-mon[34842]: Unable to set osd_memory_target on smithi189 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-10T11:13:18.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:18 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-05-10T11:13:18.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:18 smithi189 ceph-mon[34842]: Unable to set osd_memory_target on smithi076 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-05-10T11:13:18.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:18 smithi189 ceph-mon[34842]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:13:18.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:18 smithi189 ceph-mon[34842]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-05-10T11:13:18.577 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:18 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-05-10T11:13:18.577 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:18 smithi076 ceph-mon[34152]: Unable to set osd_memory_target on smithi189 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-10T11:13:18.577 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:18 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-05-10T11:13:18.577 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:18 smithi076 ceph-mon[34152]: Unable to set osd_memory_target on smithi076 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-05-10T11:13:18.577 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:18 smithi076 ceph-mon[34152]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:13:18.577 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:18 smithi076 ceph-mon[34152]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-05-10T11:13:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.363+0000 7f73d9800700 1 -- 172.21.15.27:0/1972622704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73d4109510 msgr2=0x7f73d41098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.363+0000 7f73d9800700 1 --2- 172.21.15.27:0/1972622704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73d4109510 0x7f73d41098f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009990 tx=0x7f73c4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:13:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.363+0000 7f73d9800700 1 -- 172.21.15.27:0/1972622704 shutdown_connections 2024-05-10T11:13:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.363+0000 7f73d9800700 1 --2- 172.21.15.27:0/1972622704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d406bb40 0x7f73d4110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.363+0000 7f73d9800700 1 --2- 172.21.15.27:0/1972622704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73d4109510 0x7f73d41098f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.363+0000 7f73d9800700 1 --2- 172.21.15.27:0/1972622704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73d406b180 0x7f73d406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.363+0000 7f73d9800700 1 -- 172.21.15.27:0/1972622704 >> 172.21.15.27:0/1972622704 conn(0x7f73d4079350 msgr2=0x7f73d4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.364+0000 7f73d9800700 1 -- 172.21.15.27:0/1972622704 shutdown_connections 2024-05-10T11:13:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.364+0000 7f73d9800700 1 -- 172.21.15.27:0/1972622704 wait complete. 2024-05-10T11:13:19.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.365+0000 7f73d9800700 1 Processor -- start 2024-05-10T11:13:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.365+0000 7f73d9800700 1 -- start start 2024-05-10T11:13:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.366+0000 7f73d9800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d406b180 0x7f73d419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.366+0000 7f73d9800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73d406bb40 0x7f73d419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.366+0000 7f73d9800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73d4109510 0x7f73d41a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.366+0000 7f73d9800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f73d4113640 con 0x7f73d406bb40 2024-05-10T11:13:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.366+0000 7f73d9800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f73d41134c0 con 0x7f73d4109510 2024-05-10T11:13:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.366+0000 7f73d9800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f73d41137c0 con 0x7f73d406b180 2024-05-10T11:13:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.366+0000 7f73d2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d406b180 0x7f73d419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.366+0000 7f73d2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d406b180 0x7f73d419bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42290/0 (socket says 172.21.15.27:42290) 2024-05-10T11:13:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.366+0000 7f73d2a00700 1 -- 172.21.15.27:0/529748043 learned_addr learned my addr 172.21.15.27:0/529748043 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.366+0000 7f73d2000700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73d406bb40 0x7f73d419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.366+0000 7f73d3400700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73d4109510 0x7f73d41a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.367+0000 7f73d2a00700 1 -- 172.21.15.27:0/529748043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73d4109510 msgr2=0x7f73d41a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.367+0000 7f73d2a00700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73d4109510 0x7f73d41a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.367+0000 7f73d2a00700 1 -- 172.21.15.27:0/529748043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73d406bb40 msgr2=0x7f73d419c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.367+0000 7f73d2a00700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73d406bb40 0x7f73d419c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.367+0000 7f73d2a00700 1 -- 172.21.15.27:0/529748043 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d41a0d70 con 0x7f73d406b180 2024-05-10T11:13:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.368+0000 7f73d2a00700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d406b180 0x7f73d419bd60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f73c40057f0 tx=0x7f73c400bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.368+0000 7f73cb400700 1 -- 172.21.15.27:0/529748043 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c4026070 con 0x7f73d406b180 2024-05-10T11:13:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.368+0000 7f73cb400700 1 -- 172.21.15.27:0/529748043 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73c4021410 con 0x7f73d406b180 2024-05-10T11:13:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.368+0000 7f73d9800700 1 -- 172.21.15.27:0/529748043 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73d41a1000 con 0x7f73d406b180 2024-05-10T11:13:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.368+0000 7f73cb400700 1 -- 172.21.15.27:0/529748043 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c4017650 con 0x7f73d406b180 2024-05-10T11:13:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.368+0000 7f73d9800700 1 -- 172.21.15.27:0/529748043 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73d406cec0 con 0x7f73d406b180 2024-05-10T11:13:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.370+0000 7f73d9800700 1 -- 172.21.15.27:0/529748043 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73d406a190 con 0x7f73d406b180 2024-05-10T11:13:19.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.373+0000 7f73cb400700 1 -- 172.21.15.27:0/529748043 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f73c4029070 con 0x7f73d406b180 2024-05-10T11:13:19.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.374+0000 7f73cb400700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f73b80b2b90 0x7f73b80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:19.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.375+0000 7f73cb400700 1 -- 172.21.15.27:0/529748043 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f73c40e37a0 con 0x7f73d406b180 2024-05-10T11:13:19.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.375+0000 7f73d2000700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f73b80b2b90 0x7f73b80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:19.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.376+0000 7f73d2000700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f73b80b2b90 0x7f73b80b5050 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f73d41044b0 tx=0x7f73c0009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:19.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.376+0000 7f73cb400700 1 -- 172.21.15.27:0/529748043 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f73c4013070 con 0x7f73d406b180 2024-05-10T11:13:19.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.814+0000 7f73d9800700 1 -- 172.21.15.27:0/529748043 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73d419d0c0 con 0x7f73d406b180 2024-05-10T11:13:19.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.815+0000 7f73cb400700 1 -- 172.21.15.27:0/529748043 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11936 (secure 0 0 0) 0x7f73d419d0c0 con 0x7f73d406b180 2024-05-10T11:13:19.816 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:13:19.816 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:13:10.916396+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24484,24472,24467,24488]},"id":1}]} 2024-05-10T11:13:19.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.818+0000 7f73d9800700 1 -- 172.21.15.27:0/529748043 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f73b80b2b90 msgr2=0x7f73b80b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:19.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.818+0000 7f73d9800700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f73b80b2b90 0x7f73b80b5050 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f73d41044b0 tx=0x7f73c0009260 comp rx=0 tx=0).stop 2024-05-10T11:13:19.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.818+0000 7f73d9800700 1 -- 172.21.15.27:0/529748043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d406b180 msgr2=0x7f73d419bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:19.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.818+0000 7f73d9800700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d406b180 0x7f73d419bd60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f73c40057f0 tx=0x7f73c400bfd0 comp rx=0 tx=0).stop 2024-05-10T11:13:19.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.818+0000 7f73d9800700 1 -- 172.21.15.27:0/529748043 shutdown_connections 2024-05-10T11:13:19.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.818+0000 7f73d9800700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f73b80b2b90 0x7f73b80b5050 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:19.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.818+0000 7f73d9800700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d406b180 0x7f73d419bd60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:19.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.818+0000 7f73d9800700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73d4109510 0x7f73d41a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:19.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.818+0000 7f73d9800700 1 --2- 172.21.15.27:0/529748043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73d406bb40 0x7f73d419c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:19.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.818+0000 7f73d9800700 1 -- 172.21.15.27:0/529748043 >> 172.21.15.27:0/529748043 conn(0x7f73d4079350 msgr2=0x7f73d410e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:19.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.819+0000 7f73d9800700 1 -- 172.21.15.27:0/529748043 shutdown_connections 2024-05-10T11:13:19.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:19.819+0000 7f73d9800700 1 -- 172.21.15.27:0/529748043 wait complete. 2024-05-10T11:13:19.821 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 30 2024-05-10T11:13:20.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi027 to 176.4M 2024-05-10T11:13:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: Unable to set osd_memory_target on smithi027 to 185039769: error parsing value: Value '185039769' is below minimum 939524096 2024-05-10T11:13:20.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:20.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:20.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: Updating smithi027:/etc/ceph/ceph.conf 2024-05-10T11:13:20.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: Updating smithi076:/etc/ceph/ceph.conf 2024-05-10T11:13:20.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-10T11:13:20.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:13:20.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:19 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/529748043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:13:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi027 to 176.4M 2024-05-10T11:13:20.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: Unable to set osd_memory_target on smithi027 to 185039769: error parsing value: Value '185039769' is below minimum 939524096 2024-05-10T11:13:20.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:20.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:20.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: Updating smithi027:/etc/ceph/ceph.conf 2024-05-10T11:13:20.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: Updating smithi076:/etc/ceph/ceph.conf 2024-05-10T11:13:20.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-10T11:13:20.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:13:20.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:19 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/529748043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:13:20.290 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:20.290 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:20.290 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.290 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.290 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.290 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.290 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.291 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.291 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.291 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:13:20.291 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi027 to 176.4M 2024-05-10T11:13:20.291 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: Unable to set osd_memory_target on smithi027 to 185039769: error parsing value: Value '185039769' is below minimum 939524096 2024-05-10T11:13:20.291 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:20.291 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:20.291 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: Updating smithi027:/etc/ceph/ceph.conf 2024-05-10T11:13:20.291 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: Updating smithi076:/etc/ceph/ceph.conf 2024-05-10T11:13:20.291 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-10T11:13:20.291 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:13:20.291 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:19 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/529748043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:13:20.777 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-10T11:13:20.778 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-10T11:13:20.778 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-10T11:13:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.974+0000 7f5009c00640 1 -- 172.21.15.27:0/652908737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50041067e0 msgr2=0x7f500410d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.974+0000 7f5009c00640 1 --2- 172.21.15.27:0/652908737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50041067e0 0x7f500410d070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc009ca0 tx=0x7f4ffc02a8c0 comp rx=0 tx=0).stop 2024-05-10T11:13:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.974+0000 7f5009c00640 1 -- 172.21.15.27:0/652908737 shutdown_connections 2024-05-10T11:13:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.974+0000 7f5009c00640 1 --2- 172.21.15.27:0/652908737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50041067e0 0x7f500410d070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.974+0000 7f5009c00640 1 --2- 172.21.15.27:0/652908737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5004105e20 0x7f50041062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.974+0000 7f5009c00640 1 --2- 172.21.15.27:0/652908737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5004104c30 0x7f5004105050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.974+0000 7f5009c00640 1 -- 172.21.15.27:0/652908737 >> 172.21.15.27:0/652908737 conn(0x7f50041001a0 msgr2=0x7f50041025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.974+0000 7f5009c00640 1 -- 172.21.15.27:0/652908737 shutdown_connections 2024-05-10T11:13:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.974+0000 7f5009c00640 1 -- 172.21.15.27:0/652908737 wait complete. 2024-05-10T11:13:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.975+0000 7f5009c00640 1 Processor -- start 2024-05-10T11:13:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.975+0000 7f5009c00640 1 -- start start 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.975+0000 7f5009c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5004104c30 0x7f500419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.975+0000 7f5009c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5004105e20 0x7f500419b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5009c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50041067e0 0x7f50041a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5009c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5004110790 con 0x7f50041067e0 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5009c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5004110610 con 0x7f5004104c30 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5009c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5004110910 con 0x7f5004105e20 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5003400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50041067e0 0x7f50041a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5002000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5004105e20 0x7f500419b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5003400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50041067e0 0x7f50041a23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39830/0 (socket says 172.21.15.27:39830) 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5002a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5004104c30 0x7f500419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5003400640 1 -- 172.21.15.27:0/1926625976 learned_addr learned my addr 172.21.15.27:0/1926625976 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5002000640 1 -- 172.21.15.27:0/1926625976 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5004104c30 msgr2=0x7f500419ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5002000640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5004104c30 0x7f500419ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5002000640 1 -- 172.21.15.27:0/1926625976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50041067e0 msgr2=0x7f50041a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5002000640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50041067e0 0x7f50041a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5002000640 1 -- 172.21.15.27:0/1926625976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50041a2a20 con 0x7f5004105e20 2024-05-10T11:13:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5002a00640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5004104c30 0x7f500419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.976+0000 7f5003400640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50041067e0 0x7f50041a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.977+0000 7f5002000640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5004105e20 0x7f500419b340 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000e9f0 tx=0x7f4ff000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.977+0000 7f4ffb400640 1 -- 172.21.15.27:0/1926625976 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff000cd80 con 0x7f5004105e20 2024-05-10T11:13:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.977+0000 7f4ffb400640 1 -- 172.21.15.27:0/1926625976 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ff00035f0 con 0x7f5004105e20 2024-05-10T11:13:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.977+0000 7f4ffb400640 1 -- 172.21.15.27:0/1926625976 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0010670 con 0x7f5004105e20 2024-05-10T11:13:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.977+0000 7f5009c00640 1 -- 172.21.15.27:0/1926625976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50041a2d10 con 0x7f5004105e20 2024-05-10T11:13:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.977+0000 7f5009c00640 1 -- 172.21.15.27:0/1926625976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50041a3250 con 0x7f5004105e20 2024-05-10T11:13:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.978+0000 7f5009c00640 1 -- 172.21.15.27:0/1926625976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f500410bf50 con 0x7f5004105e20 2024-05-10T11:13:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.981+0000 7f4ffb400640 1 -- 172.21.15.27:0/1926625976 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f4ff00037a0 con 0x7f5004105e20 2024-05-10T11:13:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.983+0000 7f4ffb400640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4fe80b2b00 0x7f4fe80b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.983+0000 7f4ffb400640 1 -- 172.21.15.27:0/1926625976 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f4ff00e2ee0 con 0x7f5004105e20 2024-05-10T11:13:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.983+0000 7f5002a00640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4fe80b2b00 0x7f4fe80b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:20.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.984+0000 7f5002a00640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4fe80b2b00 0x7f4fe80b4fc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5004105c80 tx=0x7f4ff400b390 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:20.987+0000 7f4ffb400640 1 -- 172.21.15.27:0/1926625976 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4ff00aa240 con 0x7f5004105e20 2024-05-10T11:13:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.203+0000 7f5009c00640 1 -- 172.21.15.27:0/1926625976 --> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5004062df0 con 0x7f4fe80b2b00 2024-05-10T11:13:21.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.222+0000 7f4ffb400640 1 -- 172.21.15.27:0/1926625976 <== mgr.25220 v2:172.21.15.27:6800/1167199542 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339737 (secure 0 0 0) 0x7f5004062df0 con 0x7f4fe80b2b00 2024-05-10T11:13:21.228 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:13:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-10T11:13:21.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.228+0000 7f5009c00640 1 -- 172.21.15.27:0/1926625976 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4fe80b2b00 msgr2=0x7f4fe80b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:21.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.228+0000 7f5009c00640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4fe80b2b00 0x7f4fe80b4fc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5004105c80 tx=0x7f4ff400b390 comp rx=0 tx=0).stop 2024-05-10T11:13:21.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.228+0000 7f5009c00640 1 -- 172.21.15.27:0/1926625976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5004105e20 msgr2=0x7f500419b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:21.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.228+0000 7f5009c00640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5004105e20 0x7f500419b340 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000e9f0 tx=0x7f4ff000eeb0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.228+0000 7f5009c00640 1 -- 172.21.15.27:0/1926625976 shutdown_connections 2024-05-10T11:13:21.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.228+0000 7f5009c00640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4fe80b2b00 0x7f4fe80b4fc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.228+0000 7f5009c00640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50041067e0 0x7f50041a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.228+0000 7f5009c00640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5004105e20 0x7f500419b340 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.228+0000 7f5009c00640 1 --2- 172.21.15.27:0/1926625976 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5004104c30 0x7f500419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.228+0000 7f5009c00640 1 -- 172.21.15.27:0/1926625976 >> 172.21.15.27:0/1926625976 conn(0x7f50041001a0 msgr2=0x7f5004109120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:21.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.228+0000 7f5009c00640 1 -- 172.21.15.27:0/1926625976 shutdown_connections 2024-05-10T11:13:21.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.228+0000 7f5009c00640 1 -- 172.21.15.27:0/1926625976 wait complete. 2024-05-10T11:13:21.249 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-05-10T11:13:21.153436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1197848,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2713,"num_read_kb":1658,"num_write":529,"num_write_kb":3635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":2952552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":916,"ondisk_log_size":916,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361736,"kb_used_data":12560,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580088,"statfs":{"total":1151940427776,"available":1151570010112,"internally_reserved":0,"allocated":12861440,"data_stored":9952238,"data_compressed":608895,"data_compressed_allocated":823296,"data_compressed_original":4497408,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3302,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":210,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.974372"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.485180+0000","last_change":"2024-05-10T11:07:47.212921+0000","last_active":"2024-05-10T11:13:13.485180+0000","last_peered":"2024-05-10T11:13:13.485180+0000","last_clean":"2024-05-10T11:13:13.485180+0000","last_became_active":"2024-05-10T11:07:47.212558+0000","last_became_peered":"2024-05-10T11:07:47.212558+0000","last_unstale":"2024-05-10T11:13:13.485180+0000","last_undegraded":"2024-05-10T11:13:13.485180+0000","last_fullsized":"2024-05-10T11:13:13.485180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:47:20.415883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201806+0000","last_change":"2024-05-10T11:07:47.211186+0000","last_active":"2024-05-10T11:13:13.201806+0000","last_peered":"2024-05-10T11:13:13.201806+0000","last_clean":"2024-05-10T11:13:13.201806+0000","last_became_active":"2024-05-10T11:07:47.203487+0000","last_became_peered":"2024-05-10T11:07:47.203487+0000","last_unstale":"2024-05-10T11:13:13.201806+0000","last_undegraded":"2024-05-10T11:13:13.201806+0000","last_fullsized":"2024-05-10T11:13:13.201806+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:51:29.674632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201303+0000","last_change":"2024-05-10T11:07:47.202718+0000","last_active":"2024-05-10T11:13:13.201303+0000","last_peered":"2024-05-10T11:13:13.201303+0000","last_clean":"2024-05-10T11:13:13.201303+0000","last_became_active":"2024-05-10T11:07:47.202048+0000","last_became_peered":"2024-05-10T11:07:47.202048+0000","last_unstale":"2024-05-10T11:13:13.201303+0000","last_undegraded":"2024-05-10T11:13:13.201303+0000","last_fullsized":"2024-05-10T11:13:13.201303+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:01:41.156850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470810+0000","last_change":"2024-05-10T11:07:47.193519+0000","last_active":"2024-05-10T11:13:13.470810+0000","last_peered":"2024-05-10T11:13:13.470810+0000","last_clean":"2024-05-10T11:13:13.470810+0000","last_became_active":"2024-05-10T11:07:47.193200+0000","last_became_peered":"2024-05-10T11:07:47.193200+0000","last_unstale":"2024-05-10T11:13:13.470810+0000","last_undegraded":"2024-05-10T11:13:13.470810+0000","last_fullsized":"2024-05-10T11:13:13.470810+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:01.949453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281822+0000","last_change":"2024-05-10T11:07:47.207750+0000","last_active":"2024-05-10T11:13:13.281822+0000","last_peered":"2024-05-10T11:13:13.281822+0000","last_clean":"2024-05-10T11:13:13.281822+0000","last_became_active":"2024-05-10T11:07:47.207384+0000","last_became_peered":"2024-05-10T11:07:47.207384+0000","last_unstale":"2024-05-10T11:13:13.281822+0000","last_undegraded":"2024-05-10T11:13:13.281822+0000","last_fullsized":"2024-05-10T11:13:13.281822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:09.432766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129613+0000","last_change":"2024-05-10T11:08:22.245930+0000","last_active":"2024-05-10T11:13:13.129613+0000","last_peered":"2024-05-10T11:13:13.129613+0000","last_clean":"2024-05-10T11:13:13.129613+0000","last_became_active":"2024-05-10T11:08:22.245553+0000","last_became_peered":"2024-05-10T11:08:22.245553+0000","last_unstale":"2024-05-10T11:13:13.129613+0000","last_undegraded":"2024-05-10T11:13:13.129613+0000","last_fullsized":"2024-05-10T11:13:13.129613+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:47:31.485623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.469888+0000","last_change":"2024-05-10T11:07:47.196666+0000","last_active":"2024-05-10T11:13:13.469888+0000","last_peered":"2024-05-10T11:13:13.469888+0000","last_clean":"2024-05-10T11:13:13.469888+0000","last_became_active":"2024-05-10T11:07:47.196300+0000","last_became_peered":"2024-05-10T11:07:47.196300+0000","last_unstale":"2024-05-10T11:13:13.469888+0000","last_undegraded":"2024-05-10T11:13:13.469888+0000","last_fullsized":"2024-05-10T11:13:13.469888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:55:24.343385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.128863+0000","last_change":"2024-05-10T11:07:47.213048+0000","last_active":"2024-05-10T11:13:13.128863+0000","last_peered":"2024-05-10T11:13:13.128863+0000","last_clean":"2024-05-10T11:13:13.128863+0000","last_became_active":"2024-05-10T11:07:47.206359+0000","last_became_peered":"2024-05-10T11:07:47.206359+0000","last_unstale":"2024-05-10T11:13:13.128863+0000","last_undegraded":"2024-05-10T11:13:13.128863+0000","last_fullsized":"2024-05-10T11:13:13.128863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:32:56.307261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477171+0000","last_change":"2024-05-10T11:07:47.190945+0000","last_active":"2024-05-10T11:13:13.477171+0000","last_peered":"2024-05-10T11:13:13.477171+0000","last_clean":"2024-05-10T11:13:13.477171+0000","last_became_active":"2024-05-10T11:07:47.190631+0000","last_became_peered":"2024-05-10T11:07:47.190631+0000","last_unstale":"2024-05-10T11:13:13.477171+0000","last_undegraded":"2024-05-10T11:13:13.477171+0000","last_fullsized":"2024-05-10T11:13:13.477171+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:34:08.683912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.130034+0000","last_change":"2024-05-10T11:07:47.189540+0000","last_active":"2024-05-10T11:13:13.130034+0000","last_peered":"2024-05-10T11:13:13.130034+0000","last_clean":"2024-05-10T11:13:13.130034+0000","last_became_active":"2024-05-10T11:07:47.189185+0000","last_became_peered":"2024-05-10T11:07:47.189185+0000","last_unstale":"2024-05-10T11:13:13.130034+0000","last_undegraded":"2024-05-10T11:13:13.130034+0000","last_fullsized":"2024-05-10T11:13:13.130034+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:55:23.892826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470578+0000","last_change":"2024-05-10T11:07:47.195688+0000","last_active":"2024-05-10T11:13:13.470578+0000","last_peered":"2024-05-10T11:13:13.470578+0000","last_clean":"2024-05-10T11:13:13.470578+0000","last_became_active":"2024-05-10T11:07:47.195081+0000","last_became_peered":"2024-05-10T11:07:47.195081+0000","last_unstale":"2024-05-10T11:13:13.470578+0000","last_undegraded":"2024-05-10T11:13:13.470578+0000","last_fullsized":"2024-05-10T11:13:13.470578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:20:55.240827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470631+0000","last_change":"2024-05-10T11:07:47.193736+0000","last_active":"2024-05-10T11:13:13.470631+0000","last_peered":"2024-05-10T11:13:13.470631+0000","last_clean":"2024-05-10T11:13:13.470631+0000","last_became_active":"2024-05-10T11:07:47.193400+0000","last_became_peered":"2024-05-10T11:07:47.193400+0000","last_unstale":"2024-05-10T11:13:13.470631+0000","last_undegraded":"2024-05-10T11:13:13.470631+0000","last_fullsized":"2024-05-10T11:13:13.470631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:46:28.507330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478284+0000","last_change":"2024-05-10T11:08:22.042849+0000","last_active":"2024-05-10T11:13:13.478284+0000","last_peered":"2024-05-10T11:13:13.478284+0000","last_clean":"2024-05-10T11:13:13.478284+0000","last_became_active":"2024-05-10T11:08:22.042024+0000","last_became_peered":"2024-05-10T11:08:22.042024+0000","last_unstale":"2024-05-10T11:13:13.478284+0000","last_undegraded":"2024-05-10T11:13:13.478284+0000","last_fullsized":"2024-05-10T11:13:13.478284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:29:08.096541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201493+0000","last_change":"2024-05-10T11:07:47.214016+0000","last_active":"2024-05-10T11:13:13.201493+0000","last_peered":"2024-05-10T11:13:13.201493+0000","last_clean":"2024-05-10T11:13:13.201493+0000","last_became_active":"2024-05-10T11:07:47.209658+0000","last_became_peered":"2024-05-10T11:07:47.209658+0000","last_unstale":"2024-05-10T11:13:13.201493+0000","last_undegraded":"2024-05-10T11:13:13.201493+0000","last_fullsized":"2024-05-10T11:13:13.201493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:44.771221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201161+0000","last_change":"2024-05-10T11:07:47.210153+0000","last_active":"2024-05-10T11:13:13.201161+0000","last_peered":"2024-05-10T11:13:13.201161+0000","last_clean":"2024-05-10T11:13:13.201161+0000","last_became_active":"2024-05-10T11:07:47.208470+0000","last_became_peered":"2024-05-10T11:07:47.208470+0000","last_unstale":"2024-05-10T11:13:13.201161+0000","last_undegraded":"2024-05-10T11:13:13.201161+0000","last_fullsized":"2024-05-10T11:13:13.201161+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:23:03.614670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.476671+0000","last_change":"2024-05-10T11:07:47.210704+0000","last_active":"2024-05-10T11:13:13.476671+0000","last_peered":"2024-05-10T11:13:13.476671+0000","last_clean":"2024-05-10T11:13:13.476671+0000","last_became_active":"2024-05-10T11:07:47.210317+0000","last_became_peered":"2024-05-10T11:07:47.210317+0000","last_unstale":"2024-05-10T11:13:13.476671+0000","last_undegraded":"2024-05-10T11:13:13.476671+0000","last_fullsized":"2024-05-10T11:13:13.476671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:47:30.848731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484975+0000","last_change":"2024-05-10T11:07:47.213116+0000","last_active":"2024-05-10T11:13:13.484975+0000","last_peered":"2024-05-10T11:13:13.484975+0000","last_clean":"2024-05-10T11:13:13.484975+0000","last_became_active":"2024-05-10T11:07:47.212567+0000","last_became_peered":"2024-05-10T11:07:47.212567+0000","last_unstale":"2024-05-10T11:13:13.484975+0000","last_undegraded":"2024-05-10T11:13:13.484975+0000","last_fullsized":"2024-05-10T11:13:13.484975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:44:32.222012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.279315+0000","last_change":"2024-05-10T11:07:47.200973+0000","last_active":"2024-05-10T11:13:13.279315+0000","last_peered":"2024-05-10T11:13:13.279315+0000","last_clean":"2024-05-10T11:13:13.279315+0000","last_became_active":"2024-05-10T11:07:47.200499+0000","last_became_peered":"2024-05-10T11:07:47.200499+0000","last_unstale":"2024-05-10T11:13:13.279315+0000","last_undegraded":"2024-05-10T11:13:13.279315+0000","last_fullsized":"2024-05-10T11:13:13.279315+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:29:24.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132918+0000","last_change":"2024-05-10T11:08:22.042254+0000","last_active":"2024-05-10T11:13:13.132918+0000","last_peered":"2024-05-10T11:13:13.132918+0000","last_clean":"2024-05-10T11:13:13.132918+0000","last_became_active":"2024-05-10T11:08:22.041906+0000","last_became_peered":"2024-05-10T11:08:22.041906+0000","last_unstale":"2024-05-10T11:13:13.132918+0000","last_undegraded":"2024-05-10T11:13:13.132918+0000","last_fullsized":"2024-05-10T11:13:13.132918+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:55:27.280478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.131809+0000","last_change":"2024-05-10T11:08:00.897746+0000","last_active":"2024-05-10T11:13:13.131809+0000","last_peered":"2024-05-10T11:13:13.131809+0000","last_clean":"2024-05-10T11:13:13.131809+0000","last_became_active":"2024-05-10T11:08:00.897428+0000","last_became_peered":"2024-05-10T11:08:00.897428+0000","last_unstale":"2024-05-10T11:13:13.131809+0000","last_undegraded":"2024-05-10T11:13:13.131809+0000","last_fullsized":"2024-05-10T11:13:13.131809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:38.846373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'14","reported_seq":306,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.590118+0000","last_change":"2024-05-10T11:07:42.139990+0000","last_active":"2024-05-10T11:13:17.590118+0000","last_peered":"2024-05-10T11:13:17.590118+0000","last_clean":"2024-05-10T11:13:17.590118+0000","last_became_active":"2024-05-10T11:07:42.139612+0000","last_became_peered":"2024-05-10T11:07:42.139612+0000","last_unstale":"2024-05-10T11:13:17.590118+0000","last_undegraded":"2024-05-10T11:13:17.590118+0000","last_fullsized":"2024-05-10T11:13:17.590118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:48:29.590570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":244,"num_read_kb":244,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129957+0000","last_change":"2024-05-10T11:07:47.214263+0000","last_active":"2024-05-10T11:13:13.129957+0000","last_peered":"2024-05-10T11:13:13.129957+0000","last_clean":"2024-05-10T11:13:13.129957+0000","last_became_active":"2024-05-10T11:07:47.213946+0000","last_became_peered":"2024-05-10T11:07:47.213946+0000","last_unstale":"2024-05-10T11:13:13.129957+0000","last_undegraded":"2024-05-10T11:13:13.129957+0000","last_fullsized":"2024-05-10T11:13:13.129957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:54:31.015838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.469566+0000","last_change":"2024-05-10T11:08:00.890571+0000","last_active":"2024-05-10T11:13:13.469566+0000","last_peered":"2024-05-10T11:13:13.469566+0000","last_clean":"2024-05-10T11:13:13.469566+0000","last_became_active":"2024-05-10T11:08:00.889468+0000","last_became_peered":"2024-05-10T11:08:00.889468+0000","last_unstale":"2024-05-10T11:13:13.469566+0000","last_undegraded":"2024-05-10T11:13:13.469566+0000","last_fullsized":"2024-05-10T11:13:13.469566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:20:54.342656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.130184+0000","last_change":"2024-05-10T11:07:42.383396+0000","last_active":"2024-05-10T11:13:13.130184+0000","last_peered":"2024-05-10T11:13:13.130184+0000","last_clean":"2024-05-10T11:13:13.130184+0000","last_became_active":"2024-05-10T11:07:42.382787+0000","last_became_peered":"2024-05-10T11:07:42.382787+0000","last_unstale":"2024-05-10T11:13:13.130184+0000","last_undegraded":"2024-05-10T11:13:13.130184+0000","last_fullsized":"2024-05-10T11:13:13.130184+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:44:34.240119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201637+0000","last_change":"2024-05-10T11:07:47.216770+0000","last_active":"2024-05-10T11:13:13.201637+0000","last_peered":"2024-05-10T11:13:13.201637+0000","last_clean":"2024-05-10T11:13:13.201637+0000","last_became_active":"2024-05-10T11:07:47.216377+0000","last_became_peered":"2024-05-10T11:07:47.216377+0000","last_unstale":"2024-05-10T11:13:13.201637+0000","last_undegraded":"2024-05-10T11:13:13.201637+0000","last_fullsized":"2024-05-10T11:13:13.201637+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:59:22.993888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470458+0000","last_change":"2024-05-10T11:08:00.900614+0000","last_active":"2024-05-10T11:13:13.470458+0000","last_peered":"2024-05-10T11:13:13.470458+0000","last_clean":"2024-05-10T11:13:13.470458+0000","last_became_active":"2024-05-10T11:08:00.900162+0000","last_became_peered":"2024-05-10T11:08:00.900162+0000","last_unstale":"2024-05-10T11:13:13.470458+0000","last_undegraded":"2024-05-10T11:13:13.470458+0000","last_fullsized":"2024-05-10T11:13:13.470458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:42:56.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.282238+0000","last_change":"2024-05-10T11:07:42.127749+0000","last_active":"2024-05-10T11:13:13.282238+0000","last_peered":"2024-05-10T11:13:13.282238+0000","last_clean":"2024-05-10T11:13:13.282238+0000","last_became_active":"2024-05-10T11:07:42.127296+0000","last_became_peered":"2024-05-10T11:07:42.127296+0000","last_unstale":"2024-05-10T11:13:13.282238+0000","last_undegraded":"2024-05-10T11:13:13.282238+0000","last_fullsized":"2024-05-10T11:13:13.282238+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:37:37.720542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477062+0000","last_change":"2024-05-10T11:07:47.198971+0000","last_active":"2024-05-10T11:13:13.477062+0000","last_peered":"2024-05-10T11:13:13.477062+0000","last_clean":"2024-05-10T11:13:13.477062+0000","last_became_active":"2024-05-10T11:07:47.198510+0000","last_became_peered":"2024-05-10T11:07:47.198510+0000","last_unstale":"2024-05-10T11:13:13.477062+0000","last_undegraded":"2024-05-10T11:13:13.477062+0000","last_fullsized":"2024-05-10T11:13:13.477062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:17:24.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470356+0000","last_change":"2024-05-10T11:08:00.878749+0000","last_active":"2024-05-10T11:13:13.470356+0000","last_peered":"2024-05-10T11:13:13.470356+0000","last_clean":"2024-05-10T11:13:13.470356+0000","last_became_active":"2024-05-10T11:08:00.878271+0000","last_became_peered":"2024-05-10T11:08:00.878271+0000","last_unstale":"2024-05-10T11:13:13.470356+0000","last_undegraded":"2024-05-10T11:13:13.470356+0000","last_fullsized":"2024-05-10T11:13:13.470356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:39:28.558766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.282229+0000","last_change":"2024-05-10T11:07:42.129593+0000","last_active":"2024-05-10T11:13:13.282229+0000","last_peered":"2024-05-10T11:13:13.282229+0000","last_clean":"2024-05-10T11:13:13.282229+0000","last_became_active":"2024-05-10T11:07:42.129250+0000","last_became_peered":"2024-05-10T11:07:42.129250+0000","last_unstale":"2024-05-10T11:13:13.282229+0000","last_undegraded":"2024-05-10T11:13:13.282229+0000","last_fullsized":"2024-05-10T11:13:13.282229+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:22:55.541079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.278965+0000","last_change":"2024-05-10T11:08:22.244535+0000","last_active":"2024-05-10T11:13:13.278965+0000","last_peered":"2024-05-10T11:13:13.278965+0000","last_clean":"2024-05-10T11:13:13.278965+0000","last_became_active":"2024-05-10T11:08:22.243716+0000","last_became_peered":"2024-05-10T11:08:22.243716+0000","last_unstale":"2024-05-10T11:13:13.278965+0000","last_undegraded":"2024-05-10T11:13:13.278965+0000","last_fullsized":"2024-05-10T11:13:13.278965+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:11.718999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132765+0000","last_change":"2024-05-10T11:08:00.869806+0000","last_active":"2024-05-10T11:13:13.132765+0000","last_peered":"2024-05-10T11:13:13.132765+0000","last_clean":"2024-05-10T11:13:13.132765+0000","last_became_active":"2024-05-10T11:08:00.869355+0000","last_became_peered":"2024-05-10T11:08:00.869355+0000","last_unstale":"2024-05-10T11:13:13.132765+0000","last_undegraded":"2024-05-10T11:13:13.132765+0000","last_fullsized":"2024-05-10T11:13:13.132765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:36.043412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":314,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.574435+0000","last_change":"2024-05-10T11:09:23.076933+0000","last_active":"2024-05-10T11:13:17.574435+0000","last_peered":"2024-05-10T11:13:17.574435+0000","last_clean":"2024-05-10T11:13:17.574435+0000","last_became_active":"2024-05-10T11:09:23.073274+0000","last_became_peered":"2024-05-10T11:09:23.073274+0000","last_unstale":"2024-05-10T11:13:17.574435+0000","last_undegraded":"2024-05-10T11:13:17.574435+0000","last_fullsized":"2024-05-10T11:13:17.574435+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:26:13.923466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":243,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281830+0000","last_change":"2024-05-10T11:07:47.200873+0000","last_active":"2024-05-10T11:13:13.281830+0000","last_peered":"2024-05-10T11:13:13.281830+0000","last_clean":"2024-05-10T11:13:13.281830+0000","last_became_active":"2024-05-10T11:07:47.200436+0000","last_became_peered":"2024-05-10T11:07:47.200436+0000","last_unstale":"2024-05-10T11:13:13.281830+0000","last_undegraded":"2024-05-10T11:13:13.281830+0000","last_fullsized":"2024-05-10T11:13:13.281830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:38:49.588593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132349+0000","last_change":"2024-05-10T11:08:00.885432+0000","last_active":"2024-05-10T11:13:13.132349+0000","last_peered":"2024-05-10T11:13:13.132349+0000","last_clean":"2024-05-10T11:13:13.132349+0000","last_became_active":"2024-05-10T11:08:00.884987+0000","last_became_peered":"2024-05-10T11:08:00.884987+0000","last_unstale":"2024-05-10T11:13:13.132349+0000","last_undegraded":"2024-05-10T11:13:13.132349+0000","last_fullsized":"2024-05-10T11:13:13.132349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:41:24.082722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129458+0000","last_change":"2024-05-10T11:09:23.093122+0000","last_active":"2024-05-10T11:13:13.129458+0000","last_peered":"2024-05-10T11:13:13.129458+0000","last_clean":"2024-05-10T11:13:13.129458+0000","last_became_active":"2024-05-10T11:09:23.084485+0000","last_became_peered":"2024-05-10T11:09:23.084485+0000","last_unstale":"2024-05-10T11:13:13.129458+0000","last_undegraded":"2024-05-10T11:13:13.129458+0000","last_fullsized":"2024-05-10T11:13:13.129458+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:12:36.793572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.288102+0000","last_change":"2024-05-10T11:08:22.036515+0000","last_active":"2024-05-10T11:13:13.288102+0000","last_peered":"2024-05-10T11:13:13.288102+0000","last_clean":"2024-05-10T11:13:13.288102+0000","last_became_active":"2024-05-10T11:08:22.036072+0000","last_became_peered":"2024-05-10T11:08:22.036072+0000","last_unstale":"2024-05-10T11:13:13.288102+0000","last_undegraded":"2024-05-10T11:13:13.288102+0000","last_fullsized":"2024-05-10T11:13:13.288102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:08.734453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6],"acting":[5,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484893+0000","last_change":"2024-05-10T11:08:00.888673+0000","last_active":"2024-05-10T11:13:13.484893+0000","last_peered":"2024-05-10T11:13:13.484893+0000","last_clean":"2024-05-10T11:13:13.484893+0000","last_became_active":"2024-05-10T11:08:00.888302+0000","last_became_peered":"2024-05-10T11:08:00.888302+0000","last_unstale":"2024-05-10T11:13:13.484893+0000","last_undegraded":"2024-05-10T11:13:13.484893+0000","last_fullsized":"2024-05-10T11:13:13.484893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:15:32.276358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478224+0000","last_change":"2024-05-10T11:07:42.140993+0000","last_active":"2024-05-10T11:13:13.478224+0000","last_peered":"2024-05-10T11:13:13.478224+0000","last_clean":"2024-05-10T11:13:13.478224+0000","last_became_active":"2024-05-10T11:07:42.140501+0000","last_became_peered":"2024-05-10T11:07:42.140501+0000","last_unstale":"2024-05-10T11:13:13.478224+0000","last_undegraded":"2024-05-10T11:13:13.478224+0000","last_fullsized":"2024-05-10T11:13:13.478224+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:01:50.324507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129032+0000","last_change":"2024-05-10T11:07:47.212616+0000","last_active":"2024-05-10T11:13:13.129032+0000","last_peered":"2024-05-10T11:13:13.129032+0000","last_clean":"2024-05-10T11:13:13.129032+0000","last_became_active":"2024-05-10T11:07:47.211081+0000","last_became_peered":"2024-05-10T11:07:47.211081+0000","last_unstale":"2024-05-10T11:13:13.129032+0000","last_undegraded":"2024-05-10T11:13:13.129032+0000","last_fullsized":"2024-05-10T11:13:13.129032+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:03:54.404355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201692+0000","last_change":"2024-05-10T11:08:00.888782+0000","last_active":"2024-05-10T11:13:13.201692+0000","last_peered":"2024-05-10T11:13:13.201692+0000","last_clean":"2024-05-10T11:13:13.201692+0000","last_became_active":"2024-05-10T11:08:00.888366+0000","last_became_peered":"2024-05-10T11:08:00.888366+0000","last_unstale":"2024-05-10T11:13:13.201692+0000","last_undegraded":"2024-05-10T11:13:13.201692+0000","last_fullsized":"2024-05-10T11:13:13.201692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:37.040186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287656+0000","last_change":"2024-05-10T11:07:42.128677+0000","last_active":"2024-05-10T11:13:13.287656+0000","last_peered":"2024-05-10T11:13:13.287656+0000","last_clean":"2024-05-10T11:13:13.287656+0000","last_became_active":"2024-05-10T11:07:42.128330+0000","last_became_peered":"2024-05-10T11:07:42.128330+0000","last_unstale":"2024-05-10T11:13:13.287656+0000","last_undegraded":"2024-05-10T11:13:13.287656+0000","last_fullsized":"2024-05-10T11:13:13.287656+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:09:11.738485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.278426+0000","last_change":"2024-05-10T11:07:47.215257+0000","last_active":"2024-05-10T11:13:13.278426+0000","last_peered":"2024-05-10T11:13:13.278426+0000","last_clean":"2024-05-10T11:13:13.278426+0000","last_became_active":"2024-05-10T11:07:47.214682+0000","last_became_peered":"2024-05-10T11:07:47.214682+0000","last_unstale":"2024-05-10T11:13:13.278426+0000","last_undegraded":"2024-05-10T11:13:13.278426+0000","last_fullsized":"2024-05-10T11:13:13.278426+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:35:45.787737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.128926+0000","last_change":"2024-05-10T11:08:00.894446+0000","last_active":"2024-05-10T11:13:13.128926+0000","last_peered":"2024-05-10T11:13:13.128926+0000","last_clean":"2024-05-10T11:13:13.128926+0000","last_became_active":"2024-05-10T11:08:00.886473+0000","last_became_peered":"2024-05-10T11:08:00.886473+0000","last_unstale":"2024-05-10T11:13:13.128926+0000","last_undegraded":"2024-05-10T11:13:13.128926+0000","last_fullsized":"2024-05-10T11:13:13.128926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:40.980328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":540,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:18.414865+0000","last_change":"2024-05-10T11:07:42.138750+0000","last_active":"2024-05-10T11:13:18.414865+0000","last_peered":"2024-05-10T11:13:18.414865+0000","last_clean":"2024-05-10T11:13:18.414865+0000","last_became_active":"2024-05-10T11:07:42.138342+0000","last_became_peered":"2024-05-10T11:07:42.138342+0000","last_unstale":"2024-05-10T11:13:18.414865+0000","last_undegraded":"2024-05-10T11:13:18.414865+0000","last_fullsized":"2024-05-10T11:13:18.414865+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:19:45.791550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":486,"num_read_kb":486,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477124+0000","last_change":"2024-05-10T11:07:47.204604+0000","last_active":"2024-05-10T11:13:13.477124+0000","last_peered":"2024-05-10T11:13:13.477124+0000","last_clean":"2024-05-10T11:13:13.477124+0000","last_became_active":"2024-05-10T11:07:47.204261+0000","last_became_peered":"2024-05-10T11:07:47.204261+0000","last_unstale":"2024-05-10T11:13:13.477124+0000","last_undegraded":"2024-05-10T11:13:13.477124+0000","last_fullsized":"2024-05-10T11:13:13.477124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:55:46.409545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281665+0000","last_change":"2024-05-10T11:08:22.038090+0000","last_active":"2024-05-10T11:13:13.281665+0000","last_peered":"2024-05-10T11:13:13.281665+0000","last_clean":"2024-05-10T11:13:13.281665+0000","last_became_active":"2024-05-10T11:08:22.037490+0000","last_became_peered":"2024-05-10T11:08:22.037490+0000","last_unstale":"2024-05-10T11:13:13.281665+0000","last_undegraded":"2024-05-10T11:13:13.281665+0000","last_fullsized":"2024-05-10T11:13:13.281665+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:01:46.319401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470116+0000","last_change":"2024-05-10T11:07:42.128636+0000","last_active":"2024-05-10T11:13:13.470116+0000","last_peered":"2024-05-10T11:13:13.470116+0000","last_clean":"2024-05-10T11:13:13.470116+0000","last_became_active":"2024-05-10T11:07:42.128194+0000","last_became_peered":"2024-05-10T11:07:42.128194+0000","last_unstale":"2024-05-10T11:13:13.470116+0000","last_undegraded":"2024-05-10T11:13:13.470116+0000","last_fullsized":"2024-05-10T11:13:13.470116+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:51:54.279292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201709+0000","last_change":"2024-05-10T11:08:22.037807+0000","last_active":"2024-05-10T11:13:13.201709+0000","last_peered":"2024-05-10T11:13:13.201709+0000","last_clean":"2024-05-10T11:13:13.201709+0000","last_became_active":"2024-05-10T11:08:22.037337+0000","last_became_peered":"2024-05-10T11:08:22.037337+0000","last_unstale":"2024-05-10T11:13:13.201709+0000","last_undegraded":"2024-05-10T11:13:13.201709+0000","last_fullsized":"2024-05-10T11:13:13.201709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:50:06.481222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470697+0000","last_change":"2024-05-10T11:08:00.893684+0000","last_active":"2024-05-10T11:13:13.470697+0000","last_peered":"2024-05-10T11:13:13.470697+0000","last_clean":"2024-05-10T11:13:13.470697+0000","last_became_active":"2024-05-10T11:08:00.893334+0000","last_became_peered":"2024-05-10T11:08:00.893334+0000","last_unstale":"2024-05-10T11:13:13.470697+0000","last_undegraded":"2024-05-10T11:13:13.470697+0000","last_fullsized":"2024-05-10T11:13:13.470697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:58:04.812308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.288233+0000","last_change":"2024-05-10T11:07:42.382086+0000","last_active":"2024-05-10T11:13:13.288233+0000","last_peered":"2024-05-10T11:13:13.288233+0000","last_clean":"2024-05-10T11:13:13.288233+0000","last_became_active":"2024-05-10T11:07:42.381744+0000","last_became_peered":"2024-05-10T11:07:42.381744+0000","last_unstale":"2024-05-10T11:13:13.288233+0000","last_undegraded":"2024-05-10T11:13:13.288233+0000","last_fullsized":"2024-05-10T11:13:13.288233+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:15:10.893834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477087+0000","last_change":"2024-05-10T11:07:47.194683+0000","last_active":"2024-05-10T11:13:13.477087+0000","last_peered":"2024-05-10T11:13:13.477087+0000","last_clean":"2024-05-10T11:13:13.477087+0000","last_became_active":"2024-05-10T11:07:47.194319+0000","last_became_peered":"2024-05-10T11:07:47.194319+0000","last_unstale":"2024-05-10T11:13:13.477087+0000","last_undegraded":"2024-05-10T11:13:13.477087+0000","last_fullsized":"2024-05-10T11:13:13.477087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:41:48.437093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.278511+0000","last_change":"2024-05-10T11:08:00.889616+0000","last_active":"2024-05-10T11:13:13.278511+0000","last_peered":"2024-05-10T11:13:13.278511+0000","last_clean":"2024-05-10T11:13:13.278511+0000","last_became_active":"2024-05-10T11:08:00.882179+0000","last_became_peered":"2024-05-10T11:08:00.882179+0000","last_unstale":"2024-05-10T11:13:13.278511+0000","last_undegraded":"2024-05-10T11:13:13.278511+0000","last_fullsized":"2024-05-10T11:13:13.278511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:23:12.407861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'45","reported_seq":593,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:18.443507+0000","last_change":"2024-05-10T11:07:42.128796+0000","last_active":"2024-05-10T11:13:18.443507+0000","last_peered":"2024-05-10T11:13:18.443507+0000","last_clean":"2024-05-10T11:13:18.443507+0000","last_became_active":"2024-05-10T11:07:42.128402+0000","last_became_peered":"2024-05-10T11:07:42.128402+0000","last_unstale":"2024-05-10T11:13:18.443507+0000","last_undegraded":"2024-05-10T11:13:18.443507+0000","last_fullsized":"2024-05-10T11:13:18.443507+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:50:09.205102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72820,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":500,"num_read_kb":322,"num_write":47,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.282034+0000","last_change":"2024-05-10T11:07:47.188304+0000","last_active":"2024-05-10T11:13:13.282034+0000","last_peered":"2024-05-10T11:13:13.282034+0000","last_clean":"2024-05-10T11:13:13.282034+0000","last_became_active":"2024-05-10T11:07:47.187798+0000","last_became_peered":"2024-05-10T11:07:47.187798+0000","last_unstale":"2024-05-10T11:13:13.282034+0000","last_undegraded":"2024-05-10T11:13:13.282034+0000","last_fullsized":"2024-05-10T11:13:13.282034+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:50:21.059060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.288522+0000","last_change":"2024-05-10T11:08:00.886669+0000","last_active":"2024-05-10T11:13:13.288522+0000","last_peered":"2024-05-10T11:13:13.288522+0000","last_clean":"2024-05-10T11:13:13.288522+0000","last_became_active":"2024-05-10T11:08:00.886249+0000","last_became_peered":"2024-05-10T11:08:00.886249+0000","last_unstale":"2024-05-10T11:13:13.288522+0000","last_undegraded":"2024-05-10T11:13:13.288522+0000","last_fullsized":"2024-05-10T11:13:13.288522+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:10:07.707462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484691+0000","last_change":"2024-05-10T11:09:23.095352+0000","last_active":"2024-05-10T11:13:13.484691+0000","last_peered":"2024-05-10T11:13:13.484691+0000","last_clean":"2024-05-10T11:13:13.484691+0000","last_became_active":"2024-05-10T11:09:23.083116+0000","last_became_peered":"2024-05-10T11:09:23.083116+0000","last_unstale":"2024-05-10T11:13:13.484691+0000","last_undegraded":"2024-05-10T11:13:13.484691+0000","last_fullsized":"2024-05-10T11:13:13.484691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:24:28.945724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478001+0000","last_change":"2024-05-10T11:07:47.207657+0000","last_active":"2024-05-10T11:13:13.478001+0000","last_peered":"2024-05-10T11:13:13.478001+0000","last_clean":"2024-05-10T11:13:13.478001+0000","last_became_active":"2024-05-10T11:07:47.207317+0000","last_became_peered":"2024-05-10T11:07:47.207317+0000","last_unstale":"2024-05-10T11:13:13.478001+0000","last_undegraded":"2024-05-10T11:13:13.478001+0000","last_fullsized":"2024-05-10T11:13:13.478001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:07:33.961642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484711+0000","last_change":"2024-05-10T11:08:00.884076+0000","last_active":"2024-05-10T11:13:13.484711+0000","last_peered":"2024-05-10T11:13:13.484711+0000","last_clean":"2024-05-10T11:13:13.484711+0000","last_became_active":"2024-05-10T11:08:00.883670+0000","last_became_peered":"2024-05-10T11:08:00.883670+0000","last_unstale":"2024-05-10T11:13:13.484711+0000","last_undegraded":"2024-05-10T11:13:13.484711+0000","last_fullsized":"2024-05-10T11:13:13.484711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:56.447836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287765+0000","last_change":"2024-05-10T11:07:42.135550+0000","last_active":"2024-05-10T11:13:13.287765+0000","last_peered":"2024-05-10T11:13:13.287765+0000","last_clean":"2024-05-10T11:13:13.287765+0000","last_became_active":"2024-05-10T11:07:42.135192+0000","last_became_peered":"2024-05-10T11:07:42.135192+0000","last_unstale":"2024-05-10T11:13:13.287765+0000","last_undegraded":"2024-05-10T11:13:13.287765+0000","last_fullsized":"2024-05-10T11:13:13.287765+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:17.100873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470885+0000","last_change":"2024-05-10T11:07:47.199529+0000","last_active":"2024-05-10T11:13:13.470885+0000","last_peered":"2024-05-10T11:13:13.470885+0000","last_clean":"2024-05-10T11:13:13.470885+0000","last_became_active":"2024-05-10T11:07:47.199070+0000","last_became_peered":"2024-05-10T11:07:47.199070+0000","last_unstale":"2024-05-10T11:13:13.470885+0000","last_undegraded":"2024-05-10T11:13:13.470885+0000","last_fullsized":"2024-05-10T11:13:13.470885+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:24.446639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.279675+0000","last_change":"2024-05-10T11:08:00.898607+0000","last_active":"2024-05-10T11:13:13.279675+0000","last_peered":"2024-05-10T11:13:13.279675+0000","last_clean":"2024-05-10T11:13:13.279675+0000","last_became_active":"2024-05-10T11:08:00.897768+0000","last_became_peered":"2024-05-10T11:08:00.897768+0000","last_unstale":"2024-05-10T11:13:13.279675+0000","last_undegraded":"2024-05-10T11:13:13.279675+0000","last_fullsized":"2024-05-10T11:13:13.279675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:24:42.125203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'10","reported_seq":307,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.546406+0000","last_change":"2024-05-10T11:07:42.126907+0000","last_active":"2024-05-10T11:13:17.546406+0000","last_peered":"2024-05-10T11:13:17.546406+0000","last_clean":"2024-05-10T11:13:17.546406+0000","last_became_active":"2024-05-10T11:07:42.126462+0000","last_became_peered":"2024-05-10T11:07:42.126462+0000","last_unstale":"2024-05-10T11:13:17.546406+0000","last_undegraded":"2024-05-10T11:13:17.546406+0000","last_fullsized":"2024-05-10T11:13:17.546406+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:18:53.971150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2212,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":250,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470856+0000","last_change":"2024-05-10T11:07:47.191888+0000","last_active":"2024-05-10T11:13:13.470856+0000","last_peered":"2024-05-10T11:13:13.470856+0000","last_clean":"2024-05-10T11:13:13.470856+0000","last_became_active":"2024-05-10T11:07:47.191557+0000","last_became_peered":"2024-05-10T11:07:47.191557+0000","last_unstale":"2024-05-10T11:13:13.470856+0000","last_undegraded":"2024-05-10T11:13:13.470856+0000","last_fullsized":"2024-05-10T11:13:13.470856+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:01:36.529493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478804+0000","last_change":"2024-05-10T11:08:00.894385+0000","last_active":"2024-05-10T11:13:13.478804+0000","last_peered":"2024-05-10T11:13:13.478804+0000","last_clean":"2024-05-10T11:13:13.478804+0000","last_became_active":"2024-05-10T11:08:00.893706+0000","last_became_peered":"2024-05-10T11:08:00.893706+0000","last_unstale":"2024-05-10T11:13:13.478804+0000","last_undegraded":"2024-05-10T11:13:13.478804+0000","last_fullsized":"2024-05-10T11:13:13.478804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:02.069878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281605+0000","last_change":"2024-05-10T11:07:42.385661+0000","last_active":"2024-05-10T11:13:13.281605+0000","last_peered":"2024-05-10T11:13:13.281605+0000","last_clean":"2024-05-10T11:13:13.281605+0000","last_became_active":"2024-05-10T11:07:42.385308+0000","last_became_peered":"2024-05-10T11:07:42.385308+0000","last_unstale":"2024-05-10T11:13:13.281605+0000","last_undegraded":"2024-05-10T11:13:13.281605+0000","last_fullsized":"2024-05-10T11:13:13.281605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:57:25.997730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287903+0000","last_change":"2024-05-10T11:07:47.191574+0000","last_active":"2024-05-10T11:13:13.287903+0000","last_peered":"2024-05-10T11:13:13.287903+0000","last_clean":"2024-05-10T11:13:13.287903+0000","last_became_active":"2024-05-10T11:07:47.191210+0000","last_became_peered":"2024-05-10T11:07:47.191210+0000","last_unstale":"2024-05-10T11:13:13.287903+0000","last_undegraded":"2024-05-10T11:13:13.287903+0000","last_fullsized":"2024-05-10T11:13:13.287903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:11:35.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287778+0000","last_change":"2024-05-10T11:08:00.890076+0000","last_active":"2024-05-10T11:13:13.287778+0000","last_peered":"2024-05-10T11:13:13.287778+0000","last_clean":"2024-05-10T11:13:13.287778+0000","last_became_active":"2024-05-10T11:08:00.889630+0000","last_became_peered":"2024-05-10T11:08:00.889630+0000","last_unstale":"2024-05-10T11:13:13.287778+0000","last_undegraded":"2024-05-10T11:13:13.287778+0000","last_fullsized":"2024-05-10T11:13:13.287778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:53:29.257873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484819+0000","last_change":"2024-05-10T11:09:23.074550+0000","last_active":"2024-05-10T11:13:13.484819+0000","last_peered":"2024-05-10T11:13:13.484819+0000","last_clean":"2024-05-10T11:13:13.484819+0000","last_became_active":"2024-05-10T11:09:23.067169+0000","last_became_peered":"2024-05-10T11:09:23.067169+0000","last_unstale":"2024-05-10T11:13:13.484819+0000","last_undegraded":"2024-05-10T11:13:13.484819+0000","last_fullsized":"2024-05-10T11:13:13.484819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:45:06.303003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484821+0000","last_change":"2024-05-10T11:07:47.192042+0000","last_active":"2024-05-10T11:13:13.484821+0000","last_peered":"2024-05-10T11:13:13.484821+0000","last_clean":"2024-05-10T11:13:13.484821+0000","last_became_active":"2024-05-10T11:07:47.191632+0000","last_became_peered":"2024-05-10T11:07:47.191632+0000","last_unstale":"2024-05-10T11:13:13.484821+0000","last_undegraded":"2024-05-10T11:13:13.484821+0000","last_fullsized":"2024-05-10T11:13:13.484821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:55:08.564841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281643+0000","last_change":"2024-05-10T11:08:22.244721+0000","last_active":"2024-05-10T11:13:13.281643+0000","last_peered":"2024-05-10T11:13:13.281643+0000","last_clean":"2024-05-10T11:13:13.281643+0000","last_became_active":"2024-05-10T11:08:22.244274+0000","last_became_peered":"2024-05-10T11:08:22.244274+0000","last_unstale":"2024-05-10T11:13:13.281643+0000","last_undegraded":"2024-05-10T11:13:13.281643+0000","last_fullsized":"2024-05-10T11:13:13.281643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:26.218071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.279764+0000","last_change":"2024-05-10T11:07:42.388358+0000","last_active":"2024-05-10T11:13:13.279764+0000","last_peered":"2024-05-10T11:13:13.279764+0000","last_clean":"2024-05-10T11:13:13.279764+0000","last_became_active":"2024-05-10T11:07:42.387806+0000","last_became_peered":"2024-05-10T11:07:42.387806+0000","last_unstale":"2024-05-10T11:13:13.279764+0000","last_undegraded":"2024-05-10T11:13:13.279764+0000","last_fullsized":"2024-05-10T11:13:13.279764+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:27:38.387332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129592+0000","last_change":"2024-05-10T11:07:47.213602+0000","last_active":"2024-05-10T11:13:13.129592+0000","last_peered":"2024-05-10T11:13:13.129592+0000","last_clean":"2024-05-10T11:13:13.129592+0000","last_became_active":"2024-05-10T11:07:47.213184+0000","last_became_peered":"2024-05-10T11:07:47.213184+0000","last_unstale":"2024-05-10T11:13:13.129592+0000","last_undegraded":"2024-05-10T11:13:13.129592+0000","last_fullsized":"2024-05-10T11:13:13.129592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T23:01:52.253279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478398+0000","last_change":"2024-05-10T11:08:00.889850+0000","last_active":"2024-05-10T11:13:13.478398+0000","last_peered":"2024-05-10T11:13:13.478398+0000","last_clean":"2024-05-10T11:13:13.478398+0000","last_became_active":"2024-05-10T11:08:00.888526+0000","last_became_peered":"2024-05-10T11:08:00.888526+0000","last_unstale":"2024-05-10T11:13:13.478398+0000","last_undegraded":"2024-05-10T11:13:13.478398+0000","last_fullsized":"2024-05-10T11:13:13.478398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:11:43.899628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'6","reported_seq":299,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.590902+0000","last_change":"2024-05-10T11:07:42.130650+0000","last_active":"2024-05-10T11:13:17.590902+0000","last_peered":"2024-05-10T11:13:17.590902+0000","last_clean":"2024-05-10T11:13:17.590902+0000","last_became_active":"2024-05-10T11:07:42.130259+0000","last_became_peered":"2024-05-10T11:07:42.130259+0000","last_unstale":"2024-05-10T11:13:17.590902+0000","last_undegraded":"2024-05-10T11:13:17.590902+0000","last_fullsized":"2024-05-10T11:13:17.590902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.870092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129319+0000","last_change":"2024-05-10T11:07:47.204159+0000","last_active":"2024-05-10T11:13:13.129319+0000","last_peered":"2024-05-10T11:13:13.129319+0000","last_clean":"2024-05-10T11:13:13.129319+0000","last_became_active":"2024-05-10T11:07:47.203723+0000","last_became_peered":"2024-05-10T11:07:47.203723+0000","last_unstale":"2024-05-10T11:13:13.129319+0000","last_undegraded":"2024-05-10T11:13:13.129319+0000","last_fullsized":"2024-05-10T11:13:13.129319+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:44:48.232321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477989+0000","last_change":"2024-05-10T11:08:00.918501+0000","last_active":"2024-05-10T11:13:13.477989+0000","last_peered":"2024-05-10T11:13:13.477989+0000","last_clean":"2024-05-10T11:13:13.477989+0000","last_became_active":"2024-05-10T11:08:00.918071+0000","last_became_peered":"2024-05-10T11:08:00.918071+0000","last_unstale":"2024-05-10T11:13:13.477989+0000","last_undegraded":"2024-05-10T11:13:13.477989+0000","last_fullsized":"2024-05-10T11:13:13.477989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:23:08.390953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287957+0000","last_change":"2024-05-10T11:08:22.036385+0000","last_active":"2024-05-10T11:13:13.287957+0000","last_peered":"2024-05-10T11:13:13.287957+0000","last_clean":"2024-05-10T11:13:13.287957+0000","last_became_active":"2024-05-10T11:08:22.035969+0000","last_became_peered":"2024-05-10T11:08:22.035969+0000","last_unstale":"2024-05-10T11:13:13.287957+0000","last_undegraded":"2024-05-10T11:13:13.287957+0000","last_fullsized":"2024-05-10T11:13:13.287957+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:14:24.069422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":299,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.545641+0000","last_change":"2024-05-10T11:07:42.381926+0000","last_active":"2024-05-10T11:13:17.545641+0000","last_peered":"2024-05-10T11:13:17.545641+0000","last_clean":"2024-05-10T11:13:17.545641+0000","last_became_active":"2024-05-10T11:07:42.381301+0000","last_became_peered":"2024-05-10T11:07:42.381301+0000","last_unstale":"2024-05-10T11:13:17.545641+0000","last_undegraded":"2024-05-10T11:13:17.545641+0000","last_fullsized":"2024-05-10T11:13:17.545641+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:48:32.291029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":244,"num_read_kb":244,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470352+0000","last_change":"2024-05-10T11:08:00.889959+0000","last_active":"2024-05-10T11:13:13.470352+0000","last_peered":"2024-05-10T11:13:13.470352+0000","last_clean":"2024-05-10T11:13:13.470352+0000","last_became_active":"2024-05-10T11:08:00.889514+0000","last_became_peered":"2024-05-10T11:08:00.889514+0000","last_unstale":"2024-05-10T11:13:13.470352+0000","last_undegraded":"2024-05-10T11:13:13.470352+0000","last_fullsized":"2024-05-10T11:13:13.470352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.557175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129670+0000","last_change":"2024-05-10T11:07:47.218025+0000","last_active":"2024-05-10T11:13:13.129670+0000","last_peered":"2024-05-10T11:13:13.129670+0000","last_clean":"2024-05-10T11:13:13.129670+0000","last_became_active":"2024-05-10T11:07:47.207089+0000","last_became_peered":"2024-05-10T11:07:47.207089+0000","last_unstale":"2024-05-10T11:13:13.129670+0000","last_undegraded":"2024-05-10T11:13:13.129670+0000","last_fullsized":"2024-05-10T11:13:13.129670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:15:50.363129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470565+0000","last_change":"2024-05-10T11:07:42.138182+0000","last_active":"2024-05-10T11:13:13.470565+0000","last_peered":"2024-05-10T11:13:13.470565+0000","last_clean":"2024-05-10T11:13:13.470565+0000","last_became_active":"2024-05-10T11:07:42.137734+0000","last_became_peered":"2024-05-10T11:07:42.137734+0000","last_unstale":"2024-05-10T11:13:13.470565+0000","last_undegraded":"2024-05-10T11:13:13.470565+0000","last_fullsized":"2024-05-10T11:13:13.470565+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:02:30.778878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'99","reported_seq":151,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:15.289134+0000","last_change":"2024-05-10T11:06:06.321257+0000","last_active":"2024-05-10T11:13:15.289134+0000","last_peered":"2024-05-10T11:13:15.289134+0000","last_clean":"2024-05-10T11:13:15.289134+0000","last_became_active":"2024-05-10T11:06:06.316928+0000","last_became_peered":"2024-05-10T11:06:06.316928+0000","last_unstale":"2024-05-10T11:13:15.289134+0000","last_undegraded":"2024-05-10T11:13:15.289134+0000","last_fullsized":"2024-05-10T11:13:15.289134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_clean_scrub_stamp":"2024-05-10T11:02:07.346111+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:32:43.670246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.288280+0000","last_change":"2024-05-10T11:08:00.918499+0000","last_active":"2024-05-10T11:13:13.288280+0000","last_peered":"2024-05-10T11:13:13.288280+0000","last_clean":"2024-05-10T11:13:13.288280+0000","last_became_active":"2024-05-10T11:08:00.917950+0000","last_became_peered":"2024-05-10T11:08:00.917950+0000","last_unstale":"2024-05-10T11:13:13.288280+0000","last_undegraded":"2024-05-10T11:13:13.288280+0000","last_fullsized":"2024-05-10T11:13:13.288280+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:07.591312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477904+0000","last_change":"2024-05-10T11:07:47.194968+0000","last_active":"2024-05-10T11:13:13.477904+0000","last_peered":"2024-05-10T11:13:13.477904+0000","last_clean":"2024-05-10T11:13:13.477904+0000","last_became_active":"2024-05-10T11:07:47.194645+0000","last_became_peered":"2024-05-10T11:07:47.194645+0000","last_unstale":"2024-05-10T11:13:13.477904+0000","last_undegraded":"2024-05-10T11:13:13.477904+0000","last_fullsized":"2024-05-10T11:13:13.477904+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:34.962163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470732+0000","last_change":"2024-05-10T11:07:42.126378+0000","last_active":"2024-05-10T11:13:13.470732+0000","last_peered":"2024-05-10T11:13:13.470732+0000","last_clean":"2024-05-10T11:13:13.470732+0000","last_became_active":"2024-05-10T11:07:42.125984+0000","last_became_peered":"2024-05-10T11:07:42.125984+0000","last_unstale":"2024-05-10T11:13:13.470732+0000","last_undegraded":"2024-05-10T11:13:13.470732+0000","last_fullsized":"2024-05-10T11:13:13.470732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:15:50.378737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484726+0000","last_change":"2024-05-10T11:08:00.872342+0000","last_active":"2024-05-10T11:13:13.484726+0000","last_peered":"2024-05-10T11:13:13.484726+0000","last_clean":"2024-05-10T11:13:13.484726+0000","last_became_active":"2024-05-10T11:08:00.871926+0000","last_became_peered":"2024-05-10T11:08:00.871926+0000","last_unstale":"2024-05-10T11:13:13.484726+0000","last_undegraded":"2024-05-10T11:13:13.484726+0000","last_fullsized":"2024-05-10T11:13:13.484726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:27:16.519752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.282072+0000","last_change":"2024-05-10T11:07:42.384236+0000","last_active":"2024-05-10T11:13:13.282072+0000","last_peered":"2024-05-10T11:13:13.282072+0000","last_clean":"2024-05-10T11:13:13.282072+0000","last_became_active":"2024-05-10T11:07:42.383908+0000","last_became_peered":"2024-05-10T11:07:42.383908+0000","last_unstale":"2024-05-10T11:13:13.282072+0000","last_undegraded":"2024-05-10T11:13:13.282072+0000","last_fullsized":"2024-05-10T11:13:13.282072+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:09:42.302506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478716+0000","last_change":"2024-05-10T11:07:47.201645+0000","last_active":"2024-05-10T11:13:13.478716+0000","last_peered":"2024-05-10T11:13:13.478716+0000","last_clean":"2024-05-10T11:13:13.478716+0000","last_became_active":"2024-05-10T11:07:47.201300+0000","last_became_peered":"2024-05-10T11:07:47.201300+0000","last_unstale":"2024-05-10T11:13:13.478716+0000","last_undegraded":"2024-05-10T11:13:13.478716+0000","last_fullsized":"2024-05-10T11:13:13.478716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:37.268530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129032+0000","last_change":"2024-05-10T11:08:00.917988+0000","last_active":"2024-05-10T11:13:13.129032+0000","last_peered":"2024-05-10T11:13:13.129032+0000","last_clean":"2024-05-10T11:13:13.129032+0000","last_became_active":"2024-05-10T11:08:00.917457+0000","last_became_peered":"2024-05-10T11:08:00.917457+0000","last_unstale":"2024-05-10T11:13:13.129032+0000","last_undegraded":"2024-05-10T11:13:13.129032+0000","last_fullsized":"2024-05-10T11:13:13.129032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:16:01.816917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'53","reported_seq":124,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.424005+0000","last_change":"2024-05-10T11:07:42.141005+0000","last_active":"2024-05-10T11:13:13.424005+0000","last_peered":"2024-05-10T11:13:13.424005+0000","last_clean":"2024-05-10T11:13:13.424005+0000","last_became_active":"2024-05-10T11:07:42.140634+0000","last_became_peered":"2024-05-10T11:07:42.140634+0000","last_unstale":"2024-05-10T11:13:13.424005+0000","last_undegraded":"2024-05-10T11:13:13.424005+0000","last_fullsized":"2024-05-10T11:13:13.424005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:00:32.029374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":94,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.130553+0000","last_change":"2024-05-10T11:07:47.206864+0000","last_active":"2024-05-10T11:13:13.130553+0000","last_peered":"2024-05-10T11:13:13.130553+0000","last_clean":"2024-05-10T11:13:13.130553+0000","last_became_active":"2024-05-10T11:07:47.206387+0000","last_became_peered":"2024-05-10T11:07:47.206387+0000","last_unstale":"2024-05-10T11:13:13.130553+0000","last_undegraded":"2024-05-10T11:13:13.130553+0000","last_fullsized":"2024-05-10T11:13:13.130553+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:12:35.242270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132847+0000","last_change":"2024-05-10T11:08:00.867872+0000","last_active":"2024-05-10T11:13:13.132847+0000","last_peered":"2024-05-10T11:13:13.132847+0000","last_clean":"2024-05-10T11:13:13.132847+0000","last_became_active":"2024-05-10T11:08:00.867461+0000","last_became_peered":"2024-05-10T11:08:00.867461+0000","last_unstale":"2024-05-10T11:13:13.132847+0000","last_undegraded":"2024-05-10T11:13:13.132847+0000","last_fullsized":"2024-05-10T11:13:13.132847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:29.617335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470710+0000","last_change":"2024-05-10T11:07:42.599520+0000","last_active":"2024-05-10T11:13:13.470710+0000","last_peered":"2024-05-10T11:13:13.470710+0000","last_clean":"2024-05-10T11:13:13.470710+0000","last_became_active":"2024-05-10T11:07:42.599209+0000","last_became_peered":"2024-05-10T11:07:42.599209+0000","last_unstale":"2024-05-10T11:13:13.470710+0000","last_undegraded":"2024-05-10T11:13:13.470710+0000","last_fullsized":"2024-05-10T11:13:13.470710+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:52.899718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201837+0000","last_change":"2024-05-10T11:07:47.218286+0000","last_active":"2024-05-10T11:13:13.201837+0000","last_peered":"2024-05-10T11:13:13.201837+0000","last_clean":"2024-05-10T11:13:13.201837+0000","last_became_active":"2024-05-10T11:07:47.217960+0000","last_became_peered":"2024-05-10T11:07:47.217960+0000","last_unstale":"2024-05-10T11:13:13.201837+0000","last_undegraded":"2024-05-10T11:13:13.201837+0000","last_fullsized":"2024-05-10T11:13:13.201837+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:11:37.208359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.471251+0000","last_change":"2024-05-10T11:08:00.919578+0000","last_active":"2024-05-10T11:13:13.471251+0000","last_peered":"2024-05-10T11:13:13.471251+0000","last_clean":"2024-05-10T11:13:13.471251+0000","last_became_active":"2024-05-10T11:08:00.918546+0000","last_became_peered":"2024-05-10T11:08:00.918546+0000","last_unstale":"2024-05-10T11:13:13.471251+0000","last_undegraded":"2024-05-10T11:13:13.471251+0000","last_fullsized":"2024-05-10T11:13:13.471251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:39:05.062210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.476737+0000","last_change":"2024-05-10T11:07:42.384577+0000","last_active":"2024-05-10T11:13:13.476737+0000","last_peered":"2024-05-10T11:13:13.476737+0000","last_clean":"2024-05-10T11:13:13.476737+0000","last_became_active":"2024-05-10T11:07:42.384301+0000","last_became_peered":"2024-05-10T11:07:42.384301+0000","last_unstale":"2024-05-10T11:13:13.476737+0000","last_undegraded":"2024-05-10T11:13:13.476737+0000","last_fullsized":"2024-05-10T11:13:13.476737+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:11:03.359490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281603+0000","last_change":"2024-05-10T11:07:47.189708+0000","last_active":"2024-05-10T11:13:13.281603+0000","last_peered":"2024-05-10T11:13:13.281603+0000","last_clean":"2024-05-10T11:13:13.281603+0000","last_became_active":"2024-05-10T11:07:47.189367+0000","last_became_peered":"2024-05-10T11:07:47.189367+0000","last_unstale":"2024-05-10T11:13:13.281603+0000","last_undegraded":"2024-05-10T11:13:13.281603+0000","last_fullsized":"2024-05-10T11:13:13.281603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:25.410477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.279787+0000","last_change":"2024-05-10T11:08:00.899359+0000","last_active":"2024-05-10T11:13:13.279787+0000","last_peered":"2024-05-10T11:13:13.279787+0000","last_clean":"2024-05-10T11:13:13.279787+0000","last_became_active":"2024-05-10T11:08:00.898924+0000","last_became_peered":"2024-05-10T11:08:00.898924+0000","last_unstale":"2024-05-10T11:13:13.279787+0000","last_undegraded":"2024-05-10T11:13:13.279787+0000","last_fullsized":"2024-05-10T11:13:13.279787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:14:17.427245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129168+0000","last_change":"2024-05-10T11:07:42.140262+0000","last_active":"2024-05-10T11:13:13.129168+0000","last_peered":"2024-05-10T11:13:13.129168+0000","last_clean":"2024-05-10T11:13:13.129168+0000","last_became_active":"2024-05-10T11:07:42.138853+0000","last_became_peered":"2024-05-10T11:07:42.138853+0000","last_unstale":"2024-05-10T11:13:13.129168+0000","last_undegraded":"2024-05-10T11:13:13.129168+0000","last_fullsized":"2024-05-10T11:13:13.129168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:05:39.995237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484555+0000","last_change":"2024-05-10T11:07:47.203637+0000","last_active":"2024-05-10T11:13:13.484555+0000","last_peered":"2024-05-10T11:13:13.484555+0000","last_clean":"2024-05-10T11:13:13.484555+0000","last_became_active":"2024-05-10T11:07:47.203286+0000","last_became_peered":"2024-05-10T11:07:47.203286+0000","last_unstale":"2024-05-10T11:13:13.484555+0000","last_undegraded":"2024-05-10T11:13:13.484555+0000","last_fullsized":"2024-05-10T11:13:13.484555+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:27:46.924625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470388+0000","last_change":"2024-05-10T11:08:00.890082+0000","last_active":"2024-05-10T11:13:13.470388+0000","last_peered":"2024-05-10T11:13:13.470388+0000","last_clean":"2024-05-10T11:13:13.470388+0000","last_became_active":"2024-05-10T11:08:00.889673+0000","last_became_peered":"2024-05-10T11:08:00.889673+0000","last_unstale":"2024-05-10T11:13:13.470388+0000","last_undegraded":"2024-05-10T11:13:13.470388+0000","last_fullsized":"2024-05-10T11:13:13.470388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:21:31.167934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:14.010400+0000","last_change":"2024-05-10T11:07:42.130703+0000","last_active":"2024-05-10T11:13:14.010400+0000","last_peered":"2024-05-10T11:13:14.010400+0000","last_clean":"2024-05-10T11:13:14.010400+0000","last_became_active":"2024-05-10T11:07:42.130098+0000","last_became_peered":"2024-05-10T11:07:42.130098+0000","last_unstale":"2024-05-10T11:13:14.010400+0000","last_undegraded":"2024-05-10T11:13:14.010400+0000","last_fullsized":"2024-05-10T11:13:14.010400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:59:39.783151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129760+0000","last_change":"2024-05-10T11:07:47.220497+0000","last_active":"2024-05-10T11:13:13.129760+0000","last_peered":"2024-05-10T11:13:13.129760+0000","last_clean":"2024-05-10T11:13:13.129760+0000","last_became_active":"2024-05-10T11:07:47.219916+0000","last_became_peered":"2024-05-10T11:07:47.219916+0000","last_unstale":"2024-05-10T11:13:13.129760+0000","last_undegraded":"2024-05-10T11:13:13.129760+0000","last_fullsized":"2024-05-10T11:13:13.129760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:26:43.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.471013+0000","last_change":"2024-05-10T11:08:00.918258+0000","last_active":"2024-05-10T11:13:13.471013+0000","last_peered":"2024-05-10T11:13:13.471013+0000","last_clean":"2024-05-10T11:13:13.471013+0000","last_became_active":"2024-05-10T11:08:00.917808+0000","last_became_peered":"2024-05-10T11:08:00.917808+0000","last_unstale":"2024-05-10T11:13:13.471013+0000","last_undegraded":"2024-05-10T11:13:13.471013+0000","last_fullsized":"2024-05-10T11:13:13.471013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:14:08.867024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281805+0000","last_change":"2024-05-10T11:07:42.127661+0000","last_active":"2024-05-10T11:13:13.281805+0000","last_peered":"2024-05-10T11:13:13.281805+0000","last_clean":"2024-05-10T11:13:13.281805+0000","last_became_active":"2024-05-10T11:07:42.127256+0000","last_became_peered":"2024-05-10T11:07:42.127256+0000","last_unstale":"2024-05-10T11:13:13.281805+0000","last_undegraded":"2024-05-10T11:13:13.281805+0000","last_fullsized":"2024-05-10T11:13:13.281805+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:14:37.821350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.469988+0000","last_change":"2024-05-10T11:07:47.199589+0000","last_active":"2024-05-10T11:13:13.469988+0000","last_peered":"2024-05-10T11:13:13.469988+0000","last_clean":"2024-05-10T11:13:13.469988+0000","last_became_active":"2024-05-10T11:07:47.199145+0000","last_became_peered":"2024-05-10T11:07:47.199145+0000","last_unstale":"2024-05-10T11:13:13.469988+0000","last_undegraded":"2024-05-10T11:13:13.469988+0000","last_fullsized":"2024-05-10T11:13:13.469988+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:46.577543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477347+0000","last_change":"2024-05-10T11:09:23.085049+0000","last_active":"2024-05-10T11:13:13.477347+0000","last_peered":"2024-05-10T11:13:13.477347+0000","last_clean":"2024-05-10T11:13:13.477347+0000","last_became_active":"2024-05-10T11:09:23.084420+0000","last_became_peered":"2024-05-10T11:09:23.084420+0000","last_unstale":"2024-05-10T11:13:13.477347+0000","last_undegraded":"2024-05-10T11:13:13.477347+0000","last_fullsized":"2024-05-10T11:13:13.477347+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:32:33.346434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":300,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.413818+0000","last_change":"2024-05-10T11:07:42.599395+0000","last_active":"2024-05-10T11:13:17.413818+0000","last_peered":"2024-05-10T11:13:17.413818+0000","last_clean":"2024-05-10T11:13:17.413818+0000","last_became_active":"2024-05-10T11:07:42.598934+0000","last_became_peered":"2024-05-10T11:07:42.598934+0000","last_unstale":"2024-05-10T11:13:17.413818+0000","last_undegraded":"2024-05-10T11:13:17.413818+0000","last_fullsized":"2024-05-10T11:13:17.413818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:49:42.414789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":246,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470676+0000","last_change":"2024-05-10T11:07:47.192087+0000","last_active":"2024-05-10T11:13:13.470676+0000","last_peered":"2024-05-10T11:13:13.470676+0000","last_clean":"2024-05-10T11:13:13.470676+0000","last_became_active":"2024-05-10T11:07:47.191041+0000","last_became_peered":"2024-05-10T11:07:47.191041+0000","last_unstale":"2024-05-10T11:13:13.470676+0000","last_undegraded":"2024-05-10T11:13:13.470676+0000","last_fullsized":"2024-05-10T11:13:13.470676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:50:14.052863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201037+0000","last_change":"2024-05-10T11:09:23.080432+0000","last_active":"2024-05-10T11:13:13.201037+0000","last_peered":"2024-05-10T11:13:13.201037+0000","last_clean":"2024-05-10T11:13:13.201037+0000","last_became_active":"2024-05-10T11:09:23.074934+0000","last_became_peered":"2024-05-10T11:09:23.074934+0000","last_unstale":"2024-05-10T11:13:13.201037+0000","last_undegraded":"2024-05-10T11:13:13.201037+0000","last_fullsized":"2024-05-10T11:13:13.201037+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:12:14.613323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'41","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.414569+0000","last_change":"2024-05-10T11:07:42.130718+0000","last_active":"2024-05-10T11:13:13.414569+0000","last_peered":"2024-05-10T11:13:13.414569+0000","last_clean":"2024-05-10T11:13:13.414569+0000","last_became_active":"2024-05-10T11:07:42.130355+0000","last_became_peered":"2024-05-10T11:07:42.130355+0000","last_unstale":"2024-05-10T11:13:13.414569+0000","last_undegraded":"2024-05-10T11:13:13.414569+0000","last_fullsized":"2024-05-10T11:13:13.414569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:16:48.093146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470333+0000","last_change":"2024-05-10T11:07:47.213569+0000","last_active":"2024-05-10T11:13:13.470333+0000","last_peered":"2024-05-10T11:13:13.470333+0000","last_clean":"2024-05-10T11:13:13.470333+0000","last_became_active":"2024-05-10T11:07:47.213148+0000","last_became_peered":"2024-05-10T11:07:47.213148+0000","last_unstale":"2024-05-10T11:13:13.470333+0000","last_undegraded":"2024-05-10T11:13:13.470333+0000","last_fullsized":"2024-05-10T11:13:13.470333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:42:41.161274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287545+0000","last_change":"2024-05-10T11:08:00.870937+0000","last_active":"2024-05-10T11:13:13.287545+0000","last_peered":"2024-05-10T11:13:13.287545+0000","last_clean":"2024-05-10T11:13:13.287545+0000","last_became_active":"2024-05-10T11:08:00.869276+0000","last_became_peered":"2024-05-10T11:08:00.869276+0000","last_unstale":"2024-05-10T11:13:13.287545+0000","last_undegraded":"2024-05-10T11:13:13.287545+0000","last_fullsized":"2024-05-10T11:13:13.287545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:37.058875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478921+0000","last_change":"2024-05-10T11:07:42.130544+0000","last_active":"2024-05-10T11:13:13.478921+0000","last_peered":"2024-05-10T11:13:13.478921+0000","last_clean":"2024-05-10T11:13:13.478921+0000","last_became_active":"2024-05-10T11:07:42.128912+0000","last_became_peered":"2024-05-10T11:07:42.128912+0000","last_unstale":"2024-05-10T11:13:13.478921+0000","last_undegraded":"2024-05-10T11:13:13.478921+0000","last_fullsized":"2024-05-10T11:13:13.478921+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:22:23.521911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470271+0000","last_change":"2024-05-10T11:07:47.194346+0000","last_active":"2024-05-10T11:13:13.470271+0000","last_peered":"2024-05-10T11:13:13.470271+0000","last_clean":"2024-05-10T11:13:13.470271+0000","last_became_active":"2024-05-10T11:07:47.194019+0000","last_became_peered":"2024-05-10T11:07:47.194019+0000","last_unstale":"2024-05-10T11:13:13.470271+0000","last_undegraded":"2024-05-10T11:13:13.470271+0000","last_fullsized":"2024-05-10T11:13:13.470271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:33:05.262347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484623+0000","last_change":"2024-05-10T11:07:47.192001+0000","last_active":"2024-05-10T11:13:13.484623+0000","last_peered":"2024-05-10T11:13:13.484623+0000","last_clean":"2024-05-10T11:13:13.484623+0000","last_became_active":"2024-05-10T11:07:47.191668+0000","last_became_peered":"2024-05-10T11:07:47.191668+0000","last_unstale":"2024-05-10T11:13:13.484623+0000","last_undegraded":"2024-05-10T11:13:13.484623+0000","last_fullsized":"2024-05-10T11:13:13.484623+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:13:33.680899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132073+0000","last_change":"2024-05-10T11:07:47.192226+0000","last_active":"2024-05-10T11:13:13.132073+0000","last_peered":"2024-05-10T11:13:13.132073+0000","last_clean":"2024-05-10T11:13:13.132073+0000","last_became_active":"2024-05-10T11:07:47.191444+0000","last_became_peered":"2024-05-10T11:07:47.191444+0000","last_unstale":"2024-05-10T11:13:13.132073+0000","last_undegraded":"2024-05-10T11:13:13.132073+0000","last_fullsized":"2024-05-10T11:13:13.132073+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:08.348986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.131701+0000","last_change":"2024-05-10T11:07:47.210801+0000","last_active":"2024-05-10T11:13:13.131701+0000","last_peered":"2024-05-10T11:13:13.131701+0000","last_clean":"2024-05-10T11:13:13.131701+0000","last_became_active":"2024-05-10T11:07:47.210496+0000","last_became_peered":"2024-05-10T11:07:47.210496+0000","last_unstale":"2024-05-10T11:13:13.131701+0000","last_undegraded":"2024-05-10T11:13:13.131701+0000","last_fullsized":"2024-05-10T11:13:13.131701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:11:03.661457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201213+0000","last_change":"2024-05-10T11:07:47.211033+0000","last_active":"2024-05-10T11:13:13.201213+0000","last_peered":"2024-05-10T11:13:13.201213+0000","last_clean":"2024-05-10T11:13:13.201213+0000","last_became_active":"2024-05-10T11:07:47.206061+0000","last_became_peered":"2024-05-10T11:07:47.206061+0000","last_unstale":"2024-05-10T11:13:13.201213+0000","last_undegraded":"2024-05-10T11:13:13.201213+0000","last_fullsized":"2024-05-10T11:13:13.201213+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:04:05.878270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478582+0000","last_change":"2024-05-10T11:07:47.214966+0000","last_active":"2024-05-10T11:13:13.478582+0000","last_peered":"2024-05-10T11:13:13.478582+0000","last_clean":"2024-05-10T11:13:13.478582+0000","last_became_active":"2024-05-10T11:07:47.214563+0000","last_became_peered":"2024-05-10T11:07:47.214563+0000","last_unstale":"2024-05-10T11:13:13.478582+0000","last_undegraded":"2024-05-10T11:13:13.478582+0000","last_fullsized":"2024-05-10T11:13:13.478582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:29:01.428893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.485051+0000","last_change":"2024-05-10T11:07:47.196740+0000","last_active":"2024-05-10T11:13:13.485051+0000","last_peered":"2024-05-10T11:13:13.485051+0000","last_clean":"2024-05-10T11:13:13.485051+0000","last_became_active":"2024-05-10T11:07:47.196388+0000","last_became_peered":"2024-05-10T11:07:47.196388+0000","last_unstale":"2024-05-10T11:13:13.485051+0000","last_undegraded":"2024-05-10T11:13:13.485051+0000","last_fullsized":"2024-05-10T11:13:13.485051+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:23:09.055338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132107+0000","last_change":"2024-05-10T11:07:47.193568+0000","last_active":"2024-05-10T11:13:13.132107+0000","last_peered":"2024-05-10T11:13:13.132107+0000","last_clean":"2024-05-10T11:13:13.132107+0000","last_became_active":"2024-05-10T11:07:47.193202+0000","last_became_peered":"2024-05-10T11:07:47.193202+0000","last_unstale":"2024-05-10T11:13:13.132107+0000","last_undegraded":"2024-05-10T11:13:13.132107+0000","last_fullsized":"2024-05-10T11:13:13.132107+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:36:24.556266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.130958+0000","last_change":"2024-05-10T11:07:47.215344+0000","last_active":"2024-05-10T11:13:13.130958+0000","last_peered":"2024-05-10T11:13:13.130958+0000","last_clean":"2024-05-10T11:13:13.130958+0000","last_became_active":"2024-05-10T11:07:47.214896+0000","last_became_peered":"2024-05-10T11:07:47.214896+0000","last_unstale":"2024-05-10T11:13:13.130958+0000","last_undegraded":"2024-05-10T11:13:13.130958+0000","last_fullsized":"2024-05-10T11:13:13.130958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:23:07.845212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287656+0000","last_change":"2024-05-10T11:07:47.219047+0000","last_active":"2024-05-10T11:13:13.287656+0000","last_peered":"2024-05-10T11:13:13.287656+0000","last_clean":"2024-05-10T11:13:13.287656+0000","last_became_active":"2024-05-10T11:07:47.218778+0000","last_became_peered":"2024-05-10T11:07:47.218778+0000","last_unstale":"2024-05-10T11:13:13.287656+0000","last_undegraded":"2024-05-10T11:13:13.287656+0000","last_fullsized":"2024-05-10T11:13:13.287656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:02:48.945259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129317+0000","last_change":"2024-05-10T11:07:47.213412+0000","last_active":"2024-05-10T11:13:13.129317+0000","last_peered":"2024-05-10T11:13:13.129317+0000","last_clean":"2024-05-10T11:13:13.129317+0000","last_became_active":"2024-05-10T11:07:47.212963+0000","last_became_peered":"2024-05-10T11:07:47.212963+0000","last_unstale":"2024-05-10T11:13:13.129317+0000","last_undegraded":"2024-05-10T11:13:13.129317+0000","last_fullsized":"2024-05-10T11:13:13.129317+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:33:05.224184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.476853+0000","last_change":"2024-05-10T11:07:47.207998+0000","last_active":"2024-05-10T11:13:13.476853+0000","last_peered":"2024-05-10T11:13:13.476853+0000","last_clean":"2024-05-10T11:13:13.476853+0000","last_became_active":"2024-05-10T11:07:47.207298+0000","last_became_peered":"2024-05-10T11:07:47.207298+0000","last_unstale":"2024-05-10T11:13:13.476853+0000","last_undegraded":"2024-05-10T11:13:13.476853+0000","last_fullsized":"2024-05-10T11:13:13.476853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:59:14.001103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478208+0000","last_change":"2024-05-10T11:07:47.202216+0000","last_active":"2024-05-10T11:13:13.478208+0000","last_peered":"2024-05-10T11:13:13.478208+0000","last_clean":"2024-05-10T11:13:13.478208+0000","last_became_active":"2024-05-10T11:07:47.201867+0000","last_became_peered":"2024-05-10T11:07:47.201867+0000","last_unstale":"2024-05-10T11:13:13.478208+0000","last_undegraded":"2024-05-10T11:13:13.478208+0000","last_fullsized":"2024-05-10T11:13:13.478208+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:48:59.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484820+0000","last_change":"2024-05-10T11:07:47.194149+0000","last_active":"2024-05-10T11:13:13.484820+0000","last_peered":"2024-05-10T11:13:13.484820+0000","last_clean":"2024-05-10T11:13:13.484820+0000","last_became_active":"2024-05-10T11:07:47.193838+0000","last_became_peered":"2024-05-10T11:07:47.193838+0000","last_unstale":"2024-05-10T11:13:13.484820+0000","last_undegraded":"2024-05-10T11:13:13.484820+0000","last_fullsized":"2024-05-10T11:13:13.484820+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:46:44.181446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107385,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":607246,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2569,"num_read_kb":1410,"num_write":392,"num_write_kb":1483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":712,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":737280,"data_stored":1821738,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678099,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":532988,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874329,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32340,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712812,"statfs":{"total":95995035648,"available":95961919488,"internally_reserved":0,"allocated":1191936,"data_stored":1325367,"data_compressed":77418,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070558,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":929749,"data_compressed":54613,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266789,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":926009,"data_compressed":54613,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863463018,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":1131718,"data_compressed":71262,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626544,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1011712,"data_stored":528332,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574988,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":602182,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738515,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":529788,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":601712,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182529,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1052465,"data_compressed":60769,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986300,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":535043,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822775,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1114112,"data_stored":1256885,"data_compressed":77418,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":74493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135202,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":394694,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":207780,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-10T11:13:21.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-10T11:13:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:21 smithi076 ceph-mon[34152]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:13:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:21 smithi076 ceph-mon[34152]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:13:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:21 smithi076 ceph-mon[34152]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:13:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:21 smithi076 ceph-mon[34152]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:13:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:21 smithi076 ceph-mon[34152]: Updating smithi027:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:13:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:21 smithi076 ceph-mon[34152]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:13:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:21 smithi076 ceph-mon[34152]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:13:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:21 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:21 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:21 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:21 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:21.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.476+0000 7fad16600640 1 -- 172.21.15.27:0/2084113602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad10104e00 msgr2=0x7fad101051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:21.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.476+0000 7fad16600640 1 --2- 172.21.15.27:0/2084113602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad10104e00 0x7fad101051e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fad00009860 tx=0x7fad0002a340 comp rx=0 tx=0).stop 2024-05-10T11:13:21.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.476+0000 7fad16600640 1 -- 172.21.15.27:0/2084113602 shutdown_connections 2024-05-10T11:13:21.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.476+0000 7fad16600640 1 --2- 172.21.15.27:0/2084113602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad10109ed0 0x7fad10111aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.476+0000 7fad16600640 1 --2- 172.21.15.27:0/2084113602 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad101057b0 0x7fad101097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.476+0000 7fad16600640 1 --2- 172.21.15.27:0/2084113602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad10104e00 0x7fad101051e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.476+0000 7fad16600640 1 -- 172.21.15.27:0/2084113602 >> 172.21.15.27:0/2084113602 conn(0x7fad10100660 msgr2=0x7fad10102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:21.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.476+0000 7fad16600640 1 -- 172.21.15.27:0/2084113602 shutdown_connections 2024-05-10T11:13:21.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.476+0000 7fad16600640 1 -- 172.21.15.27:0/2084113602 wait complete. 2024-05-10T11:13:21.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.477+0000 7fad16600640 1 Processor -- start 2024-05-10T11:13:21.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.477+0000 7fad16600640 1 -- start start 2024-05-10T11:13:21.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.477+0000 7fad16600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad10104e00 0x7fad1019b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:21.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.477+0000 7fad16600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad10109ed0 0x7fad1019b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:21.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.478+0000 7fad16600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1019fd00 0x7fad101a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:21.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.478+0000 7fad16600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fad10114e30 con 0x7fad1019fd00 2024-05-10T11:13:21.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.478+0000 7fad16600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fad10114cb0 con 0x7fad10109ed0 2024-05-10T11:13:21.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.478+0000 7fad16600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fad10114fb0 con 0x7fad10104e00 2024-05-10T11:13:21.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.478+0000 7fad0ea00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad10109ed0 0x7fad1019b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:21.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.478+0000 7fad0f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad10104e00 0x7fad1019b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:21.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.478+0000 7fad0fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1019fd00 0x7fad101a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.478+0000 7fad0ea00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad10109ed0 0x7fad1019b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:38078/0 (socket says 172.21.15.27:38078) 2024-05-10T11:13:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.478+0000 7fad0ea00640 1 -- 172.21.15.27:0/2856637600 learned_addr learned my addr 172.21.15.27:0/2856637600 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.478+0000 7fad0f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad10104e00 0x7fad1019b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42320/0 (socket says 172.21.15.27:42320) 2024-05-10T11:13:21.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.478+0000 7fad0fe00640 1 -- 172.21.15.27:0/2856637600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad10104e00 msgr2=0x7fad1019b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:21.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.478+0000 7fad0fe00640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad10104e00 0x7fad1019b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.479+0000 7fad0fe00640 1 -- 172.21.15.27:0/2856637600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad10109ed0 msgr2=0x7fad1019b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:21.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.479+0000 7fad0fe00640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad10109ed0 0x7fad1019b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.479+0000 7fad0fe00640 1 -- 172.21.15.27:0/2856637600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad101a07f0 con 0x7fad1019fd00 2024-05-10T11:13:21.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.479+0000 7fad0ea00640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad10109ed0 0x7fad1019b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:21.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.479+0000 7fad0f400640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad10104e00 0x7fad1019b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:21.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.479+0000 7fad0fe00640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1019fd00 0x7fad101a01b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fad08009fd0 tx=0x7fad0800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:21.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.479+0000 7fad07e00640 1 -- 172.21.15.27:0/2856637600 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0800cd60 con 0x7fad1019fd00 2024-05-10T11:13:21.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.479+0000 7fad16600640 1 -- 172.21.15.27:0/2856637600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad101a0ae0 con 0x7fad1019fd00 2024-05-10T11:13:21.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.479+0000 7fad07e00640 1 -- 172.21.15.27:0/2856637600 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad08005020 con 0x7fad1019fd00 2024-05-10T11:13:21.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.479+0000 7fad16600640 1 -- 172.21.15.27:0/2856637600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad101a82a0 con 0x7fad1019fd00 2024-05-10T11:13:21.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.480+0000 7fad07e00640 1 -- 172.21.15.27:0/2856637600 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad08018670 con 0x7fad1019fd00 2024-05-10T11:13:21.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.481+0000 7fad16600640 1 -- 172.21.15.27:0/2856637600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad10069c20 con 0x7fad1019fd00 2024-05-10T11:13:21.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.483+0000 7fad07e00640 1 -- 172.21.15.27:0/2856637600 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7fad080054d0 con 0x7fad1019fd00 2024-05-10T11:13:21.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.485+0000 7fad07e00640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7facf40b2a80 0x7facf40b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:21.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.485+0000 7fad07e00640 1 -- 172.21.15.27:0/2856637600 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fad080e32f0 con 0x7fad1019fd00 2024-05-10T11:13:21.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.485+0000 7fad0f400640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7facf40b2a80 0x7facf40b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:21.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:21 smithi189 ceph-mon[34842]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:13:21.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:21 smithi189 ceph-mon[34842]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:13:21.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:21 smithi189 ceph-mon[34842]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:13:21.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:21 smithi189 ceph-mon[34842]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:13:21.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:21 smithi189 ceph-mon[34842]: Updating smithi027:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:13:21.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:21 smithi189 ceph-mon[34842]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:13:21.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:21 smithi189 ceph-mon[34842]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:13:21.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:21 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:21.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:21 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:21.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:21 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:21.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:21 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:21.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.486+0000 7fad0f400640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7facf40b2a80 0x7facf40b4f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad00009830 tx=0x7fad00030340 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:21.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.491+0000 7fad07e00640 1 -- 172.21.15.27:0/2856637600 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fad080aa650 con 0x7fad1019fd00 2024-05-10T11:13:21.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:21 smithi027 ceph-mon[30054]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:13:21.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:21 smithi027 ceph-mon[30054]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:13:21.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:21 smithi027 ceph-mon[30054]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:13:21.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:21 smithi027 ceph-mon[30054]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:13:21.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:21 smithi027 ceph-mon[30054]: Updating smithi027:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-10T11:13:21.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:21 smithi027 ceph-mon[30054]: Updating smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:13:21.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:21 smithi027 ceph-mon[30054]: Updating smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:13:21.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:21 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:21.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:21 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:21.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:21 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:21.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:21 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:21.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.710+0000 7fad16600640 1 -- 172.21.15.27:0/2856637600 --> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fad101057b0 con 0x7facf40b2a80 2024-05-10T11:13:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.740+0000 7fad07e00640 1 -- 172.21.15.27:0/2856637600 <== mgr.25220 v2:172.21.15.27:6800/1167199542 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339737 (secure 0 0 0) 0x7fad101057b0 con 0x7facf40b2a80 2024-05-10T11:13:21.745 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:13:21.747 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-10T11:13:21.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.745+0000 7fad16600640 1 -- 172.21.15.27:0/2856637600 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7facf40b2a80 msgr2=0x7facf40b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:21.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.745+0000 7fad16600640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7facf40b2a80 0x7facf40b4f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad00009830 tx=0x7fad00030340 comp rx=0 tx=0).stop 2024-05-10T11:13:21.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.745+0000 7fad16600640 1 -- 172.21.15.27:0/2856637600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1019fd00 msgr2=0x7fad101a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.745+0000 7fad16600640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1019fd00 0x7fad101a01b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fad08009fd0 tx=0x7fad0800ee50 comp rx=0 tx=0).stop 2024-05-10T11:13:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.745+0000 7fad16600640 1 -- 172.21.15.27:0/2856637600 shutdown_connections 2024-05-10T11:13:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.745+0000 7fad16600640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7facf40b2a80 0x7facf40b4f40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.745+0000 7fad16600640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1019fd00 0x7fad101a01b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.745+0000 7fad16600640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad10109ed0 0x7fad1019b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.745+0000 7fad16600640 1 --2- 172.21.15.27:0/2856637600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad10104e00 0x7fad1019b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.745+0000 7fad16600640 1 -- 172.21.15.27:0/2856637600 >> 172.21.15.27:0/2856637600 conn(0x7fad10100660 msgr2=0x7fad10101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.746+0000 7fad16600640 1 -- 172.21.15.27:0/2856637600 shutdown_connections 2024-05-10T11:13:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:21.746+0000 7fad16600640 1 -- 172.21.15.27:0/2856637600 wait complete. 2024-05-10T11:13:21.766 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-05-10T11:13:21.153436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1197848,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2713,"num_read_kb":1658,"num_write":529,"num_write_kb":3635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":2952552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":916,"ondisk_log_size":916,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361736,"kb_used_data":12560,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580088,"statfs":{"total":1151940427776,"available":1151570010112,"internally_reserved":0,"allocated":12861440,"data_stored":9952238,"data_compressed":608895,"data_compressed_allocated":823296,"data_compressed_original":4497408,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3302,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":210,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.974372"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.485180+0000","last_change":"2024-05-10T11:07:47.212921+0000","last_active":"2024-05-10T11:13:13.485180+0000","last_peered":"2024-05-10T11:13:13.485180+0000","last_clean":"2024-05-10T11:13:13.485180+0000","last_became_active":"2024-05-10T11:07:47.212558+0000","last_became_peered":"2024-05-10T11:07:47.212558+0000","last_unstale":"2024-05-10T11:13:13.485180+0000","last_undegraded":"2024-05-10T11:13:13.485180+0000","last_fullsized":"2024-05-10T11:13:13.485180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:47:20.415883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201806+0000","last_change":"2024-05-10T11:07:47.211186+0000","last_active":"2024-05-10T11:13:13.201806+0000","last_peered":"2024-05-10T11:13:13.201806+0000","last_clean":"2024-05-10T11:13:13.201806+0000","last_became_active":"2024-05-10T11:07:47.203487+0000","last_became_peered":"2024-05-10T11:07:47.203487+0000","last_unstale":"2024-05-10T11:13:13.201806+0000","last_undegraded":"2024-05-10T11:13:13.201806+0000","last_fullsized":"2024-05-10T11:13:13.201806+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:51:29.674632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201303+0000","last_change":"2024-05-10T11:07:47.202718+0000","last_active":"2024-05-10T11:13:13.201303+0000","last_peered":"2024-05-10T11:13:13.201303+0000","last_clean":"2024-05-10T11:13:13.201303+0000","last_became_active":"2024-05-10T11:07:47.202048+0000","last_became_peered":"2024-05-10T11:07:47.202048+0000","last_unstale":"2024-05-10T11:13:13.201303+0000","last_undegraded":"2024-05-10T11:13:13.201303+0000","last_fullsized":"2024-05-10T11:13:13.201303+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:01:41.156850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470810+0000","last_change":"2024-05-10T11:07:47.193519+0000","last_active":"2024-05-10T11:13:13.470810+0000","last_peered":"2024-05-10T11:13:13.470810+0000","last_clean":"2024-05-10T11:13:13.470810+0000","last_became_active":"2024-05-10T11:07:47.193200+0000","last_became_peered":"2024-05-10T11:07:47.193200+0000","last_unstale":"2024-05-10T11:13:13.470810+0000","last_undegraded":"2024-05-10T11:13:13.470810+0000","last_fullsized":"2024-05-10T11:13:13.470810+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:01.949453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281822+0000","last_change":"2024-05-10T11:07:47.207750+0000","last_active":"2024-05-10T11:13:13.281822+0000","last_peered":"2024-05-10T11:13:13.281822+0000","last_clean":"2024-05-10T11:13:13.281822+0000","last_became_active":"2024-05-10T11:07:47.207384+0000","last_became_peered":"2024-05-10T11:07:47.207384+0000","last_unstale":"2024-05-10T11:13:13.281822+0000","last_undegraded":"2024-05-10T11:13:13.281822+0000","last_fullsized":"2024-05-10T11:13:13.281822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:09.432766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129613+0000","last_change":"2024-05-10T11:08:22.245930+0000","last_active":"2024-05-10T11:13:13.129613+0000","last_peered":"2024-05-10T11:13:13.129613+0000","last_clean":"2024-05-10T11:13:13.129613+0000","last_became_active":"2024-05-10T11:08:22.245553+0000","last_became_peered":"2024-05-10T11:08:22.245553+0000","last_unstale":"2024-05-10T11:13:13.129613+0000","last_undegraded":"2024-05-10T11:13:13.129613+0000","last_fullsized":"2024-05-10T11:13:13.129613+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:47:31.485623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.469888+0000","last_change":"2024-05-10T11:07:47.196666+0000","last_active":"2024-05-10T11:13:13.469888+0000","last_peered":"2024-05-10T11:13:13.469888+0000","last_clean":"2024-05-10T11:13:13.469888+0000","last_became_active":"2024-05-10T11:07:47.196300+0000","last_became_peered":"2024-05-10T11:07:47.196300+0000","last_unstale":"2024-05-10T11:13:13.469888+0000","last_undegraded":"2024-05-10T11:13:13.469888+0000","last_fullsized":"2024-05-10T11:13:13.469888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:55:24.343385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.128863+0000","last_change":"2024-05-10T11:07:47.213048+0000","last_active":"2024-05-10T11:13:13.128863+0000","last_peered":"2024-05-10T11:13:13.128863+0000","last_clean":"2024-05-10T11:13:13.128863+0000","last_became_active":"2024-05-10T11:07:47.206359+0000","last_became_peered":"2024-05-10T11:07:47.206359+0000","last_unstale":"2024-05-10T11:13:13.128863+0000","last_undegraded":"2024-05-10T11:13:13.128863+0000","last_fullsized":"2024-05-10T11:13:13.128863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:32:56.307261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477171+0000","last_change":"2024-05-10T11:07:47.190945+0000","last_active":"2024-05-10T11:13:13.477171+0000","last_peered":"2024-05-10T11:13:13.477171+0000","last_clean":"2024-05-10T11:13:13.477171+0000","last_became_active":"2024-05-10T11:07:47.190631+0000","last_became_peered":"2024-05-10T11:07:47.190631+0000","last_unstale":"2024-05-10T11:13:13.477171+0000","last_undegraded":"2024-05-10T11:13:13.477171+0000","last_fullsized":"2024-05-10T11:13:13.477171+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:34:08.683912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.130034+0000","last_change":"2024-05-10T11:07:47.189540+0000","last_active":"2024-05-10T11:13:13.130034+0000","last_peered":"2024-05-10T11:13:13.130034+0000","last_clean":"2024-05-10T11:13:13.130034+0000","last_became_active":"2024-05-10T11:07:47.189185+0000","last_became_peered":"2024-05-10T11:07:47.189185+0000","last_unstale":"2024-05-10T11:13:13.130034+0000","last_undegraded":"2024-05-10T11:13:13.130034+0000","last_fullsized":"2024-05-10T11:13:13.130034+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:55:23.892826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470578+0000","last_change":"2024-05-10T11:07:47.195688+0000","last_active":"2024-05-10T11:13:13.470578+0000","last_peered":"2024-05-10T11:13:13.470578+0000","last_clean":"2024-05-10T11:13:13.470578+0000","last_became_active":"2024-05-10T11:07:47.195081+0000","last_became_peered":"2024-05-10T11:07:47.195081+0000","last_unstale":"2024-05-10T11:13:13.470578+0000","last_undegraded":"2024-05-10T11:13:13.470578+0000","last_fullsized":"2024-05-10T11:13:13.470578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:20:55.240827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470631+0000","last_change":"2024-05-10T11:07:47.193736+0000","last_active":"2024-05-10T11:13:13.470631+0000","last_peered":"2024-05-10T11:13:13.470631+0000","last_clean":"2024-05-10T11:13:13.470631+0000","last_became_active":"2024-05-10T11:07:47.193400+0000","last_became_peered":"2024-05-10T11:07:47.193400+0000","last_unstale":"2024-05-10T11:13:13.470631+0000","last_undegraded":"2024-05-10T11:13:13.470631+0000","last_fullsized":"2024-05-10T11:13:13.470631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:46:28.507330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478284+0000","last_change":"2024-05-10T11:08:22.042849+0000","last_active":"2024-05-10T11:13:13.478284+0000","last_peered":"2024-05-10T11:13:13.478284+0000","last_clean":"2024-05-10T11:13:13.478284+0000","last_became_active":"2024-05-10T11:08:22.042024+0000","last_became_peered":"2024-05-10T11:08:22.042024+0000","last_unstale":"2024-05-10T11:13:13.478284+0000","last_undegraded":"2024-05-10T11:13:13.478284+0000","last_fullsized":"2024-05-10T11:13:13.478284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:29:08.096541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201493+0000","last_change":"2024-05-10T11:07:47.214016+0000","last_active":"2024-05-10T11:13:13.201493+0000","last_peered":"2024-05-10T11:13:13.201493+0000","last_clean":"2024-05-10T11:13:13.201493+0000","last_became_active":"2024-05-10T11:07:47.209658+0000","last_became_peered":"2024-05-10T11:07:47.209658+0000","last_unstale":"2024-05-10T11:13:13.201493+0000","last_undegraded":"2024-05-10T11:13:13.201493+0000","last_fullsized":"2024-05-10T11:13:13.201493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:44.771221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201161+0000","last_change":"2024-05-10T11:07:47.210153+0000","last_active":"2024-05-10T11:13:13.201161+0000","last_peered":"2024-05-10T11:13:13.201161+0000","last_clean":"2024-05-10T11:13:13.201161+0000","last_became_active":"2024-05-10T11:07:47.208470+0000","last_became_peered":"2024-05-10T11:07:47.208470+0000","last_unstale":"2024-05-10T11:13:13.201161+0000","last_undegraded":"2024-05-10T11:13:13.201161+0000","last_fullsized":"2024-05-10T11:13:13.201161+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:23:03.614670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.476671+0000","last_change":"2024-05-10T11:07:47.210704+0000","last_active":"2024-05-10T11:13:13.476671+0000","last_peered":"2024-05-10T11:13:13.476671+0000","last_clean":"2024-05-10T11:13:13.476671+0000","last_became_active":"2024-05-10T11:07:47.210317+0000","last_became_peered":"2024-05-10T11:07:47.210317+0000","last_unstale":"2024-05-10T11:13:13.476671+0000","last_undegraded":"2024-05-10T11:13:13.476671+0000","last_fullsized":"2024-05-10T11:13:13.476671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:47:30.848731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484975+0000","last_change":"2024-05-10T11:07:47.213116+0000","last_active":"2024-05-10T11:13:13.484975+0000","last_peered":"2024-05-10T11:13:13.484975+0000","last_clean":"2024-05-10T11:13:13.484975+0000","last_became_active":"2024-05-10T11:07:47.212567+0000","last_became_peered":"2024-05-10T11:07:47.212567+0000","last_unstale":"2024-05-10T11:13:13.484975+0000","last_undegraded":"2024-05-10T11:13:13.484975+0000","last_fullsized":"2024-05-10T11:13:13.484975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:44:32.222012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.279315+0000","last_change":"2024-05-10T11:07:47.200973+0000","last_active":"2024-05-10T11:13:13.279315+0000","last_peered":"2024-05-10T11:13:13.279315+0000","last_clean":"2024-05-10T11:13:13.279315+0000","last_became_active":"2024-05-10T11:07:47.200499+0000","last_became_peered":"2024-05-10T11:07:47.200499+0000","last_unstale":"2024-05-10T11:13:13.279315+0000","last_undegraded":"2024-05-10T11:13:13.279315+0000","last_fullsized":"2024-05-10T11:13:13.279315+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:29:24.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132918+0000","last_change":"2024-05-10T11:08:22.042254+0000","last_active":"2024-05-10T11:13:13.132918+0000","last_peered":"2024-05-10T11:13:13.132918+0000","last_clean":"2024-05-10T11:13:13.132918+0000","last_became_active":"2024-05-10T11:08:22.041906+0000","last_became_peered":"2024-05-10T11:08:22.041906+0000","last_unstale":"2024-05-10T11:13:13.132918+0000","last_undegraded":"2024-05-10T11:13:13.132918+0000","last_fullsized":"2024-05-10T11:13:13.132918+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:55:27.280478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.131809+0000","last_change":"2024-05-10T11:08:00.897746+0000","last_active":"2024-05-10T11:13:13.131809+0000","last_peered":"2024-05-10T11:13:13.131809+0000","last_clean":"2024-05-10T11:13:13.131809+0000","last_became_active":"2024-05-10T11:08:00.897428+0000","last_became_peered":"2024-05-10T11:08:00.897428+0000","last_unstale":"2024-05-10T11:13:13.131809+0000","last_undegraded":"2024-05-10T11:13:13.131809+0000","last_fullsized":"2024-05-10T11:13:13.131809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:38.846373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'14","reported_seq":306,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.590118+0000","last_change":"2024-05-10T11:07:42.139990+0000","last_active":"2024-05-10T11:13:17.590118+0000","last_peered":"2024-05-10T11:13:17.590118+0000","last_clean":"2024-05-10T11:13:17.590118+0000","last_became_active":"2024-05-10T11:07:42.139612+0000","last_became_peered":"2024-05-10T11:07:42.139612+0000","last_unstale":"2024-05-10T11:13:17.590118+0000","last_undegraded":"2024-05-10T11:13:17.590118+0000","last_fullsized":"2024-05-10T11:13:17.590118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:48:29.590570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":244,"num_read_kb":244,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129957+0000","last_change":"2024-05-10T11:07:47.214263+0000","last_active":"2024-05-10T11:13:13.129957+0000","last_peered":"2024-05-10T11:13:13.129957+0000","last_clean":"2024-05-10T11:13:13.129957+0000","last_became_active":"2024-05-10T11:07:47.213946+0000","last_became_peered":"2024-05-10T11:07:47.213946+0000","last_unstale":"2024-05-10T11:13:13.129957+0000","last_undegraded":"2024-05-10T11:13:13.129957+0000","last_fullsized":"2024-05-10T11:13:13.129957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:54:31.015838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.469566+0000","last_change":"2024-05-10T11:08:00.890571+0000","last_active":"2024-05-10T11:13:13.469566+0000","last_peered":"2024-05-10T11:13:13.469566+0000","last_clean":"2024-05-10T11:13:13.469566+0000","last_became_active":"2024-05-10T11:08:00.889468+0000","last_became_peered":"2024-05-10T11:08:00.889468+0000","last_unstale":"2024-05-10T11:13:13.469566+0000","last_undegraded":"2024-05-10T11:13:13.469566+0000","last_fullsized":"2024-05-10T11:13:13.469566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:20:54.342656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.130184+0000","last_change":"2024-05-10T11:07:42.383396+0000","last_active":"2024-05-10T11:13:13.130184+0000","last_peered":"2024-05-10T11:13:13.130184+0000","last_clean":"2024-05-10T11:13:13.130184+0000","last_became_active":"2024-05-10T11:07:42.382787+0000","last_became_peered":"2024-05-10T11:07:42.382787+0000","last_unstale":"2024-05-10T11:13:13.130184+0000","last_undegraded":"2024-05-10T11:13:13.130184+0000","last_fullsized":"2024-05-10T11:13:13.130184+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:44:34.240119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201637+0000","last_change":"2024-05-10T11:07:47.216770+0000","last_active":"2024-05-10T11:13:13.201637+0000","last_peered":"2024-05-10T11:13:13.201637+0000","last_clean":"2024-05-10T11:13:13.201637+0000","last_became_active":"2024-05-10T11:07:47.216377+0000","last_became_peered":"2024-05-10T11:07:47.216377+0000","last_unstale":"2024-05-10T11:13:13.201637+0000","last_undegraded":"2024-05-10T11:13:13.201637+0000","last_fullsized":"2024-05-10T11:13:13.201637+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:59:22.993888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470458+0000","last_change":"2024-05-10T11:08:00.900614+0000","last_active":"2024-05-10T11:13:13.470458+0000","last_peered":"2024-05-10T11:13:13.470458+0000","last_clean":"2024-05-10T11:13:13.470458+0000","last_became_active":"2024-05-10T11:08:00.900162+0000","last_became_peered":"2024-05-10T11:08:00.900162+0000","last_unstale":"2024-05-10T11:13:13.470458+0000","last_undegraded":"2024-05-10T11:13:13.470458+0000","last_fullsized":"2024-05-10T11:13:13.470458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:42:56.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.282238+0000","last_change":"2024-05-10T11:07:42.127749+0000","last_active":"2024-05-10T11:13:13.282238+0000","last_peered":"2024-05-10T11:13:13.282238+0000","last_clean":"2024-05-10T11:13:13.282238+0000","last_became_active":"2024-05-10T11:07:42.127296+0000","last_became_peered":"2024-05-10T11:07:42.127296+0000","last_unstale":"2024-05-10T11:13:13.282238+0000","last_undegraded":"2024-05-10T11:13:13.282238+0000","last_fullsized":"2024-05-10T11:13:13.282238+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:37:37.720542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477062+0000","last_change":"2024-05-10T11:07:47.198971+0000","last_active":"2024-05-10T11:13:13.477062+0000","last_peered":"2024-05-10T11:13:13.477062+0000","last_clean":"2024-05-10T11:13:13.477062+0000","last_became_active":"2024-05-10T11:07:47.198510+0000","last_became_peered":"2024-05-10T11:07:47.198510+0000","last_unstale":"2024-05-10T11:13:13.477062+0000","last_undegraded":"2024-05-10T11:13:13.477062+0000","last_fullsized":"2024-05-10T11:13:13.477062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:17:24.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470356+0000","last_change":"2024-05-10T11:08:00.878749+0000","last_active":"2024-05-10T11:13:13.470356+0000","last_peered":"2024-05-10T11:13:13.470356+0000","last_clean":"2024-05-10T11:13:13.470356+0000","last_became_active":"2024-05-10T11:08:00.878271+0000","last_became_peered":"2024-05-10T11:08:00.878271+0000","last_unstale":"2024-05-10T11:13:13.470356+0000","last_undegraded":"2024-05-10T11:13:13.470356+0000","last_fullsized":"2024-05-10T11:13:13.470356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:39:28.558766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.282229+0000","last_change":"2024-05-10T11:07:42.129593+0000","last_active":"2024-05-10T11:13:13.282229+0000","last_peered":"2024-05-10T11:13:13.282229+0000","last_clean":"2024-05-10T11:13:13.282229+0000","last_became_active":"2024-05-10T11:07:42.129250+0000","last_became_peered":"2024-05-10T11:07:42.129250+0000","last_unstale":"2024-05-10T11:13:13.282229+0000","last_undegraded":"2024-05-10T11:13:13.282229+0000","last_fullsized":"2024-05-10T11:13:13.282229+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:22:55.541079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.278965+0000","last_change":"2024-05-10T11:08:22.244535+0000","last_active":"2024-05-10T11:13:13.278965+0000","last_peered":"2024-05-10T11:13:13.278965+0000","last_clean":"2024-05-10T11:13:13.278965+0000","last_became_active":"2024-05-10T11:08:22.243716+0000","last_became_peered":"2024-05-10T11:08:22.243716+0000","last_unstale":"2024-05-10T11:13:13.278965+0000","last_undegraded":"2024-05-10T11:13:13.278965+0000","last_fullsized":"2024-05-10T11:13:13.278965+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:11.718999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132765+0000","last_change":"2024-05-10T11:08:00.869806+0000","last_active":"2024-05-10T11:13:13.132765+0000","last_peered":"2024-05-10T11:13:13.132765+0000","last_clean":"2024-05-10T11:13:13.132765+0000","last_became_active":"2024-05-10T11:08:00.869355+0000","last_became_peered":"2024-05-10T11:08:00.869355+0000","last_unstale":"2024-05-10T11:13:13.132765+0000","last_undegraded":"2024-05-10T11:13:13.132765+0000","last_fullsized":"2024-05-10T11:13:13.132765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:36.043412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":314,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.574435+0000","last_change":"2024-05-10T11:09:23.076933+0000","last_active":"2024-05-10T11:13:17.574435+0000","last_peered":"2024-05-10T11:13:17.574435+0000","last_clean":"2024-05-10T11:13:17.574435+0000","last_became_active":"2024-05-10T11:09:23.073274+0000","last_became_peered":"2024-05-10T11:09:23.073274+0000","last_unstale":"2024-05-10T11:13:17.574435+0000","last_undegraded":"2024-05-10T11:13:17.574435+0000","last_fullsized":"2024-05-10T11:13:17.574435+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:26:13.923466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":243,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281830+0000","last_change":"2024-05-10T11:07:47.200873+0000","last_active":"2024-05-10T11:13:13.281830+0000","last_peered":"2024-05-10T11:13:13.281830+0000","last_clean":"2024-05-10T11:13:13.281830+0000","last_became_active":"2024-05-10T11:07:47.200436+0000","last_became_peered":"2024-05-10T11:07:47.200436+0000","last_unstale":"2024-05-10T11:13:13.281830+0000","last_undegraded":"2024-05-10T11:13:13.281830+0000","last_fullsized":"2024-05-10T11:13:13.281830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:38:49.588593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132349+0000","last_change":"2024-05-10T11:08:00.885432+0000","last_active":"2024-05-10T11:13:13.132349+0000","last_peered":"2024-05-10T11:13:13.132349+0000","last_clean":"2024-05-10T11:13:13.132349+0000","last_became_active":"2024-05-10T11:08:00.884987+0000","last_became_peered":"2024-05-10T11:08:00.884987+0000","last_unstale":"2024-05-10T11:13:13.132349+0000","last_undegraded":"2024-05-10T11:13:13.132349+0000","last_fullsized":"2024-05-10T11:13:13.132349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:41:24.082722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129458+0000","last_change":"2024-05-10T11:09:23.093122+0000","last_active":"2024-05-10T11:13:13.129458+0000","last_peered":"2024-05-10T11:13:13.129458+0000","last_clean":"2024-05-10T11:13:13.129458+0000","last_became_active":"2024-05-10T11:09:23.084485+0000","last_became_peered":"2024-05-10T11:09:23.084485+0000","last_unstale":"2024-05-10T11:13:13.129458+0000","last_undegraded":"2024-05-10T11:13:13.129458+0000","last_fullsized":"2024-05-10T11:13:13.129458+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:12:36.793572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.288102+0000","last_change":"2024-05-10T11:08:22.036515+0000","last_active":"2024-05-10T11:13:13.288102+0000","last_peered":"2024-05-10T11:13:13.288102+0000","last_clean":"2024-05-10T11:13:13.288102+0000","last_became_active":"2024-05-10T11:08:22.036072+0000","last_became_peered":"2024-05-10T11:08:22.036072+0000","last_unstale":"2024-05-10T11:13:13.288102+0000","last_undegraded":"2024-05-10T11:13:13.288102+0000","last_fullsized":"2024-05-10T11:13:13.288102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:08.734453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6],"acting":[5,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484893+0000","last_change":"2024-05-10T11:08:00.888673+0000","last_active":"2024-05-10T11:13:13.484893+0000","last_peered":"2024-05-10T11:13:13.484893+0000","last_clean":"2024-05-10T11:13:13.484893+0000","last_became_active":"2024-05-10T11:08:00.888302+0000","last_became_peered":"2024-05-10T11:08:00.888302+0000","last_unstale":"2024-05-10T11:13:13.484893+0000","last_undegraded":"2024-05-10T11:13:13.484893+0000","last_fullsized":"2024-05-10T11:13:13.484893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:15:32.276358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478224+0000","last_change":"2024-05-10T11:07:42.140993+0000","last_active":"2024-05-10T11:13:13.478224+0000","last_peered":"2024-05-10T11:13:13.478224+0000","last_clean":"2024-05-10T11:13:13.478224+0000","last_became_active":"2024-05-10T11:07:42.140501+0000","last_became_peered":"2024-05-10T11:07:42.140501+0000","last_unstale":"2024-05-10T11:13:13.478224+0000","last_undegraded":"2024-05-10T11:13:13.478224+0000","last_fullsized":"2024-05-10T11:13:13.478224+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:01:50.324507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129032+0000","last_change":"2024-05-10T11:07:47.212616+0000","last_active":"2024-05-10T11:13:13.129032+0000","last_peered":"2024-05-10T11:13:13.129032+0000","last_clean":"2024-05-10T11:13:13.129032+0000","last_became_active":"2024-05-10T11:07:47.211081+0000","last_became_peered":"2024-05-10T11:07:47.211081+0000","last_unstale":"2024-05-10T11:13:13.129032+0000","last_undegraded":"2024-05-10T11:13:13.129032+0000","last_fullsized":"2024-05-10T11:13:13.129032+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:03:54.404355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201692+0000","last_change":"2024-05-10T11:08:00.888782+0000","last_active":"2024-05-10T11:13:13.201692+0000","last_peered":"2024-05-10T11:13:13.201692+0000","last_clean":"2024-05-10T11:13:13.201692+0000","last_became_active":"2024-05-10T11:08:00.888366+0000","last_became_peered":"2024-05-10T11:08:00.888366+0000","last_unstale":"2024-05-10T11:13:13.201692+0000","last_undegraded":"2024-05-10T11:13:13.201692+0000","last_fullsized":"2024-05-10T11:13:13.201692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:37.040186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287656+0000","last_change":"2024-05-10T11:07:42.128677+0000","last_active":"2024-05-10T11:13:13.287656+0000","last_peered":"2024-05-10T11:13:13.287656+0000","last_clean":"2024-05-10T11:13:13.287656+0000","last_became_active":"2024-05-10T11:07:42.128330+0000","last_became_peered":"2024-05-10T11:07:42.128330+0000","last_unstale":"2024-05-10T11:13:13.287656+0000","last_undegraded":"2024-05-10T11:13:13.287656+0000","last_fullsized":"2024-05-10T11:13:13.287656+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:09:11.738485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.278426+0000","last_change":"2024-05-10T11:07:47.215257+0000","last_active":"2024-05-10T11:13:13.278426+0000","last_peered":"2024-05-10T11:13:13.278426+0000","last_clean":"2024-05-10T11:13:13.278426+0000","last_became_active":"2024-05-10T11:07:47.214682+0000","last_became_peered":"2024-05-10T11:07:47.214682+0000","last_unstale":"2024-05-10T11:13:13.278426+0000","last_undegraded":"2024-05-10T11:13:13.278426+0000","last_fullsized":"2024-05-10T11:13:13.278426+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:35:45.787737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.128926+0000","last_change":"2024-05-10T11:08:00.894446+0000","last_active":"2024-05-10T11:13:13.128926+0000","last_peered":"2024-05-10T11:13:13.128926+0000","last_clean":"2024-05-10T11:13:13.128926+0000","last_became_active":"2024-05-10T11:08:00.886473+0000","last_became_peered":"2024-05-10T11:08:00.886473+0000","last_unstale":"2024-05-10T11:13:13.128926+0000","last_undegraded":"2024-05-10T11:13:13.128926+0000","last_fullsized":"2024-05-10T11:13:13.128926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:40.980328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":540,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:18.414865+0000","last_change":"2024-05-10T11:07:42.138750+0000","last_active":"2024-05-10T11:13:18.414865+0000","last_peered":"2024-05-10T11:13:18.414865+0000","last_clean":"2024-05-10T11:13:18.414865+0000","last_became_active":"2024-05-10T11:07:42.138342+0000","last_became_peered":"2024-05-10T11:07:42.138342+0000","last_unstale":"2024-05-10T11:13:18.414865+0000","last_undegraded":"2024-05-10T11:13:18.414865+0000","last_fullsized":"2024-05-10T11:13:18.414865+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:19:45.791550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":486,"num_read_kb":486,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477124+0000","last_change":"2024-05-10T11:07:47.204604+0000","last_active":"2024-05-10T11:13:13.477124+0000","last_peered":"2024-05-10T11:13:13.477124+0000","last_clean":"2024-05-10T11:13:13.477124+0000","last_became_active":"2024-05-10T11:07:47.204261+0000","last_became_peered":"2024-05-10T11:07:47.204261+0000","last_unstale":"2024-05-10T11:13:13.477124+0000","last_undegraded":"2024-05-10T11:13:13.477124+0000","last_fullsized":"2024-05-10T11:13:13.477124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:55:46.409545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281665+0000","last_change":"2024-05-10T11:08:22.038090+0000","last_active":"2024-05-10T11:13:13.281665+0000","last_peered":"2024-05-10T11:13:13.281665+0000","last_clean":"2024-05-10T11:13:13.281665+0000","last_became_active":"2024-05-10T11:08:22.037490+0000","last_became_peered":"2024-05-10T11:08:22.037490+0000","last_unstale":"2024-05-10T11:13:13.281665+0000","last_undegraded":"2024-05-10T11:13:13.281665+0000","last_fullsized":"2024-05-10T11:13:13.281665+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:01:46.319401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470116+0000","last_change":"2024-05-10T11:07:42.128636+0000","last_active":"2024-05-10T11:13:13.470116+0000","last_peered":"2024-05-10T11:13:13.470116+0000","last_clean":"2024-05-10T11:13:13.470116+0000","last_became_active":"2024-05-10T11:07:42.128194+0000","last_became_peered":"2024-05-10T11:07:42.128194+0000","last_unstale":"2024-05-10T11:13:13.470116+0000","last_undegraded":"2024-05-10T11:13:13.470116+0000","last_fullsized":"2024-05-10T11:13:13.470116+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:51:54.279292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201709+0000","last_change":"2024-05-10T11:08:22.037807+0000","last_active":"2024-05-10T11:13:13.201709+0000","last_peered":"2024-05-10T11:13:13.201709+0000","last_clean":"2024-05-10T11:13:13.201709+0000","last_became_active":"2024-05-10T11:08:22.037337+0000","last_became_peered":"2024-05-10T11:08:22.037337+0000","last_unstale":"2024-05-10T11:13:13.201709+0000","last_undegraded":"2024-05-10T11:13:13.201709+0000","last_fullsized":"2024-05-10T11:13:13.201709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:50:06.481222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470697+0000","last_change":"2024-05-10T11:08:00.893684+0000","last_active":"2024-05-10T11:13:13.470697+0000","last_peered":"2024-05-10T11:13:13.470697+0000","last_clean":"2024-05-10T11:13:13.470697+0000","last_became_active":"2024-05-10T11:08:00.893334+0000","last_became_peered":"2024-05-10T11:08:00.893334+0000","last_unstale":"2024-05-10T11:13:13.470697+0000","last_undegraded":"2024-05-10T11:13:13.470697+0000","last_fullsized":"2024-05-10T11:13:13.470697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:58:04.812308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.288233+0000","last_change":"2024-05-10T11:07:42.382086+0000","last_active":"2024-05-10T11:13:13.288233+0000","last_peered":"2024-05-10T11:13:13.288233+0000","last_clean":"2024-05-10T11:13:13.288233+0000","last_became_active":"2024-05-10T11:07:42.381744+0000","last_became_peered":"2024-05-10T11:07:42.381744+0000","last_unstale":"2024-05-10T11:13:13.288233+0000","last_undegraded":"2024-05-10T11:13:13.288233+0000","last_fullsized":"2024-05-10T11:13:13.288233+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:15:10.893834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477087+0000","last_change":"2024-05-10T11:07:47.194683+0000","last_active":"2024-05-10T11:13:13.477087+0000","last_peered":"2024-05-10T11:13:13.477087+0000","last_clean":"2024-05-10T11:13:13.477087+0000","last_became_active":"2024-05-10T11:07:47.194319+0000","last_became_peered":"2024-05-10T11:07:47.194319+0000","last_unstale":"2024-05-10T11:13:13.477087+0000","last_undegraded":"2024-05-10T11:13:13.477087+0000","last_fullsized":"2024-05-10T11:13:13.477087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:41:48.437093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.278511+0000","last_change":"2024-05-10T11:08:00.889616+0000","last_active":"2024-05-10T11:13:13.278511+0000","last_peered":"2024-05-10T11:13:13.278511+0000","last_clean":"2024-05-10T11:13:13.278511+0000","last_became_active":"2024-05-10T11:08:00.882179+0000","last_became_peered":"2024-05-10T11:08:00.882179+0000","last_unstale":"2024-05-10T11:13:13.278511+0000","last_undegraded":"2024-05-10T11:13:13.278511+0000","last_fullsized":"2024-05-10T11:13:13.278511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:23:12.407861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'45","reported_seq":593,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:18.443507+0000","last_change":"2024-05-10T11:07:42.128796+0000","last_active":"2024-05-10T11:13:18.443507+0000","last_peered":"2024-05-10T11:13:18.443507+0000","last_clean":"2024-05-10T11:13:18.443507+0000","last_became_active":"2024-05-10T11:07:42.128402+0000","last_became_peered":"2024-05-10T11:07:42.128402+0000","last_unstale":"2024-05-10T11:13:18.443507+0000","last_undegraded":"2024-05-10T11:13:18.443507+0000","last_fullsized":"2024-05-10T11:13:18.443507+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:50:09.205102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72820,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":500,"num_read_kb":322,"num_write":47,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.282034+0000","last_change":"2024-05-10T11:07:47.188304+0000","last_active":"2024-05-10T11:13:13.282034+0000","last_peered":"2024-05-10T11:13:13.282034+0000","last_clean":"2024-05-10T11:13:13.282034+0000","last_became_active":"2024-05-10T11:07:47.187798+0000","last_became_peered":"2024-05-10T11:07:47.187798+0000","last_unstale":"2024-05-10T11:13:13.282034+0000","last_undegraded":"2024-05-10T11:13:13.282034+0000","last_fullsized":"2024-05-10T11:13:13.282034+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:50:21.059060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.288522+0000","last_change":"2024-05-10T11:08:00.886669+0000","last_active":"2024-05-10T11:13:13.288522+0000","last_peered":"2024-05-10T11:13:13.288522+0000","last_clean":"2024-05-10T11:13:13.288522+0000","last_became_active":"2024-05-10T11:08:00.886249+0000","last_became_peered":"2024-05-10T11:08:00.886249+0000","last_unstale":"2024-05-10T11:13:13.288522+0000","last_undegraded":"2024-05-10T11:13:13.288522+0000","last_fullsized":"2024-05-10T11:13:13.288522+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:10:07.707462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484691+0000","last_change":"2024-05-10T11:09:23.095352+0000","last_active":"2024-05-10T11:13:13.484691+0000","last_peered":"2024-05-10T11:13:13.484691+0000","last_clean":"2024-05-10T11:13:13.484691+0000","last_became_active":"2024-05-10T11:09:23.083116+0000","last_became_peered":"2024-05-10T11:09:23.083116+0000","last_unstale":"2024-05-10T11:13:13.484691+0000","last_undegraded":"2024-05-10T11:13:13.484691+0000","last_fullsized":"2024-05-10T11:13:13.484691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:24:28.945724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478001+0000","last_change":"2024-05-10T11:07:47.207657+0000","last_active":"2024-05-10T11:13:13.478001+0000","last_peered":"2024-05-10T11:13:13.478001+0000","last_clean":"2024-05-10T11:13:13.478001+0000","last_became_active":"2024-05-10T11:07:47.207317+0000","last_became_peered":"2024-05-10T11:07:47.207317+0000","last_unstale":"2024-05-10T11:13:13.478001+0000","last_undegraded":"2024-05-10T11:13:13.478001+0000","last_fullsized":"2024-05-10T11:13:13.478001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:07:33.961642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484711+0000","last_change":"2024-05-10T11:08:00.884076+0000","last_active":"2024-05-10T11:13:13.484711+0000","last_peered":"2024-05-10T11:13:13.484711+0000","last_clean":"2024-05-10T11:13:13.484711+0000","last_became_active":"2024-05-10T11:08:00.883670+0000","last_became_peered":"2024-05-10T11:08:00.883670+0000","last_unstale":"2024-05-10T11:13:13.484711+0000","last_undegraded":"2024-05-10T11:13:13.484711+0000","last_fullsized":"2024-05-10T11:13:13.484711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:56.447836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287765+0000","last_change":"2024-05-10T11:07:42.135550+0000","last_active":"2024-05-10T11:13:13.287765+0000","last_peered":"2024-05-10T11:13:13.287765+0000","last_clean":"2024-05-10T11:13:13.287765+0000","last_became_active":"2024-05-10T11:07:42.135192+0000","last_became_peered":"2024-05-10T11:07:42.135192+0000","last_unstale":"2024-05-10T11:13:13.287765+0000","last_undegraded":"2024-05-10T11:13:13.287765+0000","last_fullsized":"2024-05-10T11:13:13.287765+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:17.100873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470885+0000","last_change":"2024-05-10T11:07:47.199529+0000","last_active":"2024-05-10T11:13:13.470885+0000","last_peered":"2024-05-10T11:13:13.470885+0000","last_clean":"2024-05-10T11:13:13.470885+0000","last_became_active":"2024-05-10T11:07:47.199070+0000","last_became_peered":"2024-05-10T11:07:47.199070+0000","last_unstale":"2024-05-10T11:13:13.470885+0000","last_undegraded":"2024-05-10T11:13:13.470885+0000","last_fullsized":"2024-05-10T11:13:13.470885+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:24.446639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.279675+0000","last_change":"2024-05-10T11:08:00.898607+0000","last_active":"2024-05-10T11:13:13.279675+0000","last_peered":"2024-05-10T11:13:13.279675+0000","last_clean":"2024-05-10T11:13:13.279675+0000","last_became_active":"2024-05-10T11:08:00.897768+0000","last_became_peered":"2024-05-10T11:08:00.897768+0000","last_unstale":"2024-05-10T11:13:13.279675+0000","last_undegraded":"2024-05-10T11:13:13.279675+0000","last_fullsized":"2024-05-10T11:13:13.279675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:24:42.125203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'10","reported_seq":307,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.546406+0000","last_change":"2024-05-10T11:07:42.126907+0000","last_active":"2024-05-10T11:13:17.546406+0000","last_peered":"2024-05-10T11:13:17.546406+0000","last_clean":"2024-05-10T11:13:17.546406+0000","last_became_active":"2024-05-10T11:07:42.126462+0000","last_became_peered":"2024-05-10T11:07:42.126462+0000","last_unstale":"2024-05-10T11:13:17.546406+0000","last_undegraded":"2024-05-10T11:13:17.546406+0000","last_fullsized":"2024-05-10T11:13:17.546406+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:18:53.971150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2212,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":250,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470856+0000","last_change":"2024-05-10T11:07:47.191888+0000","last_active":"2024-05-10T11:13:13.470856+0000","last_peered":"2024-05-10T11:13:13.470856+0000","last_clean":"2024-05-10T11:13:13.470856+0000","last_became_active":"2024-05-10T11:07:47.191557+0000","last_became_peered":"2024-05-10T11:07:47.191557+0000","last_unstale":"2024-05-10T11:13:13.470856+0000","last_undegraded":"2024-05-10T11:13:13.470856+0000","last_fullsized":"2024-05-10T11:13:13.470856+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:01:36.529493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478804+0000","last_change":"2024-05-10T11:08:00.894385+0000","last_active":"2024-05-10T11:13:13.478804+0000","last_peered":"2024-05-10T11:13:13.478804+0000","last_clean":"2024-05-10T11:13:13.478804+0000","last_became_active":"2024-05-10T11:08:00.893706+0000","last_became_peered":"2024-05-10T11:08:00.893706+0000","last_unstale":"2024-05-10T11:13:13.478804+0000","last_undegraded":"2024-05-10T11:13:13.478804+0000","last_fullsized":"2024-05-10T11:13:13.478804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:02.069878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281605+0000","last_change":"2024-05-10T11:07:42.385661+0000","last_active":"2024-05-10T11:13:13.281605+0000","last_peered":"2024-05-10T11:13:13.281605+0000","last_clean":"2024-05-10T11:13:13.281605+0000","last_became_active":"2024-05-10T11:07:42.385308+0000","last_became_peered":"2024-05-10T11:07:42.385308+0000","last_unstale":"2024-05-10T11:13:13.281605+0000","last_undegraded":"2024-05-10T11:13:13.281605+0000","last_fullsized":"2024-05-10T11:13:13.281605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:57:25.997730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287903+0000","last_change":"2024-05-10T11:07:47.191574+0000","last_active":"2024-05-10T11:13:13.287903+0000","last_peered":"2024-05-10T11:13:13.287903+0000","last_clean":"2024-05-10T11:13:13.287903+0000","last_became_active":"2024-05-10T11:07:47.191210+0000","last_became_peered":"2024-05-10T11:07:47.191210+0000","last_unstale":"2024-05-10T11:13:13.287903+0000","last_undegraded":"2024-05-10T11:13:13.287903+0000","last_fullsized":"2024-05-10T11:13:13.287903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:11:35.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287778+0000","last_change":"2024-05-10T11:08:00.890076+0000","last_active":"2024-05-10T11:13:13.287778+0000","last_peered":"2024-05-10T11:13:13.287778+0000","last_clean":"2024-05-10T11:13:13.287778+0000","last_became_active":"2024-05-10T11:08:00.889630+0000","last_became_peered":"2024-05-10T11:08:00.889630+0000","last_unstale":"2024-05-10T11:13:13.287778+0000","last_undegraded":"2024-05-10T11:13:13.287778+0000","last_fullsized":"2024-05-10T11:13:13.287778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:53:29.257873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484819+0000","last_change":"2024-05-10T11:09:23.074550+0000","last_active":"2024-05-10T11:13:13.484819+0000","last_peered":"2024-05-10T11:13:13.484819+0000","last_clean":"2024-05-10T11:13:13.484819+0000","last_became_active":"2024-05-10T11:09:23.067169+0000","last_became_peered":"2024-05-10T11:09:23.067169+0000","last_unstale":"2024-05-10T11:13:13.484819+0000","last_undegraded":"2024-05-10T11:13:13.484819+0000","last_fullsized":"2024-05-10T11:13:13.484819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:45:06.303003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484821+0000","last_change":"2024-05-10T11:07:47.192042+0000","last_active":"2024-05-10T11:13:13.484821+0000","last_peered":"2024-05-10T11:13:13.484821+0000","last_clean":"2024-05-10T11:13:13.484821+0000","last_became_active":"2024-05-10T11:07:47.191632+0000","last_became_peered":"2024-05-10T11:07:47.191632+0000","last_unstale":"2024-05-10T11:13:13.484821+0000","last_undegraded":"2024-05-10T11:13:13.484821+0000","last_fullsized":"2024-05-10T11:13:13.484821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:55:08.564841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281643+0000","last_change":"2024-05-10T11:08:22.244721+0000","last_active":"2024-05-10T11:13:13.281643+0000","last_peered":"2024-05-10T11:13:13.281643+0000","last_clean":"2024-05-10T11:13:13.281643+0000","last_became_active":"2024-05-10T11:08:22.244274+0000","last_became_peered":"2024-05-10T11:08:22.244274+0000","last_unstale":"2024-05-10T11:13:13.281643+0000","last_undegraded":"2024-05-10T11:13:13.281643+0000","last_fullsized":"2024-05-10T11:13:13.281643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:26.218071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.279764+0000","last_change":"2024-05-10T11:07:42.388358+0000","last_active":"2024-05-10T11:13:13.279764+0000","last_peered":"2024-05-10T11:13:13.279764+0000","last_clean":"2024-05-10T11:13:13.279764+0000","last_became_active":"2024-05-10T11:07:42.387806+0000","last_became_peered":"2024-05-10T11:07:42.387806+0000","last_unstale":"2024-05-10T11:13:13.279764+0000","last_undegraded":"2024-05-10T11:13:13.279764+0000","last_fullsized":"2024-05-10T11:13:13.279764+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:27:38.387332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129592+0000","last_change":"2024-05-10T11:07:47.213602+0000","last_active":"2024-05-10T11:13:13.129592+0000","last_peered":"2024-05-10T11:13:13.129592+0000","last_clean":"2024-05-10T11:13:13.129592+0000","last_became_active":"2024-05-10T11:07:47.213184+0000","last_became_peered":"2024-05-10T11:07:47.213184+0000","last_unstale":"2024-05-10T11:13:13.129592+0000","last_undegraded":"2024-05-10T11:13:13.129592+0000","last_fullsized":"2024-05-10T11:13:13.129592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T23:01:52.253279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478398+0000","last_change":"2024-05-10T11:08:00.889850+0000","last_active":"2024-05-10T11:13:13.478398+0000","last_peered":"2024-05-10T11:13:13.478398+0000","last_clean":"2024-05-10T11:13:13.478398+0000","last_became_active":"2024-05-10T11:08:00.888526+0000","last_became_peered":"2024-05-10T11:08:00.888526+0000","last_unstale":"2024-05-10T11:13:13.478398+0000","last_undegraded":"2024-05-10T11:13:13.478398+0000","last_fullsized":"2024-05-10T11:13:13.478398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:11:43.899628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'6","reported_seq":299,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.590902+0000","last_change":"2024-05-10T11:07:42.130650+0000","last_active":"2024-05-10T11:13:17.590902+0000","last_peered":"2024-05-10T11:13:17.590902+0000","last_clean":"2024-05-10T11:13:17.590902+0000","last_became_active":"2024-05-10T11:07:42.130259+0000","last_became_peered":"2024-05-10T11:07:42.130259+0000","last_unstale":"2024-05-10T11:13:17.590902+0000","last_undegraded":"2024-05-10T11:13:17.590902+0000","last_fullsized":"2024-05-10T11:13:17.590902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.870092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129319+0000","last_change":"2024-05-10T11:07:47.204159+0000","last_active":"2024-05-10T11:13:13.129319+0000","last_peered":"2024-05-10T11:13:13.129319+0000","last_clean":"2024-05-10T11:13:13.129319+0000","last_became_active":"2024-05-10T11:07:47.203723+0000","last_became_peered":"2024-05-10T11:07:47.203723+0000","last_unstale":"2024-05-10T11:13:13.129319+0000","last_undegraded":"2024-05-10T11:13:13.129319+0000","last_fullsized":"2024-05-10T11:13:13.129319+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:44:48.232321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477989+0000","last_change":"2024-05-10T11:08:00.918501+0000","last_active":"2024-05-10T11:13:13.477989+0000","last_peered":"2024-05-10T11:13:13.477989+0000","last_clean":"2024-05-10T11:13:13.477989+0000","last_became_active":"2024-05-10T11:08:00.918071+0000","last_became_peered":"2024-05-10T11:08:00.918071+0000","last_unstale":"2024-05-10T11:13:13.477989+0000","last_undegraded":"2024-05-10T11:13:13.477989+0000","last_fullsized":"2024-05-10T11:13:13.477989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:23:08.390953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287957+0000","last_change":"2024-05-10T11:08:22.036385+0000","last_active":"2024-05-10T11:13:13.287957+0000","last_peered":"2024-05-10T11:13:13.287957+0000","last_clean":"2024-05-10T11:13:13.287957+0000","last_became_active":"2024-05-10T11:08:22.035969+0000","last_became_peered":"2024-05-10T11:08:22.035969+0000","last_unstale":"2024-05-10T11:13:13.287957+0000","last_undegraded":"2024-05-10T11:13:13.287957+0000","last_fullsized":"2024-05-10T11:13:13.287957+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:14:24.069422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":299,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.545641+0000","last_change":"2024-05-10T11:07:42.381926+0000","last_active":"2024-05-10T11:13:17.545641+0000","last_peered":"2024-05-10T11:13:17.545641+0000","last_clean":"2024-05-10T11:13:17.545641+0000","last_became_active":"2024-05-10T11:07:42.381301+0000","last_became_peered":"2024-05-10T11:07:42.381301+0000","last_unstale":"2024-05-10T11:13:17.545641+0000","last_undegraded":"2024-05-10T11:13:17.545641+0000","last_fullsized":"2024-05-10T11:13:17.545641+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:48:32.291029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":244,"num_read_kb":244,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470352+0000","last_change":"2024-05-10T11:08:00.889959+0000","last_active":"2024-05-10T11:13:13.470352+0000","last_peered":"2024-05-10T11:13:13.470352+0000","last_clean":"2024-05-10T11:13:13.470352+0000","last_became_active":"2024-05-10T11:08:00.889514+0000","last_became_peered":"2024-05-10T11:08:00.889514+0000","last_unstale":"2024-05-10T11:13:13.470352+0000","last_undegraded":"2024-05-10T11:13:13.470352+0000","last_fullsized":"2024-05-10T11:13:13.470352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.557175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129670+0000","last_change":"2024-05-10T11:07:47.218025+0000","last_active":"2024-05-10T11:13:13.129670+0000","last_peered":"2024-05-10T11:13:13.129670+0000","last_clean":"2024-05-10T11:13:13.129670+0000","last_became_active":"2024-05-10T11:07:47.207089+0000","last_became_peered":"2024-05-10T11:07:47.207089+0000","last_unstale":"2024-05-10T11:13:13.129670+0000","last_undegraded":"2024-05-10T11:13:13.129670+0000","last_fullsized":"2024-05-10T11:13:13.129670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:15:50.363129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470565+0000","last_change":"2024-05-10T11:07:42.138182+0000","last_active":"2024-05-10T11:13:13.470565+0000","last_peered":"2024-05-10T11:13:13.470565+0000","last_clean":"2024-05-10T11:13:13.470565+0000","last_became_active":"2024-05-10T11:07:42.137734+0000","last_became_peered":"2024-05-10T11:07:42.137734+0000","last_unstale":"2024-05-10T11:13:13.470565+0000","last_undegraded":"2024-05-10T11:13:13.470565+0000","last_fullsized":"2024-05-10T11:13:13.470565+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:02:30.778878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'99","reported_seq":151,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:15.289134+0000","last_change":"2024-05-10T11:06:06.321257+0000","last_active":"2024-05-10T11:13:15.289134+0000","last_peered":"2024-05-10T11:13:15.289134+0000","last_clean":"2024-05-10T11:13:15.289134+0000","last_became_active":"2024-05-10T11:06:06.316928+0000","last_became_peered":"2024-05-10T11:06:06.316928+0000","last_unstale":"2024-05-10T11:13:15.289134+0000","last_undegraded":"2024-05-10T11:13:15.289134+0000","last_fullsized":"2024-05-10T11:13:15.289134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_clean_scrub_stamp":"2024-05-10T11:02:07.346111+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:32:43.670246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.288280+0000","last_change":"2024-05-10T11:08:00.918499+0000","last_active":"2024-05-10T11:13:13.288280+0000","last_peered":"2024-05-10T11:13:13.288280+0000","last_clean":"2024-05-10T11:13:13.288280+0000","last_became_active":"2024-05-10T11:08:00.917950+0000","last_became_peered":"2024-05-10T11:08:00.917950+0000","last_unstale":"2024-05-10T11:13:13.288280+0000","last_undegraded":"2024-05-10T11:13:13.288280+0000","last_fullsized":"2024-05-10T11:13:13.288280+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:07.591312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477904+0000","last_change":"2024-05-10T11:07:47.194968+0000","last_active":"2024-05-10T11:13:13.477904+0000","last_peered":"2024-05-10T11:13:13.477904+0000","last_clean":"2024-05-10T11:13:13.477904+0000","last_became_active":"2024-05-10T11:07:47.194645+0000","last_became_peered":"2024-05-10T11:07:47.194645+0000","last_unstale":"2024-05-10T11:13:13.477904+0000","last_undegraded":"2024-05-10T11:13:13.477904+0000","last_fullsized":"2024-05-10T11:13:13.477904+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:34.962163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470732+0000","last_change":"2024-05-10T11:07:42.126378+0000","last_active":"2024-05-10T11:13:13.470732+0000","last_peered":"2024-05-10T11:13:13.470732+0000","last_clean":"2024-05-10T11:13:13.470732+0000","last_became_active":"2024-05-10T11:07:42.125984+0000","last_became_peered":"2024-05-10T11:07:42.125984+0000","last_unstale":"2024-05-10T11:13:13.470732+0000","last_undegraded":"2024-05-10T11:13:13.470732+0000","last_fullsized":"2024-05-10T11:13:13.470732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:15:50.378737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484726+0000","last_change":"2024-05-10T11:08:00.872342+0000","last_active":"2024-05-10T11:13:13.484726+0000","last_peered":"2024-05-10T11:13:13.484726+0000","last_clean":"2024-05-10T11:13:13.484726+0000","last_became_active":"2024-05-10T11:08:00.871926+0000","last_became_peered":"2024-05-10T11:08:00.871926+0000","last_unstale":"2024-05-10T11:13:13.484726+0000","last_undegraded":"2024-05-10T11:13:13.484726+0000","last_fullsized":"2024-05-10T11:13:13.484726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:27:16.519752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.282072+0000","last_change":"2024-05-10T11:07:42.384236+0000","last_active":"2024-05-10T11:13:13.282072+0000","last_peered":"2024-05-10T11:13:13.282072+0000","last_clean":"2024-05-10T11:13:13.282072+0000","last_became_active":"2024-05-10T11:07:42.383908+0000","last_became_peered":"2024-05-10T11:07:42.383908+0000","last_unstale":"2024-05-10T11:13:13.282072+0000","last_undegraded":"2024-05-10T11:13:13.282072+0000","last_fullsized":"2024-05-10T11:13:13.282072+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:09:42.302506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478716+0000","last_change":"2024-05-10T11:07:47.201645+0000","last_active":"2024-05-10T11:13:13.478716+0000","last_peered":"2024-05-10T11:13:13.478716+0000","last_clean":"2024-05-10T11:13:13.478716+0000","last_became_active":"2024-05-10T11:07:47.201300+0000","last_became_peered":"2024-05-10T11:07:47.201300+0000","last_unstale":"2024-05-10T11:13:13.478716+0000","last_undegraded":"2024-05-10T11:13:13.478716+0000","last_fullsized":"2024-05-10T11:13:13.478716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:37.268530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129032+0000","last_change":"2024-05-10T11:08:00.917988+0000","last_active":"2024-05-10T11:13:13.129032+0000","last_peered":"2024-05-10T11:13:13.129032+0000","last_clean":"2024-05-10T11:13:13.129032+0000","last_became_active":"2024-05-10T11:08:00.917457+0000","last_became_peered":"2024-05-10T11:08:00.917457+0000","last_unstale":"2024-05-10T11:13:13.129032+0000","last_undegraded":"2024-05-10T11:13:13.129032+0000","last_fullsized":"2024-05-10T11:13:13.129032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:16:01.816917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'53","reported_seq":124,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.424005+0000","last_change":"2024-05-10T11:07:42.141005+0000","last_active":"2024-05-10T11:13:13.424005+0000","last_peered":"2024-05-10T11:13:13.424005+0000","last_clean":"2024-05-10T11:13:13.424005+0000","last_became_active":"2024-05-10T11:07:42.140634+0000","last_became_peered":"2024-05-10T11:07:42.140634+0000","last_unstale":"2024-05-10T11:13:13.424005+0000","last_undegraded":"2024-05-10T11:13:13.424005+0000","last_fullsized":"2024-05-10T11:13:13.424005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:00:32.029374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":94,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.130553+0000","last_change":"2024-05-10T11:07:47.206864+0000","last_active":"2024-05-10T11:13:13.130553+0000","last_peered":"2024-05-10T11:13:13.130553+0000","last_clean":"2024-05-10T11:13:13.130553+0000","last_became_active":"2024-05-10T11:07:47.206387+0000","last_became_peered":"2024-05-10T11:07:47.206387+0000","last_unstale":"2024-05-10T11:13:13.130553+0000","last_undegraded":"2024-05-10T11:13:13.130553+0000","last_fullsized":"2024-05-10T11:13:13.130553+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:12:35.242270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132847+0000","last_change":"2024-05-10T11:08:00.867872+0000","last_active":"2024-05-10T11:13:13.132847+0000","last_peered":"2024-05-10T11:13:13.132847+0000","last_clean":"2024-05-10T11:13:13.132847+0000","last_became_active":"2024-05-10T11:08:00.867461+0000","last_became_peered":"2024-05-10T11:08:00.867461+0000","last_unstale":"2024-05-10T11:13:13.132847+0000","last_undegraded":"2024-05-10T11:13:13.132847+0000","last_fullsized":"2024-05-10T11:13:13.132847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:29.617335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470710+0000","last_change":"2024-05-10T11:07:42.599520+0000","last_active":"2024-05-10T11:13:13.470710+0000","last_peered":"2024-05-10T11:13:13.470710+0000","last_clean":"2024-05-10T11:13:13.470710+0000","last_became_active":"2024-05-10T11:07:42.599209+0000","last_became_peered":"2024-05-10T11:07:42.599209+0000","last_unstale":"2024-05-10T11:13:13.470710+0000","last_undegraded":"2024-05-10T11:13:13.470710+0000","last_fullsized":"2024-05-10T11:13:13.470710+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:52.899718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201837+0000","last_change":"2024-05-10T11:07:47.218286+0000","last_active":"2024-05-10T11:13:13.201837+0000","last_peered":"2024-05-10T11:13:13.201837+0000","last_clean":"2024-05-10T11:13:13.201837+0000","last_became_active":"2024-05-10T11:07:47.217960+0000","last_became_peered":"2024-05-10T11:07:47.217960+0000","last_unstale":"2024-05-10T11:13:13.201837+0000","last_undegraded":"2024-05-10T11:13:13.201837+0000","last_fullsized":"2024-05-10T11:13:13.201837+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:11:37.208359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.471251+0000","last_change":"2024-05-10T11:08:00.919578+0000","last_active":"2024-05-10T11:13:13.471251+0000","last_peered":"2024-05-10T11:13:13.471251+0000","last_clean":"2024-05-10T11:13:13.471251+0000","last_became_active":"2024-05-10T11:08:00.918546+0000","last_became_peered":"2024-05-10T11:08:00.918546+0000","last_unstale":"2024-05-10T11:13:13.471251+0000","last_undegraded":"2024-05-10T11:13:13.471251+0000","last_fullsized":"2024-05-10T11:13:13.471251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:39:05.062210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.476737+0000","last_change":"2024-05-10T11:07:42.384577+0000","last_active":"2024-05-10T11:13:13.476737+0000","last_peered":"2024-05-10T11:13:13.476737+0000","last_clean":"2024-05-10T11:13:13.476737+0000","last_became_active":"2024-05-10T11:07:42.384301+0000","last_became_peered":"2024-05-10T11:07:42.384301+0000","last_unstale":"2024-05-10T11:13:13.476737+0000","last_undegraded":"2024-05-10T11:13:13.476737+0000","last_fullsized":"2024-05-10T11:13:13.476737+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:11:03.359490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281603+0000","last_change":"2024-05-10T11:07:47.189708+0000","last_active":"2024-05-10T11:13:13.281603+0000","last_peered":"2024-05-10T11:13:13.281603+0000","last_clean":"2024-05-10T11:13:13.281603+0000","last_became_active":"2024-05-10T11:07:47.189367+0000","last_became_peered":"2024-05-10T11:07:47.189367+0000","last_unstale":"2024-05-10T11:13:13.281603+0000","last_undegraded":"2024-05-10T11:13:13.281603+0000","last_fullsized":"2024-05-10T11:13:13.281603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:25.410477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.279787+0000","last_change":"2024-05-10T11:08:00.899359+0000","last_active":"2024-05-10T11:13:13.279787+0000","last_peered":"2024-05-10T11:13:13.279787+0000","last_clean":"2024-05-10T11:13:13.279787+0000","last_became_active":"2024-05-10T11:08:00.898924+0000","last_became_peered":"2024-05-10T11:08:00.898924+0000","last_unstale":"2024-05-10T11:13:13.279787+0000","last_undegraded":"2024-05-10T11:13:13.279787+0000","last_fullsized":"2024-05-10T11:13:13.279787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:14:17.427245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129168+0000","last_change":"2024-05-10T11:07:42.140262+0000","last_active":"2024-05-10T11:13:13.129168+0000","last_peered":"2024-05-10T11:13:13.129168+0000","last_clean":"2024-05-10T11:13:13.129168+0000","last_became_active":"2024-05-10T11:07:42.138853+0000","last_became_peered":"2024-05-10T11:07:42.138853+0000","last_unstale":"2024-05-10T11:13:13.129168+0000","last_undegraded":"2024-05-10T11:13:13.129168+0000","last_fullsized":"2024-05-10T11:13:13.129168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:05:39.995237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484555+0000","last_change":"2024-05-10T11:07:47.203637+0000","last_active":"2024-05-10T11:13:13.484555+0000","last_peered":"2024-05-10T11:13:13.484555+0000","last_clean":"2024-05-10T11:13:13.484555+0000","last_became_active":"2024-05-10T11:07:47.203286+0000","last_became_peered":"2024-05-10T11:07:47.203286+0000","last_unstale":"2024-05-10T11:13:13.484555+0000","last_undegraded":"2024-05-10T11:13:13.484555+0000","last_fullsized":"2024-05-10T11:13:13.484555+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:27:46.924625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470388+0000","last_change":"2024-05-10T11:08:00.890082+0000","last_active":"2024-05-10T11:13:13.470388+0000","last_peered":"2024-05-10T11:13:13.470388+0000","last_clean":"2024-05-10T11:13:13.470388+0000","last_became_active":"2024-05-10T11:08:00.889673+0000","last_became_peered":"2024-05-10T11:08:00.889673+0000","last_unstale":"2024-05-10T11:13:13.470388+0000","last_undegraded":"2024-05-10T11:13:13.470388+0000","last_fullsized":"2024-05-10T11:13:13.470388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:21:31.167934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:14.010400+0000","last_change":"2024-05-10T11:07:42.130703+0000","last_active":"2024-05-10T11:13:14.010400+0000","last_peered":"2024-05-10T11:13:14.010400+0000","last_clean":"2024-05-10T11:13:14.010400+0000","last_became_active":"2024-05-10T11:07:42.130098+0000","last_became_peered":"2024-05-10T11:07:42.130098+0000","last_unstale":"2024-05-10T11:13:14.010400+0000","last_undegraded":"2024-05-10T11:13:14.010400+0000","last_fullsized":"2024-05-10T11:13:14.010400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:59:39.783151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129760+0000","last_change":"2024-05-10T11:07:47.220497+0000","last_active":"2024-05-10T11:13:13.129760+0000","last_peered":"2024-05-10T11:13:13.129760+0000","last_clean":"2024-05-10T11:13:13.129760+0000","last_became_active":"2024-05-10T11:07:47.219916+0000","last_became_peered":"2024-05-10T11:07:47.219916+0000","last_unstale":"2024-05-10T11:13:13.129760+0000","last_undegraded":"2024-05-10T11:13:13.129760+0000","last_fullsized":"2024-05-10T11:13:13.129760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:26:43.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.471013+0000","last_change":"2024-05-10T11:08:00.918258+0000","last_active":"2024-05-10T11:13:13.471013+0000","last_peered":"2024-05-10T11:13:13.471013+0000","last_clean":"2024-05-10T11:13:13.471013+0000","last_became_active":"2024-05-10T11:08:00.917808+0000","last_became_peered":"2024-05-10T11:08:00.917808+0000","last_unstale":"2024-05-10T11:13:13.471013+0000","last_undegraded":"2024-05-10T11:13:13.471013+0000","last_fullsized":"2024-05-10T11:13:13.471013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:14:08.867024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281805+0000","last_change":"2024-05-10T11:07:42.127661+0000","last_active":"2024-05-10T11:13:13.281805+0000","last_peered":"2024-05-10T11:13:13.281805+0000","last_clean":"2024-05-10T11:13:13.281805+0000","last_became_active":"2024-05-10T11:07:42.127256+0000","last_became_peered":"2024-05-10T11:07:42.127256+0000","last_unstale":"2024-05-10T11:13:13.281805+0000","last_undegraded":"2024-05-10T11:13:13.281805+0000","last_fullsized":"2024-05-10T11:13:13.281805+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:14:37.821350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.469988+0000","last_change":"2024-05-10T11:07:47.199589+0000","last_active":"2024-05-10T11:13:13.469988+0000","last_peered":"2024-05-10T11:13:13.469988+0000","last_clean":"2024-05-10T11:13:13.469988+0000","last_became_active":"2024-05-10T11:07:47.199145+0000","last_became_peered":"2024-05-10T11:07:47.199145+0000","last_unstale":"2024-05-10T11:13:13.469988+0000","last_undegraded":"2024-05-10T11:13:13.469988+0000","last_fullsized":"2024-05-10T11:13:13.469988+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:46.577543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477347+0000","last_change":"2024-05-10T11:09:23.085049+0000","last_active":"2024-05-10T11:13:13.477347+0000","last_peered":"2024-05-10T11:13:13.477347+0000","last_clean":"2024-05-10T11:13:13.477347+0000","last_became_active":"2024-05-10T11:09:23.084420+0000","last_became_peered":"2024-05-10T11:09:23.084420+0000","last_unstale":"2024-05-10T11:13:13.477347+0000","last_undegraded":"2024-05-10T11:13:13.477347+0000","last_fullsized":"2024-05-10T11:13:13.477347+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:32:33.346434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":300,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.413818+0000","last_change":"2024-05-10T11:07:42.599395+0000","last_active":"2024-05-10T11:13:17.413818+0000","last_peered":"2024-05-10T11:13:17.413818+0000","last_clean":"2024-05-10T11:13:17.413818+0000","last_became_active":"2024-05-10T11:07:42.598934+0000","last_became_peered":"2024-05-10T11:07:42.598934+0000","last_unstale":"2024-05-10T11:13:17.413818+0000","last_undegraded":"2024-05-10T11:13:17.413818+0000","last_fullsized":"2024-05-10T11:13:17.413818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:49:42.414789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":246,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470676+0000","last_change":"2024-05-10T11:07:47.192087+0000","last_active":"2024-05-10T11:13:13.470676+0000","last_peered":"2024-05-10T11:13:13.470676+0000","last_clean":"2024-05-10T11:13:13.470676+0000","last_became_active":"2024-05-10T11:07:47.191041+0000","last_became_peered":"2024-05-10T11:07:47.191041+0000","last_unstale":"2024-05-10T11:13:13.470676+0000","last_undegraded":"2024-05-10T11:13:13.470676+0000","last_fullsized":"2024-05-10T11:13:13.470676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:50:14.052863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201037+0000","last_change":"2024-05-10T11:09:23.080432+0000","last_active":"2024-05-10T11:13:13.201037+0000","last_peered":"2024-05-10T11:13:13.201037+0000","last_clean":"2024-05-10T11:13:13.201037+0000","last_became_active":"2024-05-10T11:09:23.074934+0000","last_became_peered":"2024-05-10T11:09:23.074934+0000","last_unstale":"2024-05-10T11:13:13.201037+0000","last_undegraded":"2024-05-10T11:13:13.201037+0000","last_fullsized":"2024-05-10T11:13:13.201037+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:12:14.613323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'41","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.414569+0000","last_change":"2024-05-10T11:07:42.130718+0000","last_active":"2024-05-10T11:13:13.414569+0000","last_peered":"2024-05-10T11:13:13.414569+0000","last_clean":"2024-05-10T11:13:13.414569+0000","last_became_active":"2024-05-10T11:07:42.130355+0000","last_became_peered":"2024-05-10T11:07:42.130355+0000","last_unstale":"2024-05-10T11:13:13.414569+0000","last_undegraded":"2024-05-10T11:13:13.414569+0000","last_fullsized":"2024-05-10T11:13:13.414569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:16:48.093146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470333+0000","last_change":"2024-05-10T11:07:47.213569+0000","last_active":"2024-05-10T11:13:13.470333+0000","last_peered":"2024-05-10T11:13:13.470333+0000","last_clean":"2024-05-10T11:13:13.470333+0000","last_became_active":"2024-05-10T11:07:47.213148+0000","last_became_peered":"2024-05-10T11:07:47.213148+0000","last_unstale":"2024-05-10T11:13:13.470333+0000","last_undegraded":"2024-05-10T11:13:13.470333+0000","last_fullsized":"2024-05-10T11:13:13.470333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:42:41.161274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287545+0000","last_change":"2024-05-10T11:08:00.870937+0000","last_active":"2024-05-10T11:13:13.287545+0000","last_peered":"2024-05-10T11:13:13.287545+0000","last_clean":"2024-05-10T11:13:13.287545+0000","last_became_active":"2024-05-10T11:08:00.869276+0000","last_became_peered":"2024-05-10T11:08:00.869276+0000","last_unstale":"2024-05-10T11:13:13.287545+0000","last_undegraded":"2024-05-10T11:13:13.287545+0000","last_fullsized":"2024-05-10T11:13:13.287545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:37.058875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478921+0000","last_change":"2024-05-10T11:07:42.130544+0000","last_active":"2024-05-10T11:13:13.478921+0000","last_peered":"2024-05-10T11:13:13.478921+0000","last_clean":"2024-05-10T11:13:13.478921+0000","last_became_active":"2024-05-10T11:07:42.128912+0000","last_became_peered":"2024-05-10T11:07:42.128912+0000","last_unstale":"2024-05-10T11:13:13.478921+0000","last_undegraded":"2024-05-10T11:13:13.478921+0000","last_fullsized":"2024-05-10T11:13:13.478921+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:22:23.521911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470271+0000","last_change":"2024-05-10T11:07:47.194346+0000","last_active":"2024-05-10T11:13:13.470271+0000","last_peered":"2024-05-10T11:13:13.470271+0000","last_clean":"2024-05-10T11:13:13.470271+0000","last_became_active":"2024-05-10T11:07:47.194019+0000","last_became_peered":"2024-05-10T11:07:47.194019+0000","last_unstale":"2024-05-10T11:13:13.470271+0000","last_undegraded":"2024-05-10T11:13:13.470271+0000","last_fullsized":"2024-05-10T11:13:13.470271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:33:05.262347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484623+0000","last_change":"2024-05-10T11:07:47.192001+0000","last_active":"2024-05-10T11:13:13.484623+0000","last_peered":"2024-05-10T11:13:13.484623+0000","last_clean":"2024-05-10T11:13:13.484623+0000","last_became_active":"2024-05-10T11:07:47.191668+0000","last_became_peered":"2024-05-10T11:07:47.191668+0000","last_unstale":"2024-05-10T11:13:13.484623+0000","last_undegraded":"2024-05-10T11:13:13.484623+0000","last_fullsized":"2024-05-10T11:13:13.484623+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:13:33.680899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132073+0000","last_change":"2024-05-10T11:07:47.192226+0000","last_active":"2024-05-10T11:13:13.132073+0000","last_peered":"2024-05-10T11:13:13.132073+0000","last_clean":"2024-05-10T11:13:13.132073+0000","last_became_active":"2024-05-10T11:07:47.191444+0000","last_became_peered":"2024-05-10T11:07:47.191444+0000","last_unstale":"2024-05-10T11:13:13.132073+0000","last_undegraded":"2024-05-10T11:13:13.132073+0000","last_fullsized":"2024-05-10T11:13:13.132073+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:08.348986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.131701+0000","last_change":"2024-05-10T11:07:47.210801+0000","last_active":"2024-05-10T11:13:13.131701+0000","last_peered":"2024-05-10T11:13:13.131701+0000","last_clean":"2024-05-10T11:13:13.131701+0000","last_became_active":"2024-05-10T11:07:47.210496+0000","last_became_peered":"2024-05-10T11:07:47.210496+0000","last_unstale":"2024-05-10T11:13:13.131701+0000","last_undegraded":"2024-05-10T11:13:13.131701+0000","last_fullsized":"2024-05-10T11:13:13.131701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:11:03.661457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201213+0000","last_change":"2024-05-10T11:07:47.211033+0000","last_active":"2024-05-10T11:13:13.201213+0000","last_peered":"2024-05-10T11:13:13.201213+0000","last_clean":"2024-05-10T11:13:13.201213+0000","last_became_active":"2024-05-10T11:07:47.206061+0000","last_became_peered":"2024-05-10T11:07:47.206061+0000","last_unstale":"2024-05-10T11:13:13.201213+0000","last_undegraded":"2024-05-10T11:13:13.201213+0000","last_fullsized":"2024-05-10T11:13:13.201213+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:04:05.878270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478582+0000","last_change":"2024-05-10T11:07:47.214966+0000","last_active":"2024-05-10T11:13:13.478582+0000","last_peered":"2024-05-10T11:13:13.478582+0000","last_clean":"2024-05-10T11:13:13.478582+0000","last_became_active":"2024-05-10T11:07:47.214563+0000","last_became_peered":"2024-05-10T11:07:47.214563+0000","last_unstale":"2024-05-10T11:13:13.478582+0000","last_undegraded":"2024-05-10T11:13:13.478582+0000","last_fullsized":"2024-05-10T11:13:13.478582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:29:01.428893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.485051+0000","last_change":"2024-05-10T11:07:47.196740+0000","last_active":"2024-05-10T11:13:13.485051+0000","last_peered":"2024-05-10T11:13:13.485051+0000","last_clean":"2024-05-10T11:13:13.485051+0000","last_became_active":"2024-05-10T11:07:47.196388+0000","last_became_peered":"2024-05-10T11:07:47.196388+0000","last_unstale":"2024-05-10T11:13:13.485051+0000","last_undegraded":"2024-05-10T11:13:13.485051+0000","last_fullsized":"2024-05-10T11:13:13.485051+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:23:09.055338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132107+0000","last_change":"2024-05-10T11:07:47.193568+0000","last_active":"2024-05-10T11:13:13.132107+0000","last_peered":"2024-05-10T11:13:13.132107+0000","last_clean":"2024-05-10T11:13:13.132107+0000","last_became_active":"2024-05-10T11:07:47.193202+0000","last_became_peered":"2024-05-10T11:07:47.193202+0000","last_unstale":"2024-05-10T11:13:13.132107+0000","last_undegraded":"2024-05-10T11:13:13.132107+0000","last_fullsized":"2024-05-10T11:13:13.132107+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:36:24.556266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.130958+0000","last_change":"2024-05-10T11:07:47.215344+0000","last_active":"2024-05-10T11:13:13.130958+0000","last_peered":"2024-05-10T11:13:13.130958+0000","last_clean":"2024-05-10T11:13:13.130958+0000","last_became_active":"2024-05-10T11:07:47.214896+0000","last_became_peered":"2024-05-10T11:07:47.214896+0000","last_unstale":"2024-05-10T11:13:13.130958+0000","last_undegraded":"2024-05-10T11:13:13.130958+0000","last_fullsized":"2024-05-10T11:13:13.130958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:23:07.845212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287656+0000","last_change":"2024-05-10T11:07:47.219047+0000","last_active":"2024-05-10T11:13:13.287656+0000","last_peered":"2024-05-10T11:13:13.287656+0000","last_clean":"2024-05-10T11:13:13.287656+0000","last_became_active":"2024-05-10T11:07:47.218778+0000","last_became_peered":"2024-05-10T11:07:47.218778+0000","last_unstale":"2024-05-10T11:13:13.287656+0000","last_undegraded":"2024-05-10T11:13:13.287656+0000","last_fullsized":"2024-05-10T11:13:13.287656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:02:48.945259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129317+0000","last_change":"2024-05-10T11:07:47.213412+0000","last_active":"2024-05-10T11:13:13.129317+0000","last_peered":"2024-05-10T11:13:13.129317+0000","last_clean":"2024-05-10T11:13:13.129317+0000","last_became_active":"2024-05-10T11:07:47.212963+0000","last_became_peered":"2024-05-10T11:07:47.212963+0000","last_unstale":"2024-05-10T11:13:13.129317+0000","last_undegraded":"2024-05-10T11:13:13.129317+0000","last_fullsized":"2024-05-10T11:13:13.129317+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:33:05.224184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.476853+0000","last_change":"2024-05-10T11:07:47.207998+0000","last_active":"2024-05-10T11:13:13.476853+0000","last_peered":"2024-05-10T11:13:13.476853+0000","last_clean":"2024-05-10T11:13:13.476853+0000","last_became_active":"2024-05-10T11:07:47.207298+0000","last_became_peered":"2024-05-10T11:07:47.207298+0000","last_unstale":"2024-05-10T11:13:13.476853+0000","last_undegraded":"2024-05-10T11:13:13.476853+0000","last_fullsized":"2024-05-10T11:13:13.476853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:59:14.001103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478208+0000","last_change":"2024-05-10T11:07:47.202216+0000","last_active":"2024-05-10T11:13:13.478208+0000","last_peered":"2024-05-10T11:13:13.478208+0000","last_clean":"2024-05-10T11:13:13.478208+0000","last_became_active":"2024-05-10T11:07:47.201867+0000","last_became_peered":"2024-05-10T11:07:47.201867+0000","last_unstale":"2024-05-10T11:13:13.478208+0000","last_undegraded":"2024-05-10T11:13:13.478208+0000","last_fullsized":"2024-05-10T11:13:13.478208+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:48:59.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484820+0000","last_change":"2024-05-10T11:07:47.194149+0000","last_active":"2024-05-10T11:13:13.484820+0000","last_peered":"2024-05-10T11:13:13.484820+0000","last_clean":"2024-05-10T11:13:13.484820+0000","last_became_active":"2024-05-10T11:07:47.193838+0000","last_became_peered":"2024-05-10T11:07:47.193838+0000","last_unstale":"2024-05-10T11:13:13.484820+0000","last_undegraded":"2024-05-10T11:13:13.484820+0000","last_fullsized":"2024-05-10T11:13:13.484820+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:46:44.181446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107385,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":607246,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2569,"num_read_kb":1410,"num_write":392,"num_write_kb":1483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":712,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":737280,"data_stored":1821738,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678099,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":532988,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874329,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32340,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712812,"statfs":{"total":95995035648,"available":95961919488,"internally_reserved":0,"allocated":1191936,"data_stored":1325367,"data_compressed":77418,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070558,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":929749,"data_compressed":54613,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266789,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":926009,"data_compressed":54613,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863463018,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":1131718,"data_compressed":71262,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626544,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1011712,"data_stored":528332,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574988,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":602182,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738515,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":529788,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":601712,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182529,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1052465,"data_compressed":60769,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986300,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":535043,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822775,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1114112,"data_stored":1256885,"data_compressed":77418,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":74493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135202,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":394694,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":207780,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-10T11:13:21.771 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-10T11:13:21.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-10T11:13:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.017+0000 7fc716a00640 1 -- 172.21.15.27:0/3119856049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7101027d0 msgr2=0x7fc710102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.017+0000 7fc716a00640 1 --2- 172.21.15.27:0/3119856049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7101027d0 0x7fc710102c50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc009930 tx=0x7fc6fc02a420 comp rx=0 tx=0).stop 2024-05-10T11:13:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.017+0000 7fc716a00640 1 -- 172.21.15.27:0/3119856049 shutdown_connections 2024-05-10T11:13:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.017+0000 7fc716a00640 1 --2- 172.21.15.27:0/3119856049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710103190 0x7fc71010f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.017+0000 7fc716a00640 1 --2- 172.21.15.27:0/3119856049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7101027d0 0x7fc710102c50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.017+0000 7fc716a00640 1 --2- 172.21.15.27:0/3119856049 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7101087e0 0x7fc710108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.017+0000 7fc716a00640 1 -- 172.21.15.27:0/3119856049 >> 172.21.15.27:0/3119856049 conn(0x7fc7100fe2e0 msgr2=0x7fc710100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.017+0000 7fc716a00640 1 -- 172.21.15.27:0/3119856049 shutdown_connections 2024-05-10T11:13:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.017+0000 7fc716a00640 1 -- 172.21.15.27:0/3119856049 wait complete. 2024-05-10T11:13:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.018+0000 7fc716a00640 1 Processor -- start 2024-05-10T11:13:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.018+0000 7fc716a00640 1 -- start start 2024-05-10T11:13:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.019+0000 7fc716a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7101027d0 0x7fc710199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.019+0000 7fc716a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc710103190 0x7fc710199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.019+0000 7fc716a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7101087e0 0x7fc71019d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.019+0000 7fc716a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc710112ab0 con 0x7fc7101087e0 2024-05-10T11:13:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.019+0000 7fc716a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc710112930 con 0x7fc7101027d0 2024-05-10T11:13:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.019+0000 7fc716a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc710112c30 con 0x7fc710103190 2024-05-10T11:13:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.019+0000 7fc716000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7101087e0 0x7fc71019d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.019+0000 7fc714c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc710103190 0x7fc710199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.019+0000 7fc716000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7101087e0 0x7fc71019d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39856/0 (socket says 172.21.15.27:39856) 2024-05-10T11:13:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.019+0000 7fc714c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc710103190 0x7fc710199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42326/0 (socket says 172.21.15.27:42326) 2024-05-10T11:13:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.019+0000 7fc716000640 1 -- 172.21.15.27:0/1803784057 learned_addr learned my addr 172.21.15.27:0/1803784057 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.019+0000 7fc715600640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7101027d0 0x7fc710199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.020+0000 7fc716000640 1 -- 172.21.15.27:0/1803784057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc710103190 msgr2=0x7fc710199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.020+0000 7fc716000640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc710103190 0x7fc710199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.020+0000 7fc716000640 1 -- 172.21.15.27:0/1803784057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7101027d0 msgr2=0x7fc710199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.020+0000 7fc716000640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7101027d0 0x7fc710199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.020+0000 7fc716000640 1 -- 172.21.15.27:0/1803784057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc71019e040 con 0x7fc7101087e0 2024-05-10T11:13:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.020+0000 7fc716000640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7101087e0 0x7fc71019d910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc70800eb10 tx=0x7fc70800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.020+0000 7fc70e000640 1 -- 172.21.15.27:0/1803784057 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70800ce60 con 0x7fc7101087e0 2024-05-10T11:13:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.021+0000 7fc70e000640 1 -- 172.21.15.27:0/1803784057 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc708005020 con 0x7fc7101087e0 2024-05-10T11:13:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.021+0000 7fc70e000640 1 -- 172.21.15.27:0/1803784057 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc708010690 con 0x7fc7101087e0 2024-05-10T11:13:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.021+0000 7fc716a00640 1 -- 172.21.15.27:0/1803784057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc71019e330 con 0x7fc7101087e0 2024-05-10T11:13:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.021+0000 7fc716a00640 1 -- 172.21.15.27:0/1803784057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc710074be0 con 0x7fc7101087e0 2024-05-10T11:13:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.022+0000 7fc716a00640 1 -- 172.21.15.27:0/1803784057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc710103f10 con 0x7fc7101087e0 2024-05-10T11:13:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.024+0000 7fc70e000640 1 -- 172.21.15.27:0/1803784057 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7fc708005510 con 0x7fc7101087e0 2024-05-10T11:13:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.027+0000 7fc70e000640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc6f80b2b40 0x7fc6f80b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.027+0000 7fc70e000640 1 -- 172.21.15.27:0/1803784057 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fc7080e33d0 con 0x7fc7101087e0 2024-05-10T11:13:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.027+0000 7fc715600640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc6f80b2b40 0x7fc6f80b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.028+0000 7fc715600640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc6f80b2b40 0x7fc6f80b5000 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc704009b50 tx=0x7fc704005f90 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.030+0000 7fc70e000640 1 -- 172.21.15.27:0/1803784057 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fc708054d70 con 0x7fc7101087e0 2024-05-10T11:13:22.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.265+0000 7fc716a00640 1 -- 172.21.15.27:0/1803784057 --> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc71010add0 con 0x7fc6f80b2b40 2024-05-10T11:13:22.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.294+0000 7fc70e000640 1 -- 172.21.15.27:0/1803784057 <== mgr.25220 v2:172.21.15.27:6800/1167199542 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339737 (secure 0 0 0) 0x7fc70401b3f0 con 0x7fc6f80b2b40 2024-05-10T11:13:22.298 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:13:22.302 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-10T11:13:22.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.304+0000 7fc716a00640 1 -- 172.21.15.27:0/1803784057 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc6f80b2b40 msgr2=0x7fc6f80b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:22.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.305+0000 7fc716a00640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc6f80b2b40 0x7fc6f80b5000 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc704009b50 tx=0x7fc704005f90 comp rx=0 tx=0).stop 2024-05-10T11:13:22.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.305+0000 7fc716a00640 1 -- 172.21.15.27:0/1803784057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7101087e0 msgr2=0x7fc71019d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:22.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.305+0000 7fc716a00640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7101087e0 0x7fc71019d910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc70800eb10 tx=0x7fc70800c2d0 comp rx=0 tx=0).stop 2024-05-10T11:13:22.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.305+0000 7fc716a00640 1 -- 172.21.15.27:0/1803784057 shutdown_connections 2024-05-10T11:13:22.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.305+0000 7fc716a00640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc6f80b2b40 0x7fc6f80b5000 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:22.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.305+0000 7fc716a00640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7101087e0 0x7fc71019d910 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:22.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.305+0000 7fc716a00640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc710103190 0x7fc710199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:22.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.305+0000 7fc716a00640 1 --2- 172.21.15.27:0/1803784057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7101027d0 0x7fc710199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:22.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.305+0000 7fc716a00640 1 -- 172.21.15.27:0/1803784057 >> 172.21.15.27:0/1803784057 conn(0x7fc7100fe2e0 msgr2=0x7fc7100feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:22.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.305+0000 7fc716a00640 1 -- 172.21.15.27:0/1803784057 shutdown_connections 2024-05-10T11:13:22.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:22.305+0000 7fc716a00640 1 -- 172.21.15.27:0/1803784057 wait complete. 2024-05-10T11:13:22.329 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-05-10T11:13:21.153436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1197848,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2713,"num_read_kb":1658,"num_write":529,"num_write_kb":3635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":2952552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":916,"ondisk_log_size":916,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361736,"kb_used_data":12560,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580088,"statfs":{"total":1151940427776,"available":1151570010112,"internally_reserved":0,"allocated":12861440,"data_stored":9952238,"data_compressed":608895,"data_compressed_allocated":823296,"data_compressed_original":4497408,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3302,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":210,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.974372"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.485180+0000","last_change":"2024-05-10T11:07:47.212921+0000","last_active":"2024-05-10T11:13:13.485180+0000","last_peered":"2024-05-10T11:13:13.485180+0000","last_clean":"2024-05-10T11:13:13.485180+0000","last_became_active":"2024-05-10T11:07:47.212558+0000","last_became_peered":"2024-05-10T11:07:47.212558+0000","last_unstale":"2024-05-10T11:13:13.485180+0000","last_undegraded":"2024-05-10T11:13:13.485180+0000","last_fullsized":"2024-05-10T11:13:13.485180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:47:20.415883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201806+0000","last_change":"2024-05-10T11:07:47.211186+0000","last_active":"2024-05-10T11:13:13.201806+0000","last_peered":"2024-05-10T11:13:13.201806+0000","last_clean":"2024-05-10T11:13:13.201806+0000","last_became_active":"2024-05-10T11:07:47.203487+0000","last_became_peered":"2024-05-10T11:07:47.203487+0000","last_unstale":"2024-05-10T11:13:13.201806+0000","last_undegraded":"2024-05-10T11:13:13.201806+0000","last_fullsized":"2024-05-10T11:13:13.201806+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:51:29.674632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201303+0000","last_change":"2024-05-10T11:07:47.202718+0000","last_active":"2024-05-10T11:13:13.201303+0000","last_peered":"2024-05-10T11:13:13.201303+0000","last_clean":"2024-05-10T11:13:13.201303+0000","last_became_active":"2024-05-10T11:07:47.202048+0000","last_became_peered":"2024-05-10T11:07:47.202048+0000","last_unstale":"2024-05-10T11:13:13.201303+0000","last_undegraded":"2024-05-10T11:13:13.201303+0000","last_fullsized":"2024-05-10T11:13:13.201303+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:01:41.156850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470810+0000","last_change":"2024-05-10T11:07:47.193519+0000","last_active":"2024-05-10T11:13:13.470810+0000","last_peered":"2024-05-10T11:13:13.470810+0000","last_clean":"2024-05-10T11:13:13.470810+0000","last_became_active":"2024-05-10T11:07:47.193200+0000","last_became_peered":"2024-05-10T11:07:47.193200+0000","last_unstale":"2024-05-10T11:13:13.470810+0000","last_undegraded":"2024-05-10T11:13:13.470810+0000","last_fullsized":"2024-05-10T11:13:13.470810+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:01.949453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281822+0000","last_change":"2024-05-10T11:07:47.207750+0000","last_active":"2024-05-10T11:13:13.281822+0000","last_peered":"2024-05-10T11:13:13.281822+0000","last_clean":"2024-05-10T11:13:13.281822+0000","last_became_active":"2024-05-10T11:07:47.207384+0000","last_became_peered":"2024-05-10T11:07:47.207384+0000","last_unstale":"2024-05-10T11:13:13.281822+0000","last_undegraded":"2024-05-10T11:13:13.281822+0000","last_fullsized":"2024-05-10T11:13:13.281822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:09.432766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129613+0000","last_change":"2024-05-10T11:08:22.245930+0000","last_active":"2024-05-10T11:13:13.129613+0000","last_peered":"2024-05-10T11:13:13.129613+0000","last_clean":"2024-05-10T11:13:13.129613+0000","last_became_active":"2024-05-10T11:08:22.245553+0000","last_became_peered":"2024-05-10T11:08:22.245553+0000","last_unstale":"2024-05-10T11:13:13.129613+0000","last_undegraded":"2024-05-10T11:13:13.129613+0000","last_fullsized":"2024-05-10T11:13:13.129613+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:47:31.485623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.469888+0000","last_change":"2024-05-10T11:07:47.196666+0000","last_active":"2024-05-10T11:13:13.469888+0000","last_peered":"2024-05-10T11:13:13.469888+0000","last_clean":"2024-05-10T11:13:13.469888+0000","last_became_active":"2024-05-10T11:07:47.196300+0000","last_became_peered":"2024-05-10T11:07:47.196300+0000","last_unstale":"2024-05-10T11:13:13.469888+0000","last_undegraded":"2024-05-10T11:13:13.469888+0000","last_fullsized":"2024-05-10T11:13:13.469888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:55:24.343385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.128863+0000","last_change":"2024-05-10T11:07:47.213048+0000","last_active":"2024-05-10T11:13:13.128863+0000","last_peered":"2024-05-10T11:13:13.128863+0000","last_clean":"2024-05-10T11:13:13.128863+0000","last_became_active":"2024-05-10T11:07:47.206359+0000","last_became_peered":"2024-05-10T11:07:47.206359+0000","last_unstale":"2024-05-10T11:13:13.128863+0000","last_undegraded":"2024-05-10T11:13:13.128863+0000","last_fullsized":"2024-05-10T11:13:13.128863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:32:56.307261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477171+0000","last_change":"2024-05-10T11:07:47.190945+0000","last_active":"2024-05-10T11:13:13.477171+0000","last_peered":"2024-05-10T11:13:13.477171+0000","last_clean":"2024-05-10T11:13:13.477171+0000","last_became_active":"2024-05-10T11:07:47.190631+0000","last_became_peered":"2024-05-10T11:07:47.190631+0000","last_unstale":"2024-05-10T11:13:13.477171+0000","last_undegraded":"2024-05-10T11:13:13.477171+0000","last_fullsized":"2024-05-10T11:13:13.477171+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:34:08.683912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.130034+0000","last_change":"2024-05-10T11:07:47.189540+0000","last_active":"2024-05-10T11:13:13.130034+0000","last_peered":"2024-05-10T11:13:13.130034+0000","last_clean":"2024-05-10T11:13:13.130034+0000","last_became_active":"2024-05-10T11:07:47.189185+0000","last_became_peered":"2024-05-10T11:07:47.189185+0000","last_unstale":"2024-05-10T11:13:13.130034+0000","last_undegraded":"2024-05-10T11:13:13.130034+0000","last_fullsized":"2024-05-10T11:13:13.130034+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:55:23.892826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470578+0000","last_change":"2024-05-10T11:07:47.195688+0000","last_active":"2024-05-10T11:13:13.470578+0000","last_peered":"2024-05-10T11:13:13.470578+0000","last_clean":"2024-05-10T11:13:13.470578+0000","last_became_active":"2024-05-10T11:07:47.195081+0000","last_became_peered":"2024-05-10T11:07:47.195081+0000","last_unstale":"2024-05-10T11:13:13.470578+0000","last_undegraded":"2024-05-10T11:13:13.470578+0000","last_fullsized":"2024-05-10T11:13:13.470578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:20:55.240827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470631+0000","last_change":"2024-05-10T11:07:47.193736+0000","last_active":"2024-05-10T11:13:13.470631+0000","last_peered":"2024-05-10T11:13:13.470631+0000","last_clean":"2024-05-10T11:13:13.470631+0000","last_became_active":"2024-05-10T11:07:47.193400+0000","last_became_peered":"2024-05-10T11:07:47.193400+0000","last_unstale":"2024-05-10T11:13:13.470631+0000","last_undegraded":"2024-05-10T11:13:13.470631+0000","last_fullsized":"2024-05-10T11:13:13.470631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:46:28.507330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478284+0000","last_change":"2024-05-10T11:08:22.042849+0000","last_active":"2024-05-10T11:13:13.478284+0000","last_peered":"2024-05-10T11:13:13.478284+0000","last_clean":"2024-05-10T11:13:13.478284+0000","last_became_active":"2024-05-10T11:08:22.042024+0000","last_became_peered":"2024-05-10T11:08:22.042024+0000","last_unstale":"2024-05-10T11:13:13.478284+0000","last_undegraded":"2024-05-10T11:13:13.478284+0000","last_fullsized":"2024-05-10T11:13:13.478284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:29:08.096541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201493+0000","last_change":"2024-05-10T11:07:47.214016+0000","last_active":"2024-05-10T11:13:13.201493+0000","last_peered":"2024-05-10T11:13:13.201493+0000","last_clean":"2024-05-10T11:13:13.201493+0000","last_became_active":"2024-05-10T11:07:47.209658+0000","last_became_peered":"2024-05-10T11:07:47.209658+0000","last_unstale":"2024-05-10T11:13:13.201493+0000","last_undegraded":"2024-05-10T11:13:13.201493+0000","last_fullsized":"2024-05-10T11:13:13.201493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:44.771221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201161+0000","last_change":"2024-05-10T11:07:47.210153+0000","last_active":"2024-05-10T11:13:13.201161+0000","last_peered":"2024-05-10T11:13:13.201161+0000","last_clean":"2024-05-10T11:13:13.201161+0000","last_became_active":"2024-05-10T11:07:47.208470+0000","last_became_peered":"2024-05-10T11:07:47.208470+0000","last_unstale":"2024-05-10T11:13:13.201161+0000","last_undegraded":"2024-05-10T11:13:13.201161+0000","last_fullsized":"2024-05-10T11:13:13.201161+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:23:03.614670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.476671+0000","last_change":"2024-05-10T11:07:47.210704+0000","last_active":"2024-05-10T11:13:13.476671+0000","last_peered":"2024-05-10T11:13:13.476671+0000","last_clean":"2024-05-10T11:13:13.476671+0000","last_became_active":"2024-05-10T11:07:47.210317+0000","last_became_peered":"2024-05-10T11:07:47.210317+0000","last_unstale":"2024-05-10T11:13:13.476671+0000","last_undegraded":"2024-05-10T11:13:13.476671+0000","last_fullsized":"2024-05-10T11:13:13.476671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:47:30.848731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484975+0000","last_change":"2024-05-10T11:07:47.213116+0000","last_active":"2024-05-10T11:13:13.484975+0000","last_peered":"2024-05-10T11:13:13.484975+0000","last_clean":"2024-05-10T11:13:13.484975+0000","last_became_active":"2024-05-10T11:07:47.212567+0000","last_became_peered":"2024-05-10T11:07:47.212567+0000","last_unstale":"2024-05-10T11:13:13.484975+0000","last_undegraded":"2024-05-10T11:13:13.484975+0000","last_fullsized":"2024-05-10T11:13:13.484975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:44:32.222012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.279315+0000","last_change":"2024-05-10T11:07:47.200973+0000","last_active":"2024-05-10T11:13:13.279315+0000","last_peered":"2024-05-10T11:13:13.279315+0000","last_clean":"2024-05-10T11:13:13.279315+0000","last_became_active":"2024-05-10T11:07:47.200499+0000","last_became_peered":"2024-05-10T11:07:47.200499+0000","last_unstale":"2024-05-10T11:13:13.279315+0000","last_undegraded":"2024-05-10T11:13:13.279315+0000","last_fullsized":"2024-05-10T11:13:13.279315+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:29:24.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132918+0000","last_change":"2024-05-10T11:08:22.042254+0000","last_active":"2024-05-10T11:13:13.132918+0000","last_peered":"2024-05-10T11:13:13.132918+0000","last_clean":"2024-05-10T11:13:13.132918+0000","last_became_active":"2024-05-10T11:08:22.041906+0000","last_became_peered":"2024-05-10T11:08:22.041906+0000","last_unstale":"2024-05-10T11:13:13.132918+0000","last_undegraded":"2024-05-10T11:13:13.132918+0000","last_fullsized":"2024-05-10T11:13:13.132918+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:55:27.280478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.131809+0000","last_change":"2024-05-10T11:08:00.897746+0000","last_active":"2024-05-10T11:13:13.131809+0000","last_peered":"2024-05-10T11:13:13.131809+0000","last_clean":"2024-05-10T11:13:13.131809+0000","last_became_active":"2024-05-10T11:08:00.897428+0000","last_became_peered":"2024-05-10T11:08:00.897428+0000","last_unstale":"2024-05-10T11:13:13.131809+0000","last_undegraded":"2024-05-10T11:13:13.131809+0000","last_fullsized":"2024-05-10T11:13:13.131809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:38.846373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'14","reported_seq":306,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.590118+0000","last_change":"2024-05-10T11:07:42.139990+0000","last_active":"2024-05-10T11:13:17.590118+0000","last_peered":"2024-05-10T11:13:17.590118+0000","last_clean":"2024-05-10T11:13:17.590118+0000","last_became_active":"2024-05-10T11:07:42.139612+0000","last_became_peered":"2024-05-10T11:07:42.139612+0000","last_unstale":"2024-05-10T11:13:17.590118+0000","last_undegraded":"2024-05-10T11:13:17.590118+0000","last_fullsized":"2024-05-10T11:13:17.590118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:48:29.590570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":244,"num_read_kb":244,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129957+0000","last_change":"2024-05-10T11:07:47.214263+0000","last_active":"2024-05-10T11:13:13.129957+0000","last_peered":"2024-05-10T11:13:13.129957+0000","last_clean":"2024-05-10T11:13:13.129957+0000","last_became_active":"2024-05-10T11:07:47.213946+0000","last_became_peered":"2024-05-10T11:07:47.213946+0000","last_unstale":"2024-05-10T11:13:13.129957+0000","last_undegraded":"2024-05-10T11:13:13.129957+0000","last_fullsized":"2024-05-10T11:13:13.129957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:54:31.015838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.469566+0000","last_change":"2024-05-10T11:08:00.890571+0000","last_active":"2024-05-10T11:13:13.469566+0000","last_peered":"2024-05-10T11:13:13.469566+0000","last_clean":"2024-05-10T11:13:13.469566+0000","last_became_active":"2024-05-10T11:08:00.889468+0000","last_became_peered":"2024-05-10T11:08:00.889468+0000","last_unstale":"2024-05-10T11:13:13.469566+0000","last_undegraded":"2024-05-10T11:13:13.469566+0000","last_fullsized":"2024-05-10T11:13:13.469566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:20:54.342656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.130184+0000","last_change":"2024-05-10T11:07:42.383396+0000","last_active":"2024-05-10T11:13:13.130184+0000","last_peered":"2024-05-10T11:13:13.130184+0000","last_clean":"2024-05-10T11:13:13.130184+0000","last_became_active":"2024-05-10T11:07:42.382787+0000","last_became_peered":"2024-05-10T11:07:42.382787+0000","last_unstale":"2024-05-10T11:13:13.130184+0000","last_undegraded":"2024-05-10T11:13:13.130184+0000","last_fullsized":"2024-05-10T11:13:13.130184+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:44:34.240119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201637+0000","last_change":"2024-05-10T11:07:47.216770+0000","last_active":"2024-05-10T11:13:13.201637+0000","last_peered":"2024-05-10T11:13:13.201637+0000","last_clean":"2024-05-10T11:13:13.201637+0000","last_became_active":"2024-05-10T11:07:47.216377+0000","last_became_peered":"2024-05-10T11:07:47.216377+0000","last_unstale":"2024-05-10T11:13:13.201637+0000","last_undegraded":"2024-05-10T11:13:13.201637+0000","last_fullsized":"2024-05-10T11:13:13.201637+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:59:22.993888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470458+0000","last_change":"2024-05-10T11:08:00.900614+0000","last_active":"2024-05-10T11:13:13.470458+0000","last_peered":"2024-05-10T11:13:13.470458+0000","last_clean":"2024-05-10T11:13:13.470458+0000","last_became_active":"2024-05-10T11:08:00.900162+0000","last_became_peered":"2024-05-10T11:08:00.900162+0000","last_unstale":"2024-05-10T11:13:13.470458+0000","last_undegraded":"2024-05-10T11:13:13.470458+0000","last_fullsized":"2024-05-10T11:13:13.470458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:42:56.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.282238+0000","last_change":"2024-05-10T11:07:42.127749+0000","last_active":"2024-05-10T11:13:13.282238+0000","last_peered":"2024-05-10T11:13:13.282238+0000","last_clean":"2024-05-10T11:13:13.282238+0000","last_became_active":"2024-05-10T11:07:42.127296+0000","last_became_peered":"2024-05-10T11:07:42.127296+0000","last_unstale":"2024-05-10T11:13:13.282238+0000","last_undegraded":"2024-05-10T11:13:13.282238+0000","last_fullsized":"2024-05-10T11:13:13.282238+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:37:37.720542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477062+0000","last_change":"2024-05-10T11:07:47.198971+0000","last_active":"2024-05-10T11:13:13.477062+0000","last_peered":"2024-05-10T11:13:13.477062+0000","last_clean":"2024-05-10T11:13:13.477062+0000","last_became_active":"2024-05-10T11:07:47.198510+0000","last_became_peered":"2024-05-10T11:07:47.198510+0000","last_unstale":"2024-05-10T11:13:13.477062+0000","last_undegraded":"2024-05-10T11:13:13.477062+0000","last_fullsized":"2024-05-10T11:13:13.477062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:17:24.987880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470356+0000","last_change":"2024-05-10T11:08:00.878749+0000","last_active":"2024-05-10T11:13:13.470356+0000","last_peered":"2024-05-10T11:13:13.470356+0000","last_clean":"2024-05-10T11:13:13.470356+0000","last_became_active":"2024-05-10T11:08:00.878271+0000","last_became_peered":"2024-05-10T11:08:00.878271+0000","last_unstale":"2024-05-10T11:13:13.470356+0000","last_undegraded":"2024-05-10T11:13:13.470356+0000","last_fullsized":"2024-05-10T11:13:13.470356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:39:28.558766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.282229+0000","last_change":"2024-05-10T11:07:42.129593+0000","last_active":"2024-05-10T11:13:13.282229+0000","last_peered":"2024-05-10T11:13:13.282229+0000","last_clean":"2024-05-10T11:13:13.282229+0000","last_became_active":"2024-05-10T11:07:42.129250+0000","last_became_peered":"2024-05-10T11:07:42.129250+0000","last_unstale":"2024-05-10T11:13:13.282229+0000","last_undegraded":"2024-05-10T11:13:13.282229+0000","last_fullsized":"2024-05-10T11:13:13.282229+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:22:55.541079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.278965+0000","last_change":"2024-05-10T11:08:22.244535+0000","last_active":"2024-05-10T11:13:13.278965+0000","last_peered":"2024-05-10T11:13:13.278965+0000","last_clean":"2024-05-10T11:13:13.278965+0000","last_became_active":"2024-05-10T11:08:22.243716+0000","last_became_peered":"2024-05-10T11:08:22.243716+0000","last_unstale":"2024-05-10T11:13:13.278965+0000","last_undegraded":"2024-05-10T11:13:13.278965+0000","last_fullsized":"2024-05-10T11:13:13.278965+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:11.718999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132765+0000","last_change":"2024-05-10T11:08:00.869806+0000","last_active":"2024-05-10T11:13:13.132765+0000","last_peered":"2024-05-10T11:13:13.132765+0000","last_clean":"2024-05-10T11:13:13.132765+0000","last_became_active":"2024-05-10T11:08:00.869355+0000","last_became_peered":"2024-05-10T11:08:00.869355+0000","last_unstale":"2024-05-10T11:13:13.132765+0000","last_undegraded":"2024-05-10T11:13:13.132765+0000","last_fullsized":"2024-05-10T11:13:13.132765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:36.043412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":314,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.574435+0000","last_change":"2024-05-10T11:09:23.076933+0000","last_active":"2024-05-10T11:13:17.574435+0000","last_peered":"2024-05-10T11:13:17.574435+0000","last_clean":"2024-05-10T11:13:17.574435+0000","last_became_active":"2024-05-10T11:09:23.073274+0000","last_became_peered":"2024-05-10T11:09:23.073274+0000","last_unstale":"2024-05-10T11:13:17.574435+0000","last_undegraded":"2024-05-10T11:13:17.574435+0000","last_fullsized":"2024-05-10T11:13:17.574435+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:26:13.923466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":243,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281830+0000","last_change":"2024-05-10T11:07:47.200873+0000","last_active":"2024-05-10T11:13:13.281830+0000","last_peered":"2024-05-10T11:13:13.281830+0000","last_clean":"2024-05-10T11:13:13.281830+0000","last_became_active":"2024-05-10T11:07:47.200436+0000","last_became_peered":"2024-05-10T11:07:47.200436+0000","last_unstale":"2024-05-10T11:13:13.281830+0000","last_undegraded":"2024-05-10T11:13:13.281830+0000","last_fullsized":"2024-05-10T11:13:13.281830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:38:49.588593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132349+0000","last_change":"2024-05-10T11:08:00.885432+0000","last_active":"2024-05-10T11:13:13.132349+0000","last_peered":"2024-05-10T11:13:13.132349+0000","last_clean":"2024-05-10T11:13:13.132349+0000","last_became_active":"2024-05-10T11:08:00.884987+0000","last_became_peered":"2024-05-10T11:08:00.884987+0000","last_unstale":"2024-05-10T11:13:13.132349+0000","last_undegraded":"2024-05-10T11:13:13.132349+0000","last_fullsized":"2024-05-10T11:13:13.132349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:41:24.082722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129458+0000","last_change":"2024-05-10T11:09:23.093122+0000","last_active":"2024-05-10T11:13:13.129458+0000","last_peered":"2024-05-10T11:13:13.129458+0000","last_clean":"2024-05-10T11:13:13.129458+0000","last_became_active":"2024-05-10T11:09:23.084485+0000","last_became_peered":"2024-05-10T11:09:23.084485+0000","last_unstale":"2024-05-10T11:13:13.129458+0000","last_undegraded":"2024-05-10T11:13:13.129458+0000","last_fullsized":"2024-05-10T11:13:13.129458+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:12:36.793572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.288102+0000","last_change":"2024-05-10T11:08:22.036515+0000","last_active":"2024-05-10T11:13:13.288102+0000","last_peered":"2024-05-10T11:13:13.288102+0000","last_clean":"2024-05-10T11:13:13.288102+0000","last_became_active":"2024-05-10T11:08:22.036072+0000","last_became_peered":"2024-05-10T11:08:22.036072+0000","last_unstale":"2024-05-10T11:13:13.288102+0000","last_undegraded":"2024-05-10T11:13:13.288102+0000","last_fullsized":"2024-05-10T11:13:13.288102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:08.734453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6],"acting":[5,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484893+0000","last_change":"2024-05-10T11:08:00.888673+0000","last_active":"2024-05-10T11:13:13.484893+0000","last_peered":"2024-05-10T11:13:13.484893+0000","last_clean":"2024-05-10T11:13:13.484893+0000","last_became_active":"2024-05-10T11:08:00.888302+0000","last_became_peered":"2024-05-10T11:08:00.888302+0000","last_unstale":"2024-05-10T11:13:13.484893+0000","last_undegraded":"2024-05-10T11:13:13.484893+0000","last_fullsized":"2024-05-10T11:13:13.484893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:15:32.276358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478224+0000","last_change":"2024-05-10T11:07:42.140993+0000","last_active":"2024-05-10T11:13:13.478224+0000","last_peered":"2024-05-10T11:13:13.478224+0000","last_clean":"2024-05-10T11:13:13.478224+0000","last_became_active":"2024-05-10T11:07:42.140501+0000","last_became_peered":"2024-05-10T11:07:42.140501+0000","last_unstale":"2024-05-10T11:13:13.478224+0000","last_undegraded":"2024-05-10T11:13:13.478224+0000","last_fullsized":"2024-05-10T11:13:13.478224+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:01:50.324507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129032+0000","last_change":"2024-05-10T11:07:47.212616+0000","last_active":"2024-05-10T11:13:13.129032+0000","last_peered":"2024-05-10T11:13:13.129032+0000","last_clean":"2024-05-10T11:13:13.129032+0000","last_became_active":"2024-05-10T11:07:47.211081+0000","last_became_peered":"2024-05-10T11:07:47.211081+0000","last_unstale":"2024-05-10T11:13:13.129032+0000","last_undegraded":"2024-05-10T11:13:13.129032+0000","last_fullsized":"2024-05-10T11:13:13.129032+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:03:54.404355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201692+0000","last_change":"2024-05-10T11:08:00.888782+0000","last_active":"2024-05-10T11:13:13.201692+0000","last_peered":"2024-05-10T11:13:13.201692+0000","last_clean":"2024-05-10T11:13:13.201692+0000","last_became_active":"2024-05-10T11:08:00.888366+0000","last_became_peered":"2024-05-10T11:08:00.888366+0000","last_unstale":"2024-05-10T11:13:13.201692+0000","last_undegraded":"2024-05-10T11:13:13.201692+0000","last_fullsized":"2024-05-10T11:13:13.201692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:59:37.040186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287656+0000","last_change":"2024-05-10T11:07:42.128677+0000","last_active":"2024-05-10T11:13:13.287656+0000","last_peered":"2024-05-10T11:13:13.287656+0000","last_clean":"2024-05-10T11:13:13.287656+0000","last_became_active":"2024-05-10T11:07:42.128330+0000","last_became_peered":"2024-05-10T11:07:42.128330+0000","last_unstale":"2024-05-10T11:13:13.287656+0000","last_undegraded":"2024-05-10T11:13:13.287656+0000","last_fullsized":"2024-05-10T11:13:13.287656+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:09:11.738485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.278426+0000","last_change":"2024-05-10T11:07:47.215257+0000","last_active":"2024-05-10T11:13:13.278426+0000","last_peered":"2024-05-10T11:13:13.278426+0000","last_clean":"2024-05-10T11:13:13.278426+0000","last_became_active":"2024-05-10T11:07:47.214682+0000","last_became_peered":"2024-05-10T11:07:47.214682+0000","last_unstale":"2024-05-10T11:13:13.278426+0000","last_undegraded":"2024-05-10T11:13:13.278426+0000","last_fullsized":"2024-05-10T11:13:13.278426+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:35:45.787737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.128926+0000","last_change":"2024-05-10T11:08:00.894446+0000","last_active":"2024-05-10T11:13:13.128926+0000","last_peered":"2024-05-10T11:13:13.128926+0000","last_clean":"2024-05-10T11:13:13.128926+0000","last_became_active":"2024-05-10T11:08:00.886473+0000","last_became_peered":"2024-05-10T11:08:00.886473+0000","last_unstale":"2024-05-10T11:13:13.128926+0000","last_undegraded":"2024-05-10T11:13:13.128926+0000","last_fullsized":"2024-05-10T11:13:13.128926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:35:40.980328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":540,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:18.414865+0000","last_change":"2024-05-10T11:07:42.138750+0000","last_active":"2024-05-10T11:13:18.414865+0000","last_peered":"2024-05-10T11:13:18.414865+0000","last_clean":"2024-05-10T11:13:18.414865+0000","last_became_active":"2024-05-10T11:07:42.138342+0000","last_became_peered":"2024-05-10T11:07:42.138342+0000","last_unstale":"2024-05-10T11:13:18.414865+0000","last_undegraded":"2024-05-10T11:13:18.414865+0000","last_fullsized":"2024-05-10T11:13:18.414865+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:19:45.791550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":486,"num_read_kb":486,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477124+0000","last_change":"2024-05-10T11:07:47.204604+0000","last_active":"2024-05-10T11:13:13.477124+0000","last_peered":"2024-05-10T11:13:13.477124+0000","last_clean":"2024-05-10T11:13:13.477124+0000","last_became_active":"2024-05-10T11:07:47.204261+0000","last_became_peered":"2024-05-10T11:07:47.204261+0000","last_unstale":"2024-05-10T11:13:13.477124+0000","last_undegraded":"2024-05-10T11:13:13.477124+0000","last_fullsized":"2024-05-10T11:13:13.477124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:55:46.409545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281665+0000","last_change":"2024-05-10T11:08:22.038090+0000","last_active":"2024-05-10T11:13:13.281665+0000","last_peered":"2024-05-10T11:13:13.281665+0000","last_clean":"2024-05-10T11:13:13.281665+0000","last_became_active":"2024-05-10T11:08:22.037490+0000","last_became_peered":"2024-05-10T11:08:22.037490+0000","last_unstale":"2024-05-10T11:13:13.281665+0000","last_undegraded":"2024-05-10T11:13:13.281665+0000","last_fullsized":"2024-05-10T11:13:13.281665+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:01:46.319401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470116+0000","last_change":"2024-05-10T11:07:42.128636+0000","last_active":"2024-05-10T11:13:13.470116+0000","last_peered":"2024-05-10T11:13:13.470116+0000","last_clean":"2024-05-10T11:13:13.470116+0000","last_became_active":"2024-05-10T11:07:42.128194+0000","last_became_peered":"2024-05-10T11:07:42.128194+0000","last_unstale":"2024-05-10T11:13:13.470116+0000","last_undegraded":"2024-05-10T11:13:13.470116+0000","last_fullsized":"2024-05-10T11:13:13.470116+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:51:54.279292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201709+0000","last_change":"2024-05-10T11:08:22.037807+0000","last_active":"2024-05-10T11:13:13.201709+0000","last_peered":"2024-05-10T11:13:13.201709+0000","last_clean":"2024-05-10T11:13:13.201709+0000","last_became_active":"2024-05-10T11:08:22.037337+0000","last_became_peered":"2024-05-10T11:08:22.037337+0000","last_unstale":"2024-05-10T11:13:13.201709+0000","last_undegraded":"2024-05-10T11:13:13.201709+0000","last_fullsized":"2024-05-10T11:13:13.201709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:50:06.481222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470697+0000","last_change":"2024-05-10T11:08:00.893684+0000","last_active":"2024-05-10T11:13:13.470697+0000","last_peered":"2024-05-10T11:13:13.470697+0000","last_clean":"2024-05-10T11:13:13.470697+0000","last_became_active":"2024-05-10T11:08:00.893334+0000","last_became_peered":"2024-05-10T11:08:00.893334+0000","last_unstale":"2024-05-10T11:13:13.470697+0000","last_undegraded":"2024-05-10T11:13:13.470697+0000","last_fullsized":"2024-05-10T11:13:13.470697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:58:04.812308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.288233+0000","last_change":"2024-05-10T11:07:42.382086+0000","last_active":"2024-05-10T11:13:13.288233+0000","last_peered":"2024-05-10T11:13:13.288233+0000","last_clean":"2024-05-10T11:13:13.288233+0000","last_became_active":"2024-05-10T11:07:42.381744+0000","last_became_peered":"2024-05-10T11:07:42.381744+0000","last_unstale":"2024-05-10T11:13:13.288233+0000","last_undegraded":"2024-05-10T11:13:13.288233+0000","last_fullsized":"2024-05-10T11:13:13.288233+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:15:10.893834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477087+0000","last_change":"2024-05-10T11:07:47.194683+0000","last_active":"2024-05-10T11:13:13.477087+0000","last_peered":"2024-05-10T11:13:13.477087+0000","last_clean":"2024-05-10T11:13:13.477087+0000","last_became_active":"2024-05-10T11:07:47.194319+0000","last_became_peered":"2024-05-10T11:07:47.194319+0000","last_unstale":"2024-05-10T11:13:13.477087+0000","last_undegraded":"2024-05-10T11:13:13.477087+0000","last_fullsized":"2024-05-10T11:13:13.477087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:41:48.437093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.278511+0000","last_change":"2024-05-10T11:08:00.889616+0000","last_active":"2024-05-10T11:13:13.278511+0000","last_peered":"2024-05-10T11:13:13.278511+0000","last_clean":"2024-05-10T11:13:13.278511+0000","last_became_active":"2024-05-10T11:08:00.882179+0000","last_became_peered":"2024-05-10T11:08:00.882179+0000","last_unstale":"2024-05-10T11:13:13.278511+0000","last_undegraded":"2024-05-10T11:13:13.278511+0000","last_fullsized":"2024-05-10T11:13:13.278511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:23:12.407861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'45","reported_seq":593,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:18.443507+0000","last_change":"2024-05-10T11:07:42.128796+0000","last_active":"2024-05-10T11:13:18.443507+0000","last_peered":"2024-05-10T11:13:18.443507+0000","last_clean":"2024-05-10T11:13:18.443507+0000","last_became_active":"2024-05-10T11:07:42.128402+0000","last_became_peered":"2024-05-10T11:07:42.128402+0000","last_unstale":"2024-05-10T11:13:18.443507+0000","last_undegraded":"2024-05-10T11:13:18.443507+0000","last_fullsized":"2024-05-10T11:13:18.443507+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:50:09.205102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72820,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":500,"num_read_kb":322,"num_write":47,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.282034+0000","last_change":"2024-05-10T11:07:47.188304+0000","last_active":"2024-05-10T11:13:13.282034+0000","last_peered":"2024-05-10T11:13:13.282034+0000","last_clean":"2024-05-10T11:13:13.282034+0000","last_became_active":"2024-05-10T11:07:47.187798+0000","last_became_peered":"2024-05-10T11:07:47.187798+0000","last_unstale":"2024-05-10T11:13:13.282034+0000","last_undegraded":"2024-05-10T11:13:13.282034+0000","last_fullsized":"2024-05-10T11:13:13.282034+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:50:21.059060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.288522+0000","last_change":"2024-05-10T11:08:00.886669+0000","last_active":"2024-05-10T11:13:13.288522+0000","last_peered":"2024-05-10T11:13:13.288522+0000","last_clean":"2024-05-10T11:13:13.288522+0000","last_became_active":"2024-05-10T11:08:00.886249+0000","last_became_peered":"2024-05-10T11:08:00.886249+0000","last_unstale":"2024-05-10T11:13:13.288522+0000","last_undegraded":"2024-05-10T11:13:13.288522+0000","last_fullsized":"2024-05-10T11:13:13.288522+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:10:07.707462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484691+0000","last_change":"2024-05-10T11:09:23.095352+0000","last_active":"2024-05-10T11:13:13.484691+0000","last_peered":"2024-05-10T11:13:13.484691+0000","last_clean":"2024-05-10T11:13:13.484691+0000","last_became_active":"2024-05-10T11:09:23.083116+0000","last_became_peered":"2024-05-10T11:09:23.083116+0000","last_unstale":"2024-05-10T11:13:13.484691+0000","last_undegraded":"2024-05-10T11:13:13.484691+0000","last_fullsized":"2024-05-10T11:13:13.484691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:24:28.945724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478001+0000","last_change":"2024-05-10T11:07:47.207657+0000","last_active":"2024-05-10T11:13:13.478001+0000","last_peered":"2024-05-10T11:13:13.478001+0000","last_clean":"2024-05-10T11:13:13.478001+0000","last_became_active":"2024-05-10T11:07:47.207317+0000","last_became_peered":"2024-05-10T11:07:47.207317+0000","last_unstale":"2024-05-10T11:13:13.478001+0000","last_undegraded":"2024-05-10T11:13:13.478001+0000","last_fullsized":"2024-05-10T11:13:13.478001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:07:33.961642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484711+0000","last_change":"2024-05-10T11:08:00.884076+0000","last_active":"2024-05-10T11:13:13.484711+0000","last_peered":"2024-05-10T11:13:13.484711+0000","last_clean":"2024-05-10T11:13:13.484711+0000","last_became_active":"2024-05-10T11:08:00.883670+0000","last_became_peered":"2024-05-10T11:08:00.883670+0000","last_unstale":"2024-05-10T11:13:13.484711+0000","last_undegraded":"2024-05-10T11:13:13.484711+0000","last_fullsized":"2024-05-10T11:13:13.484711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:56.447836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287765+0000","last_change":"2024-05-10T11:07:42.135550+0000","last_active":"2024-05-10T11:13:13.287765+0000","last_peered":"2024-05-10T11:13:13.287765+0000","last_clean":"2024-05-10T11:13:13.287765+0000","last_became_active":"2024-05-10T11:07:42.135192+0000","last_became_peered":"2024-05-10T11:07:42.135192+0000","last_unstale":"2024-05-10T11:13:13.287765+0000","last_undegraded":"2024-05-10T11:13:13.287765+0000","last_fullsized":"2024-05-10T11:13:13.287765+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:43:17.100873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470885+0000","last_change":"2024-05-10T11:07:47.199529+0000","last_active":"2024-05-10T11:13:13.470885+0000","last_peered":"2024-05-10T11:13:13.470885+0000","last_clean":"2024-05-10T11:13:13.470885+0000","last_became_active":"2024-05-10T11:07:47.199070+0000","last_became_peered":"2024-05-10T11:07:47.199070+0000","last_unstale":"2024-05-10T11:13:13.470885+0000","last_undegraded":"2024-05-10T11:13:13.470885+0000","last_fullsized":"2024-05-10T11:13:13.470885+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:24.446639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.279675+0000","last_change":"2024-05-10T11:08:00.898607+0000","last_active":"2024-05-10T11:13:13.279675+0000","last_peered":"2024-05-10T11:13:13.279675+0000","last_clean":"2024-05-10T11:13:13.279675+0000","last_became_active":"2024-05-10T11:08:00.897768+0000","last_became_peered":"2024-05-10T11:08:00.897768+0000","last_unstale":"2024-05-10T11:13:13.279675+0000","last_undegraded":"2024-05-10T11:13:13.279675+0000","last_fullsized":"2024-05-10T11:13:13.279675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:24:42.125203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'10","reported_seq":307,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.546406+0000","last_change":"2024-05-10T11:07:42.126907+0000","last_active":"2024-05-10T11:13:17.546406+0000","last_peered":"2024-05-10T11:13:17.546406+0000","last_clean":"2024-05-10T11:13:17.546406+0000","last_became_active":"2024-05-10T11:07:42.126462+0000","last_became_peered":"2024-05-10T11:07:42.126462+0000","last_unstale":"2024-05-10T11:13:17.546406+0000","last_undegraded":"2024-05-10T11:13:17.546406+0000","last_fullsized":"2024-05-10T11:13:17.546406+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:18:53.971150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2212,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":250,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470856+0000","last_change":"2024-05-10T11:07:47.191888+0000","last_active":"2024-05-10T11:13:13.470856+0000","last_peered":"2024-05-10T11:13:13.470856+0000","last_clean":"2024-05-10T11:13:13.470856+0000","last_became_active":"2024-05-10T11:07:47.191557+0000","last_became_peered":"2024-05-10T11:07:47.191557+0000","last_unstale":"2024-05-10T11:13:13.470856+0000","last_undegraded":"2024-05-10T11:13:13.470856+0000","last_fullsized":"2024-05-10T11:13:13.470856+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:01:36.529493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478804+0000","last_change":"2024-05-10T11:08:00.894385+0000","last_active":"2024-05-10T11:13:13.478804+0000","last_peered":"2024-05-10T11:13:13.478804+0000","last_clean":"2024-05-10T11:13:13.478804+0000","last_became_active":"2024-05-10T11:08:00.893706+0000","last_became_peered":"2024-05-10T11:08:00.893706+0000","last_unstale":"2024-05-10T11:13:13.478804+0000","last_undegraded":"2024-05-10T11:13:13.478804+0000","last_fullsized":"2024-05-10T11:13:13.478804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:44:02.069878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281605+0000","last_change":"2024-05-10T11:07:42.385661+0000","last_active":"2024-05-10T11:13:13.281605+0000","last_peered":"2024-05-10T11:13:13.281605+0000","last_clean":"2024-05-10T11:13:13.281605+0000","last_became_active":"2024-05-10T11:07:42.385308+0000","last_became_peered":"2024-05-10T11:07:42.385308+0000","last_unstale":"2024-05-10T11:13:13.281605+0000","last_undegraded":"2024-05-10T11:13:13.281605+0000","last_fullsized":"2024-05-10T11:13:13.281605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:57:25.997730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287903+0000","last_change":"2024-05-10T11:07:47.191574+0000","last_active":"2024-05-10T11:13:13.287903+0000","last_peered":"2024-05-10T11:13:13.287903+0000","last_clean":"2024-05-10T11:13:13.287903+0000","last_became_active":"2024-05-10T11:07:47.191210+0000","last_became_peered":"2024-05-10T11:07:47.191210+0000","last_unstale":"2024-05-10T11:13:13.287903+0000","last_undegraded":"2024-05-10T11:13:13.287903+0000","last_fullsized":"2024-05-10T11:13:13.287903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:11:35.900635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287778+0000","last_change":"2024-05-10T11:08:00.890076+0000","last_active":"2024-05-10T11:13:13.287778+0000","last_peered":"2024-05-10T11:13:13.287778+0000","last_clean":"2024-05-10T11:13:13.287778+0000","last_became_active":"2024-05-10T11:08:00.889630+0000","last_became_peered":"2024-05-10T11:08:00.889630+0000","last_unstale":"2024-05-10T11:13:13.287778+0000","last_undegraded":"2024-05-10T11:13:13.287778+0000","last_fullsized":"2024-05-10T11:13:13.287778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:53:29.257873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484819+0000","last_change":"2024-05-10T11:09:23.074550+0000","last_active":"2024-05-10T11:13:13.484819+0000","last_peered":"2024-05-10T11:13:13.484819+0000","last_clean":"2024-05-10T11:13:13.484819+0000","last_became_active":"2024-05-10T11:09:23.067169+0000","last_became_peered":"2024-05-10T11:09:23.067169+0000","last_unstale":"2024-05-10T11:13:13.484819+0000","last_undegraded":"2024-05-10T11:13:13.484819+0000","last_fullsized":"2024-05-10T11:13:13.484819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:45:06.303003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484821+0000","last_change":"2024-05-10T11:07:47.192042+0000","last_active":"2024-05-10T11:13:13.484821+0000","last_peered":"2024-05-10T11:13:13.484821+0000","last_clean":"2024-05-10T11:13:13.484821+0000","last_became_active":"2024-05-10T11:07:47.191632+0000","last_became_peered":"2024-05-10T11:07:47.191632+0000","last_unstale":"2024-05-10T11:13:13.484821+0000","last_undegraded":"2024-05-10T11:13:13.484821+0000","last_fullsized":"2024-05-10T11:13:13.484821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:55:08.564841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281643+0000","last_change":"2024-05-10T11:08:22.244721+0000","last_active":"2024-05-10T11:13:13.281643+0000","last_peered":"2024-05-10T11:13:13.281643+0000","last_clean":"2024-05-10T11:13:13.281643+0000","last_became_active":"2024-05-10T11:08:22.244274+0000","last_became_peered":"2024-05-10T11:08:22.244274+0000","last_unstale":"2024-05-10T11:13:13.281643+0000","last_undegraded":"2024-05-10T11:13:13.281643+0000","last_fullsized":"2024-05-10T11:13:13.281643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:26.218071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.279764+0000","last_change":"2024-05-10T11:07:42.388358+0000","last_active":"2024-05-10T11:13:13.279764+0000","last_peered":"2024-05-10T11:13:13.279764+0000","last_clean":"2024-05-10T11:13:13.279764+0000","last_became_active":"2024-05-10T11:07:42.387806+0000","last_became_peered":"2024-05-10T11:07:42.387806+0000","last_unstale":"2024-05-10T11:13:13.279764+0000","last_undegraded":"2024-05-10T11:13:13.279764+0000","last_fullsized":"2024-05-10T11:13:13.279764+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:27:38.387332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129592+0000","last_change":"2024-05-10T11:07:47.213602+0000","last_active":"2024-05-10T11:13:13.129592+0000","last_peered":"2024-05-10T11:13:13.129592+0000","last_clean":"2024-05-10T11:13:13.129592+0000","last_became_active":"2024-05-10T11:07:47.213184+0000","last_became_peered":"2024-05-10T11:07:47.213184+0000","last_unstale":"2024-05-10T11:13:13.129592+0000","last_undegraded":"2024-05-10T11:13:13.129592+0000","last_fullsized":"2024-05-10T11:13:13.129592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T23:01:52.253279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478398+0000","last_change":"2024-05-10T11:08:00.889850+0000","last_active":"2024-05-10T11:13:13.478398+0000","last_peered":"2024-05-10T11:13:13.478398+0000","last_clean":"2024-05-10T11:13:13.478398+0000","last_became_active":"2024-05-10T11:08:00.888526+0000","last_became_peered":"2024-05-10T11:08:00.888526+0000","last_unstale":"2024-05-10T11:13:13.478398+0000","last_undegraded":"2024-05-10T11:13:13.478398+0000","last_fullsized":"2024-05-10T11:13:13.478398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:11:43.899628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'6","reported_seq":299,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.590902+0000","last_change":"2024-05-10T11:07:42.130650+0000","last_active":"2024-05-10T11:13:17.590902+0000","last_peered":"2024-05-10T11:13:17.590902+0000","last_clean":"2024-05-10T11:13:17.590902+0000","last_became_active":"2024-05-10T11:07:42.130259+0000","last_became_peered":"2024-05-10T11:07:42.130259+0000","last_unstale":"2024-05-10T11:13:17.590902+0000","last_undegraded":"2024-05-10T11:13:17.590902+0000","last_fullsized":"2024-05-10T11:13:17.590902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.870092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129319+0000","last_change":"2024-05-10T11:07:47.204159+0000","last_active":"2024-05-10T11:13:13.129319+0000","last_peered":"2024-05-10T11:13:13.129319+0000","last_clean":"2024-05-10T11:13:13.129319+0000","last_became_active":"2024-05-10T11:07:47.203723+0000","last_became_peered":"2024-05-10T11:07:47.203723+0000","last_unstale":"2024-05-10T11:13:13.129319+0000","last_undegraded":"2024-05-10T11:13:13.129319+0000","last_fullsized":"2024-05-10T11:13:13.129319+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:44:48.232321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477989+0000","last_change":"2024-05-10T11:08:00.918501+0000","last_active":"2024-05-10T11:13:13.477989+0000","last_peered":"2024-05-10T11:13:13.477989+0000","last_clean":"2024-05-10T11:13:13.477989+0000","last_became_active":"2024-05-10T11:08:00.918071+0000","last_became_peered":"2024-05-10T11:08:00.918071+0000","last_unstale":"2024-05-10T11:13:13.477989+0000","last_undegraded":"2024-05-10T11:13:13.477989+0000","last_fullsized":"2024-05-10T11:13:13.477989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:23:08.390953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287957+0000","last_change":"2024-05-10T11:08:22.036385+0000","last_active":"2024-05-10T11:13:13.287957+0000","last_peered":"2024-05-10T11:13:13.287957+0000","last_clean":"2024-05-10T11:13:13.287957+0000","last_became_active":"2024-05-10T11:08:22.035969+0000","last_became_peered":"2024-05-10T11:08:22.035969+0000","last_unstale":"2024-05-10T11:13:13.287957+0000","last_undegraded":"2024-05-10T11:13:13.287957+0000","last_fullsized":"2024-05-10T11:13:13.287957+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:14:24.069422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":299,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.545641+0000","last_change":"2024-05-10T11:07:42.381926+0000","last_active":"2024-05-10T11:13:17.545641+0000","last_peered":"2024-05-10T11:13:17.545641+0000","last_clean":"2024-05-10T11:13:17.545641+0000","last_became_active":"2024-05-10T11:07:42.381301+0000","last_became_peered":"2024-05-10T11:07:42.381301+0000","last_unstale":"2024-05-10T11:13:17.545641+0000","last_undegraded":"2024-05-10T11:13:17.545641+0000","last_fullsized":"2024-05-10T11:13:17.545641+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:48:32.291029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":244,"num_read_kb":244,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470352+0000","last_change":"2024-05-10T11:08:00.889959+0000","last_active":"2024-05-10T11:13:13.470352+0000","last_peered":"2024-05-10T11:13:13.470352+0000","last_clean":"2024-05-10T11:13:13.470352+0000","last_became_active":"2024-05-10T11:08:00.889514+0000","last_became_peered":"2024-05-10T11:08:00.889514+0000","last_unstale":"2024-05-10T11:13:13.470352+0000","last_undegraded":"2024-05-10T11:13:13.470352+0000","last_fullsized":"2024-05-10T11:13:13.470352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:36:06.557175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129670+0000","last_change":"2024-05-10T11:07:47.218025+0000","last_active":"2024-05-10T11:13:13.129670+0000","last_peered":"2024-05-10T11:13:13.129670+0000","last_clean":"2024-05-10T11:13:13.129670+0000","last_became_active":"2024-05-10T11:07:47.207089+0000","last_became_peered":"2024-05-10T11:07:47.207089+0000","last_unstale":"2024-05-10T11:13:13.129670+0000","last_undegraded":"2024-05-10T11:13:13.129670+0000","last_fullsized":"2024-05-10T11:13:13.129670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:15:50.363129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470565+0000","last_change":"2024-05-10T11:07:42.138182+0000","last_active":"2024-05-10T11:13:13.470565+0000","last_peered":"2024-05-10T11:13:13.470565+0000","last_clean":"2024-05-10T11:13:13.470565+0000","last_became_active":"2024-05-10T11:07:42.137734+0000","last_became_peered":"2024-05-10T11:07:42.137734+0000","last_unstale":"2024-05-10T11:13:13.470565+0000","last_undegraded":"2024-05-10T11:13:13.470565+0000","last_fullsized":"2024-05-10T11:13:13.470565+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:02:30.778878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'99","reported_seq":151,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:15.289134+0000","last_change":"2024-05-10T11:06:06.321257+0000","last_active":"2024-05-10T11:13:15.289134+0000","last_peered":"2024-05-10T11:13:15.289134+0000","last_clean":"2024-05-10T11:13:15.289134+0000","last_became_active":"2024-05-10T11:06:06.316928+0000","last_became_peered":"2024-05-10T11:06:06.316928+0000","last_unstale":"2024-05-10T11:13:15.289134+0000","last_undegraded":"2024-05-10T11:13:15.289134+0000","last_fullsized":"2024-05-10T11:13:15.289134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:02:07.346111+0000","last_clean_scrub_stamp":"2024-05-10T11:02:07.346111+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:32:43.670246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.288280+0000","last_change":"2024-05-10T11:08:00.918499+0000","last_active":"2024-05-10T11:13:13.288280+0000","last_peered":"2024-05-10T11:13:13.288280+0000","last_clean":"2024-05-10T11:13:13.288280+0000","last_became_active":"2024-05-10T11:08:00.917950+0000","last_became_peered":"2024-05-10T11:08:00.917950+0000","last_unstale":"2024-05-10T11:13:13.288280+0000","last_undegraded":"2024-05-10T11:13:13.288280+0000","last_fullsized":"2024-05-10T11:13:13.288280+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:47:07.591312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477904+0000","last_change":"2024-05-10T11:07:47.194968+0000","last_active":"2024-05-10T11:13:13.477904+0000","last_peered":"2024-05-10T11:13:13.477904+0000","last_clean":"2024-05-10T11:13:13.477904+0000","last_became_active":"2024-05-10T11:07:47.194645+0000","last_became_peered":"2024-05-10T11:07:47.194645+0000","last_unstale":"2024-05-10T11:13:13.477904+0000","last_undegraded":"2024-05-10T11:13:13.477904+0000","last_fullsized":"2024-05-10T11:13:13.477904+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:58:34.962163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470732+0000","last_change":"2024-05-10T11:07:42.126378+0000","last_active":"2024-05-10T11:13:13.470732+0000","last_peered":"2024-05-10T11:13:13.470732+0000","last_clean":"2024-05-10T11:13:13.470732+0000","last_became_active":"2024-05-10T11:07:42.125984+0000","last_became_peered":"2024-05-10T11:07:42.125984+0000","last_unstale":"2024-05-10T11:13:13.470732+0000","last_undegraded":"2024-05-10T11:13:13.470732+0000","last_fullsized":"2024-05-10T11:13:13.470732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:15:50.378737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484726+0000","last_change":"2024-05-10T11:08:00.872342+0000","last_active":"2024-05-10T11:13:13.484726+0000","last_peered":"2024-05-10T11:13:13.484726+0000","last_clean":"2024-05-10T11:13:13.484726+0000","last_became_active":"2024-05-10T11:08:00.871926+0000","last_became_peered":"2024-05-10T11:08:00.871926+0000","last_unstale":"2024-05-10T11:13:13.484726+0000","last_undegraded":"2024-05-10T11:13:13.484726+0000","last_fullsized":"2024-05-10T11:13:13.484726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:27:16.519752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.282072+0000","last_change":"2024-05-10T11:07:42.384236+0000","last_active":"2024-05-10T11:13:13.282072+0000","last_peered":"2024-05-10T11:13:13.282072+0000","last_clean":"2024-05-10T11:13:13.282072+0000","last_became_active":"2024-05-10T11:07:42.383908+0000","last_became_peered":"2024-05-10T11:07:42.383908+0000","last_unstale":"2024-05-10T11:13:13.282072+0000","last_undegraded":"2024-05-10T11:13:13.282072+0000","last_fullsized":"2024-05-10T11:13:13.282072+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:09:42.302506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478716+0000","last_change":"2024-05-10T11:07:47.201645+0000","last_active":"2024-05-10T11:13:13.478716+0000","last_peered":"2024-05-10T11:13:13.478716+0000","last_clean":"2024-05-10T11:13:13.478716+0000","last_became_active":"2024-05-10T11:07:47.201300+0000","last_became_peered":"2024-05-10T11:07:47.201300+0000","last_unstale":"2024-05-10T11:13:13.478716+0000","last_undegraded":"2024-05-10T11:13:13.478716+0000","last_fullsized":"2024-05-10T11:13:13.478716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:37.268530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129032+0000","last_change":"2024-05-10T11:08:00.917988+0000","last_active":"2024-05-10T11:13:13.129032+0000","last_peered":"2024-05-10T11:13:13.129032+0000","last_clean":"2024-05-10T11:13:13.129032+0000","last_became_active":"2024-05-10T11:08:00.917457+0000","last_became_peered":"2024-05-10T11:08:00.917457+0000","last_unstale":"2024-05-10T11:13:13.129032+0000","last_undegraded":"2024-05-10T11:13:13.129032+0000","last_fullsized":"2024-05-10T11:13:13.129032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:16:01.816917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'53","reported_seq":124,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.424005+0000","last_change":"2024-05-10T11:07:42.141005+0000","last_active":"2024-05-10T11:13:13.424005+0000","last_peered":"2024-05-10T11:13:13.424005+0000","last_clean":"2024-05-10T11:13:13.424005+0000","last_became_active":"2024-05-10T11:07:42.140634+0000","last_became_peered":"2024-05-10T11:07:42.140634+0000","last_unstale":"2024-05-10T11:13:13.424005+0000","last_undegraded":"2024-05-10T11:13:13.424005+0000","last_fullsized":"2024-05-10T11:13:13.424005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:00:32.029374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":94,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.130553+0000","last_change":"2024-05-10T11:07:47.206864+0000","last_active":"2024-05-10T11:13:13.130553+0000","last_peered":"2024-05-10T11:13:13.130553+0000","last_clean":"2024-05-10T11:13:13.130553+0000","last_became_active":"2024-05-10T11:07:47.206387+0000","last_became_peered":"2024-05-10T11:07:47.206387+0000","last_unstale":"2024-05-10T11:13:13.130553+0000","last_undegraded":"2024-05-10T11:13:13.130553+0000","last_fullsized":"2024-05-10T11:13:13.130553+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:12:35.242270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132847+0000","last_change":"2024-05-10T11:08:00.867872+0000","last_active":"2024-05-10T11:13:13.132847+0000","last_peered":"2024-05-10T11:13:13.132847+0000","last_clean":"2024-05-10T11:13:13.132847+0000","last_became_active":"2024-05-10T11:08:00.867461+0000","last_became_peered":"2024-05-10T11:08:00.867461+0000","last_unstale":"2024-05-10T11:13:13.132847+0000","last_undegraded":"2024-05-10T11:13:13.132847+0000","last_fullsized":"2024-05-10T11:13:13.132847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:29.617335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470710+0000","last_change":"2024-05-10T11:07:42.599520+0000","last_active":"2024-05-10T11:13:13.470710+0000","last_peered":"2024-05-10T11:13:13.470710+0000","last_clean":"2024-05-10T11:13:13.470710+0000","last_became_active":"2024-05-10T11:07:42.599209+0000","last_became_peered":"2024-05-10T11:07:42.599209+0000","last_unstale":"2024-05-10T11:13:13.470710+0000","last_undegraded":"2024-05-10T11:13:13.470710+0000","last_fullsized":"2024-05-10T11:13:13.470710+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T17:08:52.899718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201837+0000","last_change":"2024-05-10T11:07:47.218286+0000","last_active":"2024-05-10T11:13:13.201837+0000","last_peered":"2024-05-10T11:13:13.201837+0000","last_clean":"2024-05-10T11:13:13.201837+0000","last_became_active":"2024-05-10T11:07:47.217960+0000","last_became_peered":"2024-05-10T11:07:47.217960+0000","last_unstale":"2024-05-10T11:13:13.201837+0000","last_undegraded":"2024-05-10T11:13:13.201837+0000","last_fullsized":"2024-05-10T11:13:13.201837+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:11:37.208359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.471251+0000","last_change":"2024-05-10T11:08:00.919578+0000","last_active":"2024-05-10T11:13:13.471251+0000","last_peered":"2024-05-10T11:13:13.471251+0000","last_clean":"2024-05-10T11:13:13.471251+0000","last_became_active":"2024-05-10T11:08:00.918546+0000","last_became_peered":"2024-05-10T11:08:00.918546+0000","last_unstale":"2024-05-10T11:13:13.471251+0000","last_undegraded":"2024-05-10T11:13:13.471251+0000","last_fullsized":"2024-05-10T11:13:13.471251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:39:05.062210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.476737+0000","last_change":"2024-05-10T11:07:42.384577+0000","last_active":"2024-05-10T11:13:13.476737+0000","last_peered":"2024-05-10T11:13:13.476737+0000","last_clean":"2024-05-10T11:13:13.476737+0000","last_became_active":"2024-05-10T11:07:42.384301+0000","last_became_peered":"2024-05-10T11:07:42.384301+0000","last_unstale":"2024-05-10T11:13:13.476737+0000","last_undegraded":"2024-05-10T11:13:13.476737+0000","last_fullsized":"2024-05-10T11:13:13.476737+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:11:03.359490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281603+0000","last_change":"2024-05-10T11:07:47.189708+0000","last_active":"2024-05-10T11:13:13.281603+0000","last_peered":"2024-05-10T11:13:13.281603+0000","last_clean":"2024-05-10T11:13:13.281603+0000","last_became_active":"2024-05-10T11:07:47.189367+0000","last_became_peered":"2024-05-10T11:07:47.189367+0000","last_unstale":"2024-05-10T11:13:13.281603+0000","last_undegraded":"2024-05-10T11:13:13.281603+0000","last_fullsized":"2024-05-10T11:13:13.281603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:00:25.410477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.279787+0000","last_change":"2024-05-10T11:08:00.899359+0000","last_active":"2024-05-10T11:13:13.279787+0000","last_peered":"2024-05-10T11:13:13.279787+0000","last_clean":"2024-05-10T11:13:13.279787+0000","last_became_active":"2024-05-10T11:08:00.898924+0000","last_became_peered":"2024-05-10T11:08:00.898924+0000","last_unstale":"2024-05-10T11:13:13.279787+0000","last_undegraded":"2024-05-10T11:13:13.279787+0000","last_fullsized":"2024-05-10T11:13:13.279787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:14:17.427245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129168+0000","last_change":"2024-05-10T11:07:42.140262+0000","last_active":"2024-05-10T11:13:13.129168+0000","last_peered":"2024-05-10T11:13:13.129168+0000","last_clean":"2024-05-10T11:13:13.129168+0000","last_became_active":"2024-05-10T11:07:42.138853+0000","last_became_peered":"2024-05-10T11:07:42.138853+0000","last_unstale":"2024-05-10T11:13:13.129168+0000","last_undegraded":"2024-05-10T11:13:13.129168+0000","last_fullsized":"2024-05-10T11:13:13.129168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:05:39.995237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484555+0000","last_change":"2024-05-10T11:07:47.203637+0000","last_active":"2024-05-10T11:13:13.484555+0000","last_peered":"2024-05-10T11:13:13.484555+0000","last_clean":"2024-05-10T11:13:13.484555+0000","last_became_active":"2024-05-10T11:07:47.203286+0000","last_became_peered":"2024-05-10T11:07:47.203286+0000","last_unstale":"2024-05-10T11:13:13.484555+0000","last_undegraded":"2024-05-10T11:13:13.484555+0000","last_fullsized":"2024-05-10T11:13:13.484555+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:27:46.924625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470388+0000","last_change":"2024-05-10T11:08:00.890082+0000","last_active":"2024-05-10T11:13:13.470388+0000","last_peered":"2024-05-10T11:13:13.470388+0000","last_clean":"2024-05-10T11:13:13.470388+0000","last_became_active":"2024-05-10T11:08:00.889673+0000","last_became_peered":"2024-05-10T11:08:00.889673+0000","last_unstale":"2024-05-10T11:13:13.470388+0000","last_undegraded":"2024-05-10T11:13:13.470388+0000","last_fullsized":"2024-05-10T11:13:13.470388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:21:31.167934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:14.010400+0000","last_change":"2024-05-10T11:07:42.130703+0000","last_active":"2024-05-10T11:13:14.010400+0000","last_peered":"2024-05-10T11:13:14.010400+0000","last_clean":"2024-05-10T11:13:14.010400+0000","last_became_active":"2024-05-10T11:07:42.130098+0000","last_became_peered":"2024-05-10T11:07:42.130098+0000","last_unstale":"2024-05-10T11:13:14.010400+0000","last_undegraded":"2024-05-10T11:13:14.010400+0000","last_fullsized":"2024-05-10T11:13:14.010400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:59:39.783151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129760+0000","last_change":"2024-05-10T11:07:47.220497+0000","last_active":"2024-05-10T11:13:13.129760+0000","last_peered":"2024-05-10T11:13:13.129760+0000","last_clean":"2024-05-10T11:13:13.129760+0000","last_became_active":"2024-05-10T11:07:47.219916+0000","last_became_peered":"2024-05-10T11:07:47.219916+0000","last_unstale":"2024-05-10T11:13:13.129760+0000","last_undegraded":"2024-05-10T11:13:13.129760+0000","last_fullsized":"2024-05-10T11:13:13.129760+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:26:43.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.471013+0000","last_change":"2024-05-10T11:08:00.918258+0000","last_active":"2024-05-10T11:13:13.471013+0000","last_peered":"2024-05-10T11:13:13.471013+0000","last_clean":"2024-05-10T11:13:13.471013+0000","last_became_active":"2024-05-10T11:08:00.917808+0000","last_became_peered":"2024-05-10T11:08:00.917808+0000","last_unstale":"2024-05-10T11:13:13.471013+0000","last_undegraded":"2024-05-10T11:13:13.471013+0000","last_fullsized":"2024-05-10T11:13:13.471013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:14:08.867024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.281805+0000","last_change":"2024-05-10T11:07:42.127661+0000","last_active":"2024-05-10T11:13:13.281805+0000","last_peered":"2024-05-10T11:13:13.281805+0000","last_clean":"2024-05-10T11:13:13.281805+0000","last_became_active":"2024-05-10T11:07:42.127256+0000","last_became_peered":"2024-05-10T11:07:42.127256+0000","last_unstale":"2024-05-10T11:13:13.281805+0000","last_undegraded":"2024-05-10T11:13:13.281805+0000","last_fullsized":"2024-05-10T11:13:13.281805+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T13:14:37.821350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.469988+0000","last_change":"2024-05-10T11:07:47.199589+0000","last_active":"2024-05-10T11:13:13.469988+0000","last_peered":"2024-05-10T11:13:13.469988+0000","last_clean":"2024-05-10T11:13:13.469988+0000","last_became_active":"2024-05-10T11:07:47.199145+0000","last_became_peered":"2024-05-10T11:07:47.199145+0000","last_unstale":"2024-05-10T11:13:13.469988+0000","last_undegraded":"2024-05-10T11:13:13.469988+0000","last_fullsized":"2024-05-10T11:13:13.469988+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:47:46.577543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.477347+0000","last_change":"2024-05-10T11:09:23.085049+0000","last_active":"2024-05-10T11:13:13.477347+0000","last_peered":"2024-05-10T11:13:13.477347+0000","last_clean":"2024-05-10T11:13:13.477347+0000","last_became_active":"2024-05-10T11:09:23.084420+0000","last_became_peered":"2024-05-10T11:09:23.084420+0000","last_unstale":"2024-05-10T11:13:13.477347+0000","last_undegraded":"2024-05-10T11:13:13.477347+0000","last_fullsized":"2024-05-10T11:13:13.477347+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:32:33.346434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":300,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:17.413818+0000","last_change":"2024-05-10T11:07:42.599395+0000","last_active":"2024-05-10T11:13:17.413818+0000","last_peered":"2024-05-10T11:13:17.413818+0000","last_clean":"2024-05-10T11:13:17.413818+0000","last_became_active":"2024-05-10T11:07:42.598934+0000","last_became_peered":"2024-05-10T11:07:42.598934+0000","last_unstale":"2024-05-10T11:13:17.413818+0000","last_undegraded":"2024-05-10T11:13:17.413818+0000","last_fullsized":"2024-05-10T11:13:17.413818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:49:42.414789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":246,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470676+0000","last_change":"2024-05-10T11:07:47.192087+0000","last_active":"2024-05-10T11:13:13.470676+0000","last_peered":"2024-05-10T11:13:13.470676+0000","last_clean":"2024-05-10T11:13:13.470676+0000","last_became_active":"2024-05-10T11:07:47.191041+0000","last_became_peered":"2024-05-10T11:07:47.191041+0000","last_unstale":"2024-05-10T11:13:13.470676+0000","last_undegraded":"2024-05-10T11:13:13.470676+0000","last_fullsized":"2024-05-10T11:13:13.470676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:50:14.052863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201037+0000","last_change":"2024-05-10T11:09:23.080432+0000","last_active":"2024-05-10T11:13:13.201037+0000","last_peered":"2024-05-10T11:13:13.201037+0000","last_clean":"2024-05-10T11:13:13.201037+0000","last_became_active":"2024-05-10T11:09:23.074934+0000","last_became_peered":"2024-05-10T11:09:23.074934+0000","last_unstale":"2024-05-10T11:13:13.201037+0000","last_undegraded":"2024-05-10T11:13:13.201037+0000","last_fullsized":"2024-05-10T11:13:13.201037+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:12:14.613323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'41","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.414569+0000","last_change":"2024-05-10T11:07:42.130718+0000","last_active":"2024-05-10T11:13:13.414569+0000","last_peered":"2024-05-10T11:13:13.414569+0000","last_clean":"2024-05-10T11:13:13.414569+0000","last_became_active":"2024-05-10T11:07:42.130355+0000","last_became_peered":"2024-05-10T11:07:42.130355+0000","last_unstale":"2024-05-10T11:13:13.414569+0000","last_undegraded":"2024-05-10T11:13:13.414569+0000","last_fullsized":"2024-05-10T11:13:13.414569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:16:48.093146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470333+0000","last_change":"2024-05-10T11:07:47.213569+0000","last_active":"2024-05-10T11:13:13.470333+0000","last_peered":"2024-05-10T11:13:13.470333+0000","last_clean":"2024-05-10T11:13:13.470333+0000","last_became_active":"2024-05-10T11:07:47.213148+0000","last_became_peered":"2024-05-10T11:07:47.213148+0000","last_unstale":"2024-05-10T11:13:13.470333+0000","last_undegraded":"2024-05-10T11:13:13.470333+0000","last_fullsized":"2024-05-10T11:13:13.470333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:42:41.161274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287545+0000","last_change":"2024-05-10T11:08:00.870937+0000","last_active":"2024-05-10T11:13:13.287545+0000","last_peered":"2024-05-10T11:13:13.287545+0000","last_clean":"2024-05-10T11:13:13.287545+0000","last_became_active":"2024-05-10T11:08:00.869276+0000","last_became_peered":"2024-05-10T11:08:00.869276+0000","last_unstale":"2024-05-10T11:13:13.287545+0000","last_undegraded":"2024-05-10T11:13:13.287545+0000","last_fullsized":"2024-05-10T11:13:13.287545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:59.844349+0000","last_clean_scrub_stamp":"2024-05-10T11:07:59.844349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:08:37.058875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478921+0000","last_change":"2024-05-10T11:07:42.130544+0000","last_active":"2024-05-10T11:13:13.478921+0000","last_peered":"2024-05-10T11:13:13.478921+0000","last_clean":"2024-05-10T11:13:13.478921+0000","last_became_active":"2024-05-10T11:07:42.128912+0000","last_became_peered":"2024-05-10T11:07:42.128912+0000","last_unstale":"2024-05-10T11:13:13.478921+0000","last_undegraded":"2024-05-10T11:13:13.478921+0000","last_fullsized":"2024-05-10T11:13:13.478921+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:41.089878+0000","last_clean_scrub_stamp":"2024-05-10T11:07:41.089878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:22:23.521911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.470271+0000","last_change":"2024-05-10T11:07:47.194346+0000","last_active":"2024-05-10T11:13:13.470271+0000","last_peered":"2024-05-10T11:13:13.470271+0000","last_clean":"2024-05-10T11:13:13.470271+0000","last_became_active":"2024-05-10T11:07:47.194019+0000","last_became_peered":"2024-05-10T11:07:47.194019+0000","last_unstale":"2024-05-10T11:13:13.470271+0000","last_undegraded":"2024-05-10T11:13:13.470271+0000","last_fullsized":"2024-05-10T11:13:13.470271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T20:33:05.262347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484623+0000","last_change":"2024-05-10T11:07:47.192001+0000","last_active":"2024-05-10T11:13:13.484623+0000","last_peered":"2024-05-10T11:13:13.484623+0000","last_clean":"2024-05-10T11:13:13.484623+0000","last_became_active":"2024-05-10T11:07:47.191668+0000","last_became_peered":"2024-05-10T11:07:47.191668+0000","last_unstale":"2024-05-10T11:13:13.484623+0000","last_undegraded":"2024-05-10T11:13:13.484623+0000","last_fullsized":"2024-05-10T11:13:13.484623+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T16:13:33.680899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132073+0000","last_change":"2024-05-10T11:07:47.192226+0000","last_active":"2024-05-10T11:13:13.132073+0000","last_peered":"2024-05-10T11:13:13.132073+0000","last_clean":"2024-05-10T11:13:13.132073+0000","last_became_active":"2024-05-10T11:07:47.191444+0000","last_became_peered":"2024-05-10T11:07:47.191444+0000","last_unstale":"2024-05-10T11:13:13.132073+0000","last_undegraded":"2024-05-10T11:13:13.132073+0000","last_fullsized":"2024-05-10T11:13:13.132073+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T12:33:08.348986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.131701+0000","last_change":"2024-05-10T11:07:47.210801+0000","last_active":"2024-05-10T11:13:13.131701+0000","last_peered":"2024-05-10T11:13:13.131701+0000","last_clean":"2024-05-10T11:13:13.131701+0000","last_became_active":"2024-05-10T11:07:47.210496+0000","last_became_peered":"2024-05-10T11:07:47.210496+0000","last_unstale":"2024-05-10T11:13:13.131701+0000","last_undegraded":"2024-05-10T11:13:13.131701+0000","last_fullsized":"2024-05-10T11:13:13.131701+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:11:03.661457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.201213+0000","last_change":"2024-05-10T11:07:47.211033+0000","last_active":"2024-05-10T11:13:13.201213+0000","last_peered":"2024-05-10T11:13:13.201213+0000","last_clean":"2024-05-10T11:13:13.201213+0000","last_became_active":"2024-05-10T11:07:47.206061+0000","last_became_peered":"2024-05-10T11:07:47.206061+0000","last_unstale":"2024-05-10T11:13:13.201213+0000","last_undegraded":"2024-05-10T11:13:13.201213+0000","last_fullsized":"2024-05-10T11:13:13.201213+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T19:04:05.878270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478582+0000","last_change":"2024-05-10T11:07:47.214966+0000","last_active":"2024-05-10T11:13:13.478582+0000","last_peered":"2024-05-10T11:13:13.478582+0000","last_clean":"2024-05-10T11:13:13.478582+0000","last_became_active":"2024-05-10T11:07:47.214563+0000","last_became_peered":"2024-05-10T11:07:47.214563+0000","last_unstale":"2024-05-10T11:13:13.478582+0000","last_undegraded":"2024-05-10T11:13:13.478582+0000","last_fullsized":"2024-05-10T11:13:13.478582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T11:29:01.428893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.485051+0000","last_change":"2024-05-10T11:07:47.196740+0000","last_active":"2024-05-10T11:13:13.485051+0000","last_peered":"2024-05-10T11:13:13.485051+0000","last_clean":"2024-05-10T11:13:13.485051+0000","last_became_active":"2024-05-10T11:07:47.196388+0000","last_became_peered":"2024-05-10T11:07:47.196388+0000","last_unstale":"2024-05-10T11:13:13.485051+0000","last_undegraded":"2024-05-10T11:13:13.485051+0000","last_fullsized":"2024-05-10T11:13:13.485051+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:23:09.055338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.132107+0000","last_change":"2024-05-10T11:07:47.193568+0000","last_active":"2024-05-10T11:13:13.132107+0000","last_peered":"2024-05-10T11:13:13.132107+0000","last_clean":"2024-05-10T11:13:13.132107+0000","last_became_active":"2024-05-10T11:07:47.193202+0000","last_became_peered":"2024-05-10T11:07:47.193202+0000","last_unstale":"2024-05-10T11:13:13.132107+0000","last_undegraded":"2024-05-10T11:13:13.132107+0000","last_fullsized":"2024-05-10T11:13:13.132107+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T21:36:24.556266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.130958+0000","last_change":"2024-05-10T11:07:47.215344+0000","last_active":"2024-05-10T11:13:13.130958+0000","last_peered":"2024-05-10T11:13:13.130958+0000","last_clean":"2024-05-10T11:13:13.130958+0000","last_became_active":"2024-05-10T11:07:47.214896+0000","last_became_peered":"2024-05-10T11:07:47.214896+0000","last_unstale":"2024-05-10T11:13:13.130958+0000","last_undegraded":"2024-05-10T11:13:13.130958+0000","last_fullsized":"2024-05-10T11:13:13.130958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:23:07.845212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.287656+0000","last_change":"2024-05-10T11:07:47.219047+0000","last_active":"2024-05-10T11:13:13.287656+0000","last_peered":"2024-05-10T11:13:13.287656+0000","last_clean":"2024-05-10T11:13:13.287656+0000","last_became_active":"2024-05-10T11:07:47.218778+0000","last_became_peered":"2024-05-10T11:07:47.218778+0000","last_unstale":"2024-05-10T11:13:13.287656+0000","last_undegraded":"2024-05-10T11:13:13.287656+0000","last_fullsized":"2024-05-10T11:13:13.287656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T22:02:48.945259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.129317+0000","last_change":"2024-05-10T11:07:47.213412+0000","last_active":"2024-05-10T11:13:13.129317+0000","last_peered":"2024-05-10T11:13:13.129317+0000","last_clean":"2024-05-10T11:13:13.129317+0000","last_became_active":"2024-05-10T11:07:47.212963+0000","last_became_peered":"2024-05-10T11:07:47.212963+0000","last_unstale":"2024-05-10T11:13:13.129317+0000","last_undegraded":"2024-05-10T11:13:13.129317+0000","last_fullsized":"2024-05-10T11:13:13.129317+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T15:33:05.224184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.476853+0000","last_change":"2024-05-10T11:07:47.207998+0000","last_active":"2024-05-10T11:13:13.476853+0000","last_peered":"2024-05-10T11:13:13.476853+0000","last_clean":"2024-05-10T11:13:13.476853+0000","last_became_active":"2024-05-10T11:07:47.207298+0000","last_became_peered":"2024-05-10T11:07:47.207298+0000","last_unstale":"2024-05-10T11:13:13.476853+0000","last_undegraded":"2024-05-10T11:13:13.476853+0000","last_fullsized":"2024-05-10T11:13:13.476853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:59:14.001103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.478208+0000","last_change":"2024-05-10T11:07:47.202216+0000","last_active":"2024-05-10T11:13:13.478208+0000","last_peered":"2024-05-10T11:13:13.478208+0000","last_clean":"2024-05-10T11:13:13.478208+0000","last_became_active":"2024-05-10T11:07:47.201867+0000","last_became_peered":"2024-05-10T11:07:47.201867+0000","last_unstale":"2024-05-10T11:13:13.478208+0000","last_undegraded":"2024-05-10T11:13:13.478208+0000","last_fullsized":"2024-05-10T11:13:13.478208+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T14:48:59.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-10T11:13:13.484820+0000","last_change":"2024-05-10T11:07:47.194149+0000","last_active":"2024-05-10T11:13:13.484820+0000","last_peered":"2024-05-10T11:13:13.484820+0000","last_clean":"2024-05-10T11:13:13.484820+0000","last_became_active":"2024-05-10T11:07:47.193838+0000","last_became_peered":"2024-05-10T11:07:47.193838+0000","last_unstale":"2024-05-10T11:13:13.484820+0000","last_undegraded":"2024-05-10T11:13:13.484820+0000","last_fullsized":"2024-05-10T11:13:13.484820+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-10T11:07:46.137479+0000","last_clean_scrub_stamp":"2024-05-10T11:07:46.137479+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-11T18:46:44.181446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107385,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":607246,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2569,"num_read_kb":1410,"num_write":392,"num_write_kb":1483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":712,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":737280,"data_stored":1821738,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678099,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":532988,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874329,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32340,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712812,"statfs":{"total":95995035648,"available":95961919488,"internally_reserved":0,"allocated":1191936,"data_stored":1325367,"data_compressed":77418,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070558,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":929749,"data_compressed":54613,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266789,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":926009,"data_compressed":54613,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863463018,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":1131718,"data_compressed":71262,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626544,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1011712,"data_stored":528332,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574988,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":602182,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738515,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":529788,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":601712,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182529,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1052465,"data_compressed":60769,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986300,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":535043,"data_compressed":35467,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822775,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1114112,"data_stored":1256885,"data_compressed":77418,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35795,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":74493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135202,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":394694,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":207780,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-10T11:13:22.334 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:13:22.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:22 smithi189 ceph-mon[34842]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:13:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:22 smithi189 ceph-mon[34842]: Updating smithi027:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:13:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:22 smithi189 ceph-mon[34842]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 880 B/s wr, 19 op/s 2024-05-10T11:13:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:22 smithi189 ceph-mon[34842]: from='client.25312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:13:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:22 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:22 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:22 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:22 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:22 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:22 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:22.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:22 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:22 smithi027 ceph-mon[30054]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:13:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:22 smithi027 ceph-mon[30054]: Updating smithi027:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:13:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:22 smithi027 ceph-mon[30054]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 880 B/s wr, 19 op/s 2024-05-10T11:13:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:22 smithi027 ceph-mon[30054]: from='client.25312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:13:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:22 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:22 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:22 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:22 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:22 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:22 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:22 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:22.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:22 smithi076 ceph-mon[34152]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-05-10T11:13:22.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:22 smithi076 ceph-mon[34152]: Updating smithi027:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-10T11:13:22.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:22 smithi076 ceph-mon[34152]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 880 B/s wr, 19 op/s 2024-05-10T11:13:22.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:22 smithi076 ceph-mon[34152]: from='client.25312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:13:22.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:22 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:22.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:22 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:22.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:22 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:22.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:22 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:22.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:22 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:22.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:22 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:22.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:22 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:22.850 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:13:23.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:23 smithi027 ceph-mon[30054]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:13:23.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:23 smithi076 ceph-mon[34152]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:13:23.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:23 smithi189 ceph-mon[34842]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:13:24.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:24 smithi027 ceph-mon[30054]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:13:24.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:24 smithi027 ceph-mon[30054]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 684 B/s wr, 14 op/s 2024-05-10T11:13:24.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:24 smithi076 ceph-mon[34152]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:13:24.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:24 smithi076 ceph-mon[34152]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 684 B/s wr, 14 op/s 2024-05-10T11:13:24.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:24 smithi189 ceph-mon[34842]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-10T11:13:24.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:24 smithi189 ceph-mon[34842]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 684 B/s wr, 14 op/s 2024-05-10T11:13:25.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.237+0000 7f8848000700 1 -- 172.21.15.27:0/4187850404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f884006b180 msgr2=0x7f884006b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:25.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.237+0000 7f8848000700 1 --2- 172.21.15.27:0/4187850404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f884006b180 0x7f884006b600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f882c0098f0 tx=0x7f882c009c00 comp rx=0 tx=0).stop 2024-05-10T11:13:25.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.237+0000 7f8848000700 1 -- 172.21.15.27:0/4187850404 shutdown_connections 2024-05-10T11:13:25.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.237+0000 7f8848000700 1 --2- 172.21.15.27:0/4187850404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8840109510 0x7f88401098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:25.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.237+0000 7f8848000700 1 --2- 172.21.15.27:0/4187850404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884006bb40 0x7f8840110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:25.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.237+0000 7f8848000700 1 --2- 172.21.15.27:0/4187850404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f884006b180 0x7f884006b600 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:25.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.237+0000 7f8848000700 1 -- 172.21.15.27:0/4187850404 >> 172.21.15.27:0/4187850404 conn(0x7f8840079350 msgr2=0x7f8840079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:25.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.237+0000 7f8848000700 1 -- 172.21.15.27:0/4187850404 shutdown_connections 2024-05-10T11:13:25.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.237+0000 7f8848000700 1 -- 172.21.15.27:0/4187850404 wait complete. 2024-05-10T11:13:25.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.238+0000 7f8848000700 1 Processor -- start 2024-05-10T11:13:25.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.238+0000 7f8848000700 1 -- start start 2024-05-10T11:13:25.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.239+0000 7f8848000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f884006b180 0x7f884019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:25.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.239+0000 7f8848000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884006bb40 0x7f884019fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:25.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.239+0000 7f8848000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8840109510 0x7f8840199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:25.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.239+0000 7f8848000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8840113640 con 0x7f884006bb40 2024-05-10T11:13:25.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.239+0000 7f8848000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f88401134c0 con 0x7f8840109510 2024-05-10T11:13:25.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.239+0000 7f8848000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f88401137c0 con 0x7f884006b180 2024-05-10T11:13:25.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.239+0000 7f8844c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884006bb40 0x7f884019fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:25.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.239+0000 7f8845600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f884006b180 0x7f884019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:25.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.239+0000 7f8845600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f884006b180 0x7f884019f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:59226/0 (socket says 172.21.15.27:59226) 2024-05-10T11:13:25.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.239+0000 7f8845600700 1 -- 172.21.15.27:0/1710682864 learned_addr learned my addr 172.21.15.27:0/1710682864 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:25.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.239+0000 7f8846000700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8840109510 0x7f8840199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:25.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.240+0000 7f8845600700 1 -- 172.21.15.27:0/1710682864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8840109510 msgr2=0x7f8840199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:25.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.240+0000 7f8845600700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8840109510 0x7f8840199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:25.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.240+0000 7f8845600700 1 -- 172.21.15.27:0/1710682864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884006bb40 msgr2=0x7f884019fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:25.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.240+0000 7f8845600700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884006bb40 0x7f884019fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:25.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.240+0000 7f8845600700 1 -- 172.21.15.27:0/1710682864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884019a160 con 0x7f884006b180 2024-05-10T11:13:25.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.240+0000 7f8845600700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f884006b180 0x7f884019f840 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f883400e9f0 tx=0x7f883400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:25.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.240+0000 7f883e000700 1 -- 172.21.15.27:0/1710682864 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8834004a30 con 0x7f884006b180 2024-05-10T11:13:25.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.241+0000 7f8848000700 1 -- 172.21.15.27:0/1710682864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f884019a450 con 0x7f884006b180 2024-05-10T11:13:25.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.241+0000 7f883e000700 1 -- 172.21.15.27:0/1710682864 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f883400cc70 con 0x7f884006b180 2024-05-10T11:13:25.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.241+0000 7f883e000700 1 -- 172.21.15.27:0/1710682864 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88340055e0 con 0x7f884006b180 2024-05-10T11:13:25.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.241+0000 7f8848000700 1 -- 172.21.15.27:0/1710682864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88401a6710 con 0x7f884006b180 2024-05-10T11:13:25.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.242+0000 7f8848000700 1 -- 172.21.15.27:0/1710682864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f884006a190 con 0x7f884006b180 2024-05-10T11:13:25.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.244+0000 7f883e000700 1 -- 172.21.15.27:0/1710682864 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f883400c7c0 con 0x7f884006b180 2024-05-10T11:13:25.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.246+0000 7f883e000700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f88280b2b10 0x7f88280b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:25.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.246+0000 7f883e000700 1 -- 172.21.15.27:0/1710682864 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f88340e2fd0 con 0x7f884006b180 2024-05-10T11:13:25.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.247+0000 7f8844c00700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f88280b2b10 0x7f88280b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:25.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.247+0000 7f8844c00700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f88280b2b10 0x7f88280b4fd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f882c009fd0 tx=0x7f882c019040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:25.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.248+0000 7f883e000700 1 -- 172.21.15.27:0/1710682864 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f88340aa330 con 0x7f884006b180 2024-05-10T11:13:25.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.536+0000 7f8848000700 1 -- 172.21.15.27:0/1710682864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f884002e510 con 0x7f884006b180 2024-05-10T11:13:25.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.537+0000 7f883e000700 1 -- 172.21.15.27:0/1710682864 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11936 (secure 0 0 0) 0x7f883400ce20 con 0x7f884006b180 2024-05-10T11:13:25.538 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:13:25.538 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:13:10.916396+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24484,24472,24467,24488]},"id":1}]} 2024-05-10T11:13:25.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.541+0000 7f8848000700 1 -- 172.21.15.27:0/1710682864 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f88280b2b10 msgr2=0x7f88280b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:25.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.541+0000 7f8848000700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f88280b2b10 0x7f88280b4fd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f882c009fd0 tx=0x7f882c019040 comp rx=0 tx=0).stop 2024-05-10T11:13:25.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.541+0000 7f8848000700 1 -- 172.21.15.27:0/1710682864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f884006b180 msgr2=0x7f884019f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:25.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.541+0000 7f8848000700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f884006b180 0x7f884019f840 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f883400e9f0 tx=0x7f883400ed00 comp rx=0 tx=0).stop 2024-05-10T11:13:25.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.541+0000 7f8848000700 1 -- 172.21.15.27:0/1710682864 shutdown_connections 2024-05-10T11:13:25.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.541+0000 7f8848000700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f88280b2b10 0x7f88280b4fd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:25.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.541+0000 7f8848000700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f884006b180 0x7f884019f840 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:25.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.541+0000 7f8848000700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8840109510 0x7f8840199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:25.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.541+0000 7f8848000700 1 --2- 172.21.15.27:0/1710682864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884006bb40 0x7f884019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:25.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.541+0000 7f8848000700 1 -- 172.21.15.27:0/1710682864 >> 172.21.15.27:0/1710682864 conn(0x7f8840079350 msgr2=0x7f8840101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:25.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.541+0000 7f8848000700 1 -- 172.21.15.27:0/1710682864 shutdown_connections 2024-05-10T11:13:25.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:25.541+0000 7f8848000700 1 -- 172.21.15.27:0/1710682864 wait complete. 2024-05-10T11:13:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 30 2024-05-10T11:13:26.538 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:26 smithi027 ceph-mon[30054]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 559 B/s wr, 16 op/s 2024-05-10T11:13:26.538 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:26 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1710682864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:13:26.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:13:26.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:26 smithi076 ceph-mon[34152]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 559 B/s wr, 16 op/s 2024-05-10T11:13:26.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:26 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1710682864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:13:26.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:26 smithi189 ceph-mon[34842]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 559 B/s wr, 16 op/s 2024-05-10T11:13:26.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:26 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1710682864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:13:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:13:27.463 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:27 smithi027 ceph-mon[30054]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 511 B/s wr, 15 op/s 2024-05-10T11:13:27.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:27 smithi076 ceph-mon[34152]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 511 B/s wr, 15 op/s 2024-05-10T11:13:27.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:27 smithi189 ceph-mon[34842]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 511 B/s wr, 15 op/s 2024-05-10T11:13:29.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.566+0000 7f5fcbe00700 1 -- 172.21.15.27:0/2773395881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fcc0779d0 msgr2=0x7f5fcc077db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:29.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.566+0000 7f5fcbe00700 1 --2- 172.21.15.27:0/2773395881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fcc0779d0 0x7f5fcc077db0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009f90 tx=0x7f5fbc00b330 comp rx=0 tx=0).stop 2024-05-10T11:13:29.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.566+0000 7f5fcbe00700 1 -- 172.21.15.27:0/2773395881 shutdown_connections 2024-05-10T11:13:29.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.566+0000 7f5fcbe00700 1 --2- 172.21.15.27:0/2773395881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fcc0779d0 0x7f5fcc077db0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:29.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.566+0000 7f5fcbe00700 1 --2- 172.21.15.27:0/2773395881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fcc114250 0x7f5fcc116640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:29.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.566+0000 7f5fcbe00700 1 --2- 172.21.15.27:0/2773395881 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fcc0782f0 0x7f5fcc113d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:29.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.566+0000 7f5fcbe00700 1 -- 172.21.15.27:0/2773395881 >> 172.21.15.27:0/2773395881 conn(0x7f5fcc1010e0 msgr2=0x7f5fcc103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:29.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.566+0000 7f5fcbe00700 1 -- 172.21.15.27:0/2773395881 shutdown_connections 2024-05-10T11:13:29.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.566+0000 7f5fcbe00700 1 -- 172.21.15.27:0/2773395881 wait complete. 2024-05-10T11:13:29.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.568+0000 7f5fcbe00700 1 Processor -- start 2024-05-10T11:13:29.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.569+0000 7f5fcbe00700 1 -- start start 2024-05-10T11:13:29.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.569+0000 7f5fcbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fcc0779d0 0x7f5fcc1a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:29.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fcbe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fcc0782f0 0x7f5fcc1a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:29.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fcbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fcc114250 0x7f5fcc1a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:29.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fcbe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5fcc119c10 con 0x7f5fcc114250 2024-05-10T11:13:29.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fcbe00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5fcc119a90 con 0x7f5fcc0782f0 2024-05-10T11:13:29.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fcbe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5fcc119d90 con 0x7f5fcc0779d0 2024-05-10T11:13:29.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fc2000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fcc0782f0 0x7f5fcc1a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:29.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fcaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fcc0779d0 0x7f5fcc1a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:29.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fc2000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fcc0782f0 0x7f5fcc1a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50220/0 (socket says 172.21.15.27:50220) 2024-05-10T11:13:29.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fcaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fcc0779d0 0x7f5fcc1a0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:59242/0 (socket says 172.21.15.27:59242) 2024-05-10T11:13:29.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fc2000700 1 -- 172.21.15.27:0/4135461852 learned_addr learned my addr 172.21.15.27:0/4135461852 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:29.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fcb400700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fcc114250 0x7f5fcc1a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:29.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fcaa00700 1 -- 172.21.15.27:0/4135461852 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fcc0782f0 msgr2=0x7f5fcc1a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:29.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fcaa00700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fcc0782f0 0x7f5fcc1a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:29.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fcaa00700 1 -- 172.21.15.27:0/4135461852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fcc114250 msgr2=0x7f5fcc1a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:29.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.570+0000 7f5fcaa00700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fcc114250 0x7f5fcc1a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:29.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.571+0000 7f5fcaa00700 1 -- 172.21.15.27:0/4135461852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fcc1a51d0 con 0x7f5fcc0779d0 2024-05-10T11:13:29.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.571+0000 7f5fcaa00700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fcc0779d0 0x7f5fcc1a0210 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc000f80 tx=0x7f5fbc00ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:29.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.571+0000 7f5fc3e00700 1 -- 172.21.15.27:0/4135461852 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc021070 con 0x7f5fcc0779d0 2024-05-10T11:13:29.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.571+0000 7f5fcbe00700 1 -- 172.21.15.27:0/4135461852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fcc1a5460 con 0x7f5fcc0779d0 2024-05-10T11:13:29.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.571+0000 7f5fc3e00700 1 -- 172.21.15.27:0/4135461852 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fbc007ac0 con 0x7f5fcc0779d0 2024-05-10T11:13:29.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.571+0000 7f5fc3e00700 1 -- 172.21.15.27:0/4135461852 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc005040 con 0x7f5fcc0779d0 2024-05-10T11:13:29.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.572+0000 7f5fcbe00700 1 -- 172.21.15.27:0/4135461852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fcc1acd40 con 0x7f5fcc0779d0 2024-05-10T11:13:29.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.573+0000 7f5fcbe00700 1 -- 172.21.15.27:0/4135461852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fcc06a190 con 0x7f5fcc0779d0 2024-05-10T11:13:29.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.575+0000 7f5fc3e00700 1 -- 172.21.15.27:0/4135461852 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f5fbc024070 con 0x7f5fcc0779d0 2024-05-10T11:13:29.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.577+0000 7f5fc3e00700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f5fb00b2b10 0x7f5fb00b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:29.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.577+0000 7f5fc3e00700 1 -- 172.21.15.27:0/4135461852 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f5fbc00f020 con 0x7f5fcc0779d0 2024-05-10T11:13:29.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.577+0000 7f5fc2000700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f5fb00b2b10 0x7f5fb00b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:29.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.578+0000 7f5fc2000700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f5fb00b2b10 0x7f5fb00b4fd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4009680 tx=0x7f5fb4009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.579+0000 7f5fc3e00700 1 -- 172.21.15.27:0/4135461852 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5fbc02b020 con 0x7f5fcc0779d0 2024-05-10T11:13:29.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.820+0000 7f5fcbe00700 1 -- 172.21.15.27:0/4135461852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5fcc1a1520 con 0x7f5fcc0779d0 2024-05-10T11:13:29.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.822+0000 7f5fc3e00700 1 -- 172.21.15.27:0/4135461852 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25213 (secure 0 0 0) 0x7f5fbc0ab7e0 con 0x7f5fcc0779d0 2024-05-10T11:13:29.823 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:13:29.823 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":91,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:13:13.118756+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6},{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1021663132":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6801/1704535643":"2024-05-11T11:13:13.118713+0000","172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/1450791252":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1410017298":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/1704535643":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/2991615845":"2024-05-11T11:13:13.118713+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:13:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.825+0000 7f5fcbe00700 1 -- 172.21.15.27:0/4135461852 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f5fb00b2b10 msgr2=0x7f5fb00b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.825+0000 7f5fcbe00700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f5fb00b2b10 0x7f5fb00b4fd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4009680 tx=0x7f5fb4009260 comp rx=0 tx=0).stop 2024-05-10T11:13:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.825+0000 7f5fcbe00700 1 -- 172.21.15.27:0/4135461852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fcc0779d0 msgr2=0x7f5fcc1a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:29.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.825+0000 7f5fcbe00700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fcc0779d0 0x7f5fcc1a0210 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc000f80 tx=0x7f5fbc00ccb0 comp rx=0 tx=0).stop 2024-05-10T11:13:29.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.826+0000 7f5fcbe00700 1 -- 172.21.15.27:0/4135461852 shutdown_connections 2024-05-10T11:13:29.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.826+0000 7f5fcbe00700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f5fb00b2b10 0x7f5fb00b4fd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:29.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.826+0000 7f5fcbe00700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fcc0779d0 0x7f5fcc1a0210 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:29.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.826+0000 7f5fcbe00700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fcc114250 0x7f5fcc1a4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:29.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.826+0000 7f5fcbe00700 1 --2- 172.21.15.27:0/4135461852 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fcc0782f0 0x7f5fcc1a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:29.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.826+0000 7f5fcbe00700 1 -- 172.21.15.27:0/4135461852 >> 172.21.15.27:0/4135461852 conn(0x7f5fcc1010e0 msgr2=0x7f5fcc1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:29.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.826+0000 7f5fcbe00700 1 -- 172.21.15.27:0/4135461852 shutdown_connections 2024-05-10T11:13:29.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:29.826+0000 7f5fcbe00700 1 -- 172.21.15.27:0/4135461852 wait complete. 2024-05-10T11:13:30.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:30 smithi189 ceph-mon[34842]: pgmap v11: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 511 B/s wr, 15 op/s 2024-05-10T11:13:30.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:30 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4135461852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:13:30.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:30 smithi027 ceph-mon[30054]: pgmap v11: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 511 B/s wr, 15 op/s 2024-05-10T11:13:30.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:30 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4135461852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:13:30.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:30 smithi076 ceph-mon[34152]: pgmap v11: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 511 B/s wr, 15 op/s 2024-05-10T11:13:30.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:30 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4135461852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:13:30.815 DEBUG:tasks.fwd_scrub:Yielding 2024-05-10T11:13:30.815 INFO:teuthology.run_tasks:Running task exec... 2024-05-10T11:13:30.825 INFO:teuthology.task.exec:Executing custom commands... 2024-05-10T11:13:30.825 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-05-10T11:13:30.825 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-05-10T11:13:30.825 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-10T11:13:30.826 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:13:30.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:13:31.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.024+0000 7f4650600640 1 -- 172.21.15.27:0/2091960989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464806b320 msgr2=0x7f464806b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.024+0000 7f4650600640 1 --2- 172.21.15.27:0/2091960989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464806b320 0x7f464806b7a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4634009960 tx=0x7f463402a460 comp rx=0 tx=0).stop 2024-05-10T11:13:31.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.024+0000 7f4650600640 1 -- 172.21.15.27:0/2091960989 shutdown_connections 2024-05-10T11:13:31.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.024+0000 7f4650600640 1 --2- 172.21.15.27:0/2091960989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f464806bce0 0x7f464810f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.024+0000 7f4650600640 1 --2- 172.21.15.27:0/2091960989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464806b320 0x7f464806b7a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.024+0000 7f4650600640 1 --2- 172.21.15.27:0/2091960989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4648108a60 0x7f4648108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.024+0000 7f4650600640 1 -- 172.21.15.27:0/2091960989 >> 172.21.15.27:0/2091960989 conn(0x7f4648077bf0 msgr2=0x7f4648078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:31.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.024+0000 7f4650600640 1 -- 172.21.15.27:0/2091960989 shutdown_connections 2024-05-10T11:13:31.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.024+0000 7f4650600640 1 -- 172.21.15.27:0/2091960989 wait complete. 2024-05-10T11:13:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.025+0000 7f4650600640 1 Processor -- start 2024-05-10T11:13:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.025+0000 7f4650600640 1 -- start start 2024-05-10T11:13:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.026+0000 7f4650600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f464806b320 0x7f464819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.026+0000 7f4650600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464806bce0 0x7f464819b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.026+0000 7f4650600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4648108a60 0x7f464819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.026+0000 7f4650600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4648112c30 con 0x7f4648108a60 2024-05-10T11:13:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.026+0000 7f4650600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4648112ab0 con 0x7f464806bce0 2024-05-10T11:13:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.026+0000 7f4650600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4648112db0 con 0x7f464806b320 2024-05-10T11:13:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.026+0000 7f464d200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464806bce0 0x7f464819b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.026+0000 7f464d200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464806bce0 0x7f464819b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50240/0 (socket says 172.21.15.27:50240) 2024-05-10T11:13:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.026+0000 7f464d200640 1 -- 172.21.15.27:0/1768556104 learned_addr learned my addr 172.21.15.27:0/1768556104 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.026+0000 7f464e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4648108a60 0x7f464819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.026+0000 7f464dc00640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f464806b320 0x7f464819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.027+0000 7f464d200640 1 -- 172.21.15.27:0/1768556104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f464806b320 msgr2=0x7f464819b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.027+0000 7f464d200640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f464806b320 0x7f464819b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.027+0000 7f464d200640 1 -- 172.21.15.27:0/1768556104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4648108a60 msgr2=0x7f464819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.027+0000 7f464d200640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4648108a60 0x7f464819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.027+0000 7f464d200640 1 -- 172.21.15.27:0/1768556104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46481a0380 con 0x7f464806bce0 2024-05-10T11:13:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.027+0000 7f464d200640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464806bce0 0x7f464819b8d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4634009930 tx=0x7f4634004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:31.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.027+0000 7f4646000640 1 -- 172.21.15.27:0/1768556104 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463403f030 con 0x7f464806bce0 2024-05-10T11:13:31.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.027+0000 7f4646000640 1 -- 172.21.15.27:0/1768556104 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4634004490 con 0x7f464806bce0 2024-05-10T11:13:31.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.027+0000 7f4650600640 1 -- 172.21.15.27:0/1768556104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46481a0610 con 0x7f464806bce0 2024-05-10T11:13:31.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.027+0000 7f4646000640 1 -- 172.21.15.27:0/1768556104 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634035070 con 0x7f464806bce0 2024-05-10T11:13:31.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.027+0000 7f4650600640 1 -- 172.21.15.27:0/1768556104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46481a7f00 con 0x7f464806bce0 2024-05-10T11:13:31.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.029+0000 7f4650600640 1 -- 172.21.15.27:0/1768556104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4648050740 con 0x7f464806bce0 2024-05-10T11:13:31.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.031+0000 7f4646000640 1 -- 172.21.15.27:0/1768556104 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f46340035f0 con 0x7f464806bce0 2024-05-10T11:13:31.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.033+0000 7f4646000640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f46300b2bb0 0x7f46300b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:31.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.033+0000 7f4646000640 1 -- 172.21.15.27:0/1768556104 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f46341031f0 con 0x7f464806bce0 2024-05-10T11:13:31.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.033+0000 7f464dc00640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f46300b2bb0 0x7f46300b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:31.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.034+0000 7f464dc00640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f46300b2bb0 0x7f46300b5070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f463c009bd0 tx=0x7f463c005f90 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:31.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.037+0000 7f4646000640 1 -- 172.21.15.27:0/1768556104 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f46340ca620 con 0x7f464806bce0 2024-05-10T11:13:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.276+0000 7f4650600640 1 -- 172.21.15.27:0/1768556104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f46481a8160 con 0x7f464806bce0 2024-05-10T11:13:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.282+0000 7f4646000640 1 -- 172.21.15.27:0/1768556104 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v29)=0 v29) ==== 129+0+0 (secure 0 0 0) 0x7f46340c9f20 con 0x7f464806bce0 2024-05-10T11:13:31.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.285+0000 7f4650600640 1 -- 172.21.15.27:0/1768556104 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f46300b2bb0 msgr2=0x7f46300b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.285+0000 7f4650600640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f46300b2bb0 0x7f46300b5070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f463c009bd0 tx=0x7f463c005f90 comp rx=0 tx=0).stop 2024-05-10T11:13:31.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.285+0000 7f4650600640 1 -- 172.21.15.27:0/1768556104 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464806bce0 msgr2=0x7f464819b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.285+0000 7f4650600640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464806bce0 0x7f464819b8d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4634009930 tx=0x7f4634004060 comp rx=0 tx=0).stop 2024-05-10T11:13:31.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.286+0000 7f4650600640 1 -- 172.21.15.27:0/1768556104 shutdown_connections 2024-05-10T11:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.286+0000 7f4650600640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f46300b2bb0 0x7f46300b5070 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.286+0000 7f4650600640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4648108a60 0x7f464819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.286+0000 7f4650600640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464806bce0 0x7f464819b8d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.286+0000 7f4650600640 1 --2- 172.21.15.27:0/1768556104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f464806b320 0x7f464819b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.286+0000 7f4650600640 1 -- 172.21.15.27:0/1768556104 >> 172.21.15.27:0/1768556104 conn(0x7f4648077bf0 msgr2=0x7f464810e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.286+0000 7f4650600640 1 -- 172.21.15.27:0/1768556104 shutdown_connections 2024-05-10T11:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.286+0000 7f4650600640 1 -- 172.21.15.27:0/1768556104 wait complete. 2024-05-10T11:13:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:13:31.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-05-10T11:13:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.490+0000 7fad75e00640 1 -- 172.21.15.27:0/111676308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad70109bc0 msgr2=0x7fad7010dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.490+0000 7fad75e00640 1 --2- 172.21.15.27:0/111676308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad70109bc0 0x7fad7010dc00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fad60009ca0 tx=0x7fad6002a8c0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.490+0000 7fad75e00640 1 -- 172.21.15.27:0/111676308 shutdown_connections 2024-05-10T11:13:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.490+0000 7fad75e00640 1 --2- 172.21.15.27:0/111676308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad7010e2e0 0x7fad70115eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.490+0000 7fad75e00640 1 --2- 172.21.15.27:0/111676308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad70109bc0 0x7fad7010dc00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.490+0000 7fad75e00640 1 --2- 172.21.15.27:0/111676308 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad70109210 0x7fad701095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.490+0000 7fad75e00640 1 -- 172.21.15.27:0/111676308 >> 172.21.15.27:0/111676308 conn(0x7fad70076f40 msgr2=0x7fad70079360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.490+0000 7fad75e00640 1 -- 172.21.15.27:0/111676308 shutdown_connections 2024-05-10T11:13:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.490+0000 7fad75e00640 1 -- 172.21.15.27:0/111676308 wait complete. 2024-05-10T11:13:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.491+0000 7fad75e00640 1 Processor -- start 2024-05-10T11:13:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.491+0000 7fad75e00640 1 -- start start 2024-05-10T11:13:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.492+0000 7fad75e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad70109210 0x7fad7019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.492+0000 7fad75e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad7010e2e0 0x7fad7019fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.492+0000 7fad75e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad701a3ff0 0x7fad701a44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.492+0000 7fad75e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fad70119250 con 0x7fad701a3ff0 2024-05-10T11:13:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.492+0000 7fad75e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fad701190d0 con 0x7fad70109210 2024-05-10T11:13:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.492+0000 7fad75e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fad701193d0 con 0x7fad7010e2e0 2024-05-10T11:13:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.492+0000 7fad6ea00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad70109210 0x7fad7019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.492+0000 7fad6ea00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad70109210 0x7fad7019f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50260/0 (socket says 172.21.15.27:50260) 2024-05-10T11:13:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.492+0000 7fad6ea00640 1 -- 172.21.15.27:0/1857621843 learned_addr learned my addr 172.21.15.27:0/1857621843 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.492+0000 7fad6f400640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad701a3ff0 0x7fad701a44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.492+0000 7fad6e000640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad7010e2e0 0x7fad7019fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.493+0000 7fad6ea00640 1 -- 172.21.15.27:0/1857621843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad7010e2e0 msgr2=0x7fad7019fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.493+0000 7fad6ea00640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad7010e2e0 0x7fad7019fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.493+0000 7fad6ea00640 1 -- 172.21.15.27:0/1857621843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad701a3ff0 msgr2=0x7fad701a44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.493+0000 7fad6ea00640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad701a3ff0 0x7fad701a44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.493+0000 7fad6ea00640 1 -- 172.21.15.27:0/1857621843 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad701a4ae0 con 0x7fad70109210 2024-05-10T11:13:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.493+0000 7fad6e000640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad7010e2e0 0x7fad7019fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:13:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.493+0000 7fad6ea00640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad70109210 0x7fad7019f730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fad5800e9f0 tx=0x7fad5800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.493+0000 7fad74a00640 1 -- 172.21.15.27:0/1857621843 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5800cd80 con 0x7fad70109210 2024-05-10T11:13:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.493+0000 7fad74a00640 1 -- 172.21.15.27:0/1857621843 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad580108e0 con 0x7fad70109210 2024-05-10T11:13:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.493+0000 7fad75e00640 1 -- 172.21.15.27:0/1857621843 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad701a4dd0 con 0x7fad70109210 2024-05-10T11:13:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.493+0000 7fad74a00640 1 -- 172.21.15.27:0/1857621843 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58018980 con 0x7fad70109210 2024-05-10T11:13:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.494+0000 7fad75e00640 1 -- 172.21.15.27:0/1857621843 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad701ac5c0 con 0x7fad70109210 2024-05-10T11:13:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.495+0000 7fad75e00640 1 -- 172.21.15.27:0/1857621843 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad70050740 con 0x7fad70109210 2024-05-10T11:13:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.497+0000 7fad74a00640 1 -- 172.21.15.27:0/1857621843 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7fad58010430 con 0x7fad70109210 2024-05-10T11:13:31.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.500+0000 7fad74a00640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fad540b2b30 0x7fad540b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:31.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.500+0000 7fad74a00640 1 -- 172.21.15.27:0/1857621843 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fad580e3540 con 0x7fad70109210 2024-05-10T11:13:31.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.500+0000 7fad6e000640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fad540b2b30 0x7fad540b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.501+0000 7fad74a00640 1 -- 172.21.15.27:0/1857621843 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fad580aa900 con 0x7fad70109210 2024-05-10T11:13:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.501+0000 7fad6e000640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fad540b2b30 0x7fad540b4ff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fad701082a0 tx=0x7fad60009880 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.731+0000 7fad75e00640 1 -- 172.21.15.27:0/1857621843 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fad701a0a80 con 0x7fad70109210 2024-05-10T11:13:31.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.742+0000 7fad74a00640 1 -- 172.21.15.27:0/1857621843 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7fad701a0a80 con 0x7fad70109210 2024-05-10T11:13:31.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.745+0000 7fad75e00640 1 -- 172.21.15.27:0/1857621843 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fad540b2b30 msgr2=0x7fad540b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.745+0000 7fad75e00640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fad540b2b30 0x7fad540b4ff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fad701082a0 tx=0x7fad60009880 comp rx=0 tx=0).stop 2024-05-10T11:13:31.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.745+0000 7fad75e00640 1 -- 172.21.15.27:0/1857621843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad70109210 msgr2=0x7fad7019f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.745+0000 7fad75e00640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad70109210 0x7fad7019f730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fad5800e9f0 tx=0x7fad5800eeb0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.745+0000 7fad75e00640 1 -- 172.21.15.27:0/1857621843 shutdown_connections 2024-05-10T11:13:31.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.745+0000 7fad75e00640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fad540b2b30 0x7fad540b4ff0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.745+0000 7fad75e00640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad701a3ff0 0x7fad701a44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.745+0000 7fad75e00640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad7010e2e0 0x7fad7019fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.745+0000 7fad75e00640 1 --2- 172.21.15.27:0/1857621843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fad70109210 0x7fad7019f730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.745+0000 7fad75e00640 1 -- 172.21.15.27:0/1857621843 >> 172.21.15.27:0/1857621843 conn(0x7fad70076f40 msgr2=0x7fad70079260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:31.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.745+0000 7fad75e00640 1 -- 172.21.15.27:0/1857621843 shutdown_connections 2024-05-10T11:13:31.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.746+0000 7fad75e00640 1 -- 172.21.15.27:0/1857621843 wait complete. 2024-05-10T11:13:31.765 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-05-10T11:13:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.968+0000 7f1b9a600640 1 -- 172.21.15.27:0/1086418478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b941089a0 msgr2=0x7f1b94108d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.968+0000 7f1b9a600640 1 --2- 172.21.15.27:0/1086418478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b941089a0 0x7f1b94108d80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009930 tx=0x7f1b8402a420 comp rx=0 tx=0).stop 2024-05-10T11:13:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.968+0000 7f1b9a600640 1 -- 172.21.15.27:0/1086418478 shutdown_connections 2024-05-10T11:13:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.968+0000 7f1b9a600640 1 --2- 172.21.15.27:0/1086418478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b94103350 0x7f1b9410f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.968+0000 7f1b9a600640 1 --2- 172.21.15.27:0/1086418478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b94102990 0x7f1b94102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.968+0000 7f1b9a600640 1 --2- 172.21.15.27:0/1086418478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b941089a0 0x7f1b94108d80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.968+0000 7f1b9a600640 1 -- 172.21.15.27:0/1086418478 >> 172.21.15.27:0/1086418478 conn(0x7f1b94079e40 msgr2=0x7f1b9407a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.969+0000 7f1b9a600640 1 -- 172.21.15.27:0/1086418478 shutdown_connections 2024-05-10T11:13:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.969+0000 7f1b9a600640 1 -- 172.21.15.27:0/1086418478 wait complete. 2024-05-10T11:13:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.970+0000 7f1b9a600640 1 Processor -- start 2024-05-10T11:13:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.970+0000 7f1b9a600640 1 -- start start 2024-05-10T11:13:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.970+0000 7f1b9a600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b94102990 0x7f1b9419b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.970+0000 7f1b9a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b94103350 0x7f1b9419b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.970+0000 7f1b9a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b941089a0 0x7f1b9419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.970+0000 7f1b9a600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94112cb0 con 0x7f1b94103350 2024-05-10T11:13:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.970+0000 7f1b9a600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94112b30 con 0x7f1b94102990 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.970+0000 7f1b9a600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94112e30 con 0x7f1b941089a0 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.970+0000 7f1b93e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b941089a0 0x7f1b9419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.970+0000 7f1b92a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b94103350 0x7f1b9419b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.970+0000 7f1b93400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b94102990 0x7f1b9419b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.971+0000 7f1b93e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b941089a0 0x7f1b9419fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:59282/0 (socket says 172.21.15.27:59282) 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.971+0000 7f1b93e00640 1 -- 172.21.15.27:0/1281225204 learned_addr learned my addr 172.21.15.27:0/1281225204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.971+0000 7f1b93e00640 1 -- 172.21.15.27:0/1281225204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b94102990 msgr2=0x7f1b9419b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.971+0000 7f1b93e00640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b94102990 0x7f1b9419b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.971+0000 7f1b93e00640 1 -- 172.21.15.27:0/1281225204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b94103350 msgr2=0x7f1b9419b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.971+0000 7f1b93e00640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b94103350 0x7f1b9419b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.971+0000 7f1b93e00640 1 -- 172.21.15.27:0/1281225204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b941a02d0 con 0x7f1b941089a0 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.971+0000 7f1b93400640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b94102990 0x7f1b9419b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.971+0000 7f1b93e00640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b941089a0 0x7f1b9419fba0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00ea60 tx=0x7f1b8c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.972+0000 7f1b8be00640 1 -- 172.21.15.27:0/1281225204 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c00ce40 con 0x7f1b941089a0 2024-05-10T11:13:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.972+0000 7f1b8be00640 1 -- 172.21.15.27:0/1281225204 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b8c005020 con 0x7f1b941089a0 2024-05-10T11:13:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.972+0000 7f1b9a600640 1 -- 172.21.15.27:0/1281225204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b941a05c0 con 0x7f1b941089a0 2024-05-10T11:13:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.972+0000 7f1b8be00640 1 -- 172.21.15.27:0/1281225204 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c010670 con 0x7f1b941089a0 2024-05-10T11:13:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.972+0000 7f1b9a600640 1 -- 172.21.15.27:0/1281225204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b941a7e50 con 0x7f1b941089a0 2024-05-10T11:13:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.973+0000 7f1b9a600640 1 -- 172.21.15.27:0/1281225204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b94050740 con 0x7f1b941089a0 2024-05-10T11:13:31.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.975+0000 7f1b8be00640 1 -- 172.21.15.27:0/1281225204 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f1b8c0054b0 con 0x7f1b941089a0 2024-05-10T11:13:31.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.977+0000 7f1b8be00640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f1b780b2bb0 0x7f1b780b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:31.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.977+0000 7f1b8be00640 1 -- 172.21.15.27:0/1281225204 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f1b8c0e3280 con 0x7f1b941089a0 2024-05-10T11:13:31.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.977+0000 7f1b93400640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f1b780b2bb0 0x7f1b780b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:31.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.978+0000 7f1b93400640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f1b780b2bb0 0x7f1b780b5070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1b84031440 tx=0x7f1b84005a20 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:31.980+0000 7f1b8be00640 1 -- 172.21.15.27:0/1281225204 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f1b8c054d70 con 0x7f1b941089a0 2024-05-10T11:13:32.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.194+0000 7f1b9a600640 1 -- 172.21.15.27:0/1281225204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f1b941a8220 con 0x7f1b941089a0 2024-05-10T11:13:32.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.202+0000 7f1b8be00640 1 -- 172.21.15.27:0/1281225204 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7f1b8c0aa5e0 con 0x7f1b941089a0 2024-05-10T11:13:32.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.205+0000 7f1b9a600640 1 -- 172.21.15.27:0/1281225204 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f1b780b2bb0 msgr2=0x7f1b780b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:32.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.205+0000 7f1b9a600640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f1b780b2bb0 0x7f1b780b5070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1b84031440 tx=0x7f1b84005a20 comp rx=0 tx=0).stop 2024-05-10T11:13:32.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.205+0000 7f1b9a600640 1 -- 172.21.15.27:0/1281225204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b941089a0 msgr2=0x7f1b9419fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:32.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.205+0000 7f1b9a600640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b941089a0 0x7f1b9419fba0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00ea60 tx=0x7f1b8c00ef20 comp rx=0 tx=0).stop 2024-05-10T11:13:32.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.206+0000 7f1b9a600640 1 -- 172.21.15.27:0/1281225204 shutdown_connections 2024-05-10T11:13:32.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.206+0000 7f1b9a600640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f1b780b2bb0 0x7f1b780b5070 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.206+0000 7f1b9a600640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b941089a0 0x7f1b9419fba0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.206+0000 7f1b9a600640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b94103350 0x7f1b9419b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.206+0000 7f1b9a600640 1 --2- 172.21.15.27:0/1281225204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b94102990 0x7f1b9419b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.206+0000 7f1b9a600640 1 -- 172.21.15.27:0/1281225204 >> 172.21.15.27:0/1281225204 conn(0x7f1b94079e40 msgr2=0x7f1b9410e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:32.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.206+0000 7f1b9a600640 1 -- 172.21.15.27:0/1281225204 shutdown_connections 2024-05-10T11:13:32.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.206+0000 7f1b9a600640 1 -- 172.21.15.27:0/1281225204 wait complete. 2024-05-10T11:13:32.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-05-10T11:13:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.430+0000 7f14e3e00640 1 -- 172.21.15.27:0/3528499593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e4108ef0 msgr2=0x7f14e4109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.430+0000 7f14e3e00640 1 --2- 172.21.15.27:0/3528499593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e4108ef0 0x7f14e4109330 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009900 tx=0x7f14cc02a410 comp rx=0 tx=0).stop 2024-05-10T11:13:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.430+0000 7f14e3e00640 1 -- 172.21.15.27:0/3528499593 shutdown_connections 2024-05-10T11:13:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.430+0000 7f14e3e00640 1 --2- 172.21.15.27:0/3528499593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14e4109870 0x7f14e4115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.430+0000 7f14e3e00640 1 --2- 172.21.15.27:0/3528499593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e4108ef0 0x7f14e4109330 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.430+0000 7f14e3e00640 1 --2- 172.21.15.27:0/3528499593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e410ee80 0x7f14e410f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.430+0000 7f14e3e00640 1 -- 172.21.15.27:0/3528499593 >> 172.21.15.27:0/3528499593 conn(0x7f14e4078790 msgr2=0x7f14e41077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.430+0000 7f14e3e00640 1 -- 172.21.15.27:0/3528499593 shutdown_connections 2024-05-10T11:13:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.430+0000 7f14e3e00640 1 -- 172.21.15.27:0/3528499593 wait complete. 2024-05-10T11:13:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.431+0000 7f14e3e00640 1 Processor -- start 2024-05-10T11:13:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.432+0000 7f14e3e00640 1 -- start start 2024-05-10T11:13:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.432+0000 7f14e3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e4108ef0 0x7f14e419f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.432+0000 7f14e3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14e4109870 0x7f14e419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.432+0000 7f14e3e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e410ee80 0x7f14e41a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.432+0000 7f14e3e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f14e4119080 con 0x7f14e4108ef0 2024-05-10T11:13:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.432+0000 7f14e3e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f14e4118f00 con 0x7f14e410ee80 2024-05-10T11:13:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.432+0000 7f14e3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f14e4119200 con 0x7f14e4109870 2024-05-10T11:13:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.432+0000 7f14e2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e4108ef0 0x7f14e419f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.432+0000 7f14dbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14e4109870 0x7f14e419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.432+0000 7f14dbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14e4109870 0x7f14e419fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:59294/0 (socket says 172.21.15.27:59294) 2024-05-10T11:13:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.432+0000 7f14dbe00640 1 -- 172.21.15.27:0/644624504 learned_addr learned my addr 172.21.15.27:0/644624504 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.433+0000 7f14e3400640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e410ee80 0x7f14e41a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.433+0000 7f14dbe00640 1 -- 172.21.15.27:0/644624504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e410ee80 msgr2=0x7f14e41a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.433+0000 7f14dbe00640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e410ee80 0x7f14e41a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.433+0000 7f14dbe00640 1 -- 172.21.15.27:0/644624504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e4108ef0 msgr2=0x7f14e419f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.433+0000 7f14dbe00640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e4108ef0 0x7f14e419f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.433+0000 7f14dbe00640 1 -- 172.21.15.27:0/644624504 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e41a45b0 con 0x7f14e4109870 2024-05-10T11:13:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.433+0000 7f14dbe00640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14e4109870 0x7f14e419fb00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f14cc02aa00 tx=0x7f14cc003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.434+0000 7f14db400640 1 -- 172.21.15.27:0/644624504 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc03f070 con 0x7f14e4109870 2024-05-10T11:13:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.434+0000 7f14db400640 1 -- 172.21.15.27:0/644624504 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14cc03bb20 con 0x7f14e4109870 2024-05-10T11:13:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.434+0000 7f14db400640 1 -- 172.21.15.27:0/644624504 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc035030 con 0x7f14e4109870 2024-05-10T11:13:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.434+0000 7f14e3e00640 1 -- 172.21.15.27:0/644624504 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14e41a4840 con 0x7f14e4109870 2024-05-10T11:13:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.434+0000 7f14e3e00640 1 -- 172.21.15.27:0/644624504 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14e41ac130 con 0x7f14e4109870 2024-05-10T11:13:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.435+0000 7f14e3e00640 1 -- 172.21.15.27:0/644624504 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14e4050f90 con 0x7f14e4109870 2024-05-10T11:13:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.438+0000 7f14db400640 1 -- 172.21.15.27:0/644624504 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f14cc042070 con 0x7f14e4109870 2024-05-10T11:13:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.439+0000 7f14db400640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f14c80b2a70 0x7f14c80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.440+0000 7f14db400640 1 -- 172.21.15.27:0/644624504 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f14cc103280 con 0x7f14e4109870 2024-05-10T11:13:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.440+0000 7f14e2a00640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f14c80b2a70 0x7f14c80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:32.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.441+0000 7f14e2a00640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f14c80b2a70 0x7f14c80b4f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f14d40095f0 tx=0x7f14d4005f90 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.444+0000 7f14db400640 1 -- 172.21.15.27:0/644624504 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f14cc0ca060 con 0x7f14e4109870 2024-05-10T11:13:32.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:32 smithi027 ceph-mon[30054]: pgmap v12: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 511 B/s wr, 19 op/s 2024-05-10T11:13:32.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:32 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' 2024-05-10T11:13:32.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:32 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:32.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:32 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:32.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:32 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:32.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:32 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:32.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:32 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' 2024-05-10T11:13:32.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:32 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:32.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:32 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:32.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:32 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:32.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:32 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:32.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:32 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' 2024-05-10T11:13:32.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:32 smithi189 ceph-mon[34842]: pgmap v12: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 511 B/s wr, 19 op/s 2024-05-10T11:13:32.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:32 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' 2024-05-10T11:13:32.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:32 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:32.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:32 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:32.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:32 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:32.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:32 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:32.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:32 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' 2024-05-10T11:13:32.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:32 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:32.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:32 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:32.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:32 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:32.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:32 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:32.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:32 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' 2024-05-10T11:13:32.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:32 smithi076 ceph-mon[34152]: pgmap v12: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 511 B/s wr, 19 op/s 2024-05-10T11:13:32.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:32 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' 2024-05-10T11:13:32.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:32 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:32.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:32 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:32.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:32 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:32.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:32 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:32.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:32 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' 2024-05-10T11:13:32.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:32 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:32.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:32 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:32.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:32 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:32.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:32 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:32.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:32 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' 2024-05-10T11:13:32.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.691+0000 7f14e3e00640 1 -- 172.21.15.27:0/644624504 --> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f14e4111330 con 0x7f14c80b2a70 2024-05-10T11:13:32.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.830+0000 7f14db400640 1 -- 172.21.15.27:0/644624504 <== mgr.25220 v2:172.21.15.27:6800/1167199542 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f14e4111330 con 0x7f14c80b2a70 2024-05-10T11:13:32.831 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path / 2024-05-10T11:13:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.834+0000 7f14e3e00640 1 -- 172.21.15.27:0/644624504 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f14c80b2a70 msgr2=0x7f14c80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.834+0000 7f14e3e00640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f14c80b2a70 0x7f14c80b4f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f14d40095f0 tx=0x7f14d4005f90 comp rx=0 tx=0).stop 2024-05-10T11:13:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.834+0000 7f14e3e00640 1 -- 172.21.15.27:0/644624504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14e4109870 msgr2=0x7f14e419fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.834+0000 7f14e3e00640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14e4109870 0x7f14e419fb00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f14cc02aa00 tx=0x7f14cc003c10 comp rx=0 tx=0).stop 2024-05-10T11:13:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.834+0000 7f14e3e00640 1 -- 172.21.15.27:0/644624504 shutdown_connections 2024-05-10T11:13:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.834+0000 7f14e3e00640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f14c80b2a70 0x7f14c80b4f30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.834+0000 7f14e3e00640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e410ee80 0x7f14e41a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.834+0000 7f14e3e00640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14e4109870 0x7f14e419fb00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.834+0000 7f14e3e00640 1 --2- 172.21.15.27:0/644624504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e4108ef0 0x7f14e419f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.834+0000 7f14e3e00640 1 -- 172.21.15.27:0/644624504 >> 172.21.15.27:0/644624504 conn(0x7f14e4078790 msgr2=0x7f14e40798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.834+0000 7f14e3e00640 1 -- 172.21.15.27:0/644624504 shutdown_connections 2024-05-10T11:13:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:32.834+0000 7f14e3e00640 1 -- 172.21.15.27:0/644624504 wait complete. 2024-05-10T11:13:32.857 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-05-10T11:13:33.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.051+0000 7f24a7800640 1 -- 172.21.15.27:0/2667756534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a01029c0 msgr2=0x7f24a0102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.051+0000 7f24a7800640 1 --2- 172.21.15.27:0/2667756534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a01029c0 0x7f24a0102e40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f248c009930 tx=0x7f248c02a420 comp rx=0 tx=0).stop 2024-05-10T11:13:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.051+0000 7f24a7800640 1 -- 172.21.15.27:0/2667756534 shutdown_connections 2024-05-10T11:13:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.051+0000 7f24a7800640 1 --2- 172.21.15.27:0/2667756534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a0103380 0x7f24a010f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.051+0000 7f24a7800640 1 --2- 172.21.15.27:0/2667756534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a01029c0 0x7f24a0102e40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.051+0000 7f24a7800640 1 --2- 172.21.15.27:0/2667756534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a01089d0 0x7f24a0108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.051+0000 7f24a7800640 1 -- 172.21.15.27:0/2667756534 >> 172.21.15.27:0/2667756534 conn(0x7f24a0079e40 msgr2=0x7f24a007a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.051+0000 7f24a7800640 1 -- 172.21.15.27:0/2667756534 shutdown_connections 2024-05-10T11:13:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.051+0000 7f24a7800640 1 -- 172.21.15.27:0/2667756534 wait complete. 2024-05-10T11:13:33.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.052+0000 7f24a7800640 1 Processor -- start 2024-05-10T11:13:33.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.052+0000 7f24a7800640 1 -- start start 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.052+0000 7f24a7800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a01029c0 0x7f24a019b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.052+0000 7f24a7800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a0103380 0x7f24a019b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.052+0000 7f24a7800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a01089d0 0x7f24a019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.052+0000 7f24a7800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f24a0112c90 con 0x7f24a01029c0 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.052+0000 7f24a7800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f24a0112b10 con 0x7f24a01089d0 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.052+0000 7f24a7800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f24a0112e10 con 0x7f24a0103380 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.053+0000 7f24a4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a01029c0 0x7f24a019b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.053+0000 7f24a4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a01029c0 0x7f24a019b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45100/0 (socket says 172.21.15.27:45100) 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.053+0000 7f24a4e00640 1 -- 172.21.15.27:0/834412987 learned_addr learned my addr 172.21.15.27:0/834412987 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.053+0000 7f24a5800640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a01089d0 0x7f24a019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.053+0000 7f249fe00640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a0103380 0x7f24a019b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.053+0000 7f24a4e00640 1 -- 172.21.15.27:0/834412987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a0103380 msgr2=0x7f24a019b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.053+0000 7f24a4e00640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a0103380 0x7f24a019b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.053+0000 7f24a4e00640 1 -- 172.21.15.27:0/834412987 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a01089d0 msgr2=0x7f24a019fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.053+0000 7f24a4e00640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a01089d0 0x7f24a019fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.053+0000 7f24a4e00640 1 -- 172.21.15.27:0/834412987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a01a02e0 con 0x7f24a01029c0 2024-05-10T11:13:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.053+0000 7f249fe00640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a0103380 0x7f24a019b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:13:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.053+0000 7f24a4e00640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a01029c0 0x7f24a019b2f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f249400e9f0 tx=0x7f249400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.054+0000 7f249d600640 1 -- 172.21.15.27:0/834412987 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249400cd80 con 0x7f24a01029c0 2024-05-10T11:13:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.054+0000 7f249d600640 1 -- 172.21.15.27:0/834412987 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24940108e0 con 0x7f24a01029c0 2024-05-10T11:13:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.054+0000 7f249d600640 1 -- 172.21.15.27:0/834412987 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494018980 con 0x7f24a01029c0 2024-05-10T11:13:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.054+0000 7f24a7800640 1 -- 172.21.15.27:0/834412987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a01a05d0 con 0x7f24a01029c0 2024-05-10T11:13:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.054+0000 7f24a7800640 1 -- 172.21.15.27:0/834412987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a01a7e60 con 0x7f24a01029c0 2024-05-10T11:13:33.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.055+0000 7f24a7800640 1 -- 172.21.15.27:0/834412987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a0104100 con 0x7f24a01029c0 2024-05-10T11:13:33.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.057+0000 7f249d600640 1 -- 172.21.15.27:0/834412987 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f2494010430 con 0x7f24a01029c0 2024-05-10T11:13:33.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.059+0000 7f249d600640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f24880b2bb0 0x7f24880b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.059+0000 7f249d600640 1 -- 172.21.15.27:0/834412987 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f24940e2fe0 con 0x7f24a01029c0 2024-05-10T11:13:33.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.059+0000 7f249fe00640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f24880b2bb0 0x7f24880b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.060+0000 7f249fe00640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f24880b2bb0 0x7f24880b5070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f248c000c00 tx=0x7f248c009610 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:33.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.064+0000 7f249d600640 1 -- 172.21.15.27:0/834412987 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f24940aa340 con 0x7f24a01029c0 2024-05-10T11:13:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.315+0000 7f24a7800640 1 -- 172.21.15.27:0/834412987 --> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f24a0101eb0 con 0x7f24880b2bb0 2024-05-10T11:13:33.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:33 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:33.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:33 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:33.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:33 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:33.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:33 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:33.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.427+0000 7f249d600640 1 -- 172.21.15.27:0/834412987 <== mgr.25220 v2:172.21.15.27:6800/1167199542 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f24a0101eb0 con 0x7f24880b2bb0 2024-05-10T11:13:33.428 INFO:teuthology.orchestra.run.smithi027.stdout:Retention added to path / 2024-05-10T11:13:33.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.431+0000 7f24a7800640 1 -- 172.21.15.27:0/834412987 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f24880b2bb0 msgr2=0x7f24880b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.431+0000 7f24a7800640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f24880b2bb0 0x7f24880b5070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f248c000c00 tx=0x7f248c009610 comp rx=0 tx=0).stop 2024-05-10T11:13:33.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.431+0000 7f24a7800640 1 -- 172.21.15.27:0/834412987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a01029c0 msgr2=0x7f24a019b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.431+0000 7f24a7800640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a01029c0 0x7f24a019b2f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f249400e9f0 tx=0x7f249400eeb0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.431+0000 7f24a7800640 1 -- 172.21.15.27:0/834412987 shutdown_connections 2024-05-10T11:13:33.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.431+0000 7f24a7800640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f24880b2bb0 0x7f24880b5070 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.431+0000 7f24a7800640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a01089d0 0x7f24a019fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.431+0000 7f24a7800640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a0103380 0x7f24a019b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.431+0000 7f24a7800640 1 --2- 172.21.15.27:0/834412987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a01029c0 0x7f24a019b2f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.431+0000 7f24a7800640 1 -- 172.21.15.27:0/834412987 >> 172.21.15.27:0/834412987 conn(0x7f24a0079e40 msgr2=0x7f24a0101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:33.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.431+0000 7f24a7800640 1 -- 172.21.15.27:0/834412987 shutdown_connections 2024-05-10T11:13:33.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.431+0000 7f24a7800640 1 -- 172.21.15.27:0/834412987 wait complete. 2024-05-10T11:13:33.454 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-10T11:13:33.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:33 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:33.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:33 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:33.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:33 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:33.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:33 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:33.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.663+0000 7f360ac00640 1 -- 172.21.15.27:0/3900978383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3604103a00 msgr2=0x7f360406b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.663+0000 7f360ac00640 1 --2- 172.21.15.27:0/3900978383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3604103a00 0x7f360406b1e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009ca0 tx=0x7f35f402a8c0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.663+0000 7f360ac00640 1 -- 172.21.15.27:0/3900978383 shutdown_connections 2024-05-10T11:13:33.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.663+0000 7f360ac00640 1 --2- 172.21.15.27:0/3900978383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f360410f3a0 0x7f3604111790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.663+0000 7f360ac00640 1 --2- 172.21.15.27:0/3900978383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f360406b7b0 0x7f360406bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.663+0000 7f360ac00640 1 --2- 172.21.15.27:0/3900978383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3604103a00 0x7f360406b1e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.663+0000 7f360ac00640 1 -- 172.21.15.27:0/3900978383 >> 172.21.15.27:0/3900978383 conn(0x7f36040776b0 msgr2=0x7f3604077ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:33.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.663+0000 7f360ac00640 1 -- 172.21.15.27:0/3900978383 shutdown_connections 2024-05-10T11:13:33.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.663+0000 7f360ac00640 1 -- 172.21.15.27:0/3900978383 wait complete. 2024-05-10T11:13:33.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.664+0000 7f360ac00640 1 Processor -- start 2024-05-10T11:13:33.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.665+0000 7f360ac00640 1 -- start start 2024-05-10T11:13:33.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.665+0000 7f360ac00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360406b7b0 0x7f360419b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.665+0000 7f360ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604103a00 0x7f360419b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.665+0000 7f360ac00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f360410f3a0 0x7f360419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.665+0000 7f360ac00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3604114ea0 con 0x7f360406b7b0 2024-05-10T11:13:33.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.665+0000 7f360ac00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3604114d20 con 0x7f360410f3a0 2024-05-10T11:13:33.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.665+0000 7f360ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3604115020 con 0x7f3604103a00 2024-05-10T11:13:33.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.665+0000 7f3603e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360406b7b0 0x7f360419b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.665+0000 7f3603e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360406b7b0 0x7f360419b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45128/0 (socket says 172.21.15.27:45128) 2024-05-10T11:13:33.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.665+0000 7f3603e00640 1 -- 172.21.15.27:0/4166447790 learned_addr learned my addr 172.21.15.27:0/4166447790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:33.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.665+0000 7f35fb400640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604103a00 0x7f360419b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.665+0000 7f3608c00640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f360410f3a0 0x7f360419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.666+0000 7f3603e00640 1 -- 172.21.15.27:0/4166447790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604103a00 msgr2=0x7f360419b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.666+0000 7f3603e00640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604103a00 0x7f360419b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.666+0000 7f3603e00640 1 -- 172.21.15.27:0/4166447790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f360410f3a0 msgr2=0x7f360419fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.666+0000 7f3603e00640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f360410f3a0 0x7f360419fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.666+0000 7f3603e00640 1 -- 172.21.15.27:0/4166447790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36041a02b0 con 0x7f360406b7b0 2024-05-10T11:13:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.666+0000 7f3608c00640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f360410f3a0 0x7f360419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:13:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.666+0000 7f3603e00640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360406b7b0 0x7f360419b2c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f35f4005e30 tx=0x7f35f4031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.666+0000 7f3601600640 1 -- 172.21.15.27:0/4166447790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4003b70 con 0x7f360406b7b0 2024-05-10T11:13:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.667+0000 7f3601600640 1 -- 172.21.15.27:0/4166447790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f4003d10 con 0x7f360406b7b0 2024-05-10T11:13:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.667+0000 7f3601600640 1 -- 172.21.15.27:0/4166447790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f402fa30 con 0x7f360406b7b0 2024-05-10T11:13:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.667+0000 7f360ac00640 1 -- 172.21.15.27:0/4166447790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36041a0540 con 0x7f360406b7b0 2024-05-10T11:13:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.667+0000 7f360ac00640 1 -- 172.21.15.27:0/4166447790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36041a7e30 con 0x7f360406b7b0 2024-05-10T11:13:33.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.668+0000 7f360ac00640 1 -- 172.21.15.27:0/4166447790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f360406aa30 con 0x7f360406b7b0 2024-05-10T11:13:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.670+0000 7f3601600640 1 -- 172.21.15.27:0/4166447790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f35f402fbd0 con 0x7f360406b7b0 2024-05-10T11:13:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.672+0000 7f3601600640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f35e80b2bb0 0x7f35e80b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.672+0000 7f3601600640 1 -- 172.21.15.27:0/4166447790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f35f4103f00 con 0x7f360406b7b0 2024-05-10T11:13:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.672+0000 7f35fb400640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f35e80b2bb0 0x7f35e80b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.673+0000 7f35fb400640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f35e80b2bb0 0x7f35e80b5070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3604110e50 tx=0x7f35ec005f90 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:33 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:13:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:33 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:13:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:33 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:13:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:33 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:13:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.680+0000 7f3601600640 1 -- 172.21.15.27:0/4166447790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f35f40cb370 con 0x7f360406b7b0 2024-05-10T11:13:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.689+0000 7fcffe600700 1 -- 172.21.15.27:0/2610024000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff810fc60 msgr2=0x7fcff8112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.689+0000 7fcffe600700 1 --2- 172.21.15.27:0/2610024000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff810fc60 0x7fcff8112090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcff0009f90 tx=0x7fcff000b330 comp rx=0 tx=0).stop 2024-05-10T11:13:33.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.689+0000 7fcffe600700 1 -- 172.21.15.27:0/2610024000 shutdown_connections 2024-05-10T11:13:33.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.689+0000 7fcffe600700 1 --2- 172.21.15.27:0/2610024000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff810fc60 0x7fcff8112090 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.689+0000 7fcffe600700 1 --2- 172.21.15.27:0/2610024000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcff8104360 0x7fcff806ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.689+0000 7fcffe600700 1 --2- 172.21.15.27:0/2610024000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff806bf50 0x7fcff806c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.689+0000 7fcffe600700 1 -- 172.21.15.27:0/2610024000 >> 172.21.15.27:0/2610024000 conn(0x7fcff80791e0 msgr2=0x7fcff80795f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:33.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.689+0000 7fcffe600700 1 -- 172.21.15.27:0/2610024000 shutdown_connections 2024-05-10T11:13:33.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.689+0000 7fcffe600700 1 -- 172.21.15.27:0/2610024000 wait complete. 2024-05-10T11:13:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.691+0000 7fcffe600700 1 Processor -- start 2024-05-10T11:13:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.691+0000 7fcffe600700 1 -- start start 2024-05-10T11:13:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.691+0000 7fcffe600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff806bf50 0x7fcff81a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.691+0000 7fcffe600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff8104360 0x7fcff81a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.691+0000 7fcffe600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcff810fc60 0x7fcff819ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.691+0000 7fcffe600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81157a0 con 0x7fcff806bf50 2024-05-10T11:13:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.691+0000 7fcffe600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcff8115620 con 0x7fcff810fc60 2024-05-10T11:13:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.691+0000 7fcffe600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcff8115920 con 0x7fcff8104360 2024-05-10T11:13:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.692+0000 7fcff7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff8104360 0x7fcff81a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.692+0000 7fcffd200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff806bf50 0x7fcff81a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.692+0000 7fcff7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff8104360 0x7fcff81a1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:59346/0 (socket says 172.21.15.27:59346) 2024-05-10T11:13:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.692+0000 7fcff7e00700 1 -- 172.21.15.27:0/1886123922 learned_addr learned my addr 172.21.15.27:0/1886123922 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.692+0000 7fcffdc00700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcff810fc60 0x7fcff819ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.692+0000 7fcff7e00700 1 -- 172.21.15.27:0/1886123922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcff810fc60 msgr2=0x7fcff819ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.692+0000 7fcff7e00700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcff810fc60 0x7fcff819ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.692+0000 7fcff7e00700 1 -- 172.21.15.27:0/1886123922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff806bf50 msgr2=0x7fcff81a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.692+0000 7fcff7e00700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff806bf50 0x7fcff81a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.692+0000 7fcff7e00700 1 -- 172.21.15.27:0/1886123922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff819c180 con 0x7fcff8104360 2024-05-10T11:13:33.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.693+0000 7fcff7e00700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff8104360 0x7fcff81a1da0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400eb30 tx=0x7fcfe400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:33.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.693+0000 7fcff5600700 1 -- 172.21.15.27:0/1886123922 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe400cbd0 con 0x7fcff8104360 2024-05-10T11:13:33.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.693+0000 7fcff5600700 1 -- 172.21.15.27:0/1886123922 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfe40044d0 con 0x7fcff8104360 2024-05-10T11:13:33.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.693+0000 7fcff5600700 1 -- 172.21.15.27:0/1886123922 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4003df0 con 0x7fcff8104360 2024-05-10T11:13:33.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.693+0000 7fcffe600700 1 -- 172.21.15.27:0/1886123922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff819c470 con 0x7fcff8104360 2024-05-10T11:13:33.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.693+0000 7fcffe600700 1 -- 172.21.15.27:0/1886123922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff81a8730 con 0x7fcff8104360 2024-05-10T11:13:33.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.697+0000 7fcff5600700 1 -- 172.21.15.27:0/1886123922 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7fcfe4003f90 con 0x7fcff8104360 2024-05-10T11:13:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.698+0000 7fd00a5c8b80 1 Processor -- start 2024-05-10T11:13:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.698+0000 7fd00a5c8b80 1 -- start start 2024-05-10T11:13:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fd00a5c8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 0x55cb134bfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fd00a5c8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134ba060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fd00a5c8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134bab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fd00a5c8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cb1340caa0 con 0x55cb134ba680 2024-05-10T11:13:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fd00a5c8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cb133aa7e0 con 0x55cb134bfa60 2024-05-10T11:13:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fd00a5c8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cb13408a60 con 0x55cb134c0380 2024-05-10T11:13:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fcff5600700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcfe00b2b10 0x7fcfe00b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fcffdc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 0x55cb134bfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fcffdc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 0x55cb134bfe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50342/0 (socket says 172.21.15.27:50342) 2024-05-10T11:13:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fcffdc00700 1 -- 172.21.15.27:0/1096994882 learned_addr learned my addr 172.21.15.27:0/1096994882 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fcffd200700 1 --2- 172.21.15.27:0/1096994882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134ba060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fcffdc00700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcfe00b2b10 0x7fcfe00b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fcff5600700 1 -- 172.21.15.27:0/1886123922 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fcfe40e2ee0 con 0x7fcff8104360 2024-05-10T11:13:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.699+0000 7fcff7e00700 1 --2- 172.21.15.27:0/1096994882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134bab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.700+0000 7fcffd200700 1 -- 172.21.15.27:0/1096994882 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 msgr2=0x55cb134bfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:33.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.700+0000 7fcffd200700 1 --2- 172.21.15.27:0/1096994882 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 0x55cb134bfe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.700+0000 7fcffd200700 1 -- 172.21.15.27:0/1096994882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 msgr2=0x55cb134bab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:33.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.700+0000 7fcffd200700 1 --2- 172.21.15.27:0/1096994882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134bab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.700+0000 7fcffd200700 1 -- 172.21.15.27:0/1096994882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb13359950 con 0x55cb134c0380 2024-05-10T11:13:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.700+0000 7fcff7e00700 1 --2- 172.21.15.27:0/1096994882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134bab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.701+0000 7fcffdc00700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcfe00b2b10 0x7fcfe00b4fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcff0015000 tx=0x7fcff0009ba0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.701+0000 7fcffd200700 1 --2- 172.21.15.27:0/1096994882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134ba060 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcfec005fd0 tx=0x7fcfec0058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9a1c92392b4ae13 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.701+0000 7fcfea000700 1 -- 172.21.15.27:0/1096994882 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec011c80 con 0x55cb134c0380 2024-05-10T11:13:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.701+0000 7fcfea000700 1 -- 172.21.15.27:0/1096994882 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfec011e20 con 0x55cb134c0380 2024-05-10T11:13:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.701+0000 7fcfea000700 1 -- 172.21.15.27:0/1096994882 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec0193e0 con 0x55cb134c0380 2024-05-10T11:13:33.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.702+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/1096994882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 msgr2=0x55cb134ba060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.702+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/1096994882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134ba060 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcfec005fd0 tx=0x7fcfec0058e0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.703+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/1096994882 shutdown_connections 2024-05-10T11:13:33.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.703+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/1096994882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134bab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.703+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/1096994882 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 0x55cb134bfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.703+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/1096994882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134ba060 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.703+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/1096994882 >> 172.21.15.27:0/1096994882 conn(0x55cb133e8be0 msgr2=0x55cb134b4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:33.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.703+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/1096994882 shutdown_connections 2024-05-10T11:13:33.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.703+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/1096994882 wait complete. 2024-05-10T11:13:33.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.703+0000 7fd00a5c8b80 1 -- start start 2024-05-10T11:13:33.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.704+0000 7fd00a5c8b80 1 Processor -- start 2024-05-10T11:13:33.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.704+0000 7fd00a5c8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134ba680 0x55cb134c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.704+0000 7fd00a5c8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134bfa60 0x55cb134c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.704+0000 7fd00a5c8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.704+0000 7fd00a5c8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cb134c6e30 con 0x55cb134bfa60 2024-05-10T11:13:33.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.704+0000 7fd00a5c8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cb13372ba0 con 0x55cb134ba680 2024-05-10T11:13:33.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.704+0000 7fd00a5c8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cb134c6fb0 con 0x55cb134c0380 2024-05-10T11:13:33.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.704+0000 7fd00a5c8b80 10 client.0 _ll_register_callbacks cb 0x55cb133b3cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:13:33.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.705+0000 7fcff7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134bfa60 0x55cb134c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.705+0000 7fcff7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134bfa60 0x55cb134c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45162/0 (socket says 172.21.15.27:45162) 2024-05-10T11:13:33.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.705+0000 7fcff7e00700 1 -- 172.21.15.27:0/2132658482 learned_addr learned my addr 172.21.15.27:0/2132658482 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:33.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.705+0000 7fcffdc00700 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.705+0000 7fcff7e00700 1 -- 172.21.15.27:0/2132658482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 msgr2=0x55cb134c34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.705+0000 7fcff7e00700 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134c34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.705+0000 7fcff7e00700 1 -- 172.21.15.27:0/2132658482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134ba680 msgr2=0x55cb134c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.705+0000 7fcffd200700 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134ba680 0x55cb134c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.706+0000 7fcff7e00700 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134ba680 0x55cb134c5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.706+0000 7fcff7e00700 1 -- 172.21.15.27:0/2132658482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb133d24a0 con 0x55cb134bfa60 2024-05-10T11:13:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.706+0000 7fcff7e00700 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134bfa60 0x55cb134c6460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40046a0 tx=0x7fcfe40aa0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17b7462fa421774 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.706+0000 7fcfdea00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:13:33.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.706+0000 7fcfdea00700 1 -- 172.21.15.27:0/2132658482 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe40a94f0 con 0x55cb134bfa60 2024-05-10T11:13:33.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.706+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2132658482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfec011e20 con 0x55cb134bfa60 2024-05-10T11:13:33.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.707+0000 7fcfdea00700 1 -- 172.21.15.27:0/2132658482 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fcfe40e05d0 con 0x55cb134bfa60 2024-05-10T11:13:33.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.707+0000 7fcfdea00700 1 -- 172.21.15.27:0/2132658482 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcfe4004280 con 0x55cb134bfa60 2024-05-10T11:13:33.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.707+0000 7fd00a5c8b80 10 client.15258 fetch_fsmap learned FSMap version 30 2024-05-10T11:13:33.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.707+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2132658482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cb134d14f0 con 0x55cb134bfa60 2024-05-10T11:13:33.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.707+0000 7fcfdea00700 1 -- 172.21.15.27:0/2132658482 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfec011e20 con 0x55cb134bfa60 2024-05-10T11:13:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.708+0000 7fcfdea00700 1 -- 172.21.15.27:0/2132658482 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcfe40a5510 con 0x55cb134bfa60 2024-05-10T11:13:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.708+0000 7fd00a5c8b80 10 client.15258 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:13:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.708+0000 7fd00a5c8b80 10 client.15258 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:13:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.708+0000 7fd00a5c8b80 20 client.15258 populate_metadata read hostname 'smithi027' 2024-05-10T11:13:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.708+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cb134d3a80 0x55cb134d5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.708+0000 7fd00a5c8b80 4 client.15258 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:13:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.708+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2132658482 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb13487120 con 0x55cb134d3a80 2024-05-10T11:13:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.708+0000 7fcffd200700 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cb134d3a80 0x55cb134d5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.709+0000 7fcffd200700 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cb134d3a80 0x55cb134d5ed0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3198e5c793c83857 server_cookie=388b90f04383b452 in_seq=0 out_seq=0 2024-05-10T11:13:33.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.709+0000 7fcfdea00700 10 client.15258 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:13:33.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.710+0000 7fcfdea00700 1 -- 172.21.15.27:0/2132658482 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cb13487120 con 0x55cb134d3a80 2024-05-10T11:13:33.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.710+0000 7fcfdea00700 10 client.15258 handle_command_reply: tid=0 2024-05-10T11:13:33.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.710+0000 7fd00a5c8b80 1 client.15258 shutdown 2024-05-10T11:13:33.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.710+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2132658482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134bfa60 msgr2=0x55cb134c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.710+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134bfa60 0x55cb134c6460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40046a0 tx=0x7fcfe40aa0b0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.710+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2132658482 shutdown_connections 2024-05-10T11:13:33.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.710+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cb134d3a80 0x55cb134d5ed0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.710+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134c34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.710+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134ba680 0x55cb134c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.710+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/2132658482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134bfa60 0x55cb134c6460 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.710+0000 7fcfdea00700 0 client.15258 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:13:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.710+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2132658482 >> 172.21.15.27:0/2132658482 conn(0x55cb133e8be0 msgr2=0x55cb134af4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.711+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2132658482 shutdown_connections 2024-05-10T11:13:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.711+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2132658482 wait complete. 2024-05-10T11:13:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.711+0000 7fd00a5c8b80 20 client.15258 trim_cache size 0 max 16384 2024-05-10T11:13:33.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.736+0000 7fd00a5c8b80 1 Processor -- start 2024-05-10T11:13:33.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.736+0000 7fd00a5c8b80 1 -- start start 2024-05-10T11:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.736+0000 7fd00a5c8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134ba060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.736+0000 7fd00a5c8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134bfa60 0x55cb134c2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.736+0000 7fd00a5c8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134c0380 0x55cb134af360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.736+0000 7fd00a5c8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81b1660 con 0x55cb134ba680 2024-05-10T11:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.736+0000 7fd00a5c8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81b31a0 con 0x55cb134c0380 2024-05-10T11:13:33.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.736+0000 7fd00a5c8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81b1d30 con 0x55cb134bfa60 2024-05-10T11:13:33.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.737+0000 7fcffd200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134ba060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.737+0000 7fcffd200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134ba060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45178/0 (socket says 172.21.15.27:45178) 2024-05-10T11:13:33.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.737+0000 7fcffd200700 1 -- 172.21.15.27:0/4162533713 learned_addr learned my addr 172.21.15.27:0/4162533713 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:33.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.737+0000 7fcff7e00700 1 --2- 172.21.15.27:0/4162533713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134bfa60 0x55cb134c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.737+0000 7fcffdc00700 1 --2- 172.21.15.27:0/4162533713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134c0380 0x55cb134af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.737+0000 7fcffd200700 1 -- 172.21.15.27:0/4162533713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134bfa60 msgr2=0x55cb134c2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.737+0000 7fcffd200700 1 --2- 172.21.15.27:0/4162533713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134bfa60 0x55cb134c2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.737+0000 7fcffd200700 1 -- 172.21.15.27:0/4162533713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134c0380 msgr2=0x55cb134af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.737+0000 7fcffd200700 1 --2- 172.21.15.27:0/4162533713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134c0380 0x55cb134af360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.737+0000 7fcffd200700 1 -- 172.21.15.27:0/4162533713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb134c1830 con 0x55cb134ba680 2024-05-10T11:13:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.737+0000 7fcffdc00700 1 --2- 172.21.15.27:0/4162533713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134c0380 0x55cb134af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:13:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.737+0000 7fcff7e00700 1 --2- 172.21.15.27:0/4162533713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134bfa60 0x55cb134c2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.738+0000 7fcffd200700 1 --2- 172.21.15.27:0/4162533713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134ba060 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009720 tx=0x7fcfec0057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ece03375334c2fb server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.738+0000 7fcfe9600700 1 -- 172.21.15.27:0/4162533713 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec0050d0 con 0x55cb134ba680 2024-05-10T11:13:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.738+0000 7fcfe9600700 1 -- 172.21.15.27:0/4162533713 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfec005270 con 0x55cb134ba680 2024-05-10T11:13:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.738+0000 7fcfe9600700 1 -- 172.21.15.27:0/4162533713 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec02a210 con 0x55cb134ba680 2024-05-10T11:13:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.739+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/4162533713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 msgr2=0x55cb134ba060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.739+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/4162533713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134ba060 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009720 tx=0x7fcfec0057a0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.739+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/4162533713 shutdown_connections 2024-05-10T11:13:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.739+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/4162533713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134c0380 0x55cb134af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.739+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/4162533713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134ba060 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.739+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/4162533713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134bfa60 0x55cb134c2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.739+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/4162533713 >> 172.21.15.27:0/4162533713 conn(0x55cb133e8be0 msgr2=0x55cb134aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.739+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/4162533713 shutdown_connections 2024-05-10T11:13:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.739+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/4162533713 wait complete. 2024-05-10T11:13:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.740+0000 7fd00a5c8b80 1 -- start start 2024-05-10T11:13:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.740+0000 7fd00a5c8b80 1 Processor -- start 2024-05-10T11:13:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.740+0000 7fd00a5c8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.740+0000 7fd00a5c8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 0x55cb134c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.740+0000 7fd00a5c8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.740+0000 7fd00a5c8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81b2ad0 con 0x55cb134ba680 2024-05-10T11:13:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.740+0000 7fd00a5c8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81b2400 con 0x55cb134bfa60 2024-05-10T11:13:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.740+0000 7fd00a5c8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81af600 con 0x55cb134c0380 2024-05-10T11:13:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.741+0000 7fd00a5c8b80 10 client.0 _ll_register_callbacks cb 0x55cb133b3cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:13:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.741+0000 7fcffd200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.741+0000 7fcff7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 0x55cb134c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.741+0000 7fcff7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 0x55cb134c9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50366/0 (socket says 172.21.15.27:50366) 2024-05-10T11:13:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.741+0000 7fcff7e00700 1 -- 172.21.15.27:0/2816913462 learned_addr learned my addr 172.21.15.27:0/2816913462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.741+0000 7fcffdc00700 1 --2- 172.21.15.27:0/2816913462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.741+0000 7fcff7e00700 1 -- 172.21.15.27:0/2816913462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 msgr2=0x55cb134c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.741+0000 7fcff7e00700 1 --2- 172.21.15.27:0/2816913462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.741+0000 7fcff7e00700 1 -- 172.21.15.27:0/2816913462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 msgr2=0x55cb134cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.741+0000 7fcff7e00700 1 --2- 172.21.15.27:0/2816913462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.741+0000 7fcff7e00700 1 -- 172.21.15.27:0/2816913462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb134ca800 con 0x55cb134bfa60 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.742+0000 7fcff7e00700 1 --2- 172.21.15.27:0/2816913462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 0x55cb134c9a80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40e01f0 tx=0x7fcfe4003890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5be9e296c0902987 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.742+0000 7fcfdea00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.742+0000 7fcfdea00700 1 -- 172.21.15.27:0/2816913462 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe40a9310 con 0x55cb134bfa60 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.742+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2816913462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfec005270 con 0x55cb134bfa60 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.742+0000 7fcfdea00700 1 -- 172.21.15.27:0/2816913462 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fcfe40a9010 con 0x55cb134bfa60 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.742+0000 7fcfdea00700 1 -- 172.21.15.27:0/2816913462 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcfe40a94f0 con 0x55cb134bfa60 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.742+0000 7fd00a5c8b80 10 client.25394 fetch_fsmap learned FSMap version 30 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.742+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2816913462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cb13540370 con 0x55cb134bfa60 2024-05-10T11:13:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.743+0000 7fcfdea00700 1 -- 172.21.15.27:0/2816913462 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfec005270 con 0x55cb134bfa60 2024-05-10T11:13:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.743+0000 7fcfdea00700 1 -- 172.21.15.27:0/2816913462 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcfe40af0e0 con 0x55cb134bfa60 2024-05-10T11:13:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.743+0000 7fd00a5c8b80 10 client.25394 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:13:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.743+0000 7fd00a5c8b80 10 client.25394 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:13:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.743+0000 7fd00a5c8b80 20 client.25394 populate_metadata read hostname 'smithi027' 2024-05-10T11:13:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.743+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/2816913462 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cb135408a0 0x55cb13542cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.743+0000 7fd00a5c8b80 4 client.25394 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:13:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.743+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2816913462 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cb1341a480 con 0x55cb135408a0 2024-05-10T11:13:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.743+0000 7fcffd200700 1 --2- 172.21.15.27:0/2816913462 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cb135408a0 0x55cb13542cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.744+0000 7fcffd200700 1 --2- 172.21.15.27:0/2816913462 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cb135408a0 0x55cb13542cc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29da55e35310aad8 server_cookie=8b0c6f4b0e8de6da in_seq=0 out_seq=0 2024-05-10T11:13:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.744+0000 7fcfdea00700 10 client.25394 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.745+0000 7fcfdea00700 1 -- 172.21.15.27:0/2816913462 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cb1341a480 con 0x55cb135408a0 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.745+0000 7fcfdea00700 10 client.25394 handle_command_reply: tid=0 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.745+0000 7fd00a5c8b80 1 client.25394 shutdown 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.745+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2816913462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 msgr2=0x55cb134c9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.745+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/2816913462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 0x55cb134c9a80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40e01f0 tx=0x7fcfe4003890 comp rx=0 tx=0).stop 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.745+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2816913462 shutdown_connections 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.745+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/2816913462 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cb135408a0 0x55cb13542cc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.745+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/2816913462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb134c0380 0x55cb134c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.745+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/2816913462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb134ba680 0x55cb134cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.745+0000 7fd00a5c8b80 1 --2- 172.21.15.27:0/2816913462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cb134bfa60 0x55cb134c9a80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.745+0000 7fcfdea00700 0 client.25394 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.745+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2816913462 >> 172.21.15.27:0/2816913462 conn(0x55cb133e8be0 msgr2=0x55cb134af4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.746+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2816913462 shutdown_connections 2024-05-10T11:13:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.746+0000 7fd00a5c8b80 1 -- 172.21.15.27:0/2816913462 wait complete. 2024-05-10T11:13:33.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.746+0000 7fd00a5c8b80 20 client.25394 trim_cache size 0 max 16384 2024-05-10T11:13:33.748 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:13:33.748 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:13:33.748 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "af5c031f-c9bc-4bae-acd9-3c9146e2d897", 2024-05-10T11:13:33.748 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:13:33.748 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:13:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.747+0000 7fcffe600700 1 -- 172.21.15.27:0/1886123922 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcfe00b2b10 msgr2=0x7fcfe00b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.747+0000 7fcffe600700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcfe00b2b10 0x7fcfe00b4fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcff0015000 tx=0x7fcff0009ba0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.747+0000 7fcffe600700 1 -- 172.21.15.27:0/1886123922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff8104360 msgr2=0x7fcff81a1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.747+0000 7fcffe600700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff8104360 0x7fcff81a1da0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400eb30 tx=0x7fcfe400ee40 comp rx=0 tx=0).stop 2024-05-10T11:13:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.747+0000 7fcffe600700 1 -- 172.21.15.27:0/1886123922 shutdown_connections 2024-05-10T11:13:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.747+0000 7fcffe600700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcfe00b2b10 0x7fcfe00b4fd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.747+0000 7fcffe600700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcff810fc60 0x7fcff819ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.747+0000 7fcffe600700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff806bf50 0x7fcff81a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.747+0000 7fcffe600700 1 --2- 172.21.15.27:0/1886123922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff8104360 0x7fcff81a1da0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.747+0000 7fcffe600700 1 -- 172.21.15.27:0/1886123922 >> 172.21.15.27:0/1886123922 conn(0x7fcff80791e0 msgr2=0x7fcff8101550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:33.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.747+0000 7fcffe600700 1 -- 172.21.15.27:0/1886123922 shutdown_connections 2024-05-10T11:13:33.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.747+0000 7fcffe600700 1 -- 172.21.15.27:0/1886123922 wait complete. 2024-05-10T11:13:33.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.934+0000 7f360ac00640 1 -- 172.21.15.27:0/4166447790 --> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f3604101300 con 0x7f35e80b2bb0 2024-05-10T11:13:33.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.948+0000 7f3601600640 1 -- 172.21.15.27:0/4166447790 <== mgr.25220 v2:172.21.15.27:6800/1167199542 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f3604101300 con 0x7f35e80b2bb0 2024-05-10T11:13:33.949 INFO:teuthology.orchestra.run.smithi027.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-10T00:00:00", "created": "2024-05-10T11:13:32", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-05-10T11:13:33.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.952+0000 7f360ac00640 1 -- 172.21.15.27:0/4166447790 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f35e80b2bb0 msgr2=0x7f35e80b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.952+0000 7f360ac00640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f35e80b2bb0 0x7f35e80b5070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3604110e50 tx=0x7f35ec005f90 comp rx=0 tx=0).stop 2024-05-10T11:13:33.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.952+0000 7f360ac00640 1 -- 172.21.15.27:0/4166447790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360406b7b0 msgr2=0x7f360419b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:33.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.952+0000 7f360ac00640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360406b7b0 0x7f360419b2c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f35f4005e30 tx=0x7f35f4031840 comp rx=0 tx=0).stop 2024-05-10T11:13:33.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.952+0000 7f360ac00640 1 -- 172.21.15.27:0/4166447790 shutdown_connections 2024-05-10T11:13:33.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.952+0000 7f360ac00640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f35e80b2bb0 0x7f35e80b5070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.952+0000 7f360ac00640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f360410f3a0 0x7f360419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.952+0000 7f360ac00640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604103a00 0x7f360419b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.952+0000 7f360ac00640 1 --2- 172.21.15.27:0/4166447790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360406b7b0 0x7f360419b2c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:33.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.952+0000 7f360ac00640 1 -- 172.21.15.27:0/4166447790 >> 172.21.15.27:0/4166447790 conn(0x7f36040776b0 msgr2=0x7f3604100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:33.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.952+0000 7f360ac00640 1 -- 172.21.15.27:0/4166447790 shutdown_connections 2024-05-10T11:13:33.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:33.952+0000 7f360ac00640 1 -- 172.21.15.27:0/4166447790 wait complete. 2024-05-10T11:13:33.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-10T11:13:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.190+0000 7f34ca200640 1 -- 172.21.15.27:0/3509570834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34c4076ee0 msgr2=0x7f34c40772c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.190+0000 7f34ca200640 1 --2- 172.21.15.27:0/3509570834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34c4076ee0 0x7f34c40772c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f34b4009890 tx=0x7f34b402a360 comp rx=0 tx=0).stop 2024-05-10T11:13:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.190+0000 7f34ca200640 1 -- 172.21.15.27:0/3509570834 shutdown_connections 2024-05-10T11:13:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.190+0000 7f34ca200640 1 --2- 172.21.15.27:0/3509570834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34c4113830 0x7f34c4115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.190+0000 7f34ca200640 1 --2- 172.21.15.27:0/3509570834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c4077800 0x7f34c41132f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.190+0000 7f34ca200640 1 --2- 172.21.15.27:0/3509570834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34c4076ee0 0x7f34c40772c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.190+0000 7f34ca200640 1 -- 172.21.15.27:0/3509570834 >> 172.21.15.27:0/3509570834 conn(0x7f34c41006a0 msgr2=0x7f34c4102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.190+0000 7f34ca200640 1 -- 172.21.15.27:0/3509570834 shutdown_connections 2024-05-10T11:13:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.190+0000 7f34ca200640 1 -- 172.21.15.27:0/3509570834 wait complete. 2024-05-10T11:13:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.191+0000 7f34ca200640 1 Processor -- start 2024-05-10T11:13:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.192+0000 7f34ca200640 1 -- start start 2024-05-10T11:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.192+0000 7f34ca200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34c4076ee0 0x7f34c419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.192+0000 7f34ca200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c4077800 0x7f34c419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.192+0000 7f34ca200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34c4113830 0x7f34c41a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.192+0000 7f34ca200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f34c4119250 con 0x7f34c4077800 2024-05-10T11:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.192+0000 7f34ca200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f34c41190d0 con 0x7f34c4113830 2024-05-10T11:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.192+0000 7f34ca200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f34c41193d0 con 0x7f34c4076ee0 2024-05-10T11:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.193+0000 7f34c3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34c4076ee0 0x7f34c419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.193+0000 7f34c2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c4077800 0x7f34c419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.193+0000 7f34c3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34c4076ee0 0x7f34c419f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:45278/0 (socket says 172.21.15.27:45278) 2024-05-10T11:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.193+0000 7f34c3400640 1 -- 172.21.15.27:0/2748029241 learned_addr learned my addr 172.21.15.27:0/2748029241 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.193+0000 7f34c3e00640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34c4113830 0x7f34c41a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.193+0000 7f34c3e00640 1 -- 172.21.15.27:0/2748029241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34c4076ee0 msgr2=0x7f34c419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.193+0000 7f34c3e00640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34c4076ee0 0x7f34c419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.193+0000 7f34c3e00640 1 -- 172.21.15.27:0/2748029241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c4077800 msgr2=0x7f34c419fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.193+0000 7f34c3e00640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c4077800 0x7f34c419fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.193+0000 7f34c3e00640 1 -- 172.21.15.27:0/2748029241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c41a4750 con 0x7f34c4113830 2024-05-10T11:13:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.194+0000 7f34c3e00640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34c4113830 0x7f34c41a4020 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f34bc005fd0 tx=0x7f34bc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.194+0000 7f34bbe00640 1 -- 172.21.15.27:0/2748029241 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc00cdf0 con 0x7f34c4113830 2024-05-10T11:13:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.194+0000 7f34bbe00640 1 -- 172.21.15.27:0/2748029241 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34bc005020 con 0x7f34c4113830 2024-05-10T11:13:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.194+0000 7f34ca200640 1 -- 172.21.15.27:0/2748029241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c41a4a40 con 0x7f34c4113830 2024-05-10T11:13:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.194+0000 7f34bbe00640 1 -- 172.21.15.27:0/2748029241 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc010690 con 0x7f34c4113830 2024-05-10T11:13:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.194+0000 7f34ca200640 1 -- 172.21.15.27:0/2748029241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c41ac2d0 con 0x7f34c4113830 2024-05-10T11:13:34.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.196+0000 7f34ca200640 1 -- 172.21.15.27:0/2748029241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34a4005290 con 0x7f34c4113830 2024-05-10T11:13:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.196+0000 7f34bbe00640 1 -- 172.21.15.27:0/2748029241 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f34bc004a40 con 0x7f34c4113830 2024-05-10T11:13:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.196+0000 7f34bbe00640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f34a80b2bb0 0x7f34a80b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.196+0000 7f34bbe00640 1 -- 172.21.15.27:0/2748029241 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f34bc0e3130 con 0x7f34c4113830 2024-05-10T11:13:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.196+0000 7f34c3400640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f34a80b2bb0 0x7f34a80b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.196+0000 7f34c3400640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f34a80b2bb0 0x7f34a80b5070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f34b40095f0 tx=0x7f34b40056d0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.203+0000 7f34bbe00640 1 -- 172.21.15.27:0/2748029241 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f34bc0aa4f0 con 0x7f34c4113830 2024-05-10T11:13:34.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.448+0000 7f34ca200640 1 -- 172.21.15.27:0/2748029241 --> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f34a4000bc0 con 0x7f34a80b2bb0 2024-05-10T11:13:34.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.459+0000 7f34bbe00640 1 -- 172.21.15.27:0/2748029241 <== mgr.25220 v2:172.21.15.27:6800/1167199542 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f34a4000bc0 con 0x7f34a80b2bb0 2024-05-10T11:13:34.459 INFO:teuthology.orchestra.run.smithi027.stdout:/ 1m 6m3h 2024-05-10T11:13:34.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.462+0000 7f34ca200640 1 -- 172.21.15.27:0/2748029241 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f34a80b2bb0 msgr2=0x7f34a80b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:34.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.462+0000 7f34ca200640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f34a80b2bb0 0x7f34a80b5070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f34b40095f0 tx=0x7f34b40056d0 comp rx=0 tx=0).stop 2024-05-10T11:13:34.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.462+0000 7f34ca200640 1 -- 172.21.15.27:0/2748029241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34c4113830 msgr2=0x7f34c41a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:34.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.462+0000 7f34ca200640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34c4113830 0x7f34c41a4020 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f34bc005fd0 tx=0x7f34bc00ef10 comp rx=0 tx=0).stop 2024-05-10T11:13:34.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.462+0000 7f34ca200640 1 -- 172.21.15.27:0/2748029241 shutdown_connections 2024-05-10T11:13:34.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.462+0000 7f34ca200640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f34a80b2bb0 0x7f34a80b5070 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:34.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.462+0000 7f34ca200640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34c4113830 0x7f34c41a4020 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:34.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.462+0000 7f34ca200640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c4077800 0x7f34c419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:34.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.462+0000 7f34ca200640 1 --2- 172.21.15.27:0/2748029241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34c4076ee0 0x7f34c419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:34.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.462+0000 7f34ca200640 1 -- 172.21.15.27:0/2748029241 >> 172.21.15.27:0/2748029241 conn(0x7f34c41006a0 msgr2=0x7f34c4102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:34.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.462+0000 7f34ca200640 1 -- 172.21.15.27:0/2748029241 shutdown_connections 2024-05-10T11:13:34.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:34.462+0000 7f34ca200640 1 -- 172.21.15.27:0/2748029241 wait complete. 2024-05-10T11:13:34.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-05-10T11:13:34.490 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:34 smithi189 ceph-mon[34842]: from='client.25360 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:34.490 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:34 smithi189 ceph-mon[34842]: pgmap v13: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:13:34.553 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:34 smithi027 ceph-mon[30054]: from='client.25360 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:34.554 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:34 smithi027 ceph-mon[30054]: pgmap v13: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:13:34.572 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-05-10T11:13:34.583 INFO:teuthology.run_tasks:Running task workunit... 2024-05-10T11:13:34.598 INFO:tasks.workunit:Pulling workunits from ref d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T11:13:34.598 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-10T11:13:34.598 INFO:tasks.workunit:timeout=3h 2024-05-10T11:13:34.598 INFO:tasks.workunit:cleanup=True 2024-05-10T11:13:34.598 DEBUG:teuthology.orchestra.run.smithi027:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-10T11:13:34.659 INFO:teuthology.orchestra.run.smithi027.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-10T11:13:34.659 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-10T11:13:34.659 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 72h/114d Inode: 1099511628283 Links: 2 2024-05-10T11:13:34.659 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-10T11:13:34.659 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-10T11:13:34.659 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-05-10 11:09:35.919636271 +0000 2024-05-10T11:13:34.660 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-05-10 11:09:35.919636271 +0000 2024-05-10T11:13:34.660 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-05-10 11:13:09.862889927 +0000 2024-05-10T11:13:34.660 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-05-10 11:09:35.919636271 +0000 2024-05-10T11:13:34.660 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-10T11:13:34.660 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-10T11:13:34.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:34 smithi076 ceph-mon[34152]: from='client.25360 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:34.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:34 smithi076 ceph-mon[34152]: pgmap v13: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:13:34.763 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d39fa08867836ac070c47cb66c782fa559fd480b 2024-05-10T11:13:34.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:13:34.825 INFO:tasks.workunit.client.0.smithi027.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-10T11:13:35.242 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:35 smithi027 ceph-mon[30054]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:35.242 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:35 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:13:35.242 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:35 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:13:35.242 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:35 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:13:35.242 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:35 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:13:35.242 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:35 smithi027 ceph-mon[30054]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:35.306 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:13:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:35 smithi076 ceph-mon[34152]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:35 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:13:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:35 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:13:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:35 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:13:35.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:35 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:13:35.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:35 smithi076 ceph-mon[34152]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:35.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:35 smithi189 ceph-mon[34842]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:35.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:35 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:13:35.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:35 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:13:35.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:35 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:13:35.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:35 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:13:35.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:35 smithi189 ceph-mon[34842]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:36.508 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:36 smithi027 ceph-mon[30054]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:36.508 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:36 smithi027 ceph-mon[30054]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 76 KiB/s wr, 42 op/s 2024-05-10T11:13:36.637 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:36 smithi189 ceph-mon[34842]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:36.637 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:36 smithi189 ceph-mon[34842]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 76 KiB/s wr, 42 op/s 2024-05-10T11:13:36.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:36 smithi076 ceph-mon[34152]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:13:36.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:36 smithi076 ceph-mon[34152]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 76 KiB/s wr, 42 op/s 2024-05-10T11:13:37.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.760+0000 7f9c1a800700 1 -- 172.21.15.27:0/2924102985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c14105690 msgr2=0x7f9c14105b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:37.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.760+0000 7f9c1a800700 1 --2- 172.21.15.27:0/2924102985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c14105690 0x7f9c14105b10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc009990 tx=0x7f9bfc009ca0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.760+0000 7f9c1a800700 1 -- 172.21.15.27:0/2924102985 shutdown_connections 2024-05-10T11:13:37.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.760+0000 7f9c1a800700 1 --2- 172.21.15.27:0/2924102985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14106050 0x7f9c14112560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.761+0000 7f9c1a800700 1 --2- 172.21.15.27:0/2924102985 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c1410b690 0x7f9c1410ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.761+0000 7f9c1a800700 1 --2- 172.21.15.27:0/2924102985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c14105690 0x7f9c14105b10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.761+0000 7f9c1a800700 1 -- 172.21.15.27:0/2924102985 >> 172.21.15.27:0/2924102985 conn(0x7f9c14101180 msgr2=0x7f9c141035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:37.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.761+0000 7f9c1a800700 1 -- 172.21.15.27:0/2924102985 shutdown_connections 2024-05-10T11:13:37.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.761+0000 7f9c1a800700 1 -- 172.21.15.27:0/2924102985 wait complete. 2024-05-10T11:13:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.762+0000 7f9c1a800700 1 Processor -- start 2024-05-10T11:13:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.762+0000 7f9c1a800700 1 -- start start 2024-05-10T11:13:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.763+0000 7f9c1a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c14105690 0x7f9c1419bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.763+0000 7f9c1a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14106050 0x7f9c1419c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.763+0000 7f9c1a800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1410b690 0x7f9c141a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.763+0000 7f9c1a800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9c14115920 con 0x7f9c14106050 2024-05-10T11:13:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.763+0000 7f9c1a800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9c141157a0 con 0x7f9c14105690 2024-05-10T11:13:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.763+0000 7f9c1a800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9c14115aa0 con 0x7f9c1410b690 2024-05-10T11:13:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.763+0000 7f9c0aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14106050 0x7f9c1419c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.763+0000 7f9c13400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c14105690 0x7f9c1419bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.763+0000 7f9c13400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c14105690 0x7f9c1419bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51590/0 (socket says 172.21.15.27:51590) 2024-05-10T11:13:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.763+0000 7f9c0aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14106050 0x7f9c1419c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45270/0 (socket says 172.21.15.27:45270) 2024-05-10T11:13:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.763+0000 7f9c13400700 1 -- 172.21.15.27:0/2494502455 learned_addr learned my addr 172.21.15.27:0/2494502455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.763+0000 7f9c13e00700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1410b690 0x7f9c141a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.764+0000 7f9c13400700 1 -- 172.21.15.27:0/2494502455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1410b690 msgr2=0x7f9c141a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.764+0000 7f9c13400700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1410b690 0x7f9c141a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.764+0000 7f9c13400700 1 -- 172.21.15.27:0/2494502455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14106050 msgr2=0x7f9c1419c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.764+0000 7f9c13400700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14106050 0x7f9c1419c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.764+0000 7f9c13400700 1 -- 172.21.15.27:0/2494502455 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c141a0dc0 con 0x7f9c14105690 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.764+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14106050 0x7f9c1419c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.764+0000 7f9c13e00700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1410b690 0x7f9c141a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.764+0000 7f9c13400700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c14105690 0x7f9c1419bdb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400ef30 tx=0x7f9c0400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.765+0000 7f9c10c00700 1 -- 172.21.15.27:0/2494502455 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c040092e0 con 0x7f9c14105690 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.765+0000 7f9c1a800700 1 -- 172.21.15.27:0/2494502455 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c141a1050 con 0x7f9c14105690 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.765+0000 7f9c10c00700 1 -- 172.21.15.27:0/2494502455 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c04004bb0 con 0x7f9c14105690 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.765+0000 7f9c10c00700 1 -- 172.21.15.27:0/2494502455 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c04004080 con 0x7f9c14105690 2024-05-10T11:13:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.765+0000 7f9c1a800700 1 -- 172.21.15.27:0/2494502455 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c141a8930 con 0x7f9c14105690 2024-05-10T11:13:37.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.769+0000 7f9c10c00700 1 -- 172.21.15.27:0/2494502455 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f9c0401b030 con 0x7f9c14105690 2024-05-10T11:13:37.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.770+0000 7f9c250a1b80 1 Processor -- start 2024-05-10T11:13:37.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.770+0000 7f9c250a1b80 1 -- start start 2024-05-10T11:13:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.770+0000 7f9c250a1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b060 0x55ab7700b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.770+0000 7f9c250a1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700b980 0x55ab7700f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.770+0000 7f9c250a1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700f9d0 0x55ab7700fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c250a1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ab76f5da20 con 0x55ab7700b060 2024-05-10T11:13:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c250a1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ab76efb760 con 0x55ab7700b980 2024-05-10T11:13:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c250a1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ab76f599e0 con 0x55ab7700f9d0 2024-05-10T11:13:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c13400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700b980 0x55ab7700f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c10c00700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9bf80b2b90 0x7f9bf80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c0aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700f9d0 0x55ab7700fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c13400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700b980 0x55ab7700f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51600/0 (socket says 172.21.15.27:51600) 2024-05-10T11:13:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c13400700 1 -- 172.21.15.27:0/139010775 learned_addr learned my addr 172.21.15.27:0/139010775 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c13e00700 1 --2- 172.21.15.27:0/139010775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b060 0x55ab7700b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c0aa00700 1 -- 172.21.15.27:0/139010775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700b980 msgr2=0x55ab7700f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/139010775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700b980 0x55ab7700f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c10c00700 1 -- 172.21.15.27:0/2494502455 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f9c040e2800 con 0x7f9c14105690 2024-05-10T11:13:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c0aa00700 1 -- 172.21.15.27:0/139010775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b060 msgr2=0x55ab7700b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/139010775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b060 0x55ab7700b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.771+0000 7f9c0aa00700 1 -- 172.21.15.27:0/139010775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab76eaa8d0 con 0x55ab7700f9d0 2024-05-10T11:13:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.772+0000 7f9c13e00700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9bf80b2b90 0x7f9bf80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.772+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/139010775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700f9d0 0x55ab7700fe30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00b2e0 tx=0x7f9bfc005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfec4284e0581b80 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:37.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.772+0000 7f9c08c00700 1 -- 172.21.15.27:0/139010775 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc004240 con 0x55ab7700f9d0 2024-05-10T11:13:37.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.772+0000 7f9c08c00700 1 -- 172.21.15.27:0/139010775 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bfc027070 con 0x55ab7700f9d0 2024-05-10T11:13:37.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.772+0000 7f9c08c00700 1 -- 172.21.15.27:0/139010775 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc003d90 con 0x55ab7700f9d0 2024-05-10T11:13:37.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.773+0000 7f9c250a1b80 1 -- 172.21.15.27:0/139010775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700f9d0 msgr2=0x55ab7700fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:37.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.773+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/139010775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700f9d0 0x55ab7700fe30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00b2e0 tx=0x7f9bfc005fd0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.773+0000 7f9c13e00700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9bf80b2b90 0x7f9bf80b5050 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0096d0 tx=0x7f9c0c004750 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.773+0000 7f9c250a1b80 1 -- 172.21.15.27:0/139010775 shutdown_connections 2024-05-10T11:13:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.773+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/139010775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700f9d0 0x55ab7700fe30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.773+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/139010775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b060 0x55ab7700b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.773+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/139010775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700b980 0x55ab7700f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.773+0000 7f9c250a1b80 1 -- 172.21.15.27:0/139010775 >> 172.21.15.27:0/139010775 conn(0x55ab76f39b60 msgr2=0x55ab770057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.773+0000 7f9c250a1b80 1 -- 172.21.15.27:0/139010775 shutdown_connections 2024-05-10T11:13:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.773+0000 7f9c250a1b80 1 -- 172.21.15.27:0/139010775 wait complete. 2024-05-10T11:13:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.774+0000 7f9c250a1b80 1 -- start start 2024-05-10T11:13:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.774+0000 7f9c250a1b80 1 Processor -- start 2024-05-10T11:13:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.775+0000 7f9c250a1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b060 0x55ab77016e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.775+0000 7f9c250a1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 0x55ab77014210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.775+0000 7f9c250a1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700f9d0 0x55ab77014750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.775+0000 7f9c250a1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ab77017b80 con 0x55ab7700b060 2024-05-10T11:13:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.775+0000 7f9c250a1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ab76ec3b20 con 0x55ab7700f9d0 2024-05-10T11:13:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.775+0000 7f9c250a1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ab77017d00 con 0x55ab7700b980 2024-05-10T11:13:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.775+0000 7f9c250a1b80 10 client.0 _ll_register_callbacks cb 0x55ab76f04c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:13:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.775+0000 7f9c13400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b060 0x55ab77016e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.775+0000 7f9c0aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 0x55ab77014210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.775+0000 7f9c0aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 0x55ab77014210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:45314/0 (socket says 172.21.15.27:45314) 2024-05-10T11:13:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.775+0000 7f9c0aa00700 1 -- 172.21.15.27:0/3552860654 learned_addr learned my addr 172.21.15.27:0/3552860654 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.775+0000 7f9c13e00700 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700f9d0 0x55ab77014750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.776+0000 7f9c0aa00700 1 -- 172.21.15.27:0/3552860654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700f9d0 msgr2=0x55ab77014750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.776+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700f9d0 0x55ab77014750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.776+0000 7f9c0aa00700 1 -- 172.21.15.27:0/3552860654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b060 msgr2=0x55ab77016e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.776+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b060 0x55ab77016e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.776+0000 7f9c0aa00700 1 -- 172.21.15.27:0/3552860654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab76f23420 con 0x55ab7700b980 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.776+0000 7f9c13e00700 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700f9d0 0x55ab77014750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.776+0000 7f9c13400700 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b060 0x55ab77016e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.776+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 0x55ab77014210 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0094e0 tx=0x7f9bfc003960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d41ba38b74b6f8d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.776+0000 7f9c01600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.776+0000 7f9c01600700 1 -- 172.21.15.27:0/3552860654 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc004a30 con 0x55ab7700b980 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.776+0000 7f9c01600700 1 -- 172.21.15.27:0/3552860654 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f9bfc004bd0 con 0x55ab7700b980 2024-05-10T11:13:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.776+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3552860654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bfc027070 con 0x55ab7700b980 2024-05-10T11:13:37.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.777+0000 7f9c01600700 1 -- 172.21.15.27:0/3552860654 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9bfc03c050 con 0x55ab7700b980 2024-05-10T11:13:37.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.777+0000 7f9c250a1b80 10 client.25405 fetch_fsmap learned FSMap version 30 2024-05-10T11:13:37.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.777+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3552860654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ab77017b80 con 0x55ab7700b980 2024-05-10T11:13:37.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.777+0000 7f9c01600700 1 -- 172.21.15.27:0/3552860654 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bfc027070 con 0x55ab7700b980 2024-05-10T11:13:37.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.777+0000 7f9c01600700 1 -- 172.21.15.27:0/3552860654 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9bfc005380 con 0x55ab7700b980 2024-05-10T11:13:37.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.777+0000 7f9c250a1b80 10 client.25405 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:13:37.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.777+0000 7f9c250a1b80 10 client.25405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:13:37.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.777+0000 7f9c250a1b80 20 client.25405 populate_metadata read hostname 'smithi027' 2024-05-10T11:13:37.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.777+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ab77024700 0x55ab77026af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.777+0000 7f9c250a1b80 4 client.25405 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:13:37.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.777+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3552860654 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab76fd80a0 con 0x55ab77024700 2024-05-10T11:13:37.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.778+0000 7f9c13400700 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ab77024700 0x55ab77026af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.778+0000 7f9c13400700 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ab77024700 0x55ab77026af0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e6dbde373e4af20 server_cookie=b1fcc095499eef7a in_seq=0 out_seq=0 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.778+0000 7f9c01600700 10 client.25405 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.779+0000 7f9c01600700 1 -- 172.21.15.27:0/3552860654 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ab76fd80a0 con 0x55ab77024700 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.779+0000 7f9c01600700 10 client.25405 handle_command_reply: tid=0 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.779+0000 7f9c250a1b80 1 client.25405 shutdown 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.780+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3552860654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 msgr2=0x55ab77014210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.780+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 0x55ab77014210 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0094e0 tx=0x7f9bfc003960 comp rx=0 tx=0).stop 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.780+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3552860654 shutdown_connections 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.780+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ab77024700 0x55ab77026af0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.780+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700f9d0 0x55ab77014750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.780+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b060 0x55ab77016e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.780+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/3552860654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 0x55ab77014210 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.780+0000 7f9c01600700 0 client.25405 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.780+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3552860654 >> 172.21.15.27:0/3552860654 conn(0x55ab76f39b60 msgr2=0x55ab770002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.780+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3552860654 shutdown_connections 2024-05-10T11:13:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.780+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3552860654 wait complete. 2024-05-10T11:13:37.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.780+0000 7f9c250a1b80 20 client.25405 trim_cache size 0 max 16384 2024-05-10T11:13:37.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.803+0000 7f9c250a1b80 1 Processor -- start 2024-05-10T11:13:37.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.803+0000 7f9c250a1b80 1 -- start start 2024-05-10T11:13:37.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c250a1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700b060 0x55ab77011e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c250a1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 0x55ab77020c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c250a1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700f9d0 0x55ab77013210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c250a1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9c141b17c0 con 0x55ab7700f9d0 2024-05-10T11:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c250a1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9c141b3300 con 0x55ab7700b060 2024-05-10T11:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c250a1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9c141b1e90 con 0x55ab7700b980 2024-05-10T11:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c0aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 0x55ab77020c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c0aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 0x55ab77020c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:45326/0 (socket says 172.21.15.27:45326) 2024-05-10T11:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c0aa00700 1 -- 172.21.15.27:0/2340845894 learned_addr learned my addr 172.21.15.27:0/2340845894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c13e00700 1 --2- 172.21.15.27:0/2340845894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700f9d0 0x55ab77013210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c13400700 1 --2- 172.21.15.27:0/2340845894 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700b060 0x55ab77011e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c0aa00700 1 -- 172.21.15.27:0/2340845894 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700b060 msgr2=0x55ab77011e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:37.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/2340845894 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700b060 0x55ab77011e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c0aa00700 1 -- 172.21.15.27:0/2340845894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700f9d0 msgr2=0x55ab77013210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/2340845894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700f9d0 0x55ab77013210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c0aa00700 1 -- 172.21.15.27:0/2340845894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab770150c0 con 0x55ab7700b980 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.804+0000 7f9c13400700 1 --2- 172.21.15.27:0/2340845894 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700b060 0x55ab77011e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.805+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/2340845894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 0x55ab77020c10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00b2e0 tx=0x7f9bfc00b610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbeb6ec5cbc0252f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.805+0000 7f9c03e00700 1 -- 172.21.15.27:0/2340845894 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc003e80 con 0x55ab7700b980 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.805+0000 7f9c03e00700 1 -- 172.21.15.27:0/2340845894 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bfc027070 con 0x55ab7700b980 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.805+0000 7f9c03e00700 1 -- 172.21.15.27:0/2340845894 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc0357d0 con 0x55ab7700b980 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.806+0000 7f9c250a1b80 1 -- 172.21.15.27:0/2340845894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 msgr2=0x55ab77020c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.806+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/2340845894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 0x55ab77020c10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00b2e0 tx=0x7f9bfc00b610 comp rx=0 tx=0).stop 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.806+0000 7f9c250a1b80 1 -- 172.21.15.27:0/2340845894 shutdown_connections 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.806+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/2340845894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700f9d0 0x55ab77013210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.806+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/2340845894 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700b060 0x55ab77011e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.806+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/2340845894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b980 0x55ab77020c10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.806+0000 7f9c250a1b80 1 -- 172.21.15.27:0/2340845894 >> 172.21.15.27:0/2340845894 conn(0x55ab76f39b60 msgr2=0x55ab77012b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:37.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.806+0000 7f9c250a1b80 1 -- 172.21.15.27:0/2340845894 shutdown_connections 2024-05-10T11:13:37.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.806+0000 7f9c250a1b80 1 -- 172.21.15.27:0/2340845894 wait complete. 2024-05-10T11:13:37.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.806+0000 7f9c250a1b80 1 -- start start 2024-05-10T11:13:37.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.807+0000 7f9c250a1b80 1 Processor -- start 2024-05-10T11:13:37.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.807+0000 7f9c250a1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b060 0x55ab7701d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.807+0000 7f9c250a1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b980 0x55ab7702b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.807+0000 7f9c250a1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700f9d0 0x55ab7702ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.807+0000 7f9c250a1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9c141b2c30 con 0x55ab7700b980 2024-05-10T11:13:37.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.807+0000 7f9c250a1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9c141b2560 con 0x55ab7700f9d0 2024-05-10T11:13:37.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.807+0000 7f9c250a1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9c141af760 con 0x55ab7700b060 2024-05-10T11:13:37.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.807+0000 7f9c250a1b80 10 client.0 _ll_register_callbacks cb 0x55ab76f04c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.807+0000 7f9c0aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b980 0x55ab7702b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.807+0000 7f9c13400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b060 0x55ab7701d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.807+0000 7f9c0aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b980 0x55ab7702b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45310/0 (socket says 172.21.15.27:45310) 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.807+0000 7f9c0aa00700 1 -- 172.21.15.27:0/3078589512 learned_addr learned my addr 172.21.15.27:0/3078589512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.808+0000 7f9c13e00700 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700f9d0 0x55ab7702ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.808+0000 7f9c0aa00700 1 -- 172.21.15.27:0/3078589512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b060 msgr2=0x55ab7701d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.808+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b060 0x55ab7701d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.808+0000 7f9c0aa00700 1 -- 172.21.15.27:0/3078589512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700f9d0 msgr2=0x55ab7702ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.808+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700f9d0 0x55ab7702ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.808+0000 7f9c0aa00700 1 -- 172.21.15.27:0/3078589512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab7702bff0 con 0x55ab7700b980 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.808+0000 7f9c13400700 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b060 0x55ab7701d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.808+0000 7f9c0aa00700 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b980 0x55ab7702b540 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc01a040 tx=0x7f9bfc00f4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c176b917aeb861 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.808+0000 7f9c01600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.809+0000 7f9c01600700 1 -- 172.21.15.27:0/3078589512 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc003a80 con 0x55ab7700b980 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.809+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3078589512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bfc027070 con 0x55ab7700b980 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.809+0000 7f9c01600700 1 -- 172.21.15.27:0/3078589512 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f9bfc00f9b0 con 0x55ab7700b980 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.809+0000 7f9c01600700 1 -- 172.21.15.27:0/3078589512 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9bfc052050 con 0x55ab7700b980 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.809+0000 7f9c250a1b80 10 client.15288 fetch_fsmap learned FSMap version 30 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.809+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3078589512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ab7701ce00 con 0x55ab7700b980 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.809+0000 7f9c01600700 1 -- 172.21.15.27:0/3078589512 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bfc027070 con 0x55ab7700b980 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.810+0000 7f9c01600700 1 -- 172.21.15.27:0/3078589512 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9bfc041c60 con 0x55ab7700b980 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.810+0000 7f9c250a1b80 10 client.15288 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.810+0000 7f9c250a1b80 10 client.15288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.810+0000 7f9c250a1b80 20 client.15288 populate_metadata read hostname 'smithi027' 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.810+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ab77097900 0x55ab77099d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.810+0000 7f9c250a1b80 4 client.15288 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:13:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.810+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3078589512 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ab76f6b400 con 0x55ab77097900 2024-05-10T11:13:37.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.810+0000 7f9c13400700 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ab77097900 0x55ab77099d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c13400700 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ab77097900 0x55ab77099d20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48ad19d19640fc34 server_cookie=34dddc44cfd9b479 in_seq=0 out_seq=0 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c01600700 10 client.15288 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c01600700 1 -- 172.21.15.27:0/3078589512 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ab76f6b400 con 0x55ab77097900 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c01600700 10 client.15288 handle_command_reply: tid=0 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c250a1b80 1 client.15288 shutdown 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3078589512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b980 msgr2=0x55ab7702b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b980 0x55ab7702b540 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc01a040 tx=0x7f9bfc00f4e0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3078589512 shutdown_connections 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ab77097900 0x55ab77099d20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab7700b060 0x55ab7701d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ab7700f9d0 0x55ab7702ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c250a1b80 1 --2- 172.21.15.27:0/3078589512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ab7700b980 0x55ab7702b540 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3078589512 >> 172.21.15.27:0/3078589512 conn(0x55ab76f39b60 msgr2=0x55ab77014590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c01600700 0 client.15288 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:13:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3078589512 shutdown_connections 2024-05-10T11:13:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.811+0000 7f9c250a1b80 1 -- 172.21.15.27:0/3078589512 wait complete. 2024-05-10T11:13:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.812+0000 7f9c250a1b80 20 client.15288 trim_cache size 0 max 16384 2024-05-10T11:13:37.814 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:13:37.814 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:13:37.814 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:13:37.814 INFO:teuthology.orchestra.run.smithi027.stdout: "af5c031f-c9bc-4bae-acd9-3c9146e2d897": { 2024-05-10T11:13:37.814 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:13:37.814 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "af5c031f-c9bc-4bae-acd9-3c9146e2d897", 2024-05-10T11:13:37.814 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:13:37.815 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:13:37.815 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:13:37.815 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:13:37.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.812+0000 7f9c1a800700 1 -- 172.21.15.27:0/2494502455 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9bf80b2b90 msgr2=0x7f9bf80b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:37.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.812+0000 7f9c1a800700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9bf80b2b90 0x7f9bf80b5050 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0096d0 tx=0x7f9c0c004750 comp rx=0 tx=0).stop 2024-05-10T11:13:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.812+0000 7f9c1a800700 1 -- 172.21.15.27:0/2494502455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c14105690 msgr2=0x7f9c1419bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:13:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.812+0000 7f9c1a800700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c14105690 0x7f9c1419bdb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400ef30 tx=0x7f9c0400c3a0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.812+0000 7f9c1a800700 1 -- 172.21.15.27:0/2494502455 shutdown_connections 2024-05-10T11:13:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.813+0000 7f9c1a800700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9bf80b2b90 0x7f9bf80b5050 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.813+0000 7f9c1a800700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1410b690 0x7f9c141a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.813+0000 7f9c1a800700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c14105690 0x7f9c1419bdb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.813+0000 7f9c1a800700 1 --2- 172.21.15.27:0/2494502455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14106050 0x7f9c1419c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:13:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.813+0000 7f9c1a800700 1 -- 172.21.15.27:0/2494502455 >> 172.21.15.27:0/2494502455 conn(0x7f9c14101180 msgr2=0x7f9c14101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:13:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.813+0000 7f9c1a800700 1 -- 172.21.15.27:0/2494502455 shutdown_connections 2024-05-10T11:13:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:13:37.813+0000 7f9c1a800700 1 -- 172.21.15.27:0/2494502455 wait complete. 2024-05-10T11:13:38.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:38 smithi027 ceph-mon[30054]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 38 op/s 2024-05-10T11:13:38.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:38 smithi076 ceph-mon[34152]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 38 op/s 2024-05-10T11:13:38.678 INFO:tasks.cephfs.filesystem:scrub status for tag:af5c031f-c9bc-4bae-acd9-3c9146e2d897 - {'path': '/', 'tag': 'af5c031f-c9bc-4bae-acd9-3c9146e2d897', 'options': 'recursive,force'} 2024-05-10T11:13:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:38 smithi189 ceph-mon[34842]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 38 op/s 2024-05-10T11:13:40.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:40 smithi027 ceph-mon[30054]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 38 op/s 2024-05-10T11:13:40.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:40 smithi076 ceph-mon[34152]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 38 op/s 2024-05-10T11:13:40.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:40 smithi189 ceph-mon[34842]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 38 op/s 2024-05-10T11:13:41.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:41 smithi027 ceph-mon[30054]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-10T11:13:41.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:41 smithi076 ceph-mon[34152]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-10T11:13:41.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:41 smithi189 ceph-mon[34842]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-10T11:13:44.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:44 smithi189 ceph-mon[34842]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-10T11:13:44.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:44 smithi027 ceph-mon[30054]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-10T11:13:44.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:44 smithi076 ceph-mon[34152]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-10T11:13:46.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:46 smithi189 ceph-mon[34842]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-10T11:13:46.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:46 smithi027 ceph-mon[30054]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-10T11:13:46.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:46 smithi076 ceph-mon[34152]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-10T11:13:47.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:47 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:13:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:47 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:13:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:47 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:13:48.576 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:48 smithi076 ceph-mon[34152]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 10 op/s 2024-05-10T11:13:48.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:48 smithi027 ceph-mon[30054]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 10 op/s 2024-05-10T11:13:48.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:48 smithi189 ceph-mon[34842]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 10 op/s 2024-05-10T11:13:50.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:50 smithi027 ceph-mon[30054]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 10 op/s 2024-05-10T11:13:50.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:50 smithi076 ceph-mon[34152]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 10 op/s 2024-05-10T11:13:50.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:50 smithi189 ceph-mon[34842]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 10 op/s 2024-05-10T11:13:52.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:52 smithi027 ceph-mon[30054]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 767 B/s wr, 14 op/s 2024-05-10T11:13:52.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:52 smithi076 ceph-mon[34152]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 767 B/s wr, 14 op/s 2024-05-10T11:13:52.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:52 smithi189 ceph-mon[34842]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 767 B/s wr, 14 op/s 2024-05-10T11:13:53.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:53 smithi027 ceph-mon[30054]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:13:53.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:53 smithi076 ceph-mon[34152]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:13:53.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:53 smithi189 ceph-mon[34842]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:13:56.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:56 smithi189 ceph-mon[34842]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-10T11:13:56.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:56 smithi027 ceph-mon[30054]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-10T11:13:56.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:56 smithi076 ceph-mon[34152]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-10T11:13:58.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:13:58 smithi189 ceph-mon[34842]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:13:58.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:13:58 smithi027 ceph-mon[30054]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:13:58.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:13:58 smithi076 ceph-mon[34152]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:00.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:00 smithi027 ceph-mon[30054]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-10T11:14:00.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:14:00 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:14:00.452+0000 7fa1e2000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:14:00.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:14:00 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:14:00.452+0000 7fa1e2000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:14:00.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:14:00 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:14:00.452+0000 7fa1e2000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:14:00.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:14:00 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:14:00.452+0000 7fa1e2000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:14:00.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:14:00 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:14:00.452+0000 7fa1e2000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-10T11:14:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:00 smithi076 ceph-mon[34152]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-10T11:14:00.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:00 smithi189 ceph-mon[34842]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-10T11:14:02.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:02 smithi027 ceph-mon[30054]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-10T11:14:02.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:02 smithi027 ceph-mon[30054]: mgrmap e26: x(active, since 48s), standbys: z, y 2024-05-10T11:14:02.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:02 smithi076 ceph-mon[34152]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-10T11:14:02.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:02 smithi076 ceph-mon[34152]: mgrmap e26: x(active, since 48s), standbys: z, y 2024-05-10T11:14:02.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:02 smithi189 ceph-mon[34842]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-10T11:14:02.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:02 smithi189 ceph-mon[34842]: mgrmap e26: x(active, since 48s), standbys: z, y 2024-05-10T11:14:03.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:03 smithi027 ceph-mon[30054]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:03.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:03 smithi076 ceph-mon[34152]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:03.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:03 smithi189 ceph-mon[34842]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:06.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:06 smithi189 ceph-mon[34842]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-10T11:14:06.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:06 smithi027 ceph-mon[30054]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-10T11:14:06.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:06 smithi076 ceph-mon[34152]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-10T11:14:08.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:08 smithi027 ceph-mon[30054]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-10T11:14:08.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:08 smithi076 ceph-mon[34152]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-10T11:14:08.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:14:08.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:08 smithi189 ceph-mon[34842]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-10T11:14:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:14:10.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:10 smithi027 ceph-mon[30054]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-05-10T11:14:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:10 smithi076 ceph-mon[34152]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-05-10T11:14:10.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:10 smithi189 ceph-mon[34842]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-05-10T11:14:11.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.582+0000 7f8807200700 1 -- 172.21.15.27:0/1527931504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8808109460 msgr2=0x7f8808109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:11.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.582+0000 7f8807200700 1 --2- 172.21.15.27:0/1527931504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8808109460 0x7f8808109840 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f87f8009990 tx=0x7f87f8009ca0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.582+0000 7f8807200700 1 -- 172.21.15.27:0/1527931504 shutdown_connections 2024-05-10T11:14:11.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.582+0000 7f8807200700 1 --2- 172.21.15.27:0/1527931504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8808103d90 0x7f88081102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.582+0000 7f8807200700 1 --2- 172.21.15.27:0/1527931504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8808109460 0x7f8808109840 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.582+0000 7f8807200700 1 --2- 172.21.15.27:0/1527931504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88081033d0 0x7f8808103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.582+0000 7f8807200700 1 -- 172.21.15.27:0/1527931504 >> 172.21.15.27:0/1527931504 conn(0x7f88080fef20 msgr2=0x7f8808101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:11.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.582+0000 7f8807200700 1 -- 172.21.15.27:0/1527931504 shutdown_connections 2024-05-10T11:14:11.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.582+0000 7f8807200700 1 -- 172.21.15.27:0/1527931504 wait complete. 2024-05-10T11:14:11.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.583+0000 7f8807200700 1 Processor -- start 2024-05-10T11:14:11.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.583+0000 7f8807200700 1 -- start start 2024-05-10T11:14:11.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.584+0000 7f8807200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88081033d0 0x7f88081a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.584+0000 7f8807200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8808103d90 0x7f88081a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.584+0000 7f8807200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8808109460 0x7f880819bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.584+0000 7f8807200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88081135c0 con 0x7f8808109460 2024-05-10T11:14:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.584+0000 7f8807200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8808113440 con 0x7f88081033d0 2024-05-10T11:14:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.584+0000 7f8807200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8808113740 con 0x7f8808103d90 2024-05-10T11:14:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.584+0000 7f8805e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88081033d0 0x7f88081a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.584+0000 7f8805400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8808103d90 0x7f88081a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.584+0000 7f8805e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88081033d0 0x7f88081a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:59982/0 (socket says 172.21.15.27:59982) 2024-05-10T11:14:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.584+0000 7f8805e00700 1 -- 172.21.15.27:0/1280163456 learned_addr learned my addr 172.21.15.27:0/1280163456 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.584+0000 7f8806800700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8808109460 0x7f880819bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.585+0000 7f8805400700 1 -- 172.21.15.27:0/1280163456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88081033d0 msgr2=0x7f88081a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.585+0000 7f8805400700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88081033d0 0x7f88081a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.585+0000 7f8805400700 1 -- 172.21.15.27:0/1280163456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8808109460 msgr2=0x7f880819bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:11.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.585+0000 7f8805400700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8808109460 0x7f880819bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.585+0000 7f8805400700 1 -- 172.21.15.27:0/1280163456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880819c240 con 0x7f8808103d90 2024-05-10T11:14:11.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.585+0000 7f8805e00700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88081033d0 0x7f88081a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:11.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.585+0000 7f8805400700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8808103d90 0x7f88081a1e60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f87f400e9f0 tx=0x7f87f400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:11.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.585+0000 7f87fea00700 1 -- 172.21.15.27:0/1280163456 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f400c9c0 con 0x7f8808103d90 2024-05-10T11:14:11.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.586+0000 7f8807200700 1 -- 172.21.15.27:0/1280163456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f880819c530 con 0x7f8808103d90 2024-05-10T11:14:11.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.586+0000 7f87fea00700 1 -- 172.21.15.27:0/1280163456 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87f400cb60 con 0x7f8808103d90 2024-05-10T11:14:11.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.586+0000 7f87fea00700 1 -- 172.21.15.27:0/1280163456 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f4018680 con 0x7f8808103d90 2024-05-10T11:14:11.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.586+0000 7f8807200700 1 -- 172.21.15.27:0/1280163456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88081a87f0 con 0x7f8808103d90 2024-05-10T11:14:11.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.592+0000 7f8817508b80 1 Processor -- start 2024-05-10T11:14:11.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.592+0000 7f87fea00700 1 -- 172.21.15.27:0/1280163456 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f87f400cd10 con 0x7f8808103d90 2024-05-10T11:14:11.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.593+0000 7f8817508b80 1 -- start start 2024-05-10T11:14:11.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.593+0000 7f8817508b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b3fcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.593+0000 7f8817508b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b3f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.593+0000 7f8817508b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3f7600 0x55bd3b3f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.593+0000 7f8817508b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bd3b349a20 con 0x55bd3b3fd300 2024-05-10T11:14:11.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.593+0000 7f8817508b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55bd3b2e7760 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.593+0000 7f8817508b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bd3b3459e0 con 0x55bd3b3f7600 2024-05-10T11:14:11.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.593+0000 7f8805400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3f7600 0x55bd3b3f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.593+0000 7f8805e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b3f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.593+0000 7f8805400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3f7600 0x55bd3b3f7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:58564/0 (socket says 172.21.15.27:58564) 2024-05-10T11:14:11.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.593+0000 7f8805400700 1 -- 172.21.15.27:0/4231765038 learned_addr learned my addr 172.21.15.27:0/4231765038 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:11.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.593+0000 7f8806800700 1 --2- 172.21.15.27:0/4231765038 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b3fcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.594+0000 7f8805400700 1 -- 172.21.15.27:0/4231765038 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3fc9e0 msgr2=0x55bd3b3fcdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:11.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.594+0000 7f8805400700 1 --2- 172.21.15.27:0/4231765038 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b3fcdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.594+0000 7f8805400700 1 -- 172.21.15.27:0/4231765038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 msgr2=0x55bd3b3f6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:11.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.594+0000 7f8805400700 1 --2- 172.21.15.27:0/4231765038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b3f6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.594+0000 7f8805400700 1 -- 172.21.15.27:0/4231765038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd3b2968d0 con 0x55bd3b3f7600 2024-05-10T11:14:11.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.594+0000 7f8805400700 1 --2- 172.21.15.27:0/4231765038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3f7600 0x55bd3b3f7a80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f87f40dfe50 tx=0x7f87f40e96f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e857d2ec1a557cae server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:11.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.595+0000 7f87f3400700 1 -- 172.21.15.27:0/4231765038 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f40e9ca0 con 0x55bd3b3f7600 2024-05-10T11:14:11.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.595+0000 7f87f3400700 1 -- 172.21.15.27:0/4231765038 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87f40e9e40 con 0x55bd3b3f7600 2024-05-10T11:14:11.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.595+0000 7f87f3400700 1 -- 172.21.15.27:0/4231765038 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f40f3630 con 0x55bd3b3f7600 2024-05-10T11:14:11.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.595+0000 7f87fea00700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f87ec0b2ba0 0x7f87ec0b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.595+0000 7f87fea00700 1 -- 172.21.15.27:0/1280163456 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f87f40e2da0 con 0x7f8808103d90 2024-05-10T11:14:11.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.595+0000 7f8806800700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f87ec0b2ba0 0x7f87ec0b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.596+0000 7f8817508b80 1 -- 172.21.15.27:0/4231765038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3f7600 msgr2=0x55bd3b3f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:11.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.596+0000 7f8817508b80 1 --2- 172.21.15.27:0/4231765038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3f7600 0x55bd3b3f7a80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f87f40dfe50 tx=0x7f87f40e96f0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.596+0000 7f8817508b80 1 -- 172.21.15.27:0/4231765038 shutdown_connections 2024-05-10T11:14:11.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.596+0000 7f8817508b80 1 --2- 172.21.15.27:0/4231765038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3f7600 0x55bd3b3f7a80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.596+0000 7f8817508b80 1 --2- 172.21.15.27:0/4231765038 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b3fcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.596+0000 7f8817508b80 1 --2- 172.21.15.27:0/4231765038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b3f6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.596+0000 7f8817508b80 1 -- 172.21.15.27:0/4231765038 >> 172.21.15.27:0/4231765038 conn(0x55bd3b325b60 msgr2=0x55bd3b3f19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:11.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.596+0000 7f8817508b80 1 -- 172.21.15.27:0/4231765038 shutdown_connections 2024-05-10T11:14:11.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.596+0000 7f8817508b80 1 -- 172.21.15.27:0/4231765038 wait complete. 2024-05-10T11:14:11.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.596+0000 7f8806800700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f87ec0b2ba0 0x7f87ec0b5060 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8800012710 tx=0x7f880000b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.597+0000 7f8817508b80 1 -- start start 2024-05-10T11:14:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.597+0000 7f8817508b80 1 Processor -- start 2024-05-10T11:14:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.598+0000 7f8817508b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3f7600 0x55bd3b402f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.598+0000 7f8817508b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b400320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.598+0000 7f8817508b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b400860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.598+0000 7f8817508b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bd3b403db0 con 0x55bd3b3fd300 2024-05-10T11:14:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.598+0000 7f8817508b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55bd3b2afb20 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.598+0000 7f8817508b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bd3b403f30 con 0x55bd3b3f7600 2024-05-10T11:14:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.598+0000 7f8817508b80 10 client.0 _ll_register_callbacks cb 0x55bd3b2f0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.598+0000 7f8806800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b400860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.598+0000 7f8806800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b400860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33070/0 (socket says 172.21.15.27:33070) 2024-05-10T11:14:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.598+0000 7f8806800700 1 -- 172.21.15.27:0/1936991329 learned_addr learned my addr 172.21.15.27:0/1936991329 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.598+0000 7f8805400700 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b400320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.598+0000 7f8805e00700 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3f7600 0x55bd3b402f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.599+0000 7f8805400700 1 -- 172.21.15.27:0/1936991329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3f7600 msgr2=0x55bd3b402f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.599+0000 7f8805400700 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3f7600 0x55bd3b402f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.599+0000 7f8805400700 1 -- 172.21.15.27:0/1936991329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 msgr2=0x55bd3b400860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.599+0000 7f8805400700 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b400860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.599+0000 7f8805400700 1 -- 172.21.15.27:0/1936991329 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd3b30f420 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.599+0000 7f8806800700 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b400860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.599+0000 7f8805400700 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b400320 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f87f400e9c0 tx=0x7f87f40e3020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cc71b0eb466de82 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.599+0000 7f87e7e00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:14:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.600+0000 7f87e7e00700 1 -- 172.21.15.27:0/1936991329 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f40a5930 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.600+0000 7f87e7e00700 1 -- 172.21.15.27:0/1936991329 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f87f40a6060 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.600+0000 7f8817508b80 1 -- 172.21.15.27:0/1936991329 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bd3b3c40a0 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.600+0000 7f87e7e00700 1 -- 172.21.15.27:0/1936991329 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f87f40a5200 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.600+0000 7f8817508b80 10 client.25436 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.600+0000 7f8817508b80 1 -- 172.21.15.27:0/1936991329 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55bd3b416d30 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.600+0000 7f87e7e00700 1 -- 172.21.15.27:0/1936991329 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55bd3b3c40a0 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.601+0000 7f87e7e00700 1 -- 172.21.15.27:0/1936991329 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f87f40b2440 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.601+0000 7f8817508b80 10 client.25436 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.601+0000 7f8817508b80 10 client.25436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:14:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.601+0000 7f8817508b80 20 client.25436 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.601+0000 7f8817508b80 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55bd3b419200 0x55bd3b41b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.601+0000 7f8817508b80 4 client.25436 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:14:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.601+0000 7f8817508b80 1 -- 172.21.15.27:0/1936991329 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd3b357400 con 0x55bd3b419200 2024-05-10T11:14:11.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.601+0000 7f8805e00700 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55bd3b419200 0x55bd3b41b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.602+0000 7f8805e00700 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55bd3b419200 0x55bd3b41b5f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f69fac4c7f56c2b9 server_cookie=a1f15125cc4a58a4 in_seq=0 out_seq=0 2024-05-10T11:14:11.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.602+0000 7f87e7e00700 10 client.25436 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:11.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f87e7e00700 1 -- 172.21.15.27:0/1936991329 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bd3b357400 con 0x55bd3b419200 2024-05-10T11:14:11.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f87e7e00700 10 client.25436 handle_command_reply: tid=0 2024-05-10T11:14:11.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f8817508b80 1 client.25436 shutdown 2024-05-10T11:14:11.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f8817508b80 1 -- 172.21.15.27:0/1936991329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3fc9e0 msgr2=0x55bd3b400320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:11.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f8817508b80 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b400320 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f87f400e9c0 tx=0x7f87f40e3020 comp rx=0 tx=0).stop 2024-05-10T11:14:11.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f8817508b80 1 -- 172.21.15.27:0/1936991329 shutdown_connections 2024-05-10T11:14:11.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f8817508b80 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55bd3b419200 0x55bd3b41b5f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f8817508b80 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3f7600 0x55bd3b402f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f8817508b80 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b400860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f8817508b80 1 --2- 172.21.15.27:0/1936991329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b400320 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f8817508b80 1 -- 172.21.15.27:0/1936991329 >> 172.21.15.27:0/1936991329 conn(0x55bd3b325b60 msgr2=0x55bd3b3ec460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:11.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f87e7e00700 0 client.25436 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:11.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f8817508b80 1 -- 172.21.15.27:0/1936991329 shutdown_connections 2024-05-10T11:14:11.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.603+0000 7f8817508b80 1 -- 172.21.15.27:0/1936991329 wait complete. 2024-05-10T11:14:11.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.604+0000 7f8817508b80 20 client.25436 trim_cache size 0 max 16384 2024-05-10T11:14:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.650+0000 7f8817508b80 1 Processor -- start 2024-05-10T11:14:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.650+0000 7f8817508b80 1 -- start start 2024-05-10T11:14:11.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.650+0000 7f8817508b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3f7600 0x55bd3b3f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.650+0000 7f8817508b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b3fdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.650+0000 7f8817508b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b3ff320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.650+0000 7f8817508b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88081ad450 con 0x55bd3b3fd300 2024-05-10T11:14:11.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.650+0000 7f8817508b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f88081aef90 con 0x55bd3b3f7600 2024-05-10T11:14:11.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.650+0000 7f8817508b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f88081adb20 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.651+0000 7f8806800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b3ff320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.651+0000 7f8806800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b3ff320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33082/0 (socket says 172.21.15.27:33082) 2024-05-10T11:14:11.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.651+0000 7f8805400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b3fdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.651+0000 7f8806800700 1 -- 172.21.15.27:0/2037518211 learned_addr learned my addr 172.21.15.27:0/2037518211 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.651+0000 7f8805e00700 1 --2- 172.21.15.27:0/2037518211 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3f7600 0x55bd3b3f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.651+0000 7f8805400700 1 -- 172.21.15.27:0/2037518211 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3f7600 msgr2=0x55bd3b3f6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.651+0000 7f8805400700 1 --2- 172.21.15.27:0/2037518211 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3f7600 0x55bd3b3f6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.651+0000 7f8805400700 1 -- 172.21.15.27:0/2037518211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 msgr2=0x55bd3b3ff320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.651+0000 7f8805400700 1 --2- 172.21.15.27:0/2037518211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b3ff320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.651+0000 7f8805400700 1 -- 172.21.15.27:0/2037518211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd3b3fd100 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.651+0000 7f8805e00700 1 --2- 172.21.15.27:0/2037518211 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3f7600 0x55bd3b3f6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.652+0000 7f8805400700 1 --2- 172.21.15.27:0/2037518211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b3fdf70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f87f40a9a00 tx=0x7f87f40b2800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23426f8bb049484a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.652+0000 7f87f2a00700 1 -- 172.21.15.27:0/2037518211 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f40e9ca0 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.652+0000 7f87f2a00700 1 -- 172.21.15.27:0/2037518211 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87f40a9580 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.652+0000 7f87f2a00700 1 -- 172.21.15.27:0/2037518211 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f40a91e0 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.653+0000 7f8817508b80 1 -- 172.21.15.27:0/2037518211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3fc9e0 msgr2=0x55bd3b3fdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:11.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.653+0000 7f8817508b80 1 --2- 172.21.15.27:0/2037518211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b3fdf70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f87f40a9a00 tx=0x7f87f40b2800 comp rx=0 tx=0).stop 2024-05-10T11:14:11.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.653+0000 7f8817508b80 1 -- 172.21.15.27:0/2037518211 shutdown_connections 2024-05-10T11:14:11.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.653+0000 7f8817508b80 1 --2- 172.21.15.27:0/2037518211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b3ff320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.653+0000 7f8817508b80 1 --2- 172.21.15.27:0/2037518211 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3f7600 0x55bd3b3f6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.653+0000 7f8817508b80 1 --2- 172.21.15.27:0/2037518211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b3fdf70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.653+0000 7f8817508b80 1 -- 172.21.15.27:0/2037518211 >> 172.21.15.27:0/2037518211 conn(0x55bd3b325b60 msgr2=0x55bd3b3fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:11.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.653+0000 7f8817508b80 1 -- 172.21.15.27:0/2037518211 shutdown_connections 2024-05-10T11:14:11.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.653+0000 7f8817508b80 1 -- 172.21.15.27:0/2037518211 wait complete. 2024-05-10T11:14:11.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.654+0000 7f8817508b80 1 -- start start 2024-05-10T11:14:11.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.654+0000 7f8817508b80 1 Processor -- start 2024-05-10T11:14:11.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.654+0000 7f8817508b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3f7600 0x55bd3b41e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8817508b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b41ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8817508b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b41bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8817508b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88081ae8c0 con 0x55bd3b3fd300 2024-05-10T11:14:11.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8817508b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f88081ae1f0 con 0x55bd3b3f7600 2024-05-10T11:14:11.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8817508b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f88081ab3f0 con 0x55bd3b3fc9e0 2024-05-10T11:14:11.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8817508b80 10 client.0 _ll_register_callbacks cb 0x55bd3b2f0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:11.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8805400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b41ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8805e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3f7600 0x55bd3b41e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8805400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b41ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:58596/0 (socket says 172.21.15.27:58596) 2024-05-10T11:14:11.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8805400700 1 -- 172.21.15.27:0/828758671 learned_addr learned my addr 172.21.15.27:0/828758671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:11.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8806800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b41bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8805e00700 1 -- 172.21.15.27:0/828758671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3fc9e0 msgr2=0x55bd3b41ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:11.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8805e00700 1 --2- 172.21.15.27:0/828758671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b41ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8805e00700 1 -- 172.21.15.27:0/828758671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 msgr2=0x55bd3b41bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:11.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8805e00700 1 --2- 172.21.15.27:0/828758671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b41bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.655+0000 7f8805e00700 1 -- 172.21.15.27:0/828758671 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd3b41c7f0 con 0x55bd3b3f7600 2024-05-10T11:14:11.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.656+0000 7f8805e00700 1 --2- 172.21.15.27:0/828758671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3f7600 0x55bd3b41e6d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55bd3b403f80 tx=0x7f87f800be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1cafd68245d31ce server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:11.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.656+0000 7f87e7e00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:14:11.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.656+0000 7f87e7e00700 1 -- 172.21.15.27:0/828758671 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f802e030 con 0x55bd3b3f7600 2024-05-10T11:14:11.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.656+0000 7f8817508b80 1 -- 172.21.15.27:0/828758671 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87f40a9580 con 0x55bd3b3f7600 2024-05-10T11:14:11.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.656+0000 7f87e7e00700 1 -- 172.21.15.27:0/828758671 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f87f80052c0 con 0x55bd3b3f7600 2024-05-10T11:14:11.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.657+0000 7f87e7e00700 1 -- 172.21.15.27:0/828758671 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f87f803b050 con 0x55bd3b3f7600 2024-05-10T11:14:11.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.657+0000 7f8817508b80 10 client.25442 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:11.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.657+0000 7f8817508b80 1 -- 172.21.15.27:0/828758671 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55bd3b4836b0 con 0x55bd3b3f7600 2024-05-10T11:14:11.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.657+0000 7f87e7e00700 1 -- 172.21.15.27:0/828758671 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87f40a9580 con 0x55bd3b3f7600 2024-05-10T11:14:11.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.657+0000 7f87e7e00700 1 -- 172.21.15.27:0/828758671 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f87f802a130 con 0x55bd3b3f7600 2024-05-10T11:14:11.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.657+0000 7f8817508b80 10 client.25442 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:11.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.657+0000 7f8817508b80 10 client.25442 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:14:11.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.657+0000 7f8817508b80 20 client.25442 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:11.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.657+0000 7f8817508b80 1 --2- 172.21.15.27:0/828758671 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55bd3b483d40 0x55bd3b486160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:11.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.657+0000 7f8817508b80 4 client.25442 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:14:11.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.658+0000 7f8817508b80 1 -- 172.21.15.27:0/828758671 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bd3b3514a0 con 0x55bd3b483d40 2024-05-10T11:14:11.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.658+0000 7f8805400700 1 --2- 172.21.15.27:0/828758671 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55bd3b483d40 0x55bd3b486160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:11.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.658+0000 7f8805400700 1 --2- 172.21.15.27:0/828758671 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55bd3b483d40 0x55bd3b486160 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1e8368721098b4a server_cookie=f71d8cb70d902183 in_seq=0 out_seq=0 2024-05-10T11:14:11.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.658+0000 7f87e7e00700 10 client.25442 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:11.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f87e7e00700 1 -- 172.21.15.27:0/828758671 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bd3b3514a0 con 0x55bd3b483d40 2024-05-10T11:14:11.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f87e7e00700 10 client.25442 handle_command_reply: tid=0 2024-05-10T11:14:11.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f8817508b80 1 client.25442 shutdown 2024-05-10T11:14:11.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f8817508b80 1 -- 172.21.15.27:0/828758671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3f7600 msgr2=0x55bd3b41e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:11.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f8817508b80 1 --2- 172.21.15.27:0/828758671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3f7600 0x55bd3b41e6d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55bd3b403f80 tx=0x7f87f800be60 comp rx=0 tx=0).stop 2024-05-10T11:14:11.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f8817508b80 1 -- 172.21.15.27:0/828758671 shutdown_connections 2024-05-10T11:14:11.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f8817508b80 1 --2- 172.21.15.27:0/828758671 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55bd3b483d40 0x55bd3b486160 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f8817508b80 1 --2- 172.21.15.27:0/828758671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd3b3fd300 0x55bd3b41bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f8817508b80 1 --2- 172.21.15.27:0/828758671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd3b3f7600 0x55bd3b41e6d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f8817508b80 1 --2- 172.21.15.27:0/828758671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd3b3fc9e0 0x55bd3b41ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f87e7e00700 0 client.25442 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:11.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f8817508b80 1 -- 172.21.15.27:0/828758671 >> 172.21.15.27:0/828758671 conn(0x55bd3b325b60 msgr2=0x55bd3b3ec460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:11.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f8817508b80 1 -- 172.21.15.27:0/828758671 shutdown_connections 2024-05-10T11:14:11.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f8817508b80 1 -- 172.21.15.27:0/828758671 wait complete. 2024-05-10T11:14:11.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.659+0000 7f8817508b80 20 client.25442 trim_cache size 0 max 16384 2024-05-10T11:14:11.663 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:14:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:14:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:14:11.663 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:14:11.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.660+0000 7f8807200700 1 -- 172.21.15.27:0/1280163456 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f87ec0b2ba0 msgr2=0x7f87ec0b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:11.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.660+0000 7f8807200700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f87ec0b2ba0 0x7f87ec0b5060 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8800012710 tx=0x7f880000b040 comp rx=0 tx=0).stop 2024-05-10T11:14:11.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.660+0000 7f8807200700 1 -- 172.21.15.27:0/1280163456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8808103d90 msgr2=0x7f88081a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:11.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.660+0000 7f8807200700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8808103d90 0x7f88081a1e60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f87f400e9f0 tx=0x7f87f400ed00 comp rx=0 tx=0).stop 2024-05-10T11:14:11.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.660+0000 7f8807200700 1 -- 172.21.15.27:0/1280163456 shutdown_connections 2024-05-10T11:14:11.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.660+0000 7f8807200700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f87ec0b2ba0 0x7f87ec0b5060 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.660+0000 7f8807200700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8808109460 0x7f880819bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.660+0000 7f8807200700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88081033d0 0x7f88081a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.660+0000 7f8807200700 1 --2- 172.21.15.27:0/1280163456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8808103d90 0x7f88081a1e60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:11.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.660+0000 7f8807200700 1 -- 172.21.15.27:0/1280163456 >> 172.21.15.27:0/1280163456 conn(0x7f88080fef20 msgr2=0x7f880810eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:11.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.661+0000 7f8807200700 1 -- 172.21.15.27:0/1280163456 shutdown_connections 2024-05-10T11:14:11.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:11.661+0000 7f8807200700 1 -- 172.21.15.27:0/1280163456 wait complete. 2024-05-10T11:14:12.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:12 smithi027 ceph-mon[30054]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-05-10T11:14:12.624 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:14:12.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:14:12.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:12 smithi076 ceph-mon[34152]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-05-10T11:14:12.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:12 smithi189 ceph-mon[34842]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-05-10T11:14:13.077 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:14:14.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:14 smithi027 ceph-mon[30054]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-10T11:14:14.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:14 smithi076 ceph-mon[34152]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-10T11:14:14.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:14 smithi189 ceph-mon[34842]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-10T11:14:15.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.436+0000 7f9e0a600700 1 -- 172.21.15.27:0/2238092282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e04105f90 msgr2=0x7f9e041124a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:15.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.436+0000 7f9e0a600700 1 --2- 172.21.15.27:0/2238092282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e04105f90 0x7f9e041124a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc009f90 tx=0x7f9dfc00b330 comp rx=0 tx=0).stop 2024-05-10T11:14:15.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.436+0000 7f9e0a600700 1 -- 172.21.15.27:0/2238092282 shutdown_connections 2024-05-10T11:14:15.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.436+0000 7f9e0a600700 1 --2- 172.21.15.27:0/2238092282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e04105f90 0x7f9e041124a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:15.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.436+0000 7f9e0a600700 1 --2- 172.21.15.27:0/2238092282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0410b660 0x7f9e0410ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:15.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.436+0000 7f9e0a600700 1 --2- 172.21.15.27:0/2238092282 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e041055d0 0x7f9e04105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:15.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.436+0000 7f9e0a600700 1 -- 172.21.15.27:0/2238092282 >> 172.21.15.27:0/2238092282 conn(0x7f9e041010c0 msgr2=0x7f9e041034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:15.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.437+0000 7f9e0a600700 1 -- 172.21.15.27:0/2238092282 shutdown_connections 2024-05-10T11:14:15.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.437+0000 7f9e0a600700 1 -- 172.21.15.27:0/2238092282 wait complete. 2024-05-10T11:14:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.438+0000 7f9e0a600700 1 Processor -- start 2024-05-10T11:14:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.438+0000 7f9e0a600700 1 -- start start 2024-05-10T11:14:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.439+0000 7f9e0a600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e041055d0 0x7f9e041a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.439+0000 7f9e0a600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e04105f90 0x7f9e041a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.439+0000 7f9e0a600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0410b660 0x7f9e0419ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.439+0000 7f9e0a600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9e04115800 con 0x7f9e0410b660 2024-05-10T11:14:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.439+0000 7f9e0a600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9e04115680 con 0x7f9e041055d0 2024-05-10T11:14:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.439+0000 7f9e0a600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e04115980 con 0x7f9e04105f90 2024-05-10T11:14:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.439+0000 7f9e03e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e04105f90 0x7f9e041a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.439+0000 7f9e09200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e041055d0 0x7f9e041a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:15.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.439+0000 7f9e03e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e04105f90 0x7f9e041a1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:47044/0 (socket says 172.21.15.27:47044) 2024-05-10T11:14:15.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.439+0000 7f9e03e00700 1 -- 172.21.15.27:0/567693016 learned_addr learned my addr 172.21.15.27:0/567693016 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:15.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.439+0000 7f9e09c00700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0410b660 0x7f9e0419ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.440+0000 7f9e09200700 1 -- 172.21.15.27:0/567693016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e04105f90 msgr2=0x7f9e041a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.440+0000 7f9e09200700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e04105f90 0x7f9e041a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.440+0000 7f9e09200700 1 -- 172.21.15.27:0/567693016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0410b660 msgr2=0x7f9e0419ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.440+0000 7f9e09200700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0410b660 0x7f9e0419ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.440+0000 7f9e09200700 1 -- 172.21.15.27:0/567693016 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0419c180 con 0x7f9e041055d0 2024-05-10T11:14:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.440+0000 7f9e03e00700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e04105f90 0x7f9e041a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.440+0000 7f9e09200700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e041055d0 0x7f9e041a1810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9df800e9c0 tx=0x7f9df800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.440+0000 7f9e01600700 1 -- 172.21.15.27:0/567693016 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df8004220 con 0x7f9e041055d0 2024-05-10T11:14:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.440+0000 7f9e01600700 1 -- 172.21.15.27:0/567693016 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9df80043c0 con 0x7f9e041055d0 2024-05-10T11:14:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.440+0000 7f9e0a600700 1 -- 172.21.15.27:0/567693016 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e0419c470 con 0x7f9e041055d0 2024-05-10T11:14:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.440+0000 7f9e01600700 1 -- 172.21.15.27:0/567693016 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df8003c50 con 0x7f9e041055d0 2024-05-10T11:14:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.441+0000 7f9e0a600700 1 -- 172.21.15.27:0/567693016 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e041a86e0 con 0x7f9e041055d0 2024-05-10T11:14:15.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.442+0000 7f9e0a600700 1 -- 172.21.15.27:0/567693016 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e0406a190 con 0x7f9e041055d0 2024-05-10T11:14:15.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.444+0000 7f9e01600700 1 -- 172.21.15.27:0/567693016 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f9df800c7c0 con 0x7f9e041055d0 2024-05-10T11:14:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.445+0000 7f9e01600700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9dec0b2c20 0x7f9dec0b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.446+0000 7f9e01600700 1 -- 172.21.15.27:0/567693016 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f9df80e2e10 con 0x7f9e041055d0 2024-05-10T11:14:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.446+0000 7f9e03e00700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9dec0b2c20 0x7f9dec0b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.447+0000 7f9e03e00700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9dec0b2c20 0x7f9dec0b50e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9e0410ac30 tx=0x7f9df0009310 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.450+0000 7f9e01600700 1 -- 172.21.15.27:0/567693016 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9df80e6030 con 0x7f9e041055d0 2024-05-10T11:14:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.745+0000 7f9e0a600700 1 -- 172.21.15.27:0/567693016 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e0419cf00 con 0x7f9e041055d0 2024-05-10T11:14:15.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.746+0000 7f9e01600700 1 -- 172.21.15.27:0/567693016 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11936 (secure 0 0 0) 0x7f9df80aa1e0 con 0x7f9e041055d0 2024-05-10T11:14:15.746 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:14:15.747 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:13:10.916396+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24484,24472,24467,24488]},"id":1}]} 2024-05-10T11:14:15.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.749+0000 7f9e0a600700 1 -- 172.21.15.27:0/567693016 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9dec0b2c20 msgr2=0x7f9dec0b50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:15.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.749+0000 7f9e0a600700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9dec0b2c20 0x7f9dec0b50e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9e0410ac30 tx=0x7f9df0009310 comp rx=0 tx=0).stop 2024-05-10T11:14:15.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.749+0000 7f9e0a600700 1 -- 172.21.15.27:0/567693016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e041055d0 msgr2=0x7f9e041a1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:15.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.749+0000 7f9e0a600700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e041055d0 0x7f9e041a1810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9df800e9c0 tx=0x7f9df800ecd0 comp rx=0 tx=0).stop 2024-05-10T11:14:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.750+0000 7f9e0a600700 1 -- 172.21.15.27:0/567693016 shutdown_connections 2024-05-10T11:14:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.750+0000 7f9e0a600700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9dec0b2c20 0x7f9dec0b50e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.750+0000 7f9e0a600700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0410b660 0x7f9e0419ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.750+0000 7f9e0a600700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e041055d0 0x7f9e041a1810 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.750+0000 7f9e0a600700 1 --2- 172.21.15.27:0/567693016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e04105f90 0x7f9e041a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.750+0000 7f9e0a600700 1 -- 172.21.15.27:0/567693016 >> 172.21.15.27:0/567693016 conn(0x7f9e041010c0 msgr2=0x7f9e04102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:15.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.750+0000 7f9e0a600700 1 -- 172.21.15.27:0/567693016 shutdown_connections 2024-05-10T11:14:15.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:15.750+0000 7f9e0a600700 1 -- 172.21.15.27:0/567693016 wait complete. 2024-05-10T11:14:15.751 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 30 2024-05-10T11:14:16.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:16 smithi027 ceph-mon[30054]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-05-10T11:14:16.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:16 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/567693016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:14:16.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:14:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:16 smithi076 ceph-mon[34152]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-05-10T11:14:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:16 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/567693016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:14:16.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:16 smithi189 ceph-mon[34842]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-05-10T11:14:16.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:16 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/567693016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:14:17.140 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:14:18.599 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:18 smithi027 ceph-mon[30054]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-05-10T11:14:18.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:18 smithi076 ceph-mon[34152]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-05-10T11:14:18.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:18 smithi189 ceph-mon[34842]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-05-10T11:14:19.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.484+0000 7f0a62200700 1 -- 172.21.15.27:0/1797458852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a5c108700 msgr2=0x7f0a5c1102d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:19.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.484+0000 7f0a62200700 1 --2- 172.21.15.27:0/1797458852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a5c108700 0x7f0a5c1102d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400b0d0 tx=0x7f0a5400b3e0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.484+0000 7f0a62200700 1 -- 172.21.15.27:0/1797458852 shutdown_connections 2024-05-10T11:14:19.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.484+0000 7f0a62200700 1 --2- 172.21.15.27:0/1797458852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a5c108700 0x7f0a5c1102d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.484+0000 7f0a62200700 1 --2- 172.21.15.27:0/1797458852 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a5c103620 0x7f0a5c103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.484+0000 7f0a62200700 1 --2- 172.21.15.27:0/1797458852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a5c103fd0 0x7f0a5c108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.484+0000 7f0a62200700 1 -- 172.21.15.27:0/1797458852 >> 172.21.15.27:0/1797458852 conn(0x7f0a5c0feef0 msgr2=0x7f0a5c101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:19.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.484+0000 7f0a62200700 1 -- 172.21.15.27:0/1797458852 shutdown_connections 2024-05-10T11:14:19.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.485+0000 7f0a62200700 1 -- 172.21.15.27:0/1797458852 wait complete. 2024-05-10T11:14:19.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:19 smithi027 ceph-mon[30054]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 12 op/s 2024-05-10T11:14:19.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.486+0000 7f0a62200700 1 Processor -- start 2024-05-10T11:14:19.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.486+0000 7f0a62200700 1 -- start start 2024-05-10T11:14:19.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.486+0000 7f0a62200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a5c103620 0x7f0a5c19bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.486+0000 7f0a62200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a5c103fd0 0x7f0a5c19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.486+0000 7f0a62200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a5c108700 0x7f0a5c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.486+0000 7f0a62200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c113610 con 0x7f0a5c108700 2024-05-10T11:14:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.486+0000 7f0a62200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c113490 con 0x7f0a5c103620 2024-05-10T11:14:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.486+0000 7f0a62200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c113790 con 0x7f0a5c103fd0 2024-05-10T11:14:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.487+0000 7f0a60e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a5c103620 0x7f0a5c19bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.487+0000 7f0a5be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a5c103fd0 0x7f0a5c19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.487+0000 7f0a60e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a5c103620 0x7f0a5c19bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44716/0 (socket says 172.21.15.27:44716) 2024-05-10T11:14:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.487+0000 7f0a60e00700 1 -- 172.21.15.27:0/1569721608 learned_addr learned my addr 172.21.15.27:0/1569721608 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.487+0000 7f0a61800700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a5c108700 0x7f0a5c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.487+0000 7f0a5be00700 1 -- 172.21.15.27:0/1569721608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a5c103620 msgr2=0x7f0a5c19bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.487+0000 7f0a5be00700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a5c103620 0x7f0a5c19bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.487+0000 7f0a5be00700 1 -- 172.21.15.27:0/1569721608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a5c108700 msgr2=0x7f0a5c1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.487+0000 7f0a5be00700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a5c108700 0x7f0a5c1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.487+0000 7f0a5be00700 1 -- 172.21.15.27:0/1569721608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5c1a0d90 con 0x7f0a5c103fd0 2024-05-10T11:14:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.488+0000 7f0a5be00700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a5c103fd0 0x7f0a5c19c2c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800e9c0 tx=0x7f0a4800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.488+0000 7f0a59600700 1 -- 172.21.15.27:0/1569721608 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48004220 con 0x7f0a5c103fd0 2024-05-10T11:14:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.488+0000 7f0a59600700 1 -- 172.21.15.27:0/1569721608 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a480043c0 con 0x7f0a5c103fd0 2024-05-10T11:14:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.488+0000 7f0a62200700 1 -- 172.21.15.27:0/1569721608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a5c1a1080 con 0x7f0a5c103fd0 2024-05-10T11:14:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.488+0000 7f0a59600700 1 -- 172.21.15.27:0/1569721608 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48005020 con 0x7f0a5c103fd0 2024-05-10T11:14:19.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.488+0000 7f0a62200700 1 -- 172.21.15.27:0/1569721608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a5c1a8900 con 0x7f0a5c103fd0 2024-05-10T11:14:19.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.494+0000 7f0a59600700 1 -- 172.21.15.27:0/1569721608 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f0a48004750 con 0x7f0a5c103fd0 2024-05-10T11:14:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.494+0000 7f0a6e1c3b80 1 Processor -- start 2024-05-10T11:14:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.495+0000 7f0a6e1c3b80 1 -- start start 2024-05-10T11:14:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.495+0000 7f0a6e1c3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f69e0 0x557fb51f6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.495+0000 7f0a6e1c3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f7300 0x557fb51f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.495+0000 7f0a6e1c3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 0x557fb51f1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.495+0000 7f0a6e1c3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557fb5143a20 con 0x557fb51f1600 2024-05-10T11:14:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.495+0000 7f0a6e1c3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557fb50e1760 con 0x557fb51f7300 2024-05-10T11:14:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.495+0000 7f0a6e1c3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557fb513f9e0 con 0x557fb51f69e0 2024-05-10T11:14:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a60e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f7300 0x557fb51f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a60e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f7300 0x557fb51f0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44732/0 (socket says 172.21.15.27:44732) 2024-05-10T11:14:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a61800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f69e0 0x557fb51f6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a60e00700 1 -- 172.21.15.27:0/982866043 learned_addr learned my addr 172.21.15.27:0/982866043 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a5be00700 1 --2- 172.21.15.27:0/982866043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 0x557fb51f1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a59600700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0a440b2c20 0x7f0a440b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a61800700 1 -- 172.21.15.27:0/982866043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f7300 msgr2=0x557fb51f0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a59600700 1 -- 172.21.15.27:0/1569721608 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f0a480e2b20 con 0x7f0a5c103fd0 2024-05-10T11:14:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a61800700 1 --2- 172.21.15.27:0/982866043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f7300 0x557fb51f0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a61800700 1 -- 172.21.15.27:0/982866043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 msgr2=0x557fb51f1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a61800700 1 --2- 172.21.15.27:0/982866043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 0x557fb51f1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a61800700 1 -- 172.21.15.27:0/982866043 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fb50908d0 con 0x557fb51f69e0 2024-05-10T11:14:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a60e00700 1 --2- 172.21.15.27:0/982866043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f7300 0x557fb51f0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.496+0000 7f0a5be00700 1 --2- 172.21.15.27:0/982866043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 0x557fb51f1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:14:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.497+0000 7f0a61800700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0a440b2c20 0x7f0a440b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.497+0000 7f0a61800700 1 --2- 172.21.15.27:0/982866043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f69e0 0x557fb51f6dc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400b770 tx=0x7f0a54009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcfb57753e6b3781 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:19.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.497+0000 7f0a4e000700 1 -- 172.21.15.27:0/982866043 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5402d070 con 0x557fb51f69e0 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.497+0000 7f0a4e000700 1 -- 172.21.15.27:0/982866043 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a54020070 con 0x557fb51f69e0 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.497+0000 7f0a4e000700 1 -- 172.21.15.27:0/982866043 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a540058b0 con 0x557fb51f69e0 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.498+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/982866043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f69e0 msgr2=0x557fb51f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.498+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/982866043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f69e0 0x557fb51f6dc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400b770 tx=0x7f0a54009b80 comp rx=0 tx=0).stop 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.498+0000 7f0a61800700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0a440b2c20 0x7f0a440b50e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0a5402a000 tx=0x7f0a540075c0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.498+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/982866043 shutdown_connections 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.498+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/982866043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f69e0 0x557fb51f6dc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.498+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/982866043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 0x557fb51f1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.498+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/982866043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f7300 0x557fb51f0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.498+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/982866043 >> 172.21.15.27:0/982866043 conn(0x557fb511fb60 msgr2=0x557fb51eb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.498+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/982866043 shutdown_connections 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.498+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/982866043 wait complete. 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.499+0000 7f0a6e1c3b80 1 -- start start 2024-05-10T11:14:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.499+0000 7f0a6e1c3b80 1 Processor -- start 2024-05-10T11:14:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.500+0000 7f0a6e1c3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f1600 0x557fb51fceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.500+0000 7f0a6e1c3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f69e0 0x557fb51fa250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.500+0000 7f0a6e1c3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f7300 0x557fb51fa790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.500+0000 7f0a6e1c3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557fb51fddb0 con 0x557fb51f7300 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.500+0000 7f0a6e1c3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557fb50a9b20 con 0x557fb51f69e0 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.500+0000 7f0a6e1c3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557fb51fdf30 con 0x557fb51f1600 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.500+0000 7f0a6e1c3b80 10 client.0 _ll_register_callbacks cb 0x557fb50eac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.500+0000 7f0a5be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f69e0 0x557fb51fa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.500+0000 7f0a5be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f69e0 0x557fb51fa250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44736/0 (socket says 172.21.15.27:44736) 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.500+0000 7f0a5be00700 1 -- 172.21.15.27:0/730073703 learned_addr learned my addr 172.21.15.27:0/730073703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.500+0000 7f0a61800700 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f7300 0x557fb51fa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.500+0000 7f0a60e00700 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f1600 0x557fb51fceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.501+0000 7f0a5be00700 1 -- 172.21.15.27:0/730073703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f1600 msgr2=0x557fb51fceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.501+0000 7f0a5be00700 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f1600 0x557fb51fceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.501+0000 7f0a5be00700 1 -- 172.21.15.27:0/730073703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f7300 msgr2=0x557fb51fa790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.501+0000 7f0a5be00700 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f7300 0x557fb51fa790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.501+0000 7f0a5be00700 1 -- 172.21.15.27:0/730073703 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fb5109420 con 0x557fb51f69e0 2024-05-10T11:14:19.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.501+0000 7f0a61800700 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f7300 0x557fb51fa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:14:19.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.501+0000 7f0a5be00700 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f69e0 0x557fb51fa250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0a480dfda0 tx=0x7f0a480a9d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10a809d9cd9db9c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:19.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.501+0000 7f0a42a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:14:19.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.501+0000 7f0a42a00700 1 -- 172.21.15.27:0/730073703 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4801c070 con 0x557fb51f69e0 2024-05-10T11:14:19.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/730073703 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a54020070 con 0x557fb51f69e0 2024-05-10T11:14:19.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a42a00700 1 -- 172.21.15.27:0/730073703 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f0a480cc0f0 con 0x557fb51f69e0 2024-05-10T11:14:19.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a42a00700 1 -- 172.21.15.27:0/730073703 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0a480a8f60 con 0x557fb51f69e0 2024-05-10T11:14:19.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a6e1c3b80 10 client.25460 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:19.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/730073703 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557fb5208540 con 0x557fb51f69e0 2024-05-10T11:14:19.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a42a00700 1 -- 172.21.15.27:0/730073703 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a54020070 con 0x557fb51f69e0 2024-05-10T11:14:19.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a42a00700 1 -- 172.21.15.27:0/730073703 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0a480aab20 con 0x557fb51f69e0 2024-05-10T11:14:19.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a6e1c3b80 10 client.25460 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:19.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a6e1c3b80 10 client.25460 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:14:19.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a6e1c3b80 20 client.25460 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:19.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x557fb520aa10 0x557fb520ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a6e1c3b80 4 client.25460 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:14:19.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.502+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/730073703 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fb51be0a0 con 0x557fb520aa10 2024-05-10T11:14:19.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.503+0000 7f0a60e00700 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x557fb520aa10 0x557fb520ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.503+0000 7f0a60e00700 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x557fb520aa10 0x557fb520ce00 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f106849001bbcafe server_cookie=ade4b016ca396b36 in_seq=0 out_seq=0 2024-05-10T11:14:19.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.503+0000 7f0a42a00700 10 client.25460 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:14:19.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.504+0000 7f0a42a00700 1 -- 172.21.15.27:0/730073703 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557fb51be0a0 con 0x557fb520aa10 2024-05-10T11:14:19.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.504+0000 7f0a42a00700 10 client.25460 handle_command_reply: tid=0 2024-05-10T11:14:19.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.504+0000 7f0a6e1c3b80 1 client.25460 shutdown 2024-05-10T11:14:19.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.505+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/730073703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f69e0 msgr2=0x557fb51fa250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:19.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.505+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f69e0 0x557fb51fa250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0a480dfda0 tx=0x7f0a480a9d30 comp rx=0 tx=0).stop 2024-05-10T11:14:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.505+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/730073703 shutdown_connections 2024-05-10T11:14:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.505+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x557fb520aa10 0x557fb520ce00 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.505+0000 7f0a42a00700 0 client.25460 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:14:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.505+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f1600 0x557fb51fceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.505+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f7300 0x557fb51fa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.505+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/730073703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f69e0 0x557fb51fa250 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.505+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/730073703 >> 172.21.15.27:0/730073703 conn(0x557fb511fb60 msgr2=0x557fb51e6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.505+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/730073703 shutdown_connections 2024-05-10T11:14:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.505+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/730073703 wait complete. 2024-05-10T11:14:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.505+0000 7f0a6e1c3b80 20 client.25460 trim_cache size 0 max 16384 2024-05-10T11:14:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.522+0000 7f0a6e1c3b80 1 Processor -- start 2024-05-10T11:14:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.522+0000 7f0a6e1c3b80 1 -- start start 2024-05-10T11:14:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.522+0000 7f0a6e1c3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 0x557fb51f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.522+0000 7f0a6e1c3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f69e0 0x557fb5212630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.522+0000 7f0a6e1c3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f7300 0x557fb5212b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.522+0000 7f0a6e1c3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c1b1630 con 0x557fb51f1600 2024-05-10T11:14:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.522+0000 7f0a6e1c3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c1b3170 con 0x557fb51f69e0 2024-05-10T11:14:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.522+0000 7f0a6e1c3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c1b1d00 con 0x557fb51f7300 2024-05-10T11:14:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.522+0000 7f0a60e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 0x557fb51f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.522+0000 7f0a60e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 0x557fb51f0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39282/0 (socket says 172.21.15.27:39282) 2024-05-10T11:14:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.522+0000 7f0a60e00700 1 -- 172.21.15.27:0/3920941530 learned_addr learned my addr 172.21.15.27:0/3920941530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.522+0000 7f0a5be00700 1 --2- 172.21.15.27:0/3920941530 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f69e0 0x557fb5212630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.523+0000 7f0a61800700 1 --2- 172.21.15.27:0/3920941530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f7300 0x557fb5212b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.523+0000 7f0a60e00700 1 -- 172.21.15.27:0/3920941530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f7300 msgr2=0x557fb5212b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.523+0000 7f0a60e00700 1 --2- 172.21.15.27:0/3920941530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f7300 0x557fb5212b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.523+0000 7f0a60e00700 1 -- 172.21.15.27:0/3920941530 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f69e0 msgr2=0x557fb5212630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.523+0000 7f0a60e00700 1 --2- 172.21.15.27:0/3920941530 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f69e0 0x557fb5212630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.523+0000 7f0a60e00700 1 -- 172.21.15.27:0/3920941530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fb51f87c0 con 0x557fb51f1600 2024-05-10T11:14:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.523+0000 7f0a60e00700 1 --2- 172.21.15.27:0/3920941530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 0x557fb51f0fe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x557fb520d5d0 tx=0x7f0a50009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8446390eab813cea server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.524+0000 7f0a4d600700 1 -- 172.21.15.27:0/3920941530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50026070 con 0x557fb51f1600 2024-05-10T11:14:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.524+0000 7f0a4d600700 1 -- 172.21.15.27:0/3920941530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a50004500 con 0x557fb51f1600 2024-05-10T11:14:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.524+0000 7f0a4d600700 1 -- 172.21.15.27:0/3920941530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50020370 con 0x557fb51f1600 2024-05-10T11:14:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.524+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/3920941530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 msgr2=0x557fb51f0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.524+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/3920941530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 0x557fb51f0fe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x557fb520d5d0 tx=0x7f0a50009260 comp rx=0 tx=0).stop 2024-05-10T11:14:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.524+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/3920941530 shutdown_connections 2024-05-10T11:14:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.524+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/3920941530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f7300 0x557fb5212b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.524+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/3920941530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f1600 0x557fb51f0fe0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.524+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/3920941530 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f69e0 0x557fb5212630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.524+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/3920941530 >> 172.21.15.27:0/3920941530 conn(0x557fb511fb60 msgr2=0x557fb51ed260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.525+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/3920941530 shutdown_connections 2024-05-10T11:14:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.525+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/3920941530 wait complete. 2024-05-10T11:14:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.525+0000 7f0a6e1c3b80 1 -- start start 2024-05-10T11:14:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.525+0000 7f0a6e1c3b80 1 Processor -- start 2024-05-10T11:14:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.526+0000 7f0a6e1c3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f1600 0x557fb5212080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.526+0000 7f0a6e1c3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f69e0 0x557fb52125c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.526+0000 7f0a6e1c3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f7300 0x557fb5212b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.526+0000 7f0a6e1c3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c1b2aa0 con 0x557fb51f7300 2024-05-10T11:14:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.526+0000 7f0a6e1c3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c1b23d0 con 0x557fb51f1600 2024-05-10T11:14:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.526+0000 7f0a6e1c3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c1af5d0 con 0x557fb51f69e0 2024-05-10T11:14:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.526+0000 7f0a6e1c3b80 10 client.0 _ll_register_callbacks cb 0x557fb50eac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.526+0000 7f0a60e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f1600 0x557fb5212080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.526+0000 7f0a60e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f1600 0x557fb5212080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44746/0 (socket says 172.21.15.27:44746) 2024-05-10T11:14:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.526+0000 7f0a60e00700 1 -- 172.21.15.27:0/2150136798 learned_addr learned my addr 172.21.15.27:0/2150136798 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.526+0000 7f0a61800700 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f7300 0x557fb5212b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.526+0000 7f0a5be00700 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f69e0 0x557fb52125c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.527+0000 7f0a60e00700 1 -- 172.21.15.27:0/2150136798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f69e0 msgr2=0x557fb52125c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.527+0000 7f0a60e00700 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f69e0 0x557fb52125c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.527+0000 7f0a60e00700 1 -- 172.21.15.27:0/2150136798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f7300 msgr2=0x557fb5212b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.527+0000 7f0a60e00700 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f7300 0x557fb5212b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.527+0000 7f0a60e00700 1 -- 172.21.15.27:0/2150136798 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fb5274440 con 0x557fb51f1600 2024-05-10T11:14:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.527+0000 7f0a61800700 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f7300 0x557fb5212b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.527+0000 7f0a60e00700 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f1600 0x557fb5212080 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0a50003c90 tx=0x7f0a500123a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ed85bea8194e8c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.527+0000 7f0a42a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:14:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.527+0000 7f0a42a00700 1 -- 172.21.15.27:0/2150136798 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50030060 con 0x557fb51f1600 2024-05-10T11:14:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.527+0000 7f0a42a00700 1 -- 172.21.15.27:0/2150136798 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f0a50004250 con 0x557fb51f1600 2024-05-10T11:14:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.527+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/2150136798 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a50004500 con 0x557fb51f1600 2024-05-10T11:14:19.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.528+0000 7f0a42a00700 1 -- 172.21.15.27:0/2150136798 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0a50037050 con 0x557fb51f1600 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.528+0000 7f0a6e1c3b80 10 client.25466 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.528+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/2150136798 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557fb5275720 con 0x557fb51f1600 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.528+0000 7f0a42a00700 1 -- 172.21.15.27:0/2150136798 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a50004500 con 0x557fb51f1600 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.528+0000 7f0a42a00700 1 -- 172.21.15.27:0/2150136798 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0a50005030 con 0x557fb51f1600 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.529+0000 7f0a6e1c3b80 10 client.25466 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.529+0000 7f0a6e1c3b80 10 client.25466 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.529+0000 7f0a6e1c3b80 20 client.25466 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.529+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x557fb5275cf0 0x557fb5278110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.529+0000 7f0a6e1c3b80 4 client.25466 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.529+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/2150136798 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557fb5151400 con 0x557fb5275cf0 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.529+0000 7f0a5be00700 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x557fb5275cf0 0x557fb5278110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a5be00700 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x557fb5275cf0 0x557fb5278110 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c281bd711d74edab server_cookie=13dd76bd91c7a14e in_seq=0 out_seq=0 2024-05-10T11:14:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a42a00700 10 client.25466 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:14:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a42a00700 1 -- 172.21.15.27:0/2150136798 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557fb5151400 con 0x557fb5275cf0 2024-05-10T11:14:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a42a00700 10 client.25466 handle_command_reply: tid=0 2024-05-10T11:14:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a6e1c3b80 1 client.25466 shutdown 2024-05-10T11:14:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/2150136798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f1600 msgr2=0x557fb5212080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f1600 0x557fb5212080 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0a50003c90 tx=0x7f0a500123a0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/2150136798 shutdown_connections 2024-05-10T11:14:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fb51f7300 0x557fb5212b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557fb51f1600 0x557fb5212080 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x557fb5275cf0 0x557fb5278110 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a6e1c3b80 1 --2- 172.21.15.27:0/2150136798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fb51f69e0 0x557fb52125c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/2150136798 >> 172.21.15.27:0/2150136798 conn(0x557fb511fb60 msgr2=0x557fb51f76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.530+0000 7f0a42a00700 0 client.25466 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:14:19.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.531+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/2150136798 shutdown_connections 2024-05-10T11:14:19.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.531+0000 7f0a6e1c3b80 1 -- 172.21.15.27:0/2150136798 wait complete. 2024-05-10T11:14:19.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.531+0000 7f0a6e1c3b80 20 client.25466 trim_cache size 0 max 16384 2024-05-10T11:14:19.532 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:14:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.532+0000 7f0a62200700 1 -- 172.21.15.27:0/1569721608 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0a440b2c20 msgr2=0x7f0a440b50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.532+0000 7f0a62200700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0a440b2c20 0x7f0a440b50e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0a5402a000 tx=0x7f0a540075c0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.532+0000 7f0a62200700 1 -- 172.21.15.27:0/1569721608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a5c103fd0 msgr2=0x7f0a5c19c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.532+0000 7f0a62200700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a5c103fd0 0x7f0a5c19c2c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800e9c0 tx=0x7f0a4800ecd0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.532+0000 7f0a62200700 1 -- 172.21.15.27:0/1569721608 shutdown_connections 2024-05-10T11:14:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.532+0000 7f0a62200700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0a440b2c20 0x7f0a440b50e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.532+0000 7f0a62200700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a5c108700 0x7f0a5c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.532+0000 7f0a62200700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a5c103620 0x7f0a5c19bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.532+0000 7f0a62200700 1 --2- 172.21.15.27:0/1569721608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a5c103fd0 0x7f0a5c19c2c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:19.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.532+0000 7f0a62200700 1 -- 172.21.15.27:0/1569721608 >> 172.21.15.27:0/1569721608 conn(0x7f0a5c0feef0 msgr2=0x7f0a5c10eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:19.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.532+0000 7f0a62200700 1 -- 172.21.15.27:0/1569721608 shutdown_connections 2024-05-10T11:14:19.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:19.532+0000 7f0a62200700 1 -- 172.21.15.27:0/1569721608 wait complete. 2024-05-10T11:14:19.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:19 smithi076 ceph-mon[34152]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 12 op/s 2024-05-10T11:14:19.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:19 smithi189 ceph-mon[34842]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 12 op/s 2024-05-10T11:14:20.518 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:14:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:14:22.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:22 smithi027 ceph-mon[30054]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 852 B/s wr, 9 op/s 2024-05-10T11:14:22.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:22 smithi076 ceph-mon[34152]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 852 B/s wr, 9 op/s 2024-05-10T11:14:22.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:22 smithi189 ceph-mon[34842]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 852 B/s wr, 9 op/s 2024-05-10T11:14:23.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.342+0000 7fca5ac00700 1 -- 172.21.15.27:0/3242509897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca54102b30 msgr2=0x7fca5410f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:23.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.342+0000 7fca5ac00700 1 --2- 172.21.15.27:0/3242509897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca54102b30 0x7fca5410f7f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fca40009990 tx=0x7fca40009ca0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.342+0000 7fca5ac00700 1 -- 172.21.15.27:0/3242509897 shutdown_connections 2024-05-10T11:14:23.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.342+0000 7fca5ac00700 1 --2- 172.21.15.27:0/3242509897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca5410fd30 0x7fca54112120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.342+0000 7fca5ac00700 1 --2- 172.21.15.27:0/3242509897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca54102180 0x7fca54102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.342+0000 7fca5ac00700 1 --2- 172.21.15.27:0/3242509897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca54102b30 0x7fca5410f7f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.342+0000 7fca5ac00700 1 -- 172.21.15.27:0/3242509897 >> 172.21.15.27:0/3242509897 conn(0x7fca540fddc0 msgr2=0x7fca541001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.343+0000 7fca5ac00700 1 -- 172.21.15.27:0/3242509897 shutdown_connections 2024-05-10T11:14:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.343+0000 7fca5ac00700 1 -- 172.21.15.27:0/3242509897 wait complete. 2024-05-10T11:14:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.344+0000 7fca5ac00700 1 Processor -- start 2024-05-10T11:14:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.344+0000 7fca5ac00700 1 -- start start 2024-05-10T11:14:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.344+0000 7fca5ac00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca54102180 0x7fca541a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.344+0000 7fca5ac00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca54102b30 0x7fca541a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.344+0000 7fca5ac00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca5410fd30 0x7fca5419baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.344+0000 7fca5ac00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fca541157b0 con 0x7fca5410fd30 2024-05-10T11:14:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.344+0000 7fca5ac00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fca54115630 con 0x7fca54102b30 2024-05-10T11:14:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.344+0000 7fca5ac00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fca54115930 con 0x7fca54102180 2024-05-10T11:14:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.345+0000 7fca59800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca54102180 0x7fca541a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.345+0000 7fca5a200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca5410fd30 0x7fca5419baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.345+0000 7fca5a200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca5410fd30 0x7fca5419baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39290/0 (socket says 172.21.15.27:39290) 2024-05-10T11:14:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.345+0000 7fca59800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca54102180 0x7fca541a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:47122/0 (socket says 172.21.15.27:47122) 2024-05-10T11:14:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.345+0000 7fca5a200700 1 -- 172.21.15.27:0/234481968 learned_addr learned my addr 172.21.15.27:0/234481968 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.345+0000 7fca58e00700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca54102b30 0x7fca541a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.345+0000 7fca58e00700 1 -- 172.21.15.27:0/234481968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca54102180 msgr2=0x7fca541a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.345+0000 7fca58e00700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca54102180 0x7fca541a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.345+0000 7fca58e00700 1 -- 172.21.15.27:0/234481968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca5410fd30 msgr2=0x7fca5419baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.345+0000 7fca58e00700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca5410fd30 0x7fca5419baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.345+0000 7fca58e00700 1 -- 172.21.15.27:0/234481968 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca5419c250 con 0x7fca54102b30 2024-05-10T11:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.345+0000 7fca5a200700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca5410fd30 0x7fca5419baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.346+0000 7fca58e00700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca54102b30 0x7fca541a1e20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fca40006010 tx=0x7fca400057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.346+0000 7fca52000700 1 -- 172.21.15.27:0/234481968 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca40027070 con 0x7fca54102b30 2024-05-10T11:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.346+0000 7fca5ac00700 1 -- 172.21.15.27:0/234481968 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca5419c4e0 con 0x7fca54102b30 2024-05-10T11:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.346+0000 7fca5ac00700 1 -- 172.21.15.27:0/234481968 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca541a87b0 con 0x7fca54102b30 2024-05-10T11:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.346+0000 7fca52000700 1 -- 172.21.15.27:0/234481968 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca40022470 con 0x7fca54102b30 2024-05-10T11:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.346+0000 7fca52000700 1 -- 172.21.15.27:0/234481968 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca40017650 con 0x7fca54102b30 2024-05-10T11:14:23.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.351+0000 7fca66baab80 1 Processor -- start 2024-05-10T11:14:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.351+0000 7fca66baab80 1 -- start start 2024-05-10T11:14:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.351+0000 7fca52000700 1 -- 172.21.15.27:0/234481968 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7fca40009d10 con 0x7fca54102b30 2024-05-10T11:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.352+0000 7fca66baab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 0x5609b6ae1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.352+0000 7fca66baab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae16c0 0x5609b6adb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.352+0000 7fca66baab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6adba50 0x5609b6adbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.352+0000 7fca66baab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5609b6a2da20 con 0x5609b6adba50 2024-05-10T11:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.352+0000 7fca66baab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5609b69cb760 con 0x5609b6ae0da0 2024-05-10T11:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.352+0000 7fca66baab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5609b6a299e0 con 0x5609b6ae16c0 2024-05-10T11:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.352+0000 7fca5a200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 0x5609b6ae1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.352+0000 7fca5a200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 0x5609b6ae1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44780/0 (socket says 172.21.15.27:44780) 2024-05-10T11:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.352+0000 7fca5a200700 1 -- 172.21.15.27:0/632456021 learned_addr learned my addr 172.21.15.27:0/632456021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.352+0000 7fca59800700 1 --2- 172.21.15.27:0/632456021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae16c0 0x5609b6adb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.352+0000 7fca58e00700 1 --2- 172.21.15.27:0/632456021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6adba50 0x5609b6adbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.353+0000 7fca5a200700 1 -- 172.21.15.27:0/632456021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae16c0 msgr2=0x5609b6adb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.353+0000 7fca5a200700 1 --2- 172.21.15.27:0/632456021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae16c0 0x5609b6adb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.353+0000 7fca5a200700 1 -- 172.21.15.27:0/632456021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6adba50 msgr2=0x5609b6adbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.353+0000 7fca5a200700 1 --2- 172.21.15.27:0/632456021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6adba50 0x5609b6adbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.353+0000 7fca5a200700 1 -- 172.21.15.27:0/632456021 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609b697a8d0 con 0x5609b6ae0da0 2024-05-10T11:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.353+0000 7fca5a200700 1 --2- 172.21.15.27:0/632456021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 0x5609b6ae1180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fca4c007530 tx=0x7fca4c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5036ae732b47659 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.353+0000 7fca46a00700 1 -- 172.21.15.27:0/632456021 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c019070 con 0x5609b6ae0da0 2024-05-10T11:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.353+0000 7fca46a00700 1 -- 172.21.15.27:0/632456021 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca4c00edb0 con 0x5609b6ae0da0 2024-05-10T11:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.353+0000 7fca46a00700 1 -- 172.21.15.27:0/632456021 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c0143b0 con 0x5609b6ae0da0 2024-05-10T11:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.354+0000 7fca52000700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fca3c0b2ba0 0x7fca3c0b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.354+0000 7fca52000700 1 -- 172.21.15.27:0/234481968 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fca400e3470 con 0x7fca54102b30 2024-05-10T11:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.354+0000 7fca59800700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fca3c0b2ba0 0x7fca3c0b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.354+0000 7fca66baab80 1 -- 172.21.15.27:0/632456021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 msgr2=0x5609b6ae1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:23.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.354+0000 7fca66baab80 1 --2- 172.21.15.27:0/632456021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 0x5609b6ae1180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fca4c007530 tx=0x7fca4c009920 comp rx=0 tx=0).stop 2024-05-10T11:14:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.355+0000 7fca66baab80 1 -- 172.21.15.27:0/632456021 shutdown_connections 2024-05-10T11:14:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.355+0000 7fca66baab80 1 --2- 172.21.15.27:0/632456021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6adba50 0x5609b6adbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.355+0000 7fca66baab80 1 --2- 172.21.15.27:0/632456021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 0x5609b6ae1180 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.355+0000 7fca66baab80 1 --2- 172.21.15.27:0/632456021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae16c0 0x5609b6adb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.355+0000 7fca66baab80 1 -- 172.21.15.27:0/632456021 >> 172.21.15.27:0/632456021 conn(0x5609b6a536d0 msgr2=0x5609b6ad5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.355+0000 7fca66baab80 1 -- 172.21.15.27:0/632456021 shutdown_connections 2024-05-10T11:14:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.355+0000 7fca66baab80 1 -- 172.21.15.27:0/632456021 wait complete. 2024-05-10T11:14:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.355+0000 7fca59800700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fca3c0b2ba0 0x7fca3c0b5060 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x5609b6ad7970 tx=0x7fca48009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.356+0000 7fca66baab80 1 -- start start 2024-05-10T11:14:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.356+0000 7fca66baab80 1 Processor -- start 2024-05-10T11:14:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca66baab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6adba50 0x5609b6ae72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca66baab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 0x5609b6ae4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca66baab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6ae16c0 0x5609b6ae4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca66baab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5609b6ae8150 con 0x5609b6ae16c0 2024-05-10T11:14:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca66baab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5609b6993b20 con 0x5609b6ae0da0 2024-05-10T11:14:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca66baab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5609b6ae82d0 con 0x5609b6adba50 2024-05-10T11:14:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca66baab80 10 client.0 _ll_register_callbacks cb 0x5609b69d4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca5a200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6ae16c0 0x5609b6ae4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca59800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6adba50 0x5609b6ae72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca59800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6adba50 0x5609b6ae72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:47132/0 (socket says 172.21.15.27:47132) 2024-05-10T11:14:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca5a200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6ae16c0 0x5609b6ae4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39302/0 (socket says 172.21.15.27:39302) 2024-05-10T11:14:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca59800700 1 -- 172.21.15.27:0/1501143724 learned_addr learned my addr 172.21.15.27:0/1501143724 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca58e00700 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 0x5609b6ae4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca5a200700 1 -- 172.21.15.27:0/1501143724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6adba50 msgr2=0x5609b6ae72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.357+0000 7fca5a200700 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6adba50 0x5609b6ae72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.358+0000 7fca5a200700 1 -- 172.21.15.27:0/1501143724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 msgr2=0x5609b6ae4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.358+0000 7fca5a200700 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 0x5609b6ae4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.358+0000 7fca5a200700 1 -- 172.21.15.27:0/1501143724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609b69f3420 con 0x5609b6ae16c0 2024-05-10T11:14:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.358+0000 7fca58e00700 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 0x5609b6ae4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.358+0000 7fca5a200700 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6ae16c0 0x5609b6ae4bd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fca4c007500 tx=0x7fca4c016d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01357dccb8fbb7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.358+0000 7fca3b400700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.358+0000 7fca3b400700 1 -- 172.21.15.27:0/1501143724 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c019040 con 0x5609b6ae16c0 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.358+0000 7fca3b400700 1 -- 172.21.15.27:0/1501143724 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fca4c003a00 con 0x5609b6ae16c0 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.359+0000 7fca66baab80 1 -- 172.21.15.27:0/1501143724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca4c00edb0 con 0x5609b6ae16c0 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.359+0000 7fca3b400700 1 -- 172.21.15.27:0/1501143724 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fca4c003c00 con 0x5609b6ae16c0 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.359+0000 7fca66baab80 10 client.15348 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.359+0000 7fca66baab80 1 -- 172.21.15.27:0/1501143724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5609b6af2930 con 0x5609b6ae16c0 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.359+0000 7fca3b400700 1 -- 172.21.15.27:0/1501143724 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca4c00edb0 con 0x5609b6ae16c0 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.359+0000 7fca3b400700 1 -- 172.21.15.27:0/1501143724 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fca4c026030 con 0x5609b6ae16c0 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.359+0000 7fca66baab80 10 client.15348 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.359+0000 7fca66baab80 10 client.15348 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:14:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.360+0000 7fca66baab80 20 client.15348 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:23.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.360+0000 7fca66baab80 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5609b6af4eb0 0x5609b6af72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.360+0000 7fca66baab80 4 client.15348 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:14:23.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.360+0000 7fca66baab80 1 -- 172.21.15.27:0/1501143724 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609b6aa8140 con 0x5609b6af4eb0 2024-05-10T11:14:23.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.360+0000 7fca59800700 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5609b6af4eb0 0x5609b6af72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.361+0000 7fca3b400700 10 client.15348 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:14:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.361+0000 7fca59800700 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5609b6af4eb0 0x5609b6af72a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3adfc849883113a server_cookie=acf0d802bc2a76ce in_seq=0 out_seq=0 2024-05-10T11:14:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.362+0000 7fca3b400700 1 -- 172.21.15.27:0/1501143724 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5609b6aa8140 con 0x5609b6af4eb0 2024-05-10T11:14:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.362+0000 7fca3b400700 10 client.15348 handle_command_reply: tid=0 2024-05-10T11:14:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.362+0000 7fca66baab80 1 client.15348 shutdown 2024-05-10T11:14:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.362+0000 7fca66baab80 1 -- 172.21.15.27:0/1501143724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6ae16c0 msgr2=0x5609b6ae4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.362+0000 7fca66baab80 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6ae16c0 0x5609b6ae4bd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fca4c007500 tx=0x7fca4c016d70 comp rx=0 tx=0).stop 2024-05-10T11:14:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.362+0000 7fca66baab80 1 -- 172.21.15.27:0/1501143724 shutdown_connections 2024-05-10T11:14:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.362+0000 7fca66baab80 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5609b6af4eb0 0x5609b6af72a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.362+0000 7fca66baab80 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6adba50 0x5609b6ae72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.362+0000 7fca3b400700 0 client.15348 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:14:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.362+0000 7fca66baab80 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6ae16c0 0x5609b6ae4bd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.362+0000 7fca66baab80 1 --2- 172.21.15.27:0/1501143724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae0da0 0x5609b6ae4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.362+0000 7fca66baab80 1 -- 172.21.15.27:0/1501143724 >> 172.21.15.27:0/1501143724 conn(0x5609b6a536d0 msgr2=0x5609b6ad0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.363+0000 7fca66baab80 1 -- 172.21.15.27:0/1501143724 shutdown_connections 2024-05-10T11:14:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.363+0000 7fca66baab80 1 -- 172.21.15.27:0/1501143724 wait complete. 2024-05-10T11:14:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.363+0000 7fca66baab80 20 client.15348 trim_cache size 0 max 16384 2024-05-10T11:14:23.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.380+0000 7fca66baab80 1 Processor -- start 2024-05-10T11:14:23.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.380+0000 7fca66baab80 1 -- start start 2024-05-10T11:14:23.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.381+0000 7fca66baab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6adba50 0x5609b6adb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.381+0000 7fca66baab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae0da0 0x5609b6ae2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.381+0000 7fca66baab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6ae16c0 0x5609b6ad0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.381+0000 7fca66baab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fca3400ee20 con 0x5609b6ae16c0 2024-05-10T11:14:23.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.381+0000 7fca66baab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fca34010960 con 0x5609b6adba50 2024-05-10T11:14:23.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.381+0000 7fca66baab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fca3400f4f0 con 0x5609b6ae0da0 2024-05-10T11:14:23.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.381+0000 7fca58e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae0da0 0x5609b6ae2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.381+0000 7fca59800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6adba50 0x5609b6adb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.381+0000 7fca5a200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6ae16c0 0x5609b6ad0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.381+0000 7fca59800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6adba50 0x5609b6adb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44794/0 (socket says 172.21.15.27:44794) 2024-05-10T11:14:23.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.381+0000 7fca59800700 1 -- 172.21.15.27:0/1932718534 learned_addr learned my addr 172.21.15.27:0/1932718534 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:23.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.382+0000 7fca58e00700 1 -- 172.21.15.27:0/1932718534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6adba50 msgr2=0x5609b6adb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:23.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.382+0000 7fca58e00700 1 --2- 172.21.15.27:0/1932718534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6adba50 0x5609b6adb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.382+0000 7fca58e00700 1 -- 172.21.15.27:0/1932718534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6ae16c0 msgr2=0x5609b6ad0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:23.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.382+0000 7fca58e00700 1 --2- 172.21.15.27:0/1932718534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6ae16c0 0x5609b6ad0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.382+0000 7fca58e00700 1 -- 172.21.15.27:0/1932718534 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609b6ae6f30 con 0x5609b6ae0da0 2024-05-10T11:14:23.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.382+0000 7fca59800700 1 --2- 172.21.15.27:0/1932718534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6adba50 0x5609b6adb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:23.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.382+0000 7fca58e00700 1 --2- 172.21.15.27:0/1932718534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae0da0 0x5609b6ae2330 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fca400e05b0 tx=0x7fca400e6f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eef6c8fb5c329274 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:23.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.382+0000 7fca46000700 1 -- 172.21.15.27:0/1932718534 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca400a6090 con 0x5609b6ae0da0 2024-05-10T11:14:23.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.382+0000 7fca46000700 1 -- 172.21.15.27:0/1932718534 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca400a9ce0 con 0x5609b6ae0da0 2024-05-10T11:14:23.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.382+0000 7fca46000700 1 -- 172.21.15.27:0/1932718534 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca400a9940 con 0x5609b6ae0da0 2024-05-10T11:14:23.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.383+0000 7fca66baab80 1 -- 172.21.15.27:0/1932718534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae0da0 msgr2=0x5609b6ae2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:23.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.383+0000 7fca66baab80 1 --2- 172.21.15.27:0/1932718534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae0da0 0x5609b6ae2330 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fca400e05b0 tx=0x7fca400e6f20 comp rx=0 tx=0).stop 2024-05-10T11:14:23.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.383+0000 7fca66baab80 1 -- 172.21.15.27:0/1932718534 shutdown_connections 2024-05-10T11:14:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.383+0000 7fca66baab80 1 --2- 172.21.15.27:0/1932718534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6ae16c0 0x5609b6ad0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.383+0000 7fca66baab80 1 --2- 172.21.15.27:0/1932718534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6adba50 0x5609b6adb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.383+0000 7fca66baab80 1 --2- 172.21.15.27:0/1932718534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae0da0 0x5609b6ae2330 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.383+0000 7fca66baab80 1 -- 172.21.15.27:0/1932718534 >> 172.21.15.27:0/1932718534 conn(0x5609b6a536d0 msgr2=0x5609b6ae5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.383+0000 7fca66baab80 1 -- 172.21.15.27:0/1932718534 shutdown_connections 2024-05-10T11:14:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.383+0000 7fca66baab80 1 -- 172.21.15.27:0/1932718534 wait complete. 2024-05-10T11:14:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.384+0000 7fca66baab80 1 -- start start 2024-05-10T11:14:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.384+0000 7fca66baab80 1 Processor -- start 2024-05-10T11:14:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca66baab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6adba50 0x5609b6aedb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca66baab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae0da0 0x5609b6afbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca66baab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae16c0 0x5609b6afc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca66baab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fca34010290 con 0x5609b6adba50 2024-05-10T11:14:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca66baab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fca3400fbc0 con 0x5609b6ae16c0 2024-05-10T11:14:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca66baab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fca3400cdc0 con 0x5609b6ae0da0 2024-05-10T11:14:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca66baab80 10 client.0 _ll_register_callbacks cb 0x5609b69d4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca59800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6adba50 0x5609b6aedb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca59800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6adba50 0x5609b6aedb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39314/0 (socket says 172.21.15.27:39314) 2024-05-10T11:14:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca59800700 1 -- 172.21.15.27:0/180757214 learned_addr learned my addr 172.21.15.27:0/180757214 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca58e00700 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae0da0 0x5609b6afbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca5a200700 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae16c0 0x5609b6afc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca58e00700 1 -- 172.21.15.27:0/180757214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae16c0 msgr2=0x5609b6afc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca58e00700 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae16c0 0x5609b6afc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca58e00700 1 -- 172.21.15.27:0/180757214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6adba50 msgr2=0x5609b6aedb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca58e00700 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6adba50 0x5609b6aedb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.385+0000 7fca58e00700 1 -- 172.21.15.27:0/180757214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609b6afcd20 con 0x5609b6ae0da0 2024-05-10T11:14:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.386+0000 7fca58e00700 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae0da0 0x5609b6afbf70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fca40000c00 tx=0x7fca400e6cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0adf368b5db742 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.386+0000 7fca3b400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:14:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.386+0000 7fca3b400700 1 -- 172.21.15.27:0/180757214 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca400a9b20 con 0x5609b6ae0da0 2024-05-10T11:14:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.386+0000 7fca3b400700 1 -- 172.21.15.27:0/180757214 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fca400052c0 con 0x5609b6ae0da0 2024-05-10T11:14:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.386+0000 7fca66baab80 1 -- 172.21.15.27:0/180757214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca400a9ce0 con 0x5609b6ae0da0 2024-05-10T11:14:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.386+0000 7fca3b400700 1 -- 172.21.15.27:0/180757214 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fca400a9fa0 con 0x5609b6ae0da0 2024-05-10T11:14:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.386+0000 7fca66baab80 10 client.25477 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.386+0000 7fca66baab80 1 -- 172.21.15.27:0/180757214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5609b6b5f610 con 0x5609b6ae0da0 2024-05-10T11:14:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.387+0000 7fca3b400700 1 -- 172.21.15.27:0/180757214 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca400a9ce0 con 0x5609b6ae0da0 2024-05-10T11:14:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.387+0000 7fca3b400700 1 -- 172.21.15.27:0/180757214 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fca400e3110 con 0x5609b6ae0da0 2024-05-10T11:14:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.387+0000 7fca66baab80 10 client.25477 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.387+0000 7fca66baab80 10 client.25477 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:14:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.387+0000 7fca66baab80 20 client.25477 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.387+0000 7fca66baab80 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5609b6b5fbc0 0x5609b6b61fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.387+0000 7fca66baab80 4 client.25477 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:14:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.387+0000 7fca66baab80 1 -- 172.21.15.27:0/180757214 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5609b6a3b400 con 0x5609b6b5fbc0 2024-05-10T11:14:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.388+0000 7fca59800700 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5609b6b5fbc0 0x5609b6b61fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:23.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.388+0000 7fca59800700 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5609b6b5fbc0 0x5609b6b61fe0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4aa4651f33456b1f server_cookie=ec1bf7ac8fd6a1b0 in_seq=0 out_seq=0 2024-05-10T11:14:23.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.388+0000 7fca3b400700 10 client.25477 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:14:23.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.388+0000 7fca3b400700 1 -- 172.21.15.27:0/180757214 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5609b6a3b400 con 0x5609b6b5fbc0 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.388+0000 7fca3b400700 10 client.25477 handle_command_reply: tid=0 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.388+0000 7fca66baab80 1 client.25477 shutdown 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.389+0000 7fca66baab80 1 -- 172.21.15.27:0/180757214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae0da0 msgr2=0x5609b6afbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.389+0000 7fca66baab80 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae0da0 0x5609b6afbf70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fca40000c00 tx=0x7fca400e6cf0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.389+0000 7fca66baab80 1 -- 172.21.15.27:0/180757214 shutdown_connections 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.389+0000 7fca66baab80 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5609b6b5fbc0 0x5609b6b61fe0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.389+0000 7fca66baab80 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5609b6ae16c0 0x5609b6afc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.389+0000 7fca66baab80 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5609b6adba50 0x5609b6aedb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.389+0000 7fca66baab80 1 --2- 172.21.15.27:0/180757214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609b6ae0da0 0x5609b6afbf70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.389+0000 7fca66baab80 1 -- 172.21.15.27:0/180757214 >> 172.21.15.27:0/180757214 conn(0x5609b6a536d0 msgr2=0x5609b6ae21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.389+0000 7fca3b400700 0 client.25477 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.389+0000 7fca66baab80 1 -- 172.21.15.27:0/180757214 shutdown_connections 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.389+0000 7fca66baab80 1 -- 172.21.15.27:0/180757214 wait complete. 2024-05-10T11:14:23.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.389+0000 7fca66baab80 20 client.25477 trim_cache size 0 max 16384 2024-05-10T11:14:23.391 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:14:23.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.390+0000 7fca5ac00700 1 -- 172.21.15.27:0/234481968 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fca3c0b2ba0 msgr2=0x7fca3c0b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:23.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.390+0000 7fca5ac00700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fca3c0b2ba0 0x7fca3c0b5060 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x5609b6ad7970 tx=0x7fca48009260 comp rx=0 tx=0).stop 2024-05-10T11:14:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.390+0000 7fca5ac00700 1 -- 172.21.15.27:0/234481968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca54102b30 msgr2=0x7fca541a1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.390+0000 7fca5ac00700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca54102b30 0x7fca541a1e20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fca40006010 tx=0x7fca400057c0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.390+0000 7fca5ac00700 1 -- 172.21.15.27:0/234481968 shutdown_connections 2024-05-10T11:14:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.390+0000 7fca5ac00700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fca3c0b2ba0 0x7fca3c0b5060 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.390+0000 7fca5ac00700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca54102180 0x7fca541a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.390+0000 7fca5ac00700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca5410fd30 0x7fca5419baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.390+0000 7fca5ac00700 1 --2- 172.21.15.27:0/234481968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca54102b30 0x7fca541a1e20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.390+0000 7fca5ac00700 1 -- 172.21.15.27:0/234481968 >> 172.21.15.27:0/234481968 conn(0x7fca540fddc0 msgr2=0x7fca541001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.390+0000 7fca5ac00700 1 -- 172.21.15.27:0/234481968 shutdown_connections 2024-05-10T11:14:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:23.391+0000 7fca5ac00700 1 -- 172.21.15.27:0/234481968 wait complete. 2024-05-10T11:14:24.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:14:24.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:24 smithi027 ceph-mon[30054]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 852 B/s wr, 8 op/s 2024-05-10T11:14:24.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:24 smithi076 ceph-mon[34152]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 852 B/s wr, 8 op/s 2024-05-10T11:14:24.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:24 smithi189 ceph-mon[34842]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 852 B/s wr, 8 op/s 2024-05-10T11:14:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:14:26.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:26 smithi027 ceph-mon[30054]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 853 B/s wr, 12 op/s 2024-05-10T11:14:26.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:26 smithi076 ceph-mon[34152]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 853 B/s wr, 12 op/s 2024-05-10T11:14:26.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:26 smithi189 ceph-mon[34842]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 853 B/s wr, 12 op/s 2024-05-10T11:14:27.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.241+0000 7f3cc2000700 1 -- 172.21.15.27:0/4140783973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3cbc102450 msgr2=0x7f3cbc1062a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:27.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.241+0000 7f3cc2000700 1 --2- 172.21.15.27:0/4140783973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3cbc102450 0x7f3cbc1062a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80098f0 tx=0x7f3ca8009c00 comp rx=0 tx=0).stop 2024-05-10T11:14:27.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.241+0000 7f3cc2000700 1 -- 172.21.15.27:0/4140783973 shutdown_connections 2024-05-10T11:14:27.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.241+0000 7f3cc2000700 1 --2- 172.21.15.27:0/4140783973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc1069d0 0x7f3cbc10e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.241+0000 7f3cc2000700 1 --2- 172.21.15.27:0/4140783973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc101aa0 0x7f3cbc101e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.241+0000 7f3cc2000700 1 --2- 172.21.15.27:0/4140783973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3cbc102450 0x7f3cbc1062a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.241+0000 7f3cc2000700 1 -- 172.21.15.27:0/4140783973 >> 172.21.15.27:0/4140783973 conn(0x7f3cbc077140 msgr2=0x7f3cbc077550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:27.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.241+0000 7f3cc2000700 1 -- 172.21.15.27:0/4140783973 shutdown_connections 2024-05-10T11:14:27.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.241+0000 7f3cc2000700 1 -- 172.21.15.27:0/4140783973 wait complete. 2024-05-10T11:14:27.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.243+0000 7f3cc2000700 1 Processor -- start 2024-05-10T11:14:27.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.243+0000 7f3cc2000700 1 -- start start 2024-05-10T11:14:27.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.244+0000 7f3cc2000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc101aa0 0x7f3cbc1032b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.244+0000 7f3cc2000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3cbc102450 0x7f3cbc1037f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.244+0000 7f3cc2000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc1069d0 0x7f3cbc103d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.244+0000 7f3cc2000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc112510 con 0x7f3cbc1069d0 2024-05-10T11:14:27.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.244+0000 7f3cc2000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc112390 con 0x7f3cbc102450 2024-05-10T11:14:27.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.244+0000 7f3cc2000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc112690 con 0x7f3cbc101aa0 2024-05-10T11:14:27.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.244+0000 7f3cbbe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3cbc102450 0x7f3cbc1037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.244+0000 7f3cbbe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3cbc102450 0x7f3cbc1037f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52868/0 (socket says 172.21.15.27:52868) 2024-05-10T11:14:27.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.244+0000 7f3cbbe00700 1 -- 172.21.15.27:0/3751731726 learned_addr learned my addr 172.21.15.27:0/3751731726 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:27.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.244+0000 7f3cc0c00700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc101aa0 0x7f3cbc1032b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.245+0000 7f3cc1600700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc1069d0 0x7f3cbc103d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.245+0000 7f3cbbe00700 1 -- 172.21.15.27:0/3751731726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc101aa0 msgr2=0x7f3cbc1032b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:27.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.245+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc101aa0 0x7f3cbc1032b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.245+0000 7f3cbbe00700 1 -- 172.21.15.27:0/3751731726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc1069d0 msgr2=0x7f3cbc103d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:27.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.245+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc1069d0 0x7f3cbc103d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.245+0000 7f3cbbe00700 1 -- 172.21.15.27:0/3751731726 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cbc1a7fc0 con 0x7f3cbc102450 2024-05-10T11:14:27.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.245+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3cbc102450 0x7f3cbc1037f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8009230 tx=0x7f3ca800fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:27.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.245+0000 7f3cb9600700 1 -- 172.21.15.27:0/3751731726 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8005220 con 0x7f3cbc102450 2024-05-10T11:14:27.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.246+0000 7f3cc2000700 1 -- 172.21.15.27:0/3751731726 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cbc1a8250 con 0x7f3cbc102450 2024-05-10T11:14:27.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.246+0000 7f3cb9600700 1 -- 172.21.15.27:0/3751731726 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ca800b560 con 0x7f3cbc102450 2024-05-10T11:14:27.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.246+0000 7f3cc2000700 1 -- 172.21.15.27:0/3751731726 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cbc1a8700 con 0x7f3cbc102450 2024-05-10T11:14:27.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.246+0000 7f3cb9600700 1 -- 172.21.15.27:0/3751731726 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8003a80 con 0x7f3cbc102450 2024-05-10T11:14:27.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.249+0000 7f3cb9600700 1 -- 172.21.15.27:0/3751731726 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f3ca80053c0 con 0x7f3cbc102450 2024-05-10T11:14:27.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.251+0000 7f3cb9600700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3ca40b2c20 0x7f3ca40b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.251+0000 7f3cb9600700 1 -- 172.21.15.27:0/3751731726 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f3ca80e3aa0 con 0x7f3cbc102450 2024-05-10T11:14:27.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.252+0000 7f3cc0c00700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3ca40b2c20 0x7f3ca40b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.252+0000 7f3cc0c00700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3ca40b2c20 0x7f3ca40b50e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0005fd0 tx=0x7f3cb0009310 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.252+0000 7f3ccdfdab80 1 Processor -- start 2024-05-10T11:14:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.253+0000 7f3ccdfdab80 1 -- start start 2024-05-10T11:14:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.253+0000 7f3ccdfdab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6f9e0 0x563ba3c6fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.253+0000 7f3ccdfdab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c70300 0x563ba3c69fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.253+0000 7f3ccdfdab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c6aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.253+0000 7f3ccdfdab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563ba3bbca20 con 0x563ba3c6f9e0 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.253+0000 7f3ccdfdab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563ba3b5a760 con 0x563ba3c70300 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.253+0000 7f3ccdfdab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563ba3bb89e0 con 0x563ba3c6a600 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.254+0000 7f3cc0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6f9e0 0x563ba3c6fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.254+0000 7f3cc0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6f9e0 0x563ba3c6fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38866/0 (socket says 172.21.15.27:38866) 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.254+0000 7f3cc0c00700 1 -- 172.21.15.27:0/1678173394 learned_addr learned my addr 172.21.15.27:0/1678173394 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.254+0000 7f3cc1600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c6aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.254+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/1678173394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c70300 0x563ba3c69fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.254+0000 7f3cc1600700 1 -- 172.21.15.27:0/1678173394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c70300 msgr2=0x563ba3c69fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.254+0000 7f3cc1600700 1 --2- 172.21.15.27:0/1678173394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c70300 0x563ba3c69fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.254+0000 7f3cc1600700 1 -- 172.21.15.27:0/1678173394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6f9e0 msgr2=0x563ba3c6fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.254+0000 7f3cc1600700 1 --2- 172.21.15.27:0/1678173394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6f9e0 0x563ba3c6fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.254+0000 7f3cc1600700 1 -- 172.21.15.27:0/1678173394 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ba3b098d0 con 0x563ba3c6a600 2024-05-10T11:14:27.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.254+0000 7f3cc1600700 1 --2- 172.21.15.27:0/1678173394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c6aa80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4009310 tx=0x7f3cb4009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1095d697f820229a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:27.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.255+0000 7f3cae000700 1 -- 172.21.15.27:0/1678173394 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4019020 con 0x563ba3c6a600 2024-05-10T11:14:27.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.255+0000 7f3cae000700 1 -- 172.21.15.27:0/1678173394 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cb4015af0 con 0x563ba3c6a600 2024-05-10T11:14:27.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.255+0000 7f3cae000700 1 -- 172.21.15.27:0/1678173394 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4014a80 con 0x563ba3c6a600 2024-05-10T11:14:27.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.256+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1678173394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 msgr2=0x563ba3c6aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:27.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.256+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/1678173394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c6aa80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4009310 tx=0x7f3cb4009920 comp rx=0 tx=0).stop 2024-05-10T11:14:27.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.256+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1678173394 shutdown_connections 2024-05-10T11:14:27.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.256+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/1678173394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c6aa80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.256+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/1678173394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6f9e0 0x563ba3c6fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.256+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/1678173394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c70300 0x563ba3c69fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.256+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1678173394 >> 172.21.15.27:0/1678173394 conn(0x563ba3b98b60 msgr2=0x563ba3c649b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:27.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.256+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1678173394 shutdown_connections 2024-05-10T11:14:27.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.256+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1678173394 wait complete. 2024-05-10T11:14:27.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.257+0000 7f3ccdfdab80 1 -- start start 2024-05-10T11:14:27.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.257+0000 7f3ccdfdab80 1 Processor -- start 2024-05-10T11:14:27.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.257+0000 7f3ccdfdab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c75e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.257+0000 7f3ccdfdab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6f9e0 0x563ba3c763a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.257+0000 7f3ccdfdab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c70300 0x563ba3c73400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3ccdfdab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563ba3c76db0 con 0x563ba3c6f9e0 2024-05-10T11:14:27.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3ccdfdab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563ba3b22b20 con 0x563ba3c70300 2024-05-10T11:14:27.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3ccdfdab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563ba3c76f30 con 0x563ba3c6a600 2024-05-10T11:14:27.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3ccdfdab80 10 client.0 _ll_register_callbacks cb 0x563ba3b63c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3cbbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6f9e0 0x563ba3c763a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3cc0c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c75e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3cc0c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c75e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41946/0 (socket says 172.21.15.27:41946) 2024-05-10T11:14:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3cbbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6f9e0 0x563ba3c763a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38872/0 (socket says 172.21.15.27:38872) 2024-05-10T11:14:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3cc0c00700 1 -- 172.21.15.27:0/1682049179 learned_addr learned my addr 172.21.15.27:0/1682049179 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3cc1600700 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c70300 0x563ba3c73400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3cc0c00700 1 -- 172.21.15.27:0/1682049179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c70300 msgr2=0x563ba3c73400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3cc0c00700 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c70300 0x563ba3c73400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3cc0c00700 1 -- 172.21.15.27:0/1682049179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6f9e0 msgr2=0x563ba3c763a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3cc0c00700 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6f9e0 0x563ba3c763a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3cc0c00700 1 -- 172.21.15.27:0/1682049179 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ba3b82420 con 0x563ba3c6a600 2024-05-10T11:14:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.258+0000 7f3cc1600700 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c70300 0x563ba3c73400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:14:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.259+0000 7f3cc0c00700 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c75e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0003930 tx=0x7f3cb00047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3536f9c003b22b62 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.259+0000 7f3ca2a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:14:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.259+0000 7f3ca2a00700 1 -- 172.21.15.27:0/1682049179 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb001fa00 con 0x563ba3c6a600 2024-05-10T11:14:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.259+0000 7f3ca2a00700 1 -- 172.21.15.27:0/1682049179 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f3cb001fba0 con 0x563ba3c6a600 2024-05-10T11:14:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.259+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1682049179 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cb4015af0 con 0x563ba3c6a600 2024-05-10T11:14:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.260+0000 7f3ca2a00700 1 -- 172.21.15.27:0/1682049179 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3cb001fda0 con 0x563ba3c6a600 2024-05-10T11:14:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.260+0000 7f3ccdfdab80 10 client.25489 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.260+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1682049179 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563ba3c81520 con 0x563ba3c6a600 2024-05-10T11:14:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.260+0000 7f3ca2a00700 1 -- 172.21.15.27:0/1682049179 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cb4015af0 con 0x563ba3c6a600 2024-05-10T11:14:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.261+0000 7f3ca2a00700 1 -- 172.21.15.27:0/1682049179 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3cb0026070 con 0x563ba3c6a600 2024-05-10T11:14:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.261+0000 7f3ccdfdab80 10 client.25489 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.261+0000 7f3ccdfdab80 10 client.25489 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:14:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.261+0000 7f3ccdfdab80 20 client.25489 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.261+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ba3c83c00 0x563ba3c86050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.261+0000 7f3ccdfdab80 4 client.25489 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:14:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.261+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1682049179 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ba3c370a0 con 0x563ba3c83c00 2024-05-10T11:14:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.261+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ba3c83c00 0x563ba3c86050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.262+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ba3c83c00 0x563ba3c86050 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4600d19f2782435d server_cookie=ac62e505efcb03c0 in_seq=0 out_seq=0 2024-05-10T11:14:27.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.262+0000 7f3ca2a00700 10 client.25489 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.263+0000 7f3ca2a00700 1 -- 172.21.15.27:0/1682049179 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563ba3c370a0 con 0x563ba3c83c00 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.263+0000 7f3ca2a00700 10 client.25489 handle_command_reply: tid=0 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.263+0000 7f3ccdfdab80 1 client.25489 shutdown 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.263+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1682049179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 msgr2=0x563ba3c75e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.263+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c75e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0003930 tx=0x7f3cb00047c0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.263+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1682049179 shutdown_connections 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.263+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ba3c83c00 0x563ba3c86050 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.263+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c75e60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.263+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c70300 0x563ba3c73400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.263+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/1682049179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6f9e0 0x563ba3c763a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.263+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1682049179 >> 172.21.15.27:0/1682049179 conn(0x563ba3b98b60 msgr2=0x563ba3c5f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.264+0000 7f3ca2a00700 0 client.25489 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.264+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1682049179 shutdown_connections 2024-05-10T11:14:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.264+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/1682049179 wait complete. 2024-05-10T11:14:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.264+0000 7f3ccdfdab80 20 client.25489 trim_cache size 0 max 16384 2024-05-10T11:14:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.280+0000 7f3ccdfdab80 1 Processor -- start 2024-05-10T11:14:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.280+0000 7f3ccdfdab80 1 -- start start 2024-05-10T11:14:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.280+0000 7f3ccdfdab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6a600 0x563ba3c69fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.280+0000 7f3ccdfdab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c6f9e0 0x563ba3c75e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.280+0000 7f3ccdfdab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c70300 0x563ba3c763a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.280+0000 7f3ccdfdab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c00ef10 con 0x563ba3c6a600 2024-05-10T11:14:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.280+0000 7f3ccdfdab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c010a50 con 0x563ba3c6f9e0 2024-05-10T11:14:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.280+0000 7f3ccdfdab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c00f5e0 con 0x563ba3c70300 2024-05-10T11:14:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.281+0000 7f3cc0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6a600 0x563ba3c69fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.281+0000 7f3cc0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6a600 0x563ba3c69fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38874/0 (socket says 172.21.15.27:38874) 2024-05-10T11:14:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.281+0000 7f3cc0c00700 1 -- 172.21.15.27:0/719354352 learned_addr learned my addr 172.21.15.27:0/719354352 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.281+0000 7f3cbbe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c6f9e0 0x563ba3c75e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.281+0000 7f3cc1600700 1 --2- 172.21.15.27:0/719354352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c70300 0x563ba3c763a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.281+0000 7f3cbbe00700 1 -- 172.21.15.27:0/719354352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c70300 msgr2=0x563ba3c763a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.281+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/719354352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c70300 0x563ba3c763a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.281+0000 7f3cbbe00700 1 -- 172.21.15.27:0/719354352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6a600 msgr2=0x563ba3c69fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.281+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/719354352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6a600 0x563ba3c69fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.281+0000 7f3cbbe00700 1 -- 172.21.15.27:0/719354352 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ba3c717b0 con 0x563ba3c6f9e0 2024-05-10T11:14:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.281+0000 7f3cc0c00700 1 --2- 172.21.15.27:0/719354352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6a600 0x563ba3c69fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.281+0000 7f3cc1600700 1 --2- 172.21.15.27:0/719354352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c70300 0x563ba3c763a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.282+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/719354352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c6f9e0 0x563ba3c75e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80e0be0 tx=0x7f3ca8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f840cd4c6bcd4e65 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.282+0000 7f3cad600700 1 -- 172.21.15.27:0/719354352 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca80aa800 con 0x563ba3c6f9e0 2024-05-10T11:14:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.282+0000 7f3cad600700 1 -- 172.21.15.27:0/719354352 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ca80aebb0 con 0x563ba3c6f9e0 2024-05-10T11:14:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.282+0000 7f3cad600700 1 -- 172.21.15.27:0/719354352 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8003c20 con 0x563ba3c6f9e0 2024-05-10T11:14:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.283+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/719354352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c6f9e0 msgr2=0x563ba3c75e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.283+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/719354352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c6f9e0 0x563ba3c75e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80e0be0 tx=0x7f3ca8009260 comp rx=0 tx=0).stop 2024-05-10T11:14:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.283+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/719354352 shutdown_connections 2024-05-10T11:14:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.283+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/719354352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c70300 0x563ba3c763a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.283+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/719354352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c6a600 0x563ba3c69fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.283+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/719354352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c6f9e0 0x563ba3c75e60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.283+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/719354352 >> 172.21.15.27:0/719354352 conn(0x563ba3b98b60 msgr2=0x563ba3c727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.283+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/719354352 shutdown_connections 2024-05-10T11:14:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.283+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/719354352 wait complete. 2024-05-10T11:14:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.283+0000 7f3ccdfdab80 1 -- start start 2024-05-10T11:14:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.284+0000 7f3ccdfdab80 1 Processor -- start 2024-05-10T11:14:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.284+0000 7f3ccdfdab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c7c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.284+0000 7f3ccdfdab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c6f9e0 0x563ba3c7c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.284+0000 7f3ccdfdab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c70300 0x563ba3c8ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.284+0000 7f3ccdfdab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c010380 con 0x563ba3c70300 2024-05-10T11:14:27.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.284+0000 7f3ccdfdab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c00fcb0 con 0x563ba3c6f9e0 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.284+0000 7f3ccdfdab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c00ceb0 con 0x563ba3c6a600 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.284+0000 7f3ccdfdab80 10 client.0 _ll_register_callbacks cb 0x563ba3b63c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3cc1600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c70300 0x563ba3c8ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3cbbe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c6f9e0 0x563ba3c7c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3cc0c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c7c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3cc1600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c70300 0x563ba3c8ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38882/0 (socket says 172.21.15.27:38882) 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3cc1600700 1 -- 172.21.15.27:0/343069165 learned_addr learned my addr 172.21.15.27:0/343069165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3cc0c00700 1 -- 172.21.15.27:0/343069165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c6f9e0 msgr2=0x563ba3c7c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3cc0c00700 1 --2- 172.21.15.27:0/343069165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c6f9e0 0x563ba3c7c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3cc0c00700 1 -- 172.21.15.27:0/343069165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c70300 msgr2=0x563ba3c8ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3cc0c00700 1 --2- 172.21.15.27:0/343069165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c70300 0x563ba3c8ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3cc0c00700 1 -- 172.21.15.27:0/343069165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ba3c8b4b0 con 0x563ba3c6a600 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/343069165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c6f9e0 0x563ba3c7c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3cc0c00700 1 --2- 172.21.15.27:0/343069165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c7c430 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0013010 tx=0x7f3cb0005e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9196d650454785a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:27.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.285+0000 7f3ca2a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:14:27.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.286+0000 7f3ca2a00700 1 -- 172.21.15.27:0/343069165 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb001f800 con 0x563ba3c6a600 2024-05-10T11:14:27.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.286+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/343069165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ca80aebb0 con 0x563ba3c6a600 2024-05-10T11:14:27.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.286+0000 7f3ca2a00700 1 -- 172.21.15.27:0/343069165 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f3cb001f9a0 con 0x563ba3c6a600 2024-05-10T11:14:27.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.286+0000 7f3ca2a00700 1 -- 172.21.15.27:0/343069165 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3cb001fba0 con 0x563ba3c6a600 2024-05-10T11:14:27.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.286+0000 7f3ccdfdab80 10 client.25495 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:27.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.286+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/343069165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563ba3cee110 con 0x563ba3c6a600 2024-05-10T11:14:27.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.287+0000 7f3ca2a00700 1 -- 172.21.15.27:0/343069165 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ca80aebb0 con 0x563ba3c6a600 2024-05-10T11:14:27.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.287+0000 7f3ca2a00700 1 -- 172.21.15.27:0/343069165 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3cb0026050 con 0x563ba3c6a600 2024-05-10T11:14:27.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.287+0000 7f3ccdfdab80 10 client.25495 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:27.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.287+0000 7f3ccdfdab80 10 client.25495 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:14:27.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.287+0000 7f3ccdfdab80 20 client.25495 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:27.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.287+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/343069165 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ba3cee840 0x563ba3cf0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:27.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.287+0000 7f3ccdfdab80 4 client.25495 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.287+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/343069165 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563ba3bca400 con 0x563ba3cee840 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.287+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/343069165 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ba3cee840 0x563ba3cf0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.288+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/343069165 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ba3cee840 0x563ba3cf0c60 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b6655a9b96ac8bbb server_cookie=8e1e676aa3b7c19b in_seq=0 out_seq=0 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.288+0000 7f3ca2a00700 10 client.25495 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.288+0000 7f3ca2a00700 1 -- 172.21.15.27:0/343069165 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563ba3bca400 con 0x563ba3cee840 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.288+0000 7f3ca2a00700 10 client.25495 handle_command_reply: tid=0 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.288+0000 7f3ccdfdab80 1 client.25495 shutdown 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.288+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/343069165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 msgr2=0x563ba3c7c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.289+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/343069165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c7c430 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0013010 tx=0x7f3cb0005e10 comp rx=0 tx=0).stop 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.289+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/343069165 shutdown_connections 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.289+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/343069165 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ba3cee840 0x563ba3cf0c60 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.289+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/343069165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ba3c6a600 0x563ba3c7c430 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.289+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/343069165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ba3c70300 0x563ba3c8ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.289+0000 7f3ccdfdab80 1 --2- 172.21.15.27:0/343069165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ba3c6f9e0 0x563ba3c7c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.289+0000 7f3ca2a00700 0 client.25495 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:14:27.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.289+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/343069165 >> 172.21.15.27:0/343069165 conn(0x563ba3b98b60 msgr2=0x563ba3c75e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:27.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.289+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/343069165 shutdown_connections 2024-05-10T11:14:27.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.289+0000 7f3ccdfdab80 1 -- 172.21.15.27:0/343069165 wait complete. 2024-05-10T11:14:27.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.289+0000 7f3ccdfdab80 20 client.25495 trim_cache size 0 max 16384 2024-05-10T11:14:27.292 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:14:27.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.290+0000 7f3cc2000700 1 -- 172.21.15.27:0/3751731726 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3ca40b2c20 msgr2=0x7f3ca40b50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:27.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.290+0000 7f3cc2000700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3ca40b2c20 0x7f3ca40b50e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0005fd0 tx=0x7f3cb0009310 comp rx=0 tx=0).stop 2024-05-10T11:14:27.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.290+0000 7f3cc2000700 1 -- 172.21.15.27:0/3751731726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3cbc102450 msgr2=0x7f3cbc1037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:27.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.290+0000 7f3cc2000700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3cbc102450 0x7f3cbc1037f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8009230 tx=0x7f3ca800fb20 comp rx=0 tx=0).stop 2024-05-10T11:14:27.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.290+0000 7f3cc2000700 1 -- 172.21.15.27:0/3751731726 shutdown_connections 2024-05-10T11:14:27.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.290+0000 7f3cc2000700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3ca40b2c20 0x7f3ca40b50e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.290+0000 7f3cc2000700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc101aa0 0x7f3cbc1032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.290+0000 7f3cc2000700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc1069d0 0x7f3cbc103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.290+0000 7f3cc2000700 1 --2- 172.21.15.27:0/3751731726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3cbc102450 0x7f3cbc1037f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:27.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.290+0000 7f3cc2000700 1 -- 172.21.15.27:0/3751731726 >> 172.21.15.27:0/3751731726 conn(0x7f3cbc077140 msgr2=0x7f3cbc105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:27.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.290+0000 7f3cc2000700 1 -- 172.21.15.27:0/3751731726 shutdown_connections 2024-05-10T11:14:27.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:27.290+0000 7f3cc2000700 1 -- 172.21.15.27:0/3751731726 wait complete. 2024-05-10T11:14:28.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:14:28.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:28 smithi027 ceph-mon[30054]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:28.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:28 smithi076 ceph-mon[34152]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:28.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:28 smithi189 ceph-mon[34842]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:14:30.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:30 smithi027 ceph-mon[30054]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-10T11:14:30.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:30 smithi076 ceph-mon[34152]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-10T11:14:30.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:30 smithi189 ceph-mon[34842]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-10T11:14:31.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.251+0000 7f2f4d600700 1 -- 172.21.15.27:0/3007812838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f48102f30 msgr2=0x7f2f48106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.251+0000 7f2f4d600700 1 --2- 172.21.15.27:0/3007812838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f48102f30 0x7f2f48106f80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2f340098f0 tx=0x7f2f34009c00 comp rx=0 tx=0).stop 2024-05-10T11:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.251+0000 7f2f4d600700 1 -- 172.21.15.27:0/3007812838 shutdown_connections 2024-05-10T11:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.251+0000 7f2f4d600700 1 --2- 172.21.15.27:0/3007812838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f48107660 0x7f2f4810f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.251+0000 7f2f4d600700 1 --2- 172.21.15.27:0/3007812838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f48102580 0x7f2f48102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.251+0000 7f2f4d600700 1 --2- 172.21.15.27:0/3007812838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f48102f30 0x7f2f48106f80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.251+0000 7f2f4d600700 1 -- 172.21.15.27:0/3007812838 >> 172.21.15.27:0/3007812838 conn(0x7f2f480fde30 msgr2=0x7f2f48100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.252+0000 7f2f4d600700 1 -- 172.21.15.27:0/3007812838 shutdown_connections 2024-05-10T11:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.252+0000 7f2f4d600700 1 -- 172.21.15.27:0/3007812838 wait complete. 2024-05-10T11:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.253+0000 7f2f4d600700 1 Processor -- start 2024-05-10T11:14:31.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.254+0000 7f2f4d600700 1 -- start start 2024-05-10T11:14:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.254+0000 7f2f4d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f48102580 0x7f2f48199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.254+0000 7f2f4d600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f48102f30 0x7f2f4819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.254+0000 7f2f4d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f48107660 0x7f2f4819e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f4d600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2f48112540 con 0x7f2f48107660 2024-05-10T11:14:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f4d600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481123c0 con 0x7f2f48102f30 2024-05-10T11:14:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f4d600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481126c0 con 0x7f2f48102580 2024-05-10T11:14:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f4cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f48107660 0x7f2f4819e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f48102580 0x7f2f48199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f47400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f48102f30 0x7f2f4819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f48102580 0x7f2f48199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41972/0 (socket says 172.21.15.27:41972) 2024-05-10T11:14:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f47e00700 1 -- 172.21.15.27:0/103372653 learned_addr learned my addr 172.21.15.27:0/103372653 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f4cc00700 1 -- 172.21.15.27:0/103372653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f48102580 msgr2=0x7f2f48199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f4cc00700 1 --2- 172.21.15.27:0/103372653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f48102580 0x7f2f48199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f4cc00700 1 -- 172.21.15.27:0/103372653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f48102f30 msgr2=0x7f2f4819a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:31.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f4cc00700 1 --2- 172.21.15.27:0/103372653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f48102f30 0x7f2f4819a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.255+0000 7f2f4cc00700 1 -- 172.21.15.27:0/103372653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4819ec40 con 0x7f2f48107660 2024-05-10T11:14:31.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.256+0000 7f2f4cc00700 1 --2- 172.21.15.27:0/103372653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f48107660 0x7f2f4819e540 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000b330 tx=0x7f2f4000b640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:31.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.256+0000 7f2f44c00700 1 -- 172.21.15.27:0/103372653 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4000bdd0 con 0x7f2f48107660 2024-05-10T11:14:31.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.256+0000 7f2f44c00700 1 -- 172.21.15.27:0/103372653 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f40004bb0 con 0x7f2f48107660 2024-05-10T11:14:31.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.256+0000 7f2f44c00700 1 -- 172.21.15.27:0/103372653 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40007dc0 con 0x7f2f48107660 2024-05-10T11:14:31.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.256+0000 7f2f4d600700 1 -- 172.21.15.27:0/103372653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f4819eed0 con 0x7f2f48107660 2024-05-10T11:14:31.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.256+0000 7f2f4d600700 1 -- 172.21.15.27:0/103372653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f48103f50 con 0x7f2f48107660 2024-05-10T11:14:31.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.260+0000 7f2f44c00700 1 -- 172.21.15.27:0/103372653 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f2f40020020 con 0x7f2f48107660 2024-05-10T11:14:31.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.262+0000 7f2f44c00700 1 --2- 172.21.15.27:0/103372653 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2f2c0b2c20 0x7f2f2c0b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.262+0000 7f2f44c00700 1 -- 172.21.15.27:0/103372653 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f2f400e6210 con 0x7f2f48107660 2024-05-10T11:14:31.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.262+0000 7f2f59526b80 1 Processor -- start 2024-05-10T11:14:31.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.263+0000 7f2f47e00700 1 --2- 172.21.15.27:0/103372653 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2f2c0b2c20 0x7f2f2c0b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.263+0000 7f2f59526b80 1 -- start start 2024-05-10T11:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.263+0000 7f2f59526b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 0x561c9f639200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.263+0000 7f2f59526b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 0x561c9f63d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.263+0000 7f2f59526b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 0x561c9f63db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.263+0000 7f2f59526b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561c9f582a20 con 0x561c9f639740 2024-05-10T11:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.263+0000 7f2f59526b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561c9f520760 con 0x561c9f63d6c0 2024-05-10T11:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.263+0000 7f2f59526b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561c9f57e9e0 con 0x561c9f638e20 2024-05-10T11:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.264+0000 7f2f47e00700 1 --2- 172.21.15.27:0/103372653 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2f2c0b2c20 0x7f2f2c0b50e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2f38005d10 tx=0x7f2f38009310 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:31.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.264+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 0x561c9f639200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.264+0000 7f2f47400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 0x561c9f63d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.264+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 0x561c9f639200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41988/0 (socket says 172.21.15.27:41988) 2024-05-10T11:14:31.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.264+0000 7f2f47e00700 1 -- 172.21.15.27:0/38415160 learned_addr learned my addr 172.21.15.27:0/38415160 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:31.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.264+0000 7f2f4cc00700 1 --2- 172.21.15.27:0/38415160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 0x561c9f63db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.264+0000 7f2f47e00700 1 -- 172.21.15.27:0/38415160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 msgr2=0x561c9f63db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:31.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.264+0000 7f2f47e00700 1 --2- 172.21.15.27:0/38415160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 0x561c9f63db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.264+0000 7f2f47e00700 1 -- 172.21.15.27:0/38415160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 msgr2=0x561c9f63d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:31.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.264+0000 7f2f47e00700 1 --2- 172.21.15.27:0/38415160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 0x561c9f63d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.264+0000 7f2f47e00700 1 -- 172.21.15.27:0/38415160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c9f4cf8d0 con 0x561c9f638e20 2024-05-10T11:14:31.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.265+0000 7f2f47e00700 1 --2- 172.21.15.27:0/38415160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 0x561c9f639200 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800e5e0 tx=0x7f2f38014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd4714df363d8883 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:31.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.265+0000 7f2f3d600700 1 -- 172.21.15.27:0/38415160 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38026430 con 0x561c9f638e20 2024-05-10T11:14:31.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.265+0000 7f2f3d600700 1 -- 172.21.15.27:0/38415160 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f3801d8c0 con 0x561c9f638e20 2024-05-10T11:14:31.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.265+0000 7f2f3d600700 1 -- 172.21.15.27:0/38415160 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38025910 con 0x561c9f638e20 2024-05-10T11:14:31.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.266+0000 7f2f59526b80 1 -- 172.21.15.27:0/38415160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 msgr2=0x561c9f639200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:31.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.266+0000 7f2f59526b80 1 --2- 172.21.15.27:0/38415160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 0x561c9f639200 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800e5e0 tx=0x7f2f38014000 comp rx=0 tx=0).stop 2024-05-10T11:14:31.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.266+0000 7f2f59526b80 1 -- 172.21.15.27:0/38415160 shutdown_connections 2024-05-10T11:14:31.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.266+0000 7f2f59526b80 1 --2- 172.21.15.27:0/38415160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 0x561c9f639200 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.266+0000 7f2f59526b80 1 --2- 172.21.15.27:0/38415160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 0x561c9f63db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.266+0000 7f2f59526b80 1 --2- 172.21.15.27:0/38415160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 0x561c9f63d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.267+0000 7f2f59526b80 1 -- 172.21.15.27:0/38415160 >> 172.21.15.27:0/38415160 conn(0x561c9f5a86d0 msgr2=0x561c9f62aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:31.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.267+0000 7f2f59526b80 1 -- 172.21.15.27:0/38415160 shutdown_connections 2024-05-10T11:14:31.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.267+0000 7f2f59526b80 1 -- 172.21.15.27:0/38415160 wait complete. 2024-05-10T11:14:31.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.267+0000 7f2f59526b80 1 -- start start 2024-05-10T11:14:31.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.268+0000 7f2f59526b80 1 Processor -- start 2024-05-10T11:14:31.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.269+0000 7f2f59526b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f638e20 0x561c9f644aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.269+0000 7f2f59526b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 0x561c9f641eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.269+0000 7f2f59526b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f63d6c0 0x561c9f6423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.269+0000 7f2f59526b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561c9f645940 con 0x561c9f639740 2024-05-10T11:14:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.269+0000 7f2f59526b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561c9f4e8b20 con 0x561c9f638e20 2024-05-10T11:14:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.269+0000 7f2f59526b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561c9f645ac0 con 0x561c9f63d6c0 2024-05-10T11:14:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.269+0000 7f2f59526b80 10 client.0 _ll_register_callbacks cb 0x561c9f529c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.269+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f638e20 0x561c9f644aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.269+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f638e20 0x561c9f644aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52928/0 (socket says 172.21.15.27:52928) 2024-05-10T11:14:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.269+0000 7f2f47e00700 1 -- 172.21.15.27:0/1274249010 learned_addr learned my addr 172.21.15.27:0/1274249010 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:31.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.269+0000 7f2f47400700 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 0x561c9f641eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.269+0000 7f2f4cc00700 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f63d6c0 0x561c9f6423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.270+0000 7f2f47e00700 1 -- 172.21.15.27:0/1274249010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f63d6c0 msgr2=0x561c9f6423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:31.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.270+0000 7f2f47e00700 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f63d6c0 0x561c9f6423f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.270+0000 7f2f47e00700 1 -- 172.21.15.27:0/1274249010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 msgr2=0x561c9f641eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:31.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.270+0000 7f2f47e00700 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 0x561c9f641eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.270+0000 7f2f47e00700 1 -- 172.21.15.27:0/1274249010 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c9f548420 con 0x561c9f638e20 2024-05-10T11:14:31.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.270+0000 7f2f47e00700 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f638e20 0x561c9f644aa0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800ee30 tx=0x7f2f3800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19d04e4d0e49adb server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:31.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.270+0000 7f2f32000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:14:31.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.271+0000 7f2f32000700 1 -- 172.21.15.27:0/1274249010 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3801da70 con 0x561c9f638e20 2024-05-10T11:14:31.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.271+0000 7f2f32000700 1 -- 172.21.15.27:0/1274249010 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f2f3801dc10 con 0x561c9f638e20 2024-05-10T11:14:31.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.271+0000 7f2f59526b80 1 -- 172.21.15.27:0/1274249010 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f3801d8c0 con 0x561c9f638e20 2024-05-10T11:14:31.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.271+0000 7f2f32000700 1 -- 172.21.15.27:0/1274249010 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2f3801de10 con 0x561c9f638e20 2024-05-10T11:14:31.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.271+0000 7f2f59526b80 10 client.25514 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:31.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.271+0000 7f2f59526b80 1 -- 172.21.15.27:0/1274249010 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561c9f650140 con 0x561c9f638e20 2024-05-10T11:14:31.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.272+0000 7f2f32000700 1 -- 172.21.15.27:0/1274249010 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f3801d8c0 con 0x561c9f638e20 2024-05-10T11:14:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.272+0000 7f2f32000700 1 -- 172.21.15.27:0/1274249010 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f2f38004cb0 con 0x561c9f638e20 2024-05-10T11:14:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.272+0000 7f2f59526b80 10 client.25514 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.272+0000 7f2f59526b80 10 client.25514 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:14:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.272+0000 7f2f59526b80 20 client.25514 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.272+0000 7f2f59526b80 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561c9f652610 0x561c9f654a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.272+0000 7f2f59526b80 4 client.25514 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:14:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.272+0000 7f2f59526b80 1 -- 172.21.15.27:0/1274249010 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c9f5fd140 con 0x561c9f652610 2024-05-10T11:14:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.272+0000 7f2f47400700 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561c9f652610 0x561c9f654a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.273+0000 7f2f47400700 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561c9f652610 0x561c9f654a00 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=79a7a558f4141bc2 server_cookie=fb0b409d535519ac in_seq=0 out_seq=0 2024-05-10T11:14:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.273+0000 7f2f32000700 10 client.25514 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:14:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f32000700 1 -- 172.21.15.27:0/1274249010 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561c9f5fd140 con 0x561c9f652610 2024-05-10T11:14:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f32000700 10 client.25514 handle_command_reply: tid=0 2024-05-10T11:14:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f59526b80 1 client.25514 shutdown 2024-05-10T11:14:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f59526b80 1 -- 172.21.15.27:0/1274249010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f638e20 msgr2=0x561c9f644aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f59526b80 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f638e20 0x561c9f644aa0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800ee30 tx=0x7f2f3800ee60 comp rx=0 tx=0).stop 2024-05-10T11:14:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f59526b80 1 -- 172.21.15.27:0/1274249010 shutdown_connections 2024-05-10T11:14:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f59526b80 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561c9f652610 0x561c9f654a00 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f59526b80 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f63d6c0 0x561c9f6423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f59526b80 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f638e20 0x561c9f644aa0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f59526b80 1 --2- 172.21.15.27:0/1274249010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 0x561c9f641eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f32000700 0 client.25514 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:14:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f59526b80 1 -- 172.21.15.27:0/1274249010 >> 172.21.15.27:0/1274249010 conn(0x561c9f5a86d0 msgr2=0x561c9f63eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f59526b80 1 -- 172.21.15.27:0/1274249010 shutdown_connections 2024-05-10T11:14:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.274+0000 7f2f59526b80 1 -- 172.21.15.27:0/1274249010 wait complete. 2024-05-10T11:14:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.275+0000 7f2f59526b80 20 client.25514 trim_cache size 0 max 16384 2024-05-10T11:14:31.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.292+0000 7f2f59526b80 1 Processor -- start 2024-05-10T11:14:31.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.292+0000 7f2f59526b80 1 -- start start 2024-05-10T11:14:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.292+0000 7f2f59526b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f638e20 0x561c9f63daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.292+0000 7f2f59526b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f639740 0x561c9f6417d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.292+0000 7f2f59526b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 0x561c9f657a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.292+0000 7f2f59526b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481ad3f0 con 0x561c9f638e20 2024-05-10T11:14:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.292+0000 7f2f59526b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481aef30 con 0x561c9f63d6c0 2024-05-10T11:14:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.292+0000 7f2f59526b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481adac0 con 0x561c9f639740 2024-05-10T11:14:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.292+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f638e20 0x561c9f63daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.293+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f638e20 0x561c9f63daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38928/0 (socket says 172.21.15.27:38928) 2024-05-10T11:14:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.293+0000 7f2f47e00700 1 -- 172.21.15.27:0/4203278806 learned_addr learned my addr 172.21.15.27:0/4203278806 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.293+0000 7f2f47400700 1 --2- 172.21.15.27:0/4203278806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f639740 0x561c9f6417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.293+0000 7f2f4cc00700 1 --2- 172.21.15.27:0/4203278806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 0x561c9f657a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.293+0000 7f2f47400700 1 -- 172.21.15.27:0/4203278806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 msgr2=0x561c9f657a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.293+0000 7f2f47400700 1 --2- 172.21.15.27:0/4203278806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 0x561c9f657a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.293+0000 7f2f47400700 1 -- 172.21.15.27:0/4203278806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f638e20 msgr2=0x561c9f63daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:31.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.293+0000 7f2f47400700 1 --2- 172.21.15.27:0/4203278806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f638e20 0x561c9f63daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.293+0000 7f2f47400700 1 -- 172.21.15.27:0/4203278806 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c9f6278a0 con 0x561c9f639740 2024-05-10T11:14:31.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.293+0000 7f2f47400700 1 --2- 172.21.15.27:0/4203278806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f639740 0x561c9f6417d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2f34000c00 tx=0x7f2f3400bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a914a70e718aac0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:31.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.294+0000 7f2f3cc00700 1 -- 172.21.15.27:0/4203278806 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3400f980 con 0x561c9f639740 2024-05-10T11:14:31.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.294+0000 7f2f3cc00700 1 -- 172.21.15.27:0/4203278806 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f3402d070 con 0x561c9f639740 2024-05-10T11:14:31.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.294+0000 7f2f3cc00700 1 -- 172.21.15.27:0/4203278806 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34003b90 con 0x561c9f639740 2024-05-10T11:14:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.294+0000 7f2f59526b80 1 -- 172.21.15.27:0/4203278806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f639740 msgr2=0x561c9f6417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.294+0000 7f2f59526b80 1 --2- 172.21.15.27:0/4203278806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f639740 0x561c9f6417d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2f34000c00 tx=0x7f2f3400bf60 comp rx=0 tx=0).stop 2024-05-10T11:14:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.294+0000 7f2f59526b80 1 -- 172.21.15.27:0/4203278806 shutdown_connections 2024-05-10T11:14:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.294+0000 7f2f59526b80 1 --2- 172.21.15.27:0/4203278806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 0x561c9f657a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.294+0000 7f2f59526b80 1 --2- 172.21.15.27:0/4203278806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f638e20 0x561c9f63daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.294+0000 7f2f59526b80 1 --2- 172.21.15.27:0/4203278806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f639740 0x561c9f6417d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.294+0000 7f2f59526b80 1 -- 172.21.15.27:0/4203278806 >> 172.21.15.27:0/4203278806 conn(0x561c9f5a86d0 msgr2=0x561c9f62fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.295+0000 7f2f59526b80 1 -- 172.21.15.27:0/4203278806 shutdown_connections 2024-05-10T11:14:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.295+0000 7f2f59526b80 1 -- 172.21.15.27:0/4203278806 wait complete. 2024-05-10T11:14:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.295+0000 7f2f59526b80 1 -- start start 2024-05-10T11:14:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f59526b80 1 Processor -- start 2024-05-10T11:14:31.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f59526b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 0x561c9f658120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f59526b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 0x561c9f6554c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f59526b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 0x561c9f655c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f59526b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481ae860 con 0x561c9f639740 2024-05-10T11:14:31.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f59526b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481ae190 con 0x561c9f63d6c0 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f59526b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481ab390 con 0x561c9f638e20 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f59526b80 10 client.0 _ll_register_callbacks cb 0x561c9f529c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 0x561c9f658120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f47400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 0x561c9f6554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 0x561c9f658120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42024/0 (socket says 172.21.15.27:42024) 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f47e00700 1 -- 172.21.15.27:0/609033390 learned_addr learned my addr 172.21.15.27:0/609033390 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.296+0000 7f2f4cc00700 1 --2- 172.21.15.27:0/609033390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 0x561c9f655c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.297+0000 7f2f47e00700 1 -- 172.21.15.27:0/609033390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 msgr2=0x561c9f655c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.297+0000 7f2f47e00700 1 --2- 172.21.15.27:0/609033390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 0x561c9f655c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.297+0000 7f2f47e00700 1 -- 172.21.15.27:0/609033390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 msgr2=0x561c9f6554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.297+0000 7f2f47e00700 1 --2- 172.21.15.27:0/609033390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 0x561c9f6554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.297+0000 7f2f47e00700 1 -- 172.21.15.27:0/609033390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c9f6bbb30 con 0x561c9f638e20 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.297+0000 7f2f47e00700 1 --2- 172.21.15.27:0/609033390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 0x561c9f658120 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2f38025600 tx=0x7f2f38025d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6befb9aadf512bb server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.297+0000 7f2f32000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:14:31.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.297+0000 7f2f32000700 1 -- 172.21.15.27:0/609033390 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3801d410 con 0x561c9f638e20 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.298+0000 7f2f32000700 1 -- 172.21.15.27:0/609033390 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f2f3801d9f0 con 0x561c9f638e20 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.298+0000 7f2f59526b80 1 -- 172.21.15.27:0/609033390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f3402d070 con 0x561c9f638e20 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.298+0000 7f2f32000700 1 -- 172.21.15.27:0/609033390 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2f3800ec80 con 0x561c9f638e20 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.298+0000 7f2f59526b80 10 client.25513 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.298+0000 7f2f59526b80 1 -- 172.21.15.27:0/609033390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561c9f6bcd10 con 0x561c9f638e20 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.298+0000 7f2f32000700 1 -- 172.21.15.27:0/609033390 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f3402d070 con 0x561c9f638e20 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.299+0000 7f2f32000700 1 -- 172.21.15.27:0/609033390 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f2f38005030 con 0x561c9f638e20 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.299+0000 7f2f59526b80 10 client.25513 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.299+0000 7f2f59526b80 10 client.25513 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.299+0000 7f2f59526b80 20 client.25513 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.299+0000 7f2f59526b80 1 --2- 172.21.15.27:0/609033390 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561c9f6bd390 0x561c9f6bf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.299+0000 7f2f59526b80 4 client.25513 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.299+0000 7f2f59526b80 1 -- 172.21.15.27:0/609033390 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561c9f590400 con 0x561c9f6bd390 2024-05-10T11:14:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.299+0000 7f2f47400700 1 --2- 172.21.15.27:0/609033390 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561c9f6bd390 0x561c9f6bf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:31.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f47400700 1 --2- 172.21.15.27:0/609033390 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561c9f6bd390 0x561c9f6bf7b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ddedde5ddf8fd54e server_cookie=37ddd453a9c718e3 in_seq=0 out_seq=0 2024-05-10T11:14:31.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f32000700 10 client.25513 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:14:31.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f32000700 1 -- 172.21.15.27:0/609033390 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561c9f590400 con 0x561c9f6bd390 2024-05-10T11:14:31.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f32000700 10 client.25513 handle_command_reply: tid=0 2024-05-10T11:14:31.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f59526b80 1 client.25513 shutdown 2024-05-10T11:14:31.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f59526b80 1 -- 172.21.15.27:0/609033390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 msgr2=0x561c9f658120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:31.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f59526b80 1 --2- 172.21.15.27:0/609033390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 0x561c9f658120 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2f38025600 tx=0x7f2f38025d50 comp rx=0 tx=0).stop 2024-05-10T11:14:31.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f59526b80 1 -- 172.21.15.27:0/609033390 shutdown_connections 2024-05-10T11:14:31.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f59526b80 1 --2- 172.21.15.27:0/609033390 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561c9f6bd390 0x561c9f6bf7b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f59526b80 1 --2- 172.21.15.27:0/609033390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c9f638e20 0x561c9f658120 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f59526b80 1 --2- 172.21.15.27:0/609033390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561c9f63d6c0 0x561c9f655c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f59526b80 1 --2- 172.21.15.27:0/609033390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561c9f639740 0x561c9f6554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f59526b80 1 -- 172.21.15.27:0/609033390 >> 172.21.15.27:0/609033390 conn(0x561c9f5a86d0 msgr2=0x561c9f62fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:31.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f32000700 0 client.25513 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:14:31.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.300+0000 7f2f59526b80 1 -- 172.21.15.27:0/609033390 shutdown_connections 2024-05-10T11:14:31.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.301+0000 7f2f59526b80 1 -- 172.21.15.27:0/609033390 wait complete. 2024-05-10T11:14:31.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.301+0000 7f2f59526b80 20 client.25513 trim_cache size 0 max 16384 2024-05-10T11:14:31.303 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:14:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.301+0000 7f2f4d600700 1 -- 172.21.15.27:0/103372653 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2f2c0b2c20 msgr2=0x7f2f2c0b50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.301+0000 7f2f4d600700 1 --2- 172.21.15.27:0/103372653 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2f2c0b2c20 0x7f2f2c0b50e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2f38005d10 tx=0x7f2f38009310 comp rx=0 tx=0).stop 2024-05-10T11:14:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.301+0000 7f2f4d600700 1 -- 172.21.15.27:0/103372653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f48107660 msgr2=0x7f2f4819e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.301+0000 7f2f4d600700 1 --2- 172.21.15.27:0/103372653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f48107660 0x7f2f4819e540 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000b330 tx=0x7f2f4000b640 comp rx=0 tx=0).stop 2024-05-10T11:14:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.301+0000 7f2f4d600700 1 -- 172.21.15.27:0/103372653 shutdown_connections 2024-05-10T11:14:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.301+0000 7f2f4d600700 1 --2- 172.21.15.27:0/103372653 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2f2c0b2c20 0x7f2f2c0b50e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.301+0000 7f2f4d600700 1 --2- 172.21.15.27:0/103372653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f48102580 0x7f2f48199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.301+0000 7f2f4d600700 1 --2- 172.21.15.27:0/103372653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f48107660 0x7f2f4819e540 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.301+0000 7f2f4d600700 1 --2- 172.21.15.27:0/103372653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f48102f30 0x7f2f4819a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.301+0000 7f2f4d600700 1 -- 172.21.15.27:0/103372653 >> 172.21.15.27:0/103372653 conn(0x7f2f480fde30 msgr2=0x7f2f4806bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.302+0000 7f2f4d600700 1 -- 172.21.15.27:0/103372653 shutdown_connections 2024-05-10T11:14:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:31.302+0000 7f2f4d600700 1 -- 172.21.15.27:0/103372653 wait complete. 2024-05-10T11:14:32.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:14:32.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:32 smithi027 ceph-mon[30054]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-10T11:14:32.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:32 smithi076 ceph-mon[34152]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-10T11:14:32.729 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:14:32.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:32 smithi189 ceph-mon[34842]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-10T11:14:33.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:33 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:14:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:33 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:14:33.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:33 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:14:34.498 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:34 smithi027 ceph-mon[30054]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:34.499 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:34 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:14:34.499 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:34 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:14:34.499 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:34 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:14:34.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:34 smithi076 ceph-mon[34152]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:34.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:34 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:14:34.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:34 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:14:34.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:34 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:14:34.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:34 smithi189 ceph-mon[34842]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:34.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:34 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:14:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:34 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:14:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:34 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:14:35.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:35 smithi027 ceph-mon[30054]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-10T11:14:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:35 smithi076 ceph-mon[34152]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-10T11:14:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:35 smithi189 ceph-mon[34842]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-10T11:14:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.035+0000 7f68ab400700 1 -- 172.21.15.27:0/1103606177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a406bb40 msgr2=0x7f68a4110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.035+0000 7f68ab400700 1 --2- 172.21.15.27:0/1103606177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a406bb40 0x7f68a4110330 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f689c00b0a0 tx=0x7f689c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.035+0000 7f68ab400700 1 -- 172.21.15.27:0/1103606177 shutdown_connections 2024-05-10T11:14:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.035+0000 7f68ab400700 1 --2- 172.21.15.27:0/1103606177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a4109510 0x7f68a41098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.035+0000 7f68ab400700 1 --2- 172.21.15.27:0/1103606177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a406bb40 0x7f68a4110330 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.035+0000 7f68ab400700 1 --2- 172.21.15.27:0/1103606177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68a406b180 0x7f68a406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.035+0000 7f68ab400700 1 -- 172.21.15.27:0/1103606177 >> 172.21.15.27:0/1103606177 conn(0x7f68a4079350 msgr2=0x7f68a4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.036+0000 7f68ab400700 1 -- 172.21.15.27:0/1103606177 shutdown_connections 2024-05-10T11:14:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.037+0000 7f68ab400700 1 -- 172.21.15.27:0/1103606177 wait complete. 2024-05-10T11:14:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.038+0000 7f68ab400700 1 Processor -- start 2024-05-10T11:14:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.039+0000 7f68ab400700 1 -- start start 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.039+0000 7f68ab400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a406b180 0x7f68a419f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.039+0000 7f68ab400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a406bb40 0x7f68a419fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.039+0000 7f68ab400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68a4109510 0x7f68a41999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.039+0000 7f68ab400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f68a4113690 con 0x7f68a406bb40 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.039+0000 7f68ab400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f68a4113510 con 0x7f68a4109510 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.039+0000 7f68ab400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f68a4113810 con 0x7f68a406b180 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.040+0000 7f68a8a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a406b180 0x7f68a419f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.040+0000 7f68a3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a406bb40 0x7f68a419fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.040+0000 7f68a8a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a406b180 0x7f68a419f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50808/0 (socket says 172.21.15.27:50808) 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.040+0000 7f68a8a00700 1 -- 172.21.15.27:0/1367636400 learned_addr learned my addr 172.21.15.27:0/1367636400 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.040+0000 7f68a9400700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68a4109510 0x7f68a41999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.040+0000 7f68a3e00700 1 -- 172.21.15.27:0/1367636400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a406b180 msgr2=0x7f68a419f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.040+0000 7f68a3e00700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a406b180 0x7f68a419f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.040+0000 7f68a3e00700 1 -- 172.21.15.27:0/1367636400 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68a4109510 msgr2=0x7f68a41999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.040+0000 7f68a3e00700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68a4109510 0x7f68a41999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.040+0000 7f68a3e00700 1 -- 172.21.15.27:0/1367636400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a419a150 con 0x7f68a406bb40 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.040+0000 7f68a9400700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68a4109510 0x7f68a41999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.041+0000 7f68a3e00700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a406bb40 0x7f68a419fd70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6890007c90 tx=0x7f6890011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.041+0000 7f68a1600700 1 -- 172.21.15.27:0/1367636400 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6890011500 con 0x7f68a406bb40 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.041+0000 7f68a1600700 1 -- 172.21.15.27:0/1367636400 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68900044d0 con 0x7f68a406bb40 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.041+0000 7f68a1600700 1 -- 172.21.15.27:0/1367636400 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689000b9f0 con 0x7f68a406bb40 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.042+0000 7f68ab400700 1 -- 172.21.15.27:0/1367636400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a419a440 con 0x7f68a406bb40 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.042+0000 7f68ab400700 1 -- 172.21.15.27:0/1367636400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a41023a0 con 0x7f68a406bb40 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.046+0000 7f68a1600700 1 -- 172.21.15.27:0/1367636400 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f689000bc50 con 0x7f68a406bb40 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.047+0000 7f68b5e12b80 1 Processor -- start 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.047+0000 7f68b5e12b80 1 -- start start 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.047+0000 7f68b5e12b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37f2bc0 0x55e3b37f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.047+0000 7f68b5e12b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f34e0 0x55e3b37ed2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.047+0000 7f68b5e12b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37ed8f0 0x55e3b37edd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.047+0000 7f68b5e12b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e3b373fa20 con 0x55e3b37f2bc0 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.047+0000 7f68b5e12b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55e3b36dd760 con 0x55e3b37f34e0 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.047+0000 7f68b5e12b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e3b373b9e0 con 0x55e3b37ed8f0 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.048+0000 7f68a8a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f34e0 0x55e3b37ed2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.048+0000 7f68a8a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f34e0 0x55e3b37ed2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48148/0 (socket says 172.21.15.27:48148) 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.048+0000 7f68a8a00700 1 -- 172.21.15.27:0/4151937149 learned_addr learned my addr 172.21.15.27:0/4151937149 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.048+0000 7f68a3e00700 1 --2- 172.21.15.27:0/4151937149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37ed8f0 0x55e3b37edd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.048+0000 7f68a9400700 1 --2- 172.21.15.27:0/4151937149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37f2bc0 0x55e3b37f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.048+0000 7f68a8a00700 1 -- 172.21.15.27:0/4151937149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37ed8f0 msgr2=0x55e3b37edd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.048+0000 7f68a8a00700 1 --2- 172.21.15.27:0/4151937149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37ed8f0 0x55e3b37edd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.048+0000 7f68a8a00700 1 -- 172.21.15.27:0/4151937149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37f2bc0 msgr2=0x55e3b37f2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.048+0000 7f68a8a00700 1 --2- 172.21.15.27:0/4151937149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37f2bc0 0x55e3b37f2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.049+0000 7f68a8a00700 1 -- 172.21.15.27:0/4151937149 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3b368c8d0 con 0x55e3b37f34e0 2024-05-10T11:14:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.049+0000 7f68a9400700 1 --2- 172.21.15.27:0/4151937149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37f2bc0 0x55e3b37f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.049+0000 7f68a8a00700 1 --2- 172.21.15.27:0/4151937149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f34e0 0x55e3b37ed2d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6898009d90 tx=0x7f6898011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5690df9d284f9b6c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.049+0000 7f6896000700 1 -- 172.21.15.27:0/4151937149 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689801e070 con 0x55e3b37f34e0 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.049+0000 7f68a1600700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f688c0b2c20 0x7f688c0b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.049+0000 7f6896000700 1 -- 172.21.15.27:0/4151937149 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6898019ae0 con 0x55e3b37f34e0 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.049+0000 7f6896000700 1 -- 172.21.15.27:0/4151937149 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689800fac0 con 0x55e3b37f34e0 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.050+0000 7f68a1600700 1 -- 172.21.15.27:0/1367636400 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f68900e65f0 con 0x7f68a406bb40 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.050+0000 7f68a9400700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f688c0b2c20 0x7f688c0b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.050+0000 7f68b5e12b80 1 -- 172.21.15.27:0/4151937149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f34e0 msgr2=0x55e3b37ed2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.050+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/4151937149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f34e0 0x55e3b37ed2d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6898009d90 tx=0x7f6898011720 comp rx=0 tx=0).stop 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.051+0000 7f68b5e12b80 1 -- 172.21.15.27:0/4151937149 shutdown_connections 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.051+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/4151937149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37ed8f0 0x55e3b37edd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.051+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/4151937149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37f2bc0 0x55e3b37f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.051+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/4151937149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f34e0 0x55e3b37ed2d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.051+0000 7f68b5e12b80 1 -- 172.21.15.27:0/4151937149 >> 172.21.15.27:0/4151937149 conn(0x55e3b37656d0 msgr2=0x55e3b37e7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.051+0000 7f68a9400700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f688c0b2c20 0x7f688c0b50e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f689c00b780 tx=0x7f689c01d040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.051+0000 7f68b5e12b80 1 -- 172.21.15.27:0/4151937149 shutdown_connections 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.051+0000 7f68b5e12b80 1 -- 172.21.15.27:0/4151937149 wait complete. 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.052+0000 7f68b5e12b80 1 -- start start 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.052+0000 7f68b5e12b80 1 Processor -- start 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.053+0000 7f68b5e12b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 0x55e3b37f9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.053+0000 7f68b5e12b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 0x55e3b37f6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.053+0000 7f68b5e12b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f34e0 0x55e3b37f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.053+0000 7f68b5e12b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e3b37f9f20 con 0x55e3b37ed8f0 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.053+0000 7f68b5e12b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55e3b36a5b20 con 0x55e3b37f2bc0 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.053+0000 7f68b5e12b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e3b37fa0a0 con 0x55e3b37f34e0 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.053+0000 7f68b5e12b80 10 client.0 _ll_register_callbacks cb 0x55e3b36e6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.054+0000 7f68a3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 0x55e3b37f6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.054+0000 7f68a8a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 0x55e3b37f9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.054+0000 7f68a9400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f34e0 0x55e3b37f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.054+0000 7f68a3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 0x55e3b37f6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48156/0 (socket says 172.21.15.27:48156) 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.054+0000 7f68a8a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 0x55e3b37f9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59082/0 (socket says 172.21.15.27:59082) 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.054+0000 7f68a3e00700 1 -- 172.21.15.27:0/2392269993 learned_addr learned my addr 172.21.15.27:0/2392269993 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.054+0000 7f68a3e00700 1 -- 172.21.15.27:0/2392269993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f34e0 msgr2=0x55e3b37f6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.054+0000 7f68a3e00700 1 --2- 172.21.15.27:0/2392269993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f34e0 0x55e3b37f6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.054+0000 7f68a3e00700 1 -- 172.21.15.27:0/2392269993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 msgr2=0x55e3b37f9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.054+0000 7f68a3e00700 1 --2- 172.21.15.27:0/2392269993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 0x55e3b37f9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.054+0000 7f68a3e00700 1 -- 172.21.15.27:0/2392269993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3b3705420 con 0x55e3b37f2bc0 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.054+0000 7f68a9400700 1 --2- 172.21.15.27:0/2392269993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f34e0 0x55e3b37f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.055+0000 7f68a3e00700 1 --2- 172.21.15.27:0/2392269993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 0x55e3b37f6520 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6890007c60 tx=0x7f68900116a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de9ff59f34f11374 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.055+0000 7f688aa00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.055+0000 7f688aa00700 1 -- 172.21.15.27:0/2392269993 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6890020040 con 0x55e3b37f2bc0 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.055+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2392269993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6898019ae0 con 0x55e3b37f2bc0 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.055+0000 7f688aa00700 1 -- 172.21.15.27:0/2392269993 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f68900e63d0 con 0x55e3b37f2bc0 2024-05-10T11:14:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.055+0000 7f688aa00700 1 -- 172.21.15.27:0/2392269993 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f68900c7050 con 0x55e3b37f2bc0 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.055+0000 7f68b5e12b80 10 client.25532 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.055+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2392269993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e3b38045e0 con 0x55e3b37f2bc0 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.056+0000 7f688aa00700 1 -- 172.21.15.27:0/2392269993 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6898019ae0 con 0x55e3b37f2bc0 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.056+0000 7f688aa00700 1 -- 172.21.15.27:0/2392269993 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f68900b5c90 con 0x55e3b37f2bc0 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.056+0000 7f68b5e12b80 10 client.25532 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.056+0000 7f68b5e12b80 10 client.25532 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.056+0000 7f68b5e12b80 20 client.25532 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.056+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/2392269993 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e3b3806b70 0x55e3b3808f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.056+0000 7f68b5e12b80 4 client.25532 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.056+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2392269993 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3b37ba140 con 0x55e3b3806b70 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.057+0000 7f68a8a00700 1 --2- 172.21.15.27:0/2392269993 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e3b3806b70 0x55e3b3808f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.057+0000 7f68a8a00700 1 --2- 172.21.15.27:0/2392269993 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e3b3806b70 0x55e3b3808f60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=824ddb4301b6fb92 server_cookie=9813354449d1dad6 in_seq=0 out_seq=0 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.057+0000 7f688aa00700 10 client.25532 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.058+0000 7f688aa00700 1 -- 172.21.15.27:0/2392269993 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e3b37ba140 con 0x55e3b3806b70 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.058+0000 7f688aa00700 10 client.25532 handle_command_reply: tid=0 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.058+0000 7f68b5e12b80 1 client.25532 shutdown 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.058+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2392269993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 msgr2=0x55e3b37f6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.058+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/2392269993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 0x55e3b37f6520 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6890007c60 tx=0x7f68900116a0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.059+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2392269993 shutdown_connections 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.059+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/2392269993 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e3b3806b70 0x55e3b3808f60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.059+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/2392269993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f34e0 0x55e3b37f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.059+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/2392269993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 0x55e3b37f9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.059+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/2392269993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 0x55e3b37f6520 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.059+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2392269993 >> 172.21.15.27:0/2392269993 conn(0x55e3b37656d0 msgr2=0x55e3b37e7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.059+0000 7f688aa00700 0 client.25532 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.059+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2392269993 shutdown_connections 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.059+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2392269993 wait complete. 2024-05-10T11:14:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.059+0000 7f68b5e12b80 20 client.25532 trim_cache size 0 max 16384 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.081+0000 7f68b5e12b80 1 Processor -- start 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.081+0000 7f68b5e12b80 1 -- start start 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.082+0000 7f68b5e12b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37ed8f0 0x55e3b37ed2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.082+0000 7f68b5e12b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f2bc0 0x55e3b3802fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.082+0000 7f68b5e12b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37f34e0 0x55e3b37e7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.082+0000 7f68b5e12b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41ad380 con 0x55e3b37f34e0 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.082+0000 7f68b5e12b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41aeec0 con 0x55e3b37ed8f0 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.082+0000 7f68b5e12b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41ada50 con 0x55e3b37f2bc0 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.082+0000 7f68a8a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37ed8f0 0x55e3b37ed2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.082+0000 7f68a8a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37ed8f0 0x55e3b37ed2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48158/0 (socket says 172.21.15.27:48158) 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.082+0000 7f68a8a00700 1 -- 172.21.15.27:0/2849848762 learned_addr learned my addr 172.21.15.27:0/2849848762 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.083+0000 7f68a3e00700 1 --2- 172.21.15.27:0/2849848762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f2bc0 0x55e3b3802fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.083+0000 7f68a9400700 1 --2- 172.21.15.27:0/2849848762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37f34e0 0x55e3b37e7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.083+0000 7f68a8a00700 1 -- 172.21.15.27:0/2849848762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f2bc0 msgr2=0x55e3b3802fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.083+0000 7f68a8a00700 1 --2- 172.21.15.27:0/2849848762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f2bc0 0x55e3b3802fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.083+0000 7f68a8a00700 1 -- 172.21.15.27:0/2849848762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37f34e0 msgr2=0x55e3b37e7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.083+0000 7f68a8a00700 1 --2- 172.21.15.27:0/2849848762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37f34e0 0x55e3b37e7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.083+0000 7f68a8a00700 1 -- 172.21.15.27:0/2849848762 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3b37f62f0 con 0x55e3b37ed8f0 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.083+0000 7f68a9400700 1 --2- 172.21.15.27:0/2849848762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37f34e0 0x55e3b37e7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.083+0000 7f68a8a00700 1 --2- 172.21.15.27:0/2849848762 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37ed8f0 0x55e3b37ed2d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6898009d90 tx=0x7f689800f600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbc5a696eb670ca server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.084+0000 7f6895600700 1 -- 172.21.15.27:0/2849848762 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689801e040 con 0x55e3b37ed8f0 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.084+0000 7f6895600700 1 -- 172.21.15.27:0/2849848762 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f689801f040 con 0x55e3b37ed8f0 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.084+0000 7f6895600700 1 -- 172.21.15.27:0/2849848762 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689800f970 con 0x55e3b37ed8f0 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.085+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2849848762 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37ed8f0 msgr2=0x55e3b37ed2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.085+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/2849848762 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37ed8f0 0x55e3b37ed2d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6898009d90 tx=0x7f689800f600 comp rx=0 tx=0).stop 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.086+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2849848762 shutdown_connections 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.086+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/2849848762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37f34e0 0x55e3b37e7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.086+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/2849848762 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37ed8f0 0x55e3b37ed2d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.086+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/2849848762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f2bc0 0x55e3b3802fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.086+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2849848762 >> 172.21.15.27:0/2849848762 conn(0x55e3b37656d0 msgr2=0x55e3b37f4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.086+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2849848762 shutdown_connections 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.086+0000 7f68b5e12b80 1 -- 172.21.15.27:0/2849848762 wait complete. 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.086+0000 7f68b5e12b80 1 -- start start 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.087+0000 7f68b5e12b80 1 Processor -- start 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.087+0000 7f68b5e12b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 0x55e3b37ff330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.087+0000 7f68b5e12b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 0x55e3b37ff870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.087+0000 7f68b5e12b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f34e0 0x55e3b380d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.087+0000 7f68b5e12b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41ae7f0 con 0x55e3b37ed8f0 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.087+0000 7f68b5e12b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41ae120 con 0x55e3b37f2bc0 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.087+0000 7f68b5e12b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41ab320 con 0x55e3b37f34e0 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.087+0000 7f68b5e12b80 10 client.0 _ll_register_callbacks cb 0x55e3b36e6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.088+0000 7f68a8a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 0x55e3b37ff330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.088+0000 7f68a8a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 0x55e3b37ff330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59094/0 (socket says 172.21.15.27:59094) 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.088+0000 7f68a8a00700 1 -- 172.21.15.27:0/1517128524 learned_addr learned my addr 172.21.15.27:0/1517128524 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.088+0000 7f68a9400700 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f34e0 0x55e3b380d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.089+0000 7f68a8a00700 1 -- 172.21.15.27:0/1517128524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f34e0 msgr2=0x55e3b380d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.089+0000 7f68a3e00700 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 0x55e3b37ff870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.089+0000 7f68a8a00700 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f34e0 0x55e3b380d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.089+0000 7f68a8a00700 1 -- 172.21.15.27:0/1517128524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 msgr2=0x55e3b37ff870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.089+0000 7f68a8a00700 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 0x55e3b37ff870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.089+0000 7f68a8a00700 1 -- 172.21.15.27:0/1517128524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3b380dd10 con 0x55e3b37ed8f0 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.089+0000 7f68a3e00700 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 0x55e3b37ff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.089+0000 7f68a8a00700 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 0x55e3b37ff330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f689800f5d0 tx=0x7f6898009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390241f4a269344 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.090+0000 7f688aa00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.090+0000 7f688aa00700 1 -- 172.21.15.27:0/1517128524 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689801e040 con 0x55e3b37ed8f0 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.090+0000 7f688aa00700 1 -- 172.21.15.27:0/1517128524 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f6898004910 con 0x55e3b37ed8f0 2024-05-10T11:14:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.090+0000 7f68b5e12b80 1 -- 172.21.15.27:0/1517128524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f689801f040 con 0x55e3b37ed8f0 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.090+0000 7f688aa00700 1 -- 172.21.15.27:0/1517128524 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6898047050 con 0x55e3b37ed8f0 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.091+0000 7f68b5e12b80 10 client.15405 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.091+0000 7f68b5e12b80 1 -- 172.21.15.27:0/1517128524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e3b3870b20 con 0x55e3b37ed8f0 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.091+0000 7f688aa00700 1 -- 172.21.15.27:0/1517128524 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f689801f040 con 0x55e3b37ed8f0 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.091+0000 7f688aa00700 1 -- 172.21.15.27:0/1517128524 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f689800fcb0 con 0x55e3b37ed8f0 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.091+0000 7f68b5e12b80 10 client.15405 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.091+0000 7f68b5e12b80 10 client.15405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.091+0000 7f68b5e12b80 20 client.15405 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.091+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e3b3870ee0 0x55e3b38732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.091+0000 7f68b5e12b80 4 client.15405 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.091+0000 7f68b5e12b80 1 -- 172.21.15.27:0/1517128524 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e3b374d400 con 0x55e3b3870ee0 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.093+0000 7f68a3e00700 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e3b3870ee0 0x55e3b38732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f68a3e00700 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e3b3870ee0 0x55e3b38732d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f70abfbbe951357 server_cookie=99752acbd0c72bec in_seq=0 out_seq=0 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f688aa00700 10 client.15405 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f688aa00700 1 -- 172.21.15.27:0/1517128524 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e3b374d400 con 0x55e3b3870ee0 2024-05-10T11:14:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f688aa00700 10 client.15405 handle_command_reply: tid=0 2024-05-10T11:14:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f68b5e12b80 1 client.15405 shutdown 2024-05-10T11:14:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f68b5e12b80 1 -- 172.21.15.27:0/1517128524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 msgr2=0x55e3b37ff330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 0x55e3b37ff330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f689800f5d0 tx=0x7f6898009630 comp rx=0 tx=0).stop 2024-05-10T11:14:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f68b5e12b80 1 -- 172.21.15.27:0/1517128524 shutdown_connections 2024-05-10T11:14:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e3b3870ee0 0x55e3b38732d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e3b37f34e0 0x55e3b380d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3b37ed8f0 0x55e3b37ff330 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f688aa00700 0 client.15405 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f68b5e12b80 1 --2- 172.21.15.27:0/1517128524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e3b37f2bc0 0x55e3b37ff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f68b5e12b80 1 -- 172.21.15.27:0/1517128524 >> 172.21.15.27:0/1517128524 conn(0x55e3b37656d0 msgr2=0x55e3b37f38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:37.947 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:14:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f68b5e12b80 1 -- 172.21.15.27:0/1517128524 shutdown_connections 2024-05-10T11:14:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.094+0000 7f68b5e12b80 1 -- 172.21.15.27:0/1517128524 wait complete. 2024-05-10T11:14:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.095+0000 7f68b5e12b80 20 client.15405 trim_cache size 0 max 16384 2024-05-10T11:14:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.095+0000 7f68ab400700 1 -- 172.21.15.27:0/1367636400 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f688c0b2c20 msgr2=0x7f688c0b50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.095+0000 7f68ab400700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f688c0b2c20 0x7f688c0b50e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f689c00b780 tx=0x7f689c01d040 comp rx=0 tx=0).stop 2024-05-10T11:14:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.095+0000 7f68ab400700 1 -- 172.21.15.27:0/1367636400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a406bb40 msgr2=0x7f68a419fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.095+0000 7f68ab400700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a406bb40 0x7f68a419fd70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6890007c90 tx=0x7f6890011ed0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.096+0000 7f68ab400700 1 -- 172.21.15.27:0/1367636400 shutdown_connections 2024-05-10T11:14:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.096+0000 7f68ab400700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f688c0b2c20 0x7f688c0b50e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.096+0000 7f68ab400700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a406b180 0x7f68a419f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.096+0000 7f68ab400700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f68a4109510 0x7f68a41999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.096+0000 7f68ab400700 1 --2- 172.21.15.27:0/1367636400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a406bb40 0x7f68a419fd70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:37.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.096+0000 7f68ab400700 1 -- 172.21.15.27:0/1367636400 >> 172.21.15.27:0/1367636400 conn(0x7f68a4079350 msgr2=0x7f68a4101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:37.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.096+0000 7f68ab400700 1 -- 172.21.15.27:0/1367636400 shutdown_connections 2024-05-10T11:14:37.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:35.096+0000 7f68ab400700 1 -- 172.21.15.27:0/1367636400 wait complete. 2024-05-10T11:14:38.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:38 smithi027 ceph-mon[30054]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:38.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:38 smithi076 ceph-mon[34152]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:38 smithi189 ceph-mon[34842]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:39.832 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:14:39.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:14:40.247 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:40 smithi027 ceph-mon[30054]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-10T11:14:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:14:40.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:40 smithi076 ceph-mon[34152]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-10T11:14:40.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:40 smithi189 ceph-mon[34842]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-10T11:14:42.353 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:42 smithi027 ceph-mon[30054]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-10T11:14:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.654+0000 7fae85c00700 1 -- 172.21.15.27:0/2665358352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80104590 msgr2=0x7fae80104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.654+0000 7fae85c00700 1 --2- 172.21.15.27:0/2665358352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80104590 0x7fae80104970 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fae70009990 tx=0x7fae70009ca0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.654+0000 7fae85c00700 1 -- 172.21.15.27:0/2665358352 shutdown_connections 2024-05-10T11:14:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.654+0000 7fae85c00700 1 --2- 172.21.15.27:0/2665358352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80104590 0x7fae80104970 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.654+0000 7fae85c00700 1 --2- 172.21.15.27:0/2665358352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae8010d9c0 0x7fae8010fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.654+0000 7fae85c00700 1 --2- 172.21.15.27:0/2665358352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae80104eb0 0x7fae8010d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.654+0000 7fae85c00700 1 -- 172.21.15.27:0/2665358352 >> 172.21.15.27:0/2665358352 conn(0x7fae800fddc0 msgr2=0x7fae801001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:42.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.655+0000 7fae85c00700 1 -- 172.21.15.27:0/2665358352 shutdown_connections 2024-05-10T11:14:42.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.655+0000 7fae85c00700 1 -- 172.21.15.27:0/2665358352 wait complete. 2024-05-10T11:14:42.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.656+0000 7fae85c00700 1 Processor -- start 2024-05-10T11:14:42.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.656+0000 7fae85c00700 1 -- start start 2024-05-10T11:14:42.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.657+0000 7fae85c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae80104590 0x7fae8019bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.657+0000 7fae85c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80104eb0 0x7fae8019c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.657+0000 7fae85c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae8010d9c0 0x7fae801a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.657+0000 7fae85c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fae80113600 con 0x7fae8010d9c0 2024-05-10T11:14:42.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.657+0000 7fae85c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fae80113480 con 0x7fae80104590 2024-05-10T11:14:42.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.657+0000 7fae85c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fae80113780 con 0x7fae80104eb0 2024-05-10T11:14:42.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.657+0000 7fae7f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80104eb0 0x7fae8019c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.657+0000 7fae7f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80104eb0 0x7fae8019c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50864/0 (socket says 172.21.15.27:50864) 2024-05-10T11:14:42.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.657+0000 7fae7f400700 1 -- 172.21.15.27:0/2279267897 learned_addr learned my addr 172.21.15.27:0/2279267897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:42.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.657+0000 7fae85200700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae8010d9c0 0x7fae801a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.658+0000 7fae7f400700 1 -- 172.21.15.27:0/2279267897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae80104590 msgr2=0x7fae8019bd10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-10T11:14:42.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.658+0000 7fae7f400700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae80104590 0x7fae8019bd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.658+0000 7fae7f400700 1 -- 172.21.15.27:0/2279267897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae8010d9c0 msgr2=0x7fae801a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:42.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.658+0000 7fae7f400700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae8010d9c0 0x7fae801a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.658+0000 7fae7f400700 1 -- 172.21.15.27:0/2279267897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae801a0d20 con 0x7fae80104eb0 2024-05-10T11:14:42.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.658+0000 7fae7f400700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80104eb0 0x7fae8019c250 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fae6c00d750 tx=0x7fae6c00db10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:42.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.659+0000 7fae7cc00700 1 -- 172.21.15.27:0/2279267897 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c01c070 con 0x7fae80104eb0 2024-05-10T11:14:42.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.659+0000 7fae7cc00700 1 -- 172.21.15.27:0/2279267897 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae6c01f860 con 0x7fae80104eb0 2024-05-10T11:14:42.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.659+0000 7fae7cc00700 1 -- 172.21.15.27:0/2279267897 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c009860 con 0x7fae80104eb0 2024-05-10T11:14:42.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.659+0000 7fae85c00700 1 -- 172.21.15.27:0/2279267897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae801a1010 con 0x7fae80104eb0 2024-05-10T11:14:42.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.659+0000 7fae85c00700 1 -- 172.21.15.27:0/2279267897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae80105340 con 0x7fae80104eb0 2024-05-10T11:14:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.665+0000 7fae91b18b80 1 Processor -- start 2024-05-10T11:14:42.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.665+0000 7fae91b18b80 1 -- start start 2024-05-10T11:14:42.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.666+0000 7fae91b18b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 0x5606895f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.666+0000 7fae91b18b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606895ed060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.666+0000 7fae85200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 0x5606895f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.667+0000 7fae85200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 0x5606895f2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50874/0 (socket says 172.21.15.27:50874) 2024-05-10T11:14:42.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.667+0000 7fae91b18b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895ed680 0x5606895edb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.667+0000 7fae85200700 1 -- 172.21.15.27:0/1129603553 learned_addr learned my addr 172.21.15.27:0/1129603553 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:42.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.667+0000 7fae91b18b80 1 -- 172.21.15.27:0/1129603553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56068953faa0 con 0x5606895ed680 2024-05-10T11:14:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.667+0000 7fae91b18b80 1 -- 172.21.15.27:0/1129603553 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5606894dd7e0 con 0x5606895f3380 2024-05-10T11:14:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.667+0000 7fae91b18b80 1 -- 172.21.15.27:0/1129603553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56068953ba60 con 0x5606895f2a60 2024-05-10T11:14:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.667+0000 7fae7fe00700 1 --2- 172.21.15.27:0/1129603553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606895ed060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.668+0000 7fae7fe00700 1 -- 172.21.15.27:0/1129603553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 msgr2=0x5606895f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.668+0000 7fae7fe00700 1 --2- 172.21.15.27:0/1129603553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 0x5606895f2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.668+0000 7fae7fe00700 1 -- 172.21.15.27:0/1129603553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895ed680 msgr2=0x5606895edb00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:14:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.668+0000 7fae7fe00700 1 --2- 172.21.15.27:0/1129603553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895ed680 0x5606895edb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.668+0000 7fae7fe00700 1 -- 172.21.15.27:0/1129603553 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56068948c950 con 0x5606895f3380 2024-05-10T11:14:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.668+0000 7fae85200700 1 --2- 172.21.15.27:0/1129603553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 0x5606895f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.668+0000 7fae7cc00700 1 -- 172.21.15.27:0/2279267897 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7fae6c01fa10 con 0x7fae80104eb0 2024-05-10T11:14:42.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.668+0000 7fae7fe00700 1 --2- 172.21.15.27:0/1129603553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606895ed060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fae700098e0 tx=0x7fae700096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cc1ab102a85a5e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:42.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.669+0000 7fae75600700 1 -- 172.21.15.27:0/1129603553 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae70026070 con 0x5606895f3380 2024-05-10T11:14:42.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.669+0000 7fae75600700 1 -- 172.21.15.27:0/1129603553 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae70005470 con 0x5606895f3380 2024-05-10T11:14:42.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.669+0000 7fae75600700 1 -- 172.21.15.27:0/1129603553 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae700038e0 con 0x5606895f3380 2024-05-10T11:14:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.670+0000 7fae91b18b80 1 -- 172.21.15.27:0/1129603553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 msgr2=0x5606895ed060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.670+0000 7fae91b18b80 1 --2- 172.21.15.27:0/1129603553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606895ed060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fae700098e0 tx=0x7fae700096a0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.671+0000 7fae91b18b80 1 -- 172.21.15.27:0/1129603553 shutdown_connections 2024-05-10T11:14:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.671+0000 7fae91b18b80 1 --2- 172.21.15.27:0/1129603553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 0x5606895f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.671+0000 7fae91b18b80 1 --2- 172.21.15.27:0/1129603553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895ed680 0x5606895edb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.671+0000 7fae91b18b80 1 --2- 172.21.15.27:0/1129603553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606895ed060 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.671+0000 7fae91b18b80 1 -- 172.21.15.27:0/1129603553 >> 172.21.15.27:0/1129603553 conn(0x56068951bbe0 msgr2=0x5606895e7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.671+0000 7fae91b18b80 1 -- 172.21.15.27:0/1129603553 shutdown_connections 2024-05-10T11:14:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.671+0000 7fae7cc00700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fae640b2c00 0x7fae640b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.671+0000 7fae91b18b80 1 -- 172.21.15.27:0/1129603553 wait complete. 2024-05-10T11:14:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.671+0000 7fae7cc00700 1 -- 172.21.15.27:0/2279267897 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fae6c0eac40 con 0x7fae80104eb0 2024-05-10T11:14:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.672+0000 7fae91b18b80 1 -- start start 2024-05-10T11:14:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.673+0000 7fae91b18b80 1 Processor -- start 2024-05-10T11:14:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.673+0000 7fae7fe00700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fae640b2c00 0x7fae640b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.674+0000 7fae7fe00700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fae640b2c00 0x7fae640b50c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fae70006010 tx=0x7fae7001a040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.674+0000 7fae91b18b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895ed680 0x5606895f8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:42 smithi076 ceph-mon[34152]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-10T11:14:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.674+0000 7fae91b18b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 0x5606895f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.674+0000 7fae91b18b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606895f68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.674+0000 7fae91b18b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5606895f9dd0 con 0x5606895ed680 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.674+0000 7fae91b18b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5606894a5ba0 con 0x5606895f3380 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.674+0000 7fae91b18b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5606895f9f50 con 0x5606895f2a60 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.674+0000 7fae91b18b80 10 client.0 _ll_register_callbacks cb 0x5606894e6cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.675+0000 7fae7fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895ed680 0x5606895f8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.675+0000 7fae7fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895ed680 0x5606895f8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59110/0 (socket says 172.21.15.27:59110) 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.675+0000 7fae7fe00700 1 -- 172.21.15.27:0/3901486858 learned_addr learned my addr 172.21.15.27:0/3901486858 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.675+0000 7fae85200700 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606895f68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.675+0000 7fae7f400700 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 0x5606895f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.675+0000 7fae85200700 1 -- 172.21.15.27:0/3901486858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 msgr2=0x5606895f63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.675+0000 7fae85200700 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 0x5606895f63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.675+0000 7fae85200700 1 -- 172.21.15.27:0/3901486858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895ed680 msgr2=0x5606895f8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.675+0000 7fae85200700 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895ed680 0x5606895f8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.675+0000 7fae85200700 1 -- 172.21.15.27:0/3901486858 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606895054a0 con 0x5606895f3380 2024-05-10T11:14:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.675+0000 7fae7f400700 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 0x5606895f63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:14:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.676+0000 7fae85200700 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606895f68e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fae78007aa0 tx=0x7fae78007ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91353077312f47e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.676+0000 7fae6a000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:14:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.676+0000 7fae6a000700 1 -- 172.21.15.27:0/3901486858 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7801a030 con 0x5606895f3380 2024-05-10T11:14:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.676+0000 7fae91b18b80 1 -- 172.21.15.27:0/3901486858 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae70005470 con 0x5606895f3380 2024-05-10T11:14:42.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.677+0000 7fae6a000700 1 -- 172.21.15.27:0/3901486858 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fae78004bb0 con 0x5606895f3380 2024-05-10T11:14:42.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.677+0000 7fae6a000700 1 -- 172.21.15.27:0/3901486858 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fae78026050 con 0x5606895f3380 2024-05-10T11:14:42.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.677+0000 7fae91b18b80 10 client.25544 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:42.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.677+0000 7fae91b18b80 1 -- 172.21.15.27:0/3901486858 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560689604390 con 0x5606895f3380 2024-05-10T11:14:42.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.678+0000 7fae6a000700 1 -- 172.21.15.27:0/3901486858 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae70005470 con 0x5606895f3380 2024-05-10T11:14:42.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.678+0000 7fae6a000700 1 -- 172.21.15.27:0/3901486858 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fae78016630 con 0x5606895f3380 2024-05-10T11:14:42.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.678+0000 7fae91b18b80 10 client.25544 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:42.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.678+0000 7fae91b18b80 10 client.25544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:14:42.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.678+0000 7fae91b18b80 20 client.25544 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:42.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.678+0000 7fae91b18b80 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560689606970 0x560689608d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.678+0000 7fae91b18b80 4 client.25544 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:14:42.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.678+0000 7fae91b18b80 1 -- 172.21.15.27:0/3901486858 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606895ba120 con 0x560689606970 2024-05-10T11:14:42.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.679+0000 7fae7fe00700 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560689606970 0x560689608d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.680+0000 7fae6a000700 10 client.25544 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:42.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.680+0000 7fae7fe00700 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560689606970 0x560689608d60 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf6ca1766c09214c server_cookie=6b5d48b9c52fa87e in_seq=0 out_seq=0 2024-05-10T11:14:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.681+0000 7fae6a000700 1 -- 172.21.15.27:0/3901486858 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5606895ba120 con 0x560689606970 2024-05-10T11:14:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.681+0000 7fae6a000700 10 client.25544 handle_command_reply: tid=0 2024-05-10T11:14:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.681+0000 7fae91b18b80 1 client.25544 shutdown 2024-05-10T11:14:42.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.681+0000 7fae91b18b80 1 -- 172.21.15.27:0/3901486858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 msgr2=0x5606895f68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:42.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.681+0000 7fae91b18b80 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606895f68e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fae78007aa0 tx=0x7fae78007ad0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.682+0000 7fae91b18b80 1 -- 172.21.15.27:0/3901486858 shutdown_connections 2024-05-10T11:14:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.682+0000 7fae91b18b80 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560689606970 0x560689608d60 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.682+0000 7fae91b18b80 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606895f68e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.682+0000 7fae91b18b80 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895ed680 0x5606895f8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.682+0000 7fae91b18b80 1 --2- 172.21.15.27:0/3901486858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895f2a60 0x5606895f63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.682+0000 7fae91b18b80 1 -- 172.21.15.27:0/3901486858 >> 172.21.15.27:0/3901486858 conn(0x56068951bbe0 msgr2=0x5606895e24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.682+0000 7fae6a000700 0 client.25544 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.683+0000 7fae91b18b80 1 -- 172.21.15.27:0/3901486858 shutdown_connections 2024-05-10T11:14:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.683+0000 7fae91b18b80 1 -- 172.21.15.27:0/3901486858 wait complete. 2024-05-10T11:14:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.683+0000 7fae91b18b80 20 client.25544 trim_cache size 0 max 16384 2024-05-10T11:14:42.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.705+0000 7fae91b18b80 1 Processor -- start 2024-05-10T11:14:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.705+0000 7fae91b18b80 1 -- start start 2024-05-10T11:14:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.706+0000 7fae91b18b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895ed680 0x5606895ed060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.706+0000 7fae91b18b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895f2a60 0x5606895f3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.706+0000 7fae91b18b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x56068960e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.706+0000 7fae91b18b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fae801ad330 con 0x5606895f2a60 2024-05-10T11:14:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.706+0000 7fae91b18b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fae801aee70 con 0x5606895f3380 2024-05-10T11:14:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.706+0000 7fae91b18b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fae801ada00 con 0x5606895ed680 2024-05-10T11:14:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.706+0000 7fae85200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x56068960e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.706+0000 7fae85200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x56068960e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48192/0 (socket says 172.21.15.27:48192) 2024-05-10T11:14:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.706+0000 7fae85200700 1 -- 172.21.15.27:0/1863086390 learned_addr learned my addr 172.21.15.27:0/1863086390 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.707+0000 7fae7fe00700 1 --2- 172.21.15.27:0/1863086390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895ed680 0x5606895ed060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.707+0000 7fae85200700 1 -- 172.21.15.27:0/1863086390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895ed680 msgr2=0x5606895ed060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.707+0000 7fae85200700 1 --2- 172.21.15.27:0/1863086390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895ed680 0x5606895ed060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.707+0000 7fae85200700 1 -- 172.21.15.27:0/1863086390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895f2a60 msgr2=0x5606895f3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.707+0000 7fae85200700 1 --2- 172.21.15.27:0/1863086390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895f2a60 0x5606895f3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.707+0000 7fae85200700 1 -- 172.21.15.27:0/1863086390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606895f7110 con 0x5606895f3380 2024-05-10T11:14:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.707+0000 7fae85200700 1 --2- 172.21.15.27:0/1863086390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x56068960e610 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fae78016a00 tx=0x7fae78017e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a58099f9a5f9e593 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.708+0000 7fae74c00700 1 -- 172.21.15.27:0/1863086390 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7801a070 con 0x5606895f3380 2024-05-10T11:14:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.708+0000 7fae74c00700 1 -- 172.21.15.27:0/1863086390 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae78023070 con 0x5606895f3380 2024-05-10T11:14:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.708+0000 7fae74c00700 1 -- 172.21.15.27:0/1863086390 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae780158c0 con 0x5606895f3380 2024-05-10T11:14:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.709+0000 7fae91b18b80 1 -- 172.21.15.27:0/1863086390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 msgr2=0x56068960e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.709+0000 7fae91b18b80 1 --2- 172.21.15.27:0/1863086390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x56068960e610 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fae78016a00 tx=0x7fae78017e90 comp rx=0 tx=0).stop 2024-05-10T11:14:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.709+0000 7fae91b18b80 1 -- 172.21.15.27:0/1863086390 shutdown_connections 2024-05-10T11:14:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.709+0000 7fae91b18b80 1 --2- 172.21.15.27:0/1863086390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895ed680 0x5606895ed060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.709+0000 7fae91b18b80 1 --2- 172.21.15.27:0/1863086390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x56068960e610 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.709+0000 7fae91b18b80 1 --2- 172.21.15.27:0/1863086390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895f2a60 0x5606895f3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.709+0000 7fae91b18b80 1 -- 172.21.15.27:0/1863086390 >> 172.21.15.27:0/1863086390 conn(0x56068951bbe0 msgr2=0x5606895e24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.709+0000 7fae91b18b80 1 -- 172.21.15.27:0/1863086390 shutdown_connections 2024-05-10T11:14:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.709+0000 7fae91b18b80 1 -- 172.21.15.27:0/1863086390 wait complete. 2024-05-10T11:14:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.710+0000 7fae91b18b80 1 -- start start 2024-05-10T11:14:42.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.710+0000 7fae91b18b80 1 Processor -- start 2024-05-10T11:14:42.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.711+0000 7fae91b18b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895ed680 0x5606895fc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.711+0000 7fae91b18b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895f2a60 0x5606895fc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.711+0000 7fae91b18b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606896711d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.711+0000 7fae91b18b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fae801ae7a0 con 0x5606895f2a60 2024-05-10T11:14:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.711+0000 7fae91b18b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fae801ae0d0 con 0x5606895f3380 2024-05-10T11:14:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.711+0000 7fae91b18b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fae801ab310 con 0x5606895ed680 2024-05-10T11:14:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.711+0000 7fae91b18b80 10 client.0 _ll_register_callbacks cb 0x5606894e6cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.711+0000 7fae7fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895ed680 0x5606895fc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.711+0000 7fae7fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895ed680 0x5606895fc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50896/0 (socket says 172.21.15.27:50896) 2024-05-10T11:14:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.711+0000 7fae7fe00700 1 -- 172.21.15.27:0/986767449 learned_addr learned my addr 172.21.15.27:0/986767449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.711+0000 7fae85200700 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606896711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.711+0000 7fae7f400700 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895f2a60 0x5606895fc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.712+0000 7fae7fe00700 1 -- 172.21.15.27:0/986767449 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 msgr2=0x5606896711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.712+0000 7fae7fe00700 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606896711d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.712+0000 7fae7fe00700 1 -- 172.21.15.27:0/986767449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895f2a60 msgr2=0x5606895fc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.712+0000 7fae7fe00700 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895f2a60 0x5606895fc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.712+0000 7fae7fe00700 1 -- 172.21.15.27:0/986767449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560689671710 con 0x5606895ed680 2024-05-10T11:14:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.712+0000 7fae85200700 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606896711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.712+0000 7fae7f400700 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895f2a60 0x5606895fc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:14:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.713+0000 7fae7fe00700 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895ed680 0x5606895fc410 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fae7000bec0 tx=0x7fae700096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5afd89d20ad1b0a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.713+0000 7fae6a000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:14:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.713+0000 7fae6a000700 1 -- 172.21.15.27:0/986767449 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae70026040 con 0x5606895ed680 2024-05-10T11:14:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.713+0000 7fae6a000700 1 -- 172.21.15.27:0/986767449 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fae70005470 con 0x5606895ed680 2024-05-10T11:14:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.713+0000 7fae91b18b80 1 -- 172.21.15.27:0/986767449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae78023070 con 0x5606895ed680 2024-05-10T11:14:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.714+0000 7fae6a000700 1 -- 172.21.15.27:0/986767449 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fae70045050 con 0x5606895ed680 2024-05-10T11:14:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.714+0000 7fae91b18b80 10 client.25546 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.714+0000 7fae91b18b80 1 -- 172.21.15.27:0/986767449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5606896729c0 con 0x5606895ed680 2024-05-10T11:14:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.714+0000 7fae6a000700 1 -- 172.21.15.27:0/986767449 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae78023070 con 0x5606895ed680 2024-05-10T11:14:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.714+0000 7fae6a000700 1 -- 172.21.15.27:0/986767449 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fae78009260 con 0x5606895ed680 2024-05-10T11:14:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.715+0000 7fae91b18b80 10 client.25546 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.715+0000 7fae91b18b80 10 client.25546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:14:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.715+0000 7fae91b18b80 20 client.25546 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.715+0000 7fae91b18b80 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560689672cb0 0x5606896750a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.715+0000 7fae91b18b80 4 client.25546 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:14:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.715+0000 7fae91b18b80 1 -- 172.21.15.27:0/986767449 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56068954d480 con 0x560689672cb0 2024-05-10T11:14:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.715+0000 7fae7f400700 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560689672cb0 0x5606896750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.716+0000 7fae7f400700 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560689672cb0 0x5606896750a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd81789848265948 server_cookie=7e74fe3f6bedc6d0 in_seq=0 out_seq=0 2024-05-10T11:14:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.716+0000 7fae6a000700 10 client.25546 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:42.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.717+0000 7fae6a000700 1 -- 172.21.15.27:0/986767449 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56068954d480 con 0x560689672cb0 2024-05-10T11:14:42.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.717+0000 7fae6a000700 10 client.25546 handle_command_reply: tid=0 2024-05-10T11:14:42.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.717+0000 7fae91b18b80 1 client.25546 shutdown 2024-05-10T11:14:42.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.717+0000 7fae91b18b80 1 -- 172.21.15.27:0/986767449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895ed680 msgr2=0x5606895fc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:42.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.717+0000 7fae91b18b80 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895ed680 0x5606895fc410 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fae7000bec0 tx=0x7fae700096a0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.717+0000 7fae91b18b80 1 -- 172.21.15.27:0/986767449 shutdown_connections 2024-05-10T11:14:42.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.717+0000 7fae91b18b80 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560689672cb0 0x5606896750a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.717+0000 7fae91b18b80 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606895ed680 0x5606895fc410 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.717+0000 7fae91b18b80 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5606895f3380 0x5606896711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.717+0000 7fae91b18b80 1 --2- 172.21.15.27:0/986767449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606895f2a60 0x5606895fc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.717+0000 7fae91b18b80 1 -- 172.21.15.27:0/986767449 >> 172.21.15.27:0/986767449 conn(0x56068951bbe0 msgr2=0x5606895f78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:42.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.718+0000 7fae6a000700 0 client.25546 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:42.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.718+0000 7fae91b18b80 1 -- 172.21.15.27:0/986767449 shutdown_connections 2024-05-10T11:14:42.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.718+0000 7fae91b18b80 1 -- 172.21.15.27:0/986767449 wait complete. 2024-05-10T11:14:42.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.718+0000 7fae91b18b80 20 client.25546 trim_cache size 0 max 16384 2024-05-10T11:14:42.720 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:14:42.720 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:14:42.720 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "7c0f2f46-5984-4b52-88d8-565ba8758714", 2024-05-10T11:14:42.720 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:14:42.720 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:14:42.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.720+0000 7fae85c00700 1 -- 172.21.15.27:0/2279267897 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fae640b2c00 msgr2=0x7fae640b50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:42.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.720+0000 7fae85c00700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fae640b2c00 0x7fae640b50c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fae70006010 tx=0x7fae7001a040 comp rx=0 tx=0).stop 2024-05-10T11:14:42.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.720+0000 7fae85c00700 1 -- 172.21.15.27:0/2279267897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80104eb0 msgr2=0x7fae8019c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:42.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.720+0000 7fae85c00700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80104eb0 0x7fae8019c250 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fae6c00d750 tx=0x7fae6c00db10 comp rx=0 tx=0).stop 2024-05-10T11:14:42.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.722+0000 7fae85c00700 1 -- 172.21.15.27:0/2279267897 shutdown_connections 2024-05-10T11:14:42.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.722+0000 7fae85c00700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fae640b2c00 0x7fae640b50c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.722+0000 7fae85c00700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae8010d9c0 0x7fae801a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.722+0000 7fae85c00700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae80104590 0x7fae8019bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.722+0000 7fae85c00700 1 --2- 172.21.15.27:0/2279267897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80104eb0 0x7fae8019c250 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:42.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.722+0000 7fae85c00700 1 -- 172.21.15.27:0/2279267897 >> 172.21.15.27:0/2279267897 conn(0x7fae800fddc0 msgr2=0x7fae801001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.722+0000 7fae85c00700 1 -- 172.21.15.27:0/2279267897 shutdown_connections 2024-05-10T11:14:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:42.722+0000 7fae85c00700 1 -- 172.21.15.27:0/2279267897 wait complete. 2024-05-10T11:14:42.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:42 smithi189 ceph-mon[34842]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-10T11:14:43.657 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:14:44.227 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:14:44.568 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:44 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:14:44.568 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:44 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:14:44.568 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:44 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:14:44.568 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:44 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:14:44.568 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:44 smithi027 ceph-mon[30054]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:44.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:44 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:14:44.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:44 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:14:44.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:44 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:14:44.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:44 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:14:44.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:44 smithi076 ceph-mon[34152]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:44.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:44 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:14:44.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:44 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:14:44.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:44 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:14:44.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:44 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:14:44.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:44 smithi189 ceph-mon[34842]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:14:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.485+0000 7f80e7e00700 1 -- 172.21.15.27:0/1185006724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810d9c0 msgr2=0x7f80e810fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.485+0000 7f80e7e00700 1 --2- 172.21.15.27:0/1185006724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810d9c0 0x7f80e810fe80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f80e000b0a0 tx=0x7f80e000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.485+0000 7f80e7e00700 1 -- 172.21.15.27:0/1185006724 shutdown_connections 2024-05-10T11:14:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.485+0000 7f80e7e00700 1 --2- 172.21.15.27:0/1185006724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810d9c0 0x7f80e810fe80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.485+0000 7f80e7e00700 1 --2- 172.21.15.27:0/1185006724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e8104590 0x7f80e8104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.485+0000 7f80e7e00700 1 --2- 172.21.15.27:0/1185006724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8104eb0 0x7f80e810d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.485+0000 7f80e7e00700 1 -- 172.21.15.27:0/1185006724 >> 172.21.15.27:0/1185006724 conn(0x7f80e80fddc0 msgr2=0x7f80e81001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.486+0000 7f80e7e00700 1 -- 172.21.15.27:0/1185006724 shutdown_connections 2024-05-10T11:14:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.486+0000 7f80e7e00700 1 -- 172.21.15.27:0/1185006724 wait complete. 2024-05-10T11:14:46.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.487+0000 7f80e7e00700 1 Processor -- start 2024-05-10T11:14:46.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.488+0000 7f80e7e00700 1 -- start start 2024-05-10T11:14:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.488+0000 7f80e7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e8104590 0x7f80e81a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.488+0000 7f80e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8104eb0 0x7f80e81a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.488+0000 7f80e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e810d9c0 0x7f80e819ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.488+0000 7f80e7e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f80e8113610 con 0x7f80e8104590 2024-05-10T11:14:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.488+0000 7f80e7e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f80e8113490 con 0x7f80e810d9c0 2024-05-10T11:14:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.488+0000 7f80e7e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f80e8113790 con 0x7f80e8104eb0 2024-05-10T11:14:46.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.489+0000 7f80e6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8104eb0 0x7f80e81a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.489+0000 7f80e6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8104eb0 0x7f80e81a1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49940/0 (socket says 172.21.15.27:49940) 2024-05-10T11:14:46.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.489+0000 7f80e6000700 1 -- 172.21.15.27:0/1960279561 learned_addr learned my addr 172.21.15.27:0/1960279561 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:46.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.489+0000 7f80e7400700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e810d9c0 0x7f80e819ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.489+0000 7f80e6a00700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e8104590 0x7f80e81a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.489+0000 7f80e7400700 1 -- 172.21.15.27:0/1960279561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8104eb0 msgr2=0x7f80e81a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:46.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.489+0000 7f80e7400700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8104eb0 0x7f80e81a1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.489+0000 7f80e7400700 1 -- 172.21.15.27:0/1960279561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e8104590 msgr2=0x7f80e81a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:46.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.489+0000 7f80e7400700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e8104590 0x7f80e81a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.489+0000 7f80e7400700 1 -- 172.21.15.27:0/1960279561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e819c1e0 con 0x7f80e810d9c0 2024-05-10T11:14:46.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.490+0000 7f80e6a00700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e8104590 0x7f80e81a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:14:46.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.490+0000 7f80e7400700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e810d9c0 0x7f80e819ba80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f80e000b070 tx=0x7f80e0004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:46.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.490+0000 7f80df400700 1 -- 172.21.15.27:0/1960279561 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e0021070 con 0x7f80e810d9c0 2024-05-10T11:14:46.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.490+0000 7f80df400700 1 -- 172.21.15.27:0/1960279561 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80e0007e30 con 0x7f80e810d9c0 2024-05-10T11:14:46.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.491+0000 7f80e7e00700 1 -- 172.21.15.27:0/1960279561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e819c470 con 0x7f80e810d9c0 2024-05-10T11:14:46.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.491+0000 7f80e7e00700 1 -- 172.21.15.27:0/1960279561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e81a8790 con 0x7f80e810d9c0 2024-05-10T11:14:46.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.491+0000 7f80df400700 1 -- 172.21.15.27:0/1960279561 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e00046f0 con 0x7f80e810d9c0 2024-05-10T11:14:46.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.495+0000 7f80df400700 1 -- 172.21.15.27:0/1960279561 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f80e0003c00 con 0x7f80e810d9c0 2024-05-10T11:14:46.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.497+0000 7f80df400700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f80cc0b2a70 0x7f80cc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.497+0000 7f80f8146b80 1 Processor -- start 2024-05-10T11:14:46.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.497+0000 7f80df400700 1 -- 172.21.15.27:0/1960279561 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f80e000f020 con 0x7f80e810d9c0 2024-05-10T11:14:46.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.498+0000 7f80f8146b80 1 -- start start 2024-05-10T11:14:46.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.498+0000 7f80e6a00700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f80cc0b2a70 0x7f80cc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.498+0000 7f80f8146b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211653180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.498+0000 7f80f8146b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582116536c0 0x55821164d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.498+0000 7f80f8146b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 0x55821164ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.499+0000 7f80f8146b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55821159fa20 con 0x55821164da50 2024-05-10T11:14:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.499+0000 7f80f8146b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55821153d760 con 0x558211652da0 2024-05-10T11:14:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.499+0000 7f80f8146b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55821159b9e0 con 0x5582116536c0 2024-05-10T11:14:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.499+0000 7f80e6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582116536c0 0x55821164d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.499+0000 7f80e6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582116536c0 0x55821164d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49956/0 (socket says 172.21.15.27:49956) 2024-05-10T11:14:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.499+0000 7f80e6000700 1 -- 172.21.15.27:0/1351825479 learned_addr learned my addr 172.21.15.27:0/1351825479 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.499+0000 7f80e7400700 1 --2- 172.21.15.27:0/1351825479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 0x55821164ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.499+0000 7f80e6000700 1 -- 172.21.15.27:0/1351825479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 msgr2=0x558211653180 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:14:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.500+0000 7f80e6000700 1 --2- 172.21.15.27:0/1351825479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211653180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.500+0000 7f80e6000700 1 -- 172.21.15.27:0/1351825479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 msgr2=0x55821164ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.500+0000 7f80e6000700 1 --2- 172.21.15.27:0/1351825479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 0x55821164ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.500+0000 7f80e6a00700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f80cc0b2a70 0x7f80cc0b4f30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f80e819cbc0 tx=0x7f80d8009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.500+0000 7f80e6000700 1 -- 172.21.15.27:0/1351825479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582114ec8d0 con 0x5582116536c0 2024-05-10T11:14:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.500+0000 7f80e7400700 1 --2- 172.21.15.27:0/1351825479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 0x55821164ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:14:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.501+0000 7f80e6000700 1 --2- 172.21.15.27:0/1351825479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582116536c0 0x55821164d430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f80d4009d30 tx=0x7f80d4011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c43574861231c2a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.501+0000 7f80d3e00700 1 -- 172.21.15.27:0/1351825479 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d401e070 con 0x5582116536c0 2024-05-10T11:14:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.501+0000 7f80d3e00700 1 -- 172.21.15.27:0/1351825479 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80d4019ae0 con 0x5582116536c0 2024-05-10T11:14:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.501+0000 7f80d3e00700 1 -- 172.21.15.27:0/1351825479 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d400eb30 con 0x5582116536c0 2024-05-10T11:14:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.502+0000 7f80f8146b80 1 -- 172.21.15.27:0/1351825479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582116536c0 msgr2=0x55821164d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.502+0000 7f80f8146b80 1 --2- 172.21.15.27:0/1351825479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582116536c0 0x55821164d430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f80d4009d30 tx=0x7f80d4011720 comp rx=0 tx=0).stop 2024-05-10T11:14:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.503+0000 7f80f8146b80 1 -- 172.21.15.27:0/1351825479 shutdown_connections 2024-05-10T11:14:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.503+0000 7f80f8146b80 1 --2- 172.21.15.27:0/1351825479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 0x55821164ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.503+0000 7f80f8146b80 1 --2- 172.21.15.27:0/1351825479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211653180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.503+0000 7f80f8146b80 1 --2- 172.21.15.27:0/1351825479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582116536c0 0x55821164d430 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.503+0000 7f80f8146b80 1 -- 172.21.15.27:0/1351825479 >> 172.21.15.27:0/1351825479 conn(0x5582115c56d0 msgr2=0x558211647ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.503+0000 7f80f8146b80 1 -- 172.21.15.27:0/1351825479 shutdown_connections 2024-05-10T11:14:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.503+0000 7f80f8146b80 1 -- 172.21.15.27:0/1351825479 wait complete. 2024-05-10T11:14:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.504+0000 7f80f8146b80 1 -- start start 2024-05-10T11:14:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.504+0000 7f80f8146b80 1 Processor -- start 2024-05-10T11:14:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.505+0000 7f80f8146b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55821164da50 0x5582116593b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.505+0000 7f80f8146b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211656750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.505+0000 7f80f8146b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582116536c0 0x558211656c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.505+0000 7f80f8146b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55821165a150 con 0x5582116536c0 2024-05-10T11:14:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.505+0000 7f80f8146b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558211505b20 con 0x558211652da0 2024-05-10T11:14:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.505+0000 7f80f8146b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55821165a2d0 con 0x55821164da50 2024-05-10T11:14:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.505+0000 7f80f8146b80 10 client.0 _ll_register_callbacks cb 0x558211546c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.506+0000 7f80e6000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211656750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.506+0000 7f80e6000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211656750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:43026/0 (socket says 172.21.15.27:43026) 2024-05-10T11:14:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.506+0000 7f80e6000700 1 -- 172.21.15.27:0/2742702317 learned_addr learned my addr 172.21.15.27:0/2742702317 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.506+0000 7f80e7400700 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582116536c0 0x558211656c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.507+0000 7f80e6a00700 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55821164da50 0x5582116593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.507+0000 7f80e6000700 1 -- 172.21.15.27:0/2742702317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55821164da50 msgr2=0x5582116593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.507+0000 7f80e6000700 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55821164da50 0x5582116593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.507+0000 7f80e6000700 1 -- 172.21.15.27:0/2742702317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582116536c0 msgr2=0x558211656c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.507+0000 7f80e6000700 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582116536c0 0x558211656c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.507+0000 7f80e6000700 1 -- 172.21.15.27:0/2742702317 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558211565420 con 0x558211652da0 2024-05-10T11:14:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.507+0000 7f80e7400700 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582116536c0 0x558211656c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.507+0000 7f80e6000700 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211656750 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f80d4009230 tx=0x7f80d4005d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ad0432af2931aa5 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.507+0000 7f80d0c00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:14:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.508+0000 7f80d0c00700 1 -- 172.21.15.27:0/2742702317 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d401e040 con 0x558211652da0 2024-05-10T11:14:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.508+0000 7f80d0c00700 1 -- 172.21.15.27:0/2742702317 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f80d4004230 con 0x558211652da0 2024-05-10T11:14:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.508+0000 7f80f8146b80 1 -- 172.21.15.27:0/2742702317 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80d4019ae0 con 0x558211652da0 2024-05-10T11:14:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.508+0000 7f80d0c00700 1 -- 172.21.15.27:0/2742702317 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f80d4031050 con 0x558211652da0 2024-05-10T11:14:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.508+0000 7f80f8146b80 10 client.25559 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.508+0000 7f80f8146b80 1 -- 172.21.15.27:0/2742702317 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55821166d1e0 con 0x558211652da0 2024-05-10T11:14:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.509+0000 7f80d0c00700 1 -- 172.21.15.27:0/2742702317 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80d4019ae0 con 0x558211652da0 2024-05-10T11:14:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.509+0000 7f80d0c00700 1 -- 172.21.15.27:0/2742702317 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f80d4021050 con 0x558211652da0 2024-05-10T11:14:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.509+0000 7f80f8146b80 10 client.25559 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.509+0000 7f80f8146b80 10 client.25559 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:14:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.509+0000 7f80f8146b80 20 client.25559 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.509+0000 7f80f8146b80 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55821166f6b0 0x558211671aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.509+0000 7f80f8146b80 4 client.25559 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:14:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.509+0000 7f80f8146b80 1 -- 172.21.15.27:0/2742702317 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55821161a140 con 0x55821166f6b0 2024-05-10T11:14:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.510+0000 7f80e6a00700 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55821166f6b0 0x558211671aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.511+0000 7f80e6a00700 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55821166f6b0 0x558211671aa0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bae2f11026f9b8c server_cookie=2cbb12aa28a11452 in_seq=0 out_seq=0 2024-05-10T11:14:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.511+0000 7f80d0c00700 10 client.25559 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.512+0000 7f80d0c00700 1 -- 172.21.15.27:0/2742702317 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55821161a140 con 0x55821166f6b0 2024-05-10T11:14:46.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.512+0000 7f80d0c00700 10 client.25559 handle_command_reply: tid=0 2024-05-10T11:14:46.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.512+0000 7f80f8146b80 1 client.25559 shutdown 2024-05-10T11:14:46.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.512+0000 7f80f8146b80 1 -- 172.21.15.27:0/2742702317 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 msgr2=0x558211656750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:46.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.512+0000 7f80f8146b80 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211656750 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f80d4009230 tx=0x7f80d4005d60 comp rx=0 tx=0).stop 2024-05-10T11:14:46.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.512+0000 7f80f8146b80 1 -- 172.21.15.27:0/2742702317 shutdown_connections 2024-05-10T11:14:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.512+0000 7f80f8146b80 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55821166f6b0 0x558211671aa0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.512+0000 7f80f8146b80 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55821164da50 0x5582116593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.512+0000 7f80f8146b80 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582116536c0 0x558211656c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.512+0000 7f80f8146b80 1 --2- 172.21.15.27:0/2742702317 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211656750 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.512+0000 7f80f8146b80 1 -- 172.21.15.27:0/2742702317 >> 172.21.15.27:0/2742702317 conn(0x5582115c56d0 msgr2=0x558211642750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.512+0000 7f80d0c00700 0 client.25559 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.513+0000 7f80f8146b80 1 -- 172.21.15.27:0/2742702317 shutdown_connections 2024-05-10T11:14:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.513+0000 7f80f8146b80 1 -- 172.21.15.27:0/2742702317 wait complete. 2024-05-10T11:14:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.513+0000 7f80f8146b80 20 client.25559 trim_cache size 0 max 16384 2024-05-10T11:14:46.515 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:46 smithi027 ceph-mon[30054]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-05-10T11:14:46.515 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:46 smithi027 ceph-mon[30054]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-05-10T11:14:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.559+0000 7f80f8146b80 1 Processor -- start 2024-05-10T11:14:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.559+0000 7f80f8146b80 1 -- start start 2024-05-10T11:14:46.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.559+0000 7f80f8146b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55821164da50 0x55821166bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.559+0000 7f80f8146b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558211652da0 0x558211650be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.559+0000 7f80f8146b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582116536c0 0x558211654330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.559+0000 7f80f8146b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f80c800ee20 con 0x5582116536c0 2024-05-10T11:14:46.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.559+0000 7f80f8146b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f80c8010960 con 0x55821164da50 2024-05-10T11:14:46.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.559+0000 7f80f8146b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f80c800f4f0 con 0x558211652da0 2024-05-10T11:14:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.560+0000 7f80e6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55821164da50 0x55821166bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.560+0000 7f80e6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55821164da50 0x55821166bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:43034/0 (socket says 172.21.15.27:43034) 2024-05-10T11:14:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.560+0000 7f80e6a00700 1 -- 172.21.15.27:0/3554156722 learned_addr learned my addr 172.21.15.27:0/3554156722 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.560+0000 7f80e6000700 1 --2- 172.21.15.27:0/3554156722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558211652da0 0x558211650be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.560+0000 7f80e7400700 1 --2- 172.21.15.27:0/3554156722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582116536c0 0x558211654330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.560+0000 7f80e6a00700 1 -- 172.21.15.27:0/3554156722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558211652da0 msgr2=0x558211650be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.560+0000 7f80e6a00700 1 --2- 172.21.15.27:0/3554156722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558211652da0 0x558211650be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.562+0000 7f80e6a00700 1 -- 172.21.15.27:0/3554156722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582116536c0 msgr2=0x558211654330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.562+0000 7f80e6a00700 1 --2- 172.21.15.27:0/3554156722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582116536c0 0x558211654330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.562+0000 7f80e6a00700 1 -- 172.21.15.27:0/3554156722 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55821165a030 con 0x55821164da50 2024-05-10T11:14:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.562+0000 7f80e6000700 1 --2- 172.21.15.27:0/3554156722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558211652da0 0x558211650be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.562+0000 7f80e7400700 1 --2- 172.21.15.27:0/3554156722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582116536c0 0x558211654330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:14:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.562+0000 7f80e6a00700 1 --2- 172.21.15.27:0/3554156722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55821164da50 0x55821166bad0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f80d8010d60 tx=0x7f80d8013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb2167132f699347 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.562+0000 7f80d3400700 1 -- 172.21.15.27:0/3554156722 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d802c070 con 0x55821164da50 2024-05-10T11:14:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.563+0000 7f80d3400700 1 -- 172.21.15.27:0/3554156722 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80d80044e0 con 0x55821164da50 2024-05-10T11:14:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.563+0000 7f80d3400700 1 -- 172.21.15.27:0/3554156722 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d80038e0 con 0x55821164da50 2024-05-10T11:14:46.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.563+0000 7f80f8146b80 1 -- 172.21.15.27:0/3554156722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55821164da50 msgr2=0x55821166bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:46.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.563+0000 7f80f8146b80 1 --2- 172.21.15.27:0/3554156722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55821164da50 0x55821166bad0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f80d8010d60 tx=0x7f80d8013000 comp rx=0 tx=0).stop 2024-05-10T11:14:46.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.564+0000 7f80f8146b80 1 -- 172.21.15.27:0/3554156722 shutdown_connections 2024-05-10T11:14:46.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.564+0000 7f80f8146b80 1 --2- 172.21.15.27:0/3554156722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582116536c0 0x558211654330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.564+0000 7f80f8146b80 1 --2- 172.21.15.27:0/3554156722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55821164da50 0x55821166bad0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.564+0000 7f80f8146b80 1 --2- 172.21.15.27:0/3554156722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558211652da0 0x558211650be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.564+0000 7f80f8146b80 1 -- 172.21.15.27:0/3554156722 >> 172.21.15.27:0/3554156722 conn(0x5582115c56d0 msgr2=0x558211657e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:46.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.564+0000 7f80f8146b80 1 -- 172.21.15.27:0/3554156722 shutdown_connections 2024-05-10T11:14:46.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.564+0000 7f80f8146b80 1 -- 172.21.15.27:0/3554156722 wait complete. 2024-05-10T11:14:46.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.565+0000 7f80f8146b80 1 -- start start 2024-05-10T11:14:46.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.565+0000 7f80f8146b80 1 Processor -- start 2024-05-10T11:14:46.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.566+0000 7f80f8146b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 0x558211676d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.566+0000 7f80f8146b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211677280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.566+0000 7f80f8146b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582116536c0 0x5582116d8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.566+0000 7f80f8146b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f80c8010290 con 0x55821164da50 2024-05-10T11:14:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.566+0000 7f80f8146b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f80c800fbc0 con 0x558211652da0 2024-05-10T11:14:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.566+0000 7f80f8146b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f80c800cdc0 con 0x5582116536c0 2024-05-10T11:14:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.566+0000 7f80f8146b80 10 client.0 _ll_register_callbacks cb 0x558211546c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.566+0000 7f80e6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 0x558211676d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.566+0000 7f80e6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 0x558211676d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33210/0 (socket says 172.21.15.27:33210) 2024-05-10T11:14:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.566+0000 7f80e6a00700 1 -- 172.21.15.27:0/29228460 learned_addr learned my addr 172.21.15.27:0/29228460 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.566+0000 7f80e6000700 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211677280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.566+0000 7f80e7400700 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582116536c0 0x5582116d8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.567+0000 7f80e6a00700 1 -- 172.21.15.27:0/29228460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582116536c0 msgr2=0x5582116d8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.567+0000 7f80e6a00700 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582116536c0 0x5582116d8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.567+0000 7f80e6a00700 1 -- 172.21.15.27:0/29228460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 msgr2=0x558211677280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.567+0000 7f80e6a00700 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211677280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.567+0000 7f80e6a00700 1 -- 172.21.15.27:0/29228460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558211677b20 con 0x55821164da50 2024-05-10T11:14:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.568+0000 7f80e6a00700 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 0x558211676d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009a80 tx=0x7f80d8010910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d2e67266e3ed0e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.568+0000 7f80d0c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:14:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.568+0000 7f80d0c00700 1 -- 172.21.15.27:0/29228460 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8035060 con 0x55821164da50 2024-05-10T11:14:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.568+0000 7f80d0c00700 1 -- 172.21.15.27:0/29228460 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f80d8010d90 con 0x55821164da50 2024-05-10T11:14:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.568+0000 7f80f8146b80 1 -- 172.21.15.27:0/29228460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80d80044e0 con 0x55821164da50 2024-05-10T11:14:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.569+0000 7f80d0c00700 1 -- 172.21.15.27:0/29228460 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f80d802c040 con 0x55821164da50 2024-05-10T11:14:46.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.569+0000 7f80f8146b80 10 client.15426 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:46.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.569+0000 7f80f8146b80 1 -- 172.21.15.27:0/29228460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5582116da370 con 0x55821164da50 2024-05-10T11:14:46.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.569+0000 7f80d0c00700 1 -- 172.21.15.27:0/29228460 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80d80044e0 con 0x55821164da50 2024-05-10T11:14:46.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.569+0000 7f80d0c00700 1 -- 172.21.15.27:0/29228460 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f80d8026ce0 con 0x55821164da50 2024-05-10T11:14:46.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.569+0000 7f80f8146b80 10 client.15426 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:46.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.569+0000 7f80f8146b80 10 client.15426 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:14:46.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.569+0000 7f80f8146b80 20 client.15426 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:46.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.569+0000 7f80f8146b80 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5582116da910 0x5582116dcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:46.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.569+0000 7f80f8146b80 4 client.15426 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:14:46.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.569+0000 7f80f8146b80 1 -- 172.21.15.27:0/29228460 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5582115ad400 con 0x5582116da910 2024-05-10T11:14:46.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.570+0000 7f80e6000700 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5582116da910 0x5582116dcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.571+0000 7f80d0c00700 10 client.15426 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.571+0000 7f80e6000700 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5582116da910 0x5582116dcd30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18afecd3290b0a9d server_cookie=752734cbed5b6762 in_seq=0 out_seq=0 2024-05-10T11:14:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.571+0000 7f80d0c00700 1 -- 172.21.15.27:0/29228460 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5582115ad400 con 0x5582116da910 2024-05-10T11:14:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.571+0000 7f80d0c00700 10 client.15426 handle_command_reply: tid=0 2024-05-10T11:14:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.571+0000 7f80f8146b80 1 client.15426 shutdown 2024-05-10T11:14:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.571+0000 7f80f8146b80 1 -- 172.21.15.27:0/29228460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 msgr2=0x558211676d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.571+0000 7f80f8146b80 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 0x558211676d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009a80 tx=0x7f80d8010910 comp rx=0 tx=0).stop 2024-05-10T11:14:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.572+0000 7f80f8146b80 1 -- 172.21.15.27:0/29228460 shutdown_connections 2024-05-10T11:14:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.572+0000 7f80f8146b80 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5582116da910 0x5582116dcd30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.572+0000 7f80f8146b80 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582116536c0 0x5582116d8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.572+0000 7f80f8146b80 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55821164da50 0x558211676d40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.572+0000 7f80f8146b80 1 --2- 172.21.15.27:0/29228460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558211652da0 0x558211677280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.572+0000 7f80f8146b80 1 -- 172.21.15.27:0/29228460 >> 172.21.15.27:0/29228460 conn(0x5582115c56d0 msgr2=0x5582116541b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.572+0000 7f80d0c00700 0 client.15426 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:14:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.572+0000 7f80f8146b80 1 -- 172.21.15.27:0/29228460 shutdown_connections 2024-05-10T11:14:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.572+0000 7f80f8146b80 1 -- 172.21.15.27:0/29228460 wait complete. 2024-05-10T11:14:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.572+0000 7f80f8146b80 20 client.15426 trim_cache size 0 max 16384 2024-05-10T11:14:46.575 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:14:46.575 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:14:46.575 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:14:46.575 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:14:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.573+0000 7f80e7e00700 1 -- 172.21.15.27:0/1960279561 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f80cc0b2a70 msgr2=0x7f80cc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.573+0000 7f80e7e00700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f80cc0b2a70 0x7f80cc0b4f30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f80e819cbc0 tx=0x7f80d8009260 comp rx=0 tx=0).stop 2024-05-10T11:14:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.573+0000 7f80e7e00700 1 -- 172.21.15.27:0/1960279561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e810d9c0 msgr2=0x7f80e819ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.573+0000 7f80e7e00700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e810d9c0 0x7f80e819ba80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f80e000b070 tx=0x7f80e0004c80 comp rx=0 tx=0).stop 2024-05-10T11:14:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.574+0000 7f80e7e00700 1 -- 172.21.15.27:0/1960279561 shutdown_connections 2024-05-10T11:14:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.574+0000 7f80e7e00700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f80cc0b2a70 0x7f80cc0b4f30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.574+0000 7f80e7e00700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80e810d9c0 0x7f80e819ba80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.574+0000 7f80e7e00700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e8104590 0x7f80e81a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.574+0000 7f80e7e00700 1 --2- 172.21.15.27:0/1960279561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8104eb0 0x7f80e81a1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.574+0000 7f80e7e00700 1 -- 172.21.15.27:0/1960279561 >> 172.21.15.27:0/1960279561 conn(0x7f80e80fddc0 msgr2=0x7f80e810ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.574+0000 7f80e7e00700 1 -- 172.21.15.27:0/1960279561 shutdown_connections 2024-05-10T11:14:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:46.574+0000 7f80e7e00700 1 -- 172.21.15.27:0/1960279561 wait complete. 2024-05-10T11:14:46.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:46 smithi076 ceph-mon[34152]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-05-10T11:14:46.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:46 smithi076 ceph-mon[34152]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-05-10T11:14:46.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:46 smithi189 ceph-mon[34842]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-05-10T11:14:46.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:46 smithi189 ceph-mon[34842]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-05-10T11:14:47.464 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:14:47.464 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:14:47.540 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:47 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:14:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:47 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:14:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:47 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:14:47.905 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:14:48.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:48 smithi027 ceph-mon[30054]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-05-10T11:14:48.576 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:48 smithi076 ceph-mon[34152]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-05-10T11:14:48.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:48 smithi189 ceph-mon[34842]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-05-10T11:14:49.548 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:49 smithi027 ceph-mon[30054]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-05-10T11:14:49.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:49 smithi076 ceph-mon[34152]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-05-10T11:14:49.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:49 smithi189 ceph-mon[34842]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-05-10T11:14:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.141+0000 7f7521a00700 1 -- 172.21.15.27:0/3876745017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f751c075ff0 msgr2=0x7f751c10f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.141+0000 7f7521a00700 1 --2- 172.21.15.27:0/3876745017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f751c075ff0 0x7f751c10f9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7508009990 tx=0x7f7508009ca0 comp rx=0 tx=0).stop 2024-05-10T11:14:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.141+0000 7f7521a00700 1 -- 172.21.15.27:0/3876745017 shutdown_connections 2024-05-10T11:14:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.141+0000 7f7521a00700 1 --2- 172.21.15.27:0/3876745017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f751c10ff00 0x7f751c1122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.141+0000 7f7521a00700 1 --2- 172.21.15.27:0/3876745017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f751c0756d0 0x7f751c075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.141+0000 7f7521a00700 1 --2- 172.21.15.27:0/3876745017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f751c075ff0 0x7f751c10f9c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:50.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.141+0000 7f7521a00700 1 -- 172.21.15.27:0/3876745017 >> 172.21.15.27:0/3876745017 conn(0x7f751c07a600 msgr2=0x7f751c07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:50.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.142+0000 7f7521a00700 1 -- 172.21.15.27:0/3876745017 shutdown_connections 2024-05-10T11:14:50.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.143+0000 7f7521a00700 1 -- 172.21.15.27:0/3876745017 wait complete. 2024-05-10T11:14:50.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.144+0000 7f7521a00700 1 Processor -- start 2024-05-10T11:14:50.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.144+0000 7f7521a00700 1 -- start start 2024-05-10T11:14:50.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.144+0000 7f7521a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f751c0756d0 0x7f751c19bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.144+0000 7f7521a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f751c075ff0 0x7f751c19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.144+0000 7f7521a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f751c10ff00 0x7f751c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.144+0000 7f7521a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f751c115910 con 0x7f751c10ff00 2024-05-10T11:14:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.144+0000 7f7521a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f751c115790 con 0x7f751c0756d0 2024-05-10T11:14:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.144+0000 7f7521a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f751c115a90 con 0x7f751c075ff0 2024-05-10T11:14:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.145+0000 7f751a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f751c075ff0 0x7f751c19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.145+0000 7f751a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f751c075ff0 0x7f751c19c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50010/0 (socket says 172.21.15.27:50010) 2024-05-10T11:14:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.145+0000 7f751a000700 1 -- 172.21.15.27:0/3428125013 learned_addr learned my addr 172.21.15.27:0/3428125013 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.145+0000 7f751b400700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f751c10ff00 0x7f751c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.145+0000 7f751aa00700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f751c0756d0 0x7f751c19bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.145+0000 7f751a000700 1 -- 172.21.15.27:0/3428125013 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f751c0756d0 msgr2=0x7f751c19bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.145+0000 7f751a000700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f751c0756d0 0x7f751c19bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.145+0000 7f751a000700 1 -- 172.21.15.27:0/3428125013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f751c10ff00 msgr2=0x7f751c1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.145+0000 7f751a000700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f751c10ff00 0x7f751c1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.145+0000 7f751a000700 1 -- 172.21.15.27:0/3428125013 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f751c1a0d90 con 0x7f751c075ff0 2024-05-10T11:14:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.146+0000 7f751b400700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f751c10ff00 0x7f751c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.146+0000 7f751a000700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f751c075ff0 0x7f751c19c2c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7508000c00 tx=0x7f75080057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.146+0000 7f7513400700 1 -- 172.21.15.27:0/3428125013 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7508027070 con 0x7f751c075ff0 2024-05-10T11:14:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.147+0000 7f7521a00700 1 -- 172.21.15.27:0/3428125013 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f751c1a1020 con 0x7f751c075ff0 2024-05-10T11:14:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.147+0000 7f7521a00700 1 -- 172.21.15.27:0/3428125013 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f751c06cec0 con 0x7f751c075ff0 2024-05-10T11:14:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.147+0000 7f7513400700 1 -- 172.21.15.27:0/3428125013 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7508022470 con 0x7f751c075ff0 2024-05-10T11:14:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.147+0000 7f7513400700 1 -- 172.21.15.27:0/3428125013 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7508017650 con 0x7f751c075ff0 2024-05-10T11:14:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.149+0000 7f7521a00700 1 -- 172.21.15.27:0/3428125013 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74fc005330 con 0x7f751c075ff0 2024-05-10T11:14:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.151+0000 7f7513400700 1 -- 172.21.15.27:0/3428125013 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f750802a070 con 0x7f751c075ff0 2024-05-10T11:14:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.152+0000 7f7513400700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f75000b2a70 0x7f75000b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.153+0000 7f7513400700 1 -- 172.21.15.27:0/3428125013 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f75080e36d0 con 0x7f751c075ff0 2024-05-10T11:14:50.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.153+0000 7f751aa00700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f75000b2a70 0x7f75000b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:50.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.154+0000 7f751aa00700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f75000b2a70 0x7f75000b4f30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f750c009bf0 tx=0x7f750c009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.155+0000 7f7513400700 1 -- 172.21.15.27:0/3428125013 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7508013070 con 0x7f751c075ff0 2024-05-10T11:14:50.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.650+0000 7f7521a00700 1 -- 172.21.15.27:0/3428125013 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74fc005620 con 0x7f751c075ff0 2024-05-10T11:14:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.651+0000 7f7513400700 1 -- 172.21.15.27:0/3428125013 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11936 (secure 0 0 0) 0x7f75080aad10 con 0x7f751c075ff0 2024-05-10T11:14:50.652 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:14:50.652 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:13:10.916396+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24484,24472,24467,24488]},"id":1}]} 2024-05-10T11:14:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.655+0000 7f7521a00700 1 -- 172.21.15.27:0/3428125013 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f75000b2a70 msgr2=0x7f75000b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.655+0000 7f7521a00700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f75000b2a70 0x7f75000b4f30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f750c009bf0 tx=0x7f750c009260 comp rx=0 tx=0).stop 2024-05-10T11:14:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.655+0000 7f7521a00700 1 -- 172.21.15.27:0/3428125013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f751c075ff0 msgr2=0x7f751c19c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.655+0000 7f7521a00700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f751c075ff0 0x7f751c19c2c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7508000c00 tx=0x7f75080057c0 comp rx=0 tx=0).stop 2024-05-10T11:14:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.655+0000 7f7521a00700 1 -- 172.21.15.27:0/3428125013 shutdown_connections 2024-05-10T11:14:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.655+0000 7f7521a00700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f75000b2a70 0x7f75000b4f30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.655+0000 7f7521a00700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f751c10ff00 0x7f751c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.655+0000 7f7521a00700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f751c0756d0 0x7f751c19bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.655+0000 7f7521a00700 1 --2- 172.21.15.27:0/3428125013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f751c075ff0 0x7f751c19c2c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.655+0000 7f7521a00700 1 -- 172.21.15.27:0/3428125013 >> 172.21.15.27:0/3428125013 conn(0x7f751c07a600 msgr2=0x7f751c101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.656+0000 7f7521a00700 1 -- 172.21.15.27:0/3428125013 shutdown_connections 2024-05-10T11:14:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:50.656+0000 7f7521a00700 1 -- 172.21.15.27:0/3428125013 wait complete. 2024-05-10T11:14:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 30 2024-05-10T11:14:50.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:50 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3428125013' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:14:51.052 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:50 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3428125013' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:14:51.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:50 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3428125013' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:14:51.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:14:51.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:51 smithi027 ceph-mon[30054]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-05-10T11:14:51.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:51 smithi189 ceph-mon[34842]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-05-10T11:14:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:14:52.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:51 smithi076 ceph-mon[34152]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-05-10T11:14:54.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.330+0000 7f3b71e00700 1 -- 172.21.15.27:0/1271752543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b6c108460 msgr2=0x7f3b6c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:54.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.330+0000 7f3b71e00700 1 --2- 172.21.15.27:0/1271752543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b6c108460 0x7f3b6c108840 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009990 tx=0x7f3b5c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.330+0000 7f3b71e00700 1 -- 172.21.15.27:0/1271752543 shutdown_connections 2024-05-10T11:14:54.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.330+0000 7f3b71e00700 1 --2- 172.21.15.27:0/1271752543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6c103ef0 0x7f3b6c1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.330+0000 7f3b71e00700 1 --2- 172.21.15.27:0/1271752543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b6c108460 0x7f3b6c108840 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.330+0000 7f3b71e00700 1 --2- 172.21.15.27:0/1271752543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b6c103550 0x7f3b6c1039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.330+0000 7f3b71e00700 1 -- 172.21.15.27:0/1271752543 >> 172.21.15.27:0/1271752543 conn(0x7f3b6c07a600 msgr2=0x7f3b6c07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:54.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.330+0000 7f3b71e00700 1 -- 172.21.15.27:0/1271752543 shutdown_connections 2024-05-10T11:14:54.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.330+0000 7f3b71e00700 1 -- 172.21.15.27:0/1271752543 wait complete. 2024-05-10T11:14:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.332+0000 7f3b71e00700 1 Processor -- start 2024-05-10T11:14:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.332+0000 7f3b71e00700 1 -- start start 2024-05-10T11:14:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.333+0000 7f3b71e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6c103550 0x7f3b6c19be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.333+0000 7f3b71e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b6c103ef0 0x7f3b6c19c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.333+0000 7f3b71e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b6c108460 0x7f3b6c1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.333+0000 7f3b71e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c113680 con 0x7f3b6c103ef0 2024-05-10T11:14:54.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.333+0000 7f3b71e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c113500 con 0x7f3b6c108460 2024-05-10T11:14:54.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.333+0000 7f3b71e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c113800 con 0x7f3b6c103550 2024-05-10T11:14:54.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.333+0000 7f3b6b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b6c108460 0x7f3b6c1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.333+0000 7f3b6aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6c103550 0x7f3b6c19be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.333+0000 7f3b6b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b6c108460 0x7f3b6c1a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54192/0 (socket says 172.21.15.27:54192) 2024-05-10T11:14:54.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.333+0000 7f3b6b400700 1 -- 172.21.15.27:0/3885673066 learned_addr learned my addr 172.21.15.27:0/3885673066 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.334+0000 7f3b6a000700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b6c103ef0 0x7f3b6c19c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.334+0000 7f3b6b400700 1 -- 172.21.15.27:0/3885673066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6c103550 msgr2=0x7f3b6c19be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.334+0000 7f3b6b400700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6c103550 0x7f3b6c19be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.334+0000 7f3b6b400700 1 -- 172.21.15.27:0/3885673066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b6c103ef0 msgr2=0x7f3b6c19c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.334+0000 7f3b6b400700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b6c103ef0 0x7f3b6c19c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.334+0000 7f3b6b400700 1 -- 172.21.15.27:0/3885673066 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b6c1a0dd0 con 0x7f3b6c108460 2024-05-10T11:14:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.334+0000 7f3b6a000700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b6c103ef0 0x7f3b6c19c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.335+0000 7f3b6b400700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b6c108460 0x7f3b6c1a0690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3b64007f10 tx=0x7f3b6400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.335+0000 7f3b70a00700 1 -- 172.21.15.27:0/3885673066 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6400dd20 con 0x7f3b6c108460 2024-05-10T11:14:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.335+0000 7f3b70a00700 1 -- 172.21.15.27:0/3885673066 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b64004020 con 0x7f3b6c108460 2024-05-10T11:14:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.335+0000 7f3b70a00700 1 -- 172.21.15.27:0/3885673066 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64015660 con 0x7f3b6c108460 2024-05-10T11:14:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.335+0000 7f3b71e00700 1 -- 172.21.15.27:0/3885673066 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b6c1a10c0 con 0x7f3b6c108460 2024-05-10T11:14:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.336+0000 7f3b71e00700 1 -- 172.21.15.27:0/3885673066 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b6c1a8940 con 0x7f3b6c108460 2024-05-10T11:14:54.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.339+0000 7f3b70a00700 1 -- 172.21.15.27:0/3885673066 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3b640041d0 con 0x7f3b6c108460 2024-05-10T11:14:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.342+0000 7f3b70a00700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3b500b2a70 0x7f3b500b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.342+0000 7f3b70a00700 1 -- 172.21.15.27:0/3885673066 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f3b640e2330 con 0x7f3b6c108460 2024-05-10T11:14:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.342+0000 7f3b7c736b80 1 Processor -- start 2024-05-10T11:14:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.343+0000 7f3b6aa00700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3b500b2a70 0x7f3b500b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.343+0000 7f3b7c736b80 1 -- start start 2024-05-10T11:14:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.343+0000 7f3b7c736b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 0x559ced96a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.343+0000 7f3b7c736b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96acf0 0x559ced964ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.343+0000 7f3b7c736b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced965580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.343+0000 7f3b7c736b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559ced8aea20 con 0x559ced96a3d0 2024-05-10T11:14:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.343+0000 7f3b7c736b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x559ced84c760 con 0x559ced96acf0 2024-05-10T11:14:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.343+0000 7f3b7c736b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559ced8aa9e0 con 0x559ced965100 2024-05-10T11:14:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.344+0000 7f3b6a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96acf0 0x559ced964ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.344+0000 7f3b6a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96acf0 0x559ced964ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54204/0 (socket says 172.21.15.27:54204) 2024-05-10T11:14:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.344+0000 7f3b6a000700 1 -- 172.21.15.27:0/1236893564 learned_addr learned my addr 172.21.15.27:0/1236893564 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.344+0000 7f3b6b400700 1 --2- 172.21.15.27:0/1236893564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced965580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.344+0000 7f3b6b400700 1 -- 172.21.15.27:0/1236893564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96acf0 msgr2=0x559ced964ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.344+0000 7f3b6b400700 1 --2- 172.21.15.27:0/1236893564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96acf0 0x559ced964ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.344+0000 7f3b6b400700 1 -- 172.21.15.27:0/1236893564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 msgr2=0x559ced96a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.344+0000 7f3b6aa00700 1 --2- 172.21.15.27:0/1236893564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 0x559ced96a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.345+0000 7f3b6b400700 1 --2- 172.21.15.27:0/1236893564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 0x559ced96a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.345+0000 7f3b6b400700 1 -- 172.21.15.27:0/1236893564 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ced7fb8d0 con 0x559ced965100 2024-05-10T11:14:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.345+0000 7f3b6aa00700 1 --2- 172.21.15.27:0/1236893564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 0x559ced96a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:14:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.345+0000 7f3b6b400700 1 --2- 172.21.15.27:0/1236893564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced965580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3b640df650 tx=0x7f3b640a8b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f574a405a3b2c01 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.345+0000 7f3b6aa00700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3b500b2a70 0x7f3b500b4f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00b2e0 tx=0x7f3b5c005960 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.346+0000 7f3b60c00700 1 -- 172.21.15.27:0/1236893564 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b640a9840 con 0x559ced965100 2024-05-10T11:14:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.346+0000 7f3b60c00700 1 -- 172.21.15.27:0/1236893564 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b640a9140 con 0x559ced965100 2024-05-10T11:14:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.346+0000 7f3b60c00700 1 -- 172.21.15.27:0/1236893564 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64004bd0 con 0x559ced965100 2024-05-10T11:14:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.346+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1236893564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 msgr2=0x559ced965580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.346+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/1236893564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced965580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3b640df650 tx=0x7f3b640a8b60 comp rx=0 tx=0).stop 2024-05-10T11:14:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.347+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1236893564 shutdown_connections 2024-05-10T11:14:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.347+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/1236893564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced965580 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.347+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/1236893564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 0x559ced96a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.347+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/1236893564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96acf0 0x559ced964ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.347+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1236893564 >> 172.21.15.27:0/1236893564 conn(0x559ced8d46d0 msgr2=0x559ced956b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:54.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.347+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1236893564 shutdown_connections 2024-05-10T11:14:54.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.347+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1236893564 wait complete. 2024-05-10T11:14:54.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.348+0000 7f3b7c736b80 1 -- start start 2024-05-10T11:14:54.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.348+0000 7f3b7c736b80 1 Processor -- start 2024-05-10T11:14:54.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.349+0000 7f3b7c736b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced965100 0x559ced970a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.349+0000 7f3b7c736b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 0x559ced96de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.349+0000 7f3b7c736b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced96acf0 0x559ced96e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.349+0000 7f3b7c736b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559ced971730 con 0x559ced96a3d0 2024-05-10T11:14:54.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.349+0000 7f3b7c736b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x559ced814b20 con 0x559ced965100 2024-05-10T11:14:54.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.349+0000 7f3b7c736b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559ced9718b0 con 0x559ced96acf0 2024-05-10T11:14:54.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.349+0000 7f3b7c736b80 10 client.0 _ll_register_callbacks cb 0x559ced855c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:54.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.349+0000 7f3b6a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 0x559ced96de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.349+0000 7f3b6b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced96acf0 0x559ced96e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.349+0000 7f3b6a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 0x559ced96de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46656/0 (socket says 172.21.15.27:46656) 2024-05-10T11:14:54.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.349+0000 7f3b6a000700 1 -- 172.21.15.27:0/411409831 learned_addr learned my addr 172.21.15.27:0/411409831 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:54.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.349+0000 7f3b6aa00700 1 --2- 172.21.15.27:0/411409831 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced965100 0x559ced970a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.350+0000 7f3b6a000700 1 -- 172.21.15.27:0/411409831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced96acf0 msgr2=0x559ced96e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:54.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.350+0000 7f3b6a000700 1 --2- 172.21.15.27:0/411409831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced96acf0 0x559ced96e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.350+0000 7f3b6a000700 1 -- 172.21.15.27:0/411409831 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced965100 msgr2=0x559ced970a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:54.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.350+0000 7f3b6a000700 1 --2- 172.21.15.27:0/411409831 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced965100 0x559ced970a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.350+0000 7f3b6a000700 1 -- 172.21.15.27:0/411409831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ced874420 con 0x559ced96a3d0 2024-05-10T11:14:54.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.350+0000 7f3b6a000700 1 --2- 172.21.15.27:0/411409831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 0x559ced96de20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3b58012d20 tx=0x7f3b5800f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a8489a9fcc4332e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:54.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.350+0000 7f3b55600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:14:54.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.351+0000 7f3b55600700 1 -- 172.21.15.27:0/411409831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5800fc30 con 0x559ced96a3d0 2024-05-10T11:14:54.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.351+0000 7f3b55600700 1 -- 172.21.15.27:0/411409831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f3b5800fdd0 con 0x559ced96a3d0 2024-05-10T11:14:54.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.351+0000 7f3b7c736b80 1 -- 172.21.15.27:0/411409831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b640a9140 con 0x559ced96a3d0 2024-05-10T11:14:54.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.351+0000 7f3b55600700 1 -- 172.21.15.27:0/411409831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3b58004050 con 0x559ced96a3d0 2024-05-10T11:14:54.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.351+0000 7f3b7c736b80 10 client.15441 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:54.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.351+0000 7f3b7c736b80 1 -- 172.21.15.27:0/411409831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559ced97c060 con 0x559ced96a3d0 2024-05-10T11:14:54.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.352+0000 7f3b55600700 1 -- 172.21.15.27:0/411409831 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b640a9140 con 0x559ced96a3d0 2024-05-10T11:14:54.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.352+0000 7f3b55600700 1 -- 172.21.15.27:0/411409831 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3b58010430 con 0x559ced96a3d0 2024-05-10T11:14:54.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.352+0000 7f3b7c736b80 10 client.15441 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:54.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.352+0000 7f3b7c736b80 10 client.15441 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:14:54.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.352+0000 7f3b7c736b80 20 client.15441 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:54.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.352+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/411409831 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559ced97e6f0 0x559ced980ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.352+0000 7f3b7c736b80 4 client.15441 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:14:54.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.352+0000 7f3b7c736b80 1 -- 172.21.15.27:0/411409831 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ced929140 con 0x559ced97e6f0 2024-05-10T11:14:54.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.352+0000 7f3b6aa00700 1 --2- 172.21.15.27:0/411409831 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559ced97e6f0 0x559ced980ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.353+0000 7f3b55600700 10 client.15441 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:14:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.353+0000 7f3b6aa00700 1 --2- 172.21.15.27:0/411409831 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559ced97e6f0 0x559ced980ae0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=44b11a27fb95549f server_cookie=1355e98f0d5a05d2 in_seq=0 out_seq=0 2024-05-10T11:14:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.354+0000 7f3b55600700 1 -- 172.21.15.27:0/411409831 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559ced929140 con 0x559ced97e6f0 2024-05-10T11:14:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.354+0000 7f3b55600700 10 client.15441 handle_command_reply: tid=0 2024-05-10T11:14:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.354+0000 7f3b7c736b80 1 client.15441 shutdown 2024-05-10T11:14:54.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.355+0000 7f3b7c736b80 1 -- 172.21.15.27:0/411409831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 msgr2=0x559ced96de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:54.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.355+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/411409831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 0x559ced96de20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3b58012d20 tx=0x7f3b5800f6a0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.355+0000 7f3b7c736b80 1 -- 172.21.15.27:0/411409831 shutdown_connections 2024-05-10T11:14:54.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.355+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/411409831 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559ced97e6f0 0x559ced980ae0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.355+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/411409831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced96acf0 0x559ced96e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.355+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/411409831 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced965100 0x559ced970a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.355+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/411409831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96a3d0 0x559ced96de20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.355+0000 7f3b7c736b80 1 -- 172.21.15.27:0/411409831 >> 172.21.15.27:0/411409831 conn(0x559ced8d46d0 msgr2=0x559ced956190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.355+0000 7f3b55600700 0 client.15441 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:14:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.355+0000 7f3b7c736b80 1 -- 172.21.15.27:0/411409831 shutdown_connections 2024-05-10T11:14:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.355+0000 7f3b7c736b80 1 -- 172.21.15.27:0/411409831 wait complete. 2024-05-10T11:14:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.356+0000 7f3b7c736b80 20 client.15441 trim_cache size 0 max 16384 2024-05-10T11:14:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.373+0000 7f3b7c736b80 1 Processor -- start 2024-05-10T11:14:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.373+0000 7f3b7c736b80 1 -- start start 2024-05-10T11:14:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.374+0000 7f3b7c736b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced964ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.374+0000 7f3b7c736b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 0x559ced97a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.374+0000 7f3b7c736b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96acf0 0x559ced96b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.374+0000 7f3b7c736b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c1ad390 con 0x559ced96acf0 2024-05-10T11:14:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.374+0000 7f3b7c736b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c1aeed0 con 0x559ced96a3d0 2024-05-10T11:14:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.374+0000 7f3b7c736b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c1ada60 con 0x559ced965100 2024-05-10T11:14:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.374+0000 7f3b6a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 0x559ced97a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.374+0000 7f3b6aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced964ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.374+0000 7f3b6a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 0x559ced97a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54224/0 (socket says 172.21.15.27:54224) 2024-05-10T11:14:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.374+0000 7f3b6a000700 1 -- 172.21.15.27:0/2424364221 learned_addr learned my addr 172.21.15.27:0/2424364221 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:54.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.375+0000 7f3b6b400700 1 --2- 172.21.15.27:0/2424364221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96acf0 0x559ced96b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.375+0000 7f3b6a000700 1 -- 172.21.15.27:0/2424364221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 msgr2=0x559ced964ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:54.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.375+0000 7f3b6a000700 1 --2- 172.21.15.27:0/2424364221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced964ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.375+0000 7f3b6a000700 1 -- 172.21.15.27:0/2424364221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96acf0 msgr2=0x559ced96b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:54.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.375+0000 7f3b6a000700 1 --2- 172.21.15.27:0/2424364221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96acf0 0x559ced96b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.375+0000 7f3b6a000700 1 -- 172.21.15.27:0/2424364221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ced951ab0 con 0x559ced96a3d0 2024-05-10T11:14:54.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.376+0000 7f3b6a000700 1 --2- 172.21.15.27:0/2424364221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 0x559ced97a7c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3b58012cf0 tx=0x7f3b58009400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d270bd4e05790ae5 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.376+0000 7f3b57e00700 1 -- 172.21.15.27:0/2424364221 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b58010b80 con 0x559ced96a3d0 2024-05-10T11:14:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.376+0000 7f3b57e00700 1 -- 172.21.15.27:0/2424364221 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b58005710 con 0x559ced96a3d0 2024-05-10T11:14:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.376+0000 7f3b57e00700 1 -- 172.21.15.27:0/2424364221 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b58003680 con 0x559ced96a3d0 2024-05-10T11:14:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.377+0000 7f3b7c736b80 1 -- 172.21.15.27:0/2424364221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 msgr2=0x559ced97a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.377+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/2424364221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 0x559ced97a7c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3b58012cf0 tx=0x7f3b58009400 comp rx=0 tx=0).stop 2024-05-10T11:14:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.378+0000 7f3b7c736b80 1 -- 172.21.15.27:0/2424364221 shutdown_connections 2024-05-10T11:14:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.378+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/2424364221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced964ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.378+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/2424364221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96acf0 0x559ced96b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.378+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/2424364221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 0x559ced97a7c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.378+0000 7f3b7c736b80 1 -- 172.21.15.27:0/2424364221 >> 172.21.15.27:0/2424364221 conn(0x559ced8d46d0 msgr2=0x559ced96b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:54.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.379+0000 7f3b7c736b80 1 -- 172.21.15.27:0/2424364221 shutdown_connections 2024-05-10T11:14:54.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.379+0000 7f3b7c736b80 1 -- 172.21.15.27:0/2424364221 wait complete. 2024-05-10T11:14:54.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.379+0000 7f3b7c736b80 1 -- start start 2024-05-10T11:14:54.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.380+0000 7f3b7c736b80 1 Processor -- start 2024-05-10T11:14:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.380+0000 7f3b7c736b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced976d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.380+0000 7f3b7c736b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 0x559ced977260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.380+0000 7f3b7c736b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96acf0 0x559ced9856f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.380+0000 7f3b7c736b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c1ae800 con 0x559ced96acf0 2024-05-10T11:14:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.380+0000 7f3b7c736b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c1ae130 con 0x559ced96a3d0 2024-05-10T11:14:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.380+0000 7f3b7c736b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c1ab330 con 0x559ced965100 2024-05-10T11:14:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.381+0000 7f3b7c736b80 10 client.0 _ll_register_callbacks cb 0x559ced855c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:54.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.381+0000 7f3b6a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 0x559ced977260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.381+0000 7f3b6a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 0x559ced977260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54234/0 (socket says 172.21.15.27:54234) 2024-05-10T11:14:54.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.381+0000 7f3b6a000700 1 -- 172.21.15.27:0/1607050567 learned_addr learned my addr 172.21.15.27:0/1607050567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:54.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.381+0000 7f3b6aa00700 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced976d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.381+0000 7f3b6b400700 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96acf0 0x559ced9856f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.382+0000 7f3b6aa00700 1 -- 172.21.15.27:0/1607050567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 msgr2=0x559ced977260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:54.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.382+0000 7f3b6aa00700 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 0x559ced977260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.382+0000 7f3b6aa00700 1 -- 172.21.15.27:0/1607050567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96acf0 msgr2=0x559ced9856f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:54.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.382+0000 7f3b6aa00700 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96acf0 0x559ced9856f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.382+0000 7f3b6aa00700 1 -- 172.21.15.27:0/1607050567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ced985f30 con 0x559ced965100 2024-05-10T11:14:54.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.382+0000 7f3b6aa00700 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced976d20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0059d0 tx=0x7f3b5c009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eaf0e2472d82bc5 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:54.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.382+0000 7f3b55600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:14:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.383+0000 7f3b55600700 1 -- 172.21.15.27:0/1607050567 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c03a070 con 0x559ced965100 2024-05-10T11:14:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.383+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1607050567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b58005710 con 0x559ced965100 2024-05-10T11:14:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.383+0000 7f3b55600700 1 -- 172.21.15.27:0/1607050567 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f3b5c003880 con 0x559ced965100 2024-05-10T11:14:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.384+0000 7f3b55600700 1 -- 172.21.15.27:0/1607050567 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3b5c046050 con 0x559ced965100 2024-05-10T11:14:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.384+0000 7f3b7c736b80 10 client.25585 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.384+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1607050567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559ced9e8c20 con 0x559ced965100 2024-05-10T11:14:54.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.384+0000 7f3b55600700 1 -- 172.21.15.27:0/1607050567 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b58005710 con 0x559ced965100 2024-05-10T11:14:54.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.384+0000 7f3b55600700 1 -- 172.21.15.27:0/1607050567 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3b5c00f460 con 0x559ced965100 2024-05-10T11:14:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.385+0000 7f3b7c736b80 10 client.25585 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.385+0000 7f3b7c736b80 10 client.25585 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:14:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.385+0000 7f3b7c736b80 20 client.25585 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.385+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559ced9e92e0 0x559ced9eb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.385+0000 7f3b7c736b80 4 client.25585 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:14:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.385+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1607050567 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559ced8bc400 con 0x559ced9e92e0 2024-05-10T11:14:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.385+0000 7f3b6a000700 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559ced9e92e0 0x559ced9eb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.386+0000 7f3b6a000700 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559ced9e92e0 0x559ced9eb700 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e9f10ae9f846e97 server_cookie=b0f012b9fefd5a73 in_seq=0 out_seq=0 2024-05-10T11:14:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.386+0000 7f3b55600700 10 client.25585 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:14:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.386+0000 7f3b55600700 1 -- 172.21.15.27:0/1607050567 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559ced8bc400 con 0x559ced9e92e0 2024-05-10T11:14:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.386+0000 7f3b55600700 10 client.25585 handle_command_reply: tid=0 2024-05-10T11:14:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.386+0000 7f3b7c736b80 1 client.25585 shutdown 2024-05-10T11:14:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.387+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1607050567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 msgr2=0x559ced976d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.387+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced976d20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0059d0 tx=0x7f3b5c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.387+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1607050567 shutdown_connections 2024-05-10T11:14:54.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.387+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559ced9e92e0 0x559ced9eb700 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.387+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ced965100 0x559ced976d20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.387+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559ced96acf0 0x559ced9856f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.387+0000 7f3b7c736b80 1 --2- 172.21.15.27:0/1607050567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559ced96a3d0 0x559ced977260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.387+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1607050567 >> 172.21.15.27:0/1607050567 conn(0x559ced8d46d0 msgr2=0x559ced96b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:54.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.387+0000 7f3b55600700 0 client.25585 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:14:54.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.387+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1607050567 shutdown_connections 2024-05-10T11:14:54.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.387+0000 7f3b7c736b80 1 -- 172.21.15.27:0/1607050567 wait complete. 2024-05-10T11:14:54.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.387+0000 7f3b7c736b80 20 client.25585 trim_cache size 0 max 16384 2024-05-10T11:14:54.388 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:14:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.388+0000 7f3b71e00700 1 -- 172.21.15.27:0/3885673066 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3b500b2a70 msgr2=0x7f3b500b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.388+0000 7f3b71e00700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3b500b2a70 0x7f3b500b4f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00b2e0 tx=0x7f3b5c005960 comp rx=0 tx=0).stop 2024-05-10T11:14:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.388+0000 7f3b71e00700 1 -- 172.21.15.27:0/3885673066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b6c108460 msgr2=0x7f3b6c1a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.388+0000 7f3b71e00700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b6c108460 0x7f3b6c1a0690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3b64007f10 tx=0x7f3b6400d3b0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.389+0000 7f3b71e00700 1 -- 172.21.15.27:0/3885673066 shutdown_connections 2024-05-10T11:14:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.389+0000 7f3b71e00700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3b500b2a70 0x7f3b500b4f30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.389+0000 7f3b71e00700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6c103550 0x7f3b6c19be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.389+0000 7f3b71e00700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b6c108460 0x7f3b6c1a0690 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.389+0000 7f3b71e00700 1 --2- 172.21.15.27:0/3885673066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b6c103ef0 0x7f3b6c19c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.389+0000 7f3b71e00700 1 -- 172.21.15.27:0/3885673066 >> 172.21.15.27:0/3885673066 conn(0x7f3b6c07a600 msgr2=0x7f3b6c101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.389+0000 7f3b71e00700 1 -- 172.21.15.27:0/3885673066 shutdown_connections 2024-05-10T11:14:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:54.389+0000 7f3b71e00700 1 -- 172.21.15.27:0/3885673066 wait complete. 2024-05-10T11:14:54.414 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:54 smithi027 ceph-mon[30054]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-10T11:14:54.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:54 smithi076 ceph-mon[34152]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-10T11:14:54.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:54 smithi189 ceph-mon[34842]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-10T11:14:55.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:14:55.953 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:14:56.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:56 smithi027 ceph-mon[30054]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-05-10T11:14:56.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:56 smithi076 ceph-mon[34152]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-05-10T11:14:56.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:56 smithi189 ceph-mon[34842]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-05-10T11:14:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.432+0000 7f180a000700 1 -- 172.21.15.27:0/3251152954 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18041060a0 msgr2=0x7f18041125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.432+0000 7f180a000700 1 --2- 172.21.15.27:0/3251152954 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18041060a0 0x7f18041125b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00b0a0 tx=0x7f17fc00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.432+0000 7f180a000700 1 -- 172.21.15.27:0/3251152954 shutdown_connections 2024-05-10T11:14:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.432+0000 7f180a000700 1 --2- 172.21.15.27:0/3251152954 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18041060a0 0x7f18041125b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.432+0000 7f180a000700 1 --2- 172.21.15.27:0/3251152954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f180410b6e0 0x7f180410bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.432+0000 7f180a000700 1 --2- 172.21.15.27:0/3251152954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18041056e0 0x7f1804105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.432+0000 7f180a000700 1 -- 172.21.15.27:0/3251152954 >> 172.21.15.27:0/3251152954 conn(0x7f18041011d0 msgr2=0x7f18041035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.432+0000 7f180a000700 1 -- 172.21.15.27:0/3251152954 shutdown_connections 2024-05-10T11:14:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.433+0000 7f180a000700 1 -- 172.21.15.27:0/3251152954 wait complete. 2024-05-10T11:14:58.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.434+0000 7f180a000700 1 Processor -- start 2024-05-10T11:14:58.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.435+0000 7f180a000700 1 -- start start 2024-05-10T11:14:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.435+0000 7f180a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18041056e0 0x7f1804074bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.435+0000 7f180a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18041060a0 0x7f1804075110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.435+0000 7f180a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f180410b6e0 0x7f180406deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.435+0000 7f180a000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1804115920 con 0x7f180410b6e0 2024-05-10T11:14:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.435+0000 7f180a000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f18041157a0 con 0x7f18041060a0 2024-05-10T11:14:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.435+0000 7f180a000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1804115aa0 con 0x7f18041056e0 2024-05-10T11:14:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.435+0000 7f1803400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18041056e0 0x7f1804074bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.436+0000 7f1803400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18041056e0 0x7f1804074bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:34860/0 (socket says 172.21.15.27:34860) 2024-05-10T11:14:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.436+0000 7f1803400700 1 -- 172.21.15.27:0/1624465773 learned_addr learned my addr 172.21.15.27:0/1624465773 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.436+0000 7f1802a00700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18041060a0 0x7f1804075110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.436+0000 7f1803e00700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f180410b6e0 0x7f180406deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.436+0000 7f1802a00700 1 -- 172.21.15.27:0/1624465773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18041056e0 msgr2=0x7f1804074bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.436+0000 7f1802a00700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18041056e0 0x7f1804074bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.436+0000 7f1802a00700 1 -- 172.21.15.27:0/1624465773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f180410b6e0 msgr2=0x7f180406deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.436+0000 7f1802a00700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f180410b6e0 0x7f180406deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.436+0000 7f1802a00700 1 -- 172.21.15.27:0/1624465773 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180406e7b0 con 0x7f18041060a0 2024-05-10T11:14:58.437 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:14:58 smithi027 ceph-mon[30054]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-10T11:14:58.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.437+0000 7f1802a00700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18041060a0 0x7f1804075110 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f17f000e9f0 tx=0x7f17f000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:58.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.437+0000 7f17fbe00700 1 -- 172.21.15.27:0/1624465773 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f000cbd0 con 0x7f18041060a0 2024-05-10T11:14:58.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.437+0000 7f17fbe00700 1 -- 172.21.15.27:0/1624465773 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17f000cd70 con 0x7f18041060a0 2024-05-10T11:14:58.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.437+0000 7f17fbe00700 1 -- 172.21.15.27:0/1624465773 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0018710 con 0x7f18041060a0 2024-05-10T11:14:58.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.437+0000 7f180a000700 1 -- 172.21.15.27:0/1624465773 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180406eaa0 con 0x7f18041060a0 2024-05-10T11:14:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.437+0000 7f180a000700 1 -- 172.21.15.27:0/1624465773 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18040728d0 con 0x7f18041060a0 2024-05-10T11:14:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.444+0000 7f181493fb80 1 Processor -- start 2024-05-10T11:14:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.445+0000 7f181493fb80 1 -- start start 2024-05-10T11:14:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.445+0000 7f181493fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.445+0000 7f17fbe00700 1 -- 172.21.15.27:0/1624465773 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f17f0010430 con 0x7f18041060a0 2024-05-10T11:14:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.445+0000 7f181493fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.445+0000 7f181493fb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.445+0000 7f181493fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b0ec21ca20 con 0x55b0ec2cfbc0 2024-05-10T11:14:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.445+0000 7f181493fb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b0ec1ba760 con 0x55b0ec2ca8f0 2024-05-10T11:14:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.445+0000 7f181493fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b0ec2189e0 con 0x55b0ec2d04e0 2024-05-10T11:14:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.446+0000 7f1803e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.446+0000 7f1803400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.446+0000 7f1803400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2ca2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:34876/0 (socket says 172.21.15.27:34876) 2024-05-10T11:14:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.446+0000 7f1803400700 1 -- 172.21.15.27:0/93824845 learned_addr learned my addr 172.21.15.27:0/93824845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.446+0000 7f1803e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2cffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46712/0 (socket says 172.21.15.27:46712) 2024-05-10T11:14:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.446+0000 7f1802a00700 1 --2- 172.21.15.27:0/93824845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.447+0000 7f1803e00700 1 -- 172.21.15.27:0/93824845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2d04e0 msgr2=0x55b0ec2ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.447+0000 7f1803e00700 1 --2- 172.21.15.27:0/93824845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.447+0000 7f1803e00700 1 -- 172.21.15.27:0/93824845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2ca8f0 msgr2=0x55b0ec2cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.447+0000 7f1803e00700 1 --2- 172.21.15.27:0/93824845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.447+0000 7f1803e00700 1 -- 172.21.15.27:0/93824845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0ec1698d0 con 0x55b0ec2cfbc0 2024-05-10T11:14:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.447+0000 7f1803400700 1 --2- 172.21.15.27:0/93824845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:14:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.447+0000 7f1803e00700 1 --2- 172.21.15.27:0/93824845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2cffa0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f17fc0062a0 tx=0x7f17fc0092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e94986340f11369 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.449+0000 7f17f8c00700 1 -- 172.21.15.27:0/93824845 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc00bcb0 con 0x55b0ec2cfbc0 2024-05-10T11:14:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.449+0000 7f17f8c00700 1 -- 172.21.15.27:0/93824845 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17fc0074e0 con 0x55b0ec2cfbc0 2024-05-10T11:14:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.449+0000 7f17f8c00700 1 -- 172.21.15.27:0/93824845 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc005260 con 0x55b0ec2cfbc0 2024-05-10T11:14:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.450+0000 7f17fbe00700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f17e80b2af0 0x7f17e80b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.450+0000 7f181493fb80 1 -- 172.21.15.27:0/93824845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 msgr2=0x55b0ec2cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.450+0000 7f181493fb80 1 --2- 172.21.15.27:0/93824845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2cffa0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f17fc0062a0 tx=0x7f17fc0092e0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.450+0000 7f1803400700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f17e80b2af0 0x7f17e80b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.450+0000 7f17fbe00700 1 -- 172.21.15.27:0/1624465773 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f17f00e2bb0 con 0x7f18041060a0 2024-05-10T11:14:58.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.451+0000 7f181493fb80 1 -- 172.21.15.27:0/93824845 shutdown_connections 2024-05-10T11:14:58.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.451+0000 7f181493fb80 1 --2- 172.21.15.27:0/93824845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.451+0000 7f181493fb80 1 --2- 172.21.15.27:0/93824845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2cffa0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.451+0000 7f181493fb80 1 --2- 172.21.15.27:0/93824845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.451+0000 7f181493fb80 1 -- 172.21.15.27:0/93824845 >> 172.21.15.27:0/93824845 conn(0x55b0ec2426d0 msgr2=0x55b0ec2c4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:58.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.451+0000 7f181493fb80 1 -- 172.21.15.27:0/93824845 shutdown_connections 2024-05-10T11:14:58.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.451+0000 7f181493fb80 1 -- 172.21.15.27:0/93824845 wait complete. 2024-05-10T11:14:58.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.451+0000 7f1803400700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f17e80b2af0 0x7f17e80b4fb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55b0ec2d08e0 tx=0x7f17f4009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:58.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.452+0000 7f181493fb80 1 -- start start 2024-05-10T11:14:58.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.452+0000 7f181493fb80 1 Processor -- start 2024-05-10T11:14:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.453+0000 7f181493fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2d6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.453+0000 7f181493fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.453+0000 7f181493fb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2d3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.453+0000 7f181493fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b0ec2d6ec0 con 0x55b0ec2ca8f0 2024-05-10T11:14:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.453+0000 7f181493fb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b0ec182b20 con 0x55b0ec2d04e0 2024-05-10T11:14:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.453+0000 7f181493fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b0ec2d7040 con 0x55b0ec2cfbc0 2024-05-10T11:14:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.453+0000 7f181493fb80 10 client.0 _ll_register_callbacks cb 0x55b0ec1c3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.454+0000 7f1803400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2d6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.454+0000 7f1803400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2d6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46726/0 (socket says 172.21.15.27:46726) 2024-05-10T11:14:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.454+0000 7f1803400700 1 -- 172.21.15.27:0/3131240413 learned_addr learned my addr 172.21.15.27:0/3131240413 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.454+0000 7f1803e00700 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2d3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.454+0000 7f1803400700 1 -- 172.21.15.27:0/3131240413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2cfbc0 msgr2=0x55b0ec2d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.454+0000 7f1802a00700 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.454+0000 7f1803400700 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2d3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.455+0000 7f1803400700 1 -- 172.21.15.27:0/3131240413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2d04e0 msgr2=0x55b0ec2d3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:58.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.455+0000 7f1803400700 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2d3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.455+0000 7f1803400700 1 -- 172.21.15.27:0/3131240413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0ec1e2420 con 0x55b0ec2ca8f0 2024-05-10T11:14:58.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.457+0000 7f1803400700 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2d6180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f17f401f9e0 tx=0x7f17f40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f24eb658532de0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:58.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.457+0000 7f17ed600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:14:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.457+0000 7f17ed600700 1 -- 172.21.15.27:0/3131240413 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f4004020 con 0x55b0ec2ca8f0 2024-05-10T11:14:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.457+0000 7f17ed600700 1 -- 172.21.15.27:0/3131240413 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f17f4017030 con 0x55b0ec2ca8f0 2024-05-10T11:14:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.457+0000 7f181493fb80 1 -- 172.21.15.27:0/3131240413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17fc0074e0 con 0x55b0ec2ca8f0 2024-05-10T11:14:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.458+0000 7f17ed600700 1 -- 172.21.15.27:0/3131240413 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f17f4032050 con 0x55b0ec2ca8f0 2024-05-10T11:14:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.458+0000 7f181493fb80 10 client.15456 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.458+0000 7f181493fb80 1 -- 172.21.15.27:0/3131240413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b0ec2e1740 con 0x55b0ec2ca8f0 2024-05-10T11:14:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.458+0000 7f17ed600700 1 -- 172.21.15.27:0/3131240413 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17fc0074e0 con 0x55b0ec2ca8f0 2024-05-10T11:14:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.458+0000 7f17ed600700 1 -- 172.21.15.27:0/3131240413 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f17f4003680 con 0x55b0ec2ca8f0 2024-05-10T11:14:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.458+0000 7f181493fb80 10 client.15456 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.459+0000 7f181493fb80 10 client.15456 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:14:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.459+0000 7f181493fb80 20 client.15456 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.459+0000 7f181493fb80 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b0ec2e3d80 0x55b0ec2e6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.459+0000 7f181493fb80 4 client.15456 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:14:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.459+0000 7f181493fb80 1 -- 172.21.15.27:0/3131240413 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0ec297140 con 0x55b0ec2e3d80 2024-05-10T11:14:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.459+0000 7f1802a00700 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b0ec2e3d80 0x55b0ec2e6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.460+0000 7f1802a00700 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b0ec2e3d80 0x55b0ec2e6200 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1dd03419a8d8bdc0 server_cookie=f0c656739795e0ff in_seq=0 out_seq=0 2024-05-10T11:14:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.460+0000 7f17ed600700 10 client.15456 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:14:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.461+0000 7f17ed600700 1 -- 172.21.15.27:0/3131240413 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b0ec297140 con 0x55b0ec2e3d80 2024-05-10T11:14:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.461+0000 7f17ed600700 10 client.15456 handle_command_reply: tid=0 2024-05-10T11:14:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.461+0000 7f181493fb80 1 client.15456 shutdown 2024-05-10T11:14:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.461+0000 7f181493fb80 1 -- 172.21.15.27:0/3131240413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2ca8f0 msgr2=0x55b0ec2d6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.461+0000 7f181493fb80 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2d6180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f17f401f9e0 tx=0x7f17f40047c0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.461+0000 7f181493fb80 1 -- 172.21.15.27:0/3131240413 shutdown_connections 2024-05-10T11:14:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.461+0000 7f181493fb80 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b0ec2e3d80 0x55b0ec2e6200 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.461+0000 7f181493fb80 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2d3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.461+0000 7f181493fb80 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2d6180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.461+0000 7f181493fb80 1 --2- 172.21.15.27:0/3131240413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2d3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.461+0000 7f17ed600700 0 client.15456 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:14:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.461+0000 7f181493fb80 1 -- 172.21.15.27:0/3131240413 >> 172.21.15.27:0/3131240413 conn(0x55b0ec2426d0 msgr2=0x55b0ec2c4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:58.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.462+0000 7f181493fb80 1 -- 172.21.15.27:0/3131240413 shutdown_connections 2024-05-10T11:14:58.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.462+0000 7f181493fb80 1 -- 172.21.15.27:0/3131240413 wait complete. 2024-05-10T11:14:58.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.462+0000 7f181493fb80 20 client.15456 trim_cache size 0 max 16384 2024-05-10T11:14:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.479+0000 7f181493fb80 1 Processor -- start 2024-05-10T11:14:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.479+0000 7f181493fb80 1 -- start start 2024-05-10T11:14:58.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.479+0000 7f181493fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2dff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.479+0000 7f181493fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.479+0000 7f181493fb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.479+0000 7f181493fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f18041be460 con 0x55b0ec2cfbc0 2024-05-10T11:14:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.479+0000 7f181493fb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f18041bffa0 con 0x55b0ec2d04e0 2024-05-10T11:14:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.479+0000 7f181493fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f18041beb30 con 0x55b0ec2ca8f0 2024-05-10T11:14:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.480+0000 7f1803e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.480+0000 7f1803e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54292/0 (socket says 172.21.15.27:54292) 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.480+0000 7f1803e00700 1 -- 172.21.15.27:0/882153648 learned_addr learned my addr 172.21.15.27:0/882153648 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.480+0000 7f1803400700 1 --2- 172.21.15.27:0/882153648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2dff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.480+0000 7f1802a00700 1 --2- 172.21.15.27:0/882153648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.480+0000 7f1803e00700 1 -- 172.21.15.27:0/882153648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2ca8f0 msgr2=0x55b0ec2dff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.480+0000 7f1803e00700 1 --2- 172.21.15.27:0/882153648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2dff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.480+0000 7f1803e00700 1 -- 172.21.15.27:0/882153648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 msgr2=0x55b0ec2d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.480+0000 7f1803e00700 1 --2- 172.21.15.27:0/882153648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.480+0000 7f1803e00700 1 -- 172.21.15.27:0/882153648 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0ec2cdfc0 con 0x55b0ec2d04e0 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.481+0000 7f1802a00700 1 --2- 172.21.15.27:0/882153648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.481+0000 7f1803e00700 1 --2- 172.21.15.27:0/882153648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2c4910 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f17fc009fd0 tx=0x7f17fc00bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=557d7d12ed483ec8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.481+0000 7f17efe00700 1 -- 172.21.15.27:0/882153648 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc00bde0 con 0x55b0ec2d04e0 2024-05-10T11:14:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.481+0000 7f17efe00700 1 -- 172.21.15.27:0/882153648 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17fc02a070 con 0x55b0ec2d04e0 2024-05-10T11:14:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.481+0000 7f17efe00700 1 -- 172.21.15.27:0/882153648 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc007cd0 con 0x55b0ec2d04e0 2024-05-10T11:14:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.482+0000 7f181493fb80 1 -- 172.21.15.27:0/882153648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2d04e0 msgr2=0x55b0ec2c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.482+0000 7f181493fb80 1 --2- 172.21.15.27:0/882153648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2c4910 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f17fc009fd0 tx=0x7f17fc00bc40 comp rx=0 tx=0).stop 2024-05-10T11:14:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.482+0000 7f181493fb80 1 -- 172.21.15.27:0/882153648 shutdown_connections 2024-05-10T11:14:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.482+0000 7f181493fb80 1 --2- 172.21.15.27:0/882153648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2dff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.482+0000 7f181493fb80 1 --2- 172.21.15.27:0/882153648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2c4910 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.482+0000 7f181493fb80 1 --2- 172.21.15.27:0/882153648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.482+0000 7f181493fb80 1 -- 172.21.15.27:0/882153648 >> 172.21.15.27:0/882153648 conn(0x55b0ec2426d0 msgr2=0x55b0ec2d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.483+0000 7f181493fb80 1 -- 172.21.15.27:0/882153648 shutdown_connections 2024-05-10T11:14:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.483+0000 7f181493fb80 1 -- 172.21.15.27:0/882153648 wait complete. 2024-05-10T11:14:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.483+0000 7f181493fb80 1 -- start start 2024-05-10T11:14:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.483+0000 7f181493fb80 1 Processor -- start 2024-05-10T11:14:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.484+0000 7f181493fb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2d7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.484+0000 7f181493fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2d8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.484+0000 7f181493fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2d88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.484+0000 7f181493fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f18041bf8d0 con 0x55b0ec2d04e0 2024-05-10T11:14:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.484+0000 7f181493fb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f18041bf200 con 0x55b0ec2ca8f0 2024-05-10T11:14:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.484+0000 7f181493fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f18041bc400 con 0x55b0ec2cfbc0 2024-05-10T11:14:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.484+0000 7f181493fb80 10 client.0 _ll_register_callbacks cb 0x55b0ec1c3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:14:58.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.484+0000 7f1803e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2d88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.484+0000 7f1803e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2d88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46746/0 (socket says 172.21.15.27:46746) 2024-05-10T11:14:58.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.484+0000 7f1803e00700 1 -- 172.21.15.27:0/3531617009 learned_addr learned my addr 172.21.15.27:0/3531617009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:14:58.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.485+0000 7f1803400700 1 --2- 172.21.15.27:0/3531617009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2d7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.485+0000 7f1803e00700 1 -- 172.21.15.27:0/3531617009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2cfbc0 msgr2=0x55b0ec2d8390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:14:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.485+0000 7f1803e00700 1 --2- 172.21.15.27:0/3531617009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2d8390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.485+0000 7f1803e00700 1 -- 172.21.15.27:0/3531617009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2ca8f0 msgr2=0x55b0ec2d7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:14:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.485+0000 7f1803e00700 1 --2- 172.21.15.27:0/3531617009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2d7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.485+0000 7f1803e00700 1 -- 172.21.15.27:0/3531617009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0ec34d680 con 0x55b0ec2d04e0 2024-05-10T11:14:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.486+0000 7f1803e00700 1 --2- 172.21.15.27:0/3531617009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2d88d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f17fc003c10 tx=0x7f17fc00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c3a1c2925508b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:14:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.486+0000 7f17ed600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:14:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.486+0000 7f17ed600700 1 -- 172.21.15.27:0/3531617009 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc00bde0 con 0x55b0ec2d04e0 2024-05-10T11:14:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.486+0000 7f181493fb80 1 -- 172.21.15.27:0/3531617009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17fc02a070 con 0x55b0ec2d04e0 2024-05-10T11:14:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.486+0000 7f17ed600700 1 -- 172.21.15.27:0/3531617009 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f17fc007b60 con 0x55b0ec2d04e0 2024-05-10T11:14:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.486+0000 7f17ed600700 1 -- 172.21.15.27:0/3531617009 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f17fc03e050 con 0x55b0ec2d04e0 2024-05-10T11:14:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.487+0000 7f181493fb80 10 client.15459 fetch_fsmap learned FSMap version 30 2024-05-10T11:14:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.487+0000 7f181493fb80 1 -- 172.21.15.27:0/3531617009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b0ec34e8d0 con 0x55b0ec2d04e0 2024-05-10T11:14:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.487+0000 7f17ed600700 1 -- 172.21.15.27:0/3531617009 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17fc02a070 con 0x55b0ec2d04e0 2024-05-10T11:14:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.488+0000 7f17ed600700 1 -- 172.21.15.27:0/3531617009 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f17fc011070 con 0x55b0ec2d04e0 2024-05-10T11:14:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.488+0000 7f181493fb80 10 client.15459 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:14:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.488+0000 7f181493fb80 10 client.15459 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:14:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.488+0000 7f181493fb80 20 client.15459 populate_metadata read hostname 'smithi027' 2024-05-10T11:14:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.488+0000 7f181493fb80 1 --2- 172.21.15.27:0/3531617009 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b0ec34eaa0 0x55b0ec350e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:14:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.488+0000 7f181493fb80 4 client.15459 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:14:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.488+0000 7f181493fb80 1 -- 172.21.15.27:0/3531617009 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b0ec22a400 con 0x55b0ec34eaa0 2024-05-10T11:14:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.489+0000 7f1803400700 1 --2- 172.21.15.27:0/3531617009 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b0ec34eaa0 0x55b0ec350e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:14:58.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.489+0000 7f1803400700 1 --2- 172.21.15.27:0/3531617009 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b0ec34eaa0 0x55b0ec350e90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2b4ec6bfd47a6339 server_cookie=40a5da1d0e8f2c4b in_seq=0 out_seq=0 2024-05-10T11:14:58.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.489+0000 7f17ed600700 10 client.15459 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:14:58.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.489+0000 7f17ed600700 1 -- 172.21.15.27:0/3531617009 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b0ec22a400 con 0x55b0ec34eaa0 2024-05-10T11:14:58.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.489+0000 7f17ed600700 10 client.15459 handle_command_reply: tid=0 2024-05-10T11:14:58.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.490+0000 7f181493fb80 1 client.15459 shutdown 2024-05-10T11:14:58.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.490+0000 7f181493fb80 1 -- 172.21.15.27:0/3531617009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2d04e0 msgr2=0x55b0ec2d88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:58.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.490+0000 7f181493fb80 1 --2- 172.21.15.27:0/3531617009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2d88d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f17fc003c10 tx=0x7f17fc00b8c0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.490+0000 7f181493fb80 1 -- 172.21.15.27:0/3531617009 shutdown_connections 2024-05-10T11:14:58.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.490+0000 7f181493fb80 1 --2- 172.21.15.27:0/3531617009 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b0ec34eaa0 0x55b0ec350e90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.490+0000 7f181493fb80 1 --2- 172.21.15.27:0/3531617009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0ec2d04e0 0x55b0ec2d88d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.490+0000 7f181493fb80 1 --2- 172.21.15.27:0/3531617009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0ec2ca8f0 0x55b0ec2d7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.490+0000 7f17ed600700 0 client.15459 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:14:58.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.490+0000 7f181493fb80 1 --2- 172.21.15.27:0/3531617009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0ec2cfbc0 0x55b0ec2d8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.490+0000 7f181493fb80 1 -- 172.21.15.27:0/3531617009 >> 172.21.15.27:0/3531617009 conn(0x55b0ec2426d0 msgr2=0x55b0ec2ceef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.490+0000 7f181493fb80 1 -- 172.21.15.27:0/3531617009 shutdown_connections 2024-05-10T11:14:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.491+0000 7f181493fb80 1 -- 172.21.15.27:0/3531617009 wait complete. 2024-05-10T11:14:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.491+0000 7f181493fb80 20 client.15459 trim_cache size 0 max 16384 2024-05-10T11:14:58.492 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:14:58.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.492+0000 7f180a000700 1 -- 172.21.15.27:0/1624465773 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f17e80b2af0 msgr2=0x7f17e80b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:58.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.492+0000 7f180a000700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f17e80b2af0 0x7f17e80b4fb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55b0ec2d08e0 tx=0x7f17f4009260 comp rx=0 tx=0).stop 2024-05-10T11:14:58.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.492+0000 7f180a000700 1 -- 172.21.15.27:0/1624465773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18041060a0 msgr2=0x7f1804075110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:14:58.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.492+0000 7f180a000700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18041060a0 0x7f1804075110 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f17f000e9f0 tx=0x7f17f000ed00 comp rx=0 tx=0).stop 2024-05-10T11:14:58.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.492+0000 7f180a000700 1 -- 172.21.15.27:0/1624465773 shutdown_connections 2024-05-10T11:14:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.492+0000 7f180a000700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f17e80b2af0 0x7f17e80b4fb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.492+0000 7f180a000700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18041056e0 0x7f1804074bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.492+0000 7f180a000700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f180410b6e0 0x7f180406deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:14:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.492+0000 7f180a000700 1 --2- 172.21.15.27:0/1624465773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18041060a0 0x7f1804075110 secure :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f17f000e9f0 tx=0x7f17f000ed00 comp rx=0 tx=0).stop 2024-05-10T11:14:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.492+0000 7f180a000700 1 -- 172.21.15.27:0/1624465773 >> 172.21.15.27:0/1624465773 conn(0x7f18041011d0 msgr2=0x7f1804101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:14:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.492+0000 7f180a000700 1 -- 172.21.15.27:0/1624465773 shutdown_connections 2024-05-10T11:14:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:14:58.493+0000 7f180a000700 1 -- 172.21.15.27:0/1624465773 wait complete. 2024-05-10T11:14:58.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:14:58 smithi076 ceph-mon[34152]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-10T11:14:58.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:14:58 smithi189 ceph-mon[34842]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-10T11:14:59.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:14:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:00.353 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:00 smithi027 ceph-mon[30054]: pgmap v56: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 12 op/s 2024-05-10T11:15:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:00 smithi076 ceph-mon[34152]: pgmap v56: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 12 op/s 2024-05-10T11:15:00.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:00 smithi189 ceph-mon[34842]: pgmap v56: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 12 op/s 2024-05-10T11:15:02.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.146+0000 7f3f6aa00700 1 -- 172.21.15.27:0/51067667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f64104590 msgr2=0x7f3f64104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:02.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.146+0000 7f3f6aa00700 1 --2- 172.21.15.27:0/51067667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f64104590 0x7f3f64104970 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3f58009990 tx=0x7f3f58009ca0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.147+0000 7f3f6aa00700 1 -- 172.21.15.27:0/51067667 shutdown_connections 2024-05-10T11:15:02.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.147+0000 7f3f6aa00700 1 --2- 172.21.15.27:0/51067667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f64104590 0x7f3f64104970 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.147+0000 7f3f6aa00700 1 --2- 172.21.15.27:0/51067667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f6410d9c0 0x7f3f6410fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.147+0000 7f3f6aa00700 1 --2- 172.21.15.27:0/51067667 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f64104eb0 0x7f3f6410d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.147+0000 7f3f6aa00700 1 -- 172.21.15.27:0/51067667 >> 172.21.15.27:0/51067667 conn(0x7f3f640fddc0 msgr2=0x7f3f641001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:02.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.147+0000 7f3f6aa00700 1 -- 172.21.15.27:0/51067667 shutdown_connections 2024-05-10T11:15:02.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.147+0000 7f3f6aa00700 1 -- 172.21.15.27:0/51067667 wait complete. 2024-05-10T11:15:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.148+0000 7f3f6aa00700 1 Processor -- start 2024-05-10T11:15:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.149+0000 7f3f6aa00700 1 -- start start 2024-05-10T11:15:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.149+0000 7f3f6aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f64104590 0x7f3f6410c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.149+0000 7f3f6aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f64104eb0 0x7f3f6410cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.149+0000 7f3f6aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f6410d9c0 0x7f3f641a49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.149+0000 7f3f6aa00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f64113600 con 0x7f3f6410d9c0 2024-05-10T11:15:02.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.149+0000 7f3f6aa00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3f64113480 con 0x7f3f64104eb0 2024-05-10T11:15:02.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.149+0000 7f3f6aa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f64113780 con 0x7f3f64104590 2024-05-10T11:15:02.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.150+0000 7f3f68c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f64104eb0 0x7f3f6410cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.150+0000 7f3f68c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f64104eb0 0x7f3f6410cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54312/0 (socket says 172.21.15.27:54312) 2024-05-10T11:15:02.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.150+0000 7f3f68c00700 1 -- 172.21.15.27:0/856436140 learned_addr learned my addr 172.21.15.27:0/856436140 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.150+0000 7f3f6a000700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f6410d9c0 0x7f3f641a49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.150+0000 7f3f69600700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f64104590 0x7f3f6410c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.150+0000 7f3f68c00700 1 -- 172.21.15.27:0/856436140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f64104590 msgr2=0x7f3f6410c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.150+0000 7f3f68c00700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f64104590 0x7f3f6410c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.150+0000 7f3f68c00700 1 -- 172.21.15.27:0/856436140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f6410d9c0 msgr2=0x7f3f641a49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:02.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.150+0000 7f3f68c00700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f6410d9c0 0x7f3f641a49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.150+0000 7f3f68c00700 1 -- 172.21.15.27:0/856436140 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f641a5110 con 0x7f3f64104eb0 2024-05-10T11:15:02.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.151+0000 7f3f68c00700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f64104eb0 0x7f3f6410cc40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3f5000d720 tx=0x7f3f5000da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:02.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.151+0000 7f3f62000700 1 -- 172.21.15.27:0/856436140 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f50016070 con 0x7f3f64104eb0 2024-05-10T11:15:02.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.152+0000 7f3f62000700 1 -- 172.21.15.27:0/856436140 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f50004830 con 0x7f3f64104eb0 2024-05-10T11:15:02.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.152+0000 7f3f6aa00700 1 -- 172.21.15.27:0/856436140 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f641a5400 con 0x7f3f64104eb0 2024-05-10T11:15:02.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.152+0000 7f3f62000700 1 -- 172.21.15.27:0/856436140 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f50005520 con 0x7f3f64104eb0 2024-05-10T11:15:02.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.152+0000 7f3f6aa00700 1 -- 172.21.15.27:0/856436140 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f6406cec0 con 0x7f3f64104eb0 2024-05-10T11:15:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.155+0000 7f3f62000700 1 -- 172.21.15.27:0/856436140 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3f50004220 con 0x7f3f64104eb0 2024-05-10T11:15:02.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.157+0000 7f3f62000700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3f4c0b29f0 0x7f3f4c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.157+0000 7f3f62000700 1 -- 172.21.15.27:0/856436140 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f3f500e2770 con 0x7f3f64104eb0 2024-05-10T11:15:02.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.158+0000 7f3f69600700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3f4c0b29f0 0x7f3f4c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.159+0000 7f3f69600700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3f4c0b29f0 0x7f3f4c0b4eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3f58006010 tx=0x7f3f58005c20 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:02.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.159+0000 7f3f76a70b80 1 Processor -- start 2024-05-10T11:15:02.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.159+0000 7f3f76a70b80 1 -- start start 2024-05-10T11:15:02.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.160+0000 7f3f76a70b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4bda0 0x564264f4c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.160+0000 7f3f76a70b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f4c6c0 0x564264f46430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.160+0000 7f3f76a70b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f46a50 0x564264f46ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.160+0000 7f3f76a70b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564264e98a20 con 0x564264f4bda0 2024-05-10T11:15:02.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.160+0000 7f3f76a70b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564264e36760 con 0x564264f4c6c0 2024-05-10T11:15:02.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.160+0000 7f3f76a70b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564264e949e0 con 0x564264f46a50 2024-05-10T11:15:02.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.161+0000 7f3f68c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f4c6c0 0x564264f46430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.161+0000 7f3f68c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f4c6c0 0x564264f46430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54324/0 (socket says 172.21.15.27:54324) 2024-05-10T11:15:02.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.161+0000 7f3f68c00700 1 -- 172.21.15.27:0/3343302365 learned_addr learned my addr 172.21.15.27:0/3343302365 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:02.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.161+0000 7f3f69600700 1 --2- 172.21.15.27:0/3343302365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4bda0 0x564264f4c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.161+0000 7f3f68c00700 1 -- 172.21.15.27:0/3343302365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f46a50 msgr2=0x564264f46ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:15:02.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.161+0000 7f3f68c00700 1 --2- 172.21.15.27:0/3343302365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f46a50 0x564264f46ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.161+0000 7f3f68c00700 1 -- 172.21.15.27:0/3343302365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4bda0 msgr2=0x564264f4c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:02.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.161+0000 7f3f68c00700 1 --2- 172.21.15.27:0/3343302365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4bda0 0x564264f4c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.161+0000 7f3f68c00700 1 -- 172.21.15.27:0/3343302365 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564264de58d0 con 0x564264f4c6c0 2024-05-10T11:15:02.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.161+0000 7f3f69600700 1 --2- 172.21.15.27:0/3343302365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4bda0 0x564264f4c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:15:02.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.161+0000 7f3f68c00700 1 --2- 172.21.15.27:0/3343302365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f4c6c0 0x564264f46430 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3f50005cb0 tx=0x7f3f500a8ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5aea8faa4f532aa4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:02.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.162+0000 7f3f56a00700 1 -- 172.21.15.27:0/3343302365 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f500b2480 con 0x564264f4c6c0 2024-05-10T11:15:02.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.162+0000 7f3f56a00700 1 -- 172.21.15.27:0/3343302365 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f50004cc0 con 0x564264f4c6c0 2024-05-10T11:15:02.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.162+0000 7f3f56a00700 1 -- 172.21.15.27:0/3343302365 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f500a9880 con 0x564264f4c6c0 2024-05-10T11:15:02.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.163+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3343302365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f4c6c0 msgr2=0x564264f46430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:02.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.163+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/3343302365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f4c6c0 0x564264f46430 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3f50005cb0 tx=0x7f3f500a8ec0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.163+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3343302365 shutdown_connections 2024-05-10T11:15:02.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.163+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/3343302365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f46a50 0x564264f46ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.163+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/3343302365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4bda0 0x564264f4c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.164+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/3343302365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f4c6c0 0x564264f46430 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.164+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3343302365 >> 172.21.15.27:0/3343302365 conn(0x564264ebe6d0 msgr2=0x564264f40ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:02.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.164+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3343302365 shutdown_connections 2024-05-10T11:15:02.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.164+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3343302365 wait complete. 2024-05-10T11:15:02.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.165+0000 7f3f76a70b80 1 -- start start 2024-05-10T11:15:02.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.166+0000 7f3f76a70b80 1 Processor -- start 2024-05-10T11:15:02.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.167+0000 7f3f76a70b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f523b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.167+0000 7f3f76a70b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f4f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.167+0000 7f3f76a70b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f4fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f76a70b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564264f53150 con 0x564264f4c6c0 2024-05-10T11:15:02.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f76a70b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564264dfeb20 con 0x564264f46a50 2024-05-10T11:15:02.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f76a70b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564264f532d0 con 0x564264f4bda0 2024-05-10T11:15:02.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f69600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f69600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54336/0 (socket says 172.21.15.27:54336) 2024-05-10T11:15:02.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f69600700 1 -- 172.21.15.27:0/1313307435 learned_addr learned my addr 172.21.15.27:0/1313307435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:02.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f76a70b80 10 client.0 _ll_register_callbacks cb 0x564264e3fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:02.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f6a000700 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f4fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f68c00700 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f4f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f69600700 1 -- 172.21.15.27:0/1313307435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 msgr2=0x564264f4f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:02.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f69600700 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f4f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f69600700 1 -- 172.21.15.27:0/1313307435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 msgr2=0x564264f4fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:02.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f69600700 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f4fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f69600700 1 -- 172.21.15.27:0/1313307435 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564264e5e420 con 0x564264f46a50 2024-05-10T11:15:02.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.168+0000 7f3f6a000700 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f4fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:15:02.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.169+0000 7f3f69600700 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f523b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3f58000c00 tx=0x7f3f58009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee47ebc1ae71f0fc server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.169+0000 7f3f4b400700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.169+0000 7f3f4b400700 1 -- 172.21.15.27:0/1313307435 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f58033060 con 0x564264f46a50 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.169+0000 7f3f4b400700 1 -- 172.21.15.27:0/1313307435 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f3f5801e030 con 0x564264f46a50 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.169+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1313307435 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f50004cc0 con 0x564264f46a50 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.170+0000 7f3f4b400700 1 -- 172.21.15.27:0/1313307435 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3f5803a050 con 0x564264f46a50 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.170+0000 7f3f76a70b80 10 client.25619 fetch_fsmap learned FSMap version 30 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.170+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1313307435 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564264f5d930 con 0x564264f46a50 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.170+0000 7f3f4b400700 1 -- 172.21.15.27:0/1313307435 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f50004cc0 con 0x564264f46a50 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.170+0000 7f3f4b400700 1 -- 172.21.15.27:0/1313307435 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3f5800faa0 con 0x564264f46a50 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.171+0000 7f3f76a70b80 10 client.25619 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.171+0000 7f3f76a70b80 10 client.25619 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.171+0000 7f3f76a70b80 20 client.25619 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:02.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.171+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x564264f5ff70 0x564264f62360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.171+0000 7f3f76a70b80 4 client.25619 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:15:02.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.171+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1313307435 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564264f13140 con 0x564264f5ff70 2024-05-10T11:15:02.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.171+0000 7f3f68c00700 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x564264f5ff70 0x564264f62360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.172+0000 7f3f68c00700 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x564264f5ff70 0x564264f62360 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d25d9112b04bcf1e server_cookie=404851309674920e in_seq=0 out_seq=0 2024-05-10T11:15:02.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.172+0000 7f3f4b400700 10 client.25619 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:15:02.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.173+0000 7f3f4b400700 1 -- 172.21.15.27:0/1313307435 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564264f13140 con 0x564264f5ff70 2024-05-10T11:15:02.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.173+0000 7f3f4b400700 10 client.25619 handle_command_reply: tid=0 2024-05-10T11:15:02.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.173+0000 7f3f76a70b80 1 client.25619 shutdown 2024-05-10T11:15:02.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.173+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1313307435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 msgr2=0x564264f523b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:02.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.173+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f523b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3f58000c00 tx=0x7f3f58009310 comp rx=0 tx=0).stop 2024-05-10T11:15:02.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.173+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1313307435 shutdown_connections 2024-05-10T11:15:02.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.173+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x564264f5ff70 0x564264f62360 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.173+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f4fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.173+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f523b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.173+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/1313307435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f4f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.173+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1313307435 >> 172.21.15.27:0/1313307435 conn(0x564264ebe6d0 msgr2=0x564264f3b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:02.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.174+0000 7f3f4b400700 0 client.25619 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:15:02.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.174+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1313307435 shutdown_connections 2024-05-10T11:15:02.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.174+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1313307435 wait complete. 2024-05-10T11:15:02.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.174+0000 7f3f76a70b80 20 client.25619 trim_cache size 0 max 16384 2024-05-10T11:15:02.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.190+0000 7f3f76a70b80 1 Processor -- start 2024-05-10T11:15:02.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.191+0000 7f3f76a70b80 1 -- start start 2024-05-10T11:15:02.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.191+0000 7f3f76a70b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f46430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.191+0000 7f3f76a70b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f523b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.191+0000 7f3f76a70b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f5c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.191+0000 7f3f76a70b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f641be2e0 con 0x564264f4c6c0 2024-05-10T11:15:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.191+0000 7f3f76a70b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3f641bfe20 con 0x564264f46a50 2024-05-10T11:15:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.191+0000 7f3f76a70b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f641be9b0 con 0x564264f4bda0 2024-05-10T11:15:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.191+0000 7f3f68c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.191+0000 7f3f68c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:34956/0 (socket says 172.21.15.27:34956) 2024-05-10T11:15:02.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.191+0000 7f3f68c00700 1 -- 172.21.15.27:0/1200595131 learned_addr learned my addr 172.21.15.27:0/1200595131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:02.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.192+0000 7f3f69600700 1 --2- 172.21.15.27:0/1200595131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f46430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.192+0000 7f3f68c00700 1 -- 172.21.15.27:0/1200595131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 msgr2=0x564264f46430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:02.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.192+0000 7f3f68c00700 1 --2- 172.21.15.27:0/1200595131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f46430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.192+0000 7f3f68c00700 1 -- 172.21.15.27:0/1200595131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 msgr2=0x564264f5c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:02.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.192+0000 7f3f68c00700 1 --2- 172.21.15.27:0/1200595131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f5c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.192+0000 7f3f68c00700 1 -- 172.21.15.27:0/1200595131 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564264f4b910 con 0x564264f4bda0 2024-05-10T11:15:02.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.192+0000 7f3f69600700 1 --2- 172.21.15.27:0/1200595131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f46430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:15:02.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.192+0000 7f3f68c00700 1 --2- 172.21.15.27:0/1200595131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f523b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3f500a95b0 tx=0x7f3f50004f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed441904ee808ff5 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:02.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.192+0000 7f3f56000700 1 -- 172.21.15.27:0/1200595131 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5008f070 con 0x564264f4bda0 2024-05-10T11:15:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.193+0000 7f3f56000700 1 -- 172.21.15.27:0/1200595131 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f500049e0 con 0x564264f4bda0 2024-05-10T11:15:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.193+0000 7f3f56000700 1 -- 172.21.15.27:0/1200595131 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f500a9880 con 0x564264f4bda0 2024-05-10T11:15:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.193+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1200595131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 msgr2=0x564264f523b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.193+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/1200595131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f523b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3f500a95b0 tx=0x7f3f50004f90 comp rx=0 tx=0).stop 2024-05-10T11:15:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.193+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1200595131 shutdown_connections 2024-05-10T11:15:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.193+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/1200595131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f5c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.193+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/1200595131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f46430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.193+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/1200595131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f523b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.193+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1200595131 >> 172.21.15.27:0/1200595131 conn(0x564264ebe6d0 msgr2=0x564264f4caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.193+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1200595131 shutdown_connections 2024-05-10T11:15:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.193+0000 7f3f76a70b80 1 -- 172.21.15.27:0/1200595131 wait complete. 2024-05-10T11:15:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.194+0000 7f3f76a70b80 1 -- start start 2024-05-10T11:15:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.194+0000 7f3f76a70b80 1 Processor -- start 2024-05-10T11:15:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f76a70b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f58560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f76a70b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f58aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f76a70b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f66bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.200 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:15:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f76a70b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f641bf750 con 0x564264f4c6c0 2024-05-10T11:15:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f76a70b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3f641bf080 con 0x564264f46a50 2024-05-10T11:15:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f76a70b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f64073cc0 con 0x564264f4bda0 2024-05-10T11:15:02.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f76a70b80 10 client.0 _ll_register_callbacks cb 0x564264e3fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:02.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f6a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f66bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f69600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f58560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f6a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f66bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46806/0 (socket says 172.21.15.27:46806) 2024-05-10T11:15:02.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f69600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f58560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54358/0 (socket says 172.21.15.27:54358) 2024-05-10T11:15:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f6a000700 1 -- 172.21.15.27:0/3260368001 learned_addr learned my addr 172.21.15.27:0/3260368001 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.195+0000 7f3f68c00700 1 --2- 172.21.15.27:0/3260368001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f58aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.196+0000 7f3f69600700 1 -- 172.21.15.27:0/3260368001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 msgr2=0x564264f58aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.196+0000 7f3f69600700 1 --2- 172.21.15.27:0/3260368001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f58aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.196+0000 7f3f69600700 1 -- 172.21.15.27:0/3260368001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 msgr2=0x564264f66bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:02.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.196+0000 7f3f69600700 1 --2- 172.21.15.27:0/3260368001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f66bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.196+0000 7f3f69600700 1 -- 172.21.15.27:0/3260368001 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564264f67170 con 0x564264f46a50 2024-05-10T11:15:02.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.196+0000 7f3f69600700 1 --2- 172.21.15.27:0/3260368001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f58560 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800fe70 tx=0x7f3f5800bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5ce27ac6be774a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:02.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.196+0000 7f3f4b400700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:02.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.196+0000 7f3f4b400700 1 -- 172.21.15.27:0/3260368001 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f58033030 con 0x564264f46a50 2024-05-10T11:15:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.196+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3260368001 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f500049e0 con 0x564264f46a50 2024-05-10T11:15:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.196+0000 7f3f4b400700 1 -- 172.21.15.27:0/3260368001 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f3f5802b040 con 0x564264f46a50 2024-05-10T11:15:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.197+0000 7f3f4b400700 1 -- 172.21.15.27:0/3260368001 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3f58045050 con 0x564264f46a50 2024-05-10T11:15:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.197+0000 7f3f76a70b80 10 client.25625 fetch_fsmap learned FSMap version 30 2024-05-10T11:15:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.197+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3260368001 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564264ffd030 con 0x564264f46a50 2024-05-10T11:15:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.197+0000 7f3f4b400700 1 -- 172.21.15.27:0/3260368001 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f500049e0 con 0x564264f46a50 2024-05-10T11:15:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.198+0000 7f3f4b400700 1 -- 172.21.15.27:0/3260368001 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3f5800f740 con 0x564264f46a50 2024-05-10T11:15:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.198+0000 7f3f76a70b80 10 client.25625 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:15:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.198+0000 7f3f76a70b80 10 client.25625 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:15:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.198+0000 7f3f76a70b80 20 client.25625 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.198+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/3260368001 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x564264ffd5f0 0x564264fffa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.198+0000 7f3f76a70b80 4 client.25625 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.198+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3260368001 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564264ea6400 con 0x564264ffd5f0 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.198+0000 7f3f68c00700 1 --2- 172.21.15.27:0/3260368001 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x564264ffd5f0 0x564264fffa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f68c00700 1 --2- 172.21.15.27:0/3260368001 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x564264ffd5f0 0x564264fffa70 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=37df08a947e7fa1b server_cookie=edae38185b148685 in_seq=0 out_seq=0 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f4b400700 10 client.25625 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f4b400700 1 -- 172.21.15.27:0/3260368001 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564264ea6400 con 0x564264ffd5f0 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f4b400700 10 client.25625 handle_command_reply: tid=0 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f76a70b80 1 client.25625 shutdown 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3260368001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 msgr2=0x564264f58560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/3260368001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f58560 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800fe70 tx=0x7f3f5800bf20 comp rx=0 tx=0).stop 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3260368001 shutdown_connections 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/3260368001 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x564264ffd5f0 0x564264fffa70 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f4b400700 0 client.25625 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/3260368001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564264f4c6c0 0x564264f66bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/3260368001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564264f46a50 0x564264f58560 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.199+0000 7f3f76a70b80 1 --2- 172.21.15.27:0/3260368001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564264f4bda0 0x564264f58aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.200+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3260368001 >> 172.21.15.27:0/3260368001 conn(0x564264ebe6d0 msgr2=0x564264f4a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.200+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3260368001 shutdown_connections 2024-05-10T11:15:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.200+0000 7f3f76a70b80 1 -- 172.21.15.27:0/3260368001 wait complete. 2024-05-10T11:15:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.200+0000 7f3f76a70b80 20 client.25625 trim_cache size 0 max 16384 2024-05-10T11:15:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.202+0000 7f3f6aa00700 1 -- 172.21.15.27:0/856436140 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3f4c0b29f0 msgr2=0x7f3f4c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.202+0000 7f3f6aa00700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3f4c0b29f0 0x7f3f4c0b4eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3f58006010 tx=0x7f3f58005c20 comp rx=0 tx=0).stop 2024-05-10T11:15:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.203+0000 7f3f6aa00700 1 -- 172.21.15.27:0/856436140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f64104eb0 msgr2=0x7f3f6410cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.203+0000 7f3f6aa00700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f64104eb0 0x7f3f6410cc40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3f5000d720 tx=0x7f3f5000da30 comp rx=0 tx=0).stop 2024-05-10T11:15:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.203+0000 7f3f6aa00700 1 -- 172.21.15.27:0/856436140 shutdown_connections 2024-05-10T11:15:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.203+0000 7f3f6aa00700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3f4c0b29f0 0x7f3f4c0b4eb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.203+0000 7f3f6aa00700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f64104590 0x7f3f6410c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.203+0000 7f3f6aa00700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f6410d9c0 0x7f3f641a49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.203+0000 7f3f6aa00700 1 --2- 172.21.15.27:0/856436140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f64104eb0 0x7f3f6410cc40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:02.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.203+0000 7f3f6aa00700 1 -- 172.21.15.27:0/856436140 >> 172.21.15.27:0/856436140 conn(0x7f3f640fddc0 msgr2=0x7f3f640fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:02.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.203+0000 7f3f6aa00700 1 -- 172.21.15.27:0/856436140 shutdown_connections 2024-05-10T11:15:02.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:02.203+0000 7f3f6aa00700 1 -- 172.21.15.27:0/856436140 wait complete. 2024-05-10T11:15:02.617 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:02 smithi027 ceph-mon[30054]: pgmap v57: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-10T11:15:02.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:02 smithi076 ceph-mon[34152]: pgmap v57: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-10T11:15:02.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:02 smithi189 ceph-mon[34842]: pgmap v57: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-10T11:15:03.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:15:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:04.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:04 smithi027 ceph-mon[30054]: pgmap v58: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:15:04.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:04 smithi076 ceph-mon[34152]: pgmap v58: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:15:04.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:04 smithi189 ceph-mon[34842]: pgmap v58: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-10T11:15:05.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:05 smithi027 ceph-mon[30054]: pgmap v59: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-05-10T11:15:05.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:05 smithi076 ceph-mon[34152]: pgmap v59: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-05-10T11:15:05.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:05 smithi189 ceph-mon[34842]: pgmap v59: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-05-10T11:15:05.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.936+0000 7f839b200700 1 -- 172.21.15.27:0/2769496633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c103d90 msgr2=0x7f839c1102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:05.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.936+0000 7f839b200700 1 --2- 172.21.15.27:0/2769496633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c103d90 0x7f839c1102a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f839400b0a0 tx=0x7f839400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.936+0000 7f839b200700 1 -- 172.21.15.27:0/2769496633 shutdown_connections 2024-05-10T11:15:05.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.936+0000 7f839b200700 1 --2- 172.21.15.27:0/2769496633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c103d90 0x7f839c1102a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.936+0000 7f839b200700 1 --2- 172.21.15.27:0/2769496633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f839c109460 0x7f839c109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.936+0000 7f839b200700 1 --2- 172.21.15.27:0/2769496633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f839c1033d0 0x7f839c103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.936+0000 7f839b200700 1 -- 172.21.15.27:0/2769496633 >> 172.21.15.27:0/2769496633 conn(0x7f839c0fef20 msgr2=0x7f839c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.936+0000 7f839b200700 1 -- 172.21.15.27:0/2769496633 shutdown_connections 2024-05-10T11:15:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.936+0000 7f839b200700 1 -- 172.21.15.27:0/2769496633 wait complete. 2024-05-10T11:15:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.938+0000 7f839b200700 1 Processor -- start 2024-05-10T11:15:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.938+0000 7f839b200700 1 -- start start 2024-05-10T11:15:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.939+0000 7f839b200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f839c1033d0 0x7f839c10f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.939+0000 7f839b200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f839c103d90 0x7f839c10fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.939+0000 7f839b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c109460 0x7f839c1a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.939+0000 7f839b200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f839c1135a0 con 0x7f839c1033d0 2024-05-10T11:15:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.939+0000 7f839b200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f839c113420 con 0x7f839c103d90 2024-05-10T11:15:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.939+0000 7f839b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f839c113720 con 0x7f839c109460 2024-05-10T11:15:05.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.939+0000 7f8399e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f839c1033d0 0x7f839c10f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:05.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.939+0000 7f8399e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f839c1033d0 0x7f839c10f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34158/0 (socket says 172.21.15.27:34158) 2024-05-10T11:15:05.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.939+0000 7f8399e00700 1 -- 172.21.15.27:0/252437094 learned_addr learned my addr 172.21.15.27:0/252437094 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:05.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.940+0000 7f8399400700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f839c103d90 0x7f839c10fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:05.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.940+0000 7f839a800700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c109460 0x7f839c1a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:05.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.940+0000 7f8399e00700 1 -- 172.21.15.27:0/252437094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c109460 msgr2=0x7f839c1a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:05.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.940+0000 7f8399e00700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c109460 0x7f839c1a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.940+0000 7f8399e00700 1 -- 172.21.15.27:0/252437094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f839c103d90 msgr2=0x7f839c10fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:05.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.940+0000 7f8399e00700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f839c103d90 0x7f839c10fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.940+0000 7f8399e00700 1 -- 172.21.15.27:0/252437094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839c1a4ff0 con 0x7f839c1033d0 2024-05-10T11:15:05.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.941+0000 7f8399e00700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f839c1033d0 0x7f839c10f590 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f838c00e9a0 tx=0x7f838c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:05.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.941+0000 7f839a800700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c109460 0x7f839c1a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:15:05.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.941+0000 7f8392a00700 1 -- 172.21.15.27:0/252437094 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c00cbd0 con 0x7f839c1033d0 2024-05-10T11:15:05.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.941+0000 7f8392a00700 1 -- 172.21.15.27:0/252437094 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f838c00cd70 con 0x7f839c1033d0 2024-05-10T11:15:05.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.941+0000 7f8392a00700 1 -- 172.21.15.27:0/252437094 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c018680 con 0x7f839c1033d0 2024-05-10T11:15:05.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.943+0000 7f839b200700 1 -- 172.21.15.27:0/252437094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f839c1a52e0 con 0x7f839c1033d0 2024-05-10T11:15:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.943+0000 7f839b200700 1 -- 172.21.15.27:0/252437094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f839c1acb60 con 0x7f839c1033d0 2024-05-10T11:15:05.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.949+0000 7f8392a00700 1 -- 172.21.15.27:0/252437094 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f838c010430 con 0x7f839c1033d0 2024-05-10T11:15:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.949+0000 7f83ab68bb80 1 Processor -- start 2024-05-10T11:15:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.950+0000 7f83ab68bb80 1 -- start start 2024-05-10T11:15:05.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.950+0000 7f83ab68bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733caf180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.950+0000 7f83ab68bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733caf6c0 0x561733ca9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.950+0000 7f83ab68bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733ca9a50 0x561733ca9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.950+0000 7f83ab68bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561733bfba20 con 0x561733caeda0 2024-05-10T11:15:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.950+0000 7f83ab68bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561733b99760 con 0x561733ca9a50 2024-05-10T11:15:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.950+0000 7f83ab68bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561733bf79e0 con 0x561733caf6c0 2024-05-10T11:15:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.951+0000 7f839a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733caf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.951+0000 7f839a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733caf180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34166/0 (socket says 172.21.15.27:34166) 2024-05-10T11:15:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.951+0000 7f839a800700 1 -- 172.21.15.27:0/1151260757 learned_addr learned my addr 172.21.15.27:0/1151260757 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:05.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.951+0000 7f8399e00700 1 --2- 172.21.15.27:0/1151260757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733caf6c0 0x561733ca9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:05.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.952+0000 7f8392a00700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f83800b2a70 0x7f83800b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.952+0000 7f8399e00700 1 -- 172.21.15.27:0/1151260757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733ca9a50 msgr2=0x561733ca9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:05.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.952+0000 7f8399e00700 1 --2- 172.21.15.27:0/1151260757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733ca9a50 0x561733ca9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.952+0000 7f8399e00700 1 -- 172.21.15.27:0/1151260757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 msgr2=0x561733caf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:05.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.952+0000 7f8399e00700 1 --2- 172.21.15.27:0/1151260757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733caf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.952+0000 7f8392a00700 1 -- 172.21.15.27:0/252437094 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f838c0e2b80 con 0x7f839c1033d0 2024-05-10T11:15:05.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.952+0000 7f8399e00700 1 -- 172.21.15.27:0/1151260757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561733b488d0 con 0x561733caf6c0 2024-05-10T11:15:05.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.952+0000 7f839a800700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f83800b2a70 0x7f83800b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:05.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.953+0000 7f8399e00700 1 --2- 172.21.15.27:0/1151260757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733caf6c0 0x561733ca9430 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f838c0dfea0 tx=0x7f838c010cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8d13af51da9c84c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:05.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.953+0000 7f8387400700 1 -- 172.21.15.27:0/1151260757 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c0a58f0 con 0x561733caf6c0 2024-05-10T11:15:05.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.953+0000 7f8387400700 1 -- 172.21.15.27:0/1151260757 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f838c0583a0 con 0x561733caf6c0 2024-05-10T11:15:05.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.953+0000 7f8387400700 1 -- 172.21.15.27:0/1151260757 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c0a91a0 con 0x561733caf6c0 2024-05-10T11:15:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.954+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1151260757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733caf6c0 msgr2=0x561733ca9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.954+0000 7f839a800700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f83800b2a70 0x7f83800b4f30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f839400b780 tx=0x7f839401f040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.954+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/1151260757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733caf6c0 0x561733ca9430 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f838c0dfea0 tx=0x7f838c010cb0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.954+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1151260757 shutdown_connections 2024-05-10T11:15:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.954+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/1151260757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733ca9a50 0x561733ca9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.954+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/1151260757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733caf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.954+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/1151260757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733caf6c0 0x561733ca9430 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.954+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1151260757 >> 172.21.15.27:0/1151260757 conn(0x561733c216d0 msgr2=0x561733ca3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:05.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.955+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1151260757 shutdown_connections 2024-05-10T11:15:05.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.956+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1151260757 wait complete. 2024-05-10T11:15:05.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.956+0000 7f83ab68bb80 1 -- start start 2024-05-10T11:15:05.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.957+0000 7f83ab68bb80 1 Processor -- start 2024-05-10T11:15:05.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.958+0000 7f83ab68bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 0x561733cb5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.958+0000 7f83ab68bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733cb26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.958+0000 7f83ab68bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733caf6c0 0x561733cb2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.958+0000 7f83ab68bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561733cb60f0 con 0x561733caeda0 2024-05-10T11:15:05.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.958+0000 7f83ab68bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561733b61b20 con 0x561733caf6c0 2024-05-10T11:15:05.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.958+0000 7f83ab68bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561733cb6270 con 0x561733ca9a50 2024-05-10T11:15:05.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.958+0000 7f83ab68bb80 10 client.0 _ll_register_callbacks cb 0x561733ba2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:05.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.958+0000 7f8399e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 0x561733cb5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:05.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.958+0000 7f8399e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 0x561733cb5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51706/0 (socket says 172.21.15.27:51706) 2024-05-10T11:15:05.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.958+0000 7f8399e00700 1 -- 172.21.15.27:0/4153760801 learned_addr learned my addr 172.21.15.27:0/4153760801 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:05.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.958+0000 7f8399400700 1 --2- 172.21.15.27:0/4153760801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733cb26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.959+0000 7f8399e00700 1 -- 172.21.15.27:0/4153760801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733caf6c0 msgr2=0x561733cb2c10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:15:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.959+0000 7f8399e00700 1 --2- 172.21.15.27:0/4153760801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733caf6c0 0x561733cb2c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.959+0000 7f8399e00700 1 -- 172.21.15.27:0/4153760801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 msgr2=0x561733cb26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.959+0000 7f8399e00700 1 --2- 172.21.15.27:0/4153760801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733cb26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.959+0000 7f8399e00700 1 -- 172.21.15.27:0/4153760801 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561733bc1420 con 0x561733ca9a50 2024-05-10T11:15:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.959+0000 7f8399e00700 1 --2- 172.21.15.27:0/4153760801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 0x561733cb5330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f838c009230 tx=0x7f838c0b22c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4b1f8bd2dbe5a5a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.959+0000 7f837be00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:15:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.959+0000 7f837be00700 1 -- 172.21.15.27:0/4153760801 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c0a9380 con 0x561733ca9a50 2024-05-10T11:15:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.959+0000 7f837be00700 1 -- 172.21.15.27:0/4153760801 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f838c0cc320 con 0x561733ca9a50 2024-05-10T11:15:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.960+0000 7f83ab68bb80 1 -- 172.21.15.27:0/4153760801 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f838c0583a0 con 0x561733ca9a50 2024-05-10T11:15:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.960+0000 7f837be00700 1 -- 172.21.15.27:0/4153760801 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f838c0a9800 con 0x561733ca9a50 2024-05-10T11:15:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.960+0000 7f83ab68bb80 10 client.25621 fetch_fsmap learned FSMap version 30 2024-05-10T11:15:05.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.960+0000 7f83ab68bb80 1 -- 172.21.15.27:0/4153760801 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561733cb60f0 con 0x561733ca9a50 2024-05-10T11:15:05.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.960+0000 7f837be00700 1 -- 172.21.15.27:0/4153760801 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f838c0583a0 con 0x561733ca9a50 2024-05-10T11:15:05.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.961+0000 7f837be00700 1 -- 172.21.15.27:0/4153760801 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f838c0cace0 con 0x561733ca9a50 2024-05-10T11:15:05.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.961+0000 7f83ab68bb80 10 client.25621 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:15:05.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.961+0000 7f83ab68bb80 10 client.25621 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:15:05.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.961+0000 7f83ab68bb80 20 client.25621 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:05.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.961+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/4153760801 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561733cd3c00 0x561733cd5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.961+0000 7f83ab68bb80 4 client.25621 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:15:05.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.961+0000 7f83ab68bb80 1 -- 172.21.15.27:0/4153760801 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561733c76140 con 0x561733cd3c00 2024-05-10T11:15:05.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.961+0000 7f8399400700 1 --2- 172.21.15.27:0/4153760801 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561733cd3c00 0x561733cd5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:05.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.962+0000 7f8399400700 1 --2- 172.21.15.27:0/4153760801 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561733cd3c00 0x561733cd5ff0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=19946a27d7398374 server_cookie=433e7c40e9a637b7 in_seq=0 out_seq=0 2024-05-10T11:15:05.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.962+0000 7f837be00700 10 client.25621 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:15:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.963+0000 7f837be00700 1 -- 172.21.15.27:0/4153760801 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561733c76140 con 0x561733cd3c00 2024-05-10T11:15:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.963+0000 7f837be00700 10 client.25621 handle_command_reply: tid=0 2024-05-10T11:15:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.963+0000 7f83ab68bb80 1 client.25621 shutdown 2024-05-10T11:15:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.963+0000 7f83ab68bb80 1 -- 172.21.15.27:0/4153760801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 msgr2=0x561733cb5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.963+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/4153760801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 0x561733cb5330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f838c009230 tx=0x7f838c0b22c0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.963+0000 7f83ab68bb80 1 -- 172.21.15.27:0/4153760801 shutdown_connections 2024-05-10T11:15:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.963+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/4153760801 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561733cd3c00 0x561733cd5ff0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.964+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/4153760801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 0x561733cb5330 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.964+0000 7f837be00700 0 client.25621 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:15:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.964+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/4153760801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733caf6c0 0x561733cb2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.964+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/4153760801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733cb26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.964+0000 7f83ab68bb80 1 -- 172.21.15.27:0/4153760801 >> 172.21.15.27:0/4153760801 conn(0x561733c216d0 msgr2=0x561733c9e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.964+0000 7f83ab68bb80 1 -- 172.21.15.27:0/4153760801 shutdown_connections 2024-05-10T11:15:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.964+0000 7f83ab68bb80 1 -- 172.21.15.27:0/4153760801 wait complete. 2024-05-10T11:15:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.964+0000 7f83ab68bb80 20 client.25621 trim_cache size 0 max 16384 2024-05-10T11:15:05.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.996+0000 7f83ab68bb80 1 Processor -- start 2024-05-10T11:15:05.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.996+0000 7f83ab68bb80 1 -- start start 2024-05-10T11:15:05.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.997+0000 7f83ab68bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733ca9a50 0x561733cbf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.997+0000 7f83ab68bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733caeda0 0x561733cb1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.997+0000 7f83ab68bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caf6c0 0x561733c9e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:05.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.997+0000 7f83ab68bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f839c1ad230 con 0x561733caf6c0 2024-05-10T11:15:05.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.997+0000 7f83ab68bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f839c1aed70 con 0x561733ca9a50 2024-05-10T11:15:05.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.997+0000 7f83ab68bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f839c1ad900 con 0x561733caeda0 2024-05-10T11:15:05.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.997+0000 7f8399e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733ca9a50 0x561733cbf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:05.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.997+0000 7f8399e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733ca9a50 0x561733cbf1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55034/0 (socket says 172.21.15.27:55034) 2024-05-10T11:15:05.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.997+0000 7f8399e00700 1 -- 172.21.15.27:0/2300957951 learned_addr learned my addr 172.21.15.27:0/2300957951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.998+0000 7f8399400700 1 --2- 172.21.15.27:0/2300957951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733caeda0 0x561733cb1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.998+0000 7f839a800700 1 --2- 172.21.15.27:0/2300957951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caf6c0 0x561733c9e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.998+0000 7f8399400700 1 -- 172.21.15.27:0/2300957951 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733ca9a50 msgr2=0x561733cbf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.998+0000 7f8399400700 1 --2- 172.21.15.27:0/2300957951 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733ca9a50 0x561733cbf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.998+0000 7f8399400700 1 -- 172.21.15.27:0/2300957951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caf6c0 msgr2=0x561733c9e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.998+0000 7f8399400700 1 --2- 172.21.15.27:0/2300957951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caf6c0 0x561733c9e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.999+0000 7f8399400700 1 -- 172.21.15.27:0/2300957951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561733ca56c0 con 0x561733caeda0 2024-05-10T11:15:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.999+0000 7f839a800700 1 --2- 172.21.15.27:0/2300957951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caf6c0 0x561733c9e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:15:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.999+0000 7f8399400700 1 --2- 172.21.15.27:0/2300957951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733caeda0 0x561733cb1850 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x561733cd6720 tx=0x7f8388009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb94bd7fd5fb1b32 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.999+0000 7f8386a00700 1 -- 172.21.15.27:0/2300957951 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388010920 con 0x561733caeda0 2024-05-10T11:15:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.999+0000 7f8386a00700 1 -- 172.21.15.27:0/2300957951 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f838801c400 con 0x561733caeda0 2024-05-10T11:15:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:05.999+0000 7f8386a00700 1 -- 172.21.15.27:0/2300957951 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838800f7a0 con 0x561733caeda0 2024-05-10T11:15:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.000+0000 7f83ab68bb80 1 -- 172.21.15.27:0/2300957951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733caeda0 msgr2=0x561733cb1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.000+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/2300957951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733caeda0 0x561733cb1850 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x561733cd6720 tx=0x7f8388009f70 comp rx=0 tx=0).stop 2024-05-10T11:15:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.000+0000 7f83ab68bb80 1 -- 172.21.15.27:0/2300957951 shutdown_connections 2024-05-10T11:15:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.000+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/2300957951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caf6c0 0x561733c9e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.000+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/2300957951 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733ca9a50 0x561733cbf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.000+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/2300957951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733caeda0 0x561733cb1850 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.000+0000 7f83ab68bb80 1 -- 172.21.15.27:0/2300957951 >> 172.21.15.27:0/2300957951 conn(0x561733c216d0 msgr2=0x561733cb0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:06.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.001+0000 7f83ab68bb80 1 -- 172.21.15.27:0/2300957951 shutdown_connections 2024-05-10T11:15:06.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.001+0000 7f83ab68bb80 1 -- 172.21.15.27:0/2300957951 wait complete. 2024-05-10T11:15:06.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.001+0000 7f83ab68bb80 1 -- start start 2024-05-10T11:15:06.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.001+0000 7f83ab68bb80 1 Processor -- start 2024-05-10T11:15:06.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.002+0000 7f83ab68bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 0x561733cbb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.002+0000 7f83ab68bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733cdacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.002+0000 7f83ab68bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733caf6c0 0x561733cdb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.002+0000 7f83ab68bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f839c1ae6a0 con 0x561733caeda0 2024-05-10T11:15:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.002+0000 7f83ab68bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f839c1adfd0 con 0x561733caf6c0 2024-05-10T11:15:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.002+0000 7f83ab68bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f839c073b70 con 0x561733ca9a50 2024-05-10T11:15:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.002+0000 7f83ab68bb80 10 client.0 _ll_register_callbacks cb 0x561733ba2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.003+0000 7f8399e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 0x561733cbb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.003+0000 7f8399e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 0x561733cbb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51714/0 (socket says 172.21.15.27:51714) 2024-05-10T11:15:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.003+0000 7f8399e00700 1 -- 172.21.15.27:0/1408239281 learned_addr learned my addr 172.21.15.27:0/1408239281 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.003+0000 7f8399400700 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733cdacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.003+0000 7f839a800700 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733caf6c0 0x561733cdb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.003+0000 7f8399e00700 1 -- 172.21.15.27:0/1408239281 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733caf6c0 msgr2=0x561733cdb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.003+0000 7f8399e00700 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733caf6c0 0x561733cdb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.003+0000 7f8399e00700 1 -- 172.21.15.27:0/1408239281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 msgr2=0x561733cdacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.003+0000 7f8399e00700 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733cdacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.003+0000 7f8399e00700 1 -- 172.21.15.27:0/1408239281 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561733cdb900 con 0x561733ca9a50 2024-05-10T11:15:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.004+0000 7f8399e00700 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 0x561733cbb970 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f838c0a5fd0 tx=0x7f838c00eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a3961d5f1a93399 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.004+0000 7f837be00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:15:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.005+0000 7f837be00700 1 -- 172.21.15.27:0/1408239281 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c0c3060 con 0x561733ca9a50 2024-05-10T11:15:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.005+0000 7f837be00700 1 -- 172.21.15.27:0/1408239281 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f838c0e8040 con 0x561733ca9a50 2024-05-10T11:15:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.005+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1408239281 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f838801c400 con 0x561733ca9a50 2024-05-10T11:15:06.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.005+0000 7f837be00700 1 -- 172.21.15.27:0/1408239281 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f838c08b050 con 0x561733ca9a50 2024-05-10T11:15:06.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.005+0000 7f83ab68bb80 10 client.25627 fetch_fsmap learned FSMap version 30 2024-05-10T11:15:06.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.005+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1408239281 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561733d3e190 con 0x561733ca9a50 2024-05-10T11:15:06.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.005+0000 7f837be00700 1 -- 172.21.15.27:0/1408239281 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f838801c400 con 0x561733ca9a50 2024-05-10T11:15:06.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.006+0000 7f837be00700 1 -- 172.21.15.27:0/1408239281 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f838c0cace0 con 0x561733ca9a50 2024-05-10T11:15:06.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.006+0000 7f83ab68bb80 10 client.25627 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:15:06.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.006+0000 7f83ab68bb80 10 client.25627 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:15:06.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.006+0000 7f83ab68bb80 20 client.25627 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:06.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.006+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561733cd3c00 0x561733d40660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:06.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.006+0000 7f83ab68bb80 4 client.25627 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:15:06.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.006+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1408239281 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561733c09400 con 0x561733cd3c00 2024-05-10T11:15:06.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.006+0000 7f8399400700 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561733cd3c00 0x561733d40660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:06.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.007+0000 7f8399400700 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561733cd3c00 0x561733d40660 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f8f168a8fe5a8d57 server_cookie=31ccddb2baab6520 in_seq=0 out_seq=0 2024-05-10T11:15:06.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.007+0000 7f837be00700 10 client.25627 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:15:06.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.007+0000 7f837be00700 1 -- 172.21.15.27:0/1408239281 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561733c09400 con 0x561733cd3c00 2024-05-10T11:15:06.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.007+0000 7f837be00700 10 client.25627 handle_command_reply: tid=0 2024-05-10T11:15:06.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.007+0000 7f83ab68bb80 1 client.25627 shutdown 2024-05-10T11:15:06.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.008+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1408239281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 msgr2=0x561733cbb970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:06.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.008+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 0x561733cbb970 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f838c0a5fd0 tx=0x7f838c00eef0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.008+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1408239281 shutdown_connections 2024-05-10T11:15:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.008+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x561733cd3c00 0x561733d40660 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.008+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561733ca9a50 0x561733cbb970 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.008+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561733caf6c0 0x561733cdb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.008+0000 7f83ab68bb80 1 --2- 172.21.15.27:0/1408239281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561733caeda0 0x561733cdacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.008+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1408239281 >> 172.21.15.27:0/1408239281 conn(0x561733c216d0 msgr2=0x561733cb3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.008+0000 7f837be00700 0 client.25627 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:15:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.009+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1408239281 shutdown_connections 2024-05-10T11:15:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.010+0000 7f83ab68bb80 1 -- 172.21.15.27:0/1408239281 wait complete. 2024-05-10T11:15:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.010+0000 7f83ab68bb80 20 client.25627 trim_cache size 0 max 16384 2024-05-10T11:15:06.011 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:15:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.011+0000 7f839b200700 1 -- 172.21.15.27:0/252437094 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f83800b2a70 msgr2=0x7f83800b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:06.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.011+0000 7f839b200700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f83800b2a70 0x7f83800b4f30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f839400b780 tx=0x7f839401f040 comp rx=0 tx=0).stop 2024-05-10T11:15:06.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.011+0000 7f839b200700 1 -- 172.21.15.27:0/252437094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f839c1033d0 msgr2=0x7f839c10f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:06.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.011+0000 7f839b200700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f839c1033d0 0x7f839c10f590 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f838c00e9a0 tx=0x7f838c00ecb0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.012+0000 7f839b200700 1 -- 172.21.15.27:0/252437094 shutdown_connections 2024-05-10T11:15:06.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.012+0000 7f839b200700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f83800b2a70 0x7f83800b4f30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.012+0000 7f839b200700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c109460 0x7f839c1a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.012+0000 7f839b200700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f839c1033d0 0x7f839c10f590 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.012+0000 7f839b200700 1 --2- 172.21.15.27:0/252437094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f839c103d90 0x7f839c10fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.012+0000 7f839b200700 1 -- 172.21.15.27:0/252437094 >> 172.21.15.27:0/252437094 conn(0x7f839c0fef20 msgr2=0x7f839c0ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.012+0000 7f839b200700 1 -- 172.21.15.27:0/252437094 shutdown_connections 2024-05-10T11:15:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:06.012+0000 7f839b200700 1 -- 172.21.15.27:0/252437094 wait complete. 2024-05-10T11:15:06.939 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:15:07.368 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:08.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:08 smithi027 ceph-mon[30054]: pgmap v60: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-10T11:15:08.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:08 smithi076 ceph-mon[34152]: pgmap v60: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-10T11:15:08.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:08 smithi189 ceph-mon[34842]: pgmap v60: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-10T11:15:10.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.141+0000 7f12ea600700 1 -- 172.21.15.27:0/92190836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12e406bb40 msgr2=0x7f12e4110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.141+0000 7f12ea600700 1 --2- 172.21.15.27:0/92190836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12e406bb40 0x7f12e4110330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009f90 tx=0x7f12dc00b330 comp rx=0 tx=0).stop 2024-05-10T11:15:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.141+0000 7f12ea600700 1 -- 172.21.15.27:0/92190836 shutdown_connections 2024-05-10T11:15:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.141+0000 7f12ea600700 1 --2- 172.21.15.27:0/92190836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12e406bb40 0x7f12e4110330 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.141+0000 7f12ea600700 1 --2- 172.21.15.27:0/92190836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e4109510 0x7f12e41098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.141+0000 7f12ea600700 1 --2- 172.21.15.27:0/92190836 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12e406b180 0x7f12e406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.141+0000 7f12ea600700 1 -- 172.21.15.27:0/92190836 >> 172.21.15.27:0/92190836 conn(0x7f12e4079350 msgr2=0x7f12e4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.141+0000 7f12ea600700 1 -- 172.21.15.27:0/92190836 shutdown_connections 2024-05-10T11:15:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.142+0000 7f12ea600700 1 -- 172.21.15.27:0/92190836 wait complete. 2024-05-10T11:15:10.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.143+0000 7f12ea600700 1 Processor -- start 2024-05-10T11:15:10.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.144+0000 7f12ea600700 1 -- start start 2024-05-10T11:15:10.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.144+0000 7f12ea600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e406b180 0x7f12e419bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.144+0000 7f12ea600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12e406bb40 0x7f12e419c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.144+0000 7f12ea600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12e4109510 0x7f12e41a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.144+0000 7f12ea600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f12e4113670 con 0x7f12e406b180 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.144+0000 7f12ea600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f12e41134f0 con 0x7f12e4109510 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.144+0000 7f12ea600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12e41137f0 con 0x7f12e406bb40 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.144+0000 7f12e2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12e406bb40 0x7f12e419c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.144+0000 7f12e3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e406b180 0x7f12e419bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.144+0000 7f12e2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12e406bb40 0x7f12e419c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51730/0 (socket says 172.21.15.27:51730) 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.144+0000 7f12e2a00700 1 -- 172.21.15.27:0/3925597585 learned_addr learned my addr 172.21.15.27:0/3925597585 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.144+0000 7f12e3e00700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12e4109510 0x7f12e41a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.145+0000 7f12e2a00700 1 -- 172.21.15.27:0/3925597585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12e4109510 msgr2=0x7f12e41a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.145+0000 7f12e2a00700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12e4109510 0x7f12e41a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.145+0000 7f12e2a00700 1 -- 172.21.15.27:0/3925597585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e406b180 msgr2=0x7f12e419bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.145+0000 7f12e2a00700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e406b180 0x7f12e419bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.145+0000 7f12e2a00700 1 -- 172.21.15.27:0/3925597585 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e41a0d30 con 0x7f12e406bb40 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.145+0000 7f12e2a00700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12e406bb40 0x7f12e419c260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f12d000e9a0 tx=0x7f12d000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.145+0000 7f12dbe00700 1 -- 172.21.15.27:0/3925597585 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d000ca10 con 0x7f12e406bb40 2024-05-10T11:15:10.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.146+0000 7f12dbe00700 1 -- 172.21.15.27:0/3925597585 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12d000cbb0 con 0x7f12e406bb40 2024-05-10T11:15:10.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.146+0000 7f12dbe00700 1 -- 172.21.15.27:0/3925597585 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d0018680 con 0x7f12e406bb40 2024-05-10T11:15:10.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.146+0000 7f12ea600700 1 -- 172.21.15.27:0/3925597585 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e41a1020 con 0x7f12e406bb40 2024-05-10T11:15:10.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.146+0000 7f12ea600700 1 -- 172.21.15.27:0/3925597585 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e41a88a0 con 0x7f12e406bb40 2024-05-10T11:15:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.150+0000 7f12f5064b80 1 Processor -- start 2024-05-10T11:15:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12f5064b80 1 -- start start 2024-05-10T11:15:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12f5064b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 0x55f20c4cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12dbe00700 1 -- 172.21.15.27:0/3925597585 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f12d0010430 con 0x7f12e406bb40 2024-05-10T11:15:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12f5064b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd980 0x55f20c4d1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12f5064b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4d19d0 0x55f20c4d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12f5064b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f20c41fa20 con 0x55f20c4d19d0 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12f5064b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f20c3bd760 con 0x55f20c4cd980 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12f5064b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f20c41b9e0 con 0x55f20c4cd060 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12e3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd980 0x55f20c4d1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12e2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4d19d0 0x55f20c4d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12e3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 0x55f20c4cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12e3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd980 0x55f20c4d1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55060/0 (socket says 172.21.15.27:55060) 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.151+0000 7f12e3400700 1 -- 172.21.15.27:0/1312562560 learned_addr learned my addr 172.21.15.27:0/1312562560 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.152+0000 7f12e3400700 1 -- 172.21.15.27:0/1312562560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 msgr2=0x55f20c4cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.152+0000 7f12e3400700 1 --2- 172.21.15.27:0/1312562560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 0x55f20c4cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.152+0000 7f12e3400700 1 -- 172.21.15.27:0/1312562560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4d19d0 msgr2=0x55f20c4d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.152+0000 7f12e3400700 1 --2- 172.21.15.27:0/1312562560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4d19d0 0x55f20c4d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.152+0000 7f12e3400700 1 -- 172.21.15.27:0/1312562560 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f20c36c8d0 con 0x55f20c4cd980 2024-05-10T11:15:10.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.152+0000 7f12e3e00700 1 --2- 172.21.15.27:0/1312562560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 0x55f20c4cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:10.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.152+0000 7f12e3400700 1 --2- 172.21.15.27:0/1312562560 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd980 0x55f20c4d1490 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009e70 tx=0x7f12d4011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64bee4834e1ec673 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.152+0000 7f12d8c00700 1 -- 172.21.15.27:0/1312562560 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d4004310 con 0x55f20c4cd980 2024-05-10T11:15:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.153+0000 7f12d8c00700 1 -- 172.21.15.27:0/1312562560 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12d40044b0 con 0x55f20c4cd980 2024-05-10T11:15:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.153+0000 7f12d8c00700 1 -- 172.21.15.27:0/1312562560 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d4019870 con 0x55f20c4cd980 2024-05-10T11:15:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.153+0000 7f12dbe00700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f12c80b29f0 0x7f12c80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.153+0000 7f12f5064b80 1 -- 172.21.15.27:0/1312562560 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd980 msgr2=0x55f20c4d1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.153+0000 7f12f5064b80 1 --2- 172.21.15.27:0/1312562560 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd980 0x55f20c4d1490 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009e70 tx=0x7f12d4011720 comp rx=0 tx=0).stop 2024-05-10T11:15:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.153+0000 7f12dbe00700 1 -- 172.21.15.27:0/3925597585 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f12d00e2bd0 con 0x7f12e406bb40 2024-05-10T11:15:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.154+0000 7f12f5064b80 1 -- 172.21.15.27:0/1312562560 shutdown_connections 2024-05-10T11:15:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.154+0000 7f12f5064b80 1 --2- 172.21.15.27:0/1312562560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 0x55f20c4cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.154+0000 7f12f5064b80 1 --2- 172.21.15.27:0/1312562560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4d19d0 0x55f20c4d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.154+0000 7f12f5064b80 1 --2- 172.21.15.27:0/1312562560 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd980 0x55f20c4d1490 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.154+0000 7f12f5064b80 1 -- 172.21.15.27:0/1312562560 >> 172.21.15.27:0/1312562560 conn(0x55f20c3fbb60 msgr2=0x55f20c4c77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.154+0000 7f12e3e00700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f12c80b29f0 0x7f12c80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.154+0000 7f12f5064b80 1 -- 172.21.15.27:0/1312562560 shutdown_connections 2024-05-10T11:15:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.154+0000 7f12f5064b80 1 -- 172.21.15.27:0/1312562560 wait complete. 2024-05-10T11:15:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.154+0000 7f12e3e00700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f12c80b29f0 0x7f12c80b4eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009da0 tx=0x7f12dc009700 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.154+0000 7f12f5064b80 1 -- start start 2024-05-10T11:15:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.155+0000 7f12f5064b80 1 Processor -- start 2024-05-10T11:15:10.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.155+0000 7f12f5064b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd060 0x55f20c4d8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.155+0000 7f12f5064b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd980 0x55f20c4d6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.155+0000 7f12f5064b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4d19d0 0x55f20c4d6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.155+0000 7f12f5064b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f20c4d9be0 con 0x55f20c4cd980 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.155+0000 7f12f5064b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f20c385b20 con 0x55f20c4cd060 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.155+0000 7f12f5064b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f20c4d9d60 con 0x55f20c4d19d0 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.155+0000 7f12f5064b80 10 client.0 _ll_register_callbacks cb 0x55f20c3c6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.156+0000 7f12e2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd980 0x55f20c4d6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.156+0000 7f12e2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd980 0x55f20c4d6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34242/0 (socket says 172.21.15.27:34242) 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.156+0000 7f12e2a00700 1 -- 172.21.15.27:0/3159943005 learned_addr learned my addr 172.21.15.27:0/3159943005 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.156+0000 7f12e3400700 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd060 0x55f20c4d8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.156+0000 7f12e3e00700 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4d19d0 0x55f20c4d6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.156+0000 7f12e2a00700 1 -- 172.21.15.27:0/3159943005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4d19d0 msgr2=0x55f20c4d6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.156+0000 7f12e2a00700 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4d19d0 0x55f20c4d6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.156+0000 7f12e2a00700 1 -- 172.21.15.27:0/3159943005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd060 msgr2=0x55f20c4d8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.156+0000 7f12e2a00700 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd060 0x55f20c4d8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.156+0000 7f12e2a00700 1 -- 172.21.15.27:0/3159943005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f20c3e5420 con 0x55f20c4cd980 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.156+0000 7f12e3400700 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd060 0x55f20c4d8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.157+0000 7f12e2a00700 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd980 0x55f20c4d6210 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f12d0000c00 tx=0x7f12d00a59a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a463e599a360a0af server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.157+0000 7f12cd600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.157+0000 7f12cd600700 1 -- 172.21.15.27:0/3159943005 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d00a9430 con 0x55f20c4cd980 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.157+0000 7f12cd600700 1 -- 172.21.15.27:0/3159943005 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f12d00b26c0 con 0x55f20c4cd980 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.157+0000 7f12f5064b80 1 -- 172.21.15.27:0/3159943005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12d40044b0 con 0x55f20c4cd980 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.157+0000 7f12cd600700 1 -- 172.21.15.27:0/3159943005 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f12d00a5270 con 0x55f20c4cd980 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.157+0000 7f12f5064b80 10 client.15495 fetch_fsmap learned FSMap version 30 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.157+0000 7f12f5064b80 1 -- 172.21.15.27:0/3159943005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f20c4e44c0 con 0x55f20c4cd980 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.158+0000 7f12cd600700 1 -- 172.21.15.27:0/3159943005 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12d40044b0 con 0x55f20c4cd980 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.158+0000 7f12cd600700 1 -- 172.21.15.27:0/3159943005 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f12d0010720 con 0x55f20c4cd980 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.158+0000 7f12f5064b80 10 client.15495 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:15:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.158+0000 7f12f5064b80 10 client.15495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:15:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.158+0000 7f12f5064b80 20 client.15495 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.158+0000 7f12f5064b80 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f20c4e6b00 0x55f20c4e8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.158+0000 7f12f5064b80 4 client.15495 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:15:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.158+0000 7f12f5064b80 1 -- 172.21.15.27:0/3159943005 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f20c49a0a0 con 0x55f20c4e6b00 2024-05-10T11:15:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.159+0000 7f12e3400700 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f20c4e6b00 0x55f20c4e8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.159+0000 7f12e3400700 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f20c4e6b00 0x55f20c4e8ef0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eec5e6dc4a5e6c2 server_cookie=2dd984e58fe28d2b in_seq=0 out_seq=0 2024-05-10T11:15:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.159+0000 7f12cd600700 10 client.15495 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.160+0000 7f12cd600700 1 -- 172.21.15.27:0/3159943005 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f20c49a0a0 con 0x55f20c4e6b00 2024-05-10T11:15:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.160+0000 7f12cd600700 10 client.15495 handle_command_reply: tid=0 2024-05-10T11:15:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.160+0000 7f12f5064b80 1 client.15495 shutdown 2024-05-10T11:15:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.161+0000 7f12f5064b80 1 -- 172.21.15.27:0/3159943005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd980 msgr2=0x55f20c4d6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.161+0000 7f12f5064b80 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd980 0x55f20c4d6210 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f12d0000c00 tx=0x7f12d00a59a0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.161+0000 7f12f5064b80 1 -- 172.21.15.27:0/3159943005 shutdown_connections 2024-05-10T11:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.161+0000 7f12f5064b80 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f20c4e6b00 0x55f20c4e8ef0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.161+0000 7f12f5064b80 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4d19d0 0x55f20c4d6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.161+0000 7f12f5064b80 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd060 0x55f20c4d8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.161+0000 7f12f5064b80 1 --2- 172.21.15.27:0/3159943005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd980 0x55f20c4d6210 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.161+0000 7f12cd600700 0 client.15495 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.161+0000 7f12f5064b80 1 -- 172.21.15.27:0/3159943005 >> 172.21.15.27:0/3159943005 conn(0x55f20c3fbb60 msgr2=0x55f20c4c22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.161+0000 7f12f5064b80 1 -- 172.21.15.27:0/3159943005 shutdown_connections 2024-05-10T11:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.161+0000 7f12f5064b80 1 -- 172.21.15.27:0/3159943005 wait complete. 2024-05-10T11:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.161+0000 7f12f5064b80 20 client.15495 trim_cache size 0 max 16384 2024-05-10T11:15:10.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.178+0000 7f12f5064b80 1 Processor -- start 2024-05-10T11:15:10.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.178+0000 7f12f5064b80 1 -- start start 2024-05-10T11:15:10.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12f5064b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 0x55f20c4d6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12f5064b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd980 0x55f20c4e2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12f5064b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4d19d0 0x55f20c4d3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12f5064b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f12e41b1880 con 0x55f20c4d19d0 2024-05-10T11:15:10.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12f5064b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f12e41b33c0 con 0x55f20c4cd980 2024-05-10T11:15:10.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12f5064b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12e41b1f50 con 0x55f20c4cd060 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12e3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 0x55f20c4d6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12e3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 0x55f20c4d6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51754/0 (socket says 172.21.15.27:51754) 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12e3400700 1 -- 172.21.15.27:0/165112853 learned_addr learned my addr 172.21.15.27:0/165112853 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12e3e00700 1 --2- 172.21.15.27:0/165112853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4d19d0 0x55f20c4d3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12e2a00700 1 --2- 172.21.15.27:0/165112853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd980 0x55f20c4e2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12e3400700 1 -- 172.21.15.27:0/165112853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd980 msgr2=0x55f20c4e2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.179+0000 7f12e3400700 1 --2- 172.21.15.27:0/165112853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd980 0x55f20c4e2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.180+0000 7f12e3400700 1 -- 172.21.15.27:0/165112853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4d19d0 msgr2=0x55f20c4d3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.180+0000 7f12e3400700 1 --2- 172.21.15.27:0/165112853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4d19d0 0x55f20c4d3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.180+0000 7f12e3400700 1 -- 172.21.15.27:0/165112853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f20c4c3f90 con 0x55f20c4cd060 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.180+0000 7f12e3400700 1 --2- 172.21.15.27:0/165112853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 0x55f20c4d6840 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f12d4006010 tx=0x7f12d4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfd1979ab32cf5c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.180+0000 7f12cfe00700 1 -- 172.21.15.27:0/165112853 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d4004420 con 0x55f20c4cd060 2024-05-10T11:15:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.180+0000 7f12cfe00700 1 -- 172.21.15.27:0/165112853 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12d402b070 con 0x55f20c4cd060 2024-05-10T11:15:10.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.180+0000 7f12cfe00700 1 -- 172.21.15.27:0/165112853 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d4003c20 con 0x55f20c4cd060 2024-05-10T11:15:10.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.181+0000 7f12f5064b80 1 -- 172.21.15.27:0/165112853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 msgr2=0x55f20c4d6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:10.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.181+0000 7f12f5064b80 1 --2- 172.21.15.27:0/165112853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 0x55f20c4d6840 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f12d4006010 tx=0x7f12d4000bc0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.181+0000 7f12f5064b80 1 -- 172.21.15.27:0/165112853 shutdown_connections 2024-05-10T11:15:10.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.181+0000 7f12f5064b80 1 --2- 172.21.15.27:0/165112853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd060 0x55f20c4d6840 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.181+0000 7f12f5064b80 1 --2- 172.21.15.27:0/165112853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4d19d0 0x55f20c4d3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.181+0000 7f12f5064b80 1 --2- 172.21.15.27:0/165112853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4cd980 0x55f20c4e2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.181+0000 7f12f5064b80 1 -- 172.21.15.27:0/165112853 >> 172.21.15.27:0/165112853 conn(0x55f20c3fbb60 msgr2=0x55f20c4d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:10.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.181+0000 7f12f5064b80 1 -- 172.21.15.27:0/165112853 shutdown_connections 2024-05-10T11:15:10.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.181+0000 7f12f5064b80 1 -- 172.21.15.27:0/165112853 wait complete. 2024-05-10T11:15:10.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.182+0000 7f12f5064b80 1 -- start start 2024-05-10T11:15:10.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.182+0000 7f12f5064b80 1 Processor -- start 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.182+0000 7f12f5064b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd060 0x55f20c4df710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.182+0000 7f12f5064b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd980 0x55f20c4d2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.182+0000 7f12f5064b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4d19d0 0x55f20c4d3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.182+0000 7f12f5064b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f12e41b2cf0 con 0x55f20c4cd060 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.182+0000 7f12f5064b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f12e41b2620 con 0x55f20c4d19d0 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.182+0000 7f12f5064b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12e41af820 con 0x55f20c4cd980 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12f5064b80 10 client.0 _ll_register_callbacks cb 0x55f20c3c6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12e3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd060 0x55f20c4df710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12e3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4d19d0 0x55f20c4d3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12e3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4d19d0 0x55f20c4d3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55100/0 (socket says 172.21.15.27:55100) 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12e3e00700 1 -- 172.21.15.27:0/4152273315 learned_addr learned my addr 172.21.15.27:0/4152273315 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12e3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd060 0x55f20c4df710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34252/0 (socket says 172.21.15.27:34252) 2024-05-10T11:15:10.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12e2a00700 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd980 0x55f20c4d2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12e3e00700 1 -- 172.21.15.27:0/4152273315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd980 msgr2=0x55f20c4d2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12e3e00700 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd980 0x55f20c4d2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12e3e00700 1 -- 172.21.15.27:0/4152273315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd060 msgr2=0x55f20c4df710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12e3e00700 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd060 0x55f20c4df710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12e3e00700 1 -- 172.21.15.27:0/4152273315 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f20c4dd2b0 con 0x55f20c4d19d0 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.183+0000 7f12e2a00700 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd980 0x55f20c4d2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.184+0000 7f12e3e00700 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4d19d0 0x55f20c4d3640 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009f90 tx=0x7f12dc009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff731cfebf20b6e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.184+0000 7f12cd600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.184+0000 7f12cd600700 1 -- 172.21.15.27:0/4152273315 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc005040 con 0x55f20c4d19d0 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.184+0000 7f12f5064b80 1 -- 172.21.15.27:0/4152273315 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12d402b070 con 0x55f20c4d19d0 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.184+0000 7f12cd600700 1 -- 172.21.15.27:0/4152273315 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f12dc0051e0 con 0x55f20c4d19d0 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.184+0000 7f12cd600700 1 -- 172.21.15.27:0/4152273315 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f12dc0053e0 con 0x55f20c4d19d0 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.184+0000 7f12f5064b80 10 client.25655 fetch_fsmap learned FSMap version 30 2024-05-10T11:15:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.184+0000 7f12f5064b80 1 -- 172.21.15.27:0/4152273315 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f20c551580 con 0x55f20c4d19d0 2024-05-10T11:15:10.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.185+0000 7f12cd600700 1 -- 172.21.15.27:0/4152273315 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12d402b070 con 0x55f20c4d19d0 2024-05-10T11:15:10.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.185+0000 7f12cd600700 1 -- 172.21.15.27:0/4152273315 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f12dc011070 con 0x55f20c4d19d0 2024-05-10T11:15:10.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.185+0000 7f12f5064b80 10 client.25655 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:15:10.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.185+0000 7f12f5064b80 10 client.25655 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:15:10.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.185+0000 7f12f5064b80 20 client.25655 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:10.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.185+0000 7f12f5064b80 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f20c551c70 0x55f20c554090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.185+0000 7f12f5064b80 4 client.25655 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.185+0000 7f12f5064b80 1 -- 172.21.15.27:0/4152273315 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f20c42d400 con 0x55f20c551c70 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.185+0000 7f12e3400700 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f20c551c70 0x55f20c554090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12e3400700 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f20c551c70 0x55f20c554090 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b295f90d5dceaac7 server_cookie=d8e8ad59c3fa9900 in_seq=0 out_seq=0 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12cd600700 10 client.25655 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12cd600700 1 -- 172.21.15.27:0/4152273315 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f20c42d400 con 0x55f20c551c70 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12cd600700 10 client.25655 handle_command_reply: tid=0 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12f5064b80 1 client.25655 shutdown 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12f5064b80 1 -- 172.21.15.27:0/4152273315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4d19d0 msgr2=0x55f20c4d3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12f5064b80 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4d19d0 0x55f20c4d3640 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009f90 tx=0x7f12dc009920 comp rx=0 tx=0).stop 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12f5064b80 1 -- 172.21.15.27:0/4152273315 shutdown_connections 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12f5064b80 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f20c551c70 0x55f20c554090 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12f5064b80 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f20c4d19d0 0x55f20c4d3640 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12f5064b80 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f20c4cd060 0x55f20c4df710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12f5064b80 1 --2- 172.21.15.27:0/4152273315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f20c4cd980 0x55f20c4d2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.186+0000 7f12f5064b80 1 -- 172.21.15.27:0/4152273315 >> 172.21.15.27:0/4152273315 conn(0x55f20c3fbb60 msgr2=0x55f20c4c22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.187+0000 7f12cd600700 0 client.25655 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.187+0000 7f12f5064b80 1 -- 172.21.15.27:0/4152273315 shutdown_connections 2024-05-10T11:15:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.187+0000 7f12f5064b80 1 -- 172.21.15.27:0/4152273315 wait complete. 2024-05-10T11:15:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.187+0000 7f12f5064b80 20 client.25655 trim_cache size 0 max 16384 2024-05-10T11:15:10.190 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:15:10.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.188+0000 7f12ea600700 1 -- 172.21.15.27:0/3925597585 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f12c80b29f0 msgr2=0x7f12c80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:10.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.188+0000 7f12ea600700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f12c80b29f0 0x7f12c80b4eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009da0 tx=0x7f12dc009700 comp rx=0 tx=0).stop 2024-05-10T11:15:10.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.188+0000 7f12ea600700 1 -- 172.21.15.27:0/3925597585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12e406bb40 msgr2=0x7f12e419c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:10.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.188+0000 7f12ea600700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12e406bb40 0x7f12e419c260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f12d000e9a0 tx=0x7f12d000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.188+0000 7f12ea600700 1 -- 172.21.15.27:0/3925597585 shutdown_connections 2024-05-10T11:15:10.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.188+0000 7f12ea600700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f12c80b29f0 0x7f12c80b4eb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.188+0000 7f12ea600700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12e4109510 0x7f12e41a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.188+0000 7f12ea600700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e406b180 0x7f12e419bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.188+0000 7f12ea600700 1 --2- 172.21.15.27:0/3925597585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12e406bb40 0x7f12e419c260 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:10.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.188+0000 7f12ea600700 1 -- 172.21.15.27:0/3925597585 >> 172.21.15.27:0/3925597585 conn(0x7f12e4079350 msgr2=0x7f12e410e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:10.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.188+0000 7f12ea600700 1 -- 172.21.15.27:0/3925597585 shutdown_connections 2024-05-10T11:15:10.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:10.188+0000 7f12ea600700 1 -- 172.21.15.27:0/3925597585 wait complete. 2024-05-10T11:15:10.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:10 smithi027 ceph-mon[30054]: pgmap v61: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-05-10T11:15:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:10 smithi076 ceph-mon[34152]: pgmap v61: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-05-10T11:15:10.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:10 smithi189 ceph-mon[34842]: pgmap v61: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-05-10T11:15:11.686 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 56% (6949/12271) Updating files: 57% (6995/12271) Updating files: 58% (7118/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 92% (11362/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-05-10T11:15:12.249 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:15:12.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:15:12.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:12 smithi027 ceph-mon[30054]: pgmap v62: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-10T11:15:12.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:12 smithi076 ceph-mon[34152]: pgmap v62: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-10T11:15:12.718 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:12.727 INFO:tasks.workunit.client.0.smithi027.stderr:Note: switching to 'd39fa08867836ac070c47cb66c782fa559fd480b'. 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr:state without impacting any branches by switching back to a branch. 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr: git switch -c 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr:Or undo this operation with: 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr: git switch - 2024-05-10T11:15:12.728 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-05-10T11:15:12.729 INFO:tasks.workunit.client.0.smithi027.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-10T11:15:12.729 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-05-10T11:15:12.729 INFO:tasks.workunit.client.0.smithi027.stderr:HEAD is now at d39fa088678 client: fix file cache cap leak which can stall async read call 2024-05-10T11:15:12.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:12 smithi189 ceph-mon[34842]: pgmap v62: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-10T11:15:12.743 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-10T11:15:12.854 INFO:tasks.workunit.client.0.smithi027.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-10T11:15:12.862 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-10T11:15:12.862 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-10T11:15:14.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:14 smithi027 ceph-mon[30054]: pgmap v63: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-10T11:15:14.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:14 smithi076 ceph-mon[34152]: pgmap v63: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-10T11:15:14.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:14 smithi189 ceph-mon[34842]: pgmap v63: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-10T11:15:15.050 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-10T11:15:15.168 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-10T11:15:15.259 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-10T11:15:15.266 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-10T11:15:15.266 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-10T11:15:15.351 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-10T11:15:15.362 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T11:15:15.362 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-10T11:15:15.412 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-05-10T11:15:15.414 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-05-10T11:15:15.414 DEBUG:teuthology.orchestra.run.smithi027:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39fa08867836ac070c47cb66c782fa559fd480b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-05-10T11:15:15.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.450+0000 7f5d79a00700 1 -- 172.21.15.27:0/2497590528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d7406b180 msgr2=0x7f5d7406b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:15.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.450+0000 7f5d79a00700 1 --2- 172.21.15.27:0/2497590528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d7406b180 0x7f5d7406b600 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c009990 tx=0x7f5d5c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.450+0000 7f5d79a00700 1 -- 172.21.15.27:0/2497590528 shutdown_connections 2024-05-10T11:15:15.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.450+0000 7f5d79a00700 1 --2- 172.21.15.27:0/2497590528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7406bb40 0x7f5d74110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.450+0000 7f5d79a00700 1 --2- 172.21.15.27:0/2497590528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d74109510 0x7f5d741098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.450+0000 7f5d79a00700 1 --2- 172.21.15.27:0/2497590528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d7406b180 0x7f5d7406b600 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.450+0000 7f5d79a00700 1 -- 172.21.15.27:0/2497590528 >> 172.21.15.27:0/2497590528 conn(0x7f5d74079350 msgr2=0x7f5d74079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:15.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.450+0000 7f5d79a00700 1 -- 172.21.15.27:0/2497590528 shutdown_connections 2024-05-10T11:15:15.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.451+0000 7f5d79a00700 1 -- 172.21.15.27:0/2497590528 wait complete. 2024-05-10T11:15:15.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.452+0000 7f5d79a00700 1 Processor -- start 2024-05-10T11:15:15.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.452+0000 7f5d79a00700 1 -- start start 2024-05-10T11:15:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.453+0000 7f5d79a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7406b180 0x7f5d74102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.453+0000 7f5d79a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d7406bb40 0x7f5d741029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.453+0000 7f5d79a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d74109510 0x7f5d74102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.453+0000 7f5d79a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74113620 con 0x7f5d7406b180 2024-05-10T11:15:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.453+0000 7f5d79a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741134a0 con 0x7f5d74109510 2024-05-10T11:15:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.453+0000 7f5d79a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741137a0 con 0x7f5d7406bb40 2024-05-10T11:15:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.453+0000 7f5d72a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7406b180 0x7f5d74102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.453+0000 7f5d72a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7406b180 0x7f5d74102470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32918/0 (socket says 172.21.15.27:32918) 2024-05-10T11:15:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.453+0000 7f5d72a00700 1 -- 172.21.15.27:0/663668266 learned_addr learned my addr 172.21.15.27:0/663668266 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.453+0000 7f5d73400700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d74109510 0x7f5d74102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.454+0000 7f5d6be00700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d7406bb40 0x7f5d741029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.454+0000 7f5d73400700 1 -- 172.21.15.27:0/663668266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d7406bb40 msgr2=0x7f5d741029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:15.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.454+0000 7f5d73400700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d7406bb40 0x7f5d741029b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.454+0000 7f5d73400700 1 -- 172.21.15.27:0/663668266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7406b180 msgr2=0x7f5d74102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:15.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.454+0000 7f5d73400700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7406b180 0x7f5d74102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.454+0000 7f5d73400700 1 -- 172.21.15.27:0/663668266 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d741a8120 con 0x7f5d74109510 2024-05-10T11:15:15.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.454+0000 7f5d73400700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d74109510 0x7f5d74102ef0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00efc0 tx=0x7f5d6c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:15.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.455+0000 7f5d6b400700 1 -- 172.21.15.27:0/663668266 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c0092e0 con 0x7f5d74109510 2024-05-10T11:15:15.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.455+0000 7f5d79a00700 1 -- 172.21.15.27:0/663668266 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d741a83b0 con 0x7f5d74109510 2024-05-10T11:15:15.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.455+0000 7f5d6b400700 1 -- 172.21.15.27:0/663668266 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d6c004bb0 con 0x7f5d74109510 2024-05-10T11:15:15.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.455+0000 7f5d6b400700 1 -- 172.21.15.27:0/663668266 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c004020 con 0x7f5d74109510 2024-05-10T11:15:15.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.455+0000 7f5d79a00700 1 -- 172.21.15.27:0/663668266 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d741a88c0 con 0x7f5d74109510 2024-05-10T11:15:15.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.459+0000 7f5d6b400700 1 -- 172.21.15.27:0/663668266 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f5d6c01b030 con 0x7f5d74109510 2024-05-10T11:15:15.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.460+0000 7f5d842bdb80 1 Processor -- start 2024-05-10T11:15:15.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.460+0000 7f5d842bdb80 1 -- start start 2024-05-10T11:15:15.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.460+0000 7f5d842bdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264cfd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.460+0000 7f5d842bdb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264cf7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.460+0000 7f5d842bdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cf7970 0x560264cf7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.460+0000 7f5d842bdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560264c49aa0 con 0x560264cf7970 2024-05-10T11:15:15.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.460+0000 7f5d842bdb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x560264be77e0 con 0x560264cfd560 2024-05-10T11:15:15.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.460+0000 7f5d842bdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560264c45a60 con 0x560264cfcc40 2024-05-10T11:15:15.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.461+0000 7f5d6be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264cfd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.461+0000 7f5d6be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264cfd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:35964/0 (socket says 172.21.15.27:35964) 2024-05-10T11:15:15.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.461+0000 7f5d72a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264cf7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.461+0000 7f5d6be00700 1 -- 172.21.15.27:0/331022880 learned_addr learned my addr 172.21.15.27:0/331022880 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:15.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.461+0000 7f5d6be00700 1 --2- 172.21.15.27:0/331022880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cf7970 0x560264cf7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.461+0000 7f5d6b400700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f5d580b2a70 0x7f5d580b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.462+0000 7f5d6b400700 1 -- 172.21.15.27:0/663668266 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f5d6c0e24b0 con 0x7f5d74109510 2024-05-10T11:15:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.462+0000 7f5d6be00700 1 -- 172.21.15.27:0/331022880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 msgr2=0x560264cf7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.462+0000 7f5d6be00700 1 --2- 172.21.15.27:0/331022880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264cf7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.462+0000 7f5d6be00700 1 -- 172.21.15.27:0/331022880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cf7970 msgr2=0x560264cf7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.462+0000 7f5d6be00700 1 --2- 172.21.15.27:0/331022880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cf7970 0x560264cf7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.462+0000 7f5d6be00700 1 -- 172.21.15.27:0/331022880 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560264b96950 con 0x560264cfcc40 2024-05-10T11:15:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.462+0000 7f5d73400700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f5d580b2a70 0x7f5d580b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.462+0000 7f5d6be00700 1 --2- 172.21.15.27:0/331022880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264cfd020 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x560264cf8330 tx=0x7f5d5c005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2541737342ffe1c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:15.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.463+0000 7f5d63e00700 1 -- 172.21.15.27:0/331022880 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c00b920 con 0x560264cfcc40 2024-05-10T11:15:15.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.463+0000 7f5d63e00700 1 -- 172.21.15.27:0/331022880 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d5c003680 con 0x560264cfcc40 2024-05-10T11:15:15.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.463+0000 7f5d63e00700 1 -- 172.21.15.27:0/331022880 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c00f6c0 con 0x560264cfcc40 2024-05-10T11:15:15.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.463+0000 7f5d73400700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f5d580b2a70 0x7f5d580b4f30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c009fb0 tx=0x7f5d6c009040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:15.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.463+0000 7f5d842bdb80 1 -- 172.21.15.27:0/331022880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 msgr2=0x560264cfd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:15.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.463+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/331022880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264cfd020 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x560264cf8330 tx=0x7f5d5c005e80 comp rx=0 tx=0).stop 2024-05-10T11:15:15.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.464+0000 7f5d842bdb80 1 -- 172.21.15.27:0/331022880 shutdown_connections 2024-05-10T11:15:15.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.464+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/331022880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264cfd020 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.464+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/331022880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cf7970 0x560264cf7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.464+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/331022880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264cf7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.464+0000 7f5d842bdb80 1 -- 172.21.15.27:0/331022880 >> 172.21.15.27:0/331022880 conn(0x560264c6f750 msgr2=0x560264cf1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:15.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.465+0000 7f5d842bdb80 1 -- 172.21.15.27:0/331022880 shutdown_connections 2024-05-10T11:15:15.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.465+0000 7f5d842bdb80 1 -- 172.21.15.27:0/331022880 wait complete. 2024-05-10T11:15:15.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.465+0000 7f5d842bdb80 1 -- start start 2024-05-10T11:15:15.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.466+0000 7f5d842bdb80 1 Processor -- start 2024-05-10T11:15:15.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.466+0000 7f5d842bdb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cf7970 0x560264d03250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.466+0000 7f5d842bdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264d005f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.466+0000 7f5d842bdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cfd560 0x560264d00b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.466+0000 7f5d842bdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560264d03f40 con 0x560264cfd560 2024-05-10T11:15:15.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.466+0000 7f5d842bdb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x560264bafba0 con 0x560264cf7970 2024-05-10T11:15:15.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.466+0000 7f5d842bdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560264d040c0 con 0x560264cfcc40 2024-05-10T11:15:15.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.466+0000 7f5d842bdb80 10 client.0 _ll_register_callbacks cb 0x560264bf0cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:15.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d72a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264d005f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d72a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264d005f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:35976/0 (socket says 172.21.15.27:35976) 2024-05-10T11:15:15.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d72a00700 1 -- 172.21.15.27:0/205900576 learned_addr learned my addr 172.21.15.27:0/205900576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d6be00700 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cf7970 0x560264d03250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d6be00700 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cfd560 0x560264d00b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d72a00700 1 -- 172.21.15.27:0/205900576 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cf7970 msgr2=0x560264d03250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d72a00700 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cf7970 0x560264d03250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d72a00700 1 -- 172.21.15.27:0/205900576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cfd560 msgr2=0x560264d00b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d72a00700 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cfd560 0x560264d00b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d72a00700 1 -- 172.21.15.27:0/205900576 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560264c0f4a0 con 0x560264cfcc40 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d6be00700 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cfd560 0x560264d00b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d72a00700 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264d005f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5d64012790 tx=0x7f5d64012b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1487e0bd7785d1a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.467+0000 7f5d60c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.468+0000 7f5d60c00700 1 -- 172.21.15.27:0/205900576 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6400f5c0 con 0x560264cfcc40 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.468+0000 7f5d60c00700 1 -- 172.21.15.27:0/205900576 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f5d6400f760 con 0x560264cfcc40 2024-05-10T11:15:15.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.468+0000 7f5d842bdb80 1 -- 172.21.15.27:0/205900576 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d5c003680 con 0x560264cfcc40 2024-05-10T11:15:15.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.468+0000 7f5d60c00700 1 -- 172.21.15.27:0/205900576 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5d6400f960 con 0x560264cfcc40 2024-05-10T11:15:15.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.468+0000 7f5d842bdb80 10 client.25657 fetch_fsmap learned FSMap version 30 2024-05-10T11:15:15.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.468+0000 7f5d842bdb80 1 -- 172.21.15.27:0/205900576 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560264d0e870 con 0x560264cfcc40 2024-05-10T11:15:15.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.469+0000 7f5d60c00700 1 -- 172.21.15.27:0/205900576 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d5c003680 con 0x560264cfcc40 2024-05-10T11:15:15.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.469+0000 7f5d60c00700 1 -- 172.21.15.27:0/205900576 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5d64010b40 con 0x560264cfcc40 2024-05-10T11:15:15.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.469+0000 7f5d842bdb80 10 client.25657 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:15:15.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.469+0000 7f5d842bdb80 10 client.25657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:15:15.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.469+0000 7f5d842bdb80 20 client.25657 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:15.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.469+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560264d10f00 0x560264d132f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.469+0000 7f5d842bdb80 4 client.25657 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:15:15.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.469+0000 7f5d842bdb80 1 -- 172.21.15.27:0/205900576 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560264cc41c0 con 0x560264d10f00 2024-05-10T11:15:15.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.470+0000 7f5d6be00700 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560264d10f00 0x560264d132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.470+0000 7f5d6be00700 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560264d10f00 0x560264d132f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f21f59c608040fba server_cookie=a4858d1ae5d20f1b in_seq=0 out_seq=0 2024-05-10T11:15:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.470+0000 7f5d60c00700 10 client.25657 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.471+0000 7f5d60c00700 1 -- 172.21.15.27:0/205900576 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560264cc41c0 con 0x560264d10f00 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.471+0000 7f5d60c00700 10 client.25657 handle_command_reply: tid=0 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.471+0000 7f5d842bdb80 1 client.25657 shutdown 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.471+0000 7f5d842bdb80 1 -- 172.21.15.27:0/205900576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 msgr2=0x560264d005f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.471+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264d005f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5d64012790 tx=0x7f5d64012b50 comp rx=0 tx=0).stop 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.472+0000 7f5d842bdb80 1 -- 172.21.15.27:0/205900576 shutdown_connections 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.472+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560264d10f00 0x560264d132f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.472+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cfd560 0x560264d00b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.472+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cf7970 0x560264d03250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.472+0000 7f5d60c00700 0 client.25657 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.472+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/205900576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264d005f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.472+0000 7f5d842bdb80 1 -- 172.21.15.27:0/205900576 >> 172.21.15.27:0/205900576 conn(0x560264c6f750 msgr2=0x560264cf1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.472+0000 7f5d842bdb80 1 -- 172.21.15.27:0/205900576 shutdown_connections 2024-05-10T11:15:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.472+0000 7f5d842bdb80 1 -- 172.21.15.27:0/205900576 wait complete. 2024-05-10T11:15:15.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.472+0000 7f5d842bdb80 20 client.25657 trim_cache size 0 max 16384 2024-05-10T11:15:15.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.496+0000 7f5d842bdb80 1 Processor -- start 2024-05-10T11:15:15.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.496+0000 7f5d842bdb80 1 -- start start 2024-05-10T11:15:15.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.497+0000 7f5d842bdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cf7970 0x560264d00b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.497+0000 7f5d842bdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264d0d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.497+0000 7f5d842bdb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264cf1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.497+0000 7f5d842bdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741b1700 con 0x560264cf7970 2024-05-10T11:15:15.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.497+0000 7f5d842bdb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741b3240 con 0x560264cfd560 2024-05-10T11:15:15.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.497+0000 7f5d842bdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741b1dd0 con 0x560264cfcc40 2024-05-10T11:15:15.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.497+0000 7f5d6be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cf7970 0x560264d00b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.497+0000 7f5d72a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264d0d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.497+0000 7f5d72a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264d0d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:35992/0 (socket says 172.21.15.27:35992) 2024-05-10T11:15:15.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.497+0000 7f5d72a00700 1 -- 172.21.15.27:0/1052585708 learned_addr learned my addr 172.21.15.27:0/1052585708 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:15.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.497+0000 7f5d6be00700 1 --2- 172.21.15.27:0/1052585708 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264cf1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.498+0000 7f5d72a00700 1 -- 172.21.15.27:0/1052585708 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 msgr2=0x560264cf1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:15.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.498+0000 7f5d72a00700 1 --2- 172.21.15.27:0/1052585708 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264cf1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.498+0000 7f5d72a00700 1 -- 172.21.15.27:0/1052585708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cf7970 msgr2=0x560264d00b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:15.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.498+0000 7f5d72a00700 1 --2- 172.21.15.27:0/1052585708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cf7970 0x560264d00b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.498+0000 7f5d72a00700 1 -- 172.21.15.27:0/1052585708 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560264d13660 con 0x560264cfcc40 2024-05-10T11:15:15.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.498+0000 7f5d72a00700 1 --2- 172.21.15.27:0/1052585708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264d0d030 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5d64000c00 tx=0x7f5d64005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f3b2c0a2ea6375a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:15.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.498+0000 7f5d63400700 1 -- 172.21.15.27:0/1052585708 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6400f640 con 0x560264cfcc40 2024-05-10T11:15:15.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.498+0000 7f5d63400700 1 -- 172.21.15.27:0/1052585708 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d6401c5b0 con 0x560264cfcc40 2024-05-10T11:15:15.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.498+0000 7f5d63400700 1 -- 172.21.15.27:0/1052585708 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d64005290 con 0x560264cfcc40 2024-05-10T11:15:15.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.499+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1052585708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 msgr2=0x560264d0d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:15.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.499+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/1052585708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264d0d030 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5d64000c00 tx=0x7f5d64005740 comp rx=0 tx=0).stop 2024-05-10T11:15:15.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.499+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1052585708 shutdown_connections 2024-05-10T11:15:15.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.499+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/1052585708 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264cf1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.499+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/1052585708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cf7970 0x560264d00b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.499+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/1052585708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cfcc40 0x560264d0d030 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.499+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1052585708 >> 172.21.15.27:0/1052585708 conn(0x560264c6f750 msgr2=0x560264cfc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:15.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.499+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1052585708 shutdown_connections 2024-05-10T11:15:15.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.499+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1052585708 wait complete. 2024-05-10T11:15:15.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.500+0000 7f5d842bdb80 1 -- start start 2024-05-10T11:15:15.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.500+0000 7f5d842bdb80 1 Processor -- start 2024-05-10T11:15:15.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.501+0000 7f5d842bdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cf7970 0x560264d09870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.501+0000 7f5d842bdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cfcc40 0x560264d06380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.501+0000 7f5d842bdb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264d068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.501+0000 7f5d842bdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741b2b70 con 0x560264cfcc40 2024-05-10T11:15:15.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.501+0000 7f5d842bdb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741b24a0 con 0x560264cfd560 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.501+0000 7f5d842bdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741af6a0 con 0x560264cf7970 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.501+0000 7f5d842bdb80 10 client.0 _ll_register_callbacks cb 0x560264bf0cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.501+0000 7f5d6be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264d068c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.501+0000 7f5d72a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cfcc40 0x560264d06380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.501+0000 7f5d6be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264d068c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44532/0 (socket says 172.21.15.27:44532) 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.501+0000 7f5d6be00700 1 -- 172.21.15.27:0/1227553044 learned_addr learned my addr 172.21.15.27:0/1227553044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.501+0000 7f5d6be00700 1 --2- 172.21.15.27:0/1227553044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cf7970 0x560264d09870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.502+0000 7f5d6be00700 1 -- 172.21.15.27:0/1227553044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cf7970 msgr2=0x560264d09870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.502+0000 7f5d6be00700 1 --2- 172.21.15.27:0/1227553044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cf7970 0x560264d09870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.502+0000 7f5d6be00700 1 -- 172.21.15.27:0/1227553044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cfcc40 msgr2=0x560264d06380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.502+0000 7f5d6be00700 1 --2- 172.21.15.27:0/1227553044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cfcc40 0x560264d06380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.502+0000 7f5d6be00700 1 -- 172.21.15.27:0/1227553044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560264d07100 con 0x560264cfd560 2024-05-10T11:15:15.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.502+0000 7f5d6be00700 1 --2- 172.21.15.27:0/1227553044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264d068c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0098e0 tx=0x7f5d5c005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe228a77a2db4581 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:15.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.502+0000 7f5d60c00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:15.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.502+0000 7f5d60c00700 1 -- 172.21.15.27:0/1227553044 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c004ef0 con 0x560264cfd560 2024-05-10T11:15:15.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.502+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1227553044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d6401c5b0 con 0x560264cfd560 2024-05-10T11:15:15.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.502+0000 7f5d60c00700 1 -- 172.21.15.27:0/1227553044 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f5d5c00b720 con 0x560264cfd560 2024-05-10T11:15:15.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.503+0000 7f5d60c00700 1 -- 172.21.15.27:0/1227553044 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5d5c005090 con 0x560264cfd560 2024-05-10T11:15:15.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.503+0000 7f5d842bdb80 10 client.25670 fetch_fsmap learned FSMap version 30 2024-05-10T11:15:15.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.503+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1227553044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560264d7c7b0 con 0x560264cfd560 2024-05-10T11:15:15.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.503+0000 7f5d60c00700 1 -- 172.21.15.27:0/1227553044 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d6401c5b0 con 0x560264cfd560 2024-05-10T11:15:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.503+0000 7f5d60c00700 1 -- 172.21.15.27:0/1227553044 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5d5c00f950 con 0x560264cfd560 2024-05-10T11:15:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.504+0000 7f5d842bdb80 10 client.25670 fetch_fsmap finished waiting for FSMap version 30 2024-05-10T11:15:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.504+0000 7f5d842bdb80 10 client.25670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:15:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.504+0000 7f5d842bdb80 20 client.25670 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.504+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/1227553044 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560264d7ca40 0x560264d7ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.504+0000 7f5d842bdb80 4 client.25670 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:15:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.504+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1227553044 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560264c57480 con 0x560264d7ca40 2024-05-10T11:15:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.504+0000 7f5d72a00700 1 --2- 172.21.15.27:0/1227553044 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560264d7ca40 0x560264d7ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.505+0000 7f5d72a00700 1 --2- 172.21.15.27:0/1227553044 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560264d7ca40 0x560264d7ee30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=468301a2ebb6310b server_cookie=7d22dbc9028d2bb4 in_seq=0 out_seq=0 2024-05-10T11:15:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.505+0000 7f5d60c00700 10 client.25670 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:15.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.506+0000 7f5d60c00700 1 -- 172.21.15.27:0/1227553044 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560264c57480 con 0x560264d7ca40 2024-05-10T11:15:15.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.506+0000 7f5d60c00700 10 client.25670 handle_command_reply: tid=0 2024-05-10T11:15:15.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.506+0000 7f5d842bdb80 1 client.25670 shutdown 2024-05-10T11:15:15.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.506+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1227553044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 msgr2=0x560264d068c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:15.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.506+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/1227553044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264d068c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0098e0 tx=0x7f5d5c005e80 comp rx=0 tx=0).stop 2024-05-10T11:15:15.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.507+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1227553044 shutdown_connections 2024-05-10T11:15:15.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.507+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/1227553044 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560264d7ca40 0x560264d7ee30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.507+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/1227553044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560264cf7970 0x560264d09870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.507+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/1227553044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560264cfd560 0x560264d068c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.507+0000 7f5d842bdb80 1 --2- 172.21.15.27:0/1227553044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560264cfcc40 0x560264d06380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.507+0000 7f5d60c00700 0 client.25670 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:15.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.507+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1227553044 >> 172.21.15.27:0/1227553044 conn(0x560264c6f750 msgr2=0x560264d01bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:15.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.507+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1227553044 shutdown_connections 2024-05-10T11:15:15.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.507+0000 7f5d842bdb80 1 -- 172.21.15.27:0/1227553044 wait complete. 2024-05-10T11:15:15.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.507+0000 7f5d842bdb80 20 client.25670 trim_cache size 0 max 16384 2024-05-10T11:15:15.509 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:15:15.509 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:15:15.509 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "fc8afe3b-4f27-4f5f-a2f7-f7fc468bd0f4", 2024-05-10T11:15:15.509 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:15:15.509 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:15:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.508+0000 7f5d79a00700 1 -- 172.21.15.27:0/663668266 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f5d580b2a70 msgr2=0x7f5d580b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.508+0000 7f5d79a00700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f5d580b2a70 0x7f5d580b4f30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c009fb0 tx=0x7f5d6c009040 comp rx=0 tx=0).stop 2024-05-10T11:15:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.508+0000 7f5d79a00700 1 -- 172.21.15.27:0/663668266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d74109510 msgr2=0x7f5d74102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.508+0000 7f5d79a00700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d74109510 0x7f5d74102ef0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00efc0 tx=0x7f5d6c00c3a0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.508+0000 7f5d79a00700 1 -- 172.21.15.27:0/663668266 shutdown_connections 2024-05-10T11:15:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.508+0000 7f5d79a00700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f5d580b2a70 0x7f5d580b4f30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.508+0000 7f5d79a00700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d74109510 0x7f5d74102ef0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.508+0000 7f5d79a00700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7406b180 0x7f5d74102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.508+0000 7f5d79a00700 1 --2- 172.21.15.27:0/663668266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d7406bb40 0x7f5d741029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.508+0000 7f5d79a00700 1 -- 172.21.15.27:0/663668266 >> 172.21.15.27:0/663668266 conn(0x7f5d74079350 msgr2=0x7f5d74101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.509+0000 7f5d79a00700 1 -- 172.21.15.27:0/663668266 shutdown_connections 2024-05-10T11:15:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:15.509+0000 7f5d79a00700 1 -- 172.21.15.27:0/663668266 wait complete. 2024-05-10T11:15:15.516 INFO:tasks.workunit.client.0.smithi027.stderr:+ dbench 1 2024-05-10T11:15:15.529 INFO:tasks.workunit.client.0.smithi027.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-05-10T11:15:15.529 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-05-10T11:15:15.529 INFO:tasks.workunit.client.0.smithi027.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-05-10T11:15:15.549 INFO:tasks.workunit.client.0.smithi027.stdout:0 of 1 processes prepared for launch 0 sec 2024-05-10T11:15:15.649 INFO:tasks.workunit.client.0.smithi027.stdout:1 of 1 processes prepared for launch 0 sec 2024-05-10T11:15:15.649 INFO:tasks.workunit.client.0.smithi027.stdout:releasing clients 2024-05-10T11:15:16.519 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:15:16.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:16 smithi027 ceph-mon[30054]: pgmap v64: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-05-10T11:15:16.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:16 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:15:16.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:16 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:16.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:16 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:15:16.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:16 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:16.649 INFO:tasks.workunit.client.0.smithi027.stdout: 1 1884 26.37 MB/sec warmup 1 sec latency 12.939 ms 2024-05-10T11:15:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:16 smithi076 ceph-mon[34152]: pgmap v64: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-05-10T11:15:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:16 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:15:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:16 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:16 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:15:16.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:16 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:16.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:16 smithi189 ceph-mon[34842]: pgmap v64: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-05-10T11:15:16.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:16 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:15:16.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:16 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:16.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:16 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:15:16.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:16 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:17.067 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:17.572 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:17 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:15:17.572 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:17 smithi027 ceph-mon[30054]: pgmap v65: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-10T11:15:17.650 INFO:tasks.workunit.client.0.smithi027.stdout: 1 4179 20.04 MB/sec warmup 2 sec latency 9.920 ms 2024-05-10T11:15:17.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:17 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:15:17.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:17 smithi076 ceph-mon[34152]: pgmap v65: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-10T11:15:17.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:17 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:15:17.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:17 smithi189 ceph-mon[34842]: pgmap v65: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-10T11:15:18.575 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:18 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:15:18.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:18 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:15:18.650 INFO:tasks.workunit.client.0.smithi027.stdout: 1 6477 17.53 MB/sec warmup 3 sec latency 15.049 ms 2024-05-10T11:15:18.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:18 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:15:19.498 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:19 smithi027 ceph-mon[30054]: pgmap v66: 129 pgs: 129 active+clean; 5.2 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 368 KiB/s wr, 57 op/s 2024-05-10T11:15:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.548+0000 7fd92c200700 1 -- 172.21.15.27:0/970280095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd924108850 msgr2=0x7fd9241103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.548+0000 7fd92c200700 1 --2- 172.21.15.27:0/970280095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd924108850 0x7fd9241103d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd91c00b0d0 tx=0x7fd91c00b3e0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.548+0000 7fd92c200700 1 -- 172.21.15.27:0/970280095 shutdown_connections 2024-05-10T11:15:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.548+0000 7fd92c200700 1 --2- 172.21.15.27:0/970280095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd924108850 0x7fd9241103d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.548+0000 7fd92c200700 1 --2- 172.21.15.27:0/970280095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd924103750 0x7fd924103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.548+0000 7fd92c200700 1 --2- 172.21.15.27:0/970280095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd924104100 0x7fd924108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.548+0000 7fd92c200700 1 -- 172.21.15.27:0/970280095 >> 172.21.15.27:0/970280095 conn(0x7fd92407a600 msgr2=0x7fd92407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.548+0000 7fd92c200700 1 -- 172.21.15.27:0/970280095 shutdown_connections 2024-05-10T11:15:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.548+0000 7fd92c200700 1 -- 172.21.15.27:0/970280095 wait complete. 2024-05-10T11:15:19.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.549+0000 7fd92c200700 1 Processor -- start 2024-05-10T11:15:19.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.549+0000 7fd92c200700 1 -- start start 2024-05-10T11:15:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.550+0000 7fd92c200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd924103750 0x7fd9241a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.550+0000 7fd92c200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd924104100 0x7fd9241a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.550+0000 7fd92c200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd924108850 0x7fd92419baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.550+0000 7fd92c200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd924113670 con 0x7fd924103750 2024-05-10T11:15:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.550+0000 7fd92c200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd9241134f0 con 0x7fd924108850 2024-05-10T11:15:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.550+0000 7fd92c200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd9241137f0 con 0x7fd924104100 2024-05-10T11:15:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.550+0000 7fd929800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd924103750 0x7fd9241a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.550+0000 7fd929800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd924103750 0x7fd9241a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32964/0 (socket says 172.21.15.27:32964) 2024-05-10T11:15:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.550+0000 7fd929800700 1 -- 172.21.15.27:0/1052790773 learned_addr learned my addr 172.21.15.27:0/1052790773 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.550+0000 7fd928e00700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd924104100 0x7fd9241a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.550+0000 7fd92a200700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd924108850 0x7fd92419baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.551+0000 7fd92a200700 1 -- 172.21.15.27:0/1052790773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd924104100 msgr2=0x7fd9241a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.551+0000 7fd92a200700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd924104100 0x7fd9241a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.551+0000 7fd92a200700 1 -- 172.21.15.27:0/1052790773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd924103750 msgr2=0x7fd9241a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.551+0000 7fd92a200700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd924103750 0x7fd9241a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.551+0000 7fd92a200700 1 -- 172.21.15.27:0/1052790773 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd92419c250 con 0x7fd924108850 2024-05-10T11:15:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.551+0000 7fd928e00700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd924104100 0x7fd9241a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.551+0000 7fd92a200700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd924108850 0x7fd92419baf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd91c00b770 tx=0x7fd91c007980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.551+0000 7fd922000700 1 -- 172.21.15.27:0/1052790773 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c01f070 con 0x7fd924108850 2024-05-10T11:15:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.551+0000 7fd922000700 1 -- 172.21.15.27:0/1052790773 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd91c007dc0 con 0x7fd924108850 2024-05-10T11:15:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.551+0000 7fd92c200700 1 -- 172.21.15.27:0/1052790773 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd92419c4e0 con 0x7fd924108850 2024-05-10T11:15:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.551+0000 7fd922000700 1 -- 172.21.15.27:0/1052790773 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c01b430 con 0x7fd924108850 2024-05-10T11:15:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.552+0000 7fd92c200700 1 -- 172.21.15.27:0/1052790773 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9241a8800 con 0x7fd924108850 2024-05-10T11:15:19.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.556+0000 7fd922000700 1 -- 172.21.15.27:0/1052790773 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd91c022070 con 0x7fd924108850 2024-05-10T11:15:19.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.556+0000 7fd936c4eb80 1 Processor -- start 2024-05-10T11:15:19.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.557+0000 7fd936c4eb80 1 -- start start 2024-05-10T11:15:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.557+0000 7fd936c4eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c3223440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.557+0000 7fd936c4eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c3223980 0x5592c3227490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.557+0000 7fd936c4eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c32279d0 0x5592c3227e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.557+0000 7fd936c4eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5592c3175a20 con 0x5592c32279d0 2024-05-10T11:15:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.557+0000 7fd936c4eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5592c3113760 con 0x5592c3223980 2024-05-10T11:15:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.557+0000 7fd936c4eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5592c31719e0 con 0x5592c3223060 2024-05-10T11:15:19.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.558+0000 7fd928e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c32279d0 0x5592c3227e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.558+0000 7fd928e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c3223440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.558+0000 7fd929800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c3223980 0x5592c3227490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.558+0000 7fd928e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c3223440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36020/0 (socket says 172.21.15.27:36020) 2024-05-10T11:15:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.558+0000 7fd928e00700 1 -- 172.21.15.27:0/365841367 learned_addr learned my addr 172.21.15.27:0/365841367 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.559+0000 7fd928e00700 1 -- 172.21.15.27:0/365841367 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c3223980 msgr2=0x5592c3227490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.559+0000 7fd928e00700 1 --2- 172.21.15.27:0/365841367 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c3223980 0x5592c3227490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.559+0000 7fd928e00700 1 -- 172.21.15.27:0/365841367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c32279d0 msgr2=0x5592c3227e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.559+0000 7fd928e00700 1 --2- 172.21.15.27:0/365841367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c32279d0 0x5592c3227e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.559+0000 7fd928e00700 1 -- 172.21.15.27:0/365841367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592c30c28d0 con 0x5592c3223060 2024-05-10T11:15:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.559+0000 7fd922000700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd90c0b29f0 0x7fd90c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.559+0000 7fd928e00700 1 --2- 172.21.15.27:0/365841367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c3223440 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd910013a90 tx=0x7fd9100047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af6771d423292464 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.559+0000 7fd922000700 1 -- 172.21.15.27:0/1052790773 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fd91c0e34e0 con 0x7fd924108850 2024-05-10T11:15:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.559+0000 7fd929800700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd90c0b29f0 0x7fd90c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.559+0000 7fd916a00700 1 -- 172.21.15.27:0/365841367 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910013270 con 0x5592c3223060 2024-05-10T11:15:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.560+0000 7fd916a00700 1 -- 172.21.15.27:0/365841367 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd910010c90 con 0x5592c3223060 2024-05-10T11:15:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.560+0000 7fd916a00700 1 -- 172.21.15.27:0/365841367 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910011cd0 con 0x5592c3223060 2024-05-10T11:15:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.560+0000 7fd936c4eb80 1 -- 172.21.15.27:0/365841367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 msgr2=0x5592c3223440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.560+0000 7fd929800700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd90c0b29f0 0x7fd90c0b4eb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd918009680 tx=0x7fd918009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.560+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/365841367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c3223440 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd910013a90 tx=0x7fd9100047c0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.561+0000 7fd936c4eb80 1 -- 172.21.15.27:0/365841367 shutdown_connections 2024-05-10T11:15:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.561+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/365841367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c3223440 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.561+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/365841367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c32279d0 0x5592c3227e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.561+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/365841367 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c3223980 0x5592c3227490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.561+0000 7fd936c4eb80 1 -- 172.21.15.27:0/365841367 >> 172.21.15.27:0/365841367 conn(0x5592c3151b60 msgr2=0x5592c321d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.561+0000 7fd936c4eb80 1 -- 172.21.15.27:0/365841367 shutdown_connections 2024-05-10T11:15:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.561+0000 7fd936c4eb80 1 -- 172.21.15.27:0/365841367 wait complete. 2024-05-10T11:15:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.561+0000 7fd936c4eb80 1 -- start start 2024-05-10T11:15:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.562+0000 7fd936c4eb80 1 Processor -- start 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.562+0000 7fd936c4eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c322ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.562+0000 7fd936c4eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 0x5592c322c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.562+0000 7fd936c4eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c322c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.562+0000 7fd936c4eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5592c322fbe0 con 0x5592c3223980 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.562+0000 7fd936c4eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5592c30dbb20 con 0x5592c32279d0 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.562+0000 7fd936c4eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5592c322fd60 con 0x5592c3223060 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.562+0000 7fd936c4eb80 10 client.0 _ll_register_callbacks cb 0x5592c311cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.563+0000 7fd929800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c322ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.563+0000 7fd929800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c322ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36034/0 (socket says 172.21.15.27:36034) 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.563+0000 7fd929800700 1 -- 172.21.15.27:0/1444500274 learned_addr learned my addr 172.21.15.27:0/1444500274 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.563+0000 7fd92a200700 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c322c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.563+0000 7fd928e00700 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 0x5592c322c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.563+0000 7fd929800700 1 -- 172.21.15.27:0/1444500274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 msgr2=0x5592c322c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.563+0000 7fd929800700 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c322c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.563+0000 7fd929800700 1 -- 172.21.15.27:0/1444500274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 msgr2=0x5592c322c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.563+0000 7fd929800700 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 0x5592c322c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.563+0000 7fd929800700 1 -- 172.21.15.27:0/1444500274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592c313b420 con 0x5592c3223060 2024-05-10T11:15:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.563+0000 7fd928e00700 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 0x5592c322c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:15:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.564+0000 7fd929800700 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c322ee90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd918009650 tx=0x7fd9180047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93f72d30c195867c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.564+0000 7fd90b400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:15:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.564+0000 7fd90b400700 1 -- 172.21.15.27:0/1444500274 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd918012d00 con 0x5592c3223060 2024-05-10T11:15:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.564+0000 7fd90b400700 1 -- 172.21.15.27:0/1444500274 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fd918018030 con 0x5592c3223060 2024-05-10T11:15:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.564+0000 7fd936c4eb80 1 -- 172.21.15.27:0/1444500274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd910010c90 con 0x5592c3223060 2024-05-10T11:15:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.565+0000 7fd90b400700 1 -- 172.21.15.27:0/1444500274 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd918034050 con 0x5592c3223060 2024-05-10T11:15:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.565+0000 7fd936c4eb80 10 client.25675 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.565+0000 7fd936c4eb80 1 -- 172.21.15.27:0/1444500274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5592c323a520 con 0x5592c3223060 2024-05-10T11:15:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.565+0000 7fd90b400700 1 -- 172.21.15.27:0/1444500274 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd910010c90 con 0x5592c3223060 2024-05-10T11:15:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.565+0000 7fd90b400700 1 -- 172.21.15.27:0/1444500274 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fd918029640 con 0x5592c3223060 2024-05-10T11:15:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.565+0000 7fd936c4eb80 10 client.25675 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.565+0000 7fd936c4eb80 10 client.25675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:15:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.565+0000 7fd936c4eb80 20 client.25675 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.565+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5592c323cfe0 0x5592c323f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.566+0000 7fd936c4eb80 4 client.25675 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:15:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.566+0000 7fd936c4eb80 1 -- 172.21.15.27:0/1444500274 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592c31f00a0 con 0x5592c323cfe0 2024-05-10T11:15:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.566+0000 7fd928e00700 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5592c323cfe0 0x5592c323f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.567+0000 7fd928e00700 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5592c323cfe0 0x5592c323f400 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=551de839687beb07 server_cookie=9970b5560761d5cb in_seq=0 out_seq=0 2024-05-10T11:15:19.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.567+0000 7fd90b400700 10 client.25675 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:19.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd90b400700 1 -- 172.21.15.27:0/1444500274 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5592c31f00a0 con 0x5592c323cfe0 2024-05-10T11:15:19.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd90b400700 10 client.25675 handle_command_reply: tid=0 2024-05-10T11:15:19.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd936c4eb80 1 client.25675 shutdown 2024-05-10T11:15:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd936c4eb80 1 -- 172.21.15.27:0/1444500274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 msgr2=0x5592c322ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c322ee90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd918009650 tx=0x7fd9180047c0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd936c4eb80 1 -- 172.21.15.27:0/1444500274 shutdown_connections 2024-05-10T11:15:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5592c323cfe0 0x5592c323f400 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c322ee90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd90b400700 0 client.25675 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c322c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/1444500274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 0x5592c322c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd936c4eb80 1 -- 172.21.15.27:0/1444500274 >> 172.21.15.27:0/1444500274 conn(0x5592c3151b60 msgr2=0x5592c32182b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.568+0000 7fd936c4eb80 1 -- 172.21.15.27:0/1444500274 shutdown_connections 2024-05-10T11:15:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.569+0000 7fd936c4eb80 1 -- 172.21.15.27:0/1444500274 wait complete. 2024-05-10T11:15:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.569+0000 7fd936c4eb80 20 client.25675 trim_cache size 0 max 16384 2024-05-10T11:15:19.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.594+0000 7fd936c4eb80 1 Processor -- start 2024-05-10T11:15:19.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.594+0000 7fd936c4eb80 1 -- start start 2024-05-10T11:15:19.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.594+0000 7fd936c4eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223060 0x5592c3229af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.594+0000 7fd936c4eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223980 0x5592c3238ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.594+0000 7fd936c4eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c3228dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.594+0000 7fd936c4eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd9241b1630 con 0x5592c3223060 2024-05-10T11:15:19.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.594+0000 7fd936c4eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd9241b31d0 con 0x5592c32279d0 2024-05-10T11:15:19.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.594+0000 7fd936c4eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd9241b1d00 con 0x5592c3223980 2024-05-10T11:15:19.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.595+0000 7fd92a200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c3228dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.595+0000 7fd928e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223980 0x5592c3238ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.595+0000 7fd92a200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c3228dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44576/0 (socket says 172.21.15.27:44576) 2024-05-10T11:15:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.595+0000 7fd929800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223060 0x5592c3229af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.595+0000 7fd92a200700 1 -- 172.21.15.27:0/3510975218 learned_addr learned my addr 172.21.15.27:0/3510975218 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.595+0000 7fd928e00700 1 -- 172.21.15.27:0/3510975218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 msgr2=0x5592c3228dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.595+0000 7fd928e00700 1 --2- 172.21.15.27:0/3510975218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c3228dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.595+0000 7fd928e00700 1 -- 172.21.15.27:0/3510975218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223060 msgr2=0x5592c3229af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.595+0000 7fd928e00700 1 --2- 172.21.15.27:0/3510975218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223060 0x5592c3229af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.595+0000 7fd928e00700 1 -- 172.21.15.27:0/3510975218 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592c323fa80 con 0x5592c3223980 2024-05-10T11:15:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.595+0000 7fd92a200700 1 --2- 172.21.15.27:0/3510975218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c3228dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.596+0000 7fd928e00700 1 --2- 172.21.15.27:0/3510975218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223980 0x5592c3238ca0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x5592c323fcb0 tx=0x7fd910009dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d28332a3fce4bd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:19.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.596+0000 7fd916000700 1 -- 172.21.15.27:0/3510975218 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9100132c0 con 0x5592c3223980 2024-05-10T11:15:19.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.596+0000 7fd916000700 1 -- 172.21.15.27:0/3510975218 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9100044f0 con 0x5592c3223980 2024-05-10T11:15:19.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.596+0000 7fd916000700 1 -- 172.21.15.27:0/3510975218 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9100106b0 con 0x5592c3223980 2024-05-10T11:15:19.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.597+0000 7fd936c4eb80 1 -- 172.21.15.27:0/3510975218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223980 msgr2=0x5592c3238ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:19.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.597+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/3510975218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223980 0x5592c3238ca0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x5592c323fcb0 tx=0x7fd910009dc0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.597+0000 7fd936c4eb80 1 -- 172.21.15.27:0/3510975218 shutdown_connections 2024-05-10T11:15:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.597+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/3510975218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c3228dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.597+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/3510975218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223060 0x5592c3229af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.597+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/3510975218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223980 0x5592c3238ca0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.597+0000 7fd936c4eb80 1 -- 172.21.15.27:0/3510975218 >> 172.21.15.27:0/3510975218 conn(0x5592c3151b60 msgr2=0x5592c32182b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.597+0000 7fd936c4eb80 1 -- 172.21.15.27:0/3510975218 shutdown_connections 2024-05-10T11:15:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.597+0000 7fd936c4eb80 1 -- 172.21.15.27:0/3510975218 wait complete. 2024-05-10T11:15:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.598+0000 7fd936c4eb80 1 -- start start 2024-05-10T11:15:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.598+0000 7fd936c4eb80 1 Processor -- start 2024-05-10T11:15:19.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.598+0000 7fd936c4eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c3244490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.598+0000 7fd936c4eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 0x5592c32449d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd936c4eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c3244f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd936c4eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd9241b2aa0 con 0x5592c3223980 2024-05-10T11:15:19.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd936c4eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd9241b23d0 con 0x5592c32279d0 2024-05-10T11:15:19.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd936c4eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd9241af5d0 con 0x5592c3223060 2024-05-10T11:15:19.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd936c4eb80 10 client.0 _ll_register_callbacks cb 0x5592c311cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:19.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd929800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c3244490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd928e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 0x5592c32449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd92a200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c3244f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd928e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 0x5592c32449d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32980/0 (socket says 172.21.15.27:32980) 2024-05-10T11:15:19.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd929800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c3244490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36048/0 (socket says 172.21.15.27:36048) 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd928e00700 1 -- 172.21.15.27:0/509367044 learned_addr learned my addr 172.21.15.27:0/509367044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd92a200700 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c3244f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44578/0 (socket says 172.21.15.27:44578) 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd92a200700 1 -- 172.21.15.27:0/509367044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 msgr2=0x5592c3244490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd92a200700 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c3244490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd92a200700 1 -- 172.21.15.27:0/509367044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 msgr2=0x5592c32449d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd92a200700 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 0x5592c32449d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd92a200700 1 -- 172.21.15.27:0/509367044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592c32af0d0 con 0x5592c32279d0 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.599+0000 7fd928e00700 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 0x5592c32449d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.600+0000 7fd92a200700 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c3244f10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0e08c0 tx=0x7fd91c0e0d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84d553ba8f84f735 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.600+0000 7fd90b400700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.600+0000 7fd90b400700 1 -- 172.21.15.27:0/509367044 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c0c4060 con 0x5592c32279d0 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.600+0000 7fd936c4eb80 1 -- 172.21.15.27:0/509367044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9100044f0 con 0x5592c32279d0 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.600+0000 7fd90b400700 1 -- 172.21.15.27:0/509367044 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fd91c0aa8f0 con 0x5592c32279d0 2024-05-10T11:15:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.600+0000 7fd90b400700 1 -- 172.21.15.27:0/509367044 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd91c029040 con 0x5592c32279d0 2024-05-10T11:15:19.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.600+0000 7fd936c4eb80 10 client.25688 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:19.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.600+0000 7fd936c4eb80 1 -- 172.21.15.27:0/509367044 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5592c32b0390 con 0x5592c32279d0 2024-05-10T11:15:19.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.601+0000 7fd90b400700 1 -- 172.21.15.27:0/509367044 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9100044f0 con 0x5592c32279d0 2024-05-10T11:15:19.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.601+0000 7fd90b400700 1 -- 172.21.15.27:0/509367044 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fd91c0c4ce0 con 0x5592c32279d0 2024-05-10T11:15:19.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.601+0000 7fd936c4eb80 10 client.25688 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.601+0000 7fd936c4eb80 10 client.25688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.601+0000 7fd936c4eb80 20 client.25688 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.601+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5592c32b0a80 0x5592c32b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.601+0000 7fd936c4eb80 4 client.25688 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.601+0000 7fd936c4eb80 1 -- 172.21.15.27:0/509367044 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5592c3183400 con 0x5592c32b0a80 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.601+0000 7fd929800700 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5592c32b0a80 0x5592c32b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.602+0000 7fd929800700 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5592c32b0a80 0x5592c32b2ea0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55bc15b0077bc74 server_cookie=314d11373016d4c4 in_seq=0 out_seq=0 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.602+0000 7fd90b400700 10 client.25688 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.602+0000 7fd90b400700 1 -- 172.21.15.27:0/509367044 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5592c3183400 con 0x5592c32b0a80 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.602+0000 7fd90b400700 10 client.25688 handle_command_reply: tid=0 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.602+0000 7fd936c4eb80 1 client.25688 shutdown 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.602+0000 7fd936c4eb80 1 -- 172.21.15.27:0/509367044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 msgr2=0x5592c3244f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:19.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.602+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c3244f10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0e08c0 tx=0x7fd91c0e0d10 comp rx=0 tx=0).stop 2024-05-10T11:15:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.602+0000 7fd936c4eb80 1 -- 172.21.15.27:0/509367044 shutdown_connections 2024-05-10T11:15:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.602+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5592c32b0a80 0x5592c32b2ea0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.602+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592c3223060 0x5592c3244490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.602+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5592c32279d0 0x5592c3244f10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.602+0000 7fd936c4eb80 1 --2- 172.21.15.27:0/509367044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5592c3223980 0x5592c32449d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.603+0000 7fd936c4eb80 1 -- 172.21.15.27:0/509367044 >> 172.21.15.27:0/509367044 conn(0x5592c3151b60 msgr2=0x5592c3221e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.603+0000 7fd90b400700 0 client.25688 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.603+0000 7fd936c4eb80 1 -- 172.21.15.27:0/509367044 shutdown_connections 2024-05-10T11:15:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.603+0000 7fd936c4eb80 1 -- 172.21.15.27:0/509367044 wait complete. 2024-05-10T11:15:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.603+0000 7fd936c4eb80 20 client.25688 trim_cache size 0 max 16384 2024-05-10T11:15:19.606 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:15:19.607 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:15:19.607 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:15:19.607 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:15:19.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.605+0000 7fd92c200700 1 -- 172.21.15.27:0/1052790773 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd90c0b29f0 msgr2=0x7fd90c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:19.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.605+0000 7fd92c200700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd90c0b29f0 0x7fd90c0b4eb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd918009680 tx=0x7fd918009260 comp rx=0 tx=0).stop 2024-05-10T11:15:19.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.605+0000 7fd92c200700 1 -- 172.21.15.27:0/1052790773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd924108850 msgr2=0x7fd92419baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:19.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.605+0000 7fd92c200700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd924108850 0x7fd92419baf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd91c00b770 tx=0x7fd91c007980 comp rx=0 tx=0).stop 2024-05-10T11:15:19.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.605+0000 7fd92c200700 1 -- 172.21.15.27:0/1052790773 shutdown_connections 2024-05-10T11:15:19.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.605+0000 7fd92c200700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd90c0b29f0 0x7fd90c0b4eb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.605+0000 7fd92c200700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd924108850 0x7fd92419baf0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.605+0000 7fd92c200700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd924103750 0x7fd9241a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.605+0000 7fd92c200700 1 --2- 172.21.15.27:0/1052790773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd924104100 0x7fd9241a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:19.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.605+0000 7fd92c200700 1 -- 172.21.15.27:0/1052790773 >> 172.21.15.27:0/1052790773 conn(0x7fd92407a600 msgr2=0x7fd9241017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:19.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.605+0000 7fd92c200700 1 -- 172.21.15.27:0/1052790773 shutdown_connections 2024-05-10T11:15:19.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:19.605+0000 7fd92c200700 1 -- 172.21.15.27:0/1052790773 wait complete. 2024-05-10T11:15:19.650 INFO:tasks.workunit.client.0.smithi027.stdout: 1 8046 15.00 MB/sec warmup 4 sec latency 26.298 ms 2024-05-10T11:15:19.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:19 smithi076 ceph-mon[34152]: pgmap v66: 129 pgs: 129 active+clean; 5.2 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 368 KiB/s wr, 57 op/s 2024-05-10T11:15:19.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:19 smithi189 ceph-mon[34842]: pgmap v66: 129 pgs: 129 active+clean; 5.2 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 368 KiB/s wr, 57 op/s 2024-05-10T11:15:20.633 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:15:20.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:15:20.650 INFO:tasks.workunit.client.0.smithi027.stdout: 1 10451 14.72 MB/sec warmup 5 sec latency 169.200 ms 2024-05-10T11:15:21.112 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:21.650 INFO:tasks.workunit.client.0.smithi027.stdout: 1 10451 12.27 MB/sec warmup 6 sec latency 1169.277 ms 2024-05-10T11:15:22.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:22 smithi027 ceph-mon[30054]: pgmap v67: 129 pgs: 129 active+clean; 5.2 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 368 KiB/s wr, 53 op/s 2024-05-10T11:15:22.650 INFO:tasks.workunit.client.0.smithi027.stdout: 1 10451 10.52 MB/sec warmup 7 sec latency 2169.356 ms 2024-05-10T11:15:22.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:22 smithi076 ceph-mon[34152]: pgmap v67: 129 pgs: 129 active+clean; 5.2 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 368 KiB/s wr, 53 op/s 2024-05-10T11:15:22.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:22 smithi189 ceph-mon[34842]: pgmap v67: 129 pgs: 129 active+clean; 5.2 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 368 KiB/s wr, 53 op/s 2024-05-10T11:15:23.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.575+0000 7f71e3e00700 1 -- 172.21.15.27:0/2800065147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71e41055d0 msgr2=0x7f71e4105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:23.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.575+0000 7f71e3e00700 1 --2- 172.21.15.27:0/2800065147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71e41055d0 0x7f71e4105a50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f71d0009990 tx=0x7f71d0009ca0 comp rx=0 tx=0).stop 2024-05-10T11:15:23.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.575+0000 7f71e3e00700 1 -- 172.21.15.27:0/2800065147 shutdown_connections 2024-05-10T11:15:23.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.575+0000 7f71e3e00700 1 --2- 172.21.15.27:0/2800065147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e4105f90 0x7f71e41124a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:23.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.575+0000 7f71e3e00700 1 --2- 172.21.15.27:0/2800065147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e410b660 0x7f71e410ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:23.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.575+0000 7f71e3e00700 1 --2- 172.21.15.27:0/2800065147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71e41055d0 0x7f71e4105a50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:23.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.575+0000 7f71e3e00700 1 -- 172.21.15.27:0/2800065147 >> 172.21.15.27:0/2800065147 conn(0x7f71e41010c0 msgr2=0x7f71e41034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:23.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.575+0000 7f71e3e00700 1 -- 172.21.15.27:0/2800065147 shutdown_connections 2024-05-10T11:15:23.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.575+0000 7f71e3e00700 1 -- 172.21.15.27:0/2800065147 wait complete. 2024-05-10T11:15:23.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.577+0000 7f71e3e00700 1 Processor -- start 2024-05-10T11:15:23.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.577+0000 7f71e3e00700 1 -- start start 2024-05-10T11:15:23.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.578+0000 7f71e3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71e41055d0 0x7f71e41a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:23.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.578+0000 7f71e3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e4105f90 0x7f71e41a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:23.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.578+0000 7f71e3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e410b660 0x7f71e419ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:23.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.578+0000 7f71e3e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f71e4115820 con 0x7f71e4105f90 2024-05-10T11:15:23.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.578+0000 7f71e3e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f71e41156a0 con 0x7f71e410b660 2024-05-10T11:15:23.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.578+0000 7f71e3e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f71e41159a0 con 0x7f71e41055d0 2024-05-10T11:15:23.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.578+0000 7f71e2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71e41055d0 0x7f71e41a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:23.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.578+0000 7f71e2000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e4105f90 0x7f71e41a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:23.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.578+0000 7f71e2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71e41055d0 0x7f71e41a1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36060/0 (socket says 172.21.15.27:36060) 2024-05-10T11:15:23.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.578+0000 7f71e2a00700 1 -- 172.21.15.27:0/21399374 learned_addr learned my addr 172.21.15.27:0/21399374 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:23.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.578+0000 7f71e3400700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e410b660 0x7f71e419ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:23.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.579+0000 7f71e3400700 1 -- 172.21.15.27:0/21399374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71e41055d0 msgr2=0x7f71e41a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:23.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.579+0000 7f71e3400700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71e41055d0 0x7f71e41a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:23.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.579+0000 7f71e3400700 1 -- 172.21.15.27:0/21399374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e4105f90 msgr2=0x7f71e41a1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:23.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.579+0000 7f71e3400700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e4105f90 0x7f71e41a1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:23.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.579+0000 7f71e3400700 1 -- 172.21.15.27:0/21399374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e419c1a0 con 0x7f71e410b660 2024-05-10T11:15:23.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.579+0000 7f71e2a00700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71e41055d0 0x7f71e41a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:23.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.579+0000 7f71e3400700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e410b660 0x7f71e419ba40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00efc0 tx=0x7f71dc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.580+0000 7f71db400700 1 -- 172.21.15.27:0/21399374 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71dc0092e0 con 0x7f71e410b660 2024-05-10T11:15:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.580+0000 7f71e3e00700 1 -- 172.21.15.27:0/21399374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71e419c430 con 0x7f71e410b660 2024-05-10T11:15:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.580+0000 7f71db400700 1 -- 172.21.15.27:0/21399374 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71dc004bb0 con 0x7f71e410b660 2024-05-10T11:15:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.580+0000 7f71db400700 1 -- 172.21.15.27:0/21399374 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71dc004020 con 0x7f71e410b660 2024-05-10T11:15:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.580+0000 7f71e3e00700 1 -- 172.21.15.27:0/21399374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71e41a8750 con 0x7f71e410b660 2024-05-10T11:15:23.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.581+0000 7f71e3e00700 1 -- 172.21.15.27:0/21399374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71e406a190 con 0x7f71e410b660 2024-05-10T11:15:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.583+0000 7f71db400700 1 -- 172.21.15.27:0/21399374 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f71dc01b030 con 0x7f71e410b660 2024-05-10T11:15:23.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.585+0000 7f71db400700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f71c80b29f0 0x7f71c80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:23.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.585+0000 7f71db400700 1 -- 172.21.15.27:0/21399374 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f71dc0e2500 con 0x7f71e410b660 2024-05-10T11:15:23.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.585+0000 7f71e2a00700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f71c80b29f0 0x7f71c80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:23.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.586+0000 7f71e2a00700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f71c80b29f0 0x7f71c80b4eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f71e419cb80 tx=0x7f71d400b4a0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:23.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.589+0000 7f71db400700 1 -- 172.21.15.27:0/21399374 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f71dc022020 con 0x7f71e410b660 2024-05-10T11:15:23.650 INFO:tasks.workunit.client.0.smithi027.stdout: 1 10451 9.20 MB/sec warmup 8 sec latency 3169.487 ms 2024-05-10T11:15:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.882+0000 7f71e3e00700 1 -- 172.21.15.27:0/21399374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71e402e510 con 0x7f71e410b660 2024-05-10T11:15:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.883+0000 7f71db400700 1 -- 172.21.15.27:0/21399374 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11939 (secure 0 0 0) 0x7f71dc010040 con 0x7f71e410b660 2024-05-10T11:15:23.884 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:15:23.884 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:15:16.547777+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24488,24467,24472,24527,24484]},"id":1}]} 2024-05-10T11:15:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.886+0000 7f71e3e00700 1 -- 172.21.15.27:0/21399374 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f71c80b29f0 msgr2=0x7f71c80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.886+0000 7f71e3e00700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f71c80b29f0 0x7f71c80b4eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f71e419cb80 tx=0x7f71d400b4a0 comp rx=0 tx=0).stop 2024-05-10T11:15:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.886+0000 7f71e3e00700 1 -- 172.21.15.27:0/21399374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e410b660 msgr2=0x7f71e419ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.886+0000 7f71e3e00700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e410b660 0x7f71e419ba40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00efc0 tx=0x7f71dc00c3a0 comp rx=0 tx=0).stop 2024-05-10T11:15:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.886+0000 7f71e3e00700 1 -- 172.21.15.27:0/21399374 shutdown_connections 2024-05-10T11:15:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.886+0000 7f71e3e00700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f71c80b29f0 0x7f71c80b4eb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.886+0000 7f71e3e00700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71e41055d0 0x7f71e41a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.886+0000 7f71e3e00700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e410b660 0x7f71e419ba40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.886+0000 7f71e3e00700 1 --2- 172.21.15.27:0/21399374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e4105f90 0x7f71e41a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.887+0000 7f71e3e00700 1 -- 172.21.15.27:0/21399374 >> 172.21.15.27:0/21399374 conn(0x7f71e41010c0 msgr2=0x7f71e4102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.887+0000 7f71e3e00700 1 -- 172.21.15.27:0/21399374 shutdown_connections 2024-05-10T11:15:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:23.887+0000 7f71e3e00700 1 -- 172.21.15.27:0/21399374 wait complete. 2024-05-10T11:15:23.889 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 31 2024-05-10T11:15:24.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:24 smithi027 ceph-mon[30054]: pgmap v68: 129 pgs: 129 active+clean; 5.2 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 368 KiB/s wr, 53 op/s 2024-05-10T11:15:24.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:24 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/21399374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:15:24.650 INFO:tasks.workunit.client.0.smithi027.stdout: 1 10954 8.67 MB/sec warmup 9 sec latency 3706.337 ms 2024-05-10T11:15:24.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:24 smithi076 ceph-mon[34152]: pgmap v68: 129 pgs: 129 active+clean; 5.2 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 368 KiB/s wr, 53 op/s 2024-05-10T11:15:24.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:24 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/21399374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:15:24.685 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:15:24.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:24 smithi189 ceph-mon[34842]: pgmap v68: 129 pgs: 129 active+clean; 5.2 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 368 KiB/s wr, 53 op/s 2024-05-10T11:15:24.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:24 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/21399374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:15:25.215 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:25.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 12490 8.37 MB/sec warmup 10 sec latency 28.094 ms 2024-05-10T11:15:26.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:26 smithi027 ceph-mon[30054]: pgmap v69: 129 pgs: 129 active+clean; 6.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 536 KiB/s wr, 78 op/s 2024-05-10T11:15:26.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 14001 8.43 MB/sec warmup 11 sec latency 463.402 ms 2024-05-10T11:15:26.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:26 smithi076 ceph-mon[34152]: pgmap v69: 129 pgs: 129 active+clean; 6.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 536 KiB/s wr, 78 op/s 2024-05-10T11:15:26.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:26 smithi189 ceph-mon[34842]: pgmap v69: 129 pgs: 129 active+clean; 6.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 536 KiB/s wr, 78 op/s 2024-05-10T11:15:27.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:27 smithi027 ceph-mon[30054]: pgmap v70: 129 pgs: 129 active+clean; 6.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 536 KiB/s wr, 74 op/s 2024-05-10T11:15:27.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 14001 7.73 MB/sec warmup 12 sec latency 1463.478 ms 2024-05-10T11:15:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.666+0000 7f3e82600700 1 -- 172.21.15.27:0/777672918 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e7c105f80 msgr2=0x7f3e7c112490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.666+0000 7f3e82600700 1 --2- 172.21.15.27:0/777672918 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e7c105f80 0x7f3e7c112490 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400b0a0 tx=0x7f3e7400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.666+0000 7f3e82600700 1 -- 172.21.15.27:0/777672918 shutdown_connections 2024-05-10T11:15:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.666+0000 7f3e82600700 1 --2- 172.21.15.27:0/777672918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e7c10b650 0x7f3e7c10ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.666+0000 7f3e82600700 1 --2- 172.21.15.27:0/777672918 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e7c105f80 0x7f3e7c112490 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.666+0000 7f3e82600700 1 --2- 172.21.15.27:0/777672918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c1055c0 0x7f3e7c105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.666+0000 7f3e82600700 1 -- 172.21.15.27:0/777672918 >> 172.21.15.27:0/777672918 conn(0x7f3e7c101090 msgr2=0x7f3e7c1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.666+0000 7f3e82600700 1 -- 172.21.15.27:0/777672918 shutdown_connections 2024-05-10T11:15:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.666+0000 7f3e82600700 1 -- 172.21.15.27:0/777672918 wait complete. 2024-05-10T11:15:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.667+0000 7f3e82600700 1 Processor -- start 2024-05-10T11:15:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.668+0000 7f3e82600700 1 -- start start 2024-05-10T11:15:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.668+0000 7f3e82600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c1055c0 0x7f3e7c1a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.668+0000 7f3e82600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e7c105f80 0x7f3e7c1a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.668+0000 7f3e82600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e7c10b650 0x7f3e7c19ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.668+0000 7f3e82600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c115810 con 0x7f3e7c1055c0 2024-05-10T11:15:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.668+0000 7f3e82600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c115690 con 0x7f3e7c105f80 2024-05-10T11:15:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.668+0000 7f3e82600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c115990 con 0x7f3e7c10b650 2024-05-10T11:15:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.668+0000 7f3e81200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c1055c0 0x7f3e7c1a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.668+0000 7f3e81200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c1055c0 0x7f3e7c1a1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46562/0 (socket says 172.21.15.27:46562) 2024-05-10T11:15:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.668+0000 7f3e81200700 1 -- 172.21.15.27:0/1027580386 learned_addr learned my addr 172.21.15.27:0/1027580386 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.669+0000 7f3e81c00700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e7c10b650 0x7f3e7c19ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.669+0000 7f3e7be00700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e7c105f80 0x7f3e7c1a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.669+0000 7f3e81200700 1 -- 172.21.15.27:0/1027580386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e7c10b650 msgr2=0x7f3e7c19ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.669+0000 7f3e81200700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e7c10b650 0x7f3e7c19ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.669+0000 7f3e81200700 1 -- 172.21.15.27:0/1027580386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e7c105f80 msgr2=0x7f3e7c1a1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.669+0000 7f3e81200700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e7c105f80 0x7f3e7c1a1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.669+0000 7f3e81200700 1 -- 172.21.15.27:0/1027580386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7c19c170 con 0x7f3e7c1055c0 2024-05-10T11:15:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.669+0000 7f3e81200700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c1055c0 0x7f3e7c1a1800 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3e7000e970 tx=0x7f3e7000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.670+0000 7f3e79600700 1 -- 172.21.15.27:0/1027580386 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e70004220 con 0x7f3e7c1055c0 2024-05-10T11:15:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.670+0000 7f3e82600700 1 -- 172.21.15.27:0/1027580386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e7c19c460 con 0x7f3e7c1055c0 2024-05-10T11:15:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.670+0000 7f3e79600700 1 -- 172.21.15.27:0/1027580386 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e700043c0 con 0x7f3e7c1055c0 2024-05-10T11:15:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.670+0000 7f3e79600700 1 -- 172.21.15.27:0/1027580386 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e70003ce0 con 0x7f3e7c1055c0 2024-05-10T11:15:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.670+0000 7f3e82600700 1 -- 172.21.15.27:0/1027580386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e7c1a86d0 con 0x7f3e7c1055c0 2024-05-10T11:15:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.674+0000 7f3e79600700 1 -- 172.21.15.27:0/1027580386 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3e7000c7e0 con 0x7f3e7c1055c0 2024-05-10T11:15:27.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:27 smithi076 ceph-mon[34152]: pgmap v70: 129 pgs: 129 active+clean; 6.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 536 KiB/s wr, 74 op/s 2024-05-10T11:15:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.675+0000 7f3e8e5aab80 1 Processor -- start 2024-05-10T11:15:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.675+0000 7f3e8e5aab80 1 -- start start 2024-05-10T11:15:27.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.675+0000 7f3e8e5aab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b625180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.675+0000 7f3e8e5aab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b6256c0 0x55b81b61f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.675+0000 7f3e8e5aab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b61fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.675+0000 7f3e8e5aab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b81b571a20 con 0x55b81b624da0 2024-05-10T11:15:27.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.675+0000 7f3e8e5aab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b81b50f760 con 0x55b81b6256c0 2024-05-10T11:15:27.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.675+0000 7f3e8e5aab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b81b56d9e0 con 0x55b81b61fa50 2024-05-10T11:15:27.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.676+0000 7f3e79600700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3e640b2a70 0x7f3e640b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.676+0000 7f3e81c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b625180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.676+0000 7f3e81c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b625180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46570/0 (socket says 172.21.15.27:46570) 2024-05-10T11:15:27.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.676+0000 7f3e81c00700 1 -- 172.21.15.27:0/4227492454 learned_addr learned my addr 172.21.15.27:0/4227492454 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:27.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.676+0000 7f3e79600700 1 -- 172.21.15.27:0/1027580386 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f3e700e2be0 con 0x7f3e7c1055c0 2024-05-10T11:15:27.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.676+0000 7f3e7be00700 1 --2- 172.21.15.27:0/4227492454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b61fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.676+0000 7f3e81200700 1 --2- 172.21.15.27:0/4227492454 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b6256c0 0x55b81b61f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.676+0000 7f3e81c00700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3e640b2a70 0x7f3e640b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.676+0000 7f3e81c00700 1 -- 172.21.15.27:0/4227492454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 msgr2=0x55b81b61fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:27.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.677+0000 7f3e81c00700 1 --2- 172.21.15.27:0/4227492454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b61fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.677+0000 7f3e81c00700 1 -- 172.21.15.27:0/4227492454 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b6256c0 msgr2=0x55b81b61f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:27.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.677+0000 7f3e81c00700 1 --2- 172.21.15.27:0/4227492454 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b6256c0 0x55b81b61f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.677+0000 7f3e81c00700 1 -- 172.21.15.27:0/4227492454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b81b4be8d0 con 0x55b81b624da0 2024-05-10T11:15:27.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.677+0000 7f3e7be00700 1 --2- 172.21.15.27:0/4227492454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b61fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:27.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.677+0000 7f3e81c00700 1 --2- 172.21.15.27:0/4227492454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b625180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400d010 tx=0x7f3e7400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7a9df57a47f0a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:27.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.678+0000 7f3e81c00700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3e640b2a70 0x7f3e640b4f30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e74026000 tx=0x7f3e74029000 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:27.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.678+0000 7f3e6e000700 1 -- 172.21.15.27:0/4227492454 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e740076e0 con 0x55b81b624da0 2024-05-10T11:15:27.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.678+0000 7f3e6e000700 1 -- 172.21.15.27:0/4227492454 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e74009490 con 0x55b81b624da0 2024-05-10T11:15:27.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.678+0000 7f3e6e000700 1 -- 172.21.15.27:0/4227492454 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7402ba30 con 0x55b81b624da0 2024-05-10T11:15:27.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.678+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/4227492454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 msgr2=0x55b81b625180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:27.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.678+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/4227492454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b625180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400d010 tx=0x7f3e7400bd40 comp rx=0 tx=0).stop 2024-05-10T11:15:27.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.679+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/4227492454 shutdown_connections 2024-05-10T11:15:27.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.679+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/4227492454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b61fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.679+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/4227492454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b625180 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.679+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/4227492454 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b6256c0 0x55b81b61f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.679+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/4227492454 >> 172.21.15.27:0/4227492454 conn(0x55b81b5976d0 msgr2=0x55b81b619ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:27.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.679+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/4227492454 shutdown_connections 2024-05-10T11:15:27.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.679+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/4227492454 wait complete. 2024-05-10T11:15:27.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.679+0000 7f3e8e5aab80 1 -- start start 2024-05-10T11:15:27.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.680+0000 7f3e8e5aab80 1 Processor -- start 2024-05-10T11:15:27.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.680+0000 7f3e8e5aab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b62b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.680+0000 7f3e8e5aab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b624da0 0x55b81b628860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.680+0000 7f3e8e5aab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b6256c0 0x55b81b628da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.680+0000 7f3e8e5aab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b81b62c0f0 con 0x55b81b6256c0 2024-05-10T11:15:27.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.680+0000 7f3e8e5aab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b81b4d7b20 con 0x55b81b624da0 2024-05-10T11:15:27.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.680+0000 7f3e8e5aab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b81b62c270 con 0x55b81b61fa50 2024-05-10T11:15:27.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.680+0000 7f3e8e5aab80 10 client.0 _ll_register_callbacks cb 0x55b81b518c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:27.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e7be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b624da0 0x55b81b628860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e7be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b624da0 0x55b81b628860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37450/0 (socket says 172.21.15.27:37450) 2024-05-10T11:15:27.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e7be00700 1 -- 172.21.15.27:0/3633663740 learned_addr learned my addr 172.21.15.27:0/3633663740 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:27.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e81200700 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b62b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e81c00700 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b6256c0 0x55b81b628da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e81200700 1 -- 172.21.15.27:0/3633663740 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b624da0 msgr2=0x55b81b628860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:27.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e81200700 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b624da0 0x55b81b628860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e81200700 1 -- 172.21.15.27:0/3633663740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b6256c0 msgr2=0x55b81b628da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:27.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e81200700 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b6256c0 0x55b81b628da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e81200700 1 -- 172.21.15.27:0/3633663740 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b81b537420 con 0x55b81b61fa50 2024-05-10T11:15:27.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e7be00700 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b624da0 0x55b81b628860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:27.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e81c00700 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b6256c0 0x55b81b628da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:27.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e81200700 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b62b450 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3e700dff00 tx=0x7f3e7000cf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa7cf83a3b282ad5 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:27.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.681+0000 7f3e62a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:15:27.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.682+0000 7f3e62a00700 1 -- 172.21.15.27:0/3633663740 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e70004570 con 0x55b81b61fa50 2024-05-10T11:15:27.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.682+0000 7f3e62a00700 1 -- 172.21.15.27:0/3633663740 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f3e700a9f00 con 0x55b81b61fa50 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.682+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3633663740 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e74009490 con 0x55b81b61fa50 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.682+0000 7f3e62a00700 1 -- 172.21.15.27:0/3633663740 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3e700b2230 con 0x55b81b61fa50 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.682+0000 7f3e8e5aab80 10 client.25699 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.682+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3633663740 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b81b636a90 con 0x55b81b61fa50 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.682+0000 7f3e62a00700 1 -- 172.21.15.27:0/3633663740 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e74009490 con 0x55b81b61fa50 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.683+0000 7f3e62a00700 1 -- 172.21.15.27:0/3633663740 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f3e700b2460 con 0x55b81b61fa50 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.683+0000 7f3e8e5aab80 10 client.25699 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.683+0000 7f3e8e5aab80 10 client.25699 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.683+0000 7f3e8e5aab80 20 client.25699 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.683+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55b81b6390d0 0x55b81b63b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.683+0000 7f3e8e5aab80 4 client.25699 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.683+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3633663740 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b81b5ec140 con 0x55b81b6390d0 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.683+0000 7f3e7be00700 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55b81b6390d0 0x55b81b63b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.684+0000 7f3e7be00700 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55b81b6390d0 0x55b81b63b4c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=25baff1b189938ce server_cookie=739b7f437d1412d7 in_seq=0 out_seq=0 2024-05-10T11:15:27.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.684+0000 7f3e62a00700 10 client.25699 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:15:27.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.685+0000 7f3e62a00700 1 -- 172.21.15.27:0/3633663740 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b81b5ec140 con 0x55b81b6390d0 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.685+0000 7f3e62a00700 10 client.25699 handle_command_reply: tid=0 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.685+0000 7f3e8e5aab80 1 client.25699 shutdown 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.685+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3633663740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 msgr2=0x55b81b62b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.685+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b62b450 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3e700dff00 tx=0x7f3e7000cf50 comp rx=0 tx=0).stop 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.685+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3633663740 shutdown_connections 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.685+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55b81b6390d0 0x55b81b63b4c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.685+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b62b450 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.685+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b6256c0 0x55b81b628da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.685+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/3633663740 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b624da0 0x55b81b628860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.685+0000 7f3e62a00700 0 client.25699 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.685+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3633663740 >> 172.21.15.27:0/3633663740 conn(0x55b81b5976d0 msgr2=0x55b81b614750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.686+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3633663740 shutdown_connections 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.686+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3633663740 wait complete. 2024-05-10T11:15:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.686+0000 7f3e8e5aab80 20 client.25699 trim_cache size 0 max 16384 2024-05-10T11:15:27.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.702+0000 7f3e8e5aab80 1 Processor -- start 2024-05-10T11:15:27.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.702+0000 7f3e8e5aab80 1 -- start start 2024-05-10T11:15:27.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.703+0000 7f3e8e5aab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b61fa50 0x55b81b628dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.703+0000 7f3e8e5aab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b635180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.703+0000 7f3e8e5aab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b6256c0 0x55b81b614860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.703+0000 7f3e8e5aab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c1b1540 con 0x55b81b624da0 2024-05-10T11:15:27.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.703+0000 7f3e8e5aab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c1b30e0 con 0x55b81b61fa50 2024-05-10T11:15:27.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.703+0000 7f3e8e5aab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c1b1c10 con 0x55b81b6256c0 2024-05-10T11:15:27.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.703+0000 7f3e7be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b635180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.703+0000 7f3e7be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b635180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46584/0 (socket says 172.21.15.27:46584) 2024-05-10T11:15:27.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.703+0000 7f3e7be00700 1 -- 172.21.15.27:0/3076560098 learned_addr learned my addr 172.21.15.27:0/3076560098 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:27.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.703+0000 7f3e81200700 1 --2- 172.21.15.27:0/3076560098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b61fa50 0x55b81b628dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.703+0000 7f3e81c00700 1 --2- 172.21.15.27:0/3076560098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b6256c0 0x55b81b614860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.704+0000 7f3e81200700 1 -- 172.21.15.27:0/3076560098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b6256c0 msgr2=0x55b81b614860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.704+0000 7f3e81200700 1 --2- 172.21.15.27:0/3076560098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b6256c0 0x55b81b614860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.704+0000 7f3e81200700 1 -- 172.21.15.27:0/3076560098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 msgr2=0x55b81b635180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.704+0000 7f3e81200700 1 --2- 172.21.15.27:0/3076560098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b635180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.704+0000 7f3e81200700 1 -- 172.21.15.27:0/3076560098 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b81b635d20 con 0x55b81b61fa50 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.704+0000 7f3e7be00700 1 --2- 172.21.15.27:0/3076560098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b635180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.704+0000 7f3e81200700 1 --2- 172.21.15.27:0/3076560098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b61fa50 0x55b81b628dd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3e700dff00 tx=0x7f3e70009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23eb8330e63665a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.704+0000 7f3e6d600700 1 -- 172.21.15.27:0/3076560098 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e700ca040 con 0x55b81b61fa50 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.704+0000 7f3e6d600700 1 -- 172.21.15.27:0/3076560098 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e700c4170 con 0x55b81b61fa50 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.704+0000 7f3e6d600700 1 -- 172.21.15.27:0/3076560098 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e700a95a0 con 0x55b81b61fa50 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.705+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3076560098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b61fa50 msgr2=0x55b81b628dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.705+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/3076560098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b61fa50 0x55b81b628dd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3e700dff00 tx=0x7f3e70009840 comp rx=0 tx=0).stop 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.705+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3076560098 shutdown_connections 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.705+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/3076560098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b6256c0 0x55b81b614860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.705+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/3076560098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b61fa50 0x55b81b628dd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.705+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/3076560098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b635180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.705+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3076560098 >> 172.21.15.27:0/3076560098 conn(0x55b81b5976d0 msgr2=0x55b81b6240a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:27.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.705+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3076560098 shutdown_connections 2024-05-10T11:15:27.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.705+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/3076560098 wait complete. 2024-05-10T11:15:27.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.705+0000 7f3e8e5aab80 1 -- start start 2024-05-10T11:15:27.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.706+0000 7f3e8e5aab80 1 Processor -- start 2024-05-10T11:15:27.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.706+0000 7f3e8e5aab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b6316e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.706+0000 7f3e8e5aab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b631c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.706+0000 7f3e8e5aab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b6256c0 0x55b81b63ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.706+0000 7f3e8e5aab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c1b29b0 con 0x55b81b624da0 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.706+0000 7f3e8e5aab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c1b22e0 con 0x55b81b6256c0 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.706+0000 7f3e8e5aab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c1af4e0 con 0x55b81b61fa50 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.706+0000 7f3e8e5aab80 10 client.0 _ll_register_callbacks cb 0x55b81b518c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.707+0000 7f3e7be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b631c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.707+0000 7f3e81c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b6256c0 0x55b81b63ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.707+0000 7f3e81200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b6316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.707+0000 7f3e81c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b6256c0 0x55b81b63ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37476/0 (socket says 172.21.15.27:37476) 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.707+0000 7f3e81c00700 1 -- 172.21.15.27:0/1792841829 learned_addr learned my addr 172.21.15.27:0/1792841829 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.707+0000 7f3e81c00700 1 -- 172.21.15.27:0/1792841829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 msgr2=0x55b81b6316e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.707+0000 7f3e81c00700 1 --2- 172.21.15.27:0/1792841829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b6316e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.707+0000 7f3e81c00700 1 -- 172.21.15.27:0/1792841829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 msgr2=0x55b81b631c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.707+0000 7f3e81c00700 1 --2- 172.21.15.27:0/1792841829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b631c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.707+0000 7f3e81c00700 1 -- 172.21.15.27:0/1792841829 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b81b640800 con 0x55b81b6256c0 2024-05-10T11:15:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.708+0000 7f3e81c00700 1 --2- 172.21.15.27:0/1792841829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b6256c0 0x55b81b63ffc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400d010 tx=0x7f3e74009be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a15e58edecc9efdd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.708+0000 7f3e62a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.708+0000 7f3e62a00700 1 -- 172.21.15.27:0/1792841829 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74009490 con 0x55b81b6256c0 2024-05-10T11:15:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.708+0000 7f3e62a00700 1 -- 172.21.15.27:0/1792841829 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f3e740052c0 con 0x55b81b6256c0 2024-05-10T11:15:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.708+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/1792841829 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e700c4170 con 0x55b81b6256c0 2024-05-10T11:15:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.708+0000 7f3e62a00700 1 -- 172.21.15.27:0/1792841829 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3e740054c0 con 0x55b81b6256c0 2024-05-10T11:15:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.708+0000 7f3e8e5aab80 10 client.25712 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.708+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/1792841829 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b81b6a35a0 con 0x55b81b6256c0 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.709+0000 7f3e62a00700 1 -- 172.21.15.27:0/1792841829 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e700c4170 con 0x55b81b6256c0 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.709+0000 7f3e62a00700 1 -- 172.21.15.27:0/1792841829 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f3e74033040 con 0x55b81b6256c0 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.709+0000 7f3e8e5aab80 10 client.25712 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.709+0000 7f3e8e5aab80 10 client.25712 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.709+0000 7f3e8e5aab80 20 client.25712 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.709+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/1792841829 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55b81b6a3850 0x55b81b6a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.709+0000 7f3e8e5aab80 4 client.25712 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.709+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/1792841829 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b81b57f400 con 0x55b81b6a3850 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.709+0000 7f3e81200700 1 --2- 172.21.15.27:0/1792841829 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55b81b6a3850 0x55b81b6a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.710+0000 7f3e81200700 1 --2- 172.21.15.27:0/1792841829 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55b81b6a3850 0x55b81b6a5c40 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=abac36cbbfd4b69b server_cookie=8ab66e3373228f37 in_seq=0 out_seq=0 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.710+0000 7f3e62a00700 10 client.25712 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.710+0000 7f3e62a00700 1 -- 172.21.15.27:0/1792841829 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b81b57f400 con 0x55b81b6a3850 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.710+0000 7f3e62a00700 10 client.25712 handle_command_reply: tid=0 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.710+0000 7f3e8e5aab80 1 client.25712 shutdown 2024-05-10T11:15:27.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.710+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/1792841829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b6256c0 msgr2=0x55b81b63ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:27.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.710+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/1792841829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b6256c0 0x55b81b63ffc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400d010 tx=0x7f3e74009be0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.711+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/1792841829 shutdown_connections 2024-05-10T11:15:27.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.711+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/1792841829 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55b81b6a3850 0x55b81b6a5c40 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.711+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/1792841829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b81b61fa50 0x55b81b6316e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.711+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/1792841829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b81b6256c0 0x55b81b63ffc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.711+0000 7f3e8e5aab80 1 --2- 172.21.15.27:0/1792841829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b81b624da0 0x55b81b631c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.711+0000 7f3e62a00700 0 client.25712 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:15:27.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.711+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/1792841829 >> 172.21.15.27:0/1792841829 conn(0x55b81b5976d0 msgr2=0x55b81b634a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:27.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.711+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/1792841829 shutdown_connections 2024-05-10T11:15:27.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.711+0000 7f3e8e5aab80 1 -- 172.21.15.27:0/1792841829 wait complete. 2024-05-10T11:15:27.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.711+0000 7f3e8e5aab80 20 client.25712 trim_cache size 0 max 16384 2024-05-10T11:15:27.713 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:15:27.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.712+0000 7f3e82600700 1 -- 172.21.15.27:0/1027580386 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3e640b2a70 msgr2=0x7f3e640b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:27.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.712+0000 7f3e82600700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3e640b2a70 0x7f3e640b4f30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e74026000 tx=0x7f3e74029000 comp rx=0 tx=0).stop 2024-05-10T11:15:27.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.712+0000 7f3e82600700 1 -- 172.21.15.27:0/1027580386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c1055c0 msgr2=0x7f3e7c1a1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:27.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.712+0000 7f3e82600700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c1055c0 0x7f3e7c1a1800 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3e7000e970 tx=0x7f3e7000ec80 comp rx=0 tx=0).stop 2024-05-10T11:15:27.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.712+0000 7f3e82600700 1 -- 172.21.15.27:0/1027580386 shutdown_connections 2024-05-10T11:15:27.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.712+0000 7f3e82600700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3e640b2a70 0x7f3e640b4f30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.712+0000 7f3e82600700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e7c10b650 0x7f3e7c19ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.712+0000 7f3e82600700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c1055c0 0x7f3e7c1a1800 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.712+0000 7f3e82600700 1 --2- 172.21.15.27:0/1027580386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e7c105f80 0x7f3e7c1a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:27.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.712+0000 7f3e82600700 1 -- 172.21.15.27:0/1027580386 >> 172.21.15.27:0/1027580386 conn(0x7f3e7c101090 msgr2=0x7f3e7c1027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:27.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.712+0000 7f3e82600700 1 -- 172.21.15.27:0/1027580386 shutdown_connections 2024-05-10T11:15:27.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:27.712+0000 7f3e82600700 1 -- 172.21.15.27:0/1027580386 wait complete. 2024-05-10T11:15:27.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:27 smithi189 ceph-mon[34842]: pgmap v70: 129 pgs: 129 active+clean; 6.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 536 KiB/s wr, 74 op/s 2024-05-10T11:15:28.620 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:15:28.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 14001 7.13 MB/sec warmup 13 sec latency 2463.568 ms 2024-05-10T11:15:29.087 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:29.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 14540 6.96 MB/sec warmup 14 sec latency 2991.427 ms 2024-05-10T11:15:30.616 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:30 smithi027 ceph-mon[30054]: pgmap v71: 129 pgs: 129 active+clean; 8.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 718 KiB/s wr, 106 op/s 2024-05-10T11:15:30.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 15854 6.85 MB/sec warmup 15 sec latency 22.055 ms 2024-05-10T11:15:30.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:30 smithi076 ceph-mon[34152]: pgmap v71: 129 pgs: 129 active+clean; 8.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 718 KiB/s wr, 106 op/s 2024-05-10T11:15:30.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:30 smithi189 ceph-mon[34842]: pgmap v71: 129 pgs: 129 active+clean; 8.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 718 KiB/s wr, 106 op/s 2024-05-10T11:15:31.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.513+0000 7f615d400700 1 -- 172.21.15.27:0/1865972402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6158101f30 msgr2=0x7f6158102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:31.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.513+0000 7f615d400700 1 --2- 172.21.15.27:0/1865972402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6158101f30 0x7f6158102310 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6148009990 tx=0x7f6148009ca0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.513+0000 7f615d400700 1 -- 172.21.15.27:0/1865972402 shutdown_connections 2024-05-10T11:15:31.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.513+0000 7f615d400700 1 --2- 172.21.15.27:0/1865972402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6158101f30 0x7f6158102310 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.513+0000 7f615d400700 1 --2- 172.21.15.27:0/1865972402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61581087a0 0x7f6158110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.513+0000 7f615d400700 1 --2- 172.21.15.27:0/1865972402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61581028e0 0x7f61581080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.513+0000 7f615d400700 1 -- 172.21.15.27:0/1865972402 >> 172.21.15.27:0/1865972402 conn(0x7f6158077650 msgr2=0x7f6158077a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:31.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.513+0000 7f615d400700 1 -- 172.21.15.27:0/1865972402 shutdown_connections 2024-05-10T11:15:31.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.513+0000 7f615d400700 1 -- 172.21.15.27:0/1865972402 wait complete. 2024-05-10T11:15:31.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.515+0000 7f615d400700 1 Processor -- start 2024-05-10T11:15:31.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.515+0000 7f615d400700 1 -- start start 2024-05-10T11:15:31.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f615d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6158101f30 0x7f615819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f615d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61581028e0 0x7f615819fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f615d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61581087a0 0x7f6158199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f615d400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6158113670 con 0x7f61581028e0 2024-05-10T11:15:31.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f615d400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f61581134f0 con 0x7f6158101f30 2024-05-10T11:15:31.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f615d400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61581137f0 con 0x7f61581087a0 2024-05-10T11:15:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f6157e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6158101f30 0x7f615819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f615ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61581087a0 0x7f6158199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f6157e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6158101f30 0x7f615819f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37508/0 (socket says 172.21.15.27:37508) 2024-05-10T11:15:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f6157e00700 1 -- 172.21.15.27:0/3621022999 learned_addr learned my addr 172.21.15.27:0/3621022999 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f614f400700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61581028e0 0x7f615819fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f6157e00700 1 -- 172.21.15.27:0/3621022999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61581087a0 msgr2=0x7f6158199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f6157e00700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61581087a0 0x7f6158199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f6157e00700 1 -- 172.21.15.27:0/3621022999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61581028e0 msgr2=0x7f615819fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f6157e00700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61581028e0 0x7f615819fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.516+0000 7f6157e00700 1 -- 172.21.15.27:0/3621022999 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615819a0f0 con 0x7f6158101f30 2024-05-10T11:15:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.517+0000 7f6157e00700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6158101f30 0x7f615819f7d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6148005910 tx=0x7f614800b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:31.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.517+0000 7f6155600700 1 -- 172.21.15.27:0/3621022999 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6148027070 con 0x7f6158101f30 2024-05-10T11:15:31.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.517+0000 7f6155600700 1 -- 172.21.15.27:0/3621022999 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6148022470 con 0x7f6158101f30 2024-05-10T11:15:31.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.517+0000 7f615d400700 1 -- 172.21.15.27:0/3621022999 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615819a380 con 0x7f6158101f30 2024-05-10T11:15:31.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.517+0000 7f615d400700 1 -- 172.21.15.27:0/3621022999 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6158102de0 con 0x7f6158101f30 2024-05-10T11:15:31.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.518+0000 7f6155600700 1 -- 172.21.15.27:0/3621022999 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6148017650 con 0x7f6158101f30 2024-05-10T11:15:31.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.523+0000 7f6155600700 1 -- 172.21.15.27:0/3621022999 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f614802a070 con 0x7f6158101f30 2024-05-10T11:15:31.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.525+0000 7f616942ab80 1 Processor -- start 2024-05-10T11:15:31.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.525+0000 7f616942ab80 1 -- start start 2024-05-10T11:15:31.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.525+0000 7f616942ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x55868006fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.525+0000 7f616942ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558680070300 0x558680069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.525+0000 7f616942ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868006aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.525+0000 7f616942ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55867ffbca20 con 0x55868006f9e0 2024-05-10T11:15:31.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.525+0000 7f616942ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55867ff5a760 con 0x55868006a600 2024-05-10T11:15:31.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.525+0000 7f616942ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55867ffb89e0 con 0x558680070300 2024-05-10T11:15:31.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.526+0000 7f615ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x55868006fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.526+0000 7f615ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x55868006fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46624/0 (socket says 172.21.15.27:46624) 2024-05-10T11:15:31.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.526+0000 7f615ca00700 1 -- 172.21.15.27:0/1536419321 learned_addr learned my addr 172.21.15.27:0/1536419321 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:31.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.526+0000 7f614f400700 1 --2- 172.21.15.27:0/1536419321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868006aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.526+0000 7f6157e00700 1 --2- 172.21.15.27:0/1536419321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558680070300 0x558680069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.526+0000 7f615ca00700 1 -- 172.21.15.27:0/1536419321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558680070300 msgr2=0x558680069fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.526+0000 7f615ca00700 1 --2- 172.21.15.27:0/1536419321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558680070300 0x558680069fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.526+0000 7f615ca00700 1 -- 172.21.15.27:0/1536419321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 msgr2=0x55868006aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.526+0000 7f615ca00700 1 --2- 172.21.15.27:0/1536419321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868006aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.526+0000 7f615ca00700 1 -- 172.21.15.27:0/1536419321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55867ff098d0 con 0x55868006f9e0 2024-05-10T11:15:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.526+0000 7f614f400700 1 --2- 172.21.15.27:0/1536419321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868006aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.526+0000 7f6155600700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f613c0b2a70 0x7f613c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.527+0000 7f6155600700 1 -- 172.21.15.27:0/3621022999 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f61480e36c0 con 0x7f6158101f30 2024-05-10T11:15:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.527+0000 7f6157e00700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f613c0b2a70 0x7f613c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.527+0000 7f615ca00700 1 --2- 172.21.15.27:0/1536419321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x55868006fdc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f61500108a0 tx=0x7f615000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8a7d2632144834 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.527+0000 7f614d600700 1 -- 172.21.15.27:0/1536419321 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150003c00 con 0x55868006f9e0 2024-05-10T11:15:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.528+0000 7f614d600700 1 -- 172.21.15.27:0/1536419321 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6150003da0 con 0x55868006f9e0 2024-05-10T11:15:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.528+0000 7f614d600700 1 -- 172.21.15.27:0/1536419321 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150007830 con 0x55868006f9e0 2024-05-10T11:15:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.528+0000 7f616942ab80 1 -- 172.21.15.27:0/1536419321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 msgr2=0x55868006fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.528+0000 7f616942ab80 1 --2- 172.21.15.27:0/1536419321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x55868006fdc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f61500108a0 tx=0x7f615000b040 comp rx=0 tx=0).stop 2024-05-10T11:15:31.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.528+0000 7f6157e00700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f613c0b2a70 0x7f613c0b4f30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f61480a7870 tx=0x7f6148009310 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:31.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.529+0000 7f616942ab80 1 -- 172.21.15.27:0/1536419321 shutdown_connections 2024-05-10T11:15:31.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.529+0000 7f616942ab80 1 --2- 172.21.15.27:0/1536419321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868006aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.529+0000 7f616942ab80 1 --2- 172.21.15.27:0/1536419321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x55868006fdc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.529+0000 7f616942ab80 1 --2- 172.21.15.27:0/1536419321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558680070300 0x558680069fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.529+0000 7f616942ab80 1 -- 172.21.15.27:0/1536419321 >> 172.21.15.27:0/1536419321 conn(0x55867ff98b60 msgr2=0x5586800649b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:31.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.529+0000 7f616942ab80 1 -- 172.21.15.27:0/1536419321 shutdown_connections 2024-05-10T11:15:31.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.529+0000 7f616942ab80 1 -- 172.21.15.27:0/1536419321 wait complete. 2024-05-10T11:15:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.530+0000 7f616942ab80 1 -- start start 2024-05-10T11:15:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.530+0000 7f616942ab80 1 Processor -- start 2024-05-10T11:15:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f616942ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006a600 0x558680075eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f616942ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x558680073250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f616942ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558680070300 0x558680073790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f616942ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558680076d50 con 0x55868006f9e0 2024-05-10T11:15:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f616942ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55867ff22b20 con 0x558680070300 2024-05-10T11:15:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f616942ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558680076ed0 con 0x55868006a600 2024-05-10T11:15:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f616942ab80 10 client.0 _ll_register_callbacks cb 0x55867ff63c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f615ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006a600 0x558680075eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f615ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006a600 0x558680075eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37028/0 (socket says 172.21.15.27:37028) 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f615ca00700 1 -- 172.21.15.27:0/3829984670 learned_addr learned my addr 172.21.15.27:0/3829984670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f615ca00700 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558680070300 0x558680073790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f614f400700 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x558680073250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f615ca00700 1 -- 172.21.15.27:0/3829984670 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558680070300 msgr2=0x558680073790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f615ca00700 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558680070300 0x558680073790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f615ca00700 1 -- 172.21.15.27:0/3829984670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 msgr2=0x558680073250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.531+0000 7f615ca00700 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x558680073250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.532+0000 7f615ca00700 1 -- 172.21.15.27:0/3829984670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55867ff82420 con 0x55868006a600 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.532+0000 7f614f400700 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x558680073250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.532+0000 7f615ca00700 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006a600 0x558680075eb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f61500060b0 tx=0x7f6150006040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbcf39185420d86b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.532+0000 7f6146000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:15:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.532+0000 7f6146000700 1 -- 172.21.15.27:0/3829984670 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150004360 con 0x55868006a600 2024-05-10T11:15:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.532+0000 7f6146000700 1 -- 172.21.15.27:0/3829984670 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f6150004500 con 0x55868006a600 2024-05-10T11:15:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.532+0000 7f616942ab80 1 -- 172.21.15.27:0/3829984670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6150003da0 con 0x55868006a600 2024-05-10T11:15:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.532+0000 7f6146000700 1 -- 172.21.15.27:0/3829984670 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6150007500 con 0x55868006a600 2024-05-10T11:15:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.532+0000 7f616942ab80 10 client.25717 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.532+0000 7f616942ab80 1 -- 172.21.15.27:0/3829984670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558680081490 con 0x55868006a600 2024-05-10T11:15:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.533+0000 7f6146000700 1 -- 172.21.15.27:0/3829984670 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6150003da0 con 0x55868006a600 2024-05-10T11:15:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.533+0000 7f6146000700 1 -- 172.21.15.27:0/3829984670 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f6150032030 con 0x55868006a600 2024-05-10T11:15:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.533+0000 7f616942ab80 10 client.25717 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.533+0000 7f616942ab80 10 client.25717 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:15:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.533+0000 7f616942ab80 20 client.25717 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.533+0000 7f616942ab80 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x558680083a70 0x558680085e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.533+0000 7f616942ab80 4 client.25717 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:15:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.533+0000 7f616942ab80 1 -- 172.21.15.27:0/3829984670 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586800370a0 con 0x558680083a70 2024-05-10T11:15:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.533+0000 7f614f400700 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x558680083a70 0x558680085e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.534+0000 7f614f400700 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x558680083a70 0x558680085e60 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=52192b0b1c4e6cfd server_cookie=2104186223a9d77e in_seq=0 out_seq=0 2024-05-10T11:15:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.534+0000 7f6146000700 10 client.25717 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:15:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.535+0000 7f6146000700 1 -- 172.21.15.27:0/3829984670 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5586800370a0 con 0x558680083a70 2024-05-10T11:15:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.535+0000 7f6146000700 10 client.25717 handle_command_reply: tid=0 2024-05-10T11:15:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.535+0000 7f616942ab80 1 client.25717 shutdown 2024-05-10T11:15:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.535+0000 7f616942ab80 1 -- 172.21.15.27:0/3829984670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006a600 msgr2=0x558680075eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.535+0000 7f616942ab80 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006a600 0x558680075eb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f61500060b0 tx=0x7f6150006040 comp rx=0 tx=0).stop 2024-05-10T11:15:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.535+0000 7f616942ab80 1 -- 172.21.15.27:0/3829984670 shutdown_connections 2024-05-10T11:15:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.535+0000 7f616942ab80 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x558680083a70 0x558680085e60 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.535+0000 7f616942ab80 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006a600 0x558680075eb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.535+0000 7f616942ab80 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558680070300 0x558680073790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.535+0000 7f616942ab80 1 --2- 172.21.15.27:0/3829984670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x558680073250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.536+0000 7f616942ab80 1 -- 172.21.15.27:0/3829984670 >> 172.21.15.27:0/3829984670 conn(0x55867ff98b60 msgr2=0x55868005f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.536+0000 7f6146000700 0 client.25717 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:15:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.536+0000 7f616942ab80 1 -- 172.21.15.27:0/3829984670 shutdown_connections 2024-05-10T11:15:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.536+0000 7f616942ab80 1 -- 172.21.15.27:0/3829984670 wait complete. 2024-05-10T11:15:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.536+0000 7f616942ab80 20 client.25717 trim_cache size 0 max 16384 2024-05-10T11:15:31.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.554+0000 7f616942ab80 1 Processor -- start 2024-05-10T11:15:31.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.555+0000 7f616942ab80 1 -- start start 2024-05-10T11:15:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.555+0000 7f616942ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868008b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.555+0000 7f616942ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x55868008ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.555+0000 7f616942ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558680070300 0x55868007c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.555+0000 7f616942ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f61581ad200 con 0x55868006f9e0 2024-05-10T11:15:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.555+0000 7f616942ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f61581aeda0 con 0x55868006a600 2024-05-10T11:15:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.555+0000 7f616942ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61581ad8d0 con 0x558680070300 2024-05-10T11:15:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.555+0000 7f615ca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868008b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.555+0000 7f614f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x55868008ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.555+0000 7f615ca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868008b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37520/0 (socket says 172.21.15.27:37520) 2024-05-10T11:15:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.555+0000 7f615ca00700 1 -- 172.21.15.27:0/2669983986 learned_addr learned my addr 172.21.15.27:0/2669983986 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.556+0000 7f615ca00700 1 --2- 172.21.15.27:0/2669983986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558680070300 0x55868007c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.556+0000 7f615ca00700 1 -- 172.21.15.27:0/2669983986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558680070300 msgr2=0x55868007c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.556+0000 7f615ca00700 1 --2- 172.21.15.27:0/2669983986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558680070300 0x55868007c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.556+0000 7f615ca00700 1 -- 172.21.15.27:0/2669983986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 msgr2=0x55868008ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.556+0000 7f615ca00700 1 --2- 172.21.15.27:0/2669983986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x55868008ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.556+0000 7f615ca00700 1 -- 172.21.15.27:0/2669983986 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558680075860 con 0x55868006a600 2024-05-10T11:15:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.556+0000 7f615ca00700 1 --2- 172.21.15.27:0/2669983986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868008b550 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6150009550 tx=0x7f6150009440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b59b91311a4f8ab server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.556+0000 7f614cc00700 1 -- 172.21.15.27:0/2669983986 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615001ba90 con 0x55868006a600 2024-05-10T11:15:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.556+0000 7f614cc00700 1 -- 172.21.15.27:0/2669983986 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6150004610 con 0x55868006a600 2024-05-10T11:15:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.557+0000 7f614cc00700 1 -- 172.21.15.27:0/2669983986 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150007a40 con 0x55868006a600 2024-05-10T11:15:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.558+0000 7f616942ab80 1 -- 172.21.15.27:0/2669983986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 msgr2=0x55868008b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.558+0000 7f616942ab80 1 --2- 172.21.15.27:0/2669983986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868008b550 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6150009550 tx=0x7f6150009440 comp rx=0 tx=0).stop 2024-05-10T11:15:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.558+0000 7f616942ab80 1 -- 172.21.15.27:0/2669983986 shutdown_connections 2024-05-10T11:15:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.558+0000 7f616942ab80 1 --2- 172.21.15.27:0/2669983986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558680070300 0x55868007c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.558+0000 7f616942ab80 1 --2- 172.21.15.27:0/2669983986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868008b550 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.558+0000 7f616942ab80 1 --2- 172.21.15.27:0/2669983986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55868006f9e0 0x55868008ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.558+0000 7f616942ab80 1 -- 172.21.15.27:0/2669983986 >> 172.21.15.27:0/2669983986 conn(0x55867ff98b60 msgr2=0x558680073210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.558+0000 7f616942ab80 1 -- 172.21.15.27:0/2669983986 shutdown_connections 2024-05-10T11:15:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.558+0000 7f616942ab80 1 -- 172.21.15.27:0/2669983986 wait complete. 2024-05-10T11:15:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.558+0000 7f616942ab80 1 -- start start 2024-05-10T11:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.559+0000 7f616942ab80 1 Processor -- start 2024-05-10T11:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.559+0000 7f616942ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868007c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.559+0000 7f616942ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006f9e0 0x55868007c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.559+0000 7f616942ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558680070300 0x55868008a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.559+0000 7f616942ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f61581ae670 con 0x558680070300 2024-05-10T11:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.559+0000 7f616942ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f61581adfa0 con 0x55868006a600 2024-05-10T11:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.559+0000 7f616942ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61581ab1a0 con 0x55868006f9e0 2024-05-10T11:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.559+0000 7f616942ab80 10 client.0 _ll_register_callbacks cb 0x55867ff63c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.560+0000 7f614f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006f9e0 0x55868007c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.560+0000 7f614f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006f9e0 0x55868007c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37038/0 (socket says 172.21.15.27:37038) 2024-05-10T11:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.560+0000 7f614f400700 1 -- 172.21.15.27:0/1520143185 learned_addr learned my addr 172.21.15.27:0/1520143185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.560+0000 7f615ca00700 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868007c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.560+0000 7f615ca00700 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558680070300 0x55868008a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.560+0000 7f615ca00700 1 -- 172.21.15.27:0/1520143185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006f9e0 msgr2=0x55868007c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.560+0000 7f615ca00700 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006f9e0 0x55868007c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.560+0000 7f615ca00700 1 -- 172.21.15.27:0/1520143185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558680070300 msgr2=0x55868008a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.560+0000 7f615ca00700 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558680070300 0x55868008a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.560+0000 7f615ca00700 1 -- 172.21.15.27:0/1520143185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55868008af80 con 0x55868006a600 2024-05-10T11:15:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.561+0000 7f615ca00700 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868007c3a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f615000c050 tx=0x7f6150003c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55a496e3facba85a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.561+0000 7f6146000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.561+0000 7f6146000700 1 -- 172.21.15.27:0/1520143185 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615001b890 con 0x55868006a600 2024-05-10T11:15:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.561+0000 7f6146000700 1 -- 172.21.15.27:0/1520143185 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f615001ba30 con 0x55868006a600 2024-05-10T11:15:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.562+0000 7f616942ab80 1 -- 172.21.15.27:0/1520143185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6150004610 con 0x55868006a600 2024-05-10T11:15:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.562+0000 7f6146000700 1 -- 172.21.15.27:0/1520143185 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6150004160 con 0x55868006a600 2024-05-10T11:15:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.562+0000 7f616942ab80 10 client.25730 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.562+0000 7f616942ab80 1 -- 172.21.15.27:0/1520143185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5586800edd90 con 0x55868006a600 2024-05-10T11:15:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.562+0000 7f6146000700 1 -- 172.21.15.27:0/1520143185 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6150004610 con 0x55868006a600 2024-05-10T11:15:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.562+0000 7f6146000700 1 -- 172.21.15.27:0/1520143185 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f6150039070 con 0x55868006a600 2024-05-10T11:15:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.562+0000 7f616942ab80 10 client.25730 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.562+0000 7f616942ab80 10 client.25730 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:15:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.562+0000 7f616942ab80 20 client.25730 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.562+0000 7f616942ab80 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5586800ee3e0 0x5586800f0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.562+0000 7f616942ab80 4 client.25730 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:15:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.562+0000 7f616942ab80 1 -- 172.21.15.27:0/1520143185 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55867ffca400 con 0x5586800ee3e0 2024-05-10T11:15:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.563+0000 7f614f400700 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5586800ee3e0 0x5586800f0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.564+0000 7f614f400700 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5586800ee3e0 0x5586800f0800 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=129a9b137e0c22df server_cookie=6492a2a29b9dd9e1 in_seq=0 out_seq=0 2024-05-10T11:15:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.564+0000 7f6146000700 10 client.25730 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:15:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.564+0000 7f6146000700 1 -- 172.21.15.27:0/1520143185 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55867ffca400 con 0x5586800ee3e0 2024-05-10T11:15:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.564+0000 7f6146000700 10 client.25730 handle_command_reply: tid=0 2024-05-10T11:15:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.564+0000 7f616942ab80 1 client.25730 shutdown 2024-05-10T11:15:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.566+0000 7f616942ab80 1 -- 172.21.15.27:0/1520143185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 msgr2=0x55868007c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.566+0000 7f616942ab80 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868007c3a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f615000c050 tx=0x7f6150003c20 comp rx=0 tx=0).stop 2024-05-10T11:15:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.566+0000 7f616942ab80 1 -- 172.21.15.27:0/1520143185 shutdown_connections 2024-05-10T11:15:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.566+0000 7f616942ab80 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5586800ee3e0 0x5586800f0800 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.566+0000 7f616942ab80 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558680070300 0x55868008a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.566+0000 7f616942ab80 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55868006a600 0x55868007c3a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.566+0000 7f616942ab80 1 --2- 172.21.15.27:0/1520143185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55868006f9e0 0x55868007c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.566+0000 7f616942ab80 1 -- 172.21.15.27:0/1520143185 >> 172.21.15.27:0/1520143185 conn(0x55867ff98b60 msgr2=0x5586800706e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.566+0000 7f6146000700 0 client.25730 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:15:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.566+0000 7f616942ab80 1 -- 172.21.15.27:0/1520143185 shutdown_connections 2024-05-10T11:15:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.566+0000 7f616942ab80 1 -- 172.21.15.27:0/1520143185 wait complete. 2024-05-10T11:15:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.567+0000 7f616942ab80 20 client.25730 trim_cache size 0 max 16384 2024-05-10T11:15:31.568 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:15:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.568+0000 7f615d400700 1 -- 172.21.15.27:0/3621022999 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f613c0b2a70 msgr2=0x7f613c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.568+0000 7f615d400700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f613c0b2a70 0x7f613c0b4f30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f61480a7870 tx=0x7f6148009310 comp rx=0 tx=0).stop 2024-05-10T11:15:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.568+0000 7f615d400700 1 -- 172.21.15.27:0/3621022999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6158101f30 msgr2=0x7f615819f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.568+0000 7f615d400700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6158101f30 0x7f615819f7d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6148005910 tx=0x7f614800b670 comp rx=0 tx=0).stop 2024-05-10T11:15:31.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.568+0000 7f615d400700 1 -- 172.21.15.27:0/3621022999 shutdown_connections 2024-05-10T11:15:31.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.568+0000 7f615d400700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f613c0b2a70 0x7f613c0b4f30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.568+0000 7f615d400700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61581087a0 0x7f6158199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.568+0000 7f615d400700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6158101f30 0x7f615819f7d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.568+0000 7f615d400700 1 --2- 172.21.15.27:0/3621022999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61581028e0 0x7f615819fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:31.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.568+0000 7f615d400700 1 -- 172.21.15.27:0/3621022999 >> 172.21.15.27:0/3621022999 conn(0x7f6158077650 msgr2=0x7f61580fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:31.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.568+0000 7f615d400700 1 -- 172.21.15.27:0/3621022999 shutdown_connections 2024-05-10T11:15:31.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:31.568+0000 7f615d400700 1 -- 172.21.15.27:0/3621022999 wait complete. 2024-05-10T11:15:31.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 18068 7.26 MB/sec warmup 16 sec latency 12.146 ms 2024-05-10T11:15:32.586 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:32 smithi027 ceph-mon[30054]: pgmap v72: 129 pgs: 129 active+clean; 8.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 350 KiB/s wr, 57 op/s 2024-05-10T11:15:32.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:15:32.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 19573 7.17 MB/sec warmup 17 sec latency 27.124 ms 2024-05-10T11:15:32.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:32 smithi076 ceph-mon[34152]: pgmap v72: 129 pgs: 129 active+clean; 8.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 350 KiB/s wr, 57 op/s 2024-05-10T11:15:32.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:32 smithi189 ceph-mon[34842]: pgmap v72: 129 pgs: 129 active+clean; 8.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 350 KiB/s wr, 57 op/s 2024-05-10T11:15:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:33.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 21721 7.54 MB/sec warmup 18 sec latency 14.927 ms 2024-05-10T11:15:34.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:34 smithi027 ceph-mon[30054]: pgmap v73: 129 pgs: 129 active+clean; 8.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 350 KiB/s wr, 57 op/s 2024-05-10T11:15:34.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 23702 7.51 MB/sec warmup 19 sec latency 39.541 ms 2024-05-10T11:15:34.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:34 smithi076 ceph-mon[34152]: pgmap v73: 129 pgs: 129 active+clean; 8.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 350 KiB/s wr, 57 op/s 2024-05-10T11:15:34.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:34 smithi189 ceph-mon[34842]: pgmap v73: 129 pgs: 129 active+clean; 8.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 350 KiB/s wr, 57 op/s 2024-05-10T11:15:35.552 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:35 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:15:35.552 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:35 smithi027 ceph-mon[30054]: pgmap v74: 129 pgs: 129 active+clean; 12 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 802 KiB/s wr, 127 op/s 2024-05-10T11:15:35.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 25343 7.74 MB/sec warmup 20 sec latency 11.236 ms 2024-05-10T11:15:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:35 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:15:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:35 smithi076 ceph-mon[34152]: pgmap v74: 129 pgs: 129 active+clean; 12 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 802 KiB/s wr, 127 op/s 2024-05-10T11:15:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:35 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:15:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:35 smithi189 ceph-mon[34842]: pgmap v74: 129 pgs: 129 active+clean; 12 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 802 KiB/s wr, 127 op/s 2024-05-10T11:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.915+0000 7f8adb200700 1 -- 172.21.15.27:0/870305062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ad4109510 msgr2=0x7f8ad41098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.915+0000 7f8adb200700 1 --2- 172.21.15.27:0/870305062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ad4109510 0x7f8ad41098f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8ac40098f0 tx=0x7f8ac4009c00 comp rx=0 tx=0).stop 2024-05-10T11:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.915+0000 7f8adb200700 1 -- 172.21.15.27:0/870305062 shutdown_connections 2024-05-10T11:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.915+0000 7f8adb200700 1 --2- 172.21.15.27:0/870305062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ad4109510 0x7f8ad41098f0 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8ac40098f0 tx=0x7f8ac4009c00 comp rx=0 tx=0).stop 2024-05-10T11:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.915+0000 7f8adb200700 1 --2- 172.21.15.27:0/870305062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ad406bb40 0x7f8ad4110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.915+0000 7f8adb200700 1 --2- 172.21.15.27:0/870305062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad406b180 0x7f8ad406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.915+0000 7f8adb200700 1 -- 172.21.15.27:0/870305062 >> 172.21.15.27:0/870305062 conn(0x7f8ad4079350 msgr2=0x7f8ad4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.915+0000 7f8adb200700 1 -- 172.21.15.27:0/870305062 shutdown_connections 2024-05-10T11:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.916+0000 7f8adb200700 1 -- 172.21.15.27:0/870305062 wait complete. 2024-05-10T11:15:35.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.917+0000 7f8adb200700 1 Processor -- start 2024-05-10T11:15:35.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.917+0000 7f8adb200700 1 -- start start 2024-05-10T11:15:35.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.918+0000 7f8adb200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ad406b180 0x7f8ad419c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.918+0000 7f8adb200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ad406bb40 0x7f8ad419c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.918+0000 7f8adb200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad41a0900 0x7f8ad41a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.918+0000 7f8adb200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad4113670 con 0x7f8ad41a0900 2024-05-10T11:15:35.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.918+0000 7f8adb200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad41134f0 con 0x7f8ad406b180 2024-05-10T11:15:35.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.918+0000 7f8adb200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad41137f0 con 0x7f8ad406bb40 2024-05-10T11:15:35.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.919+0000 7f8ad3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ad406b180 0x7f8ad419c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.919+0000 7f8ad3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ad406bb40 0x7f8ad419c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.919+0000 7f8ad3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ad406b180 0x7f8ad419c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45298/0 (socket says 172.21.15.27:45298) 2024-05-10T11:15:35.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.919+0000 7f8ad3e00700 1 -- 172.21.15.27:0/2677582346 learned_addr learned my addr 172.21.15.27:0/2677582346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:35.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.919+0000 7f8ad9200700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad41a0900 0x7f8ad41a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.919+0000 7f8ad3e00700 1 -- 172.21.15.27:0/2677582346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ad406bb40 msgr2=0x7f8ad419c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:35.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.919+0000 7f8ad3e00700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ad406bb40 0x7f8ad419c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.919+0000 7f8ad3e00700 1 -- 172.21.15.27:0/2677582346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad41a0900 msgr2=0x7f8ad41a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:35.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.919+0000 7f8ad3e00700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad41a0900 0x7f8ad41a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.919+0000 7f8ad3e00700 1 -- 172.21.15.27:0/2677582346 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad41a1420 con 0x7f8ad406b180 2024-05-10T11:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.920+0000 7f8ad3e00700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ad406b180 0x7f8ad419c080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400c010 tx=0x7f8ac400fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.920+0000 7f8ad9200700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad41a0900 0x7f8ad41a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.920+0000 7f8ad0c00700 1 -- 172.21.15.27:0/2677582346 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40051c0 con 0x7f8ad406b180 2024-05-10T11:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.920+0000 7f8ad0c00700 1 -- 172.21.15.27:0/2677582346 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ac400b560 con 0x7f8ad406b180 2024-05-10T11:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.920+0000 7f8ad0c00700 1 -- 172.21.15.27:0/2677582346 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4003970 con 0x7f8ad406b180 2024-05-10T11:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.920+0000 7f8adb200700 1 -- 172.21.15.27:0/2677582346 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad41a16b0 con 0x7f8ad406b180 2024-05-10T11:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.920+0000 7f8adb200700 1 -- 172.21.15.27:0/2677582346 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad406d090 con 0x7f8ad406b180 2024-05-10T11:15:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.926+0000 7f8ad0c00700 1 -- 172.21.15.27:0/2677582346 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f8ac400b930 con 0x7f8ad406b180 2024-05-10T11:15:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.927+0000 7f8ae5bd3b80 1 Processor -- start 2024-05-10T11:15:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.927+0000 7f8ae5bd3b80 1 -- start start 2024-05-10T11:15:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.927+0000 7f8ae5bd3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c20a30 0x563632c20e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.928+0000 7f8ae5bd3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1ae60 0x563632c1b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.928+0000 7f8ae5bd3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1b820 0x563632c3b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.928+0000 7f8ae5bd3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563632b70a20 con 0x563632c1ae60 2024-05-10T11:15:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.928+0000 7f8ae5bd3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563632b0e760 con 0x563632c20a30 2024-05-10T11:15:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.928+0000 7f8ae5bd3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563632b6c9e0 con 0x563632c1b820 2024-05-10T11:15:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.928+0000 7f8ad3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1b820 0x563632c3b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.928+0000 7f8ad3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1b820 0x563632c3b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:35058/0 (socket says 172.21.15.27:35058) 2024-05-10T11:15:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.928+0000 7f8ad3400700 1 -- 172.21.15.27:0/3964105915 learned_addr learned my addr 172.21.15.27:0/3964105915 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.928+0000 7f8ad9200700 1 --2- 172.21.15.27:0/3964105915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c20a30 0x563632c20e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.929+0000 7f8ad3400700 1 -- 172.21.15.27:0/3964105915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c20a30 msgr2=0x563632c20e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.929+0000 7f8ad3400700 1 --2- 172.21.15.27:0/3964105915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c20a30 0x563632c20e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.929+0000 7f8ad3400700 1 -- 172.21.15.27:0/3964105915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1ae60 msgr2=0x563632c1b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.929+0000 7f8ad3e00700 1 --2- 172.21.15.27:0/3964105915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1ae60 0x563632c1b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.929+0000 7f8ad3400700 1 --2- 172.21.15.27:0/3964105915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1ae60 0x563632c1b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.929+0000 7f8ad3400700 1 -- 172.21.15.27:0/3964105915 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563632abd8d0 con 0x563632c1b820 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.929+0000 7f8ad3400700 1 --2- 172.21.15.27:0/3964105915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1b820 0x563632c3b6c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0009ca0 tx=0x7f8ac0011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab63c67c996bd27b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.929+0000 7f8ad0c00700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8ab80b2af0 0x7f8ab80b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.930+0000 7f8ac9600700 1 -- 172.21.15.27:0/3964105915 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac001e070 con 0x563632c1b820 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.930+0000 7f8ac9600700 1 -- 172.21.15.27:0/3964105915 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ac0011d80 con 0x563632c1b820 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.930+0000 7f8ad0c00700 1 -- 172.21.15.27:0/2677582346 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f8ac40e3d70 con 0x7f8ad406b180 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.930+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3964105915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1b820 msgr2=0x563632c3b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.931+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/3964105915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1b820 0x563632c3b6c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0009ca0 tx=0x7f8ac0011440 comp rx=0 tx=0).stop 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.931+0000 7f8ac9600700 1 -- 172.21.15.27:0/3964105915 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac000e720 con 0x563632c1b820 2024-05-10T11:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.931+0000 7f8ad9200700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8ab80b2af0 0x7f8ab80b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.932+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3964105915 shutdown_connections 2024-05-10T11:15:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.932+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/3964105915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1b820 0x563632c3b6c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.932+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/3964105915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c20a30 0x563632c20e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.932+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/3964105915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1ae60 0x563632c1b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.932+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3964105915 >> 172.21.15.27:0/3964105915 conn(0x563632b4cb60 msgr2=0x563632c187a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.932+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3964105915 shutdown_connections 2024-05-10T11:15:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.932+0000 7f8ad9200700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8ab80b2af0 0x7f8ab80b4fb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009e60 tx=0x7f8acc00b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.932+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3964105915 wait complete. 2024-05-10T11:15:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.933+0000 7f8ae5bd3b80 1 -- start start 2024-05-10T11:15:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.934+0000 7f8ae5bd3b80 1 Processor -- start 2024-05-10T11:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.934+0000 7f8ae5bd3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1ae60 0x563632c3b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.934+0000 7f8ae5bd3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1b820 0x563632c3b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.934+0000 7f8ae5bd3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c20a30 0x563632c386c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.935+0000 7f8ae5bd3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563632c3be00 con 0x563632c20a30 2024-05-10T11:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.935+0000 7f8ae5bd3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563632ad6b20 con 0x563632c1b820 2024-05-10T11:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.935+0000 7f8ae5bd3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563632c3bf80 con 0x563632c1ae60 2024-05-10T11:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.935+0000 7f8ae5bd3b80 10 client.0 _ll_register_callbacks cb 0x563632b17c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.935+0000 7f8ad3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1b820 0x563632c3b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.935+0000 7f8ad3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1b820 0x563632c3b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45306/0 (socket says 172.21.15.27:45306) 2024-05-10T11:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.935+0000 7f8ad3400700 1 -- 172.21.15.27:0/3832683157 learned_addr learned my addr 172.21.15.27:0/3832683157 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.935+0000 7f8ad3e00700 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1ae60 0x563632c3b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.935+0000 7f8ad9200700 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c20a30 0x563632c386c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.936+0000 7f8ad3400700 1 -- 172.21.15.27:0/3832683157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1ae60 msgr2=0x563632c3b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.936+0000 7f8ad3400700 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1ae60 0x563632c3b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.936+0000 7f8ad3400700 1 -- 172.21.15.27:0/3832683157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c20a30 msgr2=0x563632c386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.936+0000 7f8ad3400700 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c20a30 0x563632c386c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.936+0000 7f8ad3400700 1 -- 172.21.15.27:0/3832683157 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563632b36420 con 0x563632c1b820 2024-05-10T11:15:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.936+0000 7f8ad3400700 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1b820 0x563632c3b660 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0000c00 tx=0x7f8ac00056a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7786826be1a0ba11 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.936+0000 7f8abe000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.936+0000 7f8abe000700 1 -- 172.21.15.27:0/3832683157 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0027070 con 0x563632c1b820 2024-05-10T11:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.936+0000 7f8abe000700 1 -- 172.21.15.27:0/3832683157 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f8ac0004750 con 0x563632c1b820 2024-05-10T11:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.937+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3832683157 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ac0011d80 con 0x563632c1b820 2024-05-10T11:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.937+0000 7f8abe000700 1 -- 172.21.15.27:0/3832683157 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8ac001e040 con 0x563632c1b820 2024-05-10T11:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.937+0000 7f8ae5bd3b80 10 client.25742 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.937+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3832683157 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563632c464b0 con 0x563632c1b820 2024-05-10T11:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.937+0000 7f8abe000700 1 -- 172.21.15.27:0/3832683157 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ac0033070 con 0x563632c1b820 2024-05-10T11:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.937+0000 7f8abe000700 1 -- 172.21.15.27:0/3832683157 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f8ac0019a30 con 0x563632c1b820 2024-05-10T11:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.937+0000 7f8ae5bd3b80 10 client.25742 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.937+0000 7f8ae5bd3b80 10 client.25742 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.938+0000 7f8ae5bd3b80 20 client.25742 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.938+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563632c48a30 0x563632c4ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.938+0000 7f8ae5bd3b80 4 client.25742 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:15:35.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.938+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3832683157 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563632beb0a0 con 0x563632c48a30 2024-05-10T11:15:35.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.938+0000 7f8ad3e00700 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563632c48a30 0x563632c4ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.939+0000 7f8ad3e00700 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563632c48a30 0x563632c4ae20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac7ae8ce40ab73a8 server_cookie=dc92b7ded685c37 in_seq=0 out_seq=0 2024-05-10T11:15:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.939+0000 7f8abe000700 10 client.25742 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:15:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.940+0000 7f8abe000700 1 -- 172.21.15.27:0/3832683157 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563632beb0a0 con 0x563632c48a30 2024-05-10T11:15:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.940+0000 7f8abe000700 10 client.25742 handle_command_reply: tid=0 2024-05-10T11:15:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.940+0000 7f8ae5bd3b80 1 client.25742 shutdown 2024-05-10T11:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.940+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3832683157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1b820 msgr2=0x563632c3b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.940+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1b820 0x563632c3b660 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0000c00 tx=0x7f8ac00056a0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.940+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3832683157 shutdown_connections 2024-05-10T11:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.940+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563632c48a30 0x563632c4ae20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.940+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1ae60 0x563632c3b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.941+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c20a30 0x563632c386c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.941+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/3832683157 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1b820 0x563632c3b660 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.941+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3832683157 >> 172.21.15.27:0/3832683157 conn(0x563632b4cb60 msgr2=0x563632c132b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.941+0000 7f8abe000700 0 client.25742 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.941+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3832683157 shutdown_connections 2024-05-10T11:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.941+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/3832683157 wait complete. 2024-05-10T11:15:35.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.941+0000 7f8ae5bd3b80 20 client.25742 trim_cache size 0 max 16384 2024-05-10T11:15:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.964+0000 7f8ae5bd3b80 1 Processor -- start 2024-05-10T11:15:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.964+0000 7f8ae5bd3b80 1 -- start start 2024-05-10T11:15:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.965+0000 7f8ae5bd3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1ae60 0x563632c20e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.965+0000 7f8ae5bd3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1b820 0x563632c3b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.965+0000 7f8ae5bd3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c20a30 0x563632c3b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.965+0000 7f8ae5bd3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad4071c50 con 0x563632c1b820 2024-05-10T11:15:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.965+0000 7f8ae5bd3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad40737f0 con 0x563632c1ae60 2024-05-10T11:15:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.965+0000 7f8ae5bd3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad4072320 con 0x563632c20a30 2024-05-10T11:15:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.965+0000 7f8ad9200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c20a30 0x563632c3b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.965+0000 7f8ad9200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c20a30 0x563632c3b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:35082/0 (socket says 172.21.15.27:35082) 2024-05-10T11:15:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.965+0000 7f8ad9200700 1 -- 172.21.15.27:0/821159716 learned_addr learned my addr 172.21.15.27:0/821159716 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.965+0000 7f8ad3400700 1 --2- 172.21.15.27:0/821159716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1b820 0x563632c3b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.965+0000 7f8ad3e00700 1 --2- 172.21.15.27:0/821159716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1ae60 0x563632c20e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.966+0000 7f8ad9200700 1 -- 172.21.15.27:0/821159716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1ae60 msgr2=0x563632c20e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:35.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.966+0000 7f8ad9200700 1 --2- 172.21.15.27:0/821159716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1ae60 0x563632c20e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.966+0000 7f8ad9200700 1 -- 172.21.15.27:0/821159716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1b820 msgr2=0x563632c3b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:35.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.966+0000 7f8ad9200700 1 --2- 172.21.15.27:0/821159716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1b820 0x563632c3b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.966+0000 7f8ad9200700 1 -- 172.21.15.27:0/821159716 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563632c20150 con 0x563632c20a30 2024-05-10T11:15:35.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.966+0000 7f8ad3e00700 1 --2- 172.21.15.27:0/821159716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1ae60 0x563632c20e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:15:35.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.966+0000 7f8ad3400700 1 --2- 172.21.15.27:0/821159716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1b820 0x563632c3b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:15:35.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.966+0000 7f8ad9200700 1 --2- 172.21.15.27:0/821159716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c20a30 0x563632c3b660 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8acc0177b0 tx=0x7f8acc00d050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6be25ccf04ee77be server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:35.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.966+0000 7f8ac8c00700 1 -- 172.21.15.27:0/821159716 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc0246c0 con 0x563632c20a30 2024-05-10T11:15:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.967+0000 7f8ac8c00700 1 -- 172.21.15.27:0/821159716 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8acc029070 con 0x563632c20a30 2024-05-10T11:15:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.967+0000 7f8ac8c00700 1 -- 172.21.15.27:0/821159716 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc018c60 con 0x563632c20a30 2024-05-10T11:15:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.967+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/821159716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c20a30 msgr2=0x563632c3b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.967+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/821159716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c20a30 0x563632c3b660 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8acc0177b0 tx=0x7f8acc00d050 comp rx=0 tx=0).stop 2024-05-10T11:15:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.967+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/821159716 shutdown_connections 2024-05-10T11:15:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.967+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/821159716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c20a30 0x563632c3b660 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.967+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/821159716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c1ae60 0x563632c20e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.967+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/821159716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1b820 0x563632c3b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.967+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/821159716 >> 172.21.15.27:0/821159716 conn(0x563632b4cb60 msgr2=0x563632c1fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.968+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/821159716 shutdown_connections 2024-05-10T11:15:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.968+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/821159716 wait complete. 2024-05-10T11:15:35.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.968+0000 7f8ae5bd3b80 1 -- start start 2024-05-10T11:15:35.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.968+0000 7f8ae5bd3b80 1 Processor -- start 2024-05-10T11:15:35.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.969+0000 7f8ae5bd3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1ae60 0x563632c41260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.969+0000 7f8ae5bd3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1b820 0x563632c417a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.969+0000 7f8ae5bd3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c20a30 0x563632c1d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.969+0000 7f8ae5bd3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad40730c0 con 0x563632c1b820 2024-05-10T11:15:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.969+0000 7f8ae5bd3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad40729f0 con 0x563632c20a30 2024-05-10T11:15:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.969+0000 7f8ae5bd3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad406fbf0 con 0x563632c1ae60 2024-05-10T11:15:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.969+0000 7f8ae5bd3b80 10 client.0 _ll_register_callbacks cb 0x563632b17c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1ae60 0x563632c41260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1b820 0x563632c417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad9200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c20a30 0x563632c1d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1ae60 0x563632c41260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:35086/0 (socket says 172.21.15.27:35086) 2024-05-10T11:15:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad9200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c20a30 0x563632c1d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45324/0 (socket says 172.21.15.27:45324) 2024-05-10T11:15:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1b820 0x563632c417a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49676/0 (socket says 172.21.15.27:49676) 2024-05-10T11:15:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad3e00700 1 -- 172.21.15.27:0/2232972997 learned_addr learned my addr 172.21.15.27:0/2232972997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad9200700 1 -- 172.21.15.27:0/2232972997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1ae60 msgr2=0x563632c41260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad9200700 1 --2- 172.21.15.27:0/2232972997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1ae60 0x563632c41260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad9200700 1 -- 172.21.15.27:0/2232972997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1b820 msgr2=0x563632c417a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad9200700 1 --2- 172.21.15.27:0/2232972997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1b820 0x563632c417a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad9200700 1 -- 172.21.15.27:0/2232972997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563632c1dc20 con 0x563632c20a30 2024-05-10T11:15:35.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.970+0000 7f8ad3e00700 1 --2- 172.21.15.27:0/2232972997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1ae60 0x563632c41260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:15:35.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.971+0000 7f8ad9200700 1 --2- 172.21.15.27:0/2232972997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c20a30 0x563632c1d5e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009610 tx=0x7f8acc009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7180171cdadc420d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:35.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.971+0000 7f8abe000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:35.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.972+0000 7f8abe000700 1 -- 172.21.15.27:0/2232972997 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc033070 con 0x563632c20a30 2024-05-10T11:15:35.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.972+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/2232972997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8acc029070 con 0x563632c20a30 2024-05-10T11:15:35.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.972+0000 7f8abe000700 1 -- 172.21.15.27:0/2232972997 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f8acc003c00 con 0x563632c20a30 2024-05-10T11:15:35.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.973+0000 7f8abe000700 1 -- 172.21.15.27:0/2232972997 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8acc03e050 con 0x563632c20a30 2024-05-10T11:15:35.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.973+0000 7f8ae5bd3b80 10 client.25748 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:35.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.973+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/2232972997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563632cb2a00 con 0x563632c20a30 2024-05-10T11:15:35.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.974+0000 7f8abe000700 1 -- 172.21.15.27:0/2232972997 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8acc029070 con 0x563632c20a30 2024-05-10T11:15:35.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.974+0000 7f8abe000700 1 -- 172.21.15.27:0/2232972997 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f8acc004020 con 0x563632c20a30 2024-05-10T11:15:35.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.974+0000 7f8ae5bd3b80 10 client.25748 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:35.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.974+0000 7f8ae5bd3b80 10 client.25748 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:15:35.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.974+0000 7f8ae5bd3b80 20 client.25748 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:35.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.974+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/2232972997 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563632cb3040 0x563632cb5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:35.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.974+0000 7f8ae5bd3b80 4 client.25748 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:15:35.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.974+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/2232972997 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563632b7e400 con 0x563632cb3040 2024-05-10T11:15:35.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.974+0000 7f8ad3e00700 1 --2- 172.21.15.27:0/2232972997 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563632cb3040 0x563632cb5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:35.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.975+0000 7f8ad3e00700 1 --2- 172.21.15.27:0/2232972997 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563632cb3040 0x563632cb5460 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=790e213f391adf46 server_cookie=6a97051017405c34 in_seq=0 out_seq=0 2024-05-10T11:15:35.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.975+0000 7f8abe000700 10 client.25748 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:15:35.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.975+0000 7f8abe000700 1 -- 172.21.15.27:0/2232972997 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563632b7e400 con 0x563632cb3040 2024-05-10T11:15:35.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.975+0000 7f8abe000700 10 client.25748 handle_command_reply: tid=0 2024-05-10T11:15:35.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.976+0000 7f8ae5bd3b80 1 client.25748 shutdown 2024-05-10T11:15:35.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.976+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/2232972997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c20a30 msgr2=0x563632c1d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:35.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.976+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/2232972997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c20a30 0x563632c1d5e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009610 tx=0x7f8acc009f30 comp rx=0 tx=0).stop 2024-05-10T11:15:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.976+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/2232972997 shutdown_connections 2024-05-10T11:15:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.976+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/2232972997 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563632cb3040 0x563632cb5460 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.976+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/2232972997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563632c1ae60 0x563632c41260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.976+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/2232972997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563632c20a30 0x563632c1d5e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.976+0000 7f8ae5bd3b80 1 --2- 172.21.15.27:0/2232972997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563632c1b820 0x563632c417a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.976+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/2232972997 >> 172.21.15.27:0/2232972997 conn(0x563632b4cb60 msgr2=0x563632c3b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.976+0000 7f8abe000700 0 client.25748 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:15:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.976+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/2232972997 shutdown_connections 2024-05-10T11:15:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.976+0000 7f8ae5bd3b80 1 -- 172.21.15.27:0/2232972997 wait complete. 2024-05-10T11:15:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.977+0000 7f8ae5bd3b80 20 client.25748 trim_cache size 0 max 16384 2024-05-10T11:15:35.977 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:15:35.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.978+0000 7f8adb200700 1 -- 172.21.15.27:0/2677582346 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8ab80b2af0 msgr2=0x7f8ab80b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:35.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.978+0000 7f8adb200700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8ab80b2af0 0x7f8ab80b4fb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009e60 tx=0x7f8acc00b040 comp rx=0 tx=0).stop 2024-05-10T11:15:35.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.978+0000 7f8adb200700 1 -- 172.21.15.27:0/2677582346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ad406b180 msgr2=0x7f8ad419c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:35.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.978+0000 7f8adb200700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ad406b180 0x7f8ad419c080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400c010 tx=0x7f8ac400fa00 comp rx=0 tx=0).stop 2024-05-10T11:15:35.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.978+0000 7f8adb200700 1 -- 172.21.15.27:0/2677582346 shutdown_connections 2024-05-10T11:15:35.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.978+0000 7f8adb200700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8ab80b2af0 0x7f8ab80b4fb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.978+0000 7f8adb200700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad41a0900 0x7f8ad41a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.978+0000 7f8adb200700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ad406b180 0x7f8ad419c080 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.978+0000 7f8adb200700 1 --2- 172.21.15.27:0/2677582346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ad406bb40 0x7f8ad419c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:35.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.978+0000 7f8adb200700 1 -- 172.21.15.27:0/2677582346 >> 172.21.15.27:0/2677582346 conn(0x7f8ad4079350 msgr2=0x7f8ad410e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:35.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.978+0000 7f8adb200700 1 -- 172.21.15.27:0/2677582346 shutdown_connections 2024-05-10T11:15:35.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:35.978+0000 7f8adb200700 1 -- 172.21.15.27:0/2677582346 wait complete. 2024-05-10T11:15:36.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:36 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:15:36.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:36 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:15:36.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:36 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:15:36.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 27456 7.75 MB/sec warmup 21 sec latency 37.744 ms 2024-05-10T11:15:36.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:36 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:15:36.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:36 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:15:36.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:36 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:15:36.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:36 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:15:36.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:36 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:15:36.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:36 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:15:37.055 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:15:37.541 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:37.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:37 smithi027 ceph-mon[30054]: pgmap v75: 129 pgs: 129 active+clean; 12 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 634 KiB/s wr, 102 op/s 2024-05-10T11:15:37.651 INFO:tasks.workunit.client.0.smithi027.stdout: 1 29019 7.91 MB/sec warmup 22 sec latency 12.242 ms 2024-05-10T11:15:37.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:37 smithi076 ceph-mon[34152]: pgmap v75: 129 pgs: 129 active+clean; 12 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 634 KiB/s wr, 102 op/s 2024-05-10T11:15:37.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:37 smithi189 ceph-mon[34842]: pgmap v75: 129 pgs: 129 active+clean; 12 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 634 KiB/s wr, 102 op/s 2024-05-10T11:15:38.652 INFO:tasks.workunit.client.0.smithi027.stdout: 1 31312 8.08 MB/sec warmup 23 sec latency 26.202 ms 2024-05-10T11:15:39.652 INFO:tasks.workunit.client.0.smithi027.stdout: 1 32567 8.05 MB/sec warmup 24 sec latency 23.492 ms 2024-05-10T11:15:40.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.032+0000 7f19ef200700 1 -- 172.21.15.27:0/2583502847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e8102d60 msgr2=0x7f19e810f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:40.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.032+0000 7f19ef200700 1 --2- 172.21.15.27:0/2583502847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e8102d60 0x7f19e810f250 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009990 tx=0x7f19e0009ca0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.032+0000 7f19ef200700 1 -- 172.21.15.27:0/2583502847 shutdown_connections 2024-05-10T11:15:40.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.032+0000 7f19ef200700 1 --2- 172.21.15.27:0/2583502847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e8102d60 0x7f19e810f250 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.032+0000 7f19ef200700 1 --2- 172.21.15.27:0/2583502847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e8108410 0x7f19e81087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.032+0000 7f19ef200700 1 --2- 172.21.15.27:0/2583502847 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19e81023a0 0x7f19e8102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.032+0000 7f19ef200700 1 -- 172.21.15.27:0/2583502847 >> 172.21.15.27:0/2583502847 conn(0x7f19e8079350 msgr2=0x7f19e8079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:40.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.032+0000 7f19ef200700 1 -- 172.21.15.27:0/2583502847 shutdown_connections 2024-05-10T11:15:40.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.032+0000 7f19ef200700 1 -- 172.21.15.27:0/2583502847 wait complete. 2024-05-10T11:15:40.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.033+0000 7f19ef200700 1 Processor -- start 2024-05-10T11:15:40.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.033+0000 7f19ef200700 1 -- start start 2024-05-10T11:15:40.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.034+0000 7f19ef200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e81023a0 0x7f19e8104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.034+0000 7f19ef200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e8102d60 0x7f19e81045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.034+0000 7f19ef200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19e8108410 0x7f19e8104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.034+0000 7f19ef200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f19e8112530 con 0x7f19e81023a0 2024-05-10T11:15:40.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.034+0000 7f19ef200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f19e81123b0 con 0x7f19e8108410 2024-05-10T11:15:40.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.034+0000 7f19ef200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f19e81126b0 con 0x7f19e8102d60 2024-05-10T11:15:40.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.034+0000 7f19e7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e8102d60 0x7f19e81045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.034+0000 7f19e7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e8102d60 0x7f19e81045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:35108/0 (socket says 172.21.15.27:35108) 2024-05-10T11:15:40.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.034+0000 7f19e7400700 1 -- 172.21.15.27:0/1166443136 learned_addr learned my addr 172.21.15.27:0/1166443136 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:40.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.034+0000 7f19ed200700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19e8108410 0x7f19e8104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.034+0000 7f19e7e00700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e81023a0 0x7f19e8104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.035+0000 7f19ed200700 1 -- 172.21.15.27:0/1166443136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e8102d60 msgr2=0x7f19e81045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:40.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.035+0000 7f19ed200700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e8102d60 0x7f19e81045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.035+0000 7f19ed200700 1 -- 172.21.15.27:0/1166443136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e81023a0 msgr2=0x7f19e8104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:40.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.035+0000 7f19ed200700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e81023a0 0x7f19e8104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.035+0000 7f19ed200700 1 -- 172.21.15.27:0/1166443136 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e81a8010 con 0x7f19e8108410 2024-05-10T11:15:40.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.035+0000 7f19e7400700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e8102d60 0x7f19e81045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:40.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.035+0000 7f19ed200700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19e8108410 0x7f19e8104b00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f19e00059d0 tx=0x7f19e00057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:40.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.035+0000 7f19e4c00700 1 -- 172.21.15.27:0/1166443136 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e0026070 con 0x7f19e8108410 2024-05-10T11:15:40.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.035+0000 7f19e4c00700 1 -- 172.21.15.27:0/1166443136 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19e0021410 con 0x7f19e8108410 2024-05-10T11:15:40.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.035+0000 7f19e4c00700 1 -- 172.21.15.27:0/1166443136 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e0017650 con 0x7f19e8108410 2024-05-10T11:15:40.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.035+0000 7f19ef200700 1 -- 172.21.15.27:0/1166443136 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e81a82a0 con 0x7f19e8108410 2024-05-10T11:15:40.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.036+0000 7f19ef200700 1 -- 172.21.15.27:0/1166443136 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e81a86a0 con 0x7f19e8108410 2024-05-10T11:15:40.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.039+0000 7f19e4c00700 1 -- 172.21.15.27:0/1166443136 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f19e0029070 con 0x7f19e8108410 2024-05-10T11:15:40.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.040+0000 7f19f9c4eb80 1 Processor -- start 2024-05-10T11:15:40.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.041+0000 7f19f9c4eb80 1 -- start start 2024-05-10T11:15:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.041+0000 7f19f9c4eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100ac060 0x562b100ac440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.041+0000 7f19f9c4eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac980 0x562b100b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.041+0000 7f19f9c4eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100b09d0 0x562b100b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.041+0000 7f19f9c4eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562b0fffea20 con 0x562b100ac060 2024-05-10T11:15:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.041+0000 7f19f9c4eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562b0ff9c760 con 0x562b100b09d0 2024-05-10T11:15:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.041+0000 7f19f9c4eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562b0fffa9e0 con 0x562b100ac980 2024-05-10T11:15:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.041+0000 7f19e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac980 0x562b100b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.041+0000 7f19e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac980 0x562b100b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:35110/0 (socket says 172.21.15.27:35110) 2024-05-10T11:15:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.041+0000 7f19e7e00700 1 -- 172.21.15.27:0/1766454803 learned_addr learned my addr 172.21.15.27:0/1766454803 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.041+0000 7f19e7400700 1 --2- 172.21.15.27:0/1766454803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100ac060 0x562b100ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.041+0000 7f19e4c00700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f19cc0b2a70 0x7f19cc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.042+0000 7f19e7400700 1 --2- 172.21.15.27:0/1766454803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100b09d0 0x562b100b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.042+0000 7f19e7e00700 1 -- 172.21.15.27:0/1766454803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100b09d0 msgr2=0x562b100b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.042+0000 7f19e7e00700 1 --2- 172.21.15.27:0/1766454803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100b09d0 0x562b100b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.042+0000 7f19e4c00700 1 -- 172.21.15.27:0/1166443136 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f19e00e3680 con 0x7f19e8108410 2024-05-10T11:15:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.042+0000 7f19e7e00700 1 -- 172.21.15.27:0/1766454803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100ac060 msgr2=0x562b100ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.042+0000 7f19e7e00700 1 --2- 172.21.15.27:0/1766454803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100ac060 0x562b100ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.042+0000 7f19e7e00700 1 -- 172.21.15.27:0/1766454803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b0ff4b8d0 con 0x562b100ac980 2024-05-10T11:15:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.042+0000 7f19e7400700 1 --2- 172.21.15.27:0/1766454803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100b09d0 0x562b100b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:15:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.042+0000 7f19e7e00700 1 --2- 172.21.15.27:0/1766454803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac980 0x562b100b0490 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f19d8011650 tx=0x7f19d8009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a55a27f8ee9ecac server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.042+0000 7f19ed200700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f19cc0b2a70 0x7f19cc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.043+0000 7f19dd600700 1 -- 172.21.15.27:0/1766454803 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d801ab70 con 0x562b100ac980 2024-05-10T11:15:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.043+0000 7f19dd600700 1 -- 172.21.15.27:0/1766454803 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d801e070 con 0x562b100ac980 2024-05-10T11:15:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.043+0000 7f19dd600700 1 -- 172.21.15.27:0/1766454803 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d80236d0 con 0x562b100ac980 2024-05-10T11:15:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.043+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/1766454803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac980 msgr2=0x562b100b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.043+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/1766454803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac980 0x562b100b0490 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f19d8011650 tx=0x7f19d8009920 comp rx=0 tx=0).stop 2024-05-10T11:15:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.043+0000 7f19ed200700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f19cc0b2a70 0x7f19cc0b4f30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f19e00098e0 tx=0x7f19e0009ca0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.044+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/1766454803 shutdown_connections 2024-05-10T11:15:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.044+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/1766454803 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100b09d0 0x562b100b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.044+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/1766454803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100ac060 0x562b100ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.044+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/1766454803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac980 0x562b100b0490 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.044+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/1766454803 >> 172.21.15.27:0/1766454803 conn(0x562b0ffdab60 msgr2=0x562b100a67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:40.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.044+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/1766454803 shutdown_connections 2024-05-10T11:15:40.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.044+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/1766454803 wait complete. 2024-05-10T11:15:40.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.044+0000 7f19f9c4eb80 1 -- start start 2024-05-10T11:15:40.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.045+0000 7f19f9c4eb80 1 Processor -- start 2024-05-10T11:15:40.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.045+0000 7f19f9c4eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac060 0x562b100b7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.045+0000 7f19f9c4eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac980 0x562b100b5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.045+0000 7f19f9c4eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100b09d0 0x562b100b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.045+0000 7f19f9c4eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562b100b8be0 con 0x562b100b09d0 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.045+0000 7f19f9c4eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562b0ff64b20 con 0x562b100ac980 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.045+0000 7f19f9c4eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562b100b8d60 con 0x562b100ac060 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.045+0000 7f19f9c4eb80 10 client.0 _ll_register_callbacks cb 0x562b0ffa5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.045+0000 7f19e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac980 0x562b100b5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.046+0000 7f19e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac980 0x562b100b5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45344/0 (socket says 172.21.15.27:45344) 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.046+0000 7f19e7e00700 1 -- 172.21.15.27:0/4263408814 learned_addr learned my addr 172.21.15.27:0/4263408814 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.046+0000 7f19e7400700 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac060 0x562b100b7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.046+0000 7f19e7400700 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100b09d0 0x562b100b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.046+0000 7f19e7e00700 1 -- 172.21.15.27:0/4263408814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac060 msgr2=0x562b100b7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.046+0000 7f19e7e00700 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac060 0x562b100b7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.046+0000 7f19e7e00700 1 -- 172.21.15.27:0/4263408814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100b09d0 msgr2=0x562b100b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.046+0000 7f19e7e00700 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100b09d0 0x562b100b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.046+0000 7f19e7e00700 1 -- 172.21.15.27:0/4263408814 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b0ffc4420 con 0x562b100ac980 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.046+0000 7f19e7400700 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac060 0x562b100b7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:15:40.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.046+0000 7f19e7e00700 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac980 0x562b100b5250 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f19d8011300 tx=0x7f19d801ae50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d9e16dd75675b27 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.046+0000 7f19d2000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.047+0000 7f19d2000700 1 -- 172.21.15.27:0/4263408814 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d8031060 con 0x562b100ac980 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.047+0000 7f19d2000700 1 -- 172.21.15.27:0/4263408814 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f19d8007700 con 0x562b100ac980 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.047+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/4263408814 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19d801e070 con 0x562b100ac980 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.047+0000 7f19d2000700 1 -- 172.21.15.27:0/4263408814 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f19d8004e20 con 0x562b100ac980 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.047+0000 7f19f9c4eb80 10 client.25757 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.047+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/4263408814 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562b100c3420 con 0x562b100ac980 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.047+0000 7f19d2000700 1 -- 172.21.15.27:0/4263408814 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d801e070 con 0x562b100ac980 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.047+0000 7f19d2000700 1 -- 172.21.15.27:0/4263408814 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f19d8004400 con 0x562b100ac980 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.048+0000 7f19f9c4eb80 10 client.25757 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.048+0000 7f19f9c4eb80 10 client.25757 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.048+0000 7f19f9c4eb80 20 client.25757 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.048+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x562b100c5a00 0x562b100c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.048+0000 7f19f9c4eb80 4 client.25757 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.048+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/4263408814 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b100790a0 con 0x562b100c5a00 2024-05-10T11:15:40.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.048+0000 7f19e7400700 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x562b100c5a00 0x562b100c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.048+0000 7f19e7400700 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x562b100c5a00 0x562b100c7df0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9ac8e1f189b3d89 server_cookie=8cf388f4dd7d51f2 in_seq=0 out_seq=0 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.048+0000 7f19d2000700 10 client.25757 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19d2000700 1 -- 172.21.15.27:0/4263408814 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562b100790a0 con 0x562b100c5a00 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19d2000700 10 client.25757 handle_command_reply: tid=0 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19f9c4eb80 1 client.25757 shutdown 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/4263408814 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac980 msgr2=0x562b100b5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac980 0x562b100b5250 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f19d8011300 tx=0x7f19d801ae50 comp rx=0 tx=0).stop 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/4263408814 shutdown_connections 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x562b100c5a00 0x562b100c7df0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac060 0x562b100b7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100b09d0 0x562b100b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/4263408814 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac980 0x562b100b5250 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/4263408814 >> 172.21.15.27:0/4263408814 conn(0x562b0ffdab60 msgr2=0x562b100a12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:40.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19d2000700 0 client.25757 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:15:40.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/4263408814 shutdown_connections 2024-05-10T11:15:40.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.050+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/4263408814 wait complete. 2024-05-10T11:15:40.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.051+0000 7f19f9c4eb80 20 client.25757 trim_cache size 0 max 16384 2024-05-10T11:15:40.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.068+0000 7f19f9c4eb80 1 Processor -- start 2024-05-10T11:15:40.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.068+0000 7f19f9c4eb80 1 -- start start 2024-05-10T11:15:40.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.068+0000 7f19f9c4eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b0ffdab60 0x562b100b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19f9c4eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 0x562b100a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19f9c4eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac980 0x562b100a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19f9c4eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f19e81ad310 con 0x562b0ffdab60 2024-05-10T11:15:40.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19f9c4eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f19e81aeeb0 con 0x562b100ac060 2024-05-10T11:15:40.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19f9c4eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f19e81ad9e0 con 0x562b100ac980 2024-05-10T11:15:40.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 0x562b100a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 0x562b100a0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45348/0 (socket says 172.21.15.27:45348) 2024-05-10T11:15:40.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19e7e00700 1 -- 172.21.15.27:0/611600180 learned_addr learned my addr 172.21.15.27:0/611600180 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:40.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19e7400700 1 --2- 172.21.15.27:0/611600180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b0ffdab60 0x562b100b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19e7400700 1 --2- 172.21.15.27:0/611600180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac980 0x562b100a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19e7e00700 1 -- 172.21.15.27:0/611600180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac980 msgr2=0x562b100a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19e7e00700 1 --2- 172.21.15.27:0/611600180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac980 0x562b100a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19e7e00700 1 -- 172.21.15.27:0/611600180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b0ffdab60 msgr2=0x562b100b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19e7e00700 1 --2- 172.21.15.27:0/611600180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b0ffdab60 0x562b100b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.069+0000 7f19e7e00700 1 -- 172.21.15.27:0/611600180 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b100a2f90 con 0x562b100ac060 2024-05-10T11:15:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.070+0000 7f19e7e00700 1 --2- 172.21.15.27:0/611600180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 0x562b100a0a20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f19d8009b30 tx=0x7f19d80021d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc23c8bdb2140f19 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.070+0000 7f19dcc00700 1 -- 172.21.15.27:0/611600180 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d8031070 con 0x562b100ac060 2024-05-10T11:15:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.070+0000 7f19dcc00700 1 -- 172.21.15.27:0/611600180 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d801e040 con 0x562b100ac060 2024-05-10T11:15:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.070+0000 7f19dcc00700 1 -- 172.21.15.27:0/611600180 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d8007920 con 0x562b100ac060 2024-05-10T11:15:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.071+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/611600180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 msgr2=0x562b100a0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.071+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/611600180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 0x562b100a0a20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f19d8009b30 tx=0x7f19d80021d0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.071+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/611600180 shutdown_connections 2024-05-10T11:15:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.071+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/611600180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b100ac980 0x562b100a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.071+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/611600180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b0ffdab60 0x562b100b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.071+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/611600180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 0x562b100a0a20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.071+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/611600180 >> 172.21.15.27:0/611600180 conn(0x562b100b1110 msgr2=0x562b100b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.071+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/611600180 shutdown_connections 2024-05-10T11:15:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.071+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/611600180 wait complete. 2024-05-10T11:15:40.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.071+0000 7f19f9c4eb80 1 -- start start 2024-05-10T11:15:40.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.072+0000 7f19f9c4eb80 1 Processor -- start 2024-05-10T11:15:40.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.072+0000 7f19f9c4eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b0ffdab60 0x562b100cac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.072+0000 7f19f9c4eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 0x562b100c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.072+0000 7f19f9c4eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100ac980 0x562b100c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.072+0000 7f19f9c4eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f19e81ae780 con 0x562b100ac980 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.072+0000 7f19f9c4eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f19e81ae0b0 con 0x562b100ac060 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.072+0000 7f19f9c4eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f19e81ab2b0 con 0x562b0ffdab60 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.072+0000 7f19f9c4eb80 10 client.0 _ll_register_callbacks cb 0x562b0ffa5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.073+0000 7f19e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 0x562b100c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.073+0000 7f19e7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b0ffdab60 0x562b100cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.073+0000 7f19e7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100ac980 0x562b100c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.073+0000 7f19e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 0x562b100c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45360/0 (socket says 172.21.15.27:45360) 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.073+0000 7f19e7e00700 1 -- 172.21.15.27:0/516622573 learned_addr learned my addr 172.21.15.27:0/516622573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.073+0000 7f19e7e00700 1 -- 172.21.15.27:0/516622573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b0ffdab60 msgr2=0x562b100cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.073+0000 7f19e7e00700 1 --2- 172.21.15.27:0/516622573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b0ffdab60 0x562b100cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.073+0000 7f19e7e00700 1 -- 172.21.15.27:0/516622573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100ac980 msgr2=0x562b100c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.073+0000 7f19e7e00700 1 --2- 172.21.15.27:0/516622573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100ac980 0x562b100c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.073+0000 7f19e7e00700 1 -- 172.21.15.27:0/516622573 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b100c6a40 con 0x562b100ac060 2024-05-10T11:15:40.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.073+0000 7f19e7e00700 1 --2- 172.21.15.27:0/516622573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 0x562b100c5fc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f19d80047a0 tx=0x7f19d80098a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ed2b410db0e79d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:40.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.073+0000 7f19d2000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:40.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.074+0000 7f19d2000700 1 -- 172.21.15.27:0/516622573 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d8031070 con 0x562b100ac060 2024-05-10T11:15:40.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.074+0000 7f19d2000700 1 -- 172.21.15.27:0/516622573 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f19d8008940 con 0x562b100ac060 2024-05-10T11:15:40.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.074+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/516622573 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19d801e040 con 0x562b100ac060 2024-05-10T11:15:40.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.074+0000 7f19d2000700 1 -- 172.21.15.27:0/516622573 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f19d804c050 con 0x562b100ac060 2024-05-10T11:15:40.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.074+0000 7f19f9c4eb80 10 client.25763 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.074+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/516622573 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562b1012fc00 con 0x562b100ac060 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.074+0000 7f19d2000700 1 -- 172.21.15.27:0/516622573 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d801e040 con 0x562b100ac060 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.075+0000 7f19d2000700 1 -- 172.21.15.27:0/516622573 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f19d802c480 con 0x562b100ac060 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.075+0000 7f19f9c4eb80 10 client.25763 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.075+0000 7f19f9c4eb80 10 client.25763 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.075+0000 7f19f9c4eb80 20 client.25763 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.075+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/516622573 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x562b100b09d0 0x562b101320b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.075+0000 7f19f9c4eb80 4 client.25763 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.075+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/516622573 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562b1000c400 con 0x562b100b09d0 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.075+0000 7f19e7400700 1 --2- 172.21.15.27:0/516622573 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x562b100b09d0 0x562b101320b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19e7400700 1 --2- 172.21.15.27:0/516622573 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x562b100b09d0 0x562b101320b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74c1da4972be75c1 server_cookie=ef3dd1bbca359be2 in_seq=0 out_seq=0 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19d2000700 10 client.25763 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19d2000700 1 -- 172.21.15.27:0/516622573 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562b1000c400 con 0x562b100b09d0 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19d2000700 10 client.25763 handle_command_reply: tid=0 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19f9c4eb80 1 client.25763 shutdown 2024-05-10T11:15:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/516622573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 msgr2=0x562b100c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/516622573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 0x562b100c5fc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f19d80047a0 tx=0x7f19d80098a0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/516622573 shutdown_connections 2024-05-10T11:15:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/516622573 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x562b100b09d0 0x562b101320b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/516622573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b0ffdab60 0x562b100cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/516622573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b100ac980 0x562b100c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19f9c4eb80 1 --2- 172.21.15.27:0/516622573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b100ac060 0x562b100c5fc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/516622573 >> 172.21.15.27:0/516622573 conn(0x562b100b1110 msgr2=0x562b100a12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:40.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19d2000700 0 client.25763 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:15:40.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/516622573 shutdown_connections 2024-05-10T11:15:40.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.076+0000 7f19f9c4eb80 1 -- 172.21.15.27:0/516622573 wait complete. 2024-05-10T11:15:40.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.077+0000 7f19f9c4eb80 20 client.25763 trim_cache size 0 max 16384 2024-05-10T11:15:40.078 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:15:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.077+0000 7f19ef200700 1 -- 172.21.15.27:0/1166443136 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f19cc0b2a70 msgr2=0x7f19cc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.077+0000 7f19ef200700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f19cc0b2a70 0x7f19cc0b4f30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f19e00098e0 tx=0x7f19e0009ca0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.077+0000 7f19ef200700 1 -- 172.21.15.27:0/1166443136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19e8108410 msgr2=0x7f19e8104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.077+0000 7f19ef200700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19e8108410 0x7f19e8104b00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f19e00059d0 tx=0x7f19e00057c0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.078+0000 7f19ef200700 1 -- 172.21.15.27:0/1166443136 shutdown_connections 2024-05-10T11:15:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.078+0000 7f19ef200700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f19cc0b2a70 0x7f19cc0b4f30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.078+0000 7f19ef200700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19e8108410 0x7f19e8104b00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.078+0000 7f19ef200700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e81023a0 0x7f19e8104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.078+0000 7f19ef200700 1 --2- 172.21.15.27:0/1166443136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e8102d60 0x7f19e81045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:40.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.078+0000 7f19ef200700 1 -- 172.21.15.27:0/1166443136 >> 172.21.15.27:0/1166443136 conn(0x7f19e8079350 msgr2=0x7f19e8101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:40.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.078+0000 7f19ef200700 1 -- 172.21.15.27:0/1166443136 shutdown_connections 2024-05-10T11:15:40.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:40.078+0000 7f19ef200700 1 -- 172.21.15.27:0/1166443136 wait complete. 2024-05-10T11:15:40.585 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:40 smithi027 ceph-mon[30054]: pgmap v76: 129 pgs: 129 active+clean; 17 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-10T11:15:40.652 INFO:tasks.workunit.client.0.smithi027.stdout: 1 34896 8.22 MB/sec warmup 25 sec latency 24.626 ms 2024-05-10T11:15:40.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:40 smithi076 ceph-mon[34152]: pgmap v76: 129 pgs: 129 active+clean; 17 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-10T11:15:40.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:40 smithi189 ceph-mon[34842]: pgmap v76: 129 pgs: 129 active+clean; 17 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-10T11:15:41.126 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:15:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:41.652 INFO:tasks.workunit.client.0.smithi027.stdout: 1 36229 8.17 MB/sec warmup 26 sec latency 15.540 ms 2024-05-10T11:15:42.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:42 smithi027 ceph-mon[30054]: pgmap v77: 129 pgs: 129 active+clean; 17 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 913 KiB/s wr, 141 op/s 2024-05-10T11:15:42.652 INFO:tasks.workunit.client.0.smithi027.stdout: 1 38856 8.39 MB/sec warmup 27 sec latency 9.569 ms 2024-05-10T11:15:42.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:42 smithi076 ceph-mon[34152]: pgmap v77: 129 pgs: 129 active+clean; 17 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 913 KiB/s wr, 141 op/s 2024-05-10T11:15:42.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:42 smithi189 ceph-mon[34842]: pgmap v77: 129 pgs: 129 active+clean; 17 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 913 KiB/s wr, 141 op/s 2024-05-10T11:15:43.652 INFO:tasks.workunit.client.0.smithi027.stdout: 1 39923 8.28 MB/sec warmup 28 sec latency 38.232 ms 2024-05-10T11:15:44.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.092+0000 7f4338e00700 1 -- 172.21.15.27:0/765480866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4334102b80 msgr2=0x7f433410f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:44.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.092+0000 7f4338e00700 1 --2- 172.21.15.27:0/765480866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4334102b80 0x7f433410f840 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4320009990 tx=0x7f4320009ca0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.092+0000 7f4338e00700 1 -- 172.21.15.27:0/765480866 shutdown_connections 2024-05-10T11:15:44.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.092+0000 7f4338e00700 1 --2- 172.21.15.27:0/765480866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43341021d0 0x7f43341025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.092+0000 7f4338e00700 1 --2- 172.21.15.27:0/765480866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f433410fd80 0x7f4334112170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.092+0000 7f4338e00700 1 --2- 172.21.15.27:0/765480866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4334102b80 0x7f433410f840 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.092+0000 7f4338e00700 1 -- 172.21.15.27:0/765480866 >> 172.21.15.27:0/765480866 conn(0x7f43340fde30 msgr2=0x7f4334100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:44.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.092+0000 7f4338e00700 1 -- 172.21.15.27:0/765480866 shutdown_connections 2024-05-10T11:15:44.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.092+0000 7f4338e00700 1 -- 172.21.15.27:0/765480866 wait complete. 2024-05-10T11:15:44.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.093+0000 7f4338e00700 1 Processor -- start 2024-05-10T11:15:44.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.093+0000 7f4338e00700 1 -- start start 2024-05-10T11:15:44.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.094+0000 7f4338e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43341021d0 0x7f433419bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.094+0000 7f4338e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4334102b80 0x7f433419c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.094+0000 7f4338e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f433410fd80 0x7f43341a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.094+0000 7f4338e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f43341157e0 con 0x7f433410fd80 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.094+0000 7f4338e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4334115660 con 0x7f4334102b80 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.094+0000 7f4338e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4334115960 con 0x7f43341021d0 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.094+0000 7f4333400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43341021d0 0x7f433419bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.094+0000 7f4332a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4334102b80 0x7f433419c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.094+0000 7f4333400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43341021d0 0x7f433419bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60890/0 (socket says 172.21.15.27:60890) 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.094+0000 7f4333400700 1 -- 172.21.15.27:0/3732171238 learned_addr learned my addr 172.21.15.27:0/3732171238 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.094+0000 7f4333e00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f433410fd80 0x7f43341a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.095+0000 7f4332a00700 1 -- 172.21.15.27:0/3732171238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43341021d0 msgr2=0x7f433419bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.095+0000 7f4332a00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43341021d0 0x7f433419bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.095+0000 7f4332a00700 1 -- 172.21.15.27:0/3732171238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f433410fd80 msgr2=0x7f43341a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.095+0000 7f4332a00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f433410fd80 0x7f43341a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.095+0000 7f4332a00700 1 -- 172.21.15.27:0/3732171238 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43341a0d80 con 0x7f4334102b80 2024-05-10T11:15:44.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.095+0000 7f4333e00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f433410fd80 0x7f43341a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:15:44.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.095+0000 7f4332a00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4334102b80 0x7f433419c2b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4320006010 tx=0x7f43200057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:44.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.095+0000 7f432be00700 1 -- 172.21.15.27:0/3732171238 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320027070 con 0x7f4334102b80 2024-05-10T11:15:44.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.095+0000 7f432be00700 1 -- 172.21.15.27:0/3732171238 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4320022470 con 0x7f4334102b80 2024-05-10T11:15:44.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.095+0000 7f432be00700 1 -- 172.21.15.27:0/3732171238 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320017650 con 0x7f4334102b80 2024-05-10T11:15:44.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.095+0000 7f4338e00700 1 -- 172.21.15.27:0/3732171238 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43341a1010 con 0x7f4334102b80 2024-05-10T11:15:44.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.096+0000 7f4338e00700 1 -- 172.21.15.27:0/3732171238 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43341a88f0 con 0x7f4334102b80 2024-05-10T11:15:44.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.099+0000 7f432be00700 1 -- 172.21.15.27:0/3732171238 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f432002a070 con 0x7f4334102b80 2024-05-10T11:15:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.100+0000 7f4344d37b80 1 Processor -- start 2024-05-10T11:15:44.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.100+0000 7f4344d37b80 1 -- start start 2024-05-10T11:15:44.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.101+0000 7f432be00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f43180b2a70 0x7f43180b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.101+0000 7f432be00700 1 -- 172.21.15.27:0/3732171238 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f43200e36d0 con 0x7f4334102b80 2024-05-10T11:15:44.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.101+0000 7f4344d37b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d69e0 0x5569851d6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.101+0000 7f4344d37b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d7300 0x5569851d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.101+0000 7f4344d37b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d1600 0x5569851d1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.101+0000 7f4344d37b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556985123a20 con 0x5569851d1600 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.101+0000 7f4344d37b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5569850c1760 con 0x5569851d69e0 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.101+0000 7f4344d37b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55698511f9e0 con 0x5569851d7300 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.101+0000 7f4333400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d69e0 0x5569851d6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.101+0000 7f4333400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d69e0 0x5569851d6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57202/0 (socket says 172.21.15.27:57202) 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.101+0000 7f4333400700 1 -- 172.21.15.27:0/2702868891 learned_addr learned my addr 172.21.15.27:0/2702868891 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.101+0000 7f4332a00700 1 --2- 172.21.15.27:0/2702868891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d7300 0x5569851d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.102+0000 7f4333e00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f43180b2a70 0x7f43180b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.102+0000 7f4333400700 1 -- 172.21.15.27:0/2702868891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d7300 msgr2=0x5569851d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.102+0000 7f4333400700 1 --2- 172.21.15.27:0/2702868891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d7300 0x5569851d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.102+0000 7f4333400700 1 -- 172.21.15.27:0/2702868891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d1600 msgr2=0x5569851d1a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.102+0000 7f4333400700 1 --2- 172.21.15.27:0/2702868891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d1600 0x5569851d1a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.102+0000 7f4333400700 1 -- 172.21.15.27:0/2702868891 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569850708d0 con 0x5569851d69e0 2024-05-10T11:15:44.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.102+0000 7f4333400700 1 --2- 172.21.15.27:0/2702868891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d69e0 0x5569851d6dc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4324007da0 tx=0x7f43240106b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddf0b73fd55e2ca5 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:44.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.102+0000 7f4328c00700 1 -- 172.21.15.27:0/2702868891 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4324010e10 con 0x5569851d69e0 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.102+0000 7f4328c00700 1 -- 172.21.15.27:0/2702868891 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43240118f0 con 0x5569851d69e0 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.102+0000 7f4328c00700 1 -- 172.21.15.27:0/2702868891 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4324019870 con 0x5569851d69e0 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.103+0000 7f4344d37b80 1 -- 172.21.15.27:0/2702868891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d69e0 msgr2=0x5569851d6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.103+0000 7f4344d37b80 1 --2- 172.21.15.27:0/2702868891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d69e0 0x5569851d6dc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4324007da0 tx=0x7f43240106b0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.103+0000 7f4333e00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f43180b2a70 0x7f43180b4f30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x5569851d1fc0 tx=0x7f432c00b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.103+0000 7f4344d37b80 1 -- 172.21.15.27:0/2702868891 shutdown_connections 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.103+0000 7f4344d37b80 1 --2- 172.21.15.27:0/2702868891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d1600 0x5569851d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.103+0000 7f4344d37b80 1 --2- 172.21.15.27:0/2702868891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d69e0 0x5569851d6dc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.103+0000 7f4344d37b80 1 --2- 172.21.15.27:0/2702868891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d7300 0x5569851d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.103+0000 7f4344d37b80 1 -- 172.21.15.27:0/2702868891 >> 172.21.15.27:0/2702868891 conn(0x5569850ffb60 msgr2=0x5569851cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.104+0000 7f4344d37b80 1 -- 172.21.15.27:0/2702868891 shutdown_connections 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.104+0000 7f4344d37b80 1 -- 172.21.15.27:0/2702868891 wait complete. 2024-05-10T11:15:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.104+0000 7f4344d37b80 1 -- start start 2024-05-10T11:15:44.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.105+0000 7f4344d37b80 1 Processor -- start 2024-05-10T11:15:44.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.105+0000 7f4344d37b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d1600 0x5569851dcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.105+0000 7f4344d37b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851da340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.105+0000 7f4344d37b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d7300 0x5569851da880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.105+0000 7f4344d37b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5569851dddb0 con 0x5569851d1600 2024-05-10T11:15:44.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.105+0000 7f4344d37b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x556985089b20 con 0x5569851d7300 2024-05-10T11:15:44.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.105+0000 7f4344d37b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5569851ddf30 con 0x5569851d69e0 2024-05-10T11:15:44.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.105+0000 7f4344d37b80 10 client.0 _ll_register_callbacks cb 0x5569850cac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:44.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f4332a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851da340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f4333400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d1600 0x5569851dcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f4332a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851da340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60920/0 (socket says 172.21.15.27:60920) 2024-05-10T11:15:44.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f4332a00700 1 -- 172.21.15.27:0/3322289972 learned_addr learned my addr 172.21.15.27:0/3322289972 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f4333e00700 1 --2- 172.21.15.27:0/3322289972 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d7300 0x5569851da880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f4332a00700 1 -- 172.21.15.27:0/3322289972 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d7300 msgr2=0x5569851da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f4332a00700 1 --2- 172.21.15.27:0/3322289972 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d7300 0x5569851da880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f4332a00700 1 -- 172.21.15.27:0/3322289972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d1600 msgr2=0x5569851dcfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f4332a00700 1 --2- 172.21.15.27:0/3322289972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d1600 0x5569851dcfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f4332a00700 1 -- 172.21.15.27:0/3322289972 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569850e9420 con 0x5569851d69e0 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f4332a00700 1 --2- 172.21.15.27:0/3322289972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851da340 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f43200aa5a0 tx=0x7f43200a9fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10477cac4976a1b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f431d600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.106+0000 7f431d600700 1 -- 172.21.15.27:0/3322289972 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43200aad10 con 0x5569851d69e0 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.107+0000 7f431d600700 1 -- 172.21.15.27:0/3322289972 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f43200e3280 con 0x5569851d69e0 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.107+0000 7f4344d37b80 1 -- 172.21.15.27:0/3322289972 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43240118f0 con 0x5569851d69e0 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.107+0000 7f431d600700 1 -- 172.21.15.27:0/3322289972 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f43200aa610 con 0x5569851d69e0 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.107+0000 7f4344d37b80 10 client.25768 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:44.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.107+0000 7f4344d37b80 1 -- 172.21.15.27:0/3322289972 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5569851dddb0 con 0x5569851d69e0 2024-05-10T11:15:44.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.107+0000 7f431d600700 1 -- 172.21.15.27:0/3322289972 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43240118f0 con 0x5569851d69e0 2024-05-10T11:15:44.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.107+0000 7f431d600700 1 -- 172.21.15.27:0/3322289972 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f43200b2f50 con 0x5569851d69e0 2024-05-10T11:15:44.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.107+0000 7f4344d37b80 10 client.25768 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:44.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.107+0000 7f4344d37b80 10 client.25768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:15:44.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.108+0000 7f4344d37b80 20 client.25768 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:44.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.108+0000 7f4344d37b80 1 --2- 172.21.15.27:0/3322289972 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5569851ea9f0 0x5569851ece70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.108+0000 7f4344d37b80 4 client.25768 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:15:44.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.108+0000 7f4344d37b80 1 -- 172.21.15.27:0/3322289972 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55698519e0a0 con 0x5569851ea9f0 2024-05-10T11:15:44.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.108+0000 7f4333400700 1 --2- 172.21.15.27:0/3322289972 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5569851ea9f0 0x5569851ece70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.108+0000 7f4333400700 1 --2- 172.21.15.27:0/3322289972 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5569851ea9f0 0x5569851ece70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8567a9b69bfbd50 server_cookie=62dc423b49924c0 in_seq=0 out_seq=0 2024-05-10T11:15:44.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.108+0000 7f431d600700 10 client.25768 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.109+0000 7f431d600700 1 -- 172.21.15.27:0/3322289972 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55698519e0a0 con 0x5569851ea9f0 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.109+0000 7f431d600700 10 client.25768 handle_command_reply: tid=0 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.109+0000 7f4344d37b80 1 client.25768 shutdown 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.110+0000 7f4344d37b80 1 -- 172.21.15.27:0/3322289972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 msgr2=0x5569851da340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.110+0000 7f4344d37b80 1 --2- 172.21.15.27:0/3322289972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851da340 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f43200aa5a0 tx=0x7f43200a9fd0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.110+0000 7f4344d37b80 1 -- 172.21.15.27:0/3322289972 shutdown_connections 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.110+0000 7f4344d37b80 1 --2- 172.21.15.27:0/3322289972 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5569851ea9f0 0x5569851ece70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.110+0000 7f4344d37b80 1 --2- 172.21.15.27:0/3322289972 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d7300 0x5569851da880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.110+0000 7f4344d37b80 1 --2- 172.21.15.27:0/3322289972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d1600 0x5569851dcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.110+0000 7f4344d37b80 1 --2- 172.21.15.27:0/3322289972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851da340 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.110+0000 7f431d600700 0 client.25768 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.110+0000 7f4344d37b80 1 -- 172.21.15.27:0/3322289972 >> 172.21.15.27:0/3322289972 conn(0x5569850ffb60 msgr2=0x5569851c6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.110+0000 7f4344d37b80 1 -- 172.21.15.27:0/3322289972 shutdown_connections 2024-05-10T11:15:44.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.110+0000 7f4344d37b80 1 -- 172.21.15.27:0/3322289972 wait complete. 2024-05-10T11:15:44.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.110+0000 7f4344d37b80 20 client.25768 trim_cache size 0 max 16384 2024-05-10T11:15:44.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4344d37b80 1 Processor -- start 2024-05-10T11:15:44.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4344d37b80 1 -- start start 2024-05-10T11:15:44.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4344d37b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 0x5569851e6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4344d37b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851cc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4344d37b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d7300 0x5569851f2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4344d37b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f43341b1660 con 0x5569851d7300 2024-05-10T11:15:44.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4344d37b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f43341b3200 con 0x5569851d1600 2024-05-10T11:15:44.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4344d37b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f43341b1d30 con 0x5569851d69e0 2024-05-10T11:15:44.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4333e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d7300 0x5569851f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4333400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 0x5569851e6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4332a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4333400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 0x5569851e6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57206/0 (socket says 172.21.15.27:57206) 2024-05-10T11:15:44.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4333400700 1 -- 172.21.15.27:0/825295340 learned_addr learned my addr 172.21.15.27:0/825295340 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:44.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.128+0000 7f4333e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d7300 0x5569851f2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41186/0 (socket says 172.21.15.27:41186) 2024-05-10T11:15:44.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.129+0000 7f4332a00700 1 -- 172.21.15.27:0/825295340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 msgr2=0x5569851e6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:44.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.129+0000 7f4332a00700 1 --2- 172.21.15.27:0/825295340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 0x5569851e6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.129+0000 7f4332a00700 1 -- 172.21.15.27:0/825295340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d7300 msgr2=0x5569851f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:44.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.129+0000 7f4332a00700 1 --2- 172.21.15.27:0/825295340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d7300 0x5569851f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.129+0000 7f4332a00700 1 -- 172.21.15.27:0/825295340 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569851d4da0 con 0x5569851d69e0 2024-05-10T11:15:44.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.129+0000 7f4333400700 1 --2- 172.21.15.27:0/825295340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 0x5569851e6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:44.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.129+0000 7f4332a00700 1 --2- 172.21.15.27:0/825295340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851cc480 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f43200059d0 tx=0x7f43200b1050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a1df18fdf46ed41 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:44.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.130+0000 7f431fe00700 1 -- 172.21.15.27:0/825295340 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43200a9d80 con 0x5569851d69e0 2024-05-10T11:15:44.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.130+0000 7f431fe00700 1 -- 172.21.15.27:0/825295340 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43200aad10 con 0x5569851d69e0 2024-05-10T11:15:44.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.130+0000 7f431fe00700 1 -- 172.21.15.27:0/825295340 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320090c50 con 0x5569851d69e0 2024-05-10T11:15:44.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.130+0000 7f4344d37b80 1 -- 172.21.15.27:0/825295340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 msgr2=0x5569851cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:44.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.130+0000 7f4344d37b80 1 --2- 172.21.15.27:0/825295340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851cc480 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f43200059d0 tx=0x7f43200b1050 comp rx=0 tx=0).stop 2024-05-10T11:15:44.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.130+0000 7f4344d37b80 1 -- 172.21.15.27:0/825295340 shutdown_connections 2024-05-10T11:15:44.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.130+0000 7f4344d37b80 1 --2- 172.21.15.27:0/825295340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d7300 0x5569851f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.130+0000 7f4344d37b80 1 --2- 172.21.15.27:0/825295340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 0x5569851e6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.130+0000 7f4344d37b80 1 --2- 172.21.15.27:0/825295340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851cc480 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.130+0000 7f4344d37b80 1 -- 172.21.15.27:0/825295340 >> 172.21.15.27:0/825295340 conn(0x5569850ffb60 msgr2=0x5569851cb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:44.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.131+0000 7f4344d37b80 1 -- 172.21.15.27:0/825295340 shutdown_connections 2024-05-10T11:15:44.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.131+0000 7f4344d37b80 1 -- 172.21.15.27:0/825295340 wait complete. 2024-05-10T11:15:44.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.131+0000 7f4344d37b80 1 -- start start 2024-05-10T11:15:44.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.131+0000 7f4344d37b80 1 Processor -- start 2024-05-10T11:15:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.132+0000 7f4344d37b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 0x5569851f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.132+0000 7f4344d37b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851f2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.132+0000 7f4344d37b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d7300 0x5569851f2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.132+0000 7f4344d37b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f43341b2ad0 con 0x5569851d7300 2024-05-10T11:15:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.132+0000 7f4344d37b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f43341b2400 con 0x5569851d1600 2024-05-10T11:15:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.132+0000 7f4344d37b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f43341af600 con 0x5569851d69e0 2024-05-10T11:15:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.132+0000 7f4344d37b80 10 client.0 _ll_register_callbacks cb 0x5569851ed300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.132+0000 7f4333400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 0x5569851f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.132+0000 7f4333400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 0x5569851f1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57214/0 (socket says 172.21.15.27:57214) 2024-05-10T11:15:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.132+0000 7f4333400700 1 -- 172.21.15.27:0/136274336 learned_addr learned my addr 172.21.15.27:0/136274336 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.132+0000 7f4332a00700 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.132+0000 7f4333e00700 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d7300 0x5569851f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f4333400700 1 -- 172.21.15.27:0/136274336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 msgr2=0x5569851f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f4333400700 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851f2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f4333400700 1 -- 172.21.15.27:0/136274336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d7300 msgr2=0x5569851f2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f4333400700 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d7300 0x5569851f2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f4333400700 1 -- 172.21.15.27:0/136274336 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556985254320 con 0x5569851d1600 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f4333e00700 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d7300 0x5569851f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f4333400700 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 0x5569851f1ee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4324007580 tx=0x7f4324005d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1deb5530129ed1dd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f431d600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f431d600700 1 -- 172.21.15.27:0/136274336 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4324005190 con 0x5569851d1600 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f431d600700 1 -- 172.21.15.27:0/136274336 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f4324005330 con 0x5569851d1600 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f4344d37b80 1 -- 172.21.15.27:0/136274336 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43200aad10 con 0x5569851d1600 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f431d600700 1 -- 172.21.15.27:0/136274336 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4324036050 con 0x5569851d1600 2024-05-10T11:15:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.133+0000 7f4344d37b80 10 client.25781 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:44.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.134+0000 7f4344d37b80 1 -- 172.21.15.27:0/136274336 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556985255580 con 0x5569851d1600 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.134+0000 7f431d600700 1 -- 172.21.15.27:0/136274336 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43200aad10 con 0x5569851d1600 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.134+0000 7f431d600700 1 -- 172.21.15.27:0/136274336 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f4320022620 con 0x5569851d1600 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.134+0000 7f4344d37b80 10 client.25781 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.134+0000 7f4344d37b80 10 client.25781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.134+0000 7f4344d37b80 20 client.25781 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.134+0000 7f4344d37b80 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5569852559b0 0x556985257da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.134+0000 7f4344d37b80 4 client.25781 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.134+0000 7f4344d37b80 1 -- 172.21.15.27:0/136274336 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556985131400 con 0x5569852559b0 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.134+0000 7f4332a00700 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5569852559b0 0x556985257da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.135+0000 7f4332a00700 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5569852559b0 0x556985257da0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d45023e0e189ec4f server_cookie=f6faa35da61022a6 in_seq=0 out_seq=0 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.135+0000 7f431d600700 10 client.25781 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.135+0000 7f431d600700 1 -- 172.21.15.27:0/136274336 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556985131400 con 0x5569852559b0 2024-05-10T11:15:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.135+0000 7f431d600700 10 client.25781 handle_command_reply: tid=0 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.135+0000 7f4344d37b80 1 client.25781 shutdown 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.135+0000 7f4344d37b80 1 -- 172.21.15.27:0/136274336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 msgr2=0x5569851f1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.135+0000 7f4344d37b80 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 0x5569851f1ee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4324007580 tx=0x7f4324005d10 comp rx=0 tx=0).stop 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.136+0000 7f4344d37b80 1 -- 172.21.15.27:0/136274336 shutdown_connections 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.136+0000 7f4344d37b80 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5569852559b0 0x556985257da0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.136+0000 7f4344d37b80 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5569851d7300 0x5569851f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.136+0000 7f4344d37b80 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5569851d1600 0x5569851f1ee0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.136+0000 7f4344d37b80 1 --2- 172.21.15.27:0/136274336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5569851d69e0 0x5569851f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.136+0000 7f4344d37b80 1 -- 172.21.15.27:0/136274336 >> 172.21.15.27:0/136274336 conn(0x5569850ffb60 msgr2=0x5569851d76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.136+0000 7f431d600700 0 client.25781 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.136+0000 7f4344d37b80 1 -- 172.21.15.27:0/136274336 shutdown_connections 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.136+0000 7f4344d37b80 1 -- 172.21.15.27:0/136274336 wait complete. 2024-05-10T11:15:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.136+0000 7f4344d37b80 20 client.25781 trim_cache size 0 max 16384 2024-05-10T11:15:44.139 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:15:44.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.137+0000 7f4338e00700 1 -- 172.21.15.27:0/3732171238 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f43180b2a70 msgr2=0x7f43180b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:44.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.137+0000 7f4338e00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f43180b2a70 0x7f43180b4f30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x5569851d1fc0 tx=0x7f432c00b040 comp rx=0 tx=0).stop 2024-05-10T11:15:44.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.137+0000 7f4338e00700 1 -- 172.21.15.27:0/3732171238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4334102b80 msgr2=0x7f433419c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.137+0000 7f4338e00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4334102b80 0x7f433419c2b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4320006010 tx=0x7f43200057c0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.137+0000 7f4338e00700 1 -- 172.21.15.27:0/3732171238 shutdown_connections 2024-05-10T11:15:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.137+0000 7f4338e00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f43180b2a70 0x7f43180b4f30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.137+0000 7f4338e00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43341021d0 0x7f433419bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.137+0000 7f4338e00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f433410fd80 0x7f43341a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.137+0000 7f4338e00700 1 --2- 172.21.15.27:0/3732171238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4334102b80 0x7f433419c2b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.137+0000 7f4338e00700 1 -- 172.21.15.27:0/3732171238 >> 172.21.15.27:0/3732171238 conn(0x7f43340fde30 msgr2=0x7f4334100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.137+0000 7f4338e00700 1 -- 172.21.15.27:0/3732171238 shutdown_connections 2024-05-10T11:15:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:44.137+0000 7f4338e00700 1 -- 172.21.15.27:0/3732171238 wait complete. 2024-05-10T11:15:44.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:44 smithi027 ceph-mon[30054]: pgmap v78: 129 pgs: 129 active+clean; 17 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 913 KiB/s wr, 141 op/s 2024-05-10T11:15:44.652 INFO:tasks.workunit.client.0.smithi027.stdout: 1 42607 8.58 MB/sec warmup 29 sec latency 18.953 ms 2024-05-10T11:15:44.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:44 smithi076 ceph-mon[34152]: pgmap v78: 129 pgs: 129 active+clean; 17 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 913 KiB/s wr, 141 op/s 2024-05-10T11:15:44.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:44 smithi189 ceph-mon[34842]: pgmap v78: 129 pgs: 129 active+clean; 17 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 913 KiB/s wr, 141 op/s 2024-05-10T11:15:45.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:45 smithi027 ceph-mon[30054]: pgmap v79: 129 pgs: 129 active+clean; 22 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 1.4 MiB/s wr, 211 op/s 2024-05-10T11:15:45.652 INFO:tasks.workunit.client.0.smithi027.stdout: 1 43388 8.36 MB/sec warmup 30 sec latency 128.326 ms 2024-05-10T11:15:45.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:45 smithi076 ceph-mon[34152]: pgmap v79: 129 pgs: 129 active+clean; 22 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 1.4 MiB/s wr, 211 op/s 2024-05-10T11:15:45.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:45 smithi189 ceph-mon[34842]: pgmap v79: 129 pgs: 129 active+clean; 22 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 1.4 MiB/s wr, 211 op/s 2024-05-10T11:15:46.652 INFO:tasks.workunit.client.0.smithi027.stdout: 1 45355 8.43 MB/sec warmup 31 sec latency 249.812 ms 2024-05-10T11:15:47.652 INFO:tasks.workunit.client.0.smithi027.stdout: 1 46593 8.42 MB/sec warmup 32 sec latency 91.095 ms 2024-05-10T11:15:48.105 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:15:48.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:15:48.603 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:48.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:48 smithi027 ceph-mon[30054]: pgmap v80: 129 pgs: 129 active+clean; 22 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 942 KiB/s wr, 141 op/s 2024-05-10T11:15:48.652 INFO:tasks.workunit.client.0.smithi027.stdout: 1 48584 8.38 MB/sec warmup 33 sec latency 26.736 ms 2024-05-10T11:15:48.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:48 smithi076 ceph-mon[34152]: pgmap v80: 129 pgs: 129 active+clean; 22 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 942 KiB/s wr, 141 op/s 2024-05-10T11:15:48.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:48 smithi189 ceph-mon[34842]: pgmap v80: 129 pgs: 129 active+clean; 22 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 942 KiB/s wr, 141 op/s 2024-05-10T11:15:49.653 INFO:tasks.workunit.client.0.smithi027.stdout: 1 50090 8.49 MB/sec warmup 34 sec latency 15.055 ms 2024-05-10T11:15:50.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:50 smithi027 ceph-mon[30054]: pgmap v81: 129 pgs: 129 active+clean; 45 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-05-10T11:15:50.653 INFO:tasks.workunit.client.0.smithi027.stdout: 1 51832 8.41 MB/sec warmup 35 sec latency 31.272 ms 2024-05-10T11:15:50.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:50 smithi076 ceph-mon[34152]: pgmap v81: 129 pgs: 129 active+clean; 45 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-05-10T11:15:50.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:50 smithi189 ceph-mon[34842]: pgmap v81: 129 pgs: 129 active+clean; 45 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-05-10T11:15:50.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.966+0000 7f84e8800700 1 -- 172.21.15.27:0/721981266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84e0108460 msgr2=0x7f84e0108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:50.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.966+0000 7f84e8800700 1 --2- 172.21.15.27:0/721981266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84e0108460 0x7f84e0108840 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009990 tx=0x7f84d4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.966+0000 7f84e8800700 1 -- 172.21.15.27:0/721981266 shutdown_connections 2024-05-10T11:15:50.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.966+0000 7f84e8800700 1 --2- 172.21.15.27:0/721981266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e0103ef0 0x7f84e01103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.966+0000 7f84e8800700 1 --2- 172.21.15.27:0/721981266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84e0108460 0x7f84e0108840 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.966+0000 7f84e8800700 1 --2- 172.21.15.27:0/721981266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e0103550 0x7f84e01039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.966+0000 7f84e8800700 1 -- 172.21.15.27:0/721981266 >> 172.21.15.27:0/721981266 conn(0x7f84e007a600 msgr2=0x7f84e007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:50.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.966+0000 7f84e8800700 1 -- 172.21.15.27:0/721981266 shutdown_connections 2024-05-10T11:15:50.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.966+0000 7f84e8800700 1 -- 172.21.15.27:0/721981266 wait complete. 2024-05-10T11:15:50.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.967+0000 7f84e8800700 1 Processor -- start 2024-05-10T11:15:50.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.967+0000 7f84e8800700 1 -- start start 2024-05-10T11:15:50.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.968+0000 7f84e8800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e0103550 0x7f84e019bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:50.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.968+0000 7f84e8800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84e0103ef0 0x7f84e019c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:50.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.968+0000 7f84e8800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e0108460 0x7f84e01a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:50.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.968+0000 7f84e8800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f84e0113680 con 0x7f84e0103550 2024-05-10T11:15:50.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.968+0000 7f84e8800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f84e0113500 con 0x7f84e0103ef0 2024-05-10T11:15:50.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.968+0000 7f84e8800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f84e0113800 con 0x7f84e0108460 2024-05-10T11:15:50.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.968+0000 7f84e5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e0103550 0x7f84e019bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:50.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.968+0000 7f84e5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84e0103ef0 0x7f84e019c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:50.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.968+0000 7f84e5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84e0103ef0 0x7f84e019c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57234/0 (socket says 172.21.15.27:57234) 2024-05-10T11:15:50.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.968+0000 7f84e5400700 1 -- 172.21.15.27:0/2870501402 learned_addr learned my addr 172.21.15.27:0/2870501402 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:50.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.968+0000 7f84e6800700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e0108460 0x7f84e01a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:50.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.969+0000 7f84e5400700 1 -- 172.21.15.27:0/2870501402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e0108460 msgr2=0x7f84e01a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:50.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.969+0000 7f84e5400700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e0108460 0x7f84e01a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.969+0000 7f84e5400700 1 -- 172.21.15.27:0/2870501402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e0103550 msgr2=0x7f84e019bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:50.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.969+0000 7f84e5400700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e0103550 0x7f84e019bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.969+0000 7f84e5400700 1 -- 172.21.15.27:0/2870501402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e01a0d60 con 0x7f84e0103ef0 2024-05-10T11:15:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.969+0000 7f84e5400700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84e0103ef0 0x7f84e019c290 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f84d000e9a0 tx=0x7f84d000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.969+0000 7f84daa00700 1 -- 172.21.15.27:0/2870501402 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d000cbd0 con 0x7f84e0103ef0 2024-05-10T11:15:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.969+0000 7f84daa00700 1 -- 172.21.15.27:0/2870501402 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84d000cd70 con 0x7f84e0103ef0 2024-05-10T11:15:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.969+0000 7f84e8800700 1 -- 172.21.15.27:0/2870501402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e01a1050 con 0x7f84e0103ef0 2024-05-10T11:15:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.969+0000 7f84daa00700 1 -- 172.21.15.27:0/2870501402 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d0018680 con 0x7f84e0103ef0 2024-05-10T11:15:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.969+0000 7f84e8800700 1 -- 172.21.15.27:0/2870501402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e01a88d0 con 0x7f84e0103ef0 2024-05-10T11:15:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.974+0000 7f84f3257b80 1 Processor -- start 2024-05-10T11:15:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.974+0000 7f84daa00700 1 -- 172.21.15.27:0/2870501402 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f84d0010430 con 0x7f84e0103ef0 2024-05-10T11:15:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.975+0000 7f84f3257b80 1 -- start start 2024-05-10T11:15:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.975+0000 7f84f3257b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 0x5601cf2344c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.975+0000 7f84f3257b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf234a00 0x5601cf238510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.975+0000 7f84f3257b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf238a50 0x5601cf238eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.975+0000 7f84f3257b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5601cf186aa0 con 0x5601cf234a00 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.975+0000 7f84f3257b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5601cf1247e0 con 0x5601cf2340e0 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.975+0000 7f84f3257b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5601cf182a60 con 0x5601cf238a50 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.975+0000 7f84e6800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 0x5601cf2344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.975+0000 7f84e6800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 0x5601cf2344c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57240/0 (socket says 172.21.15.27:57240) 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.975+0000 7f84e6800700 1 -- 172.21.15.27:0/3435336881 learned_addr learned my addr 172.21.15.27:0/3435336881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.975+0000 7f84e5400700 1 --2- 172.21.15.27:0/3435336881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf238a50 0x5601cf238eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.976+0000 7f84e5e00700 1 --2- 172.21.15.27:0/3435336881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf234a00 0x5601cf238510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.976+0000 7f84e6800700 1 -- 172.21.15.27:0/3435336881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf238a50 msgr2=0x5601cf238eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.976+0000 7f84e6800700 1 --2- 172.21.15.27:0/3435336881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf238a50 0x5601cf238eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.976+0000 7f84e6800700 1 -- 172.21.15.27:0/3435336881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf234a00 msgr2=0x5601cf238510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.976+0000 7f84e6800700 1 --2- 172.21.15.27:0/3435336881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf234a00 0x5601cf238510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.976+0000 7f84e6800700 1 -- 172.21.15.27:0/3435336881 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601cf0d3950 con 0x5601cf2340e0 2024-05-10T11:15:50.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.976+0000 7f84e6800700 1 --2- 172.21.15.27:0/3435336881 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 0x5601cf2344c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f84dc009430 tx=0x7f84dc009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1672d20c3ead75cb server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:50.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.976+0000 7f84cf400700 1 -- 172.21.15.27:0/3435336881 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc013750 con 0x5601cf2340e0 2024-05-10T11:15:50.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.976+0000 7f84cf400700 1 -- 172.21.15.27:0/3435336881 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84dc013da0 con 0x5601cf2340e0 2024-05-10T11:15:50.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.976+0000 7f84cf400700 1 -- 172.21.15.27:0/3435336881 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc015e40 con 0x5601cf2340e0 2024-05-10T11:15:50.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.977+0000 7f84daa00700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f84c80b2a70 0x7f84c80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:50.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.977+0000 7f84daa00700 1 -- 172.21.15.27:0/2870501402 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f84d00e2c40 con 0x7f84e0103ef0 2024-05-10T11:15:50.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.977+0000 7f84e5e00700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f84c80b2a70 0x7f84c80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:50.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.977+0000 7f84f3257b80 1 -- 172.21.15.27:0/3435336881 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 msgr2=0x5601cf2344c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.977+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3435336881 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 0x5601cf2344c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f84dc009430 tx=0x7f84dc009920 comp rx=0 tx=0).stop 2024-05-10T11:15:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.977+0000 7f84f3257b80 1 -- 172.21.15.27:0/3435336881 shutdown_connections 2024-05-10T11:15:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.977+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3435336881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf238a50 0x5601cf238eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.977+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3435336881 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 0x5601cf2344c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.977+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3435336881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf234a00 0x5601cf238510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.977+0000 7f84f3257b80 1 -- 172.21.15.27:0/3435336881 >> 172.21.15.27:0/3435336881 conn(0x5601cf162be0 msgr2=0x5601cf22e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.978+0000 7f84f3257b80 1 -- 172.21.15.27:0/3435336881 shutdown_connections 2024-05-10T11:15:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.978+0000 7f84f3257b80 1 -- 172.21.15.27:0/3435336881 wait complete. 2024-05-10T11:15:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.978+0000 7f84e5e00700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f84c80b2a70 0x7f84c80b4f30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f84d400b2e0 tx=0x7f84d401a040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.978+0000 7f84f3257b80 1 -- start start 2024-05-10T11:15:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.979+0000 7f84f3257b80 1 Processor -- start 2024-05-10T11:15:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.979+0000 7f84f3257b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf2340e0 0x5601cf23fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.979+0000 7f84f3257b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf234a00 0x5601cf23d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84f3257b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf238a50 0x5601cf23d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84f3257b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5601cf240c60 con 0x5601cf2340e0 2024-05-10T11:15:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84f3257b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5601cf0ecba0 con 0x5601cf234a00 2024-05-10T11:15:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84f3257b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5601cf240de0 con 0x5601cf238a50 2024-05-10T11:15:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84f3257b80 10 client.0 _ll_register_callbacks cb 0x5601cf12dcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84e5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf234a00 0x5601cf23d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84e5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf234a00 0x5601cf23d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57256/0 (socket says 172.21.15.27:57256) 2024-05-10T11:15:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84e5400700 1 -- 172.21.15.27:0/3605623684 learned_addr learned my addr 172.21.15.27:0/3605623684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84e5e00700 1 --2- 172.21.15.27:0/3605623684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf2340e0 0x5601cf23fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84e5400700 1 -- 172.21.15.27:0/3605623684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf238a50 msgr2=0x5601cf23d6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:15:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84e5400700 1 --2- 172.21.15.27:0/3605623684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf238a50 0x5601cf23d6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84e5400700 1 -- 172.21.15.27:0/3605623684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf2340e0 msgr2=0x5601cf23fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84e5400700 1 --2- 172.21.15.27:0/3605623684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf2340e0 0x5601cf23fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.980+0000 7f84e5400700 1 -- 172.21.15.27:0/3605623684 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601cf14c4a0 con 0x5601cf234a00 2024-05-10T11:15:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.981+0000 7f84e5400700 1 --2- 172.21.15.27:0/3605623684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf234a00 0x5601cf23d180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f84d00a9a80 tx=0x7f84d00aa050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73b33f1eab1e993d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.981+0000 7f84c3e00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.981+0000 7f84c3e00700 1 -- 172.21.15.27:0/3605623684 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d00e6020 con 0x5601cf234a00 2024-05-10T11:15:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.981+0000 7f84c3e00700 1 -- 172.21.15.27:0/3605623684 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f84d0010930 con 0x5601cf234a00 2024-05-10T11:15:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.981+0000 7f84f3257b80 1 -- 172.21.15.27:0/3605623684 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84dc013da0 con 0x5601cf234a00 2024-05-10T11:15:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.981+0000 7f84c3e00700 1 -- 172.21.15.27:0/3605623684 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f84d00a9440 con 0x5601cf234a00 2024-05-10T11:15:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.981+0000 7f84f3257b80 10 client.25793 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.981+0000 7f84f3257b80 1 -- 172.21.15.27:0/3605623684 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5601cf24b440 con 0x5601cf234a00 2024-05-10T11:15:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.982+0000 7f84c3e00700 1 -- 172.21.15.27:0/3605623684 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84dc013da0 con 0x5601cf234a00 2024-05-10T11:15:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.982+0000 7f84c3e00700 1 -- 172.21.15.27:0/3605623684 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f84d00cc0e0 con 0x5601cf234a00 2024-05-10T11:15:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.982+0000 7f84f3257b80 10 client.25793 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.982+0000 7f84f3257b80 10 client.25793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:15:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.982+0000 7f84f3257b80 20 client.25793 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.982+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3605623684 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5601cf24d910 0x5601cf24fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.982+0000 7f84f3257b80 4 client.25793 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:15:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.982+0000 7f84f3257b80 1 -- 172.21.15.27:0/3605623684 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601cf201120 con 0x5601cf24d910 2024-05-10T11:15:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.982+0000 7f84e5e00700 1 --2- 172.21.15.27:0/3605623684 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5601cf24d910 0x5601cf24fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:50.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.983+0000 7f84e5e00700 1 --2- 172.21.15.27:0/3605623684 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5601cf24d910 0x5601cf24fd40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a301c41385db5c88 server_cookie=db3addbe7e8254d in_seq=0 out_seq=0 2024-05-10T11:15:50.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.983+0000 7f84c3e00700 10 client.25793 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:50.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.984+0000 7f84c3e00700 1 -- 172.21.15.27:0/3605623684 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5601cf201120 con 0x5601cf24d910 2024-05-10T11:15:50.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.984+0000 7f84c3e00700 10 client.25793 handle_command_reply: tid=0 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.984+0000 7f84f3257b80 1 client.25793 shutdown 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.985+0000 7f84f3257b80 1 -- 172.21.15.27:0/3605623684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf234a00 msgr2=0x5601cf23d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.985+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3605623684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf234a00 0x5601cf23d180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f84d00a9a80 tx=0x7f84d00aa050 comp rx=0 tx=0).stop 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.985+0000 7f84f3257b80 1 -- 172.21.15.27:0/3605623684 shutdown_connections 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.985+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3605623684 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5601cf24d910 0x5601cf24fd40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.985+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3605623684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf238a50 0x5601cf23d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.985+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3605623684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf2340e0 0x5601cf23fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.985+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3605623684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf234a00 0x5601cf23d180 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.985+0000 7f84c3e00700 0 client.25793 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.985+0000 7f84f3257b80 1 -- 172.21.15.27:0/3605623684 >> 172.21.15.27:0/3605623684 conn(0x5601cf162be0 msgr2=0x5601cf229330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.985+0000 7f84f3257b80 1 -- 172.21.15.27:0/3605623684 shutdown_connections 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.985+0000 7f84f3257b80 1 -- 172.21.15.27:0/3605623684 wait complete. 2024-05-10T11:15:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:50.985+0000 7f84f3257b80 20 client.25793 trim_cache size 0 max 16384 2024-05-10T11:15:51.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.012+0000 7f84f3257b80 1 Processor -- start 2024-05-10T11:15:51.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.012+0000 7f84f3257b80 1 -- start start 2024-05-10T11:15:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84f3257b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 0x5601cf2291b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84f3257b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf234a00 0x5601cf255610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84f3257b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf238a50 0x5601cf255b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84f3257b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01b1870 con 0x5601cf238a50 2024-05-10T11:15:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84f3257b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01b3410 con 0x5601cf2340e0 2024-05-10T11:15:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84f3257b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01b1f40 con 0x5601cf234a00 2024-05-10T11:15:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84e5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 0x5601cf2291b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84e5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf234a00 0x5601cf255610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84e5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 0x5601cf2291b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57258/0 (socket says 172.21.15.27:57258) 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84e5e00700 1 -- 172.21.15.27:0/3798935030 learned_addr learned my addr 172.21.15.27:0/3798935030 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84e5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf234a00 0x5601cf255610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60974/0 (socket says 172.21.15.27:60974) 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84e6800700 1 --2- 172.21.15.27:0/3798935030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf238a50 0x5601cf255b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84e5400700 1 -- 172.21.15.27:0/3798935030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 msgr2=0x5601cf2291b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84e5400700 1 --2- 172.21.15.27:0/3798935030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 0x5601cf2291b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84e5400700 1 -- 172.21.15.27:0/3798935030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf238a50 msgr2=0x5601cf255b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84e5400700 1 --2- 172.21.15.27:0/3798935030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf238a50 0x5601cf255b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.013+0000 7f84e5400700 1 -- 172.21.15.27:0/3798935030 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601cf239be0 con 0x5601cf234a00 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.014+0000 7f84e5e00700 1 --2- 172.21.15.27:0/3798935030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 0x5601cf2291b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.014+0000 7f84e5400700 1 --2- 172.21.15.27:0/3798935030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf234a00 0x5601cf255610 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f84d000cfd0 tx=0x7f84d00a5aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=762141dc2a52a0a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.014+0000 7f84cea00700 1 -- 172.21.15.27:0/3798935030 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d00a5280 con 0x5601cf234a00 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.014+0000 7f84cea00700 1 -- 172.21.15.27:0/3798935030 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84d00a9600 con 0x5601cf234a00 2024-05-10T11:15:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.014+0000 7f84cea00700 1 -- 172.21.15.27:0/3798935030 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d00a9260 con 0x5601cf234a00 2024-05-10T11:15:51.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.015+0000 7f84f3257b80 1 -- 172.21.15.27:0/3798935030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf234a00 msgr2=0x5601cf255610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:51.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.015+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3798935030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf234a00 0x5601cf255610 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f84d000cfd0 tx=0x7f84d00a5aa0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.015+0000 7f84f3257b80 1 -- 172.21.15.27:0/3798935030 shutdown_connections 2024-05-10T11:15:51.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.015+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3798935030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf238a50 0x5601cf255b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.015+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3798935030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf2340e0 0x5601cf2291b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.015+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3798935030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf234a00 0x5601cf255610 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.015+0000 7f84f3257b80 1 -- 172.21.15.27:0/3798935030 >> 172.21.15.27:0/3798935030 conn(0x5601cf162be0 msgr2=0x5601cf228aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.015+0000 7f84f3257b80 1 -- 172.21.15.27:0/3798935030 shutdown_connections 2024-05-10T11:15:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.015+0000 7f84f3257b80 1 -- 172.21.15.27:0/3798935030 wait complete. 2024-05-10T11:15:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.016+0000 7f84f3257b80 1 -- start start 2024-05-10T11:15:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.016+0000 7f84f3257b80 1 Processor -- start 2024-05-10T11:15:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84f3257b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf2340e0 0x5601cf236d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84f3257b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf234a00 0x5601cf2460d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84f3257b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf238a50 0x5601cf2466e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84f3257b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01b2ce0 con 0x5601cf2340e0 2024-05-10T11:15:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84f3257b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01b2610 con 0x5601cf238a50 2024-05-10T11:15:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84f3257b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01af810 con 0x5601cf234a00 2024-05-10T11:15:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84f3257b80 10 client.0 _ll_register_callbacks cb 0x5601cf12dcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84e6800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf238a50 0x5601cf2466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84e6800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf238a50 0x5601cf2466e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57266/0 (socket says 172.21.15.27:57266) 2024-05-10T11:15:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84e6800700 1 -- 172.21.15.27:0/3833462326 learned_addr learned my addr 172.21.15.27:0/3833462326 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84e5e00700 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf2340e0 0x5601cf236d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84e5400700 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf234a00 0x5601cf2460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84e6800700 1 -- 172.21.15.27:0/3833462326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf234a00 msgr2=0x5601cf2460d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84e6800700 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf234a00 0x5601cf2460d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84e6800700 1 -- 172.21.15.27:0/3833462326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf2340e0 msgr2=0x5601cf236d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84e6800700 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf2340e0 0x5601cf236d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84e6800700 1 -- 172.21.15.27:0/3833462326 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601cf2354f0 con 0x5601cf238a50 2024-05-10T11:15:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.017+0000 7f84e5e00700 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf2340e0 0x5601cf236d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:15:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.018+0000 7f84e6800700 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf238a50 0x5601cf2466e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f84dc013780 tx=0x7f84dc0137b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edcb7dd3de39a156 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.018+0000 7f84c3e00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.018+0000 7f84c3e00700 1 -- 172.21.15.27:0/3833462326 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc013e10 con 0x5601cf238a50 2024-05-10T11:15:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.018+0000 7f84c3e00700 1 -- 172.21.15.27:0/3833462326 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f84dc003a00 con 0x5601cf238a50 2024-05-10T11:15:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.018+0000 7f84f3257b80 1 -- 172.21.15.27:0/3833462326 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84d00a9600 con 0x5601cf238a50 2024-05-10T11:15:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.019+0000 7f84c3e00700 1 -- 172.21.15.27:0/3833462326 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f84dc02f050 con 0x5601cf238a50 2024-05-10T11:15:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.019+0000 7f84f3257b80 10 client.25799 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.019+0000 7f84f3257b80 1 -- 172.21.15.27:0/3833462326 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5601cf2b96c0 con 0x5601cf238a50 2024-05-10T11:15:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.019+0000 7f84c3e00700 1 -- 172.21.15.27:0/3833462326 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84d00a9600 con 0x5601cf238a50 2024-05-10T11:15:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.019+0000 7f84c3e00700 1 -- 172.21.15.27:0/3833462326 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f84dc005040 con 0x5601cf238a50 2024-05-10T11:15:51.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.019+0000 7f84f3257b80 10 client.25799 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:51.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.019+0000 7f84f3257b80 10 client.25799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:15:51.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.019+0000 7f84f3257b80 20 client.25799 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:51.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.019+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5601cf2b99e0 0x5601cf2bbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:51.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.020+0000 7f84f3257b80 4 client.25799 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:15:51.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.020+0000 7f84f3257b80 1 -- 172.21.15.27:0/3833462326 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5601cf194480 con 0x5601cf2b99e0 2024-05-10T11:15:51.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.020+0000 7f84e5e00700 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5601cf2b99e0 0x5601cf2bbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:51.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.020+0000 7f84e5e00700 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5601cf2b99e0 0x5601cf2bbe00 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e89b52ed3859003d server_cookie=470830413910853f in_seq=0 out_seq=0 2024-05-10T11:15:51.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.020+0000 7f84c3e00700 10 client.25799 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.021+0000 7f84c3e00700 1 -- 172.21.15.27:0/3833462326 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5601cf194480 con 0x5601cf2b99e0 2024-05-10T11:15:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.021+0000 7f84c3e00700 10 client.25799 handle_command_reply: tid=0 2024-05-10T11:15:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.021+0000 7f84f3257b80 1 client.25799 shutdown 2024-05-10T11:15:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.022+0000 7f84f3257b80 1 -- 172.21.15.27:0/3833462326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf238a50 msgr2=0x5601cf2466e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.022+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf238a50 0x5601cf2466e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f84dc013780 tx=0x7f84dc0137b0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.022+0000 7f84f3257b80 1 -- 172.21.15.27:0/3833462326 shutdown_connections 2024-05-10T11:15:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.022+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5601cf2b99e0 0x5601cf2bbe00 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.022+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5601cf238a50 0x5601cf2466e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.022+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601cf2340e0 0x5601cf236d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.022+0000 7f84f3257b80 1 --2- 172.21.15.27:0/3833462326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601cf234a00 0x5601cf2460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.022+0000 7f84c3e00700 0 client.25799 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.022+0000 7f84f3257b80 1 -- 172.21.15.27:0/3833462326 >> 172.21.15.27:0/3833462326 conn(0x5601cf162be0 msgr2=0x5601cf23b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:51.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.022+0000 7f84f3257b80 1 -- 172.21.15.27:0/3833462326 shutdown_connections 2024-05-10T11:15:51.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.022+0000 7f84f3257b80 1 -- 172.21.15.27:0/3833462326 wait complete. 2024-05-10T11:15:51.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.023+0000 7f84f3257b80 20 client.25799 trim_cache size 0 max 16384 2024-05-10T11:15:51.025 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:15:51.025 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:15:51.025 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "9be49f8d-a352-4fdf-9bba-24d307294269", 2024-05-10T11:15:51.025 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:15:51.025 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:15:51.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.024+0000 7f84e8800700 1 -- 172.21.15.27:0/2870501402 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f84c80b2a70 msgr2=0x7f84c80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:51.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.024+0000 7f84e8800700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f84c80b2a70 0x7f84c80b4f30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f84d400b2e0 tx=0x7f84d401a040 comp rx=0 tx=0).stop 2024-05-10T11:15:51.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.024+0000 7f84e8800700 1 -- 172.21.15.27:0/2870501402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84e0103ef0 msgr2=0x7f84e019c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:51.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.024+0000 7f84e8800700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84e0103ef0 0x7f84e019c290 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f84d000e9a0 tx=0x7f84d000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.025+0000 7f84e8800700 1 -- 172.21.15.27:0/2870501402 shutdown_connections 2024-05-10T11:15:51.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.025+0000 7f84e8800700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f84c80b2a70 0x7f84c80b4f30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.025+0000 7f84e8800700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e0108460 0x7f84e01a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.025+0000 7f84e8800700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e0103550 0x7f84e019bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.025+0000 7f84e8800700 1 --2- 172.21.15.27:0/2870501402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f84e0103ef0 0x7f84e019c290 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:51.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.025+0000 7f84e8800700 1 -- 172.21.15.27:0/2870501402 >> 172.21.15.27:0/2870501402 conn(0x7f84e007a600 msgr2=0x7f84e01013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:51.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.025+0000 7f84e8800700 1 -- 172.21.15.27:0/2870501402 shutdown_connections 2024-05-10T11:15:51.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:51.025+0000 7f84e8800700 1 -- 172.21.15.27:0/2870501402 wait complete. 2024-05-10T11:15:51.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:51 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:15:51.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:51 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:51.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:51 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:15:51.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:51 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:51.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:51 smithi027 ceph-mon[30054]: pgmap v82: 129 pgs: 129 active+clean; 45 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-05-10T11:15:51.653 INFO:tasks.workunit.client.0.smithi027.stdout: 1 53513 8.54 MB/sec warmup 36 sec latency 26.506 ms 2024-05-10T11:15:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:51 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:15:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:51 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:51 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:15:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:51 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:51 smithi076 ceph-mon[34152]: pgmap v82: 129 pgs: 129 active+clean; 45 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-05-10T11:15:51.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:51 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:15:51.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:51 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:51.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:51 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:15:51.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:51 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:51.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:51 smithi189 ceph-mon[34842]: pgmap v82: 129 pgs: 129 active+clean; 45 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-05-10T11:15:52.154 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:15:52.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:52 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:15:52.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:52 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:52.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:52 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:15:52.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:52 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:15:52.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:52 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:52.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:52 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:52.654 INFO:tasks.workunit.client.0.smithi027.stdout: 1 54215 8.34 MB/sec warmup 37 sec latency 95.705 ms 2024-05-10T11:15:52.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:52 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:15:52.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:52 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:52.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:52 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:15:52.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:52 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:15:52.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:52 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:52.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:52 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:52.712 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:15:52.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:52 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:15:52.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:52 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:52.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:52 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:15:52.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:52 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:15:52.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:52 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:52.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:52 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:15:53.392 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:53 smithi027 ceph-mon[30054]: pgmap v83: 129 pgs: 129 active+clean; 45 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-05-10T11:15:53.654 INFO:tasks.workunit.client.0.smithi027.stdout: 1 56586 8.48 MB/sec warmup 38 sec latency 18.088 ms 2024-05-10T11:15:53.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:53 smithi076 ceph-mon[34152]: pgmap v83: 129 pgs: 129 active+clean; 45 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-05-10T11:15:53.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:53 smithi189 ceph-mon[34842]: pgmap v83: 129 pgs: 129 active+clean; 45 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-05-10T11:15:54.654 INFO:tasks.workunit.client.0.smithi027.stdout: 1 57376 8.38 MB/sec warmup 39 sec latency 19.338 ms 2024-05-10T11:15:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.431+0000 7f2d52000700 1 -- 172.21.15.27:0/1172322526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1060a0 msgr2=0x7f2d4c1125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.431+0000 7f2d52000700 1 --2- 172.21.15.27:0/1172322526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1060a0 0x7f2d4c1125b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2d380098f0 tx=0x7f2d38009c00 comp rx=0 tx=0).stop 2024-05-10T11:15:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.432+0000 7f2d52000700 1 -- 172.21.15.27:0/1172322526 shutdown_connections 2024-05-10T11:15:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.432+0000 7f2d52000700 1 --2- 172.21.15.27:0/1172322526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1060a0 0x7f2d4c1125b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.432+0000 7f2d52000700 1 --2- 172.21.15.27:0/1172322526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c10b6e0 0x7f2d4c10bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.432+0000 7f2d52000700 1 --2- 172.21.15.27:0/1172322526 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d4c1056e0 0x7f2d4c105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.432+0000 7f2d52000700 1 -- 172.21.15.27:0/1172322526 >> 172.21.15.27:0/1172322526 conn(0x7f2d4c1011d0 msgr2=0x7f2d4c1035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.432+0000 7f2d52000700 1 -- 172.21.15.27:0/1172322526 shutdown_connections 2024-05-10T11:15:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.432+0000 7f2d52000700 1 -- 172.21.15.27:0/1172322526 wait complete. 2024-05-10T11:15:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.433+0000 7f2d52000700 1 Processor -- start 2024-05-10T11:15:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.434+0000 7f2d52000700 1 -- start start 2024-05-10T11:15:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.434+0000 7f2d52000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1056e0 0x7f2d4c19bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.434+0000 7f2d52000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d4c1060a0 0x7f2d4c19c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.434+0000 7f2d52000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c10b6e0 0x7f2d4c1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.434+0000 7f2d52000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c115970 con 0x7f2d4c10b6e0 2024-05-10T11:15:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.434+0000 7f2d52000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1157f0 con 0x7f2d4c1060a0 2024-05-10T11:15:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.434+0000 7f2d52000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c115af0 con 0x7f2d4c1056e0 2024-05-10T11:15:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.435+0000 7f2d4aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d4c1060a0 0x7f2d4c19c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.435+0000 7f2d4b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1056e0 0x7f2d4c19bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.435+0000 7f2d4aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d4c1060a0 0x7f2d4c19c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37738/0 (socket says 172.21.15.27:37738) 2024-05-10T11:15:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.435+0000 7f2d4aa00700 1 -- 172.21.15.27:0/1423924851 learned_addr learned my addr 172.21.15.27:0/1423924851 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.435+0000 7f2d4be00700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c10b6e0 0x7f2d4c1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.435+0000 7f2d4aa00700 1 -- 172.21.15.27:0/1423924851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1056e0 msgr2=0x7f2d4c19bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.435+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1056e0 0x7f2d4c19bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.435+0000 7f2d4aa00700 1 -- 172.21.15.27:0/1423924851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c10b6e0 msgr2=0x7f2d4c1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.435+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c10b6e0 0x7f2d4c1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.435+0000 7f2d4aa00700 1 -- 172.21.15.27:0/1423924851 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d4c1a0d70 con 0x7f2d4c1060a0 2024-05-10T11:15:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.435+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d4c1060a0 0x7f2d4c19c2a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00d980 tx=0x7f2d3c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.436+0000 7f2d43e00700 1 -- 172.21.15.27:0/1423924851 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c016070 con 0x7f2d4c1060a0 2024-05-10T11:15:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.436+0000 7f2d43e00700 1 -- 172.21.15.27:0/1423924851 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d3c009de0 con 0x7f2d4c1060a0 2024-05-10T11:15:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.436+0000 7f2d52000700 1 -- 172.21.15.27:0/1423924851 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d4c1a1060 con 0x7f2d4c1060a0 2024-05-10T11:15:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.436+0000 7f2d43e00700 1 -- 172.21.15.27:0/1423924851 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c00f3c0 con 0x7f2d4c1060a0 2024-05-10T11:15:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.436+0000 7f2d52000700 1 -- 172.21.15.27:0/1423924851 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d4c1a88e0 con 0x7f2d4c1060a0 2024-05-10T11:15:55.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.439+0000 7f2d43e00700 1 -- 172.21.15.27:0/1423924851 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2d3c019070 con 0x7f2d4c1060a0 2024-05-10T11:15:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.441+0000 7f2d43e00700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d300b2a70 0x7f2d300b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.441+0000 7f2d5c9a9b80 1 Processor -- start 2024-05-10T11:15:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.441+0000 7f2d43e00700 1 -- 172.21.15.27:0/1423924851 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f2d3c0e2580 con 0x7f2d4c1060a0 2024-05-10T11:15:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.441+0000 7f2d5c9a9b80 1 -- start start 2024-05-10T11:15:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.441+0000 7f2d4b400700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d300b2a70 0x7f2d300b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.442+0000 7f2d5c9a9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872060 0x561fba872440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.442+0000 7f2d5c9a9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 0x561fba876490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.442+0000 7f2d5c9a9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba8769d0 0x561fba876e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.442+0000 7f2d5c9a9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561fba7c4a20 con 0x561fba8769d0 2024-05-10T11:15:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.442+0000 7f2d5c9a9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561fba762760 con 0x561fba872060 2024-05-10T11:15:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.442+0000 7f2d5c9a9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561fba7c09e0 con 0x561fba872980 2024-05-10T11:15:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.442+0000 7f2d4aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 0x561fba876490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.442+0000 7f2d4b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872060 0x561fba872440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.442+0000 7f2d4aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 0x561fba876490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:33786/0 (socket says 172.21.15.27:33786) 2024-05-10T11:15:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.442+0000 7f2d4aa00700 1 -- 172.21.15.27:0/3842213829 learned_addr learned my addr 172.21.15.27:0/3842213829 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.443+0000 7f2d4be00700 1 --2- 172.21.15.27:0/3842213829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba8769d0 0x561fba876e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.443+0000 7f2d4b400700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d300b2a70 0x7f2d300b4f30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400f540 tx=0x7f2d4400b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.443+0000 7f2d4aa00700 1 -- 172.21.15.27:0/3842213829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872060 msgr2=0x561fba872440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.443+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/3842213829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872060 0x561fba872440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.443+0000 7f2d4aa00700 1 -- 172.21.15.27:0/3842213829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba8769d0 msgr2=0x561fba876e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.443+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/3842213829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba8769d0 0x561fba876e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.443+0000 7f2d4aa00700 1 -- 172.21.15.27:0/3842213829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fba7118d0 con 0x561fba872980 2024-05-10T11:15:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.443+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/3842213829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 0x561fba876490 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0a54b0 tx=0x7f2d3c0a9b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fea724692417b0b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.444+0000 7f2d40c00700 1 -- 172.21.15.27:0/3842213829 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0ad7e0 con 0x561fba872980 2024-05-10T11:15:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.444+0000 7f2d40c00700 1 -- 172.21.15.27:0/3842213829 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d3c0b2290 con 0x561fba872980 2024-05-10T11:15:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.444+0000 7f2d40c00700 1 -- 172.21.15.27:0/3842213829 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0b2060 con 0x561fba872980 2024-05-10T11:15:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.444+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/3842213829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 msgr2=0x561fba876490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.444+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/3842213829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 0x561fba876490 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0a54b0 tx=0x7f2d3c0a9b30 comp rx=0 tx=0).stop 2024-05-10T11:15:55.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.445+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/3842213829 shutdown_connections 2024-05-10T11:15:55.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.445+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/3842213829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba8769d0 0x561fba876e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.445+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/3842213829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872060 0x561fba872440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.445+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/3842213829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 0x561fba876490 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.445+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/3842213829 >> 172.21.15.27:0/3842213829 conn(0x561fba7a0b60 msgr2=0x561fba86c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:55.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.445+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/3842213829 shutdown_connections 2024-05-10T11:15:55.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.445+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/3842213829 wait complete. 2024-05-10T11:15:55.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.445+0000 7f2d5c9a9b80 1 -- start start 2024-05-10T11:15:55.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.446+0000 7f2d5c9a9b80 1 Processor -- start 2024-05-10T11:15:55.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.446+0000 7f2d5c9a9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872060 0x561fba87ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.446+0000 7f2d5c9a9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872980 0x561fba87b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.446+0000 7f2d5c9a9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba8769d0 0x561fba87b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.446+0000 7f2d5c9a9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561fba87ebe0 con 0x561fba872980 2024-05-10T11:15:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.446+0000 7f2d5c9a9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561fba72ab20 con 0x561fba872060 2024-05-10T11:15:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.446+0000 7f2d5c9a9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561fba87ed60 con 0x561fba8769d0 2024-05-10T11:15:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.446+0000 7f2d5c9a9b80 10 client.0 _ll_register_callbacks cb 0x561fba76bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.447+0000 7f2d4b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872060 0x561fba87ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.447+0000 7f2d4b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872060 0x561fba87ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37756/0 (socket says 172.21.15.27:37756) 2024-05-10T11:15:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.447+0000 7f2d4b400700 1 -- 172.21.15.27:0/4015101801 learned_addr learned my addr 172.21.15.27:0/4015101801 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.447+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872980 0x561fba87b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.447+0000 7f2d4be00700 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba8769d0 0x561fba87b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.447+0000 7f2d4aa00700 1 -- 172.21.15.27:0/4015101801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba8769d0 msgr2=0x561fba87b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.447+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba8769d0 0x561fba87b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.447+0000 7f2d4aa00700 1 -- 172.21.15.27:0/4015101801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872060 msgr2=0x561fba87ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.447+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872060 0x561fba87ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.447+0000 7f2d4aa00700 1 -- 172.21.15.27:0/4015101801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fba78a420 con 0x561fba872980 2024-05-10T11:15:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.447+0000 7f2d4b400700 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872060 0x561fba87ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:15:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.448+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872980 0x561fba87b2c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0a5460 tx=0x7f2d3c091040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39bb4bc887ec3ff9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.448+0000 7f2d35600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:15:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.448+0000 7f2d35600700 1 -- 172.21.15.27:0/4015101801 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0c2070 con 0x561fba872980 2024-05-10T11:15:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.448+0000 7f2d35600700 1 -- 172.21.15.27:0/4015101801 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f2d3c0a9690 con 0x561fba872980 2024-05-10T11:15:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.448+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/4015101801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d3c0b2290 con 0x561fba872980 2024-05-10T11:15:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.449+0000 7f2d35600700 1 -- 172.21.15.27:0/4015101801 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2d3c0e2340 con 0x561fba872980 2024-05-10T11:15:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.449+0000 7f2d5c9a9b80 10 client.15648 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.449+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/4015101801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561fba8894c0 con 0x561fba872980 2024-05-10T11:15:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.449+0000 7f2d35600700 1 -- 172.21.15.27:0/4015101801 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d3c0b2290 con 0x561fba872980 2024-05-10T11:15:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.449+0000 7f2d35600700 1 -- 172.21.15.27:0/4015101801 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f2d3c009930 con 0x561fba872980 2024-05-10T11:15:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.449+0000 7f2d5c9a9b80 10 client.15648 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.449+0000 7f2d5c9a9b80 10 client.15648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:15:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.449+0000 7f2d5c9a9b80 20 client.15648 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.449+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x561fba88b990 0x561fba88dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.449+0000 7f2d5c9a9b80 4 client.15648 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:15:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.449+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/4015101801 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fba83f0a0 con 0x561fba88b990 2024-05-10T11:15:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.450+0000 7f2d4b400700 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x561fba88b990 0x561fba88dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.450+0000 7f2d4b400700 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x561fba88b990 0x561fba88dd80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d15056809dc319 server_cookie=beefe9dbf607feb in_seq=0 out_seq=0 2024-05-10T11:15:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.450+0000 7f2d35600700 10 client.15648 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.452+0000 7f2d35600700 1 -- 172.21.15.27:0/4015101801 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561fba83f0a0 con 0x561fba88b990 2024-05-10T11:15:55.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.452+0000 7f2d35600700 10 client.15648 handle_command_reply: tid=0 2024-05-10T11:15:55.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.452+0000 7f2d5c9a9b80 1 client.15648 shutdown 2024-05-10T11:15:55.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.452+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/4015101801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872980 msgr2=0x561fba87b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:55.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.452+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872980 0x561fba87b2c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0a5460 tx=0x7f2d3c091040 comp rx=0 tx=0).stop 2024-05-10T11:15:55.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.452+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/4015101801 shutdown_connections 2024-05-10T11:15:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.452+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x561fba88b990 0x561fba88dd80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.452+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba8769d0 0x561fba87b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.452+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872060 0x561fba87ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.452+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/4015101801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872980 0x561fba87b2c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.452+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/4015101801 >> 172.21.15.27:0/4015101801 conn(0x561fba7a0b60 msgr2=0x561fba86c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.452+0000 7f2d35600700 0 client.15648 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.453+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/4015101801 shutdown_connections 2024-05-10T11:15:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.453+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/4015101801 wait complete. 2024-05-10T11:15:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.453+0000 7f2d5c9a9b80 20 client.15648 trim_cache size 0 max 16384 2024-05-10T11:15:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.483+0000 7f2d5c9a9b80 1 Processor -- start 2024-05-10T11:15:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.483+0000 7f2d5c9a9b80 1 -- start start 2024-05-10T11:15:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.483+0000 7f2d5c9a9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872060 0x561fba867130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.483+0000 7f2d5c9a9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 0x561fba890da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.483+0000 7f2d5c9a9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba8769d0 0x561fba8912e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.483+0000 7f2d5c9a9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1b18f0 con 0x561fba872060 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.483+0000 7f2d5c9a9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1b3490 con 0x561fba8769d0 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.483+0000 7f2d5c9a9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1b1fc0 con 0x561fba872980 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.484+0000 7f2d4aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 0x561fba890da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.484+0000 7f2d4b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872060 0x561fba867130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.484+0000 7f2d4be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba8769d0 0x561fba8912e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.484+0000 7f2d4aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 0x561fba890da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:33800/0 (socket says 172.21.15.27:33800) 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.484+0000 7f2d4aa00700 1 -- 172.21.15.27:0/884738952 learned_addr learned my addr 172.21.15.27:0/884738952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.484+0000 7f2d4aa00700 1 -- 172.21.15.27:0/884738952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba8769d0 msgr2=0x561fba8912e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.484+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/884738952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba8769d0 0x561fba8912e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.484+0000 7f2d4aa00700 1 -- 172.21.15.27:0/884738952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872060 msgr2=0x561fba867130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.484+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/884738952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872060 0x561fba867130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.484+0000 7f2d4aa00700 1 -- 172.21.15.27:0/884738952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fba872690 con 0x561fba872980 2024-05-10T11:15:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.484+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/884738952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 0x561fba890da0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0a95f0 tx=0x7f2d3c0a9b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b639dc6f48898ed7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.485+0000 7f2d37e00700 1 -- 172.21.15.27:0/884738952 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0ca020 con 0x561fba872980 2024-05-10T11:15:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.485+0000 7f2d37e00700 1 -- 172.21.15.27:0/884738952 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d3c0ad7e0 con 0x561fba872980 2024-05-10T11:15:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.485+0000 7f2d37e00700 1 -- 172.21.15.27:0/884738952 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0c2070 con 0x561fba872980 2024-05-10T11:15:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.485+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/884738952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 msgr2=0x561fba890da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.486+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/884738952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 0x561fba890da0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0a95f0 tx=0x7f2d3c0a9b30 comp rx=0 tx=0).stop 2024-05-10T11:15:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.486+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/884738952 shutdown_connections 2024-05-10T11:15:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.486+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/884738952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba8769d0 0x561fba8912e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.486+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/884738952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872060 0x561fba867130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.486+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/884738952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba872980 0x561fba890da0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.486+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/884738952 >> 172.21.15.27:0/884738952 conn(0x561fba7a0b60 msgr2=0x561fba866a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.486+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/884738952 shutdown_connections 2024-05-10T11:15:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.486+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/884738952 wait complete. 2024-05-10T11:15:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.486+0000 7f2d5c9a9b80 1 -- start start 2024-05-10T11:15:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.487+0000 7f2d5c9a9b80 1 Processor -- start 2024-05-10T11:15:55.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.487+0000 7f2d5c9a9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872060 0x561fba88e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.487+0000 7f2d5c9a9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872980 0x561fba88ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.487+0000 7f2d5c9a9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba8769d0 0x561fba88f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.487+0000 7f2d5c9a9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1b2d60 con 0x561fba872060 2024-05-10T11:15:55.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.487+0000 7f2d5c9a9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1b2690 con 0x561fba872980 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.487+0000 7f2d5c9a9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1af890 con 0x561fba8769d0 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.487+0000 7f2d5c9a9b80 10 client.0 _ll_register_callbacks cb 0x561fba76bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.487+0000 7f2d4aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872980 0x561fba88ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.487+0000 7f2d4b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872060 0x561fba88e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.487+0000 7f2d4aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872980 0x561fba88ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37774/0 (socket says 172.21.15.27:37774) 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.487+0000 7f2d4aa00700 1 -- 172.21.15.27:0/2804573978 learned_addr learned my addr 172.21.15.27:0/2804573978 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.488+0000 7f2d4be00700 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba8769d0 0x561fba88f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.488+0000 7f2d4aa00700 1 -- 172.21.15.27:0/2804573978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba8769d0 msgr2=0x561fba88f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.488+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba8769d0 0x561fba88f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.488+0000 7f2d4aa00700 1 -- 172.21.15.27:0/2804573978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872060 msgr2=0x561fba88e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.488+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872060 0x561fba88e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.488+0000 7f2d4aa00700 1 -- 172.21.15.27:0/2804573978 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fba8762d0 con 0x561fba872980 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.488+0000 7f2d4b400700 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872060 0x561fba88e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:15:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.488+0000 7f2d4aa00700 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872980 0x561fba88ec60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c009520 tx=0x7f2d3c0b20b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d789ff9c6bb46651 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:15:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.488+0000 7f2d35600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:15:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.488+0000 7f2d35600700 1 -- 172.21.15.27:0/2804573978 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0c2070 con 0x561fba872980 2024-05-10T11:15:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.488+0000 7f2d35600700 1 -- 172.21.15.27:0/2804573978 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f2d3c0a98a0 con 0x561fba872980 2024-05-10T11:15:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.488+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/2804573978 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d3c0ad7e0 con 0x561fba872980 2024-05-10T11:15:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.489+0000 7f2d35600700 1 -- 172.21.15.27:0/2804573978 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2d3c0ca040 con 0x561fba872980 2024-05-10T11:15:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.489+0000 7f2d5c9a9b80 10 client.25817 fetch_fsmap learned FSMap version 31 2024-05-10T11:15:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.489+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/2804573978 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561fba8762d0 con 0x561fba872980 2024-05-10T11:15:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.489+0000 7f2d35600700 1 -- 172.21.15.27:0/2804573978 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d3c0ad7e0 con 0x561fba872980 2024-05-10T11:15:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.489+0000 7f2d35600700 1 -- 172.21.15.27:0/2804573978 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f2d3c0a4e20 con 0x561fba872980 2024-05-10T11:15:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.489+0000 7f2d5c9a9b80 10 client.25817 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:15:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.489+0000 7f2d5c9a9b80 10 client.25817 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:15:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.489+0000 7f2d5c9a9b80 20 client.25817 populate_metadata read hostname 'smithi027' 2024-05-10T11:15:55.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.489+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x561fba8f6290 0x561fba8f86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:15:55.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.489+0000 7f2d5c9a9b80 4 client.25817 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:15:55.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.489+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/2804573978 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561fba7d2400 con 0x561fba8f6290 2024-05-10T11:15:55.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.490+0000 7f2d4b400700 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x561fba8f6290 0x561fba8f86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:15:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.490+0000 7f2d4b400700 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x561fba8f6290 0x561fba8f86b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6908e5936a0e8b1f server_cookie=bc9e5aac0e2f4a08 in_seq=0 out_seq=0 2024-05-10T11:15:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.490+0000 7f2d35600700 10 client.25817 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d35600700 1 -- 172.21.15.27:0/2804573978 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561fba7d2400 con 0x561fba8f6290 2024-05-10T11:15:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d35600700 10 client.25817 handle_command_reply: tid=0 2024-05-10T11:15:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d5c9a9b80 1 client.25817 shutdown 2024-05-10T11:15:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/2804573978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872980 msgr2=0x561fba88ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872980 0x561fba88ec60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c009520 tx=0x7f2d3c0b20b0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/2804573978 shutdown_connections 2024-05-10T11:15:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x561fba8f6290 0x561fba8f86b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fba8769d0 0x561fba88f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fba872060 0x561fba88e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d5c9a9b80 1 --2- 172.21.15.27:0/2804573978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fba872980 0x561fba88ec60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d35600700 0 client.25817 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:15:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/2804573978 >> 172.21.15.27:0/2804573978 conn(0x561fba7a0b60 msgr2=0x561fba8672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/2804573978 shutdown_connections 2024-05-10T11:15:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d5c9a9b80 1 -- 172.21.15.27:0/2804573978 wait complete. 2024-05-10T11:15:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.491+0000 7f2d5c9a9b80 20 client.25817 trim_cache size 0 max 16384 2024-05-10T11:15:55.495 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:15:55.495 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:15:55.496 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:15:55.496 INFO:teuthology.orchestra.run.smithi027.stdout: "9be49f8d-a352-4fdf-9bba-24d307294269": { 2024-05-10T11:15:55.496 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:15:55.496 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "9be49f8d-a352-4fdf-9bba-24d307294269", 2024-05-10T11:15:55.496 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:15:55.496 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:15:55.496 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:15:55.496 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:15:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.492+0000 7f2d52000700 1 -- 172.21.15.27:0/1423924851 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d300b2a70 msgr2=0x7f2d300b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.492+0000 7f2d52000700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d300b2a70 0x7f2d300b4f30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400f540 tx=0x7f2d4400b040 comp rx=0 tx=0).stop 2024-05-10T11:15:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.492+0000 7f2d52000700 1 -- 172.21.15.27:0/1423924851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d4c1060a0 msgr2=0x7f2d4c19c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:15:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.492+0000 7f2d52000700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d4c1060a0 0x7f2d4c19c2a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00d980 tx=0x7f2d3c00dc90 comp rx=0 tx=0).stop 2024-05-10T11:15:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.492+0000 7f2d52000700 1 -- 172.21.15.27:0/1423924851 shutdown_connections 2024-05-10T11:15:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.492+0000 7f2d52000700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d300b2a70 0x7f2d300b4f30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.492+0000 7f2d52000700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1056e0 0x7f2d4c19bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.492+0000 7f2d52000700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c10b6e0 0x7f2d4c1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.492+0000 7f2d52000700 1 --2- 172.21.15.27:0/1423924851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d4c1060a0 0x7f2d4c19c2a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:15:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.492+0000 7f2d52000700 1 -- 172.21.15.27:0/1423924851 >> 172.21.15.27:0/1423924851 conn(0x7f2d4c1011d0 msgr2=0x7f2d4c1018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:15:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.493+0000 7f2d52000700 1 -- 172.21.15.27:0/1423924851 shutdown_connections 2024-05-10T11:15:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:15:55.493+0000 7f2d52000700 1 -- 172.21.15.27:0/1423924851 wait complete. 2024-05-10T11:15:55.654 INFO:tasks.workunit.client.0.smithi027.stdout: 1 59361 8.37 MB/sec warmup 40 sec latency 29.314 ms 2024-05-10T11:15:56.503 INFO:tasks.cephfs.filesystem:scrub status for tag:9be49f8d-a352-4fdf-9bba-24d307294269 - {'path': '/', 'tag': '9be49f8d-a352-4fdf-9bba-24d307294269', 'options': 'recursive,force'} 2024-05-10T11:15:56.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:56 smithi027 ceph-mon[30054]: pgmap v84: 129 pgs: 129 active+clean; 47 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 3.6 MiB/s wr, 228 op/s 2024-05-10T11:15:56.654 INFO:tasks.workunit.client.0.smithi027.stdout: 1 60665 8.44 MB/sec warmup 41 sec latency 30.483 ms 2024-05-10T11:15:56.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:56 smithi076 ceph-mon[34152]: pgmap v84: 129 pgs: 129 active+clean; 47 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 3.6 MiB/s wr, 228 op/s 2024-05-10T11:15:56.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:56 smithi189 ceph-mon[34842]: pgmap v84: 129 pgs: 129 active+clean; 47 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 3.6 MiB/s wr, 228 op/s 2024-05-10T11:15:57.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:57 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:15:57.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:57 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:15:57.654 INFO:tasks.workunit.client.0.smithi027.stdout: 1 61684 8.34 MB/sec warmup 42 sec latency 39.679 ms 2024-05-10T11:15:57.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:57 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:15:57.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:57 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:15:57.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:57 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:15:57.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:57 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:15:58.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:58 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:15:58.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:58 smithi027 ceph-mon[30054]: pgmap v85: 129 pgs: 129 active+clean; 47 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 3.1 MiB/s wr, 158 op/s 2024-05-10T11:15:58.655 INFO:tasks.workunit.client.0.smithi027.stdout: 1 63961 8.46 MB/sec warmup 43 sec latency 10.304 ms 2024-05-10T11:15:58.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:58 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:15:58.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:58 smithi076 ceph-mon[34152]: pgmap v85: 129 pgs: 129 active+clean; 47 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 3.1 MiB/s wr, 158 op/s 2024-05-10T11:15:58.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:58 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:15:58.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:58 smithi189 ceph-mon[34842]: pgmap v85: 129 pgs: 129 active+clean; 47 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 3.1 MiB/s wr, 158 op/s 2024-05-10T11:15:59.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:15:59 smithi027 ceph-mon[30054]: pgmap v86: 129 pgs: 129 active+clean; 50 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 229 op/s 2024-05-10T11:15:59.655 INFO:tasks.workunit.client.0.smithi027.stdout: 1 64593 8.30 MB/sec warmup 44 sec latency 40.818 ms 2024-05-10T11:15:59.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:15:59 smithi076 ceph-mon[34152]: pgmap v86: 129 pgs: 129 active+clean; 50 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 229 op/s 2024-05-10T11:15:59.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:15:59 smithi189 ceph-mon[34842]: pgmap v86: 129 pgs: 129 active+clean; 50 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 229 op/s 2024-05-10T11:16:00.655 INFO:tasks.workunit.client.0.smithi027.stdout: 1 66466 8.29 MB/sec warmup 45 sec latency 35.636 ms 2024-05-10T11:16:01.655 INFO:tasks.workunit.client.0.smithi027.stdout: 1 67128 8.25 MB/sec warmup 46 sec latency 85.497 ms 2024-05-10T11:16:02.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:02 smithi027 ceph-mon[30054]: pgmap v87: 129 pgs: 129 active+clean; 50 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-05-10T11:16:02.655 INFO:tasks.workunit.client.0.smithi027.stdout: 1 67800 8.18 MB/sec warmup 47 sec latency 70.707 ms 2024-05-10T11:16:02.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:02 smithi076 ceph-mon[34152]: pgmap v87: 129 pgs: 129 active+clean; 50 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-05-10T11:16:02.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:02 smithi189 ceph-mon[34842]: pgmap v87: 129 pgs: 129 active+clean; 50 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-05-10T11:16:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:03 smithi027 ceph-mon[30054]: pgmap v88: 129 pgs: 129 active+clean; 50 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-05-10T11:16:03.655 INFO:tasks.workunit.client.0.smithi027.stdout: 1 68945 8.11 MB/sec warmup 48 sec latency 32.041 ms 2024-05-10T11:16:03.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:03 smithi076 ceph-mon[34152]: pgmap v88: 129 pgs: 129 active+clean; 50 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-05-10T11:16:03.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:03 smithi189 ceph-mon[34842]: pgmap v88: 129 pgs: 129 active+clean; 50 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-05-10T11:16:04.655 INFO:tasks.workunit.client.0.smithi027.stdout: 1 70968 8.20 MB/sec warmup 49 sec latency 12.409 ms 2024-05-10T11:16:05.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:05 smithi027 ceph-mon[30054]: pgmap v89: 129 pgs: 129 active+clean; 53 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 250 op/s 2024-05-10T11:16:05.655 INFO:tasks.workunit.client.0.smithi027.stdout: 1 71565 8.07 MB/sec warmup 50 sec latency 20.989 ms 2024-05-10T11:16:05.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:05 smithi076 ceph-mon[34152]: pgmap v89: 129 pgs: 129 active+clean; 53 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 250 op/s 2024-05-10T11:16:05.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:05 smithi189 ceph-mon[34842]: pgmap v89: 129 pgs: 129 active+clean; 53 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 250 op/s 2024-05-10T11:16:06.655 INFO:tasks.workunit.client.0.smithi027.stdout: 1 73426 8.05 MB/sec warmup 51 sec latency 37.119 ms 2024-05-10T11:16:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:07 smithi027 ceph-mon[30054]: pgmap v90: 129 pgs: 129 active+clean; 53 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-05-10T11:16:07.655 INFO:tasks.workunit.client.0.smithi027.stdout: 1 74781 8.12 MB/sec warmup 52 sec latency 82.808 ms 2024-05-10T11:16:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:07 smithi076 ceph-mon[34152]: pgmap v90: 129 pgs: 129 active+clean; 53 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-05-10T11:16:07.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:07 smithi189 ceph-mon[34842]: pgmap v90: 129 pgs: 129 active+clean; 53 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-05-10T11:16:08.656 INFO:tasks.workunit.client.0.smithi027.stdout: 1 75411 7.98 MB/sec warmup 53 sec latency 38.623 ms 2024-05-10T11:16:09.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:09 smithi027 ceph-mon[30054]: pgmap v91: 129 pgs: 129 active+clean; 56 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 248 op/s 2024-05-10T11:16:09.656 INFO:tasks.workunit.client.0.smithi027.stdout: 1 77339 8.03 MB/sec warmup 54 sec latency 14.570 ms 2024-05-10T11:16:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:09 smithi076 ceph-mon[34152]: pgmap v91: 129 pgs: 129 active+clean; 56 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 248 op/s 2024-05-10T11:16:09.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:09 smithi189 ceph-mon[34842]: pgmap v91: 129 pgs: 129 active+clean; 56 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 248 op/s 2024-05-10T11:16:10.656 INFO:tasks.workunit.client.0.smithi027.stdout: 1 78414 8.02 MB/sec warmup 55 sec latency 19.241 ms 2024-05-10T11:16:11.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:11 smithi027 ceph-mon[30054]: pgmap v92: 129 pgs: 129 active+clean; 56 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-05-10T11:16:11.656 INFO:tasks.workunit.client.0.smithi027.stdout: 1 79270 7.92 MB/sec warmup 56 sec latency 32.983 ms 2024-05-10T11:16:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:11 smithi076 ceph-mon[34152]: pgmap v92: 129 pgs: 129 active+clean; 56 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-05-10T11:16:11.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:11 smithi189 ceph-mon[34842]: pgmap v92: 129 pgs: 129 active+clean; 56 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-05-10T11:16:12.656 INFO:tasks.workunit.client.0.smithi027.stdout: 1 81098 7.97 MB/sec warmup 57 sec latency 47.383 ms 2024-05-10T11:16:13.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:13 smithi027 ceph-mon[30054]: pgmap v93: 129 pgs: 129 active+clean; 56 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-05-10T11:16:13.656 INFO:tasks.workunit.client.0.smithi027.stdout: 1 82043 7.94 MB/sec warmup 58 sec latency 23.601 ms 2024-05-10T11:16:13.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:13 smithi076 ceph-mon[34152]: pgmap v93: 129 pgs: 129 active+clean; 56 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-05-10T11:16:13.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:13 smithi189 ceph-mon[34842]: pgmap v93: 129 pgs: 129 active+clean; 56 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-05-10T11:16:14.657 INFO:tasks.workunit.client.0.smithi027.stdout: 1 83194 7.90 MB/sec warmup 59 sec latency 31.575 ms 2024-05-10T11:16:15.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:15 smithi027 ceph-mon[30054]: pgmap v94: 129 pgs: 129 active+clean; 58 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 255 op/s 2024-05-10T11:16:15.657 INFO:tasks.workunit.client.0.smithi027.stdout: 1 85170 7.97 MB/sec warmup 60 sec latency 15.914 ms 2024-05-10T11:16:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:15 smithi076 ceph-mon[34152]: pgmap v94: 129 pgs: 129 active+clean; 58 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 255 op/s 2024-05-10T11:16:15.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:15 smithi189 ceph-mon[34842]: pgmap v94: 129 pgs: 129 active+clean; 58 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 255 op/s 2024-05-10T11:16:16.657 INFO:tasks.workunit.client.0.smithi027.stdout: 1 85779 7.87 MB/sec warmup 61 sec latency 33.374 ms 2024-05-10T11:16:17.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:17 smithi027 ceph-mon[30054]: pgmap v95: 129 pgs: 129 active+clean; 58 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 161 op/s 2024-05-10T11:16:17.657 INFO:tasks.workunit.client.0.smithi027.stdout: 1 87554 7.85 MB/sec warmup 62 sec latency 126.359 ms 2024-05-10T11:16:17.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:17 smithi076 ceph-mon[34152]: pgmap v95: 129 pgs: 129 active+clean; 58 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 161 op/s 2024-05-10T11:16:17.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:17 smithi189 ceph-mon[34842]: pgmap v95: 129 pgs: 129 active+clean; 58 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 161 op/s 2024-05-10T11:16:18.658 INFO:tasks.workunit.client.0.smithi027.stdout: 1 89010 7.92 MB/sec warmup 63 sec latency 98.809 ms 2024-05-10T11:16:19.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:19 smithi027 ceph-mon[30054]: pgmap v96: 129 pgs: 129 active+clean; 64 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 246 op/s 2024-05-10T11:16:19.657 INFO:tasks.workunit.client.0.smithi027.stdout: 1 90046 7.84 MB/sec warmup 64 sec latency 13.028 ms 2024-05-10T11:16:19.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:19 smithi076 ceph-mon[34152]: pgmap v96: 129 pgs: 129 active+clean; 64 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 246 op/s 2024-05-10T11:16:19.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:19 smithi189 ceph-mon[34842]: pgmap v96: 129 pgs: 129 active+clean; 64 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 246 op/s 2024-05-10T11:16:20.658 INFO:tasks.workunit.client.0.smithi027.stdout: 1 92386 7.95 MB/sec warmup 65 sec latency 9.462 ms 2024-05-10T11:16:21.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:21 smithi027 ceph-mon[30054]: pgmap v97: 129 pgs: 129 active+clean; 64 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 163 op/s 2024-05-10T11:16:21.659 INFO:tasks.workunit.client.0.smithi027.stdout: 1 93004 7.86 MB/sec warmup 66 sec latency 26.950 ms 2024-05-10T11:16:21.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:21 smithi076 ceph-mon[34152]: pgmap v97: 129 pgs: 129 active+clean; 64 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 163 op/s 2024-05-10T11:16:21.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:21 smithi189 ceph-mon[34842]: pgmap v97: 129 pgs: 129 active+clean; 64 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 163 op/s 2024-05-10T11:16:22.658 INFO:tasks.workunit.client.0.smithi027.stdout: 1 95163 7.91 MB/sec warmup 67 sec latency 58.938 ms 2024-05-10T11:16:23.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:23 smithi027 ceph-mon[30054]: pgmap v98: 129 pgs: 129 active+clean; 64 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 163 op/s 2024-05-10T11:16:23.658 INFO:tasks.workunit.client.0.smithi027.stdout: 1 96396 7.90 MB/sec warmup 68 sec latency 15.680 ms 2024-05-10T11:16:23.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:23 smithi076 ceph-mon[34152]: pgmap v98: 129 pgs: 129 active+clean; 64 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 163 op/s 2024-05-10T11:16:23.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:23 smithi189 ceph-mon[34842]: pgmap v98: 129 pgs: 129 active+clean; 64 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 163 op/s 2024-05-10T11:16:24.658 INFO:tasks.workunit.client.0.smithi027.stdout: 1 98595 7.95 MB/sec warmup 69 sec latency 34.818 ms 2024-05-10T11:16:25.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:25 smithi027 ceph-mon[30054]: pgmap v99: 129 pgs: 129 active+clean; 67 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 252 op/s 2024-05-10T11:16:25.658 INFO:tasks.workunit.client.0.smithi027.stdout: 1 99796 7.95 MB/sec warmup 70 sec latency 15.220 ms 2024-05-10T11:16:25.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:25 smithi076 ceph-mon[34152]: pgmap v99: 129 pgs: 129 active+clean; 67 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 252 op/s 2024-05-10T11:16:25.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:25 smithi189 ceph-mon[34842]: pgmap v99: 129 pgs: 129 active+clean; 67 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 252 op/s 2024-05-10T11:16:26.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:16:26.658 INFO:tasks.workunit.client.0.smithi027.stdout: 1 101738 7.94 MB/sec warmup 71 sec latency 28.827 ms 2024-05-10T11:16:26.995 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:16:27.406 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:27 smithi027 ceph-mon[30054]: pgmap v100: 129 pgs: 129 active+clean; 67 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 173 op/s 2024-05-10T11:16:27.659 INFO:tasks.workunit.client.0.smithi027.stdout: 1 103219 7.99 MB/sec warmup 72 sec latency 50.860 ms 2024-05-10T11:16:27.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:27 smithi076 ceph-mon[34152]: pgmap v100: 129 pgs: 129 active+clean; 67 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 173 op/s 2024-05-10T11:16:27.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:27 smithi189 ceph-mon[34842]: pgmap v100: 129 pgs: 129 active+clean; 67 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 173 op/s 2024-05-10T11:16:28.658 INFO:tasks.workunit.client.0.smithi027.stdout: 1 104440 7.95 MB/sec warmup 73 sec latency 42.546 ms 2024-05-10T11:16:29.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.490+0000 7f506fc00700 1 -- 172.21.15.27:0/4188172036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5068075ff0 msgr2=0x7f506810f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:29.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.490+0000 7f506fc00700 1 --2- 172.21.15.27:0/4188172036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5068075ff0 0x7f506810f960 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5050009990 tx=0x7f5050009ca0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.490+0000 7f506fc00700 1 -- 172.21.15.27:0/4188172036 shutdown_connections 2024-05-10T11:16:29.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.490+0000 7f506fc00700 1 --2- 172.21.15.27:0/4188172036 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f506810fea0 0x7f5068112290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.490+0000 7f506fc00700 1 --2- 172.21.15.27:0/4188172036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50680756d0 0x7f5068075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.490+0000 7f506fc00700 1 --2- 172.21.15.27:0/4188172036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5068075ff0 0x7f506810f960 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.490+0000 7f506fc00700 1 -- 172.21.15.27:0/4188172036 >> 172.21.15.27:0/4188172036 conn(0x7f506807a600 msgr2=0x7f506807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:29.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.490+0000 7f506fc00700 1 -- 172.21.15.27:0/4188172036 shutdown_connections 2024-05-10T11:16:29.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.490+0000 7f506fc00700 1 -- 172.21.15.27:0/4188172036 wait complete. 2024-05-10T11:16:29.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:29 smithi027 ceph-mon[30054]: pgmap v101: 129 pgs: 129 active+clean; 70 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 257 op/s 2024-05-10T11:16:29.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.491+0000 7f506fc00700 1 Processor -- start 2024-05-10T11:16:29.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.492+0000 7f506fc00700 1 -- start start 2024-05-10T11:16:29.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.492+0000 7f506fc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50680756d0 0x7f506819be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.492+0000 7f506fc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5068075ff0 0x7f506819c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.492+0000 7f506fc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f506810fea0 0x7f50681a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.492+0000 7f506fc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f50681158d0 con 0x7f5068075ff0 2024-05-10T11:16:29.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.492+0000 7f506fc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5068115750 con 0x7f506810fea0 2024-05-10T11:16:29.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.492+0000 7f506fc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5068115a50 con 0x7f50680756d0 2024-05-10T11:16:29.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.493+0000 7f506d200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50680756d0 0x7f506819be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.493+0000 7f506d200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50680756d0 0x7f506819be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41628/0 (socket says 172.21.15.27:41628) 2024-05-10T11:16:29.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.493+0000 7f506d200700 1 -- 172.21.15.27:0/3490016820 learned_addr learned my addr 172.21.15.27:0/3490016820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:29.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.493+0000 7f505fe00700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5068075ff0 0x7f506819c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.493+0000 7f506dc00700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f506810fea0 0x7f50681a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.493+0000 7f506d200700 1 -- 172.21.15.27:0/3490016820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f506810fea0 msgr2=0x7f50681a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:29.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.493+0000 7f506d200700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f506810fea0 0x7f50681a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.493+0000 7f506d200700 1 -- 172.21.15.27:0/3490016820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5068075ff0 msgr2=0x7f506819c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:29.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.493+0000 7f506d200700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5068075ff0 0x7f506819c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.493+0000 7f506d200700 1 -- 172.21.15.27:0/3490016820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50681a0dc0 con 0x7f50680756d0 2024-05-10T11:16:29.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.493+0000 7f505fe00700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5068075ff0 0x7f506819c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:16:29.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.493+0000 7f506dc00700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f506810fea0 0x7f50681a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:16:29.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.494+0000 7f506d200700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50680756d0 0x7f506819be00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f505800e9a0 tx=0x7f505800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:29.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.494+0000 7f5066000700 1 -- 172.21.15.27:0/3490016820 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505800cbd0 con 0x7f50680756d0 2024-05-10T11:16:29.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.494+0000 7f5066000700 1 -- 172.21.15.27:0/3490016820 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f505800cd70 con 0x7f50680756d0 2024-05-10T11:16:29.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.494+0000 7f5066000700 1 -- 172.21.15.27:0/3490016820 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50580186f0 con 0x7f50680756d0 2024-05-10T11:16:29.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.494+0000 7f506fc00700 1 -- 172.21.15.27:0/3490016820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50681a10b0 con 0x7f50680756d0 2024-05-10T11:16:29.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.494+0000 7f506fc00700 1 -- 172.21.15.27:0/3490016820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50681a8930 con 0x7f50680756d0 2024-05-10T11:16:29.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.498+0000 7f5066000700 1 -- 172.21.15.27:0/3490016820 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f5058010a00 con 0x7f50680756d0 2024-05-10T11:16:29.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.499+0000 7f507a63db80 1 Processor -- start 2024-05-10T11:16:29.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.499+0000 7f507a63db80 1 -- start start 2024-05-10T11:16:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.499+0000 7f507a63db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11eea440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.499+0000 7f507a63db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 0x55fc11eee490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.499+0000 7f507a63db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11eeee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.500+0000 7f507a63db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fc11e3ca20 con 0x55fc11eea060 2024-05-10T11:16:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.500+0000 7f5066000700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f504c0b2a70 0x7f504c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.500+0000 7f507a63db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55fc11dda760 con 0x55fc11eea980 2024-05-10T11:16:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.500+0000 7f507a63db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fc11e389e0 con 0x55fc11eee9d0 2024-05-10T11:16:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.500+0000 7f506d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 0x55fc11eee490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.500+0000 7f5066000700 1 -- 172.21.15.27:0/3490016820 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f50580e2d10 con 0x7f50680756d0 2024-05-10T11:16:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.500+0000 7f506dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11eea440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.500+0000 7f505fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11eeee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.500+0000 7f506d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 0x55fc11eee490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60948/0 (socket says 172.21.15.27:60948) 2024-05-10T11:16:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.500+0000 7f506d200700 1 -- 172.21.15.27:0/3211279693 learned_addr learned my addr 172.21.15.27:0/3211279693 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.500+0000 7f506dc00700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f504c0b2a70 0x7f504c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.501+0000 7f506d200700 1 -- 172.21.15.27:0/3211279693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 msgr2=0x55fc11eeee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.501+0000 7f506d200700 1 --2- 172.21.15.27:0/3211279693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11eeee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.501+0000 7f506d200700 1 -- 172.21.15.27:0/3211279693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 msgr2=0x55fc11eea440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.501+0000 7f506d200700 1 --2- 172.21.15.27:0/3211279693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11eea440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.501+0000 7f506d200700 1 -- 172.21.15.27:0/3211279693 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc11d898d0 con 0x55fc11eea980 2024-05-10T11:16:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.501+0000 7f506d200700 1 --2- 172.21.15.27:0/3211279693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 0x55fc11eee490 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f50580e0090 tx=0x7f5058010da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ea7c5a8d6a6579a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.501+0000 7f506dc00700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f504c0b2a70 0x7f504c0b4f30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5060012e40 tx=0x7f5060009ca0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.501+0000 7f505e000700 1 -- 172.21.15.27:0/3211279693 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50580a96d0 con 0x55fc11eea980 2024-05-10T11:16:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.501+0000 7f505e000700 1 -- 172.21.15.27:0/3211279693 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50580aa2c0 con 0x55fc11eea980 2024-05-10T11:16:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.501+0000 7f505e000700 1 -- 172.21.15.27:0/3211279693 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50580b2590 con 0x55fc11eea980 2024-05-10T11:16:29.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.502+0000 7f507a63db80 1 -- 172.21.15.27:0/3211279693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 msgr2=0x55fc11eee490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.502+0000 7f507a63db80 1 --2- 172.21.15.27:0/3211279693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 0x55fc11eee490 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f50580e0090 tx=0x7f5058010da0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.502+0000 7f507a63db80 1 -- 172.21.15.27:0/3211279693 shutdown_connections 2024-05-10T11:16:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.502+0000 7f507a63db80 1 --2- 172.21.15.27:0/3211279693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11eeee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.502+0000 7f507a63db80 1 --2- 172.21.15.27:0/3211279693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11eea440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.502+0000 7f507a63db80 1 --2- 172.21.15.27:0/3211279693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 0x55fc11eee490 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.502+0000 7f507a63db80 1 -- 172.21.15.27:0/3211279693 >> 172.21.15.27:0/3211279693 conn(0x55fc11e18b60 msgr2=0x55fc11ee47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.502+0000 7f507a63db80 1 -- 172.21.15.27:0/3211279693 shutdown_connections 2024-05-10T11:16:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.502+0000 7f507a63db80 1 -- 172.21.15.27:0/3211279693 wait complete. 2024-05-10T11:16:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.503+0000 7f507a63db80 1 -- start start 2024-05-10T11:16:29.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.503+0000 7f507a63db80 1 Processor -- start 2024-05-10T11:16:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.504+0000 7f507a63db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11ef5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.504+0000 7f507a63db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eea980 0x55fc11ef31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.504+0000 7f507a63db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eee9d0 0x55fc11ef36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.504+0000 7f507a63db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fc11ef6b80 con 0x55fc11eea060 2024-05-10T11:16:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.504+0000 7f507a63db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55fc11da2b20 con 0x55fc11eee9d0 2024-05-10T11:16:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.504+0000 7f507a63db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fc11ef6d00 con 0x55fc11eea980 2024-05-10T11:16:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.504+0000 7f507a63db80 10 client.0 _ll_register_callbacks cb 0x55fc11de3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.504+0000 7f506d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11ef5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.504+0000 7f506dc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eee9d0 0x55fc11ef36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.504+0000 7f506dc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eee9d0 0x55fc11ef36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60964/0 (socket says 172.21.15.27:60964) 2024-05-10T11:16:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.504+0000 7f506dc00700 1 -- 172.21.15.27:0/3765725448 learned_addr learned my addr 172.21.15.27:0/3765725448 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.504+0000 7f505fe00700 1 --2- 172.21.15.27:0/3765725448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eea980 0x55fc11ef31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.505+0000 7f506dc00700 1 -- 172.21.15.27:0/3765725448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eea980 msgr2=0x55fc11ef31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.505+0000 7f506dc00700 1 --2- 172.21.15.27:0/3765725448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eea980 0x55fc11ef31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.505+0000 7f506dc00700 1 -- 172.21.15.27:0/3765725448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 msgr2=0x55fc11ef5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.505+0000 7f506dc00700 1 --2- 172.21.15.27:0/3765725448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11ef5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.505+0000 7f506dc00700 1 -- 172.21.15.27:0/3765725448 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc11e02420 con 0x55fc11eee9d0 2024-05-10T11:16:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.505+0000 7f506dc00700 1 --2- 172.21.15.27:0/3765725448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eee9d0 0x55fc11ef36e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5060008790 tx=0x7f5060009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7a5e952bc5e58af server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.505+0000 7f5056a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:16:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.505+0000 7f5056a00700 1 -- 172.21.15.27:0/3765725448 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5060029060 con 0x55fc11eee9d0 2024-05-10T11:16:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.505+0000 7f5056a00700 1 -- 172.21.15.27:0/3765725448 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f506001a850 con 0x55fc11eee9d0 2024-05-10T11:16:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.505+0000 7f507a63db80 1 -- 172.21.15.27:0/3765725448 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50580aa2c0 con 0x55fc11eee9d0 2024-05-10T11:16:29.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.506+0000 7f5056a00700 1 -- 172.21.15.27:0/3765725448 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5060030050 con 0x55fc11eee9d0 2024-05-10T11:16:29.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.506+0000 7f507a63db80 10 client.25829 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:29.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.506+0000 7f507a63db80 1 -- 172.21.15.27:0/3765725448 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fc11f013f0 con 0x55fc11eee9d0 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.506+0000 7f5056a00700 1 -- 172.21.15.27:0/3765725448 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50580aa2c0 con 0x55fc11eee9d0 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.506+0000 7f5056a00700 1 -- 172.21.15.27:0/3765725448 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f50600237a0 con 0x55fc11eee9d0 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.506+0000 7f507a63db80 10 client.25829 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.506+0000 7f507a63db80 10 client.25829 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.506+0000 7f507a63db80 20 client.25829 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.506+0000 7f507a63db80 1 --2- 172.21.15.27:0/3765725448 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55fc11f03a30 0x55fc11f05e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.506+0000 7f507a63db80 4 client.25829 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.506+0000 7f507a63db80 1 -- 172.21.15.27:0/3765725448 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc11eb70a0 con 0x55fc11f03a30 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.507+0000 7f506d200700 1 --2- 172.21.15.27:0/3765725448 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55fc11f03a30 0x55fc11f05e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.507+0000 7f506d200700 1 --2- 172.21.15.27:0/3765725448 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55fc11f03a30 0x55fc11f05e20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6af161406e8bd416 server_cookie=99da62b739a6d8e7 in_seq=0 out_seq=0 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.507+0000 7f5056a00700 10 client.25829 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.508+0000 7f5056a00700 1 -- 172.21.15.27:0/3765725448 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fc11eb70a0 con 0x55fc11f03a30 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.508+0000 7f5056a00700 10 client.25829 handle_command_reply: tid=0 2024-05-10T11:16:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.508+0000 7f507a63db80 1 client.25829 shutdown 2024-05-10T11:16:29.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.508+0000 7f507a63db80 1 -- 172.21.15.27:0/3765725448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eee9d0 msgr2=0x55fc11ef36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:29.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.508+0000 7f507a63db80 1 --2- 172.21.15.27:0/3765725448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eee9d0 0x55fc11ef36e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5060008790 tx=0x7f5060009920 comp rx=0 tx=0).stop 2024-05-10T11:16:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.509+0000 7f507a63db80 1 -- 172.21.15.27:0/3765725448 shutdown_connections 2024-05-10T11:16:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.509+0000 7f507a63db80 1 --2- 172.21.15.27:0/3765725448 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55fc11f03a30 0x55fc11f05e20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.509+0000 7f507a63db80 1 --2- 172.21.15.27:0/3765725448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eee9d0 0x55fc11ef36e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.509+0000 7f507a63db80 1 --2- 172.21.15.27:0/3765725448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11ef5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.509+0000 7f507a63db80 1 --2- 172.21.15.27:0/3765725448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eea980 0x55fc11ef31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.509+0000 7f507a63db80 1 -- 172.21.15.27:0/3765725448 >> 172.21.15.27:0/3765725448 conn(0x55fc11e18b60 msgr2=0x55fc11edf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.509+0000 7f5056a00700 0 client.25829 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:16:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.509+0000 7f507a63db80 1 -- 172.21.15.27:0/3765725448 shutdown_connections 2024-05-10T11:16:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.509+0000 7f507a63db80 1 -- 172.21.15.27:0/3765725448 wait complete. 2024-05-10T11:16:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.509+0000 7f507a63db80 20 client.25829 trim_cache size 0 max 16384 2024-05-10T11:16:29.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.533+0000 7f507a63db80 1 Processor -- start 2024-05-10T11:16:29.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.533+0000 7f507a63db80 1 -- start start 2024-05-10T11:16:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f507a63db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea060 0x55fc11effc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f507a63db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea980 0x55fc11ef0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f507a63db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11eef4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f507a63db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f50681b1890 con 0x55fc11eea980 2024-05-10T11:16:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f507a63db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f50681b3430 con 0x55fc11eea060 2024-05-10T11:16:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f507a63db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f50681b1f60 con 0x55fc11eee9d0 2024-05-10T11:16:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f506d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea060 0x55fc11effc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f506dc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11eef4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f506d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea060 0x55fc11effc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60974/0 (socket says 172.21.15.27:60974) 2024-05-10T11:16:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f506d200700 1 -- 172.21.15.27:0/2935206379 learned_addr learned my addr 172.21.15.27:0/2935206379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f505fe00700 1 --2- 172.21.15.27:0/2935206379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea980 0x55fc11ef0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f506d200700 1 -- 172.21.15.27:0/2935206379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 msgr2=0x55fc11eef4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f506d200700 1 --2- 172.21.15.27:0/2935206379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11eef4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f506d200700 1 -- 172.21.15.27:0/2935206379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea980 msgr2=0x55fc11ef0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f506d200700 1 --2- 172.21.15.27:0/2935206379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea980 0x55fc11ef0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f506d200700 1 -- 172.21.15.27:0/2935206379 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc11ef1710 con 0x55fc11eea060 2024-05-10T11:16:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.534+0000 7f506dc00700 1 --2- 172.21.15.27:0/2935206379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11eef4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:16:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.535+0000 7f506d200700 1 --2- 172.21.15.27:0/2935206379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea060 0x55fc11effc10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f50580a9b50 tx=0x7f50580cc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a37264ffc8c2a533 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.535+0000 7f505d600700 1 -- 172.21.15.27:0/2935206379 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50580a9bc0 con 0x55fc11eea060 2024-05-10T11:16:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.535+0000 7f505d600700 1 -- 172.21.15.27:0/2935206379 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50580a96d0 con 0x55fc11eea060 2024-05-10T11:16:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.535+0000 7f505d600700 1 -- 172.21.15.27:0/2935206379 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50580b27d0 con 0x55fc11eea060 2024-05-10T11:16:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.536+0000 7f507a63db80 1 -- 172.21.15.27:0/2935206379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea060 msgr2=0x55fc11effc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.536+0000 7f507a63db80 1 --2- 172.21.15.27:0/2935206379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea060 0x55fc11effc10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f50580a9b50 tx=0x7f50580cc040 comp rx=0 tx=0).stop 2024-05-10T11:16:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.536+0000 7f507a63db80 1 -- 172.21.15.27:0/2935206379 shutdown_connections 2024-05-10T11:16:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.536+0000 7f507a63db80 1 --2- 172.21.15.27:0/2935206379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11eef4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.536+0000 7f507a63db80 1 --2- 172.21.15.27:0/2935206379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea060 0x55fc11effc10 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.536+0000 7f507a63db80 1 --2- 172.21.15.27:0/2935206379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea980 0x55fc11ef0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.536+0000 7f507a63db80 1 -- 172.21.15.27:0/2935206379 >> 172.21.15.27:0/2935206379 conn(0x55fc11e18b60 msgr2=0x55fc11eeedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.536+0000 7f507a63db80 1 -- 172.21.15.27:0/2935206379 shutdown_connections 2024-05-10T11:16:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.536+0000 7f507a63db80 1 -- 172.21.15.27:0/2935206379 wait complete. 2024-05-10T11:16:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.536+0000 7f507a63db80 1 -- start start 2024-05-10T11:16:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.537+0000 7f507a63db80 1 Processor -- start 2024-05-10T11:16:29.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.537+0000 7f507a63db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11efbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.537+0000 7f507a63db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 0x55fc11efc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.537+0000 7f507a63db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11ef6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.537+0000 7f507a63db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f50681b2d00 con 0x55fc11eea060 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.537+0000 7f507a63db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f50681b2630 con 0x55fc11eea980 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.537+0000 7f507a63db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f50681af830 con 0x55fc11eee9d0 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.537+0000 7f507a63db80 10 client.0 _ll_register_callbacks cb 0x55fc11de3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.537+0000 7f505fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 0x55fc11efc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.538+0000 7f505fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 0x55fc11efc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60988/0 (socket says 172.21.15.27:60988) 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.538+0000 7f505fe00700 1 -- 172.21.15.27:0/1962729060 learned_addr learned my addr 172.21.15.27:0/1962729060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.538+0000 7f506d200700 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11efbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.538+0000 7f506dc00700 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11ef6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.538+0000 7f505fe00700 1 -- 172.21.15.27:0/1962729060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 msgr2=0x55fc11ef6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.538+0000 7f505fe00700 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11ef6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.538+0000 7f505fe00700 1 -- 172.21.15.27:0/1962729060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 msgr2=0x55fc11efbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:29.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.538+0000 7f505fe00700 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11efbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.538+0000 7f505fe00700 1 -- 172.21.15.27:0/1962729060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc11ef7600 con 0x55fc11eea980 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.538+0000 7f506dc00700 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11ef6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.538+0000 7f506d200700 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11efbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.539+0000 7f505fe00700 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 0x55fc11efc4e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5050009990 tx=0x7f5050005810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecf233c0ee827df9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.539+0000 7f5056a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.539+0000 7f5056a00700 1 -- 172.21.15.27:0/1962729060 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505000bd00 con 0x55fc11eea980 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.539+0000 7f507a63db80 1 -- 172.21.15.27:0/1962729060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50580a96d0 con 0x55fc11eea980 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.539+0000 7f5056a00700 1 -- 172.21.15.27:0/1962729060 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f5050003680 con 0x55fc11eea980 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.539+0000 7f5056a00700 1 -- 172.21.15.27:0/1962729060 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5050036050 con 0x55fc11eea980 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.539+0000 7f507a63db80 10 client.25835 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.539+0000 7f507a63db80 1 -- 172.21.15.27:0/1962729060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fc11f6ed40 con 0x55fc11eea980 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.540+0000 7f5056a00700 1 -- 172.21.15.27:0/1962729060 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50580a96d0 con 0x55fc11eea980 2024-05-10T11:16:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.540+0000 7f5056a00700 1 -- 172.21.15.27:0/1962729060 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f505000f960 con 0x55fc11eea980 2024-05-10T11:16:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.540+0000 7f507a63db80 10 client.25835 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.540+0000 7f507a63db80 10 client.25835 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:16:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.540+0000 7f507a63db80 20 client.25835 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.540+0000 7f507a63db80 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55fc11f6f4c0 0x55fc11f718e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.540+0000 7f507a63db80 4 client.25835 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:16:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.540+0000 7f507a63db80 1 -- 172.21.15.27:0/1962729060 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fc11e4a400 con 0x55fc11f6f4c0 2024-05-10T11:16:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.540+0000 7f506d200700 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55fc11f6f4c0 0x55fc11f718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f506d200700 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55fc11f6f4c0 0x55fc11f718e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf42ac8a679390e9 server_cookie=8c60e48728f57f8f in_seq=0 out_seq=0 2024-05-10T11:16:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f5056a00700 10 client.25835 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:16:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f5056a00700 1 -- 172.21.15.27:0/1962729060 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fc11e4a400 con 0x55fc11f6f4c0 2024-05-10T11:16:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f5056a00700 10 client.25835 handle_command_reply: tid=0 2024-05-10T11:16:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f507a63db80 1 client.25835 shutdown 2024-05-10T11:16:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f507a63db80 1 -- 172.21.15.27:0/1962729060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 msgr2=0x55fc11efc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f507a63db80 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 0x55fc11efc4e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5050009990 tx=0x7f5050005810 comp rx=0 tx=0).stop 2024-05-10T11:16:29.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f507a63db80 1 -- 172.21.15.27:0/1962729060 shutdown_connections 2024-05-10T11:16:29.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f507a63db80 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55fc11f6f4c0 0x55fc11f718e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f507a63db80 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc11eee9d0 0x55fc11ef6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f507a63db80 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fc11eea060 0x55fc11efbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f507a63db80 1 --2- 172.21.15.27:0/1962729060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fc11eea980 0x55fc11efc4e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f5056a00700 0 client.25835 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:16:29.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f507a63db80 1 -- 172.21.15.27:0/1962729060 >> 172.21.15.27:0/1962729060 conn(0x55fc11e18b60 msgr2=0x55fc11edf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:29.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f507a63db80 1 -- 172.21.15.27:0/1962729060 shutdown_connections 2024-05-10T11:16:29.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.541+0000 7f507a63db80 1 -- 172.21.15.27:0/1962729060 wait complete. 2024-05-10T11:16:29.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.542+0000 7f507a63db80 20 client.25835 trim_cache size 0 max 16384 2024-05-10T11:16:29.545 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:16:29.545 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:16:29.546 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:16:29.546 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:16:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.542+0000 7f506fc00700 1 -- 172.21.15.27:0/3490016820 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f504c0b2a70 msgr2=0x7f504c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.542+0000 7f506fc00700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f504c0b2a70 0x7f504c0b4f30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5060012e40 tx=0x7f5060009ca0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.542+0000 7f506fc00700 1 -- 172.21.15.27:0/3490016820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50680756d0 msgr2=0x7f506819be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.542+0000 7f506fc00700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50680756d0 0x7f506819be00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f505800e9a0 tx=0x7f505800ed60 comp rx=0 tx=0).stop 2024-05-10T11:16:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.542+0000 7f506fc00700 1 -- 172.21.15.27:0/3490016820 shutdown_connections 2024-05-10T11:16:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.542+0000 7f506fc00700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f504c0b2a70 0x7f504c0b4f30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.542+0000 7f506fc00700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50680756d0 0x7f506819be00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.542+0000 7f506fc00700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f506810fea0 0x7f50681a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.542+0000 7f506fc00700 1 --2- 172.21.15.27:0/3490016820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5068075ff0 0x7f506819c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.543+0000 7f506fc00700 1 -- 172.21.15.27:0/3490016820 >> 172.21.15.27:0/3490016820 conn(0x7f506807a600 msgr2=0x7f5068100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.543+0000 7f506fc00700 1 -- 172.21.15.27:0/3490016820 shutdown_connections 2024-05-10T11:16:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:29.543+0000 7f506fc00700 1 -- 172.21.15.27:0/3490016820 wait complete. 2024-05-10T11:16:29.658 INFO:tasks.workunit.client.0.smithi027.stdout: 1 106707 8.03 MB/sec warmup 74 sec latency 14.244 ms 2024-05-10T11:16:29.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:29 smithi076 ceph-mon[34152]: pgmap v101: 129 pgs: 129 active+clean; 70 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 257 op/s 2024-05-10T11:16:29.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:29 smithi189 ceph-mon[34842]: pgmap v101: 129 pgs: 129 active+clean; 70 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 257 op/s 2024-05-10T11:16:30.513 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:16:30.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:16:30.659 INFO:tasks.workunit.client.0.smithi027.stdout: 1 107730 7.96 MB/sec warmup 75 sec latency 33.052 ms 2024-05-10T11:16:30.997 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:16:31.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:31 smithi027 ceph-mon[30054]: pgmap v102: 129 pgs: 129 active+clean; 70 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 172 op/s 2024-05-10T11:16:31.659 INFO:tasks.workunit.client.0.smithi027.stdout: 1 110217 8.07 MB/sec warmup 76 sec latency 10.646 ms 2024-05-10T11:16:31.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:31 smithi076 ceph-mon[34152]: pgmap v102: 129 pgs: 129 active+clean; 70 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 172 op/s 2024-05-10T11:16:31.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:31 smithi189 ceph-mon[34842]: pgmap v102: 129 pgs: 129 active+clean; 70 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 172 op/s 2024-05-10T11:16:32.659 INFO:tasks.workunit.client.0.smithi027.stdout: 1 110903 7.98 MB/sec warmup 77 sec latency 62.236 ms 2024-05-10T11:16:33.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:33 smithi027 ceph-mon[30054]: pgmap v103: 129 pgs: 129 active+clean; 70 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 172 op/s 2024-05-10T11:16:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.549+0000 7f2d23e00700 1 -- 172.21.15.27:0/3786119522 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d1c106080 msgr2=0x7f2d1c112590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.549+0000 7f2d23e00700 1 --2- 172.21.15.27:0/3786119522 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d1c106080 0x7f2d1c112590 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400b0a0 tx=0x7f2d1400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:16:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.549+0000 7f2d23e00700 1 -- 172.21.15.27:0/3786119522 shutdown_connections 2024-05-10T11:16:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.549+0000 7f2d23e00700 1 --2- 172.21.15.27:0/3786119522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d1c10b6c0 0x7f2d1c10baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.549+0000 7f2d23e00700 1 --2- 172.21.15.27:0/3786119522 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d1c106080 0x7f2d1c112590 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.549+0000 7f2d23e00700 1 --2- 172.21.15.27:0/3786119522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d1c1056c0 0x7f2d1c105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.549+0000 7f2d23e00700 1 -- 172.21.15.27:0/3786119522 >> 172.21.15.27:0/3786119522 conn(0x7f2d1c1011b0 msgr2=0x7f2d1c1035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.549+0000 7f2d23e00700 1 -- 172.21.15.27:0/3786119522 shutdown_connections 2024-05-10T11:16:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.549+0000 7f2d23e00700 1 -- 172.21.15.27:0/3786119522 wait complete. 2024-05-10T11:16:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.550+0000 7f2d23e00700 1 Processor -- start 2024-05-10T11:16:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.550+0000 7f2d23e00700 1 -- start start 2024-05-10T11:16:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.551+0000 7f2d23e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d1c1056c0 0x7f2d1c19bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.551+0000 7f2d23e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d1c106080 0x7f2d1c19c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.551+0000 7f2d23e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d1c10b6c0 0x7f2d1c1a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.551+0000 7f2d23e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c115950 con 0x7f2d1c106080 2024-05-10T11:16:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.551+0000 7f2d23e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c1157d0 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.551+0000 7f2d23e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c115ad0 con 0x7f2d1c1056c0 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.551+0000 7f2d21e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d1c10b6c0 0x7f2d1c1a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.551+0000 7f2d20a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d1c106080 0x7f2d1c19c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.551+0000 7f2d21e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d1c10b6c0 0x7f2d1c1a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:32774/0 (socket says 172.21.15.27:32774) 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.551+0000 7f2d21e00700 1 -- 172.21.15.27:0/49175723 learned_addr learned my addr 172.21.15.27:0/49175723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.551+0000 7f2d21400700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d1c1056c0 0x7f2d1c19bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.552+0000 7f2d21e00700 1 -- 172.21.15.27:0/49175723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d1c1056c0 msgr2=0x7f2d1c19bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.552+0000 7f2d21e00700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d1c1056c0 0x7f2d1c19bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.552+0000 7f2d21e00700 1 -- 172.21.15.27:0/49175723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d1c106080 msgr2=0x7f2d1c19c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.552+0000 7f2d21e00700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d1c106080 0x7f2d1c19c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.552+0000 7f2d21e00700 1 -- 172.21.15.27:0/49175723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c1a0d60 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.552+0000 7f2d21e00700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d1c10b6c0 0x7f2d1c1a0620 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400b070 tx=0x7f2d14009c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.553+0000 7f2d1a000700 1 -- 172.21.15.27:0/49175723 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d14021070 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.553+0000 7f2d1a000700 1 -- 172.21.15.27:0/49175723 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d14004e40 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.553+0000 7f2d1a000700 1 -- 172.21.15.27:0/49175723 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d14004630 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.553+0000 7f2d23e00700 1 -- 172.21.15.27:0/49175723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1c1a0ff0 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.553+0000 7f2d23e00700 1 -- 172.21.15.27:0/49175723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d1c1a88d0 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.554+0000 7f2d23e00700 1 -- 172.21.15.27:0/49175723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d1c10dc50 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.557+0000 7f2d1a000700 1 -- 172.21.15.27:0/49175723 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2d14007980 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.559+0000 7f2d1a000700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d040b2a70 0x7f2d040b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:33.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.559+0000 7f2d1a000700 1 -- 172.21.15.27:0/49175723 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f2d1400f020 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.559+0000 7f2d21400700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d040b2a70 0x7f2d040b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:33.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.560+0000 7f2d21400700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d040b2a70 0x7f2d040b4f30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2d10009be0 tx=0x7f2d10009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:33.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.562+0000 7f2d1a000700 1 -- 172.21.15.27:0/49175723 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f2d1402a020 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.659 INFO:tasks.workunit.client.0.smithi027.stdout: 1 113506 8.09 MB/sec warmup 78 sec latency 42.014 ms 2024-05-10T11:16:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:33 smithi076 ceph-mon[34152]: pgmap v103: 129 pgs: 129 active+clean; 70 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 172 op/s 2024-05-10T11:16:33.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:33 smithi189 ceph-mon[34842]: pgmap v103: 129 pgs: 129 active+clean; 70 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 172 op/s 2024-05-10T11:16:33.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.919+0000 7f2d23e00700 1 -- 172.21.15.27:0/49175723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d1c19d0b0 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.920+0000 7f2d1a000700 1 -- 172.21.15.27:0/49175723 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11939 (secure 0 0 0) 0x7f2d140ab730 con 0x7f2d1c10b6c0 2024-05-10T11:16:33.921 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:16:33.921 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:15:16.547777+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24488,24467,24472,24527,24484]},"id":1}]} 2024-05-10T11:16:33.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.923+0000 7f2d23e00700 1 -- 172.21.15.27:0/49175723 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d040b2a70 msgr2=0x7f2d040b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:33.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.923+0000 7f2d23e00700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d040b2a70 0x7f2d040b4f30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2d10009be0 tx=0x7f2d10009260 comp rx=0 tx=0).stop 2024-05-10T11:16:33.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.923+0000 7f2d23e00700 1 -- 172.21.15.27:0/49175723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d1c10b6c0 msgr2=0x7f2d1c1a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:33.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.923+0000 7f2d23e00700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d1c10b6c0 0x7f2d1c1a0620 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400b070 tx=0x7f2d14009c90 comp rx=0 tx=0).stop 2024-05-10T11:16:33.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.923+0000 7f2d23e00700 1 -- 172.21.15.27:0/49175723 shutdown_connections 2024-05-10T11:16:33.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.923+0000 7f2d23e00700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d040b2a70 0x7f2d040b4f30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:33.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.923+0000 7f2d23e00700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d1c1056c0 0x7f2d1c19bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:33.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.923+0000 7f2d23e00700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d1c10b6c0 0x7f2d1c1a0620 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:33.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.923+0000 7f2d23e00700 1 --2- 172.21.15.27:0/49175723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d1c106080 0x7f2d1c19c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:33.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.923+0000 7f2d23e00700 1 -- 172.21.15.27:0/49175723 >> 172.21.15.27:0/49175723 conn(0x7f2d1c1011b0 msgr2=0x7f2d1c101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:33.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.924+0000 7f2d23e00700 1 -- 172.21.15.27:0/49175723 shutdown_connections 2024-05-10T11:16:33.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:33.924+0000 7f2d23e00700 1 -- 172.21.15.27:0/49175723 wait complete. 2024-05-10T11:16:33.926 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 31 2024-05-10T11:16:34.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:34 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/49175723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:16:34.659 INFO:tasks.workunit.client.0.smithi027.stdout: 1 114265 8.01 MB/sec warmup 79 sec latency 25.077 ms 2024-05-10T11:16:34.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:34 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/49175723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:16:34.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:34 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/49175723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:16:34.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:16:35.217 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:16:35.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:35 smithi027 ceph-mon[30054]: pgmap v104: 129 pgs: 129 active+clean; 75 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 255 op/s 2024-05-10T11:16:35.659 INFO:tasks.workunit.client.0.smithi027.stdout: 1 116905 8.09 MB/sec warmup 80 sec latency 9.154 ms 2024-05-10T11:16:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:35 smithi076 ceph-mon[34152]: pgmap v104: 129 pgs: 129 active+clean; 75 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 255 op/s 2024-05-10T11:16:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:35 smithi189 ceph-mon[34842]: pgmap v104: 129 pgs: 129 active+clean; 75 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 255 op/s 2024-05-10T11:16:36.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:36 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:16:36.659 INFO:tasks.workunit.client.0.smithi027.stdout: 1 117766 8.05 MB/sec warmup 81 sec latency 21.992 ms 2024-05-10T11:16:36.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:36 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:16:36.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:36 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:16:37.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:37 smithi027 ceph-mon[30054]: pgmap v105: 129 pgs: 129 active+clean; 75 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 166 op/s 2024-05-10T11:16:37.659 INFO:tasks.workunit.client.0.smithi027.stdout: 1 120258 8.11 MB/sec warmup 82 sec latency 10.443 ms 2024-05-10T11:16:37.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:37 smithi076 ceph-mon[34152]: pgmap v105: 129 pgs: 129 active+clean; 75 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 166 op/s 2024-05-10T11:16:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.677+0000 7f45b1000700 1 -- 172.21.15.27:0/34751336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac075870 msgr2=0x7f45ac075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.677+0000 7f45b1000700 1 --2- 172.21.15.27:0/34751336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac075870 0x7f45ac075c50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f459c009990 tx=0x7f459c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.677+0000 7f45b1000700 1 -- 172.21.15.27:0/34751336 shutdown_connections 2024-05-10T11:16:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.677+0000 7f45b1000700 1 --2- 172.21.15.27:0/34751336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac075870 0x7f45ac075c50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.677+0000 7f45b1000700 1 --2- 172.21.15.27:0/34751336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac10fc60 0x7f45ac112070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.677+0000 7f45b1000700 1 --2- 172.21.15.27:0/34751336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45ac076190 0x7f45ac10f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.677+0000 7f45b1000700 1 -- 172.21.15.27:0/34751336 >> 172.21.15.27:0/34751336 conn(0x7f45ac0feeb0 msgr2=0x7f45ac1012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:37.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.678+0000 7f45b1000700 1 -- 172.21.15.27:0/34751336 shutdown_connections 2024-05-10T11:16:37.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.678+0000 7f45b1000700 1 -- 172.21.15.27:0/34751336 wait complete. 2024-05-10T11:16:37.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.679+0000 7f45b1000700 1 Processor -- start 2024-05-10T11:16:37.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.680+0000 7f45b1000700 1 -- start start 2024-05-10T11:16:37.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.680+0000 7f45b1000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac075870 0x7f45ac19bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.680+0000 7f45b1000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45ac076190 0x7f45ac19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.680+0000 7f45b1000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac10fc60 0x7f45ac1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.680+0000 7f45b1000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac115810 con 0x7f45ac10fc60 2024-05-10T11:16:37.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.680+0000 7f45b1000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac115690 con 0x7f45ac076190 2024-05-10T11:16:37.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.680+0000 7f45b1000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac115990 con 0x7f45ac075870 2024-05-10T11:16:37.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.681+0000 7f45a3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45ac076190 0x7f45ac19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.681+0000 7f45ab400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac075870 0x7f45ac19bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.681+0000 7f45a3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45ac076190 0x7f45ac19c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41096/0 (socket says 172.21.15.27:41096) 2024-05-10T11:16:37.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.681+0000 7f45ab400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac075870 0x7f45ac19bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:44100/0 (socket says 172.21.15.27:44100) 2024-05-10T11:16:37.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.681+0000 7f45a3e00700 1 -- 172.21.15.27:0/1201283921 learned_addr learned my addr 172.21.15.27:0/1201283921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:37.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.681+0000 7f45a3e00700 1 -- 172.21.15.27:0/1201283921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac075870 msgr2=0x7f45ac19bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:37.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.681+0000 7f45a3e00700 1 --2- 172.21.15.27:0/1201283921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac075870 0x7f45ac19bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.681+0000 7f45a3e00700 1 -- 172.21.15.27:0/1201283921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac10fc60 msgr2=0x7f45ac1a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:37.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.681+0000 7f45a3e00700 1 --2- 172.21.15.27:0/1201283921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac10fc60 0x7f45ac1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.681+0000 7f45a3e00700 1 -- 172.21.15.27:0/1201283921 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac1a0d50 con 0x7f45ac076190 2024-05-10T11:16:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.681+0000 7f45a3e00700 1 --2- 172.21.15.27:0/1201283921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45ac076190 0x7f45ac19c280 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f459400e9c0 tx=0x7f459400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.682+0000 7f45a8c00700 1 -- 172.21.15.27:0/1201283921 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459400cbd0 con 0x7f45ac076190 2024-05-10T11:16:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.682+0000 7f45a8c00700 1 -- 172.21.15.27:0/1201283921 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f459400cd70 con 0x7f45ac076190 2024-05-10T11:16:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.682+0000 7f45a8c00700 1 -- 172.21.15.27:0/1201283921 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4594005610 con 0x7f45ac076190 2024-05-10T11:16:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.682+0000 7f45b1000700 1 -- 172.21.15.27:0/1201283921 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45ac1a1040 con 0x7f45ac076190 2024-05-10T11:16:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.682+0000 7f45b1000700 1 -- 172.21.15.27:0/1201283921 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45ac1a88c0 con 0x7f45ac076190 2024-05-10T11:16:37.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.687+0000 7f45a8c00700 1 -- 172.21.15.27:0/1201283921 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f4594004750 con 0x7f45ac076190 2024-05-10T11:16:37.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.689+0000 7f45bcf76b80 1 Processor -- start 2024-05-10T11:16:37.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.689+0000 7f45a8c00700 1 --2- 172.21.15.27:0/1201283921 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f45900b2a70 0x7f45900b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.689+0000 7f45bcf76b80 1 -- start start 2024-05-10T11:16:37.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.689+0000 7f45a8c00700 1 -- 172.21.15.27:0/1201283921 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f45940e2b70 con 0x7f45ac076190 2024-05-10T11:16:37.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.689+0000 7f45ab400700 1 --2- 172.21.15.27:0/1201283921 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f45900b2a70 0x7f45900b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.689+0000 7f45bcf76b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc009e0 0x5617fcc00dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.689+0000 7f45bcf76b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc01300 0x5617fcbfafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.689+0000 7f45bcf76b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcbfba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.689+0000 7f45bcf76b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5617fcb4da20 con 0x5617fcc01300 2024-05-10T11:16:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.689+0000 7f45bcf76b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5617fcaeb760 con 0x5617fcc009e0 2024-05-10T11:16:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.689+0000 7f45bcf76b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5617fcb499e0 con 0x5617fcbfb600 2024-05-10T11:16:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.690+0000 7f45abe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcbfba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.690+0000 7f45ab400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc009e0 0x5617fcc00dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.690+0000 7f45abe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcbfba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:44106/0 (socket says 172.21.15.27:44106) 2024-05-10T11:16:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.690+0000 7f45abe00700 1 -- 172.21.15.27:0/3441884983 learned_addr learned my addr 172.21.15.27:0/3441884983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.690+0000 7f45a3e00700 1 --2- 172.21.15.27:0/3441884983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc01300 0x5617fcbfafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.690+0000 7f45abe00700 1 -- 172.21.15.27:0/3441884983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc009e0 msgr2=0x5617fcc00dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.690+0000 7f45abe00700 1 --2- 172.21.15.27:0/3441884983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc009e0 0x5617fcc00dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.690+0000 7f45abe00700 1 -- 172.21.15.27:0/3441884983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc01300 msgr2=0x5617fcbfafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.690+0000 7f45abe00700 1 --2- 172.21.15.27:0/3441884983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc01300 0x5617fcbfafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.690+0000 7f45abe00700 1 -- 172.21.15.27:0/3441884983 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617fca9a8d0 con 0x5617fcbfb600 2024-05-10T11:16:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.691+0000 7f45ab400700 1 --2- 172.21.15.27:0/1201283921 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f45900b2a70 0x7f45900b4f30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f459c006010 tx=0x7f459c0051c0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.691+0000 7f45abe00700 1 --2- 172.21.15.27:0/3441884983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcbfba80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f45a400e3e0 tx=0x7f45a400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eac998ef078c450e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.691+0000 7f45a0c00700 1 -- 172.21.15.27:0/3441884983 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a400ee80 con 0x5617fcbfb600 2024-05-10T11:16:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.691+0000 7f45a0c00700 1 -- 172.21.15.27:0/3441884983 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a4004bb0 con 0x5617fcbfb600 2024-05-10T11:16:37.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.691+0000 7f45a0c00700 1 -- 172.21.15.27:0/3441884983 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a40042c0 con 0x5617fcbfb600 2024-05-10T11:16:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.692+0000 7f45bcf76b80 1 -- 172.21.15.27:0/3441884983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 msgr2=0x5617fcbfba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.692+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/3441884983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcbfba80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f45a400e3e0 tx=0x7f45a400b040 comp rx=0 tx=0).stop 2024-05-10T11:16:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.692+0000 7f45bcf76b80 1 -- 172.21.15.27:0/3441884983 shutdown_connections 2024-05-10T11:16:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.692+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/3441884983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcbfba80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.692+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/3441884983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc009e0 0x5617fcc00dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.692+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/3441884983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc01300 0x5617fcbfafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.692+0000 7f45bcf76b80 1 -- 172.21.15.27:0/3441884983 >> 172.21.15.27:0/3441884983 conn(0x5617fcb29b60 msgr2=0x5617fcbf59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.693+0000 7f45bcf76b80 1 -- 172.21.15.27:0/3441884983 shutdown_connections 2024-05-10T11:16:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.693+0000 7f45bcf76b80 1 -- 172.21.15.27:0/3441884983 wait complete. 2024-05-10T11:16:37.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.694+0000 7f45bcf76b80 1 -- start start 2024-05-10T11:16:37.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.694+0000 7f45bcf76b80 1 Processor -- start 2024-05-10T11:16:37.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.695+0000 7f45bcf76b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcbfb600 0x5617fcc06ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.695+0000 7f45bcf76b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc009e0 0x5617fcc04290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.695+0000 7f45bcf76b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcc01300 0x5617fcc047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.695+0000 7f45bcf76b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5617fcc07d50 con 0x5617fcc009e0 2024-05-10T11:16:37.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.695+0000 7f45bcf76b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5617fcab3b20 con 0x5617fcbfb600 2024-05-10T11:16:37.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.695+0000 7f45bcf76b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5617fcc07ed0 con 0x5617fcc01300 2024-05-10T11:16:37.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.695+0000 7f45bcf76b80 10 client.0 _ll_register_callbacks cb 0x5617fcaf4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:37.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.695+0000 7f45ab400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcbfb600 0x5617fcc06ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.695+0000 7f45ab400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcbfb600 0x5617fcc06ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41118/0 (socket says 172.21.15.27:41118) 2024-05-10T11:16:37.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.695+0000 7f45ab400700 1 -- 172.21.15.27:0/2156781245 learned_addr learned my addr 172.21.15.27:0/2156781245 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:37.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.695+0000 7f45abe00700 1 --2- 172.21.15.27:0/2156781245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcc01300 0x5617fcc047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.696+0000 7f45ab400700 1 -- 172.21.15.27:0/2156781245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcc01300 msgr2=0x5617fcc047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:37.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.696+0000 7f45ab400700 1 --2- 172.21.15.27:0/2156781245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcc01300 0x5617fcc047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.696+0000 7f45ab400700 1 -- 172.21.15.27:0/2156781245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc009e0 msgr2=0x5617fcc04290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:37.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.696+0000 7f45ab400700 1 --2- 172.21.15.27:0/2156781245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc009e0 0x5617fcc04290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.696+0000 7f45ab400700 1 -- 172.21.15.27:0/2156781245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617fcb13420 con 0x5617fcbfb600 2024-05-10T11:16:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.696+0000 7f45ab400700 1 --2- 172.21.15.27:0/2156781245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcbfb600 0x5617fcc06ef0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f459c005850 tx=0x7f459c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc3416754e9010d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.696+0000 7f4599600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:16:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.697+0000 7f4599600700 1 -- 172.21.15.27:0/2156781245 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c036060 con 0x5617fcbfb600 2024-05-10T11:16:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.697+0000 7f4599600700 1 -- 172.21.15.27:0/2156781245 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f459c003a90 con 0x5617fcbfb600 2024-05-10T11:16:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.697+0000 7f45bcf76b80 1 -- 172.21.15.27:0/2156781245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45a4004bb0 con 0x5617fcbfb600 2024-05-10T11:16:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.698+0000 7f4599600700 1 -- 172.21.15.27:0/2156781245 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f459c021040 con 0x5617fcbfb600 2024-05-10T11:16:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.699+0000 7f45bcf76b80 10 client.25853 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.699+0000 7f45bcf76b80 1 -- 172.21.15.27:0/2156781245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5617fcc124a0 con 0x5617fcbfb600 2024-05-10T11:16:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.699+0000 7f4599600700 1 -- 172.21.15.27:0/2156781245 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a4004bb0 con 0x5617fcbfb600 2024-05-10T11:16:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.699+0000 7f4599600700 1 -- 172.21.15.27:0/2156781245 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f459c031430 con 0x5617fcbfb600 2024-05-10T11:16:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.700+0000 7f45bcf76b80 10 client.25853 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.700+0000 7f45bcf76b80 10 client.25853 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:16:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.700+0000 7f45bcf76b80 20 client.25853 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:37.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.700+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/2156781245 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5617fcc14a30 0x5617fcc16e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.700+0000 7f45bcf76b80 4 client.25853 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:16:37.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.700+0000 7f45bcf76b80 1 -- 172.21.15.27:0/2156781245 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617fcbc80a0 con 0x5617fcc14a30 2024-05-10T11:16:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.701+0000 7f45a3e00700 1 --2- 172.21.15.27:0/2156781245 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5617fcc14a30 0x5617fcc16e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.703+0000 7f45a3e00700 1 --2- 172.21.15.27:0/2156781245 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5617fcc14a30 0x5617fcc16e20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4a2169cc86cd8d40 server_cookie=433da1c8f99beaac in_seq=0 out_seq=0 2024-05-10T11:16:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.703+0000 7f4599600700 10 client.25853 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:16:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.707+0000 7f4599600700 1 -- 172.21.15.27:0/2156781245 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5617fcbc80a0 con 0x5617fcc14a30 2024-05-10T11:16:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.707+0000 7f4599600700 10 client.25853 handle_command_reply: tid=0 2024-05-10T11:16:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.707+0000 7f45bcf76b80 1 client.25853 shutdown 2024-05-10T11:16:37.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.709+0000 7f45bcf76b80 1 -- 172.21.15.27:0/2156781245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcbfb600 msgr2=0x5617fcc06ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:37.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.709+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/2156781245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcbfb600 0x5617fcc06ef0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f459c005850 tx=0x7f459c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.709+0000 7f45bcf76b80 1 -- 172.21.15.27:0/2156781245 shutdown_connections 2024-05-10T11:16:37.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.709+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/2156781245 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5617fcc14a30 0x5617fcc16e20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.709+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/2156781245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcc01300 0x5617fcc047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.709+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/2156781245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcbfb600 0x5617fcc06ef0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.709+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/2156781245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc009e0 0x5617fcc04290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.709+0000 7f45bcf76b80 1 -- 172.21.15.27:0/2156781245 >> 172.21.15.27:0/2156781245 conn(0x5617fcb29b60 msgr2=0x5617fcbf0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:37.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.710+0000 7f4599600700 0 client.25853 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:16:37.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.710+0000 7f45bcf76b80 1 -- 172.21.15.27:0/2156781245 shutdown_connections 2024-05-10T11:16:37.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.710+0000 7f45bcf76b80 1 -- 172.21.15.27:0/2156781245 wait complete. 2024-05-10T11:16:37.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.710+0000 7f45bcf76b80 20 client.25853 trim_cache size 0 max 16384 2024-05-10T11:16:37.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:37 smithi189 ceph-mon[34842]: pgmap v105: 129 pgs: 129 active+clean; 75 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 166 op/s 2024-05-10T11:16:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.742+0000 7f45bcf76b80 1 Processor -- start 2024-05-10T11:16:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.743+0000 7f45bcf76b80 1 -- start start 2024-05-10T11:16:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.743+0000 7f45bcf76b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcbfafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.743+0000 7f45bcf76b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc009e0 0x5617fcc1c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.743+0000 7f45bcf76b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc01300 0x5617fcc1cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.743+0000 7f45bcf76b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1b18b0 con 0x5617fcc009e0 2024-05-10T11:16:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.743+0000 7f45bcf76b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1b3450 con 0x5617fcc01300 2024-05-10T11:16:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.743+0000 7f45bcf76b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1b1f80 con 0x5617fcbfb600 2024-05-10T11:16:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.744+0000 7f45abe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc01300 0x5617fcc1cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.744+0000 7f45ab400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcbfafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.744+0000 7f45abe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc01300 0x5617fcc1cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41134/0 (socket says 172.21.15.27:41134) 2024-05-10T11:16:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.744+0000 7f45abe00700 1 -- 172.21.15.27:0/4292385947 learned_addr learned my addr 172.21.15.27:0/4292385947 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.744+0000 7f45a3e00700 1 --2- 172.21.15.27:0/4292385947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc009e0 0x5617fcc1c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.745+0000 7f45ab400700 1 -- 172.21.15.27:0/4292385947 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc01300 msgr2=0x5617fcc1cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.745+0000 7f45ab400700 1 --2- 172.21.15.27:0/4292385947 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc01300 0x5617fcc1cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.745+0000 7f45ab400700 1 -- 172.21.15.27:0/4292385947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc009e0 msgr2=0x5617fcc1c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.745+0000 7f45ab400700 1 --2- 172.21.15.27:0/4292385947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc009e0 0x5617fcc1c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.745+0000 7f45ab400700 1 -- 172.21.15.27:0/4292385947 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617fcbedf10 con 0x5617fcbfb600 2024-05-10T11:16:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.745+0000 7f45a3e00700 1 --2- 172.21.15.27:0/4292385947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc009e0 0x5617fcc1c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:16:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.745+0000 7f45ab400700 1 --2- 172.21.15.27:0/4292385947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcbfafe0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f459c031830 tx=0x7f459c031630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80ec106c90a694a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.745+0000 7f459be00700 1 -- 172.21.15.27:0/4292385947 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c036070 con 0x5617fcbfb600 2024-05-10T11:16:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.746+0000 7f459be00700 1 -- 172.21.15.27:0/4292385947 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f459c02d040 con 0x5617fcbfb600 2024-05-10T11:16:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.746+0000 7f459be00700 1 -- 172.21.15.27:0/4292385947 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c040080 con 0x5617fcbfb600 2024-05-10T11:16:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.747+0000 7f45bcf76b80 1 -- 172.21.15.27:0/4292385947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 msgr2=0x5617fcbfafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.747+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/4292385947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcbfafe0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f459c031830 tx=0x7f459c031630 comp rx=0 tx=0).stop 2024-05-10T11:16:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.747+0000 7f45bcf76b80 1 -- 172.21.15.27:0/4292385947 shutdown_connections 2024-05-10T11:16:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.747+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/4292385947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcbfafe0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.747+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/4292385947 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc01300 0x5617fcc1cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.747+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/4292385947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc009e0 0x5617fcc1c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.747+0000 7f45bcf76b80 1 -- 172.21.15.27:0/4292385947 >> 172.21.15.27:0/4292385947 conn(0x5617fcb29b60 msgr2=0x5617fcc04250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.747+0000 7f45bcf76b80 1 -- 172.21.15.27:0/4292385947 shutdown_connections 2024-05-10T11:16:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.747+0000 7f45bcf76b80 1 -- 172.21.15.27:0/4292385947 wait complete. 2024-05-10T11:16:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.748+0000 7f45bcf76b80 1 -- start start 2024-05-10T11:16:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.748+0000 7f45bcf76b80 1 Processor -- start 2024-05-10T11:16:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.749+0000 7f45bcf76b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcc0d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.749+0000 7f45bcf76b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc009e0 0x5617fcc0d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.749+0000 7f45bcf76b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc01300 0x5617fcc82e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.749+0000 7f45bcf76b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1b2d20 con 0x5617fcc01300 2024-05-10T11:16:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.749+0000 7f45bcf76b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1b2650 con 0x5617fcc009e0 2024-05-10T11:16:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.749+0000 7f45bcf76b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1af850 con 0x5617fcbfb600 2024-05-10T11:16:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.749+0000 7f45bcf76b80 10 client.0 _ll_register_callbacks cb 0x5617fcaf4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.749+0000 7f45a3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc009e0 0x5617fcc0d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.749+0000 7f45a3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc009e0 0x5617fcc0d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41140/0 (socket says 172.21.15.27:41140) 2024-05-10T11:16:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.749+0000 7f45a3e00700 1 -- 172.21.15.27:0/384679955 learned_addr learned my addr 172.21.15.27:0/384679955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.750+0000 7f45a3e00700 1 -- 172.21.15.27:0/384679955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 msgr2=0x5617fcc0d160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:16:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.750+0000 7f45a3e00700 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcc0d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.750+0000 7f45a3e00700 1 -- 172.21.15.27:0/384679955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc01300 msgr2=0x5617fcc82e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:16:37.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.753+0000 7f45abe00700 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc01300 0x5617fcc82e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.754+0000 7f45a3e00700 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc01300 0x5617fcc82e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.754+0000 7f45a3e00700 1 -- 172.21.15.27:0/384679955 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617fcc83650 con 0x5617fcc009e0 2024-05-10T11:16:37.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.755+0000 7f45abe00700 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc01300 0x5617fcc82e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:16:37.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.755+0000 7f45a3e00700 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc009e0 0x5617fcc0d6a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f45940dff50 tx=0x7f4594005020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=556bb56bcefa1370 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:37.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.755+0000 7f4599600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:16:37.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.756+0000 7f4599600700 1 -- 172.21.15.27:0/384679955 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459408e070 con 0x5617fcc009e0 2024-05-10T11:16:37.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.756+0000 7f4599600700 1 -- 172.21.15.27:0/384679955 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f45940a9ef0 con 0x5617fcc009e0 2024-05-10T11:16:37.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.756+0000 7f45bcf76b80 1 -- 172.21.15.27:0/384679955 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f459c02d040 con 0x5617fcc009e0 2024-05-10T11:16:37.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.757+0000 7f4599600700 1 -- 172.21.15.27:0/384679955 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f45940aa180 con 0x5617fcc009e0 2024-05-10T11:16:37.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.758+0000 7f45bcf76b80 10 client.25859 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:37.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.758+0000 7f45bcf76b80 1 -- 172.21.15.27:0/384679955 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5617fcc98aa0 con 0x5617fcc009e0 2024-05-10T11:16:37.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.758+0000 7f4599600700 1 -- 172.21.15.27:0/384679955 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f459c02d040 con 0x5617fcc009e0 2024-05-10T11:16:37.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.758+0000 7f4599600700 1 -- 172.21.15.27:0/384679955 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f45940cc0b0 con 0x5617fcc009e0 2024-05-10T11:16:37.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.758+0000 7f45bcf76b80 10 client.25859 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:37.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.758+0000 7f45bcf76b80 10 client.25859 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:16:37.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.758+0000 7f45bcf76b80 20 client.25859 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:37.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.758+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5617fcc98f50 0x5617fcc9b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:37.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.759+0000 7f45bcf76b80 4 client.25859 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:16:37.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.759+0000 7f45bcf76b80 1 -- 172.21.15.27:0/384679955 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5617fcb5b400 con 0x5617fcc98f50 2024-05-10T11:16:37.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.759+0000 7f45ab400700 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5617fcc98f50 0x5617fcc9b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.760+0000 7f45ab400700 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5617fcc98f50 0x5617fcc9b370 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c49f3ad5d1259c63 server_cookie=fba22200b4482b8a in_seq=0 out_seq=0 2024-05-10T11:16:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.760+0000 7f4599600700 10 client.25859 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:16:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.761+0000 7f4599600700 1 -- 172.21.15.27:0/384679955 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5617fcb5b400 con 0x5617fcc98f50 2024-05-10T11:16:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.761+0000 7f4599600700 10 client.25859 handle_command_reply: tid=0 2024-05-10T11:16:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.762+0000 7f45bcf76b80 1 client.25859 shutdown 2024-05-10T11:16:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.763+0000 7f45bcf76b80 1 -- 172.21.15.27:0/384679955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc009e0 msgr2=0x5617fcc0d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.763+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc009e0 0x5617fcc0d6a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f45940dff50 tx=0x7f4594005020 comp rx=0 tx=0).stop 2024-05-10T11:16:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.763+0000 7f45bcf76b80 1 -- 172.21.15.27:0/384679955 shutdown_connections 2024-05-10T11:16:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.763+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5617fcc98f50 0x5617fcc9b370 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.763+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617fcbfb600 0x5617fcc0d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.763+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5617fcc01300 0x5617fcc82e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.763+0000 7f45bcf76b80 1 --2- 172.21.15.27:0/384679955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617fcc009e0 0x5617fcc0d6a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.763+0000 7f45bcf76b80 1 -- 172.21.15.27:0/384679955 >> 172.21.15.27:0/384679955 conn(0x5617fcb29b60 msgr2=0x5617fcc016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.763+0000 7f4599600700 0 client.25859 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:16:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.764+0000 7f45bcf76b80 1 -- 172.21.15.27:0/384679955 shutdown_connections 2024-05-10T11:16:37.765 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:16:37.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.764+0000 7f45bcf76b80 1 -- 172.21.15.27:0/384679955 wait complete. 2024-05-10T11:16:37.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.764+0000 7f45bcf76b80 20 client.25859 trim_cache size 0 max 16384 2024-05-10T11:16:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.766+0000 7f45b1000700 1 -- 172.21.15.27:0/1201283921 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f45900b2a70 msgr2=0x7f45900b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.766+0000 7f45b1000700 1 --2- 172.21.15.27:0/1201283921 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f45900b2a70 0x7f45900b4f30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f459c006010 tx=0x7f459c0051c0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.766+0000 7f45b1000700 1 -- 172.21.15.27:0/1201283921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45ac076190 msgr2=0x7f45ac19c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.766+0000 7f45b1000700 1 --2- 172.21.15.27:0/1201283921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45ac076190 0x7f45ac19c280 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f459400e9c0 tx=0x7f459400ecd0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.766+0000 7f45b1000700 1 -- 172.21.15.27:0/1201283921 shutdown_connections 2024-05-10T11:16:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.766+0000 7f45b1000700 1 --2- 172.21.15.27:0/1201283921 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f45900b2a70 0x7f45900b4f30 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f459c006010 tx=0x7f459c0051c0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.766+0000 7f45b1000700 1 --2- 172.21.15.27:0/1201283921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac075870 0x7f45ac19bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.766+0000 7f45b1000700 1 --2- 172.21.15.27:0/1201283921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45ac10fc60 0x7f45ac1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.766+0000 7f45b1000700 1 --2- 172.21.15.27:0/1201283921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45ac076190 0x7f45ac19c280 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.766+0000 7f45b1000700 1 -- 172.21.15.27:0/1201283921 >> 172.21.15.27:0/1201283921 conn(0x7f45ac0feeb0 msgr2=0x7f45ac0ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:37.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.767+0000 7f45b1000700 1 -- 172.21.15.27:0/1201283921 shutdown_connections 2024-05-10T11:16:37.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:37.767+0000 7f45b1000700 1 -- 172.21.15.27:0/1201283921 wait complete. 2024-05-10T11:16:38.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:38 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:16:38.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:38 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:16:38.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:38 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:16:38.660 INFO:tasks.workunit.client.0.smithi027.stdout: 1 120951 8.08 MB/sec warmup 83 sec latency 272.306 ms 2024-05-10T11:16:38.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:38 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:16:38.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:38 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:16:38.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:38 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:16:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:38 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:16:38.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:38 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:16:38.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:38 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:16:38.947 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:16:39.417 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:16:39.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:39 smithi027 ceph-mon[30054]: pgmap v106: 129 pgs: 129 active+clean; 77 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 243 op/s 2024-05-10T11:16:39.660 INFO:tasks.workunit.client.0.smithi027.stdout: 1 120951 7.99 MB/sec warmup 84 sec latency 1272.471 ms 2024-05-10T11:16:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:39 smithi076 ceph-mon[34152]: pgmap v106: 129 pgs: 129 active+clean; 77 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 243 op/s 2024-05-10T11:16:39.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:39 smithi189 ceph-mon[34842]: pgmap v106: 129 pgs: 129 active+clean; 77 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 243 op/s 2024-05-10T11:16:40.660 INFO:tasks.workunit.client.0.smithi027.stdout: 1 120951 7.89 MB/sec warmup 85 sec latency 2272.616 ms 2024-05-10T11:16:41.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:41 smithi027 ceph-mon[30054]: pgmap v107: 129 pgs: 129 active+clean; 77 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-05-10T11:16:41.660 INFO:tasks.workunit.client.0.smithi027.stdout: 1 120951 7.80 MB/sec warmup 86 sec latency 3272.783 ms 2024-05-10T11:16:41.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:41 smithi076 ceph-mon[34152]: pgmap v107: 129 pgs: 129 active+clean; 77 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-05-10T11:16:41.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:41 smithi189 ceph-mon[34842]: pgmap v107: 129 pgs: 129 active+clean; 77 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-05-10T11:16:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.853+0000 7efd2e400700 1 -- 172.21.15.27:0/1990303755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd28102180 msgr2=0x7efd28102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.853+0000 7efd2e400700 1 --2- 172.21.15.27:0/1990303755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd28102180 0x7efd28102560 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009990 tx=0x7efd1c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.853+0000 7efd2e400700 1 -- 172.21.15.27:0/1990303755 shutdown_connections 2024-05-10T11:16:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.853+0000 7efd2e400700 1 --2- 172.21.15.27:0/1990303755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd2810fd30 0x7efd28112120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.853+0000 7efd2e400700 1 --2- 172.21.15.27:0/1990303755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd28102180 0x7efd28102560 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.853+0000 7efd2e400700 1 --2- 172.21.15.27:0/1990303755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd28102b30 0x7efd2810f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.853+0000 7efd2e400700 1 -- 172.21.15.27:0/1990303755 >> 172.21.15.27:0/1990303755 conn(0x7efd280fddc0 msgr2=0x7efd281001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.853+0000 7efd2e400700 1 -- 172.21.15.27:0/1990303755 shutdown_connections 2024-05-10T11:16:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.853+0000 7efd2e400700 1 -- 172.21.15.27:0/1990303755 wait complete. 2024-05-10T11:16:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.854+0000 7efd2e400700 1 Processor -- start 2024-05-10T11:16:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.854+0000 7efd2e400700 1 -- start start 2024-05-10T11:16:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.855+0000 7efd2e400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd28102180 0x7efd281a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.855+0000 7efd2e400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd28102b30 0x7efd281a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.855+0000 7efd2e400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd2810fd30 0x7efd2819bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.855+0000 7efd2e400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efd28115790 con 0x7efd28102b30 2024-05-10T11:16:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.855+0000 7efd2e400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efd28115610 con 0x7efd28102180 2024-05-10T11:16:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.855+0000 7efd2e400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efd28115910 con 0x7efd2810fd30 2024-05-10T11:16:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.855+0000 7efd27e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd28102b30 0x7efd281a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.855+0000 7efd2d000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd28102180 0x7efd281a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.855+0000 7efd27e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd28102b30 0x7efd281a1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41306/0 (socket says 172.21.15.27:41306) 2024-05-10T11:16:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.855+0000 7efd27e00700 1 -- 172.21.15.27:0/3358911758 learned_addr learned my addr 172.21.15.27:0/3358911758 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.855+0000 7efd2da00700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd2810fd30 0x7efd2819bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.856+0000 7efd2d000700 1 -- 172.21.15.27:0/3358911758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd2810fd30 msgr2=0x7efd2819bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.856+0000 7efd2d000700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd2810fd30 0x7efd2819bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.856+0000 7efd2d000700 1 -- 172.21.15.27:0/3358911758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd28102b30 msgr2=0x7efd281a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.856+0000 7efd2d000700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd28102b30 0x7efd281a1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.856+0000 7efd2d000700 1 -- 172.21.15.27:0/3358911758 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2819c230 con 0x7efd28102180 2024-05-10T11:16:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.856+0000 7efd2d000700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd28102180 0x7efd281a18c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efd1c0059d0 tx=0x7efd1c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.856+0000 7efd25600700 1 -- 172.21.15.27:0/3358911758 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c027070 con 0x7efd28102180 2024-05-10T11:16:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.857+0000 7efd25600700 1 -- 172.21.15.27:0/3358911758 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd1c022470 con 0x7efd28102180 2024-05-10T11:16:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.857+0000 7efd25600700 1 -- 172.21.15.27:0/3358911758 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c017650 con 0x7efd28102180 2024-05-10T11:16:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.857+0000 7efd2e400700 1 -- 172.21.15.27:0/3358911758 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd2819c4c0 con 0x7efd28102180 2024-05-10T11:16:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.857+0000 7efd2e400700 1 -- 172.21.15.27:0/3358911758 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd281a8790 con 0x7efd28102180 2024-05-10T11:16:41.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.861+0000 7efd25600700 1 -- 172.21.15.27:0/3358911758 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7efd1c02a070 con 0x7efd28102180 2024-05-10T11:16:41.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.862+0000 7efd3a30bb80 1 Processor -- start 2024-05-10T11:16:41.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.862+0000 7efd3a30bb80 1 -- start start 2024-05-10T11:16:41.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.862+0000 7efd3a30bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb7da0 0x55d82cfb8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.862+0000 7efd3a30bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfb2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.862+0000 7efd3a30bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb2a50 0x55d82cfb2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.862+0000 7efd3a30bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d82cf04a20 con 0x55d82cfb86c0 2024-05-10T11:16:41.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.862+0000 7efd3a30bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d82cea2760 con 0x55d82cfb7da0 2024-05-10T11:16:41.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.862+0000 7efd3a30bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d82cf009e0 con 0x55d82cfb2a50 2024-05-10T11:16:41.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.862+0000 7efd25600700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd100b2a70 0x7efd100b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd25600700 1 -- 172.21.15.27:0/3358911758 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7efd1c0e35d0 con 0x7efd28102180 2024-05-10T11:16:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd27e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb2a50 0x55d82cfb2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd2d000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfb2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd27e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb2a50 0x55d82cfb2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:44172/0 (socket says 172.21.15.27:44172) 2024-05-10T11:16:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd27e00700 1 -- 172.21.15.27:0/751868753 learned_addr learned my addr 172.21.15.27:0/751868753 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd2da00700 1 --2- 172.21.15.27:0/751868753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb7da0 0x55d82cfb8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd2da00700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd100b2a70 0x7efd100b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd27e00700 1 -- 172.21.15.27:0/751868753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb7da0 msgr2=0x55d82cfb8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd27e00700 1 --2- 172.21.15.27:0/751868753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb7da0 0x55d82cfb8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd27e00700 1 -- 172.21.15.27:0/751868753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 msgr2=0x55d82cfb2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd27e00700 1 --2- 172.21.15.27:0/751868753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfb2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd27e00700 1 -- 172.21.15.27:0/751868753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d82ce518d0 con 0x55d82cfb2a50 2024-05-10T11:16:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.863+0000 7efd2d000700 1 --2- 172.21.15.27:0/751868753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfb2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:16:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.864+0000 7efd27e00700 1 --2- 172.21.15.27:0/751868753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb2a50 0x55d82cfb2ed0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efd14009e30 tx=0x7efd14011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc67570297692270 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.864+0000 7efd2da00700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd100b2a70 0x7efd100b4f30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efd20009ec0 tx=0x7efd2000b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:41.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.864+0000 7efd1a000700 1 -- 172.21.15.27:0/751868753 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1401e070 con 0x55d82cfb2a50 2024-05-10T11:16:41.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.864+0000 7efd1a000700 1 -- 172.21.15.27:0/751868753 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd14019ae0 con 0x55d82cfb2a50 2024-05-10T11:16:41.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.864+0000 7efd1a000700 1 -- 172.21.15.27:0/751868753 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1400fa70 con 0x55d82cfb2a50 2024-05-10T11:16:41.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.865+0000 7efd3a30bb80 1 -- 172.21.15.27:0/751868753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb2a50 msgr2=0x55d82cfb2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.865+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/751868753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb2a50 0x55d82cfb2ed0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efd14009e30 tx=0x7efd14011720 comp rx=0 tx=0).stop 2024-05-10T11:16:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.865+0000 7efd3a30bb80 1 -- 172.21.15.27:0/751868753 shutdown_connections 2024-05-10T11:16:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.865+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/751868753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb2a50 0x55d82cfb2ed0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.865+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/751868753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb7da0 0x55d82cfb8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.865+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/751868753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfb2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.865+0000 7efd3a30bb80 1 -- 172.21.15.27:0/751868753 >> 172.21.15.27:0/751868753 conn(0x55d82cf2a6d0 msgr2=0x55d82cfacca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.865+0000 7efd3a30bb80 1 -- 172.21.15.27:0/751868753 shutdown_connections 2024-05-10T11:16:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.865+0000 7efd3a30bb80 1 -- 172.21.15.27:0/751868753 wait complete. 2024-05-10T11:16:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.866+0000 7efd3a30bb80 1 -- start start 2024-05-10T11:16:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.866+0000 7efd3a30bb80 1 Processor -- start 2024-05-10T11:16:41.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.867+0000 7efd3a30bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb2a50 0x55d82cfbe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.867+0000 7efd3a30bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb7da0 0x55d82cfbb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.867+0000 7efd3a30bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb86c0 0x55d82cfbbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.867+0000 7efd3a30bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d82cfbf150 con 0x55d82cfb7da0 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.867+0000 7efd3a30bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d82ce6ab20 con 0x55d82cfb2a50 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.867+0000 7efd3a30bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d82cfbf2d0 con 0x55d82cfb86c0 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.867+0000 7efd3a30bb80 10 client.0 _ll_register_callbacks cb 0x55d82ceabc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.867+0000 7efd2d000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb2a50 0x55d82cfbe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.867+0000 7efd2d000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb2a50 0x55d82cfbe3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41182/0 (socket says 172.21.15.27:41182) 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.867+0000 7efd2d000700 1 -- 172.21.15.27:0/3516370227 learned_addr learned my addr 172.21.15.27:0/3516370227 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.867+0000 7efd27e00700 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb7da0 0x55d82cfbb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.867+0000 7efd2da00700 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb86c0 0x55d82cfbbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.868+0000 7efd2d000700 1 -- 172.21.15.27:0/3516370227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb86c0 msgr2=0x55d82cfbbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.868+0000 7efd2d000700 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb86c0 0x55d82cfbbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.868+0000 7efd2d000700 1 -- 172.21.15.27:0/3516370227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb7da0 msgr2=0x55d82cfbb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.868+0000 7efd2d000700 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb7da0 0x55d82cfbb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.868+0000 7efd2d000700 1 -- 172.21.15.27:0/3516370227 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d82ceca420 con 0x55d82cfb2a50 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.868+0000 7efd2d000700 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb2a50 0x55d82cfbe3b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efd1c000c00 tx=0x7efd1c074ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea379996a7ae2b63 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.868+0000 7efd0ea00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.868+0000 7efd0ea00700 1 -- 172.21.15.27:0/3516370227 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c0095d0 con 0x55d82cfb2a50 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.868+0000 7efd0ea00700 1 -- 172.21.15.27:0/3516370227 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7efd1c0cbeb0 con 0x55d82cfb2a50 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.868+0000 7efd3a30bb80 1 -- 172.21.15.27:0/3516370227 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd14019ae0 con 0x55d82cfb2a50 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.869+0000 7efd0ea00700 1 -- 172.21.15.27:0/3516370227 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efd1c0aa020 con 0x55d82cfb2a50 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.869+0000 7efd3a30bb80 10 client.25871 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.869+0000 7efd3a30bb80 1 -- 172.21.15.27:0/3516370227 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d82cfc9970 con 0x55d82cfb2a50 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.869+0000 7efd0ea00700 1 -- 172.21.15.27:0/3516370227 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd14019ae0 con 0x55d82cfb2a50 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.869+0000 7efd0ea00700 1 -- 172.21.15.27:0/3516370227 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7efd1c0e0b70 con 0x55d82cfb2a50 2024-05-10T11:16:41.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.869+0000 7efd3a30bb80 10 client.25871 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:41.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.869+0000 7efd3a30bb80 10 client.25871 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:16:41.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.869+0000 7efd3a30bb80 20 client.25871 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:41.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.869+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d82cfcbf30 0x55d82cfce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.869+0000 7efd3a30bb80 4 client.25871 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:16:41.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.869+0000 7efd3a30bb80 1 -- 172.21.15.27:0/3516370227 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d82cf7f140 con 0x55d82cfcbf30 2024-05-10T11:16:41.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.870+0000 7efd27e00700 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d82cfcbf30 0x55d82cfce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.870+0000 7efd27e00700 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d82cfcbf30 0x55d82cfce3a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b4b36da40cfb640 server_cookie=2114a82d2208493d in_seq=0 out_seq=0 2024-05-10T11:16:41.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.870+0000 7efd0ea00700 10 client.25871 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:16:41.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.871+0000 7efd0ea00700 1 -- 172.21.15.27:0/3516370227 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d82cf7f140 con 0x55d82cfcbf30 2024-05-10T11:16:41.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.871+0000 7efd0ea00700 10 client.25871 handle_command_reply: tid=0 2024-05-10T11:16:41.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.871+0000 7efd3a30bb80 1 client.25871 shutdown 2024-05-10T11:16:41.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.872+0000 7efd3a30bb80 1 -- 172.21.15.27:0/3516370227 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb2a50 msgr2=0x55d82cfbe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:41.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.872+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb2a50 0x55d82cfbe3b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efd1c000c00 tx=0x7efd1c074ce0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.872+0000 7efd3a30bb80 1 -- 172.21.15.27:0/3516370227 shutdown_connections 2024-05-10T11:16:41.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.872+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d82cfcbf30 0x55d82cfce3a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.872+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb86c0 0x55d82cfbbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.872+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb2a50 0x55d82cfbe3b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.872+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/3516370227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb7da0 0x55d82cfbb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.872+0000 7efd3a30bb80 1 -- 172.21.15.27:0/3516370227 >> 172.21.15.27:0/3516370227 conn(0x55d82cf2a6d0 msgr2=0x55d82cfa7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:41.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.872+0000 7efd0ea00700 0 client.25871 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:16:41.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.872+0000 7efd3a30bb80 1 -- 172.21.15.27:0/3516370227 shutdown_connections 2024-05-10T11:16:41.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.872+0000 7efd3a30bb80 1 -- 172.21.15.27:0/3516370227 wait complete. 2024-05-10T11:16:41.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.872+0000 7efd3a30bb80 20 client.25871 trim_cache size 0 max 16384 2024-05-10T11:16:41.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.889+0000 7efd3a30bb80 1 Processor -- start 2024-05-10T11:16:41.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.889+0000 7efd3a30bb80 1 -- start start 2024-05-10T11:16:41.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.889+0000 7efd3a30bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb2a50 0x55d82cfb2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.889+0000 7efd3a30bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb7da0 0x55d82cfc81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.889+0000 7efd3a30bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfa7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.889+0000 7efd3a30bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efd281b13f0 con 0x55d82cfb86c0 2024-05-10T11:16:41.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.889+0000 7efd3a30bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efd281b2f90 con 0x55d82cfb2a50 2024-05-10T11:16:41.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.889+0000 7efd3a30bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efd281b1ac0 con 0x55d82cfb7da0 2024-05-10T11:16:41.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.890+0000 7efd27e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb7da0 0x55d82cfc81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.890+0000 7efd2da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfa7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.890+0000 7efd27e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb7da0 0x55d82cfc81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:44180/0 (socket says 172.21.15.27:44180) 2024-05-10T11:16:41.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.890+0000 7efd27e00700 1 -- 172.21.15.27:0/38007423 learned_addr learned my addr 172.21.15.27:0/38007423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.890+0000 7efd2d000700 1 --2- 172.21.15.27:0/38007423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb2a50 0x55d82cfb2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.890+0000 7efd27e00700 1 -- 172.21.15.27:0/38007423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb2a50 msgr2=0x55d82cfb2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.890+0000 7efd27e00700 1 --2- 172.21.15.27:0/38007423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb2a50 0x55d82cfb2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.890+0000 7efd27e00700 1 -- 172.21.15.27:0/38007423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 msgr2=0x55d82cfa7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.890+0000 7efd27e00700 1 --2- 172.21.15.27:0/38007423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfa7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.890+0000 7efd27e00700 1 -- 172.21.15.27:0/38007423 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d82cfa98a0 con 0x55d82cfb7da0 2024-05-10T11:16:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.891+0000 7efd27e00700 1 --2- 172.21.15.27:0/38007423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb7da0 0x55d82cfc81e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efd14009e30 tx=0x7efd14009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5110fa859f1c14d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.891+0000 7efd19600700 1 -- 172.21.15.27:0/38007423 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1401e040 con 0x55d82cfb7da0 2024-05-10T11:16:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.891+0000 7efd19600700 1 -- 172.21.15.27:0/38007423 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd14005560 con 0x55d82cfb7da0 2024-05-10T11:16:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.891+0000 7efd19600700 1 -- 172.21.15.27:0/38007423 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd140038c0 con 0x55d82cfb7da0 2024-05-10T11:16:41.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.891+0000 7efd3a30bb80 1 -- 172.21.15.27:0/38007423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb7da0 msgr2=0x55d82cfc81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:41.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.891+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/38007423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb7da0 0x55d82cfc81e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efd14009e30 tx=0x7efd14009900 comp rx=0 tx=0).stop 2024-05-10T11:16:41.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.892+0000 7efd3a30bb80 1 -- 172.21.15.27:0/38007423 shutdown_connections 2024-05-10T11:16:41.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.892+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/38007423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfa7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.892+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/38007423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb2a50 0x55d82cfb2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.892+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/38007423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb7da0 0x55d82cfc81e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.892+0000 7efd3a30bb80 1 -- 172.21.15.27:0/38007423 >> 172.21.15.27:0/38007423 conn(0x55d82cf2a6d0 msgr2=0x55d82cfb8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.892+0000 7efd3a30bb80 1 -- 172.21.15.27:0/38007423 shutdown_connections 2024-05-10T11:16:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.892+0000 7efd3a30bb80 1 -- 172.21.15.27:0/38007423 wait complete. 2024-05-10T11:16:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.892+0000 7efd3a30bb80 1 -- start start 2024-05-10T11:16:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.892+0000 7efd3a30bb80 1 Processor -- start 2024-05-10T11:16:41.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.893+0000 7efd3a30bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb2a50 0x55d82cfc4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.893+0000 7efd3a30bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb7da0 0x55d82cfd3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.893+0000 7efd3a30bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfd35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.893+0000 7efd3a30bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efd281b2860 con 0x55d82cfb86c0 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.893+0000 7efd3a30bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efd281b2190 con 0x55d82cfb7da0 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.893+0000 7efd3a30bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efd281af390 con 0x55d82cfb2a50 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.893+0000 7efd3a30bb80 10 client.0 _ll_register_callbacks cb 0x55d82ceabc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.893+0000 7efd2da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfd35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.893+0000 7efd2d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb2a50 0x55d82cfc4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.893+0000 7efd27e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb7da0 0x55d82cfd3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.893+0000 7efd2da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfd35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41332/0 (socket says 172.21.15.27:41332) 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.893+0000 7efd2da00700 1 -- 172.21.15.27:0/2842605438 learned_addr learned my addr 172.21.15.27:0/2842605438 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.894+0000 7efd27e00700 1 -- 172.21.15.27:0/2842605438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb2a50 msgr2=0x55d82cfc4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.894+0000 7efd27e00700 1 --2- 172.21.15.27:0/2842605438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb2a50 0x55d82cfc4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.894+0000 7efd27e00700 1 -- 172.21.15.27:0/2842605438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 msgr2=0x55d82cfd35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.894+0000 7efd27e00700 1 --2- 172.21.15.27:0/2842605438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfd35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.894+0000 7efd27e00700 1 -- 172.21.15.27:0/2842605438 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d82cfd3af0 con 0x55d82cfb7da0 2024-05-10T11:16:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.894+0000 7efd27e00700 1 --2- 172.21.15.27:0/2842605438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb7da0 0x55d82cfd3070 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x55d82cfad890 tx=0x7efd14009d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd882afe97600c61 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.894+0000 7efd0ea00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:16:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.894+0000 7efd0ea00700 1 -- 172.21.15.27:0/2842605438 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd14039020 con 0x55d82cfb7da0 2024-05-10T11:16:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.895+0000 7efd0ea00700 1 -- 172.21.15.27:0/2842605438 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7efd140050b0 con 0x55d82cfb7da0 2024-05-10T11:16:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.895+0000 7efd3a30bb80 1 -- 172.21.15.27:0/2842605438 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd14005560 con 0x55d82cfb7da0 2024-05-10T11:16:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.895+0000 7efd0ea00700 1 -- 172.21.15.27:0/2842605438 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efd14044050 con 0x55d82cfb7da0 2024-05-10T11:16:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.895+0000 7efd3a30bb80 10 client.25877 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.895+0000 7efd3a30bb80 1 -- 172.21.15.27:0/2842605438 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d82d03ed80 con 0x55d82cfb7da0 2024-05-10T11:16:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.895+0000 7efd0ea00700 1 -- 172.21.15.27:0/2842605438 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd14005560 con 0x55d82cfb7da0 2024-05-10T11:16:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.895+0000 7efd0ea00700 1 -- 172.21.15.27:0/2842605438 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7efd14003cc0 con 0x55d82cfb7da0 2024-05-10T11:16:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.895+0000 7efd3a30bb80 10 client.25877 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.895+0000 7efd3a30bb80 10 client.25877 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:16:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.896+0000 7efd3a30bb80 20 client.25877 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.896+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/2842605438 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d82d03f260 0x55d82d041680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.896+0000 7efd3a30bb80 4 client.25877 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:16:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.896+0000 7efd3a30bb80 1 -- 172.21.15.27:0/2842605438 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d82cf12400 con 0x55d82d03f260 2024-05-10T11:16:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.896+0000 7efd2d000700 1 --2- 172.21.15.27:0/2842605438 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d82d03f260 0x55d82d041680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.896+0000 7efd2d000700 1 --2- 172.21.15.27:0/2842605438 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d82d03f260 0x55d82d041680 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=17ef2f11d03a2b68 server_cookie=7d626d511c6616eb in_seq=0 out_seq=0 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.896+0000 7efd0ea00700 10 client.25877 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd0ea00700 1 -- 172.21.15.27:0/2842605438 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d82cf12400 con 0x55d82d03f260 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd0ea00700 10 client.25877 handle_command_reply: tid=0 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd3a30bb80 1 client.25877 shutdown 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd3a30bb80 1 -- 172.21.15.27:0/2842605438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb7da0 msgr2=0x55d82cfd3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/2842605438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb7da0 0x55d82cfd3070 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x55d82cfad890 tx=0x7efd14009d30 comp rx=0 tx=0).stop 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd3a30bb80 1 -- 172.21.15.27:0/2842605438 shutdown_connections 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/2842605438 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d82d03f260 0x55d82d041680 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/2842605438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d82cfb2a50 0x55d82cfc4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/2842605438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d82cfb86c0 0x55d82cfd35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd3a30bb80 1 --2- 172.21.15.27:0/2842605438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d82cfb7da0 0x55d82cfd3070 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd3a30bb80 1 -- 172.21.15.27:0/2842605438 >> 172.21.15.27:0/2842605438 conn(0x55d82cf2a6d0 msgr2=0x55d82cfbcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd0ea00700 0 client.25877 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:16:41.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd3a30bb80 1 -- 172.21.15.27:0/2842605438 shutdown_connections 2024-05-10T11:16:41.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd3a30bb80 1 -- 172.21.15.27:0/2842605438 wait complete. 2024-05-10T11:16:41.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.897+0000 7efd3a30bb80 20 client.25877 trim_cache size 0 max 16384 2024-05-10T11:16:41.901 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:16:41.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.898+0000 7efd2e400700 1 -- 172.21.15.27:0/3358911758 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd100b2a70 msgr2=0x7efd100b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:41.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.898+0000 7efd2e400700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd100b2a70 0x7efd100b4f30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efd20009ec0 tx=0x7efd2000b040 comp rx=0 tx=0).stop 2024-05-10T11:16:41.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.898+0000 7efd2e400700 1 -- 172.21.15.27:0/3358911758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd28102180 msgr2=0x7efd281a18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:41.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.898+0000 7efd2e400700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd28102180 0x7efd281a18c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efd1c0059d0 tx=0x7efd1c0047b0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.898+0000 7efd2e400700 1 -- 172.21.15.27:0/3358911758 shutdown_connections 2024-05-10T11:16:41.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.898+0000 7efd2e400700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd100b2a70 0x7efd100b4f30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.898+0000 7efd2e400700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd2810fd30 0x7efd2819bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.898+0000 7efd2e400700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd28102180 0x7efd281a18c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.898+0000 7efd2e400700 1 --2- 172.21.15.27:0/3358911758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd28102b30 0x7efd281a1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:41.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.898+0000 7efd2e400700 1 -- 172.21.15.27:0/3358911758 >> 172.21.15.27:0/3358911758 conn(0x7efd280fddc0 msgr2=0x7efd281001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:41.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.898+0000 7efd2e400700 1 -- 172.21.15.27:0/3358911758 shutdown_connections 2024-05-10T11:16:41.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:41.898+0000 7efd2e400700 1 -- 172.21.15.27:0/3358911758 wait complete. 2024-05-10T11:16:42.660 INFO:tasks.workunit.client.0.smithi027.stdout: 1 120951 7.71 MB/sec warmup 87 sec latency 4272.884 ms 2024-05-10T11:16:43.066 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:16:43.538 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:16:43.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:43 smithi027 ceph-mon[30054]: pgmap v108: 129 pgs: 129 active+clean; 77 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-05-10T11:16:43.660 INFO:tasks.workunit.client.0.smithi027.stdout: 1 121079 7.63 MB/sec warmup 88 sec latency 5020.153 ms 2024-05-10T11:16:43.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:43 smithi076 ceph-mon[34152]: pgmap v108: 129 pgs: 129 active+clean; 77 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-05-10T11:16:43.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:43 smithi189 ceph-mon[34842]: pgmap v108: 129 pgs: 129 active+clean; 77 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-05-10T11:16:44.660 INFO:tasks.workunit.client.0.smithi027.stdout: 1 122929 7.62 MB/sec warmup 89 sec latency 24.962 ms 2024-05-10T11:16:45.589 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:45 smithi027 ceph-mon[30054]: pgmap v109: 129 pgs: 129 active+clean; 78 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 193 op/s 2024-05-10T11:16:45.660 INFO:tasks.workunit.client.0.smithi027.stdout: 1 124441 7.67 MB/sec warmup 90 sec latency 24.203 ms 2024-05-10T11:16:45.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:45 smithi076 ceph-mon[34152]: pgmap v109: 129 pgs: 129 active+clean; 78 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 193 op/s 2024-05-10T11:16:45.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:45 smithi189 ceph-mon[34842]: pgmap v109: 129 pgs: 129 active+clean; 78 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 193 op/s 2024-05-10T11:16:46.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.180+0000 7fd42a400700 1 -- 172.21.15.27:0/393359485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4241045e0 msgr2=0x7fd4241049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:46.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.180+0000 7fd42a400700 1 --2- 172.21.15.27:0/393359485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4241045e0 0x7fd4241049c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd418009990 tx=0x7fd418009ca0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.180+0000 7fd42a400700 1 -- 172.21.15.27:0/393359485 shutdown_connections 2024-05-10T11:16:46.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.180+0000 7fd42a400700 1 --2- 172.21.15.27:0/393359485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd42410da10 0x7fd42410fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.180+0000 7fd42a400700 1 --2- 172.21.15.27:0/393359485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4241045e0 0x7fd4241049c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.180+0000 7fd42a400700 1 --2- 172.21.15.27:0/393359485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd424104f00 0x7fd42410d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.180+0000 7fd42a400700 1 -- 172.21.15.27:0/393359485 >> 172.21.15.27:0/393359485 conn(0x7fd4240fde30 msgr2=0x7fd424100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:46.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.180+0000 7fd42a400700 1 -- 172.21.15.27:0/393359485 shutdown_connections 2024-05-10T11:16:46.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.180+0000 7fd42a400700 1 -- 172.21.15.27:0/393359485 wait complete. 2024-05-10T11:16:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.181+0000 7fd42a400700 1 Processor -- start 2024-05-10T11:16:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.182+0000 7fd42a400700 1 -- start start 2024-05-10T11:16:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.182+0000 7fd42a400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4241045e0 0x7fd42419bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.182+0000 7fd42a400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd424104f00 0x7fd42419c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.182+0000 7fd42a400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd42410da10 0x7fd4241a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.182+0000 7fd42a400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd424113650 con 0x7fd42410da10 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.182+0000 7fd42a400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd4241134d0 con 0x7fd4241045e0 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.182+0000 7fd42a400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd4241137d0 con 0x7fd424104f00 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.182+0000 7fd423e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd424104f00 0x7fd42419c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.182+0000 7fd429000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4241045e0 0x7fd42419bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.182+0000 7fd423e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd424104f00 0x7fd42419c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:33358/0 (socket says 172.21.15.27:33358) 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.182+0000 7fd423e00700 1 -- 172.21.15.27:0/3535385197 learned_addr learned my addr 172.21.15.27:0/3535385197 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.183+0000 7fd429a00700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd42410da10 0x7fd4241a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.183+0000 7fd423e00700 1 -- 172.21.15.27:0/3535385197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4241045e0 msgr2=0x7fd42419bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.183+0000 7fd423e00700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4241045e0 0x7fd42419bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.183+0000 7fd423e00700 1 -- 172.21.15.27:0/3535385197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd42410da10 msgr2=0x7fd4241a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:46.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.183+0000 7fd423e00700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd42410da10 0x7fd4241a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.183+0000 7fd423e00700 1 -- 172.21.15.27:0/3535385197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4241a0df0 con 0x7fd424104f00 2024-05-10T11:16:46.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.183+0000 7fd429000700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4241045e0 0x7fd42419bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:16:46.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.183+0000 7fd429a00700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd42410da10 0x7fd4241a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:16:46.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.183+0000 7fd423e00700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd424104f00 0x7fd42419c320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd41000e9c0 tx=0x7fd41000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:46.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.184+0000 7fd421600700 1 -- 172.21.15.27:0/3535385197 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41000cbd0 con 0x7fd424104f00 2024-05-10T11:16:46.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.184+0000 7fd421600700 1 -- 172.21.15.27:0/3535385197 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd41000cd70 con 0x7fd424104f00 2024-05-10T11:16:46.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.184+0000 7fd42a400700 1 -- 172.21.15.27:0/3535385197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4241a10e0 con 0x7fd424104f00 2024-05-10T11:16:46.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.184+0000 7fd421600700 1 -- 172.21.15.27:0/3535385197 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd410005580 con 0x7fd424104f00 2024-05-10T11:16:46.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.184+0000 7fd42a400700 1 -- 172.21.15.27:0/3535385197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4241a8960 con 0x7fd424104f00 2024-05-10T11:16:46.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.187+0000 7fd421600700 1 -- 172.21.15.27:0/3535385197 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd410004750 con 0x7fd424104f00 2024-05-10T11:16:46.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.189+0000 7fd436420b80 1 Processor -- start 2024-05-10T11:16:46.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.189+0000 7fd436420b80 1 -- start start 2024-05-10T11:16:46.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.189+0000 7fd421600700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd40c0b2a70 0x7fd40c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.189+0000 7fd436420b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc6e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd436420b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc687f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd436420b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc68e10 0x5607fdc69290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd436420b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5607fdbb2a20 con 0x5607fdc68e10 2024-05-10T11:16:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd436420b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5607fdb50760 con 0x5607fdc6e1f0 2024-05-10T11:16:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd436420b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5607fdbae9e0 con 0x5607fdc6eb10 2024-05-10T11:16:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd421600700 1 -- 172.21.15.27:0/3535385197 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fd4100e2d20 con 0x7fd424104f00 2024-05-10T11:16:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd423e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc687f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd423e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc687f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:33372/0 (socket says 172.21.15.27:33372) 2024-05-10T11:16:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd423e00700 1 -- 172.21.15.27:0/1681492820 learned_addr learned my addr 172.21.15.27:0/1681492820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd429a00700 1 --2- 172.21.15.27:0/1681492820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc68e10 0x5607fdc69290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd429a00700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd40c0b2a70 0x7fd40c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd429000700 1 --2- 172.21.15.27:0/1681492820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc6e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd423e00700 1 -- 172.21.15.27:0/1681492820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 msgr2=0x5607fdc6e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd423e00700 1 --2- 172.21.15.27:0/1681492820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc6e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd423e00700 1 -- 172.21.15.27:0/1681492820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc68e10 msgr2=0x5607fdc69290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd423e00700 1 --2- 172.21.15.27:0/1681492820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc68e10 0x5607fdc69290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd423e00700 1 -- 172.21.15.27:0/1681492820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607fdaff8d0 con 0x5607fdc6eb10 2024-05-10T11:16:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.190+0000 7fd429a00700 1 --2- 172.21.15.27:0/1681492820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc68e10 0x5607fdc69290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:16:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.191+0000 7fd423e00700 1 --2- 172.21.15.27:0/1681492820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc687f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd41000cfd0 tx=0x7fd4100a9410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7efd9155f6accf5d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.191+0000 7fd416000700 1 -- 172.21.15.27:0/1681492820 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd410004ed0 con 0x5607fdc6eb10 2024-05-10T11:16:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.191+0000 7fd416000700 1 -- 172.21.15.27:0/1681492820 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4100a96e0 con 0x5607fdc6eb10 2024-05-10T11:16:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.191+0000 7fd416000700 1 -- 172.21.15.27:0/1681492820 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4100aa070 con 0x5607fdc6eb10 2024-05-10T11:16:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.192+0000 7fd436420b80 1 -- 172.21.15.27:0/1681492820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 msgr2=0x5607fdc687f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.192+0000 7fd429a00700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd40c0b2a70 0x7fd40c0b4f30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x5607fdc697d0 tx=0x7fd41c00b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.192+0000 7fd436420b80 1 --2- 172.21.15.27:0/1681492820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc687f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd41000cfd0 tx=0x7fd4100a9410 comp rx=0 tx=0).stop 2024-05-10T11:16:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.192+0000 7fd436420b80 1 -- 172.21.15.27:0/1681492820 shutdown_connections 2024-05-10T11:16:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.192+0000 7fd436420b80 1 --2- 172.21.15.27:0/1681492820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc68e10 0x5607fdc69290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.192+0000 7fd436420b80 1 --2- 172.21.15.27:0/1681492820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc6e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.192+0000 7fd436420b80 1 --2- 172.21.15.27:0/1681492820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc687f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.192+0000 7fd436420b80 1 -- 172.21.15.27:0/1681492820 >> 172.21.15.27:0/1681492820 conn(0x5607fdb8eb60 msgr2=0x5607fdc5a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.192+0000 7fd436420b80 1 -- 172.21.15.27:0/1681492820 shutdown_connections 2024-05-10T11:16:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.192+0000 7fd436420b80 1 -- 172.21.15.27:0/1681492820 wait complete. 2024-05-10T11:16:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.193+0000 7fd436420b80 1 -- start start 2024-05-10T11:16:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.193+0000 7fd436420b80 1 Processor -- start 2024-05-10T11:16:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.194+0000 7fd436420b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc68e10 0x5607fdc74780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.194+0000 7fd436420b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc71b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.194+0000 7fd436420b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc6eb10 0x5607fdc72060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.194+0000 7fd436420b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5607fdc755c0 con 0x5607fdc6eb10 2024-05-10T11:16:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.194+0000 7fd436420b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5607fdb18b20 con 0x5607fdc6e1f0 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.194+0000 7fd436420b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5607fdc75740 con 0x5607fdc68e10 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.194+0000 7fd436420b80 10 client.0 _ll_register_callbacks cb 0x5607fdb59c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.194+0000 7fd429000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc68e10 0x5607fdc74780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.194+0000 7fd429000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc68e10 0x5607fdc74780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:33378/0 (socket says 172.21.15.27:33378) 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.194+0000 7fd429000700 1 -- 172.21.15.27:0/3105820782 learned_addr learned my addr 172.21.15.27:0/3105820782 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.194+0000 7fd423e00700 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc71b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.194+0000 7fd429a00700 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc6eb10 0x5607fdc72060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.195+0000 7fd429000700 1 -- 172.21.15.27:0/3105820782 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 msgr2=0x5607fdc71b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.195+0000 7fd429000700 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc71b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.195+0000 7fd429000700 1 -- 172.21.15.27:0/3105820782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc6eb10 msgr2=0x5607fdc72060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.195+0000 7fd429000700 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc6eb10 0x5607fdc72060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.195+0000 7fd429000700 1 -- 172.21.15.27:0/3105820782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607fdb78420 con 0x5607fdc68e10 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.195+0000 7fd423e00700 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc71b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:16:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.195+0000 7fd429000700 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc68e10 0x5607fdc74780 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd418009740 tx=0x7fd41800bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e04a720dede5989a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.195+0000 7fd40aa00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:16:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.195+0000 7fd40aa00700 1 -- 172.21.15.27:0/3105820782 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418028060 con 0x5607fdc68e10 2024-05-10T11:16:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.195+0000 7fd40aa00700 1 -- 172.21.15.27:0/3105820782 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fd418003680 con 0x5607fdc68e10 2024-05-10T11:16:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.196+0000 7fd436420b80 1 -- 172.21.15.27:0/3105820782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4100a96e0 con 0x5607fdc68e10 2024-05-10T11:16:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.196+0000 7fd40aa00700 1 -- 172.21.15.27:0/3105820782 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd41802f050 con 0x5607fdc68e10 2024-05-10T11:16:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.196+0000 7fd436420b80 10 client.25876 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.196+0000 7fd436420b80 1 -- 172.21.15.27:0/3105820782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5607fdc7fd40 con 0x5607fdc68e10 2024-05-10T11:16:46.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.196+0000 7fd40aa00700 1 -- 172.21.15.27:0/3105820782 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4100a96e0 con 0x5607fdc68e10 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.196+0000 7fd40aa00700 1 -- 172.21.15.27:0/3105820782 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fd41800f9f0 con 0x5607fdc68e10 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.196+0000 7fd436420b80 10 client.25876 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.197+0000 7fd436420b80 10 client.25876 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.197+0000 7fd436420b80 20 client.25876 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.197+0000 7fd436420b80 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5607fdc82380 0x5607fdc84790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.197+0000 7fd436420b80 4 client.25876 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.197+0000 7fd436420b80 1 -- 172.21.15.27:0/3105820782 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607fdc2d0a0 con 0x5607fdc82380 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.197+0000 7fd423e00700 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5607fdc82380 0x5607fdc84790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.198+0000 7fd423e00700 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5607fdc82380 0x5607fdc84790 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=55c02ede218a84ba server_cookie=c3b5ee332b7e4cfb in_seq=0 out_seq=0 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.198+0000 7fd40aa00700 10 client.25876 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.198+0000 7fd40aa00700 1 -- 172.21.15.27:0/3105820782 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5607fdc2d0a0 con 0x5607fdc82380 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.198+0000 7fd40aa00700 10 client.25876 handle_command_reply: tid=0 2024-05-10T11:16:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.199+0000 7fd436420b80 1 client.25876 shutdown 2024-05-10T11:16:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.199+0000 7fd436420b80 1 -- 172.21.15.27:0/3105820782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc68e10 msgr2=0x5607fdc74780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.199+0000 7fd436420b80 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc68e10 0x5607fdc74780 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd418009740 tx=0x7fd41800bf90 comp rx=0 tx=0).stop 2024-05-10T11:16:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.199+0000 7fd436420b80 1 -- 172.21.15.27:0/3105820782 shutdown_connections 2024-05-10T11:16:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.199+0000 7fd436420b80 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5607fdc82380 0x5607fdc84790 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.199+0000 7fd436420b80 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc68e10 0x5607fdc74780 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.199+0000 7fd436420b80 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc6eb10 0x5607fdc72060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.199+0000 7fd436420b80 1 --2- 172.21.15.27:0/3105820782 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc71b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.199+0000 7fd436420b80 1 -- 172.21.15.27:0/3105820782 >> 172.21.15.27:0/3105820782 conn(0x5607fdb8eb60 msgr2=0x5607fdc55460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.199+0000 7fd40aa00700 0 client.25876 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:16:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.200+0000 7fd436420b80 1 -- 172.21.15.27:0/3105820782 shutdown_connections 2024-05-10T11:16:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.200+0000 7fd436420b80 1 -- 172.21.15.27:0/3105820782 wait complete. 2024-05-10T11:16:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.200+0000 7fd436420b80 20 client.25876 trim_cache size 0 max 16384 2024-05-10T11:16:46.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.218+0000 7fd436420b80 1 Processor -- start 2024-05-10T11:16:46.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.218+0000 7fd436420b80 1 -- start start 2024-05-10T11:16:46.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.219+0000 7fd436420b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc68e10 0x5607fdc7e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.219+0000 7fd436420b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc54400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.219+0000 7fd436420b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc6f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.219+0000 7fd436420b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd4241ad4d0 con 0x5607fdc68e10 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.219+0000 7fd436420b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd4241af070 con 0x5607fdc6e1f0 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.219+0000 7fd436420b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd4241adba0 con 0x5607fdc6eb10 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.219+0000 7fd429000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc68e10 0x5607fdc7e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.219+0000 7fd423e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc54400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.219+0000 7fd423e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc54400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35400/0 (socket says 172.21.15.27:35400) 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.219+0000 7fd429a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc6f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.219+0000 7fd423e00700 1 -- 172.21.15.27:0/1327783721 learned_addr learned my addr 172.21.15.27:0/1327783721 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.220+0000 7fd423e00700 1 -- 172.21.15.27:0/1327783721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 msgr2=0x5607fdc6f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.220+0000 7fd423e00700 1 --2- 172.21.15.27:0/1327783721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc6f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.220+0000 7fd423e00700 1 -- 172.21.15.27:0/1327783721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc68e10 msgr2=0x5607fdc7e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.220+0000 7fd423e00700 1 --2- 172.21.15.27:0/1327783721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc68e10 0x5607fdc7e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.220+0000 7fd423e00700 1 -- 172.21.15.27:0/1327783721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607fdc700c0 con 0x5607fdc6e1f0 2024-05-10T11:16:46.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.220+0000 7fd423e00700 1 --2- 172.21.15.27:0/1327783721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc54400 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd4100a98d0 tx=0x7fd4100a5490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=beeaf23ee9e3143c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:46.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.220+0000 7fd415600700 1 -- 172.21.15.27:0/1327783721 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4100c42b0 con 0x5607fdc6e1f0 2024-05-10T11:16:46.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.220+0000 7fd415600700 1 -- 172.21.15.27:0/1327783721 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd41008e070 con 0x5607fdc6e1f0 2024-05-10T11:16:46.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.220+0000 7fd415600700 1 -- 172.21.15.27:0/1327783721 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4100b2a30 con 0x5607fdc6e1f0 2024-05-10T11:16:46.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.221+0000 7fd436420b80 1 -- 172.21.15.27:0/1327783721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 msgr2=0x5607fdc54400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:46.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.221+0000 7fd436420b80 1 --2- 172.21.15.27:0/1327783721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc54400 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd4100a98d0 tx=0x7fd4100a5490 comp rx=0 tx=0).stop 2024-05-10T11:16:46.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.221+0000 7fd436420b80 1 -- 172.21.15.27:0/1327783721 shutdown_connections 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.221+0000 7fd436420b80 1 --2- 172.21.15.27:0/1327783721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc6f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.221+0000 7fd436420b80 1 --2- 172.21.15.27:0/1327783721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc68e10 0x5607fdc7e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.221+0000 7fd436420b80 1 --2- 172.21.15.27:0/1327783721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc6e1f0 0x5607fdc54400 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.221+0000 7fd436420b80 1 -- 172.21.15.27:0/1327783721 >> 172.21.15.27:0/1327783721 conn(0x5607fdb8eb60 msgr2=0x5607fdc6ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.221+0000 7fd436420b80 1 -- 172.21.15.27:0/1327783721 shutdown_connections 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.221+0000 7fd436420b80 1 -- 172.21.15.27:0/1327783721 wait complete. 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.222+0000 7fd436420b80 1 -- start start 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.222+0000 7fd436420b80 1 Processor -- start 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd436420b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc68e10 0x5607fdc7ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd436420b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc6e1f0 0x5607fdc89450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd436420b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc89990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd436420b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd4241ae940 con 0x5607fdc6e1f0 2024-05-10T11:16:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd436420b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd4241ae270 con 0x5607fdc68e10 2024-05-10T11:16:46.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd436420b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd4241ab490 con 0x5607fdc6eb10 2024-05-10T11:16:46.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd436420b80 10 client.0 _ll_register_callbacks cb 0x5607fdb59c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:46.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd429000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc68e10 0x5607fdc7ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd429000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc68e10 0x5607fdc7ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35414/0 (socket says 172.21.15.27:35414) 2024-05-10T11:16:46.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd429000700 1 -- 172.21.15.27:0/1193828655 learned_addr learned my addr 172.21.15.27:0/1193828655 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:46.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd429a00700 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc89990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd423e00700 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc6e1f0 0x5607fdc89450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd429000700 1 -- 172.21.15.27:0/1193828655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 msgr2=0x5607fdc89990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd429000700 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc89990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd429000700 1 -- 172.21.15.27:0/1193828655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc6e1f0 msgr2=0x5607fdc89450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd429000700 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc6e1f0 0x5607fdc89450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.223+0000 7fd429000700 1 -- 172.21.15.27:0/1193828655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607fdc8a0a0 con 0x5607fdc68e10 2024-05-10T11:16:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.224+0000 7fd429000700 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc68e10 0x5607fdc7ae80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd418009990 tx=0x7fd418005a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6951b1cf3fc6572 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.224+0000 7fd40aa00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:16:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.224+0000 7fd40aa00700 1 -- 172.21.15.27:0/1193828655 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418028040 con 0x5607fdc68e10 2024-05-10T11:16:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.224+0000 7fd436420b80 1 -- 172.21.15.27:0/1193828655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd41008e070 con 0x5607fdc68e10 2024-05-10T11:16:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.224+0000 7fd40aa00700 1 -- 172.21.15.27:0/1193828655 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fd418017bf0 con 0x5607fdc68e10 2024-05-10T11:16:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.224+0000 7fd40aa00700 1 -- 172.21.15.27:0/1193828655 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd418041050 con 0x5607fdc68e10 2024-05-10T11:16:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.224+0000 7fd436420b80 10 client.25895 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.224+0000 7fd436420b80 1 -- 172.21.15.27:0/1193828655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5607fdcecaf0 con 0x5607fdc68e10 2024-05-10T11:16:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.225+0000 7fd40aa00700 1 -- 172.21.15.27:0/1193828655 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd41008e070 con 0x5607fdc68e10 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.225+0000 7fd40aa00700 1 -- 172.21.15.27:0/1193828655 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fd418003680 con 0x5607fdc68e10 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.225+0000 7fd436420b80 10 client.25895 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.225+0000 7fd436420b80 10 client.25895 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.225+0000 7fd436420b80 20 client.25895 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.225+0000 7fd436420b80 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5607fdced190 0x5607fdcef5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.225+0000 7fd436420b80 4 client.25895 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.225+0000 7fd436420b80 1 -- 172.21.15.27:0/1193828655 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5607fdbc0400 con 0x5607fdced190 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.225+0000 7fd423e00700 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5607fdced190 0x5607fdcef5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd423e00700 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5607fdced190 0x5607fdcef5b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=62d2fad007569dbd server_cookie=ff2dd67e380693f4 in_seq=0 out_seq=0 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd40aa00700 10 client.25895 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd40aa00700 1 -- 172.21.15.27:0/1193828655 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5607fdbc0400 con 0x5607fdced190 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd40aa00700 10 client.25895 handle_command_reply: tid=0 2024-05-10T11:16:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd436420b80 1 client.25895 shutdown 2024-05-10T11:16:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd436420b80 1 -- 172.21.15.27:0/1193828655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc68e10 msgr2=0x5607fdc7ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd436420b80 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc68e10 0x5607fdc7ae80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd418009990 tx=0x7fd418005a00 comp rx=0 tx=0).stop 2024-05-10T11:16:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd436420b80 1 -- 172.21.15.27:0/1193828655 shutdown_connections 2024-05-10T11:16:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd436420b80 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5607fdced190 0x5607fdcef5b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd436420b80 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607fdc6eb10 0x5607fdc89990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd436420b80 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5607fdc68e10 0x5607fdc7ae80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd436420b80 1 --2- 172.21.15.27:0/1193828655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607fdc6e1f0 0x5607fdc89450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd40aa00700 0 client.25895 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:16:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.226+0000 7fd436420b80 1 -- 172.21.15.27:0/1193828655 >> 172.21.15.27:0/1193828655 conn(0x5607fdb8eb60 msgr2=0x5607fdc54bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.227+0000 7fd436420b80 1 -- 172.21.15.27:0/1193828655 shutdown_connections 2024-05-10T11:16:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.227+0000 7fd436420b80 1 -- 172.21.15.27:0/1193828655 wait complete. 2024-05-10T11:16:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.227+0000 7fd436420b80 20 client.25895 trim_cache size 0 max 16384 2024-05-10T11:16:46.231 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:16:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.228+0000 7fd42a400700 1 -- 172.21.15.27:0/3535385197 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd40c0b2a70 msgr2=0x7fd40c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.228+0000 7fd42a400700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd40c0b2a70 0x7fd40c0b4f30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x5607fdc697d0 tx=0x7fd41c00b040 comp rx=0 tx=0).stop 2024-05-10T11:16:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.228+0000 7fd42a400700 1 -- 172.21.15.27:0/3535385197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd424104f00 msgr2=0x7fd42419c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.228+0000 7fd42a400700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd424104f00 0x7fd42419c320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd41000e9c0 tx=0x7fd41000ecd0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.228+0000 7fd42a400700 1 -- 172.21.15.27:0/3535385197 shutdown_connections 2024-05-10T11:16:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.228+0000 7fd42a400700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd40c0b2a70 0x7fd40c0b4f30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.228+0000 7fd42a400700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd42410da10 0x7fd4241a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.228+0000 7fd42a400700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4241045e0 0x7fd42419bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.228+0000 7fd42a400700 1 --2- 172.21.15.27:0/3535385197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd424104f00 0x7fd42419c320 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.228+0000 7fd42a400700 1 -- 172.21.15.27:0/3535385197 >> 172.21.15.27:0/3535385197 conn(0x7fd4240fde30 msgr2=0x7fd424100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.228+0000 7fd42a400700 1 -- 172.21.15.27:0/3535385197 shutdown_connections 2024-05-10T11:16:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:46.228+0000 7fd42a400700 1 -- 172.21.15.27:0/3535385197 wait complete. 2024-05-10T11:16:46.660 INFO:tasks.workunit.client.0.smithi027.stdout: 1 125109 7.59 MB/sec warmup 91 sec latency 55.371 ms 2024-05-10T11:16:47.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:16:47.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:47 smithi027 ceph-mon[30054]: pgmap v110: 129 pgs: 129 active+clean; 78 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 1.4 MiB/s wr, 110 op/s 2024-05-10T11:16:47.660 INFO:tasks.workunit.client.0.smithi027.stdout: 1 127453 7.66 MB/sec warmup 92 sec latency 12.843 ms 2024-05-10T11:16:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:47 smithi076 ceph-mon[34152]: pgmap v110: 129 pgs: 129 active+clean; 78 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 1.4 MiB/s wr, 110 op/s 2024-05-10T11:16:47.709 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:16:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:47 smithi189 ceph-mon[34842]: pgmap v110: 129 pgs: 129 active+clean; 78 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 1.4 MiB/s wr, 110 op/s 2024-05-10T11:16:48.661 INFO:tasks.workunit.client.0.smithi027.stdout: 1 128173 7.63 MB/sec warmup 93 sec latency 46.128 ms 2024-05-10T11:16:49.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:49 smithi027 ceph-mon[30054]: pgmap v111: 129 pgs: 129 active+clean; 80 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 188 op/s 2024-05-10T11:16:49.661 INFO:tasks.workunit.client.0.smithi027.stdout: 1 130073 7.62 MB/sec warmup 94 sec latency 29.105 ms 2024-05-10T11:16:49.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:49 smithi076 ceph-mon[34152]: pgmap v111: 129 pgs: 129 active+clean; 80 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 188 op/s 2024-05-10T11:16:49.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:49 smithi189 ceph-mon[34842]: pgmap v111: 129 pgs: 129 active+clean; 80 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 188 op/s 2024-05-10T11:16:50.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.160+0000 7f17d6a00700 1 -- 172.21.15.27:0/488871106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d010a490 msgr2=0x7f17d01169a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:50.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.160+0000 7f17d6a00700 1 --2- 172.21.15.27:0/488871106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d010a490 0x7f17d01169a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f17c800b0a0 tx=0x7f17c800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.160+0000 7f17d6a00700 1 -- 172.21.15.27:0/488871106 shutdown_connections 2024-05-10T11:16:50.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.160+0000 7f17d6a00700 1 --2- 172.21.15.27:0/488871106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17d010fb60 0x7f17d010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.160+0000 7f17d6a00700 1 --2- 172.21.15.27:0/488871106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d010a490 0x7f17d01169a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.160+0000 7f17d6a00700 1 --2- 172.21.15.27:0/488871106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d0109ad0 0x7f17d0109f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.160+0000 7f17d6a00700 1 -- 172.21.15.27:0/488871106 >> 172.21.15.27:0/488871106 conn(0x7f17d0076c70 msgr2=0x7f17d01082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:50.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.160+0000 7f17d6a00700 1 -- 172.21.15.27:0/488871106 shutdown_connections 2024-05-10T11:16:50.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.160+0000 7f17d6a00700 1 -- 172.21.15.27:0/488871106 wait complete. 2024-05-10T11:16:50.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.161+0000 7f17d6a00700 1 Processor -- start 2024-05-10T11:16:50.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.162+0000 7f17d6a00700 1 -- start start 2024-05-10T11:16:50.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.162+0000 7f17d6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d0109ad0 0x7f17d01a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.162+0000 7f17d6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d010a490 0x7f17d01a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.162+0000 7f17d6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17d010fb60 0x7f17d01a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.162+0000 7f17d6a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0119d20 con 0x7f17d0109ad0 2024-05-10T11:16:50.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.162+0000 7f17d6a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0119ba0 con 0x7f17d010a490 2024-05-10T11:16:50.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.162+0000 7f17d6a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0119ea0 con 0x7f17d010fb60 2024-05-10T11:16:50.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.162+0000 7f17cf400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d010a490 0x7f17d01a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.162+0000 7f17cfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d0109ad0 0x7f17d01a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.162+0000 7f17cf400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d010a490 0x7f17d01a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35430/0 (socket says 172.21.15.27:35430) 2024-05-10T11:16:50.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.162+0000 7f17cf400700 1 -- 172.21.15.27:0/1585064246 learned_addr learned my addr 172.21.15.27:0/1585064246 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:50.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.163+0000 7f17d4a00700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17d010fb60 0x7f17d01a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.163+0000 7f17cf400700 1 -- 172.21.15.27:0/1585064246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17d010fb60 msgr2=0x7f17d01a4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:50.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.163+0000 7f17cf400700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17d010fb60 0x7f17d01a4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.163+0000 7f17cf400700 1 -- 172.21.15.27:0/1585064246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d0109ad0 msgr2=0x7f17d01a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:50.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.163+0000 7f17cf400700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d0109ad0 0x7f17d01a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.163+0000 7f17cf400700 1 -- 172.21.15.27:0/1585064246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d01a5160 con 0x7f17d010a490 2024-05-10T11:16:50.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.163+0000 7f17cf400700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d010a490 0x7f17d01a0690 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00eb30 tx=0x7f17bc00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:50.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.164+0000 7f17ccc00700 1 -- 172.21.15.27:0/1585064246 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00cbd0 con 0x7f17d010a490 2024-05-10T11:16:50.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.164+0000 7f17ccc00700 1 -- 172.21.15.27:0/1585064246 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17bc0044d0 con 0x7f17d010a490 2024-05-10T11:16:50.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.164+0000 7f17d6a00700 1 -- 172.21.15.27:0/1585064246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d01a5450 con 0x7f17d010a490 2024-05-10T11:16:50.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.164+0000 7f17ccc00700 1 -- 172.21.15.27:0/1585064246 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc003e30 con 0x7f17d010a490 2024-05-10T11:16:50.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.164+0000 7f17d6a00700 1 -- 172.21.15.27:0/1585064246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d01accd0 con 0x7f17d010a490 2024-05-10T11:16:50.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.168+0000 7f17ccc00700 1 -- 172.21.15.27:0/1585064246 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f17bc004000 con 0x7f17d010a490 2024-05-10T11:16:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.169+0000 7f17e134eb80 1 Processor -- start 2024-05-10T11:16:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.169+0000 7f17e134eb80 1 -- start start 2024-05-10T11:16:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.169+0000 7f17e134eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef60187bc0 0x55ef60187fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.169+0000 7f17e134eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef601884e0 0x55ef601822d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.169+0000 7f17e134eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef60182d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.169+0000 7f17e134eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ef600d4a20 con 0x55ef60187bc0 2024-05-10T11:16:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.169+0000 7f17e134eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ef60072760 con 0x55ef601828f0 2024-05-10T11:16:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.169+0000 7f17e134eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ef600d09e0 con 0x55ef601884e0 2024-05-10T11:16:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.170+0000 7f17cfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef601884e0 0x55ef601822d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.170+0000 7f17cf400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef60182d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.170+0000 7f17d4a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef60187bc0 0x55ef60187fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.170+0000 7f17cfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef601884e0 0x55ef601822d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:33412/0 (socket says 172.21.15.27:33412) 2024-05-10T11:16:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.170+0000 7f17cfe00700 1 -- 172.21.15.27:0/2568670872 learned_addr learned my addr 172.21.15.27:0/2568670872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.170+0000 7f17cfe00700 1 -- 172.21.15.27:0/2568670872 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 msgr2=0x55ef60182d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.170+0000 7f17cfe00700 1 --2- 172.21.15.27:0/2568670872 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef60182d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.170+0000 7f17cfe00700 1 -- 172.21.15.27:0/2568670872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef60187bc0 msgr2=0x55ef60187fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.170+0000 7f17cfe00700 1 --2- 172.21.15.27:0/2568670872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef60187bc0 0x55ef60187fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.170+0000 7f17cfe00700 1 -- 172.21.15.27:0/2568670872 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef600218d0 con 0x55ef601884e0 2024-05-10T11:16:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.171+0000 7f17ccc00700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f17b40b2a70 0x7f17b40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.171+0000 7f17cfe00700 1 --2- 172.21.15.27:0/2568670872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef601884e0 0x55ef601822d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f17c0009d00 tx=0x7f17c0011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cadb53105eead4d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.171+0000 7f17ccc00700 1 -- 172.21.15.27:0/1585064246 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f17bc0e2d40 con 0x7f17d010a490 2024-05-10T11:16:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.171+0000 7f17c5600700 1 -- 172.21.15.27:0/2568670872 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c001e070 con 0x55ef601884e0 2024-05-10T11:16:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.171+0000 7f17d4a00700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f17b40b2a70 0x7f17b40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.171+0000 7f17c5600700 1 -- 172.21.15.27:0/2568670872 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c0019ae0 con 0x55ef601884e0 2024-05-10T11:16:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.171+0000 7f17c5600700 1 -- 172.21.15.27:0/2568670872 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c000ea90 con 0x55ef601884e0 2024-05-10T11:16:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.172+0000 7f17e134eb80 1 -- 172.21.15.27:0/2568670872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef601884e0 msgr2=0x55ef601822d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.172+0000 7f17d4a00700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f17b40b2a70 0x7f17b40b4f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f17c800b780 tx=0x7f17c8006210 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.172+0000 7f17e134eb80 1 --2- 172.21.15.27:0/2568670872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef601884e0 0x55ef601822d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f17c0009d00 tx=0x7f17c0011720 comp rx=0 tx=0).stop 2024-05-10T11:16:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.172+0000 7f17e134eb80 1 -- 172.21.15.27:0/2568670872 shutdown_connections 2024-05-10T11:16:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.172+0000 7f17e134eb80 1 --2- 172.21.15.27:0/2568670872 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef60182d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.172+0000 7f17e134eb80 1 --2- 172.21.15.27:0/2568670872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef60187bc0 0x55ef60187fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.172+0000 7f17e134eb80 1 --2- 172.21.15.27:0/2568670872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef601884e0 0x55ef601822d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.172+0000 7f17e134eb80 1 -- 172.21.15.27:0/2568670872 >> 172.21.15.27:0/2568670872 conn(0x55ef600fa6d0 msgr2=0x55ef6017cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.172+0000 7f17e134eb80 1 -- 172.21.15.27:0/2568670872 shutdown_connections 2024-05-10T11:16:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.172+0000 7f17e134eb80 1 -- 172.21.15.27:0/2568670872 wait complete. 2024-05-10T11:16:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.173+0000 7f17e134eb80 1 -- start start 2024-05-10T11:16:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.173+0000 7f17e134eb80 1 Processor -- start 2024-05-10T11:16:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17e134eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef6018e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17e134eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17e134eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601884e0 0x55ef6018ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17e134eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ef6018ef20 con 0x55ef601884e0 2024-05-10T11:16:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17e134eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ef6003ab20 con 0x55ef601828f0 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17e134eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ef6018f0a0 con 0x55ef60187bc0 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17e134eb80 10 client.0 _ll_register_callbacks cb 0x55ef6007bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17cfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef6018e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17cfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef6018e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35442/0 (socket says 172.21.15.27:35442) 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17cfe00700 1 -- 172.21.15.27:0/3256609820 learned_addr learned my addr 172.21.15.27:0/3256609820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17cf400700 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17d4a00700 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601884e0 0x55ef6018ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17cfe00700 1 -- 172.21.15.27:0/3256609820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 msgr2=0x55ef6018b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17cfe00700 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17cfe00700 1 -- 172.21.15.27:0/3256609820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601884e0 msgr2=0x55ef6018ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.174+0000 7f17cfe00700 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601884e0 0x55ef6018ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.175+0000 7f17cfe00700 1 -- 172.21.15.27:0/3256609820 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef6009a420 con 0x55ef601828f0 2024-05-10T11:16:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.175+0000 7f17cf400700 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:16:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.175+0000 7f17d4a00700 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601884e0 0x55ef6018ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:16:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.175+0000 7f17cfe00700 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef6018e140 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f17c0009f50 tx=0x7f17c0005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79050ef812210011 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.175+0000 7f17ba000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:16:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.175+0000 7f17ba000700 1 -- 172.21.15.27:0/3256609820 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0026070 con 0x55ef601828f0 2024-05-10T11:16:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.175+0000 7f17e134eb80 1 -- 172.21.15.27:0/3256609820 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17c0019ae0 con 0x55ef601828f0 2024-05-10T11:16:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.175+0000 7f17ba000700 1 -- 172.21.15.27:0/3256609820 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f17c0019630 con 0x55ef601828f0 2024-05-10T11:16:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.176+0000 7f17ba000700 1 -- 172.21.15.27:0/3256609820 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f17c0032050 con 0x55ef601828f0 2024-05-10T11:16:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.176+0000 7f17e134eb80 10 client.25907 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.176+0000 7f17e134eb80 1 -- 172.21.15.27:0/3256609820 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ef60199750 con 0x55ef601828f0 2024-05-10T11:16:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.176+0000 7f17ba000700 1 -- 172.21.15.27:0/3256609820 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c0019ae0 con 0x55ef601828f0 2024-05-10T11:16:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.176+0000 7f17ba000700 1 -- 172.21.15.27:0/3256609820 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f17c0021050 con 0x55ef601828f0 2024-05-10T11:16:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.176+0000 7f17e134eb80 10 client.25907 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.176+0000 7f17e134eb80 10 client.25907 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:16:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.176+0000 7f17e134eb80 20 client.25907 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.176+0000 7f17e134eb80 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ef6019bc80 0x55ef6019e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.176+0000 7f17e134eb80 4 client.25907 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:16:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.176+0000 7f17e134eb80 1 -- 172.21.15.27:0/3256609820 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef6014f140 con 0x55ef6019bc80 2024-05-10T11:16:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.177+0000 7f17cf400700 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ef6019bc80 0x55ef6019e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.177+0000 7f17cf400700 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ef6019bc80 0x55ef6019e070 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=49d9cf5ace129202 server_cookie=f8aba1dbb5742ece in_seq=0 out_seq=0 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.177+0000 7f17ba000700 10 client.25907 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.178+0000 7f17ba000700 1 -- 172.21.15.27:0/3256609820 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ef6014f140 con 0x55ef6019bc80 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.178+0000 7f17ba000700 10 client.25907 handle_command_reply: tid=0 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.178+0000 7f17e134eb80 1 client.25907 shutdown 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.178+0000 7f17e134eb80 1 -- 172.21.15.27:0/3256609820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 msgr2=0x55ef6018e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.178+0000 7f17e134eb80 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef6018e140 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f17c0009f50 tx=0x7f17c0005dc0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.178+0000 7f17e134eb80 1 -- 172.21.15.27:0/3256609820 shutdown_connections 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.178+0000 7f17e134eb80 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601884e0 0x55ef6018ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.178+0000 7f17e134eb80 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef6018e140 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.178+0000 7f17e134eb80 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ef6019bc80 0x55ef6019e070 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.178+0000 7f17e134eb80 1 --2- 172.21.15.27:0/3256609820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.178+0000 7f17e134eb80 1 -- 172.21.15.27:0/3256609820 >> 172.21.15.27:0/3256609820 conn(0x55ef600fa6d0 msgr2=0x55ef6017c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:50.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.178+0000 7f17ba000700 0 client.25907 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:16:50.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.179+0000 7f17e134eb80 1 -- 172.21.15.27:0/3256609820 shutdown_connections 2024-05-10T11:16:50.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.179+0000 7f17e134eb80 1 -- 172.21.15.27:0/3256609820 wait complete. 2024-05-10T11:16:50.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.179+0000 7f17e134eb80 20 client.25907 trim_cache size 0 max 16384 2024-05-10T11:16:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.197+0000 7f17e134eb80 1 Processor -- start 2024-05-10T11:16:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.197+0000 7f17e134eb80 1 -- start start 2024-05-10T11:16:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.198+0000 7f17e134eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601828f0 0x55ef60197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.198+0000 7f17e134eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.198+0000 7f17e134eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601884e0 0x55ef6017c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.198+0000 7f17e134eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f17d01b5cd0 con 0x55ef601828f0 2024-05-10T11:16:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.198+0000 7f17e134eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f17d01b7870 con 0x55ef601884e0 2024-05-10T11:16:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.198+0000 7f17e134eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f17d01b63a0 con 0x55ef60187bc0 2024-05-10T11:16:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.198+0000 7f17d4a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601884e0 0x55ef6017c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.198+0000 7f17d4a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601884e0 0x55ef6017c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35452/0 (socket says 172.21.15.27:35452) 2024-05-10T11:16:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.198+0000 7f17d4a00700 1 -- 172.21.15.27:0/1703810359 learned_addr learned my addr 172.21.15.27:0/1703810359 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.198+0000 7f17cf400700 1 --2- 172.21.15.27:0/1703810359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.198+0000 7f17cfe00700 1 --2- 172.21.15.27:0/1703810359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601828f0 0x55ef60197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.199+0000 7f17cf400700 1 -- 172.21.15.27:0/1703810359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601884e0 msgr2=0x55ef6017c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.199+0000 7f17cf400700 1 --2- 172.21.15.27:0/1703810359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601884e0 0x55ef6017c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.199+0000 7f17cf400700 1 -- 172.21.15.27:0/1703810359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601828f0 msgr2=0x55ef60197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.199+0000 7f17cf400700 1 --2- 172.21.15.27:0/1703810359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601828f0 0x55ef60197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.199+0000 7f17cf400700 1 -- 172.21.15.27:0/1703810359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef6019e540 con 0x55ef60187bc0 2024-05-10T11:16:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.199+0000 7f17d4a00700 1 --2- 172.21.15.27:0/1703810359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601884e0 0x55ef6017c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:16:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.199+0000 7f17cf400700 1 --2- 172.21.15.27:0/1703810359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018a660 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0e0060 tx=0x7f17bc0e67f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7155f0adbd2e32b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.199+0000 7f17c4c00700 1 -- 172.21.15.27:0/1703810359 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc0a99c0 con 0x55ef60187bc0 2024-05-10T11:16:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.199+0000 7f17c4c00700 1 -- 172.21.15.27:0/1703810359 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17bc0b2390 con 0x55ef60187bc0 2024-05-10T11:16:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.199+0000 7f17c4c00700 1 -- 172.21.15.27:0/1703810359 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc0a5380 con 0x55ef60187bc0 2024-05-10T11:16:50.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.200+0000 7f17e134eb80 1 -- 172.21.15.27:0/1703810359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 msgr2=0x55ef6018a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:50.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.200+0000 7f17e134eb80 1 --2- 172.21.15.27:0/1703810359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018a660 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0e0060 tx=0x7f17bc0e67f0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.200+0000 7f17e134eb80 1 -- 172.21.15.27:0/1703810359 shutdown_connections 2024-05-10T11:16:50.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.200+0000 7f17e134eb80 1 --2- 172.21.15.27:0/1703810359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601884e0 0x55ef6017c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.200+0000 7f17e134eb80 1 --2- 172.21.15.27:0/1703810359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601828f0 0x55ef60197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.200+0000 7f17e134eb80 1 --2- 172.21.15.27:0/1703810359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018a660 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.200+0000 7f17e134eb80 1 -- 172.21.15.27:0/1703810359 >> 172.21.15.27:0/1703810359 conn(0x55ef600fa6d0 msgr2=0x55ef60188f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:50.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.200+0000 7f17e134eb80 1 -- 172.21.15.27:0/1703810359 shutdown_connections 2024-05-10T11:16:50.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.200+0000 7f17e134eb80 1 -- 172.21.15.27:0/1703810359 wait complete. 2024-05-10T11:16:50.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.201+0000 7f17e134eb80 1 -- start start 2024-05-10T11:16:50.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.201+0000 7f17e134eb80 1 Processor -- start 2024-05-10T11:16:50.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.202+0000 7f17e134eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef60194560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.202+0000 7f17e134eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.202+0000 7f17e134eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601884e0 0x55ef6018fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.202+0000 7f17e134eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f17d01b7140 con 0x55ef601884e0 2024-05-10T11:16:50.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.202+0000 7f17e134eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f17d01b6a70 con 0x55ef601828f0 2024-05-10T11:16:50.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.202+0000 7f17e134eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f17d01b3c70 con 0x55ef60187bc0 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.202+0000 7f17e134eb80 10 client.0 _ll_register_callbacks cb 0x55ef6007bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.202+0000 7f17cfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef60194560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.202+0000 7f17cfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef60194560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35458/0 (socket says 172.21.15.27:35458) 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.202+0000 7f17cfe00700 1 -- 172.21.15.27:0/4160428306 learned_addr learned my addr 172.21.15.27:0/4160428306 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.202+0000 7f17cf400700 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.202+0000 7f17d4a00700 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601884e0 0x55ef6018fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.203+0000 7f17cfe00700 1 -- 172.21.15.27:0/4160428306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 msgr2=0x55ef6018f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.203+0000 7f17cfe00700 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.203+0000 7f17cfe00700 1 -- 172.21.15.27:0/4160428306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601884e0 msgr2=0x55ef6018fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.203+0000 7f17cfe00700 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601884e0 0x55ef6018fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.203+0000 7f17cfe00700 1 -- 172.21.15.27:0/4160428306 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef60190400 con 0x55ef601828f0 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.203+0000 7f17cf400700 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.203+0000 7f17cfe00700 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef60194560 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f17c0003880 tx=0x7f17c0003a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8cab76dc6796c9a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.203+0000 7f17ba000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:16:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.203+0000 7f17ba000700 1 -- 172.21.15.27:0/4160428306 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0026030 con 0x55ef601828f0 2024-05-10T11:16:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.203+0000 7f17e134eb80 1 -- 172.21.15.27:0/4160428306 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ef600e2400 con 0x55ef601828f0 2024-05-10T11:16:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.203+0000 7f17ba000700 1 -- 172.21.15.27:0/4160428306 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f17c0005040 con 0x55ef601828f0 2024-05-10T11:16:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.204+0000 7f17ba000700 1 -- 172.21.15.27:0/4160428306 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f17c003f050 con 0x55ef601828f0 2024-05-10T11:16:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.204+0000 7f17e134eb80 10 client.25913 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.204+0000 7f17e134eb80 1 -- 172.21.15.27:0/4160428306 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ef60206030 con 0x55ef601828f0 2024-05-10T11:16:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.204+0000 7f17ba000700 1 -- 172.21.15.27:0/4160428306 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ef600e2400 con 0x55ef601828f0 2024-05-10T11:16:50.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.204+0000 7f17ba000700 1 -- 172.21.15.27:0/4160428306 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f17c0021040 con 0x55ef601828f0 2024-05-10T11:16:50.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.204+0000 7f17e134eb80 10 client.25913 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.204+0000 7f17e134eb80 10 client.25913 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.204+0000 7f17e134eb80 20 client.25913 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.204+0000 7f17e134eb80 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ef60206620 0x55ef60208a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.204+0000 7f17e134eb80 4 client.25913 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.204+0000 7f17e134eb80 1 -- 172.21.15.27:0/4160428306 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef600dc4a0 con 0x55ef60206620 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.205+0000 7f17cf400700 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ef60206620 0x55ef60208a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.205+0000 7f17cf400700 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ef60206620 0x55ef60208a10 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=421584701cfee2c2 server_cookie=1e57c8bcb1191e69 in_seq=0 out_seq=0 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.205+0000 7f17ba000700 10 client.25913 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.205+0000 7f17ba000700 1 -- 172.21.15.27:0/4160428306 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef600dc4a0 con 0x55ef60206620 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.205+0000 7f17ba000700 10 client.25913 handle_command_reply: tid=0 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.205+0000 7f17e134eb80 1 client.25913 shutdown 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.206+0000 7f17e134eb80 1 -- 172.21.15.27:0/4160428306 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 msgr2=0x55ef60194560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.206+0000 7f17e134eb80 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef60194560 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f17c0003880 tx=0x7f17c0003a80 comp rx=0 tx=0).stop 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.206+0000 7f17e134eb80 1 -- 172.21.15.27:0/4160428306 shutdown_connections 2024-05-10T11:16:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.206+0000 7f17e134eb80 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef601884e0 0x55ef6018fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.206+0000 7f17e134eb80 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ef601828f0 0x55ef60194560 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.206+0000 7f17e134eb80 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ef60206620 0x55ef60208a10 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.206+0000 7f17e134eb80 1 --2- 172.21.15.27:0/4160428306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef60187bc0 0x55ef6018f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.206+0000 7f17e134eb80 1 -- 172.21.15.27:0/4160428306 >> 172.21.15.27:0/4160428306 conn(0x55ef600fa6d0 msgr2=0x55ef60186f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.206+0000 7f17ba000700 0 client.25913 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:16:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.206+0000 7f17e134eb80 1 -- 172.21.15.27:0/4160428306 shutdown_connections 2024-05-10T11:16:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.206+0000 7f17e134eb80 1 -- 172.21.15.27:0/4160428306 wait complete. 2024-05-10T11:16:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.206+0000 7f17e134eb80 20 client.25913 trim_cache size 0 max 16384 2024-05-10T11:16:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:16:50.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.207+0000 7f17d6a00700 1 -- 172.21.15.27:0/1585064246 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f17b40b2a70 msgr2=0x7f17b40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:50.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.207+0000 7f17d6a00700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f17b40b2a70 0x7f17b40b4f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f17c800b780 tx=0x7f17c8006210 comp rx=0 tx=0).stop 2024-05-10T11:16:50.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.207+0000 7f17d6a00700 1 -- 172.21.15.27:0/1585064246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d010a490 msgr2=0x7f17d01a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:50.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.207+0000 7f17d6a00700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d010a490 0x7f17d01a0690 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00eb30 tx=0x7f17bc00ee40 comp rx=0 tx=0).stop 2024-05-10T11:16:50.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.207+0000 7f17d6a00700 1 -- 172.21.15.27:0/1585064246 shutdown_connections 2024-05-10T11:16:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.207+0000 7f17d6a00700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f17b40b2a70 0x7f17b40b4f30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.207+0000 7f17d6a00700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17d010fb60 0x7f17d01a4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.207+0000 7f17d6a00700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d0109ad0 0x7f17d01a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.207+0000 7f17d6a00700 1 --2- 172.21.15.27:0/1585064246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d010a490 0x7f17d01a0690 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.207+0000 7f17d6a00700 1 -- 172.21.15.27:0/1585064246 >> 172.21.15.27:0/1585064246 conn(0x7f17d0076c70 msgr2=0x7f17d0077290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.208+0000 7f17d6a00700 1 -- 172.21.15.27:0/1585064246 shutdown_connections 2024-05-10T11:16:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:50.208+0000 7f17d6a00700 1 -- 172.21.15.27:0/1585064246 wait complete. 2024-05-10T11:16:50.661 INFO:tasks.workunit.client.0.smithi027.stdout: 1 131565 7.66 MB/sec warmup 95 sec latency 14.462 ms 2024-05-10T11:16:51.180 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:16:51.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:51 smithi027 ceph-mon[30054]: pgmap v112: 129 pgs: 129 active+clean; 80 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-05-10T11:16:51.661 INFO:tasks.workunit.client.0.smithi027.stdout: 1 132495 7.61 MB/sec warmup 96 sec latency 45.070 ms 2024-05-10T11:16:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:51 smithi076 ceph-mon[34152]: pgmap v112: 129 pgs: 129 active+clean; 80 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-05-10T11:16:51.686 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:16:51.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:51 smithi189 ceph-mon[34842]: pgmap v112: 129 pgs: 129 active+clean; 80 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-05-10T11:16:52.661 INFO:tasks.workunit.client.0.smithi027.stdout: 1 134935 7.69 MB/sec warmup 97 sec latency 9.593 ms 2024-05-10T11:16:53.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:53 smithi027 ceph-mon[30054]: pgmap v113: 129 pgs: 129 active+clean; 80 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-05-10T11:16:53.661 INFO:tasks.workunit.client.0.smithi027.stdout: 1 135487 7.63 MB/sec warmup 98 sec latency 80.652 ms 2024-05-10T11:16:53.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:53 smithi076 ceph-mon[34152]: pgmap v113: 129 pgs: 129 active+clean; 80 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-05-10T11:16:53.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:53 smithi189 ceph-mon[34842]: pgmap v113: 129 pgs: 129 active+clean; 80 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-05-10T11:16:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.487+0000 7efd42000700 1 -- 172.21.15.27:0/4285322953 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd3c106080 msgr2=0x7efd3c112590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.487+0000 7efd42000700 1 --2- 172.21.15.27:0/4285322953 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd3c106080 0x7efd3c112590 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efd3400b0a0 tx=0x7efd3400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.487+0000 7efd42000700 1 -- 172.21.15.27:0/4285322953 shutdown_connections 2024-05-10T11:16:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.487+0000 7efd42000700 1 --2- 172.21.15.27:0/4285322953 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd3c106080 0x7efd3c112590 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.487+0000 7efd42000700 1 --2- 172.21.15.27:0/4285322953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3c10b6c0 0x7efd3c10baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.487+0000 7efd42000700 1 --2- 172.21.15.27:0/4285322953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3c1056c0 0x7efd3c105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.487+0000 7efd42000700 1 -- 172.21.15.27:0/4285322953 >> 172.21.15.27:0/4285322953 conn(0x7efd3c1011b0 msgr2=0x7efd3c1035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.487+0000 7efd42000700 1 -- 172.21.15.27:0/4285322953 shutdown_connections 2024-05-10T11:16:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.487+0000 7efd42000700 1 -- 172.21.15.27:0/4285322953 wait complete. 2024-05-10T11:16:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.489+0000 7efd42000700 1 Processor -- start 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.489+0000 7efd42000700 1 -- start start 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.489+0000 7efd42000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3c1056c0 0x7efd3c19bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.489+0000 7efd42000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3c106080 0x7efd3c19c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.489+0000 7efd42000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd3c10b6c0 0x7efd3c1a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.489+0000 7efd42000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c115900 con 0x7efd3c106080 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.489+0000 7efd42000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c115780 con 0x7efd3c10b6c0 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.489+0000 7efd42000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c115a80 con 0x7efd3c1056c0 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.490+0000 7efd3b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3c1056c0 0x7efd3c19bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.490+0000 7efd3b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3c1056c0 0x7efd3c19bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51556/0 (socket says 172.21.15.27:51556) 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.490+0000 7efd3b400700 1 -- 172.21.15.27:0/2742887239 learned_addr learned my addr 172.21.15.27:0/2742887239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.490+0000 7efd33e00700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3c106080 0x7efd3c19c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.490+0000 7efd3be00700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd3c10b6c0 0x7efd3c1a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.490+0000 7efd3b400700 1 -- 172.21.15.27:0/2742887239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd3c10b6c0 msgr2=0x7efd3c1a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.490+0000 7efd3b400700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd3c10b6c0 0x7efd3c1a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.490+0000 7efd3b400700 1 -- 172.21.15.27:0/2742887239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3c106080 msgr2=0x7efd3c19c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.490+0000 7efd3b400700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3c106080 0x7efd3c19c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.490+0000 7efd3b400700 1 -- 172.21.15.27:0/2742887239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3c1a0d20 con 0x7efd3c1056c0 2024-05-10T11:16:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.490+0000 7efd33e00700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3c106080 0x7efd3c19c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:16:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.490+0000 7efd3b400700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3c1056c0 0x7efd3c19bd10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00e9a0 tx=0x7efd2c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.491+0000 7efd38c00700 1 -- 172.21.15.27:0/2742887239 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2c00cbd0 con 0x7efd3c1056c0 2024-05-10T11:16:54.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.491+0000 7efd42000700 1 -- 172.21.15.27:0/2742887239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd3c1a1010 con 0x7efd3c1056c0 2024-05-10T11:16:54.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.491+0000 7efd38c00700 1 -- 172.21.15.27:0/2742887239 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd2c00cd70 con 0x7efd3c1056c0 2024-05-10T11:16:54.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.491+0000 7efd38c00700 1 -- 172.21.15.27:0/2742887239 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2c018680 con 0x7efd3c1056c0 2024-05-10T11:16:54.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.491+0000 7efd42000700 1 -- 172.21.15.27:0/2742887239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd3c1a8890 con 0x7efd3c1056c0 2024-05-10T11:16:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.496+0000 7efd38c00700 1 -- 172.21.15.27:0/2742887239 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7efd2c010430 con 0x7efd3c1056c0 2024-05-10T11:16:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.497+0000 7efd4c9f1b80 1 Processor -- start 2024-05-10T11:16:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.497+0000 7efd4c9f1b80 1 -- start start 2024-05-10T11:16:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.498+0000 7efd4c9f1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d41976bc0 0x564d41976fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.498+0000 7efd4c9f1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419774e0 0x564d419712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.498+0000 7efd4c9f1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d419718f0 0x564d41971d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.498+0000 7efd4c9f1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564d418c3a20 con 0x564d41976bc0 2024-05-10T11:16:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.498+0000 7efd4c9f1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564d41861760 con 0x564d419718f0 2024-05-10T11:16:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.498+0000 7efd4c9f1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564d418bf9e0 con 0x564d419774e0 2024-05-10T11:16:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.498+0000 7efd33e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d419718f0 0x564d41971d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.498+0000 7efd3be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d41976bc0 0x564d41976fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.498+0000 7efd33e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d419718f0 0x564d41971d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60478/0 (socket says 172.21.15.27:60478) 2024-05-10T11:16:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.498+0000 7efd33e00700 1 -- 172.21.15.27:0/2610963300 learned_addr learned my addr 172.21.15.27:0/2610963300 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.498+0000 7efd3b400700 1 --2- 172.21.15.27:0/2610963300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419774e0 0x564d419712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.498+0000 7efd38c00700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd200b2a70 0x7efd200b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.499+0000 7efd33e00700 1 -- 172.21.15.27:0/2610963300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419774e0 msgr2=0x564d419712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.499+0000 7efd33e00700 1 --2- 172.21.15.27:0/2610963300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419774e0 0x564d419712d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.499+0000 7efd33e00700 1 -- 172.21.15.27:0/2610963300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d41976bc0 msgr2=0x564d41976fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.499+0000 7efd33e00700 1 --2- 172.21.15.27:0/2610963300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d41976bc0 0x564d41976fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.499+0000 7efd33e00700 1 -- 172.21.15.27:0/2610963300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d418108d0 con 0x564d419718f0 2024-05-10T11:16:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.499+0000 7efd38c00700 1 -- 172.21.15.27:0/2742887239 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7efd2c0e2b80 con 0x7efd3c1056c0 2024-05-10T11:16:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.499+0000 7efd3be00700 1 --2- 172.21.15.27:0/2610963300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d41976bc0 0x564d41976fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:16:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.499+0000 7efd3be00700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd200b2a70 0x7efd200b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.499+0000 7efd33e00700 1 --2- 172.21.15.27:0/2610963300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d419718f0 0x564d41971d70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efd24009d80 tx=0x7efd240116c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63661521f855fba server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.499+0000 7efd30c00700 1 -- 172.21.15.27:0/2610963300 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24011e20 con 0x564d419718f0 2024-05-10T11:16:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.500+0000 7efd30c00700 1 -- 172.21.15.27:0/2610963300 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd24012900 con 0x564d419718f0 2024-05-10T11:16:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.500+0000 7efd30c00700 1 -- 172.21.15.27:0/2610963300 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2400f800 con 0x564d419718f0 2024-05-10T11:16:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.500+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2610963300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d419718f0 msgr2=0x564d41971d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.500+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/2610963300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d419718f0 0x564d41971d70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efd24009d80 tx=0x7efd240116c0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.500+0000 7efd3be00700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd200b2a70 0x7efd200b4f30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efd3400b780 tx=0x7efd340099e0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.501+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2610963300 shutdown_connections 2024-05-10T11:16:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.501+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/2610963300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d419718f0 0x564d41971d70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.501+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/2610963300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d41976bc0 0x564d41976fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.501+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/2610963300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419774e0 0x564d419712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.501+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2610963300 >> 172.21.15.27:0/2610963300 conn(0x564d418e96d0 msgr2=0x564d4196bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.501+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2610963300 shutdown_connections 2024-05-10T11:16:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.501+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2610963300 wait complete. 2024-05-10T11:16:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.501+0000 7efd4c9f1b80 1 -- start start 2024-05-10T11:16:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.502+0000 7efd4c9f1b80 1 Processor -- start 2024-05-10T11:16:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.502+0000 7efd4c9f1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419718f0 0x564d4197d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.502+0000 7efd4c9f1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 0x564d4197a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.502+0000 7efd4c9f1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419774e0 0x564d4197aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.502+0000 7efd4c9f1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564d4197df20 con 0x564d419774e0 2024-05-10T11:16:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.502+0000 7efd4c9f1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564d41829b20 con 0x564d41976bc0 2024-05-10T11:16:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.502+0000 7efd4c9f1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564d4197e0a0 con 0x564d419718f0 2024-05-10T11:16:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.502+0000 7efd4c9f1b80 10 client.0 _ll_register_callbacks cb 0x564d4186ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.502+0000 7efd33e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 0x564d4197a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.502+0000 7efd3b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419718f0 0x564d4197d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.503+0000 7efd3be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419774e0 0x564d4197aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.503+0000 7efd33e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 0x564d4197a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60486/0 (socket says 172.21.15.27:60486) 2024-05-10T11:16:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.503+0000 7efd33e00700 1 -- 172.21.15.27:0/2725352669 learned_addr learned my addr 172.21.15.27:0/2725352669 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.503+0000 7efd33e00700 1 -- 172.21.15.27:0/2725352669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419718f0 msgr2=0x564d4197d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.503+0000 7efd33e00700 1 --2- 172.21.15.27:0/2725352669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419718f0 0x564d4197d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.503+0000 7efd33e00700 1 -- 172.21.15.27:0/2725352669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419774e0 msgr2=0x564d4197aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.503+0000 7efd33e00700 1 --2- 172.21.15.27:0/2725352669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419774e0 0x564d4197aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.503+0000 7efd33e00700 1 -- 172.21.15.27:0/2725352669 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d41889420 con 0x564d41976bc0 2024-05-10T11:16:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.503+0000 7efd33e00700 1 --2- 172.21.15.27:0/2725352669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 0x564d4197a590 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efd240058f0 tx=0x7efd24012660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78a783e598ec54d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.503+0000 7efd29600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:16:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.504+0000 7efd29600700 1 -- 172.21.15.27:0/2725352669 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24028020 con 0x564d41976bc0 2024-05-10T11:16:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.504+0000 7efd29600700 1 -- 172.21.15.27:0/2725352669 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7efd24012cc0 con 0x564d41976bc0 2024-05-10T11:16:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.504+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2725352669 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd24012900 con 0x564d41976bc0 2024-05-10T11:16:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.504+0000 7efd29600700 1 -- 172.21.15.27:0/2725352669 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efd24033050 con 0x564d41976bc0 2024-05-10T11:16:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.504+0000 7efd4c9f1b80 10 client.25925 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.504+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2725352669 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564d419887b0 con 0x564d41976bc0 2024-05-10T11:16:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.505+0000 7efd29600700 1 -- 172.21.15.27:0/2725352669 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd24012900 con 0x564d41976bc0 2024-05-10T11:16:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.505+0000 7efd29600700 1 -- 172.21.15.27:0/2725352669 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7efd24003ba0 con 0x564d41976bc0 2024-05-10T11:16:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.505+0000 7efd4c9f1b80 10 client.25925 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.505+0000 7efd4c9f1b80 10 client.25925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:16:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.505+0000 7efd4c9f1b80 20 client.25925 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.505+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/2725352669 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564d4198adf0 0x564d4198d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.505+0000 7efd4c9f1b80 4 client.25925 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:16:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.505+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2725352669 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d4193e140 con 0x564d4198adf0 2024-05-10T11:16:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.505+0000 7efd3b400700 1 --2- 172.21.15.27:0/2725352669 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564d4198adf0 0x564d4198d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.506+0000 7efd3b400700 1 --2- 172.21.15.27:0/2725352669 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564d4198adf0 0x564d4198d1e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c1c694bea1a1b85 server_cookie=90c262e150cc59b7 in_seq=0 out_seq=0 2024-05-10T11:16:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.506+0000 7efd29600700 10 client.25925 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:16:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.507+0000 7efd29600700 1 -- 172.21.15.27:0/2725352669 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564d4193e140 con 0x564d4198adf0 2024-05-10T11:16:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.507+0000 7efd29600700 10 client.25925 handle_command_reply: tid=0 2024-05-10T11:16:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.507+0000 7efd4c9f1b80 1 client.25925 shutdown 2024-05-10T11:16:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.507+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2725352669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 msgr2=0x564d4197a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.507+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/2725352669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 0x564d4197a590 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efd240058f0 tx=0x7efd24012660 comp rx=0 tx=0).stop 2024-05-10T11:16:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.507+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2725352669 shutdown_connections 2024-05-10T11:16:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.507+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/2725352669 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564d4198adf0 0x564d4198d1e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.507+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/2725352669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419718f0 0x564d4197d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.507+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/2725352669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419774e0 0x564d4197aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.507+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/2725352669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 0x564d4197a590 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.507+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2725352669 >> 172.21.15.27:0/2725352669 conn(0x564d418e96d0 msgr2=0x564d4196b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.508+0000 7efd29600700 0 client.25925 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:16:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.508+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2725352669 shutdown_connections 2024-05-10T11:16:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.508+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/2725352669 wait complete. 2024-05-10T11:16:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.508+0000 7efd4c9f1b80 20 client.25925 trim_cache size 0 max 16384 2024-05-10T11:16:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.528+0000 7efd4c9f1b80 1 Processor -- start 2024-05-10T11:16:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.528+0000 7efd4c9f1b80 1 -- start start 2024-05-10T11:16:54.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.528+0000 7efd4c9f1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419718f0 0x564d419712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.528+0000 7efd4c9f1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 0x564d41986fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.528+0000 7efd4c9f1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419774e0 0x564d4196b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd4c9f1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c1b1810 con 0x564d419774e0 2024-05-10T11:16:54.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd4c9f1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c1b33b0 con 0x564d41976bc0 2024-05-10T11:16:54.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd4c9f1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c1b1ee0 con 0x564d419718f0 2024-05-10T11:16:54.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd33e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 0x564d41986fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd3be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419774e0 0x564d4196b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd33e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 0x564d41986fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60500/0 (socket says 172.21.15.27:60500) 2024-05-10T11:16:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd33e00700 1 -- 172.21.15.27:0/93690632 learned_addr learned my addr 172.21.15.27:0/93690632 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd3b400700 1 --2- 172.21.15.27:0/93690632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419718f0 0x564d419712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd33e00700 1 -- 172.21.15.27:0/93690632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419718f0 msgr2=0x564d419712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd33e00700 1 --2- 172.21.15.27:0/93690632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419718f0 0x564d419712d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd33e00700 1 -- 172.21.15.27:0/93690632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419774e0 msgr2=0x564d4196b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd33e00700 1 --2- 172.21.15.27:0/93690632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419774e0 0x564d4196b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.529+0000 7efd33e00700 1 -- 172.21.15.27:0/93690632 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d4197a360 con 0x564d41976bc0 2024-05-10T11:16:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.530+0000 7efd33e00700 1 --2- 172.21.15.27:0/93690632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 0x564d41986fb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efd24011700 tx=0x7efd24024880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb0db8291737458f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.530+0000 7efd2be00700 1 -- 172.21.15.27:0/93690632 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2403c070 con 0x564d41976bc0 2024-05-10T11:16:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.530+0000 7efd2be00700 1 -- 172.21.15.27:0/93690632 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd24011a10 con 0x564d41976bc0 2024-05-10T11:16:54.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.530+0000 7efd2be00700 1 -- 172.21.15.27:0/93690632 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24028040 con 0x564d41976bc0 2024-05-10T11:16:54.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.531+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/93690632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 msgr2=0x564d41986fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:54.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.531+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/93690632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 0x564d41986fb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efd24011700 tx=0x7efd24024880 comp rx=0 tx=0).stop 2024-05-10T11:16:54.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.531+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/93690632 shutdown_connections 2024-05-10T11:16:54.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.531+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/93690632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419718f0 0x564d419712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.531+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/93690632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419774e0 0x564d4196b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.531+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/93690632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41976bc0 0x564d41986fb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.531+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/93690632 >> 172.21.15.27:0/93690632 conn(0x564d418e96d0 msgr2=0x564d41978e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:54.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.531+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/93690632 shutdown_connections 2024-05-10T11:16:54.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.531+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/93690632 wait complete. 2024-05-10T11:16:54.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.532+0000 7efd4c9f1b80 1 -- start start 2024-05-10T11:16:54.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.532+0000 7efd4c9f1b80 1 Processor -- start 2024-05-10T11:16:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd4c9f1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419718f0 0x564d41983300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd4c9f1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419774e0 0x564d41983840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd4c9f1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41991800 0x564d41991c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd4c9f1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c1b2c80 con 0x564d419718f0 2024-05-10T11:16:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd4c9f1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c1b25b0 con 0x564d41991800 2024-05-10T11:16:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd4c9f1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c1af7b0 con 0x564d419774e0 2024-05-10T11:16:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd4c9f1b80 10 client.0 _ll_register_callbacks cb 0x564d4186ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd3b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419718f0 0x564d41983300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd3be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41991800 0x564d41991c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd3be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41991800 0x564d41991c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60512/0 (socket says 172.21.15.27:60512) 2024-05-10T11:16:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd3b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419718f0 0x564d41983300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36440/0 (socket says 172.21.15.27:36440) 2024-05-10T11:16:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd3be00700 1 -- 172.21.15.27:0/3056751787 learned_addr learned my addr 172.21.15.27:0/3056751787 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd33e00700 1 --2- 172.21.15.27:0/3056751787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419774e0 0x564d41983840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd33e00700 1 -- 172.21.15.27:0/3056751787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41991800 msgr2=0x564d41991c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd33e00700 1 --2- 172.21.15.27:0/3056751787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41991800 0x564d41991c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd33e00700 1 -- 172.21.15.27:0/3056751787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419718f0 msgr2=0x564d41983300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd33e00700 1 --2- 172.21.15.27:0/3056751787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419718f0 0x564d41983300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.533+0000 7efd33e00700 1 -- 172.21.15.27:0/3056751787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d419921c0 con 0x564d419774e0 2024-05-10T11:16:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.534+0000 7efd33e00700 1 --2- 172.21.15.27:0/3056751787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419774e0 0x564d41983840 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x564d419768f0 tx=0x7efd24012890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=594a7322e8731d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.534+0000 7efd29600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:16:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.534+0000 7efd29600700 1 -- 172.21.15.27:0/3056751787 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2403c070 con 0x564d419774e0 2024-05-10T11:16:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.534+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/3056751787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd24011a10 con 0x564d419774e0 2024-05-10T11:16:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.534+0000 7efd29600700 1 -- 172.21.15.27:0/3056751787 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7efd240050b0 con 0x564d419774e0 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.535+0000 7efd29600700 1 -- 172.21.15.27:0/3056751787 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efd24028040 con 0x564d419774e0 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.535+0000 7efd4c9f1b80 10 client.25918 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.535+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/3056751787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564d419f4fe0 con 0x564d419774e0 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.535+0000 7efd29600700 1 -- 172.21.15.27:0/3056751787 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd24011a10 con 0x564d419774e0 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.535+0000 7efd29600700 1 -- 172.21.15.27:0/3056751787 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7efd240052d0 con 0x564d419774e0 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.535+0000 7efd4c9f1b80 10 client.25918 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.535+0000 7efd4c9f1b80 10 client.25918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.535+0000 7efd4c9f1b80 20 client.25918 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.535+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/3056751787 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564d419f55f0 0x564d419f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.535+0000 7efd4c9f1b80 4 client.25918 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.535+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/3056751787 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564d418d1400 con 0x564d419f55f0 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.536+0000 7efd3b400700 1 --2- 172.21.15.27:0/3056751787 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564d419f55f0 0x564d419f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.536+0000 7efd3b400700 1 --2- 172.21.15.27:0/3056751787 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564d419f55f0 0x564d419f7a10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b975b8f6d9b43ef server_cookie=8e4c33a92078c75b in_seq=0 out_seq=0 2024-05-10T11:16:54.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.536+0000 7efd29600700 10 client.25918 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.536+0000 7efd29600700 1 -- 172.21.15.27:0/3056751787 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564d418d1400 con 0x564d419f55f0 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.536+0000 7efd29600700 10 client.25918 handle_command_reply: tid=0 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.536+0000 7efd4c9f1b80 1 client.25918 shutdown 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.537+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/3056751787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419774e0 msgr2=0x564d41983840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.537+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/3056751787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419774e0 0x564d41983840 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x564d419768f0 tx=0x7efd24012890 comp rx=0 tx=0).stop 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.537+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/3056751787 shutdown_connections 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.537+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/3056751787 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564d419f55f0 0x564d419f7a10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.537+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/3056751787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d41991800 0x564d41991c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.537+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/3056751787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d419718f0 0x564d41983300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.537+0000 7efd4c9f1b80 1 --2- 172.21.15.27:0/3056751787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d419774e0 0x564d41983840 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.537+0000 7efd29600700 0 client.25918 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.537+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/3056751787 >> 172.21.15.27:0/3056751787 conn(0x564d418e96d0 msgr2=0x564d41978a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:54.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.537+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/3056751787 shutdown_connections 2024-05-10T11:16:54.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.537+0000 7efd4c9f1b80 1 -- 172.21.15.27:0/3056751787 wait complete. 2024-05-10T11:16:54.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.537+0000 7efd4c9f1b80 20 client.25918 trim_cache size 0 max 16384 2024-05-10T11:16:54.539 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:16:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.538+0000 7efd42000700 1 -- 172.21.15.27:0/2742887239 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd200b2a70 msgr2=0x7efd200b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.538+0000 7efd42000700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd200b2a70 0x7efd200b4f30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efd3400b780 tx=0x7efd340099e0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.538+0000 7efd42000700 1 -- 172.21.15.27:0/2742887239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3c1056c0 msgr2=0x7efd3c19bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.538+0000 7efd42000700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3c1056c0 0x7efd3c19bd10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00e9a0 tx=0x7efd2c00ecb0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.538+0000 7efd42000700 1 -- 172.21.15.27:0/2742887239 shutdown_connections 2024-05-10T11:16:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.538+0000 7efd42000700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd200b2a70 0x7efd200b4f30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.538+0000 7efd42000700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3c1056c0 0x7efd3c19bd10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.538+0000 7efd42000700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd3c10b6c0 0x7efd3c1a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.538+0000 7efd42000700 1 --2- 172.21.15.27:0/2742887239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3c106080 0x7efd3c19c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.538+0000 7efd42000700 1 -- 172.21.15.27:0/2742887239 >> 172.21.15.27:0/2742887239 conn(0x7efd3c1011b0 msgr2=0x7efd3c101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.538+0000 7efd42000700 1 -- 172.21.15.27:0/2742887239 shutdown_connections 2024-05-10T11:16:54.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:54.538+0000 7efd42000700 1 -- 172.21.15.27:0/2742887239 wait complete. 2024-05-10T11:16:54.661 INFO:tasks.workunit.client.0.smithi027.stdout: 1 137810 7.68 MB/sec warmup 99 sec latency 18.804 ms 2024-05-10T11:16:55.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:55 smithi027 ceph-mon[30054]: pgmap v114: 129 pgs: 129 active+clean; 84 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 193 op/s 2024-05-10T11:16:55.661 INFO:tasks.workunit.client.0.smithi027.stdout: 1 138763 7.67 MB/sec warmup 100 sec latency 42.111 ms 2024-05-10T11:16:55.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:55 smithi076 ceph-mon[34152]: pgmap v114: 129 pgs: 129 active+clean; 84 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 193 op/s 2024-05-10T11:16:55.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:55 smithi189 ceph-mon[34842]: pgmap v114: 129 pgs: 129 active+clean; 84 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 193 op/s 2024-05-10T11:16:56.588 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:16:56.588 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:16:56.661 INFO:tasks.workunit.client.0.smithi027.stdout: 1 140131 7.64 MB/sec warmup 101 sec latency 31.246 ms 2024-05-10T11:16:57.080 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:16:57.585 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:57 smithi027 ceph-mon[30054]: pgmap v115: 129 pgs: 129 active+clean; 84 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 160 op/s 2024-05-10T11:16:57.661 INFO:tasks.workunit.client.0.smithi027.stdout: 1 142251 7.70 MB/sec warmup 102 sec latency 18.305 ms 2024-05-10T11:16:57.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:57 smithi076 ceph-mon[34152]: pgmap v115: 129 pgs: 129 active+clean; 84 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 160 op/s 2024-05-10T11:16:57.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:57 smithi189 ceph-mon[34842]: pgmap v115: 129 pgs: 129 active+clean; 84 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 160 op/s 2024-05-10T11:16:58.661 INFO:tasks.workunit.client.0.smithi027.stdout: 1 143314 7.68 MB/sec warmup 103 sec latency 63.690 ms 2024-05-10T11:16:59.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:16:59 smithi027 ceph-mon[30054]: pgmap v116: 129 pgs: 129 active+clean; 88 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 245 op/s 2024-05-10T11:16:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.523+0000 7f2765400700 1 -- 172.21.15.27:0/869060530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27600761f0 msgr2=0x7f276010f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.523+0000 7f2765400700 1 --2- 172.21.15.27:0/869060530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27600761f0 0x7f276010f6d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f27480098f0 tx=0x7f2748009c00 comp rx=0 tx=0).stop 2024-05-10T11:16:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.523+0000 7f2765400700 1 -- 172.21.15.27:0/869060530 shutdown_connections 2024-05-10T11:16:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.523+0000 7f2765400700 1 --2- 172.21.15.27:0/869060530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f276010fc10 0x7f2760112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.523+0000 7f2765400700 1 --2- 172.21.15.27:0/869060530 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27600758d0 0x7f2760075cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.523+0000 7f2765400700 1 --2- 172.21.15.27:0/869060530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27600761f0 0x7f276010f6d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.523+0000 7f2765400700 1 -- 172.21.15.27:0/869060530 >> 172.21.15.27:0/869060530 conn(0x7f27600fef20 msgr2=0x7f2760101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.523+0000 7f2765400700 1 -- 172.21.15.27:0/869060530 shutdown_connections 2024-05-10T11:16:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.523+0000 7f2765400700 1 -- 172.21.15.27:0/869060530 wait complete. 2024-05-10T11:16:59.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.524+0000 7f2765400700 1 Processor -- start 2024-05-10T11:16:59.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.525+0000 7f2765400700 1 -- start start 2024-05-10T11:16:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.525+0000 7f2765400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27600758d0 0x7f276019bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.525+0000 7f2765400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27600761f0 0x7f276019c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.525+0000 7f2765400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f276010fc10 0x7f27601a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.525+0000 7f2765400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f27601157b0 con 0x7f276010fc10 2024-05-10T11:16:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.525+0000 7f2765400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2760115630 con 0x7f27600761f0 2024-05-10T11:16:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.525+0000 7f2765400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2760115930 con 0x7f27600758d0 2024-05-10T11:16:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.525+0000 7f275fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27600758d0 0x7f276019bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.525+0000 7f2757400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27600761f0 0x7f276019c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.525+0000 7f275fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27600758d0 0x7f276019bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51616/0 (socket says 172.21.15.27:51616) 2024-05-10T11:16:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.525+0000 7f275fe00700 1 -- 172.21.15.27:0/4245979857 learned_addr learned my addr 172.21.15.27:0/4245979857 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.526+0000 7f2764a00700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f276010fc10 0x7f27601a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.526+0000 7f2757400700 1 -- 172.21.15.27:0/4245979857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27600758d0 msgr2=0x7f276019bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.526+0000 7f2757400700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27600758d0 0x7f276019bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.526+0000 7f2757400700 1 -- 172.21.15.27:0/4245979857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f276010fc10 msgr2=0x7f27601a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.526+0000 7f2757400700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f276010fc10 0x7f27601a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.526+0000 7f2757400700 1 -- 172.21.15.27:0/4245979857 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27601a0d30 con 0x7f27600761f0 2024-05-10T11:16:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.526+0000 7f2757400700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27600761f0 0x7f276019c2b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2748009230 tx=0x7f274800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.527+0000 7f275d600700 1 -- 172.21.15.27:0/4245979857 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2748005140 con 0x7f27600761f0 2024-05-10T11:16:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.527+0000 7f275d600700 1 -- 172.21.15.27:0/4245979857 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f274800b560 con 0x7f27600761f0 2024-05-10T11:16:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.527+0000 7f275d600700 1 -- 172.21.15.27:0/4245979857 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2748003990 con 0x7f27600761f0 2024-05-10T11:16:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.527+0000 7f2765400700 1 -- 172.21.15.27:0/4245979857 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27601a0fc0 con 0x7f27600761f0 2024-05-10T11:16:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.527+0000 7f2765400700 1 -- 172.21.15.27:0/4245979857 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27601a88a0 con 0x7f27600761f0 2024-05-10T11:16:59.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.531+0000 7f275d600700 1 -- 172.21.15.27:0/4245979857 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f274800b9a0 con 0x7f27600761f0 2024-05-10T11:16:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.533+0000 7f275d600700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f27440b2a70 0x7f27440b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.533+0000 7f277140db80 1 Processor -- start 2024-05-10T11:16:59.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.533+0000 7f275d600700 1 -- 172.21.15.27:0/4245979857 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f27480e3930 con 0x7f27600761f0 2024-05-10T11:16:59.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.533+0000 7f277140db80 1 -- start start 2024-05-10T11:16:59.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.533+0000 7f275fe00700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f27440b2a70 0x7f27440b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f277140db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 0x55f22067d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f277140db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 0x55f220681200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f277140db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f220681740 0x55f220681ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f277140db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f2205c6aa0 con 0x55f220681740 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f277140db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f2205647e0 con 0x55f22067cea0 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f277140db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f2205c2a60 con 0x55f22067d7c0 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f2757400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 0x55f220681200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f2757400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 0x55f220681200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51624/0 (socket says 172.21.15.27:51624) 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f2757400700 1 -- 172.21.15.27:0/2670649798 learned_addr learned my addr 172.21.15.27:0/2670649798 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f2764a00700 1 --2- 172.21.15.27:0/2670649798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f220681740 0x55f220681ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f275fe00700 1 --2- 172.21.15.27:0/2670649798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 0x55f22067d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f2757400700 1 -- 172.21.15.27:0/2670649798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 msgr2=0x55f22067d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f2757400700 1 --2- 172.21.15.27:0/2670649798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 0x55f22067d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f2757400700 1 -- 172.21.15.27:0/2670649798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f220681740 msgr2=0x55f220681ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f2757400700 1 --2- 172.21.15.27:0/2670649798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f220681740 0x55f220681ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f2757400700 1 -- 172.21.15.27:0/2670649798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f220513950 con 0x55f22067d7c0 2024-05-10T11:16:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f2764a00700 1 --2- 172.21.15.27:0/2670649798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f220681740 0x55f220681ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:16:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f275fe00700 1 --2- 172.21.15.27:0/2670649798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 0x55f22067d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:16:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.534+0000 7f275fe00700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f27440b2a70 0x7f27440b4f30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2750009d80 tx=0x7f2750009310 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.535+0000 7f2757400700 1 --2- 172.21.15.27:0/2670649798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 0x55f220681200 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f27480aa800 tx=0x7f2748009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cf5365b59b2e84e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.535+0000 7f2755600700 1 -- 172.21.15.27:0/2670649798 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27480aaf70 con 0x55f22067d7c0 2024-05-10T11:16:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.535+0000 7f2755600700 1 -- 172.21.15.27:0/2670649798 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27480aa870 con 0x55f22067d7c0 2024-05-10T11:16:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.535+0000 7f2755600700 1 -- 172.21.15.27:0/2670649798 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27480b1140 con 0x55f22067d7c0 2024-05-10T11:16:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.536+0000 7f277140db80 1 -- 172.21.15.27:0/2670649798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 msgr2=0x55f220681200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.536+0000 7f277140db80 1 --2- 172.21.15.27:0/2670649798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 0x55f220681200 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f27480aa800 tx=0x7f2748009c00 comp rx=0 tx=0).stop 2024-05-10T11:16:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.536+0000 7f277140db80 1 -- 172.21.15.27:0/2670649798 shutdown_connections 2024-05-10T11:16:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.536+0000 7f277140db80 1 --2- 172.21.15.27:0/2670649798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f220681740 0x55f220681ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.536+0000 7f277140db80 1 --2- 172.21.15.27:0/2670649798 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 0x55f22067d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.536+0000 7f277140db80 1 --2- 172.21.15.27:0/2670649798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 0x55f220681200 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.536+0000 7f277140db80 1 -- 172.21.15.27:0/2670649798 >> 172.21.15.27:0/2670649798 conn(0x55f2205ec750 msgr2=0x55f22066ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.536+0000 7f277140db80 1 -- 172.21.15.27:0/2670649798 shutdown_connections 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.536+0000 7f277140db80 1 -- 172.21.15.27:0/2670649798 wait complete. 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.537+0000 7f277140db80 1 -- start start 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.537+0000 7f277140db80 1 Processor -- start 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.537+0000 7f277140db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 0x55f220688ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.537+0000 7f277140db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067d7c0 0x55f220685fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f277140db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f220681740 0x55f2206864f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f277140db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f220689780 con 0x55f22067d7c0 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f277140db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f22052cba0 con 0x55f22067cea0 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f277140db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f220689900 con 0x55f220681740 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f277140db80 10 client.0 _ll_register_callbacks cb 0x55f22056dcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f275fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 0x55f220688ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f2764a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f220681740 0x55f2206864f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f275fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 0x55f220688ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60534/0 (socket says 172.21.15.27:60534) 2024-05-10T11:16:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f275fe00700 1 -- 172.21.15.27:0/1110434701 learned_addr learned my addr 172.21.15.27:0/1110434701 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f2757400700 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067d7c0 0x55f220685fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f275fe00700 1 -- 172.21.15.27:0/1110434701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f220681740 msgr2=0x55f2206864f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f275fe00700 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f220681740 0x55f2206864f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f275fe00700 1 -- 172.21.15.27:0/1110434701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067d7c0 msgr2=0x55f220685fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f275fe00700 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067d7c0 0x55f220685fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f275fe00700 1 -- 172.21.15.27:0/1110434701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f22058c4a0 con 0x55f22067cea0 2024-05-10T11:16:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.538+0000 7f2764a00700 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f220681740 0x55f2206864f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:16:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.539+0000 7f275fe00700 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 0x55f220688ba0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f275001c980 tx=0x7f27500047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f5bfddea381e7e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.539+0000 7f274e000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:16:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.539+0000 7f274e000700 1 -- 172.21.15.27:0/1110434701 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275002a060 con 0x55f22067cea0 2024-05-10T11:16:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.539+0000 7f274e000700 1 -- 172.21.15.27:0/1110434701 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f275000fa50 con 0x55f22067cea0 2024-05-10T11:16:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.539+0000 7f277140db80 1 -- 172.21.15.27:0/1110434701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27480aa870 con 0x55f22067cea0 2024-05-10T11:16:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.539+0000 7f274e000700 1 -- 172.21.15.27:0/1110434701 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2750031050 con 0x55f22067cea0 2024-05-10T11:16:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.539+0000 7f277140db80 10 client.25943 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.539+0000 7f277140db80 1 -- 172.21.15.27:0/1110434701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f220693fa0 con 0x55f22067cea0 2024-05-10T11:16:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.540+0000 7f274e000700 1 -- 172.21.15.27:0/1110434701 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27480aa870 con 0x55f22067cea0 2024-05-10T11:16:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.540+0000 7f274e000700 1 -- 172.21.15.27:0/1110434701 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f275001a050 con 0x55f22067cea0 2024-05-10T11:16:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.540+0000 7f277140db80 10 client.25943 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.540+0000 7f277140db80 10 client.25943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:16:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.540+0000 7f277140db80 20 client.25943 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.540+0000 7f277140db80 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f2206965e0 0x55f2206989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.540+0000 7f277140db80 4 client.25943 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:16:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.540+0000 7f277140db80 1 -- 172.21.15.27:0/1110434701 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2206411c0 con 0x55f2206965e0 2024-05-10T11:16:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.541+0000 7f2757400700 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f2206965e0 0x55f2206989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.541+0000 7f2757400700 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f2206965e0 0x55f2206989d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e065a6dae1d30b8 server_cookie=1d248b131fbb8daf in_seq=0 out_seq=0 2024-05-10T11:16:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.541+0000 7f274e000700 10 client.25943 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:16:59.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.542+0000 7f274e000700 1 -- 172.21.15.27:0/1110434701 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f2206411c0 con 0x55f2206965e0 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.542+0000 7f274e000700 10 client.25943 handle_command_reply: tid=0 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.542+0000 7f277140db80 1 client.25943 shutdown 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.542+0000 7f277140db80 1 -- 172.21.15.27:0/1110434701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 msgr2=0x55f220688ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.542+0000 7f277140db80 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 0x55f220688ba0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f275001c980 tx=0x7f27500047c0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.543+0000 7f277140db80 1 -- 172.21.15.27:0/1110434701 shutdown_connections 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.543+0000 7f277140db80 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f2206965e0 0x55f2206989d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.543+0000 7f277140db80 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f220681740 0x55f2206864f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.543+0000 7f277140db80 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f22067cea0 0x55f220688ba0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.543+0000 7f277140db80 1 --2- 172.21.15.27:0/1110434701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067d7c0 0x55f220685fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.543+0000 7f277140db80 1 -- 172.21.15.27:0/1110434701 >> 172.21.15.27:0/1110434701 conn(0x55f2205ec750 msgr2=0x55f220682bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.543+0000 7f274e000700 0 client.25943 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.543+0000 7f277140db80 1 -- 172.21.15.27:0/1110434701 shutdown_connections 2024-05-10T11:16:59.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.543+0000 7f277140db80 1 -- 172.21.15.27:0/1110434701 wait complete. 2024-05-10T11:16:59.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.543+0000 7f277140db80 20 client.25943 trim_cache size 0 max 16384 2024-05-10T11:16:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.567+0000 7f277140db80 1 Processor -- start 2024-05-10T11:16:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.568+0000 7f277140db80 1 -- start start 2024-05-10T11:16:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.568+0000 7f277140db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067cea0 0x55f2206927d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.568+0000 7f277140db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067d7c0 0x55f22069e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.568+0000 7f277140db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f220681740 0x55f22069e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.568+0000 7f277140db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f27601ad1d0 con 0x55f22067d7c0 2024-05-10T11:16:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.568+0000 7f277140db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f27601aed70 con 0x55f220681740 2024-05-10T11:16:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.568+0000 7f277140db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f27601ad8a0 con 0x55f22067cea0 2024-05-10T11:16:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.568+0000 7f275fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067cea0 0x55f2206927d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.568+0000 7f275fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067cea0 0x55f2206927d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51644/0 (socket says 172.21.15.27:51644) 2024-05-10T11:16:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.568+0000 7f275fe00700 1 -- 172.21.15.27:0/3155935888 learned_addr learned my addr 172.21.15.27:0/3155935888 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.568+0000 7f2764a00700 1 --2- 172.21.15.27:0/3155935888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f220681740 0x55f22069e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.568+0000 7f2757400700 1 --2- 172.21.15.27:0/3155935888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067d7c0 0x55f22069e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.569+0000 7f275fe00700 1 -- 172.21.15.27:0/3155935888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f220681740 msgr2=0x55f22069e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.569+0000 7f275fe00700 1 --2- 172.21.15.27:0/3155935888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f220681740 0x55f22069e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.569+0000 7f275fe00700 1 -- 172.21.15.27:0/3155935888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067d7c0 msgr2=0x55f22069e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.569+0000 7f275fe00700 1 --2- 172.21.15.27:0/3155935888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067d7c0 0x55f22069e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.569+0000 7f275fe00700 1 -- 172.21.15.27:0/3155935888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2206845a0 con 0x55f22067cea0 2024-05-10T11:16:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.569+0000 7f275fe00700 1 --2- 172.21.15.27:0/3155935888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067cea0 0x55f2206927d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2750011ab0 tx=0x7f27500093a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2e897bebce66583 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:59.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.569+0000 7f2754c00700 1 -- 172.21.15.27:0/3155935888 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750039070 con 0x55f22067cea0 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.569+0000 7f2754c00700 1 -- 172.21.15.27:0/3155935888 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f275000fd50 con 0x55f22067cea0 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.569+0000 7f2754c00700 1 -- 172.21.15.27:0/3155935888 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275000e4d0 con 0x55f22067cea0 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.570+0000 7f277140db80 1 -- 172.21.15.27:0/3155935888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067cea0 msgr2=0x55f2206927d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.570+0000 7f277140db80 1 --2- 172.21.15.27:0/3155935888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067cea0 0x55f2206927d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2750011ab0 tx=0x7f27500093a0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.570+0000 7f277140db80 1 -- 172.21.15.27:0/3155935888 shutdown_connections 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.570+0000 7f277140db80 1 --2- 172.21.15.27:0/3155935888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067cea0 0x55f2206927d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.570+0000 7f277140db80 1 --2- 172.21.15.27:0/3155935888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f220681740 0x55f22069e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.570+0000 7f277140db80 1 --2- 172.21.15.27:0/3155935888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067d7c0 0x55f22069e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.570+0000 7f277140db80 1 -- 172.21.15.27:0/3155935888 >> 172.21.15.27:0/3155935888 conn(0x55f2205ec750 msgr2=0x55f220673d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.571+0000 7f277140db80 1 -- 172.21.15.27:0/3155935888 shutdown_connections 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.571+0000 7f277140db80 1 -- 172.21.15.27:0/3155935888 wait complete. 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.571+0000 7f277140db80 1 -- start start 2024-05-10T11:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.571+0000 7f277140db80 1 Processor -- start 2024-05-10T11:16:59.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.572+0000 7f277140db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067cea0 0x55f22068edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.572+0000 7f277140db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 0x55f22068d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.572+0000 7f277140db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f220681740 0x55f22068d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.572+0000 7f277140db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f27601ae640 con 0x55f22067cea0 2024-05-10T11:16:59.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.572+0000 7f277140db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f27601adf70 con 0x55f220681740 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.572+0000 7f277140db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f27601ab170 con 0x55f22067d7c0 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.572+0000 7f277140db80 10 client.0 _ll_register_callbacks cb 0x55f22056dcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.572+0000 7f2757400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 0x55f22068d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.572+0000 7f275fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067cea0 0x55f22068edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.572+0000 7f2757400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 0x55f22068d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51650/0 (socket says 172.21.15.27:51650) 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.572+0000 7f2757400700 1 -- 172.21.15.27:0/3031912838 learned_addr learned my addr 172.21.15.27:0/3031912838 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.572+0000 7f2764a00700 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f220681740 0x55f22068d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.573+0000 7f2764a00700 1 -- 172.21.15.27:0/3031912838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 msgr2=0x55f22068d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.573+0000 7f2764a00700 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 0x55f22068d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.573+0000 7f2764a00700 1 -- 172.21.15.27:0/3031912838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067cea0 msgr2=0x55f22068edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.573+0000 7f2764a00700 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067cea0 0x55f22068edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.573+0000 7f2764a00700 1 -- 172.21.15.27:0/3031912838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f22068dc30 con 0x55f220681740 2024-05-10T11:16:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.573+0000 7f2757400700 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 0x55f22068d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:16:59.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.573+0000 7f275fe00700 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067cea0 0x55f22068edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:16:59.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.573+0000 7f2764a00700 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f220681740 0x55f22068d6f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55f2206897a0 tx=0x7f2758007e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d74a032f1b12a14 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:16:59.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.573+0000 7f274e000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:16:59.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.574+0000 7f274e000700 1 -- 172.21.15.27:0/3031912838 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2758004db0 con 0x55f220681740 2024-05-10T11:16:59.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.574+0000 7f274e000700 1 -- 172.21.15.27:0/3031912838 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f2758003a00 con 0x55f220681740 2024-05-10T11:16:59.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.574+0000 7f277140db80 1 -- 172.21.15.27:0/3031912838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f275000fd50 con 0x55f220681740 2024-05-10T11:16:59.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.574+0000 7f274e000700 1 -- 172.21.15.27:0/3031912838 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2758028050 con 0x55f220681740 2024-05-10T11:16:59.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.574+0000 7f277140db80 10 client.25949 fetch_fsmap learned FSMap version 31 2024-05-10T11:16:59.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.574+0000 7f277140db80 1 -- 172.21.15.27:0/3031912838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f220701cb0 con 0x55f220681740 2024-05-10T11:16:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.574+0000 7f274e000700 1 -- 172.21.15.27:0/3031912838 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f275000fd50 con 0x55f220681740 2024-05-10T11:16:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.574+0000 7f274e000700 1 -- 172.21.15.27:0/3031912838 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f27580055c0 con 0x55f220681740 2024-05-10T11:16:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.574+0000 7f277140db80 10 client.25949 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:16:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.574+0000 7f277140db80 10 client.25949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:16:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.574+0000 7f277140db80 20 client.25949 populate_metadata read hostname 'smithi027' 2024-05-10T11:16:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.575+0000 7f277140db80 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f2207023c0 0x55f2207047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:16:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.575+0000 7f277140db80 4 client.25949 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:16:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.575+0000 7f277140db80 1 -- 172.21.15.27:0/3031912838 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f2205d4480 con 0x55f2207023c0 2024-05-10T11:16:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.575+0000 7f275fe00700 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f2207023c0 0x55f2207047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:16:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.575+0000 7f275fe00700 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f2207023c0 0x55f2207047e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98bec9c9486ad8f8 server_cookie=ea54a7c88e57dbf5 in_seq=0 out_seq=0 2024-05-10T11:16:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.575+0000 7f274e000700 10 client.25949 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:16:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.577+0000 7f274e000700 1 -- 172.21.15.27:0/3031912838 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f2205d4480 con 0x55f2207023c0 2024-05-10T11:16:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.577+0000 7f274e000700 10 client.25949 handle_command_reply: tid=0 2024-05-10T11:16:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.577+0000 7f277140db80 1 client.25949 shutdown 2024-05-10T11:16:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.577+0000 7f277140db80 1 -- 172.21.15.27:0/3031912838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f220681740 msgr2=0x55f22068d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.577+0000 7f277140db80 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f220681740 0x55f22068d6f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55f2206897a0 tx=0x7f2758007e20 comp rx=0 tx=0).stop 2024-05-10T11:16:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.577+0000 7f277140db80 1 -- 172.21.15.27:0/3031912838 shutdown_connections 2024-05-10T11:16:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.577+0000 7f277140db80 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f2207023c0 0x55f2207047e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.578+0000 7f277140db80 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f220681740 0x55f22068d6f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.578+0000 7f277140db80 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f22067cea0 0x55f22068edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.578+0000 7f277140db80 1 --2- 172.21.15.27:0/3031912838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f22067d7c0 0x55f22068d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.578+0000 7f277140db80 1 -- 172.21.15.27:0/3031912838 >> 172.21.15.27:0/3031912838 conn(0x55f2205ec750 msgr2=0x55f220680620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.578+0000 7f274e000700 0 client.25949 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:16:59.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.578+0000 7f277140db80 1 -- 172.21.15.27:0/3031912838 shutdown_connections 2024-05-10T11:16:59.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.578+0000 7f277140db80 1 -- 172.21.15.27:0/3031912838 wait complete. 2024-05-10T11:16:59.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.578+0000 7f277140db80 20 client.25949 trim_cache size 0 max 16384 2024-05-10T11:16:59.579 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:16:59.579 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:16:59.579 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "928d53f7-f27b-4c56-a984-f0379c05643b", 2024-05-10T11:16:59.579 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:16:59.579 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:16:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.579+0000 7f2765400700 1 -- 172.21.15.27:0/4245979857 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f27440b2a70 msgr2=0x7f27440b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.579+0000 7f2765400700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f27440b2a70 0x7f27440b4f30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2750009d80 tx=0x7f2750009310 comp rx=0 tx=0).stop 2024-05-10T11:16:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.579+0000 7f2765400700 1 -- 172.21.15.27:0/4245979857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27600761f0 msgr2=0x7f276019c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:16:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.579+0000 7f2765400700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27600761f0 0x7f276019c2b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2748009230 tx=0x7f274800fa00 comp rx=0 tx=0).stop 2024-05-10T11:16:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.579+0000 7f2765400700 1 -- 172.21.15.27:0/4245979857 shutdown_connections 2024-05-10T11:16:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.579+0000 7f2765400700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f27440b2a70 0x7f27440b4f30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.579+0000 7f2765400700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27600758d0 0x7f276019bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.579+0000 7f2765400700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f276010fc10 0x7f27601a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.579+0000 7f2765400700 1 --2- 172.21.15.27:0/4245979857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27600761f0 0x7f276019c2b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:16:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.579+0000 7f2765400700 1 -- 172.21.15.27:0/4245979857 >> 172.21.15.27:0/4245979857 conn(0x7f27600fef20 msgr2=0x7f27600ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:16:59.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.579+0000 7f2765400700 1 -- 172.21.15.27:0/4245979857 shutdown_connections 2024-05-10T11:16:59.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:16:59.579+0000 7f2765400700 1 -- 172.21.15.27:0/4245979857 wait complete. 2024-05-10T11:16:59.661 INFO:tasks.workunit.client.0.smithi027.stdout: 1 145679 7.74 MB/sec warmup 104 sec latency 13.806 ms 2024-05-10T11:16:59.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:16:59 smithi076 ceph-mon[34152]: pgmap v116: 129 pgs: 129 active+clean; 88 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 245 op/s 2024-05-10T11:16:59.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:16:59 smithi189 ceph-mon[34842]: pgmap v116: 129 pgs: 129 active+clean; 88 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 245 op/s 2024-05-10T11:17:00.578 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:17:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:00 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:17:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:00 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:00 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:17:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:00 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:17:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:00 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:00 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:17:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:00 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:00.662 INFO:tasks.workunit.client.0.smithi027.stdout: 1 146424 7.67 MB/sec warmup 105 sec latency 33.646 ms 2024-05-10T11:17:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:00 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:17:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:00 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:00 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:17:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:00 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:17:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:00 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:00 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:17:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:00 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:00.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:00 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:17:00.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:00 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:00.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:00 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:17:00.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:00 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:17:00.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:00 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:00.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:00 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:17:00.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:00 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:01.078 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:17:01.541 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:01 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:17:01.541 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:01 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:01.542 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:01 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:01.542 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:01 smithi027 ceph-mon[30054]: pgmap v117: 129 pgs: 129 active+clean; 88 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 167 op/s 2024-05-10T11:17:01.662 INFO:tasks.workunit.client.0.smithi027.stdout: 1 147747 7.65 MB/sec warmup 106 sec latency 77.394 ms 2024-05-10T11:17:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:01 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:17:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:01 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:01 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:01 smithi076 ceph-mon[34152]: pgmap v117: 129 pgs: 129 active+clean; 88 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 167 op/s 2024-05-10T11:17:01.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:01 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:17:01.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:01 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:01.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:01 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:17:01.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:01 smithi189 ceph-mon[34842]: pgmap v117: 129 pgs: 129 active+clean; 88 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 167 op/s 2024-05-10T11:17:02.662 INFO:tasks.workunit.client.0.smithi027.stdout: 1 149099 7.69 MB/sec warmup 107 sec latency 80.577 ms 2024-05-10T11:17:03.390 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:03 smithi027 ceph-mon[30054]: pgmap v118: 129 pgs: 129 active+clean; 88 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 167 op/s 2024-05-10T11:17:03.662 INFO:tasks.workunit.client.0.smithi027.stdout: 1 149677 7.63 MB/sec warmup 108 sec latency 85.145 ms 2024-05-10T11:17:03.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:03 smithi076 ceph-mon[34152]: pgmap v118: 129 pgs: 129 active+clean; 88 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 167 op/s 2024-05-10T11:17:03.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.689+0000 7fb4b3400700 1 -- 172.21.15.27:0/2711005434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4ac10a9e0 msgr2=0x7fb4ac10ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:03.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.689+0000 7fb4b3400700 1 --2- 172.21.15.27:0/2711005434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4ac10a9e0 0x7fb4ac10ae60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009990 tx=0x7fb4a0009ca0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.689+0000 7fb4b3400700 1 -- 172.21.15.27:0/2711005434 shutdown_connections 2024-05-10T11:17:03.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.689+0000 7fb4b3400700 1 --2- 172.21.15.27:0/2711005434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4ac11cc80 0x7fb4ac11f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.689+0000 7fb4b3400700 1 --2- 172.21.15.27:0/2711005434 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ac111c80 0x7fb4ac10a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.689+0000 7fb4b3400700 1 --2- 172.21.15.27:0/2711005434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4ac10a9e0 0x7fb4ac10ae60 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009990 tx=0x7fb4a0009ca0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.689+0000 7fb4b3400700 1 -- 172.21.15.27:0/2711005434 >> 172.21.15.27:0/2711005434 conn(0x7fb4ac06e610 msgr2=0x7fb4ac06ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:03.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.689+0000 7fb4b3400700 1 -- 172.21.15.27:0/2711005434 shutdown_connections 2024-05-10T11:17:03.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.689+0000 7fb4b3400700 1 -- 172.21.15.27:0/2711005434 wait complete. 2024-05-10T11:17:03.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.691+0000 7fb4b3400700 1 Processor -- start 2024-05-10T11:17:03.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.691+0000 7fb4b3400700 1 -- start start 2024-05-10T11:17:03.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.692+0000 7fb4b3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ac111c80 0x7fb4ac1ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.692+0000 7fb4b3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4ac11cc80 0x7fb4ac1aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.692+0000 7fb4b3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4ac1a89c0 0x7fb4ac1a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.692+0000 7fb4b3400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac122540 con 0x7fb4ac11cc80 2024-05-10T11:17:03.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.692+0000 7fb4b3400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac1223c0 con 0x7fb4ac111c80 2024-05-10T11:17:03.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.692+0000 7fb4b3400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac1226c0 con 0x7fb4ac1a89c0 2024-05-10T11:17:03.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.693+0000 7fb4b0a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ac111c80 0x7fb4ac1ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.693+0000 7fb4b0a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ac111c80 0x7fb4ac1ae800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60562/0 (socket says 172.21.15.27:60562) 2024-05-10T11:17:03.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.693+0000 7fb4b0a00700 1 -- 172.21.15.27:0/2498038713 learned_addr learned my addr 172.21.15.27:0/2498038713 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:03.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.693+0000 7fb4b1400700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4ac1a89c0 0x7fb4ac1a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.693+0000 7fb4abe00700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4ac11cc80 0x7fb4ac1aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.693+0000 7fb4b0a00700 1 -- 172.21.15.27:0/2498038713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4ac1a89c0 msgr2=0x7fb4ac1a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:03.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.693+0000 7fb4b0a00700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4ac1a89c0 0x7fb4ac1a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.693+0000 7fb4b0a00700 1 -- 172.21.15.27:0/2498038713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4ac11cc80 msgr2=0x7fb4ac1aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:03.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.693+0000 7fb4b0a00700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4ac11cc80 0x7fb4ac1aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.693+0000 7fb4b0a00700 1 -- 172.21.15.27:0/2498038713 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4ac1a9540 con 0x7fb4ac111c80 2024-05-10T11:17:03.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.694+0000 7fb4b0a00700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ac111c80 0x7fb4ac1ae800 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb49800c9b0 tx=0x7fb49800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:03.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.695+0000 7fb4a9600700 1 -- 172.21.15.27:0/2498038713 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb498007b00 con 0x7fb4ac111c80 2024-05-10T11:17:03.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.695+0000 7fb4a9600700 1 -- 172.21.15.27:0/2498038713 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb498007ca0 con 0x7fb4ac111c80 2024-05-10T11:17:03.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.695+0000 7fb4a9600700 1 -- 172.21.15.27:0/2498038713 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb498018660 con 0x7fb4ac111c80 2024-05-10T11:17:03.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.695+0000 7fb4b3400700 1 -- 172.21.15.27:0/2498038713 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4ac1b56d0 con 0x7fb4ac111c80 2024-05-10T11:17:03.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.695+0000 7fb4b3400700 1 -- 172.21.15.27:0/2498038713 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4ac1b5a70 con 0x7fb4ac111c80 2024-05-10T11:17:03.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.704+0000 7fb4bdd9bb80 1 Processor -- start 2024-05-10T11:17:03.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.704+0000 7fb4bdd9bb80 1 -- start start 2024-05-10T11:17:03.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.705+0000 7fb4bdd9bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf060 0x55cd61cdf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.705+0000 7fb4bdd9bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf980 0x55cd61ce3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.705+0000 7fb4bdd9bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61ce39d0 0x55cd61ce3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.705+0000 7fb4bdd9bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cd61c31a20 con 0x55cd61cdf980 2024-05-10T11:17:03.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.705+0000 7fb4bdd9bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cd61bcf760 con 0x55cd61ce39d0 2024-05-10T11:17:03.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.705+0000 7fb4bdd9bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cd61c2d9e0 con 0x55cd61cdf060 2024-05-10T11:17:03.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.705+0000 7fb4b1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf060 0x55cd61cdf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.706+0000 7fb4b1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf060 0x55cd61cdf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51682/0 (socket says 172.21.15.27:51682) 2024-05-10T11:17:03.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.706+0000 7fb4b1400700 1 -- 172.21.15.27:0/3625302500 learned_addr learned my addr 172.21.15.27:0/3625302500 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:03.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.706+0000 7fb4abe00700 1 --2- 172.21.15.27:0/3625302500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61ce39d0 0x55cd61ce3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.706+0000 7fb4a9600700 1 -- 172.21.15.27:0/2498038713 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb498004830 con 0x7fb4ac111c80 2024-05-10T11:17:03.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.706+0000 7fb4b1400700 1 -- 172.21.15.27:0/3625302500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61ce39d0 msgr2=0x55cd61ce3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:03.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.706+0000 7fb4b1400700 1 --2- 172.21.15.27:0/3625302500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61ce39d0 0x55cd61ce3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.707+0000 7fb4b1400700 1 -- 172.21.15.27:0/3625302500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf980 msgr2=0x55cd61ce3490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:17:03.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.708+0000 7fb4b0a00700 1 --2- 172.21.15.27:0/3625302500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf980 0x55cd61ce3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.709+0000 7fb4b1400700 1 --2- 172.21.15.27:0/3625302500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf980 0x55cd61ce3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.709+0000 7fb4b1400700 1 -- 172.21.15.27:0/3625302500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd61b7e8d0 con 0x55cd61cdf060 2024-05-10T11:17:03.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.709+0000 7fb4b0a00700 1 --2- 172.21.15.27:0/3625302500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf980 0x55cd61ce3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:17:03.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.710+0000 7fb4b1400700 1 --2- 172.21.15.27:0/3625302500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf060 0x55cd61cdf440 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4011600 tx=0x7fb4a4009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecd50f2ffc5fb8f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:03.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.710+0000 7fb49e000700 1 -- 172.21.15.27:0/3625302500 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4019030 con 0x55cd61cdf060 2024-05-10T11:17:03.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.710+0000 7fb49e000700 1 -- 172.21.15.27:0/3625302500 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4a40092e0 con 0x55cd61cdf060 2024-05-10T11:17:03.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.710+0000 7fb49e000700 1 -- 172.21.15.27:0/3625302500 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4007500 con 0x55cd61cdf060 2024-05-10T11:17:03.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.711+0000 7fb4a9600700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb4940b2af0 0x7fb4940b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.711+0000 7fb4a9600700 1 -- 172.21.15.27:0/2498038713 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fb4980e29e0 con 0x7fb4ac111c80 2024-05-10T11:17:03.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.712+0000 7fb4b0a00700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb4940b2af0 0x7fb4940b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.713+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3625302500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf060 msgr2=0x55cd61cdf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:03.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.713+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3625302500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf060 0x55cd61cdf440 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4011600 tx=0x7fb4a4009920 comp rx=0 tx=0).stop 2024-05-10T11:17:03.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.714+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3625302500 shutdown_connections 2024-05-10T11:17:03.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.714+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3625302500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf060 0x55cd61cdf440 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.714+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3625302500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61ce39d0 0x55cd61ce3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.714+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3625302500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf980 0x55cd61ce3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.714+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3625302500 >> 172.21.15.27:0/3625302500 conn(0x55cd61c0db60 msgr2=0x55cd61cd97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:03.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.714+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3625302500 shutdown_connections 2024-05-10T11:17:03.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.715+0000 7fb4b0a00700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb4940b2af0 0x7fb4940b4fb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb4980a9820 tx=0x7fb4980a5040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:03.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.715+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3625302500 wait complete. 2024-05-10T11:17:03.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.716+0000 7fb4bdd9bb80 1 -- start start 2024-05-10T11:17:03.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.717+0000 7fb4bdd9bb80 1 Processor -- start 2024-05-10T11:17:03.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.718+0000 7fb4bdd9bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 0x55cd61ceadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.718+0000 7fb4bdd9bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf980 0x55cd61ce8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.718+0000 7fb4bdd9bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61ce39d0 0x55cd61ce8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.718+0000 7fb4bdd9bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cd61cebbe0 con 0x55cd61ce39d0 2024-05-10T11:17:03.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.718+0000 7fb4bdd9bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cd61b97b20 con 0x55cd61cdf060 2024-05-10T11:17:03.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.718+0000 7fb4bdd9bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cd61cebd60 con 0x55cd61cdf980 2024-05-10T11:17:03.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.718+0000 7fb4bdd9bb80 10 client.0 _ll_register_callbacks cb 0x55cd61bd8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:17:03.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.718+0000 7fb4abe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf980 0x55cd61ce8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.718+0000 7fb4abe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf980 0x55cd61ce8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51688/0 (socket says 172.21.15.27:51688) 2024-05-10T11:17:03.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.718+0000 7fb4abe00700 1 -- 172.21.15.27:0/4249998870 learned_addr learned my addr 172.21.15.27:0/4249998870 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:03.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.719+0000 7fb4b1400700 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 0x55cd61ceadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.719+0000 7fb4abe00700 1 -- 172.21.15.27:0/4249998870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 msgr2=0x55cd61ceadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:03.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.719+0000 7fb4abe00700 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 0x55cd61ceadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.719+0000 7fb4abe00700 1 -- 172.21.15.27:0/4249998870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61ce39d0 msgr2=0x55cd61ce8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:03.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.719+0000 7fb4abe00700 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61ce39d0 0x55cd61ce8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.719+0000 7fb4abe00700 1 -- 172.21.15.27:0/4249998870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd61bf7420 con 0x55cd61cdf980 2024-05-10T11:17:03.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.719+0000 7fb4b1400700 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 0x55cd61ceadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:17:03.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.720+0000 7fb4abe00700 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf980 0x55cd61ce8150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009990 tx=0x7fb4a0005810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c5ff07dcbce6a5e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:03.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.720+0000 7fb492a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:17:03.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.720+0000 7fb492a00700 1 -- 172.21.15.27:0/4249998870 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a000bd00 con 0x55cd61cdf980 2024-05-10T11:17:03.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.720+0000 7fb492a00700 1 -- 172.21.15.27:0/4249998870 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fb4a0003680 con 0x55cd61cdf980 2024-05-10T11:17:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.720+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/4249998870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4a40092e0 con 0x55cd61cdf980 2024-05-10T11:17:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.721+0000 7fb492a00700 1 -- 172.21.15.27:0/4249998870 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb4a000bea0 con 0x55cd61cdf980 2024-05-10T11:17:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.721+0000 7fb4bdd9bb80 10 client.25948 fetch_fsmap learned FSMap version 31 2024-05-10T11:17:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.721+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/4249998870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cd61d07380 con 0x55cd61cdf980 2024-05-10T11:17:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.721+0000 7fb492a00700 1 -- 172.21.15.27:0/4249998870 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4a40092e0 con 0x55cd61cdf980 2024-05-10T11:17:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.722+0000 7fb492a00700 1 -- 172.21.15.27:0/4249998870 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fb4a000fb30 con 0x55cd61cdf980 2024-05-10T11:17:03.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.723+0000 7fb4bdd9bb80 10 client.25948 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:17:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.723+0000 7fb4bdd9bb80 10 client.25948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:17:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.723+0000 7fb4bdd9bb80 20 client.25948 populate_metadata read hostname 'smithi027' 2024-05-10T11:17:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.723+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cd61d09950 0x55cd61d0bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.723+0000 7fb4bdd9bb80 4 client.25948 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:17:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.723+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/4249998870 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd61cac0a0 con 0x55cd61d09950 2024-05-10T11:17:03.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.725+0000 7fb4b1400700 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cd61d09950 0x55cd61d0bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.726+0000 7fb4b1400700 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cd61d09950 0x55cd61d0bd70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c903fa28242c263e server_cookie=27d264efac7867e0 in_seq=0 out_seq=0 2024-05-10T11:17:03.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.726+0000 7fb492a00700 10 client.25948 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:17:03.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.728+0000 7fb492a00700 1 -- 172.21.15.27:0/4249998870 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cd61cac0a0 con 0x55cd61d09950 2024-05-10T11:17:03.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.728+0000 7fb492a00700 10 client.25948 handle_command_reply: tid=0 2024-05-10T11:17:03.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.729+0000 7fb4bdd9bb80 1 client.25948 shutdown 2024-05-10T11:17:03.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.729+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/4249998870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf980 msgr2=0x55cd61ce8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:03.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.729+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf980 0x55cd61ce8150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009990 tx=0x7fb4a0005810 comp rx=0 tx=0).stop 2024-05-10T11:17:03.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.730+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/4249998870 shutdown_connections 2024-05-10T11:17:03.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.730+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cd61d09950 0x55cd61d0bd70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.730+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61ce39d0 0x55cd61ce8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.730+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 0x55cd61ceadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.730+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/4249998870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf980 0x55cd61ce8150 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.730+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/4249998870 >> 172.21.15.27:0/4249998870 conn(0x55cd61c0db60 msgr2=0x55cd61cd42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.730+0000 7fb492a00700 0 client.25948 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:17:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.731+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/4249998870 shutdown_connections 2024-05-10T11:17:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.731+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/4249998870 wait complete. 2024-05-10T11:17:03.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.731+0000 7fb4bdd9bb80 20 client.25948 trim_cache size 0 max 16384 2024-05-10T11:17:03.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:03 smithi189 ceph-mon[34842]: pgmap v118: 129 pgs: 129 active+clean; 88 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 167 op/s 2024-05-10T11:17:03.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.781+0000 7fb4bdd9bb80 1 Processor -- start 2024-05-10T11:17:03.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.782+0000 7fb4bdd9bb80 1 -- start start 2024-05-10T11:17:03.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.782+0000 7fb4bdd9bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61c0db60 0x55cd61ce9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.782+0000 7fb4bdd9bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf060 0x55cd61d11620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.782+0000 7fb4bdd9bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf980 0x55cd61d11b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.782+0000 7fb4bdd9bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac1b6030 con 0x55cd61cdf060 2024-05-10T11:17:03.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.782+0000 7fb4bdd9bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac1b7bd0 con 0x55cd61c0db60 2024-05-10T11:17:03.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.782+0000 7fb4bdd9bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac1b6700 con 0x55cd61cdf980 2024-05-10T11:17:03.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.783+0000 7fb4abe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf060 0x55cd61d11620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.783+0000 7fb4abe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf060 0x55cd61d11620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36552/0 (socket says 172.21.15.27:36552) 2024-05-10T11:17:03.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.783+0000 7fb4abe00700 1 -- 172.21.15.27:0/3162074859 learned_addr learned my addr 172.21.15.27:0/3162074859 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:03.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.784+0000 7fb4b1400700 1 --2- 172.21.15.27:0/3162074859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61c0db60 0x55cd61ce9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.784+0000 7fb4b1400700 1 --2- 172.21.15.27:0/3162074859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf980 0x55cd61d11b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.784+0000 7fb4b1400700 1 -- 172.21.15.27:0/3162074859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf980 msgr2=0x55cd61d11b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:03.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.784+0000 7fb4b1400700 1 --2- 172.21.15.27:0/3162074859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf980 0x55cd61d11b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.784+0000 7fb4b1400700 1 -- 172.21.15.27:0/3162074859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf060 msgr2=0x55cd61d11620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:03.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.784+0000 7fb4b1400700 1 --2- 172.21.15.27:0/3162074859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf060 0x55cd61d11620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.784+0000 7fb4b1400700 1 -- 172.21.15.27:0/3162074859 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd61cda610 con 0x55cd61c0db60 2024-05-10T11:17:03.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.784+0000 7fb4b1400700 1 --2- 172.21.15.27:0/3162074859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61c0db60 0x55cd61ce9f10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009ec0 tx=0x7fb4a4009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=363c50a28ab33167 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:03.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.785+0000 7fb49d600700 1 -- 172.21.15.27:0/3162074859 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4004d10 con 0x55cd61c0db60 2024-05-10T11:17:03.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.785+0000 7fb49d600700 1 -- 172.21.15.27:0/3162074859 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4a4027680 con 0x55cd61c0db60 2024-05-10T11:17:03.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.785+0000 7fb49d600700 1 -- 172.21.15.27:0/3162074859 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4007880 con 0x55cd61c0db60 2024-05-10T11:17:03.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.788+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3162074859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61c0db60 msgr2=0x55cd61ce9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:03.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.788+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3162074859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61c0db60 0x55cd61ce9f10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009ec0 tx=0x7fb4a4009e30 comp rx=0 tx=0).stop 2024-05-10T11:17:03.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.789+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3162074859 shutdown_connections 2024-05-10T11:17:03.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.789+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3162074859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61cdf980 0x55cd61d11b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.789+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3162074859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61c0db60 0x55cd61ce9f10 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.789+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3162074859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf060 0x55cd61d11620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.789+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3162074859 >> 172.21.15.27:0/3162074859 conn(0x55cd61cd4640 msgr2=0x55cd61ce9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:03.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.790+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3162074859 shutdown_connections 2024-05-10T11:17:03.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.790+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3162074859 wait complete. 2024-05-10T11:17:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.791+0000 7fb4bdd9bb80 1 -- start start 2024-05-10T11:17:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.791+0000 7fb4bdd9bb80 1 Processor -- start 2024-05-10T11:17:03.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.793+0000 7fb4bdd9bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61c0db60 0x55cd61cf12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.794+0000 7fb4bdd9bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 0x55cd61cee020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.794+0000 7fb4bdd9bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf980 0x55cd61cee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.794+0000 7fb4bdd9bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac1b74a0 con 0x55cd61cdf980 2024-05-10T11:17:03.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.794+0000 7fb4bdd9bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac1b6dd0 con 0x55cd61cdf060 2024-05-10T11:17:03.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.794+0000 7fb4bdd9bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac118f90 con 0x55cd61c0db60 2024-05-10T11:17:03.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.794+0000 7fb4abe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 0x55cd61cee020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.794+0000 7fb4abe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 0x55cd61cee020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60596/0 (socket says 172.21.15.27:60596) 2024-05-10T11:17:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.794+0000 7fb4abe00700 1 -- 172.21.15.27:0/3424520635 learned_addr learned my addr 172.21.15.27:0/3424520635 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.794+0000 7fb4b1400700 1 --2- 172.21.15.27:0/3424520635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61c0db60 0x55cd61cf12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.795+0000 7fb4bdd9bb80 10 client.0 _ll_register_callbacks cb 0x55cd61bd8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:17:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.795+0000 7fb4abe00700 1 -- 172.21.15.27:0/3424520635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61c0db60 msgr2=0x55cd61cf12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.795+0000 7fb4abe00700 1 --2- 172.21.15.27:0/3424520635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61c0db60 0x55cd61cf12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.795+0000 7fb4abe00700 1 -- 172.21.15.27:0/3424520635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf980 msgr2=0x55cd61cee560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.795+0000 7fb4abe00700 1 --2- 172.21.15.27:0/3424520635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf980 0x55cd61cee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.795+0000 7fb4abe00700 1 -- 172.21.15.27:0/3424520635 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd61ceeaa0 con 0x55cd61cdf060 2024-05-10T11:17:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.796+0000 7fb4abe00700 1 --2- 172.21.15.27:0/3424520635 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 0x55cd61cee020 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009ce0 tx=0x7fb4a0022c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5a4f3672a3fbe0e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.796+0000 7fb492a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:17:03.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.797+0000 7fb492a00700 1 -- 172.21.15.27:0/3424520635 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0030030 con 0x55cd61cdf060 2024-05-10T11:17:03.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.797+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3424520635 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4a4027680 con 0x55cd61cdf060 2024-05-10T11:17:03.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.797+0000 7fb492a00700 1 -- 172.21.15.27:0/3424520635 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fb4a0031070 con 0x55cd61cdf060 2024-05-10T11:17:03.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.798+0000 7fb492a00700 1 -- 172.21.15.27:0/3424520635 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb4a0045050 con 0x55cd61cdf060 2024-05-10T11:17:03.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.798+0000 7fb4bdd9bb80 10 client.25961 fetch_fsmap learned FSMap version 31 2024-05-10T11:17:03.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.798+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3424520635 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cd61d74ec0 con 0x55cd61cdf060 2024-05-10T11:17:03.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.799+0000 7fb492a00700 1 -- 172.21.15.27:0/3424520635 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4a4027680 con 0x55cd61cdf060 2024-05-10T11:17:03.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.799+0000 7fb492a00700 1 -- 172.21.15.27:0/3424520635 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7fb4a000f460 con 0x55cd61cdf060 2024-05-10T11:17:03.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.799+0000 7fb4bdd9bb80 10 client.25961 fetch_fsmap finished waiting for FSMap version 31 2024-05-10T11:17:03.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.799+0000 7fb4bdd9bb80 10 client.25961 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:17:03.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.799+0000 7fb4bdd9bb80 20 client.25961 populate_metadata read hostname 'smithi027' 2024-05-10T11:17:03.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.799+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3424520635 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cd61ce39d0 0x55cd61d775b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:03.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.799+0000 7fb4bdd9bb80 4 client.25961 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:17:03.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.799+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3424520635 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cd61c3f400 con 0x55cd61ce39d0 2024-05-10T11:17:03.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.800+0000 7fb4b1400700 1 --2- 172.21.15.27:0/3424520635 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cd61ce39d0 0x55cd61d775b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:03.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.801+0000 7fb4b1400700 1 --2- 172.21.15.27:0/3424520635 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cd61ce39d0 0x55cd61d775b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d29c485cda5cd7d server_cookie=532b9620d86da473 in_seq=0 out_seq=0 2024-05-10T11:17:03.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.801+0000 7fb492a00700 10 client.25961 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:17:03.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.801+0000 7fb492a00700 1 -- 172.21.15.27:0/3424520635 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cd61c3f400 con 0x55cd61ce39d0 2024-05-10T11:17:03.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.801+0000 7fb492a00700 10 client.25961 handle_command_reply: tid=0 2024-05-10T11:17:03.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.802+0000 7fb4bdd9bb80 1 client.25961 shutdown 2024-05-10T11:17:03.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.802+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3424520635 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 msgr2=0x55cd61cee020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:03.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.802+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3424520635 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 0x55cd61cee020 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009ce0 tx=0x7fb4a0022c00 comp rx=0 tx=0).stop 2024-05-10T11:17:03.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.803+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3424520635 shutdown_connections 2024-05-10T11:17:03.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.803+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3424520635 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cd61ce39d0 0x55cd61d775b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.803+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3424520635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cd61c0db60 0x55cd61cf12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.803+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3424520635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd61cdf980 0x55cd61cee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.803+0000 7fb4bdd9bb80 1 --2- 172.21.15.27:0/3424520635 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd61cdf060 0x55cd61cee020 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.803+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3424520635 >> 172.21.15.27:0/3424520635 conn(0x55cd61cd4640 msgr2=0x55cd61cde700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.803+0000 7fb492a00700 0 client.25961 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:17:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.804+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3424520635 shutdown_connections 2024-05-10T11:17:03.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.804+0000 7fb4bdd9bb80 1 -- 172.21.15.27:0/3424520635 wait complete. 2024-05-10T11:17:03.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.804+0000 7fb4bdd9bb80 20 client.25961 trim_cache size 0 max 16384 2024-05-10T11:17:03.808 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:17:03.808 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:17:03.808 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:17:03.808 INFO:teuthology.orchestra.run.smithi027.stdout: "928d53f7-f27b-4c56-a984-f0379c05643b": { 2024-05-10T11:17:03.808 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:17:03.809 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "928d53f7-f27b-4c56-a984-f0379c05643b", 2024-05-10T11:17:03.809 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:17:03.809 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:17:03.809 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:17:03.809 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:17:03.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.810+0000 7fb4b3400700 1 -- 172.21.15.27:0/2498038713 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb4940b2af0 msgr2=0x7fb4940b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:03.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.810+0000 7fb4b3400700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb4940b2af0 0x7fb4940b4fb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb4980a9820 tx=0x7fb4980a5040 comp rx=0 tx=0).stop 2024-05-10T11:17:03.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.810+0000 7fb4b3400700 1 -- 172.21.15.27:0/2498038713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ac111c80 msgr2=0x7fb4ac1ae800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:03.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.810+0000 7fb4b3400700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ac111c80 0x7fb4ac1ae800 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb49800c9b0 tx=0x7fb49800ccc0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.811+0000 7fb4b3400700 1 -- 172.21.15.27:0/2498038713 shutdown_connections 2024-05-10T11:17:03.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.811+0000 7fb4b3400700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb4940b2af0 0x7fb4940b4fb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.811+0000 7fb4b3400700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4ac1a89c0 0x7fb4ac1a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.811+0000 7fb4b3400700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4ac111c80 0x7fb4ac1ae800 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.811+0000 7fb4b3400700 1 --2- 172.21.15.27:0/2498038713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4ac11cc80 0x7fb4ac1aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:03.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.811+0000 7fb4b3400700 1 -- 172.21.15.27:0/2498038713 >> 172.21.15.27:0/2498038713 conn(0x7fb4ac06e610 msgr2=0x7fb4ac11ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:03.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.813+0000 7fb4b3400700 1 -- 172.21.15.27:0/2498038713 shutdown_connections 2024-05-10T11:17:03.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:03.813+0000 7fb4b3400700 1 -- 172.21.15.27:0/2498038713 wait complete. 2024-05-10T11:17:04.662 INFO:tasks.workunit.client.0.smithi027.stdout: 1 151664 7.65 MB/sec warmup 109 sec latency 28.500 ms 2024-05-10T11:17:05.220 INFO:tasks.cephfs.filesystem:scrub status for tag:928d53f7-f27b-4c56-a984-f0379c05643b - {'path': '/', 'tag': '928d53f7-f27b-4c56-a984-f0379c05643b', 'options': 'recursive,force'} 2024-05-10T11:17:05.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:05 smithi027 ceph-mon[30054]: pgmap v119: 129 pgs: 129 active+clean; 91 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 283 op/s 2024-05-10T11:17:05.662 INFO:tasks.workunit.client.0.smithi027.stdout: 1 152918 7.67 MB/sec warmup 110 sec latency 25.050 ms 2024-05-10T11:17:05.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:05 smithi076 ceph-mon[34152]: pgmap v119: 129 pgs: 129 active+clean; 91 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 283 op/s 2024-05-10T11:17:05.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:05 smithi189 ceph-mon[34842]: pgmap v119: 129 pgs: 129 active+clean; 91 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 283 op/s 2024-05-10T11:17:06.662 INFO:tasks.workunit.client.0.smithi027.stdout: 1 153960 7.64 MB/sec warmup 111 sec latency 39.357 ms 2024-05-10T11:17:06.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:06 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:17:06.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:06 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:17:07.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:06 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:17:07.663 INFO:tasks.workunit.client.0.smithi027.stdout: 1 156056 7.68 MB/sec warmup 112 sec latency 14.783 ms 2024-05-10T11:17:07.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:07 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:17:07.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:07 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:17:07.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:07 smithi189 ceph-mon[34842]: pgmap v120: 129 pgs: 129 active+clean; 91 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 200 op/s 2024-05-10T11:17:08.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:07 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:17:08.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:07 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:17:08.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:07 smithi027 ceph-mon[30054]: pgmap v120: 129 pgs: 129 active+clean; 91 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 200 op/s 2024-05-10T11:17:08.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:07 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:17:08.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:07 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:17:08.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:07 smithi076 ceph-mon[34152]: pgmap v120: 129 pgs: 129 active+clean; 91 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 200 op/s 2024-05-10T11:17:08.663 INFO:tasks.workunit.client.0.smithi027.stdout: 1 156568 7.63 MB/sec warmup 113 sec latency 73.737 ms 2024-05-10T11:17:09.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:09 smithi027 ceph-mon[30054]: pgmap v121: 129 pgs: 129 active+clean; 94 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 278 op/s 2024-05-10T11:17:09.663 INFO:tasks.workunit.client.0.smithi027.stdout: 1 158496 7.63 MB/sec warmup 114 sec latency 28.248 ms 2024-05-10T11:17:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:09 smithi076 ceph-mon[34152]: pgmap v121: 129 pgs: 129 active+clean; 94 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 278 op/s 2024-05-10T11:17:09.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:09 smithi189 ceph-mon[34842]: pgmap v121: 129 pgs: 129 active+clean; 94 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 278 op/s 2024-05-10T11:17:10.663 INFO:tasks.workunit.client.0.smithi027.stdout: 1 160013 7.66 MB/sec warmup 115 sec latency 14.882 ms 2024-05-10T11:17:11.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:11 smithi027 ceph-mon[30054]: pgmap v122: 129 pgs: 129 active+clean; 94 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-05-10T11:17:11.663 INFO:tasks.workunit.client.0.smithi027.stdout: 1 161201 7.64 MB/sec warmup 116 sec latency 27.673 ms 2024-05-10T11:17:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:11 smithi076 ceph-mon[34152]: pgmap v122: 129 pgs: 129 active+clean; 94 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-05-10T11:17:11.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:11 smithi189 ceph-mon[34842]: pgmap v122: 129 pgs: 129 active+clean; 94 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-05-10T11:17:12.663 INFO:tasks.workunit.client.0.smithi027.stdout: 1 163433 7.69 MB/sec warmup 117 sec latency 15.677 ms 2024-05-10T11:17:13.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:13 smithi027 ceph-mon[30054]: pgmap v123: 129 pgs: 129 active+clean; 94 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-05-10T11:17:13.663 INFO:tasks.workunit.client.0.smithi027.stdout: 1 164276 7.65 MB/sec warmup 118 sec latency 37.394 ms 2024-05-10T11:17:13.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:13 smithi076 ceph-mon[34152]: pgmap v123: 129 pgs: 129 active+clean; 94 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-05-10T11:17:13.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:13 smithi189 ceph-mon[34842]: pgmap v123: 129 pgs: 129 active+clean; 94 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-05-10T11:17:14.663 INFO:tasks.workunit.client.0.smithi027.stdout: 1 166749 7.71 MB/sec warmup 119 sec latency 12.508 ms 2024-05-10T11:17:15.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:15 smithi027 ceph-mon[30054]: pgmap v124: 129 pgs: 129 active+clean; 99 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 261 op/s 2024-05-10T11:17:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:15 smithi076 ceph-mon[34152]: pgmap v124: 129 pgs: 129 active+clean; 99 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 261 op/s 2024-05-10T11:17:15.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:15 smithi189 ceph-mon[34842]: pgmap v124: 129 pgs: 129 active+clean; 99 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 261 op/s 2024-05-10T11:17:16.664 INFO:tasks.workunit.client.0.smithi027.stdout: 1 169303 7.79 MB/sec execute 1 sec latency 37.265 ms 2024-05-10T11:17:17.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:17 smithi027 ceph-mon[30054]: pgmap v125: 129 pgs: 129 active+clean; 99 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 146 op/s 2024-05-10T11:17:17.664 INFO:tasks.workunit.client.0.smithi027.stdout: 1 170614 9.45 MB/sec execute 2 sec latency 16.309 ms 2024-05-10T11:17:17.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:17 smithi076 ceph-mon[34152]: pgmap v125: 129 pgs: 129 active+clean; 99 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 146 op/s 2024-05-10T11:17:17.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:17 smithi189 ceph-mon[34842]: pgmap v125: 129 pgs: 129 active+clean; 99 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 146 op/s 2024-05-10T11:17:18.666 INFO:tasks.workunit.client.0.smithi027.stdout: 1 171553 7.03 MB/sec execute 3 sec latency 35.276 ms 2024-05-10T11:17:19.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:19 smithi027 ceph-mon[30054]: pgmap v126: 129 pgs: 129 active+clean; 110 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-05-10T11:17:19.666 INFO:tasks.workunit.client.0.smithi027.stdout: 1 173649 8.35 MB/sec execute 4 sec latency 31.568 ms 2024-05-10T11:17:19.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:19 smithi076 ceph-mon[34152]: pgmap v126: 129 pgs: 129 active+clean; 110 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-05-10T11:17:19.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:19 smithi189 ceph-mon[34842]: pgmap v126: 129 pgs: 129 active+clean; 110 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-05-10T11:17:20.666 INFO:tasks.workunit.client.0.smithi027.stdout: 1 174371 7.61 MB/sec execute 5 sec latency 23.194 ms 2024-05-10T11:17:21.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:21 smithi027 ceph-mon[30054]: pgmap v127: 129 pgs: 129 active+clean; 110 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-05-10T11:17:21.666 INFO:tasks.workunit.client.0.smithi027.stdout: 1 176218 7.51 MB/sec execute 6 sec latency 31.066 ms 2024-05-10T11:17:21.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:21 smithi076 ceph-mon[34152]: pgmap v127: 129 pgs: 129 active+clean; 110 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-05-10T11:17:21.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:21 smithi189 ceph-mon[34842]: pgmap v127: 129 pgs: 129 active+clean; 110 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-05-10T11:17:22.666 INFO:tasks.workunit.client.0.smithi027.stdout: 1 177639 8.12 MB/sec execute 7 sec latency 18.988 ms 2024-05-10T11:17:23.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:23 smithi027 ceph-mon[30054]: pgmap v128: 129 pgs: 129 active+clean; 110 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-05-10T11:17:23.666 INFO:tasks.workunit.client.0.smithi027.stdout: 1 178597 7.41 MB/sec execute 8 sec latency 20.400 ms 2024-05-10T11:17:23.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:23 smithi076 ceph-mon[34152]: pgmap v128: 129 pgs: 129 active+clean; 110 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-05-10T11:17:23.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:23 smithi189 ceph-mon[34842]: pgmap v128: 129 pgs: 129 active+clean; 110 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-05-10T11:17:24.666 INFO:tasks.workunit.client.0.smithi027.stdout: 1 180871 8.28 MB/sec execute 9 sec latency 42.876 ms 2024-05-10T11:17:25.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:25 smithi027 ceph-mon[30054]: pgmap v129: 129 pgs: 129 active+clean; 113 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 217 op/s 2024-05-10T11:17:25.666 INFO:tasks.workunit.client.0.smithi027.stdout: 1 181415 7.63 MB/sec execute 10 sec latency 17.250 ms 2024-05-10T11:17:25.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:25 smithi076 ceph-mon[34152]: pgmap v129: 129 pgs: 129 active+clean; 113 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 217 op/s 2024-05-10T11:17:25.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:25 smithi189 ceph-mon[34842]: pgmap v129: 129 pgs: 129 active+clean; 113 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 217 op/s 2024-05-10T11:17:26.667 INFO:tasks.workunit.client.0.smithi027.stdout: 1 183112 7.46 MB/sec execute 11 sec latency 46.302 ms 2024-05-10T11:17:27.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:27 smithi027 ceph-mon[30054]: pgmap v130: 129 pgs: 129 active+clean; 113 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 149 op/s 2024-05-10T11:17:27.667 INFO:tasks.workunit.client.0.smithi027.stdout: 1 184694 7.92 MB/sec execute 12 sec latency 18.253 ms 2024-05-10T11:17:27.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:27 smithi076 ceph-mon[34152]: pgmap v130: 129 pgs: 129 active+clean; 113 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 149 op/s 2024-05-10T11:17:27.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:27 smithi189 ceph-mon[34842]: pgmap v130: 129 pgs: 129 active+clean; 113 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 149 op/s 2024-05-10T11:17:28.667 INFO:tasks.workunit.client.0.smithi027.stdout: 1 185200 7.36 MB/sec execute 13 sec latency 45.433 ms 2024-05-10T11:17:29.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:29 smithi027 ceph-mon[30054]: pgmap v131: 129 pgs: 129 active+clean; 116 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-05-10T11:17:29.667 INFO:tasks.workunit.client.0.smithi027.stdout: 1 187046 7.39 MB/sec execute 14 sec latency 55.935 ms 2024-05-10T11:17:29.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:29 smithi076 ceph-mon[34152]: pgmap v131: 129 pgs: 129 active+clean; 116 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-05-10T11:17:29.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:29 smithi189 ceph-mon[34842]: pgmap v131: 129 pgs: 129 active+clean; 116 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-05-10T11:17:30.667 INFO:tasks.workunit.client.0.smithi027.stdout: 1 188372 7.63 MB/sec execute 15 sec latency 17.409 ms 2024-05-10T11:17:31.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:31 smithi027 ceph-mon[30054]: pgmap v132: 129 pgs: 129 active+clean; 116 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 156 op/s 2024-05-10T11:17:31.667 INFO:tasks.workunit.client.0.smithi027.stdout: 1 189160 7.28 MB/sec execute 16 sec latency 60.397 ms 2024-05-10T11:17:31.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:31 smithi076 ceph-mon[34152]: pgmap v132: 129 pgs: 129 active+clean; 116 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 156 op/s 2024-05-10T11:17:31.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:31 smithi189 ceph-mon[34842]: pgmap v132: 129 pgs: 129 active+clean; 116 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 156 op/s 2024-05-10T11:17:32.668 INFO:tasks.workunit.client.0.smithi027.stdout: 1 191354 7.58 MB/sec execute 17 sec latency 12.273 ms 2024-05-10T11:17:33.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:33 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:17:33.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:33 smithi027 ceph-mon[30054]: pgmap v133: 129 pgs: 129 active+clean; 116 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 156 op/s 2024-05-10T11:17:33.668 INFO:tasks.workunit.client.0.smithi027.stdout: 1 191990 7.42 MB/sec execute 18 sec latency 25.175 ms 2024-05-10T11:17:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:33 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:17:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:33 smithi076 ceph-mon[34152]: pgmap v133: 129 pgs: 129 active+clean; 116 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 156 op/s 2024-05-10T11:17:33.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:33 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:17:33.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:33 smithi189 ceph-mon[34842]: pgmap v133: 129 pgs: 129 active+clean; 116 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 156 op/s 2024-05-10T11:17:34.668 INFO:tasks.workunit.client.0.smithi027.stdout: 1 193525 7.32 MB/sec execute 19 sec latency 40.743 ms 2024-05-10T11:17:35.220 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:17:35.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:35 smithi027 ceph-mon[30054]: pgmap v134: 129 pgs: 129 active+clean; 119 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 231 op/s 2024-05-10T11:17:35.668 INFO:tasks.workunit.client.0.smithi027.stdout: 1 195369 7.62 MB/sec execute 20 sec latency 14.751 ms 2024-05-10T11:17:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:35 smithi076 ceph-mon[34152]: pgmap v134: 129 pgs: 129 active+clean; 119 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 231 op/s 2024-05-10T11:17:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:17:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:35 smithi189 ceph-mon[34842]: pgmap v134: 129 pgs: 129 active+clean; 119 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 231 op/s 2024-05-10T11:17:36.668 INFO:tasks.workunit.client.0.smithi027.stdout: 1 196284 7.37 MB/sec execute 21 sec latency 29.568 ms 2024-05-10T11:17:37.668 INFO:tasks.workunit.client.0.smithi027.stdout: 1 198727 7.75 MB/sec execute 22 sec latency 35.746 ms 2024-05-10T11:17:37.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:37 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:17:37.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:37 smithi027 ceph-mon[30054]: pgmap v135: 129 pgs: 129 active+clean; 119 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 152 op/s 2024-05-10T11:17:37.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:37 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:17:37.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:37 smithi076 ceph-mon[34152]: pgmap v135: 129 pgs: 129 active+clean; 119 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 152 op/s 2024-05-10T11:17:37.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:37 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:17:37.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:37 smithi189 ceph-mon[34842]: pgmap v135: 129 pgs: 129 active+clean; 119 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 152 op/s 2024-05-10T11:17:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.177+0000 7fd41b400700 1 -- 172.21.15.27:0/851542611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41406c530 msgr2=0x7fd414110350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.177+0000 7fd41b400700 1 --2- 172.21.15.27:0/851542611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41406c530 0x7fd414110350 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd40c00b0a0 tx=0x7fd40c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.177+0000 7fd41b400700 1 -- 172.21.15.27:0/851542611 shutdown_connections 2024-05-10T11:17:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.177+0000 7fd41b400700 1 --2- 172.21.15.27:0/851542611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41406c530 0x7fd414110350 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.177+0000 7fd41b400700 1 --2- 172.21.15.27:0/851542611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd414109530 0x7fd414109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.177+0000 7fd41b400700 1 --2- 172.21.15.27:0/851542611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd41406bb70 0x7fd41406bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.177+0000 7fd41b400700 1 -- 172.21.15.27:0/851542611 >> 172.21.15.27:0/851542611 conn(0x7fd414079350 msgr2=0x7fd414079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.177+0000 7fd41b400700 1 -- 172.21.15.27:0/851542611 shutdown_connections 2024-05-10T11:17:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.177+0000 7fd41b400700 1 -- 172.21.15.27:0/851542611 wait complete. 2024-05-10T11:17:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.178+0000 7fd41b400700 1 Processor -- start 2024-05-10T11:17:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.178+0000 7fd41b400700 1 -- start start 2024-05-10T11:17:38.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.179+0000 7fd41b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41406bb70 0x7fd4141a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.179+0000 7fd41b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd41406c530 0x7fd4141a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.179+0000 7fd41b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd414109530 0x7fd41419baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.179+0000 7fd41b400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd414113640 con 0x7fd414109530 2024-05-10T11:17:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.179+0000 7fd41b400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd4141134c0 con 0x7fd41406c530 2024-05-10T11:17:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.179+0000 7fd41b400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd4141137c0 con 0x7fd41406bb70 2024-05-10T11:17:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.179+0000 7fd418a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41406bb70 0x7fd4141a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.179+0000 7fd418a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41406bb70 0x7fd4141a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:57528/0 (socket says 172.21.15.27:57528) 2024-05-10T11:17:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.179+0000 7fd418a00700 1 -- 172.21.15.27:0/4083171523 learned_addr learned my addr 172.21.15.27:0/4083171523 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.179+0000 7fd40be00700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd41406c530 0x7fd4141a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.179+0000 7fd419400700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd414109530 0x7fd41419baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.180+0000 7fd418a00700 1 -- 172.21.15.27:0/4083171523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd41406c530 msgr2=0x7fd4141a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.180+0000 7fd418a00700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd41406c530 0x7fd4141a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.180+0000 7fd418a00700 1 -- 172.21.15.27:0/4083171523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd414109530 msgr2=0x7fd41419baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.180+0000 7fd418a00700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd414109530 0x7fd41419baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.180+0000 7fd418a00700 1 -- 172.21.15.27:0/4083171523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41419c250 con 0x7fd41406bb70 2024-05-10T11:17:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.180+0000 7fd418a00700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41406bb70 0x7fd4141a1930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00eb30 tx=0x7fd3fc00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.180+0000 7fd412000700 1 -- 172.21.15.27:0/4083171523 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc00cbd0 con 0x7fd41406bb70 2024-05-10T11:17:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.180+0000 7fd412000700 1 -- 172.21.15.27:0/4083171523 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3fc0044d0 con 0x7fd41406bb70 2024-05-10T11:17:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.180+0000 7fd41b400700 1 -- 172.21.15.27:0/4083171523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd41419c540 con 0x7fd41406bb70 2024-05-10T11:17:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.181+0000 7fd412000700 1 -- 172.21.15.27:0/4083171523 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc003e30 con 0x7fd41406bb70 2024-05-10T11:17:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.181+0000 7fd41b400700 1 -- 172.21.15.27:0/4083171523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4141a8800 con 0x7fd41406bb70 2024-05-10T11:17:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.184+0000 7fd412000700 1 -- 172.21.15.27:0/4083171523 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd3fc004000 con 0x7fd41406bb70 2024-05-10T11:17:38.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.186+0000 7fd412000700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd3f40b2a70 0x7fd3f40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.186+0000 7fd412000700 1 -- 172.21.15.27:0/4083171523 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fd3fc0e2ce0 con 0x7fd41406bb70 2024-05-10T11:17:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.187+0000 7fd425d9cb80 1 Processor -- start 2024-05-10T11:17:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.187+0000 7fd40be00700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd3f40b2a70 0x7fd3f40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.187+0000 7fd425d9cb80 1 -- start start 2024-05-10T11:17:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.187+0000 7fd425d9cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e04fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.187+0000 7fd425d9cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7dff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.187+0000 7fd425d9cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7dffd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd425d9cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e5e7d51a20 con 0x55e5e7e054e0 2024-05-10T11:17:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd425d9cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55e5e7cef760 con 0x55e5e7e04bc0 2024-05-10T11:17:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd425d9cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e5e7d4d9e0 con 0x55e5e7dff8f0 2024-05-10T11:17:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd418a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e04fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd419400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7dffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd40be00700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd3f40b2a70 0x7fd3f40b4f30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd404009680 tx=0x7fd404009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd418a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e04fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36938/0 (socket says 172.21.15.27:36938) 2024-05-10T11:17:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd418a00700 1 -- 172.21.15.27:0/675198898 learned_addr learned my addr 172.21.15.27:0/675198898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd40be00700 1 --2- 172.21.15.27:0/675198898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7dff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd418a00700 1 -- 172.21.15.27:0/675198898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 msgr2=0x55e5e7dffd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd418a00700 1 --2- 172.21.15.27:0/675198898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7dffd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd418a00700 1 -- 172.21.15.27:0/675198898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 msgr2=0x55e5e7dff2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd418a00700 1 --2- 172.21.15.27:0/675198898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7dff2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd418a00700 1 -- 172.21.15.27:0/675198898 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5e7c9e8d0 con 0x55e5e7e04bc0 2024-05-10T11:17:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.188+0000 7fd40be00700 1 --2- 172.21.15.27:0/675198898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7dff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:17:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.189+0000 7fd418a00700 1 --2- 172.21.15.27:0/675198898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e04fa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00c5b0 tx=0x7fd3fc0a9960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=500a90dc8b30d2f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.189+0000 7fd40a000700 1 -- 172.21.15.27:0/675198898 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0a9300 con 0x55e5e7e04bc0 2024-05-10T11:17:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.189+0000 7fd40a000700 1 -- 172.21.15.27:0/675198898 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3fc01e4f0 con 0x55e5e7e04bc0 2024-05-10T11:17:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.189+0000 7fd40a000700 1 -- 172.21.15.27:0/675198898 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0aa050 con 0x55e5e7e04bc0 2024-05-10T11:17:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.190+0000 7fd425d9cb80 1 -- 172.21.15.27:0/675198898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 msgr2=0x55e5e7e04fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.190+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/675198898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e04fa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00c5b0 tx=0x7fd3fc0a9960 comp rx=0 tx=0).stop 2024-05-10T11:17:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.190+0000 7fd425d9cb80 1 -- 172.21.15.27:0/675198898 shutdown_connections 2024-05-10T11:17:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.190+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/675198898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7dffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.190+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/675198898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e04fa0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.190+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/675198898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7dff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.190+0000 7fd425d9cb80 1 -- 172.21.15.27:0/675198898 >> 172.21.15.27:0/675198898 conn(0x55e5e7d776d0 msgr2=0x55e5e7df9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.190+0000 7fd425d9cb80 1 -- 172.21.15.27:0/675198898 shutdown_connections 2024-05-10T11:17:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.190+0000 7fd425d9cb80 1 -- 172.21.15.27:0/675198898 wait complete. 2024-05-10T11:17:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.191+0000 7fd425d9cb80 1 -- start start 2024-05-10T11:17:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.191+0000 7fd425d9cb80 1 Processor -- start 2024-05-10T11:17:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd425d9cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7e0b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd425d9cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e085f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd425d9cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7e08b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd425d9cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e5e7e0bf20 con 0x55e5e7e054e0 2024-05-10T11:17:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd425d9cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55e5e7cb7b20 con 0x55e5e7e04bc0 2024-05-10T11:17:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd425d9cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e5e7e0c0a0 con 0x55e5e7dff8f0 2024-05-10T11:17:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd425d9cb80 10 client.0 _ll_register_callbacks cb 0x55e5e7cf8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:17:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd40be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e085f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd40be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e085f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36944/0 (socket says 172.21.15.27:36944) 2024-05-10T11:17:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd40be00700 1 -- 172.21.15.27:0/720243358 learned_addr learned my addr 172.21.15.27:0/720243358 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd418a00700 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7e0b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd419400700 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7e08b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd418a00700 1 -- 172.21.15.27:0/720243358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 msgr2=0x55e5e7e085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd418a00700 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e085f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd418a00700 1 -- 172.21.15.27:0/720243358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 msgr2=0x55e5e7e08b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd418a00700 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7e08b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.192+0000 7fd418a00700 1 -- 172.21.15.27:0/720243358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5e7d17420 con 0x55e5e7dff8f0 2024-05-10T11:17:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.193+0000 7fd419400700 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7e08b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:17:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.193+0000 7fd418a00700 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7e0b250 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0a9890 tx=0x7fd3fc0aa0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e6687c9013acb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.193+0000 7fd402a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:17:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.193+0000 7fd402a00700 1 -- 172.21.15.27:0/720243358 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0a9300 con 0x55e5e7dff8f0 2024-05-10T11:17:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.193+0000 7fd402a00700 1 -- 172.21.15.27:0/720243358 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fd3fc0a6180 con 0x55e5e7dff8f0 2024-05-10T11:17:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.193+0000 7fd425d9cb80 1 -- 172.21.15.27:0/720243358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e5e7dcc140 con 0x55e5e7dff8f0 2024-05-10T11:17:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.193+0000 7fd402a00700 1 -- 172.21.15.27:0/720243358 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd3fc08b050 con 0x55e5e7dff8f0 2024-05-10T11:17:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.193+0000 7fd425d9cb80 10 client.25966 fetch_fsmap learned FSMap version 33 2024-05-10T11:17:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.193+0000 7fd425d9cb80 1 -- 172.21.15.27:0/720243358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e5e7e0bf20 con 0x55e5e7dff8f0 2024-05-10T11:17:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.194+0000 7fd402a00700 1 -- 172.21.15.27:0/720243358 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e5e7dcc140 con 0x55e5e7dff8f0 2024-05-10T11:17:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.194+0000 7fd402a00700 1 -- 172.21.15.27:0/720243358 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fd3fc0a5500 con 0x55e5e7dff8f0 2024-05-10T11:17:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.194+0000 7fd425d9cb80 10 client.25966 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:17:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.194+0000 7fd425d9cb80 10 client.25966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:17:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.194+0000 7fd425d9cb80 20 client.25966 populate_metadata read hostname 'smithi027' 2024-05-10T11:17:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.194+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5e7e18b20 0x55e5e7e1af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.194+0000 7fd425d9cb80 4 client.25966 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:17:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.194+0000 7fd425d9cb80 1 -- 172.21.15.27:0/720243358 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5e7d5f400 con 0x55e5e7e18b20 2024-05-10T11:17:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.194+0000 7fd40be00700 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5e7e18b20 0x55e5e7e1af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.195+0000 7fd40be00700 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5e7e18b20 0x55e5e7e1af10 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98ec5f2629069d09 server_cookie=faaed0de6b6e0f17 in_seq=0 out_seq=0 2024-05-10T11:17:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.195+0000 7fd402a00700 10 client.25966 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:17:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.196+0000 7fd402a00700 1 -- 172.21.15.27:0/720243358 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e5e7d5f400 con 0x55e5e7e18b20 2024-05-10T11:17:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.196+0000 7fd402a00700 10 client.25966 handle_command_reply: tid=0 2024-05-10T11:17:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.196+0000 7fd425d9cb80 1 client.25966 shutdown 2024-05-10T11:17:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.196+0000 7fd425d9cb80 1 -- 172.21.15.27:0/720243358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 msgr2=0x55e5e7e0b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.196+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7e0b250 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0a9890 tx=0x7fd3fc0aa0a0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.196+0000 7fd425d9cb80 1 -- 172.21.15.27:0/720243358 shutdown_connections 2024-05-10T11:17:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.196+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5e7e18b20 0x55e5e7e1af10 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.196+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7e0b250 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.196+0000 7fd402a00700 0 client.25966 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:17:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.196+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7e08b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.196+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/720243358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.196+0000 7fd425d9cb80 1 -- 172.21.15.27:0/720243358 >> 172.21.15.27:0/720243358 conn(0x55e5e7d776d0 msgr2=0x55e5e7df9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.197+0000 7fd425d9cb80 1 -- 172.21.15.27:0/720243358 shutdown_connections 2024-05-10T11:17:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.197+0000 7fd425d9cb80 1 -- 172.21.15.27:0/720243358 wait complete. 2024-05-10T11:17:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.197+0000 7fd425d9cb80 20 client.25966 trim_cache size 0 max 16384 2024-05-10T11:17:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.222+0000 7fd425d9cb80 1 Processor -- start 2024-05-10T11:17:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.222+0000 7fd425d9cb80 1 -- start start 2024-05-10T11:17:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.222+0000 7fd425d9cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7dff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.222+0000 7fd425d9cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e062a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd425d9cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e054e0 0x55e5e7e06db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd425d9cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd4141b1810 con 0x55e5e7e04bc0 2024-05-10T11:17:38.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd425d9cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd4141b3350 con 0x55e5e7e054e0 2024-05-10T11:17:38.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd425d9cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd4141b1ee0 con 0x55e5e7dff8f0 2024-05-10T11:17:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd418a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7dff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd419400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e054e0 0x55e5e7e06db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd418a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7dff2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:57552/0 (socket says 172.21.15.27:57552) 2024-05-10T11:17:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd418a00700 1 -- 172.21.15.27:0/2151667273 learned_addr learned my addr 172.21.15.27:0/2151667273 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd40be00700 1 --2- 172.21.15.27:0/2151667273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd419400700 1 -- 172.21.15.27:0/2151667273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 msgr2=0x55e5e7dff2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd419400700 1 --2- 172.21.15.27:0/2151667273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7dff2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd419400700 1 -- 172.21.15.27:0/2151667273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e04bc0 msgr2=0x55e5e7e062a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:38.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd419400700 1 --2- 172.21.15.27:0/2151667273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e062a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd419400700 1 -- 172.21.15.27:0/2151667273 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5e7e07ca0 con 0x55e5e7e054e0 2024-05-10T11:17:38.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd418a00700 1 --2- 172.21.15.27:0/2151667273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7dff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:17:38.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.223+0000 7fd40be00700 1 --2- 172.21.15.27:0/2151667273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:17:38.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.224+0000 7fd419400700 1 --2- 172.21.15.27:0/2151667273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e054e0 0x55e5e7e06db0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009fd0 tx=0x7fd40c009390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7561789f4ee83d41 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:38.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.224+0000 7fd409600700 1 -- 172.21.15.27:0/2151667273 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40c00bcb0 con 0x55e5e7e054e0 2024-05-10T11:17:38.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.224+0000 7fd409600700 1 -- 172.21.15.27:0/2151667273 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd40c0074e0 con 0x55e5e7e054e0 2024-05-10T11:17:38.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.224+0000 7fd409600700 1 -- 172.21.15.27:0/2151667273 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40c0052a0 con 0x55e5e7e054e0 2024-05-10T11:17:38.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.225+0000 7fd425d9cb80 1 -- 172.21.15.27:0/2151667273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e054e0 msgr2=0x55e5e7e06db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:38.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.225+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/2151667273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e054e0 0x55e5e7e06db0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009fd0 tx=0x7fd40c009390 comp rx=0 tx=0).stop 2024-05-10T11:17:38.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.225+0000 7fd425d9cb80 1 -- 172.21.15.27:0/2151667273 shutdown_connections 2024-05-10T11:17:38.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.225+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/2151667273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7dff8f0 0x55e5e7dff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.225+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/2151667273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7e054e0 0x55e5e7e06db0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.225+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/2151667273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.225+0000 7fd425d9cb80 1 -- 172.21.15.27:0/2151667273 >> 172.21.15.27:0/2151667273 conn(0x55e5e7d776d0 msgr2=0x55e5e7e03f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:38.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.226+0000 7fd425d9cb80 1 -- 172.21.15.27:0/2151667273 shutdown_connections 2024-05-10T11:17:38.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.226+0000 7fd425d9cb80 1 -- 172.21.15.27:0/2151667273 wait complete. 2024-05-10T11:17:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.226+0000 7fd425d9cb80 1 -- start start 2024-05-10T11:17:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.226+0000 7fd425d9cb80 1 Processor -- start 2024-05-10T11:17:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.227+0000 7fd425d9cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7dff8f0 0x55e5e7e1ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.227+0000 7fd425d9cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e20500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.227+0000 7fd425d9cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7e20a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.227+0000 7fd425d9cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd4141b2c80 con 0x55e5e7e054e0 2024-05-10T11:17:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.227+0000 7fd425d9cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd4141b25b0 con 0x55e5e7dff8f0 2024-05-10T11:17:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.227+0000 7fd425d9cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd4141af7b0 con 0x55e5e7e04bc0 2024-05-10T11:17:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.227+0000 7fd425d9cb80 10 client.0 _ll_register_callbacks cb 0x55e5e7cf8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:17:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.228+0000 7fd419400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7e20a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.228+0000 7fd40be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e20500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.228+0000 7fd40be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e20500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:57556/0 (socket says 172.21.15.27:57556) 2024-05-10T11:17:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.228+0000 7fd419400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7e20a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40796/0 (socket says 172.21.15.27:40796) 2024-05-10T11:17:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.228+0000 7fd40be00700 1 -- 172.21.15.27:0/1203584867 learned_addr learned my addr 172.21.15.27:0/1203584867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.228+0000 7fd418a00700 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7dff8f0 0x55e5e7e1ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.228+0000 7fd40be00700 1 -- 172.21.15.27:0/1203584867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7dff8f0 msgr2=0x55e5e7e1ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.228+0000 7fd40be00700 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7dff8f0 0x55e5e7e1ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.228+0000 7fd40be00700 1 -- 172.21.15.27:0/1203584867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 msgr2=0x55e5e7e20a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.228+0000 7fd40be00700 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7e20a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.228+0000 7fd40be00700 1 -- 172.21.15.27:0/1203584867 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5e7e82370 con 0x55e5e7e04bc0 2024-05-10T11:17:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.228+0000 7fd419400700 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7e20a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-05-10T11:17:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.229+0000 7fd40be00700 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e20500 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55e5e7e1b500 tx=0x7fd4040047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=762c0213adca973e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.229+0000 7fd402a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:17:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.229+0000 7fd402a00700 1 -- 172.21.15.27:0/1203584867 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd404004570 con 0x55e5e7e04bc0 2024-05-10T11:17:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.229+0000 7fd425d9cb80 1 -- 172.21.15.27:0/1203584867 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd40c0074e0 con 0x55e5e7e04bc0 2024-05-10T11:17:38.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.229+0000 7fd402a00700 1 -- 172.21.15.27:0/1203584867 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fd404029420 con 0x55e5e7e04bc0 2024-05-10T11:17:38.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.230+0000 7fd402a00700 1 -- 172.21.15.27:0/1203584867 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd404039050 con 0x55e5e7e04bc0 2024-05-10T11:17:38.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.230+0000 7fd425d9cb80 10 client.25972 fetch_fsmap learned FSMap version 33 2024-05-10T11:17:38.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.230+0000 7fd425d9cb80 1 -- 172.21.15.27:0/1203584867 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e5e7e83720 con 0x55e5e7e04bc0 2024-05-10T11:17:38.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.230+0000 7fd402a00700 1 -- 172.21.15.27:0/1203584867 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd40c0074e0 con 0x55e5e7e04bc0 2024-05-10T11:17:38.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.230+0000 7fd402a00700 1 -- 172.21.15.27:0/1203584867 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fd40401d910 con 0x55e5e7e04bc0 2024-05-10T11:17:38.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.230+0000 7fd425d9cb80 10 client.25972 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:17:38.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.230+0000 7fd425d9cb80 10 client.25972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:17:38.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.230+0000 7fd425d9cb80 20 client.25972 populate_metadata read hostname 'smithi027' 2024-05-10T11:17:38.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.230+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5e7e83e40 0x55e5e7e86230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:38.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.230+0000 7fd425d9cb80 4 client.25972 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:17:38.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.230+0000 7fd425d9cb80 1 -- 172.21.15.27:0/1203584867 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e5e7d594a0 con 0x55e5e7e83e40 2024-05-10T11:17:38.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.231+0000 7fd418a00700 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5e7e83e40 0x55e5e7e86230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.231+0000 7fd418a00700 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5e7e83e40 0x55e5e7e86230 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52389bcdbc18ce7c server_cookie=42ef1660b19124a6 in_seq=0 out_seq=0 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.231+0000 7fd402a00700 10 client.25972 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.231+0000 7fd402a00700 1 -- 172.21.15.27:0/1203584867 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e5e7d594a0 con 0x55e5e7e83e40 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.231+0000 7fd402a00700 10 client.25972 handle_command_reply: tid=0 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.231+0000 7fd425d9cb80 1 client.25972 shutdown 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.232+0000 7fd425d9cb80 1 -- 172.21.15.27:0/1203584867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7e04bc0 msgr2=0x55e5e7e20500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.232+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e20500 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55e5e7e1b500 tx=0x7fd4040047c0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.232+0000 7fd425d9cb80 1 -- 172.21.15.27:0/1203584867 shutdown_connections 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.232+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5e7e83e40 0x55e5e7e86230 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.232+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5e7e054e0 0x55e5e7e20a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.232+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5e7dff8f0 0x55e5e7e1ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.232+0000 7fd425d9cb80 1 --2- 172.21.15.27:0/1203584867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5e7e04bc0 0x55e5e7e20500 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.232+0000 7fd402a00700 0 client.25972 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:17:38.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.232+0000 7fd425d9cb80 1 -- 172.21.15.27:0/1203584867 >> 172.21.15.27:0/1203584867 conn(0x55e5e7d776d0 msgr2=0x55e5e7e03700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:38.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.232+0000 7fd425d9cb80 1 -- 172.21.15.27:0/1203584867 shutdown_connections 2024-05-10T11:17:38.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.232+0000 7fd425d9cb80 1 -- 172.21.15.27:0/1203584867 wait complete. 2024-05-10T11:17:38.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.232+0000 7fd425d9cb80 20 client.25972 trim_cache size 0 max 16384 2024-05-10T11:17:38.236 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:17:38.236 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:17:38.236 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:17:38.236 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:17:38.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.233+0000 7fd41b400700 1 -- 172.21.15.27:0/4083171523 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd3f40b2a70 msgr2=0x7fd3f40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:38.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.233+0000 7fd41b400700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd3f40b2a70 0x7fd3f40b4f30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd404009680 tx=0x7fd404009260 comp rx=0 tx=0).stop 2024-05-10T11:17:38.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.233+0000 7fd41b400700 1 -- 172.21.15.27:0/4083171523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41406bb70 msgr2=0x7fd4141a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:38.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.233+0000 7fd41b400700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41406bb70 0x7fd4141a1930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00eb30 tx=0x7fd3fc00ee40 comp rx=0 tx=0).stop 2024-05-10T11:17:38.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.233+0000 7fd41b400700 1 -- 172.21.15.27:0/4083171523 shutdown_connections 2024-05-10T11:17:38.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.233+0000 7fd41b400700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd3f40b2a70 0x7fd3f40b4f30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.233+0000 7fd41b400700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41406bb70 0x7fd4141a1930 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.233+0000 7fd41b400700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd414109530 0x7fd41419baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.233+0000 7fd41b400700 1 --2- 172.21.15.27:0/4083171523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd41406c530 0x7fd4141a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:38.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.233+0000 7fd41b400700 1 -- 172.21.15.27:0/4083171523 >> 172.21.15.27:0/4083171523 conn(0x7fd414079350 msgr2=0x7fd41410e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:38.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.233+0000 7fd41b400700 1 -- 172.21.15.27:0/4083171523 shutdown_connections 2024-05-10T11:17:38.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:38.233+0000 7fd41b400700 1 -- 172.21.15.27:0/4083171523 wait complete. 2024-05-10T11:17:38.668 INFO:tasks.workunit.client.0.smithi027.stdout: 1 199378 7.49 MB/sec execute 23 sec latency 27.703 ms 2024-05-10T11:17:38.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:38 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:17:38.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:38 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:17:38.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:38 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:17:39.134 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:17:39.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:17:39.629 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:17:39.668 INFO:tasks.workunit.client.0.smithi027.stdout: 1 201989 7.76 MB/sec execute 24 sec latency 32.342 ms 2024-05-10T11:17:39.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:39 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:17:39.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:39 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:17:39.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:39 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:17:39.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:39 smithi076 ceph-mon[34152]: pgmap v136: 129 pgs: 129 active+clean; 121 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 232 op/s 2024-05-10T11:17:39.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:39 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:17:39.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:39 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:17:39.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:39 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:17:39.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:39 smithi027 ceph-mon[30054]: pgmap v136: 129 pgs: 129 active+clean; 121 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 232 op/s 2024-05-10T11:17:39.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:39 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:17:39.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:39 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:17:39.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:39 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:17:39.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:39 smithi189 ceph-mon[34842]: pgmap v136: 129 pgs: 129 active+clean; 121 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 232 op/s 2024-05-10T11:17:40.668 INFO:tasks.workunit.client.0.smithi027.stdout: 1 202688 7.64 MB/sec execute 25 sec latency 58.728 ms 2024-05-10T11:17:41.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:41 smithi027 ceph-mon[30054]: pgmap v137: 129 pgs: 129 active+clean; 121 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 155 op/s 2024-05-10T11:17:41.668 INFO:tasks.workunit.client.0.smithi027.stdout: 1 204655 7.64 MB/sec execute 26 sec latency 28.718 ms 2024-05-10T11:17:41.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:41 smithi076 ceph-mon[34152]: pgmap v137: 129 pgs: 129 active+clean; 121 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 155 op/s 2024-05-10T11:17:41.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:41 smithi189 ceph-mon[34842]: pgmap v137: 129 pgs: 129 active+clean; 121 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 155 op/s 2024-05-10T11:17:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.375+0000 7f51bae00700 1 -- 172.21.15.27:0/4113776258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b4109510 msgr2=0x7f51b41098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.375+0000 7f51bae00700 1 --2- 172.21.15.27:0/4113776258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b4109510 0x7f51b41098f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009990 tx=0x7f51a4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:17:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.375+0000 7f51bae00700 1 -- 172.21.15.27:0/4113776258 shutdown_connections 2024-05-10T11:17:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.375+0000 7f51bae00700 1 --2- 172.21.15.27:0/4113776258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b4109510 0x7f51b41098f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.375+0000 7f51bae00700 1 --2- 172.21.15.27:0/4113776258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51b406bb40 0x7f51b4110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.375+0000 7f51bae00700 1 --2- 172.21.15.27:0/4113776258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b406b180 0x7f51b406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.375+0000 7f51bae00700 1 -- 172.21.15.27:0/4113776258 >> 172.21.15.27:0/4113776258 conn(0x7f51b4079350 msgr2=0x7f51b4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.375+0000 7f51bae00700 1 -- 172.21.15.27:0/4113776258 shutdown_connections 2024-05-10T11:17:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.375+0000 7f51bae00700 1 -- 172.21.15.27:0/4113776258 wait complete. 2024-05-10T11:17:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.376+0000 7f51bae00700 1 Processor -- start 2024-05-10T11:17:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.377+0000 7f51bae00700 1 -- start start 2024-05-10T11:17:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.377+0000 7f51bae00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b406b180 0x7f51b419f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.377+0000 7f51bae00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b406bb40 0x7f51b419fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.377+0000 7f51bae00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51b4109510 0x7f51b4199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.377+0000 7f51bae00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f51b4113670 con 0x7f51b406bb40 2024-05-10T11:17:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.377+0000 7f51bae00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f51b41134f0 con 0x7f51b4109510 2024-05-10T11:17:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.377+0000 7f51bae00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f51b41137f0 con 0x7f51b406b180 2024-05-10T11:17:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.377+0000 7f51b3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b406b180 0x7f51b419f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.377+0000 7f51b3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b406b180 0x7f51b419f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:57564/0 (socket says 172.21.15.27:57564) 2024-05-10T11:17:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.378+0000 7f51b3e00700 1 -- 172.21.15.27:0/3811102691 learned_addr learned my addr 172.21.15.27:0/3811102691 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.378+0000 7f51b3400700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b406bb40 0x7f51b419fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.377+0000 7f51b8e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51b4109510 0x7f51b4199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.378+0000 7f51b3e00700 1 -- 172.21.15.27:0/3811102691 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51b4109510 msgr2=0x7f51b4199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.378+0000 7f51b3e00700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51b4109510 0x7f51b4199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.378+0000 7f51b3e00700 1 -- 172.21.15.27:0/3811102691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b406bb40 msgr2=0x7f51b419fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.378+0000 7f51b3e00700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b406bb40 0x7f51b419fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.378+0000 7f51b3e00700 1 -- 172.21.15.27:0/3811102691 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b419a1d0 con 0x7f51b406b180 2024-05-10T11:17:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.378+0000 7f51b3e00700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b406b180 0x7f51b419f8b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f51a4000c00 tx=0x7f51a40057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.379+0000 7f51b0c00700 1 -- 172.21.15.27:0/3811102691 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4027070 con 0x7f51b406b180 2024-05-10T11:17:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.379+0000 7f51b0c00700 1 -- 172.21.15.27:0/3811102691 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51a4022470 con 0x7f51b406b180 2024-05-10T11:17:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.379+0000 7f51b0c00700 1 -- 172.21.15.27:0/3811102691 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4017650 con 0x7f51b406b180 2024-05-10T11:17:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.379+0000 7f51bae00700 1 -- 172.21.15.27:0/3811102691 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b419a460 con 0x7f51b406b180 2024-05-10T11:17:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.379+0000 7f51bae00700 1 -- 172.21.15.27:0/3811102691 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b41a6780 con 0x7f51b406b180 2024-05-10T11:17:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.380+0000 7f51bae00700 1 -- 172.21.15.27:0/3811102691 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51b410b9f0 con 0x7f51b406b180 2024-05-10T11:17:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.382+0000 7f51b0c00700 1 -- 172.21.15.27:0/3811102691 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f51a402a070 con 0x7f51b406b180 2024-05-10T11:17:42.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.384+0000 7f51b0c00700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f51980b2a70 0x7f51980b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.384+0000 7f51b0c00700 1 -- 172.21.15.27:0/3811102691 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f51a40e3660 con 0x7f51b406b180 2024-05-10T11:17:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.384+0000 7f51b3400700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f51980b2a70 0x7f51980b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.385+0000 7f51b3400700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f51980b2a70 0x7f51980b4f30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f51b4079a70 tx=0x7f51a0009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:42.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.388+0000 7f51b0c00700 1 -- 172.21.15.27:0/3811102691 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f51a4013070 con 0x7f51b406b180 2024-05-10T11:17:42.670 INFO:tasks.workunit.client.0.smithi027.stdout: 1 206121 7.78 MB/sec execute 27 sec latency 14.144 ms 2024-05-10T11:17:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.707+0000 7f51bae00700 1 -- 172.21.15.27:0/3811102691 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51b406a190 con 0x7f51b406b180 2024-05-10T11:17:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.708+0000 7f51b0c00700 1 -- 172.21.15.27:0/3811102691 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11936 (secure 0 0 0) 0x7f51a40aaca0 con 0x7f51b406b180 2024-05-10T11:17:42.708 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:17:42.709 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:17:36.552331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:17:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.711+0000 7f51bae00700 1 -- 172.21.15.27:0/3811102691 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f51980b2a70 msgr2=0x7f51980b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.711+0000 7f51bae00700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f51980b2a70 0x7f51980b4f30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f51b4079a70 tx=0x7f51a0009260 comp rx=0 tx=0).stop 2024-05-10T11:17:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.711+0000 7f51bae00700 1 -- 172.21.15.27:0/3811102691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b406b180 msgr2=0x7f51b419f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.711+0000 7f51bae00700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b406b180 0x7f51b419f8b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f51a4000c00 tx=0x7f51a40057c0 comp rx=0 tx=0).stop 2024-05-10T11:17:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.712+0000 7f51bae00700 1 -- 172.21.15.27:0/3811102691 shutdown_connections 2024-05-10T11:17:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.712+0000 7f51bae00700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f51980b2a70 0x7f51980b4f30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.712+0000 7f51bae00700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b406b180 0x7f51b419f8b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.712+0000 7f51bae00700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51b4109510 0x7f51b4199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.712+0000 7f51bae00700 1 --2- 172.21.15.27:0/3811102691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b406bb40 0x7f51b419fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.712+0000 7f51bae00700 1 -- 172.21.15.27:0/3811102691 >> 172.21.15.27:0/3811102691 conn(0x7f51b4079350 msgr2=0x7f51b4101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.712+0000 7f51bae00700 1 -- 172.21.15.27:0/3811102691 shutdown_connections 2024-05-10T11:17:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:42.712+0000 7f51bae00700 1 -- 172.21.15.27:0/3811102691 wait complete. 2024-05-10T11:17:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 33 2024-05-10T11:17:43.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:42 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3811102691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:17:43.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:42 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3811102691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:17:43.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:42 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3811102691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:17:43.669 INFO:tasks.workunit.client.0.smithi027.stdout: 1 207288 7.69 MB/sec execute 28 sec latency 46.150 ms 2024-05-10T11:17:43.744 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:17:43.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:43 smithi027 ceph-mon[30054]: pgmap v138: 129 pgs: 129 active+clean; 121 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 155 op/s 2024-05-10T11:17:44.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:43 smithi076 ceph-mon[34152]: pgmap v138: 129 pgs: 129 active+clean; 121 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 155 op/s 2024-05-10T11:17:44.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:43 smithi189 ceph-mon[34842]: pgmap v138: 129 pgs: 129 active+clean; 121 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 155 op/s 2024-05-10T11:17:44.241 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:17:44.669 INFO:tasks.workunit.client.0.smithi027.stdout: 1 209337 7.85 MB/sec execute 29 sec latency 91.320 ms 2024-05-10T11:17:45.556 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:45 smithi027 ceph-mon[30054]: pgmap v139: 129 pgs: 129 active+clean; 122 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 232 op/s 2024-05-10T11:17:45.669 INFO:tasks.workunit.client.0.smithi027.stdout: 1 209957 7.65 MB/sec execute 30 sec latency 33.193 ms 2024-05-10T11:17:45.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:45 smithi076 ceph-mon[34152]: pgmap v139: 129 pgs: 129 active+clean; 122 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 232 op/s 2024-05-10T11:17:45.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:45 smithi189 ceph-mon[34842]: pgmap v139: 129 pgs: 129 active+clean; 122 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 232 op/s 2024-05-10T11:17:46.669 INFO:tasks.workunit.client.0.smithi027.stdout: 1 211696 7.62 MB/sec execute 31 sec latency 69.781 ms 2024-05-10T11:17:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.935+0000 7fe9f9a00700 1 -- 172.21.15.27:0/3854841415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f40758d0 msgr2=0x7fe9f4075cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.935+0000 7fe9f9a00700 1 --2- 172.21.15.27:0/3854841415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f40758d0 0x7fe9f4075cb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009990 tx=0x7fe9e4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.935+0000 7fe9f9a00700 1 -- 172.21.15.27:0/3854841415 shutdown_connections 2024-05-10T11:17:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.935+0000 7fe9f9a00700 1 --2- 172.21.15.27:0/3854841415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9f410fc10 0x7fe9f4112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.935+0000 7fe9f9a00700 1 --2- 172.21.15.27:0/3854841415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f40758d0 0x7fe9f4075cb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.935+0000 7fe9f9a00700 1 --2- 172.21.15.27:0/3854841415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f40761f0 0x7fe9f410f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.935+0000 7fe9f9a00700 1 -- 172.21.15.27:0/3854841415 >> 172.21.15.27:0/3854841415 conn(0x7fe9f40fef20 msgr2=0x7fe9f4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.935+0000 7fe9f9a00700 1 -- 172.21.15.27:0/3854841415 shutdown_connections 2024-05-10T11:17:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.936+0000 7fe9f9a00700 1 -- 172.21.15.27:0/3854841415 wait complete. 2024-05-10T11:17:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.937+0000 7fe9f9a00700 1 Processor -- start 2024-05-10T11:17:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.937+0000 7fe9f9a00700 1 -- start start 2024-05-10T11:17:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.938+0000 7fe9f9a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f40758d0 0x7fe9f4108d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.938+0000 7fe9f9a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9f40761f0 0x7fe9f4109250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.938+0000 7fe9f9a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f410fc10 0x7fe9f4103e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.938+0000 7fe9f9a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f41157f0 con 0x7fe9f410fc10 2024-05-10T11:17:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.938+0000 7fe9f9a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f4115670 con 0x7fe9f40761f0 2024-05-10T11:17:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.938+0000 7fe9f9a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f4115970 con 0x7fe9f40758d0 2024-05-10T11:17:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.938+0000 7fe9f3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f40758d0 0x7fe9f4108d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.938+0000 7fe9f3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f40758d0 0x7fe9f4108d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50378/0 (socket says 172.21.15.27:50378) 2024-05-10T11:17:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.938+0000 7fe9f3e00700 1 -- 172.21.15.27:0/2940788823 learned_addr learned my addr 172.21.15.27:0/2940788823 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.938+0000 7fe9f9000700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f410fc10 0x7fe9f4103e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.938+0000 7fe9f3400700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9f40761f0 0x7fe9f4109250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.939+0000 7fe9f3e00700 1 -- 172.21.15.27:0/2940788823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9f40761f0 msgr2=0x7fe9f4109250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.939+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9f40761f0 0x7fe9f4109250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.939+0000 7fe9f3e00700 1 -- 172.21.15.27:0/2940788823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f410fc10 msgr2=0x7fe9f4103e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.939+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f410fc10 0x7fe9f4103e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.939+0000 7fe9f3e00700 1 -- 172.21.15.27:0/2940788823 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f4104730 con 0x7fe9f40758d0 2024-05-10T11:17:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.939+0000 7fe9f3400700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9f40761f0 0x7fe9f4109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:17:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.939+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f40758d0 0x7fe9f4108d10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4005910 tx=0x7fe9e400b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.939+0000 7fe9f0c00700 1 -- 172.21.15.27:0/2940788823 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4027070 con 0x7fe9f40758d0 2024-05-10T11:17:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.939+0000 7fe9f9a00700 1 -- 172.21.15.27:0/2940788823 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f41049c0 con 0x7fe9f40758d0 2024-05-10T11:17:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.939+0000 7fe9f0c00700 1 -- 172.21.15.27:0/2940788823 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9e4022470 con 0x7fe9f40758d0 2024-05-10T11:17:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.940+0000 7fe9f9a00700 1 -- 172.21.15.27:0/2940788823 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f410ec00 con 0x7fe9f40758d0 2024-05-10T11:17:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.940+0000 7fe9f0c00700 1 -- 172.21.15.27:0/2940788823 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4017650 con 0x7fe9f40758d0 2024-05-10T11:17:46.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.944+0000 7fe9f0c00700 1 -- 172.21.15.27:0/2940788823 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fe9e402a070 con 0x7fe9f40758d0 2024-05-10T11:17:46.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.945+0000 7fea059b3b80 1 Processor -- start 2024-05-10T11:17:46.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.945+0000 7fea059b3b80 1 -- start start 2024-05-10T11:17:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fea059b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a8cda0 0x5612a1a8d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fea059b3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8d6c0 0x5612a1a87430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fea059b3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 0x5612a1a87ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fea059b3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5612a19d9a20 con 0x5612a1a8d6c0 2024-05-10T11:17:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fea059b3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5612a1977760 con 0x5612a1a87a50 2024-05-10T11:17:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fea059b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5612a19d59e0 con 0x5612a1a8cda0 2024-05-10T11:17:46.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f0c00700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe9d80b2a70 0x7fe9d80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 0x5612a1a87ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f9000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a8cda0 0x5612a1a8d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 0x5612a1a87ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52536/0 (socket says 172.21.15.27:52536) 2024-05-10T11:17:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f3400700 1 -- 172.21.15.27:0/287381747 learned_addr learned my addr 172.21.15.27:0/287381747 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/287381747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8d6c0 0x5612a1a87430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f0c00700 1 -- 172.21.15.27:0/2940788823 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fe9e40e3710 con 0x7fe9f40758d0 2024-05-10T11:17:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f3400700 1 -- 172.21.15.27:0/287381747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a8cda0 msgr2=0x5612a1a8d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f3400700 1 --2- 172.21.15.27:0/287381747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a8cda0 0x5612a1a8d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f3400700 1 -- 172.21.15.27:0/287381747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8d6c0 msgr2=0x5612a1a87430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f3400700 1 --2- 172.21.15.27:0/287381747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8d6c0 0x5612a1a87430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f3400700 1 -- 172.21.15.27:0/287381747 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a19268d0 con 0x5612a1a87a50 2024-05-10T11:17:46.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.946+0000 7fe9f9000700 1 --2- 172.21.15.27:0/287381747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a8cda0 0x5612a1a8d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:17:46.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.947+0000 7fe9f9000700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe9d80b2a70 0x7fe9d80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.947+0000 7fe9f3400700 1 --2- 172.21.15.27:0/287381747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 0x5612a1a87ed0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009d20 tx=0x7fe9e00116c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=296760d95e71f924 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:46.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.947+0000 7fe9e9600700 1 -- 172.21.15.27:0/287381747 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0011e20 con 0x5612a1a87a50 2024-05-10T11:17:46.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.947+0000 7fe9e9600700 1 -- 172.21.15.27:0/287381747 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9e0012900 con 0x5612a1a87a50 2024-05-10T11:17:46.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.947+0000 7fe9e9600700 1 -- 172.21.15.27:0/287381747 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e000f840 con 0x5612a1a87a50 2024-05-10T11:17:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.948+0000 7fea059b3b80 1 -- 172.21.15.27:0/287381747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 msgr2=0x5612a1a87ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.948+0000 7fe9f9000700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe9d80b2a70 0x7fe9d80b4f30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009ec0 tx=0x7fe9ec009040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.948+0000 7fea059b3b80 1 --2- 172.21.15.27:0/287381747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 0x5612a1a87ed0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009d20 tx=0x7fe9e00116c0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.948+0000 7fea059b3b80 1 -- 172.21.15.27:0/287381747 shutdown_connections 2024-05-10T11:17:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.948+0000 7fea059b3b80 1 --2- 172.21.15.27:0/287381747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a8cda0 0x5612a1a8d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.948+0000 7fea059b3b80 1 --2- 172.21.15.27:0/287381747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 0x5612a1a87ed0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.948+0000 7fea059b3b80 1 --2- 172.21.15.27:0/287381747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8d6c0 0x5612a1a87430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.948+0000 7fea059b3b80 1 -- 172.21.15.27:0/287381747 >> 172.21.15.27:0/287381747 conn(0x5612a19ff6d0 msgr2=0x5612a1a81ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.948+0000 7fea059b3b80 1 -- 172.21.15.27:0/287381747 shutdown_connections 2024-05-10T11:17:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.948+0000 7fea059b3b80 1 -- 172.21.15.27:0/287381747 wait complete. 2024-05-10T11:17:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.949+0000 7fea059b3b80 1 -- start start 2024-05-10T11:17:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.949+0000 7fea059b3b80 1 Processor -- start 2024-05-10T11:17:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fea059b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 0x5612a1a93420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fea059b3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8cda0 0x5612a1a907c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fea059b3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8d6c0 0x5612a1a90d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fea059b3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5612a1a94150 con 0x5612a1a8cda0 2024-05-10T11:17:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fea059b3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5612a193fb20 con 0x5612a1a8d6c0 2024-05-10T11:17:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fea059b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5612a1a942d0 con 0x5612a1a87a50 2024-05-10T11:17:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fea059b3b80 10 client.0 _ll_register_callbacks cb 0x5612a1980c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:17:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fe9f9000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8d6c0 0x5612a1a90d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fe9f3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 0x5612a1a93420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fe9f3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8cda0 0x5612a1a907c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fe9f9000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8d6c0 0x5612a1a90d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52546/0 (socket says 172.21.15.27:52546) 2024-05-10T11:17:46.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fe9f3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 0x5612a1a93420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50402/0 (socket says 172.21.15.27:50402) 2024-05-10T11:17:46.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.950+0000 7fe9f9000700 1 -- 172.21.15.27:0/3772870886 learned_addr learned my addr 172.21.15.27:0/3772870886 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.951+0000 7fe9f9000700 1 -- 172.21.15.27:0/3772870886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 msgr2=0x5612a1a93420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.951+0000 7fe9f9000700 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 0x5612a1a93420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.951+0000 7fe9f9000700 1 -- 172.21.15.27:0/3772870886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8cda0 msgr2=0x5612a1a907c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.951+0000 7fe9f9000700 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8cda0 0x5612a1a907c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.951+0000 7fe9f9000700 1 -- 172.21.15.27:0/3772870886 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a199f420 con 0x5612a1a8d6c0 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.951+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 0x5612a1a93420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.951+0000 7fe9f3400700 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8cda0 0x5612a1a907c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.951+0000 7fe9f9000700 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8d6c0 0x5612a1a90d00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec017ed0 tx=0x7fe9ec004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=473af58dc2b3b310 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.952+0000 7fe9de000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.952+0000 7fe9de000700 1 -- 172.21.15.27:0/3772870886 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec016d40 con 0x5612a1a8d6c0 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.952+0000 7fea059b3b80 1 -- 172.21.15.27:0/3772870886 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9e0012900 con 0x5612a1a8d6c0 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.952+0000 7fe9de000700 1 -- 172.21.15.27:0/3772870886 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fe9ec022440 con 0x5612a1a8d6c0 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.952+0000 7fe9de000700 1 -- 172.21.15.27:0/3772870886 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe9ec030050 con 0x5612a1a8d6c0 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.952+0000 7fea059b3b80 10 client.25997 fetch_fsmap learned FSMap version 33 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.952+0000 7fea059b3b80 1 -- 172.21.15.27:0/3772870886 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5612a1a9ea90 con 0x5612a1a8d6c0 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.952+0000 7fe9de000700 1 -- 172.21.15.27:0/3772870886 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9e0012900 con 0x5612a1a8d6c0 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.953+0000 7fe9de000700 1 -- 172.21.15.27:0/3772870886 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fe9ec02b5e0 con 0x5612a1a8d6c0 2024-05-10T11:17:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.953+0000 7fea059b3b80 10 client.25997 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:17:46.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.953+0000 7fea059b3b80 10 client.25997 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:17:46.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.953+0000 7fea059b3b80 20 client.25997 populate_metadata read hostname 'smithi027' 2024-05-10T11:17:46.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.953+0000 7fea059b3b80 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5612a1aa1020 0x5612a1aa3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.953+0000 7fea059b3b80 4 client.25997 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:17:46.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.953+0000 7fea059b3b80 1 -- 172.21.15.27:0/3772870886 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612a1a54140 con 0x5612a1aa1020 2024-05-10T11:17:46.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.954+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5612a1aa1020 0x5612a1aa3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.954+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5612a1aa1020 0x5612a1aa3440 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7b55773062b69cd server_cookie=bc13649c4a19f8d9 in_seq=0 out_seq=0 2024-05-10T11:17:46.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.954+0000 7fe9de000700 10 client.25997 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:17:46.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.955+0000 7fe9de000700 1 -- 172.21.15.27:0/3772870886 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5612a1a54140 con 0x5612a1aa1020 2024-05-10T11:17:46.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.955+0000 7fe9de000700 10 client.25997 handle_command_reply: tid=0 2024-05-10T11:17:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.955+0000 7fea059b3b80 1 client.25997 shutdown 2024-05-10T11:17:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.956+0000 7fea059b3b80 1 -- 172.21.15.27:0/3772870886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8d6c0 msgr2=0x5612a1a90d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.956+0000 7fea059b3b80 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8d6c0 0x5612a1a90d00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec017ed0 tx=0x7fe9ec004ad0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.956+0000 7fea059b3b80 1 -- 172.21.15.27:0/3772870886 shutdown_connections 2024-05-10T11:17:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.956+0000 7fea059b3b80 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5612a1aa1020 0x5612a1aa3440 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.956+0000 7fea059b3b80 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 0x5612a1a93420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.956+0000 7fea059b3b80 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8d6c0 0x5612a1a90d00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.956+0000 7fea059b3b80 1 --2- 172.21.15.27:0/3772870886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8cda0 0x5612a1a907c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.956+0000 7fea059b3b80 1 -- 172.21.15.27:0/3772870886 >> 172.21.15.27:0/3772870886 conn(0x5612a19ff6d0 msgr2=0x5612a1a7c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.956+0000 7fe9de000700 0 client.25997 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:17:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.956+0000 7fea059b3b80 1 -- 172.21.15.27:0/3772870886 shutdown_connections 2024-05-10T11:17:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.956+0000 7fea059b3b80 1 -- 172.21.15.27:0/3772870886 wait complete. 2024-05-10T11:17:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.956+0000 7fea059b3b80 20 client.25997 trim_cache size 0 max 16384 2024-05-10T11:17:46.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.977+0000 7fea059b3b80 1 Processor -- start 2024-05-10T11:17:46.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.977+0000 7fea059b3b80 1 -- start start 2024-05-10T11:17:46.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.978+0000 7fea059b3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 0x5612a1a87430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.978+0000 7fea059b3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8cda0 0x5612a1a9d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.978+0000 7fea059b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a8d6c0 0x5612a1a7c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.978+0000 7fea059b3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f41be210 con 0x5612a1a8cda0 2024-05-10T11:17:46.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.978+0000 7fea059b3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f41bfd50 con 0x5612a1a87a50 2024-05-10T11:17:46.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.978+0000 7fea059b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f41be8e0 con 0x5612a1a8d6c0 2024-05-10T11:17:46.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.978+0000 7fe9f3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 0x5612a1a87430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.978+0000 7fe9f3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8cda0 0x5612a1a9d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.978+0000 7fe9f3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 0x5612a1a87430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52560/0 (socket says 172.21.15.27:52560) 2024-05-10T11:17:46.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.978+0000 7fe9f3e00700 1 -- 172.21.15.27:0/845827645 learned_addr learned my addr 172.21.15.27:0/845827645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:46.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.978+0000 7fe9f9000700 1 --2- 172.21.15.27:0/845827645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a8d6c0 0x5612a1a7c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.979+0000 7fe9f3e00700 1 -- 172.21.15.27:0/845827645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a8d6c0 msgr2=0x5612a1a7c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:46.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.979+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/845827645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a8d6c0 0x5612a1a7c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.979+0000 7fe9f3e00700 1 -- 172.21.15.27:0/845827645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8cda0 msgr2=0x5612a1a9d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:46.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.979+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/845827645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8cda0 0x5612a1a9d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.979+0000 7fe9f3e00700 1 -- 172.21.15.27:0/845827645 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a1a90590 con 0x5612a1a87a50 2024-05-10T11:17:46.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.979+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/845827645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 0x5612a1a87430 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40a66d0 tx=0x7fe9e4076f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=128cf5e27e924fa7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:46.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.979+0000 7fe9e8c00700 1 -- 172.21.15.27:0/845827645 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4078060 con 0x5612a1a87a50 2024-05-10T11:17:46.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.979+0000 7fe9e8c00700 1 -- 172.21.15.27:0/845827645 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9e40e2030 con 0x5612a1a87a50 2024-05-10T11:17:46.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.979+0000 7fe9e8c00700 1 -- 172.21.15.27:0/845827645 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e40aea00 con 0x5612a1a87a50 2024-05-10T11:17:46.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.980+0000 7fea059b3b80 1 -- 172.21.15.27:0/845827645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 msgr2=0x5612a1a87430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:46.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.980+0000 7fea059b3b80 1 --2- 172.21.15.27:0/845827645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 0x5612a1a87430 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40a66d0 tx=0x7fe9e4076f80 comp rx=0 tx=0).stop 2024-05-10T11:17:46.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.980+0000 7fea059b3b80 1 -- 172.21.15.27:0/845827645 shutdown_connections 2024-05-10T11:17:46.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.980+0000 7fea059b3b80 1 --2- 172.21.15.27:0/845827645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a8d6c0 0x5612a1a7c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.980+0000 7fea059b3b80 1 --2- 172.21.15.27:0/845827645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a87a50 0x5612a1a87430 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.980+0000 7fea059b3b80 1 --2- 172.21.15.27:0/845827645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8cda0 0x5612a1a9d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.980+0000 7fea059b3b80 1 -- 172.21.15.27:0/845827645 >> 172.21.15.27:0/845827645 conn(0x5612a19ff6d0 msgr2=0x5612a1a8f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:46.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.980+0000 7fea059b3b80 1 -- 172.21.15.27:0/845827645 shutdown_connections 2024-05-10T11:17:46.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.981+0000 7fea059b3b80 1 -- 172.21.15.27:0/845827645 wait complete. 2024-05-10T11:17:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.981+0000 7fea059b3b80 1 -- start start 2024-05-10T11:17:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.981+0000 7fea059b3b80 1 Processor -- start 2024-05-10T11:17:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.982+0000 7fea059b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 0x5612a1a99560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.982+0000 7fea059b3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8cda0 0x5612a1a99aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.982+0000 7fea059b3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8d6c0 0x5612a1aa7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.982+0000 7fea059b3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f41bf680 con 0x5612a1a8d6c0 2024-05-10T11:17:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.982+0000 7fea059b3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f41befb0 con 0x5612a1a8cda0 2024-05-10T11:17:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.982+0000 7fea059b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f41bc1b0 con 0x5612a1a87a50 2024-05-10T11:17:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.982+0000 7fea059b3b80 10 client.0 _ll_register_callbacks cb 0x5612a1980c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:17:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.982+0000 7fe9f3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8cda0 0x5612a1a99aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.982+0000 7fe9f3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 0x5612a1a99560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.982+0000 7fe9f3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 0x5612a1a99560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50432/0 (socket says 172.21.15.27:50432) 2024-05-10T11:17:46.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.983+0000 7fe9f3e00700 1 -- 172.21.15.27:0/952762429 learned_addr learned my addr 172.21.15.27:0/952762429 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:46.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.982+0000 7fe9f3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8cda0 0x5612a1a99aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52572/0 (socket says 172.21.15.27:52572) 2024-05-10T11:17:46.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.983+0000 7fe9f9000700 1 --2- 172.21.15.27:0/952762429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8d6c0 0x5612a1aa7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.983+0000 7fe9f3e00700 1 -- 172.21.15.27:0/952762429 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8cda0 msgr2=0x5612a1a99aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:46.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.983+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/952762429 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8cda0 0x5612a1a99aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.983+0000 7fe9f3e00700 1 -- 172.21.15.27:0/952762429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8d6c0 msgr2=0x5612a1aa7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.983+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/952762429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8d6c0 0x5612a1aa7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.983+0000 7fe9f3e00700 1 -- 172.21.15.27:0/952762429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a1aa8240 con 0x5612a1a87a50 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.983+0000 7fe9f3e00700 1 --2- 172.21.15.27:0/952762429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 0x5612a1a99560 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4000c00 tx=0x7fe9e4009710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62e53b6f63aea8c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.983+0000 7fe9de000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.983+0000 7fe9de000700 1 -- 172.21.15.27:0/952762429 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e40aa160 con 0x5612a1a87a50 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.983+0000 7fea059b3b80 1 -- 172.21.15.27:0/952762429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9e40e2030 con 0x5612a1a87a50 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.983+0000 7fe9de000700 1 -- 172.21.15.27:0/952762429 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fe9e40e0cb0 con 0x5612a1a87a50 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.984+0000 7fe9de000700 1 -- 172.21.15.27:0/952762429 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe9e40e0eb0 con 0x5612a1a87a50 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.984+0000 7fea059b3b80 10 client.25996 fetch_fsmap learned FSMap version 33 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.984+0000 7fea059b3b80 1 -- 172.21.15.27:0/952762429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5612a1b0b040 con 0x5612a1a87a50 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.984+0000 7fe9de000700 1 -- 172.21.15.27:0/952762429 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9e40e2030 con 0x5612a1a87a50 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.985+0000 7fe9de000700 1 -- 172.21.15.27:0/952762429 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fe9e40aa8b0 con 0x5612a1a87a50 2024-05-10T11:17:46.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.985+0000 7fea059b3b80 10 client.25996 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:17:46.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.985+0000 7fea059b3b80 10 client.25996 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:17:46.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.985+0000 7fea059b3b80 20 client.25996 populate_metadata read hostname 'smithi027' 2024-05-10T11:17:46.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.985+0000 7fea059b3b80 1 --2- 172.21.15.27:0/952762429 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5612a1b0b660 0x5612a1b0da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:46.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.985+0000 7fea059b3b80 4 client.25996 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:17:46.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.985+0000 7fea059b3b80 1 -- 172.21.15.27:0/952762429 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5612a19e7400 con 0x5612a1b0b660 2024-05-10T11:17:46.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.985+0000 7fe9f3400700 1 --2- 172.21.15.27:0/952762429 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5612a1b0b660 0x5612a1b0da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:46.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.986+0000 7fe9f3400700 1 --2- 172.21.15.27:0/952762429 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5612a1b0b660 0x5612a1b0da80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ce5ddc20aec9a710 server_cookie=e0e40f31b5b671a0 in_seq=0 out_seq=0 2024-05-10T11:17:46.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.986+0000 7fe9de000700 10 client.25996 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:17:46.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fe9de000700 1 -- 172.21.15.27:0/952762429 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5612a19e7400 con 0x5612a1b0b660 2024-05-10T11:17:46.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fe9de000700 10 client.25996 handle_command_reply: tid=0 2024-05-10T11:17:46.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fea059b3b80 1 client.25996 shutdown 2024-05-10T11:17:46.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fea059b3b80 1 -- 172.21.15.27:0/952762429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 msgr2=0x5612a1a99560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:46.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fea059b3b80 1 --2- 172.21.15.27:0/952762429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 0x5612a1a99560 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4000c00 tx=0x7fe9e4009710 comp rx=0 tx=0).stop 2024-05-10T11:17:46.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fea059b3b80 1 -- 172.21.15.27:0/952762429 shutdown_connections 2024-05-10T11:17:46.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fea059b3b80 1 --2- 172.21.15.27:0/952762429 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5612a1b0b660 0x5612a1b0da80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fea059b3b80 1 --2- 172.21.15.27:0/952762429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a1a87a50 0x5612a1a99560 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fea059b3b80 1 --2- 172.21.15.27:0/952762429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612a1a8d6c0 0x5612a1aa7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fea059b3b80 1 --2- 172.21.15.27:0/952762429 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5612a1a8cda0 0x5612a1a99aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fea059b3b80 1 -- 172.21.15.27:0/952762429 >> 172.21.15.27:0/952762429 conn(0x5612a19ff6d0 msgr2=0x5612a1a8eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:46.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fe9de000700 0 client.25996 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:17:46.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fea059b3b80 1 -- 172.21.15.27:0/952762429 shutdown_connections 2024-05-10T11:17:46.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.987+0000 7fea059b3b80 1 -- 172.21.15.27:0/952762429 wait complete. 2024-05-10T11:17:46.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.988+0000 7fea059b3b80 20 client.25996 trim_cache size 0 max 16384 2024-05-10T11:17:46.990 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:17:46.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.988+0000 7fe9f9a00700 1 -- 172.21.15.27:0/2940788823 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe9d80b2a70 msgr2=0x7fe9d80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:46.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.988+0000 7fe9f9a00700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe9d80b2a70 0x7fe9d80b4f30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009ec0 tx=0x7fe9ec009040 comp rx=0 tx=0).stop 2024-05-10T11:17:46.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.988+0000 7fe9f9a00700 1 -- 172.21.15.27:0/2940788823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f40758d0 msgr2=0x7fe9f4108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:46.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.988+0000 7fe9f9a00700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f40758d0 0x7fe9f4108d10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4005910 tx=0x7fe9e400b670 comp rx=0 tx=0).stop 2024-05-10T11:17:46.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.989+0000 7fe9f9a00700 1 -- 172.21.15.27:0/2940788823 shutdown_connections 2024-05-10T11:17:46.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.989+0000 7fe9f9a00700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe9d80b2a70 0x7fe9d80b4f30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.989+0000 7fe9f9a00700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f40758d0 0x7fe9f4108d10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.989+0000 7fe9f9a00700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f410fc10 0x7fe9f4103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.989+0000 7fe9f9a00700 1 --2- 172.21.15.27:0/2940788823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9f40761f0 0x7fe9f4109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:46.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.989+0000 7fe9f9a00700 1 -- 172.21.15.27:0/2940788823 >> 172.21.15.27:0/2940788823 conn(0x7fe9f40fef20 msgr2=0x7fe9f40ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:46.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.989+0000 7fe9f9a00700 1 -- 172.21.15.27:0/2940788823 shutdown_connections 2024-05-10T11:17:46.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:46.989+0000 7fe9f9a00700 1 -- 172.21.15.27:0/2940788823 wait complete. 2024-05-10T11:17:47.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:47 smithi027 ceph-mon[30054]: pgmap v140: 129 pgs: 129 active+clean; 122 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-05-10T11:17:47.669 INFO:tasks.workunit.client.0.smithi027.stdout: 1 213106 7.75 MB/sec execute 32 sec latency 24.465 ms 2024-05-10T11:17:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:47 smithi076 ceph-mon[34152]: pgmap v140: 129 pgs: 129 active+clean; 122 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-05-10T11:17:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:47 smithi189 ceph-mon[34842]: pgmap v140: 129 pgs: 129 active+clean; 122 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-05-10T11:17:47.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:17:48.368 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:17:48.669 INFO:tasks.workunit.client.0.smithi027.stdout: 1 214041 7.59 MB/sec execute 33 sec latency 29.197 ms 2024-05-10T11:17:49.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:49 smithi027 ceph-mon[30054]: pgmap v141: 129 pgs: 129 active+clean; 125 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-05-10T11:17:49.669 INFO:tasks.workunit.client.0.smithi027.stdout: 1 216418 7.82 MB/sec execute 34 sec latency 62.639 ms 2024-05-10T11:17:49.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:49 smithi076 ceph-mon[34152]: pgmap v141: 129 pgs: 129 active+clean; 125 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-05-10T11:17:49.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:49 smithi189 ceph-mon[34842]: pgmap v141: 129 pgs: 129 active+clean; 125 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-05-10T11:17:50.669 INFO:tasks.workunit.client.0.smithi027.stdout: 1 217111 7.65 MB/sec execute 35 sec latency 26.694 ms 2024-05-10T11:17:50.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.871+0000 7fd546400700 1 -- 172.21.15.27:0/4285950119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd540106080 msgr2=0x7fd540112590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:50.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.872+0000 7fd546400700 1 --2- 172.21.15.27:0/4285950119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd540106080 0x7fd540112590 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd538009f90 tx=0x7fd53800b330 comp rx=0 tx=0).stop 2024-05-10T11:17:50.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.872+0000 7fd546400700 1 -- 172.21.15.27:0/4285950119 shutdown_connections 2024-05-10T11:17:50.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.872+0000 7fd546400700 1 --2- 172.21.15.27:0/4285950119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd540106080 0x7fd540112590 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.872+0000 7fd546400700 1 --2- 172.21.15.27:0/4285950119 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd54010b6c0 0x7fd54010baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.872+0000 7fd546400700 1 --2- 172.21.15.27:0/4285950119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5401056c0 0x7fd540105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.872+0000 7fd546400700 1 -- 172.21.15.27:0/4285950119 >> 172.21.15.27:0/4285950119 conn(0x7fd5401011b0 msgr2=0x7fd5401035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:50.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.872+0000 7fd546400700 1 -- 172.21.15.27:0/4285950119 shutdown_connections 2024-05-10T11:17:50.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.872+0000 7fd546400700 1 -- 172.21.15.27:0/4285950119 wait complete. 2024-05-10T11:17:50.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.873+0000 7fd546400700 1 Processor -- start 2024-05-10T11:17:50.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.873+0000 7fd546400700 1 -- start start 2024-05-10T11:17:50.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd546400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5401056c0 0x7fd54019bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd546400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd540106080 0x7fd54019c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd546400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd54010b6c0 0x7fd5401a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd546400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd540115900 con 0x7fd54010b6c0 2024-05-10T11:17:50.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd546400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd540115780 con 0x7fd540106080 2024-05-10T11:17:50.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd546400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd540115a80 con 0x7fd5401056c0 2024-05-10T11:17:50.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd536a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd540106080 0x7fd54019c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd53f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5401056c0 0x7fd54019bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd536a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd540106080 0x7fd54019c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52602/0 (socket says 172.21.15.27:52602) 2024-05-10T11:17:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd536a00700 1 -- 172.21.15.27:0/3246076449 learned_addr learned my addr 172.21.15.27:0/3246076449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd53fe00700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd54010b6c0 0x7fd5401a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd536a00700 1 -- 172.21.15.27:0/3246076449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5401056c0 msgr2=0x7fd54019bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd536a00700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5401056c0 0x7fd54019bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd536a00700 1 -- 172.21.15.27:0/3246076449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd54010b6c0 msgr2=0x7fd5401a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd536a00700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd54010b6c0 0x7fd5401a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.874+0000 7fd536a00700 1 -- 172.21.15.27:0/3246076449 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5401a0dc0 con 0x7fd540106080 2024-05-10T11:17:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.875+0000 7fd536a00700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd540106080 0x7fd54019c2f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd52800e9a0 tx=0x7fd52800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.875+0000 7fd53cc00700 1 -- 172.21.15.27:0/3246076449 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52800c9c0 con 0x7fd540106080 2024-05-10T11:17:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.875+0000 7fd53cc00700 1 -- 172.21.15.27:0/3246076449 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd52800cb60 con 0x7fd540106080 2024-05-10T11:17:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.875+0000 7fd546400700 1 -- 172.21.15.27:0/3246076449 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5401a10b0 con 0x7fd540106080 2024-05-10T11:17:50.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.875+0000 7fd53cc00700 1 -- 172.21.15.27:0/3246076449 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528018680 con 0x7fd540106080 2024-05-10T11:17:50.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.875+0000 7fd546400700 1 -- 172.21.15.27:0/3246076449 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5401a8930 con 0x7fd540106080 2024-05-10T11:17:50.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.879+0000 7fd53cc00700 1 -- 172.21.15.27:0/3246076449 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd528010430 con 0x7fd540106080 2024-05-10T11:17:50.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.881+0000 7fd550c93b80 1 Processor -- start 2024-05-10T11:17:50.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.881+0000 7fd550c93b80 1 -- start start 2024-05-10T11:17:50.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.881+0000 7fd550c93b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b41fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.881+0000 7fd550c93b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b424e0 0x55d6d9b3c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.881+0000 7fd550c93b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b3cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.882+0000 7fd550c93b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d6d9a8ea20 con 0x55d6d9b424e0 2024-05-10T11:17:50.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.882+0000 7fd550c93b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d6d9a2c760 con 0x55d6d9b3c8f0 2024-05-10T11:17:50.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.882+0000 7fd550c93b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d6d9a8a9e0 con 0x55d6d9b41bc0 2024-05-10T11:17:50.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.882+0000 7fd53cc00700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd5240b2a70 0x7fd5240b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.882+0000 7fd53fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b41fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.882+0000 7fd53f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b424e0 0x55d6d9b3c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.882+0000 7fd53fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b41fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50446/0 (socket says 172.21.15.27:50446) 2024-05-10T11:17:50.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.882+0000 7fd53fe00700 1 -- 172.21.15.27:0/2710905579 learned_addr learned my addr 172.21.15.27:0/2710905579 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:50.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.882+0000 7fd53cc00700 1 -- 172.21.15.27:0/3246076449 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fd5280e2c40 con 0x7fd540106080 2024-05-10T11:17:50.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.882+0000 7fd53fe00700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd5240b2a70 0x7fd5240b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.882+0000 7fd536a00700 1 --2- 172.21.15.27:0/2710905579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b3cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.883+0000 7fd53f400700 1 -- 172.21.15.27:0/2710905579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 msgr2=0x55d6d9b41fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:50.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.883+0000 7fd53f400700 1 --2- 172.21.15.27:0/2710905579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b41fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.883+0000 7fd53f400700 1 -- 172.21.15.27:0/2710905579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b3c8f0 msgr2=0x55d6d9b3cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:50.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.883+0000 7fd53f400700 1 --2- 172.21.15.27:0/2710905579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b3cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.883+0000 7fd53f400700 1 -- 172.21.15.27:0/2710905579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6d99db8d0 con 0x55d6d9b424e0 2024-05-10T11:17:50.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.883+0000 7fd536a00700 1 --2- 172.21.15.27:0/2710905579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b3cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:17:50.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.884+0000 7fd53f400700 1 --2- 172.21.15.27:0/2710905579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b424e0 0x55d6d9b3c2d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd530009cc0 tx=0x7fd5300058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93bb7eac260704cc server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:50.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.884+0000 7fd53fe00700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd5240b2a70 0x7fd5240b4f30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5380060b0 tx=0x7fd538009c20 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:50.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.884+0000 7fd534c00700 1 -- 172.21.15.27:0/2710905579 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd530011ce0 con 0x55d6d9b424e0 2024-05-10T11:17:50.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.884+0000 7fd534c00700 1 -- 172.21.15.27:0/2710905579 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd530004500 con 0x55d6d9b424e0 2024-05-10T11:17:50.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.884+0000 7fd534c00700 1 -- 172.21.15.27:0/2710905579 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd530005280 con 0x55d6d9b424e0 2024-05-10T11:17:50.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.885+0000 7fd550c93b80 1 -- 172.21.15.27:0/2710905579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b424e0 msgr2=0x55d6d9b3c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:50.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.885+0000 7fd550c93b80 1 --2- 172.21.15.27:0/2710905579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b424e0 0x55d6d9b3c2d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd530009cc0 tx=0x7fd5300058e0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.885+0000 7fd550c93b80 1 -- 172.21.15.27:0/2710905579 shutdown_connections 2024-05-10T11:17:50.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.885+0000 7fd550c93b80 1 --2- 172.21.15.27:0/2710905579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b41fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.885+0000 7fd550c93b80 1 --2- 172.21.15.27:0/2710905579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b3cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.885+0000 7fd550c93b80 1 --2- 172.21.15.27:0/2710905579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b424e0 0x55d6d9b3c2d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.885+0000 7fd550c93b80 1 -- 172.21.15.27:0/2710905579 >> 172.21.15.27:0/2710905579 conn(0x55d6d9ab46d0 msgr2=0x55d6d9b36b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:50.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.886+0000 7fd550c93b80 1 -- 172.21.15.27:0/2710905579 shutdown_connections 2024-05-10T11:17:50.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.886+0000 7fd550c93b80 1 -- 172.21.15.27:0/2710905579 wait complete. 2024-05-10T11:17:50.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.886+0000 7fd550c93b80 1 -- start start 2024-05-10T11:17:50.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.887+0000 7fd550c93b80 1 Processor -- start 2024-05-10T11:17:50.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.887+0000 7fd550c93b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b481e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.887+0000 7fd550c93b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b455d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.887+0000 7fd550c93b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b424e0 0x55d6d9b45b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.887+0000 7fd550c93b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d6d9b48f20 con 0x55d6d9b424e0 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.887+0000 7fd550c93b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d6d99f4b20 con 0x55d6d9b3c8f0 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.887+0000 7fd550c93b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d6d9b490a0 con 0x55d6d9b41bc0 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.887+0000 7fd550c93b80 10 client.0 _ll_register_callbacks cb 0x55d6d9a35c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.887+0000 7fd536a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b455d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.887+0000 7fd53f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.887+0000 7fd53fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b424e0 0x55d6d9b45b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.888+0000 7fd53f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b481e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52620/0 (socket says 172.21.15.27:52620) 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.888+0000 7fd53f400700 1 -- 172.21.15.27:0/271717394 learned_addr learned my addr 172.21.15.27:0/271717394 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.888+0000 7fd536a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b455d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50450/0 (socket says 172.21.15.27:50450) 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.888+0000 7fd536a00700 1 -- 172.21.15.27:0/271717394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b3c8f0 msgr2=0x55d6d9b481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.888+0000 7fd536a00700 1 --2- 172.21.15.27:0/271717394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b481e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.888+0000 7fd536a00700 1 -- 172.21.15.27:0/271717394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b424e0 msgr2=0x55d6d9b45b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.888+0000 7fd536a00700 1 --2- 172.21.15.27:0/271717394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b424e0 0x55d6d9b45b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.888+0000 7fd536a00700 1 -- 172.21.15.27:0/271717394 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6d9a54420 con 0x55d6d9b41bc0 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.888+0000 7fd536a00700 1 --2- 172.21.15.27:0/271717394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b455d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd5280dff60 tx=0x7fd5280de220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b32dccd23b0fb155 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.888+0000 7fd52d600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.889+0000 7fd52d600700 1 -- 172.21.15.27:0/271717394 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5280a9600 con 0x55d6d9b41bc0 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.889+0000 7fd52d600700 1 -- 172.21.15.27:0/271717394 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fd5280a9f60 con 0x55d6d9b41bc0 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.889+0000 7fd550c93b80 1 -- 172.21.15.27:0/271717394 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd530004500 con 0x55d6d9b41bc0 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.889+0000 7fd52d600700 1 -- 172.21.15.27:0/271717394 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd5280aa1f0 con 0x55d6d9b41bc0 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.889+0000 7fd550c93b80 10 client.26008 fetch_fsmap learned FSMap version 33 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.889+0000 7fd550c93b80 1 -- 172.21.15.27:0/271717394 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d6d9b537a0 con 0x55d6d9b41bc0 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.890+0000 7fd52d600700 1 -- 172.21.15.27:0/271717394 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd530004500 con 0x55d6d9b41bc0 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.890+0000 7fd52d600700 1 -- 172.21.15.27:0/271717394 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fd5280cc0f0 con 0x55d6d9b41bc0 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.891+0000 7fd550c93b80 10 client.26008 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.891+0000 7fd550c93b80 10 client.26008 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.891+0000 7fd550c93b80 20 client.26008 populate_metadata read hostname 'smithi027' 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.891+0000 7fd550c93b80 1 --2- 172.21.15.27:0/271717394 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d6d9b55d30 0x55d6d9b58120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.891+0000 7fd550c93b80 4 client.26008 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:17:50.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.891+0000 7fd550c93b80 1 -- 172.21.15.27:0/271717394 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6d9b09140 con 0x55d6d9b55d30 2024-05-10T11:17:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.891+0000 7fd53f400700 1 --2- 172.21.15.27:0/271717394 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d6d9b55d30 0x55d6d9b58120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.892+0000 7fd53f400700 1 --2- 172.21.15.27:0/271717394 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d6d9b55d30 0x55d6d9b58120 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=17c3afb8017c4d37 server_cookie=2facda3bbba43be6 in_seq=0 out_seq=0 2024-05-10T11:17:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.892+0000 7fd52d600700 10 client.26008 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:17:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.892+0000 7fd52d600700 1 -- 172.21.15.27:0/271717394 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d6d9b09140 con 0x55d6d9b55d30 2024-05-10T11:17:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.892+0000 7fd52d600700 10 client.26008 handle_command_reply: tid=0 2024-05-10T11:17:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.893+0000 7fd550c93b80 1 client.26008 shutdown 2024-05-10T11:17:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.893+0000 7fd550c93b80 1 -- 172.21.15.27:0/271717394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 msgr2=0x55d6d9b455d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.893+0000 7fd550c93b80 1 --2- 172.21.15.27:0/271717394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b455d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd5280dff60 tx=0x7fd5280de220 comp rx=0 tx=0).stop 2024-05-10T11:17:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.893+0000 7fd550c93b80 1 -- 172.21.15.27:0/271717394 shutdown_connections 2024-05-10T11:17:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.893+0000 7fd550c93b80 1 --2- 172.21.15.27:0/271717394 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d6d9b55d30 0x55d6d9b58120 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.893+0000 7fd550c93b80 1 --2- 172.21.15.27:0/271717394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b424e0 0x55d6d9b45b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.893+0000 7fd550c93b80 1 --2- 172.21.15.27:0/271717394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b481e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.893+0000 7fd550c93b80 1 --2- 172.21.15.27:0/271717394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b455d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.893+0000 7fd52d600700 0 client.26008 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:17:50.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.893+0000 7fd550c93b80 1 -- 172.21.15.27:0/271717394 >> 172.21.15.27:0/271717394 conn(0x55d6d9ab46d0 msgr2=0x55d6d9b36190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:50.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.893+0000 7fd550c93b80 1 -- 172.21.15.27:0/271717394 shutdown_connections 2024-05-10T11:17:50.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.893+0000 7fd550c93b80 1 -- 172.21.15.27:0/271717394 wait complete. 2024-05-10T11:17:50.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.894+0000 7fd550c93b80 20 client.26008 trim_cache size 0 max 16384 2024-05-10T11:17:50.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.911+0000 7fd550c93b80 1 Processor -- start 2024-05-10T11:17:50.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.911+0000 7fd550c93b80 1 -- start start 2024-05-10T11:17:50.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.912+0000 7fd550c93b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b3c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.912+0000 7fd550c93b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b44530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.912+0000 7fd550c93b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b424e0 0x55d6d9b51fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.912+0000 7fd550c93b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401b1850 con 0x55d6d9b3c8f0 2024-05-10T11:17:50.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.912+0000 7fd550c93b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401b3390 con 0x55d6d9b41bc0 2024-05-10T11:17:50.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.912+0000 7fd550c93b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401b1f20 con 0x55d6d9b424e0 2024-05-10T11:17:50.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.912+0000 7fd53f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b3c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.912+0000 7fd536a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b44530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.912+0000 7fd53fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b424e0 0x55d6d9b51fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.912+0000 7fd536a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b44530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52624/0 (socket says 172.21.15.27:52624) 2024-05-10T11:17:50.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.912+0000 7fd53f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b3c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50616/0 (socket says 172.21.15.27:50616) 2024-05-10T11:17:50.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.912+0000 7fd536a00700 1 -- 172.21.15.27:0/347055916 learned_addr learned my addr 172.21.15.27:0/347055916 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:50.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.913+0000 7fd53fe00700 1 -- 172.21.15.27:0/347055916 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b41bc0 msgr2=0x55d6d9b44530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:50.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.913+0000 7fd53fe00700 1 --2- 172.21.15.27:0/347055916 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b44530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.913+0000 7fd53fe00700 1 -- 172.21.15.27:0/347055916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b3c8f0 msgr2=0x55d6d9b3c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:50.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.913+0000 7fd53fe00700 1 --2- 172.21.15.27:0/347055916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b3c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.913+0000 7fd53fe00700 1 -- 172.21.15.27:0/347055916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6d9b48550 con 0x55d6d9b424e0 2024-05-10T11:17:50.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.913+0000 7fd536a00700 1 --2- 172.21.15.27:0/347055916 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b44530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:17:50.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.913+0000 7fd53f400700 1 --2- 172.21.15.27:0/347055916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b3c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:17:50.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.913+0000 7fd53fe00700 1 --2- 172.21.15.27:0/347055916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b424e0 0x55d6d9b51fb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd5380090a0 tx=0x7fd53801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=126319c706d69d09 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:50.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.913+0000 7fd52fe00700 1 -- 172.21.15.27:0/347055916 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd538003de0 con 0x55d6d9b424e0 2024-05-10T11:17:50.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.913+0000 7fd52fe00700 1 -- 172.21.15.27:0/347055916 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd538031070 con 0x55d6d9b424e0 2024-05-10T11:17:50.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.913+0000 7fd52fe00700 1 -- 172.21.15.27:0/347055916 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd538007680 con 0x55d6d9b424e0 2024-05-10T11:17:50.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.914+0000 7fd550c93b80 1 -- 172.21.15.27:0/347055916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b424e0 msgr2=0x55d6d9b51fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:50.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.914+0000 7fd550c93b80 1 --2- 172.21.15.27:0/347055916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b424e0 0x55d6d9b51fb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd5380090a0 tx=0x7fd53801c000 comp rx=0 tx=0).stop 2024-05-10T11:17:50.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.914+0000 7fd550c93b80 1 -- 172.21.15.27:0/347055916 shutdown_connections 2024-05-10T11:17:50.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.914+0000 7fd550c93b80 1 --2- 172.21.15.27:0/347055916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b424e0 0x55d6d9b51fb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.914+0000 7fd550c93b80 1 --2- 172.21.15.27:0/347055916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b3c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.914+0000 7fd550c93b80 1 --2- 172.21.15.27:0/347055916 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b44530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.914+0000 7fd550c93b80 1 -- 172.21.15.27:0/347055916 >> 172.21.15.27:0/347055916 conn(0x55d6d9ab46d0 msgr2=0x55d6d9b40700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:50.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.914+0000 7fd550c93b80 1 -- 172.21.15.27:0/347055916 shutdown_connections 2024-05-10T11:17:50.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.915+0000 7fd550c93b80 1 -- 172.21.15.27:0/347055916 wait complete. 2024-05-10T11:17:50.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.915+0000 7fd550c93b80 1 -- start start 2024-05-10T11:17:50.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.915+0000 7fd550c93b80 1 Processor -- start 2024-05-10T11:17:50.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd550c93b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b4e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd550c93b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b5cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd550c93b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b424e0 0x55d6d9b5d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd550c93b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401b2cc0 con 0x55d6d9b41bc0 2024-05-10T11:17:50.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd550c93b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401b25f0 con 0x55d6d9b424e0 2024-05-10T11:17:50.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd550c93b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401af7f0 con 0x55d6d9b3c8f0 2024-05-10T11:17:50.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd550c93b80 10 client.0 _ll_register_callbacks cb 0x55d6d9a35c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:17:50.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd53f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b4e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd53f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b4e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50454/0 (socket says 172.21.15.27:50454) 2024-05-10T11:17:50.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd53f400700 1 -- 172.21.15.27:0/2721285889 learned_addr learned my addr 172.21.15.27:0/2721285889 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:50.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd536a00700 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b5cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd53fe00700 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b424e0 0x55d6d9b5d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.916+0000 7fd53f400700 1 -- 172.21.15.27:0/2721285889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b424e0 msgr2=0x55d6d9b5d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:50.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.917+0000 7fd53f400700 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b424e0 0x55d6d9b5d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.917+0000 7fd53f400700 1 -- 172.21.15.27:0/2721285889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b41bc0 msgr2=0x55d6d9b5cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:50.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.917+0000 7fd53f400700 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b5cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.917+0000 7fd53f400700 1 -- 172.21.15.27:0/2721285889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6d9b5d970 con 0x55d6d9b3c8f0 2024-05-10T11:17:50.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.917+0000 7fd53f400700 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b4e730 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd530006010 tx=0x7fd530005da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcbffd17ce064655 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:50.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.917+0000 7fd52d600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:17:50.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.917+0000 7fd52d600700 1 -- 172.21.15.27:0/2721285889 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5300041e0 con 0x55d6d9b3c8f0 2024-05-10T11:17:50.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.917+0000 7fd52d600700 1 -- 172.21.15.27:0/2721285889 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fd53002a050 con 0x55d6d9b3c8f0 2024-05-10T11:17:50.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.917+0000 7fd550c93b80 1 -- 172.21.15.27:0/2721285889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd538031070 con 0x55d6d9b3c8f0 2024-05-10T11:17:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.918+0000 7fd52d600700 1 -- 172.21.15.27:0/2721285889 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd530039050 con 0x55d6d9b3c8f0 2024-05-10T11:17:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.918+0000 7fd550c93b80 10 client.26014 fetch_fsmap learned FSMap version 33 2024-05-10T11:17:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.918+0000 7fd550c93b80 1 -- 172.21.15.27:0/2721285889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d6d9bc8c20 con 0x55d6d9b3c8f0 2024-05-10T11:17:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.918+0000 7fd52d600700 1 -- 172.21.15.27:0/2721285889 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd538031070 con 0x55d6d9b3c8f0 2024-05-10T11:17:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.918+0000 7fd52d600700 1 -- 172.21.15.27:0/2721285889 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fd530018070 con 0x55d6d9b3c8f0 2024-05-10T11:17:50.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.918+0000 7fd550c93b80 10 client.26014 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:17:50.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.918+0000 7fd550c93b80 10 client.26014 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:17:50.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.918+0000 7fd550c93b80 20 client.26014 populate_metadata read hostname 'smithi027' 2024-05-10T11:17:50.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.918+0000 7fd550c93b80 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d6d9bc91f0 0x55d6d9bcb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:50.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.918+0000 7fd550c93b80 4 client.26014 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:17:50.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.918+0000 7fd550c93b80 1 -- 172.21.15.27:0/2721285889 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d6d9a9c400 con 0x55d6d9bc91f0 2024-05-10T11:17:50.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.919+0000 7fd536a00700 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d6d9bc91f0 0x55d6d9bcb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:50.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.919+0000 7fd536a00700 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d6d9bc91f0 0x55d6d9bcb610 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c5eaae59cd118ddf server_cookie=216b307602c909b0 in_seq=0 out_seq=0 2024-05-10T11:17:50.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.919+0000 7fd52d600700 10 client.26014 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:17:50.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.919+0000 7fd52d600700 1 -- 172.21.15.27:0/2721285889 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d6d9a9c400 con 0x55d6d9bc91f0 2024-05-10T11:17:50.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.919+0000 7fd52d600700 10 client.26014 handle_command_reply: tid=0 2024-05-10T11:17:50.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.919+0000 7fd550c93b80 1 client.26014 shutdown 2024-05-10T11:17:50.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.920+0000 7fd550c93b80 1 -- 172.21.15.27:0/2721285889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b3c8f0 msgr2=0x55d6d9b4e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:50.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.920+0000 7fd550c93b80 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b4e730 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd530006010 tx=0x7fd530005da0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.920+0000 7fd550c93b80 1 -- 172.21.15.27:0/2721285889 shutdown_connections 2024-05-10T11:17:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.920+0000 7fd550c93b80 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55d6d9bc91f0 0x55d6d9bcb610 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.920+0000 7fd550c93b80 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d6d9b3c8f0 0x55d6d9b4e730 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.920+0000 7fd550c93b80 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d6d9b424e0 0x55d6d9b5d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.920+0000 7fd550c93b80 1 --2- 172.21.15.27:0/2721285889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6d9b41bc0 0x55d6d9b5cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.920+0000 7fd52d600700 0 client.26014 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:17:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.920+0000 7fd550c93b80 1 -- 172.21.15.27:0/2721285889 >> 172.21.15.27:0/2721285889 conn(0x55d6d9ab46d0 msgr2=0x55d6d9b46b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.920+0000 7fd550c93b80 1 -- 172.21.15.27:0/2721285889 shutdown_connections 2024-05-10T11:17:50.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.920+0000 7fd550c93b80 1 -- 172.21.15.27:0/2721285889 wait complete. 2024-05-10T11:17:50.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.920+0000 7fd550c93b80 20 client.26014 trim_cache size 0 max 16384 2024-05-10T11:17:50.928 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:17:50.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.921+0000 7fd546400700 1 -- 172.21.15.27:0/3246076449 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd5240b2a70 msgr2=0x7fd5240b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:50.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.921+0000 7fd546400700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd5240b2a70 0x7fd5240b4f30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5380060b0 tx=0x7fd538009c20 comp rx=0 tx=0).stop 2024-05-10T11:17:50.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.921+0000 7fd546400700 1 -- 172.21.15.27:0/3246076449 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd540106080 msgr2=0x7fd54019c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:50.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.921+0000 7fd546400700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd540106080 0x7fd54019c2f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd52800e9a0 tx=0x7fd52800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.921+0000 7fd546400700 1 -- 172.21.15.27:0/3246076449 shutdown_connections 2024-05-10T11:17:50.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.921+0000 7fd546400700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd5240b2a70 0x7fd5240b4f30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.921+0000 7fd546400700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5401056c0 0x7fd54019bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.921+0000 7fd546400700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd54010b6c0 0x7fd5401a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.921+0000 7fd546400700 1 --2- 172.21.15.27:0/3246076449 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd540106080 0x7fd54019c2f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:50.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.921+0000 7fd546400700 1 -- 172.21.15.27:0/3246076449 >> 172.21.15.27:0/3246076449 conn(0x7fd5401011b0 msgr2=0x7fd5401016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.921+0000 7fd546400700 1 -- 172.21.15.27:0/3246076449 shutdown_connections 2024-05-10T11:17:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:50.921+0000 7fd546400700 1 -- 172.21.15.27:0/3246076449 wait complete. 2024-05-10T11:17:51.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:51 smithi027 ceph-mon[30054]: pgmap v142: 129 pgs: 129 active+clean; 125 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-05-10T11:17:51.669 INFO:tasks.workunit.client.0.smithi027.stdout: 1 219596 7.80 MB/sec execute 36 sec latency 41.130 ms 2024-05-10T11:17:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:51 smithi076 ceph-mon[34152]: pgmap v142: 129 pgs: 129 active+clean; 125 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-05-10T11:17:51.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:51 smithi189 ceph-mon[34842]: pgmap v142: 129 pgs: 129 active+clean; 125 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-05-10T11:17:51.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:17:52.495 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:17:52.670 INFO:tasks.workunit.client.0.smithi027.stdout: 1 220478 7.75 MB/sec execute 37 sec latency 68.312 ms 2024-05-10T11:17:53.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:53 smithi027 ceph-mon[30054]: pgmap v143: 129 pgs: 129 active+clean; 125 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-05-10T11:17:53.670 INFO:tasks.workunit.client.0.smithi027.stdout: 1 222581 7.76 MB/sec execute 38 sec latency 26.436 ms 2024-05-10T11:17:53.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:53 smithi076 ceph-mon[34152]: pgmap v143: 129 pgs: 129 active+clean; 125 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-05-10T11:17:53.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:53 smithi189 ceph-mon[34842]: pgmap v143: 129 pgs: 129 active+clean; 125 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-05-10T11:17:54.670 INFO:tasks.workunit.client.0.smithi027.stdout: 1 223893 7.84 MB/sec execute 39 sec latency 53.242 ms 2024-05-10T11:17:54.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.977+0000 7f4db7200700 1 -- 172.21.15.27:0/621833713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4db0103620 msgr2=0x7f4db0103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:54.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.977+0000 7f4db7200700 1 --2- 172.21.15.27:0/621833713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4db0103620 0x7f4db0103a00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4da4009990 tx=0x7f4da4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.977+0000 7f4db7200700 1 -- 172.21.15.27:0/621833713 shutdown_connections 2024-05-10T11:17:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.977+0000 7f4db7200700 1 --2- 172.21.15.27:0/621833713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db0108700 0x7f4db01102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.977+0000 7f4db7200700 1 --2- 172.21.15.27:0/621833713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4db0103620 0x7f4db0103a00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.977+0000 7f4db7200700 1 --2- 172.21.15.27:0/621833713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4db0103fd0 0x7f4db0108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.977+0000 7f4db7200700 1 -- 172.21.15.27:0/621833713 >> 172.21.15.27:0/621833713 conn(0x7f4db00feef0 msgr2=0x7f4db0101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.977+0000 7f4db7200700 1 -- 172.21.15.27:0/621833713 shutdown_connections 2024-05-10T11:17:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.977+0000 7f4db7200700 1 -- 172.21.15.27:0/621833713 wait complete. 2024-05-10T11:17:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.979+0000 7f4db7200700 1 Processor -- start 2024-05-10T11:17:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.979+0000 7f4db7200700 1 -- start start 2024-05-10T11:17:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.979+0000 7f4db7200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4db0103620 0x7f4db019bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.979+0000 7f4db7200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db0103fd0 0x7f4db019c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.979+0000 7f4db7200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4db0108700 0x7f4db01a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.979+0000 7f4db7200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4db0113600 con 0x7f4db0103fd0 2024-05-10T11:17:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.979+0000 7f4db7200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4db0113480 con 0x7f4db0108700 2024-05-10T11:17:54.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.979+0000 7f4db7200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4db0113780 con 0x7f4db0103620 2024-05-10T11:17:54.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.980+0000 7f4db5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4db0103620 0x7f4db019bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:54.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.980+0000 7f4db5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4db0103620 0x7f4db019bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37338/0 (socket says 172.21.15.27:37338) 2024-05-10T11:17:54.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.980+0000 7f4db5e00700 1 -- 172.21.15.27:0/4179046941 learned_addr learned my addr 172.21.15.27:0/4179046941 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:54.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.980+0000 7f4db5400700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db0103fd0 0x7f4db019c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:54.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.980+0000 7f4db6800700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4db0108700 0x7f4db01a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:54.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.980+0000 7f4db6800700 1 -- 172.21.15.27:0/4179046941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4db0103620 msgr2=0x7f4db019bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:54.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.980+0000 7f4db6800700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4db0103620 0x7f4db019bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.980+0000 7f4db6800700 1 -- 172.21.15.27:0/4179046941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db0103fd0 msgr2=0x7f4db019c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:54.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.980+0000 7f4db6800700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db0103fd0 0x7f4db019c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.980+0000 7f4db6800700 1 -- 172.21.15.27:0/4179046941 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db01a0d20 con 0x7f4db0108700 2024-05-10T11:17:54.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.980+0000 7f4db5400700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db0103fd0 0x7f4db019c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:17:54.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.981+0000 7f4db6800700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4db0108700 0x7f4db01a05e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00efc0 tx=0x7f4dac00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.981+0000 7f4daaa00700 1 -- 172.21.15.27:0/4179046941 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac0092e0 con 0x7f4db0108700 2024-05-10T11:17:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.981+0000 7f4db7200700 1 -- 172.21.15.27:0/4179046941 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4db01a0fb0 con 0x7f4db0108700 2024-05-10T11:17:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.981+0000 7f4daaa00700 1 -- 172.21.15.27:0/4179046941 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4dac004bb0 con 0x7f4db0108700 2024-05-10T11:17:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.981+0000 7f4daaa00700 1 -- 172.21.15.27:0/4179046941 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac004020 con 0x7f4db0108700 2024-05-10T11:17:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.981+0000 7f4db7200700 1 -- 172.21.15.27:0/4179046941 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4db01a8890 con 0x7f4db0108700 2024-05-10T11:17:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.985+0000 7f4daaa00700 1 -- 172.21.15.27:0/4179046941 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f4dac01b030 con 0x7f4db0108700 2024-05-10T11:17:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.986+0000 7f4dc30d6b80 1 Processor -- start 2024-05-10T11:17:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.986+0000 7f4dc30d6b80 1 -- start start 2024-05-10T11:17:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.986+0000 7f4dc30d6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e70180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.986+0000 7f4dc30d6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e706c0 0x55b7c8e6a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4dc30d6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e6aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4dc30d6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b7c8dbca20 con 0x55b7c8e6fda0 2024-05-10T11:17:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4dc30d6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b7c8d5a760 con 0x55b7c8e706c0 2024-05-10T11:17:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4dc30d6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b7c8db89e0 con 0x55b7c8e6aa50 2024-05-10T11:17:54.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4db5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e70180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:54.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4db5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e70180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37818/0 (socket says 172.21.15.27:37818) 2024-05-10T11:17:54.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4db5400700 1 -- 172.21.15.27:0/1664546320 learned_addr learned my addr 172.21.15.27:0/1664546320 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:54.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4db5e00700 1 --2- 172.21.15.27:0/1664546320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e706c0 0x55b7c8e6a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:54.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4db5400700 1 --2- 172.21.15.27:0/1664546320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e6aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:54.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4db5e00700 1 -- 172.21.15.27:0/1664546320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 msgr2=0x55b7c8e6aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:54.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4db5e00700 1 --2- 172.21.15.27:0/1664546320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e6aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4db5e00700 1 -- 172.21.15.27:0/1664546320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 msgr2=0x55b7c8e70180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:54.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4db5e00700 1 --2- 172.21.15.27:0/1664546320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e70180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4db5e00700 1 -- 172.21.15.27:0/1664546320 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7c8d098d0 con 0x55b7c8e706c0 2024-05-10T11:17:54.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.987+0000 7f4db5400700 1 --2- 172.21.15.27:0/1664546320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e70180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:17:54.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.988+0000 7f4daaa00700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4d980b29f0 0x7f4d980b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:54.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.988+0000 7f4db5e00700 1 --2- 172.21.15.27:0/1664546320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e706c0 0x55b7c8e6a430 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4da40098e0 tx=0x7f4da4005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb640c00bd137d3b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:54.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.988+0000 7f4daaa00700 1 -- 172.21.15.27:0/4179046941 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f4dac0e24b0 con 0x7f4db0108700 2024-05-10T11:17:54.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.988+0000 7f4db5400700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4d980b29f0 0x7f4d980b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:54.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.988+0000 7f4d9f400700 1 -- 172.21.15.27:0/1664546320 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4027070 con 0x55b7c8e706c0 2024-05-10T11:17:54.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.988+0000 7f4d9f400700 1 -- 172.21.15.27:0/1664546320 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4da40095b0 con 0x55b7c8e706c0 2024-05-10T11:17:54.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.988+0000 7f4d9f400700 1 -- 172.21.15.27:0/1664546320 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4003a80 con 0x55b7c8e706c0 2024-05-10T11:17:54.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.989+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1664546320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e706c0 msgr2=0x55b7c8e6a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:54.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.989+0000 7f4db5400700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4d980b29f0 0x7f4d980b4eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55b7c8e6b410 tx=0x7f4da0009f90 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:54.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.989+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/1664546320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e706c0 0x55b7c8e6a430 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4da40098e0 tx=0x7f4da4005bc0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.989+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1664546320 shutdown_connections 2024-05-10T11:17:54.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.989+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/1664546320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e6aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.989+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/1664546320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e70180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.989+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/1664546320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e706c0 0x55b7c8e6a430 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.989+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1664546320 >> 172.21.15.27:0/1664546320 conn(0x55b7c8de26d0 msgr2=0x55b7c8e64ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:54.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.990+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1664546320 shutdown_connections 2024-05-10T11:17:54.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.990+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1664546320 wait complete. 2024-05-10T11:17:54.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.990+0000 7f4dc30d6b80 1 -- start start 2024-05-10T11:17:54.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.990+0000 7f4dc30d6b80 1 Processor -- start 2024-05-10T11:17:54.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.991+0000 7f4dc30d6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e763f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:54.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.991+0000 7f4dc30d6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e73790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:54.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.991+0000 7f4dc30d6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e706c0 0x55b7c8e73cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:54.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.991+0000 7f4dc30d6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b7c8e770f0 con 0x55b7c8e706c0 2024-05-10T11:17:54.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.991+0000 7f4dc30d6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b7c8d22b20 con 0x55b7c8e6fda0 2024-05-10T11:17:54.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.991+0000 7f4dc30d6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b7c8e77270 con 0x55b7c8e6aa50 2024-05-10T11:17:54.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.991+0000 7f4dc30d6b80 10 client.0 _ll_register_callbacks cb 0x55b7c8d63c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:17:54.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.991+0000 7f4db5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e73790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:54.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.991+0000 7f4db5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e763f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:54.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.991+0000 7f4db6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e706c0 0x55b7c8e73cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:54.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.991+0000 7f4db5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e73790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40244/0 (socket says 172.21.15.27:40244) 2024-05-10T11:17:54.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.991+0000 7f4db5400700 1 -- 172.21.15.27:0/2412311825 learned_addr learned my addr 172.21.15.27:0/2412311825 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:54.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.992+0000 7f4db5400700 1 -- 172.21.15.27:0/2412311825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 msgr2=0x55b7c8e763f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:54.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.992+0000 7f4db5400700 1 --2- 172.21.15.27:0/2412311825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e763f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.992+0000 7f4db5400700 1 -- 172.21.15.27:0/2412311825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e706c0 msgr2=0x55b7c8e73cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.992+0000 7f4db5400700 1 --2- 172.21.15.27:0/2412311825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e706c0 0x55b7c8e73cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.992+0000 7f4db5400700 1 -- 172.21.15.27:0/2412311825 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7c8d82420 con 0x55b7c8e6fda0 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.992+0000 7f4db5400700 1 --2- 172.21.15.27:0/2412311825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e73790 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4da0005c30 tx=0x7f4da0009c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=708d1cb6be7dad98 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.992+0000 7f4d93e00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.992+0000 7f4d93e00700 1 -- 172.21.15.27:0/2412311825 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da0021a40 con 0x55b7c8e6fda0 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.992+0000 7f4d93e00700 1 -- 172.21.15.27:0/2412311825 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f4da0021be0 con 0x55b7c8e6fda0 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.993+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/2412311825 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4da40095b0 con 0x55b7c8e6fda0 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.993+0000 7f4d93e00700 1 -- 172.21.15.27:0/2412311825 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4da0021e20 con 0x55b7c8e6fda0 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.993+0000 7f4dc30d6b80 10 client.26033 fetch_fsmap learned FSMap version 33 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.993+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/2412311825 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b7c8e819b0 con 0x55b7c8e6fda0 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.993+0000 7f4d93e00700 1 -- 172.21.15.27:0/2412311825 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4da40095b0 con 0x55b7c8e6fda0 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.993+0000 7f4d93e00700 1 -- 172.21.15.27:0/2412311825 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f4da00225d0 con 0x55b7c8e6fda0 2024-05-10T11:17:54.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.993+0000 7f4dc30d6b80 10 client.26033 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:17:54.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.993+0000 7f4dc30d6b80 10 client.26033 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:17:54.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.993+0000 7f4dc30d6b80 20 client.26033 populate_metadata read hostname 'smithi027' 2024-05-10T11:17:54.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.993+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/2412311825 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55b7c8e83ff0 0x55b7c8e863e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:54.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.993+0000 7f4dc30d6b80 4 client.26033 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:17:54.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.993+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/2412311825 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7c8e37140 con 0x55b7c8e83ff0 2024-05-10T11:17:54.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.994+0000 7f4db5e00700 1 --2- 172.21.15.27:0/2412311825 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55b7c8e83ff0 0x55b7c8e863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:54.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.994+0000 7f4db5e00700 1 --2- 172.21.15.27:0/2412311825 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55b7c8e83ff0 0x55b7c8e863e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc14fe509c38cb20 server_cookie=732812e3cb41e046 in_seq=0 out_seq=0 2024-05-10T11:17:54.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.994+0000 7f4d93e00700 10 client.26033 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:17:54.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.995+0000 7f4d93e00700 1 -- 172.21.15.27:0/2412311825 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b7c8e37140 con 0x55b7c8e83ff0 2024-05-10T11:17:54.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.995+0000 7f4d93e00700 10 client.26033 handle_command_reply: tid=0 2024-05-10T11:17:54.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4dc30d6b80 1 client.26033 shutdown 2024-05-10T11:17:54.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/2412311825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 msgr2=0x55b7c8e73790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:54.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/2412311825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e73790 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4da0005c30 tx=0x7f4da0009c70 comp rx=0 tx=0).stop 2024-05-10T11:17:54.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/2412311825 shutdown_connections 2024-05-10T11:17:54.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/2412311825 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55b7c8e83ff0 0x55b7c8e863e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/2412311825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e763f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/2412311825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e706c0 0x55b7c8e73cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/2412311825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e73790 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:54.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4d93e00700 0 client.26033 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:17:54.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/2412311825 >> 172.21.15.27:0/2412311825 conn(0x55b7c8de26d0 msgr2=0x55b7c8e5f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:54.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/2412311825 shutdown_connections 2024-05-10T11:17:54.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/2412311825 wait complete. 2024-05-10T11:17:54.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:54.996+0000 7f4dc30d6b80 20 client.26033 trim_cache size 0 max 16384 2024-05-10T11:17:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.014+0000 7f4dc30d6b80 1 Processor -- start 2024-05-10T11:17:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.014+0000 7f4dc30d6b80 1 -- start start 2024-05-10T11:17:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.015+0000 7f4dc30d6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e6a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.015+0000 7f4dc30d6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e72710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.015+0000 7f4dc30d6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e706c0 0x55b7c8e71360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.015+0000 7f4dc30d6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01b1660 con 0x55b7c8e6fda0 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.015+0000 7f4dc30d6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01b31a0 con 0x55b7c8e706c0 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.015+0000 7f4dc30d6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01b1d30 con 0x55b7c8e6aa50 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.015+0000 7f4db5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e6a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.015+0000 7f4db5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e6a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37358/0 (socket says 172.21.15.27:37358) 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.015+0000 7f4db5e00700 1 -- 172.21.15.27:0/971425109 learned_addr learned my addr 172.21.15.27:0/971425109 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.015+0000 7f4db6800700 1 --2- 172.21.15.27:0/971425109 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e706c0 0x55b7c8e71360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.015+0000 7f4db5400700 1 --2- 172.21.15.27:0/971425109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e72710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.016+0000 7f4db5e00700 1 -- 172.21.15.27:0/971425109 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e706c0 msgr2=0x55b7c8e71360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.016+0000 7f4db5e00700 1 --2- 172.21.15.27:0/971425109 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e706c0 0x55b7c8e71360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.016+0000 7f4db5e00700 1 -- 172.21.15.27:0/971425109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 msgr2=0x55b7c8e72710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.016+0000 7f4db5e00700 1 --2- 172.21.15.27:0/971425109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e72710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.016+0000 7f4db5e00700 1 -- 172.21.15.27:0/971425109 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7c8e71d50 con 0x55b7c8e6aa50 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.016+0000 7f4db6800700 1 --2- 172.21.15.27:0/971425109 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e706c0 0x55b7c8e71360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:17:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.016+0000 7f4db5400700 1 --2- 172.21.15.27:0/971425109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e72710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:17:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.016+0000 7f4db5e00700 1 --2- 172.21.15.27:0/971425109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e6a430 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4da4009350 tx=0x7f4da4005ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e06a36feb8dbd965 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.016+0000 7f4d9ea00700 1 -- 172.21.15.27:0/971425109 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4027040 con 0x55b7c8e6aa50 2024-05-10T11:17:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.016+0000 7f4d9ea00700 1 -- 172.21.15.27:0/971425109 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4da4028040 con 0x55b7c8e6aa50 2024-05-10T11:17:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.017+0000 7f4d9ea00700 1 -- 172.21.15.27:0/971425109 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4003dd0 con 0x55b7c8e6aa50 2024-05-10T11:17:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.017+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/971425109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 msgr2=0x55b7c8e6a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.017+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/971425109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e6a430 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4da4009350 tx=0x7f4da4005ac0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.017+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/971425109 shutdown_connections 2024-05-10T11:17:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.017+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/971425109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e6a430 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.017+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/971425109 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e706c0 0x55b7c8e71360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.017+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/971425109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e72710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.017+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/971425109 >> 172.21.15.27:0/971425109 conn(0x55b7c8de26d0 msgr2=0x55b7c8e74d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.017+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/971425109 shutdown_connections 2024-05-10T11:17:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.018+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/971425109 wait complete. 2024-05-10T11:17:55.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.018+0000 7f4dc30d6b80 1 -- start start 2024-05-10T11:17:55.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.018+0000 7f4dc30d6b80 1 Processor -- start 2024-05-10T11:17:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.019+0000 7f4dc30d6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e8b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.019+0000 7f4dc30d6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e8bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.019+0000 7f4dc30d6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e706c0 0x55b7c8e8c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.019+0000 7f4dc30d6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01b2ad0 con 0x55b7c8e6aa50 2024-05-10T11:17:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.019+0000 7f4dc30d6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01b2400 con 0x55b7c8e6fda0 2024-05-10T11:17:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.019+0000 7f4dc30d6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01af600 con 0x55b7c8e706c0 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.019+0000 7f4dc30d6b80 10 client.0 _ll_register_callbacks cb 0x55b7c8d63c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.019+0000 7f4db5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e8bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.019+0000 7f4db6800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e706c0 0x55b7c8e8c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.019+0000 7f4db5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e8bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40272/0 (socket says 172.21.15.27:40272) 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.019+0000 7f4db5400700 1 -- 172.21.15.27:0/1308270430 learned_addr learned my addr 172.21.15.27:0/1308270430 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.019+0000 7f4db5e00700 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e8b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.020+0000 7f4db6800700 1 -- 172.21.15.27:0/1308270430 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 msgr2=0x55b7c8e8bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.020+0000 7f4db6800700 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e8bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.020+0000 7f4db6800700 1 -- 172.21.15.27:0/1308270430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6aa50 msgr2=0x55b7c8e8b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.020+0000 7f4db6800700 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e8b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.020+0000 7f4db6800700 1 -- 172.21.15.27:0/1308270430 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7c8eed9f0 con 0x55b7c8e706c0 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.020+0000 7f4db5400700 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e8bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.020+0000 7f4db6800700 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e706c0 0x55b7c8e8c050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0df7d0 tx=0x7f4dac0a9930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71969835656d1cb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.020+0000 7f4d93e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:17:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.020+0000 7f4d93e00700 1 -- 172.21.15.27:0/1308270430 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac0c3060 con 0x55b7c8e706c0 2024-05-10T11:17:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.020+0000 7f4d93e00700 1 -- 172.21.15.27:0/1308270430 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f4dac0a8860 con 0x55b7c8e706c0 2024-05-10T11:17:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.020+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1308270430 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4da4028040 con 0x55b7c8e706c0 2024-05-10T11:17:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.021+0000 7f4d93e00700 1 -- 172.21.15.27:0/1308270430 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4dac0a8f00 con 0x55b7c8e706c0 2024-05-10T11:17:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.021+0000 7f4dc30d6b80 10 client.26032 fetch_fsmap learned FSMap version 33 2024-05-10T11:17:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.021+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1308270430 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b7c8eeec40 con 0x55b7c8e706c0 2024-05-10T11:17:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.021+0000 7f4d93e00700 1 -- 172.21.15.27:0/1308270430 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4da4028040 con 0x55b7c8e706c0 2024-05-10T11:17:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.021+0000 7f4d93e00700 1 -- 172.21.15.27:0/1308270430 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f4dac0a4d60 con 0x55b7c8e706c0 2024-05-10T11:17:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.021+0000 7f4dc30d6b80 10 client.26032 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.021+0000 7f4dc30d6b80 10 client.26032 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.021+0000 7f4dc30d6b80 20 client.26032 populate_metadata read hostname 'smithi027' 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.021+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55b7c8eef320 0x55b7c8ef1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.021+0000 7f4dc30d6b80 4 client.26032 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.021+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1308270430 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b7c8dca400 con 0x55b7c8eef320 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.022+0000 7f4db5e00700 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55b7c8eef320 0x55b7c8ef1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.022+0000 7f4db5e00700 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55b7c8eef320 0x55b7c8ef1740 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f10fa3b02f4c2a47 server_cookie=1373b05b52b0228e in_seq=0 out_seq=0 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.022+0000 7f4d93e00700 10 client.26032 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.022+0000 7f4d93e00700 1 -- 172.21.15.27:0/1308270430 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b7c8dca400 con 0x55b7c8eef320 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.022+0000 7f4d93e00700 10 client.26032 handle_command_reply: tid=0 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.022+0000 7f4dc30d6b80 1 client.26032 shutdown 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.023+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1308270430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e706c0 msgr2=0x55b7c8e8c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.023+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e706c0 0x55b7c8e8c050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0df7d0 tx=0x7f4dac0a9930 comp rx=0 tx=0).stop 2024-05-10T11:17:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.023+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1308270430 shutdown_connections 2024-05-10T11:17:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.023+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55b7c8eef320 0x55b7c8ef1740 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.023+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7c8e706c0 0x55b7c8e8c050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.023+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b7c8e6aa50 0x55b7c8e8b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.023+0000 7f4d93e00700 0 client.26032 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:17:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.023+0000 7f4dc30d6b80 1 --2- 172.21.15.27:0/1308270430 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b7c8e6fda0 0x55b7c8e8bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.023+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1308270430 >> 172.21.15.27:0/1308270430 conn(0x55b7c8de26d0 msgr2=0x55b7c8e74d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.023+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1308270430 shutdown_connections 2024-05-10T11:17:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.023+0000 7f4dc30d6b80 1 -- 172.21.15.27:0/1308270430 wait complete. 2024-05-10T11:17:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.023+0000 7f4dc30d6b80 20 client.26032 trim_cache size 0 max 16384 2024-05-10T11:17:55.026 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:17:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.024+0000 7f4db7200700 1 -- 172.21.15.27:0/4179046941 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4d980b29f0 msgr2=0x7f4d980b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.024+0000 7f4db7200700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4d980b29f0 0x7f4d980b4eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55b7c8e6b410 tx=0x7f4da0009f90 comp rx=0 tx=0).stop 2024-05-10T11:17:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.024+0000 7f4db7200700 1 -- 172.21.15.27:0/4179046941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4db0108700 msgr2=0x7f4db01a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.024+0000 7f4db7200700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4db0108700 0x7f4db01a05e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00efc0 tx=0x7f4dac00c3a0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.024+0000 7f4db7200700 1 -- 172.21.15.27:0/4179046941 shutdown_connections 2024-05-10T11:17:55.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.024+0000 7f4db7200700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4d980b29f0 0x7f4d980b4eb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.024+0000 7f4db7200700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4db0103620 0x7f4db019bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.024+0000 7f4db7200700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4db0108700 0x7f4db01a05e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.024+0000 7f4db7200700 1 --2- 172.21.15.27:0/4179046941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db0103fd0 0x7f4db019c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:55.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.024+0000 7f4db7200700 1 -- 172.21.15.27:0/4179046941 >> 172.21.15.27:0/4179046941 conn(0x7f4db00feef0 msgr2=0x7f4db01011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:55.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.025+0000 7f4db7200700 1 -- 172.21.15.27:0/4179046941 shutdown_connections 2024-05-10T11:17:55.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:55.025+0000 7f4db7200700 1 -- 172.21.15.27:0/4179046941 wait complete. 2024-05-10T11:17:55.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:55 smithi027 ceph-mon[30054]: pgmap v144: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-05-10T11:17:55.670 INFO:tasks.workunit.client.0.smithi027.stdout: 1 225292 7.78 MB/sec execute 40 sec latency 27.072 ms 2024-05-10T11:17:55.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:55 smithi076 ceph-mon[34152]: pgmap v144: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-05-10T11:17:55.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:55 smithi189 ceph-mon[34842]: pgmap v144: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-05-10T11:17:56.091 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:17:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:17:56.670 INFO:tasks.workunit.client.0.smithi027.stdout: 1 227358 7.92 MB/sec execute 41 sec latency 12.873 ms 2024-05-10T11:17:57.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:57 smithi027 ceph-mon[30054]: pgmap v145: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 166 op/s 2024-05-10T11:17:57.670 INFO:tasks.workunit.client.0.smithi027.stdout: 1 228544 7.86 MB/sec execute 42 sec latency 26.791 ms 2024-05-10T11:17:57.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:57 smithi076 ceph-mon[34152]: pgmap v145: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 166 op/s 2024-05-10T11:17:57.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:57 smithi189 ceph-mon[34842]: pgmap v145: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 166 op/s 2024-05-10T11:17:58.670 INFO:tasks.workunit.client.0.smithi027.stdout: 1 230799 7.99 MB/sec execute 43 sec latency 18.744 ms 2024-05-10T11:17:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.302+0000 7f2995c00700 1 -- 172.21.15.27:0/1644857666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29901056c0 msgr2=0x7f2990105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.302+0000 7f2995c00700 1 --2- 172.21.15.27:0/1644857666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29901056c0 0x7f2990105b40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f297c009990 tx=0x7f297c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.302+0000 7f2995c00700 1 -- 172.21.15.27:0/1644857666 shutdown_connections 2024-05-10T11:17:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.302+0000 7f2995c00700 1 --2- 172.21.15.27:0/1644857666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f299010b6c0 0x7f299010baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.302+0000 7f2995c00700 1 --2- 172.21.15.27:0/1644857666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990106080 0x7f2990112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.302+0000 7f2995c00700 1 --2- 172.21.15.27:0/1644857666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29901056c0 0x7f2990105b40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.302+0000 7f2995c00700 1 -- 172.21.15.27:0/1644857666 >> 172.21.15.27:0/1644857666 conn(0x7f29901011b0 msgr2=0x7f29901035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.302+0000 7f2995c00700 1 -- 172.21.15.27:0/1644857666 shutdown_connections 2024-05-10T11:17:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.302+0000 7f2995c00700 1 -- 172.21.15.27:0/1644857666 wait complete. 2024-05-10T11:17:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.303+0000 7f2995c00700 1 Processor -- start 2024-05-10T11:17:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.304+0000 7f2995c00700 1 -- start start 2024-05-10T11:17:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.304+0000 7f2995c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29901056c0 0x7f299019bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.304+0000 7f2995c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990106080 0x7f299019c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.304+0000 7f2995c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f299010b6c0 0x7f29901a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.304+0000 7f2995c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2990115950 con 0x7f2990106080 2024-05-10T11:17:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.304+0000 7f2995c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f29901157d0 con 0x7f299010b6c0 2024-05-10T11:17:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.304+0000 7f2995c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2990115ad0 con 0x7f29901056c0 2024-05-10T11:17:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.304+0000 7f298ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29901056c0 0x7f299019bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.304+0000 7f298ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29901056c0 0x7f299019bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37398/0 (socket says 172.21.15.27:37398) 2024-05-10T11:17:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.304+0000 7f298ea00700 1 -- 172.21.15.27:0/272402145 learned_addr learned my addr 172.21.15.27:0/272402145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.304+0000 7f298e000700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990106080 0x7f299019c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.304+0000 7f298f400700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f299010b6c0 0x7f29901a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.305+0000 7f298ea00700 1 -- 172.21.15.27:0/272402145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f299010b6c0 msgr2=0x7f29901a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:59.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.305+0000 7f298ea00700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f299010b6c0 0x7f29901a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.305+0000 7f298ea00700 1 -- 172.21.15.27:0/272402145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990106080 msgr2=0x7f299019c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:59.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.305+0000 7f298ea00700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990106080 0x7f299019c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.305+0000 7f298ea00700 1 -- 172.21.15.27:0/272402145 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29901a0d90 con 0x7f29901056c0 2024-05-10T11:17:59.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.305+0000 7f298ea00700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29901056c0 0x7f299019bd80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f298000e9a0 tx=0x7f298000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:59.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.306+0000 7f2987400700 1 -- 172.21.15.27:0/272402145 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298000ca10 con 0x7f29901056c0 2024-05-10T11:17:59.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.306+0000 7f2987400700 1 -- 172.21.15.27:0/272402145 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f298000cbb0 con 0x7f29901056c0 2024-05-10T11:17:59.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.306+0000 7f2987400700 1 -- 172.21.15.27:0/272402145 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980018680 con 0x7f29901056c0 2024-05-10T11:17:59.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.306+0000 7f2995c00700 1 -- 172.21.15.27:0/272402145 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29901a1080 con 0x7f29901056c0 2024-05-10T11:17:59.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.306+0000 7f2995c00700 1 -- 172.21.15.27:0/272402145 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29901a8900 con 0x7f29901056c0 2024-05-10T11:17:59.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.310+0000 7f2987400700 1 -- 172.21.15.27:0/272402145 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2980010430 con 0x7f29901056c0 2024-05-10T11:17:59.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.311+0000 7f29a04bdb80 1 Processor -- start 2024-05-10T11:17:59.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.311+0000 7f29a04bdb80 1 -- start start 2024-05-10T11:17:59.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.311+0000 7f29a04bdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 0x5576bec48440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.311+0000 7f29a04bdb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48980 0x5576bec4c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.311+0000 7f29a04bdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec4c9d0 0x5576bec4ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f29a04bdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5576beb9aa20 con 0x5576bec4c9d0 2024-05-10T11:17:59.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f29a04bdb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5576beb38760 con 0x5576bec48980 2024-05-10T11:17:59.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f29a04bdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5576beb969e0 con 0x5576bec48060 2024-05-10T11:17:59.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f2987400700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f29740b2a70 0x7f29740b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f298f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 0x5576bec48440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f298f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 0x5576bec48440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37402/0 (socket says 172.21.15.27:37402) 2024-05-10T11:17:59.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f298f400700 1 -- 172.21.15.27:0/1829716106 learned_addr learned my addr 172.21.15.27:0/1829716106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:59.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f298ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48980 0x5576bec4c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f298e000700 1 --2- 172.21.15.27:0/1829716106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec4c9d0 0x5576bec4ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f2987400700 1 -- 172.21.15.27:0/272402145 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f29800e2ad0 con 0x7f29901056c0 2024-05-10T11:17:59.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f298f400700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f29740b2a70 0x7f29740b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f298ea00700 1 -- 172.21.15.27:0/1829716106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 msgr2=0x5576bec48440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:59.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f298ea00700 1 --2- 172.21.15.27:0/1829716106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 0x5576bec48440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f298ea00700 1 -- 172.21.15.27:0/1829716106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec4c9d0 msgr2=0x5576bec4ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:59.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f298ea00700 1 --2- 172.21.15.27:0/1829716106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec4c9d0 0x5576bec4ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f298ea00700 1 -- 172.21.15.27:0/1829716106 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576beae78d0 con 0x5576bec48980 2024-05-10T11:17:59.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.312+0000 7f298e000700 1 --2- 172.21.15.27:0/1829716106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec4c9d0 0x5576bec4ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:17:59.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.313+0000 7f298f400700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f29740b2a70 0x7f29740b4f30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f298800fb80 tx=0x7f2988004b40 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:59.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.314+0000 7f298ea00700 1 --2- 172.21.15.27:0/1829716106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48980 0x5576bec4c490 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f29800dfdf0 tx=0x7f2980010c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e729c11a2fe5dec4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:59.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.314+0000 7f297be00700 1 -- 172.21.15.27:0/1829716106 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800b2120 con 0x5576bec48980 2024-05-10T11:17:59.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.314+0000 7f297be00700 1 -- 172.21.15.27:0/1829716106 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29800a9490 con 0x5576bec48980 2024-05-10T11:17:59.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.314+0000 7f297be00700 1 -- 172.21.15.27:0/1829716106 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800a9de0 con 0x5576bec48980 2024-05-10T11:17:59.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.315+0000 7f29a04bdb80 1 -- 172.21.15.27:0/1829716106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48980 msgr2=0x5576bec4c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:59.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.315+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/1829716106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48980 0x5576bec4c490 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f29800dfdf0 tx=0x7f2980010c10 comp rx=0 tx=0).stop 2024-05-10T11:17:59.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.315+0000 7f29a04bdb80 1 -- 172.21.15.27:0/1829716106 shutdown_connections 2024-05-10T11:17:59.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.315+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/1829716106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 0x5576bec48440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.315+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/1829716106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec4c9d0 0x5576bec4ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.315+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/1829716106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48980 0x5576bec4c490 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.315+0000 7f29a04bdb80 1 -- 172.21.15.27:0/1829716106 >> 172.21.15.27:0/1829716106 conn(0x5576beb76b60 msgr2=0x5576bec427a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.315+0000 7f29a04bdb80 1 -- 172.21.15.27:0/1829716106 shutdown_connections 2024-05-10T11:17:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.315+0000 7f29a04bdb80 1 -- 172.21.15.27:0/1829716106 wait complete. 2024-05-10T11:17:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.316+0000 7f29a04bdb80 1 -- start start 2024-05-10T11:17:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.316+0000 7f29a04bdb80 1 Processor -- start 2024-05-10T11:17:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.316+0000 7f29a04bdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 0x5576bec53e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.316+0000 7f29a04bdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48980 0x5576bec51270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.316+0000 7f29a04bdb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec4c9d0 0x5576bec517b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.316+0000 7f29a04bdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5576bec54b20 con 0x5576bec48980 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.316+0000 7f29a04bdb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5576beb00b20 con 0x5576bec4c9d0 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.316+0000 7f29a04bdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5576bec54ca0 con 0x5576bec48060 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.317+0000 7f29a04bdb80 10 client.0 _ll_register_callbacks cb 0x5576beb41c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.317+0000 7f298ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 0x5576bec53e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.317+0000 7f298e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48980 0x5576bec51270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.317+0000 7f298ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 0x5576bec53e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37410/0 (socket says 172.21.15.27:37410) 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.317+0000 7f298ea00700 1 -- 172.21.15.27:0/571101024 learned_addr learned my addr 172.21.15.27:0/571101024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.317+0000 7f298f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec4c9d0 0x5576bec517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.317+0000 7f298ea00700 1 -- 172.21.15.27:0/571101024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec4c9d0 msgr2=0x5576bec517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.317+0000 7f298ea00700 1 --2- 172.21.15.27:0/571101024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec4c9d0 0x5576bec517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.317+0000 7f298ea00700 1 -- 172.21.15.27:0/571101024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48980 msgr2=0x5576bec51270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.317+0000 7f298ea00700 1 --2- 172.21.15.27:0/571101024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48980 0x5576bec51270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.317+0000 7f298ea00700 1 -- 172.21.15.27:0/571101024 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576beb60420 con 0x5576bec48060 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.318+0000 7f298ea00700 1 --2- 172.21.15.27:0/571101024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 0x5576bec53e80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f29800a5280 tx=0x7f29800b2270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=126e36ebb9f6f8c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.318+0000 7f2978c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.318+0000 7f2978c00700 1 -- 172.21.15.27:0/571101024 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800a9de0 con 0x5576bec48060 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.318+0000 7f2978c00700 1 -- 172.21.15.27:0/571101024 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f29800b2350 con 0x5576bec48060 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.318+0000 7f29a04bdb80 1 -- 172.21.15.27:0/571101024 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29800a9490 con 0x5576bec48060 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.318+0000 7f2978c00700 1 -- 172.21.15.27:0/571101024 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f29800add30 con 0x5576bec48060 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.318+0000 7f29a04bdb80 10 client.26044 fetch_fsmap learned FSMap version 33 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.318+0000 7f29a04bdb80 1 -- 172.21.15.27:0/571101024 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5576bec54b20 con 0x5576bec48060 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.318+0000 7f2978c00700 1 -- 172.21.15.27:0/571101024 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29800a9490 con 0x5576bec48060 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.319+0000 7f2978c00700 1 -- 172.21.15.27:0/571101024 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f29800a8d70 con 0x5576bec48060 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.319+0000 7f29a04bdb80 10 client.26044 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:17:59.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.319+0000 7f29a04bdb80 10 client.26044 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:17:59.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.319+0000 7f29a04bdb80 20 client.26044 populate_metadata read hostname 'smithi027' 2024-05-10T11:17:59.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.319+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/571101024 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5576bec616b0 0x5576bec63aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.319+0000 7f29a04bdb80 4 client.26044 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:17:59.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.319+0000 7f29a04bdb80 1 -- 172.21.15.27:0/571101024 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576bec150a0 con 0x5576bec616b0 2024-05-10T11:17:59.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.319+0000 7f298e000700 1 --2- 172.21.15.27:0/571101024 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5576bec616b0 0x5576bec63aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.320+0000 7f298e000700 1 --2- 172.21.15.27:0/571101024 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5576bec616b0 0x5576bec63aa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b93989b5fc1fcfa server_cookie=99ac5ba51eba9784 in_seq=0 out_seq=0 2024-05-10T11:17:59.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.320+0000 7f2978c00700 10 client.26044 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:17:59.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f2978c00700 1 -- 172.21.15.27:0/571101024 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5576bec150a0 con 0x5576bec616b0 2024-05-10T11:17:59.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f2978c00700 10 client.26044 handle_command_reply: tid=0 2024-05-10T11:17:59.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f29a04bdb80 1 client.26044 shutdown 2024-05-10T11:17:59.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f29a04bdb80 1 -- 172.21.15.27:0/571101024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 msgr2=0x5576bec53e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:59.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/571101024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 0x5576bec53e80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f29800a5280 tx=0x7f29800b2270 comp rx=0 tx=0).stop 2024-05-10T11:17:59.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f29a04bdb80 1 -- 172.21.15.27:0/571101024 shutdown_connections 2024-05-10T11:17:59.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/571101024 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5576bec616b0 0x5576bec63aa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/571101024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48060 0x5576bec53e80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/571101024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec4c9d0 0x5576bec517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/571101024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48980 0x5576bec51270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f2978c00700 0 client.26044 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:17:59.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f29a04bdb80 1 -- 172.21.15.27:0/571101024 >> 172.21.15.27:0/571101024 conn(0x5576beb76b60 msgr2=0x5576bec3d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:59.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f29a04bdb80 1 -- 172.21.15.27:0/571101024 shutdown_connections 2024-05-10T11:17:59.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.321+0000 7f29a04bdb80 1 -- 172.21.15.27:0/571101024 wait complete. 2024-05-10T11:17:59.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.322+0000 7f29a04bdb80 20 client.26044 trim_cache size 0 max 16384 2024-05-10T11:17:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.340+0000 7f29a04bdb80 1 Processor -- start 2024-05-10T11:17:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.340+0000 7f29a04bdb80 1 -- start start 2024-05-10T11:17:59.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.341+0000 7f29a04bdb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576beb76b60 0x5576bec501d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.341+0000 7f29a04bdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48060 0x5576bec5dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.341+0000 7f29a04bdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48980 0x5576bec3d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.341+0000 7f29a04bdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f29901b1910 con 0x5576bec48060 2024-05-10T11:17:59.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.341+0000 7f29a04bdb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f29901b3450 con 0x5576beb76b60 2024-05-10T11:17:59.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.341+0000 7f29a04bdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f29901b1fe0 con 0x5576bec48980 2024-05-10T11:17:59.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.341+0000 7f298ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576beb76b60 0x5576bec501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.341+0000 7f298ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576beb76b60 0x5576bec501d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40316/0 (socket says 172.21.15.27:40316) 2024-05-10T11:17:59.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.341+0000 7f298e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48060 0x5576bec5dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.341+0000 7f298ea00700 1 -- 172.21.15.27:0/3151232103 learned_addr learned my addr 172.21.15.27:0/3151232103 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:59.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.341+0000 7f298f400700 1 --2- 172.21.15.27:0/3151232103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48980 0x5576bec3d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.342+0000 7f298ea00700 1 -- 172.21.15.27:0/3151232103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48980 msgr2=0x5576bec3d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.342+0000 7f298ea00700 1 --2- 172.21.15.27:0/3151232103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48980 0x5576bec3d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.342+0000 7f298ea00700 1 -- 172.21.15.27:0/3151232103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48060 msgr2=0x5576bec5dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.342+0000 7f298ea00700 1 --2- 172.21.15.27:0/3151232103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48060 0x5576bec5dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.342+0000 7f298ea00700 1 -- 172.21.15.27:0/3151232103 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576bec43660 con 0x5576beb76b60 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.342+0000 7f298f400700 1 --2- 172.21.15.27:0/3151232103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48980 0x5576bec3d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.342+0000 7f298ea00700 1 --2- 172.21.15.27:0/3151232103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576beb76b60 0x5576bec501d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f29800a9910 tx=0x7f2980010c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c7466716a126235 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.342+0000 7f297b400700 1 -- 172.21.15.27:0/3151232103 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800b2550 con 0x5576beb76b60 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.342+0000 7f297b400700 1 -- 172.21.15.27:0/3151232103 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29800aa080 con 0x5576beb76b60 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.342+0000 7f297b400700 1 -- 172.21.15.27:0/3151232103 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800a9e20 con 0x5576beb76b60 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.343+0000 7f29a04bdb80 1 -- 172.21.15.27:0/3151232103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576beb76b60 msgr2=0x5576bec501d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.343+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/3151232103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576beb76b60 0x5576bec501d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f29800a9910 tx=0x7f2980010c10 comp rx=0 tx=0).stop 2024-05-10T11:17:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.343+0000 7f29a04bdb80 1 -- 172.21.15.27:0/3151232103 shutdown_connections 2024-05-10T11:17:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.343+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/3151232103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576bec48980 0x5576bec3d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.343+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/3151232103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576beb76b60 0x5576bec501d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.343+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/3151232103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48060 0x5576bec5dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.343+0000 7f29a04bdb80 1 -- 172.21.15.27:0/3151232103 >> 172.21.15.27:0/3151232103 conn(0x5576bec3c610 msgr2=0x5576bec3ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.343+0000 7f29a04bdb80 1 -- 172.21.15.27:0/3151232103 shutdown_connections 2024-05-10T11:17:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.343+0000 7f29a04bdb80 1 -- 172.21.15.27:0/3151232103 wait complete. 2024-05-10T11:17:59.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.344+0000 7f29a04bdb80 1 -- start start 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.344+0000 7f29a04bdb80 1 Processor -- start 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f29a04bdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576beb76b60 0x5576bec57c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f29a04bdb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48060 0x5576bec581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f29a04bdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48980 0x5576bec586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f29a04bdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f29901b2d80 con 0x5576bec48980 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f29a04bdb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f29901b26b0 con 0x5576bec48060 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f29a04bdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f29901af8b0 con 0x5576beb76b60 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f29a04bdb80 10 client.0 _ll_register_callbacks cb 0x5576beb41c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576beb76b60 0x5576bec57c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298e000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48060 0x5576bec581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576beb76b60 0x5576bec57c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37432/0 (socket says 172.21.15.27:37432) 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298e000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48060 0x5576bec581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40330/0 (socket says 172.21.15.27:40330) 2024-05-10T11:17:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298ea00700 1 -- 172.21.15.27:0/926388519 learned_addr learned my addr 172.21.15.27:0/926388519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298f400700 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48980 0x5576bec586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298ea00700 1 -- 172.21.15.27:0/926388519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48060 msgr2=0x5576bec581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298ea00700 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48060 0x5576bec581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298ea00700 1 -- 172.21.15.27:0/926388519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48980 msgr2=0x5576bec586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298ea00700 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48980 0x5576bec586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298ea00700 1 -- 172.21.15.27:0/926388519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576beccaf40 con 0x5576beb76b60 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298f400700 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48980 0x5576bec586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.345+0000 7f298e000700 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48060 0x5576bec581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.346+0000 7f298ea00700 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576beb76b60 0x5576bec57c60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f29800adef0 tx=0x7f29800a9ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90986248c65a23f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.346+0000 7f2978c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.346+0000 7f2978c00700 1 -- 172.21.15.27:0/926388519 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800a92d0 con 0x5576beb76b60 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.346+0000 7f29a04bdb80 1 -- 172.21.15.27:0/926388519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29800aa080 con 0x5576beb76b60 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.346+0000 7f2978c00700 1 -- 172.21.15.27:0/926388519 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f29800a86d0 con 0x5576beb76b60 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.346+0000 7f2978c00700 1 -- 172.21.15.27:0/926388519 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f29800b27e0 con 0x5576beb76b60 2024-05-10T11:17:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.346+0000 7f29a04bdb80 10 client.26050 fetch_fsmap learned FSMap version 33 2024-05-10T11:17:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.346+0000 7f29a04bdb80 1 -- 172.21.15.27:0/926388519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5576beccaf40 con 0x5576beb76b60 2024-05-10T11:17:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.347+0000 7f2978c00700 1 -- 172.21.15.27:0/926388519 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29800aa080 con 0x5576beb76b60 2024-05-10T11:17:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.347+0000 7f2978c00700 1 -- 172.21.15.27:0/926388519 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f29800aefe0 con 0x5576beb76b60 2024-05-10T11:17:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.347+0000 7f29a04bdb80 10 client.26050 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:17:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.347+0000 7f29a04bdb80 10 client.26050 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:17:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.347+0000 7f29a04bdb80 20 client.26050 populate_metadata read hostname 'smithi027' 2024-05-10T11:17:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.347+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5576bec4c9d0 0x5576becce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:17:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.347+0000 7f29a04bdb80 4 client.26050 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:17:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.347+0000 7f29a04bdb80 1 -- 172.21.15.27:0/926388519 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5576beba8400 con 0x5576bec4c9d0 2024-05-10T11:17:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.347+0000 7f298e000700 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5576bec4c9d0 0x5576becce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:17:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f298e000700 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5576bec4c9d0 0x5576becce4a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac96bdd186b7d148 server_cookie=c0449bc9af907a74 in_seq=0 out_seq=0 2024-05-10T11:17:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f2978c00700 10 client.26050 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:17:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f2978c00700 1 -- 172.21.15.27:0/926388519 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5576beba8400 con 0x5576bec4c9d0 2024-05-10T11:17:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f2978c00700 10 client.26050 handle_command_reply: tid=0 2024-05-10T11:17:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f29a04bdb80 1 client.26050 shutdown 2024-05-10T11:17:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f29a04bdb80 1 -- 172.21.15.27:0/926388519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576beb76b60 msgr2=0x5576bec57c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576beb76b60 0x5576bec57c60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f29800adef0 tx=0x7f29800a9ea0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f29a04bdb80 1 -- 172.21.15.27:0/926388519 shutdown_connections 2024-05-10T11:17:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5576bec4c9d0 0x5576becce4a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5576beb76b60 0x5576bec57c60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5576bec48980 0x5576bec586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f29a04bdb80 1 --2- 172.21.15.27:0/926388519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576bec48060 0x5576bec581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f2978c00700 0 client.26050 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:17:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f29a04bdb80 1 -- 172.21.15.27:0/926388519 >> 172.21.15.27:0/926388519 conn(0x5576bec3c610 msgr2=0x5576bec3d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.348+0000 7f29a04bdb80 1 -- 172.21.15.27:0/926388519 shutdown_connections 2024-05-10T11:17:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.349+0000 7f29a04bdb80 1 -- 172.21.15.27:0/926388519 wait complete. 2024-05-10T11:17:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.349+0000 7f29a04bdb80 20 client.26050 trim_cache size 0 max 16384 2024-05-10T11:17:59.353 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:17:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.349+0000 7f2995c00700 1 -- 172.21.15.27:0/272402145 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f29740b2a70 msgr2=0x7f29740b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.349+0000 7f2995c00700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f29740b2a70 0x7f29740b4f30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f298800fb80 tx=0x7f2988004b40 comp rx=0 tx=0).stop 2024-05-10T11:17:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.349+0000 7f2995c00700 1 -- 172.21.15.27:0/272402145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29901056c0 msgr2=0x7f299019bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:17:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.349+0000 7f2995c00700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29901056c0 0x7f299019bd80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f298000e9a0 tx=0x7f298000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.350+0000 7f2995c00700 1 -- 172.21.15.27:0/272402145 shutdown_connections 2024-05-10T11:17:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.350+0000 7f2995c00700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f29740b2a70 0x7f29740b4f30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.350+0000 7f2995c00700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29901056c0 0x7f299019bd80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.350+0000 7f2995c00700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f299010b6c0 0x7f29901a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.350+0000 7f2995c00700 1 --2- 172.21.15.27:0/272402145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990106080 0x7f299019c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:17:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.350+0000 7f2995c00700 1 -- 172.21.15.27:0/272402145 >> 172.21.15.27:0/272402145 conn(0x7f29901011b0 msgr2=0x7f2990101990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:17:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.350+0000 7f2995c00700 1 -- 172.21.15.27:0/272402145 shutdown_connections 2024-05-10T11:17:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:17:59.350+0000 7f2995c00700 1 -- 172.21.15.27:0/272402145 wait complete. 2024-05-10T11:17:59.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:17:59 smithi027 ceph-mon[30054]: pgmap v146: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 251 op/s 2024-05-10T11:17:59.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 231431 7.83 MB/sec execute 44 sec latency 30.529 ms 2024-05-10T11:17:59.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:17:59 smithi076 ceph-mon[34152]: pgmap v146: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 251 op/s 2024-05-10T11:17:59.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:17:59 smithi189 ceph-mon[34842]: pgmap v146: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 251 op/s 2024-05-10T11:18:00.535 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:18:00.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 232879 7.78 MB/sec execute 45 sec latency 89.734 ms 2024-05-10T11:18:01.059 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:18:01.464 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:01 smithi027 ceph-mon[30054]: pgmap v147: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 168 op/s 2024-05-10T11:18:01.670 INFO:tasks.workunit.client.0.smithi027.stdout: 1 233980 7.79 MB/sec execute 46 sec latency 81.390 ms 2024-05-10T11:18:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:01 smithi076 ceph-mon[34152]: pgmap v147: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 168 op/s 2024-05-10T11:18:01.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:01 smithi189 ceph-mon[34842]: pgmap v147: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 168 op/s 2024-05-10T11:18:02.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 234824 7.73 MB/sec execute 47 sec latency 29.249 ms 2024-05-10T11:18:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.519+0000 7f893ee00700 1 -- 172.21.15.27:0/3174459255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8938109540 msgr2=0x7f8938109920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.519+0000 7f893ee00700 1 --2- 172.21.15.27:0/3174459255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8938109540 0x7f8938109920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f89280099c0 tx=0x7f8928009cd0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.519+0000 7f893ee00700 1 -- 172.21.15.27:0/3174459255 shutdown_connections 2024-05-10T11:18:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.519+0000 7f893ee00700 1 --2- 172.21.15.27:0/3174459255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8938103f20 0x7f8938110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.519+0000 7f893ee00700 1 --2- 172.21.15.27:0/3174459255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8938109540 0x7f8938109920 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.519+0000 7f893ee00700 1 --2- 172.21.15.27:0/3174459255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938103560 0x7f89381039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.519+0000 7f893ee00700 1 -- 172.21.15.27:0/3174459255 >> 172.21.15.27:0/3174459255 conn(0x7f893807a660 msgr2=0x7f893807aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.519+0000 7f893ee00700 1 -- 172.21.15.27:0/3174459255 shutdown_connections 2024-05-10T11:18:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.520+0000 7f893ee00700 1 -- 172.21.15.27:0/3174459255 wait complete. 2024-05-10T11:18:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.521+0000 7f893ee00700 1 Processor -- start 2024-05-10T11:18:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.521+0000 7f893ee00700 1 -- start start 2024-05-10T11:18:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.522+0000 7f893ee00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938103560 0x7f89380757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.522+0000 7f893ee00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8938103f20 0x7f8938075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.522+0000 7f893ee00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8938109540 0x7f8938076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.522+0000 7f893ee00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f89381136a0 con 0x7f8938103f20 2024-05-10T11:18:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.522+0000 7f893ee00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8938113520 con 0x7f8938109540 2024-05-10T11:18:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.522+0000 7f893ee00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8938113820 con 0x7f8938103560 2024-05-10T11:18:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.522+0000 7f8937e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938103560 0x7f89380757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.522+0000 7f893ce00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8938109540 0x7f8938076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.522+0000 7f893ce00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8938109540 0x7f8938076220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40348/0 (socket says 172.21.15.27:40348) 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.522+0000 7f8937e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938103560 0x7f89380757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37452/0 (socket says 172.21.15.27:37452) 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.522+0000 7f893ce00700 1 -- 172.21.15.27:0/1793579022 learned_addr learned my addr 172.21.15.27:0/1793579022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.522+0000 7f8937400700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8938103f20 0x7f8938075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.523+0000 7f893ce00700 1 -- 172.21.15.27:0/1793579022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938103560 msgr2=0x7f89380757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.523+0000 7f893ce00700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938103560 0x7f89380757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.523+0000 7f893ce00700 1 -- 172.21.15.27:0/1793579022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8938103f20 msgr2=0x7f8938075ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.523+0000 7f893ce00700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8938103f20 0x7f8938075ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.523+0000 7f893ce00700 1 -- 172.21.15.27:0/1793579022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89381a80b0 con 0x7f8938109540 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.523+0000 7f893ce00700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8938109540 0x7f8938076220 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f893000ef30 tx=0x7f893000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.523+0000 7f8934c00700 1 -- 172.21.15.27:0/1793579022 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89300092e0 con 0x7f8938109540 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.523+0000 7f8934c00700 1 -- 172.21.15.27:0/1793579022 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8930004bb0 con 0x7f8938109540 2024-05-10T11:18:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.523+0000 7f8934c00700 1 -- 172.21.15.27:0/1793579022 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8930004020 con 0x7f8938109540 2024-05-10T11:18:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.523+0000 7f893ee00700 1 -- 172.21.15.27:0/1793579022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89381a8340 con 0x7f8938109540 2024-05-10T11:18:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.524+0000 7f893ee00700 1 -- 172.21.15.27:0/1793579022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89381a8830 con 0x7f8938109540 2024-05-10T11:18:03.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.527+0000 7f8934c00700 1 -- 172.21.15.27:0/1793579022 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f893001b030 con 0x7f8938109540 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.529+0000 7f8949694b80 1 Processor -- start 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.529+0000 7f8934c00700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f891c0b2a70 0x7f891c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.529+0000 7f8949694b80 1 -- start start 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.529+0000 7f8934c00700 1 -- 172.21.15.27:0/1793579022 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f89300e2430 con 0x7f8938109540 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.529+0000 7f8937400700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f891c0b2a70 0x7f891c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.529+0000 7f8949694b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579da3d0 0x5562579da7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.529+0000 7f8949694b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579d4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f8949694b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579d5100 0x5562579d5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f8949694b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55625791ea20 con 0x5562579da3d0 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f8949694b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5562578bc760 con 0x5562579d5100 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f8949694b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55625791a9e0 con 0x5562579dacf0 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f8937e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579da3d0 0x5562579da7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f893ce00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579d5100 0x5562579d5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f8937e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579da3d0 0x5562579da7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37902/0 (socket says 172.21.15.27:37902) 2024-05-10T11:18:03.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f8937e00700 1 -- 172.21.15.27:0/3463111938 learned_addr learned my addr 172.21.15.27:0/3463111938 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:03.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f893ce00700 1 -- 172.21.15.27:0/3463111938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 msgr2=0x5562579d4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:03.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f893ce00700 1 --2- 172.21.15.27:0/3463111938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579d4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f893ce00700 1 -- 172.21.15.27:0/3463111938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579da3d0 msgr2=0x5562579da7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:03.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f893ce00700 1 --2- 172.21.15.27:0/3463111938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579da3d0 0x5562579da7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.530+0000 7f893ce00700 1 -- 172.21.15.27:0/3463111938 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55625786b8d0 con 0x5562579d5100 2024-05-10T11:18:03.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.531+0000 7f8937400700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f891c0b2a70 0x7f891c0b4f30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f893807ac40 tx=0x7f8924011720 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:03.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.531+0000 7f893ce00700 1 --2- 172.21.15.27:0/3463111938 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579d5100 0x5562579d5580 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f89300df750 tx=0x7f893000ca60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1549c22be5aefc35 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:03.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.531+0000 7f892d600700 1 -- 172.21.15.27:0/3463111938 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89300a9370 con 0x5562579d5100 2024-05-10T11:18:03.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.531+0000 7f892d600700 1 -- 172.21.15.27:0/3463111938 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89300ad720 con 0x5562579d5100 2024-05-10T11:18:03.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.531+0000 7f892d600700 1 -- 172.21.15.27:0/3463111938 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89300b1f10 con 0x5562579d5100 2024-05-10T11:18:03.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.532+0000 7f8949694b80 1 -- 172.21.15.27:0/3463111938 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579d5100 msgr2=0x5562579d5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:03.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.532+0000 7f8949694b80 1 --2- 172.21.15.27:0/3463111938 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579d5100 0x5562579d5580 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f89300df750 tx=0x7f893000ca60 comp rx=0 tx=0).stop 2024-05-10T11:18:03.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.532+0000 7f8949694b80 1 -- 172.21.15.27:0/3463111938 shutdown_connections 2024-05-10T11:18:03.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.532+0000 7f8949694b80 1 --2- 172.21.15.27:0/3463111938 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579d5100 0x5562579d5580 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.532+0000 7f8949694b80 1 --2- 172.21.15.27:0/3463111938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579da3d0 0x5562579da7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.532+0000 7f8949694b80 1 --2- 172.21.15.27:0/3463111938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579d4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.532+0000 7f8949694b80 1 -- 172.21.15.27:0/3463111938 >> 172.21.15.27:0/3463111938 conn(0x5562579446d0 msgr2=0x5562579c6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:03.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.532+0000 7f8949694b80 1 -- 172.21.15.27:0/3463111938 shutdown_connections 2024-05-10T11:18:03.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.532+0000 7f8949694b80 1 -- 172.21.15.27:0/3463111938 wait complete. 2024-05-10T11:18:03.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.533+0000 7f8949694b80 1 -- start start 2024-05-10T11:18:03.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.533+0000 7f8949694b80 1 Processor -- start 2024-05-10T11:18:03.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.534+0000 7f8949694b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579d5100 0x5562579e0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.534+0000 7f8949694b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 0x5562579ddde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.534+0000 7f8949694b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579dacf0 0x5562579de320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.534+0000 7f8949694b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5562579e16d0 con 0x5562579dacf0 2024-05-10T11:18:03.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.534+0000 7f8949694b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x556257884b20 con 0x5562579da3d0 2024-05-10T11:18:03.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.534+0000 7f8949694b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5562579e1850 con 0x5562579d5100 2024-05-10T11:18:03.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.534+0000 7f8949694b80 10 client.0 _ll_register_callbacks cb 0x5562578c5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:18:03.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.534+0000 7f8937400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 0x5562579ddde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.534+0000 7f8937400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 0x5562579ddde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40356/0 (socket says 172.21.15.27:40356) 2024-05-10T11:18:03.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.534+0000 7f8937400700 1 -- 172.21.15.27:0/3099119486 learned_addr learned my addr 172.21.15.27:0/3099119486 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:03.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.534+0000 7f8937e00700 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579d5100 0x5562579e0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.534+0000 7f893ce00700 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579dacf0 0x5562579de320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.535+0000 7f8937400700 1 -- 172.21.15.27:0/3099119486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579d5100 msgr2=0x5562579e0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:03.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.535+0000 7f8937400700 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579d5100 0x5562579e0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.535+0000 7f8937400700 1 -- 172.21.15.27:0/3099119486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579dacf0 msgr2=0x5562579de320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:03.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.535+0000 7f8937400700 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579dacf0 0x5562579de320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.535+0000 7f8937400700 1 -- 172.21.15.27:0/3099119486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562578e4420 con 0x5562579da3d0 2024-05-10T11:18:03.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.535+0000 7f893ce00700 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579dacf0 0x5562579de320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:18:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.535+0000 7f8937400700 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 0x5562579ddde0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f892400fa40 tx=0x7f89240047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33d1991abbefcb42 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.535+0000 7f8922000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:18:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.535+0000 7f8922000700 1 -- 172.21.15.27:0/3099119486 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924022070 con 0x5562579da3d0 2024-05-10T11:18:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.535+0000 7f8922000700 1 -- 172.21.15.27:0/3099119486 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f892400fe00 con 0x5562579da3d0 2024-05-10T11:18:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.535+0000 7f8949694b80 1 -- 172.21.15.27:0/3099119486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89300ad720 con 0x5562579da3d0 2024-05-10T11:18:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.536+0000 7f8922000700 1 -- 172.21.15.27:0/3099119486 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8924030050 con 0x5562579da3d0 2024-05-10T11:18:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.536+0000 7f8949694b80 10 client.26069 fetch_fsmap learned FSMap version 33 2024-05-10T11:18:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.536+0000 7f8949694b80 1 -- 172.21.15.27:0/3099119486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5562579ec000 con 0x5562579da3d0 2024-05-10T11:18:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.536+0000 7f8922000700 1 -- 172.21.15.27:0/3099119486 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89300ad720 con 0x5562579da3d0 2024-05-10T11:18:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.536+0000 7f8922000700 1 -- 172.21.15.27:0/3099119486 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f8924027c40 con 0x5562579da3d0 2024-05-10T11:18:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.536+0000 7f8949694b80 10 client.26069 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:18:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.536+0000 7f8949694b80 10 client.26069 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:18:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.537+0000 7f8949694b80 20 client.26069 populate_metadata read hostname 'smithi027' 2024-05-10T11:18:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.537+0000 7f8949694b80 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5562579ee640 0x5562579f0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.537+0000 7f8949694b80 4 client.26069 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:18:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.537+0000 7f8949694b80 1 -- 172.21.15.27:0/3099119486 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556257999140 con 0x5562579ee640 2024-05-10T11:18:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.537+0000 7f8937e00700 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5562579ee640 0x5562579f0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.537+0000 7f8937e00700 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5562579ee640 0x5562579f0a50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95cd67b969764d53 server_cookie=42b1047b5c77878f in_seq=0 out_seq=0 2024-05-10T11:18:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.537+0000 7f8922000700 10 client.26069 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8922000700 1 -- 172.21.15.27:0/3099119486 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556257999140 con 0x5562579ee640 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8922000700 10 client.26069 handle_command_reply: tid=0 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8949694b80 1 client.26069 shutdown 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8949694b80 1 -- 172.21.15.27:0/3099119486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 msgr2=0x5562579ddde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8949694b80 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 0x5562579ddde0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f892400fa40 tx=0x7f89240047c0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8949694b80 1 -- 172.21.15.27:0/3099119486 shutdown_connections 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8949694b80 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5562579ee640 0x5562579f0a50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8922000700 0 client.26069 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8949694b80 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579d5100 0x5562579e0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8949694b80 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579dacf0 0x5562579de320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8949694b80 1 --2- 172.21.15.27:0/3099119486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 0x5562579ddde0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8949694b80 1 -- 172.21.15.27:0/3099119486 >> 172.21.15.27:0/3099119486 conn(0x5562579446d0 msgr2=0x5562579c6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.539+0000 7f8949694b80 1 -- 172.21.15.27:0/3099119486 shutdown_connections 2024-05-10T11:18:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.540+0000 7f8949694b80 1 -- 172.21.15.27:0/3099119486 wait complete. 2024-05-10T11:18:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.540+0000 7f8949694b80 20 client.26069 trim_cache size 0 max 16384 2024-05-10T11:18:03.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.557+0000 7f8949694b80 1 Processor -- start 2024-05-10T11:18:03.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.558+0000 7f8949694b80 1 -- start start 2024-05-10T11:18:03.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.558+0000 7f8949694b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579d5100 0x5562579d4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.558+0000 7f8949694b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579da3d0 0x5562579ea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.558+0000 7f8949694b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579dce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.558+0000 7f8949694b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f89381ad4d0 con 0x5562579da3d0 2024-05-10T11:18:03.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.558+0000 7f8949694b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f89381af010 con 0x5562579d5100 2024-05-10T11:18:03.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.558+0000 7f8949694b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f89381adba0 con 0x5562579dacf0 2024-05-10T11:18:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.558+0000 7f893ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579dce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.558+0000 7f8937e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579d5100 0x5562579d4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.558+0000 7f893ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579dce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37484/0 (socket says 172.21.15.27:37484) 2024-05-10T11:18:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.558+0000 7f893ce00700 1 -- 172.21.15.27:0/3115351345 learned_addr learned my addr 172.21.15.27:0/3115351345 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.558+0000 7f8937400700 1 --2- 172.21.15.27:0/3115351345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579da3d0 0x5562579ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.559+0000 7f893ce00700 1 -- 172.21.15.27:0/3115351345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579d5100 msgr2=0x5562579d4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.559+0000 7f893ce00700 1 --2- 172.21.15.27:0/3115351345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579d5100 0x5562579d4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.559+0000 7f893ce00700 1 -- 172.21.15.27:0/3115351345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579da3d0 msgr2=0x5562579ea760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.559+0000 7f893ce00700 1 --2- 172.21.15.27:0/3115351345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579da3d0 0x5562579ea760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.559+0000 7f893ce00700 1 -- 172.21.15.27:0/3115351345 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562579daa00 con 0x5562579dacf0 2024-05-10T11:18:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.559+0000 7f893ce00700 1 --2- 172.21.15.27:0/3115351345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579dce40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f89300a53f0 tx=0x7f893000c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60adde12eae75442 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.559+0000 7f892cc00700 1 -- 172.21.15.27:0/3115351345 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89300b2140 con 0x5562579dacf0 2024-05-10T11:18:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.559+0000 7f892cc00700 1 -- 172.21.15.27:0/3115351345 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8930004d60 con 0x5562579dacf0 2024-05-10T11:18:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.559+0000 7f892cc00700 1 -- 172.21.15.27:0/3115351345 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89300a4f40 con 0x5562579dacf0 2024-05-10T11:18:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.560+0000 7f8949694b80 1 -- 172.21.15.27:0/3115351345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 msgr2=0x5562579dce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.560+0000 7f8949694b80 1 --2- 172.21.15.27:0/3115351345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579dce40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f89300a53f0 tx=0x7f893000c9f0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.560+0000 7f8949694b80 1 -- 172.21.15.27:0/3115351345 shutdown_connections 2024-05-10T11:18:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.560+0000 7f8949694b80 1 --2- 172.21.15.27:0/3115351345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579dce40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.560+0000 7f8949694b80 1 --2- 172.21.15.27:0/3115351345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579d5100 0x5562579d4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.560+0000 7f8949694b80 1 --2- 172.21.15.27:0/3115351345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579da3d0 0x5562579ea760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.560+0000 7f8949694b80 1 -- 172.21.15.27:0/3115351345 >> 172.21.15.27:0/3115351345 conn(0x5562579446d0 msgr2=0x5562579dc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.560+0000 7f8949694b80 1 -- 172.21.15.27:0/3115351345 shutdown_connections 2024-05-10T11:18:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.560+0000 7f8949694b80 1 -- 172.21.15.27:0/3115351345 wait complete. 2024-05-10T11:18:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.561+0000 7f8949694b80 1 -- start start 2024-05-10T11:18:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.561+0000 7f8949694b80 1 Processor -- start 2024-05-10T11:18:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.561+0000 7f8949694b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579d5100 0x5562579e70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.561+0000 7f8949694b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 0x5562579e2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.561+0000 7f8949694b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579e29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.561+0000 7f8949694b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f89381ae940 con 0x5562579d5100 2024-05-10T11:18:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.561+0000 7f8949694b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f89381ae270 con 0x5562579da3d0 2024-05-10T11:18:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.561+0000 7f8949694b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f89381ab470 con 0x5562579dacf0 2024-05-10T11:18:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.561+0000 7f8949694b80 10 client.0 _ll_register_callbacks cb 0x5562578c5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:18:03.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.562+0000 7f8937400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 0x5562579e2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.562+0000 7f8937400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 0x5562579e2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40372/0 (socket says 172.21.15.27:40372) 2024-05-10T11:18:03.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.562+0000 7f8937400700 1 -- 172.21.15.27:0/3548560271 learned_addr learned my addr 172.21.15.27:0/3548560271 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:03.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.562+0000 7f893ce00700 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579e29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.562+0000 7f8937e00700 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579d5100 0x5562579e70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.562+0000 7f8937400700 1 -- 172.21.15.27:0/3548560271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 msgr2=0x5562579e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:03.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.562+0000 7f8937400700 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579e29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.562+0000 7f8937400700 1 -- 172.21.15.27:0/3548560271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579d5100 msgr2=0x5562579e70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.562+0000 7f8937400700 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579d5100 0x5562579e70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.562+0000 7f8937400700 1 -- 172.21.15.27:0/3548560271 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562579e31e0 con 0x5562579da3d0 2024-05-10T11:18:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.563+0000 7f8937400700 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 0x5562579e2420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8924009740 tx=0x7f89240094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=899171aa313731cb server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.563+0000 7f8922000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:18:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.564+0000 7f8922000700 1 -- 172.21.15.27:0/3548560271 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924022050 con 0x5562579da3d0 2024-05-10T11:18:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.564+0000 7f8949694b80 1 -- 172.21.15.27:0/3548560271 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8930004d60 con 0x5562579da3d0 2024-05-10T11:18:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.564+0000 7f8922000700 1 -- 172.21.15.27:0/3548560271 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f892400fe00 con 0x5562579da3d0 2024-05-10T11:18:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.564+0000 7f8922000700 1 -- 172.21.15.27:0/3548560271 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8924017030 con 0x5562579da3d0 2024-05-10T11:18:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.564+0000 7f8949694b80 10 client.26075 fetch_fsmap learned FSMap version 33 2024-05-10T11:18:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.564+0000 7f8949694b80 1 -- 172.21.15.27:0/3548560271 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556257a58f80 con 0x5562579da3d0 2024-05-10T11:18:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.565+0000 7f8922000700 1 -- 172.21.15.27:0/3548560271 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8930004d60 con 0x5562579da3d0 2024-05-10T11:18:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.565+0000 7f8922000700 1 -- 172.21.15.27:0/3548560271 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f8924033030 con 0x5562579da3d0 2024-05-10T11:18:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.565+0000 7f8949694b80 10 client.26075 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:18:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.565+0000 7f8949694b80 10 client.26075 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:18:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.565+0000 7f8949694b80 20 client.26075 populate_metadata read hostname 'smithi027' 2024-05-10T11:18:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.565+0000 7f8949694b80 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5562579ee640 0x556257a5b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.565+0000 7f8949694b80 4 client.26075 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:18:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.565+0000 7f8949694b80 1 -- 172.21.15.27:0/3548560271 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55625792c400 con 0x5562579ee640 2024-05-10T11:18:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.566+0000 7f8937e00700 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5562579ee640 0x556257a5b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:03.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.566+0000 7f8937e00700 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5562579ee640 0x556257a5b6c0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f7766d25224b4e server_cookie=9b8e94386d122cf0 in_seq=0 out_seq=0 2024-05-10T11:18:03.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.566+0000 7f8922000700 10 client.26075 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:03.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.567+0000 7f8922000700 1 -- 172.21.15.27:0/3548560271 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55625792c400 con 0x5562579ee640 2024-05-10T11:18:03.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.567+0000 7f8922000700 10 client.26075 handle_command_reply: tid=0 2024-05-10T11:18:03.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.567+0000 7f8949694b80 1 client.26075 shutdown 2024-05-10T11:18:03.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.568+0000 7f8949694b80 1 -- 172.21.15.27:0/3548560271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 msgr2=0x5562579e2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:03.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.568+0000 7f8949694b80 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 0x5562579e2420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8924009740 tx=0x7f89240094b0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.568+0000 7f8949694b80 1 -- 172.21.15.27:0/3548560271 shutdown_connections 2024-05-10T11:18:03.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.568+0000 7f8949694b80 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5562579ee640 0x556257a5b6c0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.568+0000 7f8949694b80 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562579dacf0 0x5562579e29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.568+0000 7f8949694b80 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5562579d5100 0x5562579e70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.568+0000 7f8949694b80 1 --2- 172.21.15.27:0/3548560271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5562579da3d0 0x5562579e2420 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.568+0000 7f8949694b80 1 -- 172.21.15.27:0/3548560271 >> 172.21.15.27:0/3548560271 conn(0x5562579446d0 msgr2=0x5562579d9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:03.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.568+0000 7f8922000700 0 client.26075 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:03.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.568+0000 7f8949694b80 1 -- 172.21.15.27:0/3548560271 shutdown_connections 2024-05-10T11:18:03.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.568+0000 7f8949694b80 1 -- 172.21.15.27:0/3548560271 wait complete. 2024-05-10T11:18:03.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.568+0000 7f8949694b80 20 client.26075 trim_cache size 0 max 16384 2024-05-10T11:18:03.570 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:18:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.569+0000 7f893ee00700 1 -- 172.21.15.27:0/1793579022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f891c0b2a70 msgr2=0x7f891c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.569+0000 7f893ee00700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f891c0b2a70 0x7f891c0b4f30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f893807ac40 tx=0x7f8924011720 comp rx=0 tx=0).stop 2024-05-10T11:18:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.569+0000 7f893ee00700 1 -- 172.21.15.27:0/1793579022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8938109540 msgr2=0x7f8938076220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.569+0000 7f893ee00700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8938109540 0x7f8938076220 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f893000ef30 tx=0x7f893000c3a0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.569+0000 7f893ee00700 1 -- 172.21.15.27:0/1793579022 shutdown_connections 2024-05-10T11:18:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.569+0000 7f893ee00700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f891c0b2a70 0x7f891c0b4f30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.569+0000 7f893ee00700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938103560 0x7f89380757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.569+0000 7f893ee00700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8938109540 0x7f8938076220 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.569+0000 7f893ee00700 1 --2- 172.21.15.27:0/1793579022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8938103f20 0x7f8938075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.569+0000 7f893ee00700 1 -- 172.21.15.27:0/1793579022 >> 172.21.15.27:0/1793579022 conn(0x7f893807a660 msgr2=0x7f8938102200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.569+0000 7f893ee00700 1 -- 172.21.15.27:0/1793579022 shutdown_connections 2024-05-10T11:18:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:03.569+0000 7f893ee00700 1 -- 172.21.15.27:0/1793579022 wait complete. 2024-05-10T11:18:03.587 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:03 smithi027 ceph-mon[30054]: pgmap v148: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 168 op/s 2024-05-10T11:18:03.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 237456 7.86 MB/sec execute 48 sec latency 10.694 ms 2024-05-10T11:18:03.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:03 smithi076 ceph-mon[34152]: pgmap v148: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 168 op/s 2024-05-10T11:18:03.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:03 smithi189 ceph-mon[34842]: pgmap v148: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 168 op/s 2024-05-10T11:18:04.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 238165 7.80 MB/sec execute 49 sec latency 17.842 ms 2024-05-10T11:18:05.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:05 smithi027 ceph-mon[30054]: pgmap v149: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 277 op/s 2024-05-10T11:18:05.650 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:18:05.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:18:05.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 240103 7.79 MB/sec execute 50 sec latency 39.052 ms 2024-05-10T11:18:05.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:05 smithi076 ceph-mon[34152]: pgmap v149: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 277 op/s 2024-05-10T11:18:05.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:05 smithi189 ceph-mon[34842]: pgmap v149: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 277 op/s 2024-05-10T11:18:06.177 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:18:06.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 241608 7.87 MB/sec execute 51 sec latency 73.597 ms 2024-05-10T11:18:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:07 smithi027 ceph-mon[30054]: pgmap v150: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 192 op/s 2024-05-10T11:18:07.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 242648 7.82 MB/sec execute 52 sec latency 50.266 ms 2024-05-10T11:18:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:07 smithi076 ceph-mon[34152]: pgmap v150: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 192 op/s 2024-05-10T11:18:07.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:07 smithi189 ceph-mon[34842]: pgmap v150: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 192 op/s 2024-05-10T11:18:08.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 244904 7.91 MB/sec execute 53 sec latency 14.517 ms 2024-05-10T11:18:08.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.683+0000 7f300de00700 1 -- 172.21.15.27:0/3320108155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f300810fd50 msgr2=0x7f3008112140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:08.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.683+0000 7f300de00700 1 --2- 172.21.15.27:0/3320108155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f300810fd50 0x7f3008112140 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f30000098f0 tx=0x7f3000009c00 comp rx=0 tx=0).stop 2024-05-10T11:18:08.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.683+0000 7f300de00700 1 -- 172.21.15.27:0/3320108155 shutdown_connections 2024-05-10T11:18:08.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.683+0000 7f300de00700 1 --2- 172.21.15.27:0/3320108155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f300810fd50 0x7f3008112140 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.683+0000 7f300de00700 1 --2- 172.21.15.27:0/3320108155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30081021a0 0x7f3008102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.683+0000 7f300de00700 1 --2- 172.21.15.27:0/3320108155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3008102b50 0x7f300810f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.683+0000 7f300de00700 1 -- 172.21.15.27:0/3320108155 >> 172.21.15.27:0/3320108155 conn(0x7f30080fddc0 msgr2=0x7f30081001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:08.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.684+0000 7f300de00700 1 -- 172.21.15.27:0/3320108155 shutdown_connections 2024-05-10T11:18:08.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.684+0000 7f300de00700 1 -- 172.21.15.27:0/3320108155 wait complete. 2024-05-10T11:18:08.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.685+0000 7f300de00700 1 Processor -- start 2024-05-10T11:18:08.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.685+0000 7f300de00700 1 -- start start 2024-05-10T11:18:08.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.685+0000 7f300de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30081021a0 0x7f30081a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f300de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3008102b50 0x7f30081a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f300de00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f300810fd50 0x7f300819bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f300de00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f30081157f0 con 0x7f30081021a0 2024-05-10T11:18:08.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f300de00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3008115670 con 0x7f300810fd50 2024-05-10T11:18:08.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f300de00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3008115970 con 0x7f3008102b50 2024-05-10T11:18:08.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f3007e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3008102b50 0x7f30081a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f300ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30081021a0 0x7f30081a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f3007e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3008102b50 0x7f30081a1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48798/0 (socket says 172.21.15.27:48798) 2024-05-10T11:18:08.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f300ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30081021a0 0x7f30081a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49560/0 (socket says 172.21.15.27:49560) 2024-05-10T11:18:08.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f3007e00700 1 -- 172.21.15.27:0/4110963063 learned_addr learned my addr 172.21.15.27:0/4110963063 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:08.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f300d400700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f300810fd50 0x7f300819bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f3007e00700 1 -- 172.21.15.27:0/4110963063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f300810fd50 msgr2=0x7f300819bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:08.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f3007e00700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f300810fd50 0x7f300819bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f3007e00700 1 -- 172.21.15.27:0/4110963063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30081021a0 msgr2=0x7f30081a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:08.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f3007e00700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30081021a0 0x7f30081a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.686+0000 7f3007e00700 1 -- 172.21.15.27:0/4110963063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f300819c3f0 con 0x7f3008102b50 2024-05-10T11:18:08.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.687+0000 7f300ca00700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30081021a0 0x7f30081a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:18:08.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.687+0000 7f3007e00700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3008102b50 0x7f30081a1f80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400e9a0 tx=0x7f2ff400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:08.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.687+0000 7f3005600700 1 -- 172.21.15.27:0/4110963063 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff400ca10 con 0x7f3008102b50 2024-05-10T11:18:08.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.687+0000 7f300de00700 1 -- 172.21.15.27:0/4110963063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f300819c6e0 con 0x7f3008102b50 2024-05-10T11:18:08.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.687+0000 7f300de00700 1 -- 172.21.15.27:0/4110963063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30081a89c0 con 0x7f3008102b50 2024-05-10T11:18:08.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.687+0000 7f3005600700 1 -- 172.21.15.27:0/4110963063 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ff400cbb0 con 0x7f3008102b50 2024-05-10T11:18:08.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.687+0000 7f3005600700 1 -- 172.21.15.27:0/4110963063 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4018680 con 0x7f3008102b50 2024-05-10T11:18:08.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.691+0000 7f3005600700 1 -- 172.21.15.27:0/4110963063 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2ff4010430 con 0x7f3008102b50 2024-05-10T11:18:08.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.692+0000 7f3019d8ab80 1 Processor -- start 2024-05-10T11:18:08.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.692+0000 7f3019d8ab80 1 -- start start 2024-05-10T11:18:08.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f3005600700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2ff00b29f0 0x7f2ff00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f3019d8ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e88a60 0x5582e7e88e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f3019d8ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e89380 0x5582e7e83060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f3019d8ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 0x5582e7e83b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f3019d8ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5582e7dd5aa0 con 0x5582e7e88a60 2024-05-10T11:18:08.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f3019d8ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5582e7d737e0 con 0x5582e7e83680 2024-05-10T11:18:08.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f3019d8ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5582e7dd1a60 con 0x5582e7e89380 2024-05-10T11:18:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f3005600700 1 -- 172.21.15.27:0/4110963063 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f2ff40e2a90 con 0x7f3008102b50 2024-05-10T11:18:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f300d400700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2ff00b29f0 0x7f2ff00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f300ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e88a60 0x5582e7e88e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f300ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e88a60 0x5582e7e88e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49572/0 (socket says 172.21.15.27:49572) 2024-05-10T11:18:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f300ca00700 1 -- 172.21.15.27:0/900403115 learned_addr learned my addr 172.21.15.27:0/900403115 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.693+0000 7f300d400700 1 --2- 172.21.15.27:0/900403115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 0x5582e7e83b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.694+0000 7f3007e00700 1 --2- 172.21.15.27:0/900403115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e89380 0x5582e7e83060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.694+0000 7f300d400700 1 -- 172.21.15.27:0/900403115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e89380 msgr2=0x5582e7e83060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.694+0000 7f300d400700 1 --2- 172.21.15.27:0/900403115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e89380 0x5582e7e83060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.694+0000 7f300d400700 1 -- 172.21.15.27:0/900403115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e88a60 msgr2=0x5582e7e88e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.694+0000 7f300d400700 1 --2- 172.21.15.27:0/900403115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e88a60 0x5582e7e88e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.694+0000 7f300d400700 1 -- 172.21.15.27:0/900403115 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582e7d22950 con 0x5582e7e83680 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.694+0000 7f300d400700 1 --2- 172.21.15.27:0/900403115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 0x5582e7e83b00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3000009230 tx=0x7f300000f670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bad631441d231824 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.695+0000 7f300d400700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2ff00b29f0 0x7f2ff00b4eb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f300806ada0 tx=0x7f300001b000 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.695+0000 7f2ffa000700 1 -- 172.21.15.27:0/900403115 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300002b070 con 0x5582e7e83680 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.695+0000 7f2ffa000700 1 -- 172.21.15.27:0/900403115 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3000021040 con 0x5582e7e83680 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.695+0000 7f2ffa000700 1 -- 172.21.15.27:0/900403115 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30000039b0 con 0x5582e7e83680 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.696+0000 7f3019d8ab80 1 -- 172.21.15.27:0/900403115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 msgr2=0x5582e7e83b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.696+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/900403115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 0x5582e7e83b00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3000009230 tx=0x7f300000f670 comp rx=0 tx=0).stop 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.696+0000 7f3019d8ab80 1 -- 172.21.15.27:0/900403115 shutdown_connections 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.696+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/900403115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 0x5582e7e83b00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.696+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/900403115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e88a60 0x5582e7e88e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.696+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/900403115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e89380 0x5582e7e83060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.696+0000 7f3019d8ab80 1 -- 172.21.15.27:0/900403115 >> 172.21.15.27:0/900403115 conn(0x5582e7db1be0 msgr2=0x5582e7e7da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:08.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.696+0000 7f3019d8ab80 1 -- 172.21.15.27:0/900403115 shutdown_connections 2024-05-10T11:18:08.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.696+0000 7f3019d8ab80 1 -- 172.21.15.27:0/900403115 wait complete. 2024-05-10T11:18:08.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.697+0000 7f3019d8ab80 1 -- start start 2024-05-10T11:18:08.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.697+0000 7f3019d8ab80 1 Processor -- start 2024-05-10T11:18:08.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f3019d8ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 0x5582e7e8ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f3019d8ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e88a60 0x5582e7e8c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f3019d8ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e89380 0x5582e7e8c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f3019d8ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5582e7e8fdd0 con 0x5582e7e89380 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f3019d8ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5582e7d3bba0 con 0x5582e7e83680 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f3019d8ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5582e7e8ff50 con 0x5582e7e88a60 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f3019d8ab80 10 client.0 _ll_register_callbacks cb 0x5582e7d7ccf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f300ca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 0x5582e7e8ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f300ca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 0x5582e7e8ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55198/0 (socket says 172.21.15.27:55198) 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f300ca00700 1 -- 172.21.15.27:0/79370883 learned_addr learned my addr 172.21.15.27:0/79370883 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f300d400700 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e89380 0x5582e7e8c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f3007e00700 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e88a60 0x5582e7e8c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f300ca00700 1 -- 172.21.15.27:0/79370883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e88a60 msgr2=0x5582e7e8c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f300ca00700 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e88a60 0x5582e7e8c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f300ca00700 1 -- 172.21.15.27:0/79370883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e89380 msgr2=0x5582e7e8c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f300ca00700 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e89380 0x5582e7e8c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f300ca00700 1 -- 172.21.15.27:0/79370883 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582e7d9b4a0 con 0x5582e7e83680 2024-05-10T11:18:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.698+0000 7f300d400700 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e89380 0x5582e7e8c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:18:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.699+0000 7f300ca00700 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 0x5582e7e8ef90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00ecd0 tx=0x7f2ffc0074a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3667c33ed2e8686 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.699+0000 7f2feea00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:18:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.699+0000 7f2feea00700 1 -- 172.21.15.27:0/79370883 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc01a070 con 0x5582e7e83680 2024-05-10T11:18:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.699+0000 7f2feea00700 1 -- 172.21.15.27:0/79370883 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f2ffc003c00 con 0x5582e7e83680 2024-05-10T11:18:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.699+0000 7f3019d8ab80 1 -- 172.21.15.27:0/79370883 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3000021040 con 0x5582e7e83680 2024-05-10T11:18:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.699+0000 7f2feea00700 1 -- 172.21.15.27:0/79370883 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2ffc026050 con 0x5582e7e83680 2024-05-10T11:18:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.699+0000 7f3019d8ab80 10 client.26087 fetch_fsmap learned FSMap version 33 2024-05-10T11:18:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.699+0000 7f3019d8ab80 1 -- 172.21.15.27:0/79370883 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5582e7e9a5b0 con 0x5582e7e83680 2024-05-10T11:18:08.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.700+0000 7f2feea00700 1 -- 172.21.15.27:0/79370883 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3000021040 con 0x5582e7e83680 2024-05-10T11:18:08.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.700+0000 7f2feea00700 1 -- 172.21.15.27:0/79370883 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f2ffc0166a0 con 0x5582e7e83680 2024-05-10T11:18:08.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.700+0000 7f3019d8ab80 10 client.26087 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:18:08.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.700+0000 7f3019d8ab80 10 client.26087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:18:08.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.700+0000 7f3019d8ab80 20 client.26087 populate_metadata read hostname 'smithi027' 2024-05-10T11:18:08.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.700+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5582e7e9cc40 0x5582e7e9f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.700+0000 7f3019d8ab80 4 client.26087 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:18:08.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.700+0000 7f3019d8ab80 1 -- 172.21.15.27:0/79370883 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582e7e50120 con 0x5582e7e9cc40 2024-05-10T11:18:08.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.700+0000 7f3007e00700 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5582e7e9cc40 0x5582e7e9f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.701+0000 7f3007e00700 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5582e7e9cc40 0x5582e7e9f0e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef52c60209440a33 server_cookie=8cba54cc87438482 in_seq=0 out_seq=0 2024-05-10T11:18:08.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.701+0000 7f2feea00700 10 client.26087 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:08.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f2feea00700 1 -- 172.21.15.27:0/79370883 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5582e7e50120 con 0x5582e7e9cc40 2024-05-10T11:18:08.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f2feea00700 10 client.26087 handle_command_reply: tid=0 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f3019d8ab80 1 client.26087 shutdown 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f3019d8ab80 1 -- 172.21.15.27:0/79370883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 msgr2=0x5582e7e8ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 0x5582e7e8ef90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00ecd0 tx=0x7f2ffc0074a0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f3019d8ab80 1 -- 172.21.15.27:0/79370883 shutdown_connections 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5582e7e9cc40 0x5582e7e9f0e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e89380 0x5582e7e8c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e83680 0x5582e7e8ef90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f2feea00700 0 client.26087 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/79370883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e88a60 0x5582e7e8c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f3019d8ab80 1 -- 172.21.15.27:0/79370883 >> 172.21.15.27:0/79370883 conn(0x5582e7db1be0 msgr2=0x5582e7e784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f3019d8ab80 1 -- 172.21.15.27:0/79370883 shutdown_connections 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.702+0000 7f3019d8ab80 1 -- 172.21.15.27:0/79370883 wait complete. 2024-05-10T11:18:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.703+0000 7f3019d8ab80 20 client.26087 trim_cache size 0 max 16384 2024-05-10T11:18:08.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.726+0000 7f3019d8ab80 1 Processor -- start 2024-05-10T11:18:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.726+0000 7f3019d8ab80 1 -- start start 2024-05-10T11:18:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.727+0000 7f3019d8ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e83680 0x5582e7e83060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.727+0000 7f3019d8ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e88a60 0x5582e7e98ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.727+0000 7f3019d8ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e89380 0x5582e7e8b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.727+0000 7f3019d8ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe800ef00 con 0x5582e7e89380 2024-05-10T11:18:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.727+0000 7f3019d8ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe8010a40 con 0x5582e7e88a60 2024-05-10T11:18:08.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.727+0000 7f3019d8ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe800f5d0 con 0x5582e7e83680 2024-05-10T11:18:08.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.727+0000 7f300ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e83680 0x5582e7e83060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.727+0000 7f3007e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e88a60 0x5582e7e98ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.727+0000 7f3007e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e88a60 0x5582e7e98ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55210/0 (socket says 172.21.15.27:55210) 2024-05-10T11:18:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.727+0000 7f300ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e83680 0x5582e7e83060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48822/0 (socket says 172.21.15.27:48822) 2024-05-10T11:18:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.727+0000 7f3007e00700 1 -- 172.21.15.27:0/714858723 learned_addr learned my addr 172.21.15.27:0/714858723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.728+0000 7f3007e00700 1 -- 172.21.15.27:0/714858723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e83680 msgr2=0x5582e7e83060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.728+0000 7f3007e00700 1 --2- 172.21.15.27:0/714858723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e83680 0x5582e7e83060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.728+0000 7f3007e00700 1 -- 172.21.15.27:0/714858723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e89380 msgr2=0x5582e7e8b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.728+0000 7f300d400700 1 --2- 172.21.15.27:0/714858723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e89380 0x5582e7e8b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.728+0000 7f3007e00700 1 --2- 172.21.15.27:0/714858723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e89380 0x5582e7e8b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.728+0000 7f3007e00700 1 -- 172.21.15.27:0/714858723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582e7e8f300 con 0x5582e7e88a60 2024-05-10T11:18:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.728+0000 7f300d400700 1 --2- 172.21.15.27:0/714858723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e89380 0x5582e7e8b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:18:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.728+0000 7f3007e00700 1 --2- 172.21.15.27:0/714858723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e88a60 0x5582e7e98ec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40096b0 tx=0x7f2ff40a9790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fe1051b5ca9f565 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.728+0000 7f2ff9600700 1 -- 172.21.15.27:0/714858723 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff40a9940 con 0x5582e7e88a60 2024-05-10T11:18:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.728+0000 7f2ff9600700 1 -- 172.21.15.27:0/714858723 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ff40adcf0 con 0x5582e7e88a60 2024-05-10T11:18:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.728+0000 7f2ff9600700 1 -- 172.21.15.27:0/714858723 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff40a1590 con 0x5582e7e88a60 2024-05-10T11:18:08.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.729+0000 7f3019d8ab80 1 -- 172.21.15.27:0/714858723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e88a60 msgr2=0x5582e7e98ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:08.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.729+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/714858723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e88a60 0x5582e7e98ec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40096b0 tx=0x7f2ff40a9790 comp rx=0 tx=0).stop 2024-05-10T11:18:08.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.729+0000 7f3019d8ab80 1 -- 172.21.15.27:0/714858723 shutdown_connections 2024-05-10T11:18:08.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.729+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/714858723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e83680 0x5582e7e83060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.729+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/714858723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e89380 0x5582e7e8b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.729+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/714858723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e88a60 0x5582e7e98ec0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.729+0000 7f3019d8ab80 1 -- 172.21.15.27:0/714858723 >> 172.21.15.27:0/714858723 conn(0x5582e7db1be0 msgr2=0x5582e7e77c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:08.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.729+0000 7f3019d8ab80 1 -- 172.21.15.27:0/714858723 shutdown_connections 2024-05-10T11:18:08.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.729+0000 7f3019d8ab80 1 -- 172.21.15.27:0/714858723 wait complete. 2024-05-10T11:18:08.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.730+0000 7f3019d8ab80 1 -- start start 2024-05-10T11:18:08.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.730+0000 7f3019d8ab80 1 Processor -- start 2024-05-10T11:18:08.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3019d8ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e83680 0x5582e7e95420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3019d8ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e88a60 0x5582e7e95960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3019d8ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e89380 0x5582e7e92250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3019d8ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe8010370 con 0x5582e7e83680 2024-05-10T11:18:08.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3019d8ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe800fca0 con 0x5582e7e89380 2024-05-10T11:18:08.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3019d8ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe800cea0 con 0x5582e7e88a60 2024-05-10T11:18:08.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3019d8ab80 10 client.0 _ll_register_callbacks cb 0x5582e7d7ccf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:18:08.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3007e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e88a60 0x5582e7e95960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f300ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e83680 0x5582e7e95420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3007e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e88a60 0x5582e7e95960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48828/0 (socket says 172.21.15.27:48828) 2024-05-10T11:18:08.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3007e00700 1 -- 172.21.15.27:0/917206920 learned_addr learned my addr 172.21.15.27:0/917206920 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f300d400700 1 --2- 172.21.15.27:0/917206920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e89380 0x5582e7e92250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3007e00700 1 -- 172.21.15.27:0/917206920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e89380 msgr2=0x5582e7e92250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3007e00700 1 --2- 172.21.15.27:0/917206920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e89380 0x5582e7e92250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3007e00700 1 -- 172.21.15.27:0/917206920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e83680 msgr2=0x5582e7e95420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3007e00700 1 --2- 172.21.15.27:0/917206920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e83680 0x5582e7e95420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.731+0000 7f3007e00700 1 -- 172.21.15.27:0/917206920 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582e7e92a90 con 0x5582e7e88a60 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.732+0000 7f3007e00700 1 --2- 172.21.15.27:0/917206920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e88a60 0x5582e7e95960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40a9640 tx=0x7f2ff400cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73d679943c37e17e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.732+0000 7f2feea00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.732+0000 7f2feea00700 1 -- 172.21.15.27:0/917206920 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff408fe60 con 0x5582e7e88a60 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.732+0000 7f2feea00700 1 -- 172.21.15.27:0/917206920 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f2ff40e0030 con 0x5582e7e88a60 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.732+0000 7f3019d8ab80 1 -- 172.21.15.27:0/917206920 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ff40adcf0 con 0x5582e7e88a60 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.732+0000 7f2feea00700 1 -- 172.21.15.27:0/917206920 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2ff40e0230 con 0x5582e7e88a60 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.732+0000 7f3019d8ab80 10 client.26083 fetch_fsmap learned FSMap version 33 2024-05-10T11:18:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.732+0000 7f3019d8ab80 1 -- 172.21.15.27:0/917206920 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5582e7f108f0 con 0x5582e7e88a60 2024-05-10T11:18:08.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.733+0000 7f2feea00700 1 -- 172.21.15.27:0/917206920 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ff40adcf0 con 0x5582e7e88a60 2024-05-10T11:18:08.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.733+0000 7f2feea00700 1 -- 172.21.15.27:0/917206920 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f2ff40a9ba0 con 0x5582e7e88a60 2024-05-10T11:18:08.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.733+0000 7f3019d8ab80 10 client.26083 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:18:08.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.733+0000 7f3019d8ab80 10 client.26083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:18:08.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.733+0000 7f3019d8ab80 20 client.26083 populate_metadata read hostname 'smithi027' 2024-05-10T11:18:08.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.733+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/917206920 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5582e7f10fe0 0x5582e7f13400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:08.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.733+0000 7f3019d8ab80 4 client.26083 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:18:08.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.733+0000 7f3019d8ab80 1 -- 172.21.15.27:0/917206920 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5582e7de3480 con 0x5582e7f10fe0 2024-05-10T11:18:08.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.733+0000 7f300ca00700 1 --2- 172.21.15.27:0/917206920 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5582e7f10fe0 0x5582e7f13400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:08.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.734+0000 7f300ca00700 1 --2- 172.21.15.27:0/917206920 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5582e7f10fe0 0x5582e7f13400 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=875e71f4f59fa638 server_cookie=c5f35454bc80f976 in_seq=0 out_seq=0 2024-05-10T11:18:08.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.734+0000 7f2feea00700 10 client.26083 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:08.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f2feea00700 1 -- 172.21.15.27:0/917206920 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5582e7de3480 con 0x5582e7f10fe0 2024-05-10T11:18:08.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f2feea00700 10 client.26083 handle_command_reply: tid=0 2024-05-10T11:18:08.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f3019d8ab80 1 client.26083 shutdown 2024-05-10T11:18:08.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f3019d8ab80 1 -- 172.21.15.27:0/917206920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e88a60 msgr2=0x5582e7e95960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:08.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/917206920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e88a60 0x5582e7e95960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40a9640 tx=0x7f2ff400cf90 comp rx=0 tx=0).stop 2024-05-10T11:18:08.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f3019d8ab80 1 -- 172.21.15.27:0/917206920 shutdown_connections 2024-05-10T11:18:08.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/917206920 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5582e7f10fe0 0x5582e7f13400 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/917206920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5582e7e89380 0x5582e7e92250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/917206920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e7e83680 0x5582e7e95420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f2feea00700 0 client.26083 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:08.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f3019d8ab80 1 --2- 172.21.15.27:0/917206920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582e7e88a60 0x5582e7e95960 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f3019d8ab80 1 -- 172.21.15.27:0/917206920 >> 172.21.15.27:0/917206920 conn(0x5582e7db1be0 msgr2=0x5582e7e784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:08.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f3019d8ab80 1 -- 172.21.15.27:0/917206920 shutdown_connections 2024-05-10T11:18:08.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.735+0000 7f3019d8ab80 1 -- 172.21.15.27:0/917206920 wait complete. 2024-05-10T11:18:08.740 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:18:08.740 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:18:08.740 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "6574fb72-bc99-44ee-b0f6-9337275ae9d1", 2024-05-10T11:18:08.740 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:18:08.740 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:18:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.736+0000 7f3019d8ab80 20 client.26083 trim_cache size 0 max 16384 2024-05-10T11:18:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.737+0000 7f300de00700 1 -- 172.21.15.27:0/4110963063 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2ff00b29f0 msgr2=0x7f2ff00b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.737+0000 7f300de00700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2ff00b29f0 0x7f2ff00b4eb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f300806ada0 tx=0x7f300001b000 comp rx=0 tx=0).stop 2024-05-10T11:18:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.737+0000 7f300de00700 1 -- 172.21.15.27:0/4110963063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3008102b50 msgr2=0x7f30081a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.737+0000 7f300de00700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3008102b50 0x7f30081a1f80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400e9a0 tx=0x7f2ff400ecb0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.737+0000 7f300de00700 1 -- 172.21.15.27:0/4110963063 shutdown_connections 2024-05-10T11:18:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.737+0000 7f300de00700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2ff00b29f0 0x7f2ff00b4eb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.737+0000 7f300de00700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f300810fd50 0x7f300819bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.737+0000 7f300de00700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30081021a0 0x7f30081a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.737+0000 7f300de00700 1 --2- 172.21.15.27:0/4110963063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3008102b50 0x7f30081a1f80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.737+0000 7f300de00700 1 -- 172.21.15.27:0/4110963063 >> 172.21.15.27:0/4110963063 conn(0x7f30080fddc0 msgr2=0x7f3008111030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.737+0000 7f300de00700 1 -- 172.21.15.27:0/4110963063 shutdown_connections 2024-05-10T11:18:08.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:08.737+0000 7f300de00700 1 -- 172.21.15.27:0/4110963063 wait complete. 2024-05-10T11:18:09.548 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:09 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:18:09.548 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:09 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:09.548 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:09 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:18:09.548 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:09 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:09.639 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:18:09.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 245528 7.79 MB/sec execute 54 sec latency 19.823 ms 2024-05-10T11:18:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:09 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:18:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:09 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:09 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:18:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:09 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:09.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:09 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:18:09.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:09 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:09.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:09 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:18:09.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:09 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:18:10.560 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:10 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:18:10.560 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:10 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:10.560 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:10 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:18:10.560 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:10 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:18:10.560 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:10 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:10.560 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:10 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:10.561 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:10 smithi027 ceph-mon[30054]: pgmap v151: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 274 op/s 2024-05-10T11:18:10.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 247785 7.87 MB/sec execute 55 sec latency 28.849 ms 2024-05-10T11:18:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:10 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:18:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:10 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:10 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:18:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:10 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:18:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:10 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:10 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:10 smithi076 ceph-mon[34152]: pgmap v151: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 274 op/s 2024-05-10T11:18:10.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:10 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:18:10.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:10 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:10.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:10 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:18:10.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:10 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:18:10.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:10 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:10.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:10 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:18:10.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:10 smithi189 ceph-mon[34842]: pgmap v151: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 274 op/s 2024-05-10T11:18:11.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 248740 7.85 MB/sec execute 56 sec latency 31.845 ms 2024-05-10T11:18:12.558 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:12 smithi027 ceph-mon[30054]: pgmap v152: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-05-10T11:18:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.557+0000 7fe5eba00700 1 -- 172.21.15.27:0/2977509593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5e4109510 msgr2=0x7fe5e41098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.557+0000 7fe5eba00700 1 --2- 172.21.15.27:0/2977509593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5e4109510 0x7fe5e41098f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800b0a0 tx=0x7fe5d800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.557+0000 7fe5eba00700 1 -- 172.21.15.27:0/2977509593 shutdown_connections 2024-05-10T11:18:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.557+0000 7fe5eba00700 1 --2- 172.21.15.27:0/2977509593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5e406bb40 0x7fe5e4110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.557+0000 7fe5eba00700 1 --2- 172.21.15.27:0/2977509593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5e4109510 0x7fe5e41098f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.557+0000 7fe5eba00700 1 --2- 172.21.15.27:0/2977509593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5e406b180 0x7fe5e406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.557+0000 7fe5eba00700 1 -- 172.21.15.27:0/2977509593 >> 172.21.15.27:0/2977509593 conn(0x7fe5e4079350 msgr2=0x7fe5e4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.557+0000 7fe5eba00700 1 -- 172.21.15.27:0/2977509593 shutdown_connections 2024-05-10T11:18:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.557+0000 7fe5eba00700 1 -- 172.21.15.27:0/2977509593 wait complete. 2024-05-10T11:18:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.559+0000 7fe5eba00700 1 Processor -- start 2024-05-10T11:18:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.559+0000 7fe5eba00700 1 -- start start 2024-05-10T11:18:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.559+0000 7fe5eba00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5e406b180 0x7fe5e419be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.559+0000 7fe5eba00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5e406bb40 0x7fe5e419c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.559+0000 7fe5eba00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5e4109510 0x7fe5e41a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.559+0000 7fe5eba00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e4113670 con 0x7fe5e4109510 2024-05-10T11:18:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.559+0000 7fe5eba00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41134f0 con 0x7fe5e406bb40 2024-05-10T11:18:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.559+0000 7fe5eba00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41137f0 con 0x7fe5e406b180 2024-05-10T11:18:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.560+0000 7fe5e9000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5e406b180 0x7fe5e419be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.560+0000 7fe5e3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5e406bb40 0x7fe5e419c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.560+0000 7fe5e3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5e406bb40 0x7fe5e419c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55222/0 (socket says 172.21.15.27:55222) 2024-05-10T11:18:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.560+0000 7fe5e3e00700 1 -- 172.21.15.27:0/3868721929 learned_addr learned my addr 172.21.15.27:0/3868721929 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.560+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5e4109510 0x7fe5e41a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.560+0000 7fe5e9000700 1 -- 172.21.15.27:0/3868721929 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5e406bb40 msgr2=0x7fe5e419c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.560+0000 7fe5e9000700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5e406bb40 0x7fe5e419c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.560+0000 7fe5e9000700 1 -- 172.21.15.27:0/3868721929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5e4109510 msgr2=0x7fe5e41a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.560+0000 7fe5e9000700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5e4109510 0x7fe5e41a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.560+0000 7fe5e9000700 1 -- 172.21.15.27:0/3868721929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e41a0e30 con 0x7fe5e406b180 2024-05-10T11:18:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.560+0000 7fe5e3e00700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5e406bb40 0x7fe5e419c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:18:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.561+0000 7fe5e9000700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5e406b180 0x7fe5e419be20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800b070 tx=0x7fe5d8004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.561+0000 7fe5e1600700 1 -- 172.21.15.27:0/3868721929 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8021070 con 0x7fe5e406b180 2024-05-10T11:18:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.561+0000 7fe5e1600700 1 -- 172.21.15.27:0/3868721929 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5d8007e30 con 0x7fe5e406b180 2024-05-10T11:18:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.561+0000 7fe5eba00700 1 -- 172.21.15.27:0/3868721929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e41a10c0 con 0x7fe5e406b180 2024-05-10T11:18:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.561+0000 7fe5e1600700 1 -- 172.21.15.27:0/3868721929 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d80046a0 con 0x7fe5e406b180 2024-05-10T11:18:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.561+0000 7fe5eba00700 1 -- 172.21.15.27:0/3868721929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5e41a89a0 con 0x7fe5e406b180 2024-05-10T11:18:12.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.566+0000 7fe5e1600700 1 -- 172.21.15.27:0/3868721929 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fe5d8007980 con 0x7fe5e406b180 2024-05-10T11:18:12.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.566+0000 7fe5f62deb80 1 Processor -- start 2024-05-10T11:18:12.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.566+0000 7fe5f62deb80 1 -- start start 2024-05-10T11:18:12.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.567+0000 7fe5f62deb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd060 0x55d3811dd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.567+0000 7fe5f62deb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 0x55d3811e1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.567+0000 7fe5f62deb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811e19d0 0x55d3811e1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.567+0000 7fe5f62deb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d38112fa20 con 0x55d3811e19d0 2024-05-10T11:18:12.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.567+0000 7fe5f62deb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d3810cd760 con 0x55d3811dd980 2024-05-10T11:18:12.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.567+0000 7fe5f62deb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d38112b9e0 con 0x55d3811dd060 2024-05-10T11:18:12.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.567+0000 7fe5e9000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 0x55d3811e1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.567+0000 7fe5e9a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd060 0x55d3811dd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.567+0000 7fe5e9a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd060 0x55d3811dd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48850/0 (socket says 172.21.15.27:48850) 2024-05-10T11:18:12.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.567+0000 7fe5e9a00700 1 -- 172.21.15.27:0/1670732343 learned_addr learned my addr 172.21.15.27:0/1670732343 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:12.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.567+0000 7fe5e3e00700 1 --2- 172.21.15.27:0/1670732343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811e19d0 0x55d3811e1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.568+0000 7fe5e9a00700 1 -- 172.21.15.27:0/1670732343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 msgr2=0x55d3811e1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:12.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.568+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/1670732343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 0x55d3811e1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.568+0000 7fe5e9a00700 1 -- 172.21.15.27:0/1670732343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811e19d0 msgr2=0x55d3811e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:12.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.568+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/1670732343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811e19d0 0x55d3811e1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.568+0000 7fe5e9a00700 1 -- 172.21.15.27:0/1670732343 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d38107c8d0 con 0x55d3811dd060 2024-05-10T11:18:12.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.568+0000 7fe5e9000700 1 --2- 172.21.15.27:0/1670732343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 0x55d3811e1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:18:12.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.568+0000 7fe5e1600700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe5cc0b2a70 0x7fe5cc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.568+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/1670732343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd060 0x55d3811dd440 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009cb0 tx=0x7fe5dc011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9084dadbc2683761 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:12.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.568+0000 7fe5e1600700 1 -- 172.21.15.27:0/3868721929 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fe5d800f020 con 0x7fe5e406b180 2024-05-10T11:18:12.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.568+0000 7fe5e9000700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe5cc0b2a70 0x7fe5cc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.569+0000 7fe5d6000700 1 -- 172.21.15.27:0/1670732343 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc011d80 con 0x55d3811dd060 2024-05-10T11:18:12.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.569+0000 7fe5d6000700 1 -- 172.21.15.27:0/1670732343 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5dc0044e0 con 0x55d3811dd060 2024-05-10T11:18:12.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.569+0000 7fe5d6000700 1 -- 172.21.15.27:0/1670732343 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc019680 con 0x55d3811dd060 2024-05-10T11:18:12.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.570+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1670732343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd060 msgr2=0x55d3811dd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:12.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.570+0000 7fe5e9000700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe5cc0b2a70 0x7fe5cc0b4f30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80e0fc0 tx=0x7fe5d8009040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:12.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.570+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/1670732343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd060 0x55d3811dd440 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009cb0 tx=0x7fe5dc011440 comp rx=0 tx=0).stop 2024-05-10T11:18:12.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.570+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1670732343 shutdown_connections 2024-05-10T11:18:12.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.570+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/1670732343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd060 0x55d3811dd440 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.570+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/1670732343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811e19d0 0x55d3811e1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.570+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/1670732343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 0x55d3811e1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.570+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1670732343 >> 172.21.15.27:0/1670732343 conn(0x55d38110bb60 msgr2=0x55d3811d77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:12.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.570+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1670732343 shutdown_connections 2024-05-10T11:18:12.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.570+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1670732343 wait complete. 2024-05-10T11:18:12.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.571+0000 7fe5f62deb80 1 -- start start 2024-05-10T11:18:12.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.571+0000 7fe5f62deb80 1 Processor -- start 2024-05-10T11:18:12.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.572+0000 7fe5f62deb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811dd060 0x55d3811e8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.572+0000 7fe5f62deb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd980 0x55d3811e61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.572+0000 7fe5f62deb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811e19d0 0x55d3811e6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.572+0000 7fe5f62deb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d3811e9be0 con 0x55d3811dd060 2024-05-10T11:18:12.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.572+0000 7fe5f62deb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d381095b20 con 0x55d3811e19d0 2024-05-10T11:18:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.572+0000 7fe5f62deb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d3811e9d60 con 0x55d3811dd980 2024-05-10T11:18:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.572+0000 7fe5f62deb80 10 client.0 _ll_register_callbacks cb 0x55d3810d6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:18:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.572+0000 7fe5e3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd980 0x55d3811e61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.572+0000 7fe5e3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd980 0x55d3811e61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48866/0 (socket says 172.21.15.27:48866) 2024-05-10T11:18:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.572+0000 7fe5e9a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811dd060 0x55d3811e8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.572+0000 7fe5e3e00700 1 -- 172.21.15.27:0/3818493386 learned_addr learned my addr 172.21.15.27:0/3818493386 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.572+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811e19d0 0x55d3811e6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.573+0000 7fe5e3e00700 1 -- 172.21.15.27:0/3818493386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811e19d0 msgr2=0x55d3811e6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.573+0000 7fe5e3e00700 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811e19d0 0x55d3811e6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.573+0000 7fe5e3e00700 1 -- 172.21.15.27:0/3818493386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811dd060 msgr2=0x55d3811e8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.573+0000 7fe5e3e00700 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811dd060 0x55d3811e8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.573+0000 7fe5e3e00700 1 -- 172.21.15.27:0/3818493386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3810f5420 con 0x55d3811dd980 2024-05-10T11:18:12.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.573+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811dd060 0x55d3811e8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:18:12.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.573+0000 7fe5e3e00700 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd980 0x55d3811e61f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe5d000e7a0 tx=0x7fe5d000eb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b0bafc40c53e11d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:12.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.573+0000 7fe5caa00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:18:12.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.573+0000 7fe5caa00700 1 -- 172.21.15.27:0/3818493386 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0004460 con 0x55d3811dd980 2024-05-10T11:18:12.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.573+0000 7fe5caa00700 1 -- 172.21.15.27:0/3818493386 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fe5d0003680 con 0x55d3811dd980 2024-05-10T11:18:12.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.574+0000 7fe5f62deb80 1 -- 172.21.15.27:0/3818493386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5dc0044e0 con 0x55d3811dd980 2024-05-10T11:18:12.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.574+0000 7fe5caa00700 1 -- 172.21.15.27:0/3818493386 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe5d0003880 con 0x55d3811dd980 2024-05-10T11:18:12.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.574+0000 7fe5f62deb80 10 client.26095 fetch_fsmap learned FSMap version 33 2024-05-10T11:18:12.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.574+0000 7fe5f62deb80 1 -- 172.21.15.27:0/3818493386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d3811f44a0 con 0x55d3811dd980 2024-05-10T11:18:12.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.574+0000 7fe5caa00700 1 -- 172.21.15.27:0/3818493386 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5dc0044e0 con 0x55d3811dd980 2024-05-10T11:18:12.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.574+0000 7fe5caa00700 1 -- 172.21.15.27:0/3818493386 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fe5d000dca0 con 0x55d3811dd980 2024-05-10T11:18:12.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.575+0000 7fe5f62deb80 10 client.26095 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:18:12.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.575+0000 7fe5f62deb80 10 client.26095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:18:12.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.575+0000 7fe5f62deb80 20 client.26095 populate_metadata read hostname 'smithi027' 2024-05-10T11:18:12.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.575+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d3811f6ae0 0x55d3811f8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.575+0000 7fe5f62deb80 4 client.26095 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:18:12.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.575+0000 7fe5f62deb80 1 -- 172.21.15.27:0/3818493386 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3811aa0a0 con 0x55d3811f6ae0 2024-05-10T11:18:12.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.575+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d3811f6ae0 0x55d3811f8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.576+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d3811f6ae0 0x55d3811f8f10 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6a336acd02c2422 server_cookie=f97bae4b67f4d65e in_seq=0 out_seq=0 2024-05-10T11:18:12.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.576+0000 7fe5caa00700 10 client.26095 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.577+0000 7fe5caa00700 1 -- 172.21.15.27:0/3818493386 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d3811aa0a0 con 0x55d3811f6ae0 2024-05-10T11:18:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.577+0000 7fe5caa00700 10 client.26095 handle_command_reply: tid=0 2024-05-10T11:18:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.577+0000 7fe5f62deb80 1 client.26095 shutdown 2024-05-10T11:18:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.577+0000 7fe5f62deb80 1 -- 172.21.15.27:0/3818493386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd980 msgr2=0x55d3811e61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.577+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd980 0x55d3811e61f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe5d000e7a0 tx=0x7fe5d000eb60 comp rx=0 tx=0).stop 2024-05-10T11:18:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.578+0000 7fe5f62deb80 1 -- 172.21.15.27:0/3818493386 shutdown_connections 2024-05-10T11:18:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.578+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d3811f6ae0 0x55d3811f8f10 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.578+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811e19d0 0x55d3811e6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.578+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811dd060 0x55d3811e8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.578+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/3818493386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd980 0x55d3811e61f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.578+0000 7fe5caa00700 0 client.26095 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.578+0000 7fe5f62deb80 1 -- 172.21.15.27:0/3818493386 >> 172.21.15.27:0/3818493386 conn(0x55d38110bb60 msgr2=0x55d3811d7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.578+0000 7fe5f62deb80 1 -- 172.21.15.27:0/3818493386 shutdown_connections 2024-05-10T11:18:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.578+0000 7fe5f62deb80 1 -- 172.21.15.27:0/3818493386 wait complete. 2024-05-10T11:18:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.578+0000 7fe5f62deb80 20 client.26095 trim_cache size 0 max 16384 2024-05-10T11:18:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.604+0000 7fe5f62deb80 1 Processor -- start 2024-05-10T11:18:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.604+0000 7fe5f62deb80 1 -- start start 2024-05-10T11:18:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.604+0000 7fe5f62deb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd060 0x55d3811e3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.605+0000 7fe5f62deb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811dd980 0x55d3811e6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.605+0000 7fe5f62deb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811e19d0 0x55d3811f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.605+0000 7fe5f62deb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41ad2b0 con 0x55d3811dd980 2024-05-10T11:18:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.605+0000 7fe5f62deb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41aedf0 con 0x55d3811dd060 2024-05-10T11:18:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.605+0000 7fe5f62deb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41ad980 con 0x55d3811e19d0 2024-05-10T11:18:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.605+0000 7fe5e9a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd060 0x55d3811e3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.605+0000 7fe5e9a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd060 0x55d3811e3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55252/0 (socket says 172.21.15.27:55252) 2024-05-10T11:18:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.605+0000 7fe5e9a00700 1 -- 172.21.15.27:0/441763362 learned_addr learned my addr 172.21.15.27:0/441763362 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.605+0000 7fe5e3e00700 1 --2- 172.21.15.27:0/441763362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811dd980 0x55d3811e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.605+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/441763362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811e19d0 0x55d3811f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.606+0000 7fe5e9a00700 1 -- 172.21.15.27:0/441763362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811e19d0 msgr2=0x55d3811f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:12.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.606+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/441763362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811e19d0 0x55d3811f2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.606+0000 7fe5e9a00700 1 -- 172.21.15.27:0/441763362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811dd980 msgr2=0x55d3811e6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:12.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.606+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/441763362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811dd980 0x55d3811e6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.606+0000 7fe5e9a00700 1 -- 172.21.15.27:0/441763362 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3811f9590 con 0x55d3811dd060 2024-05-10T11:18:12.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.606+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/441763362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd060 0x55d3811e3ea0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc005950 tx=0x7fe5dc0098c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95cf7afc79eb27b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:12.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.606+0000 7fe5d5600700 1 -- 172.21.15.27:0/441763362 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc033040 con 0x55d3811dd060 2024-05-10T11:18:12.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.606+0000 7fe5d5600700 1 -- 172.21.15.27:0/441763362 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5dc003a00 con 0x55d3811dd060 2024-05-10T11:18:12.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.606+0000 7fe5d5600700 1 -- 172.21.15.27:0/441763362 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc02e620 con 0x55d3811dd060 2024-05-10T11:18:12.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.607+0000 7fe5f62deb80 1 -- 172.21.15.27:0/441763362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd060 msgr2=0x55d3811e3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:12.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.607+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/441763362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd060 0x55d3811e3ea0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc005950 tx=0x7fe5dc0098c0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.607+0000 7fe5f62deb80 1 -- 172.21.15.27:0/441763362 shutdown_connections 2024-05-10T11:18:12.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.607+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/441763362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811e19d0 0x55d3811f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.607+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/441763362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd060 0x55d3811e3ea0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.607+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/441763362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811dd980 0x55d3811e6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.607+0000 7fe5f62deb80 1 -- 172.21.15.27:0/441763362 >> 172.21.15.27:0/441763362 conn(0x55d38110bb60 msgr2=0x55d3811d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:12.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.607+0000 7fe5f62deb80 1 -- 172.21.15.27:0/441763362 shutdown_connections 2024-05-10T11:18:12.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.607+0000 7fe5f62deb80 1 -- 172.21.15.27:0/441763362 wait complete. 2024-05-10T11:18:12.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.608+0000 7fe5f62deb80 1 -- start start 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.608+0000 7fe5f62deb80 1 Processor -- start 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5f62deb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd060 0x55d3811eae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5f62deb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 0x55d3811eb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5f62deb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811e19d0 0x55d3812603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5f62deb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41ae720 con 0x55d3811e19d0 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5f62deb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41ae050 con 0x55d3811dd980 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5f62deb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e41ab250 con 0x55d3811dd060 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5f62deb80 10 client.0 _ll_register_callbacks cb 0x55d3810d6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5e3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 0x55d3811eb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5e3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 0x55d3811eb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55268/0 (socket says 172.21.15.27:55268) 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5e3e00700 1 -- 172.21.15.27:0/1979803578 learned_addr learned my addr 172.21.15.27:0/1979803578 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd060 0x55d3811eae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811e19d0 0x55d3812603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5e3e00700 1 -- 172.21.15.27:0/1979803578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd060 msgr2=0x55d3811eae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5e3e00700 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd060 0x55d3811eae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5e3e00700 1 -- 172.21.15.27:0/1979803578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811e19d0 msgr2=0x55d3812603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5e3e00700 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811e19d0 0x55d3812603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5e3e00700 1 -- 172.21.15.27:0/1979803578 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3811d8610 con 0x55d3811dd980 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.609+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811e19d0 0x55d3812603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.610+0000 7fe5e3e00700 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 0x55d3811eb3d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0000c00 tx=0x7fe5d0003d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80d6bd83210fe59 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.610+0000 7fe5caa00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.610+0000 7fe5caa00700 1 -- 172.21.15.27:0/1979803578 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d003a020 con 0x55d3811dd980 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.610+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1979803578 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5dc003a00 con 0x55d3811dd980 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.610+0000 7fe5caa00700 1 -- 172.21.15.27:0/1979803578 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fe5d002b870 con 0x55d3811dd980 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.610+0000 7fe5caa00700 1 -- 172.21.15.27:0/1979803578 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe5d0045050 con 0x55d3811dd980 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.611+0000 7fe5f62deb80 10 client.26111 fetch_fsmap learned FSMap version 33 2024-05-10T11:18:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.611+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1979803578 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d381261830 con 0x55d3811dd980 2024-05-10T11:18:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.611+0000 7fe5caa00700 1 -- 172.21.15.27:0/1979803578 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5dc003a00 con 0x55d3811dd980 2024-05-10T11:18:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.611+0000 7fe5caa00700 1 -- 172.21.15.27:0/1979803578 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fe5d0033070 con 0x55d3811dd980 2024-05-10T11:18:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.611+0000 7fe5f62deb80 10 client.26111 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:18:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.611+0000 7fe5f62deb80 10 client.26111 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:18:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.611+0000 7fe5f62deb80 20 client.26111 populate_metadata read hostname 'smithi027' 2024-05-10T11:18:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.611+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d381261ed0 0x55d3812642f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.611+0000 7fe5f62deb80 4 client.26111 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:18:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.611+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1979803578 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d38113d400 con 0x55d381261ed0 2024-05-10T11:18:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.611+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d381261ed0 0x55d3812642f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:12.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.612+0000 7fe5e9a00700 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d381261ed0 0x55d3812642f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1b5d8290c30ce0a server_cookie=b5ef11eded4dd47 in_seq=0 out_seq=0 2024-05-10T11:18:12.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.612+0000 7fe5caa00700 10 client.26111 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:12.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.612+0000 7fe5caa00700 1 -- 172.21.15.27:0/1979803578 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d38113d400 con 0x55d381261ed0 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.612+0000 7fe5caa00700 10 client.26111 handle_command_reply: tid=0 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.612+0000 7fe5f62deb80 1 client.26111 shutdown 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.612+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1979803578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 msgr2=0x55d3811eb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.612+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 0x55d3811eb3d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0000c00 tx=0x7fe5d0003d80 comp rx=0 tx=0).stop 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.613+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1979803578 shutdown_connections 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.613+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d381261ed0 0x55d3812642f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.613+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3811dd060 0x55d3811eae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.613+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3811e19d0 0x55d3812603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.613+0000 7fe5f62deb80 1 --2- 172.21.15.27:0/1979803578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3811dd980 0x55d3811eb3d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.613+0000 7fe5caa00700 0 client.26111 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.613+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1979803578 >> 172.21.15.27:0/1979803578 conn(0x55d38110bb60 msgr2=0x55d3811d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.613+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1979803578 shutdown_connections 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.613+0000 7fe5f62deb80 1 -- 172.21.15.27:0/1979803578 wait complete. 2024-05-10T11:18:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.613+0000 7fe5f62deb80 20 client.26111 trim_cache size 0 max 16384 2024-05-10T11:18:12.616 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:18:12.616 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:18:12.616 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:18:12.616 INFO:teuthology.orchestra.run.smithi027.stdout: "6574fb72-bc99-44ee-b0f6-9337275ae9d1": { 2024-05-10T11:18:12.616 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:18:12.616 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "6574fb72-bc99-44ee-b0f6-9337275ae9d1", 2024-05-10T11:18:12.616 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:18:12.616 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:18:12.617 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:18:12.617 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:18:12.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.614+0000 7fe5eba00700 1 -- 172.21.15.27:0/3868721929 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe5cc0b2a70 msgr2=0x7fe5cc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:12.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.614+0000 7fe5eba00700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe5cc0b2a70 0x7fe5cc0b4f30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80e0fc0 tx=0x7fe5d8009040 comp rx=0 tx=0).stop 2024-05-10T11:18:12.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.614+0000 7fe5eba00700 1 -- 172.21.15.27:0/3868721929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5e406b180 msgr2=0x7fe5e419be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:12.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.614+0000 7fe5eba00700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5e406b180 0x7fe5e419be20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800b070 tx=0x7fe5d8004d00 comp rx=0 tx=0).stop 2024-05-10T11:18:12.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.614+0000 7fe5eba00700 1 -- 172.21.15.27:0/3868721929 shutdown_connections 2024-05-10T11:18:12.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.614+0000 7fe5eba00700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe5cc0b2a70 0x7fe5cc0b4f30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.614+0000 7fe5eba00700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5e406b180 0x7fe5e419be20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.614+0000 7fe5eba00700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5e4109510 0x7fe5e41a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.614+0000 7fe5eba00700 1 --2- 172.21.15.27:0/3868721929 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5e406bb40 0x7fe5e419c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:12.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.614+0000 7fe5eba00700 1 -- 172.21.15.27:0/3868721929 >> 172.21.15.27:0/3868721929 conn(0x7fe5e4079350 msgr2=0x7fe5e410e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:12.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.615+0000 7fe5eba00700 1 -- 172.21.15.27:0/3868721929 shutdown_connections 2024-05-10T11:18:12.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:12.615+0000 7fe5eba00700 1 -- 172.21.15.27:0/3868721929 wait complete. 2024-05-10T11:18:12.671 INFO:tasks.workunit.client.0.smithi027.stdout: 1 250223 7.81 MB/sec execute 57 sec latency 28.195 ms 2024-05-10T11:18:12.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:12 smithi076 ceph-mon[34152]: pgmap v152: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-05-10T11:18:12.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:12 smithi189 ceph-mon[34842]: pgmap v152: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-05-10T11:18:13.452 INFO:tasks.cephfs.filesystem:scrub status for tag:6574fb72-bc99-44ee-b0f6-9337275ae9d1 - {'path': '/', 'tag': '6574fb72-bc99-44ee-b0f6-9337275ae9d1', 'options': 'recursive,force'} 2024-05-10T11:18:13.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:13 smithi027 ceph-mon[30054]: pgmap v153: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-05-10T11:18:13.672 INFO:tasks.workunit.client.0.smithi027.stdout: 1 252110 7.90 MB/sec execute 58 sec latency 16.027 ms 2024-05-10T11:18:13.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:13 smithi076 ceph-mon[34152]: pgmap v153: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-05-10T11:18:13.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:13 smithi189 ceph-mon[34842]: pgmap v153: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-05-10T11:18:14.672 INFO:tasks.workunit.client.0.smithi027.stdout: 1 252682 7.78 MB/sec execute 59 sec latency 36.674 ms 2024-05-10T11:18:15.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:15 smithi027 ceph-mon[30054]: pgmap v154: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 273 op/s 2024-05-10T11:18:15.672 INFO:tasks.workunit.client.0.smithi027.stdout: 1 254645 7.83 MB/sec execute 60 sec latency 52.862 ms 2024-05-10T11:18:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:15 smithi076 ceph-mon[34152]: pgmap v154: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 273 op/s 2024-05-10T11:18:15.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:15 smithi189 ceph-mon[34842]: pgmap v154: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 273 op/s 2024-05-10T11:18:16.672 INFO:tasks.workunit.client.0.smithi027.stdout: 1 255787 7.83 MB/sec execute 61 sec latency 23.171 ms 2024-05-10T11:18:16.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:16 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:18:16.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:16 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:18:17.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:16 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:18:17.672 INFO:tasks.workunit.client.0.smithi027.stdout: 1 256831 7.79 MB/sec execute 62 sec latency 35.755 ms 2024-05-10T11:18:17.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:17 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:18:17.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:17 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:18:17.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:17 smithi076 ceph-mon[34152]: pgmap v155: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 165 op/s 2024-05-10T11:18:17.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:17 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:18:17.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:17 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:18:17.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:17 smithi189 ceph-mon[34842]: pgmap v155: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 165 op/s 2024-05-10T11:18:18.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:17 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:18:18.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:17 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:18:18.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:17 smithi027 ceph-mon[30054]: pgmap v155: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 165 op/s 2024-05-10T11:18:18.672 INFO:tasks.workunit.client.0.smithi027.stdout: 1 259056 7.86 MB/sec execute 63 sec latency 12.087 ms 2024-05-10T11:18:19.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:19 smithi027 ceph-mon[30054]: pgmap v156: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 242 op/s 2024-05-10T11:18:19.674 INFO:tasks.workunit.client.0.smithi027.stdout: 1 259691 7.77 MB/sec execute 64 sec latency 24.869 ms 2024-05-10T11:18:19.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:19 smithi076 ceph-mon[34152]: pgmap v156: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 242 op/s 2024-05-10T11:18:19.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:19 smithi189 ceph-mon[34842]: pgmap v156: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 242 op/s 2024-05-10T11:18:20.672 INFO:tasks.workunit.client.0.smithi027.stdout: 1 261862 7.83 MB/sec execute 65 sec latency 36.608 ms 2024-05-10T11:18:21.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:21 smithi027 ceph-mon[30054]: pgmap v157: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-05-10T11:18:21.673 INFO:tasks.workunit.client.0.smithi027.stdout: 1 262929 7.82 MB/sec execute 66 sec latency 19.572 ms 2024-05-10T11:18:21.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:21 smithi076 ceph-mon[34152]: pgmap v157: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-05-10T11:18:21.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:21 smithi189 ceph-mon[34842]: pgmap v157: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-05-10T11:18:22.673 INFO:tasks.workunit.client.0.smithi027.stdout: 1 264697 7.79 MB/sec execute 67 sec latency 30.918 ms 2024-05-10T11:18:23.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:23 smithi027 ceph-mon[30054]: pgmap v158: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-05-10T11:18:23.673 INFO:tasks.workunit.client.0.smithi027.stdout: 1 266355 7.87 MB/sec execute 68 sec latency 13.978 ms 2024-05-10T11:18:23.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:23 smithi076 ceph-mon[34152]: pgmap v158: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-05-10T11:18:23.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:23 smithi189 ceph-mon[34842]: pgmap v158: 129 pgs: 129 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-05-10T11:18:24.673 INFO:tasks.workunit.client.0.smithi027.stdout: 1 267376 7.80 MB/sec execute 69 sec latency 25.609 ms 2024-05-10T11:18:25.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:25 smithi027 ceph-mon[30054]: pgmap v159: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 240 op/s 2024-05-10T11:18:25.673 INFO:tasks.workunit.client.0.smithi027.stdout: 1 269613 7.89 MB/sec execute 70 sec latency 30.225 ms 2024-05-10T11:18:25.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:25 smithi076 ceph-mon[34152]: pgmap v159: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 240 op/s 2024-05-10T11:18:25.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:25 smithi189 ceph-mon[34842]: pgmap v159: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 240 op/s 2024-05-10T11:18:26.673 INFO:tasks.workunit.client.0.smithi027.stdout: 1 270189 7.81 MB/sec execute 71 sec latency 83.685 ms 2024-05-10T11:18:27.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:27 smithi027 ceph-mon[30054]: pgmap v160: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 157 op/s 2024-05-10T11:18:27.673 INFO:tasks.workunit.client.0.smithi027.stdout: 1 272210 7.81 MB/sec execute 72 sec latency 30.338 ms 2024-05-10T11:18:27.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:27 smithi076 ceph-mon[34152]: pgmap v160: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 157 op/s 2024-05-10T11:18:27.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:27 smithi189 ceph-mon[34842]: pgmap v160: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 157 op/s 2024-05-10T11:18:28.673 INFO:tasks.workunit.client.0.smithi027.stdout: 1 273539 7.85 MB/sec execute 73 sec latency 13.917 ms 2024-05-10T11:18:29.674 INFO:tasks.workunit.client.0.smithi027.stdout: 1 274688 7.82 MB/sec execute 74 sec latency 20.534 ms 2024-05-10T11:18:29.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:29 smithi076 ceph-mon[34152]: pgmap v161: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 239 op/s 2024-05-10T11:18:29.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:29 smithi189 ceph-mon[34842]: pgmap v161: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 239 op/s 2024-05-10T11:18:29.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:29 smithi027 ceph-mon[30054]: pgmap v161: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 239 op/s 2024-05-10T11:18:30.674 INFO:tasks.workunit.client.0.smithi027.stdout: 1 276513 7.84 MB/sec execute 75 sec latency 67.081 ms 2024-05-10T11:18:31.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:31 smithi027 ceph-mon[30054]: pgmap v162: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-05-10T11:18:31.674 INFO:tasks.workunit.client.0.smithi027.stdout: 1 277144 7.80 MB/sec execute 76 sec latency 27.481 ms 2024-05-10T11:18:31.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:31 smithi076 ceph-mon[34152]: pgmap v162: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-05-10T11:18:31.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:31 smithi189 ceph-mon[34842]: pgmap v162: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-05-10T11:18:32.674 INFO:tasks.workunit.client.0.smithi027.stdout: 1 278385 7.76 MB/sec execute 77 sec latency 40.167 ms 2024-05-10T11:18:33.674 INFO:tasks.workunit.client.0.smithi027.stdout: 1 280194 7.82 MB/sec execute 78 sec latency 92.875 ms 2024-05-10T11:18:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:33 smithi076 ceph-mon[34152]: pgmap v163: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-05-10T11:18:33.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:33 smithi189 ceph-mon[34842]: pgmap v163: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-05-10T11:18:33.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:33 smithi027 ceph-mon[30054]: pgmap v163: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-05-10T11:18:34.675 INFO:tasks.workunit.client.0.smithi027.stdout: 1 280705 7.74 MB/sec execute 79 sec latency 27.200 ms 2024-05-10T11:18:35.674 INFO:tasks.workunit.client.0.smithi027.stdout: 1 282023 7.71 MB/sec execute 80 sec latency 51.225 ms 2024-05-10T11:18:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:35 smithi076 ceph-mon[34152]: pgmap v164: 129 pgs: 129 active+clean; 163 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 236 op/s 2024-05-10T11:18:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:35 smithi189 ceph-mon[34842]: pgmap v164: 129 pgs: 129 active+clean; 163 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 236 op/s 2024-05-10T11:18:35.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:35 smithi027 ceph-mon[30054]: pgmap v164: 129 pgs: 129 active+clean; 163 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 236 op/s 2024-05-10T11:18:36.674 INFO:tasks.workunit.client.0.smithi027.stdout: 1 284024 7.78 MB/sec execute 81 sec latency 23.367 ms 2024-05-10T11:18:37.675 INFO:tasks.workunit.client.0.smithi027.stdout: 1 284822 7.72 MB/sec execute 82 sec latency 31.745 ms 2024-05-10T11:18:37.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:37 smithi076 ceph-mon[34152]: pgmap v165: 129 pgs: 129 active+clean; 163 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-05-10T11:18:37.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:37 smithi189 ceph-mon[34842]: pgmap v165: 129 pgs: 129 active+clean; 163 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-05-10T11:18:37.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:37 smithi027 ceph-mon[30054]: pgmap v165: 129 pgs: 129 active+clean; 163 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-05-10T11:18:38.675 INFO:tasks.workunit.client.0.smithi027.stdout: 1 287294 7.81 MB/sec execute 83 sec latency 12.487 ms 2024-05-10T11:18:39.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:39 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:18:39.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:39 smithi027 ceph-mon[30054]: pgmap v166: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 234 op/s 2024-05-10T11:18:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:39 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:18:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:39 smithi076 ceph-mon[34152]: pgmap v166: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 234 op/s 2024-05-10T11:18:39.675 INFO:tasks.workunit.client.0.smithi027.stdout: 1 287997 7.74 MB/sec execute 84 sec latency 26.964 ms 2024-05-10T11:18:39.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:39 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:18:39.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:39 smithi189 ceph-mon[34842]: pgmap v166: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 234 op/s 2024-05-10T11:18:40.675 INFO:tasks.workunit.client.0.smithi027.stdout: 1 290247 7.79 MB/sec execute 85 sec latency 40.424 ms 2024-05-10T11:18:41.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:40 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:18:41.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:40 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:18:41.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:40 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:18:41.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:40 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:18:41.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:40 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:18:41.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:40 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:18:41.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:40 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:18:41.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:40 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:18:41.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:40 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:18:41.675 INFO:tasks.workunit.client.0.smithi027.stdout: 1 291263 7.78 MB/sec execute 86 sec latency 59.797 ms 2024-05-10T11:18:42.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:41 smithi027 ceph-mon[30054]: pgmap v167: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 152 op/s 2024-05-10T11:18:42.137 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:41 smithi189 ceph-mon[34842]: pgmap v167: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 152 op/s 2024-05-10T11:18:42.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:41 smithi076 ceph-mon[34152]: pgmap v167: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 152 op/s 2024-05-10T11:18:42.675 INFO:tasks.workunit.client.0.smithi027.stdout: 1 292370 7.75 MB/sec execute 87 sec latency 37.800 ms 2024-05-10T11:18:43.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:18:43.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:43 smithi076 ceph-mon[34152]: pgmap v168: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 152 op/s 2024-05-10T11:18:43.675 INFO:tasks.workunit.client.0.smithi027.stdout: 1 294418 7.80 MB/sec execute 88 sec latency 13.531 ms 2024-05-10T11:18:43.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:43 smithi189 ceph-mon[34842]: pgmap v168: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 152 op/s 2024-05-10T11:18:43.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:43 smithi027 ceph-mon[30054]: pgmap v168: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 152 op/s 2024-05-10T11:18:43.942 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:18:44.676 INFO:tasks.workunit.client.0.smithi027.stdout: 1 295105 7.73 MB/sec execute 89 sec latency 28.250 ms 2024-05-10T11:18:45.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:45 smithi076 ceph-mon[34152]: pgmap v169: 129 pgs: 129 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 234 op/s 2024-05-10T11:18:45.675 INFO:tasks.workunit.client.0.smithi027.stdout: 1 296923 7.73 MB/sec execute 90 sec latency 13.088 ms 2024-05-10T11:18:45.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:45 smithi189 ceph-mon[34842]: pgmap v169: 129 pgs: 129 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 234 op/s 2024-05-10T11:18:45.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:45 smithi027 ceph-mon[30054]: pgmap v169: 129 pgs: 129 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 234 op/s 2024-05-10T11:18:46.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.405+0000 7ffa89800700 1 -- 172.21.15.27:0/1729344703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa8406bad0 msgr2=0x7ffa8410fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.405+0000 7ffa89800700 1 --2- 172.21.15.27:0/1729344703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa8406bad0 0x7ffa8410fa10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009990 tx=0x7ffa70009ca0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.405+0000 7ffa89800700 1 -- 172.21.15.27:0/1729344703 shutdown_connections 2024-05-10T11:18:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.405+0000 7ffa89800700 1 --2- 172.21.15.27:0/1729344703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa8410ff50 0x7ffa84112340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.405+0000 7ffa89800700 1 --2- 172.21.15.27:0/1729344703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa8406b1b0 0x7ffa8406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.405+0000 7ffa89800700 1 --2- 172.21.15.27:0/1729344703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa8406bad0 0x7ffa8410fa10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.405+0000 7ffa89800700 1 -- 172.21.15.27:0/1729344703 >> 172.21.15.27:0/1729344703 conn(0x7ffa84079350 msgr2=0x7ffa84079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.405+0000 7ffa89800700 1 -- 172.21.15.27:0/1729344703 shutdown_connections 2024-05-10T11:18:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.405+0000 7ffa89800700 1 -- 172.21.15.27:0/1729344703 wait complete. 2024-05-10T11:18:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.406+0000 7ffa89800700 1 Processor -- start 2024-05-10T11:18:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.407+0000 7ffa89800700 1 -- start start 2024-05-10T11:18:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.407+0000 7ffa89800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa8406b1b0 0x7ffa8419bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.407+0000 7ffa89800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa8406bad0 0x7ffa8419c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.407+0000 7ffa89800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa8410ff50 0x7ffa841a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.407+0000 7ffa89800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffa841158b0 con 0x7ffa8406b1b0 2024-05-10T11:18:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.407+0000 7ffa89800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffa84115730 con 0x7ffa8410ff50 2024-05-10T11:18:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.407+0000 7ffa89800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa84115a30 con 0x7ffa8406bad0 2024-05-10T11:18:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.407+0000 7ffa82000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa8406bad0 0x7ffa8419c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.407+0000 7ffa82a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa8406b1b0 0x7ffa8419bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.408+0000 7ffa83400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa8410ff50 0x7ffa841a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.408+0000 7ffa82000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa8406bad0 0x7ffa8419c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51006/0 (socket says 172.21.15.27:51006) 2024-05-10T11:18:46.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.408+0000 7ffa82a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa8406b1b0 0x7ffa8419bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48530/0 (socket says 172.21.15.27:48530) 2024-05-10T11:18:46.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.408+0000 7ffa82000700 1 -- 172.21.15.27:0/2544618875 learned_addr learned my addr 172.21.15.27:0/2544618875 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.408+0000 7ffa82000700 1 -- 172.21.15.27:0/2544618875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa8410ff50 msgr2=0x7ffa841a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.408+0000 7ffa82000700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa8410ff50 0x7ffa841a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.408+0000 7ffa82000700 1 -- 172.21.15.27:0/2544618875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa8406b1b0 msgr2=0x7ffa8419bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.408+0000 7ffa82000700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa8406b1b0 0x7ffa8419bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.408+0000 7ffa82000700 1 -- 172.21.15.27:0/2544618875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa841a0dd0 con 0x7ffa8406bad0 2024-05-10T11:18:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.408+0000 7ffa82a00700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa8406b1b0 0x7ffa8419bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:18:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.408+0000 7ffa82000700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa8406bad0 0x7ffa8419c300 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffa70005910 tx=0x7ffa7000b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.409+0000 7ffa7b400700 1 -- 172.21.15.27:0/2544618875 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa70027070 con 0x7ffa8406bad0 2024-05-10T11:18:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.409+0000 7ffa89800700 1 -- 172.21.15.27:0/2544618875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa841a1060 con 0x7ffa8406bad0 2024-05-10T11:18:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.409+0000 7ffa7b400700 1 -- 172.21.15.27:0/2544618875 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa70022470 con 0x7ffa8406bad0 2024-05-10T11:18:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.409+0000 7ffa89800700 1 -- 172.21.15.27:0/2544618875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa841a8940 con 0x7ffa8406bad0 2024-05-10T11:18:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.409+0000 7ffa7b400700 1 -- 172.21.15.27:0/2544618875 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa700175f0 con 0x7ffa8406bad0 2024-05-10T11:18:46.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.413+0000 7ffa7b400700 1 -- 172.21.15.27:0/2544618875 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ffa7002a070 con 0x7ffa8406bad0 2024-05-10T11:18:46.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.414+0000 7ffa94246b80 1 Processor -- start 2024-05-10T11:18:46.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.414+0000 7ffa94246b80 1 -- start start 2024-05-10T11:18:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.414+0000 7ffa94246b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe95bc0 0x5623cbe95fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.414+0000 7ffa94246b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbe902d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.414+0000 7ffa94246b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe908f0 0x5623cbe90d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.414+0000 7ffa94246b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5623cbde2a20 con 0x5623cbe908f0 2024-05-10T11:18:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.414+0000 7ffa94246b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5623cbd80760 con 0x5623cbe964e0 2024-05-10T11:18:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.414+0000 7ffa94246b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5623cbdde9e0 con 0x5623cbe95bc0 2024-05-10T11:18:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa82a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbe902d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa7b400700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ffa680b29f0 0x7ffa680b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa82a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbe902d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48180/0 (socket says 172.21.15.27:48180) 2024-05-10T11:18:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa82a00700 1 -- 172.21.15.27:0/3020720251 learned_addr learned my addr 172.21.15.27:0/3020720251 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa83400700 1 --2- 172.21.15.27:0/3020720251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe95bc0 0x5623cbe95fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa7b400700 1 -- 172.21.15.27:0/2544618875 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7ffa700e37d0 con 0x7ffa8406bad0 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa82000700 1 --2- 172.21.15.27:0/3020720251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe908f0 0x5623cbe90d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa82a00700 1 -- 172.21.15.27:0/3020720251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe95bc0 msgr2=0x5623cbe95fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa82a00700 1 --2- 172.21.15.27:0/3020720251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe95bc0 0x5623cbe95fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa82a00700 1 -- 172.21.15.27:0/3020720251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe908f0 msgr2=0x5623cbe90d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa82a00700 1 --2- 172.21.15.27:0/3020720251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe908f0 0x5623cbe90d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa82a00700 1 -- 172.21.15.27:0/3020720251 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623cbd2f8d0 con 0x5623cbe964e0 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa82000700 1 --2- 172.21.15.27:0/3020720251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe908f0 0x5623cbe90d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.415+0000 7ffa83400700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ffa680b29f0 0x7ffa680b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.416+0000 7ffa82a00700 1 --2- 172.21.15.27:0/3020720251 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbe902d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009de0 tx=0x7ffa740116c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e22713b55e68e43 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.416+0000 7ffa6fe00700 1 -- 172.21.15.27:0/3020720251 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa74012450 con 0x5623cbe964e0 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.416+0000 7ffa6fe00700 1 -- 172.21.15.27:0/3020720251 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa74012aa0 con 0x5623cbe964e0 2024-05-10T11:18:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.416+0000 7ffa6fe00700 1 -- 172.21.15.27:0/3020720251 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7400f9c0 con 0x5623cbe964e0 2024-05-10T11:18:46.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.417+0000 7ffa94246b80 1 -- 172.21.15.27:0/3020720251 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 msgr2=0x5623cbe902d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:46.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.417+0000 7ffa94246b80 1 --2- 172.21.15.27:0/3020720251 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbe902d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009de0 tx=0x7ffa740116c0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.417+0000 7ffa83400700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ffa680b29f0 0x7ffa680b4eb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009ec0 tx=0x7ffa7c009040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:46.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.417+0000 7ffa94246b80 1 -- 172.21.15.27:0/3020720251 shutdown_connections 2024-05-10T11:18:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.417+0000 7ffa94246b80 1 --2- 172.21.15.27:0/3020720251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe95bc0 0x5623cbe95fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.417+0000 7ffa94246b80 1 --2- 172.21.15.27:0/3020720251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe908f0 0x5623cbe90d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.417+0000 7ffa94246b80 1 --2- 172.21.15.27:0/3020720251 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbe902d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.417+0000 7ffa94246b80 1 -- 172.21.15.27:0/3020720251 >> 172.21.15.27:0/3020720251 conn(0x5623cbe086d0 msgr2=0x5623cbe8ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.417+0000 7ffa94246b80 1 -- 172.21.15.27:0/3020720251 shutdown_connections 2024-05-10T11:18:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.418+0000 7ffa94246b80 1 -- 172.21.15.27:0/3020720251 wait complete. 2024-05-10T11:18:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.418+0000 7ffa94246b80 1 -- start start 2024-05-10T11:18:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.418+0000 7ffa94246b80 1 Processor -- start 2024-05-10T11:18:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.419+0000 7ffa94246b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 0x5623cbe9c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.419+0000 7ffa94246b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe95bc0 0x5623cbe995d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.419+0000 7ffa94246b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe964e0 0x5623cbe99b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.419+0000 7ffa94246b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5623cbe9cf20 con 0x5623cbe964e0 2024-05-10T11:18:46.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.419+0000 7ffa94246b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5623cbd48b20 con 0x5623cbe95bc0 2024-05-10T11:18:46.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.419+0000 7ffa94246b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5623cbe9d0a0 con 0x5623cbe908f0 2024-05-10T11:18:46.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.419+0000 7ffa94246b80 10 client.0 _ll_register_callbacks cb 0x5623cbd89c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:18:46.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.419+0000 7ffa82a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 0x5623cbe9c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.419+0000 7ffa82a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 0x5623cbe9c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51022/0 (socket says 172.21.15.27:51022) 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.419+0000 7ffa82a00700 1 -- 172.21.15.27:0/539234601 learned_addr learned my addr 172.21.15.27:0/539234601 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.419+0000 7ffa82000700 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe95bc0 0x5623cbe995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.420+0000 7ffa83400700 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe964e0 0x5623cbe99b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.420+0000 7ffa82000700 1 -- 172.21.15.27:0/539234601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 msgr2=0x5623cbe9c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.420+0000 7ffa82000700 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 0x5623cbe9c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.420+0000 7ffa82000700 1 -- 172.21.15.27:0/539234601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe964e0 msgr2=0x5623cbe99b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.420+0000 7ffa82000700 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe964e0 0x5623cbe99b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.420+0000 7ffa82000700 1 -- 172.21.15.27:0/539234601 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623cbda8420 con 0x5623cbe95bc0 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.420+0000 7ffa82a00700 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 0x5623cbe9c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.420+0000 7ffa83400700 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe964e0 0x5623cbe99b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.420+0000 7ffa82000700 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe95bc0 0x5623cbe995d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ffa700e0af0 tx=0x7ffa70022cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cb80f667d174cd5 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.420+0000 7ffa6cc00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:18:46.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.420+0000 7ffa6cc00700 1 -- 172.21.15.27:0/539234601 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa700a9e80 con 0x5623cbe95bc0 2024-05-10T11:18:46.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.421+0000 7ffa6cc00700 1 -- 172.21.15.27:0/539234601 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7ffa700aa970 con 0x5623cbe95bc0 2024-05-10T11:18:46.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.421+0000 7ffa94246b80 1 -- 172.21.15.27:0/539234601 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa74012aa0 con 0x5623cbe95bc0 2024-05-10T11:18:46.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.421+0000 7ffa6cc00700 1 -- 172.21.15.27:0/539234601 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ffa70027e40 con 0x5623cbe95bc0 2024-05-10T11:18:46.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.421+0000 7ffa94246b80 10 client.26123 fetch_fsmap learned FSMap version 33 2024-05-10T11:18:46.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.421+0000 7ffa94246b80 1 -- 172.21.15.27:0/539234601 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5623cbea7760 con 0x5623cbe95bc0 2024-05-10T11:18:46.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.421+0000 7ffa6cc00700 1 -- 172.21.15.27:0/539234601 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa74012aa0 con 0x5623cbe95bc0 2024-05-10T11:18:46.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.421+0000 7ffa6cc00700 1 -- 172.21.15.27:0/539234601 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ffa700b3050 con 0x5623cbe95bc0 2024-05-10T11:18:46.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.421+0000 7ffa94246b80 10 client.26123 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:18:46.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.421+0000 7ffa94246b80 10 client.26123 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:18:46.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.421+0000 7ffa94246b80 20 client.26123 populate_metadata read hostname 'smithi027' 2024-05-10T11:18:46.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.422+0000 7ffa94246b80 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5623cbea9d20 0x5623cbeac180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.422+0000 7ffa94246b80 4 client.26123 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:18:46.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.422+0000 7ffa94246b80 1 -- 172.21.15.27:0/539234601 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623cbe5d140 con 0x5623cbea9d20 2024-05-10T11:18:46.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.422+0000 7ffa82a00700 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5623cbea9d20 0x5623cbeac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.423+0000 7ffa82a00700 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5623cbea9d20 0x5623cbeac180 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d56b4c15990aac6 server_cookie=239793a1da5cf8fd in_seq=0 out_seq=0 2024-05-10T11:18:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.423+0000 7ffa6cc00700 10 client.26123 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa6cc00700 1 -- 172.21.15.27:0/539234601 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5623cbe5d140 con 0x5623cbea9d20 2024-05-10T11:18:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa6cc00700 10 client.26123 handle_command_reply: tid=0 2024-05-10T11:18:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa94246b80 1 client.26123 shutdown 2024-05-10T11:18:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa94246b80 1 -- 172.21.15.27:0/539234601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe95bc0 msgr2=0x5623cbe995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa94246b80 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe95bc0 0x5623cbe995d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ffa700e0af0 tx=0x7ffa70022cc0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa94246b80 1 -- 172.21.15.27:0/539234601 shutdown_connections 2024-05-10T11:18:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa94246b80 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5623cbea9d20 0x5623cbeac180 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa94246b80 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 0x5623cbe9c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa94246b80 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe964e0 0x5623cbe99b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa94246b80 1 --2- 172.21.15.27:0/539234601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe95bc0 0x5623cbe995d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa94246b80 1 -- 172.21.15.27:0/539234601 >> 172.21.15.27:0/539234601 conn(0x5623cbe086d0 msgr2=0x5623cbe8a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:46.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa6cc00700 0 client.26123 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:46.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.425+0000 7ffa94246b80 1 -- 172.21.15.27:0/539234601 shutdown_connections 2024-05-10T11:18:46.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.426+0000 7ffa94246b80 1 -- 172.21.15.27:0/539234601 wait complete. 2024-05-10T11:18:46.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.426+0000 7ffa94246b80 20 client.26123 trim_cache size 0 max 16384 2024-05-10T11:18:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.453+0000 7ffa94246b80 1 Processor -- start 2024-05-10T11:18:46.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.453+0000 7ffa94246b80 1 -- start start 2024-05-10T11:18:46.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.453+0000 7ffa94246b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe908f0 0x5623cbe902d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa94246b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe95bc0 0x5623cbea5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa94246b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbe8a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa94246b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffa841b1740 con 0x5623cbe908f0 2024-05-10T11:18:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa94246b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffa841b3280 con 0x5623cbe964e0 2024-05-10T11:18:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa94246b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa841b1e10 con 0x5623cbe95bc0 2024-05-10T11:18:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa83400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbe8a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa82000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe95bc0 0x5623cbea5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa83400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbe8a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48196/0 (socket says 172.21.15.27:48196) 2024-05-10T11:18:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa83400700 1 -- 172.21.15.27:0/4073600830 learned_addr learned my addr 172.21.15.27:0/4073600830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa82a00700 1 --2- 172.21.15.27:0/4073600830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe908f0 0x5623cbe902d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa82000700 1 -- 172.21.15.27:0/4073600830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 msgr2=0x5623cbe8a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa82000700 1 --2- 172.21.15.27:0/4073600830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbe8a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa82000700 1 -- 172.21.15.27:0/4073600830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe908f0 msgr2=0x5623cbe902d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa82000700 1 --2- 172.21.15.27:0/4073600830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe908f0 0x5623cbe902d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.454+0000 7ffa82000700 1 -- 172.21.15.27:0/4073600830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623cbe9be70 con 0x5623cbe95bc0 2024-05-10T11:18:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.455+0000 7ffa82000700 1 --2- 172.21.15.27:0/4073600830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe95bc0 0x5623cbea5fb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ffa700a7980 tx=0x7ffa70009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bab411b81d338aa server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.455+0000 7ffa6f400700 1 -- 172.21.15.27:0/4073600830 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa700aa220 con 0x5623cbe95bc0 2024-05-10T11:18:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.455+0000 7ffa6f400700 1 -- 172.21.15.27:0/4073600830 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa700aae10 con 0x5623cbe95bc0 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.455+0000 7ffa6f400700 1 -- 172.21.15.27:0/4073600830 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa700b3050 con 0x5623cbe95bc0 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.456+0000 7ffa94246b80 1 -- 172.21.15.27:0/4073600830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe95bc0 msgr2=0x5623cbea5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.456+0000 7ffa94246b80 1 --2- 172.21.15.27:0/4073600830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe95bc0 0x5623cbea5fb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ffa700a7980 tx=0x7ffa70009ca0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.456+0000 7ffa94246b80 1 -- 172.21.15.27:0/4073600830 shutdown_connections 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.456+0000 7ffa94246b80 1 --2- 172.21.15.27:0/4073600830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbe8a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.456+0000 7ffa94246b80 1 --2- 172.21.15.27:0/4073600830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe908f0 0x5623cbe902d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.456+0000 7ffa94246b80 1 --2- 172.21.15.27:0/4073600830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe95bc0 0x5623cbea5fb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.456+0000 7ffa94246b80 1 -- 172.21.15.27:0/4073600830 >> 172.21.15.27:0/4073600830 conn(0x5623cbe086d0 msgr2=0x5623cbe97ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.456+0000 7ffa94246b80 1 -- 172.21.15.27:0/4073600830 shutdown_connections 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.456+0000 7ffa94246b80 1 -- 172.21.15.27:0/4073600830 wait complete. 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.456+0000 7ffa94246b80 1 -- start start 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.457+0000 7ffa94246b80 1 Processor -- start 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.457+0000 7ffa94246b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 0x5623cbeb1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.457+0000 7ffa94246b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe95bc0 0x5623cbeb16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.457+0000 7ffa94246b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbeb1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.457+0000 7ffa94246b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffa841b2bb0 con 0x5623cbe95bc0 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.457+0000 7ffa94246b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffa841b24e0 con 0x5623cbe964e0 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.457+0000 7ffa94246b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa841af6e0 con 0x5623cbe908f0 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.457+0000 7ffa94246b80 10 client.0 _ll_register_callbacks cb 0x5623cbd89c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa82a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 0x5623cbeb1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa82a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 0x5623cbeb1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51046/0 (socket says 172.21.15.27:51046) 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa82a00700 1 -- 172.21.15.27:0/833117660 learned_addr learned my addr 172.21.15.27:0/833117660 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa83400700 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbeb1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa82000700 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe95bc0 0x5623cbeb16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa82a00700 1 -- 172.21.15.27:0/833117660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 msgr2=0x5623cbeb1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa82a00700 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbeb1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa82a00700 1 -- 172.21.15.27:0/833117660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe95bc0 msgr2=0x5623cbeb16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa82a00700 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe95bc0 0x5623cbeb16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa82a00700 1 -- 172.21.15.27:0/833117660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623cbf13640 con 0x5623cbe908f0 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa83400700 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbeb1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa82a00700 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 0x5623cbeb1160 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ffa740058f0 tx=0x7ffa74005390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4b0aadba2827c7b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.458+0000 7ffa6cc00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.459+0000 7ffa6cc00700 1 -- 172.21.15.27:0/833117660 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa74012710 con 0x5623cbe908f0 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.459+0000 7ffa6cc00700 1 -- 172.21.15.27:0/833117660 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7ffa740128b0 con 0x5623cbe908f0 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.459+0000 7ffa94246b80 1 -- 172.21.15.27:0/833117660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa700aae10 con 0x5623cbe908f0 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.459+0000 7ffa6cc00700 1 -- 172.21.15.27:0/833117660 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ffa74039050 con 0x5623cbe908f0 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.459+0000 7ffa94246b80 10 client.26119 fetch_fsmap learned FSMap version 33 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.459+0000 7ffa94246b80 1 -- 172.21.15.27:0/833117660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5623cbf14800 con 0x5623cbe908f0 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.459+0000 7ffa6cc00700 1 -- 172.21.15.27:0/833117660 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa700aae10 con 0x5623cbe908f0 2024-05-10T11:18:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.460+0000 7ffa6cc00700 1 -- 172.21.15.27:0/833117660 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ffa74031020 con 0x5623cbe908f0 2024-05-10T11:18:46.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.460+0000 7ffa94246b80 10 client.26119 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:18:46.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.460+0000 7ffa94246b80 10 client.26119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:18:46.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.460+0000 7ffa94246b80 20 client.26119 populate_metadata read hostname 'smithi027' 2024-05-10T11:18:46.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.460+0000 7ffa94246b80 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5623cbf14d10 0x5623cbf17100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:46.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.460+0000 7ffa94246b80 4 client.26119 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:18:46.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.460+0000 7ffa94246b80 1 -- 172.21.15.27:0/833117660 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5623cbdf0400 con 0x5623cbf14d10 2024-05-10T11:18:46.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.460+0000 7ffa82000700 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5623cbf14d10 0x5623cbf17100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:46.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.461+0000 7ffa82000700 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5623cbf14d10 0x5623cbf17100 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=492d89daf4b63bf7 server_cookie=d6bb94370f1469a6 in_seq=0 out_seq=0 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.461+0000 7ffa6cc00700 10 client.26119 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.463+0000 7ffa6cc00700 1 -- 172.21.15.27:0/833117660 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5623cbdf0400 con 0x5623cbf14d10 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.463+0000 7ffa6cc00700 10 client.26119 handle_command_reply: tid=0 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.463+0000 7ffa94246b80 1 client.26119 shutdown 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.463+0000 7ffa94246b80 1 -- 172.21.15.27:0/833117660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 msgr2=0x5623cbeb1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.463+0000 7ffa94246b80 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 0x5623cbeb1160 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ffa740058f0 tx=0x7ffa74005390 comp rx=0 tx=0).stop 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.463+0000 7ffa94246b80 1 -- 172.21.15.27:0/833117660 shutdown_connections 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.463+0000 7ffa94246b80 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5623cbf14d10 0x5623cbf17100 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.463+0000 7ffa94246b80 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5623cbe908f0 0x5623cbeb1160 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.463+0000 7ffa94246b80 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5623cbe964e0 0x5623cbeb1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.463+0000 7ffa94246b80 1 --2- 172.21.15.27:0/833117660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623cbe95bc0 0x5623cbeb16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.463+0000 7ffa6cc00700 0 client.26119 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.463+0000 7ffa94246b80 1 -- 172.21.15.27:0/833117660 >> 172.21.15.27:0/833117660 conn(0x5623cbe086d0 msgr2=0x5623cbe97150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.464+0000 7ffa94246b80 1 -- 172.21.15.27:0/833117660 shutdown_connections 2024-05-10T11:18:46.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.464+0000 7ffa94246b80 1 -- 172.21.15.27:0/833117660 wait complete. 2024-05-10T11:18:46.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.464+0000 7ffa94246b80 20 client.26119 trim_cache size 0 max 16384 2024-05-10T11:18:46.467 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:18:46.467 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:18:46.467 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:18:46.467 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:18:46.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.465+0000 7ffa89800700 1 -- 172.21.15.27:0/2544618875 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ffa680b29f0 msgr2=0x7ffa680b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:46.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.465+0000 7ffa89800700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ffa680b29f0 0x7ffa680b4eb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009ec0 tx=0x7ffa7c009040 comp rx=0 tx=0).stop 2024-05-10T11:18:46.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.465+0000 7ffa89800700 1 -- 172.21.15.27:0/2544618875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa8406bad0 msgr2=0x7ffa8419c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:46.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.465+0000 7ffa89800700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa8406bad0 0x7ffa8419c300 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffa70005910 tx=0x7ffa7000b670 comp rx=0 tx=0).stop 2024-05-10T11:18:46.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.465+0000 7ffa89800700 1 -- 172.21.15.27:0/2544618875 shutdown_connections 2024-05-10T11:18:46.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.465+0000 7ffa89800700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ffa680b29f0 0x7ffa680b4eb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.465+0000 7ffa89800700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa8410ff50 0x7ffa841a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.465+0000 7ffa89800700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa8406b1b0 0x7ffa8419bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.465+0000 7ffa89800700 1 --2- 172.21.15.27:0/2544618875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa8406bad0 0x7ffa8419c300 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:46.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.465+0000 7ffa89800700 1 -- 172.21.15.27:0/2544618875 >> 172.21.15.27:0/2544618875 conn(0x7ffa84079350 msgr2=0x7ffa84111400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:46.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.465+0000 7ffa89800700 1 -- 172.21.15.27:0/2544618875 shutdown_connections 2024-05-10T11:18:46.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:46.465+0000 7ffa89800700 1 -- 172.21.15.27:0/2544618875 wait complete. 2024-05-10T11:18:46.676 INFO:tasks.workunit.client.0.smithi027.stdout: 1 298261 7.77 MB/sec execute 91 sec latency 74.874 ms 2024-05-10T11:18:47.288 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:18:47.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:18:47.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:47 smithi027 ceph-mon[30054]: pgmap v170: 129 pgs: 129 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 160 op/s 2024-05-10T11:18:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:47 smithi076 ceph-mon[34152]: pgmap v170: 129 pgs: 129 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 160 op/s 2024-05-10T11:18:47.675 INFO:tasks.workunit.client.0.smithi027.stdout: 1 299087 7.71 MB/sec execute 92 sec latency 72.017 ms 2024-05-10T11:18:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:47 smithi189 ceph-mon[34842]: pgmap v170: 129 pgs: 129 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 160 op/s 2024-05-10T11:18:47.780 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:18:48.676 INFO:tasks.workunit.client.0.smithi027.stdout: 1 301464 7.79 MB/sec execute 93 sec latency 14.390 ms 2024-05-10T11:18:49.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:49 smithi027 ceph-mon[30054]: pgmap v171: 129 pgs: 129 active+clean; 175 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 238 op/s 2024-05-10T11:18:49.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:49 smithi076 ceph-mon[34152]: pgmap v171: 129 pgs: 129 active+clean; 175 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 238 op/s 2024-05-10T11:18:49.676 INFO:tasks.workunit.client.0.smithi027.stdout: 1 302026 7.73 MB/sec execute 94 sec latency 34.739 ms 2024-05-10T11:18:49.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:49 smithi189 ceph-mon[34842]: pgmap v171: 129 pgs: 129 active+clean; 175 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 238 op/s 2024-05-10T11:18:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.207+0000 7fc416a00700 1 -- 172.21.15.27:0/91013303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc410103550 msgr2=0x7fc4101039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.207+0000 7fc416a00700 1 --2- 172.21.15.27:0/91013303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc410103550 0x7fc4101039b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc009990 tx=0x7fc3fc009ca0 comp rx=0 tx=0).stop 2024-05-10T11:18:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.207+0000 7fc416a00700 1 -- 172.21.15.27:0/91013303 shutdown_connections 2024-05-10T11:18:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.207+0000 7fc416a00700 1 --2- 172.21.15.27:0/91013303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc410108460 0x7fc410108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.207+0000 7fc416a00700 1 --2- 172.21.15.27:0/91013303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc410103ef0 0x7fc4101103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.207+0000 7fc416a00700 1 --2- 172.21.15.27:0/91013303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc410103550 0x7fc4101039b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.207+0000 7fc416a00700 1 -- 172.21.15.27:0/91013303 >> 172.21.15.27:0/91013303 conn(0x7fc41007a600 msgr2=0x7fc41007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.207+0000 7fc416a00700 1 -- 172.21.15.27:0/91013303 shutdown_connections 2024-05-10T11:18:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.208+0000 7fc416a00700 1 -- 172.21.15.27:0/91013303 wait complete. 2024-05-10T11:18:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.209+0000 7fc416a00700 1 Processor -- start 2024-05-10T11:18:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.209+0000 7fc416a00700 1 -- start start 2024-05-10T11:18:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc416a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc410103550 0x7fc41019bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc416a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc410103ef0 0x7fc41019c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc416a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc410108460 0x7fc4101a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc416a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc4101136e0 con 0x7fc410103ef0 2024-05-10T11:18:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc416a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc410113560 con 0x7fc410103550 2024-05-10T11:18:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc416a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc410113860 con 0x7fc410108460 2024-05-10T11:18:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc40fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc410103550 0x7fc41019bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc40fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc410103550 0x7fc41019bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48230/0 (socket says 172.21.15.27:48230) 2024-05-10T11:18:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc40fe00700 1 -- 172.21.15.27:0/2321628701 learned_addr learned my addr 172.21.15.27:0/2321628701 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc40f400700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc410103ef0 0x7fc41019c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc414a00700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc410108460 0x7fc4101a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc40fe00700 1 -- 172.21.15.27:0/2321628701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc410108460 msgr2=0x7fc4101a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.210+0000 7fc40fe00700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc410108460 0x7fc4101a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.211+0000 7fc40fe00700 1 -- 172.21.15.27:0/2321628701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc410103ef0 msgr2=0x7fc41019c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.211+0000 7fc40fe00700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc410103ef0 0x7fc41019c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.211+0000 7fc40fe00700 1 -- 172.21.15.27:0/2321628701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4101a0dc0 con 0x7fc410103550 2024-05-10T11:18:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.211+0000 7fc40fe00700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc410103550 0x7fc41019bdb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc40000e970 tx=0x7fc40000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.211+0000 7fc40cc00700 1 -- 172.21.15.27:0/2321628701 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc400004220 con 0x7fc410103550 2024-05-10T11:18:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.211+0000 7fc40cc00700 1 -- 172.21.15.27:0/2321628701 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4000043c0 con 0x7fc410103550 2024-05-10T11:18:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.211+0000 7fc40cc00700 1 -- 172.21.15.27:0/2321628701 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc400003ce0 con 0x7fc410103550 2024-05-10T11:18:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.211+0000 7fc416a00700 1 -- 172.21.15.27:0/2321628701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4101a10b0 con 0x7fc410103550 2024-05-10T11:18:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.211+0000 7fc416a00700 1 -- 172.21.15.27:0/2321628701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4101a8930 con 0x7fc410103550 2024-05-10T11:18:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.213+0000 7fc416a00700 1 -- 172.21.15.27:0/2321628701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc41006a190 con 0x7fc410103550 2024-05-10T11:18:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.215+0000 7fc40cc00700 1 -- 172.21.15.27:0/2321628701 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc40000c7e0 con 0x7fc410103550 2024-05-10T11:18:50.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.217+0000 7fc40cc00700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc3f40b2a70 0x7fc3f40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:50.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.217+0000 7fc40cc00700 1 -- 172.21.15.27:0/2321628701 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fc4000e2bb0 con 0x7fc410103550 2024-05-10T11:18:50.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.217+0000 7fc40f400700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc3f40b2a70 0x7fc3f40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:50.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.218+0000 7fc40f400700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc3f40b2a70 0x7fc3f40b4f30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc410077730 tx=0x7fc3fc005fd0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.224+0000 7fc40cc00700 1 -- 172.21.15.27:0/2321628701 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fc4000aa160 con 0x7fc410103550 2024-05-10T11:18:50.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.546+0000 7fc416a00700 1 -- 172.21.15.27:0/2321628701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc41019d110 con 0x7fc410103550 2024-05-10T11:18:50.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.547+0000 7fc40cc00700 1 -- 172.21.15.27:0/2321628701 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11936 (secure 0 0 0) 0x7fc4000a9a60 con 0x7fc410103550 2024-05-10T11:18:50.550 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:18:50.551 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:17:36.552331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:18:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.551+0000 7fc416a00700 1 -- 172.21.15.27:0/2321628701 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc3f40b2a70 msgr2=0x7fc3f40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.551+0000 7fc416a00700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc3f40b2a70 0x7fc3f40b4f30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc410077730 tx=0x7fc3fc005fd0 comp rx=0 tx=0).stop 2024-05-10T11:18:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.551+0000 7fc416a00700 1 -- 172.21.15.27:0/2321628701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc410103550 msgr2=0x7fc41019bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:50.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.551+0000 7fc416a00700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc410103550 0x7fc41019bdb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc40000e970 tx=0x7fc40000ec80 comp rx=0 tx=0).stop 2024-05-10T11:18:50.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.551+0000 7fc416a00700 1 -- 172.21.15.27:0/2321628701 shutdown_connections 2024-05-10T11:18:50.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.551+0000 7fc416a00700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc3f40b2a70 0x7fc3f40b4f30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:50.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.551+0000 7fc416a00700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc410108460 0x7fc4101a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:50.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.551+0000 7fc416a00700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc410103550 0x7fc41019bdb0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:50.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.551+0000 7fc416a00700 1 --2- 172.21.15.27:0/2321628701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc410103ef0 0x7fc41019c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:50.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.551+0000 7fc416a00700 1 -- 172.21.15.27:0/2321628701 >> 172.21.15.27:0/2321628701 conn(0x7fc41007a600 msgr2=0x7fc4101013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:50.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.552+0000 7fc416a00700 1 -- 172.21.15.27:0/2321628701 shutdown_connections 2024-05-10T11:18:50.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:50.552+0000 7fc416a00700 1 -- 172.21.15.27:0/2321628701 wait complete. 2024-05-10T11:18:50.552 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 33 2024-05-10T11:18:50.676 INFO:tasks.workunit.client.0.smithi027.stdout: 1 303963 7.72 MB/sec execute 95 sec latency 30.488 ms 2024-05-10T11:18:50.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:50 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2321628701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:18:50.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:50 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2321628701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:18:50.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:50 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2321628701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:18:51.414 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:18:51.676 INFO:tasks.workunit.client.0.smithi027.stdout: 1 305320 7.76 MB/sec execute 96 sec latency 75.614 ms 2024-05-10T11:18:51.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:51 smithi027 ceph-mon[30054]: pgmap v172: 129 pgs: 129 active+clean; 175 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-05-10T11:18:51.919 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:18:51.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:51 smithi076 ceph-mon[34152]: pgmap v172: 129 pgs: 129 active+clean; 175 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-05-10T11:18:51.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:51 smithi189 ceph-mon[34842]: pgmap v172: 129 pgs: 129 active+clean; 175 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-05-10T11:18:52.676 INFO:tasks.workunit.client.0.smithi027.stdout: 1 306191 7.71 MB/sec execute 97 sec latency 29.977 ms 2024-05-10T11:18:53.491 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:53 smithi027 ceph-mon[30054]: pgmap v173: 129 pgs: 129 active+clean; 175 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-05-10T11:18:53.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:53 smithi076 ceph-mon[34152]: pgmap v173: 129 pgs: 129 active+clean; 175 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-05-10T11:18:53.676 INFO:tasks.workunit.client.0.smithi027.stdout: 1 308617 7.79 MB/sec execute 98 sec latency 11.951 ms 2024-05-10T11:18:53.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:53 smithi189 ceph-mon[34842]: pgmap v173: 129 pgs: 129 active+clean; 175 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-05-10T11:18:54.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.379+0000 7f3379200700 1 -- 172.21.15.27:0/2627225642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374103750 msgr2=0x7f3374103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.379+0000 7f3379200700 1 --2- 172.21.15.27:0/2627225642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374103750 0x7f3374103b30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3364009920 tx=0x7f3364009c30 comp rx=0 tx=0).stop 2024-05-10T11:18:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.379+0000 7f3379200700 1 -- 172.21.15.27:0/2627225642 shutdown_connections 2024-05-10T11:18:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.379+0000 7f3379200700 1 --2- 172.21.15.27:0/2627225642 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3374108850 0x7f33741103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.379+0000 7f3379200700 1 --2- 172.21.15.27:0/2627225642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374103750 0x7f3374103b30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.379+0000 7f3379200700 1 --2- 172.21.15.27:0/2627225642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3374104100 0x7f3374108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.379+0000 7f3379200700 1 -- 172.21.15.27:0/2627225642 >> 172.21.15.27:0/2627225642 conn(0x7f337407a600 msgr2=0x7f337407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:54.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.379+0000 7f3379200700 1 -- 172.21.15.27:0/2627225642 shutdown_connections 2024-05-10T11:18:54.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.379+0000 7f3379200700 1 -- 172.21.15.27:0/2627225642 wait complete. 2024-05-10T11:18:54.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.381+0000 7f3379200700 1 Processor -- start 2024-05-10T11:18:54.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.381+0000 7f3379200700 1 -- start start 2024-05-10T11:18:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.381+0000 7f3379200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3374103750 0x7f337419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3379200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374104100 0x7f337419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3379200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3374108850 0x7f3374199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3379200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3374113620 con 0x7f3374104100 2024-05-10T11:18:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3379200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f33741134a0 con 0x7f3374103750 2024-05-10T11:18:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3379200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f33741137a0 con 0x7f3374108850 2024-05-10T11:18:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3371e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3374103750 0x7f337419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3371e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3374103750 0x7f337419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51368/0 (socket says 172.21.15.27:51368) 2024-05-10T11:18:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3371e00700 1 -- 172.21.15.27:0/266263301 learned_addr learned my addr 172.21.15.27:0/266263301 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:54.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3372800700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3374108850 0x7f3374199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3369400700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374104100 0x7f337419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3371e00700 1 -- 172.21.15.27:0/266263301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3374108850 msgr2=0x7f3374199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:54.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3371e00700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3374108850 0x7f3374199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3371e00700 1 -- 172.21.15.27:0/266263301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374104100 msgr2=0x7f337419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:54.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3371e00700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374104100 0x7f337419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.382+0000 7f3371e00700 1 -- 172.21.15.27:0/266263301 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337419a0c0 con 0x7f3374103750 2024-05-10T11:18:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.383+0000 7f3371e00700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3374103750 0x7f337419f7a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3364009230 tx=0x7f336400fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.383+0000 7f336b400700 1 -- 172.21.15.27:0/266263301 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3364027070 con 0x7f3374103750 2024-05-10T11:18:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.383+0000 7f336b400700 1 -- 172.21.15.27:0/266263301 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3364005670 con 0x7f3374103750 2024-05-10T11:18:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.383+0000 7f336b400700 1 -- 172.21.15.27:0/266263301 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3364004930 con 0x7f3374103750 2024-05-10T11:18:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.383+0000 7f3379200700 1 -- 172.21.15.27:0/266263301 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337419a350 con 0x7f3374103750 2024-05-10T11:18:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.383+0000 7f3379200700 1 -- 172.21.15.27:0/266263301 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33741a6670 con 0x7f3374103750 2024-05-10T11:18:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.387+0000 7f336b400700 1 -- 172.21.15.27:0/266263301 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f336402a070 con 0x7f3374103750 2024-05-10T11:18:54.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.388+0000 7f3383b58b80 1 Processor -- start 2024-05-10T11:18:54.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.389+0000 7f336b400700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f33580b29f0 0x7f33580b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.389+0000 7f3383b58b80 1 -- start start 2024-05-10T11:18:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.389+0000 7f336b400700 1 -- 172.21.15.27:0/266263301 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f33640e33f0 con 0x7f3374103750 2024-05-10T11:18:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.389+0000 7f3372800700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f33580b29f0 0x7f33580b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.389+0000 7f3383b58b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6a470 0x55af51a6a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.389+0000 7f3383b58b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6ad90 0x55af51a6e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.389+0000 7f3383b58b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ede0 0x55af51a6f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.389+0000 7f3383b58b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55af519b8a20 con 0x55af51a6ede0 2024-05-10T11:18:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.389+0000 7f3383b58b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55af51956760 con 0x55af51a6a470 2024-05-10T11:18:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.389+0000 7f3383b58b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55af519b49e0 con 0x55af51a6ad90 2024-05-10T11:18:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.390+0000 7f3371e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6a470 0x55af51a6a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.390+0000 7f3371e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6a470 0x55af51a6a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51382/0 (socket says 172.21.15.27:51382) 2024-05-10T11:18:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.390+0000 7f3371e00700 1 -- 172.21.15.27:0/2906192011 learned_addr learned my addr 172.21.15.27:0/2906192011 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.390+0000 7f3369400700 1 --2- 172.21.15.27:0/2906192011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6ad90 0x55af51a6e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.390+0000 7f3372800700 1 --2- 172.21.15.27:0/2906192011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ede0 0x55af51a6f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.390+0000 7f3371e00700 1 -- 172.21.15.27:0/2906192011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6ad90 msgr2=0x55af51a6e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.390+0000 7f3371e00700 1 --2- 172.21.15.27:0/2906192011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6ad90 0x55af51a6e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.390+0000 7f3371e00700 1 -- 172.21.15.27:0/2906192011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ede0 msgr2=0x55af51a6f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.390+0000 7f3371e00700 1 --2- 172.21.15.27:0/2906192011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ede0 0x55af51a6f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.390+0000 7f3371e00700 1 -- 172.21.15.27:0/2906192011 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af519058d0 con 0x55af51a6a470 2024-05-10T11:18:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.390+0000 7f3372800700 1 --2- 172.21.15.27:0/2906192011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ede0 0x55af51a6f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:18:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.390+0000 7f3372800700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f33580b29f0 0x7f33580b4eb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f336c003f60 tx=0x7f336c003f90 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.391+0000 7f3371e00700 1 --2- 172.21.15.27:0/2906192011 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6a470 0x55af51a6a850 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3364024000 tx=0x7f336400bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8895fa853a8ed117 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.391+0000 7f3363400700 1 -- 172.21.15.27:0/2906192011 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33640e0e60 con 0x55af51a6a470 2024-05-10T11:18:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.391+0000 7f3363400700 1 -- 172.21.15.27:0/2906192011 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33640aa330 con 0x55af51a6a470 2024-05-10T11:18:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.391+0000 7f3363400700 1 -- 172.21.15.27:0/2906192011 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33640051c0 con 0x55af51a6a470 2024-05-10T11:18:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.392+0000 7f3383b58b80 1 -- 172.21.15.27:0/2906192011 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6a470 msgr2=0x55af51a6a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.392+0000 7f3383b58b80 1 --2- 172.21.15.27:0/2906192011 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6a470 0x55af51a6a850 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3364024000 tx=0x7f336400bab0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.392+0000 7f3383b58b80 1 -- 172.21.15.27:0/2906192011 shutdown_connections 2024-05-10T11:18:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.392+0000 7f3383b58b80 1 --2- 172.21.15.27:0/2906192011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ede0 0x55af51a6f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.392+0000 7f3383b58b80 1 --2- 172.21.15.27:0/2906192011 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6a470 0x55af51a6a850 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.392+0000 7f3383b58b80 1 --2- 172.21.15.27:0/2906192011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6ad90 0x55af51a6e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.392+0000 7f3383b58b80 1 -- 172.21.15.27:0/2906192011 >> 172.21.15.27:0/2906192011 conn(0x55af51994b60 msgr2=0x55af51a607a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.392+0000 7f3383b58b80 1 -- 172.21.15.27:0/2906192011 shutdown_connections 2024-05-10T11:18:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.392+0000 7f3383b58b80 1 -- 172.21.15.27:0/2906192011 wait complete. 2024-05-10T11:18:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.393+0000 7f3383b58b80 1 -- start start 2024-05-10T11:18:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.393+0000 7f3383b58b80 1 Processor -- start 2024-05-10T11:18:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.393+0000 7f3383b58b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a76270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.393+0000 7f3383b58b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a73680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3383b58b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a73bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3383b58b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55af51a76e00 con 0x55af51a6ad90 2024-05-10T11:18:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3383b58b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55af5191eb20 con 0x55af51a6ede0 2024-05-10T11:18:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3383b58b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55af51a76f80 con 0x55af51a6a470 2024-05-10T11:18:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3383b58b80 10 client.0 _ll_register_callbacks cb 0x55af5195fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:18:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3371e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a76270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3369400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a73680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3372800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a73bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3369400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a73680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44328/0 (socket says 172.21.15.27:44328) 2024-05-10T11:18:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3369400700 1 -- 172.21.15.27:0/2740690206 learned_addr learned my addr 172.21.15.27:0/2740690206 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3371e00700 1 -- 172.21.15.27:0/2740690206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 msgr2=0x55af51a73bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3371e00700 1 --2- 172.21.15.27:0/2740690206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a73bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3371e00700 1 -- 172.21.15.27:0/2740690206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 msgr2=0x55af51a73680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3371e00700 1 --2- 172.21.15.27:0/2740690206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a73680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3371e00700 1 -- 172.21.15.27:0/2740690206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af5197e420 con 0x55af51a6a470 2024-05-10T11:18:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.394+0000 7f3369400700 1 --2- 172.21.15.27:0/2740690206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a73680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:18:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.395+0000 7f3371e00700 1 --2- 172.21.15.27:0/2740690206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a76270 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55af51a76c90 tx=0x7f33640ae6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=579f16ebbcb8c4b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.395+0000 7f3357e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:18:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.395+0000 7f3357e00700 1 -- 172.21.15.27:0/2740690206 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33640c4060 con 0x55af51a6a470 2024-05-10T11:18:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.395+0000 7f3357e00700 1 -- 172.21.15.27:0/2740690206 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f33640aa590 con 0x55af51a6a470 2024-05-10T11:18:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.395+0000 7f3383b58b80 1 -- 172.21.15.27:0/2740690206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33640aa330 con 0x55af51a6a470 2024-05-10T11:18:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.395+0000 7f3357e00700 1 -- 172.21.15.27:0/2740690206 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f33640e0e60 con 0x55af51a6a470 2024-05-10T11:18:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.395+0000 7f3383b58b80 10 client.26137 fetch_fsmap learned FSMap version 33 2024-05-10T11:18:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.395+0000 7f3383b58b80 1 -- 172.21.15.27:0/2740690206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55af51a76e00 con 0x55af51a6a470 2024-05-10T11:18:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.396+0000 7f3357e00700 1 -- 172.21.15.27:0/2740690206 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33640aa330 con 0x55af51a6a470 2024-05-10T11:18:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.396+0000 7f3357e00700 1 -- 172.21.15.27:0/2740690206 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f33640ae160 con 0x55af51a6a470 2024-05-10T11:18:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.396+0000 7f3383b58b80 10 client.26137 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:18:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.396+0000 7f3383b58b80 10 client.26137 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:18:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.396+0000 7f3383b58b80 20 client.26137 populate_metadata read hostname 'smithi027' 2024-05-10T11:18:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.396+0000 7f3383b58b80 1 --2- 172.21.15.27:0/2740690206 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55af51a83890 0x55af51a85cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.396+0000 7f3383b58b80 4 client.26137 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:18:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.396+0000 7f3383b58b80 1 -- 172.21.15.27:0/2740690206 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af51a330a0 con 0x55af51a83890 2024-05-10T11:18:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.396+0000 7f3369400700 1 --2- 172.21.15.27:0/2740690206 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55af51a83890 0x55af51a85cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.397+0000 7f3369400700 1 --2- 172.21.15.27:0/2740690206 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55af51a83890 0x55af51a85cf0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fee3583d83e2d1dd server_cookie=c5c89ac0d3309882 in_seq=0 out_seq=0 2024-05-10T11:18:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.397+0000 7f3357e00700 10 client.26137 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:18:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.398+0000 7f3357e00700 1 -- 172.21.15.27:0/2740690206 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55af51a330a0 con 0x55af51a83890 2024-05-10T11:18:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.398+0000 7f3357e00700 10 client.26137 handle_command_reply: tid=0 2024-05-10T11:18:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.398+0000 7f3383b58b80 1 client.26137 shutdown 2024-05-10T11:18:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.398+0000 7f3383b58b80 1 -- 172.21.15.27:0/2740690206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 msgr2=0x55af51a76270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.398+0000 7f3383b58b80 1 --2- 172.21.15.27:0/2740690206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a76270 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55af51a76c90 tx=0x7f33640ae6e0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.398+0000 7f3383b58b80 1 -- 172.21.15.27:0/2740690206 shutdown_connections 2024-05-10T11:18:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.398+0000 7f3383b58b80 1 --2- 172.21.15.27:0/2740690206 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55af51a83890 0x55af51a85cf0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.398+0000 7f3383b58b80 1 --2- 172.21.15.27:0/2740690206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a76270 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.398+0000 7f3383b58b80 1 --2- 172.21.15.27:0/2740690206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a73bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.398+0000 7f3383b58b80 1 --2- 172.21.15.27:0/2740690206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a73680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.398+0000 7f3357e00700 0 client.26137 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:18:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.398+0000 7f3383b58b80 1 -- 172.21.15.27:0/2740690206 >> 172.21.15.27:0/2740690206 conn(0x55af51994b60 msgr2=0x55af51a5b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:54.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.399+0000 7f3383b58b80 1 -- 172.21.15.27:0/2740690206 shutdown_connections 2024-05-10T11:18:54.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.399+0000 7f3383b58b80 1 -- 172.21.15.27:0/2740690206 wait complete. 2024-05-10T11:18:54.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.399+0000 7f3383b58b80 20 client.26137 trim_cache size 0 max 16384 2024-05-10T11:18:54.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.419+0000 7f3383b58b80 1 Processor -- start 2024-05-10T11:18:54.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.419+0000 7f3383b58b80 1 -- start start 2024-05-10T11:18:54.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.420+0000 7f3383b58b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a71290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.420+0000 7f3383b58b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a72fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.420+0000 7f3383b58b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a7fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.420+0000 7f3383b58b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33741ad290 con 0x55af51a6ad90 2024-05-10T11:18:54.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.420+0000 7f3383b58b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f33741aedd0 con 0x55af51a6ede0 2024-05-10T11:18:54.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.420+0000 7f3383b58b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f33741ad960 con 0x55af51a6a470 2024-05-10T11:18:54.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.420+0000 7f3371e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a71290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.420+0000 7f3372800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a7fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.420+0000 7f3372800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a7fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51394/0 (socket says 172.21.15.27:51394) 2024-05-10T11:18:54.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.420+0000 7f3371e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a71290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49460/0 (socket says 172.21.15.27:49460) 2024-05-10T11:18:54.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.420+0000 7f3372800700 1 -- 172.21.15.27:0/1830009833 learned_addr learned my addr 172.21.15.27:0/1830009833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:54.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.420+0000 7f3369400700 1 --2- 172.21.15.27:0/1830009833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a72fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.421+0000 7f3372800700 1 -- 172.21.15.27:0/1830009833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 msgr2=0x55af51a71290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:54.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.421+0000 7f3372800700 1 --2- 172.21.15.27:0/1830009833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a71290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.421+0000 7f3372800700 1 -- 172.21.15.27:0/1830009833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 msgr2=0x55af51a72fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:54.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.421+0000 7f3372800700 1 --2- 172.21.15.27:0/1830009833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a72fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.421+0000 7f3372800700 1 -- 172.21.15.27:0/1830009833 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af51a5d3f0 con 0x55af51a6ede0 2024-05-10T11:18:54.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.421+0000 7f3372800700 1 --2- 172.21.15.27:0/1830009833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a7fe50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f336c009370 tx=0x7f336c009100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=158e1300eecb590d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:54.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.421+0000 7f3362a00700 1 -- 172.21.15.27:0/1830009833 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c017470 con 0x55af51a6ede0 2024-05-10T11:18:54.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.421+0000 7f3362a00700 1 -- 172.21.15.27:0/1830009833 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f336c029070 con 0x55af51a6ede0 2024-05-10T11:18:54.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.421+0000 7f3362a00700 1 -- 172.21.15.27:0/1830009833 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c018970 con 0x55af51a6ede0 2024-05-10T11:18:54.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.422+0000 7f3383b58b80 1 -- 172.21.15.27:0/1830009833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 msgr2=0x55af51a7fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:54.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.422+0000 7f3383b58b80 1 --2- 172.21.15.27:0/1830009833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a7fe50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f336c009370 tx=0x7f336c009100 comp rx=0 tx=0).stop 2024-05-10T11:18:54.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.422+0000 7f3383b58b80 1 -- 172.21.15.27:0/1830009833 shutdown_connections 2024-05-10T11:18:54.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.422+0000 7f3383b58b80 1 --2- 172.21.15.27:0/1830009833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a71290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.423+0000 7f3383b58b80 1 --2- 172.21.15.27:0/1830009833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a7fe50 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.423+0000 7f3383b58b80 1 --2- 172.21.15.27:0/1830009833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a72fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.423+0000 7f3383b58b80 1 -- 172.21.15.27:0/1830009833 >> 172.21.15.27:0/1830009833 conn(0x55af51994b60 msgr2=0x55af51a5aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:54.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.423+0000 7f3383b58b80 1 -- 172.21.15.27:0/1830009833 shutdown_connections 2024-05-10T11:18:54.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.423+0000 7f3383b58b80 1 -- 172.21.15.27:0/1830009833 wait complete. 2024-05-10T11:18:54.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.423+0000 7f3383b58b80 1 -- start start 2024-05-10T11:18:54.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.424+0000 7f3383b58b80 1 Processor -- start 2024-05-10T11:18:54.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.424+0000 7f3383b58b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a67060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.424+0000 7f3383b58b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a675a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.424+0000 7f3383b58b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a7b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.424+0000 7f3383b58b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33741ae700 con 0x55af51a6ad90 2024-05-10T11:18:54.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.424+0000 7f3383b58b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f33741ae030 con 0x55af51a6ede0 2024-05-10T11:18:54.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.424+0000 7f3383b58b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f33741ab230 con 0x55af51a6a470 2024-05-10T11:18:54.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.424+0000 7f3383b58b80 10 client.0 _ll_register_callbacks cb 0x55af5195fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:18:54.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.424+0000 7f3371e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a67060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.424+0000 7f3371e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a67060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49470/0 (socket says 172.21.15.27:49470) 2024-05-10T11:18:54.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.424+0000 7f3371e00700 1 -- 172.21.15.27:0/4188477925 learned_addr learned my addr 172.21.15.27:0/4188477925 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:54.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3369400700 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a675a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3372800700 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a7b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3371e00700 1 -- 172.21.15.27:0/4188477925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 msgr2=0x55af51a7b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:54.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3371e00700 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a7b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3371e00700 1 -- 172.21.15.27:0/4188477925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 msgr2=0x55af51a675a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3371e00700 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a675a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3371e00700 1 -- 172.21.15.27:0/4188477925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af51a7b700 con 0x55af51a6a470 2024-05-10T11:18:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3372800700 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a7b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:18:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3371e00700 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a67060 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f33640a63b0 tx=0x7f3364009480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e2bb6e2573bd6f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3357e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:18:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3357e00700 1 -- 172.21.15.27:0/4188477925 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33640c4070 con 0x55af51a6a470 2024-05-10T11:18:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3357e00700 1 -- 172.21.15.27:0/4188477925 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f33640aa7a0 con 0x55af51a6a470 2024-05-10T11:18:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.425+0000 7f3383b58b80 1 -- 172.21.15.27:0/4188477925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f336c029070 con 0x55af51a6a470 2024-05-10T11:18:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.426+0000 7f3357e00700 1 -- 172.21.15.27:0/4188477925 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f33640a9aa0 con 0x55af51a6a470 2024-05-10T11:18:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.426+0000 7f3383b58b80 10 client.26143 fetch_fsmap learned FSMap version 33 2024-05-10T11:18:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.426+0000 7f3383b58b80 1 -- 172.21.15.27:0/4188477925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55af51a7b700 con 0x55af51a6a470 2024-05-10T11:18:54.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.426+0000 7f3357e00700 1 -- 172.21.15.27:0/4188477925 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f336c029070 con 0x55af51a6a470 2024-05-10T11:18:54.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.426+0000 7f3357e00700 1 -- 172.21.15.27:0/4188477925 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f3364004ad0 con 0x55af51a6a470 2024-05-10T11:18:54.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.426+0000 7f3383b58b80 10 client.26143 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:18:54.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.426+0000 7f3383b58b80 10 client.26143 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:18:54.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.426+0000 7f3383b58b80 20 client.26143 populate_metadata read hostname 'smithi027' 2024-05-10T11:18:54.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.426+0000 7f3383b58b80 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55af51af29b0 0x55af51af4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:54.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.426+0000 7f3383b58b80 4 client.26143 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:18:54.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.426+0000 7f3383b58b80 1 -- 172.21.15.27:0/4188477925 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55af519c6400 con 0x55af51af29b0 2024-05-10T11:18:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.427+0000 7f3369400700 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55af51af29b0 0x55af51af4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.427+0000 7f3369400700 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55af51af29b0 0x55af51af4da0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d82763f1a38b282a server_cookie=95aeab59262e77ce in_seq=0 out_seq=0 2024-05-10T11:18:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.427+0000 7f3357e00700 10 client.26143 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:18:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.427+0000 7f3357e00700 1 -- 172.21.15.27:0/4188477925 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55af519c6400 con 0x55af51af29b0 2024-05-10T11:18:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.427+0000 7f3357e00700 10 client.26143 handle_command_reply: tid=0 2024-05-10T11:18:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.427+0000 7f3383b58b80 1 client.26143 shutdown 2024-05-10T11:18:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.428+0000 7f3383b58b80 1 -- 172.21.15.27:0/4188477925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 msgr2=0x55af51a67060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.428+0000 7f3383b58b80 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a67060 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f33640a63b0 tx=0x7f3364009480 comp rx=0 tx=0).stop 2024-05-10T11:18:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.428+0000 7f3383b58b80 1 -- 172.21.15.27:0/4188477925 shutdown_connections 2024-05-10T11:18:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.428+0000 7f3383b58b80 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55af51af29b0 0x55af51af4da0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.428+0000 7f3383b58b80 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af51a6a470 0x55af51a67060 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.428+0000 7f3357e00700 0 client.26143 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:18:54.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.428+0000 7f3383b58b80 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55af51a6ede0 0x55af51a7b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.428+0000 7f3383b58b80 1 --2- 172.21.15.27:0/4188477925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55af51a6ad90 0x55af51a675a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.428+0000 7f3383b58b80 1 -- 172.21.15.27:0/4188477925 >> 172.21.15.27:0/4188477925 conn(0x55af51994b60 msgr2=0x55af51a5b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:54.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.428+0000 7f3383b58b80 1 -- 172.21.15.27:0/4188477925 shutdown_connections 2024-05-10T11:18:54.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.428+0000 7f3383b58b80 1 -- 172.21.15.27:0/4188477925 wait complete. 2024-05-10T11:18:54.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.428+0000 7f3383b58b80 20 client.26143 trim_cache size 0 max 16384 2024-05-10T11:18:54.437 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:18:54.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.429+0000 7f3379200700 1 -- 172.21.15.27:0/266263301 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f33580b29f0 msgr2=0x7f33580b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:54.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.429+0000 7f3379200700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f33580b29f0 0x7f33580b4eb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f336c003f60 tx=0x7f336c003f90 comp rx=0 tx=0).stop 2024-05-10T11:18:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.429+0000 7f3379200700 1 -- 172.21.15.27:0/266263301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3374103750 msgr2=0x7f337419f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.429+0000 7f3379200700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3374103750 0x7f337419f7a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3364009230 tx=0x7f336400fa70 comp rx=0 tx=0).stop 2024-05-10T11:18:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.429+0000 7f3379200700 1 -- 172.21.15.27:0/266263301 shutdown_connections 2024-05-10T11:18:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.429+0000 7f3379200700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f33580b29f0 0x7f33580b4eb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.429+0000 7f3379200700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3374108850 0x7f3374199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.429+0000 7f3379200700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3374103750 0x7f337419f7a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.429+0000 7f3379200700 1 --2- 172.21.15.27:0/266263301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374104100 0x7f337419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.429+0000 7f3379200700 1 -- 172.21.15.27:0/266263301 >> 172.21.15.27:0/266263301 conn(0x7f337407a600 msgr2=0x7f3374101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.430+0000 7f3379200700 1 -- 172.21.15.27:0/266263301 shutdown_connections 2024-05-10T11:18:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:54.430+0000 7f3379200700 1 -- 172.21.15.27:0/266263301 wait complete. 2024-05-10T11:18:54.677 INFO:tasks.workunit.client.0.smithi027.stdout: 1 309345 7.73 MB/sec execute 99 sec latency 24.937 ms 2024-05-10T11:18:55.598 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:18:55.677 INFO:tasks.workunit.client.0.smithi027.stdout: 1 311606 7.77 MB/sec execute 100 sec latency 18.073 ms 2024-05-10T11:18:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:18:56.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:56 smithi027 ceph-mon[30054]: pgmap v174: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 240 op/s 2024-05-10T11:18:56.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:56 smithi076 ceph-mon[34152]: pgmap v174: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 240 op/s 2024-05-10T11:18:56.677 INFO:tasks.workunit.client.0.smithi027.stdout: 1 312558 7.76 MB/sec execute 101 sec latency 55.122 ms 2024-05-10T11:18:56.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:56 smithi189 ceph-mon[34842]: pgmap v174: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 240 op/s 2024-05-10T11:18:57.677 INFO:tasks.workunit.client.0.smithi027.stdout: 1 313714 7.73 MB/sec execute 102 sec latency 35.697 ms 2024-05-10T11:18:58.567 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:18:58 smithi027 ceph-mon[30054]: pgmap v175: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 158 op/s 2024-05-10T11:18:58.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:18:58 smithi076 ceph-mon[34152]: pgmap v175: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 158 op/s 2024-05-10T11:18:58.677 INFO:tasks.workunit.client.0.smithi027.stdout: 1 315840 7.78 MB/sec execute 103 sec latency 21.750 ms 2024-05-10T11:18:58.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:18:58 smithi189 ceph-mon[34842]: pgmap v175: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 158 op/s 2024-05-10T11:18:58.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.750+0000 7f82ed200700 1 -- 172.21.15.27:0/2841743551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82e810b6e0 msgr2=0x7f82e810bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:58.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.750+0000 7f82ed200700 1 --2- 172.21.15.27:0/2841743551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82e810b6e0 0x7f82e810bac0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f82d8009990 tx=0x7f82d8009ca0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.750+0000 7f82ed200700 1 -- 172.21.15.27:0/2841743551 shutdown_connections 2024-05-10T11:18:58.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.750+0000 7f82ed200700 1 --2- 172.21.15.27:0/2841743551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82e810b6e0 0x7f82e810bac0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.750+0000 7f82ed200700 1 --2- 172.21.15.27:0/2841743551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e81060a0 0x7f82e81125b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.750+0000 7f82ed200700 1 --2- 172.21.15.27:0/2841743551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82e81056e0 0x7f82e8105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.750+0000 7f82ed200700 1 -- 172.21.15.27:0/2841743551 >> 172.21.15.27:0/2841743551 conn(0x7f82e81011d0 msgr2=0x7f82e81035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:58.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.750+0000 7f82ed200700 1 -- 172.21.15.27:0/2841743551 shutdown_connections 2024-05-10T11:18:58.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.751+0000 7f82ed200700 1 -- 172.21.15.27:0/2841743551 wait complete. 2024-05-10T11:18:58.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.752+0000 7f82ed200700 1 Processor -- start 2024-05-10T11:18:58.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.752+0000 7f82ed200700 1 -- start start 2024-05-10T11:18:58.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82ed200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82e81056e0 0x7f82e819bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82ed200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82e81060a0 0x7f82e819c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82ed200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e810b6e0 0x7f82e81a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82ed200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82e8115930 con 0x7f82e810b6e0 2024-05-10T11:18:58.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82ed200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81157b0 con 0x7f82e81056e0 2024-05-10T11:18:58.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82ed200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f82e8115ab0 con 0x7f82e81060a0 2024-05-10T11:18:58.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82e5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82e81060a0 0x7f82e819c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82e5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82e81056e0 0x7f82e819bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82e5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82e81060a0 0x7f82e819c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49494/0 (socket says 172.21.15.27:49494) 2024-05-10T11:18:58.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82e5400700 1 -- 172.21.15.27:0/2919133971 learned_addr learned my addr 172.21.15.27:0/2919133971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:58.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82e6800700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e810b6e0 0x7f82e81a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82e5400700 1 -- 172.21.15.27:0/2919133971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82e81056e0 msgr2=0x7f82e819bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82e5400700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82e81056e0 0x7f82e819bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82e5400700 1 -- 172.21.15.27:0/2919133971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e810b6e0 msgr2=0x7f82e81a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82e5400700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e810b6e0 0x7f82e81a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.753+0000 7f82e5400700 1 -- 172.21.15.27:0/2919133971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e81a0df0 con 0x7f82e81060a0 2024-05-10T11:18:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.754+0000 7f82e6800700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e810b6e0 0x7f82e81a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:18:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.754+0000 7f82e5e00700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82e81056e0 0x7f82e819bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:18:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.754+0000 7f82e5400700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82e81060a0 0x7f82e819c320 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f82d400e9a0 tx=0x7f82d400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.754+0000 7f82dea00700 1 -- 172.21.15.27:0/2919133971 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d400cbd0 con 0x7f82e81060a0 2024-05-10T11:18:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.754+0000 7f82dea00700 1 -- 172.21.15.27:0/2919133971 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d400cd70 con 0x7f82e81060a0 2024-05-10T11:18:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.754+0000 7f82dea00700 1 -- 172.21.15.27:0/2919133971 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4018680 con 0x7f82e81060a0 2024-05-10T11:18:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.754+0000 7f82ed200700 1 -- 172.21.15.27:0/2919133971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e81a10e0 con 0x7f82e81060a0 2024-05-10T11:18:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.754+0000 7f82ed200700 1 -- 172.21.15.27:0/2919133971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e81a8960 con 0x7f82e81060a0 2024-05-10T11:18:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.759+0000 7f82dea00700 1 -- 172.21.15.27:0/2919133971 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f82d4010430 con 0x7f82e81060a0 2024-05-10T11:18:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.760+0000 7f82f7ad3b80 1 Processor -- start 2024-05-10T11:18:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.760+0000 7f82f7ad3b80 1 -- start start 2024-05-10T11:18:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82f7ad3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a4130cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82f7ad3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a413072d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82f7ad3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a41307d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82f7ad3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562a41259a20 con 0x562a413078f0 2024-05-10T11:18:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82f7ad3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562a411f7760 con 0x562a4130cbc0 2024-05-10T11:18:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82f7ad3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562a412559e0 con 0x562a4130d4e0 2024-05-10T11:18:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82e6800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a4130cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82e5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a413072d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82e6800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a4130cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51436/0 (socket says 172.21.15.27:51436) 2024-05-10T11:18:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82e6800700 1 -- 172.21.15.27:0/682282690 learned_addr learned my addr 172.21.15.27:0/682282690 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82e5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a413072d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49496/0 (socket says 172.21.15.27:49496) 2024-05-10T11:18:58.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82e5e00700 1 -- 172.21.15.27:0/682282690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 msgr2=0x562a4130cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:58.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.761+0000 7f82e5e00700 1 --2- 172.21.15.27:0/682282690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a4130cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.762+0000 7f82e5e00700 1 -- 172.21.15.27:0/682282690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 msgr2=0x562a41307d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:58.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.762+0000 7f82e5400700 1 --2- 172.21.15.27:0/682282690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a41307d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.762+0000 7f82e5e00700 1 --2- 172.21.15.27:0/682282690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a41307d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.762+0000 7f82e5e00700 1 -- 172.21.15.27:0/682282690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a411a68d0 con 0x562a4130d4e0 2024-05-10T11:18:58.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.762+0000 7f82e5400700 1 --2- 172.21.15.27:0/682282690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a41307d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:18:58.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.762+0000 7f82dea00700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f82cc0b2a70 0x7f82cc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.762+0000 7f82e5e00700 1 --2- 172.21.15.27:0/682282690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a413072d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f82d80098e0 tx=0x7f82d80096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=421b0468ef120a1c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.762+0000 7f82dea00700 1 -- 172.21.15.27:0/2919133971 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f82d40e2c60 con 0x7f82e81060a0 2024-05-10T11:18:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.762+0000 7f82e6800700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f82cc0b2a70 0x7f82cc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.762+0000 7f82d3400700 1 -- 172.21.15.27:0/682282690 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8027070 con 0x562a4130d4e0 2024-05-10T11:18:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.762+0000 7f82d3400700 1 -- 172.21.15.27:0/682282690 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d8003680 con 0x562a4130d4e0 2024-05-10T11:18:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.763+0000 7f82d3400700 1 -- 172.21.15.27:0/682282690 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d800f6c0 con 0x562a4130d4e0 2024-05-10T11:18:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.763+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/682282690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 msgr2=0x562a413072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.763+0000 7f82e6800700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f82cc0b2a70 0x7f82cc0b4f30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f82e00076f0 tx=0x7f82e0009040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.763+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/682282690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a413072d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f82d80098e0 tx=0x7f82d80096a0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.764+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/682282690 shutdown_connections 2024-05-10T11:18:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.764+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/682282690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a41307d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.764+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/682282690 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a4130cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.764+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/682282690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a413072d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.764+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/682282690 >> 172.21.15.27:0/682282690 conn(0x562a4127f6d0 msgr2=0x562a41301b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.764+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/682282690 shutdown_connections 2024-05-10T11:18:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.764+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/682282690 wait complete. 2024-05-10T11:18:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.764+0000 7f82f7ad3b80 1 -- start start 2024-05-10T11:18:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.765+0000 7f82f7ad3b80 1 Processor -- start 2024-05-10T11:18:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.765+0000 7f82f7ad3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a413078f0 0x562a41313180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.765+0000 7f82f7ad3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a41310520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.765+0000 7f82f7ad3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a4130d4e0 0x562a41310a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.765+0000 7f82f7ad3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562a41313ec0 con 0x562a4130d4e0 2024-05-10T11:18:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.765+0000 7f82f7ad3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562a411bfb20 con 0x562a4130cbc0 2024-05-10T11:18:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.765+0000 7f82f7ad3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562a41314040 con 0x562a413078f0 2024-05-10T11:18:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.765+0000 7f82f7ad3b80 10 client.0 _ll_register_callbacks cb 0x562a41200c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:18:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.766+0000 7f82e5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a413078f0 0x562a41313180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.766+0000 7f82e5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a413078f0 0x562a41313180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49512/0 (socket says 172.21.15.27:49512) 2024-05-10T11:18:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.766+0000 7f82e5e00700 1 -- 172.21.15.27:0/2683260966 learned_addr learned my addr 172.21.15.27:0/2683260966 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.766+0000 7f82e5400700 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a41310520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.766+0000 7f82e6800700 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a4130d4e0 0x562a41310a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.766+0000 7f82e5e00700 1 -- 172.21.15.27:0/2683260966 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 msgr2=0x562a41310520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:58.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.766+0000 7f82e5e00700 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a41310520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.766+0000 7f82e5e00700 1 -- 172.21.15.27:0/2683260966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a4130d4e0 msgr2=0x562a41310a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:58.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.766+0000 7f82e5e00700 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a4130d4e0 0x562a41310a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.766+0000 7f82e5e00700 1 -- 172.21.15.27:0/2683260966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a4121f420 con 0x562a413078f0 2024-05-10T11:18:58.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.766+0000 7f82e6800700 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a4130d4e0 0x562a41310a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:18:58.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.767+0000 7f82e5e00700 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a413078f0 0x562a41313180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f82d8000c00 tx=0x7f82d8003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2ed84f263b39c63 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:58.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.767+0000 7f82c7e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:18:58.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.767+0000 7f82c7e00700 1 -- 172.21.15.27:0/2683260966 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8030060 con 0x562a413078f0 2024-05-10T11:18:58.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.767+0000 7f82c7e00700 1 -- 172.21.15.27:0/2683260966 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f82d80052c0 con 0x562a413078f0 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.767+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2683260966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82d8003680 con 0x562a413078f0 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.767+0000 7f82c7e00700 1 -- 172.21.15.27:0/2683260966 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f82d8037050 con 0x562a413078f0 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.767+0000 7f82f7ad3b80 10 client.26155 fetch_fsmap learned FSMap version 33 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.767+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2683260966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562a41313ec0 con 0x562a413078f0 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.768+0000 7f82c7e00700 1 -- 172.21.15.27:0/2683260966 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d8003680 con 0x562a413078f0 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.768+0000 7f82c7e00700 1 -- 172.21.15.27:0/2683260966 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f82d8021030 con 0x562a413078f0 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.768+0000 7f82f7ad3b80 10 client.26155 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.768+0000 7f82f7ad3b80 10 client.26155 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.768+0000 7f82f7ad3b80 20 client.26155 populate_metadata read hostname 'smithi027' 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.768+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a41320950 0x562a41322d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.768+0000 7f82f7ad3b80 4 client.26155 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.768+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2683260966 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a412d4140 con 0x562a41320950 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.768+0000 7f82e5400700 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a41320950 0x562a41322d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.769+0000 7f82c7e00700 10 client.26155 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:18:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.769+0000 7f82e5400700 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a41320950 0x562a41322d40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9052939ef96b688b server_cookie=ca89a1e6f9a76f4 in_seq=0 out_seq=0 2024-05-10T11:18:58.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.770+0000 7f82c7e00700 1 -- 172.21.15.27:0/2683260966 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562a412d4140 con 0x562a41320950 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.770+0000 7f82c7e00700 10 client.26155 handle_command_reply: tid=0 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.770+0000 7f82f7ad3b80 1 client.26155 shutdown 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.770+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2683260966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a413078f0 msgr2=0x562a41313180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.770+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a413078f0 0x562a41313180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f82d8000c00 tx=0x7f82d8003eb0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.770+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2683260966 shutdown_connections 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.770+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a41320950 0x562a41322d40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.770+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a413078f0 0x562a41313180 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.770+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a4130d4e0 0x562a41310a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.770+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/2683260966 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a41310520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.770+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2683260966 >> 172.21.15.27:0/2683260966 conn(0x562a4127f6d0 msgr2=0x562a41301190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.770+0000 7f82c7e00700 0 client.26155 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.771+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2683260966 shutdown_connections 2024-05-10T11:18:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.771+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2683260966 wait complete. 2024-05-10T11:18:58.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.771+0000 7f82f7ad3b80 20 client.26155 trim_cache size 0 max 16384 2024-05-10T11:18:58.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.792+0000 7f82f7ad3b80 1 Processor -- start 2024-05-10T11:18:58.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.793+0000 7f82f7ad3b80 1 -- start start 2024-05-10T11:18:58.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.793+0000 7f82f7ad3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a4131cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.793+0000 7f82f7ad3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a41301910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.793+0000 7f82f7ad3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a4130ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.793+0000 7f82f7ad3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81b1780 con 0x562a413078f0 2024-05-10T11:18:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.794+0000 7f82e5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a4131cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.794+0000 7f82e5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a41301910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.794+0000 7f82e5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a4131cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44402/0 (socket says 172.21.15.27:44402) 2024-05-10T11:18:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.794+0000 7f82e5e00700 1 -- 172.21.15.27:0/984726145 learned_addr learned my addr 172.21.15.27:0/984726145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.793+0000 7f82f7ad3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81b32c0 con 0x562a4130cbc0 2024-05-10T11:18:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.794+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/984726145 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81b1e50 con 0x562a4130d4e0 2024-05-10T11:18:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.794+0000 7f82e6800700 1 --2- 172.21.15.27:0/984726145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a4130ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.795+0000 7f82e5400700 1 -- 172.21.15.27:0/984726145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 msgr2=0x562a4130ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.795+0000 7f82e5400700 1 --2- 172.21.15.27:0/984726145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a4130ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.795+0000 7f82e5400700 1 -- 172.21.15.27:0/984726145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 msgr2=0x562a4131cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.795+0000 7f82e5400700 1 --2- 172.21.15.27:0/984726145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a4131cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.795+0000 7f82e5400700 1 -- 172.21.15.27:0/984726145 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a41307610 con 0x562a4130cbc0 2024-05-10T11:18:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.795+0000 7f82e5400700 1 --2- 172.21.15.27:0/984726145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a41301910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f82d40dff80 tx=0x7f82d40e50b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22756bec0c2b9cdd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.796+0000 7f82d2a00700 1 -- 172.21.15.27:0/984726145 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d40b22b0 con 0x562a4130cbc0 2024-05-10T11:18:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.796+0000 7f82d2a00700 1 -- 172.21.15.27:0/984726145 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d40a9620 con 0x562a4130cbc0 2024-05-10T11:18:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.796+0000 7f82d2a00700 1 -- 172.21.15.27:0/984726145 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d40a9d70 con 0x562a4130cbc0 2024-05-10T11:18:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.797+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/984726145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 msgr2=0x562a41301910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.797+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/984726145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a41301910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f82d40dff80 tx=0x7f82d40e50b0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.798+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/984726145 shutdown_connections 2024-05-10T11:18:58.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.798+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/984726145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a4130ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.798+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/984726145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a4131cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.798+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/984726145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a41301910 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.798+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/984726145 >> 172.21.15.27:0/984726145 conn(0x562a4127f6d0 msgr2=0x562a41312460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:58.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.798+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/984726145 shutdown_connections 2024-05-10T11:18:58.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.798+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/984726145 wait complete. 2024-05-10T11:18:58.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.799+0000 7f82f7ad3b80 1 -- start start 2024-05-10T11:18:58.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.799+0000 7f82f7ad3b80 1 Processor -- start 2024-05-10T11:18:58.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.800+0000 7f82f7ad3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a41314e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.800+0000 7f82f7ad3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a413153b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.800+0000 7f82f7ad3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a413158f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.800+0000 7f82f7ad3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81b2bf0 con 0x562a413078f0 2024-05-10T11:18:58.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.800+0000 7f82f7ad3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81b2520 con 0x562a4130cbc0 2024-05-10T11:18:58.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.800+0000 7f82f7ad3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81af720 con 0x562a4130d4e0 2024-05-10T11:18:58.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.800+0000 7f82f7ad3b80 10 client.0 _ll_register_callbacks cb 0x562a41200c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:18:58.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.801+0000 7f82e5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a41314e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.801+0000 7f82e5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a41314e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44418/0 (socket says 172.21.15.27:44418) 2024-05-10T11:18:58.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.801+0000 7f82e5e00700 1 -- 172.21.15.27:0/2767186018 learned_addr learned my addr 172.21.15.27:0/2767186018 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:18:58.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.801+0000 7f82e5400700 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a413153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.802+0000 7f82e6800700 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a413158f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.803+0000 7f82e5e00700 1 -- 172.21.15.27:0/2767186018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 msgr2=0x562a413158f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:58.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.803+0000 7f82e5e00700 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a413158f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.803+0000 7f82e5e00700 1 -- 172.21.15.27:0/2767186018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 msgr2=0x562a413153b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:18:58.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.803+0000 7f82e5e00700 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a413153b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.803+0000 7f82e5e00700 1 -- 172.21.15.27:0/2767186018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a41382f70 con 0x562a413078f0 2024-05-10T11:18:58.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.803+0000 7f82e6800700 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a413158f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:18:58.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.803+0000 7f82e5400700 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a413153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:18:58.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.804+0000 7f82e5e00700 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a41314e70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f82d8000c00 tx=0x7f82d8022ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f4f0c13201a611 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:18:58.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.805+0000 7f82c7e00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:18:58.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.805+0000 7f82c7e00700 1 -- 172.21.15.27:0/2767186018 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8030070 con 0x562a413078f0 2024-05-10T11:18:58.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.805+0000 7f82c7e00700 1 -- 172.21.15.27:0/2767186018 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f82d80050b0 con 0x562a413078f0 2024-05-10T11:18:58.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.806+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2767186018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82d40a9620 con 0x562a413078f0 2024-05-10T11:18:58.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.806+0000 7f82c7e00700 1 -- 172.21.15.27:0/2767186018 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f82d8030070 con 0x562a413078f0 2024-05-10T11:18:58.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.806+0000 7f82f7ad3b80 10 client.15972 fetch_fsmap learned FSMap version 33 2024-05-10T11:18:58.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.806+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2767186018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562a41384250 con 0x562a413078f0 2024-05-10T11:18:58.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.808+0000 7f82c7e00700 1 -- 172.21.15.27:0/2767186018 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d8003680 con 0x562a413078f0 2024-05-10T11:18:58.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.808+0000 7f82c7e00700 1 -- 172.21.15.27:0/2767186018 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f82d8021030 con 0x562a413078f0 2024-05-10T11:18:58.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.808+0000 7f82f7ad3b80 10 client.15972 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:18:58.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.808+0000 7f82f7ad3b80 10 client.15972 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:18:58.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.808+0000 7f82f7ad3b80 20 client.15972 populate_metadata read hostname 'smithi027' 2024-05-10T11:18:58.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.808+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a413847f0 0x562a41386c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:18:58.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.808+0000 7f82f7ad3b80 4 client.15972 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:18:58.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.808+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2767186018 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f82d40a9620 con 0x562a413847f0 2024-05-10T11:18:58.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.809+0000 7f82e5400700 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a413847f0 0x562a41386c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:18:58.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.810+0000 7f82e5400700 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a413847f0 0x562a41386c10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a4bdab25ad6a5f83 server_cookie=983115fe1c73bd13 in_seq=0 out_seq=0 2024-05-10T11:18:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.810+0000 7f82c7e00700 10 client.15972 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:18:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.810+0000 7f82c7e00700 1 -- 172.21.15.27:0/2767186018 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f82d40a9620 con 0x562a413847f0 2024-05-10T11:18:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.810+0000 7f82c7e00700 10 client.15972 handle_command_reply: tid=0 2024-05-10T11:18:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.811+0000 7f82f7ad3b80 1 client.15972 shutdown 2024-05-10T11:18:58.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.811+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2767186018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 msgr2=0x562a41314e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:58.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.811+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a41314e70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f82d8000c00 tx=0x7f82d8022ac0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.812+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2767186018 shutdown_connections 2024-05-10T11:18:58.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.812+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x562a413847f0 0x562a41386c10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.812+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a4130d4e0 0x562a413158f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.812+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a413078f0 0x562a41314e70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.812+0000 7f82f7ad3b80 1 --2- 172.21.15.27:0/2767186018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562a4130cbc0 0x562a413153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.812+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2767186018 >> 172.21.15.27:0/2767186018 conn(0x562a4127f6d0 msgr2=0x562a413062c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:58.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.812+0000 7f82c7e00700 0 client.15972 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:18:58.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.812+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2767186018 shutdown_connections 2024-05-10T11:18:58.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.812+0000 7f82f7ad3b80 1 -- 172.21.15.27:0/2767186018 wait complete. 2024-05-10T11:18:58.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.813+0000 7f82f7ad3b80 20 client.15972 trim_cache size 0 max 16384 2024-05-10T11:18:58.813 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:18:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.814+0000 7f82ed200700 1 -- 172.21.15.27:0/2919133971 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f82cc0b2a70 msgr2=0x7f82cc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.814+0000 7f82ed200700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f82cc0b2a70 0x7f82cc0b4f30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f82e00076f0 tx=0x7f82e0009040 comp rx=0 tx=0).stop 2024-05-10T11:18:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.814+0000 7f82ed200700 1 -- 172.21.15.27:0/2919133971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82e81060a0 msgr2=0x7f82e819c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:18:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.814+0000 7f82ed200700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82e81060a0 0x7f82e819c320 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f82d400e9a0 tx=0x7f82d400ecb0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.815+0000 7f82ed200700 1 -- 172.21.15.27:0/2919133971 shutdown_connections 2024-05-10T11:18:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.815+0000 7f82ed200700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f82cc0b2a70 0x7f82cc0b4f30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.815+0000 7f82ed200700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e810b6e0 0x7f82e81a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.815+0000 7f82ed200700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82e81056e0 0x7f82e819bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.815+0000 7f82ed200700 1 --2- 172.21.15.27:0/2919133971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82e81060a0 0x7f82e819c320 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:18:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.815+0000 7f82ed200700 1 -- 172.21.15.27:0/2919133971 >> 172.21.15.27:0/2919133971 conn(0x7f82e81011d0 msgr2=0x7f82e8101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:18:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.815+0000 7f82ed200700 1 -- 172.21.15.27:0/2919133971 shutdown_connections 2024-05-10T11:18:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:18:58.815+0000 7f82ed200700 1 -- 172.21.15.27:0/2919133971 wait complete. 2024-05-10T11:18:59.677 INFO:tasks.workunit.client.0.smithi027.stdout: 1 316415 7.72 MB/sec execute 104 sec latency 41.906 ms 2024-05-10T11:18:59.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:19:00.396 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:19:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:00 smithi076 ceph-mon[34152]: pgmap v176: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 240 op/s 2024-05-10T11:19:00.678 INFO:tasks.workunit.client.0.smithi027.stdout: 1 317979 7.70 MB/sec execute 105 sec latency 69.877 ms 2024-05-10T11:19:00.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:00 smithi189 ceph-mon[34842]: pgmap v176: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 240 op/s 2024-05-10T11:19:00.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:00 smithi027 ceph-mon[30054]: pgmap v176: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 240 op/s 2024-05-10T11:19:01.678 INFO:tasks.workunit.client.0.smithi027.stdout: 1 318596 7.69 MB/sec execute 106 sec latency 142.788 ms 2024-05-10T11:19:02.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:02 smithi027 ceph-mon[30054]: pgmap v177: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-05-10T11:19:02.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:02 smithi076 ceph-mon[34152]: pgmap v177: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-05-10T11:19:02.678 INFO:tasks.workunit.client.0.smithi027.stdout: 1 319638 7.68 MB/sec execute 107 sec latency 24.371 ms 2024-05-10T11:19:02.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:02 smithi189 ceph-mon[34842]: pgmap v177: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-05-10T11:19:03.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.055+0000 7fb779e00700 1 -- 172.21.15.27:0/2818647648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb774109510 msgr2=0x7fb7741098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:03.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.055+0000 7fb779e00700 1 --2- 172.21.15.27:0/2818647648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb774109510 0x7fb7741098f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb764009990 tx=0x7fb764009ca0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.055+0000 7fb779e00700 1 -- 172.21.15.27:0/2818647648 shutdown_connections 2024-05-10T11:19:03.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.055+0000 7fb779e00700 1 --2- 172.21.15.27:0/2818647648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb774109510 0x7fb7741098f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.055+0000 7fb779e00700 1 --2- 172.21.15.27:0/2818647648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb77406bb40 0x7fb774110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.055+0000 7fb779e00700 1 --2- 172.21.15.27:0/2818647648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb77406b180 0x7fb77406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.055+0000 7fb779e00700 1 -- 172.21.15.27:0/2818647648 >> 172.21.15.27:0/2818647648 conn(0x7fb774079350 msgr2=0x7fb774079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.055+0000 7fb779e00700 1 -- 172.21.15.27:0/2818647648 shutdown_connections 2024-05-10T11:19:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.055+0000 7fb779e00700 1 -- 172.21.15.27:0/2818647648 wait complete. 2024-05-10T11:19:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.056+0000 7fb779e00700 1 Processor -- start 2024-05-10T11:19:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.057+0000 7fb779e00700 1 -- start start 2024-05-10T11:19:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.057+0000 7fb779e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb77406b180 0x7fb77419bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.057+0000 7fb779e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb77406bb40 0x7fb77419c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.057+0000 7fb779e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb774109510 0x7fb7741a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.057+0000 7fb779e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb774113640 con 0x7fb77406bb40 2024-05-10T11:19:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.057+0000 7fb779e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741134c0 con 0x7fb774109510 2024-05-10T11:19:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.057+0000 7fb779e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741137c0 con 0x7fb77406b180 2024-05-10T11:19:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.057+0000 7fb772a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb77406b180 0x7fb77419bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.057+0000 7fb772a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb77406b180 0x7fb77419bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49556/0 (socket says 172.21.15.27:49556) 2024-05-10T11:19:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.057+0000 7fb772a00700 1 -- 172.21.15.27:0/1428633648 learned_addr learned my addr 172.21.15.27:0/1428633648 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.057+0000 7fb773400700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb774109510 0x7fb7741a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.058+0000 7fb76a000700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb77406bb40 0x7fb77419c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.058+0000 7fb772a00700 1 -- 172.21.15.27:0/1428633648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb774109510 msgr2=0x7fb7741a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.058+0000 7fb772a00700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb774109510 0x7fb7741a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.058+0000 7fb772a00700 1 -- 172.21.15.27:0/1428633648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb77406bb40 msgr2=0x7fb77419c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.058+0000 7fb772a00700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb77406bb40 0x7fb77419c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.058+0000 7fb772a00700 1 -- 172.21.15.27:0/1428633648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7741a0d90 con 0x7fb77406b180 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.058+0000 7fb76a000700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb77406bb40 0x7fb77419c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.058+0000 7fb772a00700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb77406b180 0x7fb77419bdd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb764000c00 tx=0x7fb764009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.059+0000 7fb778a00700 1 -- 172.21.15.27:0/1428633648 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb764027070 con 0x7fb77406b180 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.059+0000 7fb778a00700 1 -- 172.21.15.27:0/1428633648 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb764022470 con 0x7fb77406b180 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.059+0000 7fb779e00700 1 -- 172.21.15.27:0/1428633648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7741a1020 con 0x7fb77406b180 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.059+0000 7fb778a00700 1 -- 172.21.15.27:0/1428633648 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb764017650 con 0x7fb77406b180 2024-05-10T11:19:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.059+0000 7fb779e00700 1 -- 172.21.15.27:0/1428633648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7741a8900 con 0x7fb77406b180 2024-05-10T11:19:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.062+0000 7fb778a00700 1 -- 172.21.15.27:0/1428633648 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb76402a070 con 0x7fb77406b180 2024-05-10T11:19:03.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.064+0000 7fb778a00700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb7540b2a70 0x7fb7540b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.064+0000 7fb78476cb80 1 Processor -- start 2024-05-10T11:19:03.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.065+0000 7fb778a00700 1 -- 172.21.15.27:0/1428633648 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fb7640e3780 con 0x7fb77406b180 2024-05-10T11:19:03.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.065+0000 7fb76a000700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb7540b2a70 0x7fb7540b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.065+0000 7fb78476cb80 1 -- start start 2024-05-10T11:19:03.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.065+0000 7fb78476cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f8769bc0 0x5654f8769fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.065+0000 7fb78476cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f876a4e0 0x5654f87642d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.065+0000 7fb78476cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f8764d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.065+0000 7fb78476cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5654f86b6a20 con 0x5654f87648f0 2024-05-10T11:19:03.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.065+0000 7fb78476cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5654f8654760 con 0x5654f876a4e0 2024-05-10T11:19:03.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.065+0000 7fb78476cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5654f86b29e0 con 0x5654f8769bc0 2024-05-10T11:19:03.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb76a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f876a4e0 0x5654f87642d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb772a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f8769bc0 0x5654f8769fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb76a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f876a4e0 0x5654f87642d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51486/0 (socket says 172.21.15.27:51486) 2024-05-10T11:19:03.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb772a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f8769bc0 0x5654f8769fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49564/0 (socket says 172.21.15.27:49564) 2024-05-10T11:19:03.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb76a000700 1 -- 172.21.15.27:0/3800930174 learned_addr learned my addr 172.21.15.27:0/3800930174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:03.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb773400700 1 --2- 172.21.15.27:0/3800930174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f8764d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb76a000700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb7540b2a70 0x7fb7540b4f30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb774110770 tx=0x7fb75c009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:03.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb772a00700 1 -- 172.21.15.27:0/3800930174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f876a4e0 msgr2=0x5654f87642d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:03.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb772a00700 1 --2- 172.21.15.27:0/3800930174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f876a4e0 0x5654f87642d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb772a00700 1 -- 172.21.15.27:0/3800930174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 msgr2=0x5654f8764d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:03.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb772a00700 1 --2- 172.21.15.27:0/3800930174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f8764d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb772a00700 1 -- 172.21.15.27:0/3800930174 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654f86038d0 con 0x5654f8769bc0 2024-05-10T11:19:03.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb76a000700 1 --2- 172.21.15.27:0/3800930174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f876a4e0 0x5654f87642d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:19:03.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.066+0000 7fb772a00700 1 --2- 172.21.15.27:0/3800930174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f8769bc0 0x5654f8769fa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb7640e0aa0 tx=0x7fb7640a9ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f16136ee8541eaf server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:03.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.067+0000 7fb768c00700 1 -- 172.21.15.27:0/3800930174 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7640aa6c0 con 0x5654f8769bc0 2024-05-10T11:19:03.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.067+0000 7fb768c00700 1 -- 172.21.15.27:0/3800930174 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7640aea70 con 0x5654f8769bc0 2024-05-10T11:19:03.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.067+0000 7fb768c00700 1 -- 172.21.15.27:0/3800930174 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb764090e40 con 0x5654f8769bc0 2024-05-10T11:19:03.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.068+0000 7fb78476cb80 1 -- 172.21.15.27:0/3800930174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f8769bc0 msgr2=0x5654f8769fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:03.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.068+0000 7fb78476cb80 1 --2- 172.21.15.27:0/3800930174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f8769bc0 0x5654f8769fa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb7640e0aa0 tx=0x7fb7640a9ed0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.068+0000 7fb78476cb80 1 -- 172.21.15.27:0/3800930174 shutdown_connections 2024-05-10T11:19:03.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.068+0000 7fb78476cb80 1 --2- 172.21.15.27:0/3800930174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f8769bc0 0x5654f8769fa0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.068+0000 7fb78476cb80 1 --2- 172.21.15.27:0/3800930174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f8764d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.068+0000 7fb78476cb80 1 --2- 172.21.15.27:0/3800930174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f876a4e0 0x5654f87642d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.068+0000 7fb78476cb80 1 -- 172.21.15.27:0/3800930174 >> 172.21.15.27:0/3800930174 conn(0x5654f86dc6d0 msgr2=0x5654f875eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:03.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.068+0000 7fb78476cb80 1 -- 172.21.15.27:0/3800930174 shutdown_connections 2024-05-10T11:19:03.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.068+0000 7fb78476cb80 1 -- 172.21.15.27:0/3800930174 wait complete. 2024-05-10T11:19:03.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.069+0000 7fb78476cb80 1 -- start start 2024-05-10T11:19:03.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.069+0000 7fb78476cb80 1 Processor -- start 2024-05-10T11:19:03.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.069+0000 7fb78476cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f87702b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.069+0000 7fb78476cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 0x5654f876d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.069+0000 7fb78476cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f876db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.069+0000 7fb78476cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5654f8770ce0 con 0x5654f87648f0 2024-05-10T11:19:03.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.069+0000 7fb78476cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5654f861cb20 con 0x5654f8769bc0 2024-05-10T11:19:03.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.069+0000 7fb78476cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5654f8770e60 con 0x5654f876a4e0 2024-05-10T11:19:03.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.070+0000 7fb78476cb80 10 client.0 _ll_register_callbacks cb 0x5654f865dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:19:03.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.070+0000 7fb76a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 0x5654f876d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.070+0000 7fb772a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f87702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.070+0000 7fb76a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 0x5654f876d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51488/0 (socket says 172.21.15.27:51488) 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.070+0000 7fb76a000700 1 -- 172.21.15.27:0/927742742 learned_addr learned my addr 172.21.15.27:0/927742742 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.070+0000 7fb773400700 1 --2- 172.21.15.27:0/927742742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f876db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.070+0000 7fb76a000700 1 -- 172.21.15.27:0/927742742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 msgr2=0x5654f876db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.070+0000 7fb76a000700 1 --2- 172.21.15.27:0/927742742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f876db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.070+0000 7fb76a000700 1 -- 172.21.15.27:0/927742742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 msgr2=0x5654f87702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.070+0000 7fb76a000700 1 --2- 172.21.15.27:0/927742742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f87702b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.070+0000 7fb76a000700 1 -- 172.21.15.27:0/927742742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654f867c420 con 0x5654f8769bc0 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.071+0000 7fb76a000700 1 --2- 172.21.15.27:0/927742742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 0x5654f876d650 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb75c012c30 tx=0x7fb75c0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55424c7f125077c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.071+0000 7fb761600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.071+0000 7fb761600700 1 -- 172.21.15.27:0/927742742 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c011e90 con 0x5654f8769bc0 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.071+0000 7fb78476cb80 1 -- 172.21.15.27:0/927742742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7640aea70 con 0x5654f8769bc0 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.071+0000 7fb761600700 1 -- 172.21.15.27:0/927742742 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fb75c01ea50 con 0x5654f8769bc0 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.071+0000 7fb761600700 1 -- 172.21.15.27:0/927742742 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb75c009660 con 0x5654f8769bc0 2024-05-10T11:19:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.071+0000 7fb78476cb80 10 client.26183 fetch_fsmap learned FSMap version 33 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.071+0000 7fb78476cb80 1 -- 172.21.15.27:0/927742742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5654f877b620 con 0x5654f8769bc0 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.072+0000 7fb761600700 1 -- 172.21.15.27:0/927742742 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7640aea70 con 0x5654f8769bc0 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.072+0000 7fb761600700 1 -- 172.21.15.27:0/927742742 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fb75c0273c0 con 0x5654f8769bc0 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.072+0000 7fb78476cb80 10 client.26183 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.072+0000 7fb78476cb80 10 client.26183 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.072+0000 7fb78476cb80 20 client.26183 populate_metadata read hostname 'smithi027' 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.072+0000 7fb78476cb80 1 --2- 172.21.15.27:0/927742742 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5654f877dbb0 0x5654f877ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.072+0000 7fb78476cb80 4 client.26183 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.072+0000 7fb78476cb80 1 -- 172.21.15.27:0/927742742 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654f8731140 con 0x5654f877dbb0 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.073+0000 7fb772a00700 1 --2- 172.21.15.27:0/927742742 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5654f877dbb0 0x5654f877ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.073+0000 7fb772a00700 1 --2- 172.21.15.27:0/927742742 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5654f877dbb0 0x5654f877ffe0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=14bbe94426af9093 server_cookie=cfca568f1acb97f7 in_seq=0 out_seq=0 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.073+0000 7fb761600700 10 client.26183 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.074+0000 7fb761600700 1 -- 172.21.15.27:0/927742742 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5654f8731140 con 0x5654f877dbb0 2024-05-10T11:19:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.074+0000 7fb761600700 10 client.26183 handle_command_reply: tid=0 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.074+0000 7fb78476cb80 1 client.26183 shutdown 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.074+0000 7fb78476cb80 1 -- 172.21.15.27:0/927742742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 msgr2=0x5654f876d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.074+0000 7fb78476cb80 1 --2- 172.21.15.27:0/927742742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 0x5654f876d650 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb75c012c30 tx=0x7fb75c0047c0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.074+0000 7fb78476cb80 1 -- 172.21.15.27:0/927742742 shutdown_connections 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.074+0000 7fb78476cb80 1 --2- 172.21.15.27:0/927742742 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5654f877dbb0 0x5654f877ffe0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.074+0000 7fb78476cb80 1 --2- 172.21.15.27:0/927742742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f876db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.074+0000 7fb78476cb80 1 --2- 172.21.15.27:0/927742742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f87702b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.074+0000 7fb78476cb80 1 --2- 172.21.15.27:0/927742742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 0x5654f876d650 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.074+0000 7fb78476cb80 1 -- 172.21.15.27:0/927742742 >> 172.21.15.27:0/927742742 conn(0x5654f86dc6d0 msgr2=0x5654f875e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.074+0000 7fb761600700 0 client.26183 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.075+0000 7fb78476cb80 1 -- 172.21.15.27:0/927742742 shutdown_connections 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.075+0000 7fb78476cb80 1 -- 172.21.15.27:0/927742742 wait complete. 2024-05-10T11:19:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.075+0000 7fb78476cb80 20 client.26183 trim_cache size 0 max 16384 2024-05-10T11:19:03.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.093+0000 7fb78476cb80 1 Processor -- start 2024-05-10T11:19:03.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.093+0000 7fb78476cb80 1 -- start start 2024-05-10T11:19:03.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.094+0000 7fb78476cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f87642d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.094+0000 7fb78476cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 0x5654f8767d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.094+0000 7fb78476cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f876dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.094+0000 7fb78476cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741b16e0 con 0x5654f87648f0 2024-05-10T11:19:03.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.094+0000 7fb78476cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741b3220 con 0x5654f8769bc0 2024-05-10T11:19:03.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.094+0000 7fb78476cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741b1db0 con 0x5654f876a4e0 2024-05-10T11:19:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.094+0000 7fb773400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f876dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.094+0000 7fb76a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 0x5654f8767d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.094+0000 7fb772a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f87642d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.094+0000 7fb773400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f876dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49594/0 (socket says 172.21.15.27:49594) 2024-05-10T11:19:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.094+0000 7fb773400700 1 -- 172.21.15.27:0/484879345 learned_addr learned my addr 172.21.15.27:0/484879345 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:03.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.095+0000 7fb773400700 1 -- 172.21.15.27:0/484879345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 msgr2=0x5654f8767d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:03.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.095+0000 7fb773400700 1 --2- 172.21.15.27:0/484879345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 0x5654f8767d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.095+0000 7fb773400700 1 -- 172.21.15.27:0/484879345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 msgr2=0x5654f87642d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:03.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.095+0000 7fb773400700 1 --2- 172.21.15.27:0/484879345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f87642d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.095+0000 7fb773400700 1 -- 172.21.15.27:0/484879345 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654f8760b50 con 0x5654f876a4e0 2024-05-10T11:19:03.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.095+0000 7fb76a000700 1 --2- 172.21.15.27:0/484879345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 0x5654f8767d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:19:03.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.095+0000 7fb772a00700 1 --2- 172.21.15.27:0/484879345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f87642d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:19:03.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.095+0000 7fb773400700 1 --2- 172.21.15.27:0/484879345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f876dd20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009f80 tx=0x7fb76c007500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=199264bfcddfbb62 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:03.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.096+0000 7fb763e00700 1 -- 172.21.15.27:0/484879345 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c020470 con 0x5654f876a4e0 2024-05-10T11:19:03.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.096+0000 7fb763e00700 1 -- 172.21.15.27:0/484879345 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb76c01b070 con 0x5654f876a4e0 2024-05-10T11:19:03.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.096+0000 7fb763e00700 1 -- 172.21.15.27:0/484879345 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0179f0 con 0x5654f876a4e0 2024-05-10T11:19:03.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.096+0000 7fb78476cb80 1 -- 172.21.15.27:0/484879345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 msgr2=0x5654f876dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:03.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.096+0000 7fb78476cb80 1 --2- 172.21.15.27:0/484879345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f876dd20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009f80 tx=0x7fb76c007500 comp rx=0 tx=0).stop 2024-05-10T11:19:03.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.096+0000 7fb78476cb80 1 -- 172.21.15.27:0/484879345 shutdown_connections 2024-05-10T11:19:03.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.096+0000 7fb78476cb80 1 --2- 172.21.15.27:0/484879345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f876dd20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.096+0000 7fb78476cb80 1 --2- 172.21.15.27:0/484879345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f87648f0 0x5654f87642d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.096+0000 7fb78476cb80 1 --2- 172.21.15.27:0/484879345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f8769bc0 0x5654f8767d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.097+0000 7fb78476cb80 1 -- 172.21.15.27:0/484879345 >> 172.21.15.27:0/484879345 conn(0x5654f86dc6d0 msgr2=0x5654f876d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.097+0000 7fb78476cb80 1 -- 172.21.15.27:0/484879345 shutdown_connections 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.097+0000 7fb78476cb80 1 -- 172.21.15.27:0/484879345 wait complete. 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.097+0000 7fb78476cb80 1 -- start start 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.097+0000 7fb78476cb80 1 Processor -- start 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.098+0000 7fb78476cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f87648f0 0x5654f87764b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.098+0000 7fb78476cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f8769bc0 0x5654f8784ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.098+0000 7fb78476cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f8785220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.098+0000 7fb78476cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741b2b50 con 0x5654f8769bc0 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.098+0000 7fb78476cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741b2480 con 0x5654f87648f0 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.098+0000 7fb78476cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741af680 con 0x5654f876a4e0 2024-05-10T11:19:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.098+0000 7fb78476cb80 10 client.0 _ll_register_callbacks cb 0x5654f865dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.098+0000 7fb772a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f87648f0 0x5654f87764b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.098+0000 7fb772a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f87648f0 0x5654f87764b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51508/0 (socket says 172.21.15.27:51508) 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.098+0000 7fb772a00700 1 -- 172.21.15.27:0/262773715 learned_addr learned my addr 172.21.15.27:0/262773715 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.098+0000 7fb76a000700 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f8769bc0 0x5654f8784ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.098+0000 7fb773400700 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f8785220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.099+0000 7fb772a00700 1 -- 172.21.15.27:0/262773715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 msgr2=0x5654f8785220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.099+0000 7fb772a00700 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f8785220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.099+0000 7fb772a00700 1 -- 172.21.15.27:0/262773715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f8769bc0 msgr2=0x5654f8784ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.099+0000 7fb772a00700 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f8769bc0 0x5654f8784ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.099+0000 7fb772a00700 1 -- 172.21.15.27:0/262773715 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654f8785890 con 0x5654f87648f0 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.099+0000 7fb773400700 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f8785220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.099+0000 7fb772a00700 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f87648f0 0x5654f87764b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb7640e0b00 tx=0x7fb7640cd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1269c4fb725ff7bd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.099+0000 7fb761600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:19:03.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.099+0000 7fb761600700 1 -- 172.21.15.27:0/262773715 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7640a9e30 con 0x5654f87648f0 2024-05-10T11:19:03.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.099+0000 7fb761600700 1 -- 172.21.15.27:0/262773715 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fb764022940 con 0x5654f87648f0 2024-05-10T11:19:03.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.099+0000 7fb78476cb80 1 -- 172.21.15.27:0/262773715 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb76c01b070 con 0x5654f87648f0 2024-05-10T11:19:03.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.100+0000 7fb761600700 1 -- 172.21.15.27:0/262773715 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb7640aea70 con 0x5654f87648f0 2024-05-10T11:19:03.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.100+0000 7fb78476cb80 10 client.26189 fetch_fsmap learned FSMap version 33 2024-05-10T11:19:03.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.100+0000 7fb78476cb80 1 -- 172.21.15.27:0/262773715 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5654f87e8330 con 0x5654f87648f0 2024-05-10T11:19:03.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.100+0000 7fb761600700 1 -- 172.21.15.27:0/262773715 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb76c01b070 con 0x5654f87648f0 2024-05-10T11:19:03.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.100+0000 7fb761600700 1 -- 172.21.15.27:0/262773715 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fb7640aa920 con 0x5654f87648f0 2024-05-10T11:19:03.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.100+0000 7fb78476cb80 10 client.26189 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:19:03.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.100+0000 7fb78476cb80 10 client.26189 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:19:03.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.100+0000 7fb78476cb80 20 client.26189 populate_metadata read hostname 'smithi027' 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.100+0000 7fb78476cb80 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5654f87e87f0 0x5654f87eabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.100+0000 7fb78476cb80 4 client.26189 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.100+0000 7fb78476cb80 1 -- 172.21.15.27:0/262773715 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5654f86c4400 con 0x5654f87e87f0 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.101+0000 7fb76a000700 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5654f87e87f0 0x5654f87eabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.101+0000 7fb76a000700 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5654f87e87f0 0x5654f87eabe0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eac33b7b43e2e57f server_cookie=d57cc03d8c7275c0 in_seq=0 out_seq=0 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.101+0000 7fb761600700 10 client.26189 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.101+0000 7fb761600700 1 -- 172.21.15.27:0/262773715 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5654f86c4400 con 0x5654f87e87f0 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.101+0000 7fb761600700 10 client.26189 handle_command_reply: tid=0 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.101+0000 7fb78476cb80 1 client.26189 shutdown 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.102+0000 7fb78476cb80 1 -- 172.21.15.27:0/262773715 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f87648f0 msgr2=0x5654f87764b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.102+0000 7fb78476cb80 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f87648f0 0x5654f87764b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb7640e0b00 tx=0x7fb7640cd040 comp rx=0 tx=0).stop 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.102+0000 7fb78476cb80 1 -- 172.21.15.27:0/262773715 shutdown_connections 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.102+0000 7fb78476cb80 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5654f87e87f0 0x5654f87eabe0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.102+0000 7fb78476cb80 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654f876a4e0 0x5654f8785220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.102+0000 7fb78476cb80 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654f87648f0 0x5654f87764b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.102+0000 7fb78476cb80 1 --2- 172.21.15.27:0/262773715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5654f8769bc0 0x5654f8784ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.102+0000 7fb761600700 0 client.26189 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:19:03.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.102+0000 7fb78476cb80 1 -- 172.21.15.27:0/262773715 >> 172.21.15.27:0/262773715 conn(0x5654f86dc6d0 msgr2=0x5654f87687b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:03.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.102+0000 7fb78476cb80 1 -- 172.21.15.27:0/262773715 shutdown_connections 2024-05-10T11:19:03.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.102+0000 7fb78476cb80 1 -- 172.21.15.27:0/262773715 wait complete. 2024-05-10T11:19:03.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.102+0000 7fb78476cb80 20 client.26189 trim_cache size 0 max 16384 2024-05-10T11:19:03.106 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:19:03.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.103+0000 7fb779e00700 1 -- 172.21.15.27:0/1428633648 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb7540b2a70 msgr2=0x7fb7540b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:03.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.103+0000 7fb779e00700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb7540b2a70 0x7fb7540b4f30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb774110770 tx=0x7fb75c009260 comp rx=0 tx=0).stop 2024-05-10T11:19:03.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.103+0000 7fb779e00700 1 -- 172.21.15.27:0/1428633648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb77406b180 msgr2=0x7fb77419bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:03.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.103+0000 7fb779e00700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb77406b180 0x7fb77419bdd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb764000c00 tx=0x7fb764009580 comp rx=0 tx=0).stop 2024-05-10T11:19:03.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.103+0000 7fb779e00700 1 -- 172.21.15.27:0/1428633648 shutdown_connections 2024-05-10T11:19:03.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.103+0000 7fb779e00700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb7540b2a70 0x7fb7540b4f30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.103+0000 7fb779e00700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb77406b180 0x7fb77419bdd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.103+0000 7fb779e00700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb774109510 0x7fb7741a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.103+0000 7fb779e00700 1 --2- 172.21.15.27:0/1428633648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb77406bb40 0x7fb77419c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:03.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.103+0000 7fb779e00700 1 -- 172.21.15.27:0/1428633648 >> 172.21.15.27:0/1428633648 conn(0x7fb774079350 msgr2=0x7fb774100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:03.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.103+0000 7fb779e00700 1 -- 172.21.15.27:0/1428633648 shutdown_connections 2024-05-10T11:19:03.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:03.103+0000 7fb779e00700 1 -- 172.21.15.27:0/1428633648 wait complete. 2024-05-10T11:19:03.678 INFO:tasks.workunit.client.0.smithi027.stdout: 1 320932 7.66 MB/sec execute 108 sec latency 31.329 ms 2024-05-10T11:19:03.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:19:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:19:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:04 smithi027 ceph-mon[30054]: pgmap v178: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-05-10T11:19:04.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:04 smithi076 ceph-mon[34152]: pgmap v178: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-05-10T11:19:04.678 INFO:tasks.workunit.client.0.smithi027.stdout: 1 323020 7.71 MB/sec execute 109 sec latency 18.204 ms 2024-05-10T11:19:04.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:04 smithi189 ceph-mon[34842]: pgmap v178: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-05-10T11:19:05.678 INFO:tasks.workunit.client.0.smithi027.stdout: 1 323834 7.66 MB/sec execute 110 sec latency 33.155 ms 2024-05-10T11:19:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:06 smithi027 ceph-mon[30054]: pgmap v179: 129 pgs: 129 active+clean; 188 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 265 op/s 2024-05-10T11:19:06.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:06 smithi076 ceph-mon[34152]: pgmap v179: 129 pgs: 129 active+clean; 188 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 265 op/s 2024-05-10T11:19:06.679 INFO:tasks.workunit.client.0.smithi027.stdout: 1 326132 7.71 MB/sec execute 111 sec latency 48.710 ms 2024-05-10T11:19:06.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:06 smithi189 ceph-mon[34842]: pgmap v179: 129 pgs: 129 active+clean; 188 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 265 op/s 2024-05-10T11:19:06.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.829+0000 7fdd44c00700 1 -- 172.21.15.27:0/3941040352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd40102380 msgr2=0x7fdd40102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:06.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.829+0000 7fdd44c00700 1 --2- 172.21.15.27:0/3941040352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd40102380 0x7fdd40102760 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009990 tx=0x7fdd28009ca0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.829+0000 7fdd44c00700 1 -- 172.21.15.27:0/3941040352 shutdown_connections 2024-05-10T11:19:06.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.829+0000 7fdd44c00700 1 --2- 172.21.15.27:0/3941040352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd40102380 0x7fdd40102760 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.829+0000 7fdd44c00700 1 --2- 172.21.15.27:0/3941040352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd40107790 0x7fdd40107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.829+0000 7fdd44c00700 1 --2- 172.21.15.27:0/3941040352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd40102ca0 0x7fdd40107170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.829+0000 7fdd44c00700 1 -- 172.21.15.27:0/3941040352 >> 172.21.15.27:0/3941040352 conn(0x7fdd400fddc0 msgr2=0x7fdd401001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:06.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.829+0000 7fdd44c00700 1 -- 172.21.15.27:0/3941040352 shutdown_connections 2024-05-10T11:19:06.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.829+0000 7fdd44c00700 1 -- 172.21.15.27:0/3941040352 wait complete. 2024-05-10T11:19:06.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.831+0000 7fdd44c00700 1 Processor -- start 2024-05-10T11:19:06.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.831+0000 7fdd44c00700 1 -- start start 2024-05-10T11:19:06.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.831+0000 7fdd44c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd40102380 0x7fdd401a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.831+0000 7fdd44c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd40102ca0 0x7fdd401a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.831+0000 7fdd44c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd40107790 0x7fdd401a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd44c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40119c20 con 0x7fdd40102380 2024-05-10T11:19:06.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd44c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40119aa0 con 0x7fdd40102ca0 2024-05-10T11:19:06.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd44c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40119da0 con 0x7fdd40107790 2024-05-10T11:19:06.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd3ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd40102ca0 0x7fdd401a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd3f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd40102380 0x7fdd401a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd3ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd40102ca0 0x7fdd401a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58508/0 (socket says 172.21.15.27:58508) 2024-05-10T11:19:06.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd3ea00700 1 -- 172.21.15.27:0/1957956589 learned_addr learned my addr 172.21.15.27:0/1957956589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:06.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd3fe00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd40107790 0x7fdd401a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd3ea00700 1 -- 172.21.15.27:0/1957956589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd40107790 msgr2=0x7fdd401a4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:06.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd3ea00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd40107790 0x7fdd401a4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd3ea00700 1 -- 172.21.15.27:0/1957956589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd40102380 msgr2=0x7fdd401a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:06.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd3ea00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd40102380 0x7fdd401a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.832+0000 7fdd3ea00700 1 -- 172.21.15.27:0/1957956589 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd401a5160 con 0x7fdd40102ca0 2024-05-10T11:19:06.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.834+0000 7fdd3ea00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd40102ca0 0x7fdd401a0690 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800d720 tx=0x7fdd3800da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:06.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.834+0000 7fdd37e00700 1 -- 172.21.15.27:0/1957956589 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38017070 con 0x7fdd40102ca0 2024-05-10T11:19:06.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.835+0000 7fdd37e00700 1 -- 172.21.15.27:0/1957956589 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd38005020 con 0x7fdd40102ca0 2024-05-10T11:19:06.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.835+0000 7fdd37e00700 1 -- 172.21.15.27:0/1957956589 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3800fa10 con 0x7fdd40102ca0 2024-05-10T11:19:06.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.835+0000 7fdd44c00700 1 -- 172.21.15.27:0/1957956589 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd401a5450 con 0x7fdd40102ca0 2024-05-10T11:19:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.835+0000 7fdd44c00700 1 -- 172.21.15.27:0/1957956589 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd401accd0 con 0x7fdd40102ca0 2024-05-10T11:19:06.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.841+0000 7fdd37e00700 1 -- 172.21.15.27:0/1957956589 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fdd3800fbd0 con 0x7fdd40102ca0 2024-05-10T11:19:06.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.843+0000 7fdd50bccb80 1 Processor -- start 2024-05-10T11:19:06.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.843+0000 7fdd50bccb80 1 -- start start 2024-05-10T11:19:06.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.844+0000 7fdd50bccb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec5a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.844+0000 7fdd50bccb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec5a6c0 0x56130ec54430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.844+0000 7fdd50bccb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec54a50 0x56130ec54ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.844+0000 7fdd50bccb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56130eba6a20 con 0x56130ec5a6c0 2024-05-10T11:19:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.845+0000 7fdd50bccb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56130eb44760 con 0x56130ec54a50 2024-05-10T11:19:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.845+0000 7fdd3fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec5a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.845+0000 7fdd3ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec54a50 0x56130ec54ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.845+0000 7fdd50bccb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56130eba29e0 con 0x56130ec59da0 2024-05-10T11:19:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.845+0000 7fdd3ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec54a50 0x56130ec54ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58514/0 (socket says 172.21.15.27:58514) 2024-05-10T11:19:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.845+0000 7fdd3ea00700 1 -- 172.21.15.27:0/1652878194 learned_addr learned my addr 172.21.15.27:0/1652878194 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.845+0000 7fdd3fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec5a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42602/0 (socket says 172.21.15.27:42602) 2024-05-10T11:19:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.846+0000 7fdd3fe00700 1 -- 172.21.15.27:0/1652878194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec54a50 msgr2=0x56130ec54ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.846+0000 7fdd3fe00700 1 --2- 172.21.15.27:0/1652878194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec54a50 0x56130ec54ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.846+0000 7fdd3fe00700 1 -- 172.21.15.27:0/1652878194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec5a6c0 msgr2=0x56130ec54430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:19:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.846+0000 7fdd37e00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fdd240b2a70 0x7fdd240b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.846+0000 7fdd37e00700 1 -- 172.21.15.27:0/1957956589 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fdd380e23c0 con 0x7fdd40102ca0 2024-05-10T11:19:06.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.846+0000 7fdd3fe00700 1 --2- 172.21.15.27:0/1652878194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec5a6c0 0x56130ec54430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.846+0000 7fdd3fe00700 1 -- 172.21.15.27:0/1652878194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56130eaf38d0 con 0x56130ec59da0 2024-05-10T11:19:06.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.847+0000 7fdd3fe00700 1 --2- 172.21.15.27:0/1652878194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec5a180 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009c90 tx=0x7fdd30007500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3182a35aa95b3dbe server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:06.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.847+0000 7fdd3fe00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fdd240b2a70 0x7fdd240b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.847+0000 7fdd34c00700 1 -- 172.21.15.27:0/1652878194 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30014610 con 0x56130ec59da0 2024-05-10T11:19:06.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.847+0000 7fdd34c00700 1 -- 172.21.15.27:0/1652878194 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd30007e30 con 0x56130ec59da0 2024-05-10T11:19:06.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.847+0000 7fdd34c00700 1 -- 172.21.15.27:0/1652878194 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30013d20 con 0x56130ec59da0 2024-05-10T11:19:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.848+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1652878194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 msgr2=0x56130ec5a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.848+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1652878194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec5a180 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009c90 tx=0x7fdd30007500 comp rx=0 tx=0).stop 2024-05-10T11:19:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.849+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1652878194 shutdown_connections 2024-05-10T11:19:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.849+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1652878194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec5a180 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.849+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1652878194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec54a50 0x56130ec54ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.849+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1652878194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec5a6c0 0x56130ec54430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.849+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1652878194 >> 172.21.15.27:0/1652878194 conn(0x56130ebcc6d0 msgr2=0x56130ec4eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.849+0000 7fdd3fe00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fdd240b2a70 0x7fdd240b4f30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdd30014fd0 tx=0x7fdd3001e340 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.849+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1652878194 shutdown_connections 2024-05-10T11:19:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.850+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1652878194 wait complete. 2024-05-10T11:19:06.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.851+0000 7fdd50bccb80 1 -- start start 2024-05-10T11:19:06.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.851+0000 7fdd50bccb80 1 Processor -- start 2024-05-10T11:19:06.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.852+0000 7fdd50bccb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec54a50 0x56130ec60400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.853+0000 7fdd50bccb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec59da0 0x56130ec5d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.853+0000 7fdd50bccb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 0x56130ec5dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.853+0000 7fdd3f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec54a50 0x56130ec60400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.853+0000 7fdd3f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec54a50 0x56130ec60400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42604/0 (socket says 172.21.15.27:42604) 2024-05-10T11:19:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.853+0000 7fdd3f400700 1 -- 172.21.15.27:0/1750851993 learned_addr learned my addr 172.21.15.27:0/1750851993 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.853+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1750851993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56130ec61150 con 0x56130ec59da0 2024-05-10T11:19:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.853+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1750851993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56130eb0cb20 con 0x56130ec5a6c0 2024-05-10T11:19:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.853+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1750851993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56130ec612d0 con 0x56130ec54a50 2024-05-10T11:19:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.853+0000 7fdd50bccb80 10 client.0 _ll_register_callbacks cb 0x56130eb4dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:19:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.853+0000 7fdd3ea00700 1 --2- 172.21.15.27:0/1750851993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec59da0 0x56130ec5d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.854+0000 7fdd3f400700 1 -- 172.21.15.27:0/1750851993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 msgr2=0x56130ec5dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:06.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.854+0000 7fdd3f400700 1 --2- 172.21.15.27:0/1750851993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 0x56130ec5dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.854+0000 7fdd3f400700 1 -- 172.21.15.27:0/1750851993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec59da0 msgr2=0x56130ec5d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:06.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.854+0000 7fdd3f400700 1 --2- 172.21.15.27:0/1750851993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec59da0 0x56130ec5d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.854+0000 7fdd3f400700 1 -- 172.21.15.27:0/1750851993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56130eb6c420 con 0x56130ec54a50 2024-05-10T11:19:06.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.854+0000 7fdd3f400700 1 --2- 172.21.15.27:0/1750851993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec54a50 0x56130ec60400 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009990 tx=0x7fdd28009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec965aed5404268 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:06.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.854+0000 7fdd2d600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:19:06.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.855+0000 7fdd2d600700 1 -- 172.21.15.27:0/1750851993 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28003a60 con 0x56130ec54a50 2024-05-10T11:19:06.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.855+0000 7fdd2d600700 1 -- 172.21.15.27:0/1750851993 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fdd28003c00 con 0x56130ec54a50 2024-05-10T11:19:06.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.855+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1750851993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd30007e30 con 0x56130ec54a50 2024-05-10T11:19:06.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.856+0000 7fdd2d600700 1 -- 172.21.15.27:0/1750851993 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdd28003680 con 0x56130ec54a50 2024-05-10T11:19:06.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.856+0000 7fdd50bccb80 10 client.26191 fetch_fsmap learned FSMap version 33 2024-05-10T11:19:06.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.856+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1750851993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56130ec6b9a0 con 0x56130ec54a50 2024-05-10T11:19:06.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.856+0000 7fdd2d600700 1 -- 172.21.15.27:0/1750851993 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd30007e30 con 0x56130ec54a50 2024-05-10T11:19:06.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.857+0000 7fdd2d600700 1 -- 172.21.15.27:0/1750851993 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fdd2800f690 con 0x56130ec54a50 2024-05-10T11:19:06.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.857+0000 7fdd50bccb80 10 client.26191 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:19:06.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.857+0000 7fdd50bccb80 10 client.26191 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:19:06.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.857+0000 7fdd50bccb80 20 client.26191 populate_metadata read hostname 'smithi027' 2024-05-10T11:19:06.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.857+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1750851993 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56130ec6dfe0 0x56130ec70410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.857+0000 7fdd50bccb80 4 client.26191 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:19:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.857+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1750851993 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56130ec21140 con 0x56130ec6dfe0 2024-05-10T11:19:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.857+0000 7fdd3ea00700 1 --2- 172.21.15.27:0/1750851993 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56130ec6dfe0 0x56130ec70410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.858+0000 7fdd3ea00700 1 --2- 172.21.15.27:0/1750851993 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56130ec6dfe0 0x56130ec70410 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef505eb04db37822 server_cookie=34bbaa9dc68ee510 in_seq=0 out_seq=0 2024-05-10T11:19:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.858+0000 7fdd2d600700 10 client.26191 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:19:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.859+0000 7fdd2d600700 1 -- 172.21.15.27:0/1750851993 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56130ec21140 con 0x56130ec6dfe0 2024-05-10T11:19:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.859+0000 7fdd2d600700 10 client.26191 handle_command_reply: tid=0 2024-05-10T11:19:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.859+0000 7fdd50bccb80 1 client.26191 shutdown 2024-05-10T11:19:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.860+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1750851993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec54a50 msgr2=0x56130ec60400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.860+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1750851993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec54a50 0x56130ec60400 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009990 tx=0x7fdd28009f90 comp rx=0 tx=0).stop 2024-05-10T11:19:06.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.860+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1750851993 shutdown_connections 2024-05-10T11:19:06.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.860+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1750851993 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56130ec6dfe0 0x56130ec70410 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.860+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1750851993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec54a50 0x56130ec60400 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.860+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1750851993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 0x56130ec5dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.860+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1750851993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec59da0 0x56130ec5d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.860+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1750851993 >> 172.21.15.27:0/1750851993 conn(0x56130ebcc6d0 msgr2=0x56130ec49750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:06.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.861+0000 7fdd2d600700 0 client.26191 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:19:06.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.861+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1750851993 shutdown_connections 2024-05-10T11:19:06.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.861+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1750851993 wait complete. 2024-05-10T11:19:06.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.861+0000 7fdd50bccb80 20 client.26191 trim_cache size 0 max 16384 2024-05-10T11:19:06.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.893+0000 7fdd50bccb80 1 Processor -- start 2024-05-10T11:19:06.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.894+0000 7fdd50bccb80 1 -- start start 2024-05-10T11:19:06.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.894+0000 7fdd50bccb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec54a50 0x56130ec54430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.894+0000 7fdd50bccb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec6a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.894+0000 7fdd50bccb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 0x56130ec49860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.894+0000 7fdd50bccb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdd401b5af0 con 0x56130ec54a50 2024-05-10T11:19:06.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.894+0000 7fdd50bccb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdd401b7630 con 0x56130ec5a6c0 2024-05-10T11:19:06.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.895+0000 7fdd50bccb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdd401b61c0 con 0x56130ec59da0 2024-05-10T11:19:06.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.895+0000 7fdd3ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec6a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.895+0000 7fdd3ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec6a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42608/0 (socket says 172.21.15.27:42608) 2024-05-10T11:19:06.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.895+0000 7fdd3ea00700 1 -- 172.21.15.27:0/3929678036 learned_addr learned my addr 172.21.15.27:0/3929678036 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:06.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.895+0000 7fdd3fe00700 1 --2- 172.21.15.27:0/3929678036 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 0x56130ec49860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.897+0000 7fdd3f400700 1 --2- 172.21.15.27:0/3929678036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec54a50 0x56130ec54430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.897+0000 7fdd3ea00700 1 -- 172.21.15.27:0/3929678036 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 msgr2=0x56130ec49860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:06.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.897+0000 7fdd3ea00700 1 --2- 172.21.15.27:0/3929678036 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 0x56130ec49860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.897+0000 7fdd3ea00700 1 -- 172.21.15.27:0/3929678036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec54a50 msgr2=0x56130ec54430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:06.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.898+0000 7fdd3ea00700 1 --2- 172.21.15.27:0/3929678036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec54a50 0x56130ec54430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.898+0000 7fdd3ea00700 1 -- 172.21.15.27:0/3929678036 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56130ec5bd10 con 0x56130ec59da0 2024-05-10T11:19:06.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.898+0000 7fdd3f400700 1 --2- 172.21.15.27:0/3929678036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec54a50 0x56130ec54430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:19:06.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.898+0000 7fdd3ea00700 1 --2- 172.21.15.27:0/3929678036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec6a1e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdd380a7250 tx=0x7fdd38012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc1b5fafe1e48d9c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:06.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.899+0000 7fdd2fe00700 1 -- 172.21.15.27:0/3929678036 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38020040 con 0x56130ec59da0 2024-05-10T11:19:06.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.899+0000 7fdd2fe00700 1 -- 172.21.15.27:0/3929678036 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd380a9af0 con 0x56130ec59da0 2024-05-10T11:19:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.900+0000 7fdd50bccb80 1 -- 172.21.15.27:0/3929678036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 msgr2=0x56130ec6a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.900+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/3929678036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec6a1e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdd380a7250 tx=0x7fdd38012000 comp rx=0 tx=0).stop 2024-05-10T11:19:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.901+0000 7fdd50bccb80 1 -- 172.21.15.27:0/3929678036 shutdown_connections 2024-05-10T11:19:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.901+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/3929678036 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 0x56130ec49860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.901+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/3929678036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec54a50 0x56130ec54430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.901+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/3929678036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec6a1e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.901+0000 7fdd50bccb80 1 -- 172.21.15.27:0/3929678036 >> 172.21.15.27:0/3929678036 conn(0x56130ebcc6d0 msgr2=0x56130ec5aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:06.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.901+0000 7fdd50bccb80 1 -- 172.21.15.27:0/3929678036 shutdown_connections 2024-05-10T11:19:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.901+0000 7fdd50bccb80 1 -- 172.21.15.27:0/3929678036 wait complete. 2024-05-10T11:19:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.902+0000 7fdd50bccb80 1 -- start start 2024-05-10T11:19:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.902+0000 7fdd50bccb80 1 Processor -- start 2024-05-10T11:19:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.904+0000 7fdd50bccb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec54a50 0x56130ec66560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.905+0000 7fdd50bccb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec66aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.905+0000 7fdd50bccb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 0x56130ecd3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.905+0000 7fdd50bccb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdd401b6f60 con 0x56130ec54a50 2024-05-10T11:19:06.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.905+0000 7fdd50bccb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdd401b6890 con 0x56130ec5a6c0 2024-05-10T11:19:06.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.905+0000 7fdd50bccb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdd401b3a90 con 0x56130ec59da0 2024-05-10T11:19:06.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.905+0000 7fdd50bccb80 10 client.0 _ll_register_callbacks cb 0x56130eb4dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:19:06.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.905+0000 7fdd3f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec54a50 0x56130ec66560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.905+0000 7fdd3ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec66aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.905+0000 7fdd3fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 0x56130ecd3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.906+0000 7fdd3fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 0x56130ecd3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58560/0 (socket says 172.21.15.27:58560) 2024-05-10T11:19:06.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.906+0000 7fdd3fe00700 1 -- 172.21.15.27:0/1008636146 learned_addr learned my addr 172.21.15.27:0/1008636146 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:06.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.905+0000 7fdd3ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec66aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42624/0 (socket says 172.21.15.27:42624) 2024-05-10T11:19:06.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.906+0000 7fdd3f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec54a50 0x56130ec66560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37636/0 (socket says 172.21.15.27:37636) 2024-05-10T11:19:06.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.906+0000 7fdd3ea00700 1 -- 172.21.15.27:0/1008636146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 msgr2=0x56130ecd3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:06.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.906+0000 7fdd3ea00700 1 --2- 172.21.15.27:0/1008636146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 0x56130ecd3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.906+0000 7fdd3ea00700 1 -- 172.21.15.27:0/1008636146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec54a50 msgr2=0x56130ec66560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.906+0000 7fdd3ea00700 1 --2- 172.21.15.27:0/1008636146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec54a50 0x56130ec66560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.906+0000 7fdd3ea00700 1 -- 172.21.15.27:0/1008636146 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56130ecd4270 con 0x56130ec59da0 2024-05-10T11:19:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.907+0000 7fdd2d600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:19:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.907+0000 7fdd3ea00700 1 --2- 172.21.15.27:0/1008636146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec66aa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x56130ec47550 tx=0x7fdd380b2220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9457d1e851cf0a6a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.908+0000 7fdd2d600700 1 -- 172.21.15.27:0/1008636146 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd380a9fe0 con 0x56130ec59da0 2024-05-10T11:19:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.908+0000 7fdd2d600700 1 -- 172.21.15.27:0/1008636146 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fdd380aa240 con 0x56130ec59da0 2024-05-10T11:19:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.908+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1008636146 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd380a9af0 con 0x56130ec59da0 2024-05-10T11:19:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.909+0000 7fdd2d600700 1 -- 172.21.15.27:0/1008636146 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdd380aa6e0 con 0x56130ec59da0 2024-05-10T11:19:06.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.910+0000 7fdd50bccb80 10 client.26197 fetch_fsmap learned FSMap version 33 2024-05-10T11:19:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.910+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1008636146 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56130ec75d40 con 0x56130ec59da0 2024-05-10T11:19:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.911+0000 7fdd2d600700 1 -- 172.21.15.27:0/1008636146 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd380a9af0 con 0x56130ec59da0 2024-05-10T11:19:06.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.911+0000 7fdd2d600700 1 -- 172.21.15.27:0/1008636146 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fdd380a5950 con 0x56130ec59da0 2024-05-10T11:19:06.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.911+0000 7fdd50bccb80 10 client.26197 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:19:06.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.911+0000 7fdd50bccb80 10 client.26197 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:19:06.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.911+0000 7fdd50bccb80 20 client.26197 populate_metadata read hostname 'smithi027' 2024-05-10T11:19:06.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.911+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1008636146 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56130ece9a90 0x56130ecebeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:06.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.911+0000 7fdd50bccb80 4 client.26197 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:19:06.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.911+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1008636146 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56130ebb4400 con 0x56130ece9a90 2024-05-10T11:19:06.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.913+0000 7fdd3f400700 1 --2- 172.21.15.27:0/1008636146 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56130ece9a90 0x56130ecebeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:06.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.913+0000 7fdd3f400700 1 --2- 172.21.15.27:0/1008636146 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56130ece9a90 0x56130ecebeb0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6548bd0dffc6802f server_cookie=937e041c1e8a28a in_seq=0 out_seq=0 2024-05-10T11:19:06.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.914+0000 7fdd2d600700 10 client.26197 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:19:06.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.914+0000 7fdd2d600700 1 -- 172.21.15.27:0/1008636146 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56130ebb4400 con 0x56130ece9a90 2024-05-10T11:19:06.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.914+0000 7fdd2d600700 10 client.26197 handle_command_reply: tid=0 2024-05-10T11:19:06.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.915+0000 7fdd50bccb80 1 client.26197 shutdown 2024-05-10T11:19:06.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.916+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1008636146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 msgr2=0x56130ec66aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:06.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.916+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1008636146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec66aa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x56130ec47550 tx=0x7fdd380b2220 comp rx=0 tx=0).stop 2024-05-10T11:19:06.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.916+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1008636146 shutdown_connections 2024-05-10T11:19:06.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.916+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1008636146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56130ec5a6c0 0x56130ecd3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.916+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1008636146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56130ec54a50 0x56130ec66560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.916+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1008636146 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56130ece9a90 0x56130ecebeb0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.917+0000 7fdd50bccb80 1 --2- 172.21.15.27:0/1008636146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56130ec59da0 0x56130ec66aa0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.917+0000 7fdd2d600700 0 client.26197 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:19:06.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.917+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1008636146 >> 172.21.15.27:0/1008636146 conn(0x56130ebcc6d0 msgr2=0x56130ec5ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:06.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.917+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1008636146 shutdown_connections 2024-05-10T11:19:06.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.918+0000 7fdd50bccb80 1 -- 172.21.15.27:0/1008636146 wait complete. 2024-05-10T11:19:06.919 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:19:06.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.918+0000 7fdd50bccb80 20 client.26197 trim_cache size 0 max 16384 2024-05-10T11:19:06.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.920+0000 7fdd44c00700 1 -- 172.21.15.27:0/1957956589 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fdd240b2a70 msgr2=0x7fdd240b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:06.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.920+0000 7fdd44c00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fdd240b2a70 0x7fdd240b4f30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdd30014fd0 tx=0x7fdd3001e340 comp rx=0 tx=0).stop 2024-05-10T11:19:06.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.920+0000 7fdd44c00700 1 -- 172.21.15.27:0/1957956589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd40102ca0 msgr2=0x7fdd401a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:06.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.920+0000 7fdd44c00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd40102ca0 0x7fdd401a0690 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800d720 tx=0x7fdd3800da30 comp rx=0 tx=0).stop 2024-05-10T11:19:06.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.920+0000 7fdd44c00700 1 -- 172.21.15.27:0/1957956589 shutdown_connections 2024-05-10T11:19:06.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.920+0000 7fdd44c00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fdd240b2a70 0x7fdd240b4f30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.920+0000 7fdd44c00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd40107790 0x7fdd401a4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.920+0000 7fdd44c00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd40102380 0x7fdd401a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.920+0000 7fdd44c00700 1 --2- 172.21.15.27:0/1957956589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd40102ca0 0x7fdd401a0690 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:06.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.920+0000 7fdd44c00700 1 -- 172.21.15.27:0/1957956589 >> 172.21.15.27:0/1957956589 conn(0x7fdd400fddc0 msgr2=0x7fdd40105140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:06.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.921+0000 7fdd44c00700 1 -- 172.21.15.27:0/1957956589 shutdown_connections 2024-05-10T11:19:06.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:06.921+0000 7fdd44c00700 1 -- 172.21.15.27:0/1957956589 wait complete. 2024-05-10T11:19:07.679 INFO:tasks.workunit.client.0.smithi027.stdout: 1 326814 7.68 MB/sec execute 112 sec latency 18.946 ms 2024-05-10T11:19:07.832 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:19:08.343 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:19:08.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:08 smithi076 ceph-mon[34152]: pgmap v180: 129 pgs: 129 active+clean; 188 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 185 op/s 2024-05-10T11:19:08.679 INFO:tasks.workunit.client.0.smithi027.stdout: 1 328755 7.68 MB/sec execute 113 sec latency 17.798 ms 2024-05-10T11:19:08.717 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:08 smithi027 ceph-mon[30054]: pgmap v180: 129 pgs: 129 active+clean; 188 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 185 op/s 2024-05-10T11:19:08.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:08 smithi189 ceph-mon[34842]: pgmap v180: 129 pgs: 129 active+clean; 188 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 185 op/s 2024-05-10T11:19:09.679 INFO:tasks.workunit.client.0.smithi027.stdout: 1 330146 7.71 MB/sec execute 114 sec latency 18.071 ms 2024-05-10T11:19:10.564 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:10 smithi027 ceph-mon[30054]: pgmap v181: 129 pgs: 129 active+clean; 193 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 268 op/s 2024-05-10T11:19:10.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:10 smithi076 ceph-mon[34152]: pgmap v181: 129 pgs: 129 active+clean; 193 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 268 op/s 2024-05-10T11:19:10.679 INFO:tasks.workunit.client.0.smithi027.stdout: 1 330854 7.65 MB/sec execute 115 sec latency 61.419 ms 2024-05-10T11:19:10.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:10 smithi189 ceph-mon[34842]: pgmap v181: 129 pgs: 129 active+clean; 193 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 268 op/s 2024-05-10T11:19:10.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.755+0000 7f69c9a00700 1 -- 172.21.15.27:0/537105185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69c4108850 msgr2=0x7f69c41103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:10.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.755+0000 7f69c9a00700 1 --2- 172.21.15.27:0/537105185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69c4108850 0x7f69c41103d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00b0d0 tx=0x7f69bc00b3e0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.756+0000 7f69c9a00700 1 -- 172.21.15.27:0/537105185 shutdown_connections 2024-05-10T11:19:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.756+0000 7f69c9a00700 1 --2- 172.21.15.27:0/537105185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69c4108850 0x7f69c41103d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.756+0000 7f69c9a00700 1 --2- 172.21.15.27:0/537105185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c4103750 0x7f69c4103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.756+0000 7f69c9a00700 1 --2- 172.21.15.27:0/537105185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69c4104100 0x7f69c4108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.756+0000 7f69c9a00700 1 -- 172.21.15.27:0/537105185 >> 172.21.15.27:0/537105185 conn(0x7f69c407a600 msgr2=0x7f69c407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.756+0000 7f69c9a00700 1 -- 172.21.15.27:0/537105185 shutdown_connections 2024-05-10T11:19:10.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.756+0000 7f69c9a00700 1 -- 172.21.15.27:0/537105185 wait complete. 2024-05-10T11:19:10.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.757+0000 7f69c9a00700 1 Processor -- start 2024-05-10T11:19:10.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.757+0000 7f69c9a00700 1 -- start start 2024-05-10T11:19:10.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.758+0000 7f69c9a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69c4103750 0x7f69c419bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.758+0000 7f69c9a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69c4104100 0x7f69c419c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.758+0000 7f69c9a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c4108850 0x7f69c41a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.758+0000 7f69c9a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f69c40769b0 con 0x7f69c4108850 2024-05-10T11:19:10.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.758+0000 7f69c9a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f69c4076830 con 0x7f69c4103750 2024-05-10T11:19:10.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.758+0000 7f69c9a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f69c4076b30 con 0x7f69c4104100 2024-05-10T11:19:10.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.758+0000 7f69c2a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69c4103750 0x7f69c419bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.758+0000 7f69c3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c4108850 0x7f69c41a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.758+0000 7f69c2a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69c4103750 0x7f69c419bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58588/0 (socket says 172.21.15.27:58588) 2024-05-10T11:19:10.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.758+0000 7f69c2a00700 1 -- 172.21.15.27:0/1163246239 learned_addr learned my addr 172.21.15.27:0/1163246239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:10.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.758+0000 7f69ba000700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69c4104100 0x7f69c419c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.759+0000 7f69c2a00700 1 -- 172.21.15.27:0/1163246239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69c4104100 msgr2=0x7f69c419c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.759+0000 7f69c2a00700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69c4104100 0x7f69c419c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.759+0000 7f69c2a00700 1 -- 172.21.15.27:0/1163246239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c4108850 msgr2=0x7f69c41a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.759+0000 7f69c2a00700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c4108850 0x7f69c41a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.759+0000 7f69c2a00700 1 -- 172.21.15.27:0/1163246239 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c41a0d60 con 0x7f69c4103750 2024-05-10T11:19:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.759+0000 7f69c2a00700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69c4103750 0x7f69c419bd50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f69b400e9a0 tx=0x7f69b400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.760+0000 7f69bbe00700 1 -- 172.21.15.27:0/1163246239 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b400ca10 con 0x7f69c4103750 2024-05-10T11:19:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.760+0000 7f69bbe00700 1 -- 172.21.15.27:0/1163246239 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69b400cbb0 con 0x7f69c4103750 2024-05-10T11:19:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.760+0000 7f69bbe00700 1 -- 172.21.15.27:0/1163246239 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4018680 con 0x7f69c4103750 2024-05-10T11:19:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.760+0000 7f69c9a00700 1 -- 172.21.15.27:0/1163246239 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c41a1050 con 0x7f69c4103750 2024-05-10T11:19:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.760+0000 7f69c9a00700 1 -- 172.21.15.27:0/1163246239 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c41a88d0 con 0x7f69c4103750 2024-05-10T11:19:10.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.764+0000 7f69bbe00700 1 -- 172.21.15.27:0/1163246239 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f69b4010430 con 0x7f69c4103750 2024-05-10T11:19:10.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.765+0000 7f69d432bb80 1 Processor -- start 2024-05-10T11:19:10.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.765+0000 7f69d432bb80 1 -- start start 2024-05-10T11:19:10.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.765+0000 7f69d432bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.765+0000 7f69d432bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70db2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.765+0000 7f69d432bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70dbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.765+0000 7f69d432bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5555f702da20 con 0x5555f70e14e0 2024-05-10T11:19:10.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.765+0000 7f69d432bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5555f6fcb760 con 0x5555f70e0bc0 2024-05-10T11:19:10.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.765+0000 7f69d432bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5555f70299e0 con 0x5555f70db8f0 2024-05-10T11:19:10.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.766+0000 7f69bbe00700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f69a80b2af0 0x7f69a80b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.766+0000 7f69bbe00700 1 -- 172.21.15.27:0/1163246239 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f69b40e2b40 con 0x7f69c4103750 2024-05-10T11:19:10.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.766+0000 7f69c3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.766+0000 7f69c3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58598/0 (socket says 172.21.15.27:58598) 2024-05-10T11:19:10.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.766+0000 7f69c3400700 1 -- 172.21.15.27:0/730682145 learned_addr learned my addr 172.21.15.27:0/730682145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:10.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.766+0000 7f69c2a00700 1 --2- 172.21.15.27:0/730682145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70db2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.766+0000 7f69c3400700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f69a80b2af0 0x7f69a80b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.766+0000 7f69ba000700 1 --2- 172.21.15.27:0/730682145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70dbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.767+0000 7f69c2a00700 1 -- 172.21.15.27:0/730682145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 msgr2=0x5555f70dbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:10.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.767+0000 7f69c2a00700 1 --2- 172.21.15.27:0/730682145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70dbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.767+0000 7f69c2a00700 1 -- 172.21.15.27:0/730682145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 msgr2=0x5555f70e0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:10.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.767+0000 7f69c2a00700 1 --2- 172.21.15.27:0/730682145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.767+0000 7f69c2a00700 1 -- 172.21.15.27:0/730682145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555f6f7a8d0 con 0x5555f70e14e0 2024-05-10T11:19:10.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.767+0000 7f69ba000700 1 --2- 172.21.15.27:0/730682145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70dbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:19:10.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.767+0000 7f69c2a00700 1 --2- 172.21.15.27:0/730682145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70db2d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f69b40dfe60 tx=0x7f69b40b2190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23efc6c996093ae8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:10.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.767+0000 7f69c3400700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f69a80b2af0 0x7f69a80b4fb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0060b0 tx=0x7f69bc009c00 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:10.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.767+0000 7f69b3e00700 1 -- 172.21.15.27:0/730682145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b40a99f0 con 0x5555f70e14e0 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.767+0000 7f69b3e00700 1 -- 172.21.15.27:0/730682145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69b40adda0 con 0x5555f70e14e0 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.767+0000 7f69b3e00700 1 -- 172.21.15.27:0/730682145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b40a9eb0 con 0x5555f70e14e0 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.768+0000 7f69d432bb80 1 -- 172.21.15.27:0/730682145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 msgr2=0x5555f70db2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.768+0000 7f69d432bb80 1 --2- 172.21.15.27:0/730682145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70db2d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f69b40dfe60 tx=0x7f69b40b2190 comp rx=0 tx=0).stop 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.768+0000 7f69d432bb80 1 -- 172.21.15.27:0/730682145 shutdown_connections 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.768+0000 7f69d432bb80 1 --2- 172.21.15.27:0/730682145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70dbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.768+0000 7f69d432bb80 1 --2- 172.21.15.27:0/730682145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.768+0000 7f69d432bb80 1 --2- 172.21.15.27:0/730682145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70db2d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.768+0000 7f69d432bb80 1 -- 172.21.15.27:0/730682145 >> 172.21.15.27:0/730682145 conn(0x5555f70536d0 msgr2=0x5555f70d5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.769+0000 7f69d432bb80 1 -- 172.21.15.27:0/730682145 shutdown_connections 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.769+0000 7f69d432bb80 1 -- 172.21.15.27:0/730682145 wait complete. 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.769+0000 7f69d432bb80 1 -- start start 2024-05-10T11:19:10.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.769+0000 7f69d432bb80 1 Processor -- start 2024-05-10T11:19:10.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.770+0000 7f69d432bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70e7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.770+0000 7f69d432bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.770+0000 7f69d432bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70e4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.770+0000 7f69d432bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5555f70e7f20 con 0x5555f70e14e0 2024-05-10T11:19:10.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.770+0000 7f69d432bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5555f6f93b20 con 0x5555f70e0bc0 2024-05-10T11:19:10.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.770+0000 7f69d432bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5555f70e80a0 con 0x5555f70db8f0 2024-05-10T11:19:10.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.770+0000 7f69d432bb80 10 client.0 _ll_register_callbacks cb 0x5555f6fd4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:19:10.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.770+0000 7f69ba000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.770+0000 7f69ba000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58602/0 (socket says 172.21.15.27:58602) 2024-05-10T11:19:10.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.770+0000 7f69ba000700 1 -- 172.21.15.27:0/2279019293 learned_addr learned my addr 172.21.15.27:0/2279019293 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.770+0000 7f69c3400700 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70e4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.771+0000 7f69c2a00700 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70e7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.771+0000 7f69ba000700 1 -- 172.21.15.27:0/2279019293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 msgr2=0x5555f70e7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.771+0000 7f69ba000700 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70e7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.771+0000 7f69ba000700 1 -- 172.21.15.27:0/2279019293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 msgr2=0x5555f70e4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.771+0000 7f69ba000700 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70e4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.771+0000 7f69ba000700 1 -- 172.21.15.27:0/2279019293 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555f6ff3420 con 0x5555f70e0bc0 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.771+0000 7f69c3400700 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70e4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.771+0000 7f69c2a00700 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70e7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.771+0000 7f69ba000700 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e45f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f69ac011970 tx=0x7f69ac011d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50eaf2619df34e48 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.771+0000 7f69b0c00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.772+0000 7f69b0c00700 1 -- 172.21.15.27:0/2279019293 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac004240 con 0x5555f70e0bc0 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.772+0000 7f69b0c00700 1 -- 172.21.15.27:0/2279019293 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f69ac0043e0 con 0x5555f70e0bc0 2024-05-10T11:19:10.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.772+0000 7f69d432bb80 1 -- 172.21.15.27:0/2279019293 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69b40adda0 con 0x5555f70e0bc0 2024-05-10T11:19:10.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.772+0000 7f69b0c00700 1 -- 172.21.15.27:0/2279019293 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f69ac025050 con 0x5555f70e0bc0 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.772+0000 7f69d432bb80 10 client.26216 fetch_fsmap learned FSMap version 33 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.772+0000 7f69d432bb80 1 -- 172.21.15.27:0/2279019293 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5555f70f2880 con 0x5555f70e0bc0 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.772+0000 7f69b0c00700 1 -- 172.21.15.27:0/2279019293 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69b40adda0 con 0x5555f70e0bc0 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.773+0000 7f69b0c00700 1 -- 172.21.15.27:0/2279019293 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f69ac019030 con 0x5555f70e0bc0 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.773+0000 7f69d432bb80 10 client.26216 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.773+0000 7f69d432bb80 10 client.26216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.773+0000 7f69d432bb80 20 client.26216 populate_metadata read hostname 'smithi027' 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.773+0000 7f69d432bb80 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5555f70f4f10 0x5555f70f7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.773+0000 7f69d432bb80 4 client.26216 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.773+0000 7f69d432bb80 1 -- 172.21.15.27:0/2279019293 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555f70a8140 con 0x5555f70f4f10 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.773+0000 7f69c2a00700 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5555f70f4f10 0x5555f70f7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.774+0000 7f69c2a00700 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5555f70f4f10 0x5555f70f7340 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35b876126bbe5340 server_cookie=e5bdc829064c94a5 in_seq=0 out_seq=0 2024-05-10T11:19:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.774+0000 7f69b0c00700 10 client.26216 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:10.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69b0c00700 1 -- 172.21.15.27:0/2279019293 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5555f70a8140 con 0x5555f70f4f10 2024-05-10T11:19:10.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69b0c00700 10 client.26216 handle_command_reply: tid=0 2024-05-10T11:19:10.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69d432bb80 1 client.26216 shutdown 2024-05-10T11:19:10.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69d432bb80 1 -- 172.21.15.27:0/2279019293 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 msgr2=0x5555f70e45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:10.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69d432bb80 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e45f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f69ac011970 tx=0x7f69ac011d30 comp rx=0 tx=0).stop 2024-05-10T11:19:10.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69d432bb80 1 -- 172.21.15.27:0/2279019293 shutdown_connections 2024-05-10T11:19:10.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69d432bb80 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5555f70f4f10 0x5555f70f7340 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69d432bb80 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70e7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69d432bb80 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70e4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69d432bb80 1 --2- 172.21.15.27:0/2279019293 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e45f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69b0c00700 0 client.26216 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:10.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69d432bb80 1 -- 172.21.15.27:0/2279019293 >> 172.21.15.27:0/2279019293 conn(0x5555f70536d0 msgr2=0x5555f70d5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:10.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69d432bb80 1 -- 172.21.15.27:0/2279019293 shutdown_connections 2024-05-10T11:19:10.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.775+0000 7f69d432bb80 1 -- 172.21.15.27:0/2279019293 wait complete. 2024-05-10T11:19:10.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.776+0000 7f69d432bb80 20 client.26216 trim_cache size 0 max 16384 2024-05-10T11:19:10.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.794+0000 7f69d432bb80 1 Processor -- start 2024-05-10T11:19:10.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.794+0000 7f69d432bb80 1 -- start start 2024-05-10T11:19:10.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.794+0000 7f69d432bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70db8f0 0x5555f70db2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.794+0000 7f69d432bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69d432bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70e14e0 0x5555f70f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69d432bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41b18a0 con 0x5555f70db8f0 2024-05-10T11:19:10.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69d432bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41b33e0 con 0x5555f70e0bc0 2024-05-10T11:19:10.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69d432bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41b1f70 con 0x5555f70e14e0 2024-05-10T11:19:10.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69ba000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69ba000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58604/0 (socket says 172.21.15.27:58604) 2024-05-10T11:19:10.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69ba000700 1 -- 172.21.15.27:0/1104125169 learned_addr learned my addr 172.21.15.27:0/1104125169 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:10.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69c2a00700 1 --2- 172.21.15.27:0/1104125169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70db8f0 0x5555f70db2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69c3400700 1 --2- 172.21.15.27:0/1104125169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70e14e0 0x5555f70f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69ba000700 1 -- 172.21.15.27:0/1104125169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70e14e0 msgr2=0x5555f70f0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:10.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69ba000700 1 --2- 172.21.15.27:0/1104125169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70e14e0 0x5555f70f0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69ba000700 1 -- 172.21.15.27:0/1104125169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70db8f0 msgr2=0x5555f70db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:10.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69ba000700 1 --2- 172.21.15.27:0/1104125169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70db8f0 0x5555f70db2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.795+0000 7f69ba000700 1 -- 172.21.15.27:0/1104125169 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555f70f14f0 con 0x5555f70e0bc0 2024-05-10T11:19:10.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.796+0000 7f69ba000700 1 --2- 172.21.15.27:0/1104125169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f69ac003da0 tx=0x7f69ac005820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c02d312aad42b4a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:10.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.796+0000 7f69b3400700 1 -- 172.21.15.27:0/1104125169 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac02e070 con 0x5555f70e0bc0 2024-05-10T11:19:10.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.796+0000 7f69b3400700 1 -- 172.21.15.27:0/1104125169 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69ac0052a0 con 0x5555f70e0bc0 2024-05-10T11:19:10.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.796+0000 7f69b3400700 1 -- 172.21.15.27:0/1104125169 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac00fdc0 con 0x5555f70e0bc0 2024-05-10T11:19:10.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.797+0000 7f69d432bb80 1 -- 172.21.15.27:0/1104125169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 msgr2=0x5555f70e4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:10.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.797+0000 7f69d432bb80 1 --2- 172.21.15.27:0/1104125169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f69ac003da0 tx=0x7f69ac005820 comp rx=0 tx=0).stop 2024-05-10T11:19:10.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.797+0000 7f69d432bb80 1 -- 172.21.15.27:0/1104125169 shutdown_connections 2024-05-10T11:19:10.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.797+0000 7f69d432bb80 1 --2- 172.21.15.27:0/1104125169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70e14e0 0x5555f70f0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.797+0000 7f69d432bb80 1 --2- 172.21.15.27:0/1104125169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70db8f0 0x5555f70db2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.797+0000 7f69d432bb80 1 --2- 172.21.15.27:0/1104125169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70e4cc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.797+0000 7f69d432bb80 1 -- 172.21.15.27:0/1104125169 >> 172.21.15.27:0/1104125169 conn(0x5555f70536d0 msgr2=0x5555f70e45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:10.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.797+0000 7f69d432bb80 1 -- 172.21.15.27:0/1104125169 shutdown_connections 2024-05-10T11:19:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.797+0000 7f69d432bb80 1 -- 172.21.15.27:0/1104125169 wait complete. 2024-05-10T11:19:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.798+0000 7f69d432bb80 1 -- start start 2024-05-10T11:19:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.798+0000 7f69d432bb80 1 Processor -- start 2024-05-10T11:19:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.798+0000 7f69d432bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70ed850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.798+0000 7f69d432bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70fc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.798+0000 7f69d432bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70fc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.798+0000 7f69d432bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41b2d10 con 0x5555f70e14e0 2024-05-10T11:19:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.798+0000 7f69d432bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41b2640 con 0x5555f70e0bc0 2024-05-10T11:19:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.798+0000 7f69d432bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41af840 con 0x5555f70db8f0 2024-05-10T11:19:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69d432bb80 10 client.0 _ll_register_callbacks cb 0x5555f6fd4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:19:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69ba000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69c3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70fc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69c3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70fc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37690/0 (socket says 172.21.15.27:37690) 2024-05-10T11:19:10.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69c3400700 1 -- 172.21.15.27:0/1822721932 learned_addr learned my addr 172.21.15.27:0/1822721932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:10.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69c2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70ed850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69ba000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58606/0 (socket says 172.21.15.27:58606) 2024-05-10T11:19:10.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69ba000700 1 -- 172.21.15.27:0/1822721932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 msgr2=0x5555f70ed850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:10.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69ba000700 1 --2- 172.21.15.27:0/1822721932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70ed850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69ba000700 1 -- 172.21.15.27:0/1822721932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 msgr2=0x5555f70fc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69ba000700 1 --2- 172.21.15.27:0/1822721932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70fc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69ba000700 1 -- 172.21.15.27:0/1822721932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555f70fcfa0 con 0x5555f70e0bc0 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.799+0000 7f69c2a00700 1 --2- 172.21.15.27:0/1822721932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70ed850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.800+0000 7f69ba000700 1 --2- 172.21.15.27:0/1822721932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70fc220 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x5555f70e8010 tx=0x7f69ac005910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=280b7bc273a6fa19 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.800+0000 7f69b0c00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.800+0000 7f69b0c00700 1 -- 172.21.15.27:0/1822721932 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac038020 con 0x5555f70e0bc0 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.800+0000 7f69b0c00700 1 -- 172.21.15.27:0/1822721932 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f69ac003680 con 0x5555f70e0bc0 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.800+0000 7f69d432bb80 1 -- 172.21.15.27:0/1822721932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69ac0052a0 con 0x5555f70e0bc0 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.800+0000 7f69b0c00700 1 -- 172.21.15.27:0/1822721932 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f69ac02e040 con 0x5555f70e0bc0 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.800+0000 7f69d432bb80 10 client.26222 fetch_fsmap learned FSMap version 33 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.800+0000 7f69d432bb80 1 -- 172.21.15.27:0/1822721932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5555f715f790 con 0x5555f70e0bc0 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.801+0000 7f69b0c00700 1 -- 172.21.15.27:0/1822721932 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69ac0052a0 con 0x5555f70e0bc0 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.801+0000 7f69b0c00700 1 -- 172.21.15.27:0/1822721932 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f69ac019040 con 0x5555f70e0bc0 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.801+0000 7f69d432bb80 10 client.26222 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:19:10.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.801+0000 7f69d432bb80 10 client.26222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:19:10.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.801+0000 7f69d432bb80 20 client.26222 populate_metadata read hostname 'smithi027' 2024-05-10T11:19:10.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.801+0000 7f69d432bb80 1 --2- 172.21.15.27:0/1822721932 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5555f715fd80 0x5555f71621a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:10.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.801+0000 7f69d432bb80 4 client.26222 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:19:10.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.801+0000 7f69d432bb80 1 -- 172.21.15.27:0/1822721932 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5555f703b400 con 0x5555f715fd80 2024-05-10T11:19:10.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.801+0000 7f69c2a00700 1 --2- 172.21.15.27:0/1822721932 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5555f715fd80 0x5555f71621a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:10.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69c2a00700 1 --2- 172.21.15.27:0/1822721932 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5555f715fd80 0x5555f71621a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2f4faf0973c5fc6 server_cookie=b7336f19c6bb34a0 in_seq=0 out_seq=0 2024-05-10T11:19:10.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69b0c00700 10 client.26222 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:10.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69b0c00700 1 -- 172.21.15.27:0/1822721932 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5555f703b400 con 0x5555f715fd80 2024-05-10T11:19:10.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69b0c00700 10 client.26222 handle_command_reply: tid=0 2024-05-10T11:19:10.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69d432bb80 1 client.26222 shutdown 2024-05-10T11:19:10.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69d432bb80 1 -- 172.21.15.27:0/1822721932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 msgr2=0x5555f70fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:10.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69d432bb80 1 --2- 172.21.15.27:0/1822721932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70fc220 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x5555f70e8010 tx=0x7f69ac005910 comp rx=0 tx=0).stop 2024-05-10T11:19:10.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69d432bb80 1 -- 172.21.15.27:0/1822721932 shutdown_connections 2024-05-10T11:19:10.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69d432bb80 1 --2- 172.21.15.27:0/1822721932 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5555f715fd80 0x5555f71621a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69d432bb80 1 --2- 172.21.15.27:0/1822721932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555f70db8f0 0x5555f70ed850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69d432bb80 1 --2- 172.21.15.27:0/1822721932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555f70e14e0 0x5555f70fc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69d432bb80 1 --2- 172.21.15.27:0/1822721932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5555f70e0bc0 0x5555f70fc220 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69d432bb80 1 -- 172.21.15.27:0/1822721932 >> 172.21.15.27:0/1822721932 conn(0x5555f70536d0 msgr2=0x5555f70e1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.802+0000 7f69b0c00700 0 client.26222 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.803+0000 7f69d432bb80 1 -- 172.21.15.27:0/1822721932 shutdown_connections 2024-05-10T11:19:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.803+0000 7f69d432bb80 1 -- 172.21.15.27:0/1822721932 wait complete. 2024-05-10T11:19:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.803+0000 7f69d432bb80 20 client.26222 trim_cache size 0 max 16384 2024-05-10T11:19:10.806 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:19:10.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.803+0000 7f69c9a00700 1 -- 172.21.15.27:0/1163246239 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f69a80b2af0 msgr2=0x7f69a80b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:10.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.803+0000 7f69c9a00700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f69a80b2af0 0x7f69a80b4fb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0060b0 tx=0x7f69bc009c00 comp rx=0 tx=0).stop 2024-05-10T11:19:10.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.803+0000 7f69c9a00700 1 -- 172.21.15.27:0/1163246239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69c4103750 msgr2=0x7f69c419bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:10.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.803+0000 7f69c9a00700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69c4103750 0x7f69c419bd50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f69b400e9a0 tx=0x7f69b400ecb0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.804+0000 7f69c9a00700 1 -- 172.21.15.27:0/1163246239 shutdown_connections 2024-05-10T11:19:10.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.804+0000 7f69c9a00700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f69a80b2af0 0x7f69a80b4fb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.804+0000 7f69c9a00700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c4108850 0x7f69c41a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.804+0000 7f69c9a00700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69c4103750 0x7f69c419bd50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.804+0000 7f69c9a00700 1 --2- 172.21.15.27:0/1163246239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69c4104100 0x7f69c419c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:10.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.804+0000 7f69c9a00700 1 -- 172.21.15.27:0/1163246239 >> 172.21.15.27:0/1163246239 conn(0x7f69c407a600 msgr2=0x7f69c4101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:10.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.804+0000 7f69c9a00700 1 -- 172.21.15.27:0/1163246239 shutdown_connections 2024-05-10T11:19:10.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:10.804+0000 7f69c9a00700 1 -- 172.21.15.27:0/1163246239 wait complete. 2024-05-10T11:19:11.679 INFO:tasks.workunit.client.0.smithi027.stdout: 1 332817 7.69 MB/sec execute 116 sec latency 87.300 ms 2024-05-10T11:19:12.683 INFO:tasks.workunit.client.0.smithi027.stdout: 1 333792 7.68 MB/sec execute 117 sec latency 23.301 ms 2024-05-10T11:19:12.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:12 smithi189 ceph-mon[34842]: pgmap v182: 129 pgs: 129 active+clean; 193 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 187 op/s 2024-05-10T11:19:12.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:12 smithi027 ceph-mon[30054]: pgmap v182: 129 pgs: 129 active+clean; 193 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 187 op/s 2024-05-10T11:19:12.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:12 smithi076 ceph-mon[34152]: pgmap v182: 129 pgs: 129 active+clean; 193 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 187 op/s 2024-05-10T11:19:13.046 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:19:13.047 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:19:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:19:13.679 INFO:tasks.workunit.client.0.smithi027.stdout: 1 334022 7.61 MB/sec execute 118 sec latency 515.644 ms 2024-05-10T11:19:14.679 INFO:tasks.workunit.client.0.smithi027.stdout: 1 334022 7.55 MB/sec execute 119 sec latency 1515.736 ms 2024-05-10T11:19:14.722 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:14 smithi027 ceph-mon[30054]: pgmap v183: 129 pgs: 129 active+clean; 193 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 187 op/s 2024-05-10T11:19:14.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:14 smithi189 ceph-mon[34842]: pgmap v183: 129 pgs: 129 active+clean; 193 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 187 op/s 2024-05-10T11:19:14.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:14 smithi076 ceph-mon[34152]: pgmap v183: 129 pgs: 129 active+clean; 193 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 187 op/s 2024-05-10T11:19:15.680 INFO:tasks.workunit.client.0.smithi027.stdout: 1 334022 7.49 MB/sec execute 120 sec latency 2515.830 ms 2024-05-10T11:19:15.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.894+0000 7f57e2c00700 1 -- 172.21.15.27:0/323595457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57dc1056c0 msgr2=0x7f57dc105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:15.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.894+0000 7f57e2c00700 1 --2- 172.21.15.27:0/323595457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57dc1056c0 0x7f57dc105b40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f57c8009990 tx=0x7f57c8009ca0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.894+0000 7f57e2c00700 1 -- 172.21.15.27:0/323595457 shutdown_connections 2024-05-10T11:19:15.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.894+0000 7f57e2c00700 1 --2- 172.21.15.27:0/323595457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57dc106080 0x7f57dc112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.894+0000 7f57e2c00700 1 --2- 172.21.15.27:0/323595457 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57dc10b6c0 0x7f57dc10baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.894+0000 7f57e2c00700 1 --2- 172.21.15.27:0/323595457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57dc1056c0 0x7f57dc105b40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.894+0000 7f57e2c00700 1 -- 172.21.15.27:0/323595457 >> 172.21.15.27:0/323595457 conn(0x7f57dc1011b0 msgr2=0x7f57dc1035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:15.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.894+0000 7f57e2c00700 1 -- 172.21.15.27:0/323595457 shutdown_connections 2024-05-10T11:19:15.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.894+0000 7f57e2c00700 1 -- 172.21.15.27:0/323595457 wait complete. 2024-05-10T11:19:15.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.895+0000 7f57e2c00700 1 Processor -- start 2024-05-10T11:19:15.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.896+0000 7f57e2c00700 1 -- start start 2024-05-10T11:19:15.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.896+0000 7f57e2c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57dc1056c0 0x7f57dc19bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.896+0000 7f57e2c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57dc106080 0x7f57dc19c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.896+0000 7f57e2c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57dc10b6c0 0x7f57dc1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.896+0000 7f57e2c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc115950 con 0x7f57dc106080 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.896+0000 7f57e2c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc1157d0 con 0x7f57dc10b6c0 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.896+0000 7f57e2c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc115ad0 con 0x7f57dc1056c0 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.897+0000 7f57dbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57dc1056c0 0x7f57dc19bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.897+0000 7f57db400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57dc106080 0x7f57dc19c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.897+0000 7f57dbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57dc1056c0 0x7f57dc19bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60072/0 (socket says 172.21.15.27:60072) 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.897+0000 7f57dbe00700 1 -- 172.21.15.27:0/77574373 learned_addr learned my addr 172.21.15.27:0/77574373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.897+0000 7f57e0c00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57dc10b6c0 0x7f57dc1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.897+0000 7f57dbe00700 1 -- 172.21.15.27:0/77574373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57dc10b6c0 msgr2=0x7f57dc1a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.897+0000 7f57dbe00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57dc10b6c0 0x7f57dc1a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.897+0000 7f57dbe00700 1 -- 172.21.15.27:0/77574373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57dc106080 msgr2=0x7f57dc19c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.897+0000 7f57dbe00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57dc106080 0x7f57dc19c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.897+0000 7f57dbe00700 1 -- 172.21.15.27:0/77574373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57dc1a0db0 con 0x7f57dc1056c0 2024-05-10T11:19:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.897+0000 7f57db400700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57dc106080 0x7f57dc19c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:19:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.898+0000 7f57dbe00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57dc1056c0 0x7f57dc19bda0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00e9a0 tx=0x7f57cc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.898+0000 7f57d8c00700 1 -- 172.21.15.27:0/77574373 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc00ca10 con 0x7f57dc1056c0 2024-05-10T11:19:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.898+0000 7f57d8c00700 1 -- 172.21.15.27:0/77574373 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57cc00cbb0 con 0x7f57dc1056c0 2024-05-10T11:19:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.898+0000 7f57d8c00700 1 -- 172.21.15.27:0/77574373 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc018680 con 0x7f57dc1056c0 2024-05-10T11:19:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.898+0000 7f57e2c00700 1 -- 172.21.15.27:0/77574373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57dc1a10a0 con 0x7f57dc1056c0 2024-05-10T11:19:15.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.898+0000 7f57e2c00700 1 -- 172.21.15.27:0/77574373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57dc1a8920 con 0x7f57dc1056c0 2024-05-10T11:19:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.902+0000 7f57d8c00700 1 -- 172.21.15.27:0/77574373 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f57cc010430 con 0x7f57dc1056c0 2024-05-10T11:19:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.903+0000 7f57ed5a4b80 1 Processor -- start 2024-05-10T11:19:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.903+0000 7f57ed5a4b80 1 -- start start 2024-05-10T11:19:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.903+0000 7f57ed5a4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb7414c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.903+0000 7f57ed5a4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb741a00 0x5627fb745510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.903+0000 7f57ed5a4b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb745a50 0x5627fb745eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.903+0000 7f57ed5a4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5627fb693aa0 con 0x5627fb7410e0 2024-05-10T11:19:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.903+0000 7f57ed5a4b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5627fb6317e0 con 0x5627fb745a50 2024-05-10T11:19:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.903+0000 7f57ed5a4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5627fb68fa60 con 0x5627fb741a00 2024-05-10T11:19:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.904+0000 7f57e0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb7414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.904+0000 7f57e0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb7414c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35984/0 (socket says 172.21.15.27:35984) 2024-05-10T11:19:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.904+0000 7f57e0c00700 1 -- 172.21.15.27:0/1480650220 learned_addr learned my addr 172.21.15.27:0/1480650220 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.904+0000 7f57db400700 1 --2- 172.21.15.27:0/1480650220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb745a50 0x5627fb745eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.904+0000 7f57dbe00700 1 --2- 172.21.15.27:0/1480650220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb741a00 0x5627fb745510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.904+0000 7f57e0c00700 1 -- 172.21.15.27:0/1480650220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb741a00 msgr2=0x5627fb745510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.904+0000 7f57e0c00700 1 --2- 172.21.15.27:0/1480650220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb741a00 0x5627fb745510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.904+0000 7f57e0c00700 1 -- 172.21.15.27:0/1480650220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb745a50 msgr2=0x5627fb745eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.904+0000 7f57e0c00700 1 --2- 172.21.15.27:0/1480650220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb745a50 0x5627fb745eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.904+0000 7f57e0c00700 1 -- 172.21.15.27:0/1480650220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627fb5e0950 con 0x5627fb7410e0 2024-05-10T11:19:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.904+0000 7f57dbe00700 1 --2- 172.21.15.27:0/1480650220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb741a00 0x5627fb745510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:19:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.905+0000 7f57d8c00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f57c00b2a70 0x7f57c00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.905+0000 7f57d8c00700 1 -- 172.21.15.27:0/77574373 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f57cc0e2ad0 con 0x7f57dc1056c0 2024-05-10T11:19:15.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.905+0000 7f57e0c00700 1 --2- 172.21.15.27:0/1480650220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb7414c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f57d4011680 tx=0x7f57d4009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562d47a594c1c555 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:15.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.905+0000 7f57dbe00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f57c00b2a70 0x7f57c00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.905+0000 7f57d1600700 1 -- 172.21.15.27:0/1480650220 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d4019030 con 0x5627fb7410e0 2024-05-10T11:19:15.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.905+0000 7f57d1600700 1 -- 172.21.15.27:0/1480650220 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57d40092e0 con 0x5627fb7410e0 2024-05-10T11:19:15.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.905+0000 7f57d1600700 1 -- 172.21.15.27:0/1480650220 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d4007500 con 0x5627fb7410e0 2024-05-10T11:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.906+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/1480650220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 msgr2=0x5627fb7414c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.906+0000 7f57dbe00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f57c00b2a70 0x7f57c00b4f30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00e970 tx=0x7f57cc009790 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.906+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/1480650220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb7414c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f57d4011680 tx=0x7f57d4009920 comp rx=0 tx=0).stop 2024-05-10T11:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.906+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/1480650220 shutdown_connections 2024-05-10T11:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.906+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/1480650220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb745a50 0x5627fb745eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.906+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/1480650220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb7414c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.906+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/1480650220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb741a00 0x5627fb745510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.906+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/1480650220 >> 172.21.15.27:0/1480650220 conn(0x5627fb66fbe0 msgr2=0x5627fb73b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.907+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/1480650220 shutdown_connections 2024-05-10T11:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.907+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/1480650220 wait complete. 2024-05-10T11:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.907+0000 7f57ed5a4b80 1 -- start start 2024-05-10T11:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.907+0000 7f57ed5a4b80 1 Processor -- start 2024-05-10T11:19:15.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.908+0000 7f57ed5a4b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb7410e0 0x5627fb74cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.908+0000 7f57ed5a4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb741a00 0x5627fb74a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.908+0000 7f57ed5a4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb745a50 0x5627fb74a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.908+0000 7f57ed5a4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5627fb74dc60 con 0x5627fb745a50 2024-05-10T11:19:15.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.908+0000 7f57ed5a4b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5627fb5f9ba0 con 0x5627fb7410e0 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.908+0000 7f57ed5a4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5627fb74dde0 con 0x5627fb741a00 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.908+0000 7f57ed5a4b80 10 client.0 _ll_register_callbacks cb 0x5627fb63acf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.908+0000 7f57e0c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb7410e0 0x5627fb74cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.908+0000 7f57db400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb741a00 0x5627fb74a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.908+0000 7f57e0c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb7410e0 0x5627fb74cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:49472/0 (socket says 172.21.15.27:49472) 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.908+0000 7f57e0c00700 1 -- 172.21.15.27:0/2115687167 learned_addr learned my addr 172.21.15.27:0/2115687167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.909+0000 7f57e0c00700 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb745a50 0x5627fb74a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.909+0000 7f57db400700 1 -- 172.21.15.27:0/2115687167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb7410e0 msgr2=0x5627fb74cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.909+0000 7f57db400700 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb7410e0 0x5627fb74cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.909+0000 7f57db400700 1 -- 172.21.15.27:0/2115687167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb745a50 msgr2=0x5627fb74a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.909+0000 7f57db400700 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb745a50 0x5627fb74a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.909+0000 7f57db400700 1 -- 172.21.15.27:0/2115687167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627fb6594a0 con 0x5627fb741a00 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.909+0000 7f57e0c00700 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb745a50 0x5627fb74a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:19:15.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.909+0000 7f57db400700 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb741a00 0x5627fb74a2b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f57c8006010 tx=0x7f57c8005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cad0194d1d42053 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:15.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.909+0000 7f57c6000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:19:15.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.910+0000 7f57c6000700 1 -- 172.21.15.27:0/2115687167 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c8029060 con 0x5627fb741a00 2024-05-10T11:19:15.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.910+0000 7f57c6000700 1 -- 172.21.15.27:0/2115687167 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f57c8003880 con 0x5627fb741a00 2024-05-10T11:19:15.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.910+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2115687167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57d40092e0 con 0x5627fb741a00 2024-05-10T11:19:15.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.910+0000 7f57c6000700 1 -- 172.21.15.27:0/2115687167 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f57c8030050 con 0x5627fb741a00 2024-05-10T11:19:15.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.910+0000 7f57ed5a4b80 10 client.26224 fetch_fsmap learned FSMap version 33 2024-05-10T11:19:15.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.910+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2115687167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5627fb758560 con 0x5627fb741a00 2024-05-10T11:19:15.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.910+0000 7f57c6000700 1 -- 172.21.15.27:0/2115687167 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57d40092e0 con 0x5627fb741a00 2024-05-10T11:19:15.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.911+0000 7f57c6000700 1 -- 172.21.15.27:0/2115687167 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f57c8024a20 con 0x5627fb741a00 2024-05-10T11:19:15.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.911+0000 7f57ed5a4b80 10 client.26224 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.911+0000 7f57ed5a4b80 10 client.26224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.911+0000 7f57ed5a4b80 20 client.26224 populate_metadata read hostname 'smithi027' 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.911+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627fb75aba0 0x5627fb75cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.911+0000 7f57ed5a4b80 4 client.26224 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.911+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2115687167 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627fb70e120 con 0x5627fb75aba0 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.911+0000 7f57e0c00700 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627fb75aba0 0x5627fb75cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.912+0000 7f57e0c00700 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627fb75aba0 0x5627fb75cf90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=353022f281cceb74 server_cookie=eae55a93c3a96c37 in_seq=0 out_seq=0 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.912+0000 7f57c6000700 10 client.26224 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.913+0000 7f57c6000700 1 -- 172.21.15.27:0/2115687167 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5627fb70e120 con 0x5627fb75aba0 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.913+0000 7f57c6000700 10 client.26224 handle_command_reply: tid=0 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.913+0000 7f57ed5a4b80 1 client.26224 shutdown 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.913+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2115687167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb741a00 msgr2=0x5627fb74a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.913+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb741a00 0x5627fb74a2b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f57c8006010 tx=0x7f57c8005bc0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.913+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2115687167 shutdown_connections 2024-05-10T11:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.913+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627fb75aba0 0x5627fb75cf90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.913+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb745a50 0x5627fb74a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.913+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb7410e0 0x5627fb74cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.913+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2115687167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb741a00 0x5627fb74a2b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.913+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2115687167 >> 172.21.15.27:0/2115687167 conn(0x5627fb66fbe0 msgr2=0x5627fb73b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.913+0000 7f57c6000700 0 client.26224 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.914+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2115687167 shutdown_connections 2024-05-10T11:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.914+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2115687167 wait complete. 2024-05-10T11:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.914+0000 7f57ed5a4b80 20 client.26224 trim_cache size 0 max 16384 2024-05-10T11:19:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.937+0000 7f57ed5a4b80 1 Processor -- start 2024-05-10T11:19:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.937+0000 7f57ed5a4b80 1 -- start start 2024-05-10T11:19:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.937+0000 7f57ed5a4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb756cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.937+0000 7f57ed5a4b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb741a00 0x5627fb749480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.937+0000 7f57ed5a4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb745a50 0x5627fb762860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.937+0000 7f57ed5a4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc1b17d0 con 0x5627fb7410e0 2024-05-10T11:19:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.937+0000 7f57ed5a4b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc1b3310 con 0x5627fb741a00 2024-05-10T11:19:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.937+0000 7f57ed5a4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc1b1ea0 con 0x5627fb745a50 2024-05-10T11:19:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.938+0000 7f57e0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb756cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.938+0000 7f57e0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb756cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36000/0 (socket says 172.21.15.27:36000) 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.938+0000 7f57e0c00700 1 -- 172.21.15.27:0/2615884414 learned_addr learned my addr 172.21.15.27:0/2615884414 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.938+0000 7f57db400700 1 --2- 172.21.15.27:0/2615884414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb741a00 0x5627fb749480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.938+0000 7f57e0c00700 1 --2- 172.21.15.27:0/2615884414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb745a50 0x5627fb762860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.938+0000 7f57e0c00700 1 -- 172.21.15.27:0/2615884414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb745a50 msgr2=0x5627fb762860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.938+0000 7f57e0c00700 1 --2- 172.21.15.27:0/2615884414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb745a50 0x5627fb762860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.938+0000 7f57e0c00700 1 -- 172.21.15.27:0/2615884414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb741a00 msgr2=0x5627fb749480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.938+0000 7f57e0c00700 1 --2- 172.21.15.27:0/2615884414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb741a00 0x5627fb749480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.938+0000 7f57e0c00700 1 -- 172.21.15.27:0/2615884414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627fb73c690 con 0x5627fb7410e0 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.939+0000 7f57e0c00700 1 --2- 172.21.15.27:0/2615884414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb756cf0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f57d4011620 tx=0x7f57d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab90a07c9a11681 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.939+0000 7f57d0c00700 1 -- 172.21.15.27:0/2615884414 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d4019040 con 0x5627fb7410e0 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.939+0000 7f57d0c00700 1 -- 172.21.15.27:0/2615884414 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57d4004c20 con 0x5627fb7410e0 2024-05-10T11:19:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.939+0000 7f57d0c00700 1 -- 172.21.15.27:0/2615884414 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d4003fc0 con 0x5627fb7410e0 2024-05-10T11:19:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.940+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2615884414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 msgr2=0x5627fb756cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.940+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2615884414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb756cf0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f57d4011620 tx=0x7f57d400b040 comp rx=0 tx=0).stop 2024-05-10T11:19:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.940+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2615884414 shutdown_connections 2024-05-10T11:19:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.940+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2615884414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb745a50 0x5627fb762860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.940+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2615884414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb756cf0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.940+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2615884414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb741a00 0x5627fb749480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.940+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2615884414 >> 172.21.15.27:0/2615884414 conn(0x5627fb66fbe0 msgr2=0x5627fb736330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.940+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2615884414 shutdown_connections 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.940+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2615884414 wait complete. 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.940+0000 7f57ed5a4b80 1 -- start start 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.941+0000 7f57ed5a4b80 1 Processor -- start 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.941+0000 7f57ed5a4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb7533f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.941+0000 7f57ed5a4b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb741a00 0x5627fb7517a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.941+0000 7f57ed5a4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb745a50 0x5627fb751ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.941+0000 7f57ed5a4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc1b2c40 con 0x5627fb7410e0 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.941+0000 7f57ed5a4b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc1b2570 con 0x5627fb741a00 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.941+0000 7f57ed5a4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc1af770 con 0x5627fb745a50 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.941+0000 7f57ed5a4b80 10 client.0 _ll_register_callbacks cb 0x5627fb63acf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:19:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.941+0000 7f57e0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb7533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.942+0000 7f57e0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb7533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36016/0 (socket says 172.21.15.27:36016) 2024-05-10T11:19:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.942+0000 7f57e0c00700 1 -- 172.21.15.27:0/2049182678 learned_addr learned my addr 172.21.15.27:0/2049182678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.942+0000 7f57db400700 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb741a00 0x5627fb7517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.942+0000 7f57e0c00700 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb745a50 0x5627fb751ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.942+0000 7f57e0c00700 1 -- 172.21.15.27:0/2049182678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb745a50 msgr2=0x5627fb751ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.942+0000 7f57e0c00700 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb745a50 0x5627fb751ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.942+0000 7f57e0c00700 1 -- 172.21.15.27:0/2049182678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb741a00 msgr2=0x5627fb7517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.942+0000 7f57e0c00700 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb741a00 0x5627fb7517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.942+0000 7f57e0c00700 1 -- 172.21.15.27:0/2049182678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627fb752390 con 0x5627fb7410e0 2024-05-10T11:19:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.942+0000 7f57db400700 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb741a00 0x5627fb7517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:19:15.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.942+0000 7f57e0c00700 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb7533f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f57d4000f80 tx=0x7f57d4009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81bf1e8ecde41152 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.942+0000 7f57c6000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.943+0000 7f57c6000700 1 -- 172.21.15.27:0/2049182678 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d4034020 con 0x5627fb7410e0 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.943+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2049182678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57d4004c20 con 0x5627fb7410e0 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.943+0000 7f57c6000700 1 -- 172.21.15.27:0/2049182678 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f57d40046f0 con 0x5627fb7410e0 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.943+0000 7f57c6000700 1 -- 172.21.15.27:0/2049182678 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f57d4019070 con 0x5627fb7410e0 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.943+0000 7f57ed5a4b80 10 client.16035 fetch_fsmap learned FSMap version 33 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.943+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2049182678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5627fb7c6350 con 0x5627fb7410e0 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.943+0000 7f57c6000700 1 -- 172.21.15.27:0/2049182678 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57d4004c20 con 0x5627fb7410e0 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.943+0000 7f57c6000700 1 -- 172.21.15.27:0/2049182678 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f57d402b020 con 0x5627fb7410e0 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.944+0000 7f57ed5a4b80 10 client.16035 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.944+0000 7f57ed5a4b80 10 client.16035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.944+0000 7f57ed5a4b80 20 client.16035 populate_metadata read hostname 'smithi027' 2024-05-10T11:19:15.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.944+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627fb7c6750 0x5627fb7c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:15.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.944+0000 7f57ed5a4b80 4 client.16035 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:19:15.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.944+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2049182678 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5627fb6a1480 con 0x5627fb7c6750 2024-05-10T11:19:15.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.944+0000 7f57db400700 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627fb7c6750 0x5627fb7c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:15.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.945+0000 7f57db400700 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627fb7c6750 0x5627fb7c8b40 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adc1d1c1e85fa780 server_cookie=a24acbeb2ffb69f in_seq=0 out_seq=0 2024-05-10T11:19:15.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.945+0000 7f57c6000700 10 client.16035 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57c6000700 1 -- 172.21.15.27:0/2049182678 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5627fb6a1480 con 0x5627fb7c6750 2024-05-10T11:19:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57c6000700 10 client.16035 handle_command_reply: tid=0 2024-05-10T11:19:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57ed5a4b80 1 client.16035 shutdown 2024-05-10T11:19:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2049182678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 msgr2=0x5627fb7533f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb7533f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f57d4000f80 tx=0x7f57d4009ac0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2049182678 shutdown_connections 2024-05-10T11:19:15.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627fb7c6750 0x5627fb7c8b40 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627fb745a50 0x5627fb751ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627fb7410e0 0x5627fb7533f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57ed5a4b80 1 --2- 172.21.15.27:0/2049182678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627fb741a00 0x5627fb7517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2049182678 >> 172.21.15.27:0/2049182678 conn(0x5627fb66fbe0 msgr2=0x5627fb73fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:15.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57c6000700 0 client.16035 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:15.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.946+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2049182678 shutdown_connections 2024-05-10T11:19:15.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.947+0000 7f57ed5a4b80 1 -- 172.21.15.27:0/2049182678 wait complete. 2024-05-10T11:19:15.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.947+0000 7f57ed5a4b80 20 client.16035 trim_cache size 0 max 16384 2024-05-10T11:19:15.952 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:19:15.953 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:19:15.953 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "6e8c66d6-0fd0-47f2-9f93-08ab8547b646", 2024-05-10T11:19:15.953 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:19:15.953 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:19:15.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.948+0000 7f57e2c00700 1 -- 172.21.15.27:0/77574373 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f57c00b2a70 msgr2=0x7f57c00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:15.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.948+0000 7f57e2c00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f57c00b2a70 0x7f57c00b4f30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00e970 tx=0x7f57cc009790 comp rx=0 tx=0).stop 2024-05-10T11:19:15.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.948+0000 7f57e2c00700 1 -- 172.21.15.27:0/77574373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57dc1056c0 msgr2=0x7f57dc19bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:15.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.948+0000 7f57e2c00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57dc1056c0 0x7f57dc19bda0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00e9a0 tx=0x7f57cc00ecb0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.948+0000 7f57e2c00700 1 -- 172.21.15.27:0/77574373 shutdown_connections 2024-05-10T11:19:15.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.948+0000 7f57e2c00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f57c00b2a70 0x7f57c00b4f30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.948+0000 7f57e2c00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57dc1056c0 0x7f57dc19bda0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.948+0000 7f57e2c00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57dc10b6c0 0x7f57dc1a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.948+0000 7f57e2c00700 1 --2- 172.21.15.27:0/77574373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57dc106080 0x7f57dc19c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:15.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.948+0000 7f57e2c00700 1 -- 172.21.15.27:0/77574373 >> 172.21.15.27:0/77574373 conn(0x7f57dc1011b0 msgr2=0x7f57dc101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:15.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.948+0000 7f57e2c00700 1 -- 172.21.15.27:0/77574373 shutdown_connections 2024-05-10T11:19:15.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:15.948+0000 7f57e2c00700 1 -- 172.21.15.27:0/77574373 wait complete. 2024-05-10T11:19:16.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:16 smithi027 ceph-mon[30054]: pgmap v184: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 251 op/s 2024-05-10T11:19:16.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:16 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:19:16.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:16 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:16.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:16 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:19:16.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:16 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:16.680 INFO:tasks.workunit.client.0.smithi027.stdout: 1 334022 7.42 MB/sec execute 121 sec latency 3515.986 ms 2024-05-10T11:19:16.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:16 smithi189 ceph-mon[34842]: pgmap v184: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 251 op/s 2024-05-10T11:19:16.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:16 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:19:16.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:16 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:16.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:16 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:19:16.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:16 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:16.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:16 smithi076 ceph-mon[34152]: pgmap v184: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 251 op/s 2024-05-10T11:19:16.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:16 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:19:16.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:16 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:16.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:16 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:19:16.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:16 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:16.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:19:17.396 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:19:17.680 INFO:tasks.workunit.client.0.smithi027.stdout: 1 334022 7.36 MB/sec execute 122 sec latency 4516.090 ms 2024-05-10T11:19:17.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:17 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:19:17.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:17 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:17.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:17 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:19:17.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:17 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:19:17.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:17 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:17.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:17 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:19:17.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:17 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:17.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:17 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:19:17.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:17 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:19:17.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:17 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:17.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:17 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:19:17.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:17 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:17.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:17 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:19:17.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:17 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:19:17.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:17 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:18.680 INFO:tasks.workunit.client.0.smithi027.stdout: 1 334022 7.30 MB/sec execute 123 sec latency 5516.223 ms 2024-05-10T11:19:18.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:18 smithi189 ceph-mon[34842]: pgmap v185: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 148 op/s 2024-05-10T11:19:18.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:18 smithi027 ceph-mon[30054]: pgmap v185: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 148 op/s 2024-05-10T11:19:18.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:18 smithi076 ceph-mon[34152]: pgmap v185: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 148 op/s 2024-05-10T11:19:19.680 INFO:tasks.workunit.client.0.smithi027.stdout: 1 334022 7.25 MB/sec execute 124 sec latency 6516.383 ms 2024-05-10T11:19:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.877+0000 7f8b8f600700 1 -- 172.21.15.27:0/3414879129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8806bbb0 msgr2=0x7f8b8806bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.877+0000 7f8b8f600700 1 --2- 172.21.15.27:0/3414879129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8806bbb0 0x7f8b8806bf90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009990 tx=0x7f8b7c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.877+0000 7f8b8f600700 1 -- 172.21.15.27:0/3414879129 shutdown_connections 2024-05-10T11:19:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.877+0000 7f8b8f600700 1 --2- 172.21.15.27:0/3414879129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b8810ffb0 0x7f8b881123a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.877+0000 7f8b8f600700 1 --2- 172.21.15.27:0/3414879129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8806bbb0 0x7f8b8806bf90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.877+0000 7f8b8f600700 1 --2- 172.21.15.27:0/3414879129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b8806c4d0 0x7f8b8810fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.877+0000 7f8b8f600700 1 -- 172.21.15.27:0/3414879129 >> 172.21.15.27:0/3414879129 conn(0x7f8b88078440 msgr2=0x7f8b88078850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:19.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.877+0000 7f8b8f600700 1 -- 172.21.15.27:0/3414879129 shutdown_connections 2024-05-10T11:19:19.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.878+0000 7f8b8f600700 1 -- 172.21.15.27:0/3414879129 wait complete. 2024-05-10T11:19:19.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.879+0000 7f8b8f600700 1 Processor -- start 2024-05-10T11:19:19.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.879+0000 7f8b8f600700 1 -- start start 2024-05-10T11:19:19.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.879+0000 7f8b8f600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b8806bbb0 0x7f8b8819be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.879+0000 7f8b8f600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b8806c4d0 0x7f8b8819c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.879+0000 7f8b8f600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8810ffb0 0x7f8b881a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.879+0000 7f8b8f600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8b88115910 con 0x7f8b8810ffb0 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.879+0000 7f8b8f600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8b88115790 con 0x7f8b8806c4d0 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.879+0000 7f8b8f600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8b88115a90 con 0x7f8b8806bbb0 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.879+0000 7f8b8d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8810ffb0 0x7f8b881a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.879+0000 7f8b87e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b8806c4d0 0x7f8b8819c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.879+0000 7f8b8d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8810ffb0 0x7f8b881a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36034/0 (socket says 172.21.15.27:36034) 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.879+0000 7f8b8d600700 1 -- 172.21.15.27:0/2707113149 learned_addr learned my addr 172.21.15.27:0/2707113149 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.880+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b8806bbb0 0x7f8b8819be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.880+0000 7f8b8d600700 1 -- 172.21.15.27:0/2707113149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b8806bbb0 msgr2=0x7f8b8819be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.880+0000 7f8b8d600700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b8806bbb0 0x7f8b8819be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.880+0000 7f8b8d600700 1 -- 172.21.15.27:0/2707113149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b8806c4d0 msgr2=0x7f8b8819c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.880+0000 7f8b8d600700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b8806c4d0 0x7f8b8819c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.880+0000 7f8b8d600700 1 -- 172.21.15.27:0/2707113149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b881a0e10 con 0x7f8b8810ffb0 2024-05-10T11:19:19.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.880+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b8806bbb0 0x7f8b8819be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:19:19.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.880+0000 7f8b87e00700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b8806c4d0 0x7f8b8819c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:19:19.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.880+0000 7f8b8d600700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8810ffb0 0x7f8b881a06d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000ef90 tx=0x7f8b8000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:19.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.881+0000 7f8b85600700 1 -- 172.21.15.27:0/2707113149 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b800092e0 con 0x7f8b8810ffb0 2024-05-10T11:19:19.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.881+0000 7f8b8f600700 1 -- 172.21.15.27:0/2707113149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b881a10a0 con 0x7f8b8810ffb0 2024-05-10T11:19:19.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.881+0000 7f8b85600700 1 -- 172.21.15.27:0/2707113149 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b80004bb0 con 0x7f8b8810ffb0 2024-05-10T11:19:19.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.881+0000 7f8b85600700 1 -- 172.21.15.27:0/2707113149 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b80004280 con 0x7f8b8810ffb0 2024-05-10T11:19:19.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.881+0000 7f8b8f600700 1 -- 172.21.15.27:0/2707113149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b881a8980 con 0x7f8b8810ffb0 2024-05-10T11:19:19.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.885+0000 7f8b85600700 1 -- 172.21.15.27:0/2707113149 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f8b8001b030 con 0x7f8b8810ffb0 2024-05-10T11:19:19.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.886+0000 7f8b99efcb80 1 Processor -- start 2024-05-10T11:19:19.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.886+0000 7f8b99efcb80 1 -- start start 2024-05-10T11:19:19.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.886+0000 7f8b99efcb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d972664440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.886+0000 7f8b99efcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664980 0x55d972668490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.886+0000 7f8b99efcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9726689d0 0x55d972668e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.886+0000 7f8b99efcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d9725b6a20 con 0x55d972664980 2024-05-10T11:19:19.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.886+0000 7f8b99efcb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d972554760 con 0x55d972664060 2024-05-10T11:19:19.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.886+0000 7f8b99efcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d9725b29e0 con 0x55d9726689d0 2024-05-10T11:19:19.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.887+0000 7f8b85600700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8b700b2a70 0x7f8b700b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.887+0000 7f8b8cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664980 0x55d972668490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.887+0000 7f8b85600700 1 -- 172.21.15.27:0/2707113149 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f8b800e24e0 con 0x7f8b8810ffb0 2024-05-10T11:19:19.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.887+0000 7f8b8cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664980 0x55d972668490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36050/0 (socket says 172.21.15.27:36050) 2024-05-10T11:19:19.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.887+0000 7f8b8cc00700 1 -- 172.21.15.27:0/3871757914 learned_addr learned my addr 172.21.15.27:0/3871757914 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:19.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.887+0000 7f8b87e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d972664440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.887+0000 7f8b8d600700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8b700b2a70 0x7f8b700b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.887+0000 7f8b87e00700 1 --2- 172.21.15.27:0/3871757914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9726689d0 0x55d972668e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.888+0000 7f8b8cc00700 1 -- 172.21.15.27:0/3871757914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9726689d0 msgr2=0x55d972668e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.888+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/3871757914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9726689d0 0x55d972668e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.888+0000 7f8b8cc00700 1 -- 172.21.15.27:0/3871757914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 msgr2=0x55d972664440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.888+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/3871757914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d972664440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.888+0000 7f8b8cc00700 1 -- 172.21.15.27:0/3871757914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9725038d0 con 0x55d972664980 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.888+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/3871757914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664980 0x55d972668490 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c005ca0 tx=0x7f8b7c0056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b196ec2d93c398 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.888+0000 7f8b8d600700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8b700b2a70 0x7f8b700b4f30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000c7a0 tx=0x7f8b80009480 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.888+0000 7f8b7a000700 1 -- 172.21.15.27:0/3871757914 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c00bc50 con 0x55d972664980 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.889+0000 7f8b7a000700 1 -- 172.21.15.27:0/3871757914 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b7c00bdf0 con 0x55d972664980 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.889+0000 7f8b7a000700 1 -- 172.21.15.27:0/3871757914 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c003cf0 con 0x55d972664980 2024-05-10T11:19:19.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.889+0000 7f8b99efcb80 1 -- 172.21.15.27:0/3871757914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664980 msgr2=0x55d972668490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:19.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.889+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/3871757914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664980 0x55d972668490 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c005ca0 tx=0x7f8b7c0056a0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.889+0000 7f8b99efcb80 1 -- 172.21.15.27:0/3871757914 shutdown_connections 2024-05-10T11:19:19.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.889+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/3871757914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9726689d0 0x55d972668e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.889+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/3871757914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d972664440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.889+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/3871757914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664980 0x55d972668490 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.889+0000 7f8b99efcb80 1 -- 172.21.15.27:0/3871757914 >> 172.21.15.27:0/3871757914 conn(0x55d972592b60 msgr2=0x55d97265e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.890+0000 7f8b99efcb80 1 -- 172.21.15.27:0/3871757914 shutdown_connections 2024-05-10T11:19:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.890+0000 7f8b99efcb80 1 -- 172.21.15.27:0/3871757914 wait complete. 2024-05-10T11:19:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.890+0000 7f8b99efcb80 1 -- start start 2024-05-10T11:19:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.890+0000 7f8b99efcb80 1 Processor -- start 2024-05-10T11:19:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.891+0000 7f8b99efcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664060 0x55d97266fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.891+0000 7f8b99efcb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664980 0x55d97266d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.891+0000 7f8b99efcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9726689d0 0x55d97266d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.891+0000 7f8b99efcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d972670b20 con 0x55d972664060 2024-05-10T11:19:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.891+0000 7f8b99efcb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d97251cb20 con 0x55d972664980 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.891+0000 7f8b99efcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d972670ca0 con 0x55d9726689d0 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.891+0000 7f8b99efcb80 10 client.0 _ll_register_callbacks cb 0x55d97255dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.891+0000 7f8b87e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664060 0x55d97266fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.891+0000 7f8b87e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664060 0x55d97266fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36056/0 (socket says 172.21.15.27:36056) 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.891+0000 7f8b87e00700 1 -- 172.21.15.27:0/1578529156 learned_addr learned my addr 172.21.15.27:0/1578529156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.892+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664980 0x55d97266d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.892+0000 7f8b87e00700 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9726689d0 0x55d97266d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.892+0000 7f8b87e00700 1 -- 172.21.15.27:0/1578529156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9726689d0 msgr2=0x55d97266d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.892+0000 7f8b87e00700 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9726689d0 0x55d97266d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.892+0000 7f8b87e00700 1 -- 172.21.15.27:0/1578529156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664980 msgr2=0x55d97266d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.892+0000 7f8b87e00700 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664980 0x55d97266d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.892+0000 7f8b87e00700 1 -- 172.21.15.27:0/1578529156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d97257c420 con 0x55d972664060 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.892+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664980 0x55d97266d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.892+0000 7f8b87e00700 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664060 0x55d97266fe90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8b74012c10 tx=0x7f8b740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2197def16cfc6a6b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:19.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.892+0000 7f8b6ea00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.893+0000 7f8b6ea00700 1 -- 172.21.15.27:0/1578529156 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74003d30 con 0x55d972664060 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.893+0000 7f8b6ea00700 1 -- 172.21.15.27:0/1578529156 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f8b74004830 con 0x55d972664060 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.893+0000 7f8b99efcb80 1 -- 172.21.15.27:0/1578529156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b7c00bdf0 con 0x55d972664060 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.893+0000 7f8b6ea00700 1 -- 172.21.15.27:0/1578529156 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8b7402c050 con 0x55d972664060 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.893+0000 7f8b99efcb80 10 client.16047 fetch_fsmap learned FSMap version 33 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.893+0000 7f8b99efcb80 1 -- 172.21.15.27:0/1578529156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d97267b3d0 con 0x55d972664060 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.893+0000 7f8b6ea00700 1 -- 172.21.15.27:0/1578529156 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b7c00bdf0 con 0x55d972664060 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.894+0000 7f8b6ea00700 1 -- 172.21.15.27:0/1578529156 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f8b7400e450 con 0x55d972664060 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.894+0000 7f8b99efcb80 10 client.16047 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.894+0000 7f8b99efcb80 10 client.16047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.894+0000 7f8b99efcb80 20 client.16047 populate_metadata read hostname 'smithi027' 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.894+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d97267da10 0x55d97267fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.894+0000 7f8b99efcb80 4 client.16047 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:19:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.894+0000 7f8b99efcb80 1 -- 172.21.15.27:0/1578529156 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9726310a0 con 0x55d97267da10 2024-05-10T11:19:19.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.894+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d97267da10 0x55d97267fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.895+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d97267da10 0x55d97267fe00 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=610374fccfdeb23e server_cookie=7012ae54f49d9ec6 in_seq=0 out_seq=0 2024-05-10T11:19:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.895+0000 7f8b6ea00700 10 client.16047 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b6ea00700 1 -- 172.21.15.27:0/1578529156 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d9726310a0 con 0x55d97267da10 2024-05-10T11:19:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b6ea00700 10 client.16047 handle_command_reply: tid=0 2024-05-10T11:19:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b99efcb80 1 client.16047 shutdown 2024-05-10T11:19:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b99efcb80 1 -- 172.21.15.27:0/1578529156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664060 msgr2=0x55d97266fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664060 0x55d97266fe90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8b74012c10 tx=0x7f8b740047c0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b99efcb80 1 -- 172.21.15.27:0/1578529156 shutdown_connections 2024-05-10T11:19:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d97267da10 0x55d97267fe00 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9726689d0 0x55d97266d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972664060 0x55d97266fe90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/1578529156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664980 0x55d97266d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b99efcb80 1 -- 172.21.15.27:0/1578529156 >> 172.21.15.27:0/1578529156 conn(0x55d972592b60 msgr2=0x55d9726592b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b6ea00700 0 client.16047 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b99efcb80 1 -- 172.21.15.27:0/1578529156 shutdown_connections 2024-05-10T11:19:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b99efcb80 1 -- 172.21.15.27:0/1578529156 wait complete. 2024-05-10T11:19:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.896+0000 7f8b99efcb80 20 client.16047 trim_cache size 0 max 16384 2024-05-10T11:19:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.919+0000 7f8b99efcb80 1 Processor -- start 2024-05-10T11:19:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.919+0000 7f8b99efcb80 1 -- start start 2024-05-10T11:19:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.919+0000 7f8b99efcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 0x55d972679bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.919+0000 7f8b99efcb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d97266ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.919+0000 7f8b99efcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d972664980 0x55d97266c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.919+0000 7f8b99efcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8b881b18a0 con 0x55d972592b60 2024-05-10T11:19:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.919+0000 7f8b99efcb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8b881b33e0 con 0x55d972664060 2024-05-10T11:19:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.919+0000 7f8b99efcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8b881b1f70 con 0x55d972664980 2024-05-10T11:19:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.920+0000 7f8b87e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 0x55d972679bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.920+0000 7f8b87e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 0x55d972679bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36064/0 (socket says 172.21.15.27:36064) 2024-05-10T11:19:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.920+0000 7f8b87e00700 1 -- 172.21.15.27:0/2629371708 learned_addr learned my addr 172.21.15.27:0/2629371708 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.920+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/2629371708 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d97266ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.920+0000 7f8b87e00700 1 --2- 172.21.15.27:0/2629371708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d972664980 0x55d97266c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.920+0000 7f8b87e00700 1 -- 172.21.15.27:0/2629371708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d972664980 msgr2=0x55d97266c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.920+0000 7f8b87e00700 1 --2- 172.21.15.27:0/2629371708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d972664980 0x55d97266c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.920+0000 7f8b87e00700 1 -- 172.21.15.27:0/2629371708 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 msgr2=0x55d97266ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.920+0000 7f8b87e00700 1 --2- 172.21.15.27:0/2629371708 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d97266ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.920+0000 7f8b87e00700 1 -- 172.21.15.27:0/2629371708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9726687b0 con 0x55d972592b60 2024-05-10T11:19:19.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.921+0000 7f8b87e00700 1 --2- 172.21.15.27:0/2629371708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 0x55d972679bb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8b74001eb0 tx=0x7f8b74009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b87465a0e1cbeae server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.921+0000 7f8b79600700 1 -- 172.21.15.27:0/2629371708 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74003df0 con 0x55d972592b60 2024-05-10T11:19:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.921+0000 7f8b79600700 1 -- 172.21.15.27:0/2629371708 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b7400e3c0 con 0x55d972592b60 2024-05-10T11:19:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.921+0000 7f8b79600700 1 -- 172.21.15.27:0/2629371708 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74004a90 con 0x55d972592b60 2024-05-10T11:19:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.922+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2629371708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 msgr2=0x55d972679bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.922+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/2629371708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 0x55d972679bb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8b74001eb0 tx=0x7f8b74009260 comp rx=0 tx=0).stop 2024-05-10T11:19:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.922+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2629371708 shutdown_connections 2024-05-10T11:19:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.922+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/2629371708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d972664980 0x55d97266c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.922+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/2629371708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 0x55d972679bb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.922+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/2629371708 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d97266ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.922+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2629371708 >> 172.21.15.27:0/2629371708 conn(0x55d972658610 msgr2=0x55d97266bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.922+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2629371708 shutdown_connections 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.922+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2629371708 wait complete. 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.922+0000 7f8b99efcb80 1 -- start start 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.923+0000 7f8b99efcb80 1 Processor -- start 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.923+0000 7f8b99efcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 0x55d972676450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.923+0000 7f8b99efcb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d972684be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.923+0000 7f8b99efcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d972664980 0x55d972685120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.923+0000 7f8b99efcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8b881b2d10 con 0x55d972592b60 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.923+0000 7f8b99efcb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8b881b2640 con 0x55d972664060 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.923+0000 7f8b99efcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8b881af840 con 0x55d972664980 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.923+0000 7f8b99efcb80 10 client.0 _ll_register_callbacks cb 0x55d97255dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.924+0000 7f8b87e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 0x55d972676450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.924+0000 7f8b87e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 0x55d972676450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36070/0 (socket says 172.21.15.27:36070) 2024-05-10T11:19:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.924+0000 7f8b87e00700 1 -- 172.21.15.27:0/2276789851 learned_addr learned my addr 172.21.15.27:0/2276789851 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.924+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d972684be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.924+0000 7f8b87e00700 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d972664980 0x55d972685120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.924+0000 7f8b8cc00700 1 -- 172.21.15.27:0/2276789851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d972664980 msgr2=0x55d972685120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.924+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d972664980 0x55d972685120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.924+0000 7f8b8cc00700 1 -- 172.21.15.27:0/2276789851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 msgr2=0x55d972676450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.924+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 0x55d972676450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.924+0000 7f8b8cc00700 1 -- 172.21.15.27:0/2276789851 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d972685940 con 0x55d972664060 2024-05-10T11:19:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.924+0000 7f8b87e00700 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 0x55d972676450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:19:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.925+0000 7f8b8cc00700 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d972684be0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009ce0 tx=0x7f8b7c00bad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69a3526e1c65d266 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.925+0000 7f8b6ea00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:19:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.925+0000 7f8b6ea00700 1 -- 172.21.15.27:0/2276789851 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c0050d0 con 0x55d972664060 2024-05-10T11:19:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.925+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2276789851 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b7400e3c0 con 0x55d972664060 2024-05-10T11:19:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.925+0000 7f8b6ea00700 1 -- 172.21.15.27:0/2276789851 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7f8b7c005270 con 0x55d972664060 2024-05-10T11:19:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.925+0000 7f8b6ea00700 1 -- 172.21.15.27:0/2276789851 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8b7c041050 con 0x55d972664060 2024-05-10T11:19:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.925+0000 7f8b99efcb80 10 client.26258 fetch_fsmap learned FSMap version 33 2024-05-10T11:19:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.925+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2276789851 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d9726e8260 con 0x55d972664060 2024-05-10T11:19:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.926+0000 7f8b6ea00700 1 -- 172.21.15.27:0/2276789851 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b7400e3c0 con 0x55d972664060 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.926+0000 7f8b6ea00700 1 -- 172.21.15.27:0/2276789851 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f8b7c0322c0 con 0x55d972664060 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.926+0000 7f8b99efcb80 10 client.26258 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.926+0000 7f8b99efcb80 10 client.26258 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.926+0000 7f8b99efcb80 20 client.26258 populate_metadata read hostname 'smithi027' 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.926+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d9726689d0 0x55d9726ea440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.926+0000 7f8b99efcb80 4 client.26258 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.926+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2276789851 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d9725c4400 con 0x55d9726689d0 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.927+0000 7f8b87e00700 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d9726689d0 0x55d9726ea440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.927+0000 7f8b87e00700 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d9726689d0 0x55d9726ea440 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab0d286fad817b60 server_cookie=1233e6f95232c4f2 in_seq=0 out_seq=0 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.927+0000 7f8b6ea00700 10 client.26258 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.927+0000 7f8b6ea00700 1 -- 172.21.15.27:0/2276789851 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55d9725c4400 con 0x55d9726689d0 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.927+0000 7f8b6ea00700 10 client.26258 handle_command_reply: tid=0 2024-05-10T11:19:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.927+0000 7f8b99efcb80 1 client.26258 shutdown 2024-05-10T11:19:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.927+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2276789851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 msgr2=0x55d972684be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.927+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d972684be0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009ce0 tx=0x7f8b7c00bad0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.928+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2276789851 shutdown_connections 2024-05-10T11:19:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.928+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d9726689d0 0x55d9726ea440 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.928+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d972664980 0x55d972685120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.928+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d972592b60 0x55d972676450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.928+0000 7f8b6ea00700 0 client.26258 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.928+0000 7f8b99efcb80 1 --2- 172.21.15.27:0/2276789851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d972664060 0x55d972684be0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.928+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2276789851 >> 172.21.15.27:0/2276789851 conn(0x55d972658610 msgr2=0x55d97266c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.928+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2276789851 shutdown_connections 2024-05-10T11:19:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.928+0000 7f8b99efcb80 1 -- 172.21.15.27:0/2276789851 wait complete. 2024-05-10T11:19:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.928+0000 7f8b99efcb80 20 client.26258 trim_cache size 0 max 16384 2024-05-10T11:19:19.932 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:19:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (1 inodes in the stack)", 2024-05-10T11:19:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:19:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "6e8c66d6-0fd0-47f2-9f93-08ab8547b646": { 2024-05-10T11:19:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:19:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "6e8c66d6-0fd0-47f2-9f93-08ab8547b646", 2024-05-10T11:19:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:19:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:19:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:19:19.932 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:19:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.929+0000 7f8b8f600700 1 -- 172.21.15.27:0/2707113149 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8b700b2a70 msgr2=0x7f8b700b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.929+0000 7f8b8f600700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8b700b2a70 0x7f8b700b4f30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000c7a0 tx=0x7f8b80009480 comp rx=0 tx=0).stop 2024-05-10T11:19:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.929+0000 7f8b8f600700 1 -- 172.21.15.27:0/2707113149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8810ffb0 msgr2=0x7f8b881a06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.929+0000 7f8b8f600700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8810ffb0 0x7f8b881a06d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000ef90 tx=0x7f8b8000c3a0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.929+0000 7f8b8f600700 1 -- 172.21.15.27:0/2707113149 shutdown_connections 2024-05-10T11:19:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.929+0000 7f8b8f600700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8b700b2a70 0x7f8b700b4f30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.929+0000 7f8b8f600700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b8806bbb0 0x7f8b8819be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.929+0000 7f8b8f600700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8810ffb0 0x7f8b881a06d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.929+0000 7f8b8f600700 1 --2- 172.21.15.27:0/2707113149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b8806c4d0 0x7f8b8819c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:19.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.929+0000 7f8b8f600700 1 -- 172.21.15.27:0/2707113149 >> 172.21.15.27:0/2707113149 conn(0x7f8b88078440 msgr2=0x7f8b88111520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:19.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.929+0000 7f8b8f600700 1 -- 172.21.15.27:0/2707113149 shutdown_connections 2024-05-10T11:19:19.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:19.929+0000 7f8b8f600700 1 -- 172.21.15.27:0/2707113149 wait complete. 2024-05-10T11:19:20.681 INFO:tasks.workunit.client.0.smithi027.stdout: 1 334022 7.19 MB/sec execute 125 sec latency 7516.946 ms 2024-05-10T11:19:20.727 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:20 smithi076 ceph-mon[34152]: pgmap v186: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 173 op/s 2024-05-10T11:19:20.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:20 smithi189 ceph-mon[34842]: pgmap v186: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 173 op/s 2024-05-10T11:19:20.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:20 smithi027 ceph-mon[30054]: pgmap v186: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 173 op/s 2024-05-10T11:19:21.225 INFO:tasks.cephfs.filesystem:scrub status for tag:6e8c66d6-0fd0-47f2-9f93-08ab8547b646 - {'path': '/', 'tag': '6e8c66d6-0fd0-47f2-9f93-08ab8547b646', 'options': 'recursive,force'} 2024-05-10T11:19:21.681 INFO:tasks.workunit.client.0.smithi027.stdout: 1 334022 7.13 MB/sec execute 126 sec latency 8517.041 ms 2024-05-10T11:19:22.681 INFO:tasks.workunit.client.0.smithi027.stdout: 1 335132 7.12 MB/sec execute 127 sec latency 8540.702 ms 2024-05-10T11:19:22.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:22 smithi189 ceph-mon[34842]: pgmap v187: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.4 MiB/s wr, 89 op/s 2024-05-10T11:19:22.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:22 smithi027 ceph-mon[30054]: pgmap v187: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.4 MiB/s wr, 89 op/s 2024-05-10T11:19:22.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:22 smithi076 ceph-mon[34152]: pgmap v187: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.4 MiB/s wr, 89 op/s 2024-05-10T11:19:23.682 INFO:tasks.workunit.client.0.smithi027.stdout: 1 336790 7.13 MB/sec execute 128 sec latency 23.322 ms 2024-05-10T11:19:23.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:23 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:23.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:23 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:23.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:23 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:19:24.681 INFO:tasks.workunit.client.0.smithi027.stdout: 1 337407 7.11 MB/sec execute 129 sec latency 22.857 ms 2024-05-10T11:19:24.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:24 smithi189 ceph-mon[34842]: pgmap v188: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.4 MiB/s wr, 89 op/s 2024-05-10T11:19:24.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:24 smithi027 ceph-mon[30054]: pgmap v188: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.4 MiB/s wr, 89 op/s 2024-05-10T11:19:24.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:24 smithi076 ceph-mon[34152]: pgmap v188: 129 pgs: 129 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.4 MiB/s wr, 89 op/s 2024-05-10T11:19:25.681 INFO:tasks.workunit.client.0.smithi027.stdout: 1 338396 7.10 MB/sec execute 130 sec latency 43.945 ms 2024-05-10T11:19:26.682 INFO:tasks.workunit.client.0.smithi027.stdout: 1 339743 7.09 MB/sec execute 131 sec latency 31.427 ms 2024-05-10T11:19:26.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:26 smithi189 ceph-mon[34842]: pgmap v189: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 2.2 MiB/s wr, 138 op/s 2024-05-10T11:19:26.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:26 smithi027 ceph-mon[30054]: pgmap v189: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 2.2 MiB/s wr, 138 op/s 2024-05-10T11:19:26.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:26 smithi076 ceph-mon[34152]: pgmap v189: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 2.2 MiB/s wr, 138 op/s 2024-05-10T11:19:27.682 INFO:tasks.workunit.client.0.smithi027.stdout: 1 340870 7.10 MB/sec execute 132 sec latency 76.141 ms 2024-05-10T11:19:27.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:27 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:19:27.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:27 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:19:27.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:27 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:19:27.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:27 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:19:27.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:27 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:19:27.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:27 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:19:28.682 INFO:tasks.workunit.client.0.smithi027.stdout: 1 341770 7.06 MB/sec execute 133 sec latency 35.171 ms 2024-05-10T11:19:28.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:28 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:19:28.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:28 smithi027 ceph-mon[30054]: pgmap v190: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 1.3 MiB/s wr, 73 op/s 2024-05-10T11:19:28.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:28 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:19:28.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:28 smithi076 ceph-mon[34152]: pgmap v190: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 1.3 MiB/s wr, 73 op/s 2024-05-10T11:19:28.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:28 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:19:28.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:28 smithi189 ceph-mon[34842]: pgmap v190: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 1.3 MiB/s wr, 73 op/s 2024-05-10T11:19:29.683 INFO:tasks.workunit.client.0.smithi027.stdout: 1 343127 7.05 MB/sec execute 134 sec latency 23.578 ms 2024-05-10T11:19:30.683 INFO:tasks.workunit.client.0.smithi027.stdout: 1 344379 7.08 MB/sec execute 135 sec latency 31.984 ms 2024-05-10T11:19:30.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:30 smithi027 ceph-mon[30054]: pgmap v191: 129 pgs: 129 active+clean; 202 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 2.3 MiB/s wr, 139 op/s 2024-05-10T11:19:30.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:30 smithi076 ceph-mon[34152]: pgmap v191: 129 pgs: 129 active+clean; 202 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 2.3 MiB/s wr, 139 op/s 2024-05-10T11:19:30.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:30 smithi189 ceph-mon[34842]: pgmap v191: 129 pgs: 129 active+clean; 202 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 2.3 MiB/s wr, 139 op/s 2024-05-10T11:19:31.683 INFO:tasks.workunit.client.0.smithi027.stdout: 1 345245 7.04 MB/sec execute 136 sec latency 37.376 ms 2024-05-10T11:19:32.683 INFO:tasks.workunit.client.0.smithi027.stdout: 1 346484 7.03 MB/sec execute 137 sec latency 95.538 ms 2024-05-10T11:19:32.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:32 smithi027 ceph-mon[30054]: pgmap v192: 129 pgs: 129 active+clean; 202 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.7 MiB/s wr, 114 op/s 2024-05-10T11:19:32.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:32 smithi076 ceph-mon[34152]: pgmap v192: 129 pgs: 129 active+clean; 202 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.7 MiB/s wr, 114 op/s 2024-05-10T11:19:32.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:32 smithi189 ceph-mon[34842]: pgmap v192: 129 pgs: 129 active+clean; 202 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.7 MiB/s wr, 114 op/s 2024-05-10T11:19:33.683 INFO:tasks.workunit.client.0.smithi027.stdout: 1 347972 7.06 MB/sec execute 138 sec latency 18.672 ms 2024-05-10T11:19:34.683 INFO:tasks.workunit.client.0.smithi027.stdout: 1 349045 7.05 MB/sec execute 139 sec latency 35.938 ms 2024-05-10T11:19:34.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:34 smithi027 ceph-mon[30054]: pgmap v193: 129 pgs: 129 active+clean; 202 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.7 MiB/s wr, 114 op/s 2024-05-10T11:19:34.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:34 smithi076 ceph-mon[34152]: pgmap v193: 129 pgs: 129 active+clean; 202 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.7 MiB/s wr, 114 op/s 2024-05-10T11:19:34.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:34 smithi189 ceph-mon[34842]: pgmap v193: 129 pgs: 129 active+clean; 202 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.7 MiB/s wr, 114 op/s 2024-05-10T11:19:35.683 INFO:tasks.workunit.client.0.smithi027.stdout: 1 350819 7.06 MB/sec execute 140 sec latency 20.612 ms 2024-05-10T11:19:36.683 INFO:tasks.workunit.client.0.smithi027.stdout: 1 351632 7.05 MB/sec execute 141 sec latency 22.047 ms 2024-05-10T11:19:36.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:36 smithi027 ceph-mon[30054]: pgmap v194: 129 pgs: 129 active+clean; 205 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 182 op/s 2024-05-10T11:19:36.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:36 smithi076 ceph-mon[34152]: pgmap v194: 129 pgs: 129 active+clean; 205 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 182 op/s 2024-05-10T11:19:36.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:36 smithi189 ceph-mon[34842]: pgmap v194: 129 pgs: 129 active+clean; 205 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 182 op/s 2024-05-10T11:19:37.684 INFO:tasks.workunit.client.0.smithi027.stdout: 1 352717 7.04 MB/sec execute 142 sec latency 97.368 ms 2024-05-10T11:19:38.684 INFO:tasks.workunit.client.0.smithi027.stdout: 1 354331 7.04 MB/sec execute 143 sec latency 40.175 ms 2024-05-10T11:19:38.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:38 smithi027 ceph-mon[30054]: pgmap v195: 129 pgs: 129 active+clean; 205 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 134 op/s 2024-05-10T11:19:38.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:38 smithi076 ceph-mon[34152]: pgmap v195: 129 pgs: 129 active+clean; 205 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 134 op/s 2024-05-10T11:19:38.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:38 smithi189 ceph-mon[34842]: pgmap v195: 129 pgs: 129 active+clean; 205 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 134 op/s 2024-05-10T11:19:39.684 INFO:tasks.workunit.client.0.smithi027.stdout: 1 355118 7.04 MB/sec execute 144 sec latency 17.777 ms 2024-05-10T11:19:40.684 INFO:tasks.workunit.client.0.smithi027.stdout: 1 356153 7.02 MB/sec execute 145 sec latency 30.285 ms 2024-05-10T11:19:40.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:40 smithi027 ceph-mon[30054]: pgmap v196: 129 pgs: 129 active+clean; 210 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 205 op/s 2024-05-10T11:19:40.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:40 smithi076 ceph-mon[34152]: pgmap v196: 129 pgs: 129 active+clean; 210 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 205 op/s 2024-05-10T11:19:40.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:40 smithi189 ceph-mon[34842]: pgmap v196: 129 pgs: 129 active+clean; 210 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 205 op/s 2024-05-10T11:19:41.684 INFO:tasks.workunit.client.0.smithi027.stdout: 1 357691 7.03 MB/sec execute 146 sec latency 25.591 ms 2024-05-10T11:19:41.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:41 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:19:41.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:41 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:19:41.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:41 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:19:42.684 INFO:tasks.workunit.client.0.smithi027.stdout: 1 358652 7.02 MB/sec execute 147 sec latency 44.744 ms 2024-05-10T11:19:42.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:42 smithi027 ceph-mon[30054]: pgmap v197: 129 pgs: 129 active+clean; 210 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 139 op/s 2024-05-10T11:19:42.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:42 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:19:42.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:42 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:19:42.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:42 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:19:42.893 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:42 smithi189 ceph-mon[34842]: pgmap v197: 129 pgs: 129 active+clean; 210 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 139 op/s 2024-05-10T11:19:42.893 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:42 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:19:42.893 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:42 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:19:42.893 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:42 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:19:42.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:42 smithi076 ceph-mon[34152]: pgmap v197: 129 pgs: 129 active+clean; 210 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 139 op/s 2024-05-10T11:19:42.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:42 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:19:42.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:42 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:19:42.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:42 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:19:43.685 INFO:tasks.workunit.client.0.smithi027.stdout: 1 359239 6.98 MB/sec execute 148 sec latency 211.908 ms 2024-05-10T11:19:44.684 INFO:tasks.workunit.client.0.smithi027.stdout: 1 360679 6.98 MB/sec execute 149 sec latency 24.535 ms 2024-05-10T11:19:44.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:44 smithi027 ceph-mon[30054]: pgmap v198: 129 pgs: 129 active+clean; 210 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 139 op/s 2024-05-10T11:19:44.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:44 smithi076 ceph-mon[34152]: pgmap v198: 129 pgs: 129 active+clean; 210 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 139 op/s 2024-05-10T11:19:44.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:44 smithi189 ceph-mon[34842]: pgmap v198: 129 pgs: 129 active+clean; 210 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 139 op/s 2024-05-10T11:19:45.685 INFO:tasks.workunit.client.0.smithi027.stdout: 1 362071 7.01 MB/sec execute 150 sec latency 16.385 ms 2024-05-10T11:19:46.685 INFO:tasks.workunit.client.0.smithi027.stdout: 1 362899 6.98 MB/sec execute 151 sec latency 35.028 ms 2024-05-10T11:19:46.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:46 smithi027 ceph-mon[30054]: pgmap v199: 129 pgs: 129 active+clean; 213 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 209 op/s 2024-05-10T11:19:46.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:46 smithi076 ceph-mon[34152]: pgmap v199: 129 pgs: 129 active+clean; 213 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 209 op/s 2024-05-10T11:19:46.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:46 smithi189 ceph-mon[34842]: pgmap v199: 129 pgs: 129 active+clean; 213 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 209 op/s 2024-05-10T11:19:47.685 INFO:tasks.workunit.client.0.smithi027.stdout: 1 364233 6.96 MB/sec execute 152 sec latency 29.049 ms 2024-05-10T11:19:48.685 INFO:tasks.workunit.client.0.smithi027.stdout: 1 365657 6.99 MB/sec execute 153 sec latency 15.828 ms 2024-05-10T11:19:48.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:48 smithi027 ceph-mon[30054]: pgmap v200: 129 pgs: 129 active+clean; 213 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-05-10T11:19:48.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:48 smithi076 ceph-mon[34152]: pgmap v200: 129 pgs: 129 active+clean; 213 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-05-10T11:19:48.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:48 smithi189 ceph-mon[34842]: pgmap v200: 129 pgs: 129 active+clean; 213 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-05-10T11:19:49.685 INFO:tasks.workunit.client.0.smithi027.stdout: 1 366444 6.96 MB/sec execute 154 sec latency 31.785 ms 2024-05-10T11:19:50.686 INFO:tasks.workunit.client.0.smithi027.stdout: 1 367737 6.95 MB/sec execute 155 sec latency 26.479 ms 2024-05-10T11:19:50.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:50 smithi027 ceph-mon[30054]: pgmap v201: 129 pgs: 129 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 193 op/s 2024-05-10T11:19:50.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:50 smithi076 ceph-mon[34152]: pgmap v201: 129 pgs: 129 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 193 op/s 2024-05-10T11:19:50.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:50 smithi189 ceph-mon[34842]: pgmap v201: 129 pgs: 129 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 193 op/s 2024-05-10T11:19:51.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:19:51.685 INFO:tasks.workunit.client.0.smithi027.stdout: 1 369154 6.98 MB/sec execute 156 sec latency 17.168 ms 2024-05-10T11:19:51.732 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:19:52.686 INFO:tasks.workunit.client.0.smithi027.stdout: 1 369928 6.95 MB/sec execute 157 sec latency 33.680 ms 2024-05-10T11:19:52.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:52 smithi027 ceph-mon[30054]: pgmap v202: 129 pgs: 129 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 123 op/s 2024-05-10T11:19:52.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:52 smithi076 ceph-mon[34152]: pgmap v202: 129 pgs: 129 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 123 op/s 2024-05-10T11:19:52.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:52 smithi189 ceph-mon[34842]: pgmap v202: 129 pgs: 129 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 123 op/s 2024-05-10T11:19:53.686 INFO:tasks.workunit.client.0.smithi027.stdout: 1 371253 6.94 MB/sec execute 158 sec latency 39.840 ms 2024-05-10T11:19:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.493+0000 7ff556200700 1 -- 172.21.15.27:0/2919985238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff550108850 msgr2=0x7ff5501103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.493+0000 7ff556200700 1 --2- 172.21.15.27:0/2919985238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff550108850 0x7ff5501103d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff54800b0d0 tx=0x7ff54800b3e0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.493+0000 7ff556200700 1 -- 172.21.15.27:0/2919985238 shutdown_connections 2024-05-10T11:19:54.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.493+0000 7ff556200700 1 --2- 172.21.15.27:0/2919985238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550103750 0x7ff550103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.493+0000 7ff556200700 1 --2- 172.21.15.27:0/2919985238 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff550108850 0x7ff5501103d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.493+0000 7ff556200700 1 --2- 172.21.15.27:0/2919985238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff550104100 0x7ff550108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.493+0000 7ff556200700 1 -- 172.21.15.27:0/2919985238 >> 172.21.15.27:0/2919985238 conn(0x7ff55007a600 msgr2=0x7ff55007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:54.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.493+0000 7ff556200700 1 -- 172.21.15.27:0/2919985238 shutdown_connections 2024-05-10T11:19:54.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.493+0000 7ff556200700 1 -- 172.21.15.27:0/2919985238 wait complete. 2024-05-10T11:19:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.494+0000 7ff556200700 1 Processor -- start 2024-05-10T11:19:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.495+0000 7ff556200700 1 -- start start 2024-05-10T11:19:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.495+0000 7ff556200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff550103750 0x7ff55019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.495+0000 7ff556200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff550104100 0x7ff55019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.495+0000 7ff556200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550108850 0x7ff5501998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.495+0000 7ff556200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff550113670 con 0x7ff550104100 2024-05-10T11:19:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.495+0000 7ff556200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501134f0 con 0x7ff550103750 2024-05-10T11:19:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.495+0000 7ff556200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501137f0 con 0x7ff550108850 2024-05-10T11:19:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.495+0000 7ff54f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff550103750 0x7ff55019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.495+0000 7ff54f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff550103750 0x7ff55019f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36814/0 (socket says 172.21.15.27:36814) 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.495+0000 7ff54f400700 1 -- 172.21.15.27:0/2688804659 learned_addr learned my addr 172.21.15.27:0/2688804659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.496+0000 7ff54ea00700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff550104100 0x7ff55019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.496+0000 7ff54fe00700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550108850 0x7ff5501998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.496+0000 7ff54f400700 1 -- 172.21.15.27:0/2688804659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550108850 msgr2=0x7ff5501998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.496+0000 7ff54f400700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550108850 0x7ff5501998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.496+0000 7ff54f400700 1 -- 172.21.15.27:0/2688804659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff550104100 msgr2=0x7ff55019fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.496+0000 7ff54f400700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff550104100 0x7ff55019fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.496+0000 7ff54f400700 1 -- 172.21.15.27:0/2688804659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55019a050 con 0x7ff550103750 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.496+0000 7ff54fe00700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550108850 0x7ff5501998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.496+0000 7ff54f400700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff550103750 0x7ff55019f6e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff54000e9a0 tx=0x7ff54000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.496+0000 7ff547e00700 1 -- 172.21.15.27:0/2688804659 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54000cbd0 con 0x7ff550103750 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.497+0000 7ff547e00700 1 -- 172.21.15.27:0/2688804659 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff54000cd70 con 0x7ff550103750 2024-05-10T11:19:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.497+0000 7ff556200700 1 -- 172.21.15.27:0/2688804659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff55019a340 con 0x7ff550103750 2024-05-10T11:19:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.497+0000 7ff547e00700 1 -- 172.21.15.27:0/2688804659 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540018680 con 0x7ff550103750 2024-05-10T11:19:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.497+0000 7ff556200700 1 -- 172.21.15.27:0/2688804659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5501a65b0 con 0x7ff550103750 2024-05-10T11:19:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.500+0000 7ff547e00700 1 -- 172.21.15.27:0/2688804659 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff540010970 con 0x7ff550103750 2024-05-10T11:19:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.502+0000 7ff560ad1b80 1 Processor -- start 2024-05-10T11:19:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.502+0000 7ff547e00700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff5340b29f0 0x7ff5340b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.503+0000 7ff560ad1b80 1 -- start start 2024-05-10T11:19:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.503+0000 7ff547e00700 1 -- 172.21.15.27:0/2688804659 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7ff5400e2e10 con 0x7ff550103750 2024-05-10T11:19:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.503+0000 7ff54fe00700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff5340b29f0 0x7ff5340b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.503+0000 7ff560ad1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054060 0x55df93054440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.503+0000 7ff560ad1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df93054980 0x55df93058490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.503+0000 7ff560ad1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df930589d0 0x55df93058e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.503+0000 7ff560ad1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55df92fa6a20 con 0x55df93054980 2024-05-10T11:19:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.503+0000 7ff560ad1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55df92f44760 con 0x55df93054060 2024-05-10T11:19:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.503+0000 7ff560ad1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55df92fa29e0 con 0x55df930589d0 2024-05-10T11:19:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.504+0000 7ff54f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054060 0x55df93054440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.504+0000 7ff54f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054060 0x55df93054440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36820/0 (socket says 172.21.15.27:36820) 2024-05-10T11:19:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.504+0000 7ff54f400700 1 -- 172.21.15.27:0/1178821589 learned_addr learned my addr 172.21.15.27:0/1178821589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.504+0000 7ff54fe00700 1 --2- 172.21.15.27:0/1178821589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df930589d0 0x55df93058e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.504+0000 7ff54ea00700 1 --2- 172.21.15.27:0/1178821589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df93054980 0x55df93058490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.504+0000 7ff54fe00700 1 -- 172.21.15.27:0/1178821589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054060 msgr2=0x55df93054440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.504+0000 7ff54fe00700 1 --2- 172.21.15.27:0/1178821589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054060 0x55df93054440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.505+0000 7ff54fe00700 1 -- 172.21.15.27:0/1178821589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df93054980 msgr2=0x55df93058490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.505+0000 7ff54fe00700 1 --2- 172.21.15.27:0/1178821589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df93054980 0x55df93058490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.505+0000 7ff54fe00700 1 -- 172.21.15.27:0/1178821589 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df92ef38d0 con 0x55df930589d0 2024-05-10T11:19:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.505+0000 7ff54f400700 1 --2- 172.21.15.27:0/1178821589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054060 0x55df93054440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:19:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.505+0000 7ff54fe00700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff5340b29f0 0x7ff5340b4eb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff548009fd0 tx=0x7ff54800b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.505+0000 7ff54fe00700 1 --2- 172.21.15.27:0/1178821589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df930589d0 0x55df93058e30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff54800b770 tx=0x7ff548026000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bd1609d2ef6e437 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.506+0000 7ff544c00700 1 -- 172.21.15.27:0/1178821589 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548025030 con 0x55df930589d0 2024-05-10T11:19:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.506+0000 7ff544c00700 1 -- 172.21.15.27:0/1178821589 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff54802e070 con 0x55df930589d0 2024-05-10T11:19:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.506+0000 7ff544c00700 1 -- 172.21.15.27:0/1178821589 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548029a90 con 0x55df930589d0 2024-05-10T11:19:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.507+0000 7ff560ad1b80 1 -- 172.21.15.27:0/1178821589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df930589d0 msgr2=0x55df93058e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.507+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/1178821589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df930589d0 0x55df93058e30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff54800b770 tx=0x7ff548026000 comp rx=0 tx=0).stop 2024-05-10T11:19:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.507+0000 7ff560ad1b80 1 -- 172.21.15.27:0/1178821589 shutdown_connections 2024-05-10T11:19:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.507+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/1178821589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df930589d0 0x55df93058e30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.507+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/1178821589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054060 0x55df93054440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.507+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/1178821589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df93054980 0x55df93058490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.507+0000 7ff560ad1b80 1 -- 172.21.15.27:0/1178821589 >> 172.21.15.27:0/1178821589 conn(0x55df92f82b60 msgr2=0x55df9304e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.507+0000 7ff560ad1b80 1 -- 172.21.15.27:0/1178821589 shutdown_connections 2024-05-10T11:19:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.507+0000 7ff560ad1b80 1 -- 172.21.15.27:0/1178821589 wait complete. 2024-05-10T11:19:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.508+0000 7ff560ad1b80 1 -- start start 2024-05-10T11:19:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.508+0000 7ff560ad1b80 1 Processor -- start 2024-05-10T11:19:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.509+0000 7ff560ad1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054060 0x55df9305fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.509+0000 7ff560ad1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 0x55df9305d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.509+0000 7ff560ad1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df930589d0 0x55df9305d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.509+0000 7ff560ad1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55df93060b80 con 0x55df930589d0 2024-05-10T11:19:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.509+0000 7ff560ad1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55df92f0cb20 con 0x55df93054980 2024-05-10T11:19:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.509+0000 7ff560ad1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55df93060d00 con 0x55df93054060 2024-05-10T11:19:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.509+0000 7ff560ad1b80 10 client.0 _ll_register_callbacks cb 0x55df92f4dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:19:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.509+0000 7ff54ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 0x55df9305d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.509+0000 7ff54ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 0x55df9305d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36836/0 (socket says 172.21.15.27:36836) 2024-05-10T11:19:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.509+0000 7ff54ea00700 1 -- 172.21.15.27:0/2122425796 learned_addr learned my addr 172.21.15.27:0/2122425796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:54.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.509+0000 7ff54f400700 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054060 0x55df9305fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.509+0000 7ff54fe00700 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df930589d0 0x55df9305d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.510+0000 7ff54ea00700 1 -- 172.21.15.27:0/2122425796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054060 msgr2=0x55df9305fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.510+0000 7ff54ea00700 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054060 0x55df9305fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.510+0000 7ff54ea00700 1 -- 172.21.15.27:0/2122425796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df930589d0 msgr2=0x55df9305d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.510+0000 7ff54ea00700 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df930589d0 0x55df9305d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.510+0000 7ff54ea00700 1 -- 172.21.15.27:0/2122425796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df92f6c420 con 0x55df93054980 2024-05-10T11:19:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.510+0000 7ff54fe00700 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df930589d0 0x55df9305d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:19:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.510+0000 7ff54ea00700 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 0x55df9305d1b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff53c012740 tx=0x7ff53c012a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb382305b829b380 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.510+0000 7ff539600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:19:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.511+0000 7ff539600700 1 -- 172.21.15.27:0/2122425796 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c011770 con 0x55df93054980 2024-05-10T11:19:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.511+0000 7ff539600700 1 -- 172.21.15.27:0/2122425796 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7ff53c011910 con 0x55df93054980 2024-05-10T11:19:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.511+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2122425796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff54802e070 con 0x55df93054980 2024-05-10T11:19:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.511+0000 7ff539600700 1 -- 172.21.15.27:0/2122425796 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff53c011b10 con 0x55df93054980 2024-05-10T11:19:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.511+0000 7ff560ad1b80 10 client.26270 fetch_fsmap learned FSMap version 33 2024-05-10T11:19:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.511+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2122425796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55df9306f840 con 0x55df93054980 2024-05-10T11:19:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.511+0000 7ff539600700 1 -- 172.21.15.27:0/2122425796 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff54802e070 con 0x55df93054980 2024-05-10T11:19:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.511+0000 7ff539600700 1 -- 172.21.15.27:0/2122425796 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff53c01c420 con 0x55df93054980 2024-05-10T11:19:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.512+0000 7ff560ad1b80 10 client.26270 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:19:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.512+0000 7ff560ad1b80 10 client.26270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:19:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.512+0000 7ff560ad1b80 20 client.26270 populate_metadata read hostname 'smithi027' 2024-05-10T11:19:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.512+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df93071f20 0x55df93074310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.512+0000 7ff560ad1b80 4 client.26270 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:19:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.512+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2122425796 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df930210a0 con 0x55df93071f20 2024-05-10T11:19:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.513+0000 7ff54f400700 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df93071f20 0x55df93074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.513+0000 7ff54f400700 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df93071f20 0x55df93074310 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31b200fa73f52e2f server_cookie=b7a5cde8ee4e89eb in_seq=0 out_seq=0 2024-05-10T11:19:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.513+0000 7ff539600700 10 client.26270 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.515+0000 7ff539600700 1 -- 172.21.15.27:0/2122425796 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55df930210a0 con 0x55df93071f20 2024-05-10T11:19:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.515+0000 7ff539600700 10 client.26270 handle_command_reply: tid=0 2024-05-10T11:19:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.515+0000 7ff560ad1b80 1 client.26270 shutdown 2024-05-10T11:19:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.515+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2122425796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 msgr2=0x55df9305d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.515+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 0x55df9305d1b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff53c012740 tx=0x7ff53c012a50 comp rx=0 tx=0).stop 2024-05-10T11:19:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.515+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2122425796 shutdown_connections 2024-05-10T11:19:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.515+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df93071f20 0x55df93074310 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.515+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054060 0x55df9305fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.515+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df930589d0 0x55df9305d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.515+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/2122425796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 0x55df9305d1b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.515+0000 7ff539600700 0 client.26270 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.515+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2122425796 >> 172.21.15.27:0/2122425796 conn(0x55df92f82b60 msgr2=0x55df930492b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.516+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2122425796 shutdown_connections 2024-05-10T11:19:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.516+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2122425796 wait complete. 2024-05-10T11:19:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.516+0000 7ff560ad1b80 20 client.26270 trim_cache size 0 max 16384 2024-05-10T11:19:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.551+0000 7ff560ad1b80 1 Processor -- start 2024-05-10T11:19:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.551+0000 7ff560ad1b80 1 -- start start 2024-05-10T11:19:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.552+0000 7ff560ad1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df92f82b60 0x55df9306e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.552+0000 7ff560ad1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054060 0x55df93049130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.552+0000 7ff560ad1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 0x55df93079c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.552+0000 7ff560ad1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501ad3f0 con 0x55df92f82b60 2024-05-10T11:19:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.552+0000 7ff560ad1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501aef30 con 0x55df93054980 2024-05-10T11:19:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.552+0000 7ff560ad1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501adac0 con 0x55df93054060 2024-05-10T11:19:54.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.552+0000 7ff54fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 0x55df93079c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.552+0000 7ff54ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054060 0x55df93049130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.552+0000 7ff54fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 0x55df93079c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36842/0 (socket says 172.21.15.27:36842) 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.552+0000 7ff54fe00700 1 -- 172.21.15.27:0/1906961042 learned_addr learned my addr 172.21.15.27:0/1906961042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.552+0000 7ff54f400700 1 --2- 172.21.15.27:0/1906961042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df92f82b60 0x55df9306e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.553+0000 7ff54ea00700 1 -- 172.21.15.27:0/1906961042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 msgr2=0x55df93079c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.553+0000 7ff54ea00700 1 --2- 172.21.15.27:0/1906961042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 0x55df93079c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.553+0000 7ff54ea00700 1 -- 172.21.15.27:0/1906961042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df92f82b60 msgr2=0x55df9306e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.553+0000 7ff54ea00700 1 --2- 172.21.15.27:0/1906961042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df92f82b60 0x55df9306e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.553+0000 7ff54ea00700 1 -- 172.21.15.27:0/1906961042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df9305b720 con 0x55df93054060 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.553+0000 7ff54f400700 1 --2- 172.21.15.27:0/1906961042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df92f82b60 0x55df9306e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.553+0000 7ff54fe00700 1 --2- 172.21.15.27:0/1906961042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 0x55df93079c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.553+0000 7ff54ea00700 1 --2- 172.21.15.27:0/1906961042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054060 0x55df93049130 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff53c01c8e0 tx=0x7ff53c011eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb759ee900548b8a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.553+0000 7ff53be00700 1 -- 172.21.15.27:0/1906961042 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c011770 con 0x55df93054060 2024-05-10T11:19:54.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.553+0000 7ff53be00700 1 -- 172.21.15.27:0/1906961042 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff53c011910 con 0x55df93054060 2024-05-10T11:19:54.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.554+0000 7ff53be00700 1 -- 172.21.15.27:0/1906961042 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c01c420 con 0x55df93054060 2024-05-10T11:19:54.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.554+0000 7ff560ad1b80 1 -- 172.21.15.27:0/1906961042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054060 msgr2=0x55df93049130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:54.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.554+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/1906961042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054060 0x55df93049130 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff53c01c8e0 tx=0x7ff53c011eb0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.554+0000 7ff560ad1b80 1 -- 172.21.15.27:0/1906961042 shutdown_connections 2024-05-10T11:19:54.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.554+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/1906961042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054980 0x55df93079c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.554+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/1906961042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df92f82b60 0x55df9306e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.554+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/1906961042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054060 0x55df93049130 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.554+0000 7ff560ad1b80 1 -- 172.21.15.27:0/1906961042 >> 172.21.15.27:0/1906961042 conn(0x55df93048610 msgr2=0x55df93048a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:54.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.554+0000 7ff560ad1b80 1 -- 172.21.15.27:0/1906961042 shutdown_connections 2024-05-10T11:19:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.555+0000 7ff560ad1b80 1 -- 172.21.15.27:0/1906961042 wait complete. 2024-05-10T11:19:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.555+0000 7ff560ad1b80 1 -- start start 2024-05-10T11:19:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.555+0000 7ff560ad1b80 1 Processor -- start 2024-05-10T11:19:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.556+0000 7ff560ad1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df92f82b60 0x55df93061160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.556+0000 7ff560ad1b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054060 0x55df930616a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.556+0000 7ff560ad1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054980 0x55df93069ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.556+0000 7ff560ad1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501ae860 con 0x55df92f82b60 2024-05-10T11:19:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.556+0000 7ff560ad1b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501ae190 con 0x55df93054060 2024-05-10T11:19:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.556+0000 7ff560ad1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501ab390 con 0x55df93054980 2024-05-10T11:19:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.556+0000 7ff560ad1b80 10 client.0 _ll_register_callbacks cb 0x55df92f4dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:19:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.556+0000 7ff54fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054980 0x55df93069ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.556+0000 7ff54f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df92f82b60 0x55df93061160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.556+0000 7ff54fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054980 0x55df93069ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51054/0 (socket says 172.21.15.27:51054) 2024-05-10T11:19:54.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.556+0000 7ff54fe00700 1 -- 172.21.15.27:0/2190764813 learned_addr learned my addr 172.21.15.27:0/2190764813 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:54.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.556+0000 7ff54ea00700 1 --2- 172.21.15.27:0/2190764813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054060 0x55df930616a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.557+0000 7ff54fe00700 1 -- 172.21.15.27:0/2190764813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054060 msgr2=0x55df930616a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:54.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.557+0000 7ff54fe00700 1 --2- 172.21.15.27:0/2190764813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054060 0x55df930616a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.557+0000 7ff54fe00700 1 -- 172.21.15.27:0/2190764813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df92f82b60 msgr2=0x55df93061160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:19:54.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.557+0000 7ff54fe00700 1 --2- 172.21.15.27:0/2190764813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df92f82b60 0x55df93061160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.557+0000 7ff54fe00700 1 -- 172.21.15.27:0/2190764813 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df9306a530 con 0x55df93054980 2024-05-10T11:19:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.557+0000 7ff54fe00700 1 --2- 172.21.15.27:0/2190764813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054980 0x55df93069ff0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff5480061e0 tx=0x7ff548007ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1568d20653c660fe server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.557+0000 7ff539600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:19:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.558+0000 7ff539600700 1 -- 172.21.15.27:0/2190764813 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548038020 con 0x55df93054980 2024-05-10T11:19:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.558+0000 7ff539600700 1 -- 172.21.15.27:0/2190764813 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7ff548029820 con 0x55df93054980 2024-05-10T11:19:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.558+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2190764813 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff53c011910 con 0x55df93054980 2024-05-10T11:19:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.558+0000 7ff539600700 1 -- 172.21.15.27:0/2190764813 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff548029a20 con 0x55df93054980 2024-05-10T11:19:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.558+0000 7ff560ad1b80 10 client.26266 fetch_fsmap learned FSMap version 33 2024-05-10T11:19:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.558+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2190764813 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55df930e1730 con 0x55df93054980 2024-05-10T11:19:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.559+0000 7ff539600700 1 -- 172.21.15.27:0/2190764813 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff53c011910 con 0x55df93054980 2024-05-10T11:19:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.559+0000 7ff539600700 1 -- 172.21.15.27:0/2190764813 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff548011070 con 0x55df93054980 2024-05-10T11:19:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.559+0000 7ff560ad1b80 10 client.26266 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:19:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.559+0000 7ff560ad1b80 10 client.26266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:19:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.560+0000 7ff560ad1b80 20 client.26266 populate_metadata read hostname 'smithi027' 2024-05-10T11:19:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.560+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/2190764813 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df930589d0 0x55df930e39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.560+0000 7ff560ad1b80 4 client.26266 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:19:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.560+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2190764813 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55df92fb4400 con 0x55df930589d0 2024-05-10T11:19:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.560+0000 7ff54f400700 1 --2- 172.21.15.27:0/2190764813 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df930589d0 0x55df930e39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:54.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff54f400700 1 --2- 172.21.15.27:0/2190764813 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df930589d0 0x55df930e39c0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7aad88f84342876a server_cookie=3115ab60c6f6e2b0 in_seq=0 out_seq=0 2024-05-10T11:19:54.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff539600700 10 client.26266 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:54.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff539600700 1 -- 172.21.15.27:0/2190764813 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55df92fb4400 con 0x55df930589d0 2024-05-10T11:19:54.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff539600700 10 client.26266 handle_command_reply: tid=0 2024-05-10T11:19:54.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff560ad1b80 1 client.26266 shutdown 2024-05-10T11:19:54.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2190764813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054980 msgr2=0x55df93069ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:54.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/2190764813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054980 0x55df93069ff0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff5480061e0 tx=0x7ff548007ee0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2190764813 shutdown_connections 2024-05-10T11:19:54.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/2190764813 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df930589d0 0x55df930e39c0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/2190764813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df93054980 0x55df93069ff0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/2190764813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df92f82b60 0x55df93061160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff539600700 0 client.26266 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:19:54.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff560ad1b80 1 --2- 172.21.15.27:0/2190764813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df93054060 0x55df930616a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.561+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2190764813 >> 172.21.15.27:0/2190764813 conn(0x55df93048610 msgr2=0x55df930492b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:54.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.562+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2190764813 shutdown_connections 2024-05-10T11:19:54.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.562+0000 7ff560ad1b80 1 -- 172.21.15.27:0/2190764813 wait complete. 2024-05-10T11:19:54.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.562+0000 7ff560ad1b80 20 client.26266 trim_cache size 0 max 16384 2024-05-10T11:19:54.565 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:19:54.565 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:19:54.565 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:19:54.565 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:19:54.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.563+0000 7ff556200700 1 -- 172.21.15.27:0/2688804659 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff5340b29f0 msgr2=0x7ff5340b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:54.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.563+0000 7ff556200700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff5340b29f0 0x7ff5340b4eb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff548009fd0 tx=0x7ff54800b040 comp rx=0 tx=0).stop 2024-05-10T11:19:54.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.563+0000 7ff556200700 1 -- 172.21.15.27:0/2688804659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff550103750 msgr2=0x7ff55019f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:54.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.563+0000 7ff556200700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff550103750 0x7ff55019f6e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff54000e9a0 tx=0x7ff54000ed60 comp rx=0 tx=0).stop 2024-05-10T11:19:54.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.563+0000 7ff556200700 1 -- 172.21.15.27:0/2688804659 shutdown_connections 2024-05-10T11:19:54.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.563+0000 7ff556200700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff5340b29f0 0x7ff5340b4eb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.563+0000 7ff556200700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550108850 0x7ff5501998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.563+0000 7ff556200700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff550103750 0x7ff55019f6e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.563+0000 7ff556200700 1 --2- 172.21.15.27:0/2688804659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff550104100 0x7ff55019fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:54.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.563+0000 7ff556200700 1 -- 172.21.15.27:0/2688804659 >> 172.21.15.27:0/2688804659 conn(0x7ff55007a600 msgr2=0x7ff550101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:54.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.563+0000 7ff556200700 1 -- 172.21.15.27:0/2688804659 shutdown_connections 2024-05-10T11:19:54.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:54.564+0000 7ff556200700 1 -- 172.21.15.27:0/2688804659 wait complete. 2024-05-10T11:19:54.686 INFO:tasks.workunit.client.0.smithi027.stdout: 1 372684 6.97 MB/sec execute 159 sec latency 16.406 ms 2024-05-10T11:19:54.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:54 smithi027 ceph-mon[30054]: pgmap v203: 129 pgs: 129 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 123 op/s 2024-05-10T11:19:54.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:54 smithi076 ceph-mon[34152]: pgmap v203: 129 pgs: 129 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 123 op/s 2024-05-10T11:19:54.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:54 smithi189 ceph-mon[34842]: pgmap v203: 129 pgs: 129 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 123 op/s 2024-05-10T11:19:55.686 INFO:tasks.workunit.client.0.smithi027.stdout: 1 373368 6.93 MB/sec execute 160 sec latency 34.907 ms 2024-05-10T11:19:55.765 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:19:55.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:19:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:19:56.686 INFO:tasks.workunit.client.0.smithi027.stdout: 1 374990 6.94 MB/sec execute 161 sec latency 20.683 ms 2024-05-10T11:19:56.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:56 smithi027 ceph-mon[30054]: pgmap v204: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-05-10T11:19:56.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:56 smithi076 ceph-mon[34152]: pgmap v204: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-05-10T11:19:56.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:56 smithi189 ceph-mon[34842]: pgmap v204: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-05-10T11:19:57.686 INFO:tasks.workunit.client.0.smithi027.stdout: 1 376246 6.96 MB/sec execute 162 sec latency 103.626 ms 2024-05-10T11:19:58.686 INFO:tasks.workunit.client.0.smithi027.stdout: 1 376861 6.92 MB/sec execute 163 sec latency 226.898 ms 2024-05-10T11:19:58.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:58 smithi027 ceph-mon[30054]: pgmap v205: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-05-10T11:19:58.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:58 smithi076 ceph-mon[34152]: pgmap v205: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-05-10T11:19:58.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:58 smithi189 ceph-mon[34842]: pgmap v205: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-05-10T11:19:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.087+0000 7fbd8b800700 1 -- 172.21.15.27:0/4043326188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84075ff0 msgr2=0x7fbd8410f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.087+0000 7fbd8b800700 1 --2- 172.21.15.27:0/4043326188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84075ff0 0x7fbd8410f9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009990 tx=0x7fbd70009ca0 comp rx=0 tx=0).stop 2024-05-10T11:19:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.087+0000 7fbd8b800700 1 -- 172.21.15.27:0/4043326188 shutdown_connections 2024-05-10T11:19:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.087+0000 7fbd8b800700 1 --2- 172.21.15.27:0/4043326188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd8410ff00 0x7fbd841122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.087+0000 7fbd8b800700 1 --2- 172.21.15.27:0/4043326188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd840756d0 0x7fbd84075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.087+0000 7fbd8b800700 1 --2- 172.21.15.27:0/4043326188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84075ff0 0x7fbd8410f9c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.087+0000 7fbd8b800700 1 -- 172.21.15.27:0/4043326188 >> 172.21.15.27:0/4043326188 conn(0x7fbd8407a600 msgr2=0x7fbd8407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.088+0000 7fbd8b800700 1 -- 172.21.15.27:0/4043326188 shutdown_connections 2024-05-10T11:19:59.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.088+0000 7fbd8b800700 1 -- 172.21.15.27:0/4043326188 wait complete. 2024-05-10T11:19:59.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.089+0000 7fbd8b800700 1 Processor -- start 2024-05-10T11:19:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.089+0000 7fbd8b800700 1 -- start start 2024-05-10T11:19:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.089+0000 7fbd8b800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd840756d0 0x7fbd8419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.089+0000 7fbd8b800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd84075ff0 0x7fbd8419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.089+0000 7fbd8b800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd8410ff00 0x7fbd841a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.089+0000 7fbd8b800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbd841158f0 con 0x7fbd840756d0 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.089+0000 7fbd8b800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbd84115770 con 0x7fbd84075ff0 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.089+0000 7fbd8b800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbd84115a70 con 0x7fbd8410ff00 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.090+0000 7fbd88e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd840756d0 0x7fbd8419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.090+0000 7fbd83e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd84075ff0 0x7fbd8419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.090+0000 7fbd88e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd840756d0 0x7fbd8419bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52674/0 (socket says 172.21.15.27:52674) 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.090+0000 7fbd83e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd84075ff0 0x7fbd8419c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36874/0 (socket says 172.21.15.27:36874) 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.090+0000 7fbd88e00700 1 -- 172.21.15.27:0/1624656263 learned_addr learned my addr 172.21.15.27:0/1624656263 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.090+0000 7fbd89800700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd8410ff00 0x7fbd841a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.090+0000 7fbd83e00700 1 -- 172.21.15.27:0/1624656263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd8410ff00 msgr2=0x7fbd841a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.090+0000 7fbd83e00700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd8410ff00 0x7fbd841a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.090+0000 7fbd83e00700 1 -- 172.21.15.27:0/1624656263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd840756d0 msgr2=0x7fbd8419bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.090+0000 7fbd83e00700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd840756d0 0x7fbd8419bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.090+0000 7fbd83e00700 1 -- 172.21.15.27:0/1624656263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd841a0d20 con 0x7fbd84075ff0 2024-05-10T11:19:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.090+0000 7fbd89800700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd8410ff00 0x7fbd841a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:19:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.091+0000 7fbd83e00700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd84075ff0 0x7fbd8419c2a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbd700098e0 tx=0x7fbd700057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.091+0000 7fbd81600700 1 -- 172.21.15.27:0/1624656263 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70027070 con 0x7fbd84075ff0 2024-05-10T11:19:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.091+0000 7fbd8b800700 1 -- 172.21.15.27:0/1624656263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd841a0fb0 con 0x7fbd84075ff0 2024-05-10T11:19:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.091+0000 7fbd81600700 1 -- 172.21.15.27:0/1624656263 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd70022470 con 0x7fbd84075ff0 2024-05-10T11:19:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.091+0000 7fbd81600700 1 -- 172.21.15.27:0/1624656263 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70017650 con 0x7fbd84075ff0 2024-05-10T11:19:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.091+0000 7fbd8b800700 1 -- 172.21.15.27:0/1624656263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd841a8890 con 0x7fbd84075ff0 2024-05-10T11:19:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.092+0000 7fbd8b800700 1 -- 172.21.15.27:0/1624656263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd84076d90 con 0x7fbd84075ff0 2024-05-10T11:19:59.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.095+0000 7fbd81600700 1 -- 172.21.15.27:0/1624656263 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fbd7002a070 con 0x7fbd84075ff0 2024-05-10T11:19:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.096+0000 7fbd81600700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbd6c0b29f0 0x7fbd6c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:19:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.097+0000 7fbd81600700 1 -- 172.21.15.27:0/1624656263 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9503+0+0 (secure 0 0 0) 0x7fbd700e39d0 con 0x7fbd84075ff0 2024-05-10T11:19:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.097+0000 7fbd88e00700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbd6c0b29f0 0x7fbd6c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:19:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.098+0000 7fbd88e00700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbd6c0b29f0 0x7fbd6c0b4eb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbd78009d20 tx=0x7fbd78009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:19:59.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.099+0000 7fbd81600700 1 -- 172.21.15.27:0/1624656263 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fbd70031080 con 0x7fbd84075ff0 2024-05-10T11:19:59.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.421+0000 7fbd8b800700 1 -- 172.21.15.27:0/1624656263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd8406a190 con 0x7fbd84075ff0 2024-05-10T11:19:59.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.422+0000 7fbd81600700 1 -- 172.21.15.27:0/1624656263 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11936 (secure 0 0 0) 0x7fbd8406a190 con 0x7fbd84075ff0 2024-05-10T11:19:59.423 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:19:59.423 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:17:36.552331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:19:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.425+0000 7fbd8b800700 1 -- 172.21.15.27:0/1624656263 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbd6c0b29f0 msgr2=0x7fbd6c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.425+0000 7fbd8b800700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbd6c0b29f0 0x7fbd6c0b4eb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbd78009d20 tx=0x7fbd78009260 comp rx=0 tx=0).stop 2024-05-10T11:19:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.425+0000 7fbd8b800700 1 -- 172.21.15.27:0/1624656263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd84075ff0 msgr2=0x7fbd8419c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:19:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.425+0000 7fbd8b800700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd84075ff0 0x7fbd8419c2a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbd700098e0 tx=0x7fbd700057c0 comp rx=0 tx=0).stop 2024-05-10T11:19:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.425+0000 7fbd8b800700 1 -- 172.21.15.27:0/1624656263 shutdown_connections 2024-05-10T11:19:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.425+0000 7fbd8b800700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbd6c0b29f0 0x7fbd6c0b4eb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.425+0000 7fbd8b800700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd8410ff00 0x7fbd841a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.425+0000 7fbd8b800700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd840756d0 0x7fbd8419bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.425+0000 7fbd8b800700 1 --2- 172.21.15.27:0/1624656263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd84075ff0 0x7fbd8419c2a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:19:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.425+0000 7fbd8b800700 1 -- 172.21.15.27:0/1624656263 >> 172.21.15.27:0/1624656263 conn(0x7fbd8407a600 msgr2=0x7fbd84100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:19:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.426+0000 7fbd8b800700 1 -- 172.21.15.27:0/1624656263 shutdown_connections 2024-05-10T11:19:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:19:59.426+0000 7fbd8b800700 1 -- 172.21.15.27:0/1624656263 wait complete. 2024-05-10T11:19:59.428 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 33 2024-05-10T11:19:59.686 INFO:tasks.workunit.client.0.smithi027.stdout: 1 378403 6.92 MB/sec execute 164 sec latency 22.052 ms 2024-05-10T11:19:59.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:19:59 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1624656263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:19:59.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:19:59 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1624656263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:19:59.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:19:59 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1624656263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:20:00.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:20:00.687 INFO:tasks.workunit.client.0.smithi027.stdout: 1 379736 6.94 MB/sec execute 165 sec latency 60.749 ms 2024-05-10T11:20:00.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:00 smithi027 ceph-mon[30054]: pgmap v206: 129 pgs: 129 active+clean; 226 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-05-10T11:20:00.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:00 smithi027 ceph-mon[30054]: overall HEALTH_OK 2024-05-10T11:20:00.923 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:20:00.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:00 smithi076 ceph-mon[34152]: pgmap v206: 129 pgs: 129 active+clean; 226 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-05-10T11:20:00.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:00 smithi076 ceph-mon[34152]: overall HEALTH_OK 2024-05-10T11:20:00.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:00 smithi189 ceph-mon[34842]: pgmap v206: 129 pgs: 129 active+clean; 226 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-05-10T11:20:00.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:00 smithi189 ceph-mon[34842]: overall HEALTH_OK 2024-05-10T11:20:01.687 INFO:tasks.workunit.client.0.smithi027.stdout: 1 379862 6.91 MB/sec execute 166 sec latency 78.996 ms 2024-05-10T11:20:02.687 INFO:tasks.workunit.client.0.smithi027.stdout: 1 380476 6.87 MB/sec execute 167 sec latency 62.153 ms 2024-05-10T11:20:02.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:02 smithi027 ceph-mon[30054]: pgmap v207: 129 pgs: 129 active+clean; 226 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.5 MiB/s wr, 118 op/s 2024-05-10T11:20:02.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:02 smithi027 ceph-mon[30054]: osdmap e92: 12 total, 12 up, 12 in 2024-05-10T11:20:02.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:02 smithi076 ceph-mon[34152]: pgmap v207: 129 pgs: 129 active+clean; 226 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.5 MiB/s wr, 118 op/s 2024-05-10T11:20:02.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:02 smithi076 ceph-mon[34152]: osdmap e92: 12 total, 12 up, 12 in 2024-05-10T11:20:02.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:02 smithi189 ceph-mon[34842]: pgmap v207: 129 pgs: 129 active+clean; 226 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.5 MiB/s wr, 118 op/s 2024-05-10T11:20:02.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:02 smithi189 ceph-mon[34842]: osdmap e92: 12 total, 12 up, 12 in 2024-05-10T11:20:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.542+0000 7fecff200700 1 -- 172.21.15.27:0/2279611380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecf806b180 msgr2=0x7fecf806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.542+0000 7fecff200700 1 --2- 172.21.15.27:0/2279611380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecf806b180 0x7fecf806b600 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fece4009990 tx=0x7fece4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.542+0000 7fecff200700 1 -- 172.21.15.27:0/2279611380 shutdown_connections 2024-05-10T11:20:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.542+0000 7fecff200700 1 --2- 172.21.15.27:0/2279611380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf806bb40 0x7fecf8110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.542+0000 7fecff200700 1 --2- 172.21.15.27:0/2279611380 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecf8109510 0x7fecf81098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.542+0000 7fecff200700 1 --2- 172.21.15.27:0/2279611380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecf806b180 0x7fecf806b600 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.542+0000 7fecff200700 1 -- 172.21.15.27:0/2279611380 >> 172.21.15.27:0/2279611380 conn(0x7fecf8079350 msgr2=0x7fecf8079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.543+0000 7fecff200700 1 -- 172.21.15.27:0/2279611380 shutdown_connections 2024-05-10T11:20:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.543+0000 7fecff200700 1 -- 172.21.15.27:0/2279611380 wait complete. 2024-05-10T11:20:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.544+0000 7fecff200700 1 Processor -- start 2024-05-10T11:20:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.544+0000 7fecff200700 1 -- start start 2024-05-10T11:20:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.544+0000 7fecff200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecf806b180 0x7fecf819be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.544+0000 7fecff200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecf806bb40 0x7fecf819c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.544+0000 7fecff200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8109510 0x7fecf81a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.544+0000 7fecff200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fecf8113670 con 0x7fecf8109510 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.544+0000 7fecff200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81134f0 con 0x7fecf806b180 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.544+0000 7fecff200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81137f0 con 0x7fecf806bb40 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecf7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecf806b180 0x7fecf819be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecf7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecf806bb40 0x7fecf819c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecf7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecf806b180 0x7fecf819be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36898/0 (socket says 172.21.15.27:36898) 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecf7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecf806bb40 0x7fecf819c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51104/0 (socket says 172.21.15.27:51104) 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecf7e00700 1 -- 172.21.15.27:0/3764868869 learned_addr learned my addr 172.21.15.27:0/3764868869 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecfd200700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8109510 0x7fecf81a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecf7400700 1 -- 172.21.15.27:0/3764868869 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecf806b180 msgr2=0x7fecf819be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecf7400700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecf806b180 0x7fecf819be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecf7400700 1 -- 172.21.15.27:0/3764868869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8109510 msgr2=0x7fecf81a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecf7400700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8109510 0x7fecf81a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecf7400700 1 -- 172.21.15.27:0/3764868869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf81a0e90 con 0x7fecf806bb40 2024-05-10T11:20:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecfd200700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8109510 0x7fecf81a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:20:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.545+0000 7fecf7400700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecf806bb40 0x7fecf819c3c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fece4000c00 tx=0x7fece40047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.546+0000 7fecf4c00700 1 -- 172.21.15.27:0/3764868869 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4026070 con 0x7fecf806bb40 2024-05-10T11:20:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.546+0000 7fecf4c00700 1 -- 172.21.15.27:0/3764868869 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fece4021410 con 0x7fecf806bb40 2024-05-10T11:20:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.546+0000 7fecf4c00700 1 -- 172.21.15.27:0/3764868869 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4017650 con 0x7fecf806bb40 2024-05-10T11:20:03.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.546+0000 7fecff200700 1 -- 172.21.15.27:0/3764868869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf81a1120 con 0x7fecf806bb40 2024-05-10T11:20:03.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.546+0000 7fecff200700 1 -- 172.21.15.27:0/3764868869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf81a8a00 con 0x7fecf806bb40 2024-05-10T11:20:03.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.550+0000 7fecf4c00700 1 -- 172.21.15.27:0/3764868869 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fece4029070 con 0x7fecf806bb40 2024-05-10T11:20:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.551+0000 7fed09c88b80 1 Processor -- start 2024-05-10T11:20:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.551+0000 7fed09c88b80 1 -- start start 2024-05-10T11:20:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fecf4c00700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fecdc0b2a70 0x7fecdc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fed09c88b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 0x559a41827fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fed09c88b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418284e0 0x559a418222d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fed09c88b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a418228f0 0x559a41822d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fed09c88b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559a41774a20 con 0x559a418284e0 2024-05-10T11:20:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fed09c88b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x559a41712760 con 0x559a41827bc0 2024-05-10T11:20:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fed09c88b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559a417709e0 con 0x559a418228f0 2024-05-10T11:20:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fecf4c00700 1 -- 172.21.15.27:0/3764868869 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9679+0+0 (secure 0 0 0) 0x7fece40e3830 con 0x7fecf806bb40 2024-05-10T11:20:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fecf7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 0x559a41827fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fecf7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 0x559a41827fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36906/0 (socket says 172.21.15.27:36906) 2024-05-10T11:20:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fecf7e00700 1 -- 172.21.15.27:0/1610601635 learned_addr learned my addr 172.21.15.27:0/1610601635 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fecfd200700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fecdc0b2a70 0x7fecdc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fecf7400700 1 --2- 172.21.15.27:0/1610601635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418284e0 0x559a418222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fecf7e00700 1 -- 172.21.15.27:0/1610601635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a418228f0 msgr2=0x559a41822d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fecf7e00700 1 --2- 172.21.15.27:0/1610601635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a418228f0 0x559a41822d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fecf7e00700 1 -- 172.21.15.27:0/1610601635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418284e0 msgr2=0x559a418222d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.552+0000 7fecf7e00700 1 --2- 172.21.15.27:0/1610601635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418284e0 0x559a418222d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.553+0000 7fecf7e00700 1 -- 172.21.15.27:0/1610601635 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a416c18d0 con 0x559a41827bc0 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.553+0000 7fecf7400700 1 --2- 172.21.15.27:0/1610601635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418284e0 0x559a418222d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.553+0000 7fecf7e00700 1 --2- 172.21.15.27:0/1610601635 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 0x559a41827fa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fece800e660 tx=0x7fece8009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bec9bdd2628cb6b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.553+0000 7feced600700 1 -- 172.21.15.27:0/1610601635 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece8003a00 con 0x559a41827bc0 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.553+0000 7fecfd200700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fecdc0b2a70 0x7fecdc0b4f30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x559a418232b0 tx=0x7fecf0009640 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.553+0000 7feced600700 1 -- 172.21.15.27:0/1610601635 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fece801a070 con 0x559a41827bc0 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.553+0000 7feced600700 1 -- 172.21.15.27:0/1610601635 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece8007760 con 0x559a41827bc0 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.554+0000 7fed09c88b80 1 -- 172.21.15.27:0/1610601635 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 msgr2=0x559a41827fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.554+0000 7fed09c88b80 1 --2- 172.21.15.27:0/1610601635 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 0x559a41827fa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fece800e660 tx=0x7fece8009920 comp rx=0 tx=0).stop 2024-05-10T11:20:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.554+0000 7fed09c88b80 1 -- 172.21.15.27:0/1610601635 shutdown_connections 2024-05-10T11:20:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.554+0000 7fed09c88b80 1 --2- 172.21.15.27:0/1610601635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a418228f0 0x559a41822d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.554+0000 7fed09c88b80 1 --2- 172.21.15.27:0/1610601635 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 0x559a41827fa0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.554+0000 7fed09c88b80 1 --2- 172.21.15.27:0/1610601635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418284e0 0x559a418222d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.554+0000 7fed09c88b80 1 -- 172.21.15.27:0/1610601635 >> 172.21.15.27:0/1610601635 conn(0x559a4179a6d0 msgr2=0x559a4181cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.554+0000 7fed09c88b80 1 -- 172.21.15.27:0/1610601635 shutdown_connections 2024-05-10T11:20:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.554+0000 7fed09c88b80 1 -- 172.21.15.27:0/1610601635 wait complete. 2024-05-10T11:20:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.555+0000 7fed09c88b80 1 -- start start 2024-05-10T11:20:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.555+0000 7fed09c88b80 1 Processor -- start 2024-05-10T11:20:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.556+0000 7fed09c88b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418228f0 0x559a4182e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.556+0000 7fed09c88b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a41827bc0 0x559a4182b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.556+0000 7fed09c88b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418284e0 0x559a4182bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.556+0000 7fed09c88b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559a4182eec0 con 0x559a418228f0 2024-05-10T11:20:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.556+0000 7fed09c88b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x559a416dab20 con 0x559a418284e0 2024-05-10T11:20:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.556+0000 7fed09c88b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559a4182f040 con 0x559a41827bc0 2024-05-10T11:20:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.556+0000 7fed09c88b80 10 client.0 _ll_register_callbacks cb 0x559a4171bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.556+0000 7fecfd200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418284e0 0x559a4182bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.556+0000 7fecf7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418228f0 0x559a4182e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.556+0000 7fecfd200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418284e0 0x559a4182bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36916/0 (socket says 172.21.15.27:36916) 2024-05-10T11:20:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.556+0000 7fecfd200700 1 -- 172.21.15.27:0/161504390 learned_addr learned my addr 172.21.15.27:0/161504390 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.556+0000 7fecf7400700 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a41827bc0 0x559a4182b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.557+0000 7fecfd200700 1 -- 172.21.15.27:0/161504390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a41827bc0 msgr2=0x559a4182b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.557+0000 7fecfd200700 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a41827bc0 0x559a4182b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.557+0000 7fecfd200700 1 -- 172.21.15.27:0/161504390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418228f0 msgr2=0x559a4182e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.557+0000 7fecfd200700 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418228f0 0x559a4182e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.557+0000 7fecfd200700 1 -- 172.21.15.27:0/161504390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a4173a420 con 0x559a418284e0 2024-05-10T11:20:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.557+0000 7fecf7e00700 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418228f0 0x559a4182e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.557+0000 7fecfd200700 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418284e0 0x559a4182bac0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fecf0005490 tx=0x7fecf0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ffdaa9aada21b47 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.557+0000 7fece2000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.557+0000 7fece2000700 1 -- 172.21.15.27:0/161504390 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf0004e60 con 0x559a418284e0 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.557+0000 7fece2000700 1 -- 172.21.15.27:0/161504390 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9679+0+0 (secure 0 0 0) 0x7fecf0025670 con 0x559a418284e0 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.557+0000 7fed09c88b80 1 -- 172.21.15.27:0/161504390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fece801a070 con 0x559a418284e0 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.558+0000 7fece2000700 1 -- 172.21.15.27:0/161504390 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fecf002e050 con 0x559a418284e0 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.558+0000 7fed09c88b80 10 client.26294 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.558+0000 7fed09c88b80 1 -- 172.21.15.27:0/161504390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559a41839710 con 0x559a418284e0 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.558+0000 7fece2000700 1 -- 172.21.15.27:0/161504390 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fece801a070 con 0x559a418284e0 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.558+0000 7fece2000700 1 -- 172.21.15.27:0/161504390 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fecf0018030 con 0x559a418284e0 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.558+0000 7fed09c88b80 10 client.26294 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.558+0000 7fed09c88b80 10 client.26294 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:20:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.558+0000 7fed09c88b80 20 client.26294 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.558+0000 7fed09c88b80 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559a4183bd50 0x559a4183e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.558+0000 7fed09c88b80 4 client.26294 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:20:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.558+0000 7fed09c88b80 1 -- 172.21.15.27:0/161504390 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a417ef140 con 0x559a4183bd50 2024-05-10T11:20:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.559+0000 7fecf7e00700 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559a4183bd50 0x559a4183e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.559+0000 7fecf7e00700 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559a4183bd50 0x559a4183e140 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=88927872edfc01e0 server_cookie=5c6ab1c54fd0d8fc in_seq=0 out_seq=0 2024-05-10T11:20:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.559+0000 7fece2000700 10 client.26294 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:20:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.560+0000 7fece2000700 1 -- 172.21.15.27:0/161504390 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559a417ef140 con 0x559a4183bd50 2024-05-10T11:20:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.560+0000 7fece2000700 10 client.26294 handle_command_reply: tid=0 2024-05-10T11:20:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.560+0000 7fed09c88b80 1 client.26294 shutdown 2024-05-10T11:20:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.560+0000 7fed09c88b80 1 -- 172.21.15.27:0/161504390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418284e0 msgr2=0x559a4182bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.560+0000 7fed09c88b80 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418284e0 0x559a4182bac0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fecf0005490 tx=0x7fecf0009260 comp rx=0 tx=0).stop 2024-05-10T11:20:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.561+0000 7fed09c88b80 1 -- 172.21.15.27:0/161504390 shutdown_connections 2024-05-10T11:20:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.561+0000 7fed09c88b80 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418284e0 0x559a4182bac0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.561+0000 7fed09c88b80 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418228f0 0x559a4182e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.561+0000 7fed09c88b80 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559a4183bd50 0x559a4183e140 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.561+0000 7fed09c88b80 1 --2- 172.21.15.27:0/161504390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a41827bc0 0x559a4182b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.561+0000 7fed09c88b80 1 -- 172.21.15.27:0/161504390 >> 172.21.15.27:0/161504390 conn(0x559a4179a6d0 msgr2=0x559a4181c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.561+0000 7fece2000700 0 client.26294 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:20:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.561+0000 7fed09c88b80 1 -- 172.21.15.27:0/161504390 shutdown_connections 2024-05-10T11:20:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.561+0000 7fed09c88b80 1 -- 172.21.15.27:0/161504390 wait complete. 2024-05-10T11:20:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.561+0000 7fed09c88b80 20 client.26294 trim_cache size 0 max 16384 2024-05-10T11:20:03.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.580+0000 7fed09c88b80 1 Processor -- start 2024-05-10T11:20:03.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.580+0000 7fed09c88b80 1 -- start start 2024-05-10T11:20:03.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.580+0000 7fed09c88b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418228f0 0x559a418222d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fed09c88b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a41827bc0 0x559a41825d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fed09c88b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418284e0 0x559a41829150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fed09c88b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81b1920 con 0x559a418284e0 2024-05-10T11:20:03.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fed09c88b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81b3460 con 0x559a418228f0 2024-05-10T11:20:03.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fed09c88b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81b1ff0 con 0x559a41827bc0 2024-05-10T11:20:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fecf7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418228f0 0x559a418222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fecf7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418228f0 0x559a418222d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36928/0 (socket says 172.21.15.27:36928) 2024-05-10T11:20:03.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fecf7e00700 1 -- 172.21.15.27:0/3954407332 learned_addr learned my addr 172.21.15.27:0/3954407332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:03.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fecf7400700 1 --2- 172.21.15.27:0/3954407332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a41827bc0 0x559a41825d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fecfd200700 1 --2- 172.21.15.27:0/3954407332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418284e0 0x559a41829150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fecf7e00700 1 -- 172.21.15.27:0/3954407332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a41827bc0 msgr2=0x559a41825d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:03.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fecf7e00700 1 --2- 172.21.15.27:0/3954407332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a41827bc0 0x559a41825d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fecf7e00700 1 -- 172.21.15.27:0/3954407332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418284e0 msgr2=0x559a41829150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:03.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fecf7e00700 1 --2- 172.21.15.27:0/3954407332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418284e0 0x559a41829150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fecf7e00700 1 -- 172.21.15.27:0/3954407332 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a4181d820 con 0x559a418228f0 2024-05-10T11:20:03.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.581+0000 7fecfd200700 1 --2- 172.21.15.27:0/3954407332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418284e0 0x559a41829150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:20:03.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.582+0000 7fecf7e00700 1 --2- 172.21.15.27:0/3954407332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418228f0 0x559a418222d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fece8004780 tx=0x7fece80047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93b16035f583d4a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:03.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.582+0000 7fececc00700 1 -- 172.21.15.27:0/3954407332 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece802d070 con 0x559a418228f0 2024-05-10T11:20:03.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.582+0000 7fececc00700 1 -- 172.21.15.27:0/3954407332 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fece801a040 con 0x559a418228f0 2024-05-10T11:20:03.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.582+0000 7fececc00700 1 -- 172.21.15.27:0/3954407332 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece8028400 con 0x559a418228f0 2024-05-10T11:20:03.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.583+0000 7fed09c88b80 1 -- 172.21.15.27:0/3954407332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418228f0 msgr2=0x559a418222d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:03.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.583+0000 7fed09c88b80 1 --2- 172.21.15.27:0/3954407332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418228f0 0x559a418222d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fece8004780 tx=0x7fece80047b0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.583+0000 7fed09c88b80 1 -- 172.21.15.27:0/3954407332 shutdown_connections 2024-05-10T11:20:03.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.583+0000 7fed09c88b80 1 --2- 172.21.15.27:0/3954407332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418284e0 0x559a41829150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.583+0000 7fed09c88b80 1 --2- 172.21.15.27:0/3954407332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a418228f0 0x559a418222d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.583+0000 7fed09c88b80 1 --2- 172.21.15.27:0/3954407332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a41827bc0 0x559a41825d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.583+0000 7fed09c88b80 1 -- 172.21.15.27:0/3954407332 >> 172.21.15.27:0/3954407332 conn(0x559a4179a6d0 msgr2=0x559a41829e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:03.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.583+0000 7fed09c88b80 1 -- 172.21.15.27:0/3954407332 shutdown_connections 2024-05-10T11:20:03.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.583+0000 7fed09c88b80 1 -- 172.21.15.27:0/3954407332 wait complete. 2024-05-10T11:20:03.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.584+0000 7fed09c88b80 1 -- start start 2024-05-10T11:20:03.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.584+0000 7fed09c88b80 1 Processor -- start 2024-05-10T11:20:03.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fed09c88b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418228f0 0x559a418344b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fed09c88b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 0x559a418349f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fed09c88b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a418284e0 0x559a41842dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fed09c88b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81b2d90 con 0x559a418228f0 2024-05-10T11:20:03.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fed09c88b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81b26c0 con 0x559a41827bc0 2024-05-10T11:20:03.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fed09c88b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81af8e0 con 0x559a418284e0 2024-05-10T11:20:03.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fed09c88b80 10 client.0 _ll_register_callbacks cb 0x559a4171bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:03.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fecf7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 0x559a418349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fecf7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418228f0 0x559a418344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fecf7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 0x559a418349f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36940/0 (socket says 172.21.15.27:36940) 2024-05-10T11:20:03.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fecf7400700 1 -- 172.21.15.27:0/3977530337 learned_addr learned my addr 172.21.15.27:0/3977530337 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:03.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fecfd200700 1 --2- 172.21.15.27:0/3977530337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a418284e0 0x559a41842dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fecf7400700 1 -- 172.21.15.27:0/3977530337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a418284e0 msgr2=0x559a41842dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fecf7400700 1 --2- 172.21.15.27:0/3977530337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a418284e0 0x559a41842dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fecf7400700 1 -- 172.21.15.27:0/3977530337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418228f0 msgr2=0x559a418344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fecf7400700 1 --2- 172.21.15.27:0/3977530337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418228f0 0x559a418344b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.585+0000 7fecf7400700 1 -- 172.21.15.27:0/3977530337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a41843600 con 0x559a41827bc0 2024-05-10T11:20:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.586+0000 7fecf7400700 1 --2- 172.21.15.27:0/3977530337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 0x559a418349f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fece402f030 tx=0x7fece40096e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d230f9c751fcb243 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.586+0000 7fece2000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:20:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.586+0000 7fece2000700 1 -- 172.21.15.27:0/3977530337 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4001430 con 0x559a41827bc0 2024-05-10T11:20:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.586+0000 7fece2000700 1 -- 172.21.15.27:0/3977530337 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9679+0+0 (secure 0 0 0) 0x7fece40e0d20 con 0x559a41827bc0 2024-05-10T11:20:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.586+0000 7fed09c88b80 1 -- 172.21.15.27:0/3977530337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fece801a040 con 0x559a41827bc0 2024-05-10T11:20:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.586+0000 7fece2000700 1 -- 172.21.15.27:0/3977530337 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fece4001250 con 0x559a41827bc0 2024-05-10T11:20:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.587+0000 7fed09c88b80 10 client.26300 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.587+0000 7fed09c88b80 1 -- 172.21.15.27:0/3977530337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559a418aea50 con 0x559a41827bc0 2024-05-10T11:20:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.587+0000 7fece2000700 1 -- 172.21.15.27:0/3977530337 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fece801a040 con 0x559a41827bc0 2024-05-10T11:20:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.587+0000 7fece2000700 1 -- 172.21.15.27:0/3977530337 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fece4090be0 con 0x559a41827bc0 2024-05-10T11:20:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.587+0000 7fed09c88b80 10 client.26300 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.587+0000 7fed09c88b80 10 client.26300 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:20:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.587+0000 7fed09c88b80 20 client.26300 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.587+0000 7fed09c88b80 1 --2- 172.21.15.27:0/3977530337 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559a418af1b0 0x559a418b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.587+0000 7fed09c88b80 4 client.26300 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:20:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.587+0000 7fed09c88b80 1 -- 172.21.15.27:0/3977530337 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559a41782400 con 0x559a418af1b0 2024-05-10T11:20:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.588+0000 7fecf7e00700 1 --2- 172.21.15.27:0/3977530337 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559a418af1b0 0x559a418b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fecf7e00700 1 --2- 172.21.15.27:0/3977530337 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559a418af1b0 0x559a418b15d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1a33d9b91b6738a7 server_cookie=7153dec4ad3eaeca in_seq=0 out_seq=0 2024-05-10T11:20:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fece2000700 10 client.26300 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:20:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fece2000700 1 -- 172.21.15.27:0/3977530337 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559a41782400 con 0x559a418af1b0 2024-05-10T11:20:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fece2000700 10 client.26300 handle_command_reply: tid=0 2024-05-10T11:20:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fed09c88b80 1 client.26300 shutdown 2024-05-10T11:20:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fed09c88b80 1 -- 172.21.15.27:0/3977530337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 msgr2=0x559a418349f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fed09c88b80 1 --2- 172.21.15.27:0/3977530337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 0x559a418349f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fece402f030 tx=0x7fece40096e0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fed09c88b80 1 -- 172.21.15.27:0/3977530337 shutdown_connections 2024-05-10T11:20:03.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fed09c88b80 1 --2- 172.21.15.27:0/3977530337 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x559a418af1b0 0x559a418b15d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fed09c88b80 1 --2- 172.21.15.27:0/3977530337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a418284e0 0x559a41842dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fed09c88b80 1 --2- 172.21.15.27:0/3977530337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a418228f0 0x559a418344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fed09c88b80 1 --2- 172.21.15.27:0/3977530337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559a41827bc0 0x559a418349f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fece2000700 0 client.26300 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:20:03.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fed09c88b80 1 -- 172.21.15.27:0/3977530337 >> 172.21.15.27:0/3977530337 conn(0x559a4179a6d0 msgr2=0x559a41826fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:03.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.589+0000 7fed09c88b80 1 -- 172.21.15.27:0/3977530337 shutdown_connections 2024-05-10T11:20:03.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.590+0000 7fed09c88b80 1 -- 172.21.15.27:0/3977530337 wait complete. 2024-05-10T11:20:03.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.590+0000 7fed09c88b80 20 client.26300 trim_cache size 0 max 16384 2024-05-10T11:20:03.607 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:20:03.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.590+0000 7fecff200700 1 -- 172.21.15.27:0/3764868869 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fecdc0b2a70 msgr2=0x7fecdc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.590+0000 7fecff200700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fecdc0b2a70 0x7fecdc0b4f30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x559a418232b0 tx=0x7fecf0009640 comp rx=0 tx=0).stop 2024-05-10T11:20:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.590+0000 7fecff200700 1 -- 172.21.15.27:0/3764868869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecf806bb40 msgr2=0x7fecf819c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.590+0000 7fecff200700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecf806bb40 0x7fecf819c3c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fece4000c00 tx=0x7fece40047b0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.591+0000 7fecff200700 1 -- 172.21.15.27:0/3764868869 shutdown_connections 2024-05-10T11:20:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.591+0000 7fecff200700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fecdc0b2a70 0x7fecdc0b4f30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.591+0000 7fecff200700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8109510 0x7fecf81a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.591+0000 7fecff200700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecf806b180 0x7fecf819be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.591+0000 7fecff200700 1 --2- 172.21.15.27:0/3764868869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecf806bb40 0x7fecf819c3c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:03.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.591+0000 7fecff200700 1 -- 172.21.15.27:0/3764868869 >> 172.21.15.27:0/3764868869 conn(0x7fecf8079350 msgr2=0x7fecf810e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:03.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.591+0000 7fecff200700 1 -- 172.21.15.27:0/3764868869 shutdown_connections 2024-05-10T11:20:03.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:03.591+0000 7fecff200700 1 -- 172.21.15.27:0/3764868869 wait complete. 2024-05-10T11:20:03.687 INFO:tasks.workunit.client.0.smithi027.stdout: 1 382042 6.87 MB/sec execute 168 sec latency 33.851 ms 2024-05-10T11:20:04.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:20:04.672 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:04 smithi027 ceph-mon[30054]: pgmap v209: 129 pgs: 129 active+clean; 226 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 142 op/s 2024-05-10T11:20:04.687 INFO:tasks.workunit.client.0.smithi027.stdout: 1 383431 6.90 MB/sec execute 169 sec latency 38.094 ms 2024-05-10T11:20:04.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:04 smithi076 ceph-mon[34152]: pgmap v209: 129 pgs: 129 active+clean; 226 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 142 op/s 2024-05-10T11:20:04.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:04 smithi189 ceph-mon[34842]: pgmap v209: 129 pgs: 129 active+clean; 226 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 142 op/s 2024-05-10T11:20:05.060 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:20:05.687 INFO:tasks.workunit.client.0.smithi027.stdout: 1 384597 6.89 MB/sec execute 170 sec latency 32.970 ms 2024-05-10T11:20:06.687 INFO:tasks.workunit.client.0.smithi027.stdout: 1 386503 6.90 MB/sec execute 171 sec latency 20.706 ms 2024-05-10T11:20:06.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:06 smithi027 ceph-mon[30054]: pgmap v210: 129 pgs: 129 active+clean; 230 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-05-10T11:20:06.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:06 smithi076 ceph-mon[34152]: pgmap v210: 129 pgs: 129 active+clean; 230 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-05-10T11:20:06.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:06 smithi189 ceph-mon[34842]: pgmap v210: 129 pgs: 129 active+clean; 230 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-05-10T11:20:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.647+0000 7f441f200700 1 -- 172.21.15.27:0/1181375713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441811cc80 msgr2=0x7f441811f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.647+0000 7f441f200700 1 --2- 172.21.15.27:0/1181375713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441811cc80 0x7f441811f070 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4410009990 tx=0x7f4410009ca0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.648+0000 7f441f200700 1 -- 172.21.15.27:0/1181375713 shutdown_connections 2024-05-10T11:20:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.648+0000 7f441f200700 1 --2- 172.21.15.27:0/1181375713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4418111c80 0x7f441810a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.648+0000 7f441f200700 1 --2- 172.21.15.27:0/1181375713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441811cc80 0x7f441811f070 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.649+0000 7f441f200700 1 --2- 172.21.15.27:0/1181375713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f441810a9e0 0x7f441810ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.649+0000 7f441f200700 1 -- 172.21.15.27:0/1181375713 >> 172.21.15.27:0/1181375713 conn(0x7f441806e610 msgr2=0x7f441806ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.649+0000 7f441f200700 1 -- 172.21.15.27:0/1181375713 shutdown_connections 2024-05-10T11:20:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.649+0000 7f441f200700 1 -- 172.21.15.27:0/1181375713 wait complete. 2024-05-10T11:20:07.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.651+0000 7f441f200700 1 Processor -- start 2024-05-10T11:20:07.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.652+0000 7f441f200700 1 -- start start 2024-05-10T11:20:07.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.652+0000 7f441f200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f441810a9e0 0x7f44181ae620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.652+0000 7f441f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4418111c80 0x7f44181aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.652+0000 7f441f200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441811cc80 0x7f44181a86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.652+0000 7f441f200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4418122510 con 0x7f441810a9e0 2024-05-10T11:20:07.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.652+0000 7f441f200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4418122390 con 0x7f441811cc80 2024-05-10T11:20:07.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.652+0000 7f441f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4418122690 con 0x7f4418111c80 2024-05-10T11:20:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.653+0000 7f440f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4418111c80 0x7f44181aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.653+0000 7f440f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4418111c80 0x7f44181aeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55264/0 (socket says 172.21.15.27:55264) 2024-05-10T11:20:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.653+0000 7f440f400700 1 -- 172.21.15.27:0/1548593789 learned_addr learned my addr 172.21.15.27:0/1548593789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.654+0000 7f441d200700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441811cc80 0x7f44181a86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.654+0000 7f4417e00700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f441810a9e0 0x7f44181ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.655+0000 7f440f400700 1 -- 172.21.15.27:0/1548593789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441811cc80 msgr2=0x7f44181a86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.655+0000 7f440f400700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441811cc80 0x7f44181a86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.655+0000 7f440f400700 1 -- 172.21.15.27:0/1548593789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f441810a9e0 msgr2=0x7f44181ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.655+0000 7f440f400700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f441810a9e0 0x7f44181ae620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.655+0000 7f440f400700 1 -- 172.21.15.27:0/1548593789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44181a8fd0 con 0x7f4418111c80 2024-05-10T11:20:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.657+0000 7f440f400700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4418111c80 0x7f44181aeb60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4400007e20 tx=0x7f440000a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:07.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.658+0000 7f4415600700 1 -- 172.21.15.27:0/1548593789 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4400016020 con 0x7f4418111c80 2024-05-10T11:20:07.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.658+0000 7f4415600700 1 -- 172.21.15.27:0/1548593789 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f440000ad40 con 0x7f4418111c80 2024-05-10T11:20:07.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.662+0000 7f4415600700 1 -- 172.21.15.27:0/1548593789 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4400011670 con 0x7f4418111c80 2024-05-10T11:20:07.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.662+0000 7f441f200700 1 -- 172.21.15.27:0/1548593789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44181a92c0 con 0x7f4418111c80 2024-05-10T11:20:07.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.663+0000 7f441f200700 1 -- 172.21.15.27:0/1548593789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44181b55a0 con 0x7f4418111c80 2024-05-10T11:20:07.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.675+0000 7f4415600700 1 -- 172.21.15.27:0/1548593789 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f4400019020 con 0x7f4418111c80 2024-05-10T11:20:07.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.678+0000 7f4415600700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f43fc0b29d0 0x7f43fc0b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.678+0000 7f4415600700 1 -- 172.21.15.27:0/1548593789 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9679+0+0 (secure 0 0 0) 0x7f44000e21a0 con 0x7f4418111c80 2024-05-10T11:20:07.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.679+0000 7f4429be7b80 1 Processor -- start 2024-05-10T11:20:07.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.679+0000 7f4429be7b80 1 -- start start 2024-05-10T11:20:07.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.679+0000 7f4417e00700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f43fc0b29d0 0x7f43fc0b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.679+0000 7f4429be7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f490cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.680+0000 7f4429be7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f49072d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.680+0000 7f4429be7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f4907d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.680+0000 7f4417e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f490cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.680+0000 7f4417e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f490cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55276/0 (socket says 172.21.15.27:55276) 2024-05-10T11:20:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.680+0000 7f4429be7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5637f4859a20 con 0x5637f490d4e0 2024-05-10T11:20:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.680+0000 7f4429be7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5637f47f7760 con 0x5637f49078f0 2024-05-10T11:20:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.680+0000 7f4429be7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5637f48559e0 con 0x5637f490cbc0 2024-05-10T11:20:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.680+0000 7f4417e00700 1 -- 172.21.15.27:0/3307608351 learned_addr learned my addr 172.21.15.27:0/3307608351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:07.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.681+0000 7f440f400700 1 --2- 172.21.15.27:0/3307608351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f49072d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.681+0000 7f441d200700 1 --2- 172.21.15.27:0/3307608351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f4907d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.681+0000 7f4417e00700 1 -- 172.21.15.27:0/3307608351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 msgr2=0x5637f4907d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:07.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.681+0000 7f4417e00700 1 --2- 172.21.15.27:0/3307608351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f4907d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.681+0000 7f4417e00700 1 -- 172.21.15.27:0/3307608351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 msgr2=0x5637f49072d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:07.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.681+0000 7f4417e00700 1 --2- 172.21.15.27:0/3307608351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f49072d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.681+0000 7f4417e00700 1 -- 172.21.15.27:0/3307608351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637f47a68d0 con 0x5637f490cbc0 2024-05-10T11:20:07.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.682+0000 7f4417e00700 1 --2- 172.21.15.27:0/3307608351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f490cfa0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4408010650 tx=0x7f440801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2287d0d87b21cecd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:07.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.682+0000 7f4417e00700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f43fc0b29d0 0x7f43fc0b4e90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4408013970 tx=0x7f440800aba0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:07.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.683+0000 7f440d600700 1 -- 172.21.15.27:0/3307608351 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440801f810 con 0x5637f490cbc0 2024-05-10T11:20:07.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.683+0000 7f440d600700 1 -- 172.21.15.27:0/3307608351 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4408010e90 con 0x5637f490cbc0 2024-05-10T11:20:07.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.683+0000 7f440d600700 1 -- 172.21.15.27:0/3307608351 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408011bd0 con 0x5637f490cbc0 2024-05-10T11:20:07.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.684+0000 7f4429be7b80 1 -- 172.21.15.27:0/3307608351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 msgr2=0x5637f490cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:07.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.684+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3307608351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f490cfa0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4408010650 tx=0x7f440801a000 comp rx=0 tx=0).stop 2024-05-10T11:20:07.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.685+0000 7f4429be7b80 1 -- 172.21.15.27:0/3307608351 shutdown_connections 2024-05-10T11:20:07.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.685+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3307608351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f490cfa0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.685+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3307608351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f4907d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.685+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3307608351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f49072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.685+0000 7f4429be7b80 1 -- 172.21.15.27:0/3307608351 >> 172.21.15.27:0/3307608351 conn(0x5637f487f6d0 msgr2=0x5637f4901b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:07.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.686+0000 7f4429be7b80 1 -- 172.21.15.27:0/3307608351 shutdown_connections 2024-05-10T11:20:07.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.686+0000 7f4429be7b80 1 -- 172.21.15.27:0/3307608351 wait complete. 2024-05-10T11:20:07.687 INFO:tasks.workunit.client.0.smithi027.stdout: 1 387156 6.89 MB/sec execute 172 sec latency 30.420 ms 2024-05-10T11:20:07.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.687+0000 7f4429be7b80 1 -- start start 2024-05-10T11:20:07.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.688+0000 7f4429be7b80 1 Processor -- start 2024-05-10T11:20:07.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.689+0000 7f4429be7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f49131a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f4429be7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f4910540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f4429be7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f4910a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f4417e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f49131a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f4429be7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5637f4913ec0 con 0x5637f490d4e0 2024-05-10T11:20:07.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f4417e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f49131a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51336/0 (socket says 172.21.15.27:51336) 2024-05-10T11:20:07.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f440f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f4910540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f4429be7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5637f47bfb20 con 0x5637f49078f0 2024-05-10T11:20:07.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f4429be7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5637f4914040 con 0x5637f490cbc0 2024-05-10T11:20:07.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f441d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f4910a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f4429be7b80 10 client.0 _ll_register_callbacks cb 0x5637f4800c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:07.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f441d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f4910a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56908/0 (socket says 172.21.15.27:56908) 2024-05-10T11:20:07.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f441d200700 1 -- 172.21.15.27:0/1042649673 learned_addr learned my addr 172.21.15.27:0/1042649673 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:07.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.690+0000 7f440f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f4910540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55278/0 (socket says 172.21.15.27:55278) 2024-05-10T11:20:07.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.691+0000 7f440f400700 1 -- 172.21.15.27:0/1042649673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 msgr2=0x5637f49131a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.691+0000 7f440f400700 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f49131a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.691+0000 7f440f400700 1 -- 172.21.15.27:0/1042649673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 msgr2=0x5637f4910a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.691+0000 7f440f400700 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f4910a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.691+0000 7f440f400700 1 -- 172.21.15.27:0/1042649673 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637f481f420 con 0x5637f490cbc0 2024-05-10T11:20:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.692+0000 7f4417e00700 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f49131a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:20:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.692+0000 7f441d200700 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f4910a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:20:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.692+0000 7f440f400700 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f4910540 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f44000df520 tx=0x7f44000df970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c89ba8fd247bae65 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.692+0000 7f4406000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:20:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.693+0000 7f4406000700 1 -- 172.21.15.27:0/1042649673 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44000c3060 con 0x5637f490cbc0 2024-05-10T11:20:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.693+0000 7f4406000700 1 -- 172.21.15.27:0/1042649673 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9679+0+0 (secure 0 0 0) 0x7f44000b1c90 con 0x5637f490cbc0 2024-05-10T11:20:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.693+0000 7f4429be7b80 1 -- 172.21.15.27:0/1042649673 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4408010e90 con 0x5637f490cbc0 2024-05-10T11:20:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.693+0000 7f4406000700 1 -- 172.21.15.27:0/1042649673 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f44000ca050 con 0x5637f490cbc0 2024-05-10T11:20:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.693+0000 7f4429be7b80 10 client.26296 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.693+0000 7f4429be7b80 1 -- 172.21.15.27:0/1042649673 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5637f491e760 con 0x5637f490cbc0 2024-05-10T11:20:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.694+0000 7f4406000700 1 -- 172.21.15.27:0/1042649673 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4408010e90 con 0x5637f490cbc0 2024-05-10T11:20:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.694+0000 7f4406000700 1 -- 172.21.15.27:0/1042649673 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f44000ad0d0 con 0x5637f490cbc0 2024-05-10T11:20:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.694+0000 7f4429be7b80 10 client.26296 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.694+0000 7f4429be7b80 10 client.26296 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:20:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.694+0000 7f4429be7b80 20 client.26296 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.694+0000 7f4429be7b80 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5637f4920da0 0x5637f4923190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.694+0000 7f4429be7b80 4 client.26296 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:20:07.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.694+0000 7f4429be7b80 1 -- 172.21.15.27:0/1042649673 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637f48d4140 con 0x5637f4920da0 2024-05-10T11:20:07.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.696+0000 7f4417e00700 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5637f4920da0 0x5637f4923190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.697+0000 7f4406000700 10 client.26296 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:20:07.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.697+0000 7f4417e00700 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5637f4920da0 0x5637f4923190 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=314a99646c0f7a69 server_cookie=cd06efe8d0c8f726 in_seq=0 out_seq=0 2024-05-10T11:20:07.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.698+0000 7f4406000700 1 -- 172.21.15.27:0/1042649673 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5637f48d4140 con 0x5637f4920da0 2024-05-10T11:20:07.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.698+0000 7f4406000700 10 client.26296 handle_command_reply: tid=0 2024-05-10T11:20:07.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.698+0000 7f4429be7b80 1 client.26296 shutdown 2024-05-10T11:20:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.700+0000 7f4429be7b80 1 -- 172.21.15.27:0/1042649673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 msgr2=0x5637f4910540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.700+0000 7f4429be7b80 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f4910540 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f44000df520 tx=0x7f44000df970 comp rx=0 tx=0).stop 2024-05-10T11:20:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.701+0000 7f4429be7b80 1 -- 172.21.15.27:0/1042649673 shutdown_connections 2024-05-10T11:20:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.701+0000 7f4429be7b80 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5637f4920da0 0x5637f4923190 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.701+0000 7f4406000700 0 client.26296 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:20:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.701+0000 7f4429be7b80 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f4910a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.701+0000 7f4429be7b80 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f49131a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.701+0000 7f4429be7b80 1 --2- 172.21.15.27:0/1042649673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f4910540 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.701+0000 7f4429be7b80 1 -- 172.21.15.27:0/1042649673 >> 172.21.15.27:0/1042649673 conn(0x5637f487f6d0 msgr2=0x5637f4901190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:07.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.702+0000 7f4429be7b80 1 -- 172.21.15.27:0/1042649673 shutdown_connections 2024-05-10T11:20:07.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.702+0000 7f4429be7b80 1 -- 172.21.15.27:0/1042649673 wait complete. 2024-05-10T11:20:07.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.702+0000 7f4429be7b80 20 client.26296 trim_cache size 0 max 16384 2024-05-10T11:20:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.720+0000 7f4429be7b80 1 Processor -- start 2024-05-10T11:20:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.720+0000 7f4429be7b80 1 -- start start 2024-05-10T11:20:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.721+0000 7f4429be7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f49072d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.721+0000 7f4429be7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f490f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.721+0000 7f4429be7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f490e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.721+0000 7f4429be7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f44181be3f0 con 0x5637f490d4e0 2024-05-10T11:20:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.721+0000 7f4429be7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f44181bff30 con 0x5637f49078f0 2024-05-10T11:20:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.721+0000 7f4429be7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f44181beac0 con 0x5637f490cbc0 2024-05-10T11:20:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.721+0000 7f4417e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f49072d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.721+0000 7f440f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f490f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.721+0000 7f4417e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f49072d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51342/0 (socket says 172.21.15.27:51342) 2024-05-10T11:20:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.721+0000 7f441d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f490e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.721+0000 7f440f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f490f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55286/0 (socket says 172.21.15.27:55286) 2024-05-10T11:20:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.721+0000 7f4417e00700 1 -- 172.21.15.27:0/3371719207 learned_addr learned my addr 172.21.15.27:0/3371719207 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.722+0000 7f4417e00700 1 -- 172.21.15.27:0/3371719207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 msgr2=0x5637f490f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.722+0000 7f4417e00700 1 --2- 172.21.15.27:0/3371719207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f490f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.722+0000 7f4417e00700 1 -- 172.21.15.27:0/3371719207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 msgr2=0x5637f490e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.722+0000 7f4417e00700 1 --2- 172.21.15.27:0/3371719207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f490e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.722+0000 7f4417e00700 1 -- 172.21.15.27:0/3371719207 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637f4923500 con 0x5637f49078f0 2024-05-10T11:20:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.722+0000 7f440f400700 1 --2- 172.21.15.27:0/3371719207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f490f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:20:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.722+0000 7f4417e00700 1 --2- 172.21.15.27:0/3371719207 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f49072d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4408011740 tx=0x7f440801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6143b991c58cd1d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.722+0000 7f440cc00700 1 -- 172.21.15.27:0/3371719207 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408039070 con 0x5637f49078f0 2024-05-10T11:20:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.722+0000 7f440cc00700 1 -- 172.21.15.27:0/3371719207 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4408034e50 con 0x5637f49078f0 2024-05-10T11:20:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.722+0000 7f440cc00700 1 -- 172.21.15.27:0/3371719207 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44080117e0 con 0x5637f49078f0 2024-05-10T11:20:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.723+0000 7f4429be7b80 1 -- 172.21.15.27:0/3371719207 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 msgr2=0x5637f49072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.723+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3371719207 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f49072d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4408011740 tx=0x7f440801d000 comp rx=0 tx=0).stop 2024-05-10T11:20:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.723+0000 7f4429be7b80 1 -- 172.21.15.27:0/3371719207 shutdown_connections 2024-05-10T11:20:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.723+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3371719207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f490d4e0 0x5637f490e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.723+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3371719207 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f49078f0 0x5637f49072d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.723+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3371719207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490cbc0 0x5637f490f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.723+0000 7f4429be7b80 1 -- 172.21.15.27:0/3371719207 >> 172.21.15.27:0/3371719207 conn(0x5637f487f6d0 msgr2=0x5637f490b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.723+0000 7f4429be7b80 1 -- 172.21.15.27:0/3371719207 shutdown_connections 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.724+0000 7f4429be7b80 1 -- 172.21.15.27:0/3371719207 wait complete. 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.724+0000 7f4429be7b80 1 -- start start 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.724+0000 7f4429be7b80 1 Processor -- start 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.725+0000 7f4429be7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f490cbc0 0x5637f49195d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.725+0000 7f4429be7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490d4e0 0x5637f4927dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.725+0000 7f4429be7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f4928310 0x5637f4928790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.725+0000 7f4429be7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f44181bf860 con 0x5637f4928310 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.725+0000 7f4429be7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f44181bf190 con 0x5637f490cbc0 2024-05-10T11:20:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.725+0000 7f4429be7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f44181bc3b0 con 0x5637f490d4e0 2024-05-10T11:20:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.725+0000 7f4429be7b80 10 client.0 _ll_register_callbacks cb 0x5637f4800c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.725+0000 7f440f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490d4e0 0x5637f4927dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.725+0000 7f4417e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f490cbc0 0x5637f49195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.725+0000 7f4417e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f490cbc0 0x5637f49195d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51358/0 (socket says 172.21.15.27:51358) 2024-05-10T11:20:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.725+0000 7f4417e00700 1 -- 172.21.15.27:0/3675577948 learned_addr learned my addr 172.21.15.27:0/3675577948 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.725+0000 7f441d200700 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f4928310 0x5637f4928790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.726+0000 7f440f400700 1 -- 172.21.15.27:0/3675577948 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f490cbc0 msgr2=0x5637f49195d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.726+0000 7f440f400700 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f490cbc0 0x5637f49195d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.726+0000 7f440f400700 1 -- 172.21.15.27:0/3675577948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f4928310 msgr2=0x5637f4928790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.726+0000 7f440f400700 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f4928310 0x5637f4928790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.726+0000 7f440f400700 1 -- 172.21.15.27:0/3675577948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637f498a590 con 0x5637f490d4e0 2024-05-10T11:20:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.726+0000 7f4417e00700 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f490cbc0 0x5637f49195d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:20:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.726+0000 7f441d200700 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f4928310 0x5637f4928790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:20:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.726+0000 7f440f400700 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490d4e0 0x5637f4927dd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4400012da0 tx=0x7f44000a8e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5f050a410be0338 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.726+0000 7f4406000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:20:07.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.726+0000 7f4406000700 1 -- 172.21.15.27:0/3675577948 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44000dfe50 con 0x5637f490d4e0 2024-05-10T11:20:07.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.726+0000 7f4406000700 1 -- 172.21.15.27:0/3675577948 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9679+0+0 (secure 0 0 0) 0x7f44000a8310 con 0x5637f490d4e0 2024-05-10T11:20:07.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.726+0000 7f4429be7b80 1 -- 172.21.15.27:0/3675577948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4408034e50 con 0x5637f490d4e0 2024-05-10T11:20:07.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.727+0000 7f4406000700 1 -- 172.21.15.27:0/3675577948 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4400087050 con 0x5637f490d4e0 2024-05-10T11:20:07.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.727+0000 7f4429be7b80 10 client.26302 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:07.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.727+0000 7f4429be7b80 1 -- 172.21.15.27:0/3675577948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5637f498b750 con 0x5637f490d4e0 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.727+0000 7f4406000700 1 -- 172.21.15.27:0/3675577948 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4408034e50 con 0x5637f490d4e0 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.727+0000 7f4406000700 1 -- 172.21.15.27:0/3675577948 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f44000a7d80 con 0x5637f490d4e0 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.727+0000 7f4429be7b80 10 client.26302 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.727+0000 7f4429be7b80 10 client.26302 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.727+0000 7f4429be7b80 20 client.26302 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.727+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5637f498be00 0x5637f498e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.727+0000 7f4429be7b80 4 client.26302 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.727+0000 7f4429be7b80 1 -- 172.21.15.27:0/3675577948 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5637f4867400 con 0x5637f498be00 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.728+0000 7f4417e00700 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5637f498be00 0x5637f498e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.728+0000 7f4417e00700 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5637f498be00 0x5637f498e1f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a39b807f5db6d6b8 server_cookie=9222b174fed88c43 in_seq=0 out_seq=0 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.728+0000 7f4406000700 10 client.26302 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.728+0000 7f4406000700 1 -- 172.21.15.27:0/3675577948 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5637f4867400 con 0x5637f498be00 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.728+0000 7f4406000700 10 client.26302 handle_command_reply: tid=0 2024-05-10T11:20:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4429be7b80 1 client.26302 shutdown 2024-05-10T11:20:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4429be7b80 1 -- 172.21.15.27:0/3675577948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490d4e0 msgr2=0x5637f4927dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490d4e0 0x5637f4927dd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4400012da0 tx=0x7f44000a8e80 comp rx=0 tx=0).stop 2024-05-10T11:20:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4429be7b80 1 -- 172.21.15.27:0/3675577948 shutdown_connections 2024-05-10T11:20:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5637f498be00 0x5637f498e1f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637f4928310 0x5637f4928790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637f490cbc0 0x5637f49195d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4429be7b80 1 --2- 172.21.15.27:0/3675577948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5637f490d4e0 0x5637f4927dd0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4406000700 0 client.26302 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:20:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4429be7b80 1 -- 172.21.15.27:0/3675577948 >> 172.21.15.27:0/3675577948 conn(0x5637f487f6d0 msgr2=0x5637f490bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4429be7b80 1 -- 172.21.15.27:0/3675577948 shutdown_connections 2024-05-10T11:20:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4429be7b80 1 -- 172.21.15.27:0/3675577948 wait complete. 2024-05-10T11:20:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.729+0000 7f4429be7b80 20 client.26302 trim_cache size 0 max 16384 2024-05-10T11:20:07.735 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:20:07.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.730+0000 7f441f200700 1 -- 172.21.15.27:0/1548593789 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f43fc0b29d0 msgr2=0x7f43fc0b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:07.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.730+0000 7f441f200700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f43fc0b29d0 0x7f43fc0b4e90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4408013970 tx=0x7f440800aba0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.730+0000 7f441f200700 1 -- 172.21.15.27:0/1548593789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4418111c80 msgr2=0x7f44181aeb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.730+0000 7f441f200700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4418111c80 0x7f44181aeb60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4400007e20 tx=0x7f440000a340 comp rx=0 tx=0).stop 2024-05-10T11:20:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.730+0000 7f441f200700 1 -- 172.21.15.27:0/1548593789 shutdown_connections 2024-05-10T11:20:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.730+0000 7f441f200700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f43fc0b29d0 0x7f43fc0b4e90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.730+0000 7f441f200700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441811cc80 0x7f44181a86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.730+0000 7f441f200700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f441810a9e0 0x7f44181ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.730+0000 7f441f200700 1 --2- 172.21.15.27:0/1548593789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4418111c80 0x7f44181aeb60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.730+0000 7f441f200700 1 -- 172.21.15.27:0/1548593789 >> 172.21.15.27:0/1548593789 conn(0x7f441806e610 msgr2=0x7f441811ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.730+0000 7f441f200700 1 -- 172.21.15.27:0/1548593789 shutdown_connections 2024-05-10T11:20:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:07.730+0000 7f441f200700 1 -- 172.21.15.27:0/1548593789 wait complete. 2024-05-10T11:20:08.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:20:08.688 INFO:tasks.workunit.client.0.smithi027.stdout: 1 388325 6.88 MB/sec execute 173 sec latency 41.155 ms 2024-05-10T11:20:08.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:08 smithi027 ceph-mon[30054]: pgmap v211: 129 pgs: 129 active+clean; 230 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-05-10T11:20:08.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:08 smithi027 ceph-mon[30054]: osdmap e93: 12 total, 12 up, 12 in 2024-05-10T11:20:08.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:08 smithi076 ceph-mon[34152]: pgmap v211: 129 pgs: 129 active+clean; 230 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-05-10T11:20:08.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:08 smithi076 ceph-mon[34152]: osdmap e93: 12 total, 12 up, 12 in 2024-05-10T11:20:08.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:08 smithi189 ceph-mon[34842]: pgmap v211: 129 pgs: 129 active+clean; 230 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-05-10T11:20:08.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:08 smithi189 ceph-mon[34842]: osdmap e93: 12 total, 12 up, 12 in 2024-05-10T11:20:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:20:09.689 INFO:tasks.workunit.client.0.smithi027.stdout: 1 390209 6.91 MB/sec execute 174 sec latency 29.276 ms 2024-05-10T11:20:10.688 INFO:tasks.workunit.client.0.smithi027.stdout: 1 390780 6.88 MB/sec execute 175 sec latency 21.026 ms 2024-05-10T11:20:10.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:10 smithi076 ceph-mon[34152]: pgmap v213: 129 pgs: 129 active+clean; 235 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 249 op/s 2024-05-10T11:20:10.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:10 smithi189 ceph-mon[34842]: pgmap v213: 129 pgs: 129 active+clean; 235 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 249 op/s 2024-05-10T11:20:11.038 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:10 smithi027 ceph-mon[30054]: pgmap v213: 129 pgs: 129 active+clean; 235 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 249 op/s 2024-05-10T11:20:11.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.688+0000 7ff3ff200700 1 -- 172.21.15.27:0/1658604206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f806c3d0 msgr2=0x7ff3f8110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.688+0000 7ff3ff200700 1 --2- 172.21.15.27:0/1658604206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f806c3d0 0x7ff3f8110320 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000b0a0 tx=0x7ff3f000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.689+0000 7ff3ff200700 1 -- 172.21.15.27:0/1658604206 shutdown_connections 2024-05-10T11:20:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.689+0000 7ff3ff200700 1 --2- 172.21.15.27:0/1658604206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f806c3d0 0x7ff3f8110320 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.689+0000 7ff3ff200700 1 --2- 172.21.15.27:0/1658604206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f8109500 0x7ff3f81098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.689+0000 7ff3ff200700 1 --2- 172.21.15.27:0/1658604206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3f806ba10 0x7ff3f806be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.689+0000 7ff3ff200700 1 -- 172.21.15.27:0/1658604206 >> 172.21.15.27:0/1658604206 conn(0x7ff3f8079310 msgr2=0x7ff3f8079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.689+0000 7ff3ff200700 1 -- 172.21.15.27:0/1658604206 shutdown_connections 2024-05-10T11:20:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.689+0000 7ff3ff200700 1 -- 172.21.15.27:0/1658604206 wait complete. 2024-05-10T11:20:11.689 INFO:tasks.workunit.client.0.smithi027.stdout: 1 391923 6.87 MB/sec execute 176 sec latency 34.163 ms 2024-05-10T11:20:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.690+0000 7ff3ff200700 1 Processor -- start 2024-05-10T11:20:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.691+0000 7ff3ff200700 1 -- start start 2024-05-10T11:20:11.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.691+0000 7ff3ff200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3f806ba10 0x7ff3f819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.691+0000 7ff3ff200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f806c3d0 0x7ff3f819c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.691+0000 7ff3ff200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f8109500 0x7ff3f81a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.691+0000 7ff3ff200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f8113660 con 0x7ff3f8109500 2024-05-10T11:20:11.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.691+0000 7ff3ff200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f81134e0 con 0x7ff3f806ba10 2024-05-10T11:20:11.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.691+0000 7ff3ff200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f81137e0 con 0x7ff3f806c3d0 2024-05-10T11:20:11.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.692+0000 7ff3f7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3f806ba10 0x7ff3f819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.692+0000 7ff3efe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f806c3d0 0x7ff3f819c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.692+0000 7ff3f7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3f806ba10 0x7ff3f819bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51374/0 (socket says 172.21.15.27:51374) 2024-05-10T11:20:11.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.692+0000 7ff3f7e00700 1 -- 172.21.15.27:0/2007973086 learned_addr learned my addr 172.21.15.27:0/2007973086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:11.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.692+0000 7ff3fd200700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f8109500 0x7ff3f81a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.692+0000 7ff3f7e00700 1 -- 172.21.15.27:0/2007973086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f806c3d0 msgr2=0x7ff3f819c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:11.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.692+0000 7ff3f7e00700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f806c3d0 0x7ff3f819c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.692+0000 7ff3f7e00700 1 -- 172.21.15.27:0/2007973086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f8109500 msgr2=0x7ff3f81a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:11.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.692+0000 7ff3f7e00700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f8109500 0x7ff3f81a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.692+0000 7ff3f7e00700 1 -- 172.21.15.27:0/2007973086 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f81a0d20 con 0x7ff3f806ba10 2024-05-10T11:20:11.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.692+0000 7ff3fd200700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f8109500 0x7ff3f81a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:20:11.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.692+0000 7ff3efe00700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f806c3d0 0x7ff3f819c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:20:11.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.693+0000 7ff3f7e00700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3f806ba10 0x7ff3f819bd10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800e9a0 tx=0x7ff3e800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:11.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.693+0000 7ff3f5600700 1 -- 172.21.15.27:0/2007973086 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e800cbd0 con 0x7ff3f806ba10 2024-05-10T11:20:11.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.693+0000 7ff3f5600700 1 -- 172.21.15.27:0/2007973086 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3e800cd70 con 0x7ff3f806ba10 2024-05-10T11:20:11.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.693+0000 7ff3ff200700 1 -- 172.21.15.27:0/2007973086 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f81a1010 con 0x7ff3f806ba10 2024-05-10T11:20:11.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.693+0000 7ff3f5600700 1 -- 172.21.15.27:0/2007973086 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8018680 con 0x7ff3f806ba10 2024-05-10T11:20:11.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.693+0000 7ff3ff200700 1 -- 172.21.15.27:0/2007973086 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f81a8890 con 0x7ff3f806ba10 2024-05-10T11:20:11.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.697+0000 7ff3f5600700 1 -- 172.21.15.27:0/2007973086 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff3e8010430 con 0x7ff3f806ba10 2024-05-10T11:20:11.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.698+0000 7ff3f5600700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff3dc0b2a70 0x7ff3dc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.699+0000 7ff3f5600700 1 -- 172.21.15.27:0/2007973086 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7ff3e80e2c50 con 0x7ff3f806ba10 2024-05-10T11:20:11.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.699+0000 7ff409bf7b80 1 Processor -- start 2024-05-10T11:20:11.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.699+0000 7ff3efe00700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff3dc0b2a70 0x7ff3dc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.699+0000 7ff409bf7b80 1 -- start start 2024-05-10T11:20:11.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.700+0000 7ff409bf7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb01fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.700+0000 7ff409bf7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 0x55c90cafc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.700+0000 7ff409bf7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cafc8f0 0x55c90cafcd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.700+0000 7ff409bf7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c90ca4ea20 con 0x55c90cb01bc0 2024-05-10T11:20:11.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.700+0000 7ff409bf7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c90c9ec760 con 0x55c90cb024e0 2024-05-10T11:20:11.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.700+0000 7ff409bf7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c90ca4a9e0 con 0x55c90cafc8f0 2024-05-10T11:20:11.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.700+0000 7ff3efe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 0x55c90cafc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.700+0000 7ff3f7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb01fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.700+0000 7ff3fd200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cafc8f0 0x55c90cafcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.700+0000 7ff3fd200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cafc8f0 0x55c90cafcd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55320/0 (socket says 172.21.15.27:55320) 2024-05-10T11:20:11.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.700+0000 7ff3fd200700 1 -- 172.21.15.27:0/189923878 learned_addr learned my addr 172.21.15.27:0/189923878 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:11.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.700+0000 7ff3efe00700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff3dc0b2a70 0x7ff3dc0b4f30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8104490 tx=0x7ff3e0009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:11.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.701+0000 7ff3efe00700 1 -- 172.21.15.27:0/189923878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cafc8f0 msgr2=0x55c90cafcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:11.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.701+0000 7ff3efe00700 1 --2- 172.21.15.27:0/189923878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cafc8f0 0x55c90cafcd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.701+0000 7ff3efe00700 1 -- 172.21.15.27:0/189923878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 msgr2=0x55c90cb01fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:11.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.701+0000 7ff3efe00700 1 --2- 172.21.15.27:0/189923878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb01fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.701+0000 7ff3efe00700 1 -- 172.21.15.27:0/189923878 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c90c99b8d0 con 0x55c90cb024e0 2024-05-10T11:20:11.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.701+0000 7ff3efe00700 1 --2- 172.21.15.27:0/189923878 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 0x55c90cafc2d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0022d20 tx=0x7ff3e0014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20609fb5cda8af79 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:11.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.701+0000 7ff3ed600700 1 -- 172.21.15.27:0/189923878 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0030070 con 0x55c90cb024e0 2024-05-10T11:20:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.702+0000 7ff3ed600700 1 -- 172.21.15.27:0/189923878 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3e0003680 con 0x55c90cb024e0 2024-05-10T11:20:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.702+0000 7ff3ed600700 1 -- 172.21.15.27:0/189923878 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0022330 con 0x55c90cb024e0 2024-05-10T11:20:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.702+0000 7ff409bf7b80 1 -- 172.21.15.27:0/189923878 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 msgr2=0x55c90cafc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.702+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/189923878 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 0x55c90cafc2d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0022d20 tx=0x7ff3e0014000 comp rx=0 tx=0).stop 2024-05-10T11:20:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.703+0000 7ff409bf7b80 1 -- 172.21.15.27:0/189923878 shutdown_connections 2024-05-10T11:20:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.703+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/189923878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cafc8f0 0x55c90cafcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.703+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/189923878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb01fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.703+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/189923878 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 0x55c90cafc2d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.703+0000 7ff409bf7b80 1 -- 172.21.15.27:0/189923878 >> 172.21.15.27:0/189923878 conn(0x55c90ca746d0 msgr2=0x55c90caf6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.703+0000 7ff409bf7b80 1 -- 172.21.15.27:0/189923878 shutdown_connections 2024-05-10T11:20:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.703+0000 7ff409bf7b80 1 -- 172.21.15.27:0/189923878 wait complete. 2024-05-10T11:20:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.703+0000 7ff409bf7b80 1 -- start start 2024-05-10T11:20:11.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.704+0000 7ff409bf7b80 1 Processor -- start 2024-05-10T11:20:11.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.704+0000 7ff409bf7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cafc8f0 0x55c90cb081c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.704+0000 7ff409bf7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb01bc0 0x55c90cb05560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.704+0000 7ff409bf7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 0x55c90cb05aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.704+0000 7ff409bf7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c90cb08f20 con 0x55c90cafc8f0 2024-05-10T11:20:11.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.704+0000 7ff409bf7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c90c9b4b20 con 0x55c90cb01bc0 2024-05-10T11:20:11.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.704+0000 7ff409bf7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c90cb090a0 con 0x55c90cb024e0 2024-05-10T11:20:11.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.704+0000 7ff409bf7b80 10 client.0 _ll_register_callbacks cb 0x55c90c9f5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3efe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb01bc0 0x55c90cb05560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3fd200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 0x55c90cb05aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3efe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb01bc0 0x55c90cb05560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51400/0 (socket says 172.21.15.27:51400) 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3fd200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 0x55c90cb05aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55330/0 (socket says 172.21.15.27:55330) 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3efe00700 1 -- 172.21.15.27:0/1015044579 learned_addr learned my addr 172.21.15.27:0/1015044579 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3f7e00700 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cafc8f0 0x55c90cb081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3efe00700 1 -- 172.21.15.27:0/1015044579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 msgr2=0x55c90cb05aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3efe00700 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 0x55c90cb05aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3efe00700 1 -- 172.21.15.27:0/1015044579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cafc8f0 msgr2=0x55c90cb081c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3efe00700 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cafc8f0 0x55c90cb081c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3efe00700 1 -- 172.21.15.27:0/1015044579 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c90ca14420 con 0x55c90cb01bc0 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3fd200700 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 0x55c90cb05aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3f7e00700 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cafc8f0 0x55c90cb081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:20:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3efe00700 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb01bc0 0x55c90cb05560 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff3e002e040 tx=0x7ff3e0003a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8edc0e5de6f89f41 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:11.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.705+0000 7ff3e6000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:20:11.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.706+0000 7ff3e6000700 1 -- 172.21.15.27:0/1015044579 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0039060 con 0x55c90cb01bc0 2024-05-10T11:20:11.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.706+0000 7ff3e6000700 1 -- 172.21.15.27:0/1015044579 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7ff3e0028040 con 0x55c90cb01bc0 2024-05-10T11:20:11.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.706+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1015044579 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3e0003680 con 0x55c90cb01bc0 2024-05-10T11:20:11.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.706+0000 7ff3e6000700 1 -- 172.21.15.27:0/1015044579 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff3e0040050 con 0x55c90cb01bc0 2024-05-10T11:20:11.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.706+0000 7ff409bf7b80 10 client.26330 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:11.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.706+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1015044579 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c90cb137b0 con 0x55c90cb01bc0 2024-05-10T11:20:11.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.706+0000 7ff3e6000700 1 -- 172.21.15.27:0/1015044579 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3e0003680 con 0x55c90cb01bc0 2024-05-10T11:20:11.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.707+0000 7ff3e6000700 1 -- 172.21.15.27:0/1015044579 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff3e002b5d0 con 0x55c90cb01bc0 2024-05-10T11:20:11.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.707+0000 7ff409bf7b80 10 client.26330 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:11.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.707+0000 7ff409bf7b80 10 client.26330 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:20:11.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.707+0000 7ff409bf7b80 20 client.26330 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:11.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.707+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55c90cb15d90 0x55c90cb18180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.707+0000 7ff409bf7b80 4 client.26330 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.707+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1015044579 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c90cac9140 con 0x55c90cb15d90 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.707+0000 7ff3f7e00700 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55c90cb15d90 0x55c90cb18180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.708+0000 7ff3f7e00700 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55c90cb15d90 0x55c90cb18180 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ab02fb91f5b7e37 server_cookie=3f40dd3a79f5a697 in_seq=0 out_seq=0 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.708+0000 7ff3e6000700 10 client.26330 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff3e6000700 1 -- 172.21.15.27:0/1015044579 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c90cac9140 con 0x55c90cb15d90 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff3e6000700 10 client.26330 handle_command_reply: tid=0 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff409bf7b80 1 client.26330 shutdown 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1015044579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb01bc0 msgr2=0x55c90cb05560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb01bc0 0x55c90cb05560 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff3e002e040 tx=0x7ff3e0003a60 comp rx=0 tx=0).stop 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1015044579 shutdown_connections 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55c90cb15d90 0x55c90cb18180 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 0x55c90cb05aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cafc8f0 0x55c90cb081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1015044579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb01bc0 0x55c90cb05560 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1015044579 >> 172.21.15.27:0/1015044579 conn(0x55c90ca746d0 msgr2=0x55c90caf6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff3e6000700 0 client.26330 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:20:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1015044579 shutdown_connections 2024-05-10T11:20:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1015044579 wait complete. 2024-05-10T11:20:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.709+0000 7ff409bf7b80 20 client.26330 trim_cache size 0 max 16384 2024-05-10T11:20:11.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.728+0000 7ff409bf7b80 1 Processor -- start 2024-05-10T11:20:11.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.729+0000 7ff409bf7b80 1 -- start start 2024-05-10T11:20:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.729+0000 7ff409bf7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cafc8f0 0x55c90cafc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.729+0000 7ff409bf7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb04560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.729+0000 7ff409bf7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 0x55c90caf6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.729+0000 7ff409bf7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f81ad580 con 0x55c90cb01bc0 2024-05-10T11:20:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.729+0000 7ff409bf7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f81af0c0 con 0x55c90cafc8f0 2024-05-10T11:20:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.729+0000 7ff409bf7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f81adc50 con 0x55c90cb024e0 2024-05-10T11:20:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.729+0000 7ff3f7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cafc8f0 0x55c90cafc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.729+0000 7ff3f7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cafc8f0 0x55c90cafc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51408/0 (socket says 172.21.15.27:51408) 2024-05-10T11:20:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.729+0000 7ff3f7e00700 1 -- 172.21.15.27:0/1291848086 learned_addr learned my addr 172.21.15.27:0/1291848086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.729+0000 7ff3fd200700 1 --2- 172.21.15.27:0/1291848086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 0x55c90caf6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.729+0000 7ff3efe00700 1 --2- 172.21.15.27:0/1291848086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb04560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.730+0000 7ff3fd200700 1 -- 172.21.15.27:0/1291848086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cafc8f0 msgr2=0x55c90cafc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:11.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.730+0000 7ff3fd200700 1 --2- 172.21.15.27:0/1291848086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cafc8f0 0x55c90cafc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.730+0000 7ff3fd200700 1 -- 172.21.15.27:0/1291848086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 msgr2=0x55c90cb04560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.730+0000 7ff3fd200700 1 --2- 172.21.15.27:0/1291848086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb04560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.730+0000 7ff3fd200700 1 -- 172.21.15.27:0/1291848086 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c90cb021f0 con 0x55c90cb024e0 2024-05-10T11:20:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.730+0000 7ff3efe00700 1 --2- 172.21.15.27:0/1291848086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb04560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:20:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.730+0000 7ff3fd200700 1 --2- 172.21.15.27:0/1291848086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 0x55c90caf6910 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x55c90cafd2b0 tx=0x7ff3f000bd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2113eeed4306fcbc server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.731+0000 7ff3ecc00700 1 -- 172.21.15.27:0/1291848086 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0029020 con 0x55c90cb024e0 2024-05-10T11:20:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.731+0000 7ff3ecc00700 1 -- 172.21.15.27:0/1291848086 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f000be00 con 0x55c90cb024e0 2024-05-10T11:20:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.731+0000 7ff3ecc00700 1 -- 172.21.15.27:0/1291848086 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0007e60 con 0x55c90cb024e0 2024-05-10T11:20:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.731+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1291848086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 msgr2=0x55c90caf6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.731+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1291848086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 0x55c90caf6910 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x55c90cafd2b0 tx=0x7ff3f000bd10 comp rx=0 tx=0).stop 2024-05-10T11:20:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.731+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1291848086 shutdown_connections 2024-05-10T11:20:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.731+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1291848086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cb024e0 0x55c90caf6910 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.731+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1291848086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cafc8f0 0x55c90cafc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.731+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1291848086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb04560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.731+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1291848086 >> 172.21.15.27:0/1291848086 conn(0x55c90ca746d0 msgr2=0x55c90cb03e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.732+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1291848086 shutdown_connections 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.732+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1291848086 wait complete. 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.732+0000 7ff409bf7b80 1 -- start start 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.732+0000 7ff409bf7b80 1 Processor -- start 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.733+0000 7ff409bf7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cafc8f0 0x55c90cb181a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.733+0000 7ff409bf7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb186e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.733+0000 7ff409bf7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 0x55c90cb7e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.733+0000 7ff409bf7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f81ae9f0 con 0x55c90cb01bc0 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.733+0000 7ff409bf7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f81ae320 con 0x55c90cb024e0 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.733+0000 7ff409bf7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f81ab540 con 0x55c90cafc8f0 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.733+0000 7ff409bf7b80 10 client.0 _ll_register_callbacks cb 0x55c90c9f5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.733+0000 7ff3fd200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 0x55c90cb7e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.733+0000 7ff3fd200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 0x55c90cb7e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51412/0 (socket says 172.21.15.27:51412) 2024-05-10T11:20:11.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.733+0000 7ff3fd200700 1 -- 172.21.15.27:0/1964919978 learned_addr learned my addr 172.21.15.27:0/1964919978 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:11.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.733+0000 7ff3f7e00700 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cafc8f0 0x55c90cb181a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.733+0000 7ff3efe00700 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb186e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.734+0000 7ff3fd200700 1 -- 172.21.15.27:0/1964919978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cafc8f0 msgr2=0x55c90cb181a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:11.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.734+0000 7ff3fd200700 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cafc8f0 0x55c90cb181a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.734+0000 7ff3fd200700 1 -- 172.21.15.27:0/1964919978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 msgr2=0x55c90cb186e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.734+0000 7ff3fd200700 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb186e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.734+0000 7ff3fd200700 1 -- 172.21.15.27:0/1964919978 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c90cb7e8b0 con 0x55c90cb024e0 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.734+0000 7ff3fd200700 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 0x55c90cb7e2a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000d010 tx=0x7ff3f0005360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79db77a987ae18bc server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.734+0000 7ff3e6000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.734+0000 7ff3e6000700 1 -- 172.21.15.27:0/1964919978 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0033060 con 0x55c90cb024e0 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.734+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1964919978 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3f000be00 con 0x55c90cb024e0 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.735+0000 7ff3e6000700 1 -- 172.21.15.27:0/1964919978 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7ff3f00074a0 con 0x55c90cb024e0 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.735+0000 7ff3e6000700 1 -- 172.21.15.27:0/1964919978 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff3f0029030 con 0x55c90cb024e0 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.735+0000 7ff409bf7b80 10 client.26336 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.735+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1964919978 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c90cb7fbb0 con 0x55c90cb024e0 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.735+0000 7ff3e6000700 1 -- 172.21.15.27:0/1964919978 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f000be00 con 0x55c90cb024e0 2024-05-10T11:20:11.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.735+0000 7ff3e6000700 1 -- 172.21.15.27:0/1964919978 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff3f0024070 con 0x55c90cb024e0 2024-05-10T11:20:11.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.735+0000 7ff409bf7b80 10 client.26336 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:11.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.735+0000 7ff409bf7b80 10 client.26336 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:20:11.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.735+0000 7ff409bf7b80 20 client.26336 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:11.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.736+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55c90cb801e0 0x55c90cb82600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:11.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.736+0000 7ff409bf7b80 4 client.26336 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:20:11.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.736+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1964919978 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c90ca5c400 con 0x55c90cb801e0 2024-05-10T11:20:11.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.736+0000 7ff3f7e00700 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55c90cb801e0 0x55c90cb82600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:11.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.736+0000 7ff3f7e00700 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55c90cb801e0 0x55c90cb82600 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bfab7a08a706b366 server_cookie=d7f36b7c7c61357d in_seq=0 out_seq=0 2024-05-10T11:20:11.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.736+0000 7ff3e6000700 10 client.26336 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:20:11.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff3e6000700 1 -- 172.21.15.27:0/1964919978 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c90ca5c400 con 0x55c90cb801e0 2024-05-10T11:20:11.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff3e6000700 10 client.26336 handle_command_reply: tid=0 2024-05-10T11:20:11.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff409bf7b80 1 client.26336 shutdown 2024-05-10T11:20:11.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1964919978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 msgr2=0x55c90cb7e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:11.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 0x55c90cb7e2a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000d010 tx=0x7ff3f0005360 comp rx=0 tx=0).stop 2024-05-10T11:20:11.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1964919978 shutdown_connections 2024-05-10T11:20:11.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55c90cb801e0 0x55c90cb82600 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c90cafc8f0 0x55c90cb181a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c90cb024e0 0x55c90cb7e2a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff409bf7b80 1 --2- 172.21.15.27:0/1964919978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c90cb01bc0 0x55c90cb186e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff3e6000700 0 client.26336 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:20:11.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1964919978 >> 172.21.15.27:0/1964919978 conn(0x55c90ca746d0 msgr2=0x55c90cb00f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:11.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1964919978 shutdown_connections 2024-05-10T11:20:11.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff409bf7b80 1 -- 172.21.15.27:0/1964919978 wait complete. 2024-05-10T11:20:11.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.737+0000 7ff409bf7b80 20 client.26336 trim_cache size 0 max 16384 2024-05-10T11:20:11.740 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:20:11.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.738+0000 7ff3ff200700 1 -- 172.21.15.27:0/2007973086 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff3dc0b2a70 msgr2=0x7ff3dc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:11.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.738+0000 7ff3ff200700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff3dc0b2a70 0x7ff3dc0b4f30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8104490 tx=0x7ff3e0009260 comp rx=0 tx=0).stop 2024-05-10T11:20:11.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.738+0000 7ff3ff200700 1 -- 172.21.15.27:0/2007973086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3f806ba10 msgr2=0x7ff3f819bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:11.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.738+0000 7ff3ff200700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3f806ba10 0x7ff3f819bd10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800e9a0 tx=0x7ff3e800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.738+0000 7ff3ff200700 1 -- 172.21.15.27:0/2007973086 shutdown_connections 2024-05-10T11:20:11.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.738+0000 7ff3ff200700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff3dc0b2a70 0x7ff3dc0b4f30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.738+0000 7ff3ff200700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f8109500 0x7ff3f81a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.738+0000 7ff3ff200700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3f806ba10 0x7ff3f819bd10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.738+0000 7ff3ff200700 1 --2- 172.21.15.27:0/2007973086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f806c3d0 0x7ff3f819c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:11.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.738+0000 7ff3ff200700 1 -- 172.21.15.27:0/2007973086 >> 172.21.15.27:0/2007973086 conn(0x7ff3f8079310 msgr2=0x7ff3f810e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:11.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.739+0000 7ff3ff200700 1 -- 172.21.15.27:0/2007973086 shutdown_connections 2024-05-10T11:20:11.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:11.739+0000 7ff3ff200700 1 -- 172.21.15.27:0/2007973086 wait complete. 2024-05-10T11:20:12.689 INFO:tasks.workunit.client.0.smithi027.stdout: 1 393885 6.90 MB/sec execute 177 sec latency 32.147 ms 2024-05-10T11:20:12.724 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:20:12.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:12 smithi076 ceph-mon[34152]: pgmap v214: 129 pgs: 129 active+clean; 235 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 204 op/s 2024-05-10T11:20:12.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:12 smithi189 ceph-mon[34842]: pgmap v214: 129 pgs: 129 active+clean; 235 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 204 op/s 2024-05-10T11:20:13.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:12 smithi027 ceph-mon[30054]: pgmap v214: 129 pgs: 129 active+clean; 235 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 204 op/s 2024-05-10T11:20:13.267 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:20:13.689 INFO:tasks.workunit.client.0.smithi027.stdout: 1 394506 6.88 MB/sec execute 178 sec latency 35.256 ms 2024-05-10T11:20:14.689 INFO:tasks.workunit.client.0.smithi027.stdout: 1 395915 6.87 MB/sec execute 179 sec latency 23.391 ms 2024-05-10T11:20:14.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:14 smithi027 ceph-mon[30054]: pgmap v215: 129 pgs: 129 active+clean; 235 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 199 op/s 2024-05-10T11:20:14.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:14 smithi076 ceph-mon[34152]: pgmap v215: 129 pgs: 129 active+clean; 235 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 199 op/s 2024-05-10T11:20:14.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:14 smithi189 ceph-mon[34842]: pgmap v215: 129 pgs: 129 active+clean; 235 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 199 op/s 2024-05-10T11:20:15.689 INFO:tasks.workunit.client.0.smithi027.stdout: 1 397650 6.90 MB/sec execute 180 sec latency 18.968 ms 2024-05-10T11:20:15.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.724+0000 7ff419000700 1 -- 172.21.15.27:0/2521318335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4141055d0 msgr2=0x7ff414105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:15.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.724+0000 7ff419000700 1 --2- 172.21.15.27:0/2521318335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4141055d0 0x7ff414105a50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff400009990 tx=0x7ff400009ca0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.724+0000 7ff419000700 1 -- 172.21.15.27:0/2521318335 shutdown_connections 2024-05-10T11:20:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.724+0000 7ff419000700 1 --2- 172.21.15.27:0/2521318335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff414105f90 0x7ff4141124a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.724+0000 7ff419000700 1 --2- 172.21.15.27:0/2521318335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff41410b660 0x7ff41410ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.724+0000 7ff419000700 1 --2- 172.21.15.27:0/2521318335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff4141055d0 0x7ff414105a50 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.724+0000 7ff419000700 1 -- 172.21.15.27:0/2521318335 >> 172.21.15.27:0/2521318335 conn(0x7ff4141010c0 msgr2=0x7ff4141034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.725+0000 7ff419000700 1 -- 172.21.15.27:0/2521318335 shutdown_connections 2024-05-10T11:20:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.725+0000 7ff419000700 1 -- 172.21.15.27:0/2521318335 wait complete. 2024-05-10T11:20:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.726+0000 7ff419000700 1 Processor -- start 2024-05-10T11:20:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.726+0000 7ff419000700 1 -- start start 2024-05-10T11:20:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.726+0000 7ff419000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4141055d0 0x7ff4141a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.726+0000 7ff419000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff414105f90 0x7ff4141a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.726+0000 7ff419000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff41410b660 0x7ff41419ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.726+0000 7ff419000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff414115820 con 0x7ff41410b660 2024-05-10T11:20:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.726+0000 7ff419000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141156a0 con 0x7ff414105f90 2024-05-10T11:20:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.726+0000 7ff419000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141159a0 con 0x7ff4141055d0 2024-05-10T11:20:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.727+0000 7ff413400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4141055d0 0x7ff4141a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.727+0000 7ff412a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff414105f90 0x7ff4141a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.727+0000 7ff413400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4141055d0 0x7ff4141a18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49406/0 (socket says 172.21.15.27:49406) 2024-05-10T11:20:15.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.727+0000 7ff413400700 1 -- 172.21.15.27:0/1780429610 learned_addr learned my addr 172.21.15.27:0/1780429610 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:15.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.727+0000 7ff413e00700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff41410b660 0x7ff41419ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.727+0000 7ff412a00700 1 -- 172.21.15.27:0/1780429610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4141055d0 msgr2=0x7ff4141a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.727+0000 7ff412a00700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4141055d0 0x7ff4141a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.727+0000 7ff412a00700 1 -- 172.21.15.27:0/1780429610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff41410b660 msgr2=0x7ff41419ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.727+0000 7ff412a00700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff41410b660 0x7ff41419ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.727+0000 7ff412a00700 1 -- 172.21.15.27:0/1780429610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff41419c1c0 con 0x7ff414105f90 2024-05-10T11:20:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.728+0000 7ff412a00700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff414105f90 0x7ff4141a1de0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff4000098e0 tx=0x7ff4000057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.728+0000 7ff40be00700 1 -- 172.21.15.27:0/1780429610 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff400027070 con 0x7ff414105f90 2024-05-10T11:20:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.728+0000 7ff40be00700 1 -- 172.21.15.27:0/1780429610 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff400022470 con 0x7ff414105f90 2024-05-10T11:20:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.728+0000 7ff40be00700 1 -- 172.21.15.27:0/1780429610 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff400017650 con 0x7ff414105f90 2024-05-10T11:20:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.728+0000 7ff419000700 1 -- 172.21.15.27:0/1780429610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff41419c450 con 0x7ff414105f90 2024-05-10T11:20:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.728+0000 7ff419000700 1 -- 172.21.15.27:0/1780429610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4141a8770 con 0x7ff414105f90 2024-05-10T11:20:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.732+0000 7ff40be00700 1 -- 172.21.15.27:0/1780429610 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff40002a070 con 0x7ff414105f90 2024-05-10T11:20:15.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.733+0000 7ff424f62b80 1 Processor -- start 2024-05-10T11:20:15.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.734+0000 7ff40be00700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff3f80b2a70 0x7ff3f80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.734+0000 7ff424f62b80 1 -- start start 2024-05-10T11:20:15.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.734+0000 7ff40be00700 1 -- 172.21.15.27:0/1780429610 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7ff4000e37c0 con 0x7ff414105f90 2024-05-10T11:20:15.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.734+0000 7ff424f62b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0d9e0 0x556bcff0ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.734+0000 7ff424f62b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0e300 0x556bcff07fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.734+0000 7ff413e00700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff3f80b2a70 0x7ff3f80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.734+0000 7ff424f62b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff08600 0x556bcff08a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.734+0000 7ff424f62b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556bcfe5aa20 con 0x556bcff08600 2024-05-10T11:20:15.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.734+0000 7ff424f62b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x556bcfdf8760 con 0x556bcff0d9e0 2024-05-10T11:20:15.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.734+0000 7ff424f62b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556bcfe569e0 con 0x556bcff0e300 2024-05-10T11:20:15.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.735+0000 7ff413400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0d9e0 0x556bcff0ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.735+0000 7ff412a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0e300 0x556bcff07fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.735+0000 7ff413400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0d9e0 0x556bcff0ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36244/0 (socket says 172.21.15.27:36244) 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.735+0000 7ff413400700 1 -- 172.21.15.27:0/3295971153 learned_addr learned my addr 172.21.15.27:0/3295971153 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.735+0000 7ff413e00700 1 --2- 172.21.15.27:0/3295971153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff08600 0x556bcff08a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.735+0000 7ff412a00700 1 -- 172.21.15.27:0/3295971153 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0d9e0 msgr2=0x556bcff0ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.735+0000 7ff412a00700 1 --2- 172.21.15.27:0/3295971153 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0d9e0 0x556bcff0ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.735+0000 7ff412a00700 1 -- 172.21.15.27:0/3295971153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff08600 msgr2=0x556bcff08a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.735+0000 7ff412a00700 1 --2- 172.21.15.27:0/3295971153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff08600 0x556bcff08a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.735+0000 7ff412a00700 1 -- 172.21.15.27:0/3295971153 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bcfda78d0 con 0x556bcff0e300 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.735+0000 7ff413e00700 1 --2- 172.21.15.27:0/3295971153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff08600 0x556bcff08a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.735+0000 7ff412a00700 1 --2- 172.21.15.27:0/3295971153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0e300 0x556bcff07fe0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff4000e0a80 tx=0x7ff400009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=accb87b6994a5cb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.736+0000 7ff413e00700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff3f80b2a70 0x7ff3f80b4f30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x556bcff08fc0 tx=0x7ff40c009920 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.736+0000 7ff408c00700 1 -- 172.21.15.27:0/3295971153 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4000aada0 con 0x556bcff0e300 2024-05-10T11:20:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.736+0000 7ff408c00700 1 -- 172.21.15.27:0/3295971153 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4000aa6a0 con 0x556bcff0e300 2024-05-10T11:20:15.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.736+0000 7ff408c00700 1 -- 172.21.15.27:0/3295971153 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff400022920 con 0x556bcff0e300 2024-05-10T11:20:15.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.736+0000 7ff424f62b80 1 -- 172.21.15.27:0/3295971153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0e300 msgr2=0x556bcff07fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:15.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.736+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3295971153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0e300 0x556bcff07fe0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff4000e0a80 tx=0x7ff400009ca0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.737+0000 7ff424f62b80 1 -- 172.21.15.27:0/3295971153 shutdown_connections 2024-05-10T11:20:15.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.737+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3295971153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff08600 0x556bcff08a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.737+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3295971153 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0d9e0 0x556bcff0ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.737+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3295971153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0e300 0x556bcff07fe0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.737+0000 7ff424f62b80 1 -- 172.21.15.27:0/3295971153 >> 172.21.15.27:0/3295971153 conn(0x556bcfe36b60 msgr2=0x556bcff029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.737+0000 7ff424f62b80 1 -- 172.21.15.27:0/3295971153 shutdown_connections 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.737+0000 7ff424f62b80 1 -- 172.21.15.27:0/3295971153 wait complete. 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.737+0000 7ff424f62b80 1 -- start start 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.738+0000 7ff424f62b80 1 Processor -- start 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.738+0000 7ff424f62b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff08600 0x556bcff13fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.738+0000 7ff424f62b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff0d9e0 0x556bcff11350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.738+0000 7ff424f62b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0e300 0x556bcff11890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.738+0000 7ff424f62b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556bcff14b70 con 0x556bcff0d9e0 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.738+0000 7ff424f62b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x556bcfdc0b20 con 0x556bcff0e300 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.738+0000 7ff424f62b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556bcff14cf0 con 0x556bcff08600 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.738+0000 7ff424f62b80 10 client.0 _ll_register_callbacks cb 0x556bcfe01c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.738+0000 7ff413400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff08600 0x556bcff13fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.739+0000 7ff412a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff0d9e0 0x556bcff11350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.739+0000 7ff413400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff08600 0x556bcff13fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49430/0 (socket says 172.21.15.27:49430) 2024-05-10T11:20:15.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.739+0000 7ff413400700 1 -- 172.21.15.27:0/3594931286 learned_addr learned my addr 172.21.15.27:0/3594931286 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:15.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.739+0000 7ff413e00700 1 --2- 172.21.15.27:0/3594931286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0e300 0x556bcff11890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.739+0000 7ff413400700 1 -- 172.21.15.27:0/3594931286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0e300 msgr2=0x556bcff11890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:15.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.739+0000 7ff413400700 1 --2- 172.21.15.27:0/3594931286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0e300 0x556bcff11890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.739+0000 7ff413400700 1 -- 172.21.15.27:0/3594931286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff0d9e0 msgr2=0x556bcff11350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:15.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.739+0000 7ff413400700 1 --2- 172.21.15.27:0/3594931286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff0d9e0 0x556bcff11350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.739+0000 7ff413400700 1 -- 172.21.15.27:0/3594931286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bcfe20420 con 0x556bcff08600 2024-05-10T11:20:15.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.739+0000 7ff413400700 1 --2- 172.21.15.27:0/3594931286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff08600 0x556bcff13fb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff404012980 tx=0x7ff404012c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51cc210a19eb0564 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:15.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.739+0000 7ff3fd600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.740+0000 7ff3fd600700 1 -- 172.21.15.27:0/3594931286 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff404004030 con 0x556bcff08600 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.740+0000 7ff3fd600700 1 -- 172.21.15.27:0/3594931286 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7ff4040041d0 con 0x556bcff08600 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.740+0000 7ff424f62b80 1 -- 172.21.15.27:0/3594931286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4000aa6a0 con 0x556bcff08600 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.740+0000 7ff3fd600700 1 -- 172.21.15.27:0/3594931286 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff404025050 con 0x556bcff08600 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.740+0000 7ff424f62b80 10 client.26332 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.740+0000 7ff424f62b80 1 -- 172.21.15.27:0/3594931286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556bcff1f240 con 0x556bcff08600 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.740+0000 7ff3fd600700 1 -- 172.21.15.27:0/3594931286 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4000aa6a0 con 0x556bcff08600 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.740+0000 7ff3fd600700 1 -- 172.21.15.27:0/3594931286 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff4040037a0 con 0x556bcff08600 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.741+0000 7ff424f62b80 10 client.26332 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.741+0000 7ff424f62b80 10 client.26332 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.741+0000 7ff424f62b80 20 client.26332 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.741+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3594931286 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x556bcff21880 0x556bcff23c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.741+0000 7ff424f62b80 4 client.26332 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:20:15.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.741+0000 7ff424f62b80 1 -- 172.21.15.27:0/3594931286 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bcfed50a0 con 0x556bcff21880 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.741+0000 7ff412a00700 1 --2- 172.21.15.27:0/3594931286 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x556bcff21880 0x556bcff23c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.741+0000 7ff412a00700 1 --2- 172.21.15.27:0/3594931286 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x556bcff21880 0x556bcff23c70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2dd35fdbe927fbe6 server_cookie=f12e92718c6a32d6 in_seq=0 out_seq=0 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.741+0000 7ff3fd600700 10 client.26332 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.742+0000 7ff3fd600700 1 -- 172.21.15.27:0/3594931286 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556bcfed50a0 con 0x556bcff21880 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.742+0000 7ff3fd600700 10 client.26332 handle_command_reply: tid=0 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.742+0000 7ff424f62b80 1 client.26332 shutdown 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.743+0000 7ff424f62b80 1 -- 172.21.15.27:0/3594931286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff08600 msgr2=0x556bcff13fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.743+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3594931286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff08600 0x556bcff13fb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff404012980 tx=0x7ff404012c90 comp rx=0 tx=0).stop 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.743+0000 7ff424f62b80 1 -- 172.21.15.27:0/3594931286 shutdown_connections 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.743+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3594931286 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x556bcff21880 0x556bcff23c70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.743+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3594931286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff08600 0x556bcff13fb0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.743+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3594931286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0e300 0x556bcff11890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.743+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3594931286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff0d9e0 0x556bcff11350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.743+0000 7ff424f62b80 1 -- 172.21.15.27:0/3594931286 >> 172.21.15.27:0/3594931286 conn(0x556bcfe36b60 msgr2=0x556bcfefd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.743+0000 7ff3fd600700 0 client.26332 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:20:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.743+0000 7ff424f62b80 1 -- 172.21.15.27:0/3594931286 shutdown_connections 2024-05-10T11:20:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.743+0000 7ff424f62b80 1 -- 172.21.15.27:0/3594931286 wait complete. 2024-05-10T11:20:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.743+0000 7ff424f62b80 20 client.26332 trim_cache size 0 max 16384 2024-05-10T11:20:15.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.761+0000 7ff424f62b80 1 Processor -- start 2024-05-10T11:20:15.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.761+0000 7ff424f62b80 1 -- start start 2024-05-10T11:20:15.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.761+0000 7ff424f62b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff08600 0x556bcff07fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.761+0000 7ff424f62b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0d9e0 0x556bcff1e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.761+0000 7ff424f62b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff0e300 0x556bcff0fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.761+0000 7ff424f62b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141b16c0 con 0x556bcff0e300 2024-05-10T11:20:15.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.761+0000 7ff424f62b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141b3200 con 0x556bcff08600 2024-05-10T11:20:15.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.761+0000 7ff424f62b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141b1d90 con 0x556bcff0d9e0 2024-05-10T11:20:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.761+0000 7ff413400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff08600 0x556bcff07fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.761+0000 7ff413400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff08600 0x556bcff07fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36272/0 (socket says 172.21.15.27:36272) 2024-05-10T11:20:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.761+0000 7ff413400700 1 -- 172.21.15.27:0/1153950316 learned_addr learned my addr 172.21.15.27:0/1153950316 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.762+0000 7ff412a00700 1 --2- 172.21.15.27:0/1153950316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0d9e0 0x556bcff1e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.762+0000 7ff413e00700 1 --2- 172.21.15.27:0/1153950316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff0e300 0x556bcff0fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.762+0000 7ff413400700 1 -- 172.21.15.27:0/1153950316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0d9e0 msgr2=0x556bcff1e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.762+0000 7ff413400700 1 --2- 172.21.15.27:0/1153950316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0d9e0 0x556bcff1e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.762+0000 7ff413400700 1 -- 172.21.15.27:0/1153950316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff0e300 msgr2=0x556bcff0fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:15.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.762+0000 7ff413400700 1 --2- 172.21.15.27:0/1153950316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff0e300 0x556bcff0fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.762+0000 7ff413400700 1 -- 172.21.15.27:0/1153950316 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bcff0f8e0 con 0x556bcff08600 2024-05-10T11:20:15.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.762+0000 7ff413400700 1 --2- 172.21.15.27:0/1153950316 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff08600 0x556bcff07fe0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff404001eb0 tx=0x7ff404009aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3cf11f6d00f09f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:15.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.762+0000 7ff3ffe00700 1 -- 172.21.15.27:0/1153950316 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff40402f070 con 0x556bcff08600 2024-05-10T11:20:15.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.762+0000 7ff3ffe00700 1 -- 172.21.15.27:0/1153950316 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff40400fce0 con 0x556bcff08600 2024-05-10T11:20:15.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.762+0000 7ff3ffe00700 1 -- 172.21.15.27:0/1153950316 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff404005250 con 0x556bcff08600 2024-05-10T11:20:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.763+0000 7ff424f62b80 1 -- 172.21.15.27:0/1153950316 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff08600 msgr2=0x556bcff07fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.763+0000 7ff424f62b80 1 --2- 172.21.15.27:0/1153950316 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff08600 0x556bcff07fe0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff404001eb0 tx=0x7ff404009aa0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.763+0000 7ff424f62b80 1 -- 172.21.15.27:0/1153950316 shutdown_connections 2024-05-10T11:20:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.763+0000 7ff424f62b80 1 --2- 172.21.15.27:0/1153950316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff0e300 0x556bcff0fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.763+0000 7ff424f62b80 1 --2- 172.21.15.27:0/1153950316 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff08600 0x556bcff07fe0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.763+0000 7ff424f62b80 1 --2- 172.21.15.27:0/1153950316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0d9e0 0x556bcff1e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.763+0000 7ff424f62b80 1 -- 172.21.15.27:0/1153950316 >> 172.21.15.27:0/1153950316 conn(0x556bcfe36b60 msgr2=0x556bcff11310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.763+0000 7ff424f62b80 1 -- 172.21.15.27:0/1153950316 shutdown_connections 2024-05-10T11:20:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.763+0000 7ff424f62b80 1 -- 172.21.15.27:0/1153950316 wait complete. 2024-05-10T11:20:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.764+0000 7ff424f62b80 1 -- start start 2024-05-10T11:20:15.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.764+0000 7ff424f62b80 1 Processor -- start 2024-05-10T11:20:15.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff424f62b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff08600 0x556bcff19f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff424f62b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0d9e0 0x556bcff1a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff424f62b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0e300 0x556bcff28530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff424f62b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141b2b30 con 0x556bcff08600 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff424f62b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141b2460 con 0x556bcff0e300 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff424f62b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff4141af680 con 0x556bcff0d9e0 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff424f62b80 10 client.0 _ll_register_callbacks cb 0x556bcfe01c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff412a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0d9e0 0x556bcff1a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff413e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0e300 0x556bcff28530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff413400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff08600 0x556bcff19f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff412a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0d9e0 0x556bcff1a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49442/0 (socket says 172.21.15.27:49442) 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff412a00700 1 -- 172.21.15.27:0/3832417528 learned_addr learned my addr 172.21.15.27:0/3832417528 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff413e00700 1 --2- 172.21.15.27:0/3832417528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0e300 0x556bcff28530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36276/0 (socket says 172.21.15.27:36276) 2024-05-10T11:20:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff413e00700 1 -- 172.21.15.27:0/3832417528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0d9e0 msgr2=0x556bcff1a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:15.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.765+0000 7ff413e00700 1 --2- 172.21.15.27:0/3832417528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0d9e0 0x556bcff1a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.766+0000 7ff413e00700 1 -- 172.21.15.27:0/3832417528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff08600 msgr2=0x556bcff19f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:15.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.766+0000 7ff413e00700 1 --2- 172.21.15.27:0/3832417528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff08600 0x556bcff19f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.766+0000 7ff413e00700 1 -- 172.21.15.27:0/3832417528 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bcff28b20 con 0x556bcff0e300 2024-05-10T11:20:15.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.766+0000 7ff413e00700 1 --2- 172.21.15.27:0/3832417528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0e300 0x556bcff28530 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009510 tx=0x7ff40c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ad90ac813423579 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:15.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.766+0000 7ff3fd600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:20:15.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.766+0000 7ff3fd600700 1 -- 172.21.15.27:0/3832417528 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff40c029030 con 0x556bcff0e300 2024-05-10T11:20:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.767+0000 7ff424f62b80 1 -- 172.21.15.27:0/3832417528 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff40400fce0 con 0x556bcff0e300 2024-05-10T11:20:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.767+0000 7ff3fd600700 1 -- 172.21.15.27:0/3832417528 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7ff40c025730 con 0x556bcff0e300 2024-05-10T11:20:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.767+0000 7ff3fd600700 1 -- 172.21.15.27:0/3832417528 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff40c01e040 con 0x556bcff0e300 2024-05-10T11:20:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.767+0000 7ff424f62b80 10 client.26354 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.767+0000 7ff424f62b80 1 -- 172.21.15.27:0/3832417528 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556bcff8ba40 con 0x556bcff0e300 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.767+0000 7ff3fd600700 1 -- 172.21.15.27:0/3832417528 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff40400fce0 con 0x556bcff0e300 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.767+0000 7ff3fd600700 1 -- 172.21.15.27:0/3832417528 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff40c0179c0 con 0x556bcff0e300 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.767+0000 7ff424f62b80 10 client.26354 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.767+0000 7ff424f62b80 10 client.26354 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.767+0000 7ff424f62b80 20 client.26354 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.767+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3832417528 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x556bcff8c120 0x556bcff8e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.768+0000 7ff424f62b80 4 client.26354 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.768+0000 7ff424f62b80 1 -- 172.21.15.27:0/3832417528 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556bcfe68400 con 0x556bcff8c120 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.768+0000 7ff413400700 1 --2- 172.21.15.27:0/3832417528 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x556bcff8c120 0x556bcff8e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.768+0000 7ff413400700 1 --2- 172.21.15.27:0/3832417528 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x556bcff8c120 0x556bcff8e540 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d26e54134926e1 server_cookie=44301efc55c53562 in_seq=0 out_seq=0 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.768+0000 7ff3fd600700 10 client.26354 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff3fd600700 1 -- 172.21.15.27:0/3832417528 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556bcfe68400 con 0x556bcff8c120 2024-05-10T11:20:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff3fd600700 10 client.26354 handle_command_reply: tid=0 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff424f62b80 1 client.26354 shutdown 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff424f62b80 1 -- 172.21.15.27:0/3832417528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0e300 msgr2=0x556bcff28530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3832417528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0e300 0x556bcff28530 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009510 tx=0x7ff40c004ad0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff424f62b80 1 -- 172.21.15.27:0/3832417528 shutdown_connections 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3832417528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556bcff0e300 0x556bcff28530 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3832417528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556bcff08600 0x556bcff19f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3832417528 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x556bcff8c120 0x556bcff8e540 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff424f62b80 1 --2- 172.21.15.27:0/3832417528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bcff0d9e0 0x556bcff1a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff3fd600700 0 client.26354 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff424f62b80 1 -- 172.21.15.27:0/3832417528 >> 172.21.15.27:0/3832417528 conn(0x556bcfe36b60 msgr2=0x556bcff0fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff424f62b80 1 -- 172.21.15.27:0/3832417528 shutdown_connections 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff424f62b80 1 -- 172.21.15.27:0/3832417528 wait complete. 2024-05-10T11:20:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.769+0000 7ff424f62b80 20 client.26354 trim_cache size 0 max 16384 2024-05-10T11:20:15.772 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:20:15.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.770+0000 7ff419000700 1 -- 172.21.15.27:0/1780429610 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff3f80b2a70 msgr2=0x7ff3f80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:15.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.770+0000 7ff419000700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff3f80b2a70 0x7ff3f80b4f30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x556bcff08fc0 tx=0x7ff40c009920 comp rx=0 tx=0).stop 2024-05-10T11:20:15.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.770+0000 7ff419000700 1 -- 172.21.15.27:0/1780429610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff414105f90 msgr2=0x7ff4141a1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:15.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.770+0000 7ff419000700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff414105f90 0x7ff4141a1de0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff4000098e0 tx=0x7ff4000057c0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.770+0000 7ff419000700 1 -- 172.21.15.27:0/1780429610 shutdown_connections 2024-05-10T11:20:15.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.770+0000 7ff419000700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff3f80b2a70 0x7ff3f80b4f30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.770+0000 7ff419000700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4141055d0 0x7ff4141a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.770+0000 7ff419000700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff41410b660 0x7ff41419ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.770+0000 7ff419000700 1 --2- 172.21.15.27:0/1780429610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff414105f90 0x7ff4141a1de0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:15.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.770+0000 7ff419000700 1 -- 172.21.15.27:0/1780429610 >> 172.21.15.27:0/1780429610 conn(0x7ff4141010c0 msgr2=0x7ff414102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:15.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.770+0000 7ff419000700 1 -- 172.21.15.27:0/1780429610 shutdown_connections 2024-05-10T11:20:15.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:15.770+0000 7ff419000700 1 -- 172.21.15.27:0/1780429610 wait complete. 2024-05-10T11:20:16.689 INFO:tasks.workunit.client.0.smithi027.stdout: 1 398567 6.88 MB/sec execute 181 sec latency 34.201 ms 2024-05-10T11:20:16.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:16 smithi076 ceph-mon[34152]: pgmap v216: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 171 op/s 2024-05-10T11:20:16.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:16 smithi189 ceph-mon[34842]: pgmap v216: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 171 op/s 2024-05-10T11:20:17.088 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:16 smithi027 ceph-mon[30054]: pgmap v216: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 171 op/s 2024-05-10T11:20:17.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:20:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:20:17.689 INFO:tasks.workunit.client.0.smithi027.stdout: 1 400229 6.90 MB/sec execute 182 sec latency 31.777 ms 2024-05-10T11:20:18.689 INFO:tasks.workunit.client.0.smithi027.stdout: 1 401218 6.89 MB/sec execute 183 sec latency 46.166 ms 2024-05-10T11:20:18.885 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:18 smithi027 ceph-mon[30054]: pgmap v217: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 171 op/s 2024-05-10T11:20:18.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:18 smithi076 ceph-mon[34152]: pgmap v217: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 171 op/s 2024-05-10T11:20:18.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:18 smithi189 ceph-mon[34842]: pgmap v217: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 171 op/s 2024-05-10T11:20:19.689 INFO:tasks.workunit.client.0.smithi027.stdout: 1 402013 6.87 MB/sec execute 184 sec latency 36.108 ms 2024-05-10T11:20:20.689 INFO:tasks.workunit.client.0.smithi027.stdout: 1 403418 6.86 MB/sec execute 185 sec latency 25.926 ms 2024-05-10T11:20:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.799+0000 7fc9d6000700 1 -- 172.21.15.27:0/2857577447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9d00758d0 msgr2=0x7fc9d0075cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.799+0000 7fc9d6000700 1 --2- 172.21.15.27:0/2857577447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9d00758d0 0x7fc9d0075cb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4009990 tx=0x7fc9c4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.799+0000 7fc9d6000700 1 -- 172.21.15.27:0/2857577447 shutdown_connections 2024-05-10T11:20:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.799+0000 7fc9d6000700 1 --2- 172.21.15.27:0/2857577447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9d010fc10 0x7fc9d0112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.799+0000 7fc9d6000700 1 --2- 172.21.15.27:0/2857577447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9d00758d0 0x7fc9d0075cb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.799+0000 7fc9d6000700 1 --2- 172.21.15.27:0/2857577447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9d00761f0 0x7fc9d010f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.799+0000 7fc9d6000700 1 -- 172.21.15.27:0/2857577447 >> 172.21.15.27:0/2857577447 conn(0x7fc9d00fef20 msgr2=0x7fc9d0101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.799+0000 7fc9d6000700 1 -- 172.21.15.27:0/2857577447 shutdown_connections 2024-05-10T11:20:20.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.799+0000 7fc9d6000700 1 -- 172.21.15.27:0/2857577447 wait complete. 2024-05-10T11:20:20.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:20 smithi027 ceph-mon[30054]: pgmap v218: 129 pgs: 129 active+clean; 246 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 239 op/s 2024-05-10T11:20:20.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.800+0000 7fc9d6000700 1 Processor -- start 2024-05-10T11:20:20.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.801+0000 7fc9d6000700 1 -- start start 2024-05-10T11:20:20.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.801+0000 7fc9d6000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9d00758d0 0x7fc9d019bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.801+0000 7fc9d6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9d00761f0 0x7fc9d019c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.801+0000 7fc9d6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9d010fc10 0x7fc9d01a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.801+0000 7fc9d6000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01157f0 con 0x7fc9d010fc10 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.801+0000 7fc9d6000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d0115670 con 0x7fc9d00758d0 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.801+0000 7fc9d6000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d0115970 con 0x7fc9d00761f0 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.802+0000 7fc9cfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9d00761f0 0x7fc9d019c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.802+0000 7fc9cfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9d00761f0 0x7fc9d019c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49460/0 (socket says 172.21.15.27:49460) 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.802+0000 7fc9cfe00700 1 -- 172.21.15.27:0/485410649 learned_addr learned my addr 172.21.15.27:0/485410649 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.802+0000 7fc9d5600700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9d010fc10 0x7fc9d01a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.802+0000 7fc9cfe00700 1 -- 172.21.15.27:0/485410649 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9d00758d0 msgr2=0x7fc9d019bd20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.802+0000 7fc9cfe00700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9d00758d0 0x7fc9d019bd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.802+0000 7fc9cfe00700 1 -- 172.21.15.27:0/485410649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9d010fc10 msgr2=0x7fc9d01a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.802+0000 7fc9cfe00700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9d010fc10 0x7fc9d01a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.802+0000 7fc9cfe00700 1 -- 172.21.15.27:0/485410649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d01a0d30 con 0x7fc9d00761f0 2024-05-10T11:20:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.802+0000 7fc9d5600700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9d010fc10 0x7fc9d01a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:20:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.803+0000 7fc9cfe00700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9d00761f0 0x7fc9d019c260 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00e9a0 tx=0x7fc9bc00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.803+0000 7fc9cd600700 1 -- 172.21.15.27:0/485410649 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc00cbd0 con 0x7fc9d00761f0 2024-05-10T11:20:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.803+0000 7fc9d6000700 1 -- 172.21.15.27:0/485410649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d01a1020 con 0x7fc9d00761f0 2024-05-10T11:20:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.803+0000 7fc9cd600700 1 -- 172.21.15.27:0/485410649 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9bc00cd70 con 0x7fc9d00761f0 2024-05-10T11:20:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.803+0000 7fc9cd600700 1 -- 172.21.15.27:0/485410649 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc018710 con 0x7fc9d00761f0 2024-05-10T11:20:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.803+0000 7fc9d6000700 1 -- 172.21.15.27:0/485410649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d01a88a0 con 0x7fc9d00761f0 2024-05-10T11:20:20.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.809+0000 7fc9cd600700 1 -- 172.21.15.27:0/485410649 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc9bc010430 con 0x7fc9d00761f0 2024-05-10T11:20:20.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.810+0000 7fc9e1f8eb80 1 Processor -- start 2024-05-10T11:20:20.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.810+0000 7fc9e1f8eb80 1 -- start start 2024-05-10T11:20:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.810+0000 7fc9e1f8eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20ce039e0 0x55d20ce03dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.810+0000 7fc9e1f8eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce04300 0x55d20cdfdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.810+0000 7fc9e1f8eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20cdfe600 0x55d20cdfea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.810+0000 7fc9e1f8eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d20cd50a20 con 0x55d20ce039e0 2024-05-10T11:20:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.810+0000 7fc9e1f8eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d20ccee760 con 0x55d20ce04300 2024-05-10T11:20:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.810+0000 7fc9e1f8eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d20cd4c9e0 con 0x55d20cdfe600 2024-05-10T11:20:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.811+0000 7fc9d4c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce04300 0x55d20cdfdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.811+0000 7fc9d5600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20ce039e0 0x55d20ce03dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.811+0000 7fc9cfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20cdfe600 0x55d20cdfea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.811+0000 7fc9d5600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20ce039e0 0x55d20ce03dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54028/0 (socket says 172.21.15.27:54028) 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.811+0000 7fc9d5600700 1 -- 172.21.15.27:0/2243816050 learned_addr learned my addr 172.21.15.27:0/2243816050 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.811+0000 7fc9d4c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce04300 0x55d20cdfdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36292/0 (socket says 172.21.15.27:36292) 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.811+0000 7fc9cd600700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9b80b2a70 0x7fc9b80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.811+0000 7fc9cfe00700 1 -- 172.21.15.27:0/2243816050 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce04300 msgr2=0x55d20cdfdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.811+0000 7fc9cfe00700 1 --2- 172.21.15.27:0/2243816050 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce04300 0x55d20cdfdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.811+0000 7fc9cfe00700 1 -- 172.21.15.27:0/2243816050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20ce039e0 msgr2=0x55d20ce03dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.811+0000 7fc9cfe00700 1 --2- 172.21.15.27:0/2243816050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20ce039e0 0x55d20ce03dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.811+0000 7fc9cd600700 1 -- 172.21.15.27:0/485410649 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7fc9bc0e2ce0 con 0x7fc9d00761f0 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.812+0000 7fc9cfe00700 1 -- 172.21.15.27:0/2243816050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d20cc9d8d0 con 0x55d20cdfe600 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.812+0000 7fc9cfe00700 1 --2- 172.21.15.27:0/2243816050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20cdfe600 0x55d20cdfea80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0e0000 tx=0x7fc9bc0a5b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f5f77384b9bb7d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.812+0000 7fc9d5600700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9b80b2a70 0x7fc9b80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.812+0000 7fc9c2000700 1 -- 172.21.15.27:0/2243816050 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc0a9300 con 0x55d20cdfe600 2024-05-10T11:20:20.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.812+0000 7fc9c2000700 1 -- 172.21.15.27:0/2243816050 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9bc0b2330 con 0x55d20cdfe600 2024-05-10T11:20:20.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.812+0000 7fc9c2000700 1 -- 172.21.15.27:0/2243816050 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc0adf40 con 0x55d20cdfe600 2024-05-10T11:20:20.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.813+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/2243816050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20cdfe600 msgr2=0x55d20cdfea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:20.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.813+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/2243816050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20cdfe600 0x55d20cdfea80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0e0000 tx=0x7fc9bc0a5b80 comp rx=0 tx=0).stop 2024-05-10T11:20:20.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.813+0000 7fc9d5600700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9b80b2a70 0x7fc9b80b4f30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc9c80079a0 tx=0x7fc9c8009040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:20.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.813+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/2243816050 shutdown_connections 2024-05-10T11:20:20.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.813+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/2243816050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20cdfe600 0x55d20cdfea80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.813+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/2243816050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20ce039e0 0x55d20ce03dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.813+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/2243816050 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce04300 0x55d20cdfdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.813+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/2243816050 >> 172.21.15.27:0/2243816050 conn(0x55d20cd2cb60 msgr2=0x55d20cdf89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:20.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.813+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/2243816050 shutdown_connections 2024-05-10T11:20:20.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.814+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/2243816050 wait complete. 2024-05-10T11:20:20.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.814+0000 7fc9e1f8eb80 1 -- start start 2024-05-10T11:20:20.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.814+0000 7fc9e1f8eb80 1 Processor -- start 2024-05-10T11:20:20.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.815+0000 7fc9e1f8eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20ce09ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.815+0000 7fc9e1f8eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce07300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.815+0000 7fc9e1f8eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce07840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.815+0000 7fc9e1f8eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d20ce0ad50 con 0x55d20cdfe600 2024-05-10T11:20:20.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.815+0000 7fc9e1f8eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d20ccb6b20 con 0x55d20ce039e0 2024-05-10T11:20:20.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.815+0000 7fc9e1f8eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d20ce0aed0 con 0x55d20ce04300 2024-05-10T11:20:20.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.815+0000 7fc9e1f8eb80 10 client.0 _ll_register_callbacks cb 0x55d20ccf7c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:20.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.815+0000 7fc9cfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce07300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.815+0000 7fc9cfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce07300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36300/0 (socket says 172.21.15.27:36300) 2024-05-10T11:20:20.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.815+0000 7fc9d5600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce07840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.815+0000 7fc9cfe00700 1 -- 172.21.15.27:0/1826614303 learned_addr learned my addr 172.21.15.27:0/1826614303 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:20.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.816+0000 7fc9d4c00700 1 --2- 172.21.15.27:0/1826614303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20ce09ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.816+0000 7fc9cfe00700 1 -- 172.21.15.27:0/1826614303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 msgr2=0x55d20ce07840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:20.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.816+0000 7fc9cfe00700 1 --2- 172.21.15.27:0/1826614303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce07840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.816+0000 7fc9cfe00700 1 -- 172.21.15.27:0/1826614303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 msgr2=0x55d20ce09ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:20.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.816+0000 7fc9cfe00700 1 --2- 172.21.15.27:0/1826614303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20ce09ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.816+0000 7fc9cfe00700 1 -- 172.21.15.27:0/1826614303 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d20cd16420 con 0x55d20ce039e0 2024-05-10T11:20:20.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.816+0000 7fc9cfe00700 1 --2- 172.21.15.27:0/1826614303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce07300 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0a9b20 tx=0x7fc9bc0a6290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a020059f2c24ae66 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:20.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.816+0000 7fc9b6a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:20:20.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.816+0000 7fc9b6a00700 1 -- 172.21.15.27:0/1826614303 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc0c2020 con 0x55d20ce039e0 2024-05-10T11:20:20.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.816+0000 7fc9b6a00700 1 -- 172.21.15.27:0/1826614303 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7fc9bc0b2560 con 0x55d20ce039e0 2024-05-10T11:20:20.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.816+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/1826614303 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d20cdcb0a0 con 0x55d20ce039e0 2024-05-10T11:20:20.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.817+0000 7fc9b6a00700 1 -- 172.21.15.27:0/1826614303 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc9bc0a9300 con 0x55d20ce039e0 2024-05-10T11:20:20.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.817+0000 7fc9e1f8eb80 10 client.26363 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:20.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.817+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/1826614303 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d20ce154d0 con 0x55d20ce039e0 2024-05-10T11:20:20.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.817+0000 7fc9b6a00700 1 -- 172.21.15.27:0/1826614303 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d20cdcb0a0 con 0x55d20ce039e0 2024-05-10T11:20:20.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.817+0000 7fc9b6a00700 1 -- 172.21.15.27:0/1826614303 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fc9bc0a5500 con 0x55d20ce039e0 2024-05-10T11:20:20.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.818+0000 7fc9e1f8eb80 10 client.26363 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:20.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.818+0000 7fc9e1f8eb80 10 client.26363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:20:20.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.818+0000 7fc9e1f8eb80 20 client.26363 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:20.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.818+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/1826614303 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d20ce179a0 0x55d20ce19d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.818+0000 7fc9e1f8eb80 4 client.26363 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:20:20.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.818+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/1826614303 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d20cd5e400 con 0x55d20ce179a0 2024-05-10T11:20:20.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.818+0000 7fc9d4c00700 1 --2- 172.21.15.27:0/1826614303 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d20ce179a0 0x55d20ce19d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.818+0000 7fc9d4c00700 1 --2- 172.21.15.27:0/1826614303 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d20ce179a0 0x55d20ce19d90 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47d0ac008a5749e8 server_cookie=12a8cdde4624b29c in_seq=0 out_seq=0 2024-05-10T11:20:20.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.818+0000 7fc9b6a00700 10 client.26363 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:20:20.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.820+0000 7fc9b6a00700 1 -- 172.21.15.27:0/1826614303 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d20cd5e400 con 0x55d20ce179a0 2024-05-10T11:20:20.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.820+0000 7fc9b6a00700 10 client.26363 handle_command_reply: tid=0 2024-05-10T11:20:20.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.820+0000 7fc9e1f8eb80 1 client.26363 shutdown 2024-05-10T11:20:20.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.820+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/1826614303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 msgr2=0x55d20ce07300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:20.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.820+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/1826614303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce07300 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0a9b20 tx=0x7fc9bc0a6290 comp rx=0 tx=0).stop 2024-05-10T11:20:20.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.820+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/1826614303 shutdown_connections 2024-05-10T11:20:20.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.820+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/1826614303 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d20ce179a0 0x55d20ce19d90 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.820+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/1826614303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce07840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.820+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/1826614303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20ce09ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.820+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/1826614303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce07300 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.820+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/1826614303 >> 172.21.15.27:0/1826614303 conn(0x55d20cd2cb60 msgr2=0x55d20cdf3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:20.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.820+0000 7fc9b6a00700 0 client.26363 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:20:20.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.821+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/1826614303 shutdown_connections 2024-05-10T11:20:20.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.821+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/1826614303 wait complete. 2024-05-10T11:20:20.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.821+0000 7fc9e1f8eb80 20 client.26363 trim_cache size 0 max 16384 2024-05-10T11:20:20.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.839+0000 7fc9e1f8eb80 1 Processor -- start 2024-05-10T11:20:20.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.839+0000 7fc9e1f8eb80 1 -- start start 2024-05-10T11:20:20.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.840+0000 7fc9e1f8eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20cdfdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.840+0000 7fc9e1f8eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce07ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.840+0000 7fc9e1f8eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce04df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.840+0000 7fc9e1f8eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01b19a0 con 0x55d20cdfe600 2024-05-10T11:20:20.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.840+0000 7fc9e1f8eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01b34e0 con 0x55d20ce039e0 2024-05-10T11:20:20.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.840+0000 7fc9e1f8eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01b2070 con 0x55d20ce04300 2024-05-10T11:20:20.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.840+0000 7fc9d4c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20cdfdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.840+0000 7fc9d5600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce04df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.840+0000 7fc9d5600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce04df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49498/0 (socket says 172.21.15.27:49498) 2024-05-10T11:20:20.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.840+0000 7fc9d4c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20cdfdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54040/0 (socket says 172.21.15.27:54040) 2024-05-10T11:20:20.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.840+0000 7fc9d5600700 1 -- 172.21.15.27:0/833906196 learned_addr learned my addr 172.21.15.27:0/833906196 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:20.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.841+0000 7fc9d5600700 1 -- 172.21.15.27:0/833906196 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 msgr2=0x55d20ce07ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:20.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.841+0000 7fc9d5600700 1 --2- 172.21.15.27:0/833906196 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce07ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.841+0000 7fc9d5600700 1 -- 172.21.15.27:0/833906196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 msgr2=0x55d20cdfdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:20.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.841+0000 7fc9d5600700 1 --2- 172.21.15.27:0/833906196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20cdfdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.841+0000 7fc9d5600700 1 -- 172.21.15.27:0/833906196 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d20ce01d80 con 0x55d20ce04300 2024-05-10T11:20:20.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.841+0000 7fc9d5600700 1 --2- 172.21.15.27:0/833906196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce04df0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8007f40 tx=0x7fc9c800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45ed4c3fa49dd72e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:20.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.842+0000 7fc9c1600700 1 -- 172.21.15.27:0/833906196 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8004bb0 con 0x55d20ce04300 2024-05-10T11:20:20.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.842+0000 7fc9c1600700 1 -- 172.21.15.27:0/833906196 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9c8004d50 con 0x55d20ce04300 2024-05-10T11:20:20.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.842+0000 7fc9c1600700 1 -- 172.21.15.27:0/833906196 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8019480 con 0x55d20ce04300 2024-05-10T11:20:20.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.842+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/833906196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 msgr2=0x55d20ce04df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:20.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.842+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/833906196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce04df0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8007f40 tx=0x7fc9c800b040 comp rx=0 tx=0).stop 2024-05-10T11:20:20.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.843+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/833906196 shutdown_connections 2024-05-10T11:20:20.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.843+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/833906196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce04df0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.843+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/833906196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20cdfdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.843+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/833906196 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce07ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.843+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/833906196 >> 172.21.15.27:0/833906196 conn(0x55d20cd2cb60 msgr2=0x55d20ce046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.843+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/833906196 shutdown_connections 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.843+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/833906196 wait complete. 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.843+0000 7fc9e1f8eb80 1 -- start start 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.843+0000 7fc9e1f8eb80 1 Processor -- start 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9e1f8eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20ce1b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9e1f8eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce1bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9e1f8eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce1c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9e1f8eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01b2e10 con 0x55d20cdfe600 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9e1f8eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01b2740 con 0x55d20ce039e0 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9e1f8eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d01af960 con 0x55d20ce04300 2024-05-10T11:20:20.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9e1f8eb80 10 client.0 _ll_register_callbacks cb 0x55d20ccf7c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9d4c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20ce1b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9cfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce1bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9cfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce1bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36328/0 (socket says 172.21.15.27:36328) 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9d4c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20ce1b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54046/0 (socket says 172.21.15.27:54046) 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9cfe00700 1 -- 172.21.15.27:0/709829284 learned_addr learned my addr 172.21.15.27:0/709829284 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.844+0000 7fc9d5600700 1 --2- 172.21.15.27:0/709829284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce1c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.845+0000 7fc9d4c00700 1 -- 172.21.15.27:0/709829284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 msgr2=0x55d20ce1c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.845+0000 7fc9d4c00700 1 --2- 172.21.15.27:0/709829284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce1c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.845+0000 7fc9d4c00700 1 -- 172.21.15.27:0/709829284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 msgr2=0x55d20ce1bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.845+0000 7fc9d4c00700 1 --2- 172.21.15.27:0/709829284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce1bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.845+0000 7fc9d4c00700 1 -- 172.21.15.27:0/709829284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d20ce88810 con 0x55d20cdfe600 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.845+0000 7fc9d4c00700 1 --2- 172.21.15.27:0/709829284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20ce1b680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55d20ce0afa0 tx=0x7fc9c400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3cbc4cfbea019be server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:20.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.845+0000 7fc9b6a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.845+0000 7fc9b6a00700 1 -- 172.21.15.27:0/709829284 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c400b920 con 0x55d20cdfe600 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.845+0000 7fc9b6a00700 1 -- 172.21.15.27:0/709829284 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7fc9c400bac0 con 0x55d20cdfe600 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.845+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/709829284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9c8004d50 con 0x55d20cdfe600 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.846+0000 7fc9b6a00700 1 -- 172.21.15.27:0/709829284 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc9c4036050 con 0x55d20cdfe600 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.846+0000 7fc9e1f8eb80 10 client.16155 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.846+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/709829284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d20ce89b70 con 0x55d20cdfe600 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.846+0000 7fc9b6a00700 1 -- 172.21.15.27:0/709829284 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9c8004d50 con 0x55d20cdfe600 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.846+0000 7fc9b6a00700 1 -- 172.21.15.27:0/709829284 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fc9c4026110 con 0x55d20cdfe600 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.846+0000 7fc9e1f8eb80 10 client.16155 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.846+0000 7fc9e1f8eb80 10 client.16155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.847+0000 7fc9e1f8eb80 20 client.16155 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.847+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/709829284 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d20ce8a180 0x55d20ce8c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.847+0000 7fc9e1f8eb80 4 client.16155 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:20:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.847+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/709829284 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d20cd584a0 con 0x55d20ce8a180 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.847+0000 7fc9cfe00700 1 --2- 172.21.15.27:0/709829284 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d20ce8a180 0x55d20ce8c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.847+0000 7fc9b6a00700 10 client.16155 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.847+0000 7fc9cfe00700 1 --2- 172.21.15.27:0/709829284 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d20ce8a180 0x55d20ce8c570 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3649923d574759c server_cookie=93eab6f25d795cde in_seq=0 out_seq=0 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9b6a00700 1 -- 172.21.15.27:0/709829284 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d20cd584a0 con 0x55d20ce8a180 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9b6a00700 10 client.16155 handle_command_reply: tid=0 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9e1f8eb80 1 client.16155 shutdown 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/709829284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 msgr2=0x55d20ce1b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/709829284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20ce1b680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55d20ce0afa0 tx=0x7fc9c400b260 comp rx=0 tx=0).stop 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/709829284 shutdown_connections 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/709829284 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d20ce8a180 0x55d20ce8c570 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/709829284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d20ce04300 0x55d20ce1c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/709829284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d20cdfe600 0x55d20ce1b680 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9e1f8eb80 1 --2- 172.21.15.27:0/709829284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d20ce039e0 0x55d20ce1bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/709829284 >> 172.21.15.27:0/709829284 conn(0x55d20cd2cb60 msgr2=0x55d20cdf2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9b6a00700 0 client.16155 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:20:20.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/709829284 shutdown_connections 2024-05-10T11:20:20.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.848+0000 7fc9e1f8eb80 1 -- 172.21.15.27:0/709829284 wait complete. 2024-05-10T11:20:20.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.849+0000 7fc9e1f8eb80 20 client.16155 trim_cache size 0 max 16384 2024-05-10T11:20:20.853 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:20:20.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.849+0000 7fc9d6000700 1 -- 172.21.15.27:0/485410649 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9b80b2a70 msgr2=0x7fc9b80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:20.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.849+0000 7fc9d6000700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9b80b2a70 0x7fc9b80b4f30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc9c80079a0 tx=0x7fc9c8009040 comp rx=0 tx=0).stop 2024-05-10T11:20:20.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.849+0000 7fc9d6000700 1 -- 172.21.15.27:0/485410649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9d00761f0 msgr2=0x7fc9d019c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:20.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.849+0000 7fc9d6000700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9d00761f0 0x7fc9d019c260 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00e9a0 tx=0x7fc9bc00ed60 comp rx=0 tx=0).stop 2024-05-10T11:20:20.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.849+0000 7fc9d6000700 1 -- 172.21.15.27:0/485410649 shutdown_connections 2024-05-10T11:20:20.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.849+0000 7fc9d6000700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9b80b2a70 0x7fc9b80b4f30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.849+0000 7fc9d6000700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9d010fc10 0x7fc9d01a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.849+0000 7fc9d6000700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9d00758d0 0x7fc9d019bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.849+0000 7fc9d6000700 1 --2- 172.21.15.27:0/485410649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9d00761f0 0x7fc9d019c260 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:20.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.849+0000 7fc9d6000700 1 -- 172.21.15.27:0/485410649 >> 172.21.15.27:0/485410649 conn(0x7fc9d00fef20 msgr2=0x7fc9d00ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.850+0000 7fc9d6000700 1 -- 172.21.15.27:0/485410649 shutdown_connections 2024-05-10T11:20:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:20.850+0000 7fc9d6000700 1 -- 172.21.15.27:0/485410649 wait complete. 2024-05-10T11:20:20.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:20 smithi076 ceph-mon[34152]: pgmap v218: 129 pgs: 129 active+clean; 246 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 239 op/s 2024-05-10T11:20:20.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:20 smithi189 ceph-mon[34842]: pgmap v218: 129 pgs: 129 active+clean; 246 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 239 op/s 2024-05-10T11:20:21.689 INFO:tasks.workunit.client.0.smithi027.stdout: 1 404703 6.88 MB/sec execute 186 sec latency 45.527 ms 2024-05-10T11:20:22.690 INFO:tasks.workunit.client.0.smithi027.stdout: 1 405324 6.85 MB/sec execute 187 sec latency 39.296 ms 2024-05-10T11:20:22.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:22 smithi076 ceph-mon[34152]: pgmap v219: 129 pgs: 129 active+clean; 246 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 144 op/s 2024-05-10T11:20:22.952 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:20:22.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:20:22.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:22 smithi189 ceph-mon[34842]: pgmap v219: 129 pgs: 129 active+clean; 246 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 144 op/s 2024-05-10T11:20:23.065 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:22 smithi027 ceph-mon[30054]: pgmap v219: 129 pgs: 129 active+clean; 246 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 144 op/s 2024-05-10T11:20:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:20:23.690 INFO:tasks.workunit.client.0.smithi027.stdout: 1 406778 6.85 MB/sec execute 188 sec latency 89.087 ms 2024-05-10T11:20:24.695 INFO:tasks.workunit.client.0.smithi027.stdout: 1 408225 6.88 MB/sec execute 189 sec latency 20.632 ms 2024-05-10T11:20:24.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:24 smithi076 ceph-mon[34152]: pgmap v220: 129 pgs: 129 active+clean; 246 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 144 op/s 2024-05-10T11:20:24.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:24 smithi189 ceph-mon[34842]: pgmap v220: 129 pgs: 129 active+clean; 246 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 144 op/s 2024-05-10T11:20:25.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:24 smithi027 ceph-mon[30054]: pgmap v220: 129 pgs: 129 active+clean; 246 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 144 op/s 2024-05-10T11:20:25.695 INFO:tasks.workunit.client.0.smithi027.stdout: 1 408938 6.85 MB/sec execute 190 sec latency 46.938 ms 2024-05-10T11:20:26.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.151+0000 7f0149200700 1 -- 172.21.15.27:0/3705729986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01441023a0 msgr2=0x7f0144102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:26.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.151+0000 7f0149200700 1 --2- 172.21.15.27:0/3705729986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01441023a0 0x7f0144102820 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0130009990 tx=0x7f0130009ca0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.151+0000 7f0149200700 1 -- 172.21.15.27:0/3705729986 shutdown_connections 2024-05-10T11:20:26.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.151+0000 7f0149200700 1 --2- 172.21.15.27:0/3705729986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144102d60 0x7f014410f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.151+0000 7f0149200700 1 --2- 172.21.15.27:0/3705729986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0144108410 0x7f01441087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.151+0000 7f0149200700 1 --2- 172.21.15.27:0/3705729986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01441023a0 0x7f0144102820 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.151+0000 7f0149200700 1 -- 172.21.15.27:0/3705729986 >> 172.21.15.27:0/3705729986 conn(0x7f0144079310 msgr2=0x7f0144079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:26.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.151+0000 7f0149200700 1 -- 172.21.15.27:0/3705729986 shutdown_connections 2024-05-10T11:20:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.151+0000 7f0149200700 1 -- 172.21.15.27:0/3705729986 wait complete. 2024-05-10T11:20:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.152+0000 7f0149200700 1 Processor -- start 2024-05-10T11:20:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.152+0000 7f0149200700 1 -- start start 2024-05-10T11:20:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.153+0000 7f0149200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01441023a0 0x7f014419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.153+0000 7f0149200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144102d60 0x7f014419fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.153+0000 7f0149200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0144108410 0x7f0144199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.153+0000 7f0149200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0144112580 con 0x7f0144102d60 2024-05-10T11:20:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.153+0000 7f0149200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0144112400 con 0x7f01441023a0 2024-05-10T11:20:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.153+0000 7f0149200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0144112700 con 0x7f0144108410 2024-05-10T11:20:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.153+0000 7f0141e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01441023a0 0x7f014419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.153+0000 7f0141e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01441023a0 0x7f014419f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54352/0 (socket says 172.21.15.27:54352) 2024-05-10T11:20:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.153+0000 7f0141e00700 1 -- 172.21.15.27:0/3710137548 learned_addr learned my addr 172.21.15.27:0/3710137548 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.153+0000 7f0141400700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144102d60 0x7f014419fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.153+0000 7f0142800700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0144108410 0x7f0144199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.154+0000 7f0141e00700 1 -- 172.21.15.27:0/3710137548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0144108410 msgr2=0x7f0144199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.154+0000 7f0141e00700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0144108410 0x7f0144199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.154+0000 7f0141e00700 1 -- 172.21.15.27:0/3710137548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144102d60 msgr2=0x7f014419fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.154+0000 7f0141e00700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144102d60 0x7f014419fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.154+0000 7f0141e00700 1 -- 172.21.15.27:0/3710137548 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014419a0f0 con 0x7f01441023a0 2024-05-10T11:20:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.154+0000 7f0141400700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144102d60 0x7f014419fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:20:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.154+0000 7f0141e00700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01441023a0 0x7f014419f7d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f013400e9a0 tx=0x7f013400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.154+0000 7f013aa00700 1 -- 172.21.15.27:0/3710137548 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013400cbd0 con 0x7f01441023a0 2024-05-10T11:20:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.154+0000 7f0149200700 1 -- 172.21.15.27:0/3710137548 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f014419a3e0 con 0x7f01441023a0 2024-05-10T11:20:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.154+0000 7f013aa00700 1 -- 172.21.15.27:0/3710137548 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f013400cd70 con 0x7f01441023a0 2024-05-10T11:20:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.154+0000 7f013aa00700 1 -- 172.21.15.27:0/3710137548 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134018680 con 0x7f01441023a0 2024-05-10T11:20:26.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.155+0000 7f0149200700 1 -- 172.21.15.27:0/3710137548 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01441a66a0 con 0x7f01441023a0 2024-05-10T11:20:26.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.158+0000 7f013aa00700 1 -- 172.21.15.27:0/3710137548 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0134010430 con 0x7f01441023a0 2024-05-10T11:20:26.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.160+0000 7f0153b48b80 1 Processor -- start 2024-05-10T11:20:26.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.160+0000 7f0153b48b80 1 -- start start 2024-05-10T11:20:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.160+0000 7f013aa00700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f01280b29f0 0x7f01280b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0153b48b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 0x562d10b2e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0153b48b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b28350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f013aa00700 1 -- 172.21.15.27:0/3710137548 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7f01340e2ce0 con 0x7f01441023a0 2024-05-10T11:20:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0153b48b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 0x562d10b28df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0153b48b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562d10a7aaa0 con 0x562d10b28970 2024-05-10T11:20:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0142800700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f01280b29f0 0x7f01280b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0153b48b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562d10a187e0 con 0x562d10b2dc40 2024-05-10T11:20:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0153b48b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562d10a76a60 con 0x562d10b2e560 2024-05-10T11:20:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0141e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 0x562d10b2e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0141400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b28350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0141e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 0x562d10b2e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54354/0 (socket says 172.21.15.27:54354) 2024-05-10T11:20:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0141400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b28350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:40452/0 (socket says 172.21.15.27:40452) 2024-05-10T11:20:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0141e00700 1 -- 172.21.15.27:0/3287046172 learned_addr learned my addr 172.21.15.27:0/3287046172 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0142800700 1 --2- 172.21.15.27:0/3287046172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 0x562d10b28df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0141e00700 1 -- 172.21.15.27:0/3287046172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 msgr2=0x562d10b28350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.161+0000 7f0141e00700 1 --2- 172.21.15.27:0/3287046172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b28350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.162+0000 7f0141e00700 1 -- 172.21.15.27:0/3287046172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 msgr2=0x562d10b28df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.162+0000 7f0141e00700 1 --2- 172.21.15.27:0/3287046172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 0x562d10b28df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.162+0000 7f0141e00700 1 -- 172.21.15.27:0/3287046172 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d109c7950 con 0x562d10b2dc40 2024-05-10T11:20:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.162+0000 7f0141e00700 1 --2- 172.21.15.27:0/3287046172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 0x562d10b2e020 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f01340a5bb0 tx=0x7f01340a52c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91d8dd3821e5fb55 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.162+0000 7f0142800700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f01280b29f0 0x7f01280b4eb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x562d10b29330 tx=0x7f013c009920 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.162+0000 7f012f400700 1 -- 172.21.15.27:0/3287046172 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01340aae60 con 0x562d10b2dc40 2024-05-10T11:20:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.162+0000 7f012f400700 1 -- 172.21.15.27:0/3287046172 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01340b22d0 con 0x562d10b2dc40 2024-05-10T11:20:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.162+0000 7f012f400700 1 -- 172.21.15.27:0/3287046172 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01340b2760 con 0x562d10b2dc40 2024-05-10T11:20:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.163+0000 7f0153b48b80 1 -- 172.21.15.27:0/3287046172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 msgr2=0x562d10b2e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.163+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3287046172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 0x562d10b2e020 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f01340a5bb0 tx=0x7f01340a52c0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.163+0000 7f0153b48b80 1 -- 172.21.15.27:0/3287046172 shutdown_connections 2024-05-10T11:20:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.163+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3287046172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 0x562d10b28df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.163+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3287046172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 0x562d10b2e020 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.163+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3287046172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b28350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.163+0000 7f0153b48b80 1 -- 172.21.15.27:0/3287046172 >> 172.21.15.27:0/3287046172 conn(0x562d10aa0750 msgr2=0x562d10b22b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.163+0000 7f0153b48b80 1 -- 172.21.15.27:0/3287046172 shutdown_connections 2024-05-10T11:20:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.163+0000 7f0153b48b80 1 -- 172.21.15.27:0/3287046172 wait complete. 2024-05-10T11:20:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.164+0000 7f0153b48b80 1 -- start start 2024-05-10T11:20:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.164+0000 7f0153b48b80 1 Processor -- start 2024-05-10T11:20:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.165+0000 7f0153b48b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 0x562d10b342d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.165+0000 7f0153b48b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2dc40 0x562d10b31670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.165+0000 7f0153b48b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2e560 0x562d10b31bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.165+0000 7f0153b48b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562d10b34f40 con 0x562d10b28970 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.165+0000 7f0153b48b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562d109e0ba0 con 0x562d10b2e560 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.165+0000 7f0153b48b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562d10b350c0 con 0x562d10b2dc40 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.165+0000 7f0153b48b80 10 client.0 _ll_register_callbacks cb 0x562d10a21cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.165+0000 7f0141400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2dc40 0x562d10b31670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.165+0000 7f0141400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2dc40 0x562d10b31670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:40464/0 (socket says 172.21.15.27:40464) 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.165+0000 7f0141400700 1 -- 172.21.15.27:0/3429949387 learned_addr learned my addr 172.21.15.27:0/3429949387 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.165+0000 7f0141e00700 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 0x562d10b342d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.165+0000 7f0142800700 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2e560 0x562d10b31bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.166+0000 7f0141400700 1 -- 172.21.15.27:0/3429949387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2e560 msgr2=0x562d10b31bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.166+0000 7f0141400700 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2e560 0x562d10b31bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.166+0000 7f0141400700 1 -- 172.21.15.27:0/3429949387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 msgr2=0x562d10b342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.166+0000 7f0141400700 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 0x562d10b342d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.166+0000 7f0141400700 1 -- 172.21.15.27:0/3429949387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d10a404a0 con 0x562d10b2dc40 2024-05-10T11:20:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.166+0000 7f0141400700 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2dc40 0x562d10b31670 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0130009990 tx=0x7f013000be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a35ffe6458ef69ad server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.166+0000 7f0123e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:20:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.166+0000 7f0123e00700 1 -- 172.21.15.27:0/3429949387 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013000bea0 con 0x562d10b2dc40 2024-05-10T11:20:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.166+0000 7f0123e00700 1 -- 172.21.15.27:0/3429949387 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7f013000fac0 con 0x562d10b2dc40 2024-05-10T11:20:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.166+0000 7f0153b48b80 1 -- 172.21.15.27:0/3429949387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562d10af51c0 con 0x562d10b2dc40 2024-05-10T11:20:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.167+0000 7f0123e00700 1 -- 172.21.15.27:0/3429949387 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0130030050 con 0x562d10b2dc40 2024-05-10T11:20:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.167+0000 7f0153b48b80 10 client.26368 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.167+0000 7f0153b48b80 1 -- 172.21.15.27:0/3429949387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562d10b3f840 con 0x562d10b2dc40 2024-05-10T11:20:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.167+0000 7f0123e00700 1 -- 172.21.15.27:0/3429949387 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562d10af51c0 con 0x562d10b2dc40 2024-05-10T11:20:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.167+0000 7f0123e00700 1 -- 172.21.15.27:0/3429949387 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f013002b030 con 0x562d10b2dc40 2024-05-10T11:20:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.167+0000 7f0153b48b80 10 client.26368 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.167+0000 7f0153b48b80 10 client.26368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:20:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.167+0000 7f0153b48b80 20 client.26368 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.167+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562d10b41e80 0x562d10b442b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.167+0000 7f0153b48b80 4 client.26368 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:20:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.167+0000 7f0153b48b80 1 -- 172.21.15.27:0/3429949387 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d10a88480 con 0x562d10b41e80 2024-05-10T11:20:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.168+0000 7f0141e00700 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562d10b41e80 0x562d10b442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.168+0000 7f0141e00700 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562d10b41e80 0x562d10b442b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f25c15372d8504c server_cookie=4e80a90c87b39a3c in_seq=0 out_seq=0 2024-05-10T11:20:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.168+0000 7f0123e00700 10 client.26368 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:20:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.169+0000 7f0123e00700 1 -- 172.21.15.27:0/3429949387 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562d10a88480 con 0x562d10b41e80 2024-05-10T11:20:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.169+0000 7f0123e00700 10 client.26368 handle_command_reply: tid=0 2024-05-10T11:20:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.169+0000 7f0153b48b80 1 client.26368 shutdown 2024-05-10T11:20:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.169+0000 7f0153b48b80 1 -- 172.21.15.27:0/3429949387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2dc40 msgr2=0x562d10b31670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.169+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2dc40 0x562d10b31670 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0130009990 tx=0x7f013000be30 comp rx=0 tx=0).stop 2024-05-10T11:20:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.170+0000 7f0153b48b80 1 -- 172.21.15.27:0/3429949387 shutdown_connections 2024-05-10T11:20:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.170+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562d10b41e80 0x562d10b442b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.170+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2e560 0x562d10b31bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.170+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 0x562d10b342d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.170+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3429949387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2dc40 0x562d10b31670 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.170+0000 7f0153b48b80 1 -- 172.21.15.27:0/3429949387 >> 172.21.15.27:0/3429949387 conn(0x562d10aa0750 msgr2=0x562d10b22210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:26.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.170+0000 7f0123e00700 0 client.26368 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:20:26.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.170+0000 7f0153b48b80 1 -- 172.21.15.27:0/3429949387 shutdown_connections 2024-05-10T11:20:26.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.170+0000 7f0153b48b80 1 -- 172.21.15.27:0/3429949387 wait complete. 2024-05-10T11:20:26.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.170+0000 7f0153b48b80 20 client.26368 trim_cache size 0 max 16384 2024-05-10T11:20:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.197+0000 7f0153b48b80 1 Processor -- start 2024-05-10T11:20:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.197+0000 7f0153b48b80 1 -- start start 2024-05-10T11:20:26.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.198+0000 7f0153b48b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b28970 0x562d10b28350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.198+0000 7f0153b48b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b2dc40 0x562d10b3dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.198+0000 7f0153b48b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b2f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.198+0000 7f0153b48b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f01441ad630 con 0x562d10b2dc40 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.198+0000 7f0153b48b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f01441af170 con 0x562d10b28970 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.198+0000 7f0153b48b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f01441add00 con 0x562d10b2e560 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.198+0000 7f0141e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b28970 0x562d10b28350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.198+0000 7f0141e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b28970 0x562d10b28350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54382/0 (socket says 172.21.15.27:54382) 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.198+0000 7f0141e00700 1 -- 172.21.15.27:0/3408285391 learned_addr learned my addr 172.21.15.27:0/3408285391 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.198+0000 7f0142800700 1 --2- 172.21.15.27:0/3408285391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b2f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.198+0000 7f0141e00700 1 -- 172.21.15.27:0/3408285391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 msgr2=0x562d10b2f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.199+0000 7f0141e00700 1 --2- 172.21.15.27:0/3408285391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b2f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.199+0000 7f0141e00700 1 -- 172.21.15.27:0/3408285391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b2dc40 msgr2=0x562d10b3dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.199+0000 7f0141400700 1 --2- 172.21.15.27:0/3408285391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b2dc40 0x562d10b3dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.199+0000 7f0141e00700 1 --2- 172.21.15.27:0/3408285391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b2dc40 0x562d10b3dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.199+0000 7f0141e00700 1 -- 172.21.15.27:0/3408285391 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d10b1db30 con 0x562d10b28970 2024-05-10T11:20:26.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.199+0000 7f0142800700 1 --2- 172.21.15.27:0/3408285391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b2f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:20:26.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.199+0000 7f0141400700 1 --2- 172.21.15.27:0/3408285391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b2dc40 0x562d10b3dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:20:26.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.199+0000 7f0141e00700 1 --2- 172.21.15.27:0/3408285391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b28970 0x562d10b28350 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f013400c5b0 tx=0x7f0134010cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1313fc1bbdc98951 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:26.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.199+0000 7f012ea00700 1 -- 172.21.15.27:0/3408285391 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013408e070 con 0x562d10b28970 2024-05-10T11:20:26.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.199+0000 7f012ea00700 1 -- 172.21.15.27:0/3408285391 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01340b2990 con 0x562d10b28970 2024-05-10T11:20:26.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.199+0000 7f012ea00700 1 -- 172.21.15.27:0/3408285391 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01340aae60 con 0x562d10b28970 2024-05-10T11:20:26.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.200+0000 7f0153b48b80 1 -- 172.21.15.27:0/3408285391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b28970 msgr2=0x562d10b28350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.200+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3408285391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b28970 0x562d10b28350 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f013400c5b0 tx=0x7f0134010cb0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.200+0000 7f0153b48b80 1 -- 172.21.15.27:0/3408285391 shutdown_connections 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.200+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3408285391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b2f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.200+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3408285391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b28970 0x562d10b28350 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.200+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3408285391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b2dc40 0x562d10b3dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.200+0000 7f0153b48b80 1 -- 172.21.15.27:0/3408285391 >> 172.21.15.27:0/3408285391 conn(0x562d10aa0750 msgr2=0x562d10b2e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.200+0000 7f0153b48b80 1 -- 172.21.15.27:0/3408285391 shutdown_connections 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.200+0000 7f0153b48b80 1 -- 172.21.15.27:0/3408285391 wait complete. 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.201+0000 7f0153b48b80 1 -- start start 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.201+0000 7f0153b48b80 1 Processor -- start 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0153b48b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 0x562d10b3a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0153b48b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 0x562d10b37320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0153b48b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b37860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0153b48b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f01441aeaa0 con 0x562d10b28970 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0153b48b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f01441ae3d0 con 0x562d10b2dc40 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0153b48b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f01441ab5f0 con 0x562d10b2e560 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0153b48b80 10 client.0 _ll_register_callbacks cb 0x562d10a21cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0141400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 0x562d10b37320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0142800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b37860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0141400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 0x562d10b37320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54390/0 (socket says 172.21.15.27:54390) 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0141400700 1 -- 172.21.15.27:0/3050941385 learned_addr learned my addr 172.21.15.27:0/3050941385 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0141e00700 1 --2- 172.21.15.27:0/3050941385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 0x562d10b3a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0141400700 1 -- 172.21.15.27:0/3050941385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 msgr2=0x562d10b37860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0141400700 1 --2- 172.21.15.27:0/3050941385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b37860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0141400700 1 -- 172.21.15.27:0/3050941385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 msgr2=0x562d10b3a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0141400700 1 --2- 172.21.15.27:0/3050941385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 0x562d10b3a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.202+0000 7f0141400700 1 -- 172.21.15.27:0/3050941385 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d10b37da0 con 0x562d10b2dc40 2024-05-10T11:20:26.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.203+0000 7f0141400700 1 --2- 172.21.15.27:0/3050941385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 0x562d10b37320 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f013000b2e0 tx=0x7f013000f9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b20e89bce8b7efc server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:26.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.203+0000 7f0123e00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:20:26.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.203+0000 7f0123e00700 1 -- 172.21.15.27:0/3050941385 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013000bea0 con 0x562d10b2dc40 2024-05-10T11:20:26.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.203+0000 7f0153b48b80 1 -- 172.21.15.27:0/3050941385 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01340b2990 con 0x562d10b2dc40 2024-05-10T11:20:26.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.203+0000 7f0123e00700 1 -- 172.21.15.27:0/3050941385 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7f0130005270 con 0x562d10b2dc40 2024-05-10T11:20:26.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.203+0000 7f0123e00700 1 -- 172.21.15.27:0/3050941385 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0130043050 con 0x562d10b2dc40 2024-05-10T11:20:26.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.203+0000 7f0153b48b80 10 client.26384 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:26.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.203+0000 7f0153b48b80 1 -- 172.21.15.27:0/3050941385 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562d10bad420 con 0x562d10b2dc40 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.204+0000 7f0123e00700 1 -- 172.21.15.27:0/3050941385 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01340b2990 con 0x562d10b2dc40 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.204+0000 7f0123e00700 1 -- 172.21.15.27:0/3050941385 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f013002b070 con 0x562d10b2dc40 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.204+0000 7f0153b48b80 10 client.26384 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.204+0000 7f0153b48b80 10 client.26384 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.204+0000 7f0153b48b80 20 client.26384 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.204+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3050941385 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562d10badaa0 0x562d10bafec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.204+0000 7f0153b48b80 4 client.26384 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.204+0000 7f0153b48b80 1 -- 172.21.15.27:0/3050941385 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562d10a82520 con 0x562d10badaa0 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.204+0000 7f0141e00700 1 --2- 172.21.15.27:0/3050941385 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562d10badaa0 0x562d10bafec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.205+0000 7f0141e00700 1 --2- 172.21.15.27:0/3050941385 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562d10badaa0 0x562d10bafec0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77259ff231bb160f server_cookie=4153b0073545c0a1 in_seq=0 out_seq=0 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.205+0000 7f0123e00700 10 client.26384 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0123e00700 1 -- 172.21.15.27:0/3050941385 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562d10a82520 con 0x562d10badaa0 2024-05-10T11:20:26.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0123e00700 10 client.26384 handle_command_reply: tid=0 2024-05-10T11:20:26.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0153b48b80 1 client.26384 shutdown 2024-05-10T11:20:26.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0153b48b80 1 -- 172.21.15.27:0/3050941385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 msgr2=0x562d10b37320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:26.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3050941385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 0x562d10b37320 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f013000b2e0 tx=0x7f013000f9f0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0153b48b80 1 -- 172.21.15.27:0/3050941385 shutdown_connections 2024-05-10T11:20:26.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3050941385 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562d10badaa0 0x562d10bafec0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3050941385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d10b2e560 0x562d10b37860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3050941385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d10b28970 0x562d10b3a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0153b48b80 1 --2- 172.21.15.27:0/3050941385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562d10b2dc40 0x562d10b37320 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0123e00700 0 client.26384 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:20:26.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0153b48b80 1 -- 172.21.15.27:0/3050941385 >> 172.21.15.27:0/3050941385 conn(0x562d10aa0750 msgr2=0x562d10b2f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:26.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0153b48b80 1 -- 172.21.15.27:0/3050941385 shutdown_connections 2024-05-10T11:20:26.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.207+0000 7f0153b48b80 1 -- 172.21.15.27:0/3050941385 wait complete. 2024-05-10T11:20:26.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.208+0000 7f0153b48b80 20 client.26384 trim_cache size 0 max 16384 2024-05-10T11:20:26.210 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:20:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:20:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "63d2eeed-c6ee-4c72-8d29-ce18158f899c", 2024-05-10T11:20:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:20:26.210 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:20:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.208+0000 7f0149200700 1 -- 172.21.15.27:0/3710137548 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f01280b29f0 msgr2=0x7f01280b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.208+0000 7f0149200700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f01280b29f0 0x7f01280b4eb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x562d10b29330 tx=0x7f013c009920 comp rx=0 tx=0).stop 2024-05-10T11:20:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.208+0000 7f0149200700 1 -- 172.21.15.27:0/3710137548 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01441023a0 msgr2=0x7f014419f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.208+0000 7f0149200700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01441023a0 0x7f014419f7d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f013400e9a0 tx=0x7f013400ecb0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.209+0000 7f0149200700 1 -- 172.21.15.27:0/3710137548 shutdown_connections 2024-05-10T11:20:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.209+0000 7f0149200700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f01280b29f0 0x7f01280b4eb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.209+0000 7f0149200700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0144108410 0x7f0144199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.209+0000 7f0149200700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01441023a0 0x7f014419f7d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.209+0000 7f0149200700 1 --2- 172.21.15.27:0/3710137548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144102d60 0x7f014419fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.209+0000 7f0149200700 1 -- 172.21.15.27:0/3710137548 >> 172.21.15.27:0/3710137548 conn(0x7f0144079310 msgr2=0x7f0144101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.209+0000 7f0149200700 1 -- 172.21.15.27:0/3710137548 shutdown_connections 2024-05-10T11:20:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:26.209+0000 7f0149200700 1 -- 172.21.15.27:0/3710137548 wait complete. 2024-05-10T11:20:26.695 INFO:tasks.workunit.client.0.smithi027.stdout: 1 410390 6.84 MB/sec execute 191 sec latency 30.667 ms 2024-05-10T11:20:26.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:26 smithi076 ceph-mon[34152]: pgmap v221: 129 pgs: 129 active+clean; 252 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-05-10T11:20:26.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:26 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:20:26.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:26 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:26.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:26 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:20:26.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:26 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:20:26.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:26 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:26.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:26 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:20:26.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:26 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:26.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:26 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:20:26.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:26 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:26.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:26 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:26.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:26 smithi189 ceph-mon[34842]: pgmap v221: 129 pgs: 129 active+clean; 252 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-05-10T11:20:26.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:26 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:20:26.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:26 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:26.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:26 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:20:26.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:26 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:20:26.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:26 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:26.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:26 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:20:26.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:26 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:26.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:26 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:20:26.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:26 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:26.988 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:26 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:27.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:26 smithi027 ceph-mon[30054]: pgmap v221: 129 pgs: 129 active+clean; 252 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-05-10T11:20:27.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:26 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:20:27.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:26 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:26 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:20:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:26 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:20:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:26 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:26 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:20:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:26 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:26 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:20:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:26 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:27.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:26 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:20:27.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:20:27.683 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:20:27.695 INFO:tasks.workunit.client.0.smithi027.stdout: 1 411862 6.87 MB/sec execute 192 sec latency 20.320 ms 2024-05-10T11:20:28.695 INFO:tasks.workunit.client.0.smithi027.stdout: 1 412894 6.86 MB/sec execute 193 sec latency 85.962 ms 2024-05-10T11:20:28.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:28 smithi076 ceph-mon[34152]: pgmap v222: 129 pgs: 129 active+clean; 252 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 137 op/s 2024-05-10T11:20:28.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:28 smithi189 ceph-mon[34842]: pgmap v222: 129 pgs: 129 active+clean; 252 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 137 op/s 2024-05-10T11:20:29.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:28 smithi027 ceph-mon[30054]: pgmap v222: 129 pgs: 129 active+clean; 252 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 137 op/s 2024-05-10T11:20:29.695 INFO:tasks.workunit.client.0.smithi027.stdout: 1 414369 6.86 MB/sec execute 194 sec latency 35.736 ms 2024-05-10T11:20:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.088+0000 7f62dd000700 1 -- 172.21.15.27:0/4019329666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d8104eb0 msgr2=0x7f62d810d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.088+0000 7f62dd000700 1 --2- 172.21.15.27:0/4019329666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d8104eb0 0x7f62d810d3a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009990 tx=0x7f62c4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.088+0000 7f62dd000700 1 -- 172.21.15.27:0/4019329666 shutdown_connections 2024-05-10T11:20:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.088+0000 7f62dd000700 1 --2- 172.21.15.27:0/4019329666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62d810d9c0 0x7f62d810fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.088+0000 7f62dd000700 1 --2- 172.21.15.27:0/4019329666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62d8104590 0x7f62d8104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.088+0000 7f62dd000700 1 --2- 172.21.15.27:0/4019329666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d8104eb0 0x7f62d810d3a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.088+0000 7f62dd000700 1 -- 172.21.15.27:0/4019329666 >> 172.21.15.27:0/4019329666 conn(0x7f62d80fddc0 msgr2=0x7f62d81001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.088+0000 7f62dd000700 1 -- 172.21.15.27:0/4019329666 shutdown_connections 2024-05-10T11:20:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.088+0000 7f62dd000700 1 -- 172.21.15.27:0/4019329666 wait complete. 2024-05-10T11:20:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.089+0000 7f62dd000700 1 Processor -- start 2024-05-10T11:20:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.090+0000 7f62dd000700 1 -- start start 2024-05-10T11:20:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.090+0000 7f62dd000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62d8104590 0x7f62d819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.090+0000 7f62dd000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62d8104eb0 0x7f62d819c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.090+0000 7f62dd000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d810d9c0 0x7f62d81a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.090+0000 7f62dd000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f62d81135e0 con 0x7f62d8104590 2024-05-10T11:20:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.090+0000 7f62dd000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f62d8113460 con 0x7f62d8104eb0 2024-05-10T11:20:30.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.090+0000 7f62dd000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f62d8113760 con 0x7f62d810d9c0 2024-05-10T11:20:30.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.090+0000 7f62d6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62d8104eb0 0x7f62d819c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.090+0000 7f62d7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62d8104590 0x7f62d819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.090+0000 7f62d6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62d8104eb0 0x7f62d819c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54410/0 (socket says 172.21.15.27:54410) 2024-05-10T11:20:30.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.091+0000 7f62d6a00700 1 -- 172.21.15.27:0/2569034984 learned_addr learned my addr 172.21.15.27:0/2569034984 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:30.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.091+0000 7f62d7e00700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d810d9c0 0x7f62d81a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.091+0000 7f62d6a00700 1 -- 172.21.15.27:0/2569034984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d810d9c0 msgr2=0x7f62d81a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:30.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.091+0000 7f62d6a00700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d810d9c0 0x7f62d81a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.091+0000 7f62d6a00700 1 -- 172.21.15.27:0/2569034984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62d8104590 msgr2=0x7f62d819bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:30.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.091+0000 7f62d6a00700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62d8104590 0x7f62d819bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.091+0000 7f62d6a00700 1 -- 172.21.15.27:0/2569034984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d81a0d20 con 0x7f62d8104eb0 2024-05-10T11:20:30.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.091+0000 7f62d6a00700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62d8104eb0 0x7f62d819c250 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f62c4000c00 tx=0x7f62c40047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:30.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.092+0000 7f62cfe00700 1 -- 172.21.15.27:0/2569034984 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4026070 con 0x7f62d8104eb0 2024-05-10T11:20:30.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.092+0000 7f62dd000700 1 -- 172.21.15.27:0/2569034984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d81a0fb0 con 0x7f62d8104eb0 2024-05-10T11:20:30.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.092+0000 7f62cfe00700 1 -- 172.21.15.27:0/2569034984 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c4021410 con 0x7f62d8104eb0 2024-05-10T11:20:30.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.092+0000 7f62cfe00700 1 -- 172.21.15.27:0/2569034984 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4017650 con 0x7f62d8104eb0 2024-05-10T11:20:30.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.092+0000 7f62dd000700 1 -- 172.21.15.27:0/2569034984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d81a8890 con 0x7f62d8104eb0 2024-05-10T11:20:30.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.096+0000 7f62cfe00700 1 -- 172.21.15.27:0/2569034984 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f62c4029070 con 0x7f62d8104eb0 2024-05-10T11:20:30.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.097+0000 7f62e9012b80 1 Processor -- start 2024-05-10T11:20:30.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.097+0000 7f62cfe00700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f62bc0b29f0 0x7f62bc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.097+0000 7f62e9012b80 1 -- start start 2024-05-10T11:20:30.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.098+0000 7f62cfe00700 1 -- 172.21.15.27:0/2569034984 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7f62c40e3780 con 0x7f62d8104eb0 2024-05-10T11:20:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.098+0000 7f62d7e00700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f62bc0b29f0 0x7f62bc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.098+0000 7f62e9012b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4eda0 0x55d15aa4f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.098+0000 7f62e9012b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa4f6c0 0x55d15aa49430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.098+0000 7f62e9012b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa49a50 0x55d15aa49ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.098+0000 7f62e9012b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d15a99ba20 con 0x55d15aa4f6c0 2024-05-10T11:20:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.098+0000 7f62e9012b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d15a939760 con 0x55d15aa4eda0 2024-05-10T11:20:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.098+0000 7f62e9012b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d15a9979e0 con 0x55d15aa49a50 2024-05-10T11:20:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.098+0000 7f62d7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4eda0 0x55d15aa4f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.098+0000 7f62d7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4eda0 0x55d15aa4f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54424/0 (socket says 172.21.15.27:54424) 2024-05-10T11:20:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.098+0000 7f62d7400700 1 -- 172.21.15.27:0/3534693021 learned_addr learned my addr 172.21.15.27:0/3534693021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.098+0000 7f62d6a00700 1 --2- 172.21.15.27:0/3534693021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa4f6c0 0x55d15aa49430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.099+0000 7f62d7e00700 1 --2- 172.21.15.27:0/3534693021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa49a50 0x55d15aa49ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.099+0000 7f62d7400700 1 -- 172.21.15.27:0/3534693021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa49a50 msgr2=0x55d15aa49ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:30.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.099+0000 7f62d7400700 1 --2- 172.21.15.27:0/3534693021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa49a50 0x55d15aa49ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.099+0000 7f62d7400700 1 -- 172.21.15.27:0/3534693021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa4f6c0 msgr2=0x55d15aa49430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:30.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.099+0000 7f62d7400700 1 --2- 172.21.15.27:0/3534693021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa4f6c0 0x55d15aa49430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.099+0000 7f62d7400700 1 -- 172.21.15.27:0/3534693021 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d15a8e88d0 con 0x55d15aa4eda0 2024-05-10T11:20:30.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.099+0000 7f62d7e00700 1 --2- 172.21.15.27:0/3534693021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa49a50 0x55d15aa49ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:20:30.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.099+0000 7f62d6a00700 1 --2- 172.21.15.27:0/3534693021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa4f6c0 0x55d15aa49430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:20:30.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.099+0000 7f62d7400700 1 --2- 172.21.15.27:0/3534693021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4eda0 0x55d15aa4f180 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009d80 tx=0x7f62c8011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=690f23409d6be627 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:30.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.100+0000 7f62d7e00700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f62bc0b29f0 0x7f62bc0b4eb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55d15aa4a410 tx=0x7f62d0009920 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:30.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.099+0000 7f62ccc00700 1 -- 172.21.15.27:0/3534693021 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c8004230 con 0x55d15aa4eda0 2024-05-10T11:20:30.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.100+0000 7f62ccc00700 1 -- 172.21.15.27:0/3534693021 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c80043d0 con 0x55d15aa4eda0 2024-05-10T11:20:30.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.100+0000 7f62ccc00700 1 -- 172.21.15.27:0/3534693021 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c8019830 con 0x55d15aa4eda0 2024-05-10T11:20:30.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.100+0000 7f62e9012b80 1 -- 172.21.15.27:0/3534693021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4eda0 msgr2=0x55d15aa4f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:30.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.100+0000 7f62e9012b80 1 --2- 172.21.15.27:0/3534693021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4eda0 0x55d15aa4f180 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009d80 tx=0x7f62c8011720 comp rx=0 tx=0).stop 2024-05-10T11:20:30.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.101+0000 7f62e9012b80 1 -- 172.21.15.27:0/3534693021 shutdown_connections 2024-05-10T11:20:30.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.101+0000 7f62e9012b80 1 --2- 172.21.15.27:0/3534693021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa49a50 0x55d15aa49ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.101+0000 7f62e9012b80 1 --2- 172.21.15.27:0/3534693021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4eda0 0x55d15aa4f180 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.101+0000 7f62e9012b80 1 --2- 172.21.15.27:0/3534693021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa4f6c0 0x55d15aa49430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.101+0000 7f62e9012b80 1 -- 172.21.15.27:0/3534693021 >> 172.21.15.27:0/3534693021 conn(0x55d15a9c16d0 msgr2=0x55d15aa43ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:30.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.101+0000 7f62e9012b80 1 -- 172.21.15.27:0/3534693021 shutdown_connections 2024-05-10T11:20:30.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.101+0000 7f62e9012b80 1 -- 172.21.15.27:0/3534693021 wait complete. 2024-05-10T11:20:30.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.101+0000 7f62e9012b80 1 -- start start 2024-05-10T11:20:30.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.102+0000 7f62e9012b80 1 Processor -- start 2024-05-10T11:20:30.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.102+0000 7f62e9012b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa49a50 0x55d15aa55370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.102+0000 7f62e9012b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa4eda0 0x55d15aa52710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.102+0000 7f62e9012b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4f6c0 0x55d15aa52c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.102+0000 7f62e9012b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d15aa56150 con 0x55d15aa4eda0 2024-05-10T11:20:30.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.102+0000 7f62e9012b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d15a901b20 con 0x55d15aa49a50 2024-05-10T11:20:30.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.102+0000 7f62e9012b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d15aa562d0 con 0x55d15aa4f6c0 2024-05-10T11:20:30.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.103+0000 7f62e9012b80 10 client.0 _ll_register_callbacks cb 0x55d15a942c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:30.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.103+0000 7f62d7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4f6c0 0x55d15aa52c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.103+0000 7f62d7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa49a50 0x55d15aa55370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.103+0000 7f62d6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa4eda0 0x55d15aa52710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.103+0000 7f62d7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4f6c0 0x55d15aa52c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:40524/0 (socket says 172.21.15.27:40524) 2024-05-10T11:20:30.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.103+0000 7f62d7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa49a50 0x55d15aa55370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54432/0 (socket says 172.21.15.27:54432) 2024-05-10T11:20:30.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.103+0000 7f62d7e00700 1 -- 172.21.15.27:0/4094684561 learned_addr learned my addr 172.21.15.27:0/4094684561 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:30.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.103+0000 7f62d7400700 1 -- 172.21.15.27:0/4094684561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4f6c0 msgr2=0x55d15aa52c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:30.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.103+0000 7f62d7400700 1 --2- 172.21.15.27:0/4094684561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4f6c0 0x55d15aa52c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.103+0000 7f62d7400700 1 -- 172.21.15.27:0/4094684561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa4eda0 msgr2=0x55d15aa52710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:30.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.103+0000 7f62d7400700 1 --2- 172.21.15.27:0/4094684561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa4eda0 0x55d15aa52710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.103+0000 7f62d7400700 1 -- 172.21.15.27:0/4094684561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d15a961420 con 0x55d15aa49a50 2024-05-10T11:20:30.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.104+0000 7f62d7400700 1 --2- 172.21.15.27:0/4094684561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa49a50 0x55d15aa55370 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f62c8005750 tx=0x7f62c8005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f66d8a4ec97436bf server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:30.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.104+0000 7f62c1600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:20:30.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.104+0000 7f62c1600700 1 -- 172.21.15.27:0/4094684561 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c8027070 con 0x55d15aa49a50 2024-05-10T11:20:30.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.104+0000 7f62c1600700 1 -- 172.21.15.27:0/4094684561 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7f62c8004b60 con 0x55d15aa49a50 2024-05-10T11:20:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.104+0000 7f62e9012b80 1 -- 172.21.15.27:0/4094684561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62c80043d0 con 0x55d15aa49a50 2024-05-10T11:20:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.104+0000 7f62c1600700 1 -- 172.21.15.27:0/4094684561 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f62c8032050 con 0x55d15aa49a50 2024-05-10T11:20:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.104+0000 7f62e9012b80 10 client.26396 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.104+0000 7f62e9012b80 1 -- 172.21.15.27:0/4094684561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d15aa608c0 con 0x55d15aa49a50 2024-05-10T11:20:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.105+0000 7f62c1600700 1 -- 172.21.15.27:0/4094684561 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c80043d0 con 0x55d15aa49a50 2024-05-10T11:20:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.105+0000 7f62c1600700 1 -- 172.21.15.27:0/4094684561 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f62c80223c0 con 0x55d15aa49a50 2024-05-10T11:20:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.105+0000 7f62e9012b80 10 client.26396 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.105+0000 7f62e9012b80 10 client.26396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:20:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.105+0000 7f62e9012b80 20 client.26396 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.105+0000 7f62e9012b80 1 --2- 172.21.15.27:0/4094684561 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d15aa62df0 0x55d15aa651e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.105+0000 7f62e9012b80 4 client.26396 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:20:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.105+0000 7f62e9012b80 1 -- 172.21.15.27:0/4094684561 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d15aa16140 con 0x55d15aa62df0 2024-05-10T11:20:30.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.106+0000 7f62d6a00700 1 --2- 172.21.15.27:0/4094684561 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d15aa62df0 0x55d15aa651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.106+0000 7f62d6a00700 1 --2- 172.21.15.27:0/4094684561 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d15aa62df0 0x55d15aa651e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a9cf9e1c54272f server_cookie=37dca8c710275b1f in_seq=0 out_seq=0 2024-05-10T11:20:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.106+0000 7f62c1600700 10 client.26396 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:20:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.107+0000 7f62c1600700 1 -- 172.21.15.27:0/4094684561 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d15aa16140 con 0x55d15aa62df0 2024-05-10T11:20:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.107+0000 7f62c1600700 10 client.26396 handle_command_reply: tid=0 2024-05-10T11:20:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.107+0000 7f62e9012b80 1 client.26396 shutdown 2024-05-10T11:20:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.108+0000 7f62e9012b80 1 -- 172.21.15.27:0/4094684561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa49a50 msgr2=0x55d15aa55370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.108+0000 7f62e9012b80 1 --2- 172.21.15.27:0/4094684561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa49a50 0x55d15aa55370 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f62c8005750 tx=0x7f62c8005780 comp rx=0 tx=0).stop 2024-05-10T11:20:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.108+0000 7f62e9012b80 1 -- 172.21.15.27:0/4094684561 shutdown_connections 2024-05-10T11:20:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.108+0000 7f62e9012b80 1 --2- 172.21.15.27:0/4094684561 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d15aa62df0 0x55d15aa651e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.108+0000 7f62e9012b80 1 --2- 172.21.15.27:0/4094684561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4f6c0 0x55d15aa52c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.108+0000 7f62e9012b80 1 --2- 172.21.15.27:0/4094684561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa49a50 0x55d15aa55370 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.108+0000 7f62e9012b80 1 --2- 172.21.15.27:0/4094684561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa4eda0 0x55d15aa52710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.108+0000 7f62e9012b80 1 -- 172.21.15.27:0/4094684561 >> 172.21.15.27:0/4094684561 conn(0x55d15a9c16d0 msgr2=0x55d15aa3e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.108+0000 7f62c1600700 0 client.26396 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:20:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.108+0000 7f62e9012b80 1 -- 172.21.15.27:0/4094684561 shutdown_connections 2024-05-10T11:20:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.108+0000 7f62e9012b80 1 -- 172.21.15.27:0/4094684561 wait complete. 2024-05-10T11:20:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.108+0000 7f62e9012b80 20 client.26396 trim_cache size 0 max 16384 2024-05-10T11:20:30.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.133+0000 7f62e9012b80 1 Processor -- start 2024-05-10T11:20:30.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.133+0000 7f62e9012b80 1 -- start start 2024-05-10T11:20:30.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.133+0000 7f62e9012b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa49a50 0x55d15aa5f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.133+0000 7f62e9012b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4eda0 0x55d15aa518d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.133+0000 7f62e9012b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4f6c0 0x55d15aa3e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.133+0000 7f62e9012b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f62d81ad360 con 0x55d15aa49a50 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.133+0000 7f62e9012b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f62d81aeea0 con 0x55d15aa4f6c0 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.133+0000 7f62e9012b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f62d81ada30 con 0x55d15aa4eda0 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.133+0000 7f62d6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4eda0 0x55d15aa518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.134+0000 7f62d7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4f6c0 0x55d15aa3e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.134+0000 7f62d7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa49a50 0x55d15aa5f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.134+0000 7f62d6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4eda0 0x55d15aa518d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:40532/0 (socket says 172.21.15.27:40532) 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.134+0000 7f62d7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4f6c0 0x55d15aa3e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54446/0 (socket says 172.21.15.27:54446) 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.134+0000 7f62d6a00700 1 -- 172.21.15.27:0/3725062125 learned_addr learned my addr 172.21.15.27:0/3725062125 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.134+0000 7f62d6a00700 1 -- 172.21.15.27:0/3725062125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4f6c0 msgr2=0x55d15aa3e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.134+0000 7f62d6a00700 1 --2- 172.21.15.27:0/3725062125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4f6c0 0x55d15aa3e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.134+0000 7f62d6a00700 1 -- 172.21.15.27:0/3725062125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa49a50 msgr2=0x55d15aa5f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.134+0000 7f62d6a00700 1 --2- 172.21.15.27:0/3725062125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa49a50 0x55d15aa5f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.134+0000 7f62d6a00700 1 -- 172.21.15.27:0/3725062125 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d15aa408a0 con 0x55d15aa4eda0 2024-05-10T11:20:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.134+0000 7f62d6a00700 1 --2- 172.21.15.27:0/3725062125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4eda0 0x55d15aa518d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f62c40a66e0 tx=0x7f62c4021c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cdbbb49285cda21 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.135+0000 7f62c3e00700 1 -- 172.21.15.27:0/3725062125 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c40aa660 con 0x55d15aa4eda0 2024-05-10T11:20:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.135+0000 7f62c3e00700 1 -- 172.21.15.27:0/3725062125 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c40aea10 con 0x55d15aa4eda0 2024-05-10T11:20:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.135+0000 7f62c3e00700 1 -- 172.21.15.27:0/3725062125 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c40218c0 con 0x55d15aa4eda0 2024-05-10T11:20:30.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.135+0000 7f62e9012b80 1 -- 172.21.15.27:0/3725062125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4eda0 msgr2=0x55d15aa518d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:30.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.135+0000 7f62e9012b80 1 --2- 172.21.15.27:0/3725062125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4eda0 0x55d15aa518d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f62c40a66e0 tx=0x7f62c4021c60 comp rx=0 tx=0).stop 2024-05-10T11:20:30.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.136+0000 7f62e9012b80 1 -- 172.21.15.27:0/3725062125 shutdown_connections 2024-05-10T11:20:30.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.136+0000 7f62e9012b80 1 --2- 172.21.15.27:0/3725062125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4f6c0 0x55d15aa3e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.136+0000 7f62e9012b80 1 --2- 172.21.15.27:0/3725062125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa49a50 0x55d15aa5f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.136+0000 7f62e9012b80 1 --2- 172.21.15.27:0/3725062125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4eda0 0x55d15aa518d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.136+0000 7f62e9012b80 1 -- 172.21.15.27:0/3725062125 >> 172.21.15.27:0/3725062125 conn(0x55d15a9c16d0 msgr2=0x55d15aa4d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.136+0000 7f62e9012b80 1 -- 172.21.15.27:0/3725062125 shutdown_connections 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.136+0000 7f62e9012b80 1 -- 172.21.15.27:0/3725062125 wait complete. 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.136+0000 7f62e9012b80 1 -- start start 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.137+0000 7f62e9012b80 1 Processor -- start 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.137+0000 7f62e9012b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa49a50 0x55d15aa5ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.137+0000 7f62e9012b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4eda0 0x55d15aa56da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.137+0000 7f62e9012b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4f6c0 0x55d15aa572e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.137+0000 7f62e9012b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f62d81ae7d0 con 0x55d15aa49a50 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.137+0000 7f62e9012b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f62d81ae100 con 0x55d15aa4eda0 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.137+0000 7f62e9012b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f62d81ab360 con 0x55d15aa4f6c0 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.137+0000 7f62e9012b80 10 client.0 _ll_register_callbacks cb 0x55d15a942c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62d7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa49a50 0x55d15aa5ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62d7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa49a50 0x55d15aa5ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55624/0 (socket says 172.21.15.27:55624) 2024-05-10T11:20:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62d7400700 1 -- 172.21.15.27:0/2340491472 learned_addr learned my addr 172.21.15.27:0/2340491472 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62d6a00700 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4eda0 0x55d15aa56da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62d7e00700 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4f6c0 0x55d15aa572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62d7400700 1 -- 172.21.15.27:0/2340491472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4f6c0 msgr2=0x55d15aa572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62d7400700 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4f6c0 0x55d15aa572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62d7400700 1 -- 172.21.15.27:0/2340491472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4eda0 msgr2=0x55d15aa56da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62d7400700 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4eda0 0x55d15aa56da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62d7400700 1 -- 172.21.15.27:0/2340491472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d15aa57b20 con 0x55d15aa49a50 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62d7e00700 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4f6c0 0x55d15aa572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62d7400700 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa49a50 0x55d15aa5ba20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f62c80227c0 tx=0x7f62c8005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3abff7bf3771355 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.138+0000 7f62c1600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.139+0000 7f62c1600700 1 -- 172.21.15.27:0/2340491472 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c8027040 con 0x55d15aa49a50 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.139+0000 7f62c1600700 1 -- 172.21.15.27:0/2340491472 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9591+0+0 (secure 0 0 0) 0x7f62c8004960 con 0x55d15aa49a50 2024-05-10T11:20:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.139+0000 7f62e9012b80 1 -- 172.21.15.27:0/2340491472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62c40aea10 con 0x55d15aa49a50 2024-05-10T11:20:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.139+0000 7f62c1600700 1 -- 172.21.15.27:0/2340491472 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f62c8005b40 con 0x55d15aa49a50 2024-05-10T11:20:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.139+0000 7f62e9012b80 10 client.16188 fetch_fsmap learned FSMap version 33 2024-05-10T11:20:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.139+0000 7f62e9012b80 1 -- 172.21.15.27:0/2340491472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d15aacd4c0 con 0x55d15aa49a50 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.140+0000 7f62c1600700 1 -- 172.21.15.27:0/2340491472 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c40aea10 con 0x55d15aa49a50 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.140+0000 7f62c1600700 1 -- 172.21.15.27:0/2340491472 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f62c8003680 con 0x55d15aa49a50 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.140+0000 7f62e9012b80 10 client.16188 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.140+0000 7f62e9012b80 10 client.16188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.140+0000 7f62e9012b80 20 client.16188 populate_metadata read hostname 'smithi027' 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.140+0000 7f62e9012b80 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d15aacdbc0 0x55d15aacffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.140+0000 7f62e9012b80 4 client.16188 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.140+0000 7f62e9012b80 1 -- 172.21.15.27:0/2340491472 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d15a9a9400 con 0x55d15aacdbc0 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.141+0000 7f62d6a00700 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d15aacdbc0 0x55d15aacffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.141+0000 7f62d6a00700 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d15aacdbc0 0x55d15aacffe0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49cc14c9c3a2710a server_cookie=26bf99c83f1476db in_seq=0 out_seq=0 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.141+0000 7f62c1600700 10 client.16188 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.142+0000 7f62c1600700 1 -- 172.21.15.27:0/2340491472 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d15a9a9400 con 0x55d15aacdbc0 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.142+0000 7f62c1600700 10 client.16188 handle_command_reply: tid=0 2024-05-10T11:20:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.142+0000 7f62e9012b80 1 client.16188 shutdown 2024-05-10T11:20:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.142+0000 7f62e9012b80 1 -- 172.21.15.27:0/2340491472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa49a50 msgr2=0x55d15aa5ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.142+0000 7f62e9012b80 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa49a50 0x55d15aa5ba20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f62c80227c0 tx=0x7f62c8005da0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.142+0000 7f62e9012b80 1 -- 172.21.15.27:0/2340491472 shutdown_connections 2024-05-10T11:20:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.142+0000 7f62e9012b80 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d15aacdbc0 0x55d15aacffe0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.142+0000 7f62e9012b80 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d15aa4f6c0 0x55d15aa572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.142+0000 7f62e9012b80 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d15aa49a50 0x55d15aa5ba20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.142+0000 7f62e9012b80 1 --2- 172.21.15.27:0/2340491472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d15aa4eda0 0x55d15aa56da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.142+0000 7f62e9012b80 1 -- 172.21.15.27:0/2340491472 >> 172.21.15.27:0/2340491472 conn(0x55d15a9c16d0 msgr2=0x55d15aa53cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.143+0000 7f62c1600700 0 client.16188 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:20:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.143+0000 7f62e9012b80 1 -- 172.21.15.27:0/2340491472 shutdown_connections 2024-05-10T11:20:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.143+0000 7f62e9012b80 1 -- 172.21.15.27:0/2340491472 wait complete. 2024-05-10T11:20:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.143+0000 7f62e9012b80 20 client.16188 trim_cache size 0 max 16384 2024-05-10T11:20:30.146 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:20:30.146 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:20:30.146 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:20:30.146 INFO:teuthology.orchestra.run.smithi027.stdout: "63d2eeed-c6ee-4c72-8d29-ce18158f899c": { 2024-05-10T11:20:30.146 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:20:30.146 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "63d2eeed-c6ee-4c72-8d29-ce18158f899c", 2024-05-10T11:20:30.146 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:20:30.146 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:20:30.146 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:20:30.146 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:20:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.144+0000 7f62dd000700 1 -- 172.21.15.27:0/2569034984 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f62bc0b29f0 msgr2=0x7f62bc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.144+0000 7f62dd000700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f62bc0b29f0 0x7f62bc0b4eb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55d15aa4a410 tx=0x7f62d0009920 comp rx=0 tx=0).stop 2024-05-10T11:20:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.144+0000 7f62dd000700 1 -- 172.21.15.27:0/2569034984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62d8104eb0 msgr2=0x7f62d819c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:20:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.144+0000 7f62dd000700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62d8104eb0 0x7f62d819c250 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f62c4000c00 tx=0x7f62c40047b0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.144+0000 7f62dd000700 1 -- 172.21.15.27:0/2569034984 shutdown_connections 2024-05-10T11:20:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.144+0000 7f62dd000700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f62bc0b29f0 0x7f62bc0b4eb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.144+0000 7f62dd000700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d810d9c0 0x7f62d81a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.144+0000 7f62dd000700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62d8104590 0x7f62d819bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.144+0000 7f62dd000700 1 --2- 172.21.15.27:0/2569034984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62d8104eb0 0x7f62d819c250 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:20:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.144+0000 7f62dd000700 1 -- 172.21.15.27:0/2569034984 >> 172.21.15.27:0/2569034984 conn(0x7f62d80fddc0 msgr2=0x7f62d81001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:20:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.144+0000 7f62dd000700 1 -- 172.21.15.27:0/2569034984 shutdown_connections 2024-05-10T11:20:30.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:20:30.144+0000 7f62dd000700 1 -- 172.21.15.27:0/2569034984 wait complete. 2024-05-10T11:20:30.695 INFO:tasks.workunit.client.0.smithi027.stdout: 1 415433 6.86 MB/sec execute 195 sec latency 16.539 ms 2024-05-10T11:20:30.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:30 smithi027 ceph-mon[30054]: pgmap v223: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-05-10T11:20:30.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:30 smithi189 ceph-mon[34842]: pgmap v223: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-05-10T11:20:31.082 INFO:tasks.cephfs.filesystem:scrub status for tag:63d2eeed-c6ee-4c72-8d29-ce18158f899c - {'path': '/', 'tag': '63d2eeed-c6ee-4c72-8d29-ce18158f899c', 'options': 'recursive,force'} 2024-05-10T11:20:31.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:30 smithi076 ceph-mon[34152]: pgmap v223: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-05-10T11:20:31.695 INFO:tasks.workunit.client.0.smithi027.stdout: 1 416496 6.85 MB/sec execute 196 sec latency 43.337 ms 2024-05-10T11:20:31.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:31 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:20:32.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:31 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:20:32.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:31 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:20:32.696 INFO:tasks.workunit.client.0.smithi027.stdout: 1 418301 6.87 MB/sec execute 197 sec latency 25.499 ms 2024-05-10T11:20:32.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:32 smithi189 ceph-mon[34842]: pgmap v224: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 134 op/s 2024-05-10T11:20:32.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:32 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:20:32.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:32 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:20:33.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:32 smithi027 ceph-mon[30054]: pgmap v224: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 134 op/s 2024-05-10T11:20:33.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:32 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:20:33.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:32 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:20:33.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:32 smithi076 ceph-mon[34152]: pgmap v224: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 134 op/s 2024-05-10T11:20:33.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:32 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:20:33.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:32 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:20:33.696 INFO:tasks.workunit.client.0.smithi027.stdout: 1 419055 6.85 MB/sec execute 198 sec latency 18.566 ms 2024-05-10T11:20:34.696 INFO:tasks.workunit.client.0.smithi027.stdout: 1 420269 6.85 MB/sec execute 199 sec latency 33.815 ms 2024-05-10T11:20:34.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:34 smithi189 ceph-mon[34842]: pgmap v225: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 134 op/s 2024-05-10T11:20:35.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:34 smithi027 ceph-mon[30054]: pgmap v225: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 134 op/s 2024-05-10T11:20:35.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:34 smithi076 ceph-mon[34152]: pgmap v225: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 134 op/s 2024-05-10T11:20:35.696 INFO:tasks.workunit.client.0.smithi027.stdout: 1 422200 6.87 MB/sec execute 200 sec latency 21.537 ms 2024-05-10T11:20:36.696 INFO:tasks.workunit.client.0.smithi027.stdout: 1 422820 6.85 MB/sec execute 201 sec latency 20.803 ms 2024-05-10T11:20:36.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:36 smithi189 ceph-mon[34842]: pgmap v226: 129 pgs: 129 active+clean; 262 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 205 op/s 2024-05-10T11:20:37.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:36 smithi027 ceph-mon[30054]: pgmap v226: 129 pgs: 129 active+clean; 262 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 205 op/s 2024-05-10T11:20:37.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:36 smithi076 ceph-mon[34152]: pgmap v226: 129 pgs: 129 active+clean; 262 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 205 op/s 2024-05-10T11:20:37.698 INFO:tasks.workunit.client.0.smithi027.stdout: 1 424112 6.84 MB/sec execute 202 sec latency 22.070 ms 2024-05-10T11:20:38.697 INFO:tasks.workunit.client.0.smithi027.stdout: 1 425814 6.87 MB/sec execute 203 sec latency 46.363 ms 2024-05-10T11:20:38.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:38 smithi189 ceph-mon[34842]: pgmap v227: 129 pgs: 129 active+clean; 262 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 140 op/s 2024-05-10T11:20:39.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:38 smithi027 ceph-mon[30054]: pgmap v227: 129 pgs: 129 active+clean; 262 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 140 op/s 2024-05-10T11:20:39.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:38 smithi076 ceph-mon[34152]: pgmap v227: 129 pgs: 129 active+clean; 262 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 140 op/s 2024-05-10T11:20:39.697 INFO:tasks.workunit.client.0.smithi027.stdout: 1 426476 6.84 MB/sec execute 204 sec latency 24.933 ms 2024-05-10T11:20:40.697 INFO:tasks.workunit.client.0.smithi027.stdout: 1 427814 6.84 MB/sec execute 205 sec latency 62.965 ms 2024-05-10T11:20:40.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:40 smithi189 ceph-mon[34842]: pgmap v228: 129 pgs: 129 active+clean; 266 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 209 op/s 2024-05-10T11:20:41.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:40 smithi027 ceph-mon[30054]: pgmap v228: 129 pgs: 129 active+clean; 266 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 209 op/s 2024-05-10T11:20:41.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:40 smithi076 ceph-mon[34152]: pgmap v228: 129 pgs: 129 active+clean; 266 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 209 op/s 2024-05-10T11:20:41.697 INFO:tasks.workunit.client.0.smithi027.stdout: 1 429431 6.87 MB/sec execute 206 sec latency 21.838 ms 2024-05-10T11:20:42.697 INFO:tasks.workunit.client.0.smithi027.stdout: 1 430012 6.84 MB/sec execute 207 sec latency 36.231 ms 2024-05-10T11:20:42.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:42 smithi189 ceph-mon[34842]: pgmap v229: 129 pgs: 129 active+clean; 266 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 139 op/s 2024-05-10T11:20:42.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:42 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:20:43.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:42 smithi027 ceph-mon[30054]: pgmap v229: 129 pgs: 129 active+clean; 266 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 139 op/s 2024-05-10T11:20:43.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:42 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:20:43.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:42 smithi076 ceph-mon[34152]: pgmap v229: 129 pgs: 129 active+clean; 266 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 139 op/s 2024-05-10T11:20:43.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:42 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:20:43.697 INFO:tasks.workunit.client.0.smithi027.stdout: 1 431422 6.83 MB/sec execute 208 sec latency 83.377 ms 2024-05-10T11:20:44.697 INFO:tasks.workunit.client.0.smithi027.stdout: 1 433044 6.86 MB/sec execute 209 sec latency 19.451 ms 2024-05-10T11:20:44.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:44 smithi189 ceph-mon[34842]: pgmap v230: 129 pgs: 129 active+clean; 266 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 139 op/s 2024-05-10T11:20:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:44 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:20:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:44 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:20:45.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:44 smithi027 ceph-mon[30054]: pgmap v230: 129 pgs: 129 active+clean; 266 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 139 op/s 2024-05-10T11:20:45.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:44 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:20:45.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:44 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:20:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:44 smithi076 ceph-mon[34152]: pgmap v230: 129 pgs: 129 active+clean; 266 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 139 op/s 2024-05-10T11:20:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:44 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:20:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:44 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:20:45.697 INFO:tasks.workunit.client.0.smithi027.stdout: 1 433976 6.84 MB/sec execute 210 sec latency 34.537 ms 2024-05-10T11:20:45.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:45 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:20:45.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:45 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:20:45.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:45 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:20:46.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:45 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:20:46.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:45 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:20:46.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:45 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:20:46.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:45 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:20:46.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:45 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:20:46.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:45 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:20:46.697 INFO:tasks.workunit.client.0.smithi027.stdout: 1 435641 6.85 MB/sec execute 211 sec latency 23.124 ms 2024-05-10T11:20:47.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:47 smithi076 ceph-mon[34152]: pgmap v231: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 207 op/s 2024-05-10T11:20:47.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:47 smithi189 ceph-mon[34842]: pgmap v231: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 207 op/s 2024-05-10T11:20:47.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:47 smithi027 ceph-mon[30054]: pgmap v231: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 207 op/s 2024-05-10T11:20:47.698 INFO:tasks.workunit.client.0.smithi027.stdout: 1 436736 6.85 MB/sec execute 212 sec latency 15.055 ms 2024-05-10T11:20:48.698 INFO:tasks.workunit.client.0.smithi027.stdout: 1 437801 6.84 MB/sec execute 213 sec latency 87.628 ms 2024-05-10T11:20:49.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:49 smithi076 ceph-mon[34152]: pgmap v232: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 136 op/s 2024-05-10T11:20:49.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:49 smithi189 ceph-mon[34842]: pgmap v232: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 136 op/s 2024-05-10T11:20:49.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:49 smithi027 ceph-mon[30054]: pgmap v232: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 136 op/s 2024-05-10T11:20:49.698 INFO:tasks.workunit.client.0.smithi027.stdout: 1 439645 6.86 MB/sec execute 214 sec latency 36.719 ms 2024-05-10T11:20:50.698 INFO:tasks.workunit.client.0.smithi027.stdout: 1 440458 6.85 MB/sec execute 215 sec latency 17.623 ms 2024-05-10T11:20:51.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:51 smithi076 ceph-mon[34152]: pgmap v233: 129 pgs: 129 active+clean; 276 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-05-10T11:20:51.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:51 smithi189 ceph-mon[34842]: pgmap v233: 129 pgs: 129 active+clean; 276 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-05-10T11:20:51.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:51 smithi027 ceph-mon[30054]: pgmap v233: 129 pgs: 129 active+clean; 276 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-05-10T11:20:51.698 INFO:tasks.workunit.client.0.smithi027.stdout: 1 441700 6.84 MB/sec execute 216 sec latency 31.803 ms 2024-05-10T11:20:52.698 INFO:tasks.workunit.client.0.smithi027.stdout: 1 443599 6.87 MB/sec execute 217 sec latency 18.405 ms 2024-05-10T11:20:53.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:53 smithi076 ceph-mon[34152]: pgmap v234: 129 pgs: 129 active+clean; 276 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 139 op/s 2024-05-10T11:20:53.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:53 smithi189 ceph-mon[34842]: pgmap v234: 129 pgs: 129 active+clean; 276 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 139 op/s 2024-05-10T11:20:53.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:53 smithi027 ceph-mon[30054]: pgmap v234: 129 pgs: 129 active+clean; 276 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 139 op/s 2024-05-10T11:20:53.699 INFO:tasks.workunit.client.0.smithi027.stdout: 1 444278 6.84 MB/sec execute 218 sec latency 30.172 ms 2024-05-10T11:20:54.699 INFO:tasks.workunit.client.0.smithi027.stdout: 1 445605 6.84 MB/sec execute 219 sec latency 72.387 ms 2024-05-10T11:20:55.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:55 smithi076 ceph-mon[34152]: pgmap v235: 129 pgs: 129 active+clean; 276 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 139 op/s 2024-05-10T11:20:55.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:55 smithi189 ceph-mon[34842]: pgmap v235: 129 pgs: 129 active+clean; 276 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 139 op/s 2024-05-10T11:20:55.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:55 smithi027 ceph-mon[30054]: pgmap v235: 129 pgs: 129 active+clean; 276 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 139 op/s 2024-05-10T11:20:55.699 INFO:tasks.workunit.client.0.smithi027.stdout: 1 447066 6.86 MB/sec execute 220 sec latency 22.577 ms 2024-05-10T11:20:56.699 INFO:tasks.workunit.client.0.smithi027.stdout: 1 447611 6.83 MB/sec execute 221 sec latency 46.906 ms 2024-05-10T11:20:57.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:57 smithi076 ceph-mon[34152]: pgmap v236: 129 pgs: 129 active+clean; 282 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 212 op/s 2024-05-10T11:20:57.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:57 smithi189 ceph-mon[34842]: pgmap v236: 129 pgs: 129 active+clean; 282 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 212 op/s 2024-05-10T11:20:57.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:57 smithi027 ceph-mon[30054]: pgmap v236: 129 pgs: 129 active+clean; 282 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 212 op/s 2024-05-10T11:20:57.699 INFO:tasks.workunit.client.0.smithi027.stdout: 1 448605 6.83 MB/sec execute 222 sec latency 28.951 ms 2024-05-10T11:20:58.699 INFO:tasks.workunit.client.0.smithi027.stdout: 1 450207 6.84 MB/sec execute 223 sec latency 24.745 ms 2024-05-10T11:20:59.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:20:59 smithi189 ceph-mon[34842]: pgmap v237: 129 pgs: 129 active+clean; 282 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-05-10T11:20:59.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:20:59 smithi027 ceph-mon[30054]: pgmap v237: 129 pgs: 129 active+clean; 282 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-05-10T11:20:59.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:20:59 smithi076 ceph-mon[34152]: pgmap v237: 129 pgs: 129 active+clean; 282 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-05-10T11:20:59.699 INFO:tasks.workunit.client.0.smithi027.stdout: 1 450879 6.83 MB/sec execute 224 sec latency 89.346 ms 2024-05-10T11:21:00.700 INFO:tasks.workunit.client.0.smithi027.stdout: 1 451930 6.82 MB/sec execute 225 sec latency 32.387 ms 2024-05-10T11:21:01.084 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:21:01.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:01 smithi189 ceph-mon[34842]: pgmap v238: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 202 op/s 2024-05-10T11:21:01.587 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:21:01.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:01 smithi027 ceph-mon[30054]: pgmap v238: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 202 op/s 2024-05-10T11:21:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:01 smithi076 ceph-mon[34152]: pgmap v238: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 202 op/s 2024-05-10T11:21:01.700 INFO:tasks.workunit.client.0.smithi027.stdout: 1 452769 6.79 MB/sec execute 226 sec latency 62.000 ms 2024-05-10T11:21:02.700 INFO:tasks.workunit.client.0.smithi027.stdout: 1 453504 6.79 MB/sec execute 227 sec latency 85.911 ms 2024-05-10T11:21:03.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:03 smithi189 ceph-mon[34842]: pgmap v239: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 131 op/s 2024-05-10T11:21:03.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:03 smithi189 ceph-mon[34842]: osdmap e94: 12 total, 12 up, 12 in 2024-05-10T11:21:03.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:03 smithi027 ceph-mon[30054]: pgmap v239: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 131 op/s 2024-05-10T11:21:03.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:03 smithi027 ceph-mon[30054]: osdmap e94: 12 total, 12 up, 12 in 2024-05-10T11:21:03.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:03 smithi076 ceph-mon[34152]: pgmap v239: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 131 op/s 2024-05-10T11:21:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:03 smithi076 ceph-mon[34152]: osdmap e94: 12 total, 12 up, 12 in 2024-05-10T11:21:03.700 INFO:tasks.workunit.client.0.smithi027.stdout: 1 454354 6.79 MB/sec execute 228 sec latency 303.991 ms 2024-05-10T11:21:04.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.254+0000 7fee6a600700 1 -- 172.21.15.27:0/2096446150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64075ff0 msgr2=0x7fee6410f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:04.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.254+0000 7fee6a600700 1 --2- 172.21.15.27:0/2096446150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64075ff0 0x7fee6410f9c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fee4c009990 tx=0x7fee4c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.254+0000 7fee6a600700 1 -- 172.21.15.27:0/2096446150 shutdown_connections 2024-05-10T11:21:04.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.254+0000 7fee6a600700 1 --2- 172.21.15.27:0/2096446150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee6410ff00 0x7fee641122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.254+0000 7fee6a600700 1 --2- 172.21.15.27:0/2096446150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee640756d0 0x7fee64075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.254+0000 7fee6a600700 1 --2- 172.21.15.27:0/2096446150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64075ff0 0x7fee6410f9c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.254+0000 7fee6a600700 1 -- 172.21.15.27:0/2096446150 >> 172.21.15.27:0/2096446150 conn(0x7fee6407a600 msgr2=0x7fee6407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:04.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.255+0000 7fee6a600700 1 -- 172.21.15.27:0/2096446150 shutdown_connections 2024-05-10T11:21:04.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.255+0000 7fee6a600700 1 -- 172.21.15.27:0/2096446150 wait complete. 2024-05-10T11:21:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.256+0000 7fee6a600700 1 Processor -- start 2024-05-10T11:21:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.256+0000 7fee6a600700 1 -- start start 2024-05-10T11:21:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.256+0000 7fee6a600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee640756d0 0x7fee6419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.256+0000 7fee6a600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee64075ff0 0x7fee6419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.256+0000 7fee6a600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee6410ff00 0x7fee641a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.256+0000 7fee6a600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fee641158f0 con 0x7fee6410ff00 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.256+0000 7fee6a600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fee64115770 con 0x7fee640756d0 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.256+0000 7fee6a600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fee64115a70 con 0x7fee64075ff0 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.257+0000 7fee63400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee640756d0 0x7fee6419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.257+0000 7fee63e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee6410ff00 0x7fee641a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.257+0000 7fee5aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee64075ff0 0x7fee6419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.257+0000 7fee63400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee640756d0 0x7fee6419bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35484/0 (socket says 172.21.15.27:35484) 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.257+0000 7fee63400700 1 -- 172.21.15.27:0/791285431 learned_addr learned my addr 172.21.15.27:0/791285431 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.257+0000 7fee63400700 1 -- 172.21.15.27:0/791285431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee64075ff0 msgr2=0x7fee6419c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.257+0000 7fee63400700 1 --2- 172.21.15.27:0/791285431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee64075ff0 0x7fee6419c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.257+0000 7fee63400700 1 -- 172.21.15.27:0/791285431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee6410ff00 msgr2=0x7fee641a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.257+0000 7fee63400700 1 --2- 172.21.15.27:0/791285431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee6410ff00 0x7fee641a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.257+0000 7fee63400700 1 -- 172.21.15.27:0/791285431 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee641a0d70 con 0x7fee640756d0 2024-05-10T11:21:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.257+0000 7fee63400700 1 --2- 172.21.15.27:0/791285431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee640756d0 0x7fee6419bd60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fee5400c370 tx=0x7fee5400c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.258+0000 7fee60c00700 1 -- 172.21.15.27:0/791285431 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee540092e0 con 0x7fee640756d0 2024-05-10T11:21:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.258+0000 7fee6a600700 1 -- 172.21.15.27:0/791285431 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee641a1000 con 0x7fee640756d0 2024-05-10T11:21:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.258+0000 7fee60c00700 1 -- 172.21.15.27:0/791285431 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee5400cbe0 con 0x7fee640756d0 2024-05-10T11:21:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.258+0000 7fee60c00700 1 -- 172.21.15.27:0/791285431 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee540046b0 con 0x7fee640756d0 2024-05-10T11:21:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.258+0000 7fee6a600700 1 -- 172.21.15.27:0/791285431 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee641a88e0 con 0x7fee640756d0 2024-05-10T11:21:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.261+0000 7fee60c00700 1 -- 172.21.15.27:0/791285431 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fee5401a020 con 0x7fee640756d0 2024-05-10T11:21:04.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.263+0000 7fee60c00700 1 --2- 172.21.15.27:0/791285431 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fee480b29f0 0x7fee480b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.263+0000 7fee75046b80 1 Processor -- start 2024-05-10T11:21:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.263+0000 7fee75046b80 1 -- start start 2024-05-10T11:21:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.263+0000 7fee60c00700 1 -- 172.21.15.27:0/791285431 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9679+0+0 (secure 0 0 0) 0x7fee540e2460 con 0x7fee640756d0 2024-05-10T11:21:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.263+0000 7fee5aa00700 1 --2- 172.21.15.27:0/791285431 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fee480b29f0 0x7fee480b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.263+0000 7fee75046b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3ebc0 0x562409b3efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.263+0000 7fee75046b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3f4e0 0x562409b392d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.263+0000 7fee75046b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b398f0 0x562409b39d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.263+0000 7fee75046b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562409a8ba20 con 0x562409b3f4e0 2024-05-10T11:21:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.263+0000 7fee75046b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562409a29760 con 0x562409b398f0 2024-05-10T11:21:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.263+0000 7fee75046b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562409a879e0 con 0x562409b3ebc0 2024-05-10T11:21:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.264+0000 7fee63400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3ebc0 0x562409b3efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.264+0000 7fee63400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3ebc0 0x562409b3efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:34412/0 (socket says 172.21.15.27:34412) 2024-05-10T11:21:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.264+0000 7fee63400700 1 -- 172.21.15.27:0/3974623127 learned_addr learned my addr 172.21.15.27:0/3974623127 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.264+0000 7fee63e00700 1 --2- 172.21.15.27:0/3974623127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b398f0 0x562409b39d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.264+0000 7fee5aa00700 1 --2- 172.21.15.27:0/3974623127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3f4e0 0x562409b392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.264+0000 7fee63400700 1 -- 172.21.15.27:0/3974623127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b398f0 msgr2=0x562409b39d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:04.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.264+0000 7fee63400700 1 --2- 172.21.15.27:0/3974623127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b398f0 0x562409b39d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.264+0000 7fee63400700 1 -- 172.21.15.27:0/3974623127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3f4e0 msgr2=0x562409b392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:04.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.264+0000 7fee63400700 1 --2- 172.21.15.27:0/3974623127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3f4e0 0x562409b392d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.264+0000 7fee63400700 1 -- 172.21.15.27:0/3974623127 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624099d88d0 con 0x562409b3ebc0 2024-05-10T11:21:04.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.264+0000 7fee5aa00700 1 --2- 172.21.15.27:0/3974623127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3f4e0 0x562409b392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:21:04.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.265+0000 7fee5aa00700 1 --2- 172.21.15.27:0/791285431 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fee480b29f0 0x7fee480b4eb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fee64111ae0 tx=0x7fee4c009630 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:04.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.265+0000 7fee63400700 1 --2- 172.21.15.27:0/3974623127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3ebc0 0x562409b3efa0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fee540a52e0 tx=0x7fee540a89d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94cbd2d3cfcfd174 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.265+0000 7fee58c00700 1 -- 172.21.15.27:0/3974623127 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee540a9260 con 0x562409b3ebc0 2024-05-10T11:21:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.265+0000 7fee58c00700 1 -- 172.21.15.27:0/3974623127 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee540ad610 con 0x562409b3ebc0 2024-05-10T11:21:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.265+0000 7fee58c00700 1 -- 172.21.15.27:0/3974623127 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54004bd0 con 0x562409b3ebc0 2024-05-10T11:21:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.266+0000 7fee75046b80 1 -- 172.21.15.27:0/3974623127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3ebc0 msgr2=0x562409b3efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.266+0000 7fee75046b80 1 --2- 172.21.15.27:0/3974623127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3ebc0 0x562409b3efa0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fee540a52e0 tx=0x7fee540a89d0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.266+0000 7fee75046b80 1 -- 172.21.15.27:0/3974623127 shutdown_connections 2024-05-10T11:21:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.266+0000 7fee75046b80 1 --2- 172.21.15.27:0/3974623127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3ebc0 0x562409b3efa0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.266+0000 7fee75046b80 1 --2- 172.21.15.27:0/3974623127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b398f0 0x562409b39d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.266+0000 7fee75046b80 1 --2- 172.21.15.27:0/3974623127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3f4e0 0x562409b392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.266+0000 7fee75046b80 1 -- 172.21.15.27:0/3974623127 >> 172.21.15.27:0/3974623127 conn(0x562409ab16d0 msgr2=0x562409b33b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.266+0000 7fee75046b80 1 -- 172.21.15.27:0/3974623127 shutdown_connections 2024-05-10T11:21:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.266+0000 7fee75046b80 1 -- 172.21.15.27:0/3974623127 wait complete. 2024-05-10T11:21:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.267+0000 7fee75046b80 1 -- start start 2024-05-10T11:21:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.267+0000 7fee75046b80 1 Processor -- start 2024-05-10T11:21:04.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.267+0000 7fee75046b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b398f0 0x562409b45260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.267+0000 7fee75046b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3ebc0 0x562409b42600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.267+0000 7fee75046b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3f4e0 0x562409b42b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.267+0000 7fee75046b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562409b45f20 con 0x562409b3ebc0 2024-05-10T11:21:04.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.267+0000 7fee75046b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5624099f1b20 con 0x562409b398f0 2024-05-10T11:21:04.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.267+0000 7fee75046b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562409b460a0 con 0x562409b3f4e0 2024-05-10T11:21:04.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee75046b80 10 client.0 _ll_register_callbacks cb 0x562409a32c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:04.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee63400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b398f0 0x562409b45260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee63400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b398f0 0x562409b45260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35496/0 (socket says 172.21.15.27:35496) 2024-05-10T11:21:04.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee63400700 1 -- 172.21.15.27:0/4284741992 learned_addr learned my addr 172.21.15.27:0/4284741992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:04.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee63e00700 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3f4e0 0x562409b42b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee5aa00700 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3ebc0 0x562409b42600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee63e00700 1 -- 172.21.15.27:0/4284741992 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b398f0 msgr2=0x562409b45260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee63e00700 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b398f0 0x562409b45260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee63e00700 1 -- 172.21.15.27:0/4284741992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3ebc0 msgr2=0x562409b42600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee63e00700 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3ebc0 0x562409b42600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee63e00700 1 -- 172.21.15.27:0/4284741992 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562409a51420 con 0x562409b3f4e0 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee63400700 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b398f0 0x562409b45260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.268+0000 7fee5aa00700 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3ebc0 0x562409b42600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.269+0000 7fee63e00700 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3f4e0 0x562409b42b40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fee6406bcd0 tx=0x7fee5c011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78af9a84bed67d7c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.269+0000 7fee51600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.269+0000 7fee51600700 1 -- 172.21.15.27:0/4284741992 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c004300 con 0x562409b3f4e0 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.269+0000 7fee75046b80 1 -- 172.21.15.27:0/4284741992 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee540ad610 con 0x562409b3f4e0 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.269+0000 7fee51600700 1 -- 172.21.15.27:0/4284741992 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9679+0+0 (secure 0 0 0) 0x7fee5c0044a0 con 0x562409b3f4e0 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.269+0000 7fee51600700 1 -- 172.21.15.27:0/4284741992 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fee5c024050 con 0x562409b3f4e0 2024-05-10T11:21:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.269+0000 7fee75046b80 10 client.26395 fetch_fsmap learned FSMap version 33 2024-05-10T11:21:04.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.269+0000 7fee75046b80 1 -- 172.21.15.27:0/4284741992 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562409b50840 con 0x562409b3f4e0 2024-05-10T11:21:04.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.270+0000 7fee51600700 1 -- 172.21.15.27:0/4284741992 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee540ad610 con 0x562409b3f4e0 2024-05-10T11:21:04.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.270+0000 7fee51600700 1 -- 172.21.15.27:0/4284741992 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fee5c01b810 con 0x562409b3f4e0 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.270+0000 7fee75046b80 10 client.26395 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.270+0000 7fee75046b80 10 client.26395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.270+0000 7fee75046b80 20 client.26395 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.270+0000 7fee75046b80 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562409b52dd0 0x562409b551c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.270+0000 7fee75046b80 4 client.26395 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.270+0000 7fee75046b80 1 -- 172.21.15.27:0/4284741992 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562409b06140 con 0x562409b52dd0 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.271+0000 7fee63400700 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562409b52dd0 0x562409b551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.271+0000 7fee63400700 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562409b52dd0 0x562409b551c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f469c955489ec00f server_cookie=3fb2b4f75c43baae in_seq=0 out_seq=0 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.271+0000 7fee51600700 10 client.26395 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.272+0000 7fee51600700 1 -- 172.21.15.27:0/4284741992 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562409b06140 con 0x562409b52dd0 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.272+0000 7fee51600700 10 client.26395 handle_command_reply: tid=0 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.272+0000 7fee75046b80 1 client.26395 shutdown 2024-05-10T11:21:04.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.273+0000 7fee75046b80 1 -- 172.21.15.27:0/4284741992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3f4e0 msgr2=0x562409b42b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:04.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.273+0000 7fee75046b80 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3f4e0 0x562409b42b40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fee6406bcd0 tx=0x7fee5c011720 comp rx=0 tx=0).stop 2024-05-10T11:21:04.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.273+0000 7fee75046b80 1 -- 172.21.15.27:0/4284741992 shutdown_connections 2024-05-10T11:21:04.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.273+0000 7fee75046b80 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562409b52dd0 0x562409b551c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.273+0000 7fee75046b80 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b3f4e0 0x562409b42b40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.273+0000 7fee75046b80 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b398f0 0x562409b45260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.273+0000 7fee75046b80 1 --2- 172.21.15.27:0/4284741992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3ebc0 0x562409b42600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.273+0000 7fee51600700 0 client.26395 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:04.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.273+0000 7fee75046b80 1 -- 172.21.15.27:0/4284741992 >> 172.21.15.27:0/4284741992 conn(0x562409ab16d0 msgr2=0x562409b33190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:04.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.273+0000 7fee75046b80 1 -- 172.21.15.27:0/4284741992 shutdown_connections 2024-05-10T11:21:04.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.273+0000 7fee75046b80 1 -- 172.21.15.27:0/4284741992 wait complete. 2024-05-10T11:21:04.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.273+0000 7fee75046b80 20 client.26395 trim_cache size 0 max 16384 2024-05-10T11:21:04.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.296+0000 7fee75046b80 1 Processor -- start 2024-05-10T11:21:04.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.296+0000 7fee75046b80 1 -- start start 2024-05-10T11:21:04.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee75046b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 0x562409b392d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee75046b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b3ebc0 0x562409b3cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee75046b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3f4e0 0x562409b4efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee75046b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fee641b1a90 con 0x562409b3f4e0 2024-05-10T11:21:04.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee75046b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fee641b35d0 con 0x562409b3ebc0 2024-05-10T11:21:04.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee75046b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fee641b2160 con 0x562409b398f0 2024-05-10T11:21:04.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee5aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b3ebc0 0x562409b3cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee5aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b3ebc0 0x562409b3cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35500/0 (socket says 172.21.15.27:35500) 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee5aa00700 1 -- 172.21.15.27:0/2262736197 learned_addr learned my addr 172.21.15.27:0/2262736197 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee63e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3f4e0 0x562409b4efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee63400700 1 --2- 172.21.15.27:0/2262736197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 0x562409b392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee5aa00700 1 -- 172.21.15.27:0/2262736197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 msgr2=0x562409b392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee5aa00700 1 --2- 172.21.15.27:0/2262736197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 0x562409b392d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee5aa00700 1 -- 172.21.15.27:0/2262736197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3f4e0 msgr2=0x562409b4efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee5aa00700 1 --2- 172.21.15.27:0/2262736197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3f4e0 0x562409b4efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee5aa00700 1 -- 172.21.15.27:0/2262736197 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562409b55820 con 0x562409b3ebc0 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.297+0000 7fee63400700 1 --2- 172.21.15.27:0/2262736197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 0x562409b392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.298+0000 7fee5aa00700 1 --2- 172.21.15.27:0/2262736197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b3ebc0 0x562409b3cd80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fee4c009990 tx=0x7fee4c00b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef48db5a73527f3e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.298+0000 7fee53e00700 1 -- 172.21.15.27:0/2262736197 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c00be60 con 0x562409b3ebc0 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.298+0000 7fee53e00700 1 -- 172.21.15.27:0/2262736197 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee4c003b30 con 0x562409b3ebc0 2024-05-10T11:21:04.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.298+0000 7fee53e00700 1 -- 172.21.15.27:0/2262736197 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c00f8b0 con 0x562409b3ebc0 2024-05-10T11:21:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.299+0000 7fee75046b80 1 -- 172.21.15.27:0/2262736197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b3ebc0 msgr2=0x562409b3cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.299+0000 7fee75046b80 1 --2- 172.21.15.27:0/2262736197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b3ebc0 0x562409b3cd80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fee4c009990 tx=0x7fee4c00b260 comp rx=0 tx=0).stop 2024-05-10T11:21:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.299+0000 7fee75046b80 1 -- 172.21.15.27:0/2262736197 shutdown_connections 2024-05-10T11:21:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.299+0000 7fee75046b80 1 --2- 172.21.15.27:0/2262736197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 0x562409b392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.299+0000 7fee75046b80 1 --2- 172.21.15.27:0/2262736197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3f4e0 0x562409b4efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.299+0000 7fee75046b80 1 --2- 172.21.15.27:0/2262736197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b3ebc0 0x562409b3cd80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.299+0000 7fee75046b80 1 -- 172.21.15.27:0/2262736197 >> 172.21.15.27:0/2262736197 conn(0x562409ab16d0 msgr2=0x562409b3f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.299+0000 7fee75046b80 1 -- 172.21.15.27:0/2262736197 shutdown_connections 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.299+0000 7fee75046b80 1 -- 172.21.15.27:0/2262736197 wait complete. 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.300+0000 7fee75046b80 1 -- start start 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.300+0000 7fee75046b80 1 Processor -- start 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.300+0000 7fee75046b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 0x562409b4b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.300+0000 7fee75046b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3ebc0 0x562409b5a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.300+0000 7fee75046b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b3f4e0 0x562409b5a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.300+0000 7fee75046b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fee641b2f00 con 0x562409b3ebc0 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.300+0000 7fee75046b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fee641b2830 con 0x562409b3f4e0 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.300+0000 7fee75046b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fee641afa50 con 0x562409b398f0 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.301+0000 7fee75046b80 10 client.0 _ll_register_callbacks cb 0x562409a32c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.301+0000 7fee63400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 0x562409b4b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.301+0000 7fee63400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 0x562409b4b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:34434/0 (socket says 172.21.15.27:34434) 2024-05-10T11:21:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.301+0000 7fee63400700 1 -- 172.21.15.27:0/4101120854 learned_addr learned my addr 172.21.15.27:0/4101120854 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:04.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.301+0000 7fee63e00700 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b3f4e0 0x562409b5a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.301+0000 7fee5aa00700 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3ebc0 0x562409b5a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.301+0000 7fee63400700 1 -- 172.21.15.27:0/4101120854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b3f4e0 msgr2=0x562409b5a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.301+0000 7fee63400700 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b3f4e0 0x562409b5a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.301+0000 7fee63400700 1 -- 172.21.15.27:0/4101120854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3ebc0 msgr2=0x562409b5a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.301+0000 7fee63400700 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3ebc0 0x562409b5a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.301+0000 7fee63400700 1 -- 172.21.15.27:0/4101120854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562409b55530 con 0x562409b398f0 2024-05-10T11:21:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.302+0000 7fee63400700 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 0x562409b4b800 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fee540a8b40 tx=0x7fee540a8a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5d66eabcaa10b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.302+0000 7fee51600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:21:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.302+0000 7fee51600700 1 -- 172.21.15.27:0/4101120854 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54009c20 con 0x562409b398f0 2024-05-10T11:21:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.302+0000 7fee51600700 1 -- 172.21.15.27:0/4101120854 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9679+0+0 (secure 0 0 0) 0x7fee540a5850 con 0x562409b398f0 2024-05-10T11:21:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.302+0000 7fee75046b80 1 -- 172.21.15.27:0/4101120854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee4c003b30 con 0x562409b398f0 2024-05-10T11:21:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.302+0000 7fee51600700 1 -- 172.21.15.27:0/4101120854 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fee540a9260 con 0x562409b398f0 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.302+0000 7fee75046b80 10 client.26401 fetch_fsmap learned FSMap version 33 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.302+0000 7fee75046b80 1 -- 172.21.15.27:0/4101120854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562409bbd3a0 con 0x562409b398f0 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.302+0000 7fee51600700 1 -- 172.21.15.27:0/4101120854 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee4c003b30 con 0x562409b398f0 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.303+0000 7fee51600700 1 -- 172.21.15.27:0/4101120854 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fee540ad120 con 0x562409b398f0 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.303+0000 7fee75046b80 10 client.26401 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.303+0000 7fee75046b80 10 client.26401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.303+0000 7fee75046b80 20 client.26401 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.303+0000 7fee75046b80 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562409bbd690 0x562409bbfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.303+0000 7fee75046b80 4 client.26401 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.303+0000 7fee75046b80 1 -- 172.21.15.27:0/4101120854 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562409a99400 con 0x562409bbd690 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.303+0000 7fee5aa00700 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562409bbd690 0x562409bbfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee5aa00700 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562409bbd690 0x562409bbfa80 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=545fe362f514c0c3 server_cookie=dd74be1ac10a388 in_seq=0 out_seq=0 2024-05-10T11:21:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee51600700 10 client.26401 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee51600700 1 -- 172.21.15.27:0/4101120854 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562409a99400 con 0x562409bbd690 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee51600700 10 client.26401 handle_command_reply: tid=0 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee75046b80 1 client.26401 shutdown 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee75046b80 1 -- 172.21.15.27:0/4101120854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 msgr2=0x562409b4b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee75046b80 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 0x562409b4b800 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fee540a8b40 tx=0x7fee540a8a40 comp rx=0 tx=0).stop 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee75046b80 1 -- 172.21.15.27:0/4101120854 shutdown_connections 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee75046b80 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562409bbd690 0x562409bbfa80 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee75046b80 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562409b398f0 0x562409b4b800 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee75046b80 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562409b3f4e0 0x562409b5a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee75046b80 1 --2- 172.21.15.27:0/4101120854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562409b3ebc0 0x562409b5a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee51600700 0 client.26401 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee75046b80 1 -- 172.21.15.27:0/4101120854 >> 172.21.15.27:0/4101120854 conn(0x562409ab16d0 msgr2=0x562409b40150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee75046b80 1 -- 172.21.15.27:0/4101120854 shutdown_connections 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.304+0000 7fee75046b80 1 -- 172.21.15.27:0/4101120854 wait complete. 2024-05-10T11:21:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.305+0000 7fee75046b80 20 client.26401 trim_cache size 0 max 16384 2024-05-10T11:21:04.308 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:21:04.308 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:21:04.308 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:21:04.308 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:21:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.305+0000 7fee6a600700 1 -- 172.21.15.27:0/791285431 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fee480b29f0 msgr2=0x7fee480b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.305+0000 7fee6a600700 1 --2- 172.21.15.27:0/791285431 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fee480b29f0 0x7fee480b4eb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fee64111ae0 tx=0x7fee4c009630 comp rx=0 tx=0).stop 2024-05-10T11:21:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.305+0000 7fee6a600700 1 -- 172.21.15.27:0/791285431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee640756d0 msgr2=0x7fee6419bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.305+0000 7fee6a600700 1 --2- 172.21.15.27:0/791285431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee640756d0 0x7fee6419bd60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fee5400c370 tx=0x7fee5400c680 comp rx=0 tx=0).stop 2024-05-10T11:21:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.305+0000 7fee6a600700 1 -- 172.21.15.27:0/791285431 shutdown_connections 2024-05-10T11:21:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.305+0000 7fee6a600700 1 --2- 172.21.15.27:0/791285431 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fee480b29f0 0x7fee480b4eb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.305+0000 7fee6a600700 1 --2- 172.21.15.27:0/791285431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee6410ff00 0x7fee641a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.305+0000 7fee6a600700 1 --2- 172.21.15.27:0/791285431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee640756d0 0x7fee6419bd60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.305+0000 7fee6a600700 1 --2- 172.21.15.27:0/791285431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee64075ff0 0x7fee6419c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.305+0000 7fee6a600700 1 -- 172.21.15.27:0/791285431 >> 172.21.15.27:0/791285431 conn(0x7fee6407a600 msgr2=0x7fee64100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.306+0000 7fee6a600700 1 -- 172.21.15.27:0/791285431 shutdown_connections 2024-05-10T11:21:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:04.306+0000 7fee6a600700 1 -- 172.21.15.27:0/791285431 wait complete. 2024-05-10T11:21:04.700 INFO:tasks.workunit.client.0.smithi027.stdout: 1 454354 6.76 MB/sec execute 229 sec latency 1304.095 ms 2024-05-10T11:21:05.249 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:21:05.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:21:05.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:05 smithi027 ceph-mon[30054]: pgmap v241: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-05-10T11:21:05.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:05 smithi189 ceph-mon[34842]: pgmap v241: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-05-10T11:21:05.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:05 smithi076 ceph-mon[34152]: pgmap v241: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-05-10T11:21:05.700 INFO:tasks.workunit.client.0.smithi027.stdout: 1 454354 6.73 MB/sec execute 230 sec latency 2304.176 ms 2024-05-10T11:21:05.713 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:21:06.700 INFO:tasks.workunit.client.0.smithi027.stdout: 1 454354 6.70 MB/sec execute 231 sec latency 3304.257 ms 2024-05-10T11:21:07.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:07 smithi027 ceph-mon[30054]: pgmap v242: 129 pgs: 129 active+clean; 294 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 173 op/s 2024-05-10T11:21:07.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:07 smithi189 ceph-mon[34842]: pgmap v242: 129 pgs: 129 active+clean; 294 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 173 op/s 2024-05-10T11:21:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:07 smithi076 ceph-mon[34152]: pgmap v242: 129 pgs: 129 active+clean; 294 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 173 op/s 2024-05-10T11:21:07.700 INFO:tasks.workunit.client.0.smithi027.stdout: 1 454354 6.67 MB/sec execute 232 sec latency 4304.401 ms 2024-05-10T11:21:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.236+0000 7fc76e400700 1 -- 172.21.15.27:0/3193960976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc768075ff0 msgr2=0x7fc76810f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.236+0000 7fc76e400700 1 --2- 172.21.15.27:0/3193960976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc768075ff0 0x7fc76810f960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc750009990 tx=0x7fc750009ca0 comp rx=0 tx=0).stop 2024-05-10T11:21:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.236+0000 7fc76e400700 1 -- 172.21.15.27:0/3193960976 shutdown_connections 2024-05-10T11:21:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.236+0000 7fc76e400700 1 --2- 172.21.15.27:0/3193960976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76810fea0 0x7fc768112290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.236+0000 7fc76e400700 1 --2- 172.21.15.27:0/3193960976 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc7680756d0 0x7fc768075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.236+0000 7fc76e400700 1 --2- 172.21.15.27:0/3193960976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc768075ff0 0x7fc76810f960 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.236+0000 7fc76e400700 1 -- 172.21.15.27:0/3193960976 >> 172.21.15.27:0/3193960976 conn(0x7fc76807a600 msgr2=0x7fc76807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.236+0000 7fc76e400700 1 -- 172.21.15.27:0/3193960976 shutdown_connections 2024-05-10T11:21:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.236+0000 7fc76e400700 1 -- 172.21.15.27:0/3193960976 wait complete. 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.237+0000 7fc76e400700 1 Processor -- start 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.237+0000 7fc76e400700 1 -- start start 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.238+0000 7fc76e400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7680756d0 0x7fc76819bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.238+0000 7fc76e400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc768075ff0 0x7fc76819c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.238+0000 7fc76e400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76810fea0 0x7fc7681a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.238+0000 7fc76e400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc7681158d0 con 0x7fc7680756d0 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.238+0000 7fc76e400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc768115750 con 0x7fc768075ff0 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.238+0000 7fc76e400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc768115a50 con 0x7fc76810fea0 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.238+0000 7fc767400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7680756d0 0x7fc76819bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.238+0000 7fc75ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc768075ff0 0x7fc76819c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.238+0000 7fc767400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7680756d0 0x7fc76819bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43538/0 (socket says 172.21.15.27:43538) 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.238+0000 7fc75ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc768075ff0 0x7fc76819c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35536/0 (socket says 172.21.15.27:35536) 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.238+0000 7fc767400700 1 -- 172.21.15.27:0/1798801334 learned_addr learned my addr 172.21.15.27:0/1798801334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.238+0000 7fc767e00700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76810fea0 0x7fc7681a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.239+0000 7fc75ea00700 1 -- 172.21.15.27:0/1798801334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76810fea0 msgr2=0x7fc7681a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.239+0000 7fc75ea00700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76810fea0 0x7fc7681a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.239+0000 7fc75ea00700 1 -- 172.21.15.27:0/1798801334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7680756d0 msgr2=0x7fc76819bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.239+0000 7fc75ea00700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7680756d0 0x7fc76819bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.239+0000 7fc75ea00700 1 -- 172.21.15.27:0/1798801334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7681a0d60 con 0x7fc768075ff0 2024-05-10T11:21:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.239+0000 7fc75ea00700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc768075ff0 0x7fc76819c2e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc7500098e0 tx=0x7fc7500057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.239+0000 7fc764c00700 1 -- 172.21.15.27:0/1798801334 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc750027070 con 0x7fc768075ff0 2024-05-10T11:21:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.240+0000 7fc764c00700 1 -- 172.21.15.27:0/1798801334 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc750022470 con 0x7fc768075ff0 2024-05-10T11:21:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.240+0000 7fc76e400700 1 -- 172.21.15.27:0/1798801334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7681a0ff0 con 0x7fc768075ff0 2024-05-10T11:21:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.240+0000 7fc764c00700 1 -- 172.21.15.27:0/1798801334 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc750017650 con 0x7fc768075ff0 2024-05-10T11:21:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.240+0000 7fc76e400700 1 -- 172.21.15.27:0/1798801334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7681a88d0 con 0x7fc768075ff0 2024-05-10T11:21:08.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.241+0000 7fc76e400700 1 -- 172.21.15.27:0/1798801334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc76806a190 con 0x7fc768075ff0 2024-05-10T11:21:08.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.243+0000 7fc764c00700 1 -- 172.21.15.27:0/1798801334 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc75002a070 con 0x7fc768075ff0 2024-05-10T11:21:08.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.245+0000 7fc764c00700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc74c0b2a70 0x7fc74c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:08.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.245+0000 7fc764c00700 1 -- 172.21.15.27:0/1798801334 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9679+0+0 (secure 0 0 0) 0x7fc7500e36e0 con 0x7fc768075ff0 2024-05-10T11:21:08.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.245+0000 7fc767400700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc74c0b2a70 0x7fc74c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:08.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.246+0000 7fc767400700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc74c0b2a70 0x7fc74c0b4f30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc758009440 tx=0x7fc75800b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:08.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.247+0000 7fc764c00700 1 -- 172.21.15.27:0/1798801334 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fc750013070 con 0x7fc768075ff0 2024-05-10T11:21:08.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.548+0000 7fc76e400700 1 -- 172.21.15.27:0/1798801334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc76802e510 con 0x7fc768075ff0 2024-05-10T11:21:08.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.549+0000 7fc764c00700 1 -- 172.21.15.27:0/1798801334 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11936 (secure 0 0 0) 0x7fc7500aac70 con 0x7fc768075ff0 2024-05-10T11:21:08.550 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:21:08.550 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:17:36.552331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:21:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.552+0000 7fc76e400700 1 -- 172.21.15.27:0/1798801334 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc74c0b2a70 msgr2=0x7fc74c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.552+0000 7fc76e400700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc74c0b2a70 0x7fc74c0b4f30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc758009440 tx=0x7fc75800b040 comp rx=0 tx=0).stop 2024-05-10T11:21:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.552+0000 7fc76e400700 1 -- 172.21.15.27:0/1798801334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc768075ff0 msgr2=0x7fc76819c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.552+0000 7fc76e400700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc768075ff0 0x7fc76819c2e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc7500098e0 tx=0x7fc7500057c0 comp rx=0 tx=0).stop 2024-05-10T11:21:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.553+0000 7fc76e400700 1 -- 172.21.15.27:0/1798801334 shutdown_connections 2024-05-10T11:21:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.553+0000 7fc76e400700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc74c0b2a70 0x7fc74c0b4f30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.553+0000 7fc76e400700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76810fea0 0x7fc7681a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.553+0000 7fc76e400700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7680756d0 0x7fc76819bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.553+0000 7fc76e400700 1 --2- 172.21.15.27:0/1798801334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc768075ff0 0x7fc76819c2e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.553+0000 7fc76e400700 1 -- 172.21.15.27:0/1798801334 >> 172.21.15.27:0/1798801334 conn(0x7fc76807a600 msgr2=0x7fc7681006b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.553+0000 7fc76e400700 1 -- 172.21.15.27:0/1798801334 shutdown_connections 2024-05-10T11:21:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:08.553+0000 7fc76e400700 1 -- 172.21.15.27:0/1798801334 wait complete. 2024-05-10T11:21:08.555 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 33 2024-05-10T11:21:08.701 INFO:tasks.workunit.client.0.smithi027.stdout: 1 454490 6.65 MB/sec execute 233 sec latency 5010.225 ms 2024-05-10T11:21:09.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:09 smithi027 ceph-mon[30054]: pgmap v243: 129 pgs: 129 active+clean; 294 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 173 op/s 2024-05-10T11:21:09.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:09 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1798801334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:21:09.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:09 smithi027 ceph-mon[30054]: osdmap e95: 12 total, 12 up, 12 in 2024-05-10T11:21:09.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:21:09.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:09 smithi189 ceph-mon[34842]: pgmap v243: 129 pgs: 129 active+clean; 294 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 173 op/s 2024-05-10T11:21:09.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:09 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1798801334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:21:09.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:09 smithi189 ceph-mon[34842]: osdmap e95: 12 total, 12 up, 12 in 2024-05-10T11:21:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:09 smithi076 ceph-mon[34152]: pgmap v243: 129 pgs: 129 active+clean; 294 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 173 op/s 2024-05-10T11:21:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:09 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1798801334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:21:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:09 smithi076 ceph-mon[34152]: osdmap e95: 12 total, 12 up, 12 in 2024-05-10T11:21:09.701 INFO:tasks.workunit.client.0.smithi027.stdout: 1 455568 6.64 MB/sec execute 234 sec latency 59.689 ms 2024-05-10T11:21:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:21:10.701 INFO:tasks.workunit.client.0.smithi027.stdout: 1 457346 6.65 MB/sec execute 235 sec latency 22.553 ms 2024-05-10T11:21:11.486 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:11 smithi027 ceph-mon[30054]: pgmap v245: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 2.4 MiB/s wr, 177 op/s 2024-05-10T11:21:11.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:11 smithi189 ceph-mon[34842]: pgmap v245: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 2.4 MiB/s wr, 177 op/s 2024-05-10T11:21:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:11 smithi076 ceph-mon[34152]: pgmap v245: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 2.4 MiB/s wr, 177 op/s 2024-05-10T11:21:11.702 INFO:tasks.workunit.client.0.smithi027.stdout: 1 458040 6.64 MB/sec execute 236 sec latency 21.001 ms 2024-05-10T11:21:12.702 INFO:tasks.workunit.client.0.smithi027.stdout: 1 458725 6.67 MB/sec execute 237 sec latency 98.578 ms 2024-05-10T11:21:12.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.769+0000 7ff25ae00700 1 -- 172.21.15.27:0/4258136873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff254076160 msgr2=0x7ff254113ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:12.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.769+0000 7ff25ae00700 1 --2- 172.21.15.27:0/4258136873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff254076160 0x7ff254113ea0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff240009990 tx=0x7ff240009ca0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.769+0000 7ff25ae00700 1 -- 172.21.15.27:0/4258136873 shutdown_connections 2024-05-10T11:21:12.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.769+0000 7ff25ae00700 1 --2- 172.21.15.27:0/4258136873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff254075840 0x7ff254075c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.769+0000 7ff25ae00700 1 --2- 172.21.15.27:0/4258136873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2541143e0 0x7ff2541167d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.769+0000 7ff25ae00700 1 --2- 172.21.15.27:0/4258136873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff254076160 0x7ff254113ea0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.769+0000 7ff25ae00700 1 -- 172.21.15.27:0/4258136873 >> 172.21.15.27:0/4258136873 conn(0x7ff2540fef20 msgr2=0x7ff254101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:12.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.769+0000 7ff25ae00700 1 -- 172.21.15.27:0/4258136873 shutdown_connections 2024-05-10T11:21:12.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.769+0000 7ff25ae00700 1 -- 172.21.15.27:0/4258136873 wait complete. 2024-05-10T11:21:12.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.770+0000 7ff25ae00700 1 Processor -- start 2024-05-10T11:21:12.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.770+0000 7ff25ae00700 1 -- start start 2024-05-10T11:21:12.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.771+0000 7ff25ae00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff254075840 0x7ff2541a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.771+0000 7ff25ae00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff254076160 0x7ff2541a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.771+0000 7ff25ae00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2541143e0 0x7ff2541a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.771+0000 7ff25ae00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff254119c00 con 0x7ff254076160 2024-05-10T11:21:12.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.771+0000 7ff25ae00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff254119a80 con 0x7ff2541143e0 2024-05-10T11:21:12.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.771+0000 7ff25ae00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff254119d80 con 0x7ff254075840 2024-05-10T11:21:12.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.771+0000 7ff259a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff254075840 0x7ff2541a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.771+0000 7ff259a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff254075840 0x7ff2541a00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:34458/0 (socket says 172.21.15.27:34458) 2024-05-10T11:21:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.771+0000 7ff259000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff254076160 0x7ff2541a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.771+0000 7ff259a00700 1 -- 172.21.15.27:0/3306344099 learned_addr learned my addr 172.21.15.27:0/3306344099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.771+0000 7ff25a400700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2541143e0 0x7ff2541a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.772+0000 7ff259a00700 1 -- 172.21.15.27:0/3306344099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2541143e0 msgr2=0x7ff2541a49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.772+0000 7ff259a00700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2541143e0 0x7ff2541a49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.772+0000 7ff259a00700 1 -- 172.21.15.27:0/3306344099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff254076160 msgr2=0x7ff2541a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.772+0000 7ff259a00700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff254076160 0x7ff2541a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.772+0000 7ff259a00700 1 -- 172.21.15.27:0/3306344099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2541a5100 con 0x7ff254075840 2024-05-10T11:21:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.772+0000 7ff259a00700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff254075840 0x7ff2541a00f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff24800eb30 tx=0x7ff24800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.772+0000 7ff252000700 1 -- 172.21.15.27:0/3306344099 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24800cbd0 con 0x7ff254075840 2024-05-10T11:21:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.772+0000 7ff25ae00700 1 -- 172.21.15.27:0/3306344099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2541a53f0 con 0x7ff254075840 2024-05-10T11:21:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.772+0000 7ff252000700 1 -- 172.21.15.27:0/3306344099 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2480044d0 con 0x7ff254075840 2024-05-10T11:21:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.772+0000 7ff252000700 1 -- 172.21.15.27:0/3306344099 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248003e30 con 0x7ff254075840 2024-05-10T11:21:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.772+0000 7ff25ae00700 1 -- 172.21.15.27:0/3306344099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2541acc70 con 0x7ff254075840 2024-05-10T11:21:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.776+0000 7ff252000700 1 -- 172.21.15.27:0/3306344099 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff248004000 con 0x7ff254075840 2024-05-10T11:21:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.777+0000 7ff266d3cb80 1 Processor -- start 2024-05-10T11:21:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff266d3cb80 1 -- start start 2024-05-10T11:21:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff252000700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff23c0b29f0 0x7ff23c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff252000700 1 -- 172.21.15.27:0/3306344099 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7ff2480e2e40 con 0x7ff254075840 2024-05-10T11:21:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff266d3cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fcda0 0x5628378fd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff266d3cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x5628378f7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff25a400700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff23c0b29f0 0x7ff23c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff266d3cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378f7a50 0x5628378f7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff266d3cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562837849a20 con 0x5628378fcda0 2024-05-10T11:21:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff266d3cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5628377e7760 con 0x5628378fd6c0 2024-05-10T11:21:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff266d3cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5628378459e0 con 0x5628378f7a50 2024-05-10T11:21:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff25a400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378f7a50 0x5628378f7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff259000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x5628378f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff259000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x5628378f7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35556/0 (socket says 172.21.15.27:35556) 2024-05-10T11:21:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.778+0000 7ff259000700 1 -- 172.21.15.27:0/1325165204 learned_addr learned my addr 172.21.15.27:0/1325165204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.779+0000 7ff259a00700 1 --2- 172.21.15.27:0/1325165204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fcda0 0x5628378fd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.779+0000 7ff259000700 1 -- 172.21.15.27:0/1325165204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378f7a50 msgr2=0x5628378f7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.779+0000 7ff259000700 1 --2- 172.21.15.27:0/1325165204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378f7a50 0x5628378f7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.779+0000 7ff259000700 1 -- 172.21.15.27:0/1325165204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fcda0 msgr2=0x5628378fd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.779+0000 7ff259000700 1 --2- 172.21.15.27:0/1325165204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fcda0 0x5628378fd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.779+0000 7ff259000700 1 -- 172.21.15.27:0/1325165204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628377968d0 con 0x5628378fd6c0 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.779+0000 7ff259a00700 1 --2- 172.21.15.27:0/1325165204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fcda0 0x5628378fd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.779+0000 7ff25a400700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff23c0b29f0 0x7ff23c0b4eb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff24c009ec0 tx=0x7ff24c013480 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.779+0000 7ff259000700 1 --2- 172.21.15.27:0/1325165204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x5628378f7430 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff240015000 tx=0x7ff2400096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3515f45614615c9c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.780+0000 7ff246a00700 1 -- 172.21.15.27:0/1325165204 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240027070 con 0x5628378fd6c0 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.780+0000 7ff246a00700 1 -- 172.21.15.27:0/1325165204 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2400056d0 con 0x5628378fd6c0 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.780+0000 7ff246a00700 1 -- 172.21.15.27:0/1325165204 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240003aa0 con 0x5628378fd6c0 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.780+0000 7ff266d3cb80 1 -- 172.21.15.27:0/1325165204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 msgr2=0x5628378f7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.780+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/1325165204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x5628378f7430 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff240015000 tx=0x7ff2400096a0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.780+0000 7ff266d3cb80 1 -- 172.21.15.27:0/1325165204 shutdown_connections 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.780+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/1325165204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378f7a50 0x5628378f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.780+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/1325165204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fcda0 0x5628378fd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.780+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/1325165204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x5628378f7430 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.781+0000 7ff266d3cb80 1 -- 172.21.15.27:0/1325165204 >> 172.21.15.27:0/1325165204 conn(0x56283786f6d0 msgr2=0x5628378f1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.781+0000 7ff266d3cb80 1 -- 172.21.15.27:0/1325165204 shutdown_connections 2024-05-10T11:21:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.781+0000 7ff266d3cb80 1 -- 172.21.15.27:0/1325165204 wait complete. 2024-05-10T11:21:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.781+0000 7ff266d3cb80 1 -- start start 2024-05-10T11:21:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.781+0000 7ff266d3cb80 1 Processor -- start 2024-05-10T11:21:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.782+0000 7ff266d3cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378f7a50 0x562837903370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.782+0000 7ff266d3cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fcda0 0x562837900710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.782+0000 7ff266d3cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x562837900c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.782+0000 7ff266d3cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5628379040f0 con 0x5628378fcda0 2024-05-10T11:21:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.782+0000 7ff266d3cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5628377afb20 con 0x5628378fd6c0 2024-05-10T11:21:12.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.782+0000 7ff266d3cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562837904270 con 0x5628378f7a50 2024-05-10T11:21:12.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.782+0000 7ff266d3cb80 10 client.0 _ll_register_callbacks cb 0x5628377f0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:12.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.782+0000 7ff259a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378f7a50 0x562837903370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.782+0000 7ff25a400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x562837900c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.782+0000 7ff259a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378f7a50 0x562837903370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:34468/0 (socket says 172.21.15.27:34468) 2024-05-10T11:21:12.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.782+0000 7ff259a00700 1 -- 172.21.15.27:0/2546867230 learned_addr learned my addr 172.21.15.27:0/2546867230 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:12.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.782+0000 7ff259000700 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fcda0 0x562837900710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff259a00700 1 -- 172.21.15.27:0/2546867230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 msgr2=0x562837900c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:12.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff259a00700 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x562837900c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff259a00700 1 -- 172.21.15.27:0/2546867230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fcda0 msgr2=0x562837900710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:12.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff259a00700 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fcda0 0x562837900710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff259a00700 1 -- 172.21.15.27:0/2546867230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56283780f420 con 0x5628378f7a50 2024-05-10T11:21:12.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff259000700 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fcda0 0x562837900710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:21:12.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff25a400700 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x562837900c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:21:12.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff259a00700 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378f7a50 0x562837903370 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff2480a6f00 tx=0x7ff2480a5c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d731295f5c654154 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:12.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff23b400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:21:12.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff23b400700 1 -- 172.21.15.27:0/2546867230 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2480a97a0 con 0x5628378f7a50 2024-05-10T11:21:12.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff23b400700 1 -- 172.21.15.27:0/2546867230 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7ff2480c4310 con 0x5628378f7a50 2024-05-10T11:21:12.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff266d3cb80 1 -- 172.21.15.27:0/2546867230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2400056d0 con 0x5628378f7a50 2024-05-10T11:21:12.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.783+0000 7ff23b400700 1 -- 172.21.15.27:0/2546867230 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff2480042f0 con 0x5628378f7a50 2024-05-10T11:21:12.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.784+0000 7ff266d3cb80 10 client.26419 fetch_fsmap learned FSMap version 33 2024-05-10T11:21:12.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.784+0000 7ff266d3cb80 1 -- 172.21.15.27:0/2546867230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56283790e950 con 0x5628378f7a50 2024-05-10T11:21:12.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.784+0000 7ff23b400700 1 -- 172.21.15.27:0/2546867230 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2400056d0 con 0x5628378f7a50 2024-05-10T11:21:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.784+0000 7ff23b400700 1 -- 172.21.15.27:0/2546867230 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff2480b2660 con 0x5628378f7a50 2024-05-10T11:21:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.785+0000 7ff266d3cb80 10 client.26419 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:21:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.785+0000 7ff266d3cb80 10 client.26419 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:21:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.785+0000 7ff266d3cb80 20 client.26419 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.785+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x562837910f90 0x562837913380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.785+0000 7ff266d3cb80 4 client.26419 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:21:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.785+0000 7ff266d3cb80 1 -- 172.21.15.27:0/2546867230 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628378c4140 con 0x562837910f90 2024-05-10T11:21:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.785+0000 7ff259000700 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x562837910f90 0x562837913380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.786+0000 7ff259000700 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x562837910f90 0x562837913380 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f214360bd1838deb server_cookie=dfa810af27221723 in_seq=0 out_seq=0 2024-05-10T11:21:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.786+0000 7ff23b400700 10 client.26419 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:21:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.786+0000 7ff23b400700 1 -- 172.21.15.27:0/2546867230 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5628378c4140 con 0x562837910f90 2024-05-10T11:21:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff23b400700 10 client.26419 handle_command_reply: tid=0 2024-05-10T11:21:12.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff266d3cb80 1 client.26419 shutdown 2024-05-10T11:21:12.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff266d3cb80 1 -- 172.21.15.27:0/2546867230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378f7a50 msgr2=0x562837903370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:12.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378f7a50 0x562837903370 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff2480a6f00 tx=0x7ff2480a5c60 comp rx=0 tx=0).stop 2024-05-10T11:21:12.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff266d3cb80 1 -- 172.21.15.27:0/2546867230 shutdown_connections 2024-05-10T11:21:12.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x562837910f90 0x562837913380 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378f7a50 0x562837903370 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x562837900c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/2546867230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fcda0 0x562837900710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff266d3cb80 1 -- 172.21.15.27:0/2546867230 >> 172.21.15.27:0/2546867230 conn(0x56283786f6d0 msgr2=0x5628378ec750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:12.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff23b400700 0 client.26419 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:21:12.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff266d3cb80 1 -- 172.21.15.27:0/2546867230 shutdown_connections 2024-05-10T11:21:12.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff266d3cb80 1 -- 172.21.15.27:0/2546867230 wait complete. 2024-05-10T11:21:12.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.787+0000 7ff266d3cb80 20 client.26419 trim_cache size 0 max 16384 2024-05-10T11:21:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.806+0000 7ff266d3cb80 1 Processor -- start 2024-05-10T11:21:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.806+0000 7ff266d3cb80 1 -- start start 2024-05-10T11:21:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.806+0000 7ff266d3cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378f7a50 0x56283790d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.806+0000 7ff266d3cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378fcda0 0x5628378fe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.806+0000 7ff266d3cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x5628378ec860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.806+0000 7ff266d3cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2541b5c00 con 0x5628378f7a50 2024-05-10T11:21:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.806+0000 7ff266d3cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff2541b7740 con 0x5628378fd6c0 2024-05-10T11:21:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.806+0000 7ff266d3cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2541b62d0 con 0x5628378fcda0 2024-05-10T11:21:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.807+0000 7ff259a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378f7a50 0x56283790d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.807+0000 7ff259000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378fcda0 0x5628378fe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.807+0000 7ff259a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378f7a50 0x56283790d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43586/0 (socket says 172.21.15.27:43586) 2024-05-10T11:21:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.807+0000 7ff259a00700 1 -- 172.21.15.27:0/3485118021 learned_addr learned my addr 172.21.15.27:0/3485118021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.807+0000 7ff25a400700 1 --2- 172.21.15.27:0/3485118021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x5628378ec860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.807+0000 7ff259a00700 1 -- 172.21.15.27:0/3485118021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378fcda0 msgr2=0x5628378fe1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.807+0000 7ff259a00700 1 --2- 172.21.15.27:0/3485118021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378fcda0 0x5628378fe1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.807+0000 7ff259a00700 1 -- 172.21.15.27:0/3485118021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 msgr2=0x5628378ec860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.807+0000 7ff259a00700 1 --2- 172.21.15.27:0/3485118021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x5628378ec860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.807+0000 7ff259a00700 1 -- 172.21.15.27:0/3485118021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628378ee8a0 con 0x5628378f7a50 2024-05-10T11:21:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.807+0000 7ff25a400700 1 --2- 172.21.15.27:0/3485118021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x5628378ec860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:21:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.808+0000 7ff259a00700 1 --2- 172.21.15.27:0/3485118021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378f7a50 0x56283790d180 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff2480046a0 tx=0x7ff248003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193830c78b348bb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.808+0000 7ff246000700 1 -- 172.21.15.27:0/3485118021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2480a97a0 con 0x5628378f7a50 2024-05-10T11:21:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.808+0000 7ff246000700 1 -- 172.21.15.27:0/3485118021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2480a9c90 con 0x5628378f7a50 2024-05-10T11:21:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.808+0000 7ff246000700 1 -- 172.21.15.27:0/3485118021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2480b2af0 con 0x5628378f7a50 2024-05-10T11:21:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.809+0000 7ff266d3cb80 1 -- 172.21.15.27:0/3485118021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378f7a50 msgr2=0x56283790d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.809+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/3485118021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378f7a50 0x56283790d180 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff2480046a0 tx=0x7ff248003890 comp rx=0 tx=0).stop 2024-05-10T11:21:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.809+0000 7ff266d3cb80 1 -- 172.21.15.27:0/3485118021 shutdown_connections 2024-05-10T11:21:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.809+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/3485118021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378fd6c0 0x5628378ec860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.809+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/3485118021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378f7a50 0x56283790d180 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.809+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/3485118021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378fcda0 0x5628378fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.809+0000 7ff266d3cb80 1 -- 172.21.15.27:0/3485118021 >> 172.21.15.27:0/3485118021 conn(0x56283786f6d0 msgr2=0x5628378fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.809+0000 7ff266d3cb80 1 -- 172.21.15.27:0/3485118021 shutdown_connections 2024-05-10T11:21:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.809+0000 7ff266d3cb80 1 -- 172.21.15.27:0/3485118021 wait complete. 2024-05-10T11:21:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.810+0000 7ff266d3cb80 1 -- start start 2024-05-10T11:21:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.810+0000 7ff266d3cb80 1 Processor -- start 2024-05-10T11:21:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff266d3cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378f7a50 0x5628379096e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff266d3cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378fcda0 0x562837909c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff266d3cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fd6c0 0x562837917c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff266d3cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2541b7070 con 0x5628378fd6c0 2024-05-10T11:21:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff266d3cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff2541b69a0 con 0x5628378f7a50 2024-05-10T11:21:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff266d3cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2541b3bc0 con 0x5628378fcda0 2024-05-10T11:21:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff266d3cb80 10 client.0 _ll_register_callbacks cb 0x5628377f0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff259a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378f7a50 0x5628379096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff259a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378f7a50 0x5628379096e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:35592/0 (socket says 172.21.15.27:35592) 2024-05-10T11:21:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff259a00700 1 -- 172.21.15.27:0/522046293 learned_addr learned my addr 172.21.15.27:0/522046293 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff259000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378fcda0 0x562837909c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff25a400700 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fd6c0 0x562837917c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff259000700 1 -- 172.21.15.27:0/522046293 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378f7a50 msgr2=0x5628379096e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff259000700 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378f7a50 0x5628379096e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff259000700 1 -- 172.21.15.27:0/522046293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fd6c0 msgr2=0x562837917c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff259000700 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fd6c0 0x562837917c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.811+0000 7ff259000700 1 -- 172.21.15.27:0/522046293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628378f37a0 con 0x5628378fcda0 2024-05-10T11:21:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.812+0000 7ff259a00700 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378f7a50 0x5628379096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:21:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.812+0000 7ff259000700 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378fcda0 0x562837909c20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x5628378f3970 tx=0x7ff240005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29455d28ecb9967c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.812+0000 7ff23b400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:21:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.812+0000 7ff23b400700 1 -- 172.21.15.27:0/522046293 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240027040 con 0x5628378fcda0 2024-05-10T11:21:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.812+0000 7ff23b400700 1 -- 172.21.15.27:0/522046293 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7ff24000bbf0 con 0x5628378fcda0 2024-05-10T11:21:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.812+0000 7ff266d3cb80 1 -- 172.21.15.27:0/522046293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2480a9c90 con 0x5628378fcda0 2024-05-10T11:21:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.812+0000 7ff23b400700 1 -- 172.21.15.27:0/522046293 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff240042050 con 0x5628378fcda0 2024-05-10T11:21:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.812+0000 7ff266d3cb80 10 client.26425 fetch_fsmap learned FSMap version 33 2024-05-10T11:21:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.812+0000 7ff266d3cb80 1 -- 172.21.15.27:0/522046293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56283797b0e0 con 0x5628378fcda0 2024-05-10T11:21:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.813+0000 7ff23b400700 1 -- 172.21.15.27:0/522046293 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2480a9c90 con 0x5628378fcda0 2024-05-10T11:21:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.813+0000 7ff23b400700 1 -- 172.21.15.27:0/522046293 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff240021070 con 0x5628378fcda0 2024-05-10T11:21:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.813+0000 7ff266d3cb80 10 client.26425 fetch_fsmap finished waiting for FSMap version 33 2024-05-10T11:21:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.813+0000 7ff266d3cb80 10 client.26425 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:21:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.813+0000 7ff266d3cb80 20 client.26425 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.813+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56283797b2b0 0x56283797d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.813+0000 7ff266d3cb80 4 client.26425 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.813+0000 7ff266d3cb80 1 -- 172.21.15.27:0/522046293 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562837857400 con 0x56283797b2b0 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.813+0000 7ff259a00700 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56283797b2b0 0x56283797d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff259a00700 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56283797b2b0 0x56283797d6a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=637e4fb2e54d0f65 server_cookie=d2fe2aff0d7c53a0 in_seq=0 out_seq=0 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff23b400700 10 client.26425 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff23b400700 1 -- 172.21.15.27:0/522046293 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562837857400 con 0x56283797b2b0 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff23b400700 10 client.26425 handle_command_reply: tid=0 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff266d3cb80 1 client.26425 shutdown 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff266d3cb80 1 -- 172.21.15.27:0/522046293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378fcda0 msgr2=0x562837909c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378fcda0 0x562837909c20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x5628378f3970 tx=0x7ff240005e80 comp rx=0 tx=0).stop 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff266d3cb80 1 -- 172.21.15.27:0/522046293 shutdown_connections 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628378fd6c0 0x562837917c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5628378f7a50 0x5628379096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56283797b2b0 0x56283797d6a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff266d3cb80 1 --2- 172.21.15.27:0/522046293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628378fcda0 0x562837909c20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff23b400700 0 client.26425 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:21:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.814+0000 7ff266d3cb80 1 -- 172.21.15.27:0/522046293 >> 172.21.15.27:0/522046293 conn(0x56283786f6d0 msgr2=0x562837901d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.815+0000 7ff266d3cb80 1 -- 172.21.15.27:0/522046293 shutdown_connections 2024-05-10T11:21:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.815+0000 7ff266d3cb80 1 -- 172.21.15.27:0/522046293 wait complete. 2024-05-10T11:21:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.815+0000 7ff266d3cb80 20 client.26425 trim_cache size 0 max 16384 2024-05-10T11:21:12.832 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:21:12.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.816+0000 7ff25ae00700 1 -- 172.21.15.27:0/3306344099 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff23c0b29f0 msgr2=0x7ff23c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.816+0000 7ff25ae00700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff23c0b29f0 0x7ff23c0b4eb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff24c009ec0 tx=0x7ff24c013480 comp rx=0 tx=0).stop 2024-05-10T11:21:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.816+0000 7ff25ae00700 1 -- 172.21.15.27:0/3306344099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff254075840 msgr2=0x7ff2541a00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.816+0000 7ff25ae00700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff254075840 0x7ff2541a00f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff24800eb30 tx=0x7ff24800ee40 comp rx=0 tx=0).stop 2024-05-10T11:21:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.816+0000 7ff25ae00700 1 -- 172.21.15.27:0/3306344099 shutdown_connections 2024-05-10T11:21:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.816+0000 7ff25ae00700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff23c0b29f0 0x7ff23c0b4eb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.816+0000 7ff25ae00700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff254075840 0x7ff2541a00f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.816+0000 7ff25ae00700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2541143e0 0x7ff2541a49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.816+0000 7ff25ae00700 1 --2- 172.21.15.27:0/3306344099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff254076160 0x7ff2541a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:12.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.816+0000 7ff25ae00700 1 -- 172.21.15.27:0/3306344099 >> 172.21.15.27:0/3306344099 conn(0x7ff2540fef20 msgr2=0x7ff254100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:12.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.816+0000 7ff25ae00700 1 -- 172.21.15.27:0/3306344099 shutdown_connections 2024-05-10T11:21:12.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:12.816+0000 7ff25ae00700 1 -- 172.21.15.27:0/3306344099 wait complete. 2024-05-10T11:21:13.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:13 smithi189 ceph-mon[34842]: pgmap v246: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-05-10T11:21:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:13 smithi027 ceph-mon[30054]: pgmap v246: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-05-10T11:21:13.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:13 smithi076 ceph-mon[34152]: pgmap v246: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-05-10T11:21:13.702 INFO:tasks.workunit.client.0.smithi027.stdout: 1 459772 6.70 MB/sec execute 238 sec latency 25.999 ms 2024-05-10T11:21:13.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:21:14.362 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:21:14.702 INFO:tasks.workunit.client.0.smithi027.stdout: 1 460946 6.68 MB/sec execute 239 sec latency 150.392 ms 2024-05-10T11:21:15.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:15 smithi027 ceph-mon[30054]: pgmap v247: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 1.9 MiB/s wr, 142 op/s 2024-05-10T11:21:15.501 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:15 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:21:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:15 smithi076 ceph-mon[34152]: pgmap v247: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 1.9 MiB/s wr, 142 op/s 2024-05-10T11:21:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:15 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:21:15.702 INFO:tasks.workunit.client.0.smithi027.stdout: 1 461619 6.68 MB/sec execute 240 sec latency 744.280 ms 2024-05-10T11:21:15.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:15 smithi189 ceph-mon[34842]: pgmap v247: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 1.9 MiB/s wr, 142 op/s 2024-05-10T11:21:15.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:15 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:21:16.702 INFO:tasks.workunit.client.0.smithi027.stdout: 1 461619 6.65 MB/sec execute 241 sec latency 1744.417 ms 2024-05-10T11:21:16.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.808+0000 7fbbfca00700 1 -- 172.21.15.27:0/258155041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbf8109500 msgr2=0x7fbbf81098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:16.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.808+0000 7fbbfca00700 1 --2- 172.21.15.27:0/258155041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbf8109500 0x7fbbf81098e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009990 tx=0x7fbbe8009ca0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.808+0000 7fbbfca00700 1 -- 172.21.15.27:0/258155041 shutdown_connections 2024-05-10T11:21:16.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.808+0000 7fbbfca00700 1 --2- 172.21.15.27:0/258155041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf806c3d0 0x7fbbf8110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.808+0000 7fbbfca00700 1 --2- 172.21.15.27:0/258155041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbf8109500 0x7fbbf81098e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.808+0000 7fbbfca00700 1 --2- 172.21.15.27:0/258155041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbf806ba10 0x7fbbf806be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.808+0000 7fbbfca00700 1 -- 172.21.15.27:0/258155041 >> 172.21.15.27:0/258155041 conn(0x7fbbf8079310 msgr2=0x7fbbf8079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.809+0000 7fbbfca00700 1 -- 172.21.15.27:0/258155041 shutdown_connections 2024-05-10T11:21:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.809+0000 7fbbfca00700 1 -- 172.21.15.27:0/258155041 wait complete. 2024-05-10T11:21:16.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.810+0000 7fbbfca00700 1 Processor -- start 2024-05-10T11:21:16.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.810+0000 7fbbfca00700 1 -- start start 2024-05-10T11:21:16.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.810+0000 7fbbfca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbf806ba10 0x7fbbf819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.810+0000 7fbbfca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf806c3d0 0x7fbbf819fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.810+0000 7fbbfca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbf8109500 0x7fbbf8199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbfca00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8113610 con 0x7fbbf8109500 2024-05-10T11:21:16.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbfca00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8113490 con 0x7fbbf806ba10 2024-05-10T11:21:16.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbfca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8113790 con 0x7fbbf806c3d0 2024-05-10T11:21:16.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbf5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbf806ba10 0x7fbbf819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbf5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf806c3d0 0x7fbbf819fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbf5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbf806ba10 0x7fbbf819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42082/0 (socket says 172.21.15.27:42082) 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbf5e00700 1 -- 172.21.15.27:0/853506823 learned_addr learned my addr 172.21.15.27:0/853506823 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbf6800700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbf8109500 0x7fbbf8199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbf5e00700 1 -- 172.21.15.27:0/853506823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf806c3d0 msgr2=0x7fbbf819fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf806c3d0 0x7fbbf819fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbf5e00700 1 -- 172.21.15.27:0/853506823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbf8109500 msgr2=0x7fbbf8199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbf8109500 0x7fbbf8199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.811+0000 7fbbf5e00700 1 -- 172.21.15.27:0/853506823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf819a0b0 con 0x7fbbf806ba10 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.812+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbf806ba10 0x7fbbf819f790 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8000c00 tx=0x7fbbe800bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.812+0000 7fbbeea00700 1 -- 172.21.15.27:0/853506823 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8027070 con 0x7fbbf806ba10 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.812+0000 7fbbfca00700 1 -- 172.21.15.27:0/853506823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf819a340 con 0x7fbbf806ba10 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.812+0000 7fbbeea00700 1 -- 172.21.15.27:0/853506823 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbe8022470 con 0x7fbbf806ba10 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.812+0000 7fbbeea00700 1 -- 172.21.15.27:0/853506823 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8017650 con 0x7fbbf806ba10 2024-05-10T11:21:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.812+0000 7fbbfca00700 1 -- 172.21.15.27:0/853506823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf81a6660 con 0x7fbbf806ba10 2024-05-10T11:21:16.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.816+0000 7fbbeea00700 1 -- 172.21.15.27:0/853506823 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fbbe802a070 con 0x7fbbf806ba10 2024-05-10T11:21:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.817+0000 7fbc073d6b80 1 Processor -- start 2024-05-10T11:21:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.817+0000 7fbc073d6b80 1 -- start start 2024-05-10T11:21:16.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.817+0000 7fbc073d6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe57fd0 0x56377fe583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.817+0000 7fbc073d6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe588f0 0x56377fe526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.817+0000 7fbc073d6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe52d00 0x56377fe53180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.817+0000 7fbc073d6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56377fda0a20 con 0x56377fe57fd0 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.817+0000 7fbc073d6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56377fd3e760 con 0x56377fe588f0 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.817+0000 7fbc073d6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56377fd9c9e0 con 0x56377fe52d00 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe57fd0 0x56377fe583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe52d00 0x56377fe53180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe52d00 0x56377fe53180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:32930/0 (socket says 172.21.15.27:32930) 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf5400700 1 -- 172.21.15.27:0/3420444123 learned_addr learned my addr 172.21.15.27:0/3420444123 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe57fd0 0x56377fe583b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40338/0 (socket says 172.21.15.27:40338) 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/3420444123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe588f0 0x56377fe526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbeea00700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbbdc0b29f0 0x7fbbdc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf5400700 1 -- 172.21.15.27:0/3420444123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe588f0 msgr2=0x56377fe526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf5400700 1 --2- 172.21.15.27:0/3420444123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe588f0 0x56377fe526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf5400700 1 -- 172.21.15.27:0/3420444123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe57fd0 msgr2=0x56377fe583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf5400700 1 --2- 172.21.15.27:0/3420444123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe57fd0 0x56377fe583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf5400700 1 -- 172.21.15.27:0/3420444123 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56377fced8d0 con 0x56377fe52d00 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf6800700 1 --2- 172.21.15.27:0/3420444123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe57fd0 0x56377fe583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.818+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/3420444123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe588f0 0x56377fe526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.819+0000 7fbbeea00700 1 -- 172.21.15.27:0/853506823 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7fbbe80e3720 con 0x7fbbf806ba10 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.819+0000 7fbbf5400700 1 --2- 172.21.15.27:0/3420444123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe52d00 0x56377fe53180 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009d20 tx=0x7fbbe4011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c3f316cc08fd19b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.819+0000 7fbbf6800700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbbdc0b29f0 0x7fbbdc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.819+0000 7fbbe3400700 1 -- 172.21.15.27:0/3420444123 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe401e070 con 0x56377fe52d00 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.819+0000 7fbbe3400700 1 -- 172.21.15.27:0/3420444123 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbe4019ae0 con 0x56377fe52d00 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.819+0000 7fbbe3400700 1 -- 172.21.15.27:0/3420444123 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe400ea30 con 0x56377fe52d00 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.820+0000 7fbc073d6b80 1 -- 172.21.15.27:0/3420444123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe52d00 msgr2=0x56377fe53180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.820+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/3420444123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe52d00 0x56377fe53180 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009d20 tx=0x7fbbe4011720 comp rx=0 tx=0).stop 2024-05-10T11:21:16.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.820+0000 7fbbf6800700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbbdc0b29f0 0x7fbbdc0b4eb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009ec0 tx=0x7fbbf0009040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:16.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.820+0000 7fbc073d6b80 1 -- 172.21.15.27:0/3420444123 shutdown_connections 2024-05-10T11:21:16.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.820+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/3420444123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe52d00 0x56377fe53180 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.820+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/3420444123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe57fd0 0x56377fe583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.820+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/3420444123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe588f0 0x56377fe526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.820+0000 7fbc073d6b80 1 -- 172.21.15.27:0/3420444123 >> 172.21.15.27:0/3420444123 conn(0x56377fdc66d0 msgr2=0x56377fe48b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:16.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.820+0000 7fbc073d6b80 1 -- 172.21.15.27:0/3420444123 shutdown_connections 2024-05-10T11:21:16.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.820+0000 7fbc073d6b80 1 -- 172.21.15.27:0/3420444123 wait complete. 2024-05-10T11:21:16.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.821+0000 7fbc073d6b80 1 -- start start 2024-05-10T11:21:16.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.821+0000 7fbc073d6b80 1 Processor -- start 2024-05-10T11:21:16.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbc073d6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 0x56377fe5e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbc073d6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 0x56377fe5b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbc073d6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe588f0 0x56377fe5bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbc073d6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56377fe5f330 con 0x56377fe588f0 2024-05-10T11:21:16.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbc073d6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56377fd06b20 con 0x56377fe52d00 2024-05-10T11:21:16.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbc073d6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56377fe5f4b0 con 0x56377fe57fd0 2024-05-10T11:21:16.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbc073d6b80 10 client.0 _ll_register_callbacks cb 0x56377fd47c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:16.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbbf5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 0x56377fe5e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbbf5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 0x56377fe5b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbbf5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 0x56377fe5e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42086/0 (socket says 172.21.15.27:42086) 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbbf5e00700 1 -- 172.21.15.27:0/2321373558 learned_addr learned my addr 172.21.15.27:0/2321373558 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbbf6800700 1 --2- 172.21.15.27:0/2321373558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe588f0 0x56377fe5bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbbf5e00700 1 -- 172.21.15.27:0/2321373558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 msgr2=0x56377fe5b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/2321373558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 0x56377fe5b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbbf5e00700 1 -- 172.21.15.27:0/2321373558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe588f0 msgr2=0x56377fe5bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.822+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/2321373558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe588f0 0x56377fe5bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.823+0000 7fbbf5e00700 1 -- 172.21.15.27:0/2321373558 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56377fd66420 con 0x56377fe52d00 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.823+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/2321373558 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 0x56377fe5e5f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80e0a40 tx=0x7fbbe8022c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b1d1762b960c915 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.823+0000 7fbbd7e00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.823+0000 7fbbd7e00700 1 -- 172.21.15.27:0/2321373558 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe80aa860 con 0x56377fe52d00 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.823+0000 7fbbd7e00700 1 -- 172.21.15.27:0/2321373558 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7fbbe80b2f40 con 0x56377fe52d00 2024-05-10T11:21:16.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.823+0000 7fbc073d6b80 1 -- 172.21.15.27:0/2321373558 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbe4019ae0 con 0x56377fe52d00 2024-05-10T11:21:16.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.823+0000 7fbbd7e00700 1 -- 172.21.15.27:0/2321373558 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbbe80aa600 con 0x56377fe52d00 2024-05-10T11:21:16.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.824+0000 7fbc073d6b80 10 client.26456 fetch_fsmap learned FSMap version 34 2024-05-10T11:21:16.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.824+0000 7fbc073d6b80 1 -- 172.21.15.27:0/2321373558 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56377fe69bb0 con 0x56377fe52d00 2024-05-10T11:21:16.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.824+0000 7fbbd7e00700 1 -- 172.21.15.27:0/2321373558 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbe4019ae0 con 0x56377fe52d00 2024-05-10T11:21:16.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.824+0000 7fbbd7e00700 1 -- 172.21.15.27:0/2321373558 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fbbe80e3230 con 0x56377fe52d00 2024-05-10T11:21:16.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.824+0000 7fbc073d6b80 10 client.26456 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:21:16.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.824+0000 7fbc073d6b80 10 client.26456 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:21:16.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.824+0000 7fbc073d6b80 20 client.26456 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:16.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.824+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/2321373558 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56377fe6c080 0x56377fe6e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.824+0000 7fbc073d6b80 4 client.26456 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.824+0000 7fbc073d6b80 1 -- 172.21.15.27:0/2321373558 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56377fe1b140 con 0x56377fe6c080 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.825+0000 7fbbf5400700 1 --2- 172.21.15.27:0/2321373558 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56377fe6c080 0x56377fe6e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.825+0000 7fbbf5400700 1 --2- 172.21.15.27:0/2321373558 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56377fe6c080 0x56377fe6e470 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5f3b56ad64833eef server_cookie=aa349d2a9fa4e990 in_seq=0 out_seq=0 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.825+0000 7fbbd7e00700 10 client.26456 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.826+0000 7fbbd7e00700 1 -- 172.21.15.27:0/2321373558 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56377fe1b140 con 0x56377fe6c080 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.826+0000 7fbbd7e00700 10 client.26456 handle_command_reply: tid=0 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.826+0000 7fbc073d6b80 1 client.26456 shutdown 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.827+0000 7fbc073d6b80 1 -- 172.21.15.27:0/2321373558 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 msgr2=0x56377fe5e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.827+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/2321373558 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 0x56377fe5e5f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80e0a40 tx=0x7fbbe8022c20 comp rx=0 tx=0).stop 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.827+0000 7fbc073d6b80 1 -- 172.21.15.27:0/2321373558 shutdown_connections 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.827+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/2321373558 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56377fe6c080 0x56377fe6e470 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.827+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/2321373558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe588f0 0x56377fe5bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.827+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/2321373558 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 0x56377fe5e5f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.827+0000 7fbbd7e00700 0 client.26456 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:21:16.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.827+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/2321373558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 0x56377fe5b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.827+0000 7fbc073d6b80 1 -- 172.21.15.27:0/2321373558 >> 172.21.15.27:0/2321373558 conn(0x56377fdc66d0 msgr2=0x56377fe48190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:16.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.827+0000 7fbc073d6b80 1 -- 172.21.15.27:0/2321373558 shutdown_connections 2024-05-10T11:21:16.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.827+0000 7fbc073d6b80 1 -- 172.21.15.27:0/2321373558 wait complete. 2024-05-10T11:21:16.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.827+0000 7fbc073d6b80 20 client.26456 trim_cache size 0 max 16384 2024-05-10T11:21:16.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.844+0000 7fbc073d6b80 1 Processor -- start 2024-05-10T11:21:16.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.844+0000 7fbc073d6b80 1 -- start start 2024-05-10T11:21:16.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.844+0000 7fbc073d6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 0x56377fe526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbc073d6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 0x56377fe683c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbc073d6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe588f0 0x56377fe48910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbc073d6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81ad350 con 0x56377fe588f0 2024-05-10T11:21:16.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbc073d6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81aeec0 con 0x56377fe52d00 2024-05-10T11:21:16.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbc073d6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81ada20 con 0x56377fe57fd0 2024-05-10T11:21:16.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbbf5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 0x56377fe526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbbf6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe588f0 0x56377fe48910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbbf5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 0x56377fe526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42092/0 (socket says 172.21.15.27:42092) 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbbf5e00700 1 -- 172.21.15.27:0/364311166 learned_addr learned my addr 172.21.15.27:0/364311166 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbbf5400700 1 --2- 172.21.15.27:0/364311166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 0x56377fe683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbbf5e00700 1 -- 172.21.15.27:0/364311166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 msgr2=0x56377fe683c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/364311166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 0x56377fe683c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbbf5e00700 1 -- 172.21.15.27:0/364311166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe588f0 msgr2=0x56377fe48910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/364311166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe588f0 0x56377fe48910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.845+0000 7fbbf5e00700 1 -- 172.21.15.27:0/364311166 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56377fe58600 con 0x56377fe52d00 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.846+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/364311166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 0x56377fe526e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbbe802e040 tx=0x7fbbe8023fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28e9d18d79d2a8ab server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.846+0000 7fbbe2a00700 1 -- 172.21.15.27:0/364311166 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe80aa110 con 0x56377fe52d00 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.846+0000 7fbbe2a00700 1 -- 172.21.15.27:0/364311166 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbe80aad00 con 0x56377fe52d00 2024-05-10T11:21:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.846+0000 7fbbe2a00700 1 -- 172.21.15.27:0/364311166 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe80a9b90 con 0x56377fe52d00 2024-05-10T11:21:16.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.847+0000 7fbc073d6b80 1 -- 172.21.15.27:0/364311166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 msgr2=0x56377fe526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:16.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.847+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/364311166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 0x56377fe526e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbbe802e040 tx=0x7fbbe8023fc0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.847+0000 7fbc073d6b80 1 -- 172.21.15.27:0/364311166 shutdown_connections 2024-05-10T11:21:16.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.847+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/364311166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe588f0 0x56377fe48910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.847+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/364311166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe52d00 0x56377fe526e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.847+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/364311166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 0x56377fe683c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.847+0000 7fbc073d6b80 1 -- 172.21.15.27:0/364311166 >> 172.21.15.27:0/364311166 conn(0x56377fdc66d0 msgr2=0x56377fe56b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.847+0000 7fbc073d6b80 1 -- 172.21.15.27:0/364311166 shutdown_connections 2024-05-10T11:21:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.847+0000 7fbc073d6b80 1 -- 172.21.15.27:0/364311166 wait complete. 2024-05-10T11:21:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.848+0000 7fbc073d6b80 1 -- start start 2024-05-10T11:21:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.848+0000 7fbc073d6b80 1 Processor -- start 2024-05-10T11:21:16.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.848+0000 7fbc073d6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe52d00 0x56377fe64740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.848+0000 7fbc073d6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 0x56377fe64c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.848+0000 7fbc073d6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe588f0 0x56377fe61710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.848+0000 7fbc073d6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81ae7f0 con 0x56377fe52d00 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.848+0000 7fbc073d6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81ae120 con 0x56377fe588f0 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.848+0000 7fbc073d6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81ab310 con 0x56377fe57fd0 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.848+0000 7fbc073d6b80 10 client.0 _ll_register_callbacks cb 0x56377fd47c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.849+0000 7fbbf5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe52d00 0x56377fe64740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.849+0000 7fbbf5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe52d00 0x56377fe64740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40352/0 (socket says 172.21.15.27:40352) 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.849+0000 7fbbf5e00700 1 -- 172.21.15.27:0/877515535 learned_addr learned my addr 172.21.15.27:0/877515535 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.849+0000 7fbbf5400700 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 0x56377fe64c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.849+0000 7fbbf6800700 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe588f0 0x56377fe61710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.849+0000 7fbbf5e00700 1 -- 172.21.15.27:0/877515535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 msgr2=0x56377fe64c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.849+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 0x56377fe64c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.849+0000 7fbbf5e00700 1 -- 172.21.15.27:0/877515535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe588f0 msgr2=0x56377fe61710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.849+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe588f0 0x56377fe61710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.849+0000 7fbbf5e00700 1 -- 172.21.15.27:0/877515535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56377fe61d90 con 0x56377fe52d00 2024-05-10T11:21:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.850+0000 7fbbf5e00700 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe52d00 0x56377fe64740 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80b8040 tx=0x7fbbe80b31d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9abc299107ba77 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.850+0000 7fbbd7e00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:21:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.850+0000 7fbbd7e00700 1 -- 172.21.15.27:0/877515535 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe80c3060 con 0x56377fe52d00 2024-05-10T11:21:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.850+0000 7fbbd7e00700 1 -- 172.21.15.27:0/877515535 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7fbbe80afd50 con 0x56377fe52d00 2024-05-10T11:21:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.850+0000 7fbc073d6b80 1 -- 172.21.15.27:0/877515535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbe80aad00 con 0x56377fe52d00 2024-05-10T11:21:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.850+0000 7fbbd7e00700 1 -- 172.21.15.27:0/877515535 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbbe80aa110 con 0x56377fe52d00 2024-05-10T11:21:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.850+0000 7fbc073d6b80 10 client.16245 fetch_fsmap learned FSMap version 34 2024-05-10T11:21:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.850+0000 7fbc073d6b80 1 -- 172.21.15.27:0/877515535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56377fe61d90 con 0x56377fe52d00 2024-05-10T11:21:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.851+0000 7fbbd7e00700 1 -- 172.21.15.27:0/877515535 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbe80aad00 con 0x56377fe52d00 2024-05-10T11:21:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.851+0000 7fbbd7e00700 1 -- 172.21.15.27:0/877515535 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fbbe80a5f70 con 0x56377fe52d00 2024-05-10T11:21:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.851+0000 7fbc073d6b80 10 client.16245 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:21:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.851+0000 7fbc073d6b80 10 client.16245 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:21:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.851+0000 7fbc073d6b80 20 client.16245 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.851+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56377fed7510 0x56377fed9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.851+0000 7fbc073d6b80 4 client.16245 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:21:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.851+0000 7fbc073d6b80 1 -- 172.21.15.27:0/877515535 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56377fdae400 con 0x56377fed7510 2024-05-10T11:21:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.851+0000 7fbbf5400700 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56377fed7510 0x56377fed9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbbf5400700 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56377fed7510 0x56377fed9900 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4fb9347a51653349 server_cookie=2d0dad5d05ab6565 in_seq=0 out_seq=0 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbbd7e00700 10 client.16245 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbbd7e00700 1 -- 172.21.15.27:0/877515535 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56377fdae400 con 0x56377fed7510 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbbd7e00700 10 client.16245 handle_command_reply: tid=0 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbc073d6b80 1 client.16245 shutdown 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbc073d6b80 1 -- 172.21.15.27:0/877515535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe52d00 msgr2=0x56377fe64740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe52d00 0x56377fe64740 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80b8040 tx=0x7fbbe80b31d0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbc073d6b80 1 -- 172.21.15.27:0/877515535 shutdown_connections 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56377fed7510 0x56377fed9900 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56377fe588f0 0x56377fe61710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56377fe52d00 0x56377fe64740 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbc073d6b80 1 --2- 172.21.15.27:0/877515535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377fe57fd0 0x56377fe64c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbbd7e00700 0 client.16245 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.852+0000 7fbc073d6b80 1 -- 172.21.15.27:0/877515535 >> 172.21.15.27:0/877515535 conn(0x56377fdc66d0 msgr2=0x56377fe5cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.853+0000 7fbc073d6b80 1 -- 172.21.15.27:0/877515535 shutdown_connections 2024-05-10T11:21:16.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.853+0000 7fbc073d6b80 1 -- 172.21.15.27:0/877515535 wait complete. 2024-05-10T11:21:16.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.853+0000 7fbc073d6b80 20 client.16245 trim_cache size 0 max 16384 2024-05-10T11:21:16.856 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:21:16.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.853+0000 7fbbfca00700 1 -- 172.21.15.27:0/853506823 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbbdc0b29f0 msgr2=0x7fbbdc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.853+0000 7fbbfca00700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbbdc0b29f0 0x7fbbdc0b4eb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009ec0 tx=0x7fbbf0009040 comp rx=0 tx=0).stop 2024-05-10T11:21:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.853+0000 7fbbfca00700 1 -- 172.21.15.27:0/853506823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbf806ba10 msgr2=0x7fbbf819f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.853+0000 7fbbfca00700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbf806ba10 0x7fbbf819f790 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8000c00 tx=0x7fbbe800bfd0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.854+0000 7fbbfca00700 1 -- 172.21.15.27:0/853506823 shutdown_connections 2024-05-10T11:21:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.854+0000 7fbbfca00700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbbdc0b29f0 0x7fbbdc0b4eb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.854+0000 7fbbfca00700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbf8109500 0x7fbbf8199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.854+0000 7fbbfca00700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbf806ba10 0x7fbbf819f790 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.854+0000 7fbbfca00700 1 --2- 172.21.15.27:0/853506823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf806c3d0 0x7fbbf819fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.854+0000 7fbbfca00700 1 -- 172.21.15.27:0/853506823 >> 172.21.15.27:0/853506823 conn(0x7fbbf8079310 msgr2=0x7fbbf8101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.854+0000 7fbbfca00700 1 -- 172.21.15.27:0/853506823 shutdown_connections 2024-05-10T11:21:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:16.854+0000 7fbbfca00700 1 -- 172.21.15.27:0/853506823 wait complete. 2024-05-10T11:21:17.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:17 smithi027 ceph-mon[30054]: pgmap v248: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-05-10T11:21:17.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:17 smithi076 ceph-mon[34152]: pgmap v248: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-05-10T11:21:17.702 INFO:tasks.workunit.client.0.smithi027.stdout: 1 461619 6.62 MB/sec execute 242 sec latency 2744.584 ms 2024-05-10T11:21:17.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:17 smithi189 ceph-mon[34842]: pgmap v248: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-05-10T11:21:17.789 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:21:18.267 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:21:18.703 INFO:tasks.workunit.client.0.smithi027.stdout: 1 461619 6.60 MB/sec execute 243 sec latency 3744.726 ms 2024-05-10T11:21:19.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:19 smithi027 ceph-mon[30054]: pgmap v249: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-05-10T11:21:19.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:19 smithi076 ceph-mon[34152]: pgmap v249: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-05-10T11:21:19.703 INFO:tasks.workunit.client.0.smithi027.stdout: 1 462129 6.59 MB/sec execute 244 sec latency 4223.630 ms 2024-05-10T11:21:19.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:19 smithi189 ceph-mon[34842]: pgmap v249: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 1.6 MiB/s wr, 113 op/s 2024-05-10T11:21:20.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.699+0000 7ff776400700 1 -- 172.21.15.27:0/1781290988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff770103750 msgr2=0x7ff770103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:20.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.699+0000 7ff776400700 1 --2- 172.21.15.27:0/1781290988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff770103750 0x7ff770103b30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff7600099c0 tx=0x7ff760009cd0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.700+0000 7ff776400700 1 -- 172.21.15.27:0/1781290988 shutdown_connections 2024-05-10T11:21:20.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.700+0000 7ff776400700 1 --2- 172.21.15.27:0/1781290988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff770108850 0x7ff7701103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.700+0000 7ff776400700 1 --2- 172.21.15.27:0/1781290988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff770103750 0x7ff770103b30 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.700+0000 7ff776400700 1 --2- 172.21.15.27:0/1781290988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff770104100 0x7ff770108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.700+0000 7ff776400700 1 -- 172.21.15.27:0/1781290988 >> 172.21.15.27:0/1781290988 conn(0x7ff77007a600 msgr2=0x7ff77007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:20.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.700+0000 7ff776400700 1 -- 172.21.15.27:0/1781290988 shutdown_connections 2024-05-10T11:21:20.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.700+0000 7ff776400700 1 -- 172.21.15.27:0/1781290988 wait complete. 2024-05-10T11:21:20.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.701+0000 7ff776400700 1 Processor -- start 2024-05-10T11:21:20.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.701+0000 7ff776400700 1 -- start start 2024-05-10T11:21:20.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff776400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff770103750 0x7ff7700757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff776400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff770104100 0x7ff770075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff776400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff770108850 0x7ff770076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff776400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff770113690 con 0x7ff770104100 2024-05-10T11:21:20.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff776400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff770113510 con 0x7ff770108850 2024-05-10T11:21:20.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff776400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff770113810 con 0x7ff770103750 2024-05-10T11:21:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff76f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff770103750 0x7ff7700757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff76f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff770103750 0x7ff7700757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:32962/0 (socket says 172.21.15.27:32962) 2024-05-10T11:21:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff76f400700 1 -- 172.21.15.27:0/3237009597 learned_addr learned my addr 172.21.15.27:0/3237009597 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff76fe00700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff770108850 0x7ff770076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff767e00700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff770104100 0x7ff770075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.704 INFO:tasks.workunit.client.0.smithi027.stdout: 1 463011 6.57 MB/sec execute 245 sec latency 40.355 ms 2024-05-10T11:21:20.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff76f400700 1 -- 172.21.15.27:0/3237009597 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff770108850 msgr2=0x7ff770076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:20.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff76f400700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff770108850 0x7ff770076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff76f400700 1 -- 172.21.15.27:0/3237009597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff770104100 msgr2=0x7ff770075ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.702+0000 7ff76f400700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff770104100 0x7ff770075ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.703+0000 7ff76f400700 1 -- 172.21.15.27:0/3237009597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7701a8070 con 0x7ff770103750 2024-05-10T11:21:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.703+0000 7ff76fe00700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff770108850 0x7ff770076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:21:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.703+0000 7ff76f400700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff770103750 0x7ff7700757a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff760006010 tx=0x7ff7600055f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.703+0000 7ff76cc00700 1 -- 172.21.15.27:0/3237009597 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760026070 con 0x7ff770103750 2024-05-10T11:21:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.703+0000 7ff76cc00700 1 -- 172.21.15.27:0/3237009597 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff76000f460 con 0x7ff770103750 2024-05-10T11:21:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.703+0000 7ff776400700 1 -- 172.21.15.27:0/3237009597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7701a8300 con 0x7ff770103750 2024-05-10T11:21:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.703+0000 7ff76cc00700 1 -- 172.21.15.27:0/3237009597 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760009e60 con 0x7ff770103750 2024-05-10T11:21:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.704+0000 7ff776400700 1 -- 172.21.15.27:0/3237009597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7701a87b0 con 0x7ff770103750 2024-05-10T11:21:20.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.708+0000 7ff76cc00700 1 -- 172.21.15.27:0/3237009597 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff760029070 con 0x7ff770103750 2024-05-10T11:21:20.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.709+0000 7ff780dbcb80 1 Processor -- start 2024-05-10T11:21:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.709+0000 7ff780dbcb80 1 -- start start 2024-05-10T11:21:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.710+0000 7ff780dbcb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b08e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.710+0000 7ff780dbcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b092490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.710+0000 7ff780dbcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b092e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.710+0000 7ff780dbcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55626afe0a20 con 0x55626b08e980 2024-05-10T11:21:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.710+0000 7ff780dbcb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55626af7e760 con 0x55626b08e060 2024-05-10T11:21:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.710+0000 7ff780dbcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55626afdc9e0 con 0x55626b0929d0 2024-05-10T11:21:20.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.710+0000 7ff76fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b08e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.710+0000 7ff76fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b08e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42114/0 (socket says 172.21.15.27:42114) 2024-05-10T11:21:20.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.710+0000 7ff76fe00700 1 -- 172.21.15.27:0/2444475633 learned_addr learned my addr 172.21.15.27:0/2444475633 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:20.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.710+0000 7ff767e00700 1 --2- 172.21.15.27:0/2444475633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b092e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.710+0000 7ff76f400700 1 --2- 172.21.15.27:0/2444475633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b092490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.710+0000 7ff76cc00700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff7540b2a70 0x7ff7540b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.711+0000 7ff767e00700 1 -- 172.21.15.27:0/2444475633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 msgr2=0x55626b08e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.711+0000 7ff767e00700 1 --2- 172.21.15.27:0/2444475633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b08e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.711+0000 7ff767e00700 1 -- 172.21.15.27:0/2444475633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 msgr2=0x55626b092490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.711+0000 7ff767e00700 1 --2- 172.21.15.27:0/2444475633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b092490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.711+0000 7ff767e00700 1 -- 172.21.15.27:0/2444475633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55626af2d8d0 con 0x55626b0929d0 2024-05-10T11:21:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.711+0000 7ff767e00700 1 --2- 172.21.15.27:0/2444475633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b092e30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff770077230 tx=0x7ff758009f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf21d00d8d401f4b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.711+0000 7ff76fe00700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff7540b2a70 0x7ff7540b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.711+0000 7ff764c00700 1 -- 172.21.15.27:0/2444475633 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff758011cb0 con 0x55626b0929d0 2024-05-10T11:21:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.711+0000 7ff764c00700 1 -- 172.21.15.27:0/2444475633 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff758011e50 con 0x55626b0929d0 2024-05-10T11:21:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.712+0000 7ff764c00700 1 -- 172.21.15.27:0/2444475633 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff758005280 con 0x55626b0929d0 2024-05-10T11:21:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.712+0000 7ff76cc00700 1 -- 172.21.15.27:0/3237009597 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7ff7600e31f0 con 0x7ff770103750 2024-05-10T11:21:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.712+0000 7ff780dbcb80 1 -- 172.21.15.27:0/2444475633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 msgr2=0x55626b092e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.712+0000 7ff76fe00700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff7540b2a70 0x7ff7540b4f30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff768009fd0 tx=0x7ff76800b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.712+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/2444475633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b092e30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff770077230 tx=0x7ff758009f40 comp rx=0 tx=0).stop 2024-05-10T11:21:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.713+0000 7ff780dbcb80 1 -- 172.21.15.27:0/2444475633 shutdown_connections 2024-05-10T11:21:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.713+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/2444475633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b092e30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.713+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/2444475633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b08e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.713+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/2444475633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b092490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.713+0000 7ff780dbcb80 1 -- 172.21.15.27:0/2444475633 >> 172.21.15.27:0/2444475633 conn(0x55626afbcb60 msgr2=0x55626b0887a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.713+0000 7ff780dbcb80 1 -- 172.21.15.27:0/2444475633 shutdown_connections 2024-05-10T11:21:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.713+0000 7ff780dbcb80 1 -- 172.21.15.27:0/2444475633 wait complete. 2024-05-10T11:21:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.713+0000 7ff780dbcb80 1 -- start start 2024-05-10T11:21:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.714+0000 7ff780dbcb80 1 Processor -- start 2024-05-10T11:21:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.714+0000 7ff780dbcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e060 0x55626b099ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.714+0000 7ff780dbcb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e980 0x55626b097280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.714+0000 7ff780dbcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.714+0000 7ff780dbcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55626b09abe0 con 0x55626b08e060 2024-05-10T11:21:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.714+0000 7ff780dbcb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55626af46b20 con 0x55626b08e980 2024-05-10T11:21:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.714+0000 7ff780dbcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55626b09ad60 con 0x55626b0929d0 2024-05-10T11:21:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.715+0000 7ff780dbcb80 10 client.0 _ll_register_callbacks cb 0x55626af87c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.715+0000 7ff767e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e980 0x55626b097280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.715+0000 7ff767e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e980 0x55626b097280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42130/0 (socket says 172.21.15.27:42130) 2024-05-10T11:21:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.715+0000 7ff767e00700 1 -- 172.21.15.27:0/4159231227 learned_addr learned my addr 172.21.15.27:0/4159231227 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.715+0000 7ff76f400700 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e060 0x55626b099ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.715+0000 7ff76fe00700 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.715+0000 7ff767e00700 1 -- 172.21.15.27:0/4159231227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 msgr2=0x55626b0977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.715+0000 7ff767e00700 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.715+0000 7ff767e00700 1 -- 172.21.15.27:0/4159231227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e060 msgr2=0x55626b099ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.715+0000 7ff767e00700 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e060 0x55626b099ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.715+0000 7ff767e00700 1 -- 172.21.15.27:0/4159231227 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55626afa6420 con 0x55626b08e980 2024-05-10T11:21:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.715+0000 7ff76fe00700 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:21:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.716+0000 7ff767e00700 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e980 0x55626b097280 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff758004de0 tx=0x7ff758005d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b7cd128d120243 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.716+0000 7ff75d600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:21:20.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.716+0000 7ff75d600700 1 -- 172.21.15.27:0/4159231227 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff758026070 con 0x55626b08e980 2024-05-10T11:21:20.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.716+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4159231227 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff758011e50 con 0x55626b08e980 2024-05-10T11:21:20.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.716+0000 7ff75d600700 1 -- 172.21.15.27:0/4159231227 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7ff758005020 con 0x55626b08e980 2024-05-10T11:21:20.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.716+0000 7ff75d600700 1 -- 172.21.15.27:0/4159231227 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff758032050 con 0x55626b08e980 2024-05-10T11:21:20.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.716+0000 7ff780dbcb80 10 client.26474 fetch_fsmap learned FSMap version 34 2024-05-10T11:21:20.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.716+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4159231227 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55626b0a5540 con 0x55626b08e980 2024-05-10T11:21:20.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.717+0000 7ff75d600700 1 -- 172.21.15.27:0/4159231227 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff758011e50 con 0x55626b08e980 2024-05-10T11:21:20.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.717+0000 7ff75d600700 1 -- 172.21.15.27:0/4159231227 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7ff758018070 con 0x55626b08e980 2024-05-10T11:21:20.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.717+0000 7ff780dbcb80 10 client.26474 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:21:20.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.717+0000 7ff780dbcb80 10 client.26474 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.717+0000 7ff780dbcb80 20 client.26474 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.717+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55626b0a7a70 0x55626b0a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.717+0000 7ff780dbcb80 4 client.26474 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.717+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4159231227 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55626b05b0a0 con 0x55626b0a7a70 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.717+0000 7ff76f400700 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55626b0a7a70 0x55626b0a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.718+0000 7ff76f400700 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55626b0a7a70 0x55626b0a9e60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c2b99469c09a3b39 server_cookie=265015078de99544 in_seq=0 out_seq=0 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.718+0000 7ff75d600700 10 client.26474 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff75d600700 1 -- 172.21.15.27:0/4159231227 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55626b05b0a0 con 0x55626b0a7a70 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff75d600700 10 client.26474 handle_command_reply: tid=0 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff780dbcb80 1 client.26474 shutdown 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4159231227 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e980 msgr2=0x55626b097280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e980 0x55626b097280 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff758004de0 tx=0x7ff758005d70 comp rx=0 tx=0).stop 2024-05-10T11:21:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4159231227 shutdown_connections 2024-05-10T11:21:20.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55626b0a7a70 0x55626b0a9e60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e060 0x55626b099ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/4159231227 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e980 0x55626b097280 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff75d600700 0 client.26474 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:21:20.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4159231227 >> 172.21.15.27:0/4159231227 conn(0x55626afbcb60 msgr2=0x55626b0832b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:20.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.719+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4159231227 shutdown_connections 2024-05-10T11:21:20.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.720+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4159231227 wait complete. 2024-05-10T11:21:20.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.720+0000 7ff780dbcb80 20 client.26474 trim_cache size 0 max 16384 2024-05-10T11:21:20.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.737+0000 7ff780dbcb80 1 Processor -- start 2024-05-10T11:21:20.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.737+0000 7ff780dbcb80 1 -- start start 2024-05-10T11:21:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff780dbcb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b094ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff780dbcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b097950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff780dbcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff780dbcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff7701b1850 con 0x55626b08e980 2024-05-10T11:21:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff780dbcb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff7701b33c0 con 0x55626b08e060 2024-05-10T11:21:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff780dbcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7701b1f20 con 0x55626b0929d0 2024-05-10T11:21:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff76f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b094ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff76f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b094ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42144/0 (socket says 172.21.15.27:42144) 2024-05-10T11:21:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff76f400700 1 -- 172.21.15.27:0/655152162 learned_addr learned my addr 172.21.15.27:0/655152162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff767e00700 1 --2- 172.21.15.27:0/655152162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b097950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff76fe00700 1 --2- 172.21.15.27:0/655152162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff76f400700 1 -- 172.21.15.27:0/655152162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 msgr2=0x55626b0a3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:20.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff76f400700 1 --2- 172.21.15.27:0/655152162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0a3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff76f400700 1 -- 172.21.15.27:0/655152162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 msgr2=0x55626b097950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff76f400700 1 --2- 172.21.15.27:0/655152162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b097950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff76f400700 1 -- 172.21.15.27:0/655152162 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55626b097050 con 0x55626b08e060 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.738+0000 7ff76fe00700 1 --2- 172.21.15.27:0/655152162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.739+0000 7ff76f400700 1 --2- 172.21.15.27:0/655152162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b094ee0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff760005b40 tx=0x7ff7600e3470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2462c714106d7067 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.739+0000 7ff75fe00700 1 -- 172.21.15.27:0/655152162 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7600a9ea0 con 0x55626b08e060 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.739+0000 7ff75fe00700 1 -- 172.21.15.27:0/655152162 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7600a9be0 con 0x55626b08e060 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.739+0000 7ff75fe00700 1 -- 172.21.15.27:0/655152162 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7600a9840 con 0x55626b08e060 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.740+0000 7ff780dbcb80 1 -- 172.21.15.27:0/655152162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 msgr2=0x55626b094ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.740+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/655152162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b094ee0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff760005b40 tx=0x7ff7600e3470 comp rx=0 tx=0).stop 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.740+0000 7ff780dbcb80 1 -- 172.21.15.27:0/655152162 shutdown_connections 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.740+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/655152162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.740+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/655152162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b094ee0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.740+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/655152162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b097950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.740+0000 7ff780dbcb80 1 -- 172.21.15.27:0/655152162 >> 172.21.15.27:0/655152162 conn(0x55626afbcb60 msgr2=0x55626b097240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.740+0000 7ff780dbcb80 1 -- 172.21.15.27:0/655152162 shutdown_connections 2024-05-10T11:21:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.740+0000 7ff780dbcb80 1 -- 172.21.15.27:0/655152162 wait complete. 2024-05-10T11:21:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.741+0000 7ff780dbcb80 1 -- start start 2024-05-10T11:21:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.741+0000 7ff780dbcb80 1 Processor -- start 2024-05-10T11:21:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.741+0000 7ff780dbcb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b0a0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff780dbcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b0a0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff780dbcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff780dbcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff7701b2cf0 con 0x55626b08e980 2024-05-10T11:21:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff780dbcb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff7701b2620 con 0x55626b08e060 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff780dbcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7701af810 con 0x55626b0929d0 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff780dbcb80 10 client.0 _ll_register_callbacks cb 0x55626af87c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff76f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b0a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff76f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b0a0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42150/0 (socket says 172.21.15.27:42150) 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff76f400700 1 -- 172.21.15.27:0/4029340952 learned_addr learned my addr 172.21.15.27:0/4029340952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff76fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff767e00700 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b0a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff76f400700 1 -- 172.21.15.27:0/4029340952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 msgr2=0x55626b0aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff76f400700 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0aebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff76f400700 1 -- 172.21.15.27:0/4029340952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 msgr2=0x55626b0a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff76f400700 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b0a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff76f400700 1 -- 172.21.15.27:0/4029340952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55626b0af420 con 0x55626b08e060 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff76fe00700 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:21:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.742+0000 7ff767e00700 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b0a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:21:20.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.743+0000 7ff76f400700 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b0a0230 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff7600a6150 tx=0x7ff7600a9fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4928f8c49d25c7f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:20.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.743+0000 7ff75d600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:21:20.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.743+0000 7ff75d600700 1 -- 172.21.15.27:0/4029340952 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7600a9a20 con 0x55626b08e060 2024-05-10T11:21:20.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.743+0000 7ff75d600700 1 -- 172.21.15.27:0/4029340952 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7ff7600ae120 con 0x55626b08e060 2024-05-10T11:21:20.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.743+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4029340952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7600a9be0 con 0x55626b08e060 2024-05-10T11:21:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.743+0000 7ff75d600700 1 -- 172.21.15.27:0/4029340952 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff7600aa7d0 con 0x55626b08e060 2024-05-10T11:21:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.743+0000 7ff780dbcb80 10 client.26480 fetch_fsmap learned FSMap version 34 2024-05-10T11:21:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.743+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4029340952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55626b111e50 con 0x55626b08e060 2024-05-10T11:21:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.744+0000 7ff75d600700 1 -- 172.21.15.27:0/4029340952 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7600a9be0 con 0x55626b08e060 2024-05-10T11:21:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.744+0000 7ff75d600700 1 -- 172.21.15.27:0/4029340952 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7ff7600cc040 con 0x55626b08e060 2024-05-10T11:21:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.744+0000 7ff780dbcb80 10 client.26480 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:21:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.744+0000 7ff780dbcb80 10 client.26480 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:21:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.744+0000 7ff780dbcb80 20 client.26480 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.744+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55626b1123d0 0x55626b1147f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.744+0000 7ff780dbcb80 4 client.26480 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:21:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.744+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4029340952 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55626afee400 con 0x55626b1123d0 2024-05-10T11:21:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.744+0000 7ff767e00700 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55626b1123d0 0x55626b1147f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff767e00700 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55626b1123d0 0x55626b1147f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f0cb13db85fc07d server_cookie=5e0a1432d74156cd in_seq=0 out_seq=0 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff75d600700 10 client.26480 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff75d600700 1 -- 172.21.15.27:0/4029340952 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55626afee400 con 0x55626b1123d0 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff75d600700 10 client.26480 handle_command_reply: tid=0 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff780dbcb80 1 client.26480 shutdown 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4029340952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 msgr2=0x55626b0a0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b0a0230 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff7600a6150 tx=0x7ff7600a9fd0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4029340952 shutdown_connections 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55626b1123d0 0x55626b1147f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff75d600700 0 client.26480 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55626b0929d0 0x55626b0aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55626b08e060 0x55626b0a0230 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff780dbcb80 1 --2- 172.21.15.27:0/4029340952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55626b08e980 0x55626b0a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.745+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4029340952 >> 172.21.15.27:0/4029340952 conn(0x55626afbcb60 msgr2=0x55626b082a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:20.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.746+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4029340952 shutdown_connections 2024-05-10T11:21:20.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.746+0000 7ff780dbcb80 1 -- 172.21.15.27:0/4029340952 wait complete. 2024-05-10T11:21:20.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.746+0000 7ff780dbcb80 20 client.26480 trim_cache size 0 max 16384 2024-05-10T11:21:20.749 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:21:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.747+0000 7ff776400700 1 -- 172.21.15.27:0/3237009597 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff7540b2a70 msgr2=0x7ff7540b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.747+0000 7ff776400700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff7540b2a70 0x7ff7540b4f30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff768009fd0 tx=0x7ff76800b040 comp rx=0 tx=0).stop 2024-05-10T11:21:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.747+0000 7ff776400700 1 -- 172.21.15.27:0/3237009597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff770103750 msgr2=0x7ff7700757a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.747+0000 7ff776400700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff770103750 0x7ff7700757a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff760006010 tx=0x7ff7600055f0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.747+0000 7ff776400700 1 -- 172.21.15.27:0/3237009597 shutdown_connections 2024-05-10T11:21:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.747+0000 7ff776400700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff7540b2a70 0x7ff7540b4f30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.747+0000 7ff776400700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff770103750 0x7ff7700757a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.747+0000 7ff776400700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff770108850 0x7ff770076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.747+0000 7ff776400700 1 --2- 172.21.15.27:0/3237009597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff770104100 0x7ff770075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.747+0000 7ff776400700 1 -- 172.21.15.27:0/3237009597 >> 172.21.15.27:0/3237009597 conn(0x7ff77007a600 msgr2=0x7ff7701016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.747+0000 7ff776400700 1 -- 172.21.15.27:0/3237009597 shutdown_connections 2024-05-10T11:21:20.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:20.747+0000 7ff776400700 1 -- 172.21.15.27:0/3237009597 wait complete. 2024-05-10T11:21:21.540 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:21 smithi027 ceph-mon[30054]: pgmap v250: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 1.9 MiB/s wr, 136 op/s 2024-05-10T11:21:21.610 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:21:21.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:21 smithi076 ceph-mon[34152]: pgmap v250: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 1.9 MiB/s wr, 136 op/s 2024-05-10T11:21:21.703 INFO:tasks.workunit.client.0.smithi027.stdout: 1 464351 6.57 MB/sec execute 246 sec latency 52.843 ms 2024-05-10T11:21:21.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:21 smithi189 ceph-mon[34842]: pgmap v250: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 1.9 MiB/s wr, 136 op/s 2024-05-10T11:21:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:21:22.703 INFO:tasks.workunit.client.0.smithi027.stdout: 1 465815 6.59 MB/sec execute 247 sec latency 19.617 ms 2024-05-10T11:21:23.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:23 smithi027 ceph-mon[30054]: pgmap v251: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2024-05-10T11:21:23.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:23 smithi076 ceph-mon[34152]: pgmap v251: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2024-05-10T11:21:23.703 INFO:tasks.workunit.client.0.smithi027.stdout: 1 466856 6.58 MB/sec execute 248 sec latency 35.402 ms 2024-05-10T11:21:23.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:23 smithi189 ceph-mon[34842]: pgmap v251: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2024-05-10T11:21:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.659+0000 7f0f45200700 1 -- 172.21.15.27:0/1567539428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f401055d0 msgr2=0x7f0f40105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.659+0000 7f0f45200700 1 --2- 172.21.15.27:0/1567539428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f401055d0 0x7f0f40105a50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009990 tx=0x7f0f2c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.659+0000 7f0f45200700 1 -- 172.21.15.27:0/1567539428 shutdown_connections 2024-05-10T11:21:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.659+0000 7f0f45200700 1 --2- 172.21.15.27:0/1567539428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f40105f90 0x7f0f401124a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.659+0000 7f0f45200700 1 --2- 172.21.15.27:0/1567539428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f4010b660 0x7f0f4010ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.659+0000 7f0f45200700 1 --2- 172.21.15.27:0/1567539428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f401055d0 0x7f0f40105a50 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.659+0000 7f0f45200700 1 -- 172.21.15.27:0/1567539428 >> 172.21.15.27:0/1567539428 conn(0x7f0f40101080 msgr2=0x7f0f401034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.659+0000 7f0f45200700 1 -- 172.21.15.27:0/1567539428 shutdown_connections 2024-05-10T11:21:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.659+0000 7f0f45200700 1 -- 172.21.15.27:0/1567539428 wait complete. 2024-05-10T11:21:24.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.661+0000 7f0f45200700 1 Processor -- start 2024-05-10T11:21:24.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.661+0000 7f0f45200700 1 -- start start 2024-05-10T11:21:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.661+0000 7f0f45200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f401055d0 0x7f0f401a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.661+0000 7f0f45200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f40105f90 0x7f0f401a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.661+0000 7f0f45200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f4010b660 0x7f0f4019ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.661+0000 7f0f45200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0f40115800 con 0x7f0f401055d0 2024-05-10T11:21:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.661+0000 7f0f45200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0f40115680 con 0x7f0f4010b660 2024-05-10T11:21:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.661+0000 7f0f45200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0f40115980 con 0x7f0f40105f90 2024-05-10T11:21:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.662+0000 7f0f3fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f4010b660 0x7f0f4019ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.662+0000 7f0f3fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f4010b660 0x7f0f4019ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36148/0 (socket says 172.21.15.27:36148) 2024-05-10T11:21:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.662+0000 7f0f3fe00700 1 -- 172.21.15.27:0/29476705 learned_addr learned my addr 172.21.15.27:0/29476705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.662+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f40105f90 0x7f0f401a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.662+0000 7f0f3f400700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f401055d0 0x7f0f401a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.662+0000 7f0f3fe00700 1 -- 172.21.15.27:0/29476705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f40105f90 msgr2=0x7f0f401a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:24.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.662+0000 7f0f3fe00700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f40105f90 0x7f0f401a1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.662+0000 7f0f3fe00700 1 -- 172.21.15.27:0/29476705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f401055d0 msgr2=0x7f0f401a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:24.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.662+0000 7f0f3fe00700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f401055d0 0x7f0f401a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.662+0000 7f0f3fe00700 1 -- 172.21.15.27:0/29476705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f4019c1e0 con 0x7f0f4010b660 2024-05-10T11:21:24.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.662+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f40105f90 0x7f0f401a1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:21:24.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.662+0000 7f0f3fe00700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f4010b660 0x7f0f4019ba80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0f3800c370 tx=0x7f0f3800c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:24.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.663+0000 7f0f37e00700 1 -- 172.21.15.27:0/29476705 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f380092e0 con 0x7f0f4010b660 2024-05-10T11:21:24.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.663+0000 7f0f37e00700 1 -- 172.21.15.27:0/29476705 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f38003a00 con 0x7f0f4010b660 2024-05-10T11:21:24.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.663+0000 7f0f37e00700 1 -- 172.21.15.27:0/29476705 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f380046f0 con 0x7f0f4010b660 2024-05-10T11:21:24.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.663+0000 7f0f45200700 1 -- 172.21.15.27:0/29476705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f4019c470 con 0x7f0f4010b660 2024-05-10T11:21:24.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.663+0000 7f0f45200700 1 -- 172.21.15.27:0/29476705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f401a8790 con 0x7f0f4010b660 2024-05-10T11:21:24.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.667+0000 7f0f37e00700 1 -- 172.21.15.27:0/29476705 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0f3801a020 con 0x7f0f4010b660 2024-05-10T11:21:24.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.669+0000 7f0f51237b80 1 Processor -- start 2024-05-10T11:21:24.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.669+0000 7f0f51237b80 1 -- start start 2024-05-10T11:21:24.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.669+0000 7f0f51237b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b19e0 0x56380a0b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.669+0000 7f0f51237b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b2300 0x56380a0abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.669+0000 7f0f51237b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 0x56380a0aca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.669+0000 7f0f51237b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563809ffea20 con 0x56380a0b19e0 2024-05-10T11:21:24.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.669+0000 7f0f51237b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563809f9c760 con 0x56380a0b2300 2024-05-10T11:21:24.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.669+0000 7f0f51237b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563809ffa9e0 con 0x56380a0ac600 2024-05-10T11:21:24.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.670+0000 7f0f37e00700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0f240b2a70 0x7f0f240b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.670+0000 7f0f3ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b19e0 0x56380a0b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.670+0000 7f0f37e00700 1 -- 172.21.15.27:0/29476705 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7f0f380e22d0 con 0x7f0f4010b660 2024-05-10T11:21:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.670+0000 7f0f3f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b2300 0x56380a0abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.670+0000 7f0f3ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b19e0 0x56380a0b1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51224/0 (socket says 172.21.15.27:51224) 2024-05-10T11:21:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.670+0000 7f0f3ea00700 1 -- 172.21.15.27:0/1528206170 learned_addr learned my addr 172.21.15.27:0/1528206170 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.670+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/1528206170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 0x56380a0aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.671+0000 7f0f3fe00700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0f240b2a70 0x7f0f240b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.671+0000 7f0f3ea00700 1 -- 172.21.15.27:0/1528206170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 msgr2=0x56380a0aca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.671+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/1528206170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 0x56380a0aca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.671+0000 7f0f3ea00700 1 -- 172.21.15.27:0/1528206170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b2300 msgr2=0x56380a0abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.671+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/1528206170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b2300 0x56380a0abfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.671+0000 7f0f3ea00700 1 -- 172.21.15.27:0/1528206170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563809f4b8d0 con 0x56380a0b19e0 2024-05-10T11:21:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.671+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/1528206170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b19e0 0x56380a0b1dc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009990 tx=0x7f0f2c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b509b1c14ac62d53 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.672+0000 7f0f34c00700 1 -- 172.21.15.27:0/1528206170 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c005380 con 0x56380a0b19e0 2024-05-10T11:21:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.672+0000 7f0f3fe00700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0f240b2a70 0x7f0f240b4f30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0f3800c340 tx=0x7f0f38009630 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.672+0000 7f0f34c00700 1 -- 172.21.15.27:0/1528206170 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f2c022040 con 0x56380a0b19e0 2024-05-10T11:21:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.672+0000 7f0f34c00700 1 -- 172.21.15.27:0/1528206170 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c00f6c0 con 0x56380a0b19e0 2024-05-10T11:21:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.673+0000 7f0f51237b80 1 -- 172.21.15.27:0/1528206170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b19e0 msgr2=0x56380a0b1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.673+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1528206170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b19e0 0x56380a0b1dc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009990 tx=0x7f0f2c005fd0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.673+0000 7f0f51237b80 1 -- 172.21.15.27:0/1528206170 shutdown_connections 2024-05-10T11:21:24.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.673+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1528206170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 0x56380a0aca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.673+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1528206170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b19e0 0x56380a0b1dc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.673+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1528206170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b2300 0x56380a0abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.673+0000 7f0f51237b80 1 -- 172.21.15.27:0/1528206170 >> 172.21.15.27:0/1528206170 conn(0x563809fdab60 msgr2=0x56380a0a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:24.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.674+0000 7f0f51237b80 1 -- 172.21.15.27:0/1528206170 shutdown_connections 2024-05-10T11:21:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.674+0000 7f0f51237b80 1 -- 172.21.15.27:0/1528206170 wait complete. 2024-05-10T11:21:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.675+0000 7f0f51237b80 1 -- start start 2024-05-10T11:21:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.675+0000 7f0f51237b80 1 Processor -- start 2024-05-10T11:21:24.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.675+0000 7f0f51237b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 0x56380a0b7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.675+0000 7f0f51237b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 0x56380a0b5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.675+0000 7f0f51237b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b2300 0x56380a0b57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.675+0000 7f0f51237b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56380a0b8d50 con 0x56380a0b2300 2024-05-10T11:21:24.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.675+0000 7f0f51237b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563809f64b20 con 0x56380a0b19e0 2024-05-10T11:21:24.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.675+0000 7f0f51237b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56380a0b8ed0 con 0x56380a0ac600 2024-05-10T11:21:24.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.676+0000 7f0f51237b80 10 client.0 _ll_register_callbacks cb 0x563809fa5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:24.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.676+0000 7f0f3f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 0x56380a0b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.676+0000 7f0f3ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 0x56380a0b7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.676+0000 7f0f3ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b2300 0x56380a0b57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.676+0000 7f0f3f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 0x56380a0b5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36162/0 (socket says 172.21.15.27:36162) 2024-05-10T11:21:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.676+0000 7f0f3f400700 1 -- 172.21.15.27:0/1993703450 learned_addr learned my addr 172.21.15.27:0/1993703450 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.676+0000 7f0f3f400700 1 -- 172.21.15.27:0/1993703450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 msgr2=0x56380a0b7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.676+0000 7f0f3f400700 1 --2- 172.21.15.27:0/1993703450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 0x56380a0b7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.676+0000 7f0f3f400700 1 -- 172.21.15.27:0/1993703450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b2300 msgr2=0x56380a0b57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.676+0000 7f0f3f400700 1 --2- 172.21.15.27:0/1993703450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b2300 0x56380a0b57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.676+0000 7f0f3f400700 1 -- 172.21.15.27:0/1993703450 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563809fc4420 con 0x56380a0b19e0 2024-05-10T11:21:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.677+0000 7f0f3f400700 1 --2- 172.21.15.27:0/1993703450 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 0x56380a0b5290 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0f30012840 tx=0x7f0f30012b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37645a995d5f78fd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.677+0000 7f0f29600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:21:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.677+0000 7f0f29600700 1 -- 172.21.15.27:0/1993703450 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3000f4e0 con 0x56380a0b19e0 2024-05-10T11:21:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.677+0000 7f0f51237b80 1 -- 172.21.15.27:0/1993703450 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f2c022040 con 0x56380a0b19e0 2024-05-10T11:21:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.677+0000 7f0f29600700 1 -- 172.21.15.27:0/1993703450 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7f0f3000f680 con 0x56380a0b19e0 2024-05-10T11:21:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.677+0000 7f0f29600700 1 -- 172.21.15.27:0/1993703450 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0f3000f880 con 0x56380a0b19e0 2024-05-10T11:21:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.677+0000 7f0f51237b80 10 client.26492 fetch_fsmap learned FSMap version 34 2024-05-10T11:21:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.677+0000 7f0f51237b80 1 -- 172.21.15.27:0/1993703450 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56380a0cbcf0 con 0x56380a0b19e0 2024-05-10T11:21:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.678+0000 7f0f29600700 1 -- 172.21.15.27:0/1993703450 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f2c022040 con 0x56380a0b19e0 2024-05-10T11:21:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.678+0000 7f0f29600700 1 -- 172.21.15.27:0/1993703450 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f0f300109a0 con 0x56380a0b19e0 2024-05-10T11:21:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.678+0000 7f0f51237b80 10 client.26492 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:21:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.678+0000 7f0f51237b80 10 client.26492 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:21:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.678+0000 7f0f51237b80 20 client.26492 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.678+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1993703450 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56380a0ce380 0x56380a0d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.678+0000 7f0f51237b80 4 client.26492 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:21:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.678+0000 7f0f51237b80 1 -- 172.21.15.27:0/1993703450 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56380a0790a0 con 0x56380a0ce380 2024-05-10T11:21:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.679+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/1993703450 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56380a0ce380 0x56380a0d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.679+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/1993703450 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56380a0ce380 0x56380a0d0770 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=21fc45806a7d3d28 server_cookie=fcb4dd65999706d6 in_seq=0 out_seq=0 2024-05-10T11:21:24.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.679+0000 7f0f29600700 10 client.26492 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:21:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.680+0000 7f0f29600700 1 -- 172.21.15.27:0/1993703450 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56380a0790a0 con 0x56380a0ce380 2024-05-10T11:21:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.680+0000 7f0f29600700 10 client.26492 handle_command_reply: tid=0 2024-05-10T11:21:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.680+0000 7f0f51237b80 1 client.26492 shutdown 2024-05-10T11:21:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.680+0000 7f0f51237b80 1 -- 172.21.15.27:0/1993703450 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 msgr2=0x56380a0b5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.680+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1993703450 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 0x56380a0b5290 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0f30012840 tx=0x7f0f30012b50 comp rx=0 tx=0).stop 2024-05-10T11:21:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.681+0000 7f0f51237b80 1 -- 172.21.15.27:0/1993703450 shutdown_connections 2024-05-10T11:21:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.681+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1993703450 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56380a0ce380 0x56380a0d0770 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.681+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1993703450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 0x56380a0b7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.681+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1993703450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b2300 0x56380a0b57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.681+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1993703450 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 0x56380a0b5290 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.681+0000 7f0f29600700 0 client.26492 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:21:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.681+0000 7f0f51237b80 1 -- 172.21.15.27:0/1993703450 >> 172.21.15.27:0/1993703450 conn(0x563809fdab60 msgr2=0x56380a0a1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.681+0000 7f0f51237b80 1 -- 172.21.15.27:0/1993703450 shutdown_connections 2024-05-10T11:21:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.681+0000 7f0f51237b80 1 -- 172.21.15.27:0/1993703450 wait complete. 2024-05-10T11:21:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.681+0000 7f0f51237b80 20 client.26492 trim_cache size 0 max 16384 2024-05-10T11:21:24.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.699+0000 7f0f51237b80 1 Processor -- start 2024-05-10T11:21:24.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.699+0000 7f0f51237b80 1 -- start start 2024-05-10T11:21:24.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.700+0000 7f0f51237b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0ac600 0x56380a0abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.700+0000 7f0f51237b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b19e0 0x56380a0b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.700+0000 7f0f51237b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0b2300 0x56380a0b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.700+0000 7f0f51237b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0f401ad3c0 con 0x56380a0b19e0 2024-05-10T11:21:24.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.700+0000 7f0f51237b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0f401aef30 con 0x56380a0ac600 2024-05-10T11:21:24.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.700+0000 7f0f51237b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0f401ada90 con 0x56380a0b2300 2024-05-10T11:21:24.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.700+0000 7f0f3ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0ac600 0x56380a0abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.700+0000 7f0f3f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b19e0 0x56380a0b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.700+0000 7f0f3ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0ac600 0x56380a0abfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36174/0 (socket says 172.21.15.27:36174) 2024-05-10T11:21:24.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.700+0000 7f0f3ea00700 1 -- 172.21.15.27:0/3026581997 learned_addr learned my addr 172.21.15.27:0/3026581997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:24.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.700+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/3026581997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0b2300 0x56380a0b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.701+0000 7f0f3ea00700 1 -- 172.21.15.27:0/3026581997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0b2300 msgr2=0x56380a0b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:24.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.701+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/3026581997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0b2300 0x56380a0b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.701+0000 7f0f3ea00700 1 -- 172.21.15.27:0/3026581997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b19e0 msgr2=0x56380a0b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:24.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.701+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/3026581997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b19e0 0x56380a0b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.701+0000 7f0f3ea00700 1 -- 172.21.15.27:0/3026581997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56380a0b5860 con 0x56380a0ac600 2024-05-10T11:21:24.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.701+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/3026581997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0ac600 0x56380a0abfe0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c003db0 tx=0x7f0f2c003d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8f3b44b6bb82f5a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:24.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.701+0000 7f0f2be00700 1 -- 172.21.15.27:0/3026581997 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c032040 con 0x56380a0ac600 2024-05-10T11:21:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.701+0000 7f0f2be00700 1 -- 172.21.15.27:0/3026581997 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f2c003b30 con 0x56380a0ac600 2024-05-10T11:21:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.701+0000 7f0f2be00700 1 -- 172.21.15.27:0/3026581997 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c00f460 con 0x56380a0ac600 2024-05-10T11:21:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.702+0000 7f0f51237b80 1 -- 172.21.15.27:0/3026581997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0ac600 msgr2=0x56380a0abfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.702+0000 7f0f51237b80 1 --2- 172.21.15.27:0/3026581997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0ac600 0x56380a0abfe0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c003db0 tx=0x7f0f2c003d40 comp rx=0 tx=0).stop 2024-05-10T11:21:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.702+0000 7f0f51237b80 1 -- 172.21.15.27:0/3026581997 shutdown_connections 2024-05-10T11:21:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.702+0000 7f0f51237b80 1 --2- 172.21.15.27:0/3026581997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0b2300 0x56380a0b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.702+0000 7f0f51237b80 1 --2- 172.21.15.27:0/3026581997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0ac600 0x56380a0abfe0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.702+0000 7f0f51237b80 1 --2- 172.21.15.27:0/3026581997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b19e0 0x56380a0b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.702+0000 7f0f51237b80 1 -- 172.21.15.27:0/3026581997 >> 172.21.15.27:0/3026581997 conn(0x563809fdab60 msgr2=0x56380a0b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.702+0000 7f0f51237b80 1 -- 172.21.15.27:0/3026581997 shutdown_connections 2024-05-10T11:21:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.702+0000 7f0f51237b80 1 -- 172.21.15.27:0/3026581997 wait complete. 2024-05-10T11:21:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.703+0000 7f0f51237b80 1 -- start start 2024-05-10T11:21:24.705 INFO:tasks.workunit.client.0.smithi027.stdout: 1 468017 6.57 MB/sec execute 249 sec latency 102.014 ms 2024-05-10T11:21:24.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.703+0000 7f0f51237b80 1 Processor -- start 2024-05-10T11:21:24.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.704+0000 7f0f51237b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 0x56380a0c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.704+0000 7f0f51237b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 0x56380a0c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.704+0000 7f0f51237b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b2300 0x56380a0c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.704+0000 7f0f51237b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0f401ae860 con 0x56380a0b2300 2024-05-10T11:21:24.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.704+0000 7f0f51237b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0f401ae190 con 0x56380a0b19e0 2024-05-10T11:21:24.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.704+0000 7f0f51237b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0f401ab3a0 con 0x56380a0ac600 2024-05-10T11:21:24.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.704+0000 7f0f51237b80 10 client.0 _ll_register_callbacks cb 0x563809fa5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:24.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.704+0000 7f0f3f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 0x56380a0c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.704+0000 7f0f3f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 0x56380a0c6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36188/0 (socket says 172.21.15.27:36188) 2024-05-10T11:21:24.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.704+0000 7f0f3f400700 1 -- 172.21.15.27:0/1414745601 learned_addr learned my addr 172.21.15.27:0/1414745601 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:24.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.704+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b2300 0x56380a0c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.704+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 0x56380a0c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.705+0000 7f0f3f400700 1 -- 172.21.15.27:0/1414745601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 msgr2=0x56380a0c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.705+0000 7f0f3f400700 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 0x56380a0c6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.705+0000 7f0f3f400700 1 -- 172.21.15.27:0/1414745601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b2300 msgr2=0x56380a0c39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.705+0000 7f0f3f400700 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b2300 0x56380a0c39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.705+0000 7f0f3f400700 1 -- 172.21.15.27:0/1414745601 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56380a0c3fb0 con 0x56380a0b19e0 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.705+0000 7f0f3f400700 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 0x56380a0c6f60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0f30005a80 tx=0x7f0f3000fcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=792e3dc3b0faeb34 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.705+0000 7f0f29600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.705+0000 7f0f29600700 1 -- 172.21.15.27:0/1414745601 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f30005250 con 0x56380a0b19e0 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.705+0000 7f0f29600700 1 -- 172.21.15.27:0/1414745601 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7f0f30018040 con 0x56380a0b19e0 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.705+0000 7f0f51237b80 1 -- 172.21.15.27:0/1414745601 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f2c003b30 con 0x56380a0b19e0 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.706+0000 7f0f29600700 1 -- 172.21.15.27:0/1414745601 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0f30038050 con 0x56380a0b19e0 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.706+0000 7f0f51237b80 10 client.26498 fetch_fsmap learned FSMap version 34 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.706+0000 7f0f51237b80 1 -- 172.21.15.27:0/1414745601 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56380a139460 con 0x56380a0b19e0 2024-05-10T11:21:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.706+0000 7f0f29600700 1 -- 172.21.15.27:0/1414745601 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f2c003b30 con 0x56380a0b19e0 2024-05-10T11:21:24.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.706+0000 7f0f29600700 1 -- 172.21.15.27:0/1414745601 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f0f30003bf0 con 0x56380a0b19e0 2024-05-10T11:21:24.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.706+0000 7f0f51237b80 10 client.26498 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:21:24.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.706+0000 7f0f51237b80 10 client.26498 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:21:24.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.706+0000 7f0f51237b80 20 client.26498 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:24.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.706+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56380a139aa0 0x56380a13bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:24.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.706+0000 7f0f51237b80 4 client.26498 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:21:24.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.706+0000 7f0f51237b80 1 -- 172.21.15.27:0/1414745601 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56380a00c400 con 0x56380a139aa0 2024-05-10T11:21:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.707+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56380a139aa0 0x56380a13bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.707+0000 7f0f3ea00700 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56380a139aa0 0x56380a13bec0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fd6e2e6ea3a00f2e server_cookie=50deb002d471c36b in_seq=0 out_seq=0 2024-05-10T11:21:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.707+0000 7f0f29600700 10 client.26498 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:21:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.707+0000 7f0f29600700 1 -- 172.21.15.27:0/1414745601 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56380a00c400 con 0x56380a139aa0 2024-05-10T11:21:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.707+0000 7f0f29600700 10 client.26498 handle_command_reply: tid=0 2024-05-10T11:21:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.707+0000 7f0f51237b80 1 client.26498 shutdown 2024-05-10T11:21:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.708+0000 7f0f51237b80 1 -- 172.21.15.27:0/1414745601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 msgr2=0x56380a0c6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.708+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 0x56380a0c6f60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0f30005a80 tx=0x7f0f3000fcd0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.708+0000 7f0f51237b80 1 -- 172.21.15.27:0/1414745601 shutdown_connections 2024-05-10T11:21:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.708+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x56380a139aa0 0x56380a13bec0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.708+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56380a0ac600 0x56380a0c6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.708+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56380a0b2300 0x56380a0c39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.708+0000 7f0f51237b80 1 --2- 172.21.15.27:0/1414745601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56380a0b19e0 0x56380a0c6f60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.708+0000 7f0f29600700 0 client.26498 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:21:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.708+0000 7f0f51237b80 1 -- 172.21.15.27:0/1414745601 >> 172.21.15.27:0/1414745601 conn(0x563809fdab60 msgr2=0x56380a0a1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.708+0000 7f0f51237b80 1 -- 172.21.15.27:0/1414745601 shutdown_connections 2024-05-10T11:21:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.708+0000 7f0f51237b80 1 -- 172.21.15.27:0/1414745601 wait complete. 2024-05-10T11:21:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.708+0000 7f0f51237b80 20 client.26498 trim_cache size 0 max 16384 2024-05-10T11:21:24.713 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:21:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.709+0000 7f0f45200700 1 -- 172.21.15.27:0/29476705 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0f240b2a70 msgr2=0x7f0f240b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.709+0000 7f0f45200700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0f240b2a70 0x7f0f240b4f30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0f3800c340 tx=0x7f0f38009630 comp rx=0 tx=0).stop 2024-05-10T11:21:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.709+0000 7f0f45200700 1 -- 172.21.15.27:0/29476705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f4010b660 msgr2=0x7f0f4019ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.709+0000 7f0f45200700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f4010b660 0x7f0f4019ba80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0f3800c370 tx=0x7f0f3800c680 comp rx=0 tx=0).stop 2024-05-10T11:21:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.709+0000 7f0f45200700 1 -- 172.21.15.27:0/29476705 shutdown_connections 2024-05-10T11:21:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.709+0000 7f0f45200700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0f240b2a70 0x7f0f240b4f30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.709+0000 7f0f45200700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f4010b660 0x7f0f4019ba80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.709+0000 7f0f45200700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f401055d0 0x7f0f401a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.709+0000 7f0f45200700 1 --2- 172.21.15.27:0/29476705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f40105f90 0x7f0f401a1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:24.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.709+0000 7f0f45200700 1 -- 172.21.15.27:0/29476705 >> 172.21.15.27:0/29476705 conn(0x7f0f40101080 msgr2=0x7f0f401026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:24.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.709+0000 7f0f45200700 1 -- 172.21.15.27:0/29476705 shutdown_connections 2024-05-10T11:21:24.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:24.709+0000 7f0f45200700 1 -- 172.21.15.27:0/29476705 wait complete. 2024-05-10T11:21:25.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:25 smithi027 ceph-mon[30054]: pgmap v252: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2024-05-10T11:21:25.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:21:25.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:25 smithi076 ceph-mon[34152]: pgmap v252: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2024-05-10T11:21:25.704 INFO:tasks.workunit.client.0.smithi027.stdout: 1 469378 6.58 MB/sec execute 250 sec latency 27.532 ms 2024-05-10T11:21:25.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:25 smithi189 ceph-mon[34842]: pgmap v252: 129 pgs: 129 active+clean; 295 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2024-05-10T11:21:26.020 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:21:26.704 INFO:tasks.workunit.client.0.smithi027.stdout: 1 470552 6.58 MB/sec execute 251 sec latency 34.274 ms 2024-05-10T11:21:27.597 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:27 smithi027 ceph-mon[30054]: pgmap v253: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 159 op/s 2024-05-10T11:21:27.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:27 smithi076 ceph-mon[34152]: pgmap v253: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 159 op/s 2024-05-10T11:21:27.704 INFO:tasks.workunit.client.0.smithi027.stdout: 1 472367 6.59 MB/sec execute 252 sec latency 26.802 ms 2024-05-10T11:21:27.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:27 smithi189 ceph-mon[34842]: pgmap v253: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 159 op/s 2024-05-10T11:21:28.704 INFO:tasks.workunit.client.0.smithi027.stdout: 1 473279 6.58 MB/sec execute 253 sec latency 37.287 ms 2024-05-10T11:21:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.737+0000 7f2d78600700 1 -- 172.21.15.27:0/1338365645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d700756d0 msgr2=0x7f2d70075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.737+0000 7f2d78600700 1 --2- 172.21.15.27:0/1338365645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d700756d0 0x7f2d70075ab0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009990 tx=0x7f2d64009ca0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.737+0000 7f2d78600700 1 -- 172.21.15.27:0/1338365645 shutdown_connections 2024-05-10T11:21:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.737+0000 7f2d78600700 1 --2- 172.21.15.27:0/1338365645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d700756d0 0x7f2d70075ab0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.737+0000 7f2d78600700 1 --2- 172.21.15.27:0/1338365645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d7010ff00 0x7f2d701122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.737+0000 7f2d78600700 1 --2- 172.21.15.27:0/1338365645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d70075ff0 0x7f2d7010f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.737+0000 7f2d78600700 1 -- 172.21.15.27:0/1338365645 >> 172.21.15.27:0/1338365645 conn(0x7f2d7007a600 msgr2=0x7f2d7007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.737+0000 7f2d78600700 1 -- 172.21.15.27:0/1338365645 shutdown_connections 2024-05-10T11:21:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.737+0000 7f2d78600700 1 -- 172.21.15.27:0/1338365645 wait complete. 2024-05-10T11:21:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.738+0000 7f2d78600700 1 Processor -- start 2024-05-10T11:21:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.739+0000 7f2d78600700 1 -- start start 2024-05-10T11:21:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.739+0000 7f2d78600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d700756d0 0x7f2d7019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.739+0000 7f2d78600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d70075ff0 0x7f2d7019c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.739+0000 7f2d78600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d7010ff00 0x7f2d701a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.739+0000 7f2d78600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d701158e0 con 0x7f2d7010ff00 2024-05-10T11:21:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.739+0000 7f2d78600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2d70115760 con 0x7f2d700756d0 2024-05-10T11:21:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.739+0000 7f2d78600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2d70115a60 con 0x7f2d70075ff0 2024-05-10T11:21:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.739+0000 7f2d75200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d70075ff0 0x7f2d7019c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.739+0000 7f2d75200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d70075ff0 0x7f2d7019c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60698/0 (socket says 172.21.15.27:60698) 2024-05-10T11:21:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.739+0000 7f2d75c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d700756d0 0x7f2d7019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.739+0000 7f2d75200700 1 -- 172.21.15.27:0/3922408854 learned_addr learned my addr 172.21.15.27:0/3922408854 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:28.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.739+0000 7f2d76600700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d7010ff00 0x7f2d701a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.740+0000 7f2d75200700 1 -- 172.21.15.27:0/3922408854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d700756d0 msgr2=0x7f2d7019bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:28.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.740+0000 7f2d75200700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d700756d0 0x7f2d7019bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.740+0000 7f2d75200700 1 -- 172.21.15.27:0/3922408854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d7010ff00 msgr2=0x7f2d701a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:28.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.740+0000 7f2d75200700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d7010ff00 0x7f2d701a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.740+0000 7f2d75200700 1 -- 172.21.15.27:0/3922408854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d701a0d40 con 0x7f2d70075ff0 2024-05-10T11:21:28.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.740+0000 7f2d75200700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d70075ff0 0x7f2d7019c270 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c00e9a0 tx=0x7f2d5c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:28.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.741+0000 7f2d6e000700 1 -- 172.21.15.27:0/3922408854 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c00ca10 con 0x7f2d70075ff0 2024-05-10T11:21:28.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.741+0000 7f2d6e000700 1 -- 172.21.15.27:0/3922408854 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d5c00cbb0 con 0x7f2d70075ff0 2024-05-10T11:21:28.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.741+0000 7f2d78600700 1 -- 172.21.15.27:0/3922408854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d701a1030 con 0x7f2d70075ff0 2024-05-10T11:21:28.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.741+0000 7f2d6e000700 1 -- 172.21.15.27:0/3922408854 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c018680 con 0x7f2d70075ff0 2024-05-10T11:21:28.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.741+0000 7f2d78600700 1 -- 172.21.15.27:0/3922408854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d701a88b0 con 0x7f2d70075ff0 2024-05-10T11:21:28.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.744+0000 7f2d6e000700 1 -- 172.21.15.27:0/3922408854 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2d5c010430 con 0x7f2d70075ff0 2024-05-10T11:21:28.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.746+0000 7f2d6e000700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d580b2a70 0x7f2d580b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.746+0000 7f2d6e000700 1 -- 172.21.15.27:0/3922408854 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7f2d5c0e2ba0 con 0x7f2d70075ff0 2024-05-10T11:21:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.746+0000 7f2d75c00700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d580b2a70 0x7f2d580b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.746+0000 7f2d83008b80 1 Processor -- start 2024-05-10T11:21:28.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.747+0000 7f2d83008b80 1 -- start start 2024-05-10T11:21:28.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.747+0000 7f2d83008b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa6030060 0x55cfa6030440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.747+0000 7f2d83008b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 0x55cfa6034490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.747+0000 7f2d83008b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa60349d0 0x55cfa6034e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.747+0000 7f2d83008b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cfa5f82a20 con 0x55cfa6030060 2024-05-10T11:21:28.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.747+0000 7f2d83008b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cfa5f20760 con 0x55cfa6030980 2024-05-10T11:21:28.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.747+0000 7f2d83008b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cfa5f7e9e0 con 0x55cfa60349d0 2024-05-10T11:21:28.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.748+0000 7f2d75c00700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d580b2a70 0x7f2d580b4f30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d64006010 tx=0x7f2d6400b260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:28.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.748+0000 7f2d75200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 0x55cfa6034490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.748+0000 7f2d75200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 0x55cfa6034490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36208/0 (socket says 172.21.15.27:36208) 2024-05-10T11:21:28.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.748+0000 7f2d75200700 1 -- 172.21.15.27:0/4091753570 learned_addr learned my addr 172.21.15.27:0/4091753570 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:28.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.748+0000 7f2d75c00700 1 --2- 172.21.15.27:0/4091753570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa6030060 0x55cfa6030440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.748+0000 7f2d76600700 1 --2- 172.21.15.27:0/4091753570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa60349d0 0x55cfa6034e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.748+0000 7f2d75200700 1 -- 172.21.15.27:0/4091753570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa60349d0 msgr2=0x55cfa6034e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.748+0000 7f2d75200700 1 --2- 172.21.15.27:0/4091753570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa60349d0 0x55cfa6034e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.748+0000 7f2d75200700 1 -- 172.21.15.27:0/4091753570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa6030060 msgr2=0x55cfa6030440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.748+0000 7f2d75200700 1 --2- 172.21.15.27:0/4091753570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa6030060 0x55cfa6030440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.748+0000 7f2d75200700 1 -- 172.21.15.27:0/4091753570 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfa5ecf8d0 con 0x55cfa6030980 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.748+0000 7f2d75200700 1 --2- 172.21.15.27:0/4091753570 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 0x55cfa6034490 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0a5a70 tx=0x7f2d5c0a9500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27866df1371ec681 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.749+0000 7f2d62a00700 1 -- 172.21.15.27:0/4091753570 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c0a99f0 con 0x55cfa6030980 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.749+0000 7f2d62a00700 1 -- 172.21.15.27:0/4091753570 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d5c0adda0 con 0x55cfa6030980 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.749+0000 7f2d62a00700 1 -- 172.21.15.27:0/4091753570 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c0b23c0 con 0x55cfa6030980 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.750+0000 7f2d83008b80 1 -- 172.21.15.27:0/4091753570 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 msgr2=0x55cfa6034490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.750+0000 7f2d83008b80 1 --2- 172.21.15.27:0/4091753570 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 0x55cfa6034490 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0a5a70 tx=0x7f2d5c0a9500 comp rx=0 tx=0).stop 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.750+0000 7f2d83008b80 1 -- 172.21.15.27:0/4091753570 shutdown_connections 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.750+0000 7f2d83008b80 1 --2- 172.21.15.27:0/4091753570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa60349d0 0x55cfa6034e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.750+0000 7f2d83008b80 1 --2- 172.21.15.27:0/4091753570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa6030060 0x55cfa6030440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.750+0000 7f2d83008b80 1 --2- 172.21.15.27:0/4091753570 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 0x55cfa6034490 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.750+0000 7f2d83008b80 1 -- 172.21.15.27:0/4091753570 >> 172.21.15.27:0/4091753570 conn(0x55cfa5f5eb60 msgr2=0x55cfa602a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.750+0000 7f2d83008b80 1 -- 172.21.15.27:0/4091753570 shutdown_connections 2024-05-10T11:21:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.750+0000 7f2d83008b80 1 -- 172.21.15.27:0/4091753570 wait complete. 2024-05-10T11:21:28.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.750+0000 7f2d83008b80 1 -- start start 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.751+0000 7f2d83008b80 1 Processor -- start 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.751+0000 7f2d83008b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 0x55cfa603bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.751+0000 7f2d83008b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa6030980 0x55cfa6039350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.751+0000 7f2d83008b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa60349d0 0x55cfa6039890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.751+0000 7f2d83008b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cfa603cbe0 con 0x55cfa60349d0 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.751+0000 7f2d83008b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55cfa5ee8b20 con 0x55cfa6030060 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.751+0000 7f2d83008b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cfa603cd60 con 0x55cfa6030980 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.751+0000 7f2d83008b80 10 client.0 _ll_register_callbacks cb 0x55cfa5f29c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d75c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 0x55cfa603bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d75200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa6030980 0x55cfa6039350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d75c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 0x55cfa603bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36210/0 (socket says 172.21.15.27:36210) 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d75c00700 1 -- 172.21.15.27:0/2868418568 learned_addr learned my addr 172.21.15.27:0/2868418568 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d76600700 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa60349d0 0x55cfa6039890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d75c00700 1 -- 172.21.15.27:0/2868418568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa6030980 msgr2=0x55cfa6039350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:28.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d75c00700 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa6030980 0x55cfa6039350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d75c00700 1 -- 172.21.15.27:0/2868418568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa60349d0 msgr2=0x55cfa6039890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:28.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d75c00700 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa60349d0 0x55cfa6039890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d75c00700 1 -- 172.21.15.27:0/2868418568 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfa5f48420 con 0x55cfa6030060 2024-05-10T11:21:28.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d76600700 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa60349d0 0x55cfa6039890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:21:28.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d75200700 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa6030980 0x55cfa6039350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:21:28.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d75c00700 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 0x55cfa603bf40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400be50 tx=0x7f2d64009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb50e2880950e73c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:28.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.752+0000 7f2d57400700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:21:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.753+0000 7f2d57400700 1 -- 172.21.15.27:0/2868418568 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d64003a10 con 0x55cfa6030060 2024-05-10T11:21:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.753+0000 7f2d57400700 1 -- 172.21.15.27:0/2868418568 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7f2d64003bb0 con 0x55cfa6030060 2024-05-10T11:21:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.753+0000 7f2d83008b80 1 -- 172.21.15.27:0/2868418568 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d5c0adda0 con 0x55cfa6030060 2024-05-10T11:21:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.753+0000 7f2d57400700 1 -- 172.21.15.27:0/2868418568 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2d6403a050 con 0x55cfa6030060 2024-05-10T11:21:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.753+0000 7f2d83008b80 10 client.26510 fetch_fsmap learned FSMap version 34 2024-05-10T11:21:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.753+0000 7f2d83008b80 1 -- 172.21.15.27:0/2868418568 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cfa6047600 con 0x55cfa6030060 2024-05-10T11:21:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.753+0000 7f2d57400700 1 -- 172.21.15.27:0/2868418568 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d5c0adda0 con 0x55cfa6030060 2024-05-10T11:21:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.754+0000 7f2d57400700 1 -- 172.21.15.27:0/2868418568 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f2d640344e0 con 0x55cfa6030060 2024-05-10T11:21:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.754+0000 7f2d83008b80 10 client.26510 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:21:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.754+0000 7f2d83008b80 10 client.26510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:21:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.754+0000 7f2d83008b80 20 client.26510 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.754+0000 7f2d83008b80 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cfa6049b90 0x55cfa604bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.754+0000 7f2d83008b80 4 client.26510 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:21:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.754+0000 7f2d83008b80 1 -- 172.21.15.27:0/2868418568 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfa5ffd0a0 con 0x55cfa6049b90 2024-05-10T11:21:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.754+0000 7f2d75200700 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cfa6049b90 0x55cfa604bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.755+0000 7f2d75200700 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cfa6049b90 0x55cfa604bfd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2079917d98e9ba server_cookie=ab7e7ff5ce0e5a90 in_seq=0 out_seq=0 2024-05-10T11:21:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.755+0000 7f2d57400700 10 client.26510 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d57400700 1 -- 172.21.15.27:0/2868418568 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cfa5ffd0a0 con 0x55cfa6049b90 2024-05-10T11:21:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d57400700 10 client.26510 handle_command_reply: tid=0 2024-05-10T11:21:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d83008b80 1 client.26510 shutdown 2024-05-10T11:21:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d83008b80 1 -- 172.21.15.27:0/2868418568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 msgr2=0x55cfa603bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d83008b80 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 0x55cfa603bf40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400be50 tx=0x7f2d64009310 comp rx=0 tx=0).stop 2024-05-10T11:21:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d83008b80 1 -- 172.21.15.27:0/2868418568 shutdown_connections 2024-05-10T11:21:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d83008b80 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cfa6049b90 0x55cfa604bfd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d83008b80 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa60349d0 0x55cfa6039890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d83008b80 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 0x55cfa603bf40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d83008b80 1 --2- 172.21.15.27:0/2868418568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa6030980 0x55cfa6039350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d57400700 0 client.26510 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d83008b80 1 -- 172.21.15.27:0/2868418568 >> 172.21.15.27:0/2868418568 conn(0x55cfa5f5eb60 msgr2=0x55cfa60252b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.756+0000 7f2d83008b80 1 -- 172.21.15.27:0/2868418568 shutdown_connections 2024-05-10T11:21:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.757+0000 7f2d83008b80 1 -- 172.21.15.27:0/2868418568 wait complete. 2024-05-10T11:21:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.757+0000 7f2d83008b80 20 client.26510 trim_cache size 0 max 16384 2024-05-10T11:21:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.774+0000 7f2d83008b80 1 Processor -- start 2024-05-10T11:21:28.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.774+0000 7f2d83008b80 1 -- start start 2024-05-10T11:21:28.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.774+0000 7f2d83008b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 0x55cfa6045c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.774+0000 7f2d83008b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa6030980 0x55cfa6025130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.774+0000 7f2d83008b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa60349d0 0x55cfa60518a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.774+0000 7f2d83008b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d701b1860 con 0x55cfa60349d0 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.774+0000 7f2d83008b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2d701b33d0 con 0x55cfa6030060 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.774+0000 7f2d83008b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2d701b1f30 con 0x55cfa6030980 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.775+0000 7f2d75c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 0x55cfa6045c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.775+0000 7f2d75c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 0x55cfa6045c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36220/0 (socket says 172.21.15.27:36220) 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.775+0000 7f2d75c00700 1 -- 172.21.15.27:0/1471845173 learned_addr learned my addr 172.21.15.27:0/1471845173 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.775+0000 7f2d76600700 1 --2- 172.21.15.27:0/1471845173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa60349d0 0x55cfa60518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.775+0000 7f2d75200700 1 --2- 172.21.15.27:0/1471845173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa6030980 0x55cfa6025130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.775+0000 7f2d75c00700 1 -- 172.21.15.27:0/1471845173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa6030980 msgr2=0x55cfa6025130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.775+0000 7f2d75c00700 1 --2- 172.21.15.27:0/1471845173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa6030980 0x55cfa6025130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.775+0000 7f2d75c00700 1 -- 172.21.15.27:0/1471845173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa60349d0 msgr2=0x55cfa60518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.775+0000 7f2d75c00700 1 --2- 172.21.15.27:0/1471845173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa60349d0 0x55cfa60518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.775+0000 7f2d75c00700 1 -- 172.21.15.27:0/1471845173 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfa6039df0 con 0x55cfa6030060 2024-05-10T11:21:28.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.776+0000 7f2d75c00700 1 --2- 172.21.15.27:0/1471845173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 0x55cfa6045c70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2d640098e0 tx=0x7f2d6400b310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f875a46d05291f80 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.776+0000 7f2d62000700 1 -- 172.21.15.27:0/1471845173 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d64028040 con 0x55cfa6030060 2024-05-10T11:21:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.776+0000 7f2d62000700 1 -- 172.21.15.27:0/1471845173 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d64003e10 con 0x55cfa6030060 2024-05-10T11:21:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.776+0000 7f2d62000700 1 -- 172.21.15.27:0/1471845173 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6400f6c0 con 0x55cfa6030060 2024-05-10T11:21:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.776+0000 7f2d83008b80 1 -- 172.21.15.27:0/1471845173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 msgr2=0x55cfa6045c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.776+0000 7f2d83008b80 1 --2- 172.21.15.27:0/1471845173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 0x55cfa6045c70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2d640098e0 tx=0x7f2d6400b310 comp rx=0 tx=0).stop 2024-05-10T11:21:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.777+0000 7f2d83008b80 1 -- 172.21.15.27:0/1471845173 shutdown_connections 2024-05-10T11:21:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.777+0000 7f2d83008b80 1 --2- 172.21.15.27:0/1471845173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa60349d0 0x55cfa60518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.777+0000 7f2d83008b80 1 --2- 172.21.15.27:0/1471845173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030060 0x55cfa6045c70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.777+0000 7f2d83008b80 1 --2- 172.21.15.27:0/1471845173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa6030980 0x55cfa6025130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.777+0000 7f2d83008b80 1 -- 172.21.15.27:0/1471845173 >> 172.21.15.27:0/1471845173 conn(0x55cfa5f5eb60 msgr2=0x55cfa6024a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.777+0000 7f2d83008b80 1 -- 172.21.15.27:0/1471845173 shutdown_connections 2024-05-10T11:21:28.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.777+0000 7f2d83008b80 1 -- 172.21.15.27:0/1471845173 wait complete. 2024-05-10T11:21:28.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.777+0000 7f2d83008b80 1 -- start start 2024-05-10T11:21:28.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.777+0000 7f2d83008b80 1 Processor -- start 2024-05-10T11:21:28.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.778+0000 7f2d83008b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa6030060 0x55cfa6041dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.778+0000 7f2d83008b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 0x55cfa6042310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.778+0000 7f2d83008b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa60349d0 0x55cfa603f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.778+0000 7f2d83008b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d701b2d00 con 0x55cfa6030060 2024-05-10T11:21:28.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.778+0000 7f2d83008b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2d701b2630 con 0x55cfa6030980 2024-05-10T11:21:28.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.778+0000 7f2d83008b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2d701af820 con 0x55cfa60349d0 2024-05-10T11:21:28.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.778+0000 7f2d83008b80 10 client.0 _ll_register_callbacks cb 0x55cfa5f29c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:28.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.778+0000 7f2d75c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa6030060 0x55cfa6041dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.778+0000 7f2d75200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 0x55cfa6042310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.778+0000 7f2d75c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa6030060 0x55cfa6041dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51306/0 (socket says 172.21.15.27:51306) 2024-05-10T11:21:28.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.778+0000 7f2d75c00700 1 -- 172.21.15.27:0/1357719679 learned_addr learned my addr 172.21.15.27:0/1357719679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.778+0000 7f2d76600700 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa60349d0 0x55cfa603f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.779+0000 7f2d75200700 1 -- 172.21.15.27:0/1357719679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa60349d0 msgr2=0x55cfa603f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.779+0000 7f2d75200700 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa60349d0 0x55cfa603f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.779+0000 7f2d75200700 1 -- 172.21.15.27:0/1357719679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa6030060 msgr2=0x55cfa6041dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.779+0000 7f2d75200700 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa6030060 0x55cfa6041dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.779+0000 7f2d75200700 1 -- 172.21.15.27:0/1357719679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfa603fb20 con 0x55cfa6030980 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.779+0000 7f2d75c00700 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa6030060 0x55cfa6041dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.779+0000 7f2d75200700 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 0x55cfa6042310 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c019000 tx=0x7f2d5c0a98f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adcff8d22a74222e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.779+0000 7f2d57400700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.779+0000 7f2d57400700 1 -- 172.21.15.27:0/1357719679 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c0adda0 con 0x55cfa6030980 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.779+0000 7f2d83008b80 1 -- 172.21.15.27:0/1357719679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d64003e10 con 0x55cfa6030980 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.779+0000 7f2d57400700 1 -- 172.21.15.27:0/1357719679 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7f2d5c0a9e60 con 0x55cfa6030980 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.779+0000 7f2d57400700 1 -- 172.21.15.27:0/1357719679 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2d5c0a99f0 con 0x55cfa6030980 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.780+0000 7f2d83008b80 10 client.26516 fetch_fsmap learned FSMap version 34 2024-05-10T11:21:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.780+0000 7f2d83008b80 1 -- 172.21.15.27:0/1357719679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cfa60b3f50 con 0x55cfa6030980 2024-05-10T11:21:28.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.780+0000 7f2d57400700 1 -- 172.21.15.27:0/1357719679 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d64003e10 con 0x55cfa6030980 2024-05-10T11:21:28.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.780+0000 7f2d57400700 1 -- 172.21.15.27:0/1357719679 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f2d5c010820 con 0x55cfa6030980 2024-05-10T11:21:28.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.780+0000 7f2d83008b80 10 client.26516 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:21:28.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.780+0000 7f2d83008b80 10 client.26516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:21:28.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.780+0000 7f2d83008b80 20 client.26516 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:28.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.780+0000 7f2d83008b80 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cfa60b4180 0x55cfa60b6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:28.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.780+0000 7f2d83008b80 4 client.26516 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:21:28.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.780+0000 7f2d83008b80 1 -- 172.21.15.27:0/1357719679 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cfa5f90400 con 0x55cfa60b4180 2024-05-10T11:21:28.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.780+0000 7f2d75c00700 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cfa60b4180 0x55cfa60b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:28.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.781+0000 7f2d75c00700 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cfa60b4180 0x55cfa60b6570 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1080496a6ae0ff7 server_cookie=292592e81f09dc91 in_seq=0 out_seq=0 2024-05-10T11:21:28.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.781+0000 7f2d57400700 10 client.26516 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:28.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.782+0000 7f2d57400700 1 -- 172.21.15.27:0/1357719679 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cfa5f90400 con 0x55cfa60b4180 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.782+0000 7f2d57400700 10 client.26516 handle_command_reply: tid=0 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.782+0000 7f2d83008b80 1 client.26516 shutdown 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.783+0000 7f2d83008b80 1 -- 172.21.15.27:0/1357719679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 msgr2=0x55cfa6042310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.783+0000 7f2d83008b80 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 0x55cfa6042310 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c019000 tx=0x7f2d5c0a98f0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.783+0000 7f2d83008b80 1 -- 172.21.15.27:0/1357719679 shutdown_connections 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.783+0000 7f2d83008b80 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55cfa60b4180 0x55cfa60b6570 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.783+0000 7f2d83008b80 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfa60349d0 0x55cfa603f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.783+0000 7f2d57400700 0 client.26516 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.783+0000 7f2d83008b80 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cfa6030060 0x55cfa6041dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.783+0000 7f2d83008b80 1 --2- 172.21.15.27:0/1357719679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cfa6030980 0x55cfa6042310 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.783+0000 7f2d83008b80 1 -- 172.21.15.27:0/1357719679 >> 172.21.15.27:0/1357719679 conn(0x55cfa5f5eb60 msgr2=0x55cfa60252b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.783+0000 7f2d83008b80 1 -- 172.21.15.27:0/1357719679 shutdown_connections 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.783+0000 7f2d83008b80 1 -- 172.21.15.27:0/1357719679 wait complete. 2024-05-10T11:21:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.783+0000 7f2d83008b80 20 client.26516 trim_cache size 0 max 16384 2024-05-10T11:21:28.786 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:21:28.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.784+0000 7f2d78600700 1 -- 172.21.15.27:0/3922408854 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d580b2a70 msgr2=0x7f2d580b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:28.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.784+0000 7f2d78600700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d580b2a70 0x7f2d580b4f30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d64006010 tx=0x7f2d6400b260 comp rx=0 tx=0).stop 2024-05-10T11:21:28.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.784+0000 7f2d78600700 1 -- 172.21.15.27:0/3922408854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d70075ff0 msgr2=0x7f2d7019c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:28.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.784+0000 7f2d78600700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d70075ff0 0x7f2d7019c270 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c00e9a0 tx=0x7f2d5c00ecb0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.785+0000 7f2d78600700 1 -- 172.21.15.27:0/3922408854 shutdown_connections 2024-05-10T11:21:28.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.785+0000 7f2d78600700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2d580b2a70 0x7f2d580b4f30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.785+0000 7f2d78600700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d7010ff00 0x7f2d701a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.785+0000 7f2d78600700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d700756d0 0x7f2d7019bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.785+0000 7f2d78600700 1 --2- 172.21.15.27:0/3922408854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d70075ff0 0x7f2d7019c270 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:28.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.785+0000 7f2d78600700 1 -- 172.21.15.27:0/3922408854 >> 172.21.15.27:0/3922408854 conn(0x7f2d7007a600 msgr2=0x7f2d70100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:28.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.785+0000 7f2d78600700 1 -- 172.21.15.27:0/3922408854 shutdown_connections 2024-05-10T11:21:28.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:28.785+0000 7f2d78600700 1 -- 172.21.15.27:0/3922408854 wait complete. 2024-05-10T11:21:29.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:29 smithi027 ceph-mon[30054]: pgmap v254: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1.4 MiB/s wr, 97 op/s 2024-05-10T11:21:29.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:29 smithi076 ceph-mon[34152]: pgmap v254: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1.4 MiB/s wr, 97 op/s 2024-05-10T11:21:29.704 INFO:tasks.workunit.client.0.smithi027.stdout: 1 474521 6.58 MB/sec execute 254 sec latency 72.576 ms 2024-05-10T11:21:29.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:29 smithi189 ceph-mon[34842]: pgmap v254: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1.4 MiB/s wr, 97 op/s 2024-05-10T11:21:30.704 INFO:tasks.workunit.client.0.smithi027.stdout: 1 476300 6.60 MB/sec execute 255 sec latency 26.517 ms 2024-05-10T11:21:31.009 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:21:31.010 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:21:31.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:31 smithi027 ceph-mon[30054]: pgmap v255: 129 pgs: 129 active+clean; 307 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 2.5 MiB/s wr, 172 op/s 2024-05-10T11:21:31.514 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:21:31.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:31 smithi076 ceph-mon[34152]: pgmap v255: 129 pgs: 129 active+clean; 307 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 2.5 MiB/s wr, 172 op/s 2024-05-10T11:21:31.705 INFO:tasks.workunit.client.0.smithi027.stdout: 1 477032 6.58 MB/sec execute 256 sec latency 30.326 ms 2024-05-10T11:21:31.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:31 smithi189 ceph-mon[34842]: pgmap v255: 129 pgs: 129 active+clean; 307 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 2.5 MiB/s wr, 172 op/s 2024-05-10T11:21:32.705 INFO:tasks.workunit.client.0.smithi027.stdout: 1 478474 6.58 MB/sec execute 257 sec latency 26.236 ms 2024-05-10T11:21:33.537 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:33 smithi027 ceph-mon[30054]: pgmap v256: 129 pgs: 129 active+clean; 307 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-05-10T11:21:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:33 smithi076 ceph-mon[34152]: pgmap v256: 129 pgs: 129 active+clean; 307 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-05-10T11:21:33.705 INFO:tasks.workunit.client.0.smithi027.stdout: 1 479940 6.60 MB/sec execute 258 sec latency 19.594 ms 2024-05-10T11:21:33.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:33 smithi189 ceph-mon[34842]: pgmap v256: 129 pgs: 129 active+clean; 307 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-05-10T11:21:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.854+0000 7f3b29800700 1 -- 172.21.15.27:0/3527197641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b240758d0 msgr2=0x7f3b24075cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.854+0000 7f3b29800700 1 --2- 172.21.15.27:0/3527197641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b240758d0 0x7f3b24075cb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3b140098f0 tx=0x7f3b14009c00 comp rx=0 tx=0).stop 2024-05-10T11:21:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.854+0000 7f3b29800700 1 -- 172.21.15.27:0/3527197641 shutdown_connections 2024-05-10T11:21:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.854+0000 7f3b29800700 1 --2- 172.21.15.27:0/3527197641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b240758d0 0x7f3b24075cb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.854+0000 7f3b29800700 1 --2- 172.21.15.27:0/3527197641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b2410fc10 0x7f3b24112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.854+0000 7f3b29800700 1 --2- 172.21.15.27:0/3527197641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b240761f0 0x7f3b2410f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.854+0000 7f3b29800700 1 -- 172.21.15.27:0/3527197641 >> 172.21.15.27:0/3527197641 conn(0x7f3b240fef20 msgr2=0x7f3b24101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.855+0000 7f3b29800700 1 -- 172.21.15.27:0/3527197641 shutdown_connections 2024-05-10T11:21:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.855+0000 7f3b29800700 1 -- 172.21.15.27:0/3527197641 wait complete. 2024-05-10T11:21:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.856+0000 7f3b29800700 1 Processor -- start 2024-05-10T11:21:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.856+0000 7f3b29800700 1 -- start start 2024-05-10T11:21:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.856+0000 7f3b29800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240758d0 0x7f3b2419bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.856+0000 7f3b29800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b240761f0 0x7f3b2419c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b29800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b2410fc10 0x7f3b241a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b29800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3b241157f0 con 0x7f3b240758d0 2024-05-10T11:21:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b29800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3b24115670 con 0x7f3b2410fc10 2024-05-10T11:21:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b29800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3b24115970 con 0x7f3b240761f0 2024-05-10T11:21:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b23400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b240761f0 0x7f3b2419c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b23400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b240761f0 0x7f3b2419c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60754/0 (socket says 172.21.15.27:60754) 2024-05-10T11:21:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b23400700 1 -- 172.21.15.27:0/3139955022 learned_addr learned my addr 172.21.15.27:0/3139955022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b23e00700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240758d0 0x7f3b2419bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b28e00700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b2410fc10 0x7f3b241a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b23400700 1 -- 172.21.15.27:0/3139955022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b2410fc10 msgr2=0x7f3b241a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b23400700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b2410fc10 0x7f3b241a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b23400700 1 -- 172.21.15.27:0/3139955022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240758d0 msgr2=0x7f3b2419bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b23400700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240758d0 0x7f3b2419bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.857+0000 7f3b23400700 1 -- 172.21.15.27:0/3139955022 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b241a0d70 con 0x7f3b240761f0 2024-05-10T11:21:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.858+0000 7f3b23400700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b240761f0 0x7f3b2419c2f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000e9c0 tx=0x7f3b1000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.858+0000 7f3b20c00700 1 -- 172.21.15.27:0/3139955022 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b10004a30 con 0x7f3b240761f0 2024-05-10T11:21:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.858+0000 7f3b29800700 1 -- 172.21.15.27:0/3139955022 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b241a1060 con 0x7f3b240761f0 2024-05-10T11:21:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.858+0000 7f3b20c00700 1 -- 172.21.15.27:0/3139955022 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b1000cc70 con 0x7f3b240761f0 2024-05-10T11:21:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.858+0000 7f3b20c00700 1 -- 172.21.15.27:0/3139955022 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b100054a0 con 0x7f3b240761f0 2024-05-10T11:21:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.858+0000 7f3b29800700 1 -- 172.21.15.27:0/3139955022 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b241a88e0 con 0x7f3b240761f0 2024-05-10T11:21:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.863+0000 7f3b20c00700 1 -- 172.21.15.27:0/3139955022 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3b10004bd0 con 0x7f3b240761f0 2024-05-10T11:21:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.863+0000 7f3b356e8b80 1 Processor -- start 2024-05-10T11:21:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.863+0000 7f3b356e8b80 1 -- start start 2024-05-10T11:21:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.864+0000 7f3b356e8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506ae20 0x55f56506b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.864+0000 7f3b356e8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506b740 0x55f5650654b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.864+0000 7f3b356e8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f565065ad0 0x55f565065f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.864+0000 7f3b356e8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f564fb7aa0 con 0x55f565065ad0 2024-05-10T11:21:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.864+0000 7f3b356e8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f564f557e0 con 0x55f56506ae20 2024-05-10T11:21:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.864+0000 7f3b356e8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f564fb3a60 con 0x55f56506b740 2024-05-10T11:21:33.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.864+0000 7f3b28e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506ae20 0x55f56506b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.864+0000 7f3b23e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506b740 0x55f5650654b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.864+0000 7f3b28e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506ae20 0x55f56506b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36256/0 (socket says 172.21.15.27:36256) 2024-05-10T11:21:33.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.864+0000 7f3b28e00700 1 -- 172.21.15.27:0/2321424017 learned_addr learned my addr 172.21.15.27:0/2321424017 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:33.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.864+0000 7f3b23400700 1 --2- 172.21.15.27:0/2321424017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f565065ad0 0x55f565065f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.865+0000 7f3b28e00700 1 -- 172.21.15.27:0/2321424017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506b740 msgr2=0x55f5650654b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.865+0000 7f3b28e00700 1 --2- 172.21.15.27:0/2321424017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506b740 0x55f5650654b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.865+0000 7f3b28e00700 1 -- 172.21.15.27:0/2321424017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f565065ad0 msgr2=0x55f565065f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.865+0000 7f3b28e00700 1 --2- 172.21.15.27:0/2321424017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f565065ad0 0x55f565065f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.865+0000 7f3b28e00700 1 -- 172.21.15.27:0/2321424017 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f564f04950 con 0x55f56506ae20 2024-05-10T11:21:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.865+0000 7f3b23e00700 1 --2- 172.21.15.27:0/2321424017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506b740 0x55f5650654b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:21:33.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.866+0000 7f3b28e00700 1 --2- 172.21.15.27:0/2321424017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506ae20 0x55f56506b200 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c011680 tx=0x7f3b1c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9138302abae60c49 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:33.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.866+0000 7f3b20c00700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3b080b2a70 0x7f3b080b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.866+0000 7f3b19600700 1 -- 172.21.15.27:0/2321424017 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c019030 con 0x55f56506ae20 2024-05-10T11:21:33.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.866+0000 7f3b19600700 1 -- 172.21.15.27:0/2321424017 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b1c0092e0 con 0x55f56506ae20 2024-05-10T11:21:33.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.866+0000 7f3b19600700 1 -- 172.21.15.27:0/2321424017 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c007500 con 0x55f56506ae20 2024-05-10T11:21:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.866+0000 7f3b20c00700 1 -- 172.21.15.27:0/3139955022 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7f3b100e2b30 con 0x7f3b240761f0 2024-05-10T11:21:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.866+0000 7f3b23e00700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3b080b2a70 0x7f3b080b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.867+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2321424017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506ae20 msgr2=0x55f56506b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.867+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/2321424017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506ae20 0x55f56506b200 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c011680 tx=0x7f3b1c009920 comp rx=0 tx=0).stop 2024-05-10T11:21:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.867+0000 7f3b23e00700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3b080b2a70 0x7f3b080b4f30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3b140098c0 tx=0x7f3b14019040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.867+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2321424017 shutdown_connections 2024-05-10T11:21:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.867+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/2321424017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f565065ad0 0x55f565065f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.867+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/2321424017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506ae20 0x55f56506b200 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.867+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/2321424017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506b740 0x55f5650654b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.867+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2321424017 >> 172.21.15.27:0/2321424017 conn(0x55f564fdd750 msgr2=0x55f56505fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.868+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2321424017 shutdown_connections 2024-05-10T11:21:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.868+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2321424017 wait complete. 2024-05-10T11:21:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.868+0000 7f3b356e8b80 1 -- start start 2024-05-10T11:21:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.869+0000 7f3b356e8b80 1 Processor -- start 2024-05-10T11:21:33.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.869+0000 7f3b356e8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f565065ad0 0x55f565071410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.869+0000 7f3b356e8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f56506ae20 0x55f56506e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.869+0000 7f3b356e8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506b740 0x55f56506ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.870+0000 7f3b356e8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f5650721d0 con 0x55f56506ae20 2024-05-10T11:21:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.870+0000 7f3b356e8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f564f1dba0 con 0x55f56506b740 2024-05-10T11:21:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.870+0000 7f3b356e8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f565072350 con 0x55f565065ad0 2024-05-10T11:21:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.870+0000 7f3b356e8b80 10 client.0 _ll_register_callbacks cb 0x55f564f5ecf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.870+0000 7f3b28e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506b740 0x55f56506ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.870+0000 7f3b28e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506b740 0x55f56506ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36260/0 (socket says 172.21.15.27:36260) 2024-05-10T11:21:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.870+0000 7f3b28e00700 1 -- 172.21.15.27:0/2254336704 learned_addr learned my addr 172.21.15.27:0/2254336704 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.870+0000 7f3b23400700 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f56506ae20 0x55f56506e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.870+0000 7f3b23e00700 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f565065ad0 0x55f565071410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.871+0000 7f3b28e00700 1 -- 172.21.15.27:0/2254336704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f565065ad0 msgr2=0x55f565071410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.871+0000 7f3b28e00700 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f565065ad0 0x55f565071410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.871+0000 7f3b28e00700 1 -- 172.21.15.27:0/2254336704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f56506ae20 msgr2=0x55f56506e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.871+0000 7f3b28e00700 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f56506ae20 0x55f56506e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.871+0000 7f3b28e00700 1 -- 172.21.15.27:0/2254336704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f564f7d4a0 con 0x55f56506b740 2024-05-10T11:21:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.871+0000 7f3b23400700 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f56506ae20 0x55f56506e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:21:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.871+0000 7f3b28e00700 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506b740 0x55f56506ecf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0045b0 tx=0x7f3b1c0094d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5cce14ca486b168 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.871+0000 7f3b0e000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:21:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.872+0000 7f3b0e000700 1 -- 172.21.15.27:0/2254336704 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c023070 con 0x55f56506b740 2024-05-10T11:21:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.872+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2254336704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b1c0092e0 con 0x55f56506b740 2024-05-10T11:21:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.872+0000 7f3b0e000700 1 -- 172.21.15.27:0/2254336704 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7f3b1c003ed0 con 0x55f56506b740 2024-05-10T11:21:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.872+0000 7f3b0e000700 1 -- 172.21.15.27:0/2254336704 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3b1c031050 con 0x55f56506b740 2024-05-10T11:21:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.872+0000 7f3b356e8b80 10 client.26528 fetch_fsmap learned FSMap version 34 2024-05-10T11:21:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.872+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2254336704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55f5650851e0 con 0x55f56506b740 2024-05-10T11:21:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.872+0000 7f3b0e000700 1 -- 172.21.15.27:0/2254336704 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b1c0092e0 con 0x55f56506b740 2024-05-10T11:21:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.872+0000 7f3b0e000700 1 -- 172.21.15.27:0/2254336704 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f3b1c028580 con 0x55f56506b740 2024-05-10T11:21:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.873+0000 7f3b356e8b80 10 client.26528 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:21:33.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.873+0000 7f3b356e8b80 10 client.26528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:21:33.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.873+0000 7f3b356e8b80 20 client.26528 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:33.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.873+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f565087760 0x55f565089b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.873+0000 7f3b356e8b80 4 client.26528 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:21:33.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.873+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2254336704 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5650321c0 con 0x55f565087760 2024-05-10T11:21:33.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.873+0000 7f3b23e00700 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f565087760 0x55f565089b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.874+0000 7f3b23e00700 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f565087760 0x55f565089b50 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=767da5540bed8811 server_cookie=a3344cf370814bd6 in_seq=0 out_seq=0 2024-05-10T11:21:33.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.874+0000 7f3b0e000700 10 client.26528 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:33.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.874+0000 7f3b0e000700 1 -- 172.21.15.27:0/2254336704 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f5650321c0 con 0x55f565087760 2024-05-10T11:21:33.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.874+0000 7f3b0e000700 10 client.26528 handle_command_reply: tid=0 2024-05-10T11:21:33.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b356e8b80 1 client.26528 shutdown 2024-05-10T11:21:33.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2254336704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506b740 msgr2=0x55f56506ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:33.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506b740 0x55f56506ecf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0045b0 tx=0x7f3b1c0094d0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2254336704 shutdown_connections 2024-05-10T11:21:33.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f565087760 0x55f565089b50 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f565065ad0 0x55f565071410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f56506b740 0x55f56506ecf0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/2254336704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f56506ae20 0x55f56506e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2254336704 >> 172.21.15.27:0/2254336704 conn(0x55f564fdd750 msgr2=0x55f56505a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:33.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b0e000700 0 client.26528 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:33.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2254336704 shutdown_connections 2024-05-10T11:21:33.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b356e8b80 1 -- 172.21.15.27:0/2254336704 wait complete. 2024-05-10T11:21:33.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.875+0000 7f3b356e8b80 20 client.26528 trim_cache size 0 max 16384 2024-05-10T11:21:33.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.900+0000 7f3b356e8b80 1 Processor -- start 2024-05-10T11:21:33.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.900+0000 7f3b356e8b80 1 -- start start 2024-05-10T11:21:33.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b356e8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f564fdd750 0x55f5650654b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b356e8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f565065ad0 0x55f56505a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b356e8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 0x55f56508cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b356e8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3b241ad530 con 0x55f565065ad0 2024-05-10T11:21:33.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b356e8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3b241af0a0 con 0x55f564fdd750 2024-05-10T11:21:33.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b356e8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3b241adc00 con 0x55f56506ae20 2024-05-10T11:21:33.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b28e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 0x55f56508cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b23e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f564fdd750 0x55f5650654b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b23400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f565065ad0 0x55f56505a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b28e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 0x55f56508cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60788/0 (socket says 172.21.15.27:60788) 2024-05-10T11:21:33.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b28e00700 1 -- 172.21.15.27:0/489474702 learned_addr learned my addr 172.21.15.27:0/489474702 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b28e00700 1 -- 172.21.15.27:0/489474702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f564fdd750 msgr2=0x55f5650654b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b28e00700 1 --2- 172.21.15.27:0/489474702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f564fdd750 0x55f5650654b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b28e00700 1 -- 172.21.15.27:0/489474702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f565065ad0 msgr2=0x55f56505a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.901+0000 7f3b28e00700 1 --2- 172.21.15.27:0/489474702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f565065ad0 0x55f56505a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.902+0000 7f3b28e00700 1 -- 172.21.15.27:0/489474702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f56505c920 con 0x55f56506ae20 2024-05-10T11:21:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.902+0000 7f3b23e00700 1 --2- 172.21.15.27:0/489474702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f564fdd750 0x55f5650654b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:21:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.902+0000 7f3b28e00700 1 --2- 172.21.15.27:0/489474702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 0x55f56508cb70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c012010 tx=0x7f3b1c006000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=953f24f7f42dfc52 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.902+0000 7f3b18c00700 1 -- 172.21.15.27:0/489474702 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c019020 con 0x55f56506ae20 2024-05-10T11:21:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.902+0000 7f3b18c00700 1 -- 172.21.15.27:0/489474702 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b1c0092e0 con 0x55f56506ae20 2024-05-10T11:21:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.902+0000 7f3b18c00700 1 -- 172.21.15.27:0/489474702 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c023070 con 0x55f56506ae20 2024-05-10T11:21:33.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.903+0000 7f3b356e8b80 1 -- 172.21.15.27:0/489474702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 msgr2=0x55f56508cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:33.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.903+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/489474702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 0x55f56508cb70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c012010 tx=0x7f3b1c006000 comp rx=0 tx=0).stop 2024-05-10T11:21:33.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.903+0000 7f3b356e8b80 1 -- 172.21.15.27:0/489474702 shutdown_connections 2024-05-10T11:21:33.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.903+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/489474702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 0x55f56508cb70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.903+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/489474702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f564fdd750 0x55f5650654b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.903+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/489474702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f565065ad0 0x55f56505a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.903+0000 7f3b356e8b80 1 -- 172.21.15.27:0/489474702 >> 172.21.15.27:0/489474702 conn(0x55f56506dc50 msgr2=0x55f56506a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.903+0000 7f3b356e8b80 1 -- 172.21.15.27:0/489474702 shutdown_connections 2024-05-10T11:21:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.903+0000 7f3b356e8b80 1 -- 172.21.15.27:0/489474702 wait complete. 2024-05-10T11:21:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.904+0000 7f3b356e8b80 1 -- start start 2024-05-10T11:21:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.904+0000 7f3b356e8b80 1 Processor -- start 2024-05-10T11:21:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b356e8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f564fdd750 0x55f56508d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b356e8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f565065ad0 0x55f56508b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b356e8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 0x55f56508baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b356e8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3b241ae9d0 con 0x55f564fdd750 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b356e8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3b241ae300 con 0x55f565065ad0 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b356e8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3b241ab510 con 0x55f56506ae20 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b356e8b80 10 client.0 _ll_register_callbacks cb 0x55f564f5ecf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b28e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 0x55f56508baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b23e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f564fdd750 0x55f56508d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b28e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 0x55f56508baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60794/0 (socket says 172.21.15.27:60794) 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b23e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f564fdd750 0x55f56508d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51368/0 (socket says 172.21.15.27:51368) 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b23400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f565065ad0 0x55f56508b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b28e00700 1 -- 172.21.15.27:0/285618052 learned_addr learned my addr 172.21.15.27:0/285618052 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b28e00700 1 -- 172.21.15.27:0/285618052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f565065ad0 msgr2=0x55f56508b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b28e00700 1 --2- 172.21.15.27:0/285618052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f565065ad0 0x55f56508b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b28e00700 1 -- 172.21.15.27:0/285618052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f564fdd750 msgr2=0x55f56508d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b28e00700 1 --2- 172.21.15.27:0/285618052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f564fdd750 0x55f56508d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.905+0000 7f3b28e00700 1 -- 172.21.15.27:0/285618052 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f565066100 con 0x55f56506ae20 2024-05-10T11:21:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.906+0000 7f3b28e00700 1 --2- 172.21.15.27:0/285618052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 0x55f56508baa0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0288f0 tx=0x7f3b1c028cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f65dd9cb3dbd56e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.906+0000 7f3b0e000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:21:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.906+0000 7f3b0e000700 1 -- 172.21.15.27:0/285618052 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c03a060 con 0x55f56506ae20 2024-05-10T11:21:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.906+0000 7f3b0e000700 1 -- 172.21.15.27:0/285618052 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7f3b1c003cd0 con 0x55f56506ae20 2024-05-10T11:21:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.906+0000 7f3b356e8b80 1 -- 172.21.15.27:0/285618052 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b1c0092e0 con 0x55f56506ae20 2024-05-10T11:21:33.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.906+0000 7f3b0e000700 1 -- 172.21.15.27:0/285618052 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3b1c041050 con 0x55f56506ae20 2024-05-10T11:21:33.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.906+0000 7f3b356e8b80 10 client.26515 fetch_fsmap learned FSMap version 34 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.907+0000 7f3b356e8b80 1 -- 172.21.15.27:0/285618052 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55f5650f2b90 con 0x55f56506ae20 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.907+0000 7f3b0e000700 1 -- 172.21.15.27:0/285618052 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b1c0092e0 con 0x55f56506ae20 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.907+0000 7f3b0e000700 1 -- 172.21.15.27:0/285618052 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f3b1c01f650 con 0x55f56506ae20 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.907+0000 7f3b356e8b80 10 client.26515 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.907+0000 7f3b356e8b80 10 client.26515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.907+0000 7f3b356e8b80 20 client.26515 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.907+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/285618052 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f56506b740 0x55f5650f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.907+0000 7f3b356e8b80 4 client.26515 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.907+0000 7f3b356e8b80 1 -- 172.21.15.27:0/285618052 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f564fc5480 con 0x55f56506b740 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.907+0000 7f3b23e00700 1 --2- 172.21.15.27:0/285618052 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f56506b740 0x55f5650f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.908+0000 7f3b23e00700 1 --2- 172.21.15.27:0/285618052 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f56506b740 0x55f5650f51c0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24d6308140bcb15f server_cookie=9aed9c3d5fa50cbd in_seq=0 out_seq=0 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.908+0000 7f3b0e000700 10 client.26515 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.909+0000 7f3b0e000700 1 -- 172.21.15.27:0/285618052 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f564fc5480 con 0x55f56506b740 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.909+0000 7f3b0e000700 10 client.26515 handle_command_reply: tid=0 2024-05-10T11:21:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.909+0000 7f3b356e8b80 1 client.26515 shutdown 2024-05-10T11:21:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.909+0000 7f3b356e8b80 1 -- 172.21.15.27:0/285618052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 msgr2=0x55f56508baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.909+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/285618052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 0x55f56508baa0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0288f0 tx=0x7f3b1c028cf0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.910+0000 7f3b356e8b80 1 -- 172.21.15.27:0/285618052 shutdown_connections 2024-05-10T11:21:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.910+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/285618052 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f56506b740 0x55f5650f51c0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.910+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/285618052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f56506ae20 0x55f56508baa0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.910+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/285618052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f564fdd750 0x55f56508d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.910+0000 7f3b356e8b80 1 --2- 172.21.15.27:0/285618052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f565065ad0 0x55f56508b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.910+0000 7f3b0e000700 0 client.26515 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.910+0000 7f3b356e8b80 1 -- 172.21.15.27:0/285618052 >> 172.21.15.27:0/285618052 conn(0x55f56506dc50 msgr2=0x55f565069990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.910+0000 7f3b356e8b80 1 -- 172.21.15.27:0/285618052 shutdown_connections 2024-05-10T11:21:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.910+0000 7f3b356e8b80 1 -- 172.21.15.27:0/285618052 wait complete. 2024-05-10T11:21:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.911+0000 7f3b356e8b80 20 client.26515 trim_cache size 0 max 16384 2024-05-10T11:21:33.911 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:21:33.911 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:21:33.911 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "04f467e0-e94f-4a02-98e5-e952170330ff", 2024-05-10T11:21:33.911 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:21:33.912 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:21:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.911+0000 7f3b29800700 1 -- 172.21.15.27:0/3139955022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3b080b2a70 msgr2=0x7f3b080b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.911+0000 7f3b29800700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3b080b2a70 0x7f3b080b4f30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3b140098c0 tx=0x7f3b14019040 comp rx=0 tx=0).stop 2024-05-10T11:21:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.911+0000 7f3b29800700 1 -- 172.21.15.27:0/3139955022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b240761f0 msgr2=0x7f3b2419c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.911+0000 7f3b29800700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b240761f0 0x7f3b2419c2f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000e9c0 tx=0x7f3b1000ecd0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.912+0000 7f3b29800700 1 -- 172.21.15.27:0/3139955022 shutdown_connections 2024-05-10T11:21:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.912+0000 7f3b29800700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3b080b2a70 0x7f3b080b4f30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.912+0000 7f3b29800700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b2410fc10 0x7f3b241a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.912+0000 7f3b29800700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240758d0 0x7f3b2419bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.912+0000 7f3b29800700 1 --2- 172.21.15.27:0/3139955022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b240761f0 0x7f3b2419c2f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.912+0000 7f3b29800700 1 -- 172.21.15.27:0/3139955022 >> 172.21.15.27:0/3139955022 conn(0x7f3b240fef20 msgr2=0x7f3b240ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.912+0000 7f3b29800700 1 -- 172.21.15.27:0/3139955022 shutdown_connections 2024-05-10T11:21:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:33.912+0000 7f3b29800700 1 -- 172.21.15.27:0/3139955022 wait complete. 2024-05-10T11:21:34.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:34 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:21:34.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:34 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:34.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:34 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:21:34.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:34 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:34.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:34 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:21:34.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:34 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:34.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:34 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:21:34.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:34 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:34.705 INFO:tasks.workunit.client.0.smithi027.stdout: 1 480612 6.58 MB/sec execute 259 sec latency 39.719 ms 2024-05-10T11:21:34.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:34 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:21:34.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:34 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:34 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:21:34.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:34 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:35.095 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:21:35.574 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:21:35.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:35 smithi027 ceph-mon[30054]: pgmap v257: 129 pgs: 129 active+clean; 307 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-05-10T11:21:35.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:35 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:21:35.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:35 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:35.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:35 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:21:35.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:35 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:21:35.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:35 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:35.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:35 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:35 smithi076 ceph-mon[34152]: pgmap v257: 129 pgs: 129 active+clean; 307 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-05-10T11:21:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:35 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:21:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:35 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:35 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:21:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:35 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:21:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:35 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:35 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:35.705 INFO:tasks.workunit.client.0.smithi027.stdout: 1 482147 6.58 MB/sec execute 260 sec latency 47.730 ms 2024-05-10T11:21:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:35 smithi189 ceph-mon[34842]: pgmap v257: 129 pgs: 129 active+clean; 307 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-05-10T11:21:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:35 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:21:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:35 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:35 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:21:35.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:35 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:21:35.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:35 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:35.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:35 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:21:36.705 INFO:tasks.workunit.client.0.smithi027.stdout: 1 483591 6.60 MB/sec execute 261 sec latency 15.166 ms 2024-05-10T11:21:37.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:37 smithi027 ceph-mon[30054]: pgmap v258: 129 pgs: 129 active+clean; 312 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 210 op/s 2024-05-10T11:21:37.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:37 smithi076 ceph-mon[34152]: pgmap v258: 129 pgs: 129 active+clean; 312 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 210 op/s 2024-05-10T11:21:37.705 INFO:tasks.workunit.client.0.smithi027.stdout: 1 484653 6.59 MB/sec execute 262 sec latency 42.817 ms 2024-05-10T11:21:37.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:37 smithi189 ceph-mon[34842]: pgmap v258: 129 pgs: 129 active+clean; 312 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 210 op/s 2024-05-10T11:21:38.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.695+0000 7f5204400700 1 -- 172.21.15.27:0/4044389797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51fc103eb0 msgr2=0x7f51fc110390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.695+0000 7f5204400700 1 --2- 172.21.15.27:0/4044389797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51fc103eb0 0x7f51fc110390 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f51f400b0a0 tx=0x7f51f400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.696+0000 7f5204400700 1 -- 172.21.15.27:0/4044389797 shutdown_connections 2024-05-10T11:21:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.696+0000 7f5204400700 1 --2- 172.21.15.27:0/4044389797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51fc109600 0x7f51fc1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.696+0000 7f5204400700 1 --2- 172.21.15.27:0/4044389797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51fc103eb0 0x7f51fc110390 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.696+0000 7f5204400700 1 --2- 172.21.15.27:0/4044389797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc1034f0 0x7f51fc103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.696+0000 7f5204400700 1 -- 172.21.15.27:0/4044389797 >> 172.21.15.27:0/4044389797 conn(0x7f51fc07a650 msgr2=0x7f51fc07aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.696+0000 7f5204400700 1 -- 172.21.15.27:0/4044389797 shutdown_connections 2024-05-10T11:21:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.696+0000 7f5204400700 1 -- 172.21.15.27:0/4044389797 wait complete. 2024-05-10T11:21:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.697+0000 7f5204400700 1 Processor -- start 2024-05-10T11:21:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.697+0000 7f5204400700 1 -- start start 2024-05-10T11:21:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.698+0000 7f5204400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51fc1034f0 0x7f51fc19bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.698+0000 7f5204400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc103eb0 0x7f51fc19c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.698+0000 7f5204400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51fc109600 0x7f51fc1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.698+0000 7f5204400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc1136a0 con 0x7f51fc103eb0 2024-05-10T11:21:38.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.698+0000 7f5204400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc113520 con 0x7f51fc1034f0 2024-05-10T11:21:38.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.698+0000 7f5204400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc113820 con 0x7f51fc109600 2024-05-10T11:21:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.698+0000 7f5201a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51fc1034f0 0x7f51fc19bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.698+0000 7f5201000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc103eb0 0x7f51fc19c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.698+0000 7f5201a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51fc1034f0 0x7f51fc19bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36970/0 (socket says 172.21.15.27:36970) 2024-05-10T11:21:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.698+0000 7f5201a00700 1 -- 172.21.15.27:0/2910383170 learned_addr learned my addr 172.21.15.27:0/2910383170 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.698+0000 7f5202400700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51fc109600 0x7f51fc1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.699+0000 7f5201a00700 1 -- 172.21.15.27:0/2910383170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51fc109600 msgr2=0x7f51fc1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.699+0000 7f5201a00700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51fc109600 0x7f51fc1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.699+0000 7f5201a00700 1 -- 172.21.15.27:0/2910383170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc103eb0 msgr2=0x7f51fc19c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.699+0000 7f5201a00700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc103eb0 0x7f51fc19c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.699+0000 7f5201a00700 1 -- 172.21.15.27:0/2910383170 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51fc1a0d70 con 0x7f51fc1034f0 2024-05-10T11:21:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.699+0000 7f5202400700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51fc109600 0x7f51fc1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:21:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.699+0000 7f5201a00700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51fc1034f0 0x7f51fc19bd60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f51f000e9a0 tx=0x7f51f000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.699+0000 7f51fa000700 1 -- 172.21.15.27:0/2910383170 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f000ca10 con 0x7f51fc1034f0 2024-05-10T11:21:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.700+0000 7f51fa000700 1 -- 172.21.15.27:0/2910383170 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51f000cbb0 con 0x7f51fc1034f0 2024-05-10T11:21:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.700+0000 7f51fa000700 1 -- 172.21.15.27:0/2910383170 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f0018680 con 0x7f51fc1034f0 2024-05-10T11:21:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.700+0000 7f5204400700 1 -- 172.21.15.27:0/2910383170 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51fc1a1060 con 0x7f51fc1034f0 2024-05-10T11:21:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.700+0000 7f5204400700 1 -- 172.21.15.27:0/2910383170 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51fc1a88e0 con 0x7f51fc1034f0 2024-05-10T11:21:38.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.705+0000 7f51fa000700 1 -- 172.21.15.27:0/2910383170 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f51f0010430 con 0x7f51fc1034f0 2024-05-10T11:21:38.705 INFO:tasks.workunit.client.0.smithi027.stdout: 1 486215 6.60 MB/sec execute 263 sec latency 34.292 ms 2024-05-10T11:21:38.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.707+0000 7f520ecf9b80 1 Processor -- start 2024-05-10T11:21:38.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.707+0000 7f520ecf9b80 1 -- start start 2024-05-10T11:21:38.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.707+0000 7f520ecf9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b25b3060 0x5644b25b3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.707+0000 7f520ecf9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3980 0x5644b25b7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.707+0000 7f520ecf9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b79d0 0x5644b25b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.707+0000 7f520ecf9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5644b2505a20 con 0x5644b25b3980 2024-05-10T11:21:38.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.707+0000 7f520ecf9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5644b24a3760 con 0x5644b25b3060 2024-05-10T11:21:38.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.707+0000 7f520ecf9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5644b25019e0 con 0x5644b25b79d0 2024-05-10T11:21:38.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.708+0000 7f5202400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b25b3060 0x5644b25b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.708+0000 7f5201000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b79d0 0x5644b25b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.708+0000 7f5202400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b25b3060 0x5644b25b3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36978/0 (socket says 172.21.15.27:36978) 2024-05-10T11:21:38.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.708+0000 7f5202400700 1 -- 172.21.15.27:0/187478374 learned_addr learned my addr 172.21.15.27:0/187478374 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:38.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.708+0000 7f5201a00700 1 --2- 172.21.15.27:0/187478374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3980 0x5644b25b7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.708+0000 7f5201000700 1 -- 172.21.15.27:0/187478374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b25b3060 msgr2=0x5644b25b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:38.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.708+0000 7f5201000700 1 --2- 172.21.15.27:0/187478374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b25b3060 0x5644b25b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.708+0000 7f5201000700 1 -- 172.21.15.27:0/187478374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3980 msgr2=0x5644b25b7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:38.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.708+0000 7f5201000700 1 --2- 172.21.15.27:0/187478374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3980 0x5644b25b7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.708+0000 7f5201000700 1 -- 172.21.15.27:0/187478374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644b24528d0 con 0x5644b25b79d0 2024-05-10T11:21:38.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.708+0000 7f5201a00700 1 --2- 172.21.15.27:0/187478374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3980 0x5644b25b7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:21:38.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.708+0000 7f51fa000700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f51e40b2b40 0x7f51e40b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.709+0000 7f5201000700 1 --2- 172.21.15.27:0/187478374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b79d0 0x5644b25b7e30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009c80 tx=0x7f51e8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16e0d5122a0667f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.709+0000 7f51fa000700 1 -- 172.21.15.27:0/2910383170 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7f51f00e2d10 con 0x7f51fc1034f0 2024-05-10T11:21:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.709+0000 7f5202400700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f51e40b2b40 0x7f51e40b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.709+0000 7f51eea00700 1 -- 172.21.15.27:0/187478374 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e8011cb0 con 0x5644b25b79d0 2024-05-10T11:21:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.709+0000 7f51eea00700 1 -- 172.21.15.27:0/187478374 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e8011e50 con 0x5644b25b79d0 2024-05-10T11:21:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.709+0000 7f51eea00700 1 -- 172.21.15.27:0/187478374 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e8005220 con 0x5644b25b79d0 2024-05-10T11:21:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.710+0000 7f520ecf9b80 1 -- 172.21.15.27:0/187478374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b79d0 msgr2=0x5644b25b7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.710+0000 7f5202400700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f51e40b2b40 0x7f51e40b5000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009e60 tx=0x7f51f401d040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:38.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.710+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/187478374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b79d0 0x5644b25b7e30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009c80 tx=0x7f51e8009f90 comp rx=0 tx=0).stop 2024-05-10T11:21:38.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.710+0000 7f520ecf9b80 1 -- 172.21.15.27:0/187478374 shutdown_connections 2024-05-10T11:21:38.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.710+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/187478374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b79d0 0x5644b25b7e30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.710+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/187478374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b25b3060 0x5644b25b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.710+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/187478374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3980 0x5644b25b7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.710+0000 7f520ecf9b80 1 -- 172.21.15.27:0/187478374 >> 172.21.15.27:0/187478374 conn(0x5644b24e1b60 msgr2=0x5644b25ad7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:38.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.710+0000 7f520ecf9b80 1 -- 172.21.15.27:0/187478374 shutdown_connections 2024-05-10T11:21:38.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.711+0000 7f520ecf9b80 1 -- 172.21.15.27:0/187478374 wait complete. 2024-05-10T11:21:38.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.711+0000 7f520ecf9b80 1 -- start start 2024-05-10T11:21:38.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.711+0000 7f520ecf9b80 1 Processor -- start 2024-05-10T11:21:38.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.712+0000 7f520ecf9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 0x5644b25bedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.712+0000 7f520ecf9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b25b3980 0x5644b25bc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.712+0000 7f520ecf9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b79d0 0x5644b25bc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.712+0000 7f520ecf9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5644b25bfbe0 con 0x5644b25b79d0 2024-05-10T11:21:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.712+0000 7f520ecf9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5644b246bb20 con 0x5644b25b3980 2024-05-10T11:21:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.712+0000 7f520ecf9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5644b25bfd60 con 0x5644b25b3060 2024-05-10T11:21:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.712+0000 7f520ecf9b80 10 client.0 _ll_register_callbacks cb 0x5644b24acc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.712+0000 7f5201000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b25b3980 0x5644b25bc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.712+0000 7f5201a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 0x5644b25bedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.712+0000 7f5201000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b25b3980 0x5644b25bc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36992/0 (socket says 172.21.15.27:36992) 2024-05-10T11:21:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.712+0000 7f5201000700 1 -- 172.21.15.27:0/1363260679 learned_addr learned my addr 172.21.15.27:0/1363260679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.712+0000 7f5202400700 1 --2- 172.21.15.27:0/1363260679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b79d0 0x5644b25bc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.713+0000 7f5201a00700 1 -- 172.21.15.27:0/1363260679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b25b3980 msgr2=0x5644b25bc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.713+0000 7f5201a00700 1 --2- 172.21.15.27:0/1363260679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b25b3980 0x5644b25bc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.713+0000 7f5201a00700 1 -- 172.21.15.27:0/1363260679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b79d0 msgr2=0x5644b25bc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.713+0000 7f5201a00700 1 --2- 172.21.15.27:0/1363260679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b79d0 0x5644b25bc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.713+0000 7f5201a00700 1 -- 172.21.15.27:0/1363260679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644b24cb420 con 0x5644b25b3060 2024-05-10T11:21:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.713+0000 7f5201a00700 1 --2- 172.21.15.27:0/1363260679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 0x5644b25bedb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f51f00dffd0 tx=0x7f51f00a53e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3a4ad6eb25cb48c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.713+0000 7f51e3400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:21:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.713+0000 7f51e3400700 1 -- 172.21.15.27:0/1363260679 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f00a9b60 con 0x5644b25b3060 2024-05-10T11:21:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.713+0000 7f51e3400700 1 -- 172.21.15.27:0/1363260679 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7f51f00a9fd0 con 0x5644b25b3060 2024-05-10T11:21:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.713+0000 7f520ecf9b80 1 -- 172.21.15.27:0/1363260679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e8011e50 con 0x5644b25b3060 2024-05-10T11:21:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.714+0000 7f51e3400700 1 -- 172.21.15.27:0/1363260679 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f51f00adf10 con 0x5644b25b3060 2024-05-10T11:21:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.714+0000 7f520ecf9b80 10 client.26527 fetch_fsmap learned FSMap version 34 2024-05-10T11:21:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.714+0000 7f520ecf9b80 1 -- 172.21.15.27:0/1363260679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5644b25ca3c0 con 0x5644b25b3060 2024-05-10T11:21:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.714+0000 7f51e3400700 1 -- 172.21.15.27:0/1363260679 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e8011e50 con 0x5644b25b3060 2024-05-10T11:21:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.714+0000 7f51e3400700 1 -- 172.21.15.27:0/1363260679 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f51f00e7040 con 0x5644b25b3060 2024-05-10T11:21:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.714+0000 7f520ecf9b80 10 client.26527 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:21:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.714+0000 7f520ecf9b80 10 client.26527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:21:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.714+0000 7f520ecf9b80 20 client.26527 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.714+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/1363260679 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5644b25cca00 0x5644b25cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.714+0000 7f520ecf9b80 4 client.26527 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:21:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.714+0000 7f520ecf9b80 1 -- 172.21.15.27:0/1363260679 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644b25800a0 con 0x5644b25cca00 2024-05-10T11:21:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.715+0000 7f5201000700 1 --2- 172.21.15.27:0/1363260679 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5644b25cca00 0x5644b25cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.715+0000 7f5201000700 1 --2- 172.21.15.27:0/1363260679 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5644b25cca00 0x5644b25cedf0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a618130454b342b7 server_cookie=ffeca52e12f26398 in_seq=0 out_seq=0 2024-05-10T11:21:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.715+0000 7f51e3400700 10 client.26527 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.716+0000 7f51e3400700 1 -- 172.21.15.27:0/1363260679 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5644b25800a0 con 0x5644b25cca00 2024-05-10T11:21:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.716+0000 7f51e3400700 10 client.26527 handle_command_reply: tid=0 2024-05-10T11:21:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.716+0000 7f520ecf9b80 1 client.26527 shutdown 2024-05-10T11:21:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.716+0000 7f520ecf9b80 1 -- 172.21.15.27:0/1363260679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 msgr2=0x5644b25bedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.716+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/1363260679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 0x5644b25bedb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f51f00dffd0 tx=0x7f51f00a53e0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.716+0000 7f520ecf9b80 1 -- 172.21.15.27:0/1363260679 shutdown_connections 2024-05-10T11:21:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.716+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/1363260679 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5644b25cca00 0x5644b25cedf0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.716+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/1363260679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 0x5644b25bedb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.716+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/1363260679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b79d0 0x5644b25bc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.716+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/1363260679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b25b3980 0x5644b25bc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.717+0000 7f520ecf9b80 1 -- 172.21.15.27:0/1363260679 >> 172.21.15.27:0/1363260679 conn(0x5644b24e1b60 msgr2=0x5644b25a82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.717+0000 7f51e3400700 0 client.26527 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.717+0000 7f520ecf9b80 1 -- 172.21.15.27:0/1363260679 shutdown_connections 2024-05-10T11:21:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.717+0000 7f520ecf9b80 1 -- 172.21.15.27:0/1363260679 wait complete. 2024-05-10T11:21:38.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.717+0000 7f520ecf9b80 20 client.26527 trim_cache size 0 max 16384 2024-05-10T11:21:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.744+0000 7f520ecf9b80 1 Processor -- start 2024-05-10T11:21:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.744+0000 7f520ecf9b80 1 -- start start 2024-05-10T11:21:38.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.744+0000 7f520ecf9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b24e1b60 0x5644b25c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.744+0000 7f520ecf9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3060 0x5644b25a8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.744+0000 7f520ecf9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3980 0x5644b25d4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.744+0000 7f520ecf9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc1b19d0 con 0x5644b25b3060 2024-05-10T11:21:38.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.744+0000 7f520ecf9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc1b3540 con 0x5644b24e1b60 2024-05-10T11:21:38.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.744+0000 7f520ecf9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc1b20a0 con 0x5644b25b3980 2024-05-10T11:21:38.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.745+0000 7f5201a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b24e1b60 0x5644b25c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.745+0000 7f5201a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b24e1b60 0x5644b25c8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36994/0 (socket says 172.21.15.27:36994) 2024-05-10T11:21:38.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.745+0000 7f5201a00700 1 -- 172.21.15.27:0/3212196693 learned_addr learned my addr 172.21.15.27:0/3212196693 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:38.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.745+0000 7f5202400700 1 --2- 172.21.15.27:0/3212196693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3980 0x5644b25d4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.745+0000 7f5201000700 1 --2- 172.21.15.27:0/3212196693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3060 0x5644b25a8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.745+0000 7f5201a00700 1 -- 172.21.15.27:0/3212196693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3980 msgr2=0x5644b25d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:38.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.745+0000 7f5201a00700 1 --2- 172.21.15.27:0/3212196693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3980 0x5644b25d4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.745+0000 7f5201a00700 1 -- 172.21.15.27:0/3212196693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3060 msgr2=0x5644b25a8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:38.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.745+0000 7f5201a00700 1 --2- 172.21.15.27:0/3212196693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3060 0x5644b25a8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.745+0000 7f5201a00700 1 -- 172.21.15.27:0/3212196693 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644b25b7800 con 0x5644b24e1b60 2024-05-10T11:21:38.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.746+0000 7f5201a00700 1 --2- 172.21.15.27:0/3212196693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b24e1b60 0x5644b25c8c70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f51f00a9af0 tx=0x7f51f00e3000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6063148b9028fdf0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:38.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.746+0000 7f51ee000700 1 -- 172.21.15.27:0/3212196693 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f00e5070 con 0x5644b24e1b60 2024-05-10T11:21:38.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.746+0000 7f51ee000700 1 -- 172.21.15.27:0/3212196693 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51f00b29c0 con 0x5644b24e1b60 2024-05-10T11:21:38.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.746+0000 7f51ee000700 1 -- 172.21.15.27:0/3212196693 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f00b2790 con 0x5644b24e1b60 2024-05-10T11:21:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.747+0000 7f520ecf9b80 1 -- 172.21.15.27:0/3212196693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b24e1b60 msgr2=0x5644b25c8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.747+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/3212196693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b24e1b60 0x5644b25c8c70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f51f00a9af0 tx=0x7f51f00e3000 comp rx=0 tx=0).stop 2024-05-10T11:21:38.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.747+0000 7f520ecf9b80 1 -- 172.21.15.27:0/3212196693 shutdown_connections 2024-05-10T11:21:38.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.747+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/3212196693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3980 0x5644b25d4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.747+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/3212196693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b24e1b60 0x5644b25c8c70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.747+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/3212196693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3060 0x5644b25a8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.747+0000 7f520ecf9b80 1 -- 172.21.15.27:0/3212196693 >> 172.21.15.27:0/3212196693 conn(0x5644b25a7610 msgr2=0x5644b25a7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:38.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.747+0000 7f520ecf9b80 1 -- 172.21.15.27:0/3212196693 shutdown_connections 2024-05-10T11:21:38.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.747+0000 7f520ecf9b80 1 -- 172.21.15.27:0/3212196693 wait complete. 2024-05-10T11:21:38.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.748+0000 7f520ecf9b80 1 -- start start 2024-05-10T11:21:38.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.748+0000 7f520ecf9b80 1 Processor -- start 2024-05-10T11:21:38.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.748+0000 7f520ecf9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b24e1b60 0x5644b25b6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f520ecf9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 0x5644b25b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f520ecf9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3980 0x5644b25b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f520ecf9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc1b2e70 con 0x5644b25b3980 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f520ecf9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc1b27a0 con 0x5644b24e1b60 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f520ecf9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc1af990 con 0x5644b25b3060 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f520ecf9b80 10 client.0 _ll_register_callbacks cb 0x5644b24acc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f5201000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 0x5644b25b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f5201000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 0x5644b25b6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55848/0 (socket says 172.21.15.27:55848) 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f5201000700 1 -- 172.21.15.27:0/2315846544 learned_addr learned my addr 172.21.15.27:0/2315846544 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f5202400700 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3980 0x5644b25b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f5201000700 1 -- 172.21.15.27:0/2315846544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b24e1b60 msgr2=0x5644b25b6110 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f5201000700 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b24e1b60 0x5644b25b6110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f5201000700 1 -- 172.21.15.27:0/2315846544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3980 msgr2=0x5644b25b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:21:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f5201000700 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3980 0x5644b25b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f5201000700 1 -- 172.21.15.27:0/2315846544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644b25aa3f0 con 0x5644b25b3060 2024-05-10T11:21:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.749+0000 7f5202400700 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3980 0x5644b25b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:21:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.750+0000 7f5201000700 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 0x5644b25b6650 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f51e8000c00 tx=0x7f51e8011c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15372ee95ef56369 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:21:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.750+0000 7f51e3400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:21:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.750+0000 7f51e3400700 1 -- 172.21.15.27:0/2315846544 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e80049e0 con 0x5644b25b3060 2024-05-10T11:21:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.750+0000 7f520ecf9b80 1 -- 172.21.15.27:0/2315846544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51f00b29c0 con 0x5644b25b3060 2024-05-10T11:21:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.750+0000 7f51e3400700 1 -- 172.21.15.27:0/2315846544 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9591+0+0 (secure 0 0 0) 0x7f51e8004050 con 0x5644b25b3060 2024-05-10T11:21:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.750+0000 7f51e3400700 1 -- 172.21.15.27:0/2315846544 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f51e803a050 con 0x5644b25b3060 2024-05-10T11:21:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.750+0000 7f520ecf9b80 10 client.26533 fetch_fsmap learned FSMap version 34 2024-05-10T11:21:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.750+0000 7f520ecf9b80 1 -- 172.21.15.27:0/2315846544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5644b2637550 con 0x5644b25b3060 2024-05-10T11:21:38.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.751+0000 7f51e3400700 1 -- 172.21.15.27:0/2315846544 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51f00b29c0 con 0x5644b25b3060 2024-05-10T11:21:38.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.751+0000 7f51e3400700 1 -- 172.21.15.27:0/2315846544 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f51e8018070 con 0x5644b25b3060 2024-05-10T11:21:38.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.751+0000 7f520ecf9b80 10 client.26533 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:21:38.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.751+0000 7f520ecf9b80 10 client.26533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:21:38.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.751+0000 7f520ecf9b80 20 client.26533 populate_metadata read hostname 'smithi027' 2024-05-10T11:21:38.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.751+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5644b25b79d0 0x5644b2639b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:21:38.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.751+0000 7f520ecf9b80 4 client.26533 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:21:38.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.751+0000 7f520ecf9b80 1 -- 172.21.15.27:0/2315846544 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5644b2513400 con 0x5644b25b79d0 2024-05-10T11:21:38.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.752+0000 7f5201a00700 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5644b25b79d0 0x5644b2639b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:21:38.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.752+0000 7f5201a00700 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5644b25b79d0 0x5644b2639b40 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=227398ab1cb55f server_cookie=823b2e81edeb910 in_seq=0 out_seq=0 2024-05-10T11:21:38.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.752+0000 7f51e3400700 10 client.26533 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.752+0000 7f51e3400700 1 -- 172.21.15.27:0/2315846544 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5644b2513400 con 0x5644b25b79d0 2024-05-10T11:21:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.752+0000 7f51e3400700 10 client.26533 handle_command_reply: tid=0 2024-05-10T11:21:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.752+0000 7f520ecf9b80 1 client.26533 shutdown 2024-05-10T11:21:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.752+0000 7f520ecf9b80 1 -- 172.21.15.27:0/2315846544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 msgr2=0x5644b25b6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.752+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 0x5644b25b6650 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f51e8000c00 tx=0x7f51e8011c80 comp rx=0 tx=0).stop 2024-05-10T11:21:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.753+0000 7f520ecf9b80 1 -- 172.21.15.27:0/2315846544 shutdown_connections 2024-05-10T11:21:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.753+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5644b25b79d0 0x5644b2639b40 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.753+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b25b3980 0x5644b25b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.753+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5644b24e1b60 0x5644b25b6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.753+0000 7f520ecf9b80 1 --2- 172.21.15.27:0/2315846544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644b25b3060 0x5644b25b6650 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.753+0000 7f51e3400700 0 client.26533 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:21:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.753+0000 7f520ecf9b80 1 -- 172.21.15.27:0/2315846544 >> 172.21.15.27:0/2315846544 conn(0x5644b25a7610 msgr2=0x5644b25a82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:38.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.753+0000 7f520ecf9b80 1 -- 172.21.15.27:0/2315846544 shutdown_connections 2024-05-10T11:21:38.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.753+0000 7f520ecf9b80 1 -- 172.21.15.27:0/2315846544 wait complete. 2024-05-10T11:21:38.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.753+0000 7f520ecf9b80 20 client.26533 trim_cache size 0 max 16384 2024-05-10T11:21:38.758 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:21:38.758 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:21:38.759 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:21:38.759 INFO:teuthology.orchestra.run.smithi027.stdout: "04f467e0-e94f-4a02-98e5-e952170330ff": { 2024-05-10T11:21:38.759 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:21:38.759 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "04f467e0-e94f-4a02-98e5-e952170330ff", 2024-05-10T11:21:38.759 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:21:38.759 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:21:38.759 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:21:38.759 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:21:38.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.754+0000 7f5204400700 1 -- 172.21.15.27:0/2910383170 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f51e40b2b40 msgr2=0x7f51e40b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:38.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.754+0000 7f5204400700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f51e40b2b40 0x7f51e40b5000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009e60 tx=0x7f51f401d040 comp rx=0 tx=0).stop 2024-05-10T11:21:38.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.754+0000 7f5204400700 1 -- 172.21.15.27:0/2910383170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51fc1034f0 msgr2=0x7f51fc19bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:21:38.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.754+0000 7f5204400700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51fc1034f0 0x7f51fc19bd60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f51f000e9a0 tx=0x7f51f000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.754+0000 7f5204400700 1 -- 172.21.15.27:0/2910383170 shutdown_connections 2024-05-10T11:21:38.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.754+0000 7f5204400700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f51e40b2b40 0x7f51e40b5000 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.754+0000 7f5204400700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51fc109600 0x7f51fc1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.754+0000 7f5204400700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51fc1034f0 0x7f51fc19bd60 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.754+0000 7f5204400700 1 --2- 172.21.15.27:0/2910383170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc103eb0 0x7f51fc19c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:21:38.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.754+0000 7f5204400700 1 -- 172.21.15.27:0/2910383170 >> 172.21.15.27:0/2910383170 conn(0x7f51fc07a650 msgr2=0x7f51fc10e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:21:38.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.754+0000 7f5204400700 1 -- 172.21.15.27:0/2910383170 shutdown_connections 2024-05-10T11:21:38.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:21:38.754+0000 7f5204400700 1 -- 172.21.15.27:0/2910383170 wait complete. 2024-05-10T11:21:39.611 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:39 smithi027 ceph-mon[30054]: pgmap v259: 129 pgs: 129 active+clean; 312 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 2.1 MiB/s wr, 141 op/s 2024-05-10T11:21:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:39 smithi076 ceph-mon[34152]: pgmap v259: 129 pgs: 129 active+clean; 312 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 2.1 MiB/s wr, 141 op/s 2024-05-10T11:21:39.684 INFO:tasks.cephfs.filesystem:scrub status for tag:04f467e0-e94f-4a02-98e5-e952170330ff - {'path': '/', 'tag': '04f467e0-e94f-4a02-98e5-e952170330ff', 'options': 'recursive,force'} 2024-05-10T11:21:39.705 INFO:tasks.workunit.client.0.smithi027.stdout: 1 487327 6.60 MB/sec execute 264 sec latency 38.427 ms 2024-05-10T11:21:39.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:39 smithi189 ceph-mon[34842]: pgmap v259: 129 pgs: 129 active+clean; 312 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 2.1 MiB/s wr, 141 op/s 2024-05-10T11:21:40.706 INFO:tasks.workunit.client.0.smithi027.stdout: 1 488540 6.59 MB/sec execute 265 sec latency 33.551 ms 2024-05-10T11:21:41.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:41 smithi027 ceph-mon[30054]: pgmap v260: 129 pgs: 129 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-05-10T11:21:41.681 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:41 smithi076 ceph-mon[34152]: pgmap v260: 129 pgs: 129 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-05-10T11:21:41.707 INFO:tasks.workunit.client.0.smithi027.stdout: 1 490510 6.62 MB/sec execute 266 sec latency 19.936 ms 2024-05-10T11:21:41.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:41 smithi189 ceph-mon[34842]: pgmap v260: 129 pgs: 129 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-05-10T11:21:42.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:42 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:21:42.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:42 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:21:42.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:42 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:21:42.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:42 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:21:42.706 INFO:tasks.workunit.client.0.smithi027.stdout: 1 491221 6.60 MB/sec execute 267 sec latency 30.857 ms 2024-05-10T11:21:42.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:42 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:21:42.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:42 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:21:43.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:43 smithi027 ceph-mon[30054]: pgmap v261: 129 pgs: 129 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-05-10T11:21:43.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:43 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:21:43.636 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:43 smithi189 ceph-mon[34842]: pgmap v261: 129 pgs: 129 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-05-10T11:21:43.637 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:43 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:21:43.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:43 smithi076 ceph-mon[34152]: pgmap v261: 129 pgs: 129 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-05-10T11:21:43.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:43 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:21:43.706 INFO:tasks.workunit.client.0.smithi027.stdout: 1 492600 6.60 MB/sec execute 268 sec latency 79.915 ms 2024-05-10T11:21:44.706 INFO:tasks.workunit.client.0.smithi027.stdout: 1 494192 6.62 MB/sec execute 269 sec latency 16.310 ms 2024-05-10T11:21:45.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:45 smithi027 ceph-mon[30054]: pgmap v262: 129 pgs: 129 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-05-10T11:21:45.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:45 smithi076 ceph-mon[34152]: pgmap v262: 129 pgs: 129 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-05-10T11:21:45.707 INFO:tasks.workunit.client.0.smithi027.stdout: 1 495110 6.60 MB/sec execute 270 sec latency 45.853 ms 2024-05-10T11:21:45.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:45 smithi189 ceph-mon[34842]: pgmap v262: 129 pgs: 129 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-05-10T11:21:46.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:46 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:21:46.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:46 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:46.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:46 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:46.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:46 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:46.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:46 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:46.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:46 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:21:46.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:46 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:46.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:46 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:46.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:46 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:46.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:46 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:46.706 INFO:tasks.workunit.client.0.smithi027.stdout: 1 496821 6.61 MB/sec execute 271 sec latency 37.014 ms 2024-05-10T11:21:46.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:46 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:21:46.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:46 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:46.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:46 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:46.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:46 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:46.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:46 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:47.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:47 smithi027 ceph-mon[30054]: pgmap v263: 129 pgs: 129 active+clean; 323 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 187 op/s 2024-05-10T11:21:47.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:47 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:21:47.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:47 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:21:47.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:47 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:47 smithi076 ceph-mon[34152]: pgmap v263: 129 pgs: 129 active+clean; 323 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 187 op/s 2024-05-10T11:21:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:47 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:21:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:47 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:21:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:47 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:47.706 INFO:tasks.workunit.client.0.smithi027.stdout: 1 497771 6.61 MB/sec execute 272 sec latency 18.992 ms 2024-05-10T11:21:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:47 smithi189 ceph-mon[34842]: pgmap v263: 129 pgs: 129 active+clean; 323 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 187 op/s 2024-05-10T11:21:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:47 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:21:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:47 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:21:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:47 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:21:48.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:48 smithi027 ceph-mon[30054]: pgmap v264: 129 pgs: 129 active+clean; 323 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-05-10T11:21:48.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:48 smithi076 ceph-mon[34152]: pgmap v264: 129 pgs: 129 active+clean; 323 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-05-10T11:21:48.707 INFO:tasks.workunit.client.0.smithi027.stdout: 1 498631 6.60 MB/sec execute 273 sec latency 38.663 ms 2024-05-10T11:21:48.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:48 smithi189 ceph-mon[34842]: pgmap v264: 129 pgs: 129 active+clean; 323 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 1.5 MiB/s wr, 120 op/s 2024-05-10T11:21:49.707 INFO:tasks.workunit.client.0.smithi027.stdout: 1 499855 6.59 MB/sec execute 274 sec latency 28.102 ms 2024-05-10T11:21:50.707 INFO:tasks.workunit.client.0.smithi027.stdout: 1 501178 6.61 MB/sec execute 275 sec latency 20.461 ms 2024-05-10T11:21:50.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:50 smithi189 ceph-mon[34842]: pgmap v265: 129 pgs: 129 active+clean; 348 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 192 op/s 2024-05-10T11:21:50.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:50 smithi027 ceph-mon[30054]: pgmap v265: 129 pgs: 129 active+clean; 348 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 192 op/s 2024-05-10T11:21:50.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:50 smithi076 ceph-mon[34152]: pgmap v265: 129 pgs: 129 active+clean; 348 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 192 op/s 2024-05-10T11:21:51.707 INFO:tasks.workunit.client.0.smithi027.stdout: 1 501754 6.59 MB/sec execute 276 sec latency 39.179 ms 2024-05-10T11:21:52.707 INFO:tasks.workunit.client.0.smithi027.stdout: 1 503032 6.58 MB/sec execute 277 sec latency 25.884 ms 2024-05-10T11:21:52.727 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:52 smithi076 ceph-mon[34152]: pgmap v266: 129 pgs: 129 active+clean; 348 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.3 MiB/s wr, 129 op/s 2024-05-10T11:21:52.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:52 smithi189 ceph-mon[34842]: pgmap v266: 129 pgs: 129 active+clean; 348 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.3 MiB/s wr, 129 op/s 2024-05-10T11:21:52.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:52 smithi027 ceph-mon[30054]: pgmap v266: 129 pgs: 129 active+clean; 348 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.3 MiB/s wr, 129 op/s 2024-05-10T11:21:53.707 INFO:tasks.workunit.client.0.smithi027.stdout: 1 504647 6.60 MB/sec execute 278 sec latency 21.231 ms 2024-05-10T11:21:54.707 INFO:tasks.workunit.client.0.smithi027.stdout: 1 505282 6.59 MB/sec execute 279 sec latency 46.896 ms 2024-05-10T11:21:54.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:54 smithi189 ceph-mon[34842]: pgmap v267: 129 pgs: 129 active+clean; 348 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.3 MiB/s wr, 129 op/s 2024-05-10T11:21:54.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:54 smithi027 ceph-mon[30054]: pgmap v267: 129 pgs: 129 active+clean; 348 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.3 MiB/s wr, 129 op/s 2024-05-10T11:21:54.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:54 smithi076 ceph-mon[34152]: pgmap v267: 129 pgs: 129 active+clean; 348 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.3 MiB/s wr, 129 op/s 2024-05-10T11:21:55.707 INFO:tasks.workunit.client.0.smithi027.stdout: 1 506381 6.58 MB/sec execute 280 sec latency 30.877 ms 2024-05-10T11:21:56.707 INFO:tasks.workunit.client.0.smithi027.stdout: 1 508095 6.60 MB/sec execute 281 sec latency 43.952 ms 2024-05-10T11:21:56.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:56 smithi189 ceph-mon[34842]: pgmap v268: 129 pgs: 129 active+clean; 354 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 180 op/s 2024-05-10T11:21:56.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:56 smithi027 ceph-mon[30054]: pgmap v268: 129 pgs: 129 active+clean; 354 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 180 op/s 2024-05-10T11:21:56.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:56 smithi076 ceph-mon[34152]: pgmap v268: 129 pgs: 129 active+clean; 354 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 180 op/s 2024-05-10T11:21:57.708 INFO:tasks.workunit.client.0.smithi027.stdout: 1 508638 6.58 MB/sec execute 282 sec latency 36.936 ms 2024-05-10T11:21:58.708 INFO:tasks.workunit.client.0.smithi027.stdout: 1 509738 6.58 MB/sec execute 283 sec latency 24.038 ms 2024-05-10T11:21:58.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:21:58 smithi189 ceph-mon[34842]: pgmap v269: 129 pgs: 129 active+clean; 354 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 122 op/s 2024-05-10T11:21:58.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:21:58 smithi027 ceph-mon[30054]: pgmap v269: 129 pgs: 129 active+clean; 354 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 122 op/s 2024-05-10T11:21:58.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:21:58 smithi076 ceph-mon[34152]: pgmap v269: 129 pgs: 129 active+clean; 354 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 122 op/s 2024-05-10T11:21:59.708 INFO:tasks.workunit.client.0.smithi027.stdout: 1 511543 6.60 MB/sec execute 284 sec latency 23.540 ms 2024-05-10T11:22:00.708 INFO:tasks.workunit.client.0.smithi027.stdout: 1 512155 6.58 MB/sec execute 285 sec latency 28.944 ms 2024-05-10T11:22:00.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:00 smithi189 ceph-mon[34842]: pgmap v270: 129 pgs: 129 active+clean; 359 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-10T11:22:00.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:00 smithi027 ceph-mon[30054]: pgmap v270: 129 pgs: 129 active+clean; 359 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-10T11:22:00.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:00 smithi076 ceph-mon[34152]: pgmap v270: 129 pgs: 129 active+clean; 359 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-10T11:22:01.708 INFO:tasks.workunit.client.0.smithi027.stdout: 1 512610 6.56 MB/sec execute 286 sec latency 99.193 ms 2024-05-10T11:22:02.709 INFO:tasks.workunit.client.0.smithi027.stdout: 1 513359 6.55 MB/sec execute 287 sec latency 66.756 ms 2024-05-10T11:22:02.873 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:02 smithi027 ceph-mon[30054]: pgmap v271: 129 pgs: 129 active+clean; 359 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 116 op/s 2024-05-10T11:22:02.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:02 smithi076 ceph-mon[34152]: pgmap v271: 129 pgs: 129 active+clean; 359 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 116 op/s 2024-05-10T11:22:02.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:02 smithi189 ceph-mon[34842]: pgmap v271: 129 pgs: 129 active+clean; 359 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 116 op/s 2024-05-10T11:22:03.709 INFO:tasks.workunit.client.0.smithi027.stdout: 1 515214 6.57 MB/sec execute 288 sec latency 20.696 ms 2024-05-10T11:22:03.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:03 smithi027 ceph-mon[30054]: osdmap e96: 12 total, 12 up, 12 in 2024-05-10T11:22:03.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:03 smithi076 ceph-mon[34152]: osdmap e96: 12 total, 12 up, 12 in 2024-05-10T11:22:03.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:03 smithi189 ceph-mon[34842]: osdmap e96: 12 total, 12 up, 12 in 2024-05-10T11:22:04.709 INFO:tasks.workunit.client.0.smithi027.stdout: 1 515869 6.56 MB/sec execute 289 sec latency 35.815 ms 2024-05-10T11:22:04.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:04 smithi027 ceph-mon[30054]: pgmap v273: 129 pgs: 129 active+clean; 359 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-05-10T11:22:04.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:04 smithi076 ceph-mon[34152]: pgmap v273: 129 pgs: 129 active+clean; 359 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-05-10T11:22:04.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:04 smithi189 ceph-mon[34842]: pgmap v273: 129 pgs: 129 active+clean; 359 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-05-10T11:22:05.709 INFO:tasks.workunit.client.0.smithi027.stdout: 1 517103 6.55 MB/sec execute 290 sec latency 31.944 ms 2024-05-10T11:22:06.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:06 smithi027 ceph-mon[30054]: pgmap v274: 129 pgs: 129 active+clean; 361 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-05-10T11:22:06.709 INFO:tasks.workunit.client.0.smithi027.stdout: 1 518764 6.57 MB/sec execute 291 sec latency 44.359 ms 2024-05-10T11:22:06.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:06 smithi076 ceph-mon[34152]: pgmap v274: 129 pgs: 129 active+clean; 361 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-05-10T11:22:06.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:06 smithi189 ceph-mon[34842]: pgmap v274: 129 pgs: 129 active+clean; 361 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-05-10T11:22:07.709 INFO:tasks.workunit.client.0.smithi027.stdout: 1 519405 6.56 MB/sec execute 292 sec latency 38.729 ms 2024-05-10T11:22:08.709 INFO:tasks.workunit.client.0.smithi027.stdout: 1 520499 6.55 MB/sec execute 293 sec latency 31.902 ms 2024-05-10T11:22:08.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:08 smithi027 ceph-mon[30054]: pgmap v275: 129 pgs: 129 active+clean; 361 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-05-10T11:22:08.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:08 smithi076 ceph-mon[34152]: pgmap v275: 129 pgs: 129 active+clean; 361 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-05-10T11:22:08.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:08 smithi189 ceph-mon[34842]: pgmap v275: 129 pgs: 129 active+clean; 361 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-05-10T11:22:09.685 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:22:09.709 INFO:tasks.workunit.client.0.smithi027.stdout: 1 522416 6.57 MB/sec execute 294 sec latency 25.992 ms 2024-05-10T11:22:09.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:09 smithi027 ceph-mon[30054]: osdmap e97: 12 total, 12 up, 12 in 2024-05-10T11:22:09.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:09 smithi076 ceph-mon[34152]: osdmap e97: 12 total, 12 up, 12 in 2024-05-10T11:22:09.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:09 smithi189 ceph-mon[34842]: osdmap e97: 12 total, 12 up, 12 in 2024-05-10T11:22:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:22:10.710 INFO:tasks.workunit.client.0.smithi027.stdout: 1 523205 6.56 MB/sec execute 295 sec latency 45.933 ms 2024-05-10T11:22:10.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:10 smithi027 ceph-mon[30054]: pgmap v277: 129 pgs: 129 active+clean; 367 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 251 op/s 2024-05-10T11:22:10.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:10 smithi076 ceph-mon[34152]: pgmap v277: 129 pgs: 129 active+clean; 367 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 251 op/s 2024-05-10T11:22:10.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:10 smithi189 ceph-mon[34842]: pgmap v277: 129 pgs: 129 active+clean; 367 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 251 op/s 2024-05-10T11:22:11.710 INFO:tasks.workunit.client.0.smithi027.stdout: 1 524584 6.55 MB/sec execute 296 sec latency 79.644 ms 2024-05-10T11:22:12.710 INFO:tasks.workunit.client.0.smithi027.stdout: 1 526028 6.57 MB/sec execute 297 sec latency 35.399 ms 2024-05-10T11:22:12.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.728+0000 7f2e96a00700 1 -- 172.21.15.27:0/687556408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9010fc70 msgr2=0x7f2e90112060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:12.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.728+0000 7f2e96a00700 1 --2- 172.21.15.27:0/687556408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9010fc70 0x7f2e90112060 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2e88009f90 tx=0x7f2e8800b330 comp rx=0 tx=0).stop 2024-05-10T11:22:12.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.728+0000 7f2e96a00700 1 -- 172.21.15.27:0/687556408 shutdown_connections 2024-05-10T11:22:12.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.728+0000 7f2e96a00700 1 --2- 172.21.15.27:0/687556408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e900757c0 0x7f2e90075ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.728+0000 7f2e96a00700 1 --2- 172.21.15.27:0/687556408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9010fc70 0x7f2e90112060 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.728+0000 7f2e96a00700 1 --2- 172.21.15.27:0/687556408 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e900760e0 0x7f2e9010f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.728+0000 7f2e96a00700 1 -- 172.21.15.27:0/687556408 >> 172.21.15.27:0/687556408 conn(0x7f2e900feec0 msgr2=0x7f2e901012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:12.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.728+0000 7f2e96a00700 1 -- 172.21.15.27:0/687556408 shutdown_connections 2024-05-10T11:22:12.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.728+0000 7f2e96a00700 1 -- 172.21.15.27:0/687556408 wait complete. 2024-05-10T11:22:12.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.730+0000 7f2e96a00700 1 Processor -- start 2024-05-10T11:22:12.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.730+0000 7f2e96a00700 1 -- start start 2024-05-10T11:22:12.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e96a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e900757c0 0x7f2e901a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e96a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e900760e0 0x7f2e901a1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e96a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9010fc70 0x7f2e9019ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e96a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2e901157b0 con 0x7f2e9010fc70 2024-05-10T11:22:12.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e96a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2e90115630 con 0x7f2e900760e0 2024-05-10T11:22:12.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e96a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2e90115930 con 0x7f2e900757c0 2024-05-10T11:22:12.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e95600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e900757c0 0x7f2e901a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e94c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e900760e0 0x7f2e901a1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e95600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e900757c0 0x7f2e901a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54796/0 (socket says 172.21.15.27:54796) 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e95600700 1 -- 172.21.15.27:0/1370194433 learned_addr learned my addr 172.21.15.27:0/1370194433 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e96000700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9010fc70 0x7f2e9019ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e95600700 1 -- 172.21.15.27:0/1370194433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e900760e0 msgr2=0x7f2e901a1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e95600700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e900760e0 0x7f2e901a1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e95600700 1 -- 172.21.15.27:0/1370194433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9010fc70 msgr2=0x7f2e9019ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e95600700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9010fc70 0x7f2e9019ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e95600700 1 -- 172.21.15.27:0/1370194433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9019c1f0 con 0x7f2e900757c0 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.731+0000 7f2e96000700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9010fc70 0x7f2e9019ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.732+0000 7f2e95600700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e900757c0 0x7f2e901a18d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400ec10 tx=0x7f2e8400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.732+0000 7f2e8e000700 1 -- 172.21.15.27:0/1370194433 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8400cd40 con 0x7f2e900757c0 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.732+0000 7f2e8e000700 1 -- 172.21.15.27:0/1370194433 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e840044d0 con 0x7f2e900757c0 2024-05-10T11:22:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.732+0000 7f2e96a00700 1 -- 172.21.15.27:0/1370194433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e9019c4e0 con 0x7f2e900757c0 2024-05-10T11:22:12.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.732+0000 7f2e8e000700 1 -- 172.21.15.27:0/1370194433 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e84010690 con 0x7f2e900757c0 2024-05-10T11:22:12.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.732+0000 7f2e96a00700 1 -- 172.21.15.27:0/1370194433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e901a87a0 con 0x7f2e900757c0 2024-05-10T11:22:12.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.738+0000 7f2e8e000700 1 -- 172.21.15.27:0/1370194433 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2e84003e50 con 0x7f2e900757c0 2024-05-10T11:22:12.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.739+0000 7f2ea29b7b80 1 Processor -- start 2024-05-10T11:22:12.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.739+0000 7f2ea29b7b80 1 -- start start 2024-05-10T11:22:12.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2ea29b7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70c9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2ea29b7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 0x55b6e70c3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2ea29b7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c3a50 0x55b6e70c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2ea29b7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b6e7015a20 con 0x55b6e70c8da0 2024-05-10T11:22:12.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2ea29b7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b6e6fb3760 con 0x55b6e70c96c0 2024-05-10T11:22:12.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2ea29b7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b6e70119e0 con 0x55b6e70c3a50 2024-05-10T11:22:12.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2e8e000700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2e780b2a70 0x7f2e780b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2e95600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 0x55b6e70c3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2e95600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 0x55b6e70c3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42352/0 (socket says 172.21.15.27:42352) 2024-05-10T11:22:12.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2e95600700 1 -- 172.21.15.27:0/3195485908 learned_addr learned my addr 172.21.15.27:0/3195485908 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:12.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2e94c00700 1 --2- 172.21.15.27:0/3195485908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c3a50 0x55b6e70c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2e96000700 1 --2- 172.21.15.27:0/3195485908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.740+0000 7f2e8e000700 1 -- 172.21.15.27:0/1370194433 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f2e840e2d20 con 0x7f2e900757c0 2024-05-10T11:22:12.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.741+0000 7f2e96000700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2e780b2a70 0x7f2e780b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.741+0000 7f2e94c00700 1 -- 172.21.15.27:0/3195485908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 msgr2=0x55b6e70c3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:12.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.741+0000 7f2e94c00700 1 --2- 172.21.15.27:0/3195485908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 0x55b6e70c3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.741+0000 7f2e94c00700 1 -- 172.21.15.27:0/3195485908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 msgr2=0x55b6e70c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:12.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.741+0000 7f2e94c00700 1 --2- 172.21.15.27:0/3195485908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70c9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.741+0000 7f2e94c00700 1 -- 172.21.15.27:0/3195485908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6e6f628d0 con 0x55b6e70c3a50 2024-05-10T11:22:12.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.741+0000 7f2e96000700 1 --2- 172.21.15.27:0/3195485908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:22:12.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.741+0000 7f2e95600700 1 --2- 172.21.15.27:0/3195485908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 0x55b6e70c3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:12.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.742+0000 7f2e94c00700 1 --2- 172.21.15.27:0/3195485908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c3a50 0x55b6e70c3ed0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009c80 tx=0x7f2e7c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0e4fed4f09bbd46 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.742+0000 7f2e82a00700 1 -- 172.21.15.27:0/3195485908 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c01e070 con 0x55b6e70c3a50 2024-05-10T11:22:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.742+0000 7f2e82a00700 1 -- 172.21.15.27:0/3195485908 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e7c011cb0 con 0x55b6e70c3a50 2024-05-10T11:22:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.742+0000 7f2e82a00700 1 -- 172.21.15.27:0/3195485908 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c00e6a0 con 0x55b6e70c3a50 2024-05-10T11:22:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.743+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/3195485908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c3a50 msgr2=0x55b6e70c3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.743+0000 7f2e96000700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2e780b2a70 0x7f2e780b4f30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800b6c0 tx=0x7f2e88009920 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.743+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/3195485908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c3a50 0x55b6e70c3ed0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009c80 tx=0x7f2e7c009f90 comp rx=0 tx=0).stop 2024-05-10T11:22:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.743+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/3195485908 shutdown_connections 2024-05-10T11:22:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.743+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/3195485908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c3a50 0x55b6e70c3ed0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.743+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/3195485908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.743+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/3195485908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 0x55b6e70c3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.743+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/3195485908 >> 172.21.15.27:0/3195485908 conn(0x55b6e703b6d0 msgr2=0x55b6e70bdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.743+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/3195485908 shutdown_connections 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.743+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/3195485908 wait complete. 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.744+0000 7f2ea29b7b80 1 -- start start 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.744+0000 7f2ea29b7b80 1 Processor -- start 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.744+0000 7f2ea29b7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c3a50 0x55b6e70cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2ea29b7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2ea29b7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 0x55b6e70ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2ea29b7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b6e70d0150 con 0x55b6e70c8da0 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2ea29b7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b6e6f7bb20 con 0x55b6e70c96c0 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2ea29b7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b6e70d02d0 con 0x55b6e70c3a50 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2ea29b7b80 10 client.0 _ll_register_callbacks cb 0x55b6e6fbcc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2e96000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 0x55b6e70ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2e95600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c3a50 0x55b6e70cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2e95600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c3a50 0x55b6e70cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54816/0 (socket says 172.21.15.27:54816) 2024-05-10T11:22:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2e95600700 1 -- 172.21.15.27:0/132300070 learned_addr learned my addr 172.21.15.27:0/132300070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:12.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2e94c00700 1 --2- 172.21.15.27:0/132300070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2e96000700 1 -- 172.21.15.27:0/132300070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c3a50 msgr2=0x55b6e70cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:12.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2e96000700 1 --2- 172.21.15.27:0/132300070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c3a50 0x55b6e70cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2e96000700 1 -- 172.21.15.27:0/132300070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 msgr2=0x55b6e70cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:12.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2e96000700 1 --2- 172.21.15.27:0/132300070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.745+0000 7f2e96000700 1 -- 172.21.15.27:0/132300070 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6e6fdb420 con 0x55b6e70c96c0 2024-05-10T11:22:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.746+0000 7f2e96000700 1 --2- 172.21.15.27:0/132300070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 0x55b6e70ccc90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2e88015000 tx=0x7f2e8800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30cda49c56ae6019 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.746+0000 7f2e77400700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:22:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.746+0000 7f2e77400700 1 -- 172.21.15.27:0/132300070 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8802d070 con 0x55b6e70c96c0 2024-05-10T11:22:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.746+0000 7f2e77400700 1 -- 172.21.15.27:0/132300070 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f2e88007830 con 0x55b6e70c96c0 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.746+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/132300070 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e7c011cb0 con 0x55b6e70c96c0 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.746+0000 7f2e77400700 1 -- 172.21.15.27:0/132300070 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2e88007a30 con 0x55b6e70c96c0 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.746+0000 7f2ea29b7b80 10 client.26561 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.746+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/132300070 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55b6e70da970 con 0x55b6e70c96c0 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.747+0000 7f2e77400700 1 -- 172.21.15.27:0/132300070 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e7c011cb0 con 0x55b6e70c96c0 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.747+0000 7f2e77400700 1 -- 172.21.15.27:0/132300070 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f2e88011070 con 0x55b6e70c96c0 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.747+0000 7f2ea29b7b80 10 client.26561 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.747+0000 7f2ea29b7b80 10 client.26561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.747+0000 7f2ea29b7b80 20 client.26561 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.747+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/132300070 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55b6e70dcf00 0x55b6e70df2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.747+0000 7f2ea29b7b80 4 client.26561 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.747+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/132300070 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6e7090140 con 0x55b6e70dcf00 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.748+0000 7f2e95600700 1 --2- 172.21.15.27:0/132300070 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55b6e70dcf00 0x55b6e70df2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.748+0000 7f2e95600700 1 --2- 172.21.15.27:0/132300070 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55b6e70dcf00 0x55b6e70df2f0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4cb810245db0d66 server_cookie=7cd0d146ef759bcc in_seq=0 out_seq=0 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.748+0000 7f2e77400700 10 client.26561 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.749+0000 7f2e77400700 1 -- 172.21.15.27:0/132300070 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b6e7090140 con 0x55b6e70dcf00 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.749+0000 7f2e77400700 10 client.26561 handle_command_reply: tid=0 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.749+0000 7f2ea29b7b80 1 client.26561 shutdown 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.749+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/132300070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 msgr2=0x55b6e70ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.749+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/132300070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 0x55b6e70ccc90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2e88015000 tx=0x7f2e8800b330 comp rx=0 tx=0).stop 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.749+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/132300070 shutdown_connections 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.749+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/132300070 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55b6e70dcf00 0x55b6e70df2f0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.749+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/132300070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c3a50 0x55b6e70cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.749+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/132300070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c96c0 0x55b6e70ccc90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.749+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/132300070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.749+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/132300070 >> 172.21.15.27:0/132300070 conn(0x55b6e703b6d0 msgr2=0x55b6e70b8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.749+0000 7f2e77400700 0 client.26561 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.750+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/132300070 shutdown_connections 2024-05-10T11:22:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.750+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/132300070 wait complete. 2024-05-10T11:22:12.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.750+0000 7f2ea29b7b80 20 client.26561 trim_cache size 0 max 16384 2024-05-10T11:22:12.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.775+0000 7f2ea29b7b80 1 Processor -- start 2024-05-10T11:22:12.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.776+0000 7f2ea29b7b80 1 -- start start 2024-05-10T11:22:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.776+0000 7f2ea29b7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e703b6d0 0x55b6e70c3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.776+0000 7f2ea29b7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c3a50 0x55b6e70d91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.776+0000 7f2ea29b7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c8da0 0x55b6e70b8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.776+0000 7f2ea29b7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2e901b17f0 con 0x55b6e70c3a50 2024-05-10T11:22:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.776+0000 7f2ea29b7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2e901b3360 con 0x55b6e703b6d0 2024-05-10T11:22:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.776+0000 7f2ea29b7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2e901b1ec0 con 0x55b6e70c8da0 2024-05-10T11:22:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.776+0000 7f2e95600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e703b6d0 0x55b6e70c3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.776+0000 7f2e96000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c8da0 0x55b6e70b8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.776+0000 7f2e95600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e703b6d0 0x55b6e70c3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42376/0 (socket says 172.21.15.27:42376) 2024-05-10T11:22:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.776+0000 7f2e95600700 1 -- 172.21.15.27:0/993688424 learned_addr learned my addr 172.21.15.27:0/993688424 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.776+0000 7f2e94c00700 1 --2- 172.21.15.27:0/993688424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c3a50 0x55b6e70d91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.777+0000 7f2e95600700 1 -- 172.21.15.27:0/993688424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c8da0 msgr2=0x55b6e70b8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.777+0000 7f2e95600700 1 --2- 172.21.15.27:0/993688424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c8da0 0x55b6e70b8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.777+0000 7f2e95600700 1 -- 172.21.15.27:0/993688424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c3a50 msgr2=0x55b6e70d91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:12.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.777+0000 7f2e95600700 1 --2- 172.21.15.27:0/993688424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c3a50 0x55b6e70d91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.777+0000 7f2e95600700 1 -- 172.21.15.27:0/993688424 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6e70cdbb0 con 0x55b6e703b6d0 2024-05-10T11:22:12.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.777+0000 7f2e96000700 1 --2- 172.21.15.27:0/993688424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c8da0 0x55b6e70b8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:12.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.777+0000 7f2e94c00700 1 --2- 172.21.15.27:0/993688424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c3a50 0x55b6e70d91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:22:12.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.777+0000 7f2e95600700 1 --2- 172.21.15.27:0/993688424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e703b6d0 0x55b6e70c3430 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2e84003730 tx=0x7f2e84076500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5522f5ad7fd42a94 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:12.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.777+0000 7f2e82000700 1 -- 172.21.15.27:0/993688424 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e840a9520 con 0x55b6e703b6d0 2024-05-10T11:22:12.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.777+0000 7f2e82000700 1 -- 172.21.15.27:0/993688424 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e840b2370 con 0x55b6e703b6d0 2024-05-10T11:22:12.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.777+0000 7f2e82000700 1 -- 172.21.15.27:0/993688424 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e840a5360 con 0x55b6e703b6d0 2024-05-10T11:22:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.778+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/993688424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e703b6d0 msgr2=0x55b6e70c3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.778+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/993688424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e703b6d0 0x55b6e70c3430 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2e84003730 tx=0x7f2e84076500 comp rx=0 tx=0).stop 2024-05-10T11:22:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.778+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/993688424 shutdown_connections 2024-05-10T11:22:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.778+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/993688424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e70c8da0 0x55b6e70b8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.778+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/993688424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e703b6d0 0x55b6e70c3430 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.778+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/993688424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c3a50 0x55b6e70d91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.778+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/993688424 >> 172.21.15.27:0/993688424 conn(0x55b6e70bfbe0 msgr2=0x55b6e70c9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:12.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.779+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/993688424 shutdown_connections 2024-05-10T11:22:12.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.779+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/993688424 wait complete. 2024-05-10T11:22:12.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.779+0000 7f2ea29b7b80 1 -- start start 2024-05-10T11:22:12.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.779+0000 7f2ea29b7b80 1 Processor -- start 2024-05-10T11:22:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.780+0000 7f2ea29b7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e703b6d0 0x55b6e70d5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.780+0000 7f2ea29b7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c3a50 0x55b6e70d5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.780+0000 7f2ea29b7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70d2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.780+0000 7f2ea29b7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2e901b2c90 con 0x55b6e70c8da0 2024-05-10T11:22:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.780+0000 7f2ea29b7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2e901b25c0 con 0x55b6e70c3a50 2024-05-10T11:22:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.780+0000 7f2ea29b7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2e901af7b0 con 0x55b6e703b6d0 2024-05-10T11:22:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.780+0000 7f2ea29b7b80 10 client.0 _ll_register_callbacks cb 0x55b6e6fbcc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.780+0000 7f2e95600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e703b6d0 0x55b6e70d5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.780+0000 7f2e95600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e703b6d0 0x55b6e70d5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54840/0 (socket says 172.21.15.27:54840) 2024-05-10T11:22:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.780+0000 7f2e95600700 1 -- 172.21.15.27:0/1049598822 learned_addr learned my addr 172.21.15.27:0/1049598822 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.780+0000 7f2e94c00700 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c3a50 0x55b6e70d5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.780+0000 7f2e96000700 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70d2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.781+0000 7f2e95600700 1 -- 172.21.15.27:0/1049598822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c3a50 msgr2=0x55b6e70d5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.781+0000 7f2e95600700 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c3a50 0x55b6e70d5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.781+0000 7f2e95600700 1 -- 172.21.15.27:0/1049598822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 msgr2=0x55b6e70d2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.781+0000 7f2e95600700 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70d2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.781+0000 7f2e95600700 1 -- 172.21.15.27:0/1049598822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6e70d2a70 con 0x55b6e703b6d0 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.781+0000 7f2e95600700 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e703b6d0 0x55b6e70d5560 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2e84009750 tx=0x7f2e840039b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fdc3bd7084a0c22 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.781+0000 7f2e77400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.781+0000 7f2e77400700 1 -- 172.21.15.27:0/1049598822 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e840c3030 con 0x55b6e703b6d0 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.781+0000 7f2e77400700 1 -- 172.21.15.27:0/1049598822 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f2e840a61c0 con 0x55b6e703b6d0 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.781+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/1049598822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b6e7023400 con 0x55b6e703b6d0 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.782+0000 7f2e77400700 1 -- 172.21.15.27:0/1049598822 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2e840a9340 con 0x55b6e703b6d0 2024-05-10T11:22:12.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.782+0000 7f2ea29b7b80 10 client.26551 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:12.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.782+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/1049598822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55b6e7147ef0 con 0x55b6e703b6d0 2024-05-10T11:22:12.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.782+0000 7f2e77400700 1 -- 172.21.15.27:0/1049598822 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b6e7023400 con 0x55b6e703b6d0 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.782+0000 7f2e77400700 1 -- 172.21.15.27:0/1049598822 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f2e840a5040 con 0x55b6e703b6d0 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.782+0000 7f2ea29b7b80 10 client.26551 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.782+0000 7f2ea29b7b80 10 client.26551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.782+0000 7f2ea29b7b80 20 client.26551 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.782+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55b6e70c96c0 0x55b6e714a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.782+0000 7f2ea29b7b80 4 client.26551 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.782+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/1049598822 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b6e701d4a0 con 0x55b6e70c96c0 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.783+0000 7f2e94c00700 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55b6e70c96c0 0x55b6e714a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.783+0000 7f2e77400700 10 client.26551 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.783+0000 7f2e94c00700 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55b6e70c96c0 0x55b6e714a400 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1732bfdfad118c1 server_cookie=e3790155acacdeae in_seq=0 out_seq=0 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.784+0000 7f2e77400700 1 -- 172.21.15.27:0/1049598822 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b6e701d4a0 con 0x55b6e70c96c0 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.784+0000 7f2e77400700 10 client.26551 handle_command_reply: tid=0 2024-05-10T11:22:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.784+0000 7f2ea29b7b80 1 client.26551 shutdown 2024-05-10T11:22:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.784+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/1049598822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e703b6d0 msgr2=0x55b6e70d5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.784+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e703b6d0 0x55b6e70d5560 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2e84009750 tx=0x7f2e840039b0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.784+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/1049598822 shutdown_connections 2024-05-10T11:22:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.784+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55b6e70c96c0 0x55b6e714a400 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.784+0000 7f2e77400700 0 client.26551 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.784+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6e703b6d0 0x55b6e70d5560 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.784+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b6e70c8da0 0x55b6e70d2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.784+0000 7f2ea29b7b80 1 --2- 172.21.15.27:0/1049598822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b6e70c3a50 0x55b6e70d5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.784+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/1049598822 >> 172.21.15.27:0/1049598822 conn(0x55b6e70bfbe0 msgr2=0x55b6e70c80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:12.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.785+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/1049598822 shutdown_connections 2024-05-10T11:22:12.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.785+0000 7f2ea29b7b80 1 -- 172.21.15.27:0/1049598822 wait complete. 2024-05-10T11:22:12.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.785+0000 7f2ea29b7b80 20 client.26551 trim_cache size 0 max 16384 2024-05-10T11:22:12.786 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:22:12.786 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:22:12.786 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:22:12.786 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:22:12.787 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:12 smithi027 ceph-mon[30054]: pgmap v278: 129 pgs: 129 active+clean; 367 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 225 op/s 2024-05-10T11:22:12.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.786+0000 7f2e96a00700 1 -- 172.21.15.27:0/1370194433 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2e780b2a70 msgr2=0x7f2e780b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:12.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.786+0000 7f2e96a00700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2e780b2a70 0x7f2e780b4f30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800b6c0 tx=0x7f2e88009920 comp rx=0 tx=0).stop 2024-05-10T11:22:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.786+0000 7f2e96a00700 1 -- 172.21.15.27:0/1370194433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e900757c0 msgr2=0x7f2e901a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.786+0000 7f2e96a00700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e900757c0 0x7f2e901a18d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400ec10 tx=0x7f2e8400ef20 comp rx=0 tx=0).stop 2024-05-10T11:22:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.786+0000 7f2e96a00700 1 -- 172.21.15.27:0/1370194433 shutdown_connections 2024-05-10T11:22:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.786+0000 7f2e96a00700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2e780b2a70 0x7f2e780b4f30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.786+0000 7f2e96a00700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e900757c0 0x7f2e901a18d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.786+0000 7f2e96a00700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9010fc70 0x7f2e9019ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.786+0000 7f2e96a00700 1 --2- 172.21.15.27:0/1370194433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e900760e0 0x7f2e901a1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.786+0000 7f2e96a00700 1 -- 172.21.15.27:0/1370194433 >> 172.21.15.27:0/1370194433 conn(0x7f2e900feec0 msgr2=0x7f2e900ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.787+0000 7f2e96a00700 1 -- 172.21.15.27:0/1370194433 shutdown_connections 2024-05-10T11:22:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:12.787+0000 7f2e96a00700 1 -- 172.21.15.27:0/1370194433 wait complete. 2024-05-10T11:22:12.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:12 smithi076 ceph-mon[34152]: pgmap v278: 129 pgs: 129 active+clean; 367 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 225 op/s 2024-05-10T11:22:12.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:12 smithi189 ceph-mon[34842]: pgmap v278: 129 pgs: 129 active+clean; 367 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 225 op/s 2024-05-10T11:22:13.710 INFO:tasks.workunit.client.0.smithi027.stdout: 1 526688 6.55 MB/sec execute 298 sec latency 42.008 ms 2024-05-10T11:22:13.769 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:22:13.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:22:14.309 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:22:14.710 INFO:tasks.workunit.client.0.smithi027.stdout: 1 528132 6.55 MB/sec execute 299 sec latency 37.252 ms 2024-05-10T11:22:14.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:14 smithi027 ceph-mon[30054]: pgmap v279: 129 pgs: 129 active+clean; 367 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 201 op/s 2024-05-10T11:22:14.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:14 smithi076 ceph-mon[34152]: pgmap v279: 129 pgs: 129 active+clean; 367 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 201 op/s 2024-05-10T11:22:14.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:14 smithi189 ceph-mon[34842]: pgmap v279: 129 pgs: 129 active+clean; 367 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 201 op/s 2024-05-10T11:22:15.710 INFO:tasks.workunit.client.0.smithi027.stdout: 1 529541 6.57 MB/sec execute 300 sec latency 29.292 ms 2024-05-10T11:22:16.710 INFO:tasks.workunit.client.0.smithi027.stdout: 1 530193 6.55 MB/sec execute 301 sec latency 61.376 ms 2024-05-10T11:22:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.717+0000 7f0199800700 1 -- 172.21.15.27:0/1932293534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0194109510 msgr2=0x7f01941098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.717+0000 7f0199800700 1 --2- 172.21.15.27:0/1932293534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0194109510 0x7f01941098f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0184009990 tx=0x7f0184009ca0 comp rx=0 tx=0).stop 2024-05-10T11:22:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.717+0000 7f0199800700 1 -- 172.21.15.27:0/1932293534 shutdown_connections 2024-05-10T11:22:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.717+0000 7f0199800700 1 --2- 172.21.15.27:0/1932293534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0194109510 0x7f01941098f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.717+0000 7f0199800700 1 --2- 172.21.15.27:0/1932293534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f019406bb40 0x7f0194110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.717+0000 7f0199800700 1 --2- 172.21.15.27:0/1932293534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f019406b180 0x7f019406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.717+0000 7f0199800700 1 -- 172.21.15.27:0/1932293534 >> 172.21.15.27:0/1932293534 conn(0x7f0194079350 msgr2=0x7f0194079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.717+0000 7f0199800700 1 -- 172.21.15.27:0/1932293534 shutdown_connections 2024-05-10T11:22:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.717+0000 7f0199800700 1 -- 172.21.15.27:0/1932293534 wait complete. 2024-05-10T11:22:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.718+0000 7f0199800700 1 Processor -- start 2024-05-10T11:22:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.718+0000 7f0199800700 1 -- start start 2024-05-10T11:22:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.719+0000 7f0199800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f019406b180 0x7f019419bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.719+0000 7f0199800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f019406bb40 0x7f019419c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.719+0000 7f0199800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0194109510 0x7f01941a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.719+0000 7f0199800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0194113620 con 0x7f019406bb40 2024-05-10T11:22:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.719+0000 7f0199800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f01941134a0 con 0x7f019406b180 2024-05-10T11:22:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.719+0000 7f0199800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f01941137a0 con 0x7f0194109510 2024-05-10T11:22:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.719+0000 7f0192a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f019406b180 0x7f019419bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.719+0000 7f0193400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0194109510 0x7f01941a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.719+0000 7f0192a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f019406b180 0x7f019419bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58470/0 (socket says 172.21.15.27:58470) 2024-05-10T11:22:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.719+0000 7f0192a00700 1 -- 172.21.15.27:0/94944853 learned_addr learned my addr 172.21.15.27:0/94944853 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.719+0000 7f0192000700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f019406bb40 0x7f019419c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.720+0000 7f0192a00700 1 -- 172.21.15.27:0/94944853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0194109510 msgr2=0x7f01941a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.720+0000 7f0192a00700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0194109510 0x7f01941a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.720+0000 7f0192a00700 1 -- 172.21.15.27:0/94944853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f019406bb40 msgr2=0x7f019419c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.720+0000 7f0192a00700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f019406bb40 0x7f019419c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.720+0000 7f0192a00700 1 -- 172.21.15.27:0/94944853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01941a0d70 con 0x7f019406b180 2024-05-10T11:22:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.721+0000 7f0192a00700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f019406b180 0x7f019419bdb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f01840057c0 tx=0x7f0184009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.721+0000 7f018b400700 1 -- 172.21.15.27:0/94944853 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184026070 con 0x7f019406b180 2024-05-10T11:22:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.721+0000 7f018b400700 1 -- 172.21.15.27:0/94944853 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0184021410 con 0x7f019406b180 2024-05-10T11:22:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.721+0000 7f018b400700 1 -- 172.21.15.27:0/94944853 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184017650 con 0x7f019406b180 2024-05-10T11:22:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.721+0000 7f0199800700 1 -- 172.21.15.27:0/94944853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01941a1000 con 0x7f019406b180 2024-05-10T11:22:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.721+0000 7f0199800700 1 -- 172.21.15.27:0/94944853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01941a88e0 con 0x7f019406b180 2024-05-10T11:22:16.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.722+0000 7f0199800700 1 -- 172.21.15.27:0/94944853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f019406a190 con 0x7f019406b180 2024-05-10T11:22:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.725+0000 7f018b400700 1 -- 172.21.15.27:0/94944853 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0184029070 con 0x7f019406b180 2024-05-10T11:22:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.726+0000 7f018b400700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f01780b2a70 0x7f01780b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.726+0000 7f018b400700 1 -- 172.21.15.27:0/94944853 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f01840e36e0 con 0x7f019406b180 2024-05-10T11:22:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.727+0000 7f0192000700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f01780b2a70 0x7f01780b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:16.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.728+0000 7f0192000700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f01780b2a70 0x7f01780b4f30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f01941106e0 tx=0x7f0180009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:16.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:16.730+0000 7f018b400700 1 -- 172.21.15.27:0/94944853 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f0184013070 con 0x7f019406b180 2024-05-10T11:22:16.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:16 smithi027 ceph-mon[30054]: pgmap v280: 129 pgs: 129 active+clean; 375 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-05-10T11:22:16.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:16 smithi076 ceph-mon[34152]: pgmap v280: 129 pgs: 129 active+clean; 375 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-05-10T11:22:16.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:16 smithi189 ceph-mon[34842]: pgmap v280: 129 pgs: 129 active+clean; 375 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-05-10T11:22:17.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.112+0000 7f0199800700 1 -- 172.21.15.27:0/94944853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01941a8b70 con 0x7f019406b180 2024-05-10T11:22:17.113 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:22:17.113 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:21:14.172772+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24484,24472,24467,24488]},"id":1}]} 2024-05-10T11:22:17.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.113+0000 7f018b400700 1 -- 172.21.15.27:0/94944853 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11938 (secure 0 0 0) 0x7f01840aad50 con 0x7f019406b180 2024-05-10T11:22:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.118+0000 7f0199800700 1 -- 172.21.15.27:0/94944853 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f01780b2a70 msgr2=0x7f01780b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.118+0000 7f0199800700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f01780b2a70 0x7f01780b4f30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f01941106e0 tx=0x7f0180009260 comp rx=0 tx=0).stop 2024-05-10T11:22:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.118+0000 7f0199800700 1 -- 172.21.15.27:0/94944853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f019406b180 msgr2=0x7f019419bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.118+0000 7f0199800700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f019406b180 0x7f019419bdb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f01840057c0 tx=0x7f0184009580 comp rx=0 tx=0).stop 2024-05-10T11:22:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.119+0000 7f0199800700 1 -- 172.21.15.27:0/94944853 shutdown_connections 2024-05-10T11:22:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.119+0000 7f0199800700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f01780b2a70 0x7f01780b4f30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.119+0000 7f0199800700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0194109510 0x7f01941a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:17.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.119+0000 7f0199800700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f019406b180 0x7f019419bdb0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:17.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.119+0000 7f0199800700 1 --2- 172.21.15.27:0/94944853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f019406bb40 0x7f019419c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:17.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.119+0000 7f0199800700 1 -- 172.21.15.27:0/94944853 >> 172.21.15.27:0/94944853 conn(0x7f0194079350 msgr2=0x7f0194101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:17.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.119+0000 7f0199800700 1 -- 172.21.15.27:0/94944853 shutdown_connections 2024-05-10T11:22:17.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:17.119+0000 7f0199800700 1 -- 172.21.15.27:0/94944853 wait complete. 2024-05-10T11:22:17.121 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 34 2024-05-10T11:22:17.711 INFO:tasks.workunit.client.0.smithi027.stdout: 1 531541 6.55 MB/sec execute 302 sec latency 40.688 ms 2024-05-10T11:22:17.822 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:17 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/94944853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:22:17.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:17 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/94944853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:22:17.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:17 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/94944853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:22:18.043 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:22:18.547 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:22:18.711 INFO:tasks.workunit.client.0.smithi027.stdout: 1 533108 6.57 MB/sec execute 303 sec latency 20.638 ms 2024-05-10T11:22:18.882 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:18 smithi027 ceph-mon[30054]: pgmap v281: 129 pgs: 129 active+clean; 375 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-05-10T11:22:18.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:18 smithi076 ceph-mon[34152]: pgmap v281: 129 pgs: 129 active+clean; 375 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-05-10T11:22:18.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:18 smithi189 ceph-mon[34842]: pgmap v281: 129 pgs: 129 active+clean; 375 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-05-10T11:22:19.711 INFO:tasks.workunit.client.0.smithi027.stdout: 1 533627 6.55 MB/sec execute 304 sec latency 36.600 ms 2024-05-10T11:22:20.711 INFO:tasks.workunit.client.0.smithi027.stdout: 1 534866 6.54 MB/sec execute 305 sec latency 24.652 ms 2024-05-10T11:22:20.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:20 smithi076 ceph-mon[34152]: pgmap v282: 129 pgs: 129 active+clean; 381 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 232 op/s 2024-05-10T11:22:20.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:20 smithi189 ceph-mon[34842]: pgmap v282: 129 pgs: 129 active+clean; 381 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 232 op/s 2024-05-10T11:22:21.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:20 smithi027 ceph-mon[30054]: pgmap v282: 129 pgs: 129 active+clean; 381 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 232 op/s 2024-05-10T11:22:21.711 INFO:tasks.workunit.client.0.smithi027.stdout: 1 536528 6.56 MB/sec execute 306 sec latency 107.377 ms 2024-05-10T11:22:21.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.713+0000 7fc26ea00700 1 -- 172.21.15.27:0/3777886357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc268103ef0 msgr2=0x7fc2681103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:21.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.713+0000 7fc26ea00700 1 --2- 172.21.15.27:0/3777886357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc268103ef0 0x7fc2681103d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc260009990 tx=0x7fc260009ca0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.713+0000 7fc26ea00700 1 -- 172.21.15.27:0/3777886357 shutdown_connections 2024-05-10T11:22:21.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.713+0000 7fc26ea00700 1 --2- 172.21.15.27:0/3777886357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc268108460 0x7fc268108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.713+0000 7fc26ea00700 1 --2- 172.21.15.27:0/3777886357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc268103ef0 0x7fc2681103d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.713+0000 7fc26ea00700 1 --2- 172.21.15.27:0/3777886357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc268103550 0x7fc2681039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.713+0000 7fc26ea00700 1 -- 172.21.15.27:0/3777886357 >> 172.21.15.27:0/3777886357 conn(0x7fc26807a600 msgr2=0x7fc26807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:21.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.713+0000 7fc26ea00700 1 -- 172.21.15.27:0/3777886357 shutdown_connections 2024-05-10T11:22:21.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.713+0000 7fc26ea00700 1 -- 172.21.15.27:0/3777886357 wait complete. 2024-05-10T11:22:21.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.715+0000 7fc26ea00700 1 Processor -- start 2024-05-10T11:22:21.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.715+0000 7fc26ea00700 1 -- start start 2024-05-10T11:22:21.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.715+0000 7fc26ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc268103550 0x7fc26819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.715+0000 7fc26ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc268103ef0 0x7fc26819fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc26ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc268108460 0x7fc268199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc26ea00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc268113730 con 0x7fc268108460 2024-05-10T11:22:21.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc26ea00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc2681135b0 con 0x7fc268103ef0 2024-05-10T11:22:21.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc26ea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc2681138b0 con 0x7fc268103550 2024-05-10T11:22:21.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc267e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc268103550 0x7fc26819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc25f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc268103ef0 0x7fc26819fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc267e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc268103550 0x7fc26819f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37814/0 (socket says 172.21.15.27:37814) 2024-05-10T11:22:21.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc25f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc268103ef0 0x7fc26819fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58496/0 (socket says 172.21.15.27:58496) 2024-05-10T11:22:21.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc267e00700 1 -- 172.21.15.27:0/899217585 learned_addr learned my addr 172.21.15.27:0/899217585 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:21.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc26ca00700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc268108460 0x7fc268199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc25f400700 1 -- 172.21.15.27:0/899217585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc268103550 msgr2=0x7fc26819f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:21.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc25f400700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc268103550 0x7fc26819f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc25f400700 1 -- 172.21.15.27:0/899217585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc268108460 msgr2=0x7fc268199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:21.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc25f400700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc268108460 0x7fc268199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc25f400700 1 -- 172.21.15.27:0/899217585 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc26819a0f0 con 0x7fc268103ef0 2024-05-10T11:22:21.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.716+0000 7fc267e00700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc268103550 0x7fc26819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:21.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.717+0000 7fc25f400700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc268103ef0 0x7fc26819fd10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc25000e9f0 tx=0x7fc25000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:21.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.717+0000 7fc265600700 1 -- 172.21.15.27:0/899217585 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25000c9c0 con 0x7fc268103ef0 2024-05-10T11:22:21.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.717+0000 7fc265600700 1 -- 172.21.15.27:0/899217585 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc25000cb60 con 0x7fc268103ef0 2024-05-10T11:22:21.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.717+0000 7fc265600700 1 -- 172.21.15.27:0/899217585 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc250018680 con 0x7fc268103ef0 2024-05-10T11:22:21.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.717+0000 7fc26ea00700 1 -- 172.21.15.27:0/899217585 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc26819a3e0 con 0x7fc268103ef0 2024-05-10T11:22:21.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.718+0000 7fc26ea00700 1 -- 172.21.15.27:0/899217585 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2681a66a0 con 0x7fc268103ef0 2024-05-10T11:22:21.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.722+0000 7fc265600700 1 -- 172.21.15.27:0/899217585 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc250010430 con 0x7fc268103ef0 2024-05-10T11:22:21.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.723+0000 7fc265600700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc24c0b2b40 0x7fc24c0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.723+0000 7fc2793a3b80 1 Processor -- start 2024-05-10T11:22:21.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.723+0000 7fc265600700 1 -- 172.21.15.27:0/899217585 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7fc2500e2b20 con 0x7fc268103ef0 2024-05-10T11:22:21.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.724+0000 7fc2793a3b80 1 -- start start 2024-05-10T11:22:21.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.724+0000 7fc267e00700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc24c0b2b40 0x7fc24c0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.724+0000 7fc2793a3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82b2bc0 0x5556d82b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.724+0000 7fc2793a3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.724+0000 7fc2793a3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82ad8f0 0x5556d82add70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.724+0000 7fc2793a3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5556d81ffa20 con 0x5556d82ad8f0 2024-05-10T11:22:21.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.724+0000 7fc2793a3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5556d819d760 con 0x5556d82b34e0 2024-05-10T11:22:21.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.724+0000 7fc2793a3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5556d81fb9e0 con 0x5556d82b2bc0 2024-05-10T11:22:21.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.724+0000 7fc25f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.724+0000 7fc267e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82b2bc0 0x5556d82b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc25f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82ad2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58508/0 (socket says 172.21.15.27:58508) 2024-05-10T11:22:21.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc25f400700 1 -- 172.21.15.27:0/2954791432 learned_addr learned my addr 172.21.15.27:0/2954791432 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:21.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc26ca00700 1 --2- 172.21.15.27:0/2954791432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82ad8f0 0x5556d82add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc25f400700 1 -- 172.21.15.27:0/2954791432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82b2bc0 msgr2=0x5556d82b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc25f400700 1 --2- 172.21.15.27:0/2954791432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82b2bc0 0x5556d82b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc25f400700 1 -- 172.21.15.27:0/2954791432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82ad8f0 msgr2=0x5556d82add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc25f400700 1 --2- 172.21.15.27:0/2954791432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82ad8f0 0x5556d82add70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc25f400700 1 -- 172.21.15.27:0/2954791432 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556d814c8d0 con 0x5556d82b34e0 2024-05-10T11:22:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc25f400700 1 --2- 172.21.15.27:0/2954791432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82ad2d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc2500a9900 tx=0x7fc2500a9480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=291f3e775fffe286 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc267e00700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc24c0b2b40 0x7fc24c0b5000 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc26819aad0 tx=0x7fc25800b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc25d600700 1 -- 172.21.15.27:0/2954791432 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2500a9970 con 0x5556d82b34e0 2024-05-10T11:22:21.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc25d600700 1 -- 172.21.15.27:0/2954791432 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2500add20 con 0x5556d82b34e0 2024-05-10T11:22:21.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.725+0000 7fc25d600700 1 -- 172.21.15.27:0/2954791432 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2500b2340 con 0x5556d82b34e0 2024-05-10T11:22:21.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.726+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2954791432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 msgr2=0x5556d82ad2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:21.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.726+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/2954791432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82ad2d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc2500a9900 tx=0x7fc2500a9480 comp rx=0 tx=0).stop 2024-05-10T11:22:21.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.726+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2954791432 shutdown_connections 2024-05-10T11:22:21.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.726+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/2954791432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82b2bc0 0x5556d82b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.726+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/2954791432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82ad8f0 0x5556d82add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.726+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/2954791432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82ad2d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.726+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2954791432 >> 172.21.15.27:0/2954791432 conn(0x5556d82256d0 msgr2=0x5556d82a7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:21.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.726+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2954791432 shutdown_connections 2024-05-10T11:22:21.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.727+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2954791432 wait complete. 2024-05-10T11:22:21.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.727+0000 7fc2793a3b80 1 -- start start 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.727+0000 7fc2793a3b80 1 Processor -- start 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc2793a3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 0x5556d82b9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc2793a3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b2bc0 0x5556d82b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc2793a3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc2793a3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5556d82b9f20 con 0x5556d82b2bc0 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc2793a3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5556d8165b20 con 0x5556d82b34e0 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc2793a3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5556d82ba0a0 con 0x5556d82ad8f0 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc2793a3b80 10 client.0 _ll_register_callbacks cb 0x5556d81a6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc26ca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc25f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b2bc0 0x5556d82b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc267e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 0x5556d82b9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc26ca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82b6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58518/0 (socket says 172.21.15.27:58518) 2024-05-10T11:22:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc26ca00700 1 -- 172.21.15.27:0/2770797208 learned_addr learned my addr 172.21.15.27:0/2770797208 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.728+0000 7fc267e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 0x5556d82b9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37838/0 (socket says 172.21.15.27:37838) 2024-05-10T11:22:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.729+0000 7fc267e00700 1 -- 172.21.15.27:0/2770797208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 msgr2=0x5556d82b6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.729+0000 7fc267e00700 1 --2- 172.21.15.27:0/2770797208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82b6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.729+0000 7fc267e00700 1 -- 172.21.15.27:0/2770797208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b2bc0 msgr2=0x5556d82b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.729+0000 7fc267e00700 1 --2- 172.21.15.27:0/2770797208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b2bc0 0x5556d82b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.729+0000 7fc267e00700 1 -- 172.21.15.27:0/2770797208 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556d81c5420 con 0x5556d82ad8f0 2024-05-10T11:22:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.729+0000 7fc25f400700 1 --2- 172.21.15.27:0/2770797208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b2bc0 0x5556d82b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.729+0000 7fc267e00700 1 --2- 172.21.15.27:0/2770797208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 0x5556d82b9250 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc258009fd0 tx=0x7fc258004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7735df9186fbd715 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.729+0000 7fc256000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:22:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.729+0000 7fc256000700 1 -- 172.21.15.27:0/2770797208 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25801e250 con 0x5556d82ad8f0 2024-05-10T11:22:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.729+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2770797208 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2500add20 con 0x5556d82ad8f0 2024-05-10T11:22:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.730+0000 7fc256000700 1 -- 172.21.15.27:0/2770797208 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7fc25801d4f0 con 0x5556d82ad8f0 2024-05-10T11:22:21.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.730+0000 7fc256000700 1 -- 172.21.15.27:0/2770797208 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc258035050 con 0x5556d82ad8f0 2024-05-10T11:22:21.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.730+0000 7fc2793a3b80 10 client.26569 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:21.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.730+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2770797208 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5556d82c4870 con 0x5556d82ad8f0 2024-05-10T11:22:21.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.730+0000 7fc256000700 1 -- 172.21.15.27:0/2770797208 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2500add20 con 0x5556d82ad8f0 2024-05-10T11:22:21.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.730+0000 7fc256000700 1 -- 172.21.15.27:0/2770797208 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fc258026070 con 0x5556d82ad8f0 2024-05-10T11:22:21.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.731+0000 7fc2793a3b80 10 client.26569 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:21.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.731+0000 7fc2793a3b80 10 client.26569 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:22:21.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.731+0000 7fc2793a3b80 20 client.26569 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.731+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/2770797208 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5556d82c6e00 0x5556d82c9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.731+0000 7fc2793a3b80 4 client.26569 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.731+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2770797208 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556d827a140 con 0x5556d82c6e00 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.731+0000 7fc25f400700 1 --2- 172.21.15.27:0/2770797208 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5556d82c6e00 0x5556d82c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.731+0000 7fc25f400700 1 --2- 172.21.15.27:0/2770797208 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5556d82c6e00 0x5556d82c9210 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=296f748d82dcc734 server_cookie=3da28af37e178768 in_seq=0 out_seq=0 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.731+0000 7fc256000700 10 client.26569 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.732+0000 7fc256000700 1 -- 172.21.15.27:0/2770797208 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5556d827a140 con 0x5556d82c6e00 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.732+0000 7fc256000700 10 client.26569 handle_command_reply: tid=0 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.732+0000 7fc2793a3b80 1 client.26569 shutdown 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.732+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2770797208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 msgr2=0x5556d82b9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.732+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/2770797208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 0x5556d82b9250 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc258009fd0 tx=0x7fc258004ad0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.733+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2770797208 shutdown_connections 2024-05-10T11:22:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.733+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/2770797208 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5556d82c6e00 0x5556d82c9210 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.733+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/2770797208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 0x5556d82b9250 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.733+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/2770797208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.733+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/2770797208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b2bc0 0x5556d82b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.733+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2770797208 >> 172.21.15.27:0/2770797208 conn(0x5556d82256d0 msgr2=0x5556d82a7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:21.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.733+0000 7fc256000700 0 client.26569 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:22:21.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.733+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2770797208 shutdown_connections 2024-05-10T11:22:21.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.733+0000 7fc2793a3b80 1 -- 172.21.15.27:0/2770797208 wait complete. 2024-05-10T11:22:21.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.733+0000 7fc2793a3b80 20 client.26569 trim_cache size 0 max 16384 2024-05-10T11:22:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.755+0000 7fc2793a3b80 1 Processor -- start 2024-05-10T11:22:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.755+0000 7fc2793a3b80 1 -- start start 2024-05-10T11:22:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.756+0000 7fc2793a3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 0x5556d82b5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.756+0000 7fc2793a3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b2bc0 0x5556d82b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.756+0000 7fc2793a3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.756+0000 7fc2793a3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc2681ad630 con 0x5556d82b2bc0 2024-05-10T11:22:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.756+0000 7fc2793a3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc2681af1a0 con 0x5556d82b34e0 2024-05-10T11:22:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.756+0000 7fc2793a3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc2681add00 con 0x5556d82ad8f0 2024-05-10T11:22:21.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.756+0000 7fc267e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 0x5556d82b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.756+0000 7fc267e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 0x5556d82b5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37842/0 (socket says 172.21.15.27:37842) 2024-05-10T11:22:21.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.756+0000 7fc25f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b2bc0 0x5556d82b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.756+0000 7fc26ca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.756+0000 7fc267e00700 1 -- 172.21.15.27:0/1142214932 learned_addr learned my addr 172.21.15.27:0/1142214932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:21.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.757+0000 7fc267e00700 1 -- 172.21.15.27:0/1142214932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 msgr2=0x5556d82c2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:21.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.757+0000 7fc267e00700 1 --2- 172.21.15.27:0/1142214932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82c2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.757+0000 7fc267e00700 1 -- 172.21.15.27:0/1142214932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b2bc0 msgr2=0x5556d82b41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.757+0000 7fc267e00700 1 --2- 172.21.15.27:0/1142214932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b2bc0 0x5556d82b41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.757+0000 7fc267e00700 1 -- 172.21.15.27:0/1142214932 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556d82a2ab0 con 0x5556d82ad8f0 2024-05-10T11:22:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.757+0000 7fc267e00700 1 --2- 172.21.15.27:0/1142214932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 0x5556d82b5550 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc258017650 tx=0x7fc258017440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca0cdeca7dbd65c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.757+0000 7fc25cc00700 1 -- 172.21.15.27:0/1142214932 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc258017ac0 con 0x5556d82ad8f0 2024-05-10T11:22:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.757+0000 7fc25cc00700 1 -- 172.21.15.27:0/1142214932 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc25802c070 con 0x5556d82ad8f0 2024-05-10T11:22:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.757+0000 7fc25cc00700 1 -- 172.21.15.27:0/1142214932 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25801daf0 con 0x5556d82ad8f0 2024-05-10T11:22:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.758+0000 7fc2793a3b80 1 -- 172.21.15.27:0/1142214932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 msgr2=0x5556d82b5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.758+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/1142214932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 0x5556d82b5550 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc258017650 tx=0x7fc258017440 comp rx=0 tx=0).stop 2024-05-10T11:22:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.758+0000 7fc2793a3b80 1 -- 172.21.15.27:0/1142214932 shutdown_connections 2024-05-10T11:22:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.758+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/1142214932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82ad8f0 0x5556d82b5550 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.758+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/1142214932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82b34e0 0x5556d82c2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.758+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/1142214932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b2bc0 0x5556d82b41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.758+0000 7fc2793a3b80 1 -- 172.21.15.27:0/1142214932 >> 172.21.15.27:0/1142214932 conn(0x5556d82256d0 msgr2=0x5556d82b1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.758+0000 7fc2793a3b80 1 -- 172.21.15.27:0/1142214932 shutdown_connections 2024-05-10T11:22:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.759+0000 7fc2793a3b80 1 -- 172.21.15.27:0/1142214932 wait complete. 2024-05-10T11:22:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.759+0000 7fc2793a3b80 1 -- start start 2024-05-10T11:22:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.759+0000 7fc2793a3b80 1 Processor -- start 2024-05-10T11:22:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.760+0000 7fc2793a3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82ad8f0 0x5556d82bf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.760+0000 7fc2793a3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82b2bc0 0x5556d82bf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.760+0000 7fc2793a3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b34e0 0x5556d82cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.760+0000 7fc2793a3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc2681aead0 con 0x5556d82b34e0 2024-05-10T11:22:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.760+0000 7fc2793a3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc2681ae400 con 0x5556d82ad8f0 2024-05-10T11:22:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.760+0000 7fc2793a3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc2681ab5f0 con 0x5556d82b2bc0 2024-05-10T11:22:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.760+0000 7fc2793a3b80 10 client.0 _ll_register_callbacks cb 0x5556d81a6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.760+0000 7fc25f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82b2bc0 0x5556d82bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.760+0000 7fc25f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82b2bc0 0x5556d82bf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37846/0 (socket says 172.21.15.27:37846) 2024-05-10T11:22:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.760+0000 7fc25f400700 1 -- 172.21.15.27:0/4196682657 learned_addr learned my addr 172.21.15.27:0/4196682657 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.760+0000 7fc267e00700 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82ad8f0 0x5556d82bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.761+0000 7fc26ca00700 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b34e0 0x5556d82cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.761+0000 7fc25f400700 1 -- 172.21.15.27:0/4196682657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82ad8f0 msgr2=0x5556d82bf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.761+0000 7fc25f400700 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82ad8f0 0x5556d82bf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.761+0000 7fc25f400700 1 -- 172.21.15.27:0/4196682657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b34e0 msgr2=0x5556d82cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.761+0000 7fc25f400700 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b34e0 0x5556d82cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.761+0000 7fc25f400700 1 -- 172.21.15.27:0/4196682657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556d82b95c0 con 0x5556d82b2bc0 2024-05-10T11:22:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.761+0000 7fc26ca00700 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b34e0 0x5556d82cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:22:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.761+0000 7fc25f400700 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82b2bc0 0x5556d82bf870 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc2500a9670 tx=0x7fc2500aa1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2204cbc749aa62b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.761+0000 7fc256000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:22:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.761+0000 7fc256000700 1 -- 172.21.15.27:0/4196682657 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2500add20 con 0x5556d82b2bc0 2024-05-10T11:22:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.761+0000 7fc2793a3b80 1 -- 172.21.15.27:0/4196682657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc25802c070 con 0x5556d82b2bc0 2024-05-10T11:22:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.761+0000 7fc256000700 1 -- 172.21.15.27:0/4196682657 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7fc2500a9de0 con 0x5556d82b2bc0 2024-05-10T11:22:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.762+0000 7fc256000700 1 -- 172.21.15.27:0/4196682657 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc2500a9970 con 0x5556d82b2bc0 2024-05-10T11:22:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.762+0000 7fc2793a3b80 10 client.26575 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.762+0000 7fc2793a3b80 1 -- 172.21.15.27:0/4196682657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5556d8330ef0 con 0x5556d82b2bc0 2024-05-10T11:22:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.762+0000 7fc256000700 1 -- 172.21.15.27:0/4196682657 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc25802c070 con 0x5556d82b2bc0 2024-05-10T11:22:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.762+0000 7fc256000700 1 -- 172.21.15.27:0/4196682657 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fc2500c8070 con 0x5556d82b2bc0 2024-05-10T11:22:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.762+0000 7fc2793a3b80 10 client.26575 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.762+0000 7fc2793a3b80 10 client.26575 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:22:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.762+0000 7fc2793a3b80 20 client.26575 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.763+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5556d8331180 0x5556d8333570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.763+0000 7fc2793a3b80 4 client.26575 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:22:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.763+0000 7fc2793a3b80 1 -- 172.21.15.27:0/4196682657 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5556d820d400 con 0x5556d8331180 2024-05-10T11:22:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.763+0000 7fc267e00700 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5556d8331180 0x5556d8333570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.763+0000 7fc267e00700 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5556d8331180 0x5556d8333570 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ba620ddc8d72a39c server_cookie=677de912d8ab893f in_seq=0 out_seq=0 2024-05-10T11:22:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.763+0000 7fc256000700 10 client.26575 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:22:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc256000700 1 -- 172.21.15.27:0/4196682657 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5556d820d400 con 0x5556d8331180 2024-05-10T11:22:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc256000700 10 client.26575 handle_command_reply: tid=0 2024-05-10T11:22:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc2793a3b80 1 client.26575 shutdown 2024-05-10T11:22:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc2793a3b80 1 -- 172.21.15.27:0/4196682657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82b2bc0 msgr2=0x5556d82bf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82b2bc0 0x5556d82bf870 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc2500a9670 tx=0x7fc2500aa1b0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc2793a3b80 1 -- 172.21.15.27:0/4196682657 shutdown_connections 2024-05-10T11:22:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556d82b34e0 0x5556d82cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5556d82ad8f0 0x5556d82bf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5556d8331180 0x5556d8333570 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc2793a3b80 1 --2- 172.21.15.27:0/4196682657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d82b2bc0 0x5556d82bf870 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc2793a3b80 1 -- 172.21.15.27:0/4196682657 >> 172.21.15.27:0/4196682657 conn(0x5556d82256d0 msgr2=0x5556d82b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:21.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc256000700 0 client.26575 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:22:21.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc2793a3b80 1 -- 172.21.15.27:0/4196682657 shutdown_connections 2024-05-10T11:22:21.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc2793a3b80 1 -- 172.21.15.27:0/4196682657 wait complete. 2024-05-10T11:22:21.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.764+0000 7fc2793a3b80 20 client.26575 trim_cache size 0 max 16384 2024-05-10T11:22:21.768 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:22:21.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.765+0000 7fc26ea00700 1 -- 172.21.15.27:0/899217585 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc24c0b2b40 msgr2=0x7fc24c0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:21.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.765+0000 7fc26ea00700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc24c0b2b40 0x7fc24c0b5000 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc26819aad0 tx=0x7fc25800b040 comp rx=0 tx=0).stop 2024-05-10T11:22:21.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.765+0000 7fc26ea00700 1 -- 172.21.15.27:0/899217585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc268103ef0 msgr2=0x7fc26819fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:21.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.765+0000 7fc26ea00700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc268103ef0 0x7fc26819fd10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc25000e9f0 tx=0x7fc25000ed00 comp rx=0 tx=0).stop 2024-05-10T11:22:21.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.765+0000 7fc26ea00700 1 -- 172.21.15.27:0/899217585 shutdown_connections 2024-05-10T11:22:21.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.765+0000 7fc26ea00700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc24c0b2b40 0x7fc24c0b5000 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.765+0000 7fc26ea00700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc268103550 0x7fc26819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.765+0000 7fc26ea00700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc268108460 0x7fc268199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.765+0000 7fc26ea00700 1 --2- 172.21.15.27:0/899217585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc268103ef0 0x7fc26819fd10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:21.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.765+0000 7fc26ea00700 1 -- 172.21.15.27:0/899217585 >> 172.21.15.27:0/899217585 conn(0x7fc26807a600 msgr2=0x7fc2681015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:21.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.766+0000 7fc26ea00700 1 -- 172.21.15.27:0/899217585 shutdown_connections 2024-05-10T11:22:21.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:21.766+0000 7fc26ea00700 1 -- 172.21.15.27:0/899217585 wait complete. 2024-05-10T11:22:22.711 INFO:tasks.workunit.client.0.smithi027.stdout: 1 537116 6.55 MB/sec execute 307 sec latency 41.224 ms 2024-05-10T11:22:22.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:22 smithi076 ceph-mon[34152]: pgmap v283: 129 pgs: 129 active+clean; 381 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-05-10T11:22:22.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:22 smithi027 ceph-mon[30054]: pgmap v283: 129 pgs: 129 active+clean; 381 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-05-10T11:22:22.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:22 smithi189 ceph-mon[34842]: pgmap v283: 129 pgs: 129 active+clean; 381 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-05-10T11:22:23.047 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:22:23.529 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:22:23.711 INFO:tasks.workunit.client.0.smithi027.stdout: 1 538522 6.54 MB/sec execute 308 sec latency 25.811 ms 2024-05-10T11:22:24.712 INFO:tasks.workunit.client.0.smithi027.stdout: 1 540085 6.56 MB/sec execute 309 sec latency 22.181 ms 2024-05-10T11:22:24.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:24 smithi076 ceph-mon[34152]: pgmap v284: 129 pgs: 129 active+clean; 381 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-05-10T11:22:24.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:24 smithi189 ceph-mon[34842]: pgmap v284: 129 pgs: 129 active+clean; 381 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-05-10T11:22:25.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:24 smithi027 ceph-mon[30054]: pgmap v284: 129 pgs: 129 active+clean; 381 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-05-10T11:22:25.712 INFO:tasks.workunit.client.0.smithi027.stdout: 1 540690 6.54 MB/sec execute 310 sec latency 40.796 ms 2024-05-10T11:22:26.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.075+0000 7f9e76a00700 1 -- 172.21.15.27:0/1057617322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e70109500 msgr2=0x7f9e701098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:26.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.075+0000 7f9e76a00700 1 --2- 172.21.15.27:0/1057617322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e70109500 0x7f9e701098e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000b0a0 tx=0x7f9e6000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.075+0000 7f9e76a00700 1 -- 172.21.15.27:0/1057617322 shutdown_connections 2024-05-10T11:22:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.075+0000 7f9e76a00700 1 --2- 172.21.15.27:0/1057617322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7006c3d0 0x7f9e70110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.075+0000 7f9e76a00700 1 --2- 172.21.15.27:0/1057617322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e70109500 0x7f9e701098e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.075+0000 7f9e76a00700 1 --2- 172.21.15.27:0/1057617322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e7006ba10 0x7f9e7006be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.075+0000 7f9e76a00700 1 -- 172.21.15.27:0/1057617322 >> 172.21.15.27:0/1057617322 conn(0x7f9e70079310 msgr2=0x7f9e70079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.075+0000 7f9e76a00700 1 -- 172.21.15.27:0/1057617322 shutdown_connections 2024-05-10T11:22:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.075+0000 7f9e76a00700 1 -- 172.21.15.27:0/1057617322 wait complete. 2024-05-10T11:22:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.077+0000 7f9e76a00700 1 Processor -- start 2024-05-10T11:22:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.077+0000 7f9e76a00700 1 -- start start 2024-05-10T11:22:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e76a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e7006ba10 0x7f9e7019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e76a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7006c3d0 0x7f9e7019fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e76a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70109500 0x7f9e701999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e76a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9e70113660 con 0x7f9e70109500 2024-05-10T11:22:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e76a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9e701134e0 con 0x7f9e7006ba10 2024-05-10T11:22:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e76a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e701137e0 con 0x7f9e7006c3d0 2024-05-10T11:22:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e6fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e7006ba10 0x7f9e7019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e6fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e7006ba10 0x7f9e7019f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45202/0 (socket says 172.21.15.27:45202) 2024-05-10T11:22:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e6fe00700 1 -- 172.21.15.27:0/1654386392 learned_addr learned my addr 172.21.15.27:0/1654386392 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e67400700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7006c3d0 0x7f9e7019fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e74a00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70109500 0x7f9e701999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e6fe00700 1 -- 172.21.15.27:0/1654386392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7006c3d0 msgr2=0x7f9e7019fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:26.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.078+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7006c3d0 0x7f9e7019fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.079+0000 7f9e6fe00700 1 -- 172.21.15.27:0/1654386392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70109500 msgr2=0x7f9e701999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:26.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.079+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70109500 0x7f9e701999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.079+0000 7f9e6fe00700 1 -- 172.21.15.27:0/1654386392 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e7019a140 con 0x7f9e7006ba10 2024-05-10T11:22:26.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.079+0000 7f9e67400700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7006c3d0 0x7f9e7019fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:26.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.079+0000 7f9e74a00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70109500 0x7f9e701999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:22:26.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.079+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e7006ba10 0x7f9e7019f7d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000d010 tx=0x7f9e60004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:26.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.079+0000 7f9e6d600700 1 -- 172.21.15.27:0/1654386392 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60021070 con 0x7f9e7006ba10 2024-05-10T11:22:26.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.079+0000 7f9e6d600700 1 -- 172.21.15.27:0/1654386392 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e60007c40 con 0x7f9e7006ba10 2024-05-10T11:22:26.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.079+0000 7f9e6d600700 1 -- 172.21.15.27:0/1654386392 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60004760 con 0x7f9e7006ba10 2024-05-10T11:22:26.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.079+0000 7f9e76a00700 1 -- 172.21.15.27:0/1654386392 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e7019a3d0 con 0x7f9e7006ba10 2024-05-10T11:22:26.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.080+0000 7f9e76a00700 1 -- 172.21.15.27:0/1654386392 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e701a66a0 con 0x7f9e7006ba10 2024-05-10T11:22:26.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.084+0000 7f9e6d600700 1 -- 172.21.15.27:0/1654386392 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f9e60003a00 con 0x7f9e7006ba10 2024-05-10T11:22:26.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.085+0000 7f9e81388b80 1 Processor -- start 2024-05-10T11:22:26.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.085+0000 7f9e81388b80 1 -- start start 2024-05-10T11:22:26.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.086+0000 7f9e81388b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a45bc0 0x5574e3a45fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.086+0000 7f9e81388b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a464e0 0x5574e3a402d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.086+0000 7f9e81388b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a408f0 0x5574e3a40d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.086+0000 7f9e81388b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5574e3992a20 con 0x5574e3a45bc0 2024-05-10T11:22:26.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.086+0000 7f9e81388b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5574e3930760 con 0x5574e3a464e0 2024-05-10T11:22:26.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.086+0000 7f9e81388b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5574e398e9e0 con 0x5574e3a408f0 2024-05-10T11:22:26.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.086+0000 7f9e6fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a464e0 0x5574e3a402d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.086+0000 7f9e6fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a464e0 0x5574e3a402d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45212/0 (socket says 172.21.15.27:45212) 2024-05-10T11:22:26.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.086+0000 7f9e6fe00700 1 -- 172.21.15.27:0/1994826107 learned_addr learned my addr 172.21.15.27:0/1994826107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:26.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.086+0000 7f9e74a00700 1 --2- 172.21.15.27:0/1994826107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a45bc0 0x5574e3a45fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.086+0000 7f9e67400700 1 --2- 172.21.15.27:0/1994826107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a408f0 0x5574e3a40d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.086+0000 7f9e6d600700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9e540b2a70 0x7f9e540b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.087+0000 7f9e6fe00700 1 -- 172.21.15.27:0/1994826107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a408f0 msgr2=0x5574e3a40d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:26.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.087+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/1994826107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a408f0 0x5574e3a40d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.087+0000 7f9e6fe00700 1 -- 172.21.15.27:0/1994826107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a45bc0 msgr2=0x5574e3a45fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:26.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.087+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/1994826107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a45bc0 0x5574e3a45fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.087+0000 7f9e6fe00700 1 -- 172.21.15.27:0/1994826107 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574e38df8d0 con 0x5574e3a464e0 2024-05-10T11:22:26.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.087+0000 7f9e67400700 1 --2- 172.21.15.27:0/1994826107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a408f0 0x5574e3a40d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:26.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.087+0000 7f9e6d600700 1 -- 172.21.15.27:0/1654386392 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f9e6000f020 con 0x7f9e7006ba10 2024-05-10T11:22:26.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.087+0000 7f9e74a00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9e540b2a70 0x7f9e540b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.087+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/1994826107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a464e0 0x5574e3a402d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9e600e11b0 tx=0x7f9e600097e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a67107dbf3d5649 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:26.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.088+0000 7f9e65600700 1 -- 172.21.15.27:0/1994826107 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e600aac30 con 0x5574e3a464e0 2024-05-10T11:22:26.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.088+0000 7f9e65600700 1 -- 172.21.15.27:0/1994826107 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e600aa970 con 0x5574e3a464e0 2024-05-10T11:22:26.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.088+0000 7f9e65600700 1 -- 172.21.15.27:0/1994826107 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e600aa5d0 con 0x5574e3a464e0 2024-05-10T11:22:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.089+0000 7f9e81388b80 1 -- 172.21.15.27:0/1994826107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a464e0 msgr2=0x5574e3a402d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.089+0000 7f9e81388b80 1 --2- 172.21.15.27:0/1994826107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a464e0 0x5574e3a402d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9e600e11b0 tx=0x7f9e600097e0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.089+0000 7f9e74a00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9e540b2a70 0x7f9e540b4f30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009dc0 tx=0x7f9e68004750 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.089+0000 7f9e81388b80 1 -- 172.21.15.27:0/1994826107 shutdown_connections 2024-05-10T11:22:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.089+0000 7f9e81388b80 1 --2- 172.21.15.27:0/1994826107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a408f0 0x5574e3a40d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.089+0000 7f9e81388b80 1 --2- 172.21.15.27:0/1994826107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a45bc0 0x5574e3a45fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.089+0000 7f9e81388b80 1 --2- 172.21.15.27:0/1994826107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a464e0 0x5574e3a402d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.089+0000 7f9e81388b80 1 -- 172.21.15.27:0/1994826107 >> 172.21.15.27:0/1994826107 conn(0x5574e39b86d0 msgr2=0x5574e3a3ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.090+0000 7f9e81388b80 1 -- 172.21.15.27:0/1994826107 shutdown_connections 2024-05-10T11:22:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.090+0000 7f9e81388b80 1 -- 172.21.15.27:0/1994826107 wait complete. 2024-05-10T11:22:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.090+0000 7f9e81388b80 1 -- start start 2024-05-10T11:22:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.091+0000 7f9e81388b80 1 Processor -- start 2024-05-10T11:22:26.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.091+0000 7f9e81388b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 0x5574e3a4c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.091+0000 7f9e81388b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a45bc0 0x5574e3a49480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.091+0000 7f9e81388b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a464e0 0x5574e3a499c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.091+0000 7f9e81388b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5574e3a4cce0 con 0x5574e3a464e0 2024-05-10T11:22:26.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.091+0000 7f9e81388b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5574e38f8b20 con 0x5574e3a408f0 2024-05-10T11:22:26.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.091+0000 7f9e81388b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5574e3a4ce60 con 0x5574e3a45bc0 2024-05-10T11:22:26.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.091+0000 7f9e81388b80 10 client.0 _ll_register_callbacks cb 0x5574e3939c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:26.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.091+0000 7f9e6fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 0x5574e3a4c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.092+0000 7f9e6fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 0x5574e3a4c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45226/0 (socket says 172.21.15.27:45226) 2024-05-10T11:22:26.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.092+0000 7f9e6fe00700 1 -- 172.21.15.27:0/29123255 learned_addr learned my addr 172.21.15.27:0/29123255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:26.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.092+0000 7f9e67400700 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a45bc0 0x5574e3a49480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.092+0000 7f9e74a00700 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a464e0 0x5574e3a499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.092+0000 7f9e67400700 1 -- 172.21.15.27:0/29123255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 msgr2=0x5574e3a4c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:26.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.092+0000 7f9e67400700 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 0x5574e3a4c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.092+0000 7f9e67400700 1 -- 172.21.15.27:0/29123255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a464e0 msgr2=0x5574e3a499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:26.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.092+0000 7f9e67400700 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a464e0 0x5574e3a499c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.092+0000 7f9e67400700 1 -- 172.21.15.27:0/29123255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574e3958420 con 0x5574e3a45bc0 2024-05-10T11:22:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.093+0000 7f9e67400700 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a45bc0 0x5574e3a49480 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x5574e3a412b0 tx=0x7f9e5800d8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21e36d492caa9829 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.093+0000 7f9e5e000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:22:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.093+0000 7f9e5e000700 1 -- 172.21.15.27:0/29123255 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58004020 con 0x5574e3a45bc0 2024-05-10T11:22:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.093+0000 7f9e81388b80 1 -- 172.21.15.27:0/29123255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e600aa970 con 0x5574e3a45bc0 2024-05-10T11:22:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.093+0000 7f9e5e000700 1 -- 172.21.15.27:0/29123255 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f9e580041c0 con 0x5574e3a45bc0 2024-05-10T11:22:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.093+0000 7f9e5e000700 1 -- 172.21.15.27:0/29123255 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9e580043c0 con 0x5574e3a45bc0 2024-05-10T11:22:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.093+0000 7f9e81388b80 10 client.26587 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.093+0000 7f9e81388b80 1 -- 172.21.15.27:0/29123255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5574e3a57440 con 0x5574e3a45bc0 2024-05-10T11:22:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.094+0000 7f9e5e000700 1 -- 172.21.15.27:0/29123255 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e600aa970 con 0x5574e3a45bc0 2024-05-10T11:22:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.094+0000 7f9e5e000700 1 -- 172.21.15.27:0/29123255 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f9e58005360 con 0x5574e3a45bc0 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.094+0000 7f9e81388b80 10 client.26587 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.094+0000 7f9e81388b80 10 client.26587 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.095+0000 7f9e81388b80 20 client.26587 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.095+0000 7f9e81388b80 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5574e3a599d0 0x5574e3a5be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.095+0000 7f9e81388b80 4 client.26587 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.095+0000 7f9e81388b80 1 -- 172.21.15.27:0/29123255 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574e3a0d140 con 0x5574e3a599d0 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.095+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5574e3a599d0 0x5574e3a5be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.096+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5574e3a599d0 0x5574e3a5be20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c1e6b3c29aafd51b server_cookie=7150051633bd76d5 in_seq=0 out_seq=0 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.096+0000 7f9e5e000700 10 client.26587 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.097+0000 7f9e5e000700 1 -- 172.21.15.27:0/29123255 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5574e3a0d140 con 0x5574e3a599d0 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.097+0000 7f9e5e000700 10 client.26587 handle_command_reply: tid=0 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.097+0000 7f9e81388b80 1 client.26587 shutdown 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.098+0000 7f9e81388b80 1 -- 172.21.15.27:0/29123255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a45bc0 msgr2=0x5574e3a49480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.098+0000 7f9e81388b80 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a45bc0 0x5574e3a49480 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x5574e3a412b0 tx=0x7f9e5800d8a0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.098+0000 7f9e81388b80 1 -- 172.21.15.27:0/29123255 shutdown_connections 2024-05-10T11:22:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.098+0000 7f9e81388b80 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5574e3a599d0 0x5574e3a5be20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.098+0000 7f9e81388b80 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a464e0 0x5574e3a499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.098+0000 7f9e81388b80 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 0x5574e3a4c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.098+0000 7f9e81388b80 1 --2- 172.21.15.27:0/29123255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a45bc0 0x5574e3a49480 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.098+0000 7f9e81388b80 1 -- 172.21.15.27:0/29123255 >> 172.21.15.27:0/29123255 conn(0x5574e39b86d0 msgr2=0x5574e3a3a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.098+0000 7f9e5e000700 0 client.26587 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:22:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.098+0000 7f9e81388b80 1 -- 172.21.15.27:0/29123255 shutdown_connections 2024-05-10T11:22:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.098+0000 7f9e81388b80 1 -- 172.21.15.27:0/29123255 wait complete. 2024-05-10T11:22:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.098+0000 7f9e81388b80 20 client.26587 trim_cache size 0 max 16384 2024-05-10T11:22:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.116+0000 7f9e81388b80 1 Processor -- start 2024-05-10T11:22:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.116+0000 7f9e81388b80 1 -- start start 2024-05-10T11:22:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.117+0000 7f9e81388b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a408f0 0x5574e3a55ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.117+0000 7f9e81388b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a45bc0 0x5574e3a46fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.117+0000 7f9e81388b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a464e0 0x5574e3a3a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.117+0000 7f9e81388b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9e701ad440 con 0x5574e3a408f0 2024-05-10T11:22:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.117+0000 7f9e81388b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9e701aefb0 con 0x5574e3a45bc0 2024-05-10T11:22:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.117+0000 7f9e81388b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e701adb10 con 0x5574e3a464e0 2024-05-10T11:22:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.117+0000 7f9e67400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a45bc0 0x5574e3a46fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.117+0000 7f9e74a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a464e0 0x5574e3a3a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.117+0000 7f9e67400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a45bc0 0x5574e3a46fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45228/0 (socket says 172.21.15.27:45228) 2024-05-10T11:22:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.117+0000 7f9e67400700 1 -- 172.21.15.27:0/1550564349 learned_addr learned my addr 172.21.15.27:0/1550564349 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.117+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/1550564349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a408f0 0x5574e3a55ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.118+0000 7f9e67400700 1 -- 172.21.15.27:0/1550564349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a464e0 msgr2=0x5574e3a3a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.118+0000 7f9e67400700 1 --2- 172.21.15.27:0/1550564349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a464e0 0x5574e3a3a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.118+0000 7f9e67400700 1 -- 172.21.15.27:0/1550564349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a408f0 msgr2=0x5574e3a55ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.118+0000 7f9e67400700 1 --2- 172.21.15.27:0/1550564349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a408f0 0x5574e3a55ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.118+0000 7f9e67400700 1 -- 172.21.15.27:0/1550564349 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574e3a5c180 con 0x5574e3a45bc0 2024-05-10T11:22:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.118+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/1550564349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a408f0 0x5574e3a55ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:22:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.118+0000 7f9e67400700 1 --2- 172.21.15.27:0/1550564349 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a45bc0 0x5574e3a46fd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9e58003770 tx=0x7f9e58003a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3e5f646ed82ef5e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.118+0000 7f9e64c00700 1 -- 172.21.15.27:0/1550564349 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58038070 con 0x5574e3a45bc0 2024-05-10T11:22:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.118+0000 7f9e64c00700 1 -- 172.21.15.27:0/1550564349 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e580041a0 con 0x5574e3a45bc0 2024-05-10T11:22:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.118+0000 7f9e64c00700 1 -- 172.21.15.27:0/1550564349 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58021a10 con 0x5574e3a45bc0 2024-05-10T11:22:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.119+0000 7f9e81388b80 1 -- 172.21.15.27:0/1550564349 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a45bc0 msgr2=0x5574e3a46fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.119+0000 7f9e81388b80 1 --2- 172.21.15.27:0/1550564349 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a45bc0 0x5574e3a46fd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9e58003770 tx=0x7f9e58003a80 comp rx=0 tx=0).stop 2024-05-10T11:22:26.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.119+0000 7f9e81388b80 1 -- 172.21.15.27:0/1550564349 shutdown_connections 2024-05-10T11:22:26.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.119+0000 7f9e81388b80 1 --2- 172.21.15.27:0/1550564349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a464e0 0x5574e3a3a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.119+0000 7f9e81388b80 1 --2- 172.21.15.27:0/1550564349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a408f0 0x5574e3a55ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.119+0000 7f9e81388b80 1 --2- 172.21.15.27:0/1550564349 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a45bc0 0x5574e3a46fd0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.119+0000 7f9e81388b80 1 -- 172.21.15.27:0/1550564349 >> 172.21.15.27:0/1550564349 conn(0x5574e39b86d0 msgr2=0x5574e3a468c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:26.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.119+0000 7f9e81388b80 1 -- 172.21.15.27:0/1550564349 shutdown_connections 2024-05-10T11:22:26.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.119+0000 7f9e81388b80 1 -- 172.21.15.27:0/1550564349 wait complete. 2024-05-10T11:22:26.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.120+0000 7f9e81388b80 1 -- start start 2024-05-10T11:22:26.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.120+0000 7f9e81388b80 1 Processor -- start 2024-05-10T11:22:26.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.121+0000 7f9e81388b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 0x5574e3a526d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.121+0000 7f9e81388b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a45bc0 0x5574e3a60dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.121+0000 7f9e81388b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a464e0 0x5574e3a61500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.121+0000 7f9e81388b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9e701ae8e0 con 0x5574e3a464e0 2024-05-10T11:22:26.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.121+0000 7f9e81388b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9e701ae210 con 0x5574e3a408f0 2024-05-10T11:22:26.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.121+0000 7f9e81388b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e701ab400 con 0x5574e3a45bc0 2024-05-10T11:22:26.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.121+0000 7f9e81388b80 10 client.0 _ll_register_callbacks cb 0x5574e3939c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.121+0000 7f9e6fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 0x5574e3a526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.121+0000 7f9e6fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 0x5574e3a526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45236/0 (socket says 172.21.15.27:45236) 2024-05-10T11:22:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.121+0000 7f9e6fe00700 1 -- 172.21.15.27:0/4292962209 learned_addr learned my addr 172.21.15.27:0/4292962209 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.121+0000 7f9e67400700 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a45bc0 0x5574e3a60dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.121+0000 7f9e74a00700 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a464e0 0x5574e3a61500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.122+0000 7f9e67400700 1 -- 172.21.15.27:0/4292962209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 msgr2=0x5574e3a526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.122+0000 7f9e67400700 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 0x5574e3a526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.122+0000 7f9e67400700 1 -- 172.21.15.27:0/4292962209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a464e0 msgr2=0x5574e3a61500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.122+0000 7f9e67400700 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a464e0 0x5574e3a61500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.122+0000 7f9e67400700 1 -- 172.21.15.27:0/4292962209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574e3ac32e0 con 0x5574e3a45bc0 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.122+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 0x5574e3a526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.122+0000 7f9e67400700 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a45bc0 0x5574e3a60dc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9e58036040 tx=0x7f9e58021880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2392ca3665e8bd65 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.122+0000 7f9e5e000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.122+0000 7f9e5e000700 1 -- 172.21.15.27:0/4292962209 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58038040 con 0x5574e3a45bc0 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.122+0000 7f9e81388b80 1 -- 172.21.15.27:0/4292962209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e580041a0 con 0x5574e3a45bc0 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.122+0000 7f9e5e000700 1 -- 172.21.15.27:0/4292962209 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f9e58021c20 con 0x5574e3a45bc0 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.122+0000 7f9e5e000700 1 -- 172.21.15.27:0/4292962209 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9e58021460 con 0x5574e3a45bc0 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.123+0000 7f9e81388b80 10 client.26593 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.123+0000 7f9e81388b80 1 -- 172.21.15.27:0/4292962209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5574e3ac4470 con 0x5574e3a45bc0 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.123+0000 7f9e5e000700 1 -- 172.21.15.27:0/4292962209 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e580041a0 con 0x5574e3a45bc0 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.123+0000 7f9e5e000700 1 -- 172.21.15.27:0/4292962209 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f9e5802a940 con 0x5574e3a45bc0 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.123+0000 7f9e81388b80 10 client.26593 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.123+0000 7f9e81388b80 10 client.26593 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:22:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.123+0000 7f9e81388b80 20 client.26593 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.123+0000 7f9e81388b80 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5574e3ac4a60 0x5574e3ac6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.123+0000 7f9e81388b80 4 client.26593 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:22:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.123+0000 7f9e81388b80 1 -- 172.21.15.27:0/4292962209 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5574e39a0400 con 0x5574e3ac4a60 2024-05-10T11:22:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.124+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5574e3ac4a60 0x5574e3ac6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.124+0000 7f9e6fe00700 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5574e3ac4a60 0x5574e3ac6e50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e476306504f168ad server_cookie=fa62e4fafbc67486 in_seq=0 out_seq=0 2024-05-10T11:22:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.124+0000 7f9e5e000700 10 client.26593 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:22:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.124+0000 7f9e5e000700 1 -- 172.21.15.27:0/4292962209 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5574e39a0400 con 0x5574e3ac4a60 2024-05-10T11:22:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.124+0000 7f9e5e000700 10 client.26593 handle_command_reply: tid=0 2024-05-10T11:22:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.124+0000 7f9e81388b80 1 client.26593 shutdown 2024-05-10T11:22:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.124+0000 7f9e81388b80 1 -- 172.21.15.27:0/4292962209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a45bc0 msgr2=0x5574e3a60dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.124+0000 7f9e81388b80 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a45bc0 0x5574e3a60dc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9e58036040 tx=0x7f9e58021880 comp rx=0 tx=0).stop 2024-05-10T11:22:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.125+0000 7f9e81388b80 1 -- 172.21.15.27:0/4292962209 shutdown_connections 2024-05-10T11:22:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.125+0000 7f9e81388b80 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5574e3ac4a60 0x5574e3ac6e50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.125+0000 7f9e81388b80 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5574e3a464e0 0x5574e3a61500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.125+0000 7f9e81388b80 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574e3a408f0 0x5574e3a526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.125+0000 7f9e81388b80 1 --2- 172.21.15.27:0/4292962209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574e3a45bc0 0x5574e3a60dc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.125+0000 7f9e81388b80 1 -- 172.21.15.27:0/4292962209 >> 172.21.15.27:0/4292962209 conn(0x5574e39b86d0 msgr2=0x5574e3a481f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.125+0000 7f9e5e000700 0 client.26593 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:22:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.125+0000 7f9e81388b80 1 -- 172.21.15.27:0/4292962209 shutdown_connections 2024-05-10T11:22:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.125+0000 7f9e81388b80 1 -- 172.21.15.27:0/4292962209 wait complete. 2024-05-10T11:22:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.125+0000 7f9e81388b80 20 client.26593 trim_cache size 0 max 16384 2024-05-10T11:22:26.131 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:22:26.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.126+0000 7f9e76a00700 1 -- 172.21.15.27:0/1654386392 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9e540b2a70 msgr2=0x7f9e540b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:26.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.126+0000 7f9e76a00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9e540b2a70 0x7f9e540b4f30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009dc0 tx=0x7f9e68004750 comp rx=0 tx=0).stop 2024-05-10T11:22:26.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.126+0000 7f9e76a00700 1 -- 172.21.15.27:0/1654386392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e7006ba10 msgr2=0x7f9e7019f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:26.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.126+0000 7f9e76a00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e7006ba10 0x7f9e7019f7d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000d010 tx=0x7f9e60004d00 comp rx=0 tx=0).stop 2024-05-10T11:22:26.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.126+0000 7f9e76a00700 1 -- 172.21.15.27:0/1654386392 shutdown_connections 2024-05-10T11:22:26.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.126+0000 7f9e76a00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9e540b2a70 0x7f9e540b4f30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.126+0000 7f9e76a00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70109500 0x7f9e701999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.126+0000 7f9e76a00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e7006ba10 0x7f9e7019f7d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.126+0000 7f9e76a00700 1 --2- 172.21.15.27:0/1654386392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7006c3d0 0x7f9e7019fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:26.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.126+0000 7f9e76a00700 1 -- 172.21.15.27:0/1654386392 >> 172.21.15.27:0/1654386392 conn(0x7f9e70079310 msgr2=0x7f9e701010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:26.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.126+0000 7f9e76a00700 1 -- 172.21.15.27:0/1654386392 shutdown_connections 2024-05-10T11:22:26.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:26.127+0000 7f9e76a00700 1 -- 172.21.15.27:0/1654386392 wait complete. 2024-05-10T11:22:26.712 INFO:tasks.workunit.client.0.smithi027.stdout: 1 541894 6.54 MB/sec execute 311 sec latency 67.366 ms 2024-05-10T11:22:26.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:26 smithi027 ceph-mon[30054]: pgmap v285: 129 pgs: 129 active+clean; 388 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 208 op/s 2024-05-10T11:22:26.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:26 smithi076 ceph-mon[34152]: pgmap v285: 129 pgs: 129 active+clean; 388 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 208 op/s 2024-05-10T11:22:26.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:26 smithi189 ceph-mon[34842]: pgmap v285: 129 pgs: 129 active+clean; 388 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 208 op/s 2024-05-10T11:22:27.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:22:27.687 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:22:27.712 INFO:tasks.workunit.client.0.smithi027.stdout: 1 543608 6.56 MB/sec execute 312 sec latency 83.658 ms 2024-05-10T11:22:28.712 INFO:tasks.workunit.client.0.smithi027.stdout: 1 544255 6.54 MB/sec execute 313 sec latency 36.471 ms 2024-05-10T11:22:28.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:28 smithi027 ceph-mon[30054]: pgmap v286: 129 pgs: 129 active+clean; 388 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 134 op/s 2024-05-10T11:22:28.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:28 smithi076 ceph-mon[34152]: pgmap v286: 129 pgs: 129 active+clean; 388 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 134 op/s 2024-05-10T11:22:28.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:28 smithi189 ceph-mon[34842]: pgmap v286: 129 pgs: 129 active+clean; 388 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 134 op/s 2024-05-10T11:22:29.712 INFO:tasks.workunit.client.0.smithi027.stdout: 1 545499 6.54 MB/sec execute 314 sec latency 31.302 ms 2024-05-10T11:22:30.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.203+0000 7f2583200700 1 -- 172.21.15.27:0/1114851230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c103d90 msgr2=0x7f257c1102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:30.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.203+0000 7f2583200700 1 --2- 172.21.15.27:0/1114851230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c103d90 0x7f257c1102a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2578009f90 tx=0x7f257800b330 comp rx=0 tx=0).stop 2024-05-10T11:22:30.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.204+0000 7f2583200700 1 -- 172.21.15.27:0/1114851230 shutdown_connections 2024-05-10T11:22:30.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.204+0000 7f2583200700 1 --2- 172.21.15.27:0/1114851230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c103d90 0x7f257c1102a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.204+0000 7f2583200700 1 --2- 172.21.15.27:0/1114851230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f257c109460 0x7f257c109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.204+0000 7f2583200700 1 --2- 172.21.15.27:0/1114851230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f257c1033d0 0x7f257c103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.204+0000 7f2583200700 1 -- 172.21.15.27:0/1114851230 >> 172.21.15.27:0/1114851230 conn(0x7f257c0fef20 msgr2=0x7f257c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:30.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.204+0000 7f2583200700 1 -- 172.21.15.27:0/1114851230 shutdown_connections 2024-05-10T11:22:30.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.204+0000 7f2583200700 1 -- 172.21.15.27:0/1114851230 wait complete. 2024-05-10T11:22:30.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.206+0000 7f2583200700 1 Processor -- start 2024-05-10T11:22:30.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.206+0000 7f2583200700 1 -- start start 2024-05-10T11:22:30.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.206+0000 7f2583200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c1033d0 0x7f257c19f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.206+0000 7f2583200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f257c103d90 0x7f257c19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2583200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f257c109460 0x7f257c1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2583200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f257c1135c0 con 0x7f257c103d90 2024-05-10T11:22:30.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2583200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f257c113440 con 0x7f257c109460 2024-05-10T11:22:30.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2583200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f257c113740 con 0x7f257c1033d0 2024-05-10T11:22:30.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2581e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c1033d0 0x7f257c19f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2582800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f257c109460 0x7f257c1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2581e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c1033d0 0x7f257c19f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42242/0 (socket says 172.21.15.27:42242) 2024-05-10T11:22:30.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2581e00700 1 -- 172.21.15.27:0/92838286 learned_addr learned my addr 172.21.15.27:0/92838286 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:30.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2581400700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f257c103d90 0x7f257c19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2581e00700 1 -- 172.21.15.27:0/92838286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f257c109460 msgr2=0x7f257c1998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:30.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2581e00700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f257c109460 0x7f257c1998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2581e00700 1 -- 172.21.15.27:0/92838286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f257c103d90 msgr2=0x7f257c19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:30.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2581e00700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f257c103d90 0x7f257c19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.207+0000 7f2581e00700 1 -- 172.21.15.27:0/92838286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257c19a020 con 0x7f257c1033d0 2024-05-10T11:22:30.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.208+0000 7f2581e00700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c1033d0 0x7f257c19f6b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f257000e9c0 tx=0x7f257000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:30.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.208+0000 7f2576a00700 1 -- 172.21.15.27:0/92838286 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257000cbd0 con 0x7f257c1033d0 2024-05-10T11:22:30.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.208+0000 7f2583200700 1 -- 172.21.15.27:0/92838286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f257c19a310 con 0x7f257c1033d0 2024-05-10T11:22:30.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.208+0000 7f2576a00700 1 -- 172.21.15.27:0/92838286 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f257000cd70 con 0x7f257c1033d0 2024-05-10T11:22:30.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.208+0000 7f2576a00700 1 -- 172.21.15.27:0/92838286 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2570005550 con 0x7f257c1033d0 2024-05-10T11:22:30.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.208+0000 7f2583200700 1 -- 172.21.15.27:0/92838286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f257c0756d0 con 0x7f257c1033d0 2024-05-10T11:22:30.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.212+0000 7f2576a00700 1 -- 172.21.15.27:0/92838286 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2570004750 con 0x7f257c1033d0 2024-05-10T11:22:30.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.213+0000 7f258f0dbb80 1 Processor -- start 2024-05-10T11:22:30.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.214+0000 7f2576a00700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f25640b29f0 0x7f25640b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.214+0000 7f2576a00700 1 -- 172.21.15.27:0/92838286 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f25700e2c50 con 0x7f257c1033d0 2024-05-10T11:22:30.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.214+0000 7f258f0dbb80 1 -- start start 2024-05-10T11:22:30.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.214+0000 7f2582800700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f25640b29f0 0x7f25640b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.214+0000 7f258f0dbb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7e5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.214+0000 7f258f0dbb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7dffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.214+0000 7f258f0dbb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7e0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.214+0000 7f258f0dbb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56032a732a20 con 0x56032a7e59e0 2024-05-10T11:22:30.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.214+0000 7f258f0dbb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56032a6d0760 con 0x56032a7e0600 2024-05-10T11:22:30.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.214+0000 7f258f0dbb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56032a72e9e0 con 0x56032a7e6300 2024-05-10T11:22:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.215+0000 7f2581400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7dffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.215+0000 7f2581e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7e5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.215+0000 7f2581400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7dffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42250/0 (socket says 172.21.15.27:42250) 2024-05-10T11:22:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.215+0000 7f2581400700 1 -- 172.21.15.27:0/694740031 learned_addr learned my addr 172.21.15.27:0/694740031 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.215+0000 7f2582800700 1 --2- 172.21.15.27:0/694740031 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7e0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.215+0000 7f2581400700 1 -- 172.21.15.27:0/694740031 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 msgr2=0x56032a7e0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.215+0000 7f2581400700 1 --2- 172.21.15.27:0/694740031 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7e0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.215+0000 7f2581400700 1 -- 172.21.15.27:0/694740031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 msgr2=0x56032a7e5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.215+0000 7f2581400700 1 --2- 172.21.15.27:0/694740031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7e5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.215+0000 7f2581400700 1 -- 172.21.15.27:0/694740031 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56032a67f8d0 con 0x56032a7e6300 2024-05-10T11:22:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.215+0000 7f2582800700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f25640b29f0 0x7f25640b4eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2578009da0 tx=0x7f2578009700 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.216+0000 7f2581400700 1 --2- 172.21.15.27:0/694740031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7dffe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f256c009cb0 tx=0x7f256c0113e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7d1453c78ef66da server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:30.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.216+0000 7f256b400700 1 -- 172.21.15.27:0/694740031 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c011c60 con 0x56032a7e6300 2024-05-10T11:22:30.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.216+0000 7f256b400700 1 -- 172.21.15.27:0/694740031 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f256c011e00 con 0x56032a7e6300 2024-05-10T11:22:30.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.216+0000 7f256b400700 1 -- 172.21.15.27:0/694740031 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c00f7c0 con 0x56032a7e6300 2024-05-10T11:22:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.217+0000 7f258f0dbb80 1 -- 172.21.15.27:0/694740031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 msgr2=0x56032a7dffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.217+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/694740031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7dffe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f256c009cb0 tx=0x7f256c0113e0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.217+0000 7f258f0dbb80 1 -- 172.21.15.27:0/694740031 shutdown_connections 2024-05-10T11:22:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.217+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/694740031 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7e0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.217+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/694740031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7e5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.217+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/694740031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7dffe0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.217+0000 7f258f0dbb80 1 -- 172.21.15.27:0/694740031 >> 172.21.15.27:0/694740031 conn(0x56032a70eb60 msgr2=0x56032a7da9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.217+0000 7f258f0dbb80 1 -- 172.21.15.27:0/694740031 shutdown_connections 2024-05-10T11:22:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.217+0000 7f258f0dbb80 1 -- 172.21.15.27:0/694740031 wait complete. 2024-05-10T11:22:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.218+0000 7f258f0dbb80 1 -- start start 2024-05-10T11:22:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.218+0000 7f258f0dbb80 1 Processor -- start 2024-05-10T11:22:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.219+0000 7f258f0dbb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7ebf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.219+0000 7f258f0dbb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7e92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.219+0000 7f258f0dbb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7e9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.219+0000 7f258f0dbb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56032a7ecdb0 con 0x56032a7e59e0 2024-05-10T11:22:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.219+0000 7f258f0dbb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56032a698b20 con 0x56032a7e0600 2024-05-10T11:22:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.219+0000 7f258f0dbb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56032a7ecf30 con 0x56032a7e6300 2024-05-10T11:22:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.219+0000 7f258f0dbb80 10 client.0 _ll_register_callbacks cb 0x56032a6d9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.219+0000 7f2581400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7e92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.220+0000 7f2581400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7e92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35468/0 (socket says 172.21.15.27:35468) 2024-05-10T11:22:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.220+0000 7f2581400700 1 -- 172.21.15.27:0/402687040 learned_addr learned my addr 172.21.15.27:0/402687040 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.220+0000 7f2581e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7ebf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.220+0000 7f2582800700 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7e9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.220+0000 7f2581400700 1 -- 172.21.15.27:0/402687040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 msgr2=0x56032a7e9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.220+0000 7f2581400700 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7e9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.220+0000 7f2581400700 1 -- 172.21.15.27:0/402687040 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 msgr2=0x56032a7ebf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.220+0000 7f2581400700 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7ebf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.220+0000 7f2581400700 1 -- 172.21.15.27:0/402687040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56032a6f8420 con 0x56032a7e59e0 2024-05-10T11:22:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.220+0000 7f2581e00700 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7ebf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.221+0000 7f2581400700 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7e92e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f256c012f70 tx=0x7f256c004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94238030fce0504 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.221+0000 7f255fe00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:22:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.221+0000 7f255fe00700 1 -- 172.21.15.27:0/402687040 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c027020 con 0x56032a7e59e0 2024-05-10T11:22:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.221+0000 7f255fe00700 1 -- 172.21.15.27:0/402687040 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f256c005250 con 0x56032a7e59e0 2024-05-10T11:22:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.221+0000 7f258f0dbb80 1 -- 172.21.15.27:0/402687040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f256c011e00 con 0x56032a7e59e0 2024-05-10T11:22:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.221+0000 7f255fe00700 1 -- 172.21.15.27:0/402687040 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f256c005450 con 0x56032a7e59e0 2024-05-10T11:22:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.221+0000 7f258f0dbb80 10 client.16404 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.221+0000 7f258f0dbb80 1 -- 172.21.15.27:0/402687040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56032a7ffd20 con 0x56032a7e59e0 2024-05-10T11:22:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.222+0000 7f255fe00700 1 -- 172.21.15.27:0/402687040 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f256c011e00 con 0x56032a7e59e0 2024-05-10T11:22:30.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.222+0000 7f255fe00700 1 -- 172.21.15.27:0/402687040 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f256c0048f0 con 0x56032a7e59e0 2024-05-10T11:22:30.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.222+0000 7f258f0dbb80 10 client.16404 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:30.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.222+0000 7f258f0dbb80 10 client.16404 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:22:30.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.222+0000 7f258f0dbb80 20 client.16404 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:30.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.222+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x56032a8021f0 0x56032a8045e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.222+0000 7f258f0dbb80 4 client.16404 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:22:30.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.222+0000 7f258f0dbb80 1 -- 172.21.15.27:0/402687040 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56032a7ad0a0 con 0x56032a8021f0 2024-05-10T11:22:30.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.223+0000 7f2581e00700 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x56032a8021f0 0x56032a8045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.223+0000 7f2581e00700 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x56032a8021f0 0x56032a8045e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d3a04300aea00935 server_cookie=55103d786627cc2f in_seq=0 out_seq=0 2024-05-10T11:22:30.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.223+0000 7f255fe00700 10 client.16404 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:22:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.224+0000 7f255fe00700 1 -- 172.21.15.27:0/402687040 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56032a7ad0a0 con 0x56032a8021f0 2024-05-10T11:22:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.224+0000 7f255fe00700 10 client.16404 handle_command_reply: tid=0 2024-05-10T11:22:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.224+0000 7f258f0dbb80 1 client.16404 shutdown 2024-05-10T11:22:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.224+0000 7f258f0dbb80 1 -- 172.21.15.27:0/402687040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 msgr2=0x56032a7e92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.224+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7e92e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f256c012f70 tx=0x7f256c004250 comp rx=0 tx=0).stop 2024-05-10T11:22:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.225+0000 7f258f0dbb80 1 -- 172.21.15.27:0/402687040 shutdown_connections 2024-05-10T11:22:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.225+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x56032a8021f0 0x56032a8045e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.225+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7e9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.225+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7ebf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.225+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/402687040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7e92e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.225+0000 7f258f0dbb80 1 -- 172.21.15.27:0/402687040 >> 172.21.15.27:0/402687040 conn(0x56032a70eb60 msgr2=0x56032a7d5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.225+0000 7f255fe00700 0 client.16404 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:22:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.225+0000 7f258f0dbb80 1 -- 172.21.15.27:0/402687040 shutdown_connections 2024-05-10T11:22:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.225+0000 7f258f0dbb80 1 -- 172.21.15.27:0/402687040 wait complete. 2024-05-10T11:22:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.225+0000 7f258f0dbb80 20 client.16404 trim_cache size 0 max 16384 2024-05-10T11:22:30.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.243+0000 7f258f0dbb80 1 Processor -- start 2024-05-10T11:22:30.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.243+0000 7f258f0dbb80 1 -- start start 2024-05-10T11:22:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.244+0000 7f258f0dbb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e0600 0x56032a7dffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.244+0000 7f258f0dbb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e59e0 0x56032a7ebf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.244+0000 7f258f0dbb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7e7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.244+0000 7f258f0dbb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f257c1ad4e0 con 0x56032a7e0600 2024-05-10T11:22:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.244+0000 7f258f0dbb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f257c1af050 con 0x56032a7e59e0 2024-05-10T11:22:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.244+0000 7f258f0dbb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f257c1adbb0 con 0x56032a7e6300 2024-05-10T11:22:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.244+0000 7f2581400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e59e0 0x56032a7ebf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.244+0000 7f2582800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7e7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.244+0000 7f2582800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7e7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:42266/0 (socket says 172.21.15.27:42266) 2024-05-10T11:22:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.244+0000 7f2581400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e59e0 0x56032a7ebf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45268/0 (socket says 172.21.15.27:45268) 2024-05-10T11:22:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.244+0000 7f2582800700 1 -- 172.21.15.27:0/1723390717 learned_addr learned my addr 172.21.15.27:0/1723390717 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.244+0000 7f2581e00700 1 --2- 172.21.15.27:0/1723390717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e0600 0x56032a7dffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.245+0000 7f2582800700 1 -- 172.21.15.27:0/1723390717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e59e0 msgr2=0x56032a7ebf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.245+0000 7f2582800700 1 --2- 172.21.15.27:0/1723390717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e59e0 0x56032a7ebf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.245+0000 7f2582800700 1 -- 172.21.15.27:0/1723390717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e0600 msgr2=0x56032a7dffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.245+0000 7f2582800700 1 --2- 172.21.15.27:0/1723390717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e0600 0x56032a7dffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.245+0000 7f2582800700 1 -- 172.21.15.27:0/1723390717 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56032a7dbaa0 con 0x56032a7e6300 2024-05-10T11:22:30.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.245+0000 7f2582800700 1 --2- 172.21.15.27:0/1723390717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7e7bf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2578015000 tx=0x7f2578026000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f2b25cf76d47ae1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:30.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.246+0000 7f256aa00700 1 -- 172.21.15.27:0/1723390717 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257800bbf0 con 0x56032a7e6300 2024-05-10T11:22:30.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.246+0000 7f256aa00700 1 -- 172.21.15.27:0/1723390717 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f257802d070 con 0x56032a7e6300 2024-05-10T11:22:30.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.246+0000 7f256aa00700 1 -- 172.21.15.27:0/1723390717 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2578005310 con 0x56032a7e6300 2024-05-10T11:22:30.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.247+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1723390717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 msgr2=0x56032a7e7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.247+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/1723390717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7e7bf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2578015000 tx=0x7f2578026000 comp rx=0 tx=0).stop 2024-05-10T11:22:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.247+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1723390717 shutdown_connections 2024-05-10T11:22:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.247+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/1723390717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a7e7bf0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.247+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/1723390717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e0600 0x56032a7dffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.247+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/1723390717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e59e0 0x56032a7ebf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.247+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1723390717 >> 172.21.15.27:0/1723390717 conn(0x56032a70eb60 msgr2=0x56032a7e34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.247+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1723390717 shutdown_connections 2024-05-10T11:22:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.247+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1723390717 wait complete. 2024-05-10T11:22:30.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.247+0000 7f258f0dbb80 1 -- start start 2024-05-10T11:22:30.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.248+0000 7f258f0dbb80 1 Processor -- start 2024-05-10T11:22:30.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.248+0000 7f258f0dbb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7faa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.248+0000 7f258f0dbb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7faf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.248+0000 7f258f0dbb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a808ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.248+0000 7f258f0dbb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f257c1ae980 con 0x56032a7e59e0 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.248+0000 7f258f0dbb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f257c1ae2b0 con 0x56032a7e0600 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.248+0000 7f258f0dbb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f257c1ab4a0 con 0x56032a7e6300 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.248+0000 7f258f0dbb80 10 client.0 _ll_register_callbacks cb 0x56032a6d9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.249+0000 7f2581e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7faa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.249+0000 7f2581400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7faf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.249+0000 7f2581e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7faa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45280/0 (socket says 172.21.15.27:45280) 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.249+0000 7f2581e00700 1 -- 172.21.15.27:0/1401521911 learned_addr learned my addr 172.21.15.27:0/1401521911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.249+0000 7f2582800700 1 --2- 172.21.15.27:0/1401521911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a808ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.249+0000 7f2581e00700 1 -- 172.21.15.27:0/1401521911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 msgr2=0x56032a808ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.249+0000 7f2581e00700 1 --2- 172.21.15.27:0/1401521911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a808ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.249+0000 7f2581e00700 1 -- 172.21.15.27:0/1401521911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 msgr2=0x56032a7faf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.249+0000 7f2581e00700 1 --2- 172.21.15.27:0/1401521911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7faf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.249+0000 7f2581e00700 1 -- 172.21.15.27:0/1401521911 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56032a8090e0 con 0x56032a7e0600 2024-05-10T11:22:30.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.249+0000 7f2581e00700 1 --2- 172.21.15.27:0/1401521911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7faa50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f25700a9a30 tx=0x7f2570091040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28be6f2f39f1ddad server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:30.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.249+0000 7f255fe00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:22:30.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.250+0000 7f255fe00700 1 -- 172.21.15.27:0/1401521911 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25700c2070 con 0x56032a7e0600 2024-05-10T11:22:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.250+0000 7f255fe00700 1 -- 172.21.15.27:0/1401521911 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f25700b2040 con 0x56032a7e0600 2024-05-10T11:22:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.250+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1401521911 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f257802d070 con 0x56032a7e0600 2024-05-10T11:22:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.250+0000 7f255fe00700 1 -- 172.21.15.27:0/1401521911 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f25700aa1a0 con 0x56032a7e0600 2024-05-10T11:22:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.250+0000 7f258f0dbb80 10 client.26627 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.250+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1401521911 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56032a86bf60 con 0x56032a7e0600 2024-05-10T11:22:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.250+0000 7f255fe00700 1 -- 172.21.15.27:0/1401521911 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f257802d070 con 0x56032a7e0600 2024-05-10T11:22:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.250+0000 7f255fe00700 1 -- 172.21.15.27:0/1401521911 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f25700cc0b0 con 0x56032a7e0600 2024-05-10T11:22:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.250+0000 7f258f0dbb80 10 client.26627 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.250+0000 7f258f0dbb80 10 client.26627 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:22:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.250+0000 7f258f0dbb80 20 client.26627 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.251+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/1401521911 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x56032a86c650 0x56032a86ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.251+0000 7f258f0dbb80 4 client.26627 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.251+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1401521911 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56032a740400 con 0x56032a86c650 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.251+0000 7f2581400700 1 --2- 172.21.15.27:0/1401521911 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x56032a86c650 0x56032a86ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.251+0000 7f2581400700 1 --2- 172.21.15.27:0/1401521911 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x56032a86c650 0x56032a86ea70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=301f47adeab976f5 server_cookie=a6ea290392af5dea in_seq=0 out_seq=0 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.251+0000 7f255fe00700 10 client.26627 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f255fe00700 1 -- 172.21.15.27:0/1401521911 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56032a740400 con 0x56032a86c650 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f255fe00700 10 client.26627 handle_command_reply: tid=0 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f258f0dbb80 1 client.26627 shutdown 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1401521911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 msgr2=0x56032a7faa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/1401521911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7faa50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f25700a9a30 tx=0x7f2570091040 comp rx=0 tx=0).stop 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1401521911 shutdown_connections 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/1401521911 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x56032a86c650 0x56032a86ea70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/1401521911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56032a7e6300 0x56032a808ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/1401521911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56032a7e0600 0x56032a7faa50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f258f0dbb80 1 --2- 172.21.15.27:0/1401521911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56032a7e59e0 0x56032a7faf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f255fe00700 0 client.26627 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:22:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1401521911 >> 172.21.15.27:0/1401521911 conn(0x56032a70eb60 msgr2=0x56032a7e7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1401521911 shutdown_connections 2024-05-10T11:22:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.252+0000 7f258f0dbb80 1 -- 172.21.15.27:0/1401521911 wait complete. 2024-05-10T11:22:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.253+0000 7f258f0dbb80 20 client.26627 trim_cache size 0 max 16384 2024-05-10T11:22:30.256 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:22:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.253+0000 7f2583200700 1 -- 172.21.15.27:0/92838286 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f25640b29f0 msgr2=0x7f25640b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.253+0000 7f2583200700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f25640b29f0 0x7f25640b4eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2578009da0 tx=0x7f2578009700 comp rx=0 tx=0).stop 2024-05-10T11:22:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.253+0000 7f2583200700 1 -- 172.21.15.27:0/92838286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c1033d0 msgr2=0x7f257c19f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.253+0000 7f2583200700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c1033d0 0x7f257c19f6b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f257000e9c0 tx=0x7f257000ecd0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.253+0000 7f2583200700 1 -- 172.21.15.27:0/92838286 shutdown_connections 2024-05-10T11:22:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.253+0000 7f2583200700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f25640b29f0 0x7f25640b4eb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.253+0000 7f2583200700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c1033d0 0x7f257c19f6b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.253+0000 7f2583200700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f257c109460 0x7f257c1998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.253+0000 7f2583200700 1 --2- 172.21.15.27:0/92838286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f257c103d90 0x7f257c19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.253+0000 7f2583200700 1 -- 172.21.15.27:0/92838286 >> 172.21.15.27:0/92838286 conn(0x7f257c0fef20 msgr2=0x7f257c0ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.254+0000 7f2583200700 1 -- 172.21.15.27:0/92838286 shutdown_connections 2024-05-10T11:22:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:30.254+0000 7f2583200700 1 -- 172.21.15.27:0/92838286 wait complete. 2024-05-10T11:22:30.712 INFO:tasks.workunit.client.0.smithi027.stdout: 1 547226 6.56 MB/sec execute 315 sec latency 22.579 ms 2024-05-10T11:22:30.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:30 smithi076 ceph-mon[34152]: pgmap v287: 129 pgs: 129 active+clean; 394 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 204 op/s 2024-05-10T11:22:30.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:30 smithi189 ceph-mon[34842]: pgmap v287: 129 pgs: 129 active+clean; 394 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 204 op/s 2024-05-10T11:22:31.086 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:30 smithi027 ceph-mon[30054]: pgmap v287: 129 pgs: 129 active+clean; 394 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 204 op/s 2024-05-10T11:22:31.149 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:22:31.642 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:22:31.716 INFO:tasks.workunit.client.0.smithi027.stdout: 1 547879 6.54 MB/sec execute 316 sec latency 65.025 ms 2024-05-10T11:22:32.712 INFO:tasks.workunit.client.0.smithi027.stdout: 1 549455 6.54 MB/sec execute 317 sec latency 27.117 ms 2024-05-10T11:22:32.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:32 smithi076 ceph-mon[34152]: pgmap v288: 129 pgs: 129 active+clean; 394 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 135 op/s 2024-05-10T11:22:32.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:32 smithi189 ceph-mon[34842]: pgmap v288: 129 pgs: 129 active+clean; 394 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 135 op/s 2024-05-10T11:22:33.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:32 smithi027 ceph-mon[30054]: pgmap v288: 129 pgs: 129 active+clean; 394 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 135 op/s 2024-05-10T11:22:33.712 INFO:tasks.workunit.client.0.smithi027.stdout: 1 550943 6.56 MB/sec execute 318 sec latency 16.037 ms 2024-05-10T11:22:34.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.197+0000 7fb9d8600700 1 -- 172.21.15.27:0/3549741865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9d0109510 msgr2=0x7fb9d01098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:34.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.197+0000 7fb9d8600700 1 --2- 172.21.15.27:0/3549741865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9d0109510 0x7fb9d01098f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400b0a0 tx=0x7fb9c400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.197+0000 7fb9d8600700 1 -- 172.21.15.27:0/3549741865 shutdown_connections 2024-05-10T11:22:34.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.197+0000 7fb9d8600700 1 --2- 172.21.15.27:0/3549741865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d006bb40 0x7fb9d0110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.197+0000 7fb9d8600700 1 --2- 172.21.15.27:0/3549741865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9d0109510 0x7fb9d01098f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.197+0000 7fb9d8600700 1 --2- 172.21.15.27:0/3549741865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9d006b180 0x7fb9d006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.197+0000 7fb9d8600700 1 -- 172.21.15.27:0/3549741865 >> 172.21.15.27:0/3549741865 conn(0x7fb9d0079350 msgr2=0x7fb9d0079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.197+0000 7fb9d8600700 1 -- 172.21.15.27:0/3549741865 shutdown_connections 2024-05-10T11:22:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.197+0000 7fb9d8600700 1 -- 172.21.15.27:0/3549741865 wait complete. 2024-05-10T11:22:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.199+0000 7fb9d8600700 1 Processor -- start 2024-05-10T11:22:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.199+0000 7fb9d8600700 1 -- start start 2024-05-10T11:22:34.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.200+0000 7fb9d8600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9d006b180 0x7fb9d019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.200+0000 7fb9d8600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d006bb40 0x7fb9d019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.200+0000 7fb9d8600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9d0109510 0x7fb9d0199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.200+0000 7fb9d8600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d0113640 con 0x7fb9d006b180 2024-05-10T11:22:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.200+0000 7fb9d8600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01134c0 con 0x7fb9d0109510 2024-05-10T11:22:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.200+0000 7fb9d8600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01137c0 con 0x7fb9d006bb40 2024-05-10T11:22:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.200+0000 7fb9d5c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9d006b180 0x7fb9d019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.200+0000 7fb9d5c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9d006b180 0x7fb9d019f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54746/0 (socket says 172.21.15.27:54746) 2024-05-10T11:22:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.200+0000 7fb9d5c00700 1 -- 172.21.15.27:0/2841734812 learned_addr learned my addr 172.21.15.27:0/2841734812 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.200+0000 7fb9d5200700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d006bb40 0x7fb9d019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.200+0000 7fb9d6600700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9d0109510 0x7fb9d0199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.201+0000 7fb9d5200700 1 -- 172.21.15.27:0/2841734812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9d0109510 msgr2=0x7fb9d0199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.201+0000 7fb9d5200700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9d0109510 0x7fb9d0199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.201+0000 7fb9d5200700 1 -- 172.21.15.27:0/2841734812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9d006b180 msgr2=0x7fb9d019f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.201+0000 7fb9d5200700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9d006b180 0x7fb9d019f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.201+0000 7fb9d5200700 1 -- 172.21.15.27:0/2841734812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d019a0e0 con 0x7fb9d006bb40 2024-05-10T11:22:34.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.201+0000 7fb9d5c00700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9d006b180 0x7fb9d019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:34.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.201+0000 7fb9d5200700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d006bb40 0x7fb9d019fd00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00d950 tx=0x7fb9bc00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:34.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.201+0000 7fb9ce000700 1 -- 172.21.15.27:0/2841734812 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc016070 con 0x7fb9d006bb40 2024-05-10T11:22:34.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.201+0000 7fb9ce000700 1 -- 172.21.15.27:0/2841734812 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9bc00fd40 con 0x7fb9d006bb40 2024-05-10T11:22:34.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.201+0000 7fb9ce000700 1 -- 172.21.15.27:0/2841734812 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc00e3b0 con 0x7fb9d006bb40 2024-05-10T11:22:34.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.201+0000 7fb9d8600700 1 -- 172.21.15.27:0/2841734812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d019a3d0 con 0x7fb9d006bb40 2024-05-10T11:22:34.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.202+0000 7fb9d8600700 1 -- 172.21.15.27:0/2841734812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9d01a6690 con 0x7fb9d006bb40 2024-05-10T11:22:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.206+0000 7fb9ce000700 1 -- 172.21.15.27:0/2841734812 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb9bc019070 con 0x7fb9d006bb40 2024-05-10T11:22:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.206+0000 7fb9e2ebbb80 1 Processor -- start 2024-05-10T11:22:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9e2ebbb80 1 -- start start 2024-05-10T11:22:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9e2ebbb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149ebc0 0x55c48149efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9e2ebbb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149f4e0 0x55c4814992d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9e2ebbb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c481499d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9e2ebbb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c4813eba20 con 0x55c48149ebc0 2024-05-10T11:22:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9e2ebbb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c481389760 con 0x55c4814998f0 2024-05-10T11:22:34.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9e2ebbb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c4813e79e0 con 0x55c48149f4e0 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9d6600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149ebc0 0x55c48149efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9d5c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149f4e0 0x55c4814992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9d6600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149ebc0 0x55c48149efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54752/0 (socket says 172.21.15.27:54752) 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9d6600700 1 -- 172.21.15.27:0/278408024 learned_addr learned my addr 172.21.15.27:0/278408024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9d5200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c481499d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.207+0000 7fb9ce000700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb9b80b29f0 0x7fb9b80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.208+0000 7fb9d5c00700 1 -- 172.21.15.27:0/278408024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 msgr2=0x55c481499d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.208+0000 7fb9d5c00700 1 --2- 172.21.15.27:0/278408024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c481499d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.208+0000 7fb9d5c00700 1 -- 172.21.15.27:0/278408024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149ebc0 msgr2=0x55c48149efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.208+0000 7fb9d5c00700 1 --2- 172.21.15.27:0/278408024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149ebc0 0x55c48149efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.208+0000 7fb9d5c00700 1 -- 172.21.15.27:0/278408024 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4813388d0 con 0x55c48149f4e0 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.208+0000 7fb9ce000700 1 -- 172.21.15.27:0/2841734812 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7fb9bc0e24b0 con 0x7fb9d006bb40 2024-05-10T11:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.208+0000 7fb9d6600700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb9b80b29f0 0x7fb9b80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.208+0000 7fb9d5c00700 1 --2- 172.21.15.27:0/278408024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149f4e0 0x55c4814992d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb9d019aac0 tx=0x7fb9c40092e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=857b6ea93f50cc50 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.209+0000 7fb9c2a00700 1 -- 172.21.15.27:0/278408024 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4021070 con 0x55c48149f4e0 2024-05-10T11:22:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.209+0000 7fb9c2a00700 1 -- 172.21.15.27:0/278408024 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9c4003c00 con 0x55c48149f4e0 2024-05-10T11:22:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.209+0000 7fb9c2a00700 1 -- 172.21.15.27:0/278408024 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4005040 con 0x55c48149f4e0 2024-05-10T11:22:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.210+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/278408024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149f4e0 msgr2=0x55c4814992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.210+0000 7fb9d6600700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb9b80b29f0 0x7fb9b80b4eb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009fb0 tx=0x7fb9c8004750 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.210+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/278408024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149f4e0 0x55c4814992d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb9d019aac0 tx=0x7fb9c40092e0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.210+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/278408024 shutdown_connections 2024-05-10T11:22:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.210+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/278408024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c481499d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.210+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/278408024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149ebc0 0x55c48149efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.210+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/278408024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149f4e0 0x55c4814992d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.210+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/278408024 >> 172.21.15.27:0/278408024 conn(0x55c4814116d0 msgr2=0x55c481493b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.210+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/278408024 shutdown_connections 2024-05-10T11:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.210+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/278408024 wait complete. 2024-05-10T11:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.211+0000 7fb9e2ebbb80 1 -- start start 2024-05-10T11:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.211+0000 7fb9e2ebbb80 1 Processor -- start 2024-05-10T11:22:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.211+0000 7fb9e2ebbb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4814998f0 0x55c4814a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.211+0000 7fb9e2ebbb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c48149ebc0 0x55c4814a2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.212+0000 7fb9e2ebbb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c4814a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.212+0000 7fb9e2ebbb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c4814a5ec0 con 0x55c48149f4e0 2024-05-10T11:22:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.212+0000 7fb9e2ebbb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c481351b20 con 0x55c48149ebc0 2024-05-10T11:22:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.212+0000 7fb9e2ebbb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c4814a6040 con 0x55c4814998f0 2024-05-10T11:22:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.212+0000 7fb9e2ebbb80 10 client.0 _ll_register_callbacks cb 0x55c481392c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.212+0000 7fb9d6600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c4814a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.212+0000 7fb9d5c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4814998f0 0x55c4814a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.212+0000 7fb9d5c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4814998f0 0x55c4814a51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:53100/0 (socket says 172.21.15.27:53100) 2024-05-10T11:22:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.212+0000 7fb9d5c00700 1 -- 172.21.15.27:0/1880668789 learned_addr learned my addr 172.21.15.27:0/1880668789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.212+0000 7fb9d5200700 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c48149ebc0 0x55c4814a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.213+0000 7fb9d5c00700 1 -- 172.21.15.27:0/1880668789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c48149ebc0 msgr2=0x55c4814a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.213+0000 7fb9d5c00700 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c48149ebc0 0x55c4814a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.213+0000 7fb9d5c00700 1 -- 172.21.15.27:0/1880668789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 msgr2=0x55c4814a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.213+0000 7fb9d5c00700 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c4814a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.213+0000 7fb9d5c00700 1 -- 172.21.15.27:0/1880668789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4813b1420 con 0x55c4814998f0 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.213+0000 7fb9d6600700 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c4814a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.213+0000 7fb9d5c00700 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4814998f0 0x55c4814a51a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4000f80 tx=0x7fb9c40076d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5548274092f2b838 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.213+0000 7fb9b7400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.214+0000 7fb9b7400700 1 -- 172.21.15.27:0/1880668789 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4029070 con 0x55c4814998f0 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.214+0000 7fb9b7400700 1 -- 172.21.15.27:0/1880668789 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7fb9c4003a00 con 0x55c4814998f0 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.214+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/1880668789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9c4003c00 con 0x55c4814998f0 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.214+0000 7fb9b7400700 1 -- 172.21.15.27:0/1880668789 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb9c4035050 con 0x55c4814998f0 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.214+0000 7fb9e2ebbb80 10 client.26623 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.214+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/1880668789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55c4814a5ec0 con 0x55c4814998f0 2024-05-10T11:22:34.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.215+0000 7fb9b7400700 1 -- 172.21.15.27:0/1880668789 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9c4003c00 con 0x55c4814998f0 2024-05-10T11:22:34.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.215+0000 7fb9b7400700 1 -- 172.21.15.27:0/1880668789 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fb9c4011070 con 0x55c4814998f0 2024-05-10T11:22:34.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.215+0000 7fb9e2ebbb80 10 client.26623 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:34.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.215+0000 7fb9e2ebbb80 10 client.26623 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:22:34.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.215+0000 7fb9e2ebbb80 20 client.26623 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:34.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.215+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c4814b2a00 0x55c4814b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.215+0000 7fb9e2ebbb80 4 client.26623 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:22:34.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.215+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/1880668789 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c481466140 con 0x55c4814b2a00 2024-05-10T11:22:34.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.215+0000 7fb9d5200700 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c4814b2a00 0x55c4814b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.216+0000 7fb9d5200700 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c4814b2a00 0x55c4814b4df0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1b7c78092367ed18 server_cookie=5bb49fd270004f03 in_seq=0 out_seq=0 2024-05-10T11:22:34.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.216+0000 7fb9b7400700 10 client.26623 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:22:34.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.217+0000 7fb9b7400700 1 -- 172.21.15.27:0/1880668789 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c481466140 con 0x55c4814b2a00 2024-05-10T11:22:34.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.217+0000 7fb9b7400700 10 client.26623 handle_command_reply: tid=0 2024-05-10T11:22:34.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.217+0000 7fb9e2ebbb80 1 client.26623 shutdown 2024-05-10T11:22:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.217+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/1880668789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4814998f0 msgr2=0x55c4814a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.217+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4814998f0 0x55c4814a51a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4000f80 tx=0x7fb9c40076d0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.217+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/1880668789 shutdown_connections 2024-05-10T11:22:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.217+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c4814b2a00 0x55c4814b4df0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.217+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4814998f0 0x55c4814a51a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.217+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c4814a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.217+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/1880668789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c48149ebc0 0x55c4814a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.217+0000 7fb9b7400700 0 client.26623 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:22:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.217+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/1880668789 >> 172.21.15.27:0/1880668789 conn(0x55c4814116d0 msgr2=0x55c481493190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.218+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/1880668789 shutdown_connections 2024-05-10T11:22:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.218+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/1880668789 wait complete. 2024-05-10T11:22:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.218+0000 7fb9e2ebbb80 20 client.26623 trim_cache size 0 max 16384 2024-05-10T11:22:34.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.239+0000 7fb9e2ebbb80 1 Processor -- start 2024-05-10T11:22:34.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.239+0000 7fb9e2ebbb80 1 -- start start 2024-05-10T11:22:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9e2ebbb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c4814992d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9e2ebbb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 0x55c4814a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9e2ebbb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c4814aef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9e2ebbb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01af660 con 0x55c48149f4e0 2024-05-10T11:22:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9e2ebbb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01b11d0 con 0x55c4814998f0 2024-05-10T11:22:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9e2ebbb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01afd30 con 0x55c48149ebc0 2024-05-10T11:22:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9d5c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c4814992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9d5c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c4814992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40948/0 (socket says 172.21.15.27:40948) 2024-05-10T11:22:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9d5200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 0x55c4814a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9d5c00700 1 -- 172.21.15.27:0/926894270 learned_addr learned my addr 172.21.15.27:0/926894270 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9d6600700 1 --2- 172.21.15.27:0/926894270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c4814aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9d5200700 1 -- 172.21.15.27:0/926894270 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 msgr2=0x55c4814992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9d5200700 1 --2- 172.21.15.27:0/926894270 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c4814992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9d5200700 1 -- 172.21.15.27:0/926894270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 msgr2=0x55c4814aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9d5200700 1 --2- 172.21.15.27:0/926894270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c4814aef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9d5200700 1 -- 172.21.15.27:0/926894270 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4814953b0 con 0x55c48149ebc0 2024-05-10T11:22:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.240+0000 7fb9d5c00700 1 --2- 172.21.15.27:0/926894270 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c4814992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.241+0000 7fb9d5200700 1 --2- 172.21.15.27:0/926894270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 0x55c4814a17e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0df770 tx=0x7fb9bc0a5a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5b6fedd8b53bbb3 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.241+0000 7fb9c2000700 1 -- 172.21.15.27:0/926894270 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc020040 con 0x55c48149ebc0 2024-05-10T11:22:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.241+0000 7fb9c2000700 1 -- 172.21.15.27:0/926894270 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9bc0a8e10 con 0x55c48149ebc0 2024-05-10T11:22:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.241+0000 7fb9c2000700 1 -- 172.21.15.27:0/926894270 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc0b2160 con 0x55c48149ebc0 2024-05-10T11:22:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.242+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/926894270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 msgr2=0x55c4814a17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.242+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/926894270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 0x55c4814a17e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0df770 tx=0x7fb9bc0a5a80 comp rx=0 tx=0).stop 2024-05-10T11:22:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.242+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/926894270 shutdown_connections 2024-05-10T11:22:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.242+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/926894270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c4814aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.242+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/926894270 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c4814992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.242+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/926894270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 0x55c4814a17e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.242+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/926894270 >> 172.21.15.27:0/926894270 conn(0x55c4814116d0 msgr2=0x55c48149ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.242+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/926894270 shutdown_connections 2024-05-10T11:22:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.242+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/926894270 wait complete. 2024-05-10T11:22:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.243+0000 7fb9e2ebbb80 1 -- start start 2024-05-10T11:22:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.243+0000 7fb9e2ebbb80 1 Processor -- start 2024-05-10T11:22:34.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.243+0000 7fb9e2ebbb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c4814ba7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.243+0000 7fb9e2ebbb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 0x55c481492960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.243+0000 7fb9e2ebbb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c481492ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.243+0000 7fb9e2ebbb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01b0b00 con 0x55c48149f4e0 2024-05-10T11:22:34.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.243+0000 7fb9e2ebbb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01b0430 con 0x55c4814998f0 2024-05-10T11:22:34.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.243+0000 7fb9e2ebbb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01ad620 con 0x55c48149ebc0 2024-05-10T11:22:34.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.243+0000 7fb9e2ebbb80 10 client.0 _ll_register_callbacks cb 0x55c481495cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:34.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9d5200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 0x55c481492960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9d5c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c4814ba7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9d5200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 0x55c481492960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:53108/0 (socket says 172.21.15.27:53108) 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9d5200700 1 -- 172.21.15.27:0/4178156239 learned_addr learned my addr 172.21.15.27:0/4178156239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9d6600700 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c481492ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9d5200700 1 -- 172.21.15.27:0/4178156239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 msgr2=0x55c4814ba7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9d5200700 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c4814ba7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9d5200700 1 -- 172.21.15.27:0/4178156239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 msgr2=0x55c481492ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9d5200700 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c481492ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9d5200700 1 -- 172.21.15.27:0/4178156239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c481493570 con 0x55c48149ebc0 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9d5c00700 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c4814ba7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9d5200700 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 0x55c481492960 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0df7d0 tx=0x7fb9bc0a8af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bbdcacc83a59f7f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.244+0000 7fb9b7400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:22:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.245+0000 7fb9b7400700 1 -- 172.21.15.27:0/4178156239 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc020070 con 0x55c48149ebc0 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.245+0000 7fb9b7400700 1 -- 172.21.15.27:0/4178156239 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7fb9bc0a9560 con 0x55c48149ebc0 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.245+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/4178156239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9bc0a8e10 con 0x55c48149ebc0 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.245+0000 7fb9b7400700 1 -- 172.21.15.27:0/4178156239 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb9bc0dfbf0 con 0x55c48149ebc0 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.245+0000 7fb9e2ebbb80 10 client.26629 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.245+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/4178156239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55c481520f10 con 0x55c48149ebc0 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.245+0000 7fb9b7400700 1 -- 172.21.15.27:0/4178156239 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9bc0a8e10 con 0x55c48149ebc0 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.245+0000 7fb9b7400700 1 -- 172.21.15.27:0/4178156239 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fb9bc0b1cd0 con 0x55c48149ebc0 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.245+0000 7fb9e2ebbb80 10 client.26629 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.245+0000 7fb9e2ebbb80 10 client.26629 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.245+0000 7fb9e2ebbb80 20 client.26629 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.245+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c4815215b0 0x55c4815239d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.246+0000 7fb9e2ebbb80 4 client.26629 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.246+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/4178156239 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c4813f9400 con 0x55c4815215b0 2024-05-10T11:22:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.246+0000 7fb9d5c00700 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c4815215b0 0x55c4815239d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.246+0000 7fb9d5c00700 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c4815215b0 0x55c4815239d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f512788dcb0b5091 server_cookie=cdb0d10c70193697 in_seq=0 out_seq=0 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.246+0000 7fb9b7400700 10 client.26629 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9b7400700 1 -- 172.21.15.27:0/4178156239 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c4813f9400 con 0x55c4815215b0 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9b7400700 10 client.26629 handle_command_reply: tid=0 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9e2ebbb80 1 client.26629 shutdown 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/4178156239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 msgr2=0x55c481492960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 0x55c481492960 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0df7d0 tx=0x7fb9bc0a8af0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/4178156239 shutdown_connections 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c48149f4e0 0x55c481492ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c4814998f0 0x55c4814ba7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c4815215b0 0x55c4815239d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9e2ebbb80 1 --2- 172.21.15.27:0/4178156239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48149ebc0 0x55c481492960 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9b7400700 0 client.26629 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:22:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/4178156239 >> 172.21.15.27:0/4178156239 conn(0x55c4814116d0 msgr2=0x55c48149df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:34.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/4178156239 shutdown_connections 2024-05-10T11:22:34.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9e2ebbb80 1 -- 172.21.15.27:0/4178156239 wait complete. 2024-05-10T11:22:34.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.247+0000 7fb9e2ebbb80 20 client.26629 trim_cache size 0 max 16384 2024-05-10T11:22:34.251 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:22:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.248+0000 7fb9d8600700 1 -- 172.21.15.27:0/2841734812 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb9b80b29f0 msgr2=0x7fb9b80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.248+0000 7fb9d8600700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb9b80b29f0 0x7fb9b80b4eb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009fb0 tx=0x7fb9c8004750 comp rx=0 tx=0).stop 2024-05-10T11:22:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.248+0000 7fb9d8600700 1 -- 172.21.15.27:0/2841734812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d006bb40 msgr2=0x7fb9d019fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.248+0000 7fb9d8600700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d006bb40 0x7fb9d019fd00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00d950 tx=0x7fb9bc00dc60 comp rx=0 tx=0).stop 2024-05-10T11:22:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.248+0000 7fb9d8600700 1 -- 172.21.15.27:0/2841734812 shutdown_connections 2024-05-10T11:22:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.248+0000 7fb9d8600700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb9b80b29f0 0x7fb9b80b4eb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.248+0000 7fb9d8600700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9d0109510 0x7fb9d0199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.248+0000 7fb9d8600700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9d006b180 0x7fb9d019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.248+0000 7fb9d8600700 1 --2- 172.21.15.27:0/2841734812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d006bb40 0x7fb9d019fd00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.248+0000 7fb9d8600700 1 -- 172.21.15.27:0/2841734812 >> 172.21.15.27:0/2841734812 conn(0x7fb9d0079350 msgr2=0x7fb9d01011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.248+0000 7fb9d8600700 1 -- 172.21.15.27:0/2841734812 shutdown_connections 2024-05-10T11:22:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:34.248+0000 7fb9d8600700 1 -- 172.21.15.27:0/2841734812 wait complete. 2024-05-10T11:22:34.713 INFO:tasks.workunit.client.0.smithi027.stdout: 1 552083 6.56 MB/sec execute 319 sec latency 30.099 ms 2024-05-10T11:22:34.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:34 smithi076 ceph-mon[34152]: pgmap v289: 129 pgs: 129 active+clean; 394 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 135 op/s 2024-05-10T11:22:34.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:34 smithi189 ceph-mon[34842]: pgmap v289: 129 pgs: 129 active+clean; 394 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 135 op/s 2024-05-10T11:22:35.067 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:34 smithi027 ceph-mon[30054]: pgmap v289: 129 pgs: 129 active+clean; 394 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 135 op/s 2024-05-10T11:22:35.243 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:22:35.713 INFO:tasks.workunit.client.0.smithi027.stdout: 1 553886 6.56 MB/sec execute 320 sec latency 24.811 ms 2024-05-10T11:22:35.774 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:22:36.713 INFO:tasks.workunit.client.0.smithi027.stdout: 1 554569 6.56 MB/sec execute 321 sec latency 54.648 ms 2024-05-10T11:22:36.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:36 smithi189 ceph-mon[34842]: pgmap v290: 129 pgs: 129 active+clean; 400 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 203 op/s 2024-05-10T11:22:37.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:36 smithi027 ceph-mon[30054]: pgmap v290: 129 pgs: 129 active+clean; 400 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 203 op/s 2024-05-10T11:22:37.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:36 smithi076 ceph-mon[34152]: pgmap v290: 129 pgs: 129 active+clean; 400 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 203 op/s 2024-05-10T11:22:37.715 INFO:tasks.workunit.client.0.smithi027.stdout: 1 555722 6.55 MB/sec execute 322 sec latency 62.363 ms 2024-05-10T11:22:38.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.240+0000 7fbc6d200700 1 -- 172.21.15.27:0/362545498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc68107200 msgr2=0x7fbc681075e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:38.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.240+0000 7fbc6d200700 1 --2- 172.21.15.27:0/362545498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc68107200 0x7fbc681075e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009990 tx=0x7fbc58009ca0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.240+0000 7fbc6d200700 1 -- 172.21.15.27:0/362545498 shutdown_connections 2024-05-10T11:22:38.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.240+0000 7fbc6d200700 1 --2- 172.21.15.27:0/362545498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc68102430 0x7fbc681028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.240+0000 7fbc6d200700 1 --2- 172.21.15.27:0/362545498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc68107200 0x7fbc681075e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.240+0000 7fbc6d200700 1 --2- 172.21.15.27:0/362545498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc68107bb0 0x7fbc68101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.240+0000 7fbc6d200700 1 -- 172.21.15.27:0/362545498 >> 172.21.15.27:0/362545498 conn(0x7fbc680776c0 msgr2=0x7fbc68077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:38.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.241+0000 7fbc6d200700 1 -- 172.21.15.27:0/362545498 shutdown_connections 2024-05-10T11:22:38.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.241+0000 7fbc6d200700 1 -- 172.21.15.27:0/362545498 wait complete. 2024-05-10T11:22:38.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.242+0000 7fbc6d200700 1 Processor -- start 2024-05-10T11:22:38.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.242+0000 7fbc6d200700 1 -- start start 2024-05-10T11:22:38.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.243+0000 7fbc6d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc68102430 0x7fbc681a0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.243+0000 7fbc6d200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc68107200 0x7fbc681a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.243+0000 7fbc6d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc68107bb0 0x7fbc681a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.243+0000 7fbc6d200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbc68119dc0 con 0x7fbc68107bb0 2024-05-10T11:22:38.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.243+0000 7fbc6d200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbc68119c40 con 0x7fbc68102430 2024-05-10T11:22:38.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.243+0000 7fbc6d200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbc68119f40 con 0x7fbc68107200 2024-05-10T11:22:38.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.243+0000 7fbc65e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc68102430 0x7fbc681a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.243+0000 7fbc65400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc68107200 0x7fbc681a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.243+0000 7fbc65e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc68102430 0x7fbc681a0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40966/0 (socket says 172.21.15.27:40966) 2024-05-10T11:22:38.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.243+0000 7fbc65e00700 1 -- 172.21.15.27:0/2203749611 learned_addr learned my addr 172.21.15.27:0/2203749611 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:38.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.244+0000 7fbc66800700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc68107bb0 0x7fbc681a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.244+0000 7fbc65e00700 1 -- 172.21.15.27:0/2203749611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc68107200 msgr2=0x7fbc681a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.244+0000 7fbc65e00700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc68107200 0x7fbc681a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.244+0000 7fbc65e00700 1 -- 172.21.15.27:0/2203749611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc68107bb0 msgr2=0x7fbc681a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.244+0000 7fbc65e00700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc68107bb0 0x7fbc681a4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.244+0000 7fbc65e00700 1 -- 172.21.15.27:0/2203749611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc681a5260 con 0x7fbc68102430 2024-05-10T11:22:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.244+0000 7fbc65e00700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc68102430 0x7fbc681a0250 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbc58000c00 tx=0x7fbc580057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.245+0000 7fbc5ea00700 1 -- 172.21.15.27:0/2203749611 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc58027070 con 0x7fbc68102430 2024-05-10T11:22:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.245+0000 7fbc5ea00700 1 -- 172.21.15.27:0/2203749611 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc58022470 con 0x7fbc68102430 2024-05-10T11:22:38.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.245+0000 7fbc6d200700 1 -- 172.21.15.27:0/2203749611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc681a54f0 con 0x7fbc68102430 2024-05-10T11:22:38.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.245+0000 7fbc6d200700 1 -- 172.21.15.27:0/2203749611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc681acdd0 con 0x7fbc68102430 2024-05-10T11:22:38.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.245+0000 7fbc5ea00700 1 -- 172.21.15.27:0/2203749611 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc58017650 con 0x7fbc68102430 2024-05-10T11:22:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.249+0000 7fbc5ea00700 1 -- 172.21.15.27:0/2203749611 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fbc5802a070 con 0x7fbc68102430 2024-05-10T11:22:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.251+0000 7fbc77adeb80 1 Processor -- start 2024-05-10T11:22:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.251+0000 7fbc77adeb80 1 -- start start 2024-05-10T11:22:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.251+0000 7fbc77adeb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febbc060 0x5559febbc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.251+0000 7fbc77adeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.251+0000 7fbc77adeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febc09d0 0x5559febc0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.251+0000 7fbc77adeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5559feb0ea20 con 0x5559febc09d0 2024-05-10T11:22:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.251+0000 7fbc77adeb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5559feaac760 con 0x5559febbc060 2024-05-10T11:22:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.251+0000 7fbc77adeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5559feb0a9e0 con 0x5559febbc980 2024-05-10T11:22:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.252+0000 7fbc66800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febbc060 0x5559febbc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.252+0000 7fbc66800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febbc060 0x5559febbc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:40972/0 (socket says 172.21.15.27:40972) 2024-05-10T11:22:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.252+0000 7fbc66800700 1 -- 172.21.15.27:0/2650500946 learned_addr learned my addr 172.21.15.27:0/2650500946 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:38.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.252+0000 7fbc65e00700 1 --2- 172.21.15.27:0/2650500946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.252+0000 7fbc5ea00700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbc4c0b2a70 0x7fbc4c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.252+0000 7fbc66800700 1 -- 172.21.15.27:0/2650500946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 msgr2=0x5559febc0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.252+0000 7fbc66800700 1 --2- 172.21.15.27:0/2650500946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.252+0000 7fbc66800700 1 -- 172.21.15.27:0/2650500946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febc09d0 msgr2=0x5559febc0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.252+0000 7fbc66800700 1 --2- 172.21.15.27:0/2650500946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febc09d0 0x5559febc0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.253+0000 7fbc66800700 1 -- 172.21.15.27:0/2650500946 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559fea5b8d0 con 0x5559febbc060 2024-05-10T11:22:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.253+0000 7fbc65e00700 1 --2- 172.21.15.27:0/2650500946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.253+0000 7fbc5ea00700 1 -- 172.21.15.27:0/2203749611 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7fbc580e3810 con 0x7fbc68102430 2024-05-10T11:22:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.253+0000 7fbc66800700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbc4c0b2a70 0x7fbc4c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.253+0000 7fbc66800700 1 --2- 172.21.15.27:0/2650500946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febbc060 0x5559febbc440 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbc60007530 tx=0x7fbc60007840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cafe76606424fa1b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.254+0000 7fbc53400700 1 -- 172.21.15.27:0/2650500946 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc60023640 con 0x5559febbc060 2024-05-10T11:22:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.254+0000 7fbc53400700 1 -- 172.21.15.27:0/2650500946 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc60023c90 con 0x5559febbc060 2024-05-10T11:22:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.254+0000 7fbc53400700 1 -- 172.21.15.27:0/2650500946 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc60017b60 con 0x5559febbc060 2024-05-10T11:22:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.255+0000 7fbc77adeb80 1 -- 172.21.15.27:0/2650500946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febbc060 msgr2=0x5559febbc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.255+0000 7fbc66800700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbc4c0b2a70 0x7fbc4c0b4f30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbc60020000 tx=0x7fbc60021000 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.255+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/2650500946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febbc060 0x5559febbc440 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbc60007530 tx=0x7fbc60007840 comp rx=0 tx=0).stop 2024-05-10T11:22:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.255+0000 7fbc77adeb80 1 -- 172.21.15.27:0/2650500946 shutdown_connections 2024-05-10T11:22:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.255+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/2650500946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febc09d0 0x5559febc0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.255+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/2650500946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febbc060 0x5559febbc440 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.255+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/2650500946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.255+0000 7fbc77adeb80 1 -- 172.21.15.27:0/2650500946 >> 172.21.15.27:0/2650500946 conn(0x5559feaeab60 msgr2=0x5559febb67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:38.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.255+0000 7fbc77adeb80 1 -- 172.21.15.27:0/2650500946 shutdown_connections 2024-05-10T11:22:38.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.255+0000 7fbc77adeb80 1 -- 172.21.15.27:0/2650500946 wait complete. 2024-05-10T11:22:38.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.255+0000 7fbc77adeb80 1 -- start start 2024-05-10T11:22:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.256+0000 7fbc77adeb80 1 Processor -- start 2024-05-10T11:22:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.256+0000 7fbc77adeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febbc060 0x5559febc7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.256+0000 7fbc77adeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.256+0000 7fbc77adeb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febc09d0 0x5559febc56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.256+0000 7fbc77adeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5559febc8be0 con 0x5559febbc060 2024-05-10T11:22:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.256+0000 7fbc77adeb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5559fea74b20 con 0x5559febc09d0 2024-05-10T11:22:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.256+0000 7fbc77adeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5559febc8d60 con 0x5559febbc980 2024-05-10T11:22:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.256+0000 7fbc77adeb80 10 client.0 _ll_register_callbacks cb 0x5559feab5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.257+0000 7fbc65400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.257+0000 7fbc65e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febbc060 0x5559febc7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.257+0000 7fbc65400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:53138/0 (socket says 172.21.15.27:53138) 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.257+0000 7fbc65400700 1 -- 172.21.15.27:0/3354708098 learned_addr learned my addr 172.21.15.27:0/3354708098 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.257+0000 7fbc66800700 1 --2- 172.21.15.27:0/3354708098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febc09d0 0x5559febc56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.257+0000 7fbc65400700 1 -- 172.21.15.27:0/3354708098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febc09d0 msgr2=0x5559febc56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.257+0000 7fbc65400700 1 --2- 172.21.15.27:0/3354708098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febc09d0 0x5559febc56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.257+0000 7fbc65400700 1 -- 172.21.15.27:0/3354708098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febbc060 msgr2=0x5559febc7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.257+0000 7fbc65400700 1 --2- 172.21.15.27:0/3354708098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febbc060 0x5559febc7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.257+0000 7fbc65400700 1 -- 172.21.15.27:0/3354708098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559fead4420 con 0x5559febbc980 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.257+0000 7fbc65400700 1 --2- 172.21.15.27:0/3354708098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc5160 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbc54007ed0 tx=0x7fbc540106b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61be573ba7aa5634 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.257+0000 7fbc47e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.258+0000 7fbc47e00700 1 -- 172.21.15.27:0/3354708098 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54010e20 con 0x5559febbc980 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.258+0000 7fbc47e00700 1 -- 172.21.15.27:0/3354708098 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7fbc54011440 con 0x5559febbc980 2024-05-10T11:22:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.258+0000 7fbc77adeb80 1 -- 172.21.15.27:0/3354708098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc60023c90 con 0x5559febbc980 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.258+0000 7fbc47e00700 1 -- 172.21.15.27:0/3354708098 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbc5401f6e0 con 0x5559febbc980 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.258+0000 7fbc77adeb80 10 client.26641 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.258+0000 7fbc77adeb80 1 -- 172.21.15.27:0/3354708098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5559febd34e0 con 0x5559febbc980 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.258+0000 7fbc47e00700 1 -- 172.21.15.27:0/3354708098 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc60023c90 con 0x5559febbc980 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.259+0000 7fbc47e00700 1 -- 172.21.15.27:0/3354708098 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fbc5401a430 con 0x5559febbc980 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.259+0000 7fbc77adeb80 10 client.26641 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.259+0000 7fbc77adeb80 10 client.26641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.259+0000 7fbc77adeb80 20 client.26641 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.259+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/3354708098 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5559febd5a70 0x5559febd7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.259+0000 7fbc77adeb80 4 client.26641 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.259+0000 7fbc77adeb80 1 -- 172.21.15.27:0/3354708098 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559feb890a0 con 0x5559febd5a70 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.259+0000 7fbc65e00700 1 --2- 172.21.15.27:0/3354708098 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5559febd5a70 0x5559febd7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.260+0000 7fbc65e00700 1 --2- 172.21.15.27:0/3354708098 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5559febd5a70 0x5559febd7e60 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16544cb397a06c3 server_cookie=c0542b662a8d1341 in_seq=0 out_seq=0 2024-05-10T11:22:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.260+0000 7fbc47e00700 10 client.26641 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc47e00700 1 -- 172.21.15.27:0/3354708098 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5559feb890a0 con 0x5559febd5a70 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc47e00700 10 client.26641 handle_command_reply: tid=0 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc77adeb80 1 client.26641 shutdown 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc77adeb80 1 -- 172.21.15.27:0/3354708098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 msgr2=0x5559febc5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/3354708098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc5160 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbc54007ed0 tx=0x7fbc540106b0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc77adeb80 1 -- 172.21.15.27:0/3354708098 shutdown_connections 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/3354708098 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5559febd5a70 0x5559febd7e60 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/3354708098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febc09d0 0x5559febc56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/3354708098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febbc060 0x5559febc7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/3354708098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc5160 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc47e00700 0 client.26641 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc77adeb80 1 -- 172.21.15.27:0/3354708098 >> 172.21.15.27:0/3354708098 conn(0x5559feaeab60 msgr2=0x5559febb12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.261+0000 7fbc77adeb80 1 -- 172.21.15.27:0/3354708098 shutdown_connections 2024-05-10T11:22:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.262+0000 7fbc77adeb80 1 -- 172.21.15.27:0/3354708098 wait complete. 2024-05-10T11:22:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.262+0000 7fbc77adeb80 20 client.26641 trim_cache size 0 max 16384 2024-05-10T11:22:38.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.280+0000 7fbc77adeb80 1 Processor -- start 2024-05-10T11:22:38.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.280+0000 7fbc77adeb80 1 -- start start 2024-05-10T11:22:38.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.280+0000 7fbc77adeb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febbc060 0x5559febd1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc77adeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc77adeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febc09d0 0x5559febdd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc77adeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4800ef50 con 0x5559febc09d0 2024-05-10T11:22:38.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc77adeb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbc48010ac0 con 0x5559febbc060 2024-05-10T11:22:38.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc77adeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4800f620 con 0x5559febbc980 2024-05-10T11:22:38.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc65400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc65400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:53144/0 (socket says 172.21.15.27:53144) 2024-05-10T11:22:38.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc65400700 1 -- 172.21.15.27:0/2560473183 learned_addr learned my addr 172.21.15.27:0/2560473183 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:38.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc65e00700 1 --2- 172.21.15.27:0/2560473183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febbc060 0x5559febd1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc66800700 1 --2- 172.21.15.27:0/2560473183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febc09d0 0x5559febdd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc65400700 1 -- 172.21.15.27:0/2560473183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febbc060 msgr2=0x5559febd1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc65400700 1 --2- 172.21.15.27:0/2560473183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febbc060 0x5559febd1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc65400700 1 -- 172.21.15.27:0/2560473183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febc09d0 msgr2=0x5559febdd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc65400700 1 --2- 172.21.15.27:0/2560473183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febc09d0 0x5559febdd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.281+0000 7fbc65400700 1 -- 172.21.15.27:0/2560473183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559febc07a0 con 0x5559febbc980 2024-05-10T11:22:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.282+0000 7fbc65400700 1 --2- 172.21.15.27:0/2560473183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc1aa0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbc54007fd0 tx=0x7fbc54007a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0b169e35c7bcf4a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.282+0000 7fbc52a00700 1 -- 172.21.15.27:0/2560473183 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5401a630 con 0x5559febbc980 2024-05-10T11:22:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.282+0000 7fbc52a00700 1 -- 172.21.15.27:0/2560473183 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc5402c070 con 0x5559febbc980 2024-05-10T11:22:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.282+0000 7fbc52a00700 1 -- 172.21.15.27:0/2560473183 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5401f4b0 con 0x5559febbc980 2024-05-10T11:22:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.283+0000 7fbc77adeb80 1 -- 172.21.15.27:0/2560473183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 msgr2=0x5559febc1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.283+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/2560473183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc1aa0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbc54007fd0 tx=0x7fbc54007a80 comp rx=0 tx=0).stop 2024-05-10T11:22:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.283+0000 7fbc77adeb80 1 -- 172.21.15.27:0/2560473183 shutdown_connections 2024-05-10T11:22:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.283+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/2560473183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febc09d0 0x5559febdd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.283+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/2560473183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febbc060 0x5559febd1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.283+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/2560473183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc1aa0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.283+0000 7fbc77adeb80 1 -- 172.21.15.27:0/2560473183 >> 172.21.15.27:0/2560473183 conn(0x5559feaeab60 msgr2=0x5559febc1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.283+0000 7fbc77adeb80 1 -- 172.21.15.27:0/2560473183 shutdown_connections 2024-05-10T11:22:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.283+0000 7fbc77adeb80 1 -- 172.21.15.27:0/2560473183 wait complete. 2024-05-10T11:22:38.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.284+0000 7fbc77adeb80 1 -- start start 2024-05-10T11:22:38.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.284+0000 7fbc77adeb80 1 Processor -- start 2024-05-10T11:22:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.285+0000 7fbc77adeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febbc060 0x5559febce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.285+0000 7fbc77adeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.285+0000 7fbc77adeb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febc09d0 0x5559febca040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.285+0000 7fbc77adeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbc480103f0 con 0x5559febbc060 2024-05-10T11:22:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.285+0000 7fbc77adeb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4800fd20 con 0x5559febc09d0 2024-05-10T11:22:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.285+0000 7fbc77adeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4800cf10 con 0x5559febbc980 2024-05-10T11:22:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.285+0000 7fbc77adeb80 10 client.0 _ll_register_callbacks cb 0x5559feab5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.285+0000 7fbc65e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febbc060 0x5559febce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.285+0000 7fbc65400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.285+0000 7fbc65e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febbc060 0x5559febce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54832/0 (socket says 172.21.15.27:54832) 2024-05-10T11:22:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.285+0000 7fbc65e00700 1 -- 172.21.15.27:0/1887263058 learned_addr learned my addr 172.21.15.27:0/1887263058 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.285+0000 7fbc66800700 1 --2- 172.21.15.27:0/1887263058 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febc09d0 0x5559febca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.286+0000 7fbc65400700 1 -- 172.21.15.27:0/1887263058 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febc09d0 msgr2=0x5559febca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.286+0000 7fbc65400700 1 --2- 172.21.15.27:0/1887263058 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febc09d0 0x5559febca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.286+0000 7fbc65400700 1 -- 172.21.15.27:0/1887263058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febbc060 msgr2=0x5559febce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.286+0000 7fbc65400700 1 --2- 172.21.15.27:0/1887263058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febbc060 0x5559febce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.286+0000 7fbc65400700 1 -- 172.21.15.27:0/1887263058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559febca850 con 0x5559febbc980 2024-05-10T11:22:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.286+0000 7fbc65400700 1 --2- 172.21.15.27:0/1887263058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc9a30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbc54010b30 tx=0x7fbc5401a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73932985b6cb5188 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.286+0000 7fbc47e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:22:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.286+0000 7fbc47e00700 1 -- 172.21.15.27:0/1887263058 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5401f250 con 0x5559febbc980 2024-05-10T11:22:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.286+0000 7fbc77adeb80 1 -- 172.21.15.27:0/1887263058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc5402c070 con 0x5559febbc980 2024-05-10T11:22:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.286+0000 7fbc47e00700 1 -- 172.21.15.27:0/1887263058 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7fbc5401f3f0 con 0x5559febbc980 2024-05-10T11:22:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.287+0000 7fbc47e00700 1 -- 172.21.15.27:0/1887263058 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbc5401f880 con 0x5559febbc980 2024-05-10T11:22:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.287+0000 7fbc77adeb80 10 client.26647 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.287+0000 7fbc77adeb80 1 -- 172.21.15.27:0/1887263058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5559fec40410 con 0x5559febbc980 2024-05-10T11:22:38.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.287+0000 7fbc47e00700 1 -- 172.21.15.27:0/1887263058 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc5402c070 con 0x5559febbc980 2024-05-10T11:22:38.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.287+0000 7fbc47e00700 1 -- 172.21.15.27:0/1887263058 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fbc54005640 con 0x5559febbc980 2024-05-10T11:22:38.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.287+0000 7fbc77adeb80 10 client.26647 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.287+0000 7fbc77adeb80 10 client.26647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:22:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.287+0000 7fbc77adeb80 20 client.26647 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.287+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/1887263058 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5559fec40ac0 0x5559fec42ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.287+0000 7fbc77adeb80 4 client.26647 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:22:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.287+0000 7fbc77adeb80 1 -- 172.21.15.27:0/1887263058 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5559feb1c400 con 0x5559fec40ac0 2024-05-10T11:22:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.288+0000 7fbc65e00700 1 --2- 172.21.15.27:0/1887263058 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5559fec40ac0 0x5559fec42ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.288+0000 7fbc65e00700 1 --2- 172.21.15.27:0/1887263058 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5559fec40ac0 0x5559fec42ee0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62c339ec73142fc4 server_cookie=37b9f45a43cbae1f in_seq=0 out_seq=0 2024-05-10T11:22:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.288+0000 7fbc47e00700 10 client.26647 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.288+0000 7fbc47e00700 1 -- 172.21.15.27:0/1887263058 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5559feb1c400 con 0x5559fec40ac0 2024-05-10T11:22:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.288+0000 7fbc47e00700 10 client.26647 handle_command_reply: tid=0 2024-05-10T11:22:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.288+0000 7fbc77adeb80 1 client.26647 shutdown 2024-05-10T11:22:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.289+0000 7fbc77adeb80 1 -- 172.21.15.27:0/1887263058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 msgr2=0x5559febc9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.289+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/1887263058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc9a30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbc54010b30 tx=0x7fbc5401a660 comp rx=0 tx=0).stop 2024-05-10T11:22:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.289+0000 7fbc77adeb80 1 -- 172.21.15.27:0/1887263058 shutdown_connections 2024-05-10T11:22:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.289+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/1887263058 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5559fec40ac0 0x5559fec42ee0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.289+0000 7fbc47e00700 0 client.26647 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.289+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/1887263058 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5559febc09d0 0x5559febca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.289+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/1887263058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5559febbc060 0x5559febce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.289+0000 7fbc77adeb80 1 --2- 172.21.15.27:0/1887263058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559febbc980 0x5559febc9a30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.289+0000 7fbc77adeb80 1 -- 172.21.15.27:0/1887263058 >> 172.21.15.27:0/1887263058 conn(0x5559feaeab60 msgr2=0x5559febb12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.289+0000 7fbc77adeb80 1 -- 172.21.15.27:0/1887263058 shutdown_connections 2024-05-10T11:22:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.289+0000 7fbc77adeb80 1 -- 172.21.15.27:0/1887263058 wait complete. 2024-05-10T11:22:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.289+0000 7fbc77adeb80 20 client.26647 trim_cache size 0 max 16384 2024-05-10T11:22:38.297 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:22:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.290+0000 7fbc6d200700 1 -- 172.21.15.27:0/2203749611 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbc4c0b2a70 msgr2=0x7fbc4c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.290+0000 7fbc6d200700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbc4c0b2a70 0x7fbc4c0b4f30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbc60020000 tx=0x7fbc60021000 comp rx=0 tx=0).stop 2024-05-10T11:22:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.290+0000 7fbc6d200700 1 -- 172.21.15.27:0/2203749611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc68102430 msgr2=0x7fbc681a0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.290+0000 7fbc6d200700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc68102430 0x7fbc681a0250 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbc58000c00 tx=0x7fbc580057c0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.290+0000 7fbc6d200700 1 -- 172.21.15.27:0/2203749611 shutdown_connections 2024-05-10T11:22:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.291+0000 7fbc6d200700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbc4c0b2a70 0x7fbc4c0b4f30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.291+0000 7fbc6d200700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc68107bb0 0x7fbc681a4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.291+0000 7fbc6d200700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc68102430 0x7fbc681a0250 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.291+0000 7fbc6d200700 1 --2- 172.21.15.27:0/2203749611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc68107200 0x7fbc681a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.291+0000 7fbc6d200700 1 -- 172.21.15.27:0/2203749611 >> 172.21.15.27:0/2203749611 conn(0x7fbc680776c0 msgr2=0x7fbc68102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.291+0000 7fbc6d200700 1 -- 172.21.15.27:0/2203749611 shutdown_connections 2024-05-10T11:22:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:38.291+0000 7fbc6d200700 1 -- 172.21.15.27:0/2203749611 wait complete. 2024-05-10T11:22:38.713 INFO:tasks.workunit.client.0.smithi027.stdout: 1 557318 6.56 MB/sec execute 323 sec latency 81.759 ms 2024-05-10T11:22:38.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:38 smithi189 ceph-mon[34842]: pgmap v291: 129 pgs: 129 active+clean; 400 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 137 op/s 2024-05-10T11:22:39.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:38 smithi027 ceph-mon[30054]: pgmap v291: 129 pgs: 129 active+clean; 400 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 137 op/s 2024-05-10T11:22:39.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:38 smithi076 ceph-mon[34152]: pgmap v291: 129 pgs: 129 active+clean; 400 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 137 op/s 2024-05-10T11:22:39.713 INFO:tasks.workunit.client.0.smithi027.stdout: 1 558132 6.56 MB/sec execute 324 sec latency 20.619 ms 2024-05-10T11:22:40.233 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:22:40.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:22:40.713 INFO:tasks.workunit.client.0.smithi027.stdout: 1 559264 6.55 MB/sec execute 325 sec latency 30.754 ms 2024-05-10T11:22:40.725 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:22:40.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:40 smithi189 ceph-mon[34842]: pgmap v292: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-05-10T11:22:41.075 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:40 smithi027 ceph-mon[30054]: pgmap v292: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-05-10T11:22:41.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:40 smithi076 ceph-mon[34152]: pgmap v292: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-05-10T11:22:41.713 INFO:tasks.workunit.client.0.smithi027.stdout: 1 560913 6.56 MB/sec execute 326 sec latency 24.077 ms 2024-05-10T11:22:42.713 INFO:tasks.workunit.client.0.smithi027.stdout: 1 561688 6.55 MB/sec execute 327 sec latency 32.402 ms 2024-05-10T11:22:42.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:42 smithi027 ceph-mon[30054]: pgmap v293: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 135 op/s 2024-05-10T11:22:42.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:42 smithi189 ceph-mon[34842]: pgmap v293: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 135 op/s 2024-05-10T11:22:43.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:42 smithi076 ceph-mon[34152]: pgmap v293: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 135 op/s 2024-05-10T11:22:43.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.580+0000 7f4afce00700 1 -- 172.21.15.27:0/3786260099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af00a4dd0 msgr2=0x7f4af00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:43.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.580+0000 7f4afce00700 1 --2- 172.21.15.27:0/3786260099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af00a4dd0 0x7f4af00a5250 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4009990 tx=0x7f4ae4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.584+0000 7f4afce00700 1 -- 172.21.15.27:0/3786260099 shutdown_connections 2024-05-10T11:22:43.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.584+0000 7f4afce00700 1 --2- 172.21.15.27:0/3786260099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af00a5790 0x7f4af00b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.584+0000 7f4afce00700 1 --2- 172.21.15.27:0/3786260099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4af00abd50 0x7f4af00ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.584+0000 7f4afce00700 1 --2- 172.21.15.27:0/3786260099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af00a4dd0 0x7f4af00a5250 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.584+0000 7f4afce00700 1 -- 172.21.15.27:0/3786260099 >> 172.21.15.27:0/3786260099 conn(0x7f4af001a690 msgr2=0x7f4af001aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.584+0000 7f4afce00700 1 -- 172.21.15.27:0/3786260099 shutdown_connections 2024-05-10T11:22:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.584+0000 7f4afce00700 1 -- 172.21.15.27:0/3786260099 wait complete. 2024-05-10T11:22:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.586+0000 7f4afce00700 1 Processor -- start 2024-05-10T11:22:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.587+0000 7f4afce00700 1 -- start start 2024-05-10T11:22:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.587+0000 7f4afce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af00a4dd0 0x7f4af013d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.587+0000 7f4afce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af00a5790 0x7f4af013d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.587+0000 7f4afce00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4af00abd50 0x7f4af0141c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.587+0000 7f4afce00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4af00b6f90 con 0x7f4af00a4dd0 2024-05-10T11:22:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.587+0000 7f4afce00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4af00b6e10 con 0x7f4af00abd50 2024-05-10T11:22:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.587+0000 7f4afce00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4af00b7110 con 0x7f4af00a5790 2024-05-10T11:22:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.587+0000 7f4af7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af00a4dd0 0x7f4af013d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.588+0000 7f4af7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af00a4dd0 0x7f4af013d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54846/0 (socket says 172.21.15.27:54846) 2024-05-10T11:22:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.588+0000 7f4af7400700 1 -- 172.21.15.27:0/1519204796 learned_addr learned my addr 172.21.15.27:0/1519204796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.588+0000 7f4af7e00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4af00abd50 0x7f4af0141c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.588+0000 7f4af6a00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af00a5790 0x7f4af013d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.588+0000 7f4af7e00700 1 -- 172.21.15.27:0/1519204796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af00a5790 msgr2=0x7f4af013d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.588+0000 7f4af7e00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af00a5790 0x7f4af013d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.588+0000 7f4af7e00700 1 -- 172.21.15.27:0/1519204796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af00a4dd0 msgr2=0x7f4af013d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.588+0000 7f4af7e00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af00a4dd0 0x7f4af013d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.588+0000 7f4af7e00700 1 -- 172.21.15.27:0/1519204796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af01423d0 con 0x7f4af00abd50 2024-05-10T11:22:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.589+0000 7f4af7e00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4af00abd50 0x7f4af0141c90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4af8069f30 tx=0x7f4af806cac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.589+0000 7f4aebe00700 1 -- 172.21.15.27:0/1519204796 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af806a6a0 con 0x7f4af00abd50 2024-05-10T11:22:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.589+0000 7f4aebe00700 1 -- 172.21.15.27:0/1519204796 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4af807b070 con 0x7f4af00abd50 2024-05-10T11:22:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.589+0000 7f4aebe00700 1 -- 172.21.15.27:0/1519204796 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8075b30 con 0x7f4af00abd50 2024-05-10T11:22:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.589+0000 7f4afce00700 1 -- 172.21.15.27:0/1519204796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af01426c0 con 0x7f4af00abd50 2024-05-10T11:22:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.589+0000 7f4afce00700 1 -- 172.21.15.27:0/1519204796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af0149f40 con 0x7f4af00abd50 2024-05-10T11:22:43.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.594+0000 7f4aebe00700 1 -- 172.21.15.27:0/1519204796 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f4af8073050 con 0x7f4af00abd50 2024-05-10T11:22:43.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.594+0000 7f4b08ce8b80 1 Processor -- start 2024-05-10T11:22:43.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.595+0000 7f4b08ce8b80 1 -- start start 2024-05-10T11:22:43.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.595+0000 7f4b08ce8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee1a60 0x5653caee1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.595+0000 7f4b08ce8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee2380 0x5653caedc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.595+0000 7f4b08ce8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caedcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.595+0000 7f4b08ce8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5653cae2eaa0 con 0x5653caee1a60 2024-05-10T11:22:43.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.595+0000 7f4b08ce8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5653cadcc7e0 con 0x5653caedc680 2024-05-10T11:22:43.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.595+0000 7f4b08ce8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5653cae2aa60 con 0x5653caee2380 2024-05-10T11:22:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.595+0000 7f4af7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee2380 0x5653caedc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.595+0000 7f4af7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee2380 0x5653caedc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:53180/0 (socket says 172.21.15.27:53180) 2024-05-10T11:22:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.595+0000 7f4af7400700 1 -- 172.21.15.27:0/1922123873 learned_addr learned my addr 172.21.15.27:0/1922123873 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.595+0000 7f4af6a00700 1 --2- 172.21.15.27:0/1922123873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee1a60 0x5653caee1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.596+0000 7f4aebe00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4ae00b2a70 0x7f4ae00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.596+0000 7f4af6a00700 1 --2- 172.21.15.27:0/1922123873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caedcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.596+0000 7f4aebe00700 1 -- 172.21.15.27:0/1519204796 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f4af8147680 con 0x7f4af00abd50 2024-05-10T11:22:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.596+0000 7f4af7e00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4ae00b2a70 0x7f4ae00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.596+0000 7f4af7400700 1 -- 172.21.15.27:0/1922123873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 msgr2=0x5653caedcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.596+0000 7f4af7400700 1 --2- 172.21.15.27:0/1922123873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caedcb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.596+0000 7f4af7400700 1 -- 172.21.15.27:0/1922123873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee1a60 msgr2=0x5653caee1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:43.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.596+0000 7f4af7400700 1 --2- 172.21.15.27:0/1922123873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee1a60 0x5653caee1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.596+0000 7f4af7400700 1 -- 172.21.15.27:0/1922123873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653cad7b950 con 0x5653caee2380 2024-05-10T11:22:43.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.596+0000 7f4af6a00700 1 --2- 172.21.15.27:0/1922123873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee1a60 0x5653caee1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:43.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.597+0000 7f4af7400700 1 --2- 172.21.15.27:0/1922123873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee2380 0x5653caedc060 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4aec007da0 tx=0x7f4aec0106b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c3aa360ea8fa4b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.597+0000 7f4af7e00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4ae00b2a70 0x7f4ae00b4f30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4af8051460 tx=0x7f4af8068f00 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.597+0000 7f4ae8c00700 1 -- 172.21.15.27:0/1922123873 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec010e10 con 0x5653caee2380 2024-05-10T11:22:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.597+0000 7f4ae8c00700 1 -- 172.21.15.27:0/1922123873 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aec0118f0 con 0x5653caee2380 2024-05-10T11:22:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.597+0000 7f4ae8c00700 1 -- 172.21.15.27:0/1922123873 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec019870 con 0x5653caee2380 2024-05-10T11:22:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.598+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/1922123873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee2380 msgr2=0x5653caedc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.598+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/1922123873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee2380 0x5653caedc060 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4aec007da0 tx=0x7f4aec0106b0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.598+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/1922123873 shutdown_connections 2024-05-10T11:22:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.598+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/1922123873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caedcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.598+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/1922123873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee1a60 0x5653caee1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.598+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/1922123873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee2380 0x5653caedc060 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.598+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/1922123873 >> 172.21.15.27:0/1922123873 conn(0x5653cae0abe0 msgr2=0x5653caed6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.598+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/1922123873 shutdown_connections 2024-05-10T11:22:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.598+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/1922123873 wait complete. 2024-05-10T11:22:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.599+0000 7f4b08ce8b80 1 -- start start 2024-05-10T11:22:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.599+0000 7f4b08ce8b80 1 Processor -- start 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.600+0000 7f4b08ce8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caedc680 0x5653caee7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.600+0000 7f4b08ce8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caee52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.600+0000 7f4b08ce8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caee2380 0x5653caee5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.600+0000 7f4b08ce8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5653caee8e30 con 0x5653caedc680 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.600+0000 7f4b08ce8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5653cad94ba0 con 0x5653caee2380 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.600+0000 7f4b08ce8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5653caee8fb0 con 0x5653caee1a60 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.600+0000 7f4b08ce8b80 10 client.0 _ll_register_callbacks cb 0x5653cadd5cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.600+0000 7f4af6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caedc680 0x5653caee7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.600+0000 7f4af6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caedc680 0x5653caee7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54878/0 (socket says 172.21.15.27:54878) 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.600+0000 7f4af6a00700 1 -- 172.21.15.27:0/3819204370 learned_addr learned my addr 172.21.15.27:0/3819204370 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.600+0000 7f4af7400700 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caee52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.600+0000 7f4af6a00700 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caee2380 0x5653caee5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.601+0000 7f4af6a00700 1 -- 172.21.15.27:0/3819204370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 msgr2=0x5653caee52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.601+0000 7f4af6a00700 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caee52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.601+0000 7f4af6a00700 1 -- 172.21.15.27:0/3819204370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caee2380 msgr2=0x5653caee5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.601+0000 7f4af6a00700 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caee2380 0x5653caee5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.601+0000 7f4af6a00700 1 -- 172.21.15.27:0/3819204370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653cadf44a0 con 0x5653caedc680 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.601+0000 7f4af7400700 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caee52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.601+0000 7f4af6a00700 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caedc680 0x5653caee7f30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4009990 tx=0x7f4ae4009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccce9a546cb53c7f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.601+0000 7f4add600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.601+0000 7f4add600700 1 -- 172.21.15.27:0/3819204370 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae40054c0 con 0x5653caedc680 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.601+0000 7f4add600700 1 -- 172.21.15.27:0/3819204370 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f4ae4005660 con 0x5653caedc680 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.601+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/3819204370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4aec0118f0 con 0x5653caedc680 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.602+0000 7f4add600700 1 -- 172.21.15.27:0/3819204370 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4ae4035050 con 0x5653caedc680 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.602+0000 7f4b08ce8b80 10 client.16452 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.602+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/3819204370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5653caef7a00 con 0x5653caedc680 2024-05-10T11:22:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.602+0000 7f4add600700 1 -- 172.21.15.27:0/3819204370 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aec0118f0 con 0x5653caedc680 2024-05-10T11:22:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.602+0000 7f4add600700 1 -- 172.21.15.27:0/3819204370 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f4ae4029710 con 0x5653caedc680 2024-05-10T11:22:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.602+0000 7f4b08ce8b80 10 client.16452 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.602+0000 7f4b08ce8b80 10 client.16452 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:22:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.602+0000 7f4b08ce8b80 20 client.16452 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.602+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5653caefa040 0x5653caefc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.602+0000 7f4b08ce8b80 4 client.16452 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:22:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.602+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/3819204370 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653caea9120 con 0x5653caefa040 2024-05-10T11:22:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.603+0000 7f4af7400700 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5653caefa040 0x5653caefc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.603+0000 7f4af7400700 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5653caefa040 0x5653caefc490 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb43cc95d34cb2a3 server_cookie=945189eb9927c441 in_seq=0 out_seq=0 2024-05-10T11:22:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.603+0000 7f4add600700 10 client.16452 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.604+0000 7f4add600700 1 -- 172.21.15.27:0/3819204370 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5653caea9120 con 0x5653caefa040 2024-05-10T11:22:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.604+0000 7f4add600700 10 client.16452 handle_command_reply: tid=0 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.604+0000 7f4b08ce8b80 1 client.16452 shutdown 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.604+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/3819204370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caedc680 msgr2=0x5653caee7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.604+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caedc680 0x5653caee7f30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4009990 tx=0x7f4ae4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.605+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/3819204370 shutdown_connections 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.605+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5653caefa040 0x5653caefc490 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.605+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caee2380 0x5653caee5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.605+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caedc680 0x5653caee7f30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.605+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/3819204370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caee52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.605+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/3819204370 >> 172.21.15.27:0/3819204370 conn(0x5653cae0abe0 msgr2=0x5653caed14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.605+0000 7f4add600700 0 client.16452 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.605+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/3819204370 shutdown_connections 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.605+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/3819204370 wait complete. 2024-05-10T11:22:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.605+0000 7f4b08ce8b80 20 client.16452 trim_cache size 0 max 16384 2024-05-10T11:22:43.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.628+0000 7f4b08ce8b80 1 Processor -- start 2024-05-10T11:22:43.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4b08ce8b80 1 -- start start 2024-05-10T11:22:43.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4b08ce8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caedc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4b08ce8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caee4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4b08ce8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee2380 0x5653caee7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4b08ce8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4af0150d90 con 0x5653caee2380 2024-05-10T11:22:43.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4b08ce8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4af0152900 con 0x5653caedc680 2024-05-10T11:22:43.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4b08ce8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4af0151460 con 0x5653caee1a60 2024-05-10T11:22:43.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4af6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caedc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4af7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caee4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4af6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caedc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41040/0 (socket says 172.21.15.27:41040) 2024-05-10T11:22:43.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4af6a00700 1 -- 172.21.15.27:0/404691278 learned_addr learned my addr 172.21.15.27:0/404691278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:43.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4af7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caee4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:53198/0 (socket says 172.21.15.27:53198) 2024-05-10T11:22:43.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.629+0000 7f4af6a00700 1 --2- 172.21.15.27:0/404691278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee2380 0x5653caee7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.630+0000 7f4af7400700 1 -- 172.21.15.27:0/404691278 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 msgr2=0x5653caedc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:43.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.630+0000 7f4af7400700 1 --2- 172.21.15.27:0/404691278 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caedc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.630+0000 7f4af7400700 1 -- 172.21.15.27:0/404691278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee2380 msgr2=0x5653caee7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.630+0000 7f4af7400700 1 --2- 172.21.15.27:0/404691278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee2380 0x5653caee7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.630+0000 7f4af7400700 1 -- 172.21.15.27:0/404691278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653caefcb10 con 0x5653caee1a60 2024-05-10T11:22:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.630+0000 7f4af7400700 1 --2- 172.21.15.27:0/404691278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caee4250 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4aec007d70 tx=0x7f4aec007ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15cfd852a9c75905 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.630+0000 7f4adfe00700 1 -- 172.21.15.27:0/404691278 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec02a020 con 0x5653caee1a60 2024-05-10T11:22:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.631+0000 7f4adfe00700 1 -- 172.21.15.27:0/404691278 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aec010d30 con 0x5653caee1a60 2024-05-10T11:22:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.631+0000 7f4adfe00700 1 -- 172.21.15.27:0/404691278 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec003be0 con 0x5653caee1a60 2024-05-10T11:22:43.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.631+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/404691278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 msgr2=0x5653caee4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.631+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/404691278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caee4250 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4aec007d70 tx=0x7f4aec007ca0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.631+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/404691278 shutdown_connections 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.631+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/404691278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee2380 0x5653caee7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.631+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/404691278 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caedc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.631+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/404691278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caee4250 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.631+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/404691278 >> 172.21.15.27:0/404691278 conn(0x5653cae0abe0 msgr2=0x5653caed0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.632+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/404691278 shutdown_connections 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.632+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/404691278 wait complete. 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.632+0000 7f4b08ce8b80 1 -- start start 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.632+0000 7f4b08ce8b80 1 Processor -- start 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.633+0000 7f4b08ce8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caef2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.633+0000 7f4b08ce8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caef2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.633+0000 7f4b08ce8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee2380 0x5653caeef730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.633+0000 7f4b08ce8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4af0152230 con 0x5653caee2380 2024-05-10T11:22:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.633+0000 7f4b08ce8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4af0151b60 con 0x5653caedc680 2024-05-10T11:22:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.633+0000 7f4b08ce8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4af014eda0 con 0x5653caee1a60 2024-05-10T11:22:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.633+0000 7f4b08ce8b80 10 client.0 _ll_register_callbacks cb 0x5653cadd5cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.633+0000 7f4af7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caef2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.633+0000 7f4af7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caef2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:53212/0 (socket says 172.21.15.27:53212) 2024-05-10T11:22:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.633+0000 7f4af6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caef2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.633+0000 7f4af7400700 1 -- 172.21.15.27:0/2129228151 learned_addr learned my addr 172.21.15.27:0/2129228151 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.633+0000 7f4af6a00700 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee2380 0x5653caeef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.634+0000 7f4af7400700 1 -- 172.21.15.27:0/2129228151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 msgr2=0x5653caef2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.634+0000 7f4af7400700 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caef2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.634+0000 7f4af7400700 1 -- 172.21.15.27:0/2129228151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee2380 msgr2=0x5653caeef730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.634+0000 7f4af7400700 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee2380 0x5653caeef730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.634+0000 7f4af7400700 1 -- 172.21.15.27:0/2129228151 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653caeefe30 con 0x5653caee1a60 2024-05-10T11:22:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.634+0000 7f4af6a00700 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee2380 0x5653caeef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:22:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.634+0000 7f4af7400700 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caef2ca0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4aec011010 tx=0x7f4aec007480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=153be943cbb15cb server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.634+0000 7f4add600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:22:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.634+0000 7f4add600700 1 -- 172.21.15.27:0/2129228151 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec02a030 con 0x5653caee1a60 2024-05-10T11:22:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.634+0000 7f4add600700 1 -- 172.21.15.27:0/2129228151 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7f4aec005490 con 0x5653caee1a60 2024-05-10T11:22:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.634+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/2129228151 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4aec010d30 con 0x5653caee1a60 2024-05-10T11:22:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.635+0000 7f4add600700 1 -- 172.21.15.27:0/2129228151 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4aec03e050 con 0x5653caee1a60 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.635+0000 7f4b08ce8b80 10 client.26665 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.635+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/2129228151 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5653caf653f0 con 0x5653caee1a60 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.635+0000 7f4add600700 1 -- 172.21.15.27:0/2129228151 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aec010d30 con 0x5653caee1a60 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.635+0000 7f4add600700 1 -- 172.21.15.27:0/2129228151 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f4aec011600 con 0x5653caee1a60 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.635+0000 7f4b08ce8b80 10 client.26665 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.635+0000 7f4b08ce8b80 10 client.26665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.635+0000 7f4b08ce8b80 20 client.26665 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.635+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5653caf65950 0x5653caf67d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.635+0000 7f4b08ce8b80 4 client.26665 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.635+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/2129228151 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5653cae3c480 con 0x5653caf65950 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.636+0000 7f4af6a00700 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5653caf65950 0x5653caf67d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.636+0000 7f4af6a00700 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5653caf65950 0x5653caf67d70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5745767845b54873 server_cookie=ebb57be000dd32aa in_seq=0 out_seq=0 2024-05-10T11:22:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.636+0000 7f4add600700 10 client.26665 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:43.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.637+0000 7f4add600700 1 -- 172.21.15.27:0/2129228151 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5653cae3c480 con 0x5653caf65950 2024-05-10T11:22:43.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.637+0000 7f4add600700 10 client.26665 handle_command_reply: tid=0 2024-05-10T11:22:43.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.637+0000 7f4b08ce8b80 1 client.26665 shutdown 2024-05-10T11:22:43.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.637+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/2129228151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 msgr2=0x5653caef2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.637+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caef2ca0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4aec011010 tx=0x7f4aec007480 comp rx=0 tx=0).stop 2024-05-10T11:22:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.637+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/2129228151 shutdown_connections 2024-05-10T11:22:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.637+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5653caf65950 0x5653caf67d70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.637+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653caee2380 0x5653caeef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.637+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653caedc680 0x5653caef2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.637+0000 7f4b08ce8b80 1 --2- 172.21.15.27:0/2129228151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653caee1a60 0x5653caef2ca0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.637+0000 7f4add600700 0 client.26665 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.637+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/2129228151 >> 172.21.15.27:0/2129228151 conn(0x5653cae0abe0 msgr2=0x5653caee2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.638+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/2129228151 shutdown_connections 2024-05-10T11:22:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.638+0000 7f4b08ce8b80 1 -- 172.21.15.27:0/2129228151 wait complete. 2024-05-10T11:22:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.638+0000 7f4b08ce8b80 20 client.26665 trim_cache size 0 max 16384 2024-05-10T11:22:43.641 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:22:43.641 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:22:43.641 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "2d5fa5d1-4c1b-4015-9fd6-9c95fdd8431a", 2024-05-10T11:22:43.641 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:22:43.641 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:22:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.639+0000 7f4afce00700 1 -- 172.21.15.27:0/1519204796 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4ae00b2a70 msgr2=0x7f4ae00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.639+0000 7f4afce00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4ae00b2a70 0x7f4ae00b4f30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4af8051460 tx=0x7f4af8068f00 comp rx=0 tx=0).stop 2024-05-10T11:22:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.639+0000 7f4afce00700 1 -- 172.21.15.27:0/1519204796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4af00abd50 msgr2=0x7f4af0141c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.639+0000 7f4afce00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4af00abd50 0x7f4af0141c90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4af8069f30 tx=0x7f4af806cac0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.639+0000 7f4afce00700 1 -- 172.21.15.27:0/1519204796 shutdown_connections 2024-05-10T11:22:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.639+0000 7f4afce00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4ae00b2a70 0x7f4ae00b4f30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.639+0000 7f4afce00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4af00abd50 0x7f4af0141c90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.639+0000 7f4afce00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af00a4dd0 0x7f4af013d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.639+0000 7f4afce00700 1 --2- 172.21.15.27:0/1519204796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af00a5790 0x7f4af013d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.639+0000 7f4afce00700 1 -- 172.21.15.27:0/1519204796 >> 172.21.15.27:0/1519204796 conn(0x7f4af001a690 msgr2=0x7f4af00a2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.639+0000 7f4afce00700 1 -- 172.21.15.27:0/1519204796 shutdown_connections 2024-05-10T11:22:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:43.639+0000 7f4afce00700 1 -- 172.21.15.27:0/1519204796 wait complete. 2024-05-10T11:22:43.713 INFO:tasks.workunit.client.0.smithi027.stdout: 1 562760 6.55 MB/sec execute 328 sec latency 49.535 ms 2024-05-10T11:22:43.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:43 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:22:43.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:43 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:43.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:43 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:22:43.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:43 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:22:43.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:43 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:43.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:43 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:22:44.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:43 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:22:44.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:43 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:44.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:43 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:22:44.523 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:22:44.713 INFO:tasks.workunit.client.0.smithi027.stdout: 1 564263 6.55 MB/sec execute 329 sec latency 25.849 ms 2024-05-10T11:22:44.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:44 smithi027 ceph-mon[30054]: pgmap v294: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 135 op/s 2024-05-10T11:22:44.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:44 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:22:44.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:44 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:44.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:44 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:22:44.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:44 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:22:44.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:44 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:44.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:44 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:44.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:44 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:44.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:44 smithi189 ceph-mon[34842]: pgmap v294: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 135 op/s 2024-05-10T11:22:44.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:44 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:22:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:44 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:44 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:22:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:44 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:22:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:44 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:44 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:44.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:44 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:22:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:44 smithi076 ceph-mon[34152]: pgmap v294: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 135 op/s 2024-05-10T11:22:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:44 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:22:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:44 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:44 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:22:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:44 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:22:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:44 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:44 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:44 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:22:45.714 INFO:tasks.workunit.client.0.smithi027.stdout: 1 565262 6.55 MB/sec execute 330 sec latency 17.408 ms 2024-05-10T11:22:46.714 INFO:tasks.workunit.client.0.smithi027.stdout: 1 566300 6.55 MB/sec execute 331 sec latency 28.802 ms 2024-05-10T11:22:46.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:46 smithi189 ceph-mon[34842]: pgmap v295: 129 pgs: 129 active+clean; 411 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 202 op/s 2024-05-10T11:22:47.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:46 smithi027 ceph-mon[30054]: pgmap v295: 129 pgs: 129 active+clean; 411 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 202 op/s 2024-05-10T11:22:47.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:46 smithi076 ceph-mon[34152]: pgmap v295: 129 pgs: 129 active+clean; 411 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 202 op/s 2024-05-10T11:22:47.714 INFO:tasks.workunit.client.0.smithi027.stdout: 1 567716 6.55 MB/sec execute 332 sec latency 91.985 ms 2024-05-10T11:22:47.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.903+0000 7fa532600700 1 -- 172.21.15.27:0/1101765366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52c06b180 msgr2=0x7fa52c06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:47.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.903+0000 7fa532600700 1 --2- 172.21.15.27:0/1101765366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52c06b180 0x7fa52c06b600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa518009990 tx=0x7fa518009ca0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.903+0000 7fa532600700 1 -- 172.21.15.27:0/1101765366 shutdown_connections 2024-05-10T11:22:47.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.903+0000 7fa532600700 1 --2- 172.21.15.27:0/1101765366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa52c06bb40 0x7fa52c110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.903+0000 7fa532600700 1 --2- 172.21.15.27:0/1101765366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c109510 0x7fa52c1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.903+0000 7fa532600700 1 --2- 172.21.15.27:0/1101765366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52c06b180 0x7fa52c06b600 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.903+0000 7fa532600700 1 -- 172.21.15.27:0/1101765366 >> 172.21.15.27:0/1101765366 conn(0x7fa52c079350 msgr2=0x7fa52c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:47.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.904+0000 7fa532600700 1 -- 172.21.15.27:0/1101765366 shutdown_connections 2024-05-10T11:22:47.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.904+0000 7fa532600700 1 -- 172.21.15.27:0/1101765366 wait complete. 2024-05-10T11:22:47.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.905+0000 7fa532600700 1 Processor -- start 2024-05-10T11:22:47.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.905+0000 7fa532600700 1 -- start start 2024-05-10T11:22:47.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.906+0000 7fa532600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa52c06b180 0x7fa52c19bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.906+0000 7fa532600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c06bb40 0x7fa52c19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.906+0000 7fa532600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52c109510 0x7fa52c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.906+0000 7fa532600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c115870 con 0x7fa52c06bb40 2024-05-10T11:22:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.906+0000 7fa532600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1156f0 con 0x7fa52c06b180 2024-05-10T11:22:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.906+0000 7fa532600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1159f0 con 0x7fa52c109510 2024-05-10T11:22:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.906+0000 7fa52be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52c109510 0x7fa52c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.907+0000 7fa52be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52c109510 0x7fa52c1a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36820/0 (socket says 172.21.15.27:36820) 2024-05-10T11:22:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.907+0000 7fa52be00700 1 -- 172.21.15.27:0/3969726742 learned_addr learned my addr 172.21.15.27:0/3969726742 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:47.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.907+0000 7fa52b400700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa52c06b180 0x7fa52c19bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.907+0000 7fa52aa00700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c06bb40 0x7fa52c19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.908+0000 7fa52be00700 1 -- 172.21.15.27:0/3969726742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa52c06b180 msgr2=0x7fa52c19bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:47.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.908+0000 7fa52be00700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa52c06b180 0x7fa52c19bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.908+0000 7fa52be00700 1 -- 172.21.15.27:0/3969726742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c06bb40 msgr2=0x7fa52c19c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:47.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.908+0000 7fa52be00700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c06bb40 0x7fa52c19c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.908+0000 7fa52be00700 1 -- 172.21.15.27:0/3969726742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52c1a0d90 con 0x7fa52c109510 2024-05-10T11:22:47.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.908+0000 7fa52aa00700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c06bb40 0x7fa52c19c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:22:47.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.908+0000 7fa52be00700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52c109510 0x7fa52c1a0650 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa52400ef30 tx=0x7fa52400c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:47.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.909+0000 7fa523e00700 1 -- 172.21.15.27:0/3969726742 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5240092e0 con 0x7fa52c109510 2024-05-10T11:22:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.909+0000 7fa523e00700 1 -- 172.21.15.27:0/3969726742 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa524004bb0 con 0x7fa52c109510 2024-05-10T11:22:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.909+0000 7fa523e00700 1 -- 172.21.15.27:0/3969726742 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa524004280 con 0x7fa52c109510 2024-05-10T11:22:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.909+0000 7fa532600700 1 -- 172.21.15.27:0/3969726742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa52c1a1020 con 0x7fa52c109510 2024-05-10T11:22:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.909+0000 7fa532600700 1 -- 172.21.15.27:0/3969726742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa52c1a8900 con 0x7fa52c109510 2024-05-10T11:22:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.915+0000 7fa523e00700 1 -- 172.21.15.27:0/3969726742 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa52401b030 con 0x7fa52c109510 2024-05-10T11:22:47.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.918+0000 7fa523e00700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa5100b2a70 0x7fa5100b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.918+0000 7fa523e00700 1 -- 172.21.15.27:0/3969726742 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7fa5240e2690 con 0x7fa52c109510 2024-05-10T11:22:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.918+0000 7fa52b400700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa5100b2a70 0x7fa5100b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.918+0000 7fa53d00cb80 1 Processor -- start 2024-05-10T11:22:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.919+0000 7fa53d00cb80 1 -- start start 2024-05-10T11:22:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.919+0000 7fa53d00cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.919+0000 7fa53d00cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a2690 0x5611a03a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.919+0000 7fa53d00cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a3050 0x5611a03aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.919+0000 7fa53d00cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5611a02eca20 con 0x5611a03a6fc0 2024-05-10T11:22:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.919+0000 7fa53d00cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5611a028a760 con 0x5611a03a3050 2024-05-10T11:22:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.919+0000 7fa53d00cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5611a02e89e0 con 0x5611a03a2690 2024-05-10T11:22:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.920+0000 7fa52aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a2690 0x5611a03a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.920+0000 7fa52b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.920+0000 7fa52aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a2690 0x5611a03a2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36824/0 (socket says 172.21.15.27:36824) 2024-05-10T11:22:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.920+0000 7fa52aa00700 1 -- 172.21.15.27:0/3127976222 learned_addr learned my addr 172.21.15.27:0/3127976222 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.920+0000 7fa52be00700 1 --2- 172.21.15.27:0/3127976222 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a3050 0x5611a03aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.920+0000 7fa52be00700 1 -- 172.21.15.27:0/3127976222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a2690 msgr2=0x5611a03a2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.920+0000 7fa52be00700 1 --2- 172.21.15.27:0/3127976222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a2690 0x5611a03a2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.920+0000 7fa52be00700 1 -- 172.21.15.27:0/3127976222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 msgr2=0x5611a03a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.920+0000 7fa52be00700 1 --2- 172.21.15.27:0/3127976222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.920+0000 7fa52be00700 1 -- 172.21.15.27:0/3127976222 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611a02398d0 con 0x5611a03a3050 2024-05-10T11:22:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.920+0000 7fa52b400700 1 --2- 172.21.15.27:0/3127976222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:22:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.920+0000 7fa52b400700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa5100b2a70 0x7fa5100b4f30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa51c005fd0 tx=0x7fa51c005f60 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.921+0000 7fa52be00700 1 --2- 172.21.15.27:0/3127976222 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a3050 0x5611a03aeee0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa5240df9b0 tx=0x7fa5240a90e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5579b1d9fb6ac69 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.922+0000 7fa520c00700 1 -- 172.21.15.27:0/3127976222 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5240e2070 con 0x5611a03a3050 2024-05-10T11:22:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.922+0000 7fa520c00700 1 -- 172.21.15.27:0/3127976222 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5240ad980 con 0x5611a03a3050 2024-05-10T11:22:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.922+0000 7fa520c00700 1 -- 172.21.15.27:0/3127976222 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5240b2170 con 0x5611a03a3050 2024-05-10T11:22:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.923+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3127976222 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a3050 msgr2=0x5611a03aeee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.923+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/3127976222 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a3050 0x5611a03aeee0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa5240df9b0 tx=0x7fa5240a90e0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.923+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3127976222 shutdown_connections 2024-05-10T11:22:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.923+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/3127976222 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a3050 0x5611a03aeee0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.923+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/3127976222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.923+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/3127976222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a2690 0x5611a03a2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.923+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3127976222 >> 172.21.15.27:0/3127976222 conn(0x5611a02c8b60 msgr2=0x5611a03947a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.923+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3127976222 shutdown_connections 2024-05-10T11:22:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.923+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3127976222 wait complete. 2024-05-10T11:22:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.924+0000 7fa53d00cb80 1 -- start start 2024-05-10T11:22:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.924+0000 7fa53d00cb80 1 Processor -- start 2024-05-10T11:22:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.925+0000 7fa53d00cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a2690 0x5611a03aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.925+0000 7fa53d00cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a3050 0x5611a03abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.925+0000 7fa53d00cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a6fc0 0x5611a03ac490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.925+0000 7fa53d00cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5611a03af480 con 0x5611a03a3050 2024-05-10T11:22:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.925+0000 7fa53d00cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5611a0252b20 con 0x5611a03a6fc0 2024-05-10T11:22:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.925+0000 7fa53d00cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5611a03af600 con 0x5611a03a2690 2024-05-10T11:22:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.925+0000 7fa53d00cb80 10 client.0 _ll_register_callbacks cb 0x5611a0293c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.926+0000 7fa52b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a2690 0x5611a03aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.926+0000 7fa52be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a6fc0 0x5611a03ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.926+0000 7fa52aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a3050 0x5611a03abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.926+0000 7fa52be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a6fc0 0x5611a03ac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:49634/0 (socket says 172.21.15.27:49634) 2024-05-10T11:22:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.926+0000 7fa52be00700 1 -- 172.21.15.27:0/3765560033 learned_addr learned my addr 172.21.15.27:0/3765560033 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.927+0000 7fa52be00700 1 -- 172.21.15.27:0/3765560033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a2690 msgr2=0x5611a03aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.927+0000 7fa52be00700 1 --2- 172.21.15.27:0/3765560033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a2690 0x5611a03aebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.927+0000 7fa52be00700 1 -- 172.21.15.27:0/3765560033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a3050 msgr2=0x5611a03abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.927+0000 7fa52be00700 1 --2- 172.21.15.27:0/3765560033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a3050 0x5611a03abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.927+0000 7fa52be00700 1 -- 172.21.15.27:0/3765560033 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611a02b2420 con 0x5611a03a6fc0 2024-05-10T11:22:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.927+0000 7fa52b400700 1 --2- 172.21.15.27:0/3765560033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a2690 0x5611a03aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:22:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.927+0000 7fa52be00700 1 --2- 172.21.15.27:0/3765560033 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a6fc0 0x5611a03ac490 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa5240dfa10 tx=0x7fa524004f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50867b93181d4ab0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.927+0000 7fa515600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:22:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.928+0000 7fa515600700 1 -- 172.21.15.27:0/3765560033 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5240e2050 con 0x5611a03a6fc0 2024-05-10T11:22:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.928+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3765560033 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5240ad980 con 0x5611a03a6fc0 2024-05-10T11:22:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.928+0000 7fa515600700 1 -- 172.21.15.27:0/3765560033 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7fa5240b1f10 con 0x5611a03a6fc0 2024-05-10T11:22:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.928+0000 7fa515600700 1 -- 172.21.15.27:0/3765560033 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa5240b2110 con 0x5611a03a6fc0 2024-05-10T11:22:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.928+0000 7fa53d00cb80 10 client.26690 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.928+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3765560033 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5611a03b9d50 con 0x5611a03a6fc0 2024-05-10T11:22:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.928+0000 7fa515600700 1 -- 172.21.15.27:0/3765560033 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5240ad980 con 0x5611a03a6fc0 2024-05-10T11:22:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.929+0000 7fa515600700 1 -- 172.21.15.27:0/3765560033 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fa5240aecb0 con 0x5611a03a6fc0 2024-05-10T11:22:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.929+0000 7fa53d00cb80 10 client.26690 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.929+0000 7fa53d00cb80 10 client.26690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:22:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.929+0000 7fa53d00cb80 20 client.26690 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.929+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/3765560033 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5611a03bc220 0x5611a03be610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.929+0000 7fa53d00cb80 4 client.26690 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:22:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.929+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3765560033 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611a03670a0 con 0x5611a03bc220 2024-05-10T11:22:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.929+0000 7fa52b400700 1 --2- 172.21.15.27:0/3765560033 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5611a03bc220 0x5611a03be610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.930+0000 7fa52b400700 1 --2- 172.21.15.27:0/3765560033 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5611a03bc220 0x5611a03be610 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7641f151edc7a8e server_cookie=f3d103926f42cc2 in_seq=0 out_seq=0 2024-05-10T11:22:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.930+0000 7fa515600700 10 client.26690 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.931+0000 7fa515600700 1 -- 172.21.15.27:0/3765560033 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5611a03670a0 con 0x5611a03bc220 2024-05-10T11:22:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.931+0000 7fa515600700 10 client.26690 handle_command_reply: tid=0 2024-05-10T11:22:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.931+0000 7fa53d00cb80 1 client.26690 shutdown 2024-05-10T11:22:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.931+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3765560033 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a6fc0 msgr2=0x5611a03ac490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.931+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/3765560033 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a6fc0 0x5611a03ac490 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa5240dfa10 tx=0x7fa524004f90 comp rx=0 tx=0).stop 2024-05-10T11:22:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.931+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3765560033 shutdown_connections 2024-05-10T11:22:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.931+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/3765560033 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5611a03bc220 0x5611a03be610 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.931+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/3765560033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a2690 0x5611a03aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.931+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/3765560033 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a6fc0 0x5611a03ac490 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.931+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/3765560033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a3050 0x5611a03abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.931+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3765560033 >> 172.21.15.27:0/3765560033 conn(0x5611a02c8b60 msgr2=0x5611a038f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.931+0000 7fa515600700 0 client.26690 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.932+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3765560033 shutdown_connections 2024-05-10T11:22:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.932+0000 7fa53d00cb80 1 -- 172.21.15.27:0/3765560033 wait complete. 2024-05-10T11:22:47.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.932+0000 7fa53d00cb80 20 client.26690 trim_cache size 0 max 16384 2024-05-10T11:22:47.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.961+0000 7fa53d00cb80 1 Processor -- start 2024-05-10T11:22:47.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.961+0000 7fa53d00cb80 1 -- start start 2024-05-10T11:22:47.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:47 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:22:47.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.962+0000 7fa53d00cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 0x5611a03aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.962+0000 7fa53d00cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a3050 0x5611a03c1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.962+0000 7fa53d00cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03c1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.962+0000 7fa53d00cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1ad550 con 0x5611a03a6fc0 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.962+0000 7fa53d00cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1af0c0 con 0x5611a03a2690 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.962+0000 7fa53d00cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1adc20 con 0x5611a03a3050 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.962+0000 7fa52aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a3050 0x5611a03c1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.962+0000 7fa52aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a3050 0x5611a03c1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36848/0 (socket says 172.21.15.27:36848) 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.962+0000 7fa52aa00700 1 -- 172.21.15.27:0/1911302972 learned_addr learned my addr 172.21.15.27:0/1911302972 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.962+0000 7fa52b400700 1 --2- 172.21.15.27:0/1911302972 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 0x5611a03aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.963+0000 7fa52be00700 1 --2- 172.21.15.27:0/1911302972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03c1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.963+0000 7fa52b400700 1 -- 172.21.15.27:0/1911302972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a3050 msgr2=0x5611a03c1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.963+0000 7fa52b400700 1 --2- 172.21.15.27:0/1911302972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a3050 0x5611a03c1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.963+0000 7fa52b400700 1 -- 172.21.15.27:0/1911302972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 msgr2=0x5611a03c1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.963+0000 7fa52b400700 1 --2- 172.21.15.27:0/1911302972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03c1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.963+0000 7fa52b400700 1 -- 172.21.15.27:0/1911302972 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611a03c1cd0 con 0x5611a03a2690 2024-05-10T11:22:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.963+0000 7fa52be00700 1 --2- 172.21.15.27:0/1911302972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03c1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:22:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.963+0000 7fa52b400700 1 --2- 172.21.15.27:0/1911302972 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 0x5611a03aaf70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x5611a03af540 tx=0x7fa51c013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bfe28b2a64bbb1c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.964+0000 7fa517e00700 1 -- 172.21.15.27:0/1911302972 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c011910 con 0x5611a03a2690 2024-05-10T11:22:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.964+0000 7fa517e00700 1 -- 172.21.15.27:0/1911302972 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c02d070 con 0x5611a03a2690 2024-05-10T11:22:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.964+0000 7fa517e00700 1 -- 172.21.15.27:0/1911302972 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c020400 con 0x5611a03a2690 2024-05-10T11:22:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.964+0000 7fa53d00cb80 1 -- 172.21.15.27:0/1911302972 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 msgr2=0x5611a03aaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.964+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/1911302972 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 0x5611a03aaf70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x5611a03af540 tx=0x7fa51c013000 comp rx=0 tx=0).stop 2024-05-10T11:22:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.965+0000 7fa53d00cb80 1 -- 172.21.15.27:0/1911302972 shutdown_connections 2024-05-10T11:22:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.965+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/1911302972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03c1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.965+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/1911302972 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 0x5611a03aaf70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.965+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/1911302972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a3050 0x5611a03c1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.965+0000 7fa53d00cb80 1 -- 172.21.15.27:0/1911302972 >> 172.21.15.27:0/1911302972 conn(0x5611a02c8b60 msgr2=0x5611a03aa860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.965+0000 7fa53d00cb80 1 -- 172.21.15.27:0/1911302972 shutdown_connections 2024-05-10T11:22:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.965+0000 7fa53d00cb80 1 -- 172.21.15.27:0/1911302972 wait complete. 2024-05-10T11:22:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.965+0000 7fa53d00cb80 1 -- start start 2024-05-10T11:22:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.965+0000 7fa53d00cb80 1 Processor -- start 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.966+0000 7fa53d00cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 0x5611a03c1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.966+0000 7fa53d00cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a3050 0x5611a03be9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.966+0000 7fa53d00cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03bef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.966+0000 7fa53d00cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1ae9f0 con 0x5611a03a6fc0 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.966+0000 7fa53d00cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1ae320 con 0x5611a03a2690 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.966+0000 7fa53d00cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1ab510 con 0x5611a03a3050 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.966+0000 7fa53d00cb80 10 client.0 _ll_register_callbacks cb 0x5611a0293c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.966+0000 7fa52b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 0x5611a03c1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.967+0000 7fa52b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 0x5611a03c1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:49646/0 (socket says 172.21.15.27:49646) 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.967+0000 7fa52b400700 1 -- 172.21.15.27:0/841620451 learned_addr learned my addr 172.21.15.27:0/841620451 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.967+0000 7fa52aa00700 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a3050 0x5611a03be9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.967+0000 7fa52be00700 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03bef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.967+0000 7fa52b400700 1 -- 172.21.15.27:0/841620451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a3050 msgr2=0x5611a03be9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:47.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.967+0000 7fa52b400700 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a3050 0x5611a03be9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.967+0000 7fa52b400700 1 -- 172.21.15.27:0/841620451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 msgr2=0x5611a03bef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:22:47.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.967+0000 7fa52b400700 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03bef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.967+0000 7fa52b400700 1 -- 172.21.15.27:0/841620451 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611a03bf7a0 con 0x5611a03a2690 2024-05-10T11:22:47.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.967+0000 7fa52b400700 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 0x5611a03c1640 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa51c02a040 tx=0x7fa51c00e690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e98ebfde8f761386 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:22:47.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.967+0000 7fa515600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:22:47.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.968+0000 7fa515600700 1 -- 172.21.15.27:0/841620451 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c035020 con 0x5611a03a2690 2024-05-10T11:22:47.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.968+0000 7fa515600700 1 -- 172.21.15.27:0/841620451 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9591+0+0 (secure 0 0 0) 0x7fa51c027310 con 0x5611a03a2690 2024-05-10T11:22:47.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.968+0000 7fa53d00cb80 1 -- 172.21.15.27:0/841620451 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa51c02d070 con 0x5611a03a2690 2024-05-10T11:22:47.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.968+0000 7fa515600700 1 -- 172.21.15.27:0/841620451 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa51c040050 con 0x5611a03a2690 2024-05-10T11:22:47.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.968+0000 7fa53d00cb80 10 client.26696 fetch_fsmap learned FSMap version 34 2024-05-10T11:22:47.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.968+0000 7fa53d00cb80 1 -- 172.21.15.27:0/841620451 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5611a0426800 con 0x5611a03a2690 2024-05-10T11:22:47.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.968+0000 7fa515600700 1 -- 172.21.15.27:0/841620451 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c02d070 con 0x5611a03a2690 2024-05-10T11:22:47.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.969+0000 7fa515600700 1 -- 172.21.15.27:0/841620451 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fa51c027ce0 con 0x5611a03a2690 2024-05-10T11:22:47.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.969+0000 7fa53d00cb80 10 client.26696 fetch_fsmap finished waiting for FSMap version 34 2024-05-10T11:22:47.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.969+0000 7fa53d00cb80 10 client.26696 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:22:47.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.969+0000 7fa53d00cb80 20 client.26696 populate_metadata read hostname 'smithi027' 2024-05-10T11:22:47.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.969+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5611a0426e80 0x5611a04292a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:22:47.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.969+0000 7fa53d00cb80 4 client.26696 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:22:47.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.969+0000 7fa53d00cb80 1 -- 172.21.15.27:0/841620451 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5611a02fa400 con 0x5611a0426e80 2024-05-10T11:22:47.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.970+0000 7fa52aa00700 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5611a0426e80 0x5611a04292a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:22:47.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.970+0000 7fa52aa00700 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5611a0426e80 0x5611a04292a0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eed5f4c88636ba4 server_cookie=66478cc6adb9f609 in_seq=0 out_seq=0 2024-05-10T11:22:47.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.970+0000 7fa515600700 10 client.26696 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:47.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa515600700 1 -- 172.21.15.27:0/841620451 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5611a02fa400 con 0x5611a0426e80 2024-05-10T11:22:47.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa515600700 10 client.26696 handle_command_reply: tid=0 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa53d00cb80 1 client.26696 shutdown 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa53d00cb80 1 -- 172.21.15.27:0/841620451 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 msgr2=0x5611a03c1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 0x5611a03c1640 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa51c02a040 tx=0x7fa51c00e690 comp rx=0 tx=0).stop 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa53d00cb80 1 -- 172.21.15.27:0/841620451 shutdown_connections 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5611a0426e80 0x5611a04292a0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5611a03a6fc0 0x5611a03bef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5611a03a2690 0x5611a03c1640 secure :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa51c02a040 tx=0x7fa51c00e690 comp rx=0 tx=0).stop 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa53d00cb80 1 --2- 172.21.15.27:0/841620451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a03a3050 0x5611a03be9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa53d00cb80 1 -- 172.21.15.27:0/841620451 >> 172.21.15.27:0/841620451 conn(0x5611a02c8b60 msgr2=0x5611a038f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa515600700 0 client.26696 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.971+0000 7fa53d00cb80 1 -- 172.21.15.27:0/841620451 shutdown_connections 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.972+0000 7fa53d00cb80 1 -- 172.21.15.27:0/841620451 wait complete. 2024-05-10T11:22:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.972+0000 7fa53d00cb80 20 client.26696 trim_cache size 0 max 16384 2024-05-10T11:22:47.974 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:22:47.974 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:22:47.974 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:22:47.974 INFO:teuthology.orchestra.run.smithi027.stdout: "2d5fa5d1-4c1b-4015-9fd6-9c95fdd8431a": { 2024-05-10T11:22:47.974 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:22:47.974 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "2d5fa5d1-4c1b-4015-9fd6-9c95fdd8431a", 2024-05-10T11:22:47.974 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:22:47.974 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:22:47.974 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:22:47.974 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:22:47.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.973+0000 7fa532600700 1 -- 172.21.15.27:0/3969726742 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa5100b2a70 msgr2=0x7fa5100b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:47.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.973+0000 7fa532600700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa5100b2a70 0x7fa5100b4f30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa51c005fd0 tx=0x7fa51c005f60 comp rx=0 tx=0).stop 2024-05-10T11:22:47.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.973+0000 7fa532600700 1 -- 172.21.15.27:0/3969726742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52c109510 msgr2=0x7fa52c1a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:22:47.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.973+0000 7fa532600700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52c109510 0x7fa52c1a0650 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa52400ef30 tx=0x7fa52400c450 comp rx=0 tx=0).stop 2024-05-10T11:22:47.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.973+0000 7fa532600700 1 -- 172.21.15.27:0/3969726742 shutdown_connections 2024-05-10T11:22:47.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.973+0000 7fa532600700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa5100b2a70 0x7fa5100b4f30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.973+0000 7fa532600700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52c109510 0x7fa52c1a0650 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.973+0000 7fa532600700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa52c06b180 0x7fa52c19bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.973+0000 7fa532600700 1 --2- 172.21.15.27:0/3969726742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c06bb40 0x7fa52c19c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:22:47.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.973+0000 7fa532600700 1 -- 172.21.15.27:0/3969726742 >> 172.21.15.27:0/3969726742 conn(0x7fa52c079350 msgr2=0x7fa52c101040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:22:47.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.973+0000 7fa532600700 1 -- 172.21.15.27:0/3969726742 shutdown_connections 2024-05-10T11:22:47.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:22:47.973+0000 7fa532600700 1 -- 172.21.15.27:0/3969726742 wait complete. 2024-05-10T11:22:47.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:47 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:22:48.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:47 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:22:48.714 INFO:tasks.workunit.client.0.smithi027.stdout: 1 568760 6.55 MB/sec execute 333 sec latency 25.086 ms 2024-05-10T11:22:48.977 INFO:tasks.cephfs.filesystem:scrub status for tag:2d5fa5d1-4c1b-4015-9fd6-9c95fdd8431a - {'path': '/', 'tag': '2d5fa5d1-4c1b-4015-9fd6-9c95fdd8431a', 'options': 'recursive,force'} 2024-05-10T11:22:48.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:48 smithi189 ceph-mon[34842]: pgmap v296: 129 pgs: 129 active+clean; 411 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 134 op/s 2024-05-10T11:22:48.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:48 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:22:48.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:48 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:22:48.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:48 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:22:49.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:48 smithi027 ceph-mon[30054]: pgmap v296: 129 pgs: 129 active+clean; 411 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 134 op/s 2024-05-10T11:22:49.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:48 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:22:49.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:48 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:22:49.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:48 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:22:49.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:48 smithi076 ceph-mon[34152]: pgmap v296: 129 pgs: 129 active+clean; 411 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 134 op/s 2024-05-10T11:22:49.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:48 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:22:49.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:48 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:22:49.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:48 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:22:49.714 INFO:tasks.workunit.client.0.smithi027.stdout: 1 569832 6.55 MB/sec execute 334 sec latency 52.481 ms 2024-05-10T11:22:50.714 INFO:tasks.workunit.client.0.smithi027.stdout: 1 571220 6.55 MB/sec execute 335 sec latency 25.212 ms 2024-05-10T11:22:50.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:50 smithi189 ceph-mon[34842]: pgmap v297: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 202 op/s 2024-05-10T11:22:51.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:50 smithi027 ceph-mon[30054]: pgmap v297: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 202 op/s 2024-05-10T11:22:51.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:50 smithi076 ceph-mon[34152]: pgmap v297: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 202 op/s 2024-05-10T11:22:51.714 INFO:tasks.workunit.client.0.smithi027.stdout: 1 572256 6.55 MB/sec execute 336 sec latency 19.064 ms 2024-05-10T11:22:51.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:51 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:22:52.075 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:51 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:22:52.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:51 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:22:52.714 INFO:tasks.workunit.client.0.smithi027.stdout: 1 573139 6.54 MB/sec execute 337 sec latency 54.707 ms 2024-05-10T11:22:53.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:52 smithi027 ceph-mon[30054]: pgmap v298: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 134 op/s 2024-05-10T11:22:53.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:52 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:22:53.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:52 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:22:53.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:52 smithi076 ceph-mon[34152]: pgmap v298: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 134 op/s 2024-05-10T11:22:53.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:52 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:22:53.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:52 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:22:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:52 smithi189 ceph-mon[34842]: pgmap v298: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 134 op/s 2024-05-10T11:22:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:52 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:22:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:52 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:22:53.714 INFO:tasks.workunit.client.0.smithi027.stdout: 1 574677 6.54 MB/sec execute 338 sec latency 29.731 ms 2024-05-10T11:22:54.715 INFO:tasks.workunit.client.0.smithi027.stdout: 1 575874 6.55 MB/sec execute 339 sec latency 14.614 ms 2024-05-10T11:22:55.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:54 smithi027 ceph-mon[30054]: pgmap v299: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 134 op/s 2024-05-10T11:22:55.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:54 smithi076 ceph-mon[34152]: pgmap v299: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 134 op/s 2024-05-10T11:22:55.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:54 smithi189 ceph-mon[34842]: pgmap v299: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 134 op/s 2024-05-10T11:22:55.715 INFO:tasks.workunit.client.0.smithi027.stdout: 1 577093 6.54 MB/sec execute 340 sec latency 26.219 ms 2024-05-10T11:22:56.715 INFO:tasks.workunit.client.0.smithi027.stdout: 1 578990 6.56 MB/sec execute 341 sec latency 20.649 ms 2024-05-10T11:22:57.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:56 smithi027 ceph-mon[30054]: pgmap v300: 129 pgs: 129 active+clean; 424 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 204 op/s 2024-05-10T11:22:57.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:56 smithi076 ceph-mon[34152]: pgmap v300: 129 pgs: 129 active+clean; 424 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 204 op/s 2024-05-10T11:22:57.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:56 smithi189 ceph-mon[34842]: pgmap v300: 129 pgs: 129 active+clean; 424 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 204 op/s 2024-05-10T11:22:57.715 INFO:tasks.workunit.client.0.smithi027.stdout: 1 579644 6.55 MB/sec execute 342 sec latency 47.923 ms 2024-05-10T11:22:58.718 INFO:tasks.workunit.client.0.smithi027.stdout: 1 580888 6.54 MB/sec execute 343 sec latency 24.111 ms 2024-05-10T11:22:59.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:22:58 smithi027 ceph-mon[30054]: pgmap v301: 129 pgs: 129 active+clean; 424 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 137 op/s 2024-05-10T11:22:59.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:22:58 smithi076 ceph-mon[34152]: pgmap v301: 129 pgs: 129 active+clean; 424 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 137 op/s 2024-05-10T11:22:59.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:22:58 smithi189 ceph-mon[34842]: pgmap v301: 129 pgs: 129 active+clean; 424 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 137 op/s 2024-05-10T11:22:59.715 INFO:tasks.workunit.client.0.smithi027.stdout: 1 582725 6.56 MB/sec execute 344 sec latency 20.089 ms 2024-05-10T11:23:00.715 INFO:tasks.workunit.client.0.smithi027.stdout: 1 583339 6.55 MB/sec execute 345 sec latency 45.720 ms 2024-05-10T11:23:01.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:00 smithi027 ceph-mon[30054]: pgmap v302: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 207 op/s 2024-05-10T11:23:01.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:00 smithi076 ceph-mon[34152]: pgmap v302: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 207 op/s 2024-05-10T11:23:01.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:00 smithi189 ceph-mon[34842]: pgmap v302: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 207 op/s 2024-05-10T11:23:01.715 INFO:tasks.workunit.client.0.smithi027.stdout: 1 584498 6.54 MB/sec execute 346 sec latency 39.971 ms 2024-05-10T11:23:02.715 INFO:tasks.workunit.client.0.smithi027.stdout: 1 584820 6.53 MB/sec execute 347 sec latency 148.232 ms 2024-05-10T11:23:03.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:02 smithi027 ceph-mon[30054]: pgmap v303: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 139 op/s 2024-05-10T11:23:03.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:02 smithi027 ceph-mon[30054]: osdmap e98: 12 total, 12 up, 12 in 2024-05-10T11:23:03.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:02 smithi076 ceph-mon[34152]: pgmap v303: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 139 op/s 2024-05-10T11:23:03.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:02 smithi076 ceph-mon[34152]: osdmap e98: 12 total, 12 up, 12 in 2024-05-10T11:23:03.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:02 smithi189 ceph-mon[34842]: pgmap v303: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 139 op/s 2024-05-10T11:23:03.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:02 smithi189 ceph-mon[34842]: osdmap e98: 12 total, 12 up, 12 in 2024-05-10T11:23:03.716 INFO:tasks.workunit.client.0.smithi027.stdout: 1 586191 6.54 MB/sec execute 348 sec latency 63.503 ms 2024-05-10T11:23:04.716 INFO:tasks.workunit.client.0.smithi027.stdout: 1 586747 6.53 MB/sec execute 349 sec latency 25.172 ms 2024-05-10T11:23:05.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:04 smithi027 ceph-mon[30054]: pgmap v305: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 167 op/s 2024-05-10T11:23:05.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:04 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:23:05.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:04 smithi076 ceph-mon[34152]: pgmap v305: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 167 op/s 2024-05-10T11:23:05.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:04 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:23:05.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:04 smithi189 ceph-mon[34842]: pgmap v305: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 167 op/s 2024-05-10T11:23:05.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:04 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:23:05.716 INFO:tasks.workunit.client.0.smithi027.stdout: 1 587763 6.52 MB/sec execute 350 sec latency 27.135 ms 2024-05-10T11:23:06.716 INFO:tasks.workunit.client.0.smithi027.stdout: 1 589435 6.53 MB/sec execute 351 sec latency 22.319 ms 2024-05-10T11:23:07.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:06 smithi027 ceph-mon[30054]: pgmap v306: 129 pgs: 129 active+clean; 434 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 198 op/s 2024-05-10T11:23:07.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:06 smithi076 ceph-mon[34152]: pgmap v306: 129 pgs: 129 active+clean; 434 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 198 op/s 2024-05-10T11:23:07.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:06 smithi189 ceph-mon[34842]: pgmap v306: 129 pgs: 129 active+clean; 434 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 198 op/s 2024-05-10T11:23:07.716 INFO:tasks.workunit.client.0.smithi027.stdout: 1 590166 6.52 MB/sec execute 352 sec latency 54.930 ms 2024-05-10T11:23:08.716 INFO:tasks.workunit.client.0.smithi027.stdout: 1 591182 6.52 MB/sec execute 353 sec latency 39.191 ms 2024-05-10T11:23:09.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:08 smithi027 ceph-mon[30054]: pgmap v307: 129 pgs: 129 active+clean; 434 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 198 op/s 2024-05-10T11:23:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:08 smithi027 ceph-mon[30054]: osdmap e99: 12 total, 12 up, 12 in 2024-05-10T11:23:09.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:08 smithi076 ceph-mon[34152]: pgmap v307: 129 pgs: 129 active+clean; 434 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 198 op/s 2024-05-10T11:23:09.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:08 smithi076 ceph-mon[34152]: osdmap e99: 12 total, 12 up, 12 in 2024-05-10T11:23:09.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:08 smithi189 ceph-mon[34842]: pgmap v307: 129 pgs: 129 active+clean; 434 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 198 op/s 2024-05-10T11:23:09.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:08 smithi189 ceph-mon[34842]: osdmap e99: 12 total, 12 up, 12 in 2024-05-10T11:23:09.716 INFO:tasks.workunit.client.0.smithi027.stdout: 1 592549 6.52 MB/sec execute 354 sec latency 28.862 ms 2024-05-10T11:23:10.716 INFO:tasks.workunit.client.0.smithi027.stdout: 1 593490 6.52 MB/sec execute 355 sec latency 18.303 ms 2024-05-10T11:23:11.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:10 smithi027 ceph-mon[30054]: pgmap v309: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 240 op/s 2024-05-10T11:23:11.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:10 smithi076 ceph-mon[34152]: pgmap v309: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 240 op/s 2024-05-10T11:23:11.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:10 smithi189 ceph-mon[34842]: pgmap v309: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 240 op/s 2024-05-10T11:23:11.716 INFO:tasks.workunit.client.0.smithi027.stdout: 1 594321 6.51 MB/sec execute 356 sec latency 40.758 ms 2024-05-10T11:23:12.716 INFO:tasks.workunit.client.0.smithi027.stdout: 1 595589 6.51 MB/sec execute 357 sec latency 53.517 ms 2024-05-10T11:23:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:12 smithi027 ceph-mon[30054]: pgmap v310: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 199 op/s 2024-05-10T11:23:13.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:12 smithi076 ceph-mon[34152]: pgmap v310: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 199 op/s 2024-05-10T11:23:13.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:12 smithi189 ceph-mon[34842]: pgmap v310: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 199 op/s 2024-05-10T11:23:13.718 INFO:tasks.workunit.client.0.smithi027.stdout: 1 597021 6.52 MB/sec execute 358 sec latency 16.713 ms 2024-05-10T11:23:14.717 INFO:tasks.workunit.client.0.smithi027.stdout: 1 598121 6.52 MB/sec execute 359 sec latency 23.130 ms 2024-05-10T11:23:15.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:14 smithi027 ceph-mon[30054]: pgmap v311: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 192 op/s 2024-05-10T11:23:15.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:14 smithi076 ceph-mon[34152]: pgmap v311: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 192 op/s 2024-05-10T11:23:15.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:14 smithi189 ceph-mon[34842]: pgmap v311: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 192 op/s 2024-05-10T11:23:15.717 INFO:tasks.workunit.client.0.smithi027.stdout: 1 599293 6.51 MB/sec execute 360 sec latency 24.211 ms 2024-05-10T11:23:16.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:15 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:23:16.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:15 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:23:16.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:15 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:23:16.717 INFO:tasks.workunit.client.0.smithi027.stdout: 1 600518 6.52 MB/sec execute 361 sec latency 21.511 ms 2024-05-10T11:23:17.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:16 smithi027 ceph-mon[30054]: pgmap v312: 129 pgs: 129 active+clean; 446 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 158 op/s 2024-05-10T11:23:17.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:16 smithi076 ceph-mon[34152]: pgmap v312: 129 pgs: 129 active+clean; 446 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 158 op/s 2024-05-10T11:23:17.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:16 smithi189 ceph-mon[34842]: pgmap v312: 129 pgs: 129 active+clean; 446 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 158 op/s 2024-05-10T11:23:17.717 INFO:tasks.workunit.client.0.smithi027.stdout: 1 601039 6.50 MB/sec execute 362 sec latency 91.101 ms 2024-05-10T11:23:18.717 INFO:tasks.workunit.client.0.smithi027.stdout: 1 602346 6.50 MB/sec execute 363 sec latency 41.220 ms 2024-05-10T11:23:18.978 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:23:19.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:18 smithi027 ceph-mon[30054]: pgmap v313: 129 pgs: 129 active+clean; 446 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 158 op/s 2024-05-10T11:23:19.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:18 smithi076 ceph-mon[34152]: pgmap v313: 129 pgs: 129 active+clean; 446 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 158 op/s 2024-05-10T11:23:19.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:18 smithi189 ceph-mon[34842]: pgmap v313: 129 pgs: 129 active+clean; 446 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 158 op/s 2024-05-10T11:23:19.465 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:23:19.717 INFO:tasks.workunit.client.0.smithi027.stdout: 1 604078 6.52 MB/sec execute 364 sec latency 20.442 ms 2024-05-10T11:23:20.717 INFO:tasks.workunit.client.0.smithi027.stdout: 1 605024 6.51 MB/sec execute 365 sec latency 31.317 ms 2024-05-10T11:23:21.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:20 smithi027 ceph-mon[30054]: pgmap v314: 129 pgs: 129 active+clean; 453 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 217 op/s 2024-05-10T11:23:21.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:20 smithi076 ceph-mon[34152]: pgmap v314: 129 pgs: 129 active+clean; 453 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 217 op/s 2024-05-10T11:23:21.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:20 smithi189 ceph-mon[34842]: pgmap v314: 129 pgs: 129 active+clean; 453 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 217 op/s 2024-05-10T11:23:21.718 INFO:tasks.workunit.client.0.smithi027.stdout: 1 606692 6.52 MB/sec execute 366 sec latency 23.850 ms 2024-05-10T11:23:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.010+0000 7fea3f600700 1 -- 172.21.15.27:0/2252415147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea3806b180 msgr2=0x7fea3806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.010+0000 7fea3f600700 1 --2- 172.21.15.27:0/2252415147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea3806b180 0x7fea3806b600 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fea20009990 tx=0x7fea20009ca0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.011+0000 7fea3f600700 1 -- 172.21.15.27:0/2252415147 shutdown_connections 2024-05-10T11:23:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.011+0000 7fea3f600700 1 --2- 172.21.15.27:0/2252415147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea3806bb40 0x7fea38110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.011+0000 7fea3f600700 1 --2- 172.21.15.27:0/2252415147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea38109510 0x7fea381098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.011+0000 7fea3f600700 1 --2- 172.21.15.27:0/2252415147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea3806b180 0x7fea3806b600 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.011+0000 7fea3f600700 1 -- 172.21.15.27:0/2252415147 >> 172.21.15.27:0/2252415147 conn(0x7fea38079350 msgr2=0x7fea38079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.011+0000 7fea3f600700 1 -- 172.21.15.27:0/2252415147 shutdown_connections 2024-05-10T11:23:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.011+0000 7fea3f600700 1 -- 172.21.15.27:0/2252415147 wait complete. 2024-05-10T11:23:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.013+0000 7fea3f600700 1 Processor -- start 2024-05-10T11:23:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.013+0000 7fea3f600700 1 -- start start 2024-05-10T11:23:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.014+0000 7fea3f600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea3806b180 0x7fea3819bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.014+0000 7fea3f600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea3806bb40 0x7fea3819c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.014+0000 7fea3f600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea38109510 0x7fea381a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.014+0000 7fea3f600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fea38113670 con 0x7fea3806bb40 2024-05-10T11:23:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.014+0000 7fea3f600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fea381134f0 con 0x7fea38109510 2024-05-10T11:23:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.014+0000 7fea3f600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fea381137f0 con 0x7fea3806b180 2024-05-10T11:23:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.014+0000 7fea3cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea3806b180 0x7fea3819bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.014+0000 7fea2fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea3806bb40 0x7fea3819c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.014+0000 7fea3cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea3806b180 0x7fea3819bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:53280/0 (socket says 172.21.15.27:53280) 2024-05-10T11:23:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.014+0000 7fea3cc00700 1 -- 172.21.15.27:0/3528625680 learned_addr learned my addr 172.21.15.27:0/3528625680 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.014+0000 7fea3d600700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea38109510 0x7fea381a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.015+0000 7fea3cc00700 1 -- 172.21.15.27:0/3528625680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea38109510 msgr2=0x7fea381a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:22.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.015+0000 7fea3cc00700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea38109510 0x7fea381a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.015+0000 7fea3cc00700 1 -- 172.21.15.27:0/3528625680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea3806bb40 msgr2=0x7fea3819c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.015+0000 7fea3cc00700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea3806bb40 0x7fea3819c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.015+0000 7fea3cc00700 1 -- 172.21.15.27:0/3528625680 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea381a0d90 con 0x7fea3806b180 2024-05-10T11:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.015+0000 7fea3cc00700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea3806b180 0x7fea3819bd80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fea2800d950 tx=0x7fea2800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.015+0000 7fea36000700 1 -- 172.21.15.27:0/3528625680 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28016070 con 0x7fea3806b180 2024-05-10T11:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.015+0000 7fea36000700 1 -- 172.21.15.27:0/3528625680 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea28009df0 con 0x7fea3806b180 2024-05-10T11:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.015+0000 7fea36000700 1 -- 172.21.15.27:0/3528625680 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2800f3c0 con 0x7fea3806b180 2024-05-10T11:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.015+0000 7fea3f600700 1 -- 172.21.15.27:0/3528625680 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea381a1080 con 0x7fea3806b180 2024-05-10T11:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.016+0000 7fea3f600700 1 -- 172.21.15.27:0/3528625680 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea381a8900 con 0x7fea3806b180 2024-05-10T11:23:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.019+0000 7fea36000700 1 -- 172.21.15.27:0/3528625680 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fea28019070 con 0x7fea3806b180 2024-05-10T11:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.020+0000 7fea4a063b80 1 Processor -- start 2024-05-10T11:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.021+0000 7fea4a063b80 1 -- start start 2024-05-10T11:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.021+0000 7fea36000700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fea1c0b2a70 0x7fea1c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.021+0000 7fea36000700 1 -- 172.21.15.27:0/3528625680 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7fea280e25c0 con 0x7fea3806b180 2024-05-10T11:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.021+0000 7fea4a063b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b2e060 0x5603e9b2e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.021+0000 7fea4a063b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e980 0x5603e9b32490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.021+0000 7fea4a063b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b329d0 0x5603e9b32e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.021+0000 7fea4a063b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5603e9a80a20 con 0x5603e9b329d0 2024-05-10T11:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.021+0000 7fea4a063b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5603e9a1e760 con 0x5603e9b2e980 2024-05-10T11:23:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.021+0000 7fea4a063b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5603e9a7c9e0 con 0x5603e9b2e060 2024-05-10T11:23:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.022+0000 7fea3cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b2e060 0x5603e9b2e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.022+0000 7fea3d600700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fea1c0b2a70 0x7fea1c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.022+0000 7fea2fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e980 0x5603e9b32490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.022+0000 7fea3cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b2e060 0x5603e9b2e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:53282/0 (socket says 172.21.15.27:53282) 2024-05-10T11:23:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.022+0000 7fea3cc00700 1 -- 172.21.15.27:0/2558770799 learned_addr learned my addr 172.21.15.27:0/2558770799 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.022+0000 7fea2fe00700 1 -- 172.21.15.27:0/2558770799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b2e060 msgr2=0x5603e9b2e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.022+0000 7fea2fe00700 1 --2- 172.21.15.27:0/2558770799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b2e060 0x5603e9b2e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.022+0000 7fea2fe00700 1 -- 172.21.15.27:0/2558770799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b329d0 msgr2=0x5603e9b32e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.022+0000 7fea2fe00700 1 --2- 172.21.15.27:0/2558770799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b329d0 0x5603e9b32e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.022+0000 7fea2fe00700 1 -- 172.21.15.27:0/2558770799 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603e99cd8d0 con 0x5603e9b2e980 2024-05-10T11:23:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.023+0000 7fea2fe00700 1 --2- 172.21.15.27:0/2558770799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e980 0x5603e9b32490 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fea20006010 tx=0x7fea20009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce694a6c8b886d14 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.023+0000 7fea3d600700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fea1c0b2a70 0x7fea1c0b4f30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fea30009e00 tx=0x7fea300093f0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.023+0000 7fea2e000700 1 -- 172.21.15.27:0/2558770799 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea20003a60 con 0x5603e9b2e980 2024-05-10T11:23:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.023+0000 7fea2e000700 1 -- 172.21.15.27:0/2558770799 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea20026070 con 0x5603e9b2e980 2024-05-10T11:23:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.023+0000 7fea2e000700 1 -- 172.21.15.27:0/2558770799 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2000ba30 con 0x5603e9b2e980 2024-05-10T11:23:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.024+0000 7fea4a063b80 1 -- 172.21.15.27:0/2558770799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e980 msgr2=0x5603e9b32490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.024+0000 7fea4a063b80 1 --2- 172.21.15.27:0/2558770799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e980 0x5603e9b32490 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fea20006010 tx=0x7fea20009f90 comp rx=0 tx=0).stop 2024-05-10T11:23:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.024+0000 7fea4a063b80 1 -- 172.21.15.27:0/2558770799 shutdown_connections 2024-05-10T11:23:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.024+0000 7fea4a063b80 1 --2- 172.21.15.27:0/2558770799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b2e060 0x5603e9b2e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.024+0000 7fea4a063b80 1 --2- 172.21.15.27:0/2558770799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b329d0 0x5603e9b32e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.024+0000 7fea4a063b80 1 --2- 172.21.15.27:0/2558770799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e980 0x5603e9b32490 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.024+0000 7fea4a063b80 1 -- 172.21.15.27:0/2558770799 >> 172.21.15.27:0/2558770799 conn(0x5603e9a5cb60 msgr2=0x5603e9b287a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.024+0000 7fea4a063b80 1 -- 172.21.15.27:0/2558770799 shutdown_connections 2024-05-10T11:23:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.024+0000 7fea4a063b80 1 -- 172.21.15.27:0/2558770799 wait complete. 2024-05-10T11:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.025+0000 7fea4a063b80 1 -- start start 2024-05-10T11:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.025+0000 7fea4a063b80 1 Processor -- start 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea4a063b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b39e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea4a063b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b371f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea4a063b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b37730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea4a063b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5603e9b3ab80 con 0x5603e9b2e980 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea4a063b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5603e99e6b20 con 0x5603e9b2e060 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea4a063b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5603e9b3ad00 con 0x5603e9b329d0 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea4a063b80 10 client.0 _ll_register_callbacks cb 0x5603e9a27c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea3cc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b39e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea3cc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b39e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42670/0 (socket says 172.21.15.27:42670) 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea3cc00700 1 -- 172.21.15.27:0/1005034536 learned_addr learned my addr 172.21.15.27:0/1005034536 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea2fe00700 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea3d600700 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b37730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea3cc00700 1 -- 172.21.15.27:0/1005034536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 msgr2=0x5603e9b37730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea3cc00700 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b37730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea3cc00700 1 -- 172.21.15.27:0/1005034536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 msgr2=0x5603e9b371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea3cc00700 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b371f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea3cc00700 1 -- 172.21.15.27:0/1005034536 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603e9a46420 con 0x5603e9b2e060 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.026+0000 7fea2fe00700 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b371f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.027+0000 7fea3cc00700 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b39e50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fea28000c00 tx=0x7fea28010940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4746c1f3cfe25e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.027+0000 7fea26a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.027+0000 7fea26a00700 1 -- 172.21.15.27:0/1005034536 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28020070 con 0x5603e9b2e060 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.027+0000 7fea26a00700 1 -- 172.21.15.27:0/1005034536 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7fea280e2360 con 0x5603e9b2e060 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.027+0000 7fea4a063b80 1 -- 172.21.15.27:0/1005034536 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea20026070 con 0x5603e9b2e060 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.027+0000 7fea26a00700 1 -- 172.21.15.27:0/1005034536 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fea280b2270 con 0x5603e9b2e060 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.027+0000 7fea4a063b80 10 client.26708 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.027+0000 7fea4a063b80 1 -- 172.21.15.27:0/1005034536 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5603e9b45430 con 0x5603e9b2e060 2024-05-10T11:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.027+0000 7fea26a00700 1 -- 172.21.15.27:0/1005034536 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea20026070 con 0x5603e9b2e060 2024-05-10T11:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.028+0000 7fea26a00700 1 -- 172.21.15.27:0/1005034536 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fea280ca310 con 0x5603e9b2e060 2024-05-10T11:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.028+0000 7fea4a063b80 10 client.26708 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.028+0000 7fea4a063b80 10 client.26708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.028+0000 7fea4a063b80 20 client.26708 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.028+0000 7fea4a063b80 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5603e9b47900 0x5603e9b49d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.028+0000 7fea4a063b80 4 client.26708 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.028+0000 7fea4a063b80 1 -- 172.21.15.27:0/1005034536 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603e9afb0a0 con 0x5603e9b47900 2024-05-10T11:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.028+0000 7fea2fe00700 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5603e9b47900 0x5603e9b49d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.029+0000 7fea2fe00700 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5603e9b47900 0x5603e9b49d60 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=985e054aa7e55084 server_cookie=1634d06071e5f5d3 in_seq=0 out_seq=0 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.029+0000 7fea26a00700 10 client.26708 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.031+0000 7fea26a00700 1 -- 172.21.15.27:0/1005034536 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5603e9afb0a0 con 0x5603e9b47900 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.031+0000 7fea26a00700 10 client.26708 handle_command_reply: tid=0 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.031+0000 7fea4a063b80 1 client.26708 shutdown 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.032+0000 7fea4a063b80 1 -- 172.21.15.27:0/1005034536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 msgr2=0x5603e9b39e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.032+0000 7fea4a063b80 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b39e50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fea28000c00 tx=0x7fea28010940 comp rx=0 tx=0).stop 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.032+0000 7fea4a063b80 1 -- 172.21.15.27:0/1005034536 shutdown_connections 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.032+0000 7fea4a063b80 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5603e9b47900 0x5603e9b49d60 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.032+0000 7fea4a063b80 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b37730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.032+0000 7fea4a063b80 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b39e50 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.032+0000 7fea4a063b80 1 --2- 172.21.15.27:0/1005034536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b371f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.032+0000 7fea4a063b80 1 -- 172.21.15.27:0/1005034536 >> 172.21.15.27:0/1005034536 conn(0x5603e9a5cb60 msgr2=0x5603e9b232b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.032+0000 7fea26a00700 0 client.26708 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:22.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.032+0000 7fea4a063b80 1 -- 172.21.15.27:0/1005034536 shutdown_connections 2024-05-10T11:23:22.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.032+0000 7fea4a063b80 1 -- 172.21.15.27:0/1005034536 wait complete. 2024-05-10T11:23:22.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.032+0000 7fea4a063b80 20 client.26708 trim_cache size 0 max 16384 2024-05-10T11:23:22.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.056+0000 7fea4a063b80 1 Processor -- start 2024-05-10T11:23:22.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.056+0000 7fea4a063b80 1 -- start start 2024-05-10T11:23:22.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.056+0000 7fea4a063b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b43c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.056+0000 7fea4a063b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b4f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.056+0000 7fea4a063b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b4fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.056+0000 7fea4a063b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fea381b18a0 con 0x5603e9b2e980 2024-05-10T11:23:22.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.056+0000 7fea4a063b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fea381b33e0 con 0x5603e9b2e060 2024-05-10T11:23:22.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.056+0000 7fea4a063b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fea381b1f70 con 0x5603e9b329d0 2024-05-10T11:23:22.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.057+0000 7fea3cc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b43c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.057+0000 7fea3cc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b43c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42678/0 (socket says 172.21.15.27:42678) 2024-05-10T11:23:22.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.057+0000 7fea3cc00700 1 -- 172.21.15.27:0/3941410960 learned_addr learned my addr 172.21.15.27:0/3941410960 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:22.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.057+0000 7fea3d600700 1 --2- 172.21.15.27:0/3941410960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b4fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.057+0000 7fea2fe00700 1 --2- 172.21.15.27:0/3941410960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b4f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.057+0000 7fea3d600700 1 -- 172.21.15.27:0/3941410960 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 msgr2=0x5603e9b43c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.057+0000 7fea3d600700 1 --2- 172.21.15.27:0/3941410960 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b43c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.057+0000 7fea3d600700 1 -- 172.21.15.27:0/3941410960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 msgr2=0x5603e9b4f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.057+0000 7fea3d600700 1 --2- 172.21.15.27:0/3941410960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b4f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.058+0000 7fea3d600700 1 -- 172.21.15.27:0/3941410960 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603e9b32770 con 0x5603e9b329d0 2024-05-10T11:23:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.058+0000 7fea3d600700 1 --2- 172.21.15.27:0/3941410960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b4fb70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fea30007b70 tx=0x7fea30016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b04330f95586067 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.058+0000 7fea2d600700 1 -- 172.21.15.27:0/3941410960 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea30012040 con 0x5603e9b329d0 2024-05-10T11:23:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.058+0000 7fea2d600700 1 -- 172.21.15.27:0/3941410960 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea30028070 con 0x5603e9b329d0 2024-05-10T11:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.058+0000 7fea2d600700 1 -- 172.21.15.27:0/3941410960 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea30023e60 con 0x5603e9b329d0 2024-05-10T11:23:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.059+0000 7fea4a063b80 1 -- 172.21.15.27:0/3941410960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 msgr2=0x5603e9b4fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.059+0000 7fea4a063b80 1 --2- 172.21.15.27:0/3941410960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b4fb70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fea30007b70 tx=0x7fea30016000 comp rx=0 tx=0).stop 2024-05-10T11:23:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.059+0000 7fea4a063b80 1 -- 172.21.15.27:0/3941410960 shutdown_connections 2024-05-10T11:23:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.059+0000 7fea4a063b80 1 --2- 172.21.15.27:0/3941410960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b4fb70 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.059+0000 7fea4a063b80 1 --2- 172.21.15.27:0/3941410960 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b43c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.059+0000 7fea4a063b80 1 --2- 172.21.15.27:0/3941410960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b4f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.059+0000 7fea4a063b80 1 -- 172.21.15.27:0/3941410960 >> 172.21.15.27:0/3941410960 conn(0x5603e9a5cb60 msgr2=0x5603e9b37520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.059+0000 7fea4a063b80 1 -- 172.21.15.27:0/3941410960 shutdown_connections 2024-05-10T11:23:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.059+0000 7fea4a063b80 1 -- 172.21.15.27:0/3941410960 wait complete. 2024-05-10T11:23:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.060+0000 7fea4a063b80 1 -- start start 2024-05-10T11:23:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.060+0000 7fea4a063b80 1 Processor -- start 2024-05-10T11:23:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea4a063b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b3ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea4a063b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b404e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea4a063b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b3af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea4a063b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fea381b2d10 con 0x5603e9b2e980 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea4a063b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fea381b2640 con 0x5603e9b2e060 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea4a063b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fea381af860 con 0x5603e9b329d0 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea4a063b80 10 client.0 _ll_register_callbacks cb 0x5603e9a27c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea3cc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b3ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea3cc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b3ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42686/0 (socket says 172.21.15.27:42686) 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea3cc00700 1 -- 172.21.15.27:0/2243257051 learned_addr learned my addr 172.21.15.27:0/2243257051 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea3d600700 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b3af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea2fe00700 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea3cc00700 1 -- 172.21.15.27:0/2243257051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 msgr2=0x5603e9b3af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea3cc00700 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b3af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea3cc00700 1 -- 172.21.15.27:0/2243257051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 msgr2=0x5603e9b404e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea3cc00700 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b404e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.061+0000 7fea3cc00700 1 -- 172.21.15.27:0/2243257051 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603e9b3b750 con 0x5603e9b2e060 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.062+0000 7fea3cc00700 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b3ffa0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fea280a93a0 tx=0x7fea280a9a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2116b59c2f7ff855 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.062+0000 7fea26a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.062+0000 7fea26a00700 1 -- 172.21.15.27:0/2243257051 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28020070 con 0x5603e9b2e060 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.062+0000 7fea26a00700 1 -- 172.21.15.27:0/2243257051 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7fea280a5a00 con 0x5603e9b2e060 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.062+0000 7fea4a063b80 1 -- 172.21.15.27:0/2243257051 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea30028070 con 0x5603e9b2e060 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.062+0000 7fea26a00700 1 -- 172.21.15.27:0/2243257051 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fea280a52d0 con 0x5603e9b2e060 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.062+0000 7fea4a063b80 10 client.26714 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.062+0000 7fea4a063b80 1 -- 172.21.15.27:0/2243257051 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5603e9bb2c70 con 0x5603e9b2e060 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.062+0000 7fea26a00700 1 -- 172.21.15.27:0/2243257051 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea30028070 con 0x5603e9b2e060 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.063+0000 7fea26a00700 1 -- 172.21.15.27:0/2243257051 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fea280ca020 con 0x5603e9b2e060 2024-05-10T11:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.063+0000 7fea4a063b80 10 client.26714 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.063+0000 7fea4a063b80 10 client.26714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.063+0000 7fea4a063b80 20 client.26714 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.063+0000 7fea4a063b80 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5603e9bb31c0 0x5603e9bb55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.063+0000 7fea4a063b80 4 client.26714 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.063+0000 7fea4a063b80 1 -- 172.21.15.27:0/2243257051 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5603e9a8e400 con 0x5603e9bb31c0 2024-05-10T11:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.063+0000 7fea2fe00700 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5603e9bb31c0 0x5603e9bb55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea2fe00700 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5603e9bb31c0 0x5603e9bb55e0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c181e5ac6816beb2 server_cookie=b4e447fb1ad5064b in_seq=0 out_seq=0 2024-05-10T11:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea26a00700 10 client.26714 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea26a00700 1 -- 172.21.15.27:0/2243257051 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5603e9a8e400 con 0x5603e9bb31c0 2024-05-10T11:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea26a00700 10 client.26714 handle_command_reply: tid=0 2024-05-10T11:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea4a063b80 1 client.26714 shutdown 2024-05-10T11:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea4a063b80 1 -- 172.21.15.27:0/2243257051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 msgr2=0x5603e9b3ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea4a063b80 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b3ffa0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fea280a93a0 tx=0x7fea280a9a30 comp rx=0 tx=0).stop 2024-05-10T11:23:22.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea4a063b80 1 -- 172.21.15.27:0/2243257051 shutdown_connections 2024-05-10T11:23:22.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea4a063b80 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5603e9bb31c0 0x5603e9bb55e0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea4a063b80 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603e9b329d0 0x5603e9b3af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea26a00700 0 client.26714 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:22.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea4a063b80 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603e9b2e060 0x5603e9b3ffa0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea4a063b80 1 --2- 172.21.15.27:0/2243257051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5603e9b2e980 0x5603e9b404e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea4a063b80 1 -- 172.21.15.27:0/2243257051 >> 172.21.15.27:0/2243257051 conn(0x5603e9a5cb60 msgr2=0x5603e9b35b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:22.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea4a063b80 1 -- 172.21.15.27:0/2243257051 shutdown_connections 2024-05-10T11:23:22.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.064+0000 7fea4a063b80 1 -- 172.21.15.27:0/2243257051 wait complete. 2024-05-10T11:23:22.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.065+0000 7fea4a063b80 20 client.26714 trim_cache size 0 max 16384 2024-05-10T11:23:22.070 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:23:22.070 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:23:22.070 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:23:22.070 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:23:22.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.065+0000 7fea3f600700 1 -- 172.21.15.27:0/3528625680 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fea1c0b2a70 msgr2=0x7fea1c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.065+0000 7fea3f600700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fea1c0b2a70 0x7fea1c0b4f30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fea30009e00 tx=0x7fea300093f0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.065+0000 7fea3f600700 1 -- 172.21.15.27:0/3528625680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea3806b180 msgr2=0x7fea3819bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.065+0000 7fea3f600700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea3806b180 0x7fea3819bd80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fea2800d950 tx=0x7fea2800dc60 comp rx=0 tx=0).stop 2024-05-10T11:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.066+0000 7fea3f600700 1 -- 172.21.15.27:0/3528625680 shutdown_connections 2024-05-10T11:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.066+0000 7fea3f600700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fea1c0b2a70 0x7fea1c0b4f30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.066+0000 7fea3f600700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea3806b180 0x7fea3819bd80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.066+0000 7fea3f600700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea38109510 0x7fea381a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.066+0000 7fea3f600700 1 --2- 172.21.15.27:0/3528625680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea3806bb40 0x7fea3819c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.066+0000 7fea3f600700 1 -- 172.21.15.27:0/3528625680 >> 172.21.15.27:0/3528625680 conn(0x7fea38079350 msgr2=0x7fea3810e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.066+0000 7fea3f600700 1 -- 172.21.15.27:0/3528625680 shutdown_connections 2024-05-10T11:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:22.066+0000 7fea3f600700 1 -- 172.21.15.27:0/3528625680 wait complete. 2024-05-10T11:23:22.718 INFO:tasks.workunit.client.0.smithi027.stdout: 1 607772 6.52 MB/sec execute 367 sec latency 41.064 ms 2024-05-10T11:23:22.974 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:23:22.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:23:23.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:22 smithi027 ceph-mon[30054]: pgmap v315: 129 pgs: 129 active+clean; 453 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 131 op/s 2024-05-10T11:23:23.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:22 smithi076 ceph-mon[34152]: pgmap v315: 129 pgs: 129 active+clean; 453 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 131 op/s 2024-05-10T11:23:23.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:22 smithi189 ceph-mon[34842]: pgmap v315: 129 pgs: 129 active+clean; 453 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 131 op/s 2024-05-10T11:23:23.507 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:23:23.717 INFO:tasks.workunit.client.0.smithi027.stdout: 1 609036 6.51 MB/sec execute 368 sec latency 48.972 ms 2024-05-10T11:23:24.718 INFO:tasks.workunit.client.0.smithi027.stdout: 1 610988 6.53 MB/sec execute 369 sec latency 20.136 ms 2024-05-10T11:23:25.091 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:24 smithi027 ceph-mon[30054]: pgmap v316: 129 pgs: 129 active+clean; 453 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 131 op/s 2024-05-10T11:23:25.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:24 smithi076 ceph-mon[34152]: pgmap v316: 129 pgs: 129 active+clean; 453 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 131 op/s 2024-05-10T11:23:25.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:24 smithi189 ceph-mon[34842]: pgmap v316: 129 pgs: 129 active+clean; 453 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 131 op/s 2024-05-10T11:23:25.718 INFO:tasks.workunit.client.0.smithi027.stdout: 1 611616 6.52 MB/sec execute 370 sec latency 36.482 ms 2024-05-10T11:23:25.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.966+0000 7f617b200700 1 -- 172.21.15.27:0/242506151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61741033d0 msgr2=0x7f6174103850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.966+0000 7f617b200700 1 --2- 172.21.15.27:0/242506151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61741033d0 0x7f6174103850 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6164009990 tx=0x7f6164009ca0 comp rx=0 tx=0).stop 2024-05-10T11:23:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.966+0000 7f617b200700 1 -- 172.21.15.27:0/242506151 shutdown_connections 2024-05-10T11:23:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.966+0000 7f617b200700 1 --2- 172.21.15.27:0/242506151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6174103d90 0x7f61741102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.966+0000 7f617b200700 1 --2- 172.21.15.27:0/242506151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6174109460 0x7f6174109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.966+0000 7f617b200700 1 --2- 172.21.15.27:0/242506151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61741033d0 0x7f6174103850 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.966+0000 7f617b200700 1 -- 172.21.15.27:0/242506151 >> 172.21.15.27:0/242506151 conn(0x7f61740fef20 msgr2=0x7f6174101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.967+0000 7f617b200700 1 -- 172.21.15.27:0/242506151 shutdown_connections 2024-05-10T11:23:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.967+0000 7f617b200700 1 -- 172.21.15.27:0/242506151 wait complete. 2024-05-10T11:23:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.968+0000 7f617b200700 1 Processor -- start 2024-05-10T11:23:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.969+0000 7f617b200700 1 -- start start 2024-05-10T11:23:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.969+0000 7f617b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61741033d0 0x7f61740757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.969+0000 7f617b200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174103d90 0x7f6174075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.969+0000 7f617b200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6174109460 0x7f6174076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f6179400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174103d90 0x7f6174075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f6179e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61741033d0 0x7f61740757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f6179400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174103d90 0x7f6174075ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57750/0 (socket says 172.21.15.27:57750) 2024-05-10T11:23:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f6179e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61741033d0 0x7f61740757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54164/0 (socket says 172.21.15.27:54164) 2024-05-10T11:23:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f6179400700 1 -- 172.21.15.27:0/3398620022 learned_addr learned my addr 172.21.15.27:0/3398620022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f61741135c0 con 0x7f6174109460 2024-05-10T11:23:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6174113440 con 0x7f6174103d90 2024-05-10T11:23:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6174113740 con 0x7f61741033d0 2024-05-10T11:23:25.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f6179400700 1 -- 172.21.15.27:0/3398620022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61741033d0 msgr2=0x7f61740757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:25.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f6179400700 1 --2- 172.21.15.27:0/3398620022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61741033d0 0x7f61740757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:25.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f6179400700 1 -- 172.21.15.27:0/3398620022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6174109460 msgr2=0x7f6174076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:25.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f6179400700 1 --2- 172.21.15.27:0/3398620022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6174109460 0x7f6174076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:25.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.970+0000 7f6179400700 1 -- 172.21.15.27:0/3398620022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61741a8060 con 0x7f6174103d90 2024-05-10T11:23:25.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.971+0000 7f6179400700 1 --2- 172.21.15.27:0/3398620022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174103d90 0x7f6174075ce0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6164000c00 tx=0x7f61640057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.971+0000 7f616aa00700 1 -- 172.21.15.27:0/3398620022 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6164027070 con 0x7f6174103d90 2024-05-10T11:23:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.971+0000 7f616aa00700 1 -- 172.21.15.27:0/3398620022 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6164022470 con 0x7f6174103d90 2024-05-10T11:23:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.971+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61741a82f0 con 0x7f6174103d90 2024-05-10T11:23:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.972+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61741a87a0 con 0x7f6174103d90 2024-05-10T11:23:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.972+0000 7f616aa00700 1 -- 172.21.15.27:0/3398620022 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6164017650 con 0x7f6174103d90 2024-05-10T11:23:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.973+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6158005330 con 0x7f6174103d90 2024-05-10T11:23:25.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.975+0000 7f616aa00700 1 -- 172.21.15.27:0/3398620022 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f616402a070 con 0x7f6174103d90 2024-05-10T11:23:25.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.977+0000 7f616aa00700 1 --2- 172.21.15.27:0/3398620022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f61600b2a70 0x7f61600b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:25.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.977+0000 7f616aa00700 1 -- 172.21.15.27:0/3398620022 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f61640e36a0 con 0x7f6174103d90 2024-05-10T11:23:25.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.978+0000 7f6179e00700 1 --2- 172.21.15.27:0/3398620022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f61600b2a70 0x7f61600b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:25.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.978+0000 7f6179e00700 1 --2- 172.21.15.27:0/3398620022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f61600b2a70 0x7f61600b4f30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f616c00f540 tx=0x7f616c00b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:25.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:25.981+0000 7f616aa00700 1 -- 172.21.15.27:0/3398620022 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6164013070 con 0x7f6174103d90 2024-05-10T11:23:26.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.317+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61580058f0 con 0x7f6174103d90 2024-05-10T11:23:26.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.318+0000 7f616aa00700 1 -- 172.21.15.27:0/3398620022 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11936 (secure 0 0 0) 0x7f61640aac80 con 0x7f6174103d90 2024-05-10T11:23:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:23:26.320 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:23:14.174773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24484,24527,24472,24488,24467]},"id":1}]} 2024-05-10T11:23:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.322+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f61600b2a70 msgr2=0x7f61600b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.322+0000 7f617b200700 1 --2- 172.21.15.27:0/3398620022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f61600b2a70 0x7f61600b4f30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f616c00f540 tx=0x7f616c00b040 comp rx=0 tx=0).stop 2024-05-10T11:23:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.322+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174103d90 msgr2=0x7f6174075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.322+0000 7f617b200700 1 --2- 172.21.15.27:0/3398620022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174103d90 0x7f6174075ce0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6164000c00 tx=0x7f61640057c0 comp rx=0 tx=0).stop 2024-05-10T11:23:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.322+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 shutdown_connections 2024-05-10T11:23:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.322+0000 7f617b200700 1 --2- 172.21.15.27:0/3398620022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f61600b2a70 0x7f61600b4f30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.322+0000 7f617b200700 1 --2- 172.21.15.27:0/3398620022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61741033d0 0x7f61740757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.322+0000 7f617b200700 1 --2- 172.21.15.27:0/3398620022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6174109460 0x7f6174076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.322+0000 7f617b200700 1 --2- 172.21.15.27:0/3398620022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174103d90 0x7f6174075ce0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.322+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 >> 172.21.15.27:0/3398620022 conn(0x7f61740fef20 msgr2=0x7f61740ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.322+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 shutdown_connections 2024-05-10T11:23:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:26.322+0000 7f617b200700 1 -- 172.21.15.27:0/3398620022 wait complete. 2024-05-10T11:23:26.324 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-05-10T11:23:26.718 INFO:tasks.workunit.client.0.smithi027.stdout: 1 612755 6.51 MB/sec execute 371 sec latency 31.348 ms 2024-05-10T11:23:27.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:26 smithi027 ceph-mon[30054]: pgmap v317: 129 pgs: 129 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 203 op/s 2024-05-10T11:23:27.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:26 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3398620022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:23:27.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:23:27.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:26 smithi076 ceph-mon[34152]: pgmap v317: 129 pgs: 129 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 203 op/s 2024-05-10T11:23:27.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:26 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3398620022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:23:27.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:26 smithi189 ceph-mon[34842]: pgmap v317: 129 pgs: 129 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 203 op/s 2024-05-10T11:23:27.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:26 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3398620022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:23:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:23:27.718 INFO:tasks.workunit.client.0.smithi027.stdout: 1 614574 6.53 MB/sec execute 372 sec latency 23.697 ms 2024-05-10T11:23:28.718 INFO:tasks.workunit.client.0.smithi027.stdout: 1 615033 6.52 MB/sec execute 373 sec latency 66.700 ms 2024-05-10T11:23:29.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:28 smithi027 ceph-mon[30054]: pgmap v318: 129 pgs: 129 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 135 op/s 2024-05-10T11:23:29.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:28 smithi076 ceph-mon[34152]: pgmap v318: 129 pgs: 129 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 135 op/s 2024-05-10T11:23:29.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:28 smithi189 ceph-mon[34842]: pgmap v318: 129 pgs: 129 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 135 op/s 2024-05-10T11:23:29.718 INFO:tasks.workunit.client.0.smithi027.stdout: 1 616141 6.51 MB/sec execute 374 sec latency 35.531 ms 2024-05-10T11:23:30.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.038+0000 7f6689400700 1 -- 172.21.15.27:0/165332728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6684103fd0 msgr2=0x7f6684108020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:30.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.038+0000 7f6689400700 1 --2- 172.21.15.27:0/165332728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6684103fd0 0x7f6684108020 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f666c00b0a0 tx=0x7f666c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.038+0000 7f6689400700 1 -- 172.21.15.27:0/165332728 shutdown_connections 2024-05-10T11:23:30.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.038+0000 7f6689400700 1 --2- 172.21.15.27:0/165332728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6684103620 0x7f6684103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.038+0000 7f6689400700 1 --2- 172.21.15.27:0/165332728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6684108700 0x7f66841102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.038+0000 7f6689400700 1 --2- 172.21.15.27:0/165332728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6684103fd0 0x7f6684108020 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.038+0000 7f6689400700 1 -- 172.21.15.27:0/165332728 >> 172.21.15.27:0/165332728 conn(0x7f66840feeb0 msgr2=0x7f66841012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:30.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.039+0000 7f6689400700 1 -- 172.21.15.27:0/165332728 shutdown_connections 2024-05-10T11:23:30.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.039+0000 7f6689400700 1 -- 172.21.15.27:0/165332728 wait complete. 2024-05-10T11:23:30.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.040+0000 7f6689400700 1 Processor -- start 2024-05-10T11:23:30.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.040+0000 7f6689400700 1 -- start start 2024-05-10T11:23:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6689400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6684103620 0x7f668419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6689400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6684103fd0 0x7f668419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6689400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6684108700 0x7f66841a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6689400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6684076220 con 0x7f6684103fd0 2024-05-10T11:23:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6689400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f66840760a0 con 0x7f6684108700 2024-05-10T11:23:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6689400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66840763a0 con 0x7f6684103620 2024-05-10T11:23:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6688a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6684108700 0x7f66841a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6683e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6684103620 0x7f668419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f667b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6684103fd0 0x7f668419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6683e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6684103620 0x7f668419bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54184/0 (socket says 172.21.15.27:54184) 2024-05-10T11:23:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6683e00700 1 -- 172.21.15.27:0/1452110969 learned_addr learned my addr 172.21.15.27:0/1452110969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6688a00700 1 -- 172.21.15.27:0/1452110969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6684103620 msgr2=0x7f668419bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6688a00700 1 --2- 172.21.15.27:0/1452110969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6684103620 0x7f668419bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6688a00700 1 -- 172.21.15.27:0/1452110969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6684103fd0 msgr2=0x7f668419c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6688a00700 1 --2- 172.21.15.27:0/1452110969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6684103fd0 0x7f668419c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.041+0000 7f6688a00700 1 -- 172.21.15.27:0/1452110969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66841a0d20 con 0x7f6684108700 2024-05-10T11:23:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.042+0000 7f6688a00700 1 --2- 172.21.15.27:0/1452110969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6684108700 0x7f66841a05e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f667c00e970 tx=0x7f667c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:30.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.042+0000 7f6681600700 1 -- 172.21.15.27:0/1452110969 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c004220 con 0x7f6684108700 2024-05-10T11:23:30.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.042+0000 7f6681600700 1 -- 172.21.15.27:0/1452110969 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f667c0043c0 con 0x7f6684108700 2024-05-10T11:23:30.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.042+0000 7f6681600700 1 -- 172.21.15.27:0/1452110969 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c010700 con 0x7f6684108700 2024-05-10T11:23:30.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.042+0000 7f6689400700 1 -- 172.21.15.27:0/1452110969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66841a1010 con 0x7f6684108700 2024-05-10T11:23:30.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.042+0000 7f6689400700 1 -- 172.21.15.27:0/1452110969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66841a8890 con 0x7f6684108700 2024-05-10T11:23:30.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.046+0000 7f6681600700 1 -- 172.21.15.27:0/1452110969 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f667c004750 con 0x7f6684108700 2024-05-10T11:23:30.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.047+0000 7f669546bb80 1 Processor -- start 2024-05-10T11:23:30.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.047+0000 7f669546bb80 1 -- start start 2024-05-10T11:23:30.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.047+0000 7f6681600700 1 --2- 172.21.15.27:0/1452110969 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f66680b2a70 0x7f66680b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f6681600700 1 -- 172.21.15.27:0/1452110969 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f667c0e27b0 con 0x7f6684108700 2024-05-10T11:23:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f669546bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 0x55ca11ff8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f669546bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff9300 0x55ca11ff2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f669546bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff3600 0x55ca11ff3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f669546bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ca11f45a20 con 0x55ca11ff89e0 2024-05-10T11:23:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f669546bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ca11ee3760 con 0x55ca11ff3600 2024-05-10T11:23:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f669546bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ca11f419e0 con 0x55ca11ff9300 2024-05-10T11:23:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f667b400700 1 --2- 172.21.15.27:0/1452110969 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f66680b2a70 0x7f66680b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f6683e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 0x55ca11ff8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f667b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff9300 0x55ca11ff2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f6683e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 0x55ca11ff8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56416/0 (socket says 172.21.15.27:56416) 2024-05-10T11:23:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f6683e00700 1 -- 172.21.15.27:0/3197640713 learned_addr learned my addr 172.21.15.27:0/3197640713 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:30.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.048+0000 7f6688a00700 1 --2- 172.21.15.27:0/3197640713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff3600 0x55ca11ff3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.049+0000 7f6683e00700 1 -- 172.21.15.27:0/3197640713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff9300 msgr2=0x55ca11ff2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:30.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.049+0000 7f6683e00700 1 --2- 172.21.15.27:0/3197640713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff9300 0x55ca11ff2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.049+0000 7f6683e00700 1 -- 172.21.15.27:0/3197640713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff3600 msgr2=0x55ca11ff3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:30.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.049+0000 7f6683e00700 1 --2- 172.21.15.27:0/3197640713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff3600 0x55ca11ff3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.049+0000 7f6683e00700 1 -- 172.21.15.27:0/3197640713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca11e928d0 con 0x55ca11ff89e0 2024-05-10T11:23:30.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.049+0000 7f667b400700 1 --2- 172.21.15.27:0/1452110969 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f66680b2a70 0x7f66680b4f30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f666c0062a0 tx=0x7f666c009380 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:30.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.049+0000 7f6683e00700 1 --2- 172.21.15.27:0/3197640713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 0x55ca11ff8dc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6674009e50 tx=0x7f6674011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81feb0e7e827ea4d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:30.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.050+0000 7f6679600700 1 -- 172.21.15.27:0/3197640713 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674004a30 con 0x55ca11ff89e0 2024-05-10T11:23:30.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.050+0000 7f6679600700 1 -- 172.21.15.27:0/3197640713 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66740054d0 con 0x55ca11ff89e0 2024-05-10T11:23:30.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.050+0000 7f6679600700 1 -- 172.21.15.27:0/3197640713 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674019870 con 0x55ca11ff89e0 2024-05-10T11:23:30.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.050+0000 7f669546bb80 1 -- 172.21.15.27:0/3197640713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 msgr2=0x55ca11ff8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:30.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.050+0000 7f669546bb80 1 --2- 172.21.15.27:0/3197640713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 0x55ca11ff8dc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6674009e50 tx=0x7f6674011720 comp rx=0 tx=0).stop 2024-05-10T11:23:30.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.051+0000 7f669546bb80 1 -- 172.21.15.27:0/3197640713 shutdown_connections 2024-05-10T11:23:30.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.051+0000 7f669546bb80 1 --2- 172.21.15.27:0/3197640713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff3600 0x55ca11ff3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.051+0000 7f669546bb80 1 --2- 172.21.15.27:0/3197640713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 0x55ca11ff8dc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.051+0000 7f669546bb80 1 --2- 172.21.15.27:0/3197640713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff9300 0x55ca11ff2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.051+0000 7f669546bb80 1 -- 172.21.15.27:0/3197640713 >> 172.21.15.27:0/3197640713 conn(0x55ca11f21b60 msgr2=0x55ca11fed9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:30.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.051+0000 7f669546bb80 1 -- 172.21.15.27:0/3197640713 shutdown_connections 2024-05-10T11:23:30.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.051+0000 7f669546bb80 1 -- 172.21.15.27:0/3197640713 wait complete. 2024-05-10T11:23:30.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.051+0000 7f669546bb80 1 -- start start 2024-05-10T11:23:30.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.052+0000 7f669546bb80 1 Processor -- start 2024-05-10T11:23:30.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.052+0000 7f669546bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff3600 0x55ca11ffee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.052+0000 7f669546bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 0x55ca11fff380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.052+0000 7f669546bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff9300 0x55ca11ffc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.052+0000 7f669546bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ca11fffdb0 con 0x55ca11ff9300 2024-05-10T11:23:30.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.052+0000 7f669546bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ca11eabb20 con 0x55ca11ff89e0 2024-05-10T11:23:30.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.052+0000 7f669546bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ca11ffff30 con 0x55ca11ff3600 2024-05-10T11:23:30.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.052+0000 7f669546bb80 10 client.0 _ll_register_callbacks cb 0x55ca11eecc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:30.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f6683e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff3600 0x55ca11ffee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f667b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 0x55ca11fff380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f6683e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff3600 0x55ca11ffee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54206/0 (socket says 172.21.15.27:54206) 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f667b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 0x55ca11fff380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57784/0 (socket says 172.21.15.27:57784) 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f6683e00700 1 -- 172.21.15.27:0/403511082 learned_addr learned my addr 172.21.15.27:0/403511082 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f6688a00700 1 --2- 172.21.15.27:0/403511082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff9300 0x55ca11ffc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f667b400700 1 -- 172.21.15.27:0/403511082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff3600 msgr2=0x55ca11ffee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f667b400700 1 --2- 172.21.15.27:0/403511082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff3600 0x55ca11ffee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f667b400700 1 -- 172.21.15.27:0/403511082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff9300 msgr2=0x55ca11ffc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f667b400700 1 --2- 172.21.15.27:0/403511082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff9300 0x55ca11ffc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f667b400700 1 -- 172.21.15.27:0/403511082 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca11f0b420 con 0x55ca11ff89e0 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f667b400700 1 --2- 172.21.15.27:0/403511082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 0x55ca11fff380 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f666c00b070 tx=0x7f666c00b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=186997e1973ea660 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.053+0000 7f6672000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.054+0000 7f6672000700 1 -- 172.21.15.27:0/403511082 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c031060 con 0x55ca11ff89e0 2024-05-10T11:23:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.054+0000 7f669546bb80 1 -- 172.21.15.27:0/403511082 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66740054d0 con 0x55ca11ff89e0 2024-05-10T11:23:30.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.054+0000 7f6672000700 1 -- 172.21.15.27:0/403511082 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f666c007ad0 con 0x55ca11ff89e0 2024-05-10T11:23:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.054+0000 7f6672000700 1 -- 172.21.15.27:0/403511082 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f666c038050 con 0x55ca11ff89e0 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.054+0000 7f669546bb80 10 client.26732 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.054+0000 7f669546bb80 1 -- 172.21.15.27:0/403511082 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ca1200a460 con 0x55ca11ff89e0 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.054+0000 7f6672000700 1 -- 172.21.15.27:0/403511082 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66740054d0 con 0x55ca11ff89e0 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.055+0000 7f6672000700 1 -- 172.21.15.27:0/403511082 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f666c011070 con 0x55ca11ff89e0 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.055+0000 7f669546bb80 10 client.26732 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.055+0000 7f669546bb80 10 client.26732 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.055+0000 7f669546bb80 20 client.26732 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.055+0000 7f669546bb80 1 --2- 172.21.15.27:0/403511082 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ca1200c9f0 0x55ca1200ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.055+0000 7f669546bb80 4 client.26732 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.055+0000 7f669546bb80 1 -- 172.21.15.27:0/403511082 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca11fc00a0 con 0x55ca1200c9f0 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.055+0000 7f6683e00700 1 --2- 172.21.15.27:0/403511082 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ca1200c9f0 0x55ca1200ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.056+0000 7f6683e00700 1 --2- 172.21.15.27:0/403511082 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ca1200c9f0 0x55ca1200ede0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fb6286cf157afa7a server_cookie=de8a9f5a43a0d46c in_seq=0 out_seq=0 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.056+0000 7f6672000700 10 client.26732 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.056+0000 7f6672000700 1 -- 172.21.15.27:0/403511082 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ca11fc00a0 con 0x55ca1200c9f0 2024-05-10T11:23:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.056+0000 7f6672000700 10 client.26732 handle_command_reply: tid=0 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f669546bb80 1 client.26732 shutdown 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f669546bb80 1 -- 172.21.15.27:0/403511082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 msgr2=0x55ca11fff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f669546bb80 1 --2- 172.21.15.27:0/403511082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 0x55ca11fff380 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f666c00b070 tx=0x7f666c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f669546bb80 1 -- 172.21.15.27:0/403511082 shutdown_connections 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f669546bb80 1 --2- 172.21.15.27:0/403511082 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ca1200c9f0 0x55ca1200ede0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f669546bb80 1 --2- 172.21.15.27:0/403511082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff3600 0x55ca11ffee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f669546bb80 1 --2- 172.21.15.27:0/403511082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff9300 0x55ca11ffc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f669546bb80 1 --2- 172.21.15.27:0/403511082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 0x55ca11fff380 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f6672000700 0 client.26732 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f669546bb80 1 -- 172.21.15.27:0/403511082 >> 172.21.15.27:0/403511082 conn(0x55ca11f21b60 msgr2=0x55ca11fe8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f669546bb80 1 -- 172.21.15.27:0/403511082 shutdown_connections 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f669546bb80 1 -- 172.21.15.27:0/403511082 wait complete. 2024-05-10T11:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.057+0000 7f669546bb80 20 client.26732 trim_cache size 0 max 16384 2024-05-10T11:23:30.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.077+0000 7f669546bb80 1 Processor -- start 2024-05-10T11:23:30.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.077+0000 7f669546bb80 1 -- start start 2024-05-10T11:23:30.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.077+0000 7f669546bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff3600 0x55ca11ff2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.077+0000 7f669546bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 0x55ca11ffccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.077+0000 7f669546bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff9300 0x55ca11ffaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.078+0000 7f669546bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f66841b17a0 con 0x55ca11ff89e0 2024-05-10T11:23:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.078+0000 7f669546bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f66841b32e0 con 0x55ca11ff3600 2024-05-10T11:23:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.078+0000 7f669546bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66841b1e70 con 0x55ca11ff9300 2024-05-10T11:23:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.078+0000 7f6683e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff3600 0x55ca11ff2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.078+0000 7f667b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 0x55ca11ffccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.078+0000 7f6683e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff3600 0x55ca11ff2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57796/0 (socket says 172.21.15.27:57796) 2024-05-10T11:23:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.078+0000 7f6683e00700 1 -- 172.21.15.27:0/3891708373 learned_addr learned my addr 172.21.15.27:0/3891708373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.078+0000 7f6688a00700 1 --2- 172.21.15.27:0/3891708373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff9300 0x55ca11ffaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.078+0000 7f6688a00700 1 -- 172.21.15.27:0/3891708373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff3600 msgr2=0x55ca11ff2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.079+0000 7f6688a00700 1 --2- 172.21.15.27:0/3891708373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff3600 0x55ca11ff2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.079+0000 7f6688a00700 1 -- 172.21.15.27:0/3891708373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 msgr2=0x55ca11ffccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.079+0000 7f6688a00700 1 --2- 172.21.15.27:0/3891708373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 0x55ca11ffccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.079+0000 7f6688a00700 1 -- 172.21.15.27:0/3891708373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca11fed760 con 0x55ca11ff9300 2024-05-10T11:23:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.079+0000 7f667b400700 1 --2- 172.21.15.27:0/3891708373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 0x55ca11ffccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.079+0000 7f6683e00700 1 --2- 172.21.15.27:0/3891708373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff3600 0x55ca11ff2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.079+0000 7f6688a00700 1 --2- 172.21.15.27:0/3891708373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff9300 0x55ca11ffaaf0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f667c0dfa70 tx=0x7f667c005100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94de31fbbbd0cb17 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.079+0000 7f6678c00700 1 -- 172.21.15.27:0/3891708373 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c01c030 con 0x55ca11ff9300 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.079+0000 7f6678c00700 1 -- 172.21.15.27:0/3891708373 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f667c0ad9b0 con 0x55ca11ff9300 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.079+0000 7f6678c00700 1 -- 172.21.15.27:0/3891708373 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c0a9860 con 0x55ca11ff9300 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.080+0000 7f669546bb80 1 -- 172.21.15.27:0/3891708373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff9300 msgr2=0x55ca11ffaaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.080+0000 7f669546bb80 1 --2- 172.21.15.27:0/3891708373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff9300 0x55ca11ffaaf0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f667c0dfa70 tx=0x7f667c005100 comp rx=0 tx=0).stop 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.081+0000 7f669546bb80 1 -- 172.21.15.27:0/3891708373 shutdown_connections 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.081+0000 7f669546bb80 1 --2- 172.21.15.27:0/3891708373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff9300 0x55ca11ffaaf0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.081+0000 7f669546bb80 1 --2- 172.21.15.27:0/3891708373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff3600 0x55ca11ff2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.081+0000 7f669546bb80 1 --2- 172.21.15.27:0/3891708373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff89e0 0x55ca11ffccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.081+0000 7f669546bb80 1 -- 172.21.15.27:0/3891708373 >> 172.21.15.27:0/3891708373 conn(0x55ca11f21b60 msgr2=0x55ca11ffee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.081+0000 7f669546bb80 1 -- 172.21.15.27:0/3891708373 shutdown_connections 2024-05-10T11:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.081+0000 7f669546bb80 1 -- 172.21.15.27:0/3891708373 wait complete. 2024-05-10T11:23:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.081+0000 7f669546bb80 1 -- start start 2024-05-10T11:23:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.082+0000 7f669546bb80 1 Processor -- start 2024-05-10T11:23:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.082+0000 7f669546bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff3600 0x55ca120057d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.082+0000 7f669546bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 0x55ca12013d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.082+0000 7f669546bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff9300 0x55ca120142c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.082+0000 7f669546bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f66841b2c10 con 0x55ca11ff9300 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.082+0000 7f669546bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f66841b2540 con 0x55ca11ff89e0 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.082+0000 7f669546bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66841af760 con 0x55ca11ff3600 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.083+0000 7f669546bb80 10 client.0 _ll_register_callbacks cb 0x55ca11eecc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.083+0000 7f667b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 0x55ca12013d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.083+0000 7f667b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 0x55ca12013d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57800/0 (socket says 172.21.15.27:57800) 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.083+0000 7f667b400700 1 -- 172.21.15.27:0/3823672581 learned_addr learned my addr 172.21.15.27:0/3823672581 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.083+0000 7f6683e00700 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff3600 0x55ca120057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.083+0000 7f667b400700 1 -- 172.21.15.27:0/3823672581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff3600 msgr2=0x55ca120057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.083+0000 7f667b400700 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff3600 0x55ca120057d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.083+0000 7f667b400700 1 -- 172.21.15.27:0/3823672581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff9300 msgr2=0x55ca120142c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.083+0000 7f6688a00700 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff9300 0x55ca120142c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.083+0000 7f667b400700 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff9300 0x55ca120142c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.083+0000 7f667b400700 1 -- 172.21.15.27:0/3823672581 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca12014b00 con 0x55ca11ff89e0 2024-05-10T11:23:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.083+0000 7f6688a00700 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff9300 0x55ca120142c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.084+0000 7f667b400700 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 0x55ca12013d00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f666c004fd0 tx=0x7f666c006160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ea091aaa4f50de9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.084+0000 7f6672000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.084+0000 7f6672000700 1 -- 172.21.15.27:0/3823672581 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c041060 con 0x55ca11ff89e0 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.084+0000 7f669546bb80 1 -- 172.21.15.27:0/3823672581 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f667c0ad9b0 con 0x55ca11ff89e0 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.084+0000 7f6672000700 1 -- 172.21.15.27:0/3823672581 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f666c007ce0 con 0x55ca11ff89e0 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.084+0000 7f6672000700 1 -- 172.21.15.27:0/3823672581 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f666c031070 con 0x55ca11ff89e0 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.084+0000 7f669546bb80 10 client.26738 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.084+0000 7f669546bb80 1 -- 172.21.15.27:0/3823672581 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ca1207fb70 con 0x55ca11ff89e0 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.085+0000 7f6672000700 1 -- 172.21.15.27:0/3823672581 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f667c0ad9b0 con 0x55ca11ff89e0 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.085+0000 7f6672000700 1 -- 172.21.15.27:0/3823672581 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f666c011040 con 0x55ca11ff89e0 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.085+0000 7f669546bb80 10 client.26738 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.085+0000 7f669546bb80 10 client.26738 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.085+0000 7f669546bb80 20 client.26738 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.085+0000 7f669546bb80 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ca120801c0 0x55ca120825e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.085+0000 7f669546bb80 4 client.26738 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:23:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.085+0000 7f669546bb80 1 -- 172.21.15.27:0/3823672581 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ca11f53400 con 0x55ca120801c0 2024-05-10T11:23:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.085+0000 7f6683e00700 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ca120801c0 0x55ca120825e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.086+0000 7f6683e00700 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ca120801c0 0x55ca120825e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=25a21827d337dbd1 server_cookie=db31acf9c7e3e2f in_seq=0 out_seq=0 2024-05-10T11:23:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.086+0000 7f6672000700 10 client.26738 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:23:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.086+0000 7f6672000700 1 -- 172.21.15.27:0/3823672581 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ca11f53400 con 0x55ca120801c0 2024-05-10T11:23:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.086+0000 7f6672000700 10 client.26738 handle_command_reply: tid=0 2024-05-10T11:23:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.086+0000 7f669546bb80 1 client.26738 shutdown 2024-05-10T11:23:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.086+0000 7f669546bb80 1 -- 172.21.15.27:0/3823672581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 msgr2=0x55ca12013d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.086+0000 7f669546bb80 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 0x55ca12013d00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f666c004fd0 tx=0x7f666c006160 comp rx=0 tx=0).stop 2024-05-10T11:23:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.087+0000 7f669546bb80 1 -- 172.21.15.27:0/3823672581 shutdown_connections 2024-05-10T11:23:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.087+0000 7f669546bb80 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ca120801c0 0x55ca120825e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.087+0000 7f669546bb80 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca11ff3600 0x55ca120057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.087+0000 7f669546bb80 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca11ff9300 0x55ca120142c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.087+0000 7f6672000700 0 client.26738 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:23:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.087+0000 7f669546bb80 1 --2- 172.21.15.27:0/3823672581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca11ff89e0 0x55ca12013d00 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.087+0000 7f669546bb80 1 -- 172.21.15.27:0/3823672581 >> 172.21.15.27:0/3823672581 conn(0x55ca11f21b60 msgr2=0x55ca11ffaab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.087+0000 7f669546bb80 1 -- 172.21.15.27:0/3823672581 shutdown_connections 2024-05-10T11:23:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.087+0000 7f669546bb80 1 -- 172.21.15.27:0/3823672581 wait complete. 2024-05-10T11:23:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.087+0000 7f669546bb80 20 client.26738 trim_cache size 0 max 16384 2024-05-10T11:23:30.089 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:23:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.088+0000 7f6689400700 1 -- 172.21.15.27:0/1452110969 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f66680b2a70 msgr2=0x7f66680b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.088+0000 7f6689400700 1 --2- 172.21.15.27:0/1452110969 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f66680b2a70 0x7f66680b4f30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f666c0062a0 tx=0x7f666c009380 comp rx=0 tx=0).stop 2024-05-10T11:23:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.088+0000 7f6689400700 1 -- 172.21.15.27:0/1452110969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6684108700 msgr2=0x7f66841a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.088+0000 7f6689400700 1 --2- 172.21.15.27:0/1452110969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6684108700 0x7f66841a05e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f667c00e970 tx=0x7f667c00ec80 comp rx=0 tx=0).stop 2024-05-10T11:23:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.088+0000 7f6689400700 1 -- 172.21.15.27:0/1452110969 shutdown_connections 2024-05-10T11:23:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.088+0000 7f6689400700 1 --2- 172.21.15.27:0/1452110969 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f66680b2a70 0x7f66680b4f30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.088+0000 7f6689400700 1 --2- 172.21.15.27:0/1452110969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6684103620 0x7f668419bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.088+0000 7f6689400700 1 --2- 172.21.15.27:0/1452110969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6684108700 0x7f66841a05e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.088+0000 7f6689400700 1 --2- 172.21.15.27:0/1452110969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6684103fd0 0x7f668419c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.088+0000 7f6689400700 1 -- 172.21.15.27:0/1452110969 >> 172.21.15.27:0/1452110969 conn(0x7f66840feeb0 msgr2=0x7f6684109180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.088+0000 7f6689400700 1 -- 172.21.15.27:0/1452110969 shutdown_connections 2024-05-10T11:23:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:30.088+0000 7f6689400700 1 -- 172.21.15.27:0/1452110969 wait complete. 2024-05-10T11:23:30.718 INFO:tasks.workunit.client.0.smithi027.stdout: 1 617944 6.53 MB/sec execute 375 sec latency 23.093 ms 2024-05-10T11:23:31.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:23:31.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:30 smithi027 ceph-mon[30054]: pgmap v319: 129 pgs: 129 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 206 op/s 2024-05-10T11:23:31.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:30 smithi076 ceph-mon[34152]: pgmap v319: 129 pgs: 129 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 206 op/s 2024-05-10T11:23:31.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:30 smithi189 ceph-mon[34842]: pgmap v319: 129 pgs: 129 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 206 op/s 2024-05-10T11:23:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:23:31.719 INFO:tasks.workunit.client.0.smithi027.stdout: 1 618586 6.51 MB/sec execute 376 sec latency 23.019 ms 2024-05-10T11:23:32.719 INFO:tasks.workunit.client.0.smithi027.stdout: 1 619687 6.51 MB/sec execute 377 sec latency 37.677 ms 2024-05-10T11:23:33.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:32 smithi076 ceph-mon[34152]: pgmap v320: 129 pgs: 129 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 142 op/s 2024-05-10T11:23:33.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:32 smithi189 ceph-mon[34842]: pgmap v320: 129 pgs: 129 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 142 op/s 2024-05-10T11:23:33.305 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:32 smithi027 ceph-mon[30054]: pgmap v320: 129 pgs: 129 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 142 op/s 2024-05-10T11:23:33.719 INFO:tasks.workunit.client.0.smithi027.stdout: 1 621360 6.52 MB/sec execute 378 sec latency 51.609 ms 2024-05-10T11:23:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.868+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3707683748 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ac00756d0 msgr2=0x7f6ac0075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.868+0000 7f6ac7a00700 1 --2- 172.21.15.27:0/3707683748 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ac00756d0 0x7f6ac0075ab0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009990 tx=0x7f6ab4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.868+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3707683748 shutdown_connections 2024-05-10T11:23:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.868+0000 7f6ac7a00700 1 --2- 172.21.15.27:0/3707683748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac010fea0 0x7f6ac0112290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.868+0000 7f6ac7a00700 1 --2- 172.21.15.27:0/3707683748 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ac00756d0 0x7f6ac0075ab0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.868+0000 7f6ac7a00700 1 --2- 172.21.15.27:0/3707683748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac0075ff0 0x7f6ac010f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.869+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3707683748 >> 172.21.15.27:0/3707683748 conn(0x7f6ac007a600 msgr2=0x7f6ac007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:33.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.869+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3707683748 shutdown_connections 2024-05-10T11:23:33.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.869+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3707683748 wait complete. 2024-05-10T11:23:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.870+0000 7f6ac7a00700 1 Processor -- start 2024-05-10T11:23:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.870+0000 7f6ac7a00700 1 -- start start 2024-05-10T11:23:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.870+0000 7f6ac7a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac00756d0 0x7f6ac019be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.870+0000 7f6ac7a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ac0075ff0 0x7f6ac019c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.870+0000 7f6ac7a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac010fea0 0x7f6ac01a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.870+0000 7f6ac7a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac0115890 con 0x7f6ac00756d0 2024-05-10T11:23:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.870+0000 7f6ac7a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac0115710 con 0x7f6ac0075ff0 2024-05-10T11:23:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.870+0000 7f6ac7a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac0115a10 con 0x7f6ac010fea0 2024-05-10T11:23:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.871+0000 7f6abfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ac0075ff0 0x7f6ac019c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.871+0000 7f6abfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ac0075ff0 0x7f6ac019c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57816/0 (socket says 172.21.15.27:57816) 2024-05-10T11:23:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.871+0000 7f6abfe00700 1 -- 172.21.15.27:0/3497067247 learned_addr learned my addr 172.21.15.27:0/3497067247 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.871+0000 7f6ac5000700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac00756d0 0x7f6ac019be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.871+0000 7f6ac5a00700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac010fea0 0x7f6ac01a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.871+0000 7f6abfe00700 1 -- 172.21.15.27:0/3497067247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac010fea0 msgr2=0x7f6ac01a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.871+0000 7f6abfe00700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac010fea0 0x7f6ac01a0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.871+0000 7f6abfe00700 1 -- 172.21.15.27:0/3497067247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac00756d0 msgr2=0x7f6ac019be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.871+0000 7f6abfe00700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac00756d0 0x7f6ac019be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.871+0000 7f6abfe00700 1 -- 172.21.15.27:0/3497067247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac01a0e70 con 0x7f6ac0075ff0 2024-05-10T11:23:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.872+0000 7f6abfe00700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ac0075ff0 0x7f6ac019c3a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00e9c0 tx=0x7f6aac00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.872+0000 7f6abd600700 1 -- 172.21.15.27:0/3497067247 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac00cbd0 con 0x7f6ac0075ff0 2024-05-10T11:23:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.872+0000 7f6abd600700 1 -- 172.21.15.27:0/3497067247 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6aac00ee60 con 0x7f6ac0075ff0 2024-05-10T11:23:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.872+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3497067247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac01a1160 con 0x7f6ac0075ff0 2024-05-10T11:23:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.872+0000 7f6abd600700 1 -- 172.21.15.27:0/3497067247 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac0055c0 con 0x7f6ac0075ff0 2024-05-10T11:23:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.872+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3497067247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac01a89e0 con 0x7f6ac0075ff0 2024-05-10T11:23:33.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.876+0000 7f6abd600700 1 -- 172.21.15.27:0/3497067247 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f6aac004750 con 0x7f6ac0075ff0 2024-05-10T11:23:33.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.877+0000 7f6ad237db80 1 Processor -- start 2024-05-10T11:23:33.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.878+0000 7f6abd600700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6aa80b2a70 0x7f6aa80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.878+0000 7f6ad237db80 1 -- start start 2024-05-10T11:23:33.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.878+0000 7f6abd600700 1 -- 172.21.15.27:0/3497067247 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f6aac0e2cf0 con 0x7f6ac0075ff0 2024-05-10T11:23:33.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.878+0000 7f6ad237db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f060 0x55b48d71f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.878+0000 7f6ad237db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d71f980 0x55b48d723490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.878+0000 7f6ac5a00700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6aa80b2a70 0x7f6aa80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.878+0000 7f6ad237db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d7239d0 0x55b48d723e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.878+0000 7f6ad237db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b48d671a20 con 0x55b48d71f060 2024-05-10T11:23:33.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.878+0000 7f6ad237db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b48d60f760 con 0x55b48d7239d0 2024-05-10T11:23:33.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.878+0000 7f6ad237db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b48d66d9e0 con 0x55b48d71f980 2024-05-10T11:23:33.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.879+0000 7f6ac5000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f060 0x55b48d71f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.879+0000 7f6ac5000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f060 0x55b48d71f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56470/0 (socket says 172.21.15.27:56470) 2024-05-10T11:23:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.879+0000 7f6ac5000700 1 -- 172.21.15.27:0/3898424199 learned_addr learned my addr 172.21.15.27:0/3898424199 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.879+0000 7f6ac5a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d7239d0 0x55b48d723e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.879+0000 7f6abfe00700 1 --2- 172.21.15.27:0/3898424199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d71f980 0x55b48d723490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.879+0000 7f6ac5000700 1 -- 172.21.15.27:0/3898424199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d71f980 msgr2=0x55b48d723490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.879+0000 7f6ac5000700 1 --2- 172.21.15.27:0/3898424199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d71f980 0x55b48d723490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.879+0000 7f6ac5000700 1 -- 172.21.15.27:0/3898424199 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d7239d0 msgr2=0x55b48d723e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.879+0000 7f6ac5000700 1 --2- 172.21.15.27:0/3898424199 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d7239d0 0x55b48d723e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.879+0000 7f6ac5000700 1 -- 172.21.15.27:0/3898424199 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b48d5be8d0 con 0x55b48d71f060 2024-05-10T11:23:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.879+0000 7f6ac5a00700 1 --2- 172.21.15.27:0/3898424199 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d7239d0 0x55b48d723e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.880+0000 7f6ac5a00700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6aa80b2a70 0x7f6aa80b4f30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800f870 tx=0x7f6ab800fc30 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.880+0000 7f6ac5000700 1 --2- 172.21.15.27:0/3898424199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f060 0x55b48d71f440 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40096a0 tx=0x7f6ab40096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1a3cce60b8e19 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:33.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.880+0000 7f6ab2000700 1 -- 172.21.15.27:0/3898424199 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4027070 con 0x55b48d71f060 2024-05-10T11:23:33.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.880+0000 7f6ab2000700 1 -- 172.21.15.27:0/3898424199 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ab40056d0 con 0x55b48d71f060 2024-05-10T11:23:33.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.880+0000 7f6ab2000700 1 -- 172.21.15.27:0/3898424199 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4003aa0 con 0x55b48d71f060 2024-05-10T11:23:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.881+0000 7f6ad237db80 1 -- 172.21.15.27:0/3898424199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f060 msgr2=0x55b48d71f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.881+0000 7f6ad237db80 1 --2- 172.21.15.27:0/3898424199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f060 0x55b48d71f440 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40096a0 tx=0x7f6ab40096d0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.881+0000 7f6ad237db80 1 -- 172.21.15.27:0/3898424199 shutdown_connections 2024-05-10T11:23:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.881+0000 7f6ad237db80 1 --2- 172.21.15.27:0/3898424199 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d7239d0 0x55b48d723e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.881+0000 7f6ad237db80 1 --2- 172.21.15.27:0/3898424199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f060 0x55b48d71f440 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.881+0000 7f6ad237db80 1 --2- 172.21.15.27:0/3898424199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d71f980 0x55b48d723490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.882+0000 7f6ad237db80 1 -- 172.21.15.27:0/3898424199 >> 172.21.15.27:0/3898424199 conn(0x55b48d64db60 msgr2=0x55b48d7197a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.882+0000 7f6ad237db80 1 -- 172.21.15.27:0/3898424199 shutdown_connections 2024-05-10T11:23:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.882+0000 7f6ad237db80 1 -- 172.21.15.27:0/3898424199 wait complete. 2024-05-10T11:23:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.882+0000 7f6ad237db80 1 -- start start 2024-05-10T11:23:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.883+0000 7f6ad237db80 1 Processor -- start 2024-05-10T11:23:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.883+0000 7f6ad237db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 0x55b48d72ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.883+0000 7f6ad237db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d7281b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.883+0000 7f6ad237db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 0x55b48d7286f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.883+0000 7f6ad237db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b48d72bb80 con 0x55b48d71f980 2024-05-10T11:23:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.883+0000 7f6ad237db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b48d5d7b20 con 0x55b48d71f060 2024-05-10T11:23:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.883+0000 7f6ad237db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b48d72bd00 con 0x55b48d7239d0 2024-05-10T11:23:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.883+0000 7f6ad237db80 10 client.0 _ll_register_callbacks cb 0x55b48d618c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6ac5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 0x55b48d7286f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6ac5000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 0x55b48d72ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6ac5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 0x55b48d7286f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54258/0 (socket says 172.21.15.27:54258) 2024-05-10T11:23:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6ac5a00700 1 -- 172.21.15.27:0/3493205970 learned_addr learned my addr 172.21.15.27:0/3493205970 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6abfe00700 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d7281b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6ac5a00700 1 -- 172.21.15.27:0/3493205970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 msgr2=0x55b48d72ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6ac5a00700 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 0x55b48d72ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6ac5a00700 1 -- 172.21.15.27:0/3493205970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 msgr2=0x55b48d7281b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6ac5a00700 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d7281b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6ac5a00700 1 -- 172.21.15.27:0/3493205970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b48d637420 con 0x55b48d7239d0 2024-05-10T11:23:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6ac5000700 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 0x55b48d72ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6ac5a00700 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 0x55b48d7286f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80074d0 tx=0x7f6ab8004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eabc4bd4353f2a32 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.884+0000 7f6aa6a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:23:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.885+0000 7f6aa6a00700 1 -- 172.21.15.27:0/3493205970 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8027020 con 0x55b48d7239d0 2024-05-10T11:23:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.885+0000 7f6ad237db80 1 -- 172.21.15.27:0/3493205970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ab40056d0 con 0x55b48d7239d0 2024-05-10T11:23:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.885+0000 7f6aa6a00700 1 -- 172.21.15.27:0/3493205970 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f6ab8008d80 con 0x55b48d7239d0 2024-05-10T11:23:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.885+0000 7f6aa6a00700 1 -- 172.21.15.27:0/3493205970 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6ab8032050 con 0x55b48d7239d0 2024-05-10T11:23:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.885+0000 7f6ad237db80 10 client.26737 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.885+0000 7f6ad237db80 1 -- 172.21.15.27:0/3493205970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b48d736400 con 0x55b48d7239d0 2024-05-10T11:23:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.885+0000 7f6aa6a00700 1 -- 172.21.15.27:0/3493205970 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ab40056d0 con 0x55b48d7239d0 2024-05-10T11:23:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.886+0000 7f6aa6a00700 1 -- 172.21.15.27:0/3493205970 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f6ab801ace0 con 0x55b48d7239d0 2024-05-10T11:23:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.886+0000 7f6ad237db80 10 client.26737 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.886+0000 7f6ad237db80 10 client.26737 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:23:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.886+0000 7f6ad237db80 20 client.26737 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.886+0000 7f6ad237db80 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b48d738a40 0x55b48d73ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.886+0000 7f6ad237db80 4 client.26737 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:23:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.886+0000 7f6ad237db80 1 -- 172.21.15.27:0/3493205970 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b48d6ec0a0 con 0x55b48d738a40 2024-05-10T11:23:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.886+0000 7f6ac5000700 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b48d738a40 0x55b48d73ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.887+0000 7f6ac5000700 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b48d738a40 0x55b48d73ae60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=12ff64362e1bb3c server_cookie=d469f0371c8eff45 in_seq=0 out_seq=0 2024-05-10T11:23:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.887+0000 7f6aa6a00700 10 client.26737 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:23:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.887+0000 7f6aa6a00700 1 -- 172.21.15.27:0/3493205970 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b48d6ec0a0 con 0x55b48d738a40 2024-05-10T11:23:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.887+0000 7f6aa6a00700 10 client.26737 handle_command_reply: tid=0 2024-05-10T11:23:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6ad237db80 1 client.26737 shutdown 2024-05-10T11:23:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6ad237db80 1 -- 172.21.15.27:0/3493205970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 msgr2=0x55b48d7286f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6ad237db80 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 0x55b48d7286f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80074d0 tx=0x7f6ab8004b40 comp rx=0 tx=0).stop 2024-05-10T11:23:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6ad237db80 1 -- 172.21.15.27:0/3493205970 shutdown_connections 2024-05-10T11:23:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6ad237db80 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b48d738a40 0x55b48d73ae60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6ad237db80 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 0x55b48d7286f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6ad237db80 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 0x55b48d72ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6ad237db80 1 --2- 172.21.15.27:0/3493205970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d7281b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6ad237db80 1 -- 172.21.15.27:0/3493205970 >> 172.21.15.27:0/3493205970 conn(0x55b48d64db60 msgr2=0x55b48d7142b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6aa6a00700 0 client.26737 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:23:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6ad237db80 1 -- 172.21.15.27:0/3493205970 shutdown_connections 2024-05-10T11:23:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6ad237db80 1 -- 172.21.15.27:0/3493205970 wait complete. 2024-05-10T11:23:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.888+0000 7f6ad237db80 20 client.26737 trim_cache size 0 max 16384 2024-05-10T11:23:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.906+0000 7f6ad237db80 1 Processor -- start 2024-05-10T11:23:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.906+0000 7f6ad237db80 1 -- start start 2024-05-10T11:23:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.907+0000 7f6ad237db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d71f060 0x55b48d7287e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.907+0000 7f6ad237db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d728dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.907+0000 7f6ad237db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d7239d0 0x55b48d734c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.907+0000 7f6ad237db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac01b1950 con 0x55b48d71f980 2024-05-10T11:23:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.907+0000 7f6ad237db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac01b3490 con 0x55b48d7239d0 2024-05-10T11:23:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.907+0000 7f6ad237db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac01b2020 con 0x55b48d71f060 2024-05-10T11:23:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.907+0000 7f6ac5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d71f060 0x55b48d7287e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.907+0000 7f6ac5a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d7239d0 0x55b48d734c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.907+0000 7f6ac5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d71f060 0x55b48d7287e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54274/0 (socket says 172.21.15.27:54274) 2024-05-10T11:23:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.907+0000 7f6abfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d728dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.907+0000 7f6ac5000700 1 -- 172.21.15.27:0/356968301 learned_addr learned my addr 172.21.15.27:0/356968301 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:33.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.908+0000 7f6ac5000700 1 -- 172.21.15.27:0/356968301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d7239d0 msgr2=0x55b48d734c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:33.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.908+0000 7f6ac5000700 1 --2- 172.21.15.27:0/356968301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d7239d0 0x55b48d734c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.908+0000 7f6ac5000700 1 -- 172.21.15.27:0/356968301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 msgr2=0x55b48d728dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:33.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.908+0000 7f6ac5000700 1 --2- 172.21.15.27:0/356968301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d728dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.908+0000 7f6ac5000700 1 -- 172.21.15.27:0/356968301 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b48d723800 con 0x55b48d71f060 2024-05-10T11:23:33.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.908+0000 7f6ac5a00700 1 --2- 172.21.15.27:0/356968301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d7239d0 0x55b48d734c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:33.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.908+0000 7f6ac5000700 1 --2- 172.21.15.27:0/356968301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d71f060 0x55b48d7287e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4005320 tx=0x7f6ab400b310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40203cbb8866c913 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:33.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.908+0000 7f6ab1600700 1 -- 172.21.15.27:0/356968301 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4027040 con 0x55b48d71f060 2024-05-10T11:23:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.908+0000 7f6ab1600700 1 -- 172.21.15.27:0/356968301 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ab4028040 con 0x55b48d71f060 2024-05-10T11:23:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.908+0000 7f6ab1600700 1 -- 172.21.15.27:0/356968301 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4032200 con 0x55b48d71f060 2024-05-10T11:23:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.909+0000 7f6ad237db80 1 -- 172.21.15.27:0/356968301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d71f060 msgr2=0x55b48d7287e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.909+0000 7f6ad237db80 1 --2- 172.21.15.27:0/356968301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d71f060 0x55b48d7287e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4005320 tx=0x7f6ab400b310 comp rx=0 tx=0).stop 2024-05-10T11:23:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.909+0000 7f6ad237db80 1 -- 172.21.15.27:0/356968301 shutdown_connections 2024-05-10T11:23:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.909+0000 7f6ad237db80 1 --2- 172.21.15.27:0/356968301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d71f060 0x55b48d7287e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.909+0000 7f6ad237db80 1 --2- 172.21.15.27:0/356968301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d7239d0 0x55b48d734c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.909+0000 7f6ad237db80 1 --2- 172.21.15.27:0/356968301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d728dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.909+0000 7f6ad237db80 1 -- 172.21.15.27:0/356968301 >> 172.21.15.27:0/356968301 conn(0x55b48d64db60 msgr2=0x55b48d713a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.910+0000 7f6ad237db80 1 -- 172.21.15.27:0/356968301 shutdown_connections 2024-05-10T11:23:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.910+0000 7f6ad237db80 1 -- 172.21.15.27:0/356968301 wait complete. 2024-05-10T11:23:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.910+0000 7f6ad237db80 1 -- start start 2024-05-10T11:23:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.910+0000 7f6ad237db80 1 Processor -- start 2024-05-10T11:23:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.911+0000 7f6ad237db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 0x55b48d721550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.911+0000 7f6ad237db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d721a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.911+0000 7f6ad237db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 0x55b48d721ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.911+0000 7f6ad237db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac01b2dc0 con 0x55b48d71f980 2024-05-10T11:23:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.911+0000 7f6ad237db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac01b26f0 con 0x55b48d71f060 2024-05-10T11:23:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.911+0000 7f6ad237db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac01af910 con 0x55b48d7239d0 2024-05-10T11:23:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.911+0000 7f6ad237db80 10 client.0 _ll_register_callbacks cb 0x55b48d618c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.911+0000 7f6ac5000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 0x55b48d721550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.911+0000 7f6abfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d721a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.911+0000 7f6ac5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 0x55b48d721ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.911+0000 7f6ac5000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 0x55b48d721550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57844/0 (socket says 172.21.15.27:57844) 2024-05-10T11:23:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.911+0000 7f6ac5000700 1 -- 172.21.15.27:0/2866704442 learned_addr learned my addr 172.21.15.27:0/2866704442 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.912+0000 7f6ac5000700 1 -- 172.21.15.27:0/2866704442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 msgr2=0x55b48d721ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.912+0000 7f6ac5000700 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 0x55b48d721ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.912+0000 7f6ac5000700 1 -- 172.21.15.27:0/2866704442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 msgr2=0x55b48d721a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.912+0000 7f6ac5000700 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d721a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.912+0000 7f6ac5000700 1 -- 172.21.15.27:0/2866704442 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b48d7302c0 con 0x55b48d71f060 2024-05-10T11:23:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.912+0000 7f6ac5a00700 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 0x55b48d721ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.912+0000 7f6abfe00700 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d721a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.912+0000 7f6ac5000700 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 0x55b48d721550 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4006010 tx=0x7f6ab4005700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac2ad4fdaf620791 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.912+0000 7f6aa6a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.912+0000 7f6aa6a00700 1 -- 172.21.15.27:0/2866704442 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4043060 con 0x55b48d71f060 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.913+0000 7f6ad237db80 1 -- 172.21.15.27:0/2866704442 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ab4028040 con 0x55b48d71f060 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.913+0000 7f6aa6a00700 1 -- 172.21.15.27:0/2866704442 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f6ab4003cc0 con 0x55b48d71f060 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.913+0000 7f6aa6a00700 1 -- 172.21.15.27:0/2866704442 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6ab404a050 con 0x55b48d71f060 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.913+0000 7f6ad237db80 10 client.26756 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.913+0000 7f6ad237db80 1 -- 172.21.15.27:0/2866704442 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b48d7315c0 con 0x55b48d71f060 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.913+0000 7f6aa6a00700 1 -- 172.21.15.27:0/2866704442 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ab4028040 con 0x55b48d71f060 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.913+0000 7f6aa6a00700 1 -- 172.21.15.27:0/2866704442 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f6ab403d070 con 0x55b48d71f060 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.913+0000 7f6ad237db80 10 client.26756 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.914+0000 7f6ad237db80 10 client.26756 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:23:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.914+0000 7f6ad237db80 20 client.26756 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.914+0000 7f6ad237db80 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b48d7a32c0 0x55b48d7a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.914+0000 7f6ad237db80 4 client.26756 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.914+0000 7f6ad237db80 1 -- 172.21.15.27:0/2866704442 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b48d67f400 con 0x55b48d7a32c0 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.914+0000 7f6abfe00700 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b48d7a32c0 0x55b48d7a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.914+0000 7f6abfe00700 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b48d7a32c0 0x55b48d7a56b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7ddf69d7b5bfe9cc server_cookie=969935516e948414 in_seq=0 out_seq=0 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.914+0000 7f6aa6a00700 10 client.26756 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6aa6a00700 1 -- 172.21.15.27:0/2866704442 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b48d67f400 con 0x55b48d7a32c0 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6aa6a00700 10 client.26756 handle_command_reply: tid=0 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6ad237db80 1 client.26756 shutdown 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6ad237db80 1 -- 172.21.15.27:0/2866704442 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 msgr2=0x55b48d721550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6ad237db80 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 0x55b48d721550 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4006010 tx=0x7f6ab4005700 comp rx=0 tx=0).stop 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6ad237db80 1 -- 172.21.15.27:0/2866704442 shutdown_connections 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6ad237db80 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55b48d7a32c0 0x55b48d7a56b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6ad237db80 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b48d7239d0 0x55b48d721ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6ad237db80 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b48d71f060 0x55b48d721550 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6aa6a00700 0 client.26756 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:23:33.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6ad237db80 1 --2- 172.21.15.27:0/2866704442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b48d71f980 0x55b48d721a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6ad237db80 1 -- 172.21.15.27:0/2866704442 >> 172.21.15.27:0/2866704442 conn(0x55b48d64db60 msgr2=0x55b48d726690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:33.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6ad237db80 1 -- 172.21.15.27:0/2866704442 shutdown_connections 2024-05-10T11:23:33.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6ad237db80 1 -- 172.21.15.27:0/2866704442 wait complete. 2024-05-10T11:23:33.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.915+0000 7f6ad237db80 20 client.26756 trim_cache size 0 max 16384 2024-05-10T11:23:33.917 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:23:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.916+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3497067247 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6aa80b2a70 msgr2=0x7f6aa80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.916+0000 7f6ac7a00700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6aa80b2a70 0x7f6aa80b4f30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800f870 tx=0x7f6ab800fc30 comp rx=0 tx=0).stop 2024-05-10T11:23:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.916+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3497067247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ac0075ff0 msgr2=0x7f6ac019c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.916+0000 7f6ac7a00700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ac0075ff0 0x7f6ac019c3a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00e9c0 tx=0x7f6aac00ecd0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.916+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3497067247 shutdown_connections 2024-05-10T11:23:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.916+0000 7f6ac7a00700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6aa80b2a70 0x7f6aa80b4f30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.916+0000 7f6ac7a00700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac010fea0 0x7f6ac01a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.916+0000 7f6ac7a00700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac00756d0 0x7f6ac019be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.916+0000 7f6ac7a00700 1 --2- 172.21.15.27:0/3497067247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ac0075ff0 0x7f6ac019c3a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.916+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3497067247 >> 172.21.15.27:0/3497067247 conn(0x7f6ac007a600 msgr2=0x7f6ac0101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.916+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3497067247 shutdown_connections 2024-05-10T11:23:33.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:33.917+0000 7f6ac7a00700 1 -- 172.21.15.27:0/3497067247 wait complete. 2024-05-10T11:23:34.719 INFO:tasks.workunit.client.0.smithi027.stdout: 1 622068 6.51 MB/sec execute 379 sec latency 22.166 ms 2024-05-10T11:23:34.847 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:23:35.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:34 smithi027 ceph-mon[30054]: pgmap v321: 129 pgs: 129 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 142 op/s 2024-05-10T11:23:35.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:34 smithi076 ceph-mon[34152]: pgmap v321: 129 pgs: 129 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 142 op/s 2024-05-10T11:23:35.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:34 smithi189 ceph-mon[34842]: pgmap v321: 129 pgs: 129 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 142 op/s 2024-05-10T11:23:35.372 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:23:35.719 INFO:tasks.workunit.client.0.smithi027.stdout: 1 623306 6.51 MB/sec execute 380 sec latency 43.958 ms 2024-05-10T11:23:36.720 INFO:tasks.workunit.client.0.smithi027.stdout: 1 625160 6.53 MB/sec execute 381 sec latency 33.256 ms 2024-05-10T11:23:37.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:36 smithi027 ceph-mon[30054]: pgmap v322: 129 pgs: 129 active+clean; 474 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 207 op/s 2024-05-10T11:23:37.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:36 smithi076 ceph-mon[34152]: pgmap v322: 129 pgs: 129 active+clean; 474 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 207 op/s 2024-05-10T11:23:37.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:36 smithi189 ceph-mon[34842]: pgmap v322: 129 pgs: 129 active+clean; 474 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 207 op/s 2024-05-10T11:23:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.702+0000 7f94edc00700 1 -- 172.21.15.27:0/1738639970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e8105f90 msgr2=0x7f94e8112450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.702+0000 7f94edc00700 1 --2- 172.21.15.27:0/1738639970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e8105f90 0x7f94e8112450 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f94e000b0a0 tx=0x7f94e000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.702+0000 7f94edc00700 1 -- 172.21.15.27:0/1738639970 shutdown_connections 2024-05-10T11:23:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.702+0000 7f94edc00700 1 --2- 172.21.15.27:0/1738639970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e8105f90 0x7f94e8112450 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.702+0000 7f94edc00700 1 --2- 172.21.15.27:0/1738639970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e810b680 0x7f94e810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.702+0000 7f94edc00700 1 --2- 172.21.15.27:0/1738639970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94e81055d0 0x7f94e8105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.702+0000 7f94edc00700 1 -- 172.21.15.27:0/1738639970 >> 172.21.15.27:0/1738639970 conn(0x7f94e8101080 msgr2=0x7f94e81034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.702+0000 7f94edc00700 1 -- 172.21.15.27:0/1738639970 shutdown_connections 2024-05-10T11:23:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.702+0000 7f94edc00700 1 -- 172.21.15.27:0/1738639970 wait complete. 2024-05-10T11:23:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.704+0000 7f94edc00700 1 Processor -- start 2024-05-10T11:23:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.704+0000 7f94edc00700 1 -- start start 2024-05-10T11:23:37.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.704+0000 7f94edc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94e81055d0 0x7f94e81a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94edc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e8105f90 0x7f94e81a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94edc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e810b680 0x7f94e819ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94edc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f94e8115830 con 0x7f94e8105f90 2024-05-10T11:23:37.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94edc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81156b0 con 0x7f94e81055d0 2024-05-10T11:23:37.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94edc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81159b0 con 0x7f94e810b680 2024-05-10T11:23:37.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94e81055d0 0x7f94e81a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94e81055d0 0x7f94e81a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42864/0 (socket says 172.21.15.27:42864) 2024-05-10T11:23:37.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94e7e00700 1 -- 172.21.15.27:0/2484552012 learned_addr learned my addr 172.21.15.27:0/2484552012 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:37.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94ed200700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e810b680 0x7f94e819ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94e7e00700 1 -- 172.21.15.27:0/2484552012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e810b680 msgr2=0x7f94e819ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:37.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94e7e00700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e810b680 0x7f94e819ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94e7e00700 1 -- 172.21.15.27:0/2484552012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e8105f90 msgr2=0x7f94e81a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:37.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94e7e00700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e8105f90 0x7f94e81a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.705+0000 7f94e7e00700 1 -- 172.21.15.27:0/2484552012 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e819c1b0 con 0x7f94e81055d0 2024-05-10T11:23:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.706+0000 7f94e7e00700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94e81055d0 0x7f94e81a1840 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f94d800bf20 tx=0x7f94d800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.706+0000 7f94e4c00700 1 -- 172.21.15.27:0/2484552012 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8021680 con 0x7f94e81055d0 2024-05-10T11:23:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.706+0000 7f94e4c00700 1 -- 172.21.15.27:0/2484552012 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94d8021cd0 con 0x7f94e81055d0 2024-05-10T11:23:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.706+0000 7f94edc00700 1 -- 172.21.15.27:0/2484552012 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e819c4a0 con 0x7f94e81055d0 2024-05-10T11:23:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.707+0000 7f94e4c00700 1 -- 172.21.15.27:0/2484552012 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8009cb0 con 0x7f94e81055d0 2024-05-10T11:23:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.707+0000 7f94edc00700 1 -- 172.21.15.27:0/2484552012 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e80778e0 con 0x7f94e81055d0 2024-05-10T11:23:37.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.711+0000 7f94e4c00700 1 -- 172.21.15.27:0/2484552012 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f94d8021820 con 0x7f94e81055d0 2024-05-10T11:23:37.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.713+0000 7f94f9c46b80 1 Processor -- start 2024-05-10T11:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.713+0000 7f94e4c00700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f94cc0b2a70 0x7f94cc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.713+0000 7f94f9c46b80 1 -- start start 2024-05-10T11:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.714+0000 7f94e4c00700 1 -- 172.21.15.27:0/2484552012 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f94d80eaee0 con 0x7f94e81055d0 2024-05-10T11:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.714+0000 7f94f9c46b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d478dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.714+0000 7f94f9c46b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d479300 0x55d54d472fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.714+0000 7f94f9c46b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d473600 0x55d54d473a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.714+0000 7f94f9c46b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d54d3c5a20 con 0x55d54d4789e0 2024-05-10T11:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.714+0000 7f94f9c46b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d54d363760 con 0x55d54d473600 2024-05-10T11:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.714+0000 7f94f9c46b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d54d3c19e0 con 0x55d54d479300 2024-05-10T11:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.714+0000 7f94e7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d479300 0x55d54d472fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.714+0000 7f94e7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d479300 0x55d54d472fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:45228/0 (socket says 172.21.15.27:45228) 2024-05-10T11:23:37.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.714+0000 7f94e7400700 1 -- 172.21.15.27:0/2828652421 learned_addr learned my addr 172.21.15.27:0/2828652421 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:37.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.714+0000 7f94ed200700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f94cc0b2a70 0x7f94cc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.714+0000 7f94e7e00700 1 --2- 172.21.15.27:0/2828652421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d478dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.715+0000 7f94e7400700 1 -- 172.21.15.27:0/2828652421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d473600 msgr2=0x55d54d473a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:23:37.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.715+0000 7f94e7400700 1 --2- 172.21.15.27:0/2828652421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d473600 0x55d54d473a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.715+0000 7f94e7400700 1 -- 172.21.15.27:0/2828652421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 msgr2=0x55d54d478dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:37.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.715+0000 7f94e7400700 1 --2- 172.21.15.27:0/2828652421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d478dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.715+0000 7f94e7400700 1 -- 172.21.15.27:0/2828652421 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d54d3128d0 con 0x55d54d479300 2024-05-10T11:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.715+0000 7f94e7400700 1 --2- 172.21.15.27:0/2828652421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d479300 0x55d54d472fe0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009c70 tx=0x7f94d4009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72cd4d6f3dfa2ebc server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.715+0000 7f94dd600700 1 -- 172.21.15.27:0/2828652421 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d400ec10 con 0x55d54d479300 2024-05-10T11:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.716+0000 7f94dd600700 1 -- 172.21.15.27:0/2828652421 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94d400edb0 con 0x55d54d479300 2024-05-10T11:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.716+0000 7f94dd600700 1 -- 172.21.15.27:0/2828652421 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d4010430 con 0x55d54d479300 2024-05-10T11:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.717+0000 7f94f9c46b80 1 -- 172.21.15.27:0/2828652421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d479300 msgr2=0x55d54d472fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.717+0000 7f94ed200700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f94cc0b2a70 0x7f94cc0b4f30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55d54d473fc0 tx=0x7f94e000b700 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.717+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/2828652421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d479300 0x55d54d472fe0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009c70 tx=0x7f94d4009f80 comp rx=0 tx=0).stop 2024-05-10T11:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.717+0000 7f94f9c46b80 1 -- 172.21.15.27:0/2828652421 shutdown_connections 2024-05-10T11:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.717+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/2828652421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d473600 0x55d54d473a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.717+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/2828652421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d478dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.717+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/2828652421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d479300 0x55d54d472fe0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.717+0000 7f94f9c46b80 1 -- 172.21.15.27:0/2828652421 >> 172.21.15.27:0/2828652421 conn(0x55d54d3a1b60 msgr2=0x55d54d46d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:37.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.717+0000 7f94f9c46b80 1 -- 172.21.15.27:0/2828652421 shutdown_connections 2024-05-10T11:23:37.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.717+0000 7f94f9c46b80 1 -- 172.21.15.27:0/2828652421 wait complete. 2024-05-10T11:23:37.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.718+0000 7f94f9c46b80 1 -- start start 2024-05-10T11:23:37.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.718+0000 7f94f9c46b80 1 Processor -- start 2024-05-10T11:23:37.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94f9c46b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 0x55d54d47efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94f9c46b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d47c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94f9c46b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d479300 0x55d54d47c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94f9c46b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d54d47fdb0 con 0x55d54d4789e0 2024-05-10T11:23:37.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94f9c46b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d54d32bb20 con 0x55d54d479300 2024-05-10T11:23:37.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94f9c46b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d54d47ff30 con 0x55d54d473600 2024-05-10T11:23:37.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94f9c46b80 10 client.0 _ll_register_callbacks cb 0x55d54d36cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:37.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 0x55d54d47efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94e7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d47c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 0x55d54d47efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:45230/0 (socket says 172.21.15.27:45230) 2024-05-10T11:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94e7e00700 1 -- 172.21.15.27:0/1461684332 learned_addr learned my addr 172.21.15.27:0/1461684332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94ed200700 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d479300 0x55d54d47c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94e7e00700 1 -- 172.21.15.27:0/1461684332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d479300 msgr2=0x55d54d47c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94e7e00700 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d479300 0x55d54d47c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94e7e00700 1 -- 172.21.15.27:0/1461684332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 msgr2=0x55d54d47c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94e7e00700 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d47c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94e7e00700 1 -- 172.21.15.27:0/1461684332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d54d38b420 con 0x55d54d473600 2024-05-10T11:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.719+0000 7f94e7400700 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d47c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.720+0000 7f94e7e00700 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 0x55d54d47efa0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f94d80e8310 tx=0x7f94d80096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e00d8c64b2c5ba82 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.720+0000 7f94d2000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:23:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.720+0000 7f94d2000700 1 -- 172.21.15.27:0/1461684332 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d80b4030 con 0x55d54d473600 2024-05-10T11:23:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.720+0000 7f94d2000700 1 -- 172.21.15.27:0/1461684332 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f94d80ae430 con 0x55d54d473600 2024-05-10T11:23:37.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 625903 6.51 MB/sec execute 382 sec latency 37.066 ms 2024-05-10T11:23:37.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.720+0000 7f94f9c46b80 1 -- 172.21.15.27:0/1461684332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94d400edb0 con 0x55d54d473600 2024-05-10T11:23:37.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.720+0000 7f94d2000700 1 -- 172.21.15.27:0/1461684332 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f94d80b2540 con 0x55d54d473600 2024-05-10T11:23:37.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.720+0000 7f94f9c46b80 10 client.26755 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:37.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.720+0000 7f94f9c46b80 1 -- 172.21.15.27:0/1461684332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d54d48a620 con 0x55d54d473600 2024-05-10T11:23:37.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.721+0000 7f94d2000700 1 -- 172.21.15.27:0/1461684332 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94d400edb0 con 0x55d54d473600 2024-05-10T11:23:37.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.721+0000 7f94d2000700 1 -- 172.21.15.27:0/1461684332 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f94d8029030 con 0x55d54d473600 2024-05-10T11:23:37.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.721+0000 7f94f9c46b80 10 client.26755 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:37.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.721+0000 7f94f9c46b80 10 client.26755 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:23:37.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.721+0000 7f94f9c46b80 20 client.26755 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:37.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.721+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55d54d48cbb0 0x55d54d48efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.721+0000 7f94f9c46b80 4 client.26755 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:23:37.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.721+0000 7f94f9c46b80 1 -- 172.21.15.27:0/1461684332 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d54d4400a0 con 0x55d54d48cbb0 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.721+0000 7f94e7400700 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55d54d48cbb0 0x55d54d48efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.722+0000 7f94e7400700 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55d54d48cbb0 0x55d54d48efa0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=334bf44231f318fe server_cookie=3289744cf23fbc65 in_seq=0 out_seq=0 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.722+0000 7f94d2000700 10 client.26755 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94d2000700 1 -- 172.21.15.27:0/1461684332 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d54d4400a0 con 0x55d54d48cbb0 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94d2000700 10 client.26755 handle_command_reply: tid=0 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94f9c46b80 1 client.26755 shutdown 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94f9c46b80 1 -- 172.21.15.27:0/1461684332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 msgr2=0x55d54d47efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 0x55d54d47efa0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f94d80e8310 tx=0x7f94d80096a0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94f9c46b80 1 -- 172.21.15.27:0/1461684332 shutdown_connections 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55d54d48cbb0 0x55d54d48efa0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 0x55d54d47efa0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d479300 0x55d54d47c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94d2000700 0 client.26755 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/1461684332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d47c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94f9c46b80 1 -- 172.21.15.27:0/1461684332 >> 172.21.15.27:0/1461684332 conn(0x55d54d3a1b60 msgr2=0x55d54d468460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:37.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94f9c46b80 1 -- 172.21.15.27:0/1461684332 shutdown_connections 2024-05-10T11:23:37.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.723+0000 7f94f9c46b80 1 -- 172.21.15.27:0/1461684332 wait complete. 2024-05-10T11:23:37.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.724+0000 7f94f9c46b80 20 client.26755 trim_cache size 0 max 16384 2024-05-10T11:23:37.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.741+0000 7f94f9c46b80 1 Processor -- start 2024-05-10T11:23:37.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.741+0000 7f94f9c46b80 1 -- start start 2024-05-10T11:23:37.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.741+0000 7f94f9c46b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d473600 0x55d54d488e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.741+0000 7f94f9c46b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d473a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.741+0000 7f94f9c46b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d479300 0x55d54d494800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.741+0000 7f94f9c46b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81ad310 con 0x55d54d4789e0 2024-05-10T11:23:37.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.741+0000 7f94f9c46b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81aee50 con 0x55d54d473600 2024-05-10T11:23:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.741+0000 7f94f9c46b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81ad9e0 con 0x55d54d479300 2024-05-10T11:23:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.741+0000 7f94e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d473600 0x55d54d488e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.742+0000 7f94e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d473600 0x55d54d488e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42896/0 (socket says 172.21.15.27:42896) 2024-05-10T11:23:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.742+0000 7f94e7e00700 1 -- 172.21.15.27:0/3973414494 learned_addr learned my addr 172.21.15.27:0/3973414494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.742+0000 7f94ed200700 1 --2- 172.21.15.27:0/3973414494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d479300 0x55d54d494800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.742+0000 7f94e7400700 1 --2- 172.21.15.27:0/3973414494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d473a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.742+0000 7f94e7e00700 1 -- 172.21.15.27:0/3973414494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d479300 msgr2=0x55d54d494800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.742+0000 7f94e7e00700 1 --2- 172.21.15.27:0/3973414494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d479300 0x55d54d494800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.742+0000 7f94e7e00700 1 -- 172.21.15.27:0/3973414494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 msgr2=0x55d54d473a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.742+0000 7f94e7e00700 1 --2- 172.21.15.27:0/3973414494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d473a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.742+0000 7f94e7e00700 1 -- 172.21.15.27:0/3973414494 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d54d47a8b0 con 0x55d54d473600 2024-05-10T11:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.742+0000 7f94e7e00700 1 --2- 172.21.15.27:0/3973414494 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d473600 0x55d54d488e40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f94d80af0b0 tx=0x7f94d8016f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=564d0f3c6113bc1c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.743+0000 7f94dcc00700 1 -- 172.21.15.27:0/3973414494 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8091070 con 0x55d54d473600 2024-05-10T11:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.743+0000 7f94dcc00700 1 -- 172.21.15.27:0/3973414494 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94d80b61f0 con 0x55d54d473600 2024-05-10T11:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.743+0000 7f94dcc00700 1 -- 172.21.15.27:0/3973414494 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d80b4070 con 0x55d54d473600 2024-05-10T11:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.743+0000 7f94f9c46b80 1 -- 172.21.15.27:0/3973414494 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d473600 msgr2=0x55d54d488e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.743+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/3973414494 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d473600 0x55d54d488e40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f94d80af0b0 tx=0x7f94d8016f90 comp rx=0 tx=0).stop 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.743+0000 7f94f9c46b80 1 -- 172.21.15.27:0/3973414494 shutdown_connections 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.743+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/3973414494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d479300 0x55d54d494800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.743+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/3973414494 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d473600 0x55d54d488e40 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.743+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/3973414494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d4789e0 0x55d54d473a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.743+0000 7f94f9c46b80 1 -- 172.21.15.27:0/3973414494 >> 172.21.15.27:0/3973414494 conn(0x55d54d3a1b60 msgr2=0x55d54d47c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.744+0000 7f94f9c46b80 1 -- 172.21.15.27:0/3973414494 shutdown_connections 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.744+0000 7f94f9c46b80 1 -- 172.21.15.27:0/3973414494 wait complete. 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.744+0000 7f94f9c46b80 1 -- start start 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.744+0000 7f94f9c46b80 1 Processor -- start 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94f9c46b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 0x55d54d4853a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94f9c46b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d4789e0 0x55d54d4858e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94f9c46b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d479300 0x55d54d493b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94f9c46b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81ae780 con 0x55d54d479300 2024-05-10T11:23:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94f9c46b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81ae0b0 con 0x55d54d4789e0 2024-05-10T11:23:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94f9c46b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81ab2d0 con 0x55d54d473600 2024-05-10T11:23:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94f9c46b80 10 client.0 _ll_register_callbacks cb 0x55d54d36cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 0x55d54d4853a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94e7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d4789e0 0x55d54d4858e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 0x55d54d4853a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:45236/0 (socket says 172.21.15.27:45236) 2024-05-10T11:23:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94e7e00700 1 -- 172.21.15.27:0/4098348724 learned_addr learned my addr 172.21.15.27:0/4098348724 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94ed200700 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d479300 0x55d54d493b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94e7400700 1 -- 172.21.15.27:0/4098348724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 msgr2=0x55d54d4853a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94e7400700 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 0x55d54d4853a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94e7400700 1 -- 172.21.15.27:0/4098348724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d479300 msgr2=0x55d54d493b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94e7400700 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d479300 0x55d54d493b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.745+0000 7f94e7400700 1 -- 172.21.15.27:0/4098348724 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d54d4943b0 con 0x55d54d4789e0 2024-05-10T11:23:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.746+0000 7f94ed200700 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d479300 0x55d54d493b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.746+0000 7f94e7400700 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d4789e0 0x55d54d4858e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x55d54d48f7a0 tx=0x7f94d4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a253f029e031c57 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.746+0000 7f94d2000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.746+0000 7f94d2000700 1 -- 172.21.15.27:0/4098348724 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d401d040 con 0x55d54d4789e0 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.746+0000 7f94f9c46b80 1 -- 172.21.15.27:0/4098348724 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94d80b61f0 con 0x55d54d4789e0 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.746+0000 7f94d2000700 1 -- 172.21.15.27:0/4098348724 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f94d400ea10 con 0x55d54d4789e0 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.746+0000 7f94d2000700 1 -- 172.21.15.27:0/4098348724 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f94d4039050 con 0x55d54d4789e0 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.747+0000 7f94f9c46b80 10 client.26771 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.747+0000 7f94f9c46b80 1 -- 172.21.15.27:0/4098348724 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d54d4f7010 con 0x55d54d4789e0 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.747+0000 7f94d2000700 1 -- 172.21.15.27:0/4098348724 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94d80b61f0 con 0x55d54d4789e0 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.747+0000 7f94d2000700 1 -- 172.21.15.27:0/4098348724 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f94d4004950 con 0x55d54d4789e0 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.747+0000 7f94f9c46b80 10 client.26771 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.747+0000 7f94f9c46b80 10 client.26771 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.747+0000 7f94f9c46b80 20 client.26771 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.748+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55d54d4f74a0 0x55d54d4f98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.748+0000 7f94f9c46b80 4 client.26771 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:23:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.748+0000 7f94f9c46b80 1 -- 172.21.15.27:0/4098348724 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d54d3d3400 con 0x55d54d4f74a0 2024-05-10T11:23:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.748+0000 7f94e7e00700 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55d54d4f74a0 0x55d54d4f98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.748+0000 7f94e7e00700 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55d54d4f74a0 0x55d54d4f98c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8f8b9c43ef545671 server_cookie=7a999a180a4298f0 in_seq=0 out_seq=0 2024-05-10T11:23:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.748+0000 7f94d2000700 10 client.26771 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:23:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94d2000700 1 -- 172.21.15.27:0/4098348724 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d54d3d3400 con 0x55d54d4f74a0 2024-05-10T11:23:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94d2000700 10 client.26771 handle_command_reply: tid=0 2024-05-10T11:23:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94f9c46b80 1 client.26771 shutdown 2024-05-10T11:23:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94f9c46b80 1 -- 172.21.15.27:0/4098348724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d4789e0 msgr2=0x55d54d4858e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d4789e0 0x55d54d4858e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x55d54d48f7a0 tx=0x7f94d4004750 comp rx=0 tx=0).stop 2024-05-10T11:23:37.751 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:23:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94f9c46b80 1 -- 172.21.15.27:0/4098348724 shutdown_connections 2024-05-10T11:23:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55d54d4f74a0 0x55d54d4f98c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d54d473600 0x55d54d4853a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d54d479300 0x55d54d493b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94f9c46b80 1 --2- 172.21.15.27:0/4098348724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54d4789e0 0x55d54d4858e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94f9c46b80 1 -- 172.21.15.27:0/4098348724 >> 172.21.15.27:0/4098348724 conn(0x55d54d3a1b60 msgr2=0x55d54d479780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94d2000700 0 client.26771 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94f9c46b80 1 -- 172.21.15.27:0/4098348724 shutdown_connections 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94f9c46b80 1 -- 172.21.15.27:0/4098348724 wait complete. 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.749+0000 7f94f9c46b80 20 client.26771 trim_cache size 0 max 16384 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.750+0000 7f94edc00700 1 -- 172.21.15.27:0/2484552012 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f94cc0b2a70 msgr2=0x7f94cc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.750+0000 7f94edc00700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f94cc0b2a70 0x7f94cc0b4f30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55d54d473fc0 tx=0x7f94e000b700 comp rx=0 tx=0).stop 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.750+0000 7f94edc00700 1 -- 172.21.15.27:0/2484552012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94e81055d0 msgr2=0x7f94e81a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.750+0000 7f94edc00700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94e81055d0 0x7f94e81a1840 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f94d800bf20 tx=0x7f94d800dc30 comp rx=0 tx=0).stop 2024-05-10T11:23:37.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.750+0000 7f94edc00700 1 -- 172.21.15.27:0/2484552012 shutdown_connections 2024-05-10T11:23:37.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.750+0000 7f94edc00700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f94cc0b2a70 0x7f94cc0b4f30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.750+0000 7f94edc00700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e810b680 0x7f94e819ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.750+0000 7f94edc00700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94e81055d0 0x7f94e81a1840 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.750+0000 7f94edc00700 1 --2- 172.21.15.27:0/2484552012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e8105f90 0x7f94e81a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:37.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.750+0000 7f94edc00700 1 -- 172.21.15.27:0/2484552012 >> 172.21.15.27:0/2484552012 conn(0x7f94e8101080 msgr2=0x7f94e81026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:37.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.751+0000 7f94edc00700 1 -- 172.21.15.27:0/2484552012 shutdown_connections 2024-05-10T11:23:37.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:37.751+0000 7f94edc00700 1 -- 172.21.15.27:0/2484552012 wait complete. 2024-05-10T11:23:38.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:23:38.719 INFO:tasks.workunit.client.0.smithi027.stdout: 1 627411 6.51 MB/sec execute 383 sec latency 49.513 ms 2024-05-10T11:23:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:23:39.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:38 smithi076 ceph-mon[34152]: pgmap v323: 129 pgs: 129 active+clean; 474 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 136 op/s 2024-05-10T11:23:39.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:38 smithi189 ceph-mon[34842]: pgmap v323: 129 pgs: 129 active+clean; 474 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 136 op/s 2024-05-10T11:23:39.311 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:38 smithi027 ceph-mon[30054]: pgmap v323: 129 pgs: 129 active+clean; 474 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 136 op/s 2024-05-10T11:23:39.720 INFO:tasks.workunit.client.0.smithi027.stdout: 1 628921 6.53 MB/sec execute 384 sec latency 21.861 ms 2024-05-10T11:23:40.720 INFO:tasks.workunit.client.0.smithi027.stdout: 1 629804 6.52 MB/sec execute 385 sec latency 50.534 ms 2024-05-10T11:23:41.174 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:40 smithi027 ceph-mon[30054]: pgmap v324: 129 pgs: 129 active+clean; 481 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-05-10T11:23:41.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:40 smithi076 ceph-mon[34152]: pgmap v324: 129 pgs: 129 active+clean; 481 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-05-10T11:23:41.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:40 smithi189 ceph-mon[34842]: pgmap v324: 129 pgs: 129 active+clean; 481 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-05-10T11:23:41.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.679+0000 7f87ca000700 1 -- 172.21.15.27:0/3877299384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c40756d0 msgr2=0x7f87c4075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:41.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.679+0000 7f87ca000700 1 --2- 172.21.15.27:0/3877299384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c40756d0 0x7f87c4075ab0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009990 tx=0x7f87b4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.679+0000 7f87ca000700 1 -- 172.21.15.27:0/3877299384 shutdown_connections 2024-05-10T11:23:41.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.679+0000 7f87ca000700 1 --2- 172.21.15.27:0/3877299384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87c410ff00 0x7f87c41122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.679+0000 7f87ca000700 1 --2- 172.21.15.27:0/3877299384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c40756d0 0x7f87c4075ab0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.679+0000 7f87ca000700 1 --2- 172.21.15.27:0/3877299384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c4075ff0 0x7f87c410f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.679+0000 7f87ca000700 1 -- 172.21.15.27:0/3877299384 >> 172.21.15.27:0/3877299384 conn(0x7f87c407a580 msgr2=0x7f87c407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:41.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.679+0000 7f87ca000700 1 -- 172.21.15.27:0/3877299384 shutdown_connections 2024-05-10T11:23:41.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.679+0000 7f87ca000700 1 -- 172.21.15.27:0/3877299384 wait complete. 2024-05-10T11:23:41.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.681+0000 7f87ca000700 1 Processor -- start 2024-05-10T11:23:41.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.681+0000 7f87ca000700 1 -- start start 2024-05-10T11:23:41.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.681+0000 7f87ca000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87c40756d0 0x7f87c4103620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.681+0000 7f87ca000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c4075ff0 0x7f87c4103b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.681+0000 7f87ca000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c410ff00 0x7f87c41040a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.681+0000 7f87ca000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41158e0 con 0x7f87c410ff00 2024-05-10T11:23:41.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.681+0000 7f87ca000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f87c4115760 con 0x7f87c40756d0 2024-05-10T11:23:41.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.681+0000 7f87ca000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87c4115a60 con 0x7f87c4075ff0 2024-05-10T11:23:41.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.682+0000 7f87c3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87c40756d0 0x7f87c4103620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.682+0000 7f87baa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c4075ff0 0x7f87c4103b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.682+0000 7f87c3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87c40756d0 0x7f87c4103620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42922/0 (socket says 172.21.15.27:42922) 2024-05-10T11:23:41.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.682+0000 7f87c3400700 1 -- 172.21.15.27:0/10064411 learned_addr learned my addr 172.21.15.27:0/10064411 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:41.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.682+0000 7f87c3e00700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c410ff00 0x7f87c41040a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.682+0000 7f87c3400700 1 -- 172.21.15.27:0/10064411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c4075ff0 msgr2=0x7f87c4103b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:41.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.682+0000 7f87c3400700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c4075ff0 0x7f87c4103b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.682+0000 7f87c3400700 1 -- 172.21.15.27:0/10064411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c410ff00 msgr2=0x7f87c41040a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:41.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.682+0000 7f87c3400700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c410ff00 0x7f87c41040a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.682+0000 7f87c3400700 1 -- 172.21.15.27:0/10064411 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c41ac480 con 0x7f87c40756d0 2024-05-10T11:23:41.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.683+0000 7f87c3400700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87c40756d0 0x7f87c4103620 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f87b4000c00 tx=0x7f87b40057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:41.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.683+0000 7f87c0c00700 1 -- 172.21.15.27:0/10064411 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4027070 con 0x7f87c40756d0 2024-05-10T11:23:41.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.683+0000 7f87c0c00700 1 -- 172.21.15.27:0/10064411 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87b4022470 con 0x7f87c40756d0 2024-05-10T11:23:41.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.683+0000 7f87ca000700 1 -- 172.21.15.27:0/10064411 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87c41ac710 con 0x7f87c40756d0 2024-05-10T11:23:41.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.683+0000 7f87c0c00700 1 -- 172.21.15.27:0/10064411 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4017650 con 0x7f87c40756d0 2024-05-10T11:23:41.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.683+0000 7f87ca000700 1 -- 172.21.15.27:0/10064411 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87c41acc40 con 0x7f87c40756d0 2024-05-10T11:23:41.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.687+0000 7f87c0c00700 1 -- 172.21.15.27:0/10064411 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f87b402a070 con 0x7f87c40756d0 2024-05-10T11:23:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.688+0000 7f87d4974b80 1 Processor -- start 2024-05-10T11:23:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.688+0000 7f87c0c00700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f87a80b2aa0 0x7f87a80b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.689+0000 7f87d4974b80 1 -- start start 2024-05-10T11:23:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.689+0000 7f87c0c00700 1 -- 172.21.15.27:0/10064411 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f87b40e37b0 con 0x7f87c40756d0 2024-05-10T11:23:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.689+0000 7f87c3e00700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f87a80b2aa0 0x7f87a80b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.689+0000 7f87d4974b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a22fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.689+0000 7f87d4974b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a234e0 0x555a06a1d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.689+0000 7f87d4974b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a1d8f0 0x555a06a1dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.689+0000 7f87d4974b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555a0696fa20 con 0x555a06a234e0 2024-05-10T11:23:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.689+0000 7f87d4974b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x555a0690d760 con 0x555a06a1d8f0 2024-05-10T11:23:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.689+0000 7f87d4974b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555a0696b9e0 con 0x555a06a22bc0 2024-05-10T11:23:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.689+0000 7f87c3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a22fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.690+0000 7f87c3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a22fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:45266/0 (socket says 172.21.15.27:45266) 2024-05-10T11:23:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.690+0000 7f87c3400700 1 -- 172.21.15.27:0/2948822151 learned_addr learned my addr 172.21.15.27:0/2948822151 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.690+0000 7f87baa00700 1 --2- 172.21.15.27:0/2948822151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a234e0 0x555a06a1d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.690+0000 7f87c3e00700 1 --2- 172.21.15.27:0/2948822151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a1d8f0 0x555a06a1dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.690+0000 7f87c3400700 1 -- 172.21.15.27:0/2948822151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a1d8f0 msgr2=0x555a06a1dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.690+0000 7f87c3400700 1 --2- 172.21.15.27:0/2948822151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a1d8f0 0x555a06a1dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.690+0000 7f87c3400700 1 -- 172.21.15.27:0/2948822151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a234e0 msgr2=0x555a06a1d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:41.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.690+0000 7f87c3400700 1 --2- 172.21.15.27:0/2948822151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a234e0 0x555a06a1d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.690+0000 7f87c3400700 1 -- 172.21.15.27:0/2948822151 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a068bc8d0 con 0x555a06a22bc0 2024-05-10T11:23:41.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.690+0000 7f87c3e00700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f87a80b2aa0 0x7f87a80b4f60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009da0 tx=0x7f87bc009920 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:41.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.690+0000 7f87c3400700 1 --2- 172.21.15.27:0/2948822151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a22fa0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f87b40aa390 tx=0x7f87b40aa4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2b3b1939bd293a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:41.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.691+0000 7f87b8c00700 1 -- 172.21.15.27:0/2948822151 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b40aad90 con 0x555a06a22bc0 2024-05-10T11:23:41.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.691+0000 7f87b8c00700 1 -- 172.21.15.27:0/2948822151 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87b40aa690 con 0x555a06a22bc0 2024-05-10T11:23:41.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.691+0000 7f87b8c00700 1 -- 172.21.15.27:0/2948822151 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b40aa8f0 con 0x555a06a22bc0 2024-05-10T11:23:41.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.692+0000 7f87d4974b80 1 -- 172.21.15.27:0/2948822151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 msgr2=0x555a06a22fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:41.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.692+0000 7f87d4974b80 1 --2- 172.21.15.27:0/2948822151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a22fa0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f87b40aa390 tx=0x7f87b40aa4e0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.692+0000 7f87d4974b80 1 -- 172.21.15.27:0/2948822151 shutdown_connections 2024-05-10T11:23:41.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.692+0000 7f87d4974b80 1 --2- 172.21.15.27:0/2948822151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a22fa0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.692+0000 7f87d4974b80 1 --2- 172.21.15.27:0/2948822151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a1d8f0 0x555a06a1dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.692+0000 7f87d4974b80 1 --2- 172.21.15.27:0/2948822151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a234e0 0x555a06a1d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.692+0000 7f87d4974b80 1 -- 172.21.15.27:0/2948822151 >> 172.21.15.27:0/2948822151 conn(0x555a069956d0 msgr2=0x555a06a17b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.692+0000 7f87d4974b80 1 -- 172.21.15.27:0/2948822151 shutdown_connections 2024-05-10T11:23:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.692+0000 7f87d4974b80 1 -- 172.21.15.27:0/2948822151 wait complete. 2024-05-10T11:23:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.692+0000 7f87d4974b80 1 -- start start 2024-05-10T11:23:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.693+0000 7f87d4974b80 1 Processor -- start 2024-05-10T11:23:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.693+0000 7f87d4974b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a1d8f0 0x555a06a29250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.693+0000 7f87d4974b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a22bc0 0x555a06a265f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.693+0000 7f87d4974b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a234e0 0x555a06a26b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.693+0000 7f87d4974b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555a06a29ce0 con 0x555a06a22bc0 2024-05-10T11:23:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.693+0000 7f87d4974b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x555a068d5b20 con 0x555a06a234e0 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.693+0000 7f87d4974b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555a06a29e60 con 0x555a06a1d8f0 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.693+0000 7f87d4974b80 10 client.0 _ll_register_callbacks cb 0x555a06916c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.693+0000 7f87c3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a1d8f0 0x555a06a29250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.694+0000 7f87c3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a1d8f0 0x555a06a29250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:45272/0 (socket says 172.21.15.27:45272) 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.694+0000 7f87c3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a234e0 0x555a06a26b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.694+0000 7f87c3400700 1 -- 172.21.15.27:0/2079996809 learned_addr learned my addr 172.21.15.27:0/2079996809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.694+0000 7f87baa00700 1 --2- 172.21.15.27:0/2079996809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a22bc0 0x555a06a265f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.694+0000 7f87c3400700 1 -- 172.21.15.27:0/2079996809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a234e0 msgr2=0x555a06a26b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.694+0000 7f87c3400700 1 --2- 172.21.15.27:0/2079996809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a234e0 0x555a06a26b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.694+0000 7f87c3400700 1 -- 172.21.15.27:0/2079996809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a22bc0 msgr2=0x555a06a265f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.694+0000 7f87c3400700 1 --2- 172.21.15.27:0/2079996809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a22bc0 0x555a06a265f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.694+0000 7f87c3400700 1 -- 172.21.15.27:0/2079996809 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a06935420 con 0x555a06a1d8f0 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.694+0000 7f87c3400700 1 --2- 172.21.15.27:0/2079996809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a1d8f0 0x555a06a29250 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f87b4006010 tx=0x7f87b4022bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a5e8691fe2ff8e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.694+0000 7f87b1600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.694+0000 7f87b1600700 1 -- 172.21.15.27:0/2079996809 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b40c3070 con 0x555a06a1d8f0 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.695+0000 7f87d4974b80 1 -- 172.21.15.27:0/2079996809 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87b40aa690 con 0x555a06a1d8f0 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.695+0000 7f87b1600700 1 -- 172.21.15.27:0/2079996809 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f87b40aab00 con 0x555a06a1d8f0 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.695+0000 7f87b1600700 1 -- 172.21.15.27:0/2079996809 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f87b40aea40 con 0x555a06a1d8f0 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.695+0000 7f87d4974b80 10 client.26773 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.695+0000 7f87d4974b80 1 -- 172.21.15.27:0/2079996809 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555a06a29ce0 con 0x555a06a1d8f0 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.695+0000 7f87b1600700 1 -- 172.21.15.27:0/2079996809 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87b40aa690 con 0x555a06a1d8f0 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.696+0000 7f87b1600700 1 -- 172.21.15.27:0/2079996809 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f87b4090bb0 con 0x555a06a1d8f0 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.696+0000 7f87d4974b80 10 client.26773 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.696+0000 7f87d4974b80 10 client.26773 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.696+0000 7f87d4974b80 20 client.26773 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.696+0000 7f87d4974b80 1 --2- 172.21.15.27:0/2079996809 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x555a06a368c0 0x555a06a38cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.696+0000 7f87d4974b80 4 client.26773 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:23:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.696+0000 7f87d4974b80 1 -- 172.21.15.27:0/2079996809 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a069ea140 con 0x555a06a368c0 2024-05-10T11:23:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.696+0000 7f87baa00700 1 --2- 172.21.15.27:0/2079996809 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x555a06a368c0 0x555a06a38cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.697+0000 7f87baa00700 1 --2- 172.21.15.27:0/2079996809 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x555a06a368c0 0x555a06a38cb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa3b8a40ccf90c9a server_cookie=c6473f2ac5ee5da8 in_seq=0 out_seq=0 2024-05-10T11:23:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.697+0000 7f87b1600700 10 client.26773 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:23:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.697+0000 7f87b1600700 1 -- 172.21.15.27:0/2079996809 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555a069ea140 con 0x555a06a368c0 2024-05-10T11:23:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.697+0000 7f87b1600700 10 client.26773 handle_command_reply: tid=0 2024-05-10T11:23:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.697+0000 7f87d4974b80 1 client.26773 shutdown 2024-05-10T11:23:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.698+0000 7f87d4974b80 1 -- 172.21.15.27:0/2079996809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a1d8f0 msgr2=0x555a06a29250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.698+0000 7f87d4974b80 1 --2- 172.21.15.27:0/2079996809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a1d8f0 0x555a06a29250 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f87b4006010 tx=0x7f87b4022bb0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.698+0000 7f87d4974b80 1 -- 172.21.15.27:0/2079996809 shutdown_connections 2024-05-10T11:23:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.698+0000 7f87d4974b80 1 --2- 172.21.15.27:0/2079996809 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x555a06a368c0 0x555a06a38cb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.698+0000 7f87d4974b80 1 --2- 172.21.15.27:0/2079996809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a1d8f0 0x555a06a29250 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.698+0000 7f87d4974b80 1 --2- 172.21.15.27:0/2079996809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a234e0 0x555a06a26b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.698+0000 7f87d4974b80 1 --2- 172.21.15.27:0/2079996809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a22bc0 0x555a06a265f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.698+0000 7f87d4974b80 1 -- 172.21.15.27:0/2079996809 >> 172.21.15.27:0/2079996809 conn(0x555a069956d0 msgr2=0x555a06a17190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:41.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.698+0000 7f87b1600700 0 client.26773 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:23:41.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.698+0000 7f87d4974b80 1 -- 172.21.15.27:0/2079996809 shutdown_connections 2024-05-10T11:23:41.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.698+0000 7f87d4974b80 1 -- 172.21.15.27:0/2079996809 wait complete. 2024-05-10T11:23:41.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.698+0000 7f87d4974b80 20 client.26773 trim_cache size 0 max 16384 2024-05-10T11:23:41.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.717+0000 7f87d4974b80 1 Processor -- start 2024-05-10T11:23:41.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.717+0000 7f87d4974b80 1 -- start start 2024-05-10T11:23:41.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.717+0000 7f87d4974b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a1d8f0 0x555a06a1d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.717+0000 7f87d4974b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a255b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.717+0000 7f87d4974b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a234e0 0x555a06a32d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87d4974b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41b5d40 con 0x555a06a234e0 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87d4974b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41b7880 con 0x555a06a1d8f0 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87d4974b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41b6410 con 0x555a06a22bc0 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87c3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a1d8f0 0x555a06a1d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87baa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a255b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87c3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a1d8f0 0x555a06a1d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:42946/0 (socket says 172.21.15.27:42946) 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87c3400700 1 -- 172.21.15.27:0/1063979487 learned_addr learned my addr 172.21.15.27:0/1063979487 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87c3e00700 1 --2- 172.21.15.27:0/1063979487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a234e0 0x555a06a32d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87c3400700 1 -- 172.21.15.27:0/1063979487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 msgr2=0x555a06a255b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87c3400700 1 --2- 172.21.15.27:0/1063979487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a255b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87c3400700 1 -- 172.21.15.27:0/1063979487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a234e0 msgr2=0x555a06a32d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:41.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87c3400700 1 --2- 172.21.15.27:0/1063979487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a234e0 0x555a06a32d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87c3400700 1 -- 172.21.15.27:0/1063979487 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a06a12ab0 con 0x555a06a1d8f0 2024-05-10T11:23:41.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.718+0000 7f87c3e00700 1 --2- 172.21.15.27:0/1063979487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a234e0 0x555a06a32d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:23:41.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.719+0000 7f87c3400700 1 --2- 172.21.15.27:0/1063979487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a1d8f0 0x555a06a1d2d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f87b40cd0d0 tx=0x7f87b4005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cba6fa5e5065da8e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:41.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.719+0000 7f87b3e00700 1 -- 172.21.15.27:0/1063979487 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b40c3070 con 0x555a06a1d8f0 2024-05-10T11:23:41.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.719+0000 7f87b3e00700 1 -- 172.21.15.27:0/1063979487 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87b40aa690 con 0x555a06a1d8f0 2024-05-10T11:23:41.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.719+0000 7f87b3e00700 1 -- 172.21.15.27:0/1063979487 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4090bb0 con 0x555a06a1d8f0 2024-05-10T11:23:41.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 631224 6.51 MB/sec execute 386 sec latency 22.706 ms 2024-05-10T11:23:41.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.720+0000 7f87d4974b80 1 -- 172.21.15.27:0/1063979487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a1d8f0 msgr2=0x555a06a1d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:41.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.720+0000 7f87d4974b80 1 --2- 172.21.15.27:0/1063979487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a1d8f0 0x555a06a1d2d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f87b40cd0d0 tx=0x7f87b4005f10 comp rx=0 tx=0).stop 2024-05-10T11:23:41.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.720+0000 7f87d4974b80 1 -- 172.21.15.27:0/1063979487 shutdown_connections 2024-05-10T11:23:41.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.720+0000 7f87d4974b80 1 --2- 172.21.15.27:0/1063979487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a234e0 0x555a06a32d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.720+0000 7f87d4974b80 1 --2- 172.21.15.27:0/1063979487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a1d8f0 0x555a06a1d2d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.720+0000 7f87d4974b80 1 --2- 172.21.15.27:0/1063979487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a255b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.720+0000 7f87d4974b80 1 -- 172.21.15.27:0/1063979487 >> 172.21.15.27:0/1063979487 conn(0x555a069956d0 msgr2=0x555a06a21740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:41.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.720+0000 7f87d4974b80 1 -- 172.21.15.27:0/1063979487 shutdown_connections 2024-05-10T11:23:41.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.720+0000 7f87d4974b80 1 -- 172.21.15.27:0/1063979487 wait complete. 2024-05-10T11:23:41.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.720+0000 7f87d4974b80 1 -- start start 2024-05-10T11:23:41.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.721+0000 7f87d4974b80 1 Processor -- start 2024-05-10T11:23:41.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.721+0000 7f87d4974b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a1d8f0 0x555a06a2f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.721+0000 7f87d4974b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a2f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.721+0000 7f87d4974b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a234e0 0x555a06a3d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.721+0000 7f87d4974b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41b71b0 con 0x555a06a1d8f0 2024-05-10T11:23:41.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.721+0000 7f87d4974b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41b6ae0 con 0x555a06a234e0 2024-05-10T11:23:41.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.721+0000 7f87d4974b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41b3d00 con 0x555a06a22bc0 2024-05-10T11:23:41.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.721+0000 7f87d4974b80 10 client.0 _ll_register_callbacks cb 0x555a06916c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:41.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.722+0000 7f87baa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a2f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.722+0000 7f87baa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a2f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:45280/0 (socket says 172.21.15.27:45280) 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.722+0000 7f87baa00700 1 -- 172.21.15.27:0/3328404741 learned_addr learned my addr 172.21.15.27:0/3328404741 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.722+0000 7f87c3e00700 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a234e0 0x555a06a3d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.722+0000 7f87c3400700 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a1d8f0 0x555a06a2f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.722+0000 7f87baa00700 1 -- 172.21.15.27:0/3328404741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a234e0 msgr2=0x555a06a3d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.722+0000 7f87baa00700 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a234e0 0x555a06a3d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.722+0000 7f87baa00700 1 -- 172.21.15.27:0/3328404741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a1d8f0 msgr2=0x555a06a2f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.722+0000 7f87baa00700 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a1d8f0 0x555a06a2f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.722+0000 7f87baa00700 1 -- 172.21.15.27:0/3328404741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a06a3dec0 con 0x555a06a22bc0 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.722+0000 7f87c3e00700 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a234e0 0x555a06a3d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.723+0000 7f87baa00700 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a2f7d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x555a06a238e0 tx=0x7f87ac011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5df0c4355c32fd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.723+0000 7f87b1600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.723+0000 7f87b1600700 1 -- 172.21.15.27:0/3328404741 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ac020070 con 0x555a06a22bc0 2024-05-10T11:23:41.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.723+0000 7f87b1600700 1 -- 172.21.15.27:0/3328404741 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f87ac004230 con 0x555a06a22bc0 2024-05-10T11:23:41.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.723+0000 7f87d4974b80 1 -- 172.21.15.27:0/3328404741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87b40aa690 con 0x555a06a22bc0 2024-05-10T11:23:41.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.723+0000 7f87b1600700 1 -- 172.21.15.27:0/3328404741 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f87ac02c050 con 0x555a06a22bc0 2024-05-10T11:23:41.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.723+0000 7f87d4974b80 10 client.26779 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:41.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.724+0000 7f87d4974b80 1 -- 172.21.15.27:0/3328404741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555a06aa0d50 con 0x555a06a22bc0 2024-05-10T11:23:41.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.724+0000 7f87b1600700 1 -- 172.21.15.27:0/3328404741 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87b40aa690 con 0x555a06a22bc0 2024-05-10T11:23:41.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.724+0000 7f87b1600700 1 -- 172.21.15.27:0/3328404741 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f87ac003680 con 0x555a06a22bc0 2024-05-10T11:23:41.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.724+0000 7f87d4974b80 10 client.26779 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:41.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.724+0000 7f87d4974b80 10 client.26779 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:23:41.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.724+0000 7f87d4974b80 20 client.26779 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:41.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.724+0000 7f87d4974b80 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x555a06aa1440 0x555a06aa3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:41.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.724+0000 7f87d4974b80 4 client.26779 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:23:41.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.724+0000 7f87d4974b80 1 -- 172.21.15.27:0/3328404741 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555a0697d400 con 0x555a06aa1440 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.725+0000 7f87c3400700 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x555a06aa1440 0x555a06aa3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.725+0000 7f87c3400700 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x555a06aa1440 0x555a06aa3860 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea955a013c3e4a0c server_cookie=c19eee3e4bddef04 in_seq=0 out_seq=0 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.725+0000 7f87b1600700 10 client.26779 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.725+0000 7f87b1600700 1 -- 172.21.15.27:0/3328404741 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555a0697d400 con 0x555a06aa1440 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.725+0000 7f87b1600700 10 client.26779 handle_command_reply: tid=0 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.725+0000 7f87d4974b80 1 client.26779 shutdown 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.726+0000 7f87d4974b80 1 -- 172.21.15.27:0/3328404741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 msgr2=0x555a06a2f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.726+0000 7f87d4974b80 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a2f7d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x555a06a238e0 tx=0x7f87ac011720 comp rx=0 tx=0).stop 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.726+0000 7f87d4974b80 1 -- 172.21.15.27:0/3328404741 shutdown_connections 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.726+0000 7f87d4974b80 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555a06a234e0 0x555a06a3d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.726+0000 7f87d4974b80 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a06a1d8f0 0x555a06a2f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.726+0000 7f87d4974b80 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x555a06aa1440 0x555a06aa3860 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.726+0000 7f87d4974b80 1 --2- 172.21.15.27:0/3328404741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a06a22bc0 0x555a06a2f7d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.726+0000 7f87b1600700 0 client.26779 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:23:41.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.726+0000 7f87d4974b80 1 -- 172.21.15.27:0/3328404741 >> 172.21.15.27:0/3328404741 conn(0x555a069956d0 msgr2=0x555a06a21740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:41.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.726+0000 7f87d4974b80 1 -- 172.21.15.27:0/3328404741 shutdown_connections 2024-05-10T11:23:41.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.726+0000 7f87d4974b80 1 -- 172.21.15.27:0/3328404741 wait complete. 2024-05-10T11:23:41.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.726+0000 7f87d4974b80 20 client.26779 trim_cache size 0 max 16384 2024-05-10T11:23:41.729 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:23:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.727+0000 7f87ca000700 1 -- 172.21.15.27:0/10064411 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f87a80b2aa0 msgr2=0x7f87a80b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.727+0000 7f87ca000700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f87a80b2aa0 0x7f87a80b4f60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009da0 tx=0x7f87bc009920 comp rx=0 tx=0).stop 2024-05-10T11:23:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.727+0000 7f87ca000700 1 -- 172.21.15.27:0/10064411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87c40756d0 msgr2=0x7f87c4103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.727+0000 7f87ca000700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87c40756d0 0x7f87c4103620 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f87b4000c00 tx=0x7f87b40057c0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.727+0000 7f87ca000700 1 -- 172.21.15.27:0/10064411 shutdown_connections 2024-05-10T11:23:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.727+0000 7f87ca000700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f87a80b2aa0 0x7f87a80b4f60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.727+0000 7f87ca000700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c410ff00 0x7f87c41040a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.727+0000 7f87ca000700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87c40756d0 0x7f87c4103620 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.727+0000 7f87ca000700 1 --2- 172.21.15.27:0/10064411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c4075ff0 0x7f87c4103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.727+0000 7f87ca000700 1 -- 172.21.15.27:0/10064411 >> 172.21.15.27:0/10064411 conn(0x7f87c407a580 msgr2=0x7f87c4100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.727+0000 7f87ca000700 1 -- 172.21.15.27:0/10064411 shutdown_connections 2024-05-10T11:23:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:41.727+0000 7f87ca000700 1 -- 172.21.15.27:0/10064411 wait complete. 2024-05-10T11:23:42.662 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:23:42.720 INFO:tasks.workunit.client.0.smithi027.stdout: 1 632485 6.53 MB/sec execute 387 sec latency 19.012 ms 2024-05-10T11:23:43.148 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:23:43.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:42 smithi076 ceph-mon[34152]: pgmap v325: 129 pgs: 129 active+clean; 481 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-05-10T11:23:43.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:42 smithi189 ceph-mon[34842]: pgmap v325: 129 pgs: 129 active+clean; 481 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-05-10T11:23:43.344 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:42 smithi027 ceph-mon[30054]: pgmap v325: 129 pgs: 129 active+clean; 481 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-05-10T11:23:43.720 INFO:tasks.workunit.client.0.smithi027.stdout: 1 633345 6.51 MB/sec execute 388 sec latency 71.319 ms 2024-05-10T11:23:44.720 INFO:tasks.workunit.client.0.smithi027.stdout: 1 634604 6.51 MB/sec execute 389 sec latency 46.940 ms 2024-05-10T11:23:45.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:44 smithi027 ceph-mon[30054]: pgmap v326: 129 pgs: 129 active+clean; 481 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-05-10T11:23:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:44 smithi076 ceph-mon[34152]: pgmap v326: 129 pgs: 129 active+clean; 481 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-05-10T11:23:45.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:44 smithi189 ceph-mon[34842]: pgmap v326: 129 pgs: 129 active+clean; 481 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-05-10T11:23:45.720 INFO:tasks.workunit.client.0.smithi027.stdout: 1 635963 6.52 MB/sec execute 390 sec latency 21.451 ms 2024-05-10T11:23:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.228+0000 7fa37f000700 1 -- 172.21.15.27:0/937115467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa37806b180 msgr2=0x7fa37806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.228+0000 7fa37f000700 1 --2- 172.21.15.27:0/937115467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa37806b180 0x7fa37806b600 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa3640098f0 tx=0x7fa364009c00 comp rx=0 tx=0).stop 2024-05-10T11:23:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.228+0000 7fa37f000700 1 -- 172.21.15.27:0/937115467 shutdown_connections 2024-05-10T11:23:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.228+0000 7fa37f000700 1 --2- 172.21.15.27:0/937115467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa378109510 0x7fa3781098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.228+0000 7fa37f000700 1 --2- 172.21.15.27:0/937115467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa37806bb40 0x7fa378110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.228+0000 7fa37f000700 1 --2- 172.21.15.27:0/937115467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa37806b180 0x7fa37806b600 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.228+0000 7fa37f000700 1 -- 172.21.15.27:0/937115467 >> 172.21.15.27:0/937115467 conn(0x7fa378079350 msgr2=0x7fa378079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.228+0000 7fa37f000700 1 -- 172.21.15.27:0/937115467 shutdown_connections 2024-05-10T11:23:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.229+0000 7fa37f000700 1 -- 172.21.15.27:0/937115467 wait complete. 2024-05-10T11:23:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.230+0000 7fa37f000700 1 Processor -- start 2024-05-10T11:23:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.230+0000 7fa37f000700 1 -- start start 2024-05-10T11:23:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.230+0000 7fa37f000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37806b180 0x7fa378102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.230+0000 7fa37f000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa37806bb40 0x7fa3781029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.230+0000 7fa37f000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa378109510 0x7fa378102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.230+0000 7fa37f000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa378113690 con 0x7fa378109510 2024-05-10T11:23:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.230+0000 7fa37f000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa378113510 con 0x7fa37806bb40 2024-05-10T11:23:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.230+0000 7fa37f000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa378113810 con 0x7fa37806b180 2024-05-10T11:23:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.231+0000 7fa377e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37806b180 0x7fa378102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.231+0000 7fa377400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa37806bb40 0x7fa3781029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.231+0000 7fa377e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37806b180 0x7fa378102470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37540/0 (socket says 172.21.15.27:37540) 2024-05-10T11:23:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.231+0000 7fa377e00700 1 -- 172.21.15.27:0/187361022 learned_addr learned my addr 172.21.15.27:0/187361022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.231+0000 7fa377400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa37806bb40 0x7fa3781029b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52722/0 (socket says 172.21.15.27:52722) 2024-05-10T11:23:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.231+0000 7fa37d000700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa378109510 0x7fa378102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.231+0000 7fa377400700 1 -- 172.21.15.27:0/187361022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37806b180 msgr2=0x7fa378102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.231+0000 7fa377400700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37806b180 0x7fa378102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.231+0000 7fa377400700 1 -- 172.21.15.27:0/187361022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa378109510 msgr2=0x7fa378102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.231+0000 7fa377400700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa378109510 0x7fa378102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.231+0000 7fa377400700 1 -- 172.21.15.27:0/187361022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3781a8170 con 0x7fa37806bb40 2024-05-10T11:23:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.231+0000 7fa377e00700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37806b180 0x7fa378102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.232+0000 7fa377400700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa37806bb40 0x7fa3781029b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa36400c010 tx=0x7fa36400fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.232+0000 7fa374c00700 1 -- 172.21.15.27:0/187361022 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3640050c0 con 0x7fa37806bb40 2024-05-10T11:23:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.232+0000 7fa374c00700 1 -- 172.21.15.27:0/187361022 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa36400b560 con 0x7fa37806bb40 2024-05-10T11:23:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.232+0000 7fa374c00700 1 -- 172.21.15.27:0/187361022 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa364003a00 con 0x7fa37806bb40 2024-05-10T11:23:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.232+0000 7fa37f000700 1 -- 172.21.15.27:0/187361022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3781a8400 con 0x7fa37806bb40 2024-05-10T11:23:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.232+0000 7fa37f000700 1 -- 172.21.15.27:0/187361022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3781a8800 con 0x7fa37806bb40 2024-05-10T11:23:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.238+0000 7fa3898d2b80 1 Processor -- start 2024-05-10T11:23:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.238+0000 7fa3898d2b80 1 -- start start 2024-05-10T11:23:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.238+0000 7fa3898d2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479adc060 0x562479adc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa3898d2b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479ae0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa3898d2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479ae09d0 0x562479ae0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa3898d2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562479a2ea20 con 0x562479ae09d0 2024-05-10T11:23:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa3898d2b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5624799cc760 con 0x562479adc980 2024-05-10T11:23:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa3898d2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562479a2a9e0 con 0x562479adc060 2024-05-10T11:23:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa37d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479adc060 0x562479adc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa37d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479adc060 0x562479adc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37544/0 (socket says 172.21.15.27:37544) 2024-05-10T11:23:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa37d000700 1 -- 172.21.15.27:0/1743210681 learned_addr learned my addr 172.21.15.27:0/1743210681 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa374c00700 1 -- 172.21.15.27:0/187361022 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa36400b710 con 0x7fa37806bb40 2024-05-10T11:23:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa377e00700 1 --2- 172.21.15.27:0/1743210681 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479ae0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa37d000700 1 -- 172.21.15.27:0/1743210681 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 msgr2=0x562479ae0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa37d000700 1 --2- 172.21.15.27:0/1743210681 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479ae0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa37d000700 1 -- 172.21.15.27:0/1743210681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479ae09d0 msgr2=0x562479ae0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.239+0000 7fa37d000700 1 --2- 172.21.15.27:0/1743210681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479ae09d0 0x562479ae0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.240+0000 7fa37d000700 1 -- 172.21.15.27:0/1743210681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56247997b8d0 con 0x562479adc060 2024-05-10T11:23:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.240+0000 7fa37d000700 1 --2- 172.21.15.27:0/1743210681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479adc060 0x562479adc440 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa3700093a0 tx=0x7fa3700099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddc363bf168947d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.240+0000 7fa36d600700 1 -- 172.21.15.27:0/1743210681 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa370003c00 con 0x562479adc060 2024-05-10T11:23:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.240+0000 7fa36d600700 1 -- 172.21.15.27:0/1743210681 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa370003da0 con 0x562479adc060 2024-05-10T11:23:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.240+0000 7fa36d600700 1 -- 172.21.15.27:0/1743210681 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa370017990 con 0x562479adc060 2024-05-10T11:23:46.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.241+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1743210681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479adc060 msgr2=0x562479adc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:46.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.241+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/1743210681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479adc060 0x562479adc440 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa3700093a0 tx=0x7fa3700099e0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.241+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1743210681 shutdown_connections 2024-05-10T11:23:46.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.241+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/1743210681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479adc060 0x562479adc440 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.241+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/1743210681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479ae09d0 0x562479ae0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.241+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/1743210681 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479ae0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.241+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1743210681 >> 172.21.15.27:0/1743210681 conn(0x562479a0ab60 msgr2=0x562479ad67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:46.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.242+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1743210681 shutdown_connections 2024-05-10T11:23:46.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.242+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1743210681 wait complete. 2024-05-10T11:23:46.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.242+0000 7fa374c00700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa35c0b2a00 0x7fa35c0b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.242+0000 7fa374c00700 1 -- 172.21.15.27:0/187361022 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7fa3640e37d0 con 0x7fa37806bb40 2024-05-10T11:23:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.242+0000 7fa3898d2b80 1 -- start start 2024-05-10T11:23:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.242+0000 7fa377e00700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa35c0b2a00 0x7fa35c0b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.243+0000 7fa3898d2b80 1 Processor -- start 2024-05-10T11:23:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.243+0000 7fa3898d2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479adc060 0x562479ae7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.243+0000 7fa377e00700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa35c0b2a00 0x7fa35c0b4ec0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa368011630 tx=0x7fa368009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.243+0000 7fa3898d2b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479ae5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.243+0000 7fa3898d2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479ae09d0 0x562479ae56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.243+0000 7fa3898d2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562479ae8be0 con 0x562479adc060 2024-05-10T11:23:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.243+0000 7fa3898d2b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562479994b20 con 0x562479adc980 2024-05-10T11:23:46.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.243+0000 7fa3898d2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562479ae8d60 con 0x562479ae09d0 2024-05-10T11:23:46.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.243+0000 7fa3898d2b80 10 client.0 _ll_register_callbacks cb 0x5624799d5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:46.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.244+0000 7fa377400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479ae5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.244+0000 7fa377400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479ae5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52736/0 (socket says 172.21.15.27:52736) 2024-05-10T11:23:46.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.244+0000 7fa377400700 1 -- 172.21.15.27:0/1756173694 learned_addr learned my addr 172.21.15.27:0/1756173694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:46.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.244+0000 7fa377e00700 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479adc060 0x562479ae7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.244+0000 7fa37d000700 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479ae09d0 0x562479ae56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.244+0000 7fa377400700 1 -- 172.21.15.27:0/1756173694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479ae09d0 msgr2=0x562479ae56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.244+0000 7fa377400700 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479ae09d0 0x562479ae56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.244+0000 7fa377400700 1 -- 172.21.15.27:0/1756173694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479adc060 msgr2=0x562479ae7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.244+0000 7fa377400700 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479adc060 0x562479ae7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.244+0000 7fa377400700 1 -- 172.21.15.27:0/1756173694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624799f4420 con 0x562479adc980 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.244+0000 7fa377e00700 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479adc060 0x562479ae7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.244+0000 7fa377400700 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479ae5190 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa364000c00 tx=0x7fa3640aa530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65c659856b26e1bb server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.245+0000 7fa362000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.245+0000 7fa362000700 1 -- 172.21.15.27:0/1756173694 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3640aadb0 con 0x562479adc980 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.245+0000 7fa362000700 1 -- 172.21.15.27:0/1756173694 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7fa3640e0d10 con 0x562479adc980 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.245+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1756173694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa370003da0 con 0x562479adc980 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.245+0000 7fa362000700 1 -- 172.21.15.27:0/1756173694 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fa3640aa6b0 con 0x562479adc980 2024-05-10T11:23:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.245+0000 7fa3898d2b80 10 client.26801 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.245+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1756173694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x562479af3410 con 0x562479adc980 2024-05-10T11:23:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.245+0000 7fa362000700 1 -- 172.21.15.27:0/1756173694 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa370003da0 con 0x562479adc980 2024-05-10T11:23:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.246+0000 7fa362000700 1 -- 172.21.15.27:0/1756173694 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fa364090bd0 con 0x562479adc980 2024-05-10T11:23:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.246+0000 7fa3898d2b80 10 client.26801 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.246+0000 7fa3898d2b80 10 client.26801 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:23:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.246+0000 7fa3898d2b80 20 client.26801 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.246+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562479af58e0 0x562479af7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.246+0000 7fa3898d2b80 4 client.26801 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:23:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.246+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1756173694 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562479aa90a0 con 0x562479af58e0 2024-05-10T11:23:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.246+0000 7fa377e00700 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562479af58e0 0x562479af7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.247+0000 7fa377e00700 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562479af58e0 0x562479af7d00 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=199c61816d19cfdb server_cookie=af926172a83083e1 in_seq=0 out_seq=0 2024-05-10T11:23:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.247+0000 7fa362000700 10 client.26801 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa362000700 1 -- 172.21.15.27:0/1756173694 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562479aa90a0 con 0x562479af58e0 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa362000700 10 client.26801 handle_command_reply: tid=0 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa3898d2b80 1 client.26801 shutdown 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1756173694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 msgr2=0x562479ae5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479ae5190 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa364000c00 tx=0x7fa3640aa530 comp rx=0 tx=0).stop 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1756173694 shutdown_connections 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562479af58e0 0x562479af7d00 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479ae09d0 0x562479ae56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479adc060 0x562479ae7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/1756173694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479ae5190 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa362000700 0 client.26801 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1756173694 >> 172.21.15.27:0/1756173694 conn(0x562479a0ab60 msgr2=0x562479ad12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1756173694 shutdown_connections 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.248+0000 7fa3898d2b80 1 -- 172.21.15.27:0/1756173694 wait complete. 2024-05-10T11:23:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.249+0000 7fa3898d2b80 20 client.26801 trim_cache size 0 max 16384 2024-05-10T11:23:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.267+0000 7fa3898d2b80 1 Processor -- start 2024-05-10T11:23:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.267+0000 7fa3898d2b80 1 -- start start 2024-05-10T11:23:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.268+0000 7fa3898d2b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc060 0x562479af1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.268+0000 7fa3898d2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479adc980 0x562479ae1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.268+0000 7fa3898d2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479ae09d0 0x562479ad1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.268+0000 7fa3898d2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781b17b0 con 0x562479adc980 2024-05-10T11:23:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.268+0000 7fa3898d2b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781b32f0 con 0x562479adc060 2024-05-10T11:23:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.268+0000 7fa3898d2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781b1e80 con 0x562479ae09d0 2024-05-10T11:23:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.268+0000 7fa377e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc060 0x562479af1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.268+0000 7fa377e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc060 0x562479af1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52752/0 (socket says 172.21.15.27:52752) 2024-05-10T11:23:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.268+0000 7fa377e00700 1 -- 172.21.15.27:0/3379394406 learned_addr learned my addr 172.21.15.27:0/3379394406 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.268+0000 7fa377400700 1 --2- 172.21.15.27:0/3379394406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479adc980 0x562479ae1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.268+0000 7fa37d000700 1 --2- 172.21.15.27:0/3379394406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479ae09d0 0x562479ad1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.269+0000 7fa377e00700 1 -- 172.21.15.27:0/3379394406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479ae09d0 msgr2=0x562479ad1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.269+0000 7fa377e00700 1 --2- 172.21.15.27:0/3379394406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479ae09d0 0x562479ad1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.269+0000 7fa377e00700 1 -- 172.21.15.27:0/3379394406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479adc980 msgr2=0x562479ae1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.269+0000 7fa377e00700 1 --2- 172.21.15.27:0/3379394406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479adc980 0x562479ae1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.269+0000 7fa377e00700 1 -- 172.21.15.27:0/3379394406 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562479ae3760 con 0x562479adc060 2024-05-10T11:23:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.269+0000 7fa377e00700 1 --2- 172.21.15.27:0/3379394406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc060 0x562479af1ca0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x562479af8540 tx=0x7fa368018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33dc75fdce1bcdeb server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.269+0000 7fa36cc00700 1 -- 172.21.15.27:0/3379394406 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36801cda0 con 0x562479adc060 2024-05-10T11:23:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.269+0000 7fa36cc00700 1 -- 172.21.15.27:0/3379394406 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa368029e50 con 0x562479adc060 2024-05-10T11:23:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.269+0000 7fa36cc00700 1 -- 172.21.15.27:0/3379394406 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368029a50 con 0x562479adc060 2024-05-10T11:23:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.270+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3379394406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc060 msgr2=0x562479af1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.270+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/3379394406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc060 0x562479af1ca0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x562479af8540 tx=0x7fa368018000 comp rx=0 tx=0).stop 2024-05-10T11:23:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.270+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3379394406 shutdown_connections 2024-05-10T11:23:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.270+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/3379394406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479ae09d0 0x562479ad1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.270+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/3379394406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc060 0x562479af1ca0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.270+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/3379394406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479adc980 0x562479ae1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.270+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3379394406 >> 172.21.15.27:0/3379394406 conn(0x562479a0ab60 msgr2=0x562479ad0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.270+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3379394406 shutdown_connections 2024-05-10T11:23:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.270+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3379394406 wait complete. 2024-05-10T11:23:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.271+0000 7fa3898d2b80 1 -- start start 2024-05-10T11:23:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.271+0000 7fa3898d2b80 1 Processor -- start 2024-05-10T11:23:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.272+0000 7fa3898d2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479adc060 0x562479aee200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.272+0000 7fa3898d2b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479aee740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.272+0000 7fa3898d2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479ae09d0 0x562479aeb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.272+0000 7fa3898d2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781b2c20 con 0x562479ae09d0 2024-05-10T11:23:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.272+0000 7fa3898d2b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781b2550 con 0x562479adc980 2024-05-10T11:23:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.272+0000 7fa3898d2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781af770 con 0x562479adc060 2024-05-10T11:23:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.272+0000 7fa3898d2b80 10 client.0 _ll_register_callbacks cb 0x5624799d5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.272+0000 7fa377400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479aee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.272+0000 7fa377e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479adc060 0x562479aee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.272+0000 7fa377400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479aee740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52756/0 (socket says 172.21.15.27:52756) 2024-05-10T11:23:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.272+0000 7fa377400700 1 -- 172.21.15.27:0/3608815506 learned_addr learned my addr 172.21.15.27:0/3608815506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.272+0000 7fa37d000700 1 --2- 172.21.15.27:0/3608815506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479ae09d0 0x562479aeb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.273+0000 7fa377400700 1 -- 172.21.15.27:0/3608815506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479adc060 msgr2=0x562479aee200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.273+0000 7fa377400700 1 --2- 172.21.15.27:0/3608815506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479adc060 0x562479aee200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.274+0000 7fa377400700 1 -- 172.21.15.27:0/3608815506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479ae09d0 msgr2=0x562479aeb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.274+0000 7fa377400700 1 --2- 172.21.15.27:0/3608815506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479ae09d0 0x562479aeb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.274+0000 7fa377400700 1 -- 172.21.15.27:0/3608815506 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562479aeb8a0 con 0x562479adc980 2024-05-10T11:23:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.274+0000 7fa377400700 1 --2- 172.21.15.27:0/3608815506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479aee740 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa3640aaaa0 tx=0x7fa3640095d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3a5ce6b86921729 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.274+0000 7fa362000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:23:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.274+0000 7fa362000700 1 -- 172.21.15.27:0/3608815506 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3640aadb0 con 0x562479adc980 2024-05-10T11:23:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.274+0000 7fa362000700 1 -- 172.21.15.27:0/3608815506 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7fa3640e0d10 con 0x562479adc980 2024-05-10T11:23:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.274+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3608815506 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa368029e50 con 0x562479adc980 2024-05-10T11:23:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.275+0000 7fa362000700 1 -- 172.21.15.27:0/3608815506 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fa3640e0f10 con 0x562479adc980 2024-05-10T11:23:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.275+0000 7fa3898d2b80 10 client.26807 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.275+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3608815506 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x562479b69530 con 0x562479adc980 2024-05-10T11:23:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.275+0000 7fa362000700 1 -- 172.21.15.27:0/3608815506 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa368029e50 con 0x562479adc980 2024-05-10T11:23:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.275+0000 7fa362000700 1 -- 172.21.15.27:0/3608815506 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fa364003cc0 con 0x562479adc980 2024-05-10T11:23:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.275+0000 7fa3898d2b80 10 client.26807 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.275+0000 7fa3898d2b80 10 client.26807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:23:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.275+0000 7fa3898d2b80 20 client.26807 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.275+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/3608815506 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562479b69ab0 0x562479b6bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.275+0000 7fa3898d2b80 4 client.26807 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:23:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.275+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3608815506 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562479a3c400 con 0x562479b69ab0 2024-05-10T11:23:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.276+0000 7fa377e00700 1 --2- 172.21.15.27:0/3608815506 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562479b69ab0 0x562479b6bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.276+0000 7fa377e00700 1 --2- 172.21.15.27:0/3608815506 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562479b69ab0 0x562479b6bed0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b40c55fc5b93d65d server_cookie=ccef74e3c60b96f6 in_seq=0 out_seq=0 2024-05-10T11:23:46.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.276+0000 7fa362000700 10 client.26807 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:46.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa362000700 1 -- 172.21.15.27:0/3608815506 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562479a3c400 con 0x562479b69ab0 2024-05-10T11:23:46.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa362000700 10 client.26807 handle_command_reply: tid=0 2024-05-10T11:23:46.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa3898d2b80 1 client.26807 shutdown 2024-05-10T11:23:46.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3608815506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 msgr2=0x562479aee740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:46.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/3608815506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479aee740 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa3640aaaa0 tx=0x7fa3640095d0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3608815506 shutdown_connections 2024-05-10T11:23:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/3608815506 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562479b69ab0 0x562479b6bed0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa362000700 0 client.26807 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/3608815506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562479adc060 0x562479aee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/3608815506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562479ae09d0 0x562479aeb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa3898d2b80 1 --2- 172.21.15.27:0/3608815506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562479adc980 0x562479aee740 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3608815506 >> 172.21.15.27:0/3608815506 conn(0x562479a0ab60 msgr2=0x562479ad12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3608815506 shutdown_connections 2024-05-10T11:23:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa3898d2b80 1 -- 172.21.15.27:0/3608815506 wait complete. 2024-05-10T11:23:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.277+0000 7fa3898d2b80 20 client.26807 trim_cache size 0 max 16384 2024-05-10T11:23:46.280 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:23:46.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.278+0000 7fa37f000700 1 -- 172.21.15.27:0/187361022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa35c0b2a00 msgr2=0x7fa35c0b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:46.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.278+0000 7fa37f000700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa35c0b2a00 0x7fa35c0b4ec0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa368011630 tx=0x7fa368009260 comp rx=0 tx=0).stop 2024-05-10T11:23:46.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.278+0000 7fa37f000700 1 -- 172.21.15.27:0/187361022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa37806bb40 msgr2=0x7fa3781029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:46.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.278+0000 7fa37f000700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa37806bb40 0x7fa3781029b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa36400c010 tx=0x7fa36400fa00 comp rx=0 tx=0).stop 2024-05-10T11:23:46.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.279+0000 7fa37f000700 1 -- 172.21.15.27:0/187361022 shutdown_connections 2024-05-10T11:23:46.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.279+0000 7fa37f000700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa35c0b2a00 0x7fa35c0b4ec0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.279+0000 7fa37f000700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37806b180 0x7fa378102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.279+0000 7fa37f000700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa378109510 0x7fa378102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.279+0000 7fa37f000700 1 --2- 172.21.15.27:0/187361022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa37806bb40 0x7fa3781029b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:46.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.279+0000 7fa37f000700 1 -- 172.21.15.27:0/187361022 >> 172.21.15.27:0/187361022 conn(0x7fa378079350 msgr2=0x7fa378101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:46.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.279+0000 7fa37f000700 1 -- 172.21.15.27:0/187361022 shutdown_connections 2024-05-10T11:23:46.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:46.279+0000 7fa37f000700 1 -- 172.21.15.27:0/187361022 wait complete. 2024-05-10T11:23:46.720 INFO:tasks.workunit.client.0.smithi027.stdout: 1 636633 6.51 MB/sec execute 391 sec latency 34.318 ms 2024-05-10T11:23:47.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:46 smithi076 ceph-mon[34152]: pgmap v327: 129 pgs: 129 active+clean; 490 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 202 op/s 2024-05-10T11:23:47.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:46 smithi189 ceph-mon[34842]: pgmap v327: 129 pgs: 129 active+clean; 490 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 202 op/s 2024-05-10T11:23:47.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:46 smithi027 ceph-mon[30054]: pgmap v327: 129 pgs: 129 active+clean; 490 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 202 op/s 2024-05-10T11:23:47.720 INFO:tasks.workunit.client.0.smithi027.stdout: 1 637689 6.51 MB/sec execute 392 sec latency 65.059 ms 2024-05-10T11:23:48.513 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:23:48.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:23:48.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 639342 6.52 MB/sec execute 393 sec latency 75.746 ms 2024-05-10T11:23:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:23:49.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:48 smithi076 ceph-mon[34152]: pgmap v328: 129 pgs: 129 active+clean; 490 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 137 op/s 2024-05-10T11:23:49.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:48 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:23:49.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:48 smithi027 ceph-mon[30054]: pgmap v328: 129 pgs: 129 active+clean; 490 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 137 op/s 2024-05-10T11:23:49.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:48 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:23:49.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:48 smithi189 ceph-mon[34842]: pgmap v328: 129 pgs: 129 active+clean; 490 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 137 op/s 2024-05-10T11:23:49.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:48 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:23:49.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 640062 6.51 MB/sec execute 394 sec latency 31.142 ms 2024-05-10T11:23:50.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 641456 6.51 MB/sec execute 395 sec latency 31.803 ms 2024-05-10T11:23:51.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:50 smithi027 ceph-mon[30054]: pgmap v329: 129 pgs: 129 active+clean; 494 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 201 op/s 2024-05-10T11:23:51.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:50 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:51.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:50 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:51.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:50 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:51.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:50 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:51.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:50 smithi189 ceph-mon[34842]: pgmap v329: 129 pgs: 129 active+clean; 494 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 201 op/s 2024-05-10T11:23:51.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:50 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:51.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:50 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:51.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:50 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:51.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:50 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:51.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:50 smithi076 ceph-mon[34152]: pgmap v329: 129 pgs: 129 active+clean; 494 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 201 op/s 2024-05-10T11:23:51.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:50 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:51.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:50 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:51.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:50 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:51.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:50 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:51.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 643117 6.52 MB/sec execute 396 sec latency 22.594 ms 2024-05-10T11:23:52.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:51 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.268 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:51 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.869+0000 7f0358200700 1 -- 172.21.15.27:0/3156086472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0350103eb0 msgr2=0x7f0350110390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.869+0000 7f0358200700 1 --2- 172.21.15.27:0/3156086472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0350103eb0 0x7f0350110390 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f034800b0a0 tx=0x7f034800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.869+0000 7f0358200700 1 -- 172.21.15.27:0/3156086472 shutdown_connections 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.869+0000 7f0358200700 1 --2- 172.21.15.27:0/3156086472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0350103eb0 0x7f0350110390 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.869+0000 7f0358200700 1 --2- 172.21.15.27:0/3156086472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0350109600 0x7f03501099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.869+0000 7f0358200700 1 --2- 172.21.15.27:0/3156086472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03501034f0 0x7f0350103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.869+0000 7f0358200700 1 -- 172.21.15.27:0/3156086472 >> 172.21.15.27:0/3156086472 conn(0x7f035007a650 msgr2=0x7f035007aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.869+0000 7f0358200700 1 -- 172.21.15.27:0/3156086472 shutdown_connections 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.869+0000 7f0358200700 1 -- 172.21.15.27:0/3156086472 wait complete. 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.870+0000 7f0358200700 1 Processor -- start 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.870+0000 7f0358200700 1 -- start start 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0358200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f03501034f0 0x7f035019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0358200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0350103eb0 0x7f035019fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0358200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350109600 0x7f0350199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0358200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f03501136a0 con 0x7f0350109600 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0358200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0350113520 con 0x7f03501034f0 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0358200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0350113820 con 0x7f0350103eb0 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0355800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f03501034f0 0x7f035019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0355800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f03501034f0 0x7f035019f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52782/0 (socket says 172.21.15.27:52782) 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0354e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0350103eb0 0x7f035019fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0355800700 1 -- 172.21.15.27:0/2613199594 learned_addr learned my addr 172.21.15.27:0/2613199594 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0356200700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350109600 0x7f0350199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0355800700 1 -- 172.21.15.27:0/2613199594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0350103eb0 msgr2=0x7f035019fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0355800700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0350103eb0 0x7f035019fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0355800700 1 -- 172.21.15.27:0/2613199594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350109600 msgr2=0x7f0350199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0355800700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350109600 0x7f0350199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0355800700 1 -- 172.21.15.27:0/2613199594 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035019a0a0 con 0x7f03501034f0 2024-05-10T11:23:52.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.871+0000 7f0356200700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350109600 0x7f0350199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.872+0000 7f0355800700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f03501034f0 0x7f035019f730 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f034400e9a0 tx=0x7f034400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.872+0000 7f034e000700 1 -- 172.21.15.27:0/2613199594 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034400cbd0 con 0x7f03501034f0 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.872+0000 7f0358200700 1 -- 172.21.15.27:0/2613199594 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f035019a390 con 0x7f03501034f0 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.872+0000 7f034e000700 1 -- 172.21.15.27:0/2613199594 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f034400cd70 con 0x7f03501034f0 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.872+0000 7f034e000700 1 -- 172.21.15.27:0/2613199594 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344018680 con 0x7f03501034f0 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.872+0000 7f0358200700 1 -- 172.21.15.27:0/2613199594 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03501a6600 con 0x7f03501034f0 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.878+0000 7f034e000700 1 -- 172.21.15.27:0/2613199594 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0344010430 con 0x7f03501034f0 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.878+0000 7f0362b85b80 1 Processor -- start 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.878+0000 7f0362b85b80 1 -- start start 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.879+0000 7f0362b85b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x55942630e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.879+0000 7f0362b85b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55942630ea00 0x559426312510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.879+0000 7f0362b85b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559426312a50 0x559426312eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.879+0000 7f0362b85b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559426260aa0 con 0x55942630e0e0 2024-05-10T11:23:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.879+0000 7f0362b85b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5594261fe7e0 con 0x559426312a50 2024-05-10T11:23:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.879+0000 7f0362b85b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55942625ca60 con 0x55942630ea00 2024-05-10T11:23:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.879+0000 7f0356200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x55942630e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.879+0000 7f0356200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x55942630e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42778/0 (socket says 172.21.15.27:42778) 2024-05-10T11:23:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.879+0000 7f0356200700 1 -- 172.21.15.27:0/1203667564 learned_addr learned my addr 172.21.15.27:0/1203667564 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.879+0000 7f0355800700 1 --2- 172.21.15.27:0/1203667564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55942630ea00 0x559426312510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.879+0000 7f0354e00700 1 --2- 172.21.15.27:0/1203667564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559426312a50 0x559426312eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.880+0000 7f0356200700 1 -- 172.21.15.27:0/1203667564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55942630ea00 msgr2=0x559426312510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.880+0000 7f0356200700 1 --2- 172.21.15.27:0/1203667564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55942630ea00 0x559426312510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.880+0000 7f0356200700 1 -- 172.21.15.27:0/1203667564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559426312a50 msgr2=0x559426312eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.880+0000 7f0356200700 1 --2- 172.21.15.27:0/1203667564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559426312a50 0x559426312eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.880+0000 7f0356200700 1 -- 172.21.15.27:0/1203667564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594261ad950 con 0x55942630e0e0 2024-05-10T11:23:52.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.880+0000 7f0354e00700 1 --2- 172.21.15.27:0/1203667564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559426312a50 0x559426312eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:52.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.880+0000 7f034e000700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f03380b2a70 0x7f03380b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.880+0000 7f0356200700 1 --2- 172.21.15.27:0/1203667564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x55942630e4c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0348015000 tx=0x7f03480094f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1478a62ba822130 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:52.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.880+0000 7f034e000700 1 -- 172.21.15.27:0/2613199594 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f03440e2c90 con 0x7f03501034f0 2024-05-10T11:23:52.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.880+0000 7f0355800700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f03380b2a70 0x7f03380b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.881+0000 7f0342a00700 1 -- 172.21.15.27:0/1203667564 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034800bcb0 con 0x55942630e0e0 2024-05-10T11:23:52.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.881+0000 7f0342a00700 1 -- 172.21.15.27:0/1203667564 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03480074e0 con 0x55942630e0e0 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.881+0000 7f0342a00700 1 -- 172.21.15.27:0/1203667564 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03480052a0 con 0x55942630e0e0 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.881+0000 7f0362b85b80 1 -- 172.21.15.27:0/1203667564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 msgr2=0x55942630e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.881+0000 7f0362b85b80 1 --2- 172.21.15.27:0/1203667564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x55942630e4c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0348015000 tx=0x7f03480094f0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.881+0000 7f0355800700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f03380b2a70 0x7f03380b4f30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f03440a5b60 tx=0x7f03440e6740 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.882+0000 7f0362b85b80 1 -- 172.21.15.27:0/1203667564 shutdown_connections 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.882+0000 7f0362b85b80 1 --2- 172.21.15.27:0/1203667564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559426312a50 0x559426312eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.882+0000 7f0362b85b80 1 --2- 172.21.15.27:0/1203667564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x55942630e4c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.882+0000 7f0362b85b80 1 --2- 172.21.15.27:0/1203667564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55942630ea00 0x559426312510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.882+0000 7f0362b85b80 1 -- 172.21.15.27:0/1203667564 >> 172.21.15.27:0/1203667564 conn(0x55942623cbe0 msgr2=0x559426308820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.882+0000 7f0362b85b80 1 -- 172.21.15.27:0/1203667564 shutdown_connections 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.882+0000 7f0362b85b80 1 -- 172.21.15.27:0/1203667564 wait complete. 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.882+0000 7f0362b85b80 1 -- start start 2024-05-10T11:23:52.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.883+0000 7f0362b85b80 1 Processor -- start 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.883+0000 7f0362b85b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x559426319f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.883+0000 7f0362b85b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x5594263172b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.883+0000 7f0362b85b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559426312a50 0x5594263177f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.883+0000 7f0362b85b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55942631ac00 con 0x55942630e0e0 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.883+0000 7f0362b85b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5594261c6ba0 con 0x55942630ea00 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.883+0000 7f0362b85b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55942631ad80 con 0x559426312a50 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.883+0000 7f0362b85b80 10 client.0 _ll_register_callbacks cb 0x559426207cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0354e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x5594263172b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0354e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x5594263172b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52798/0 (socket says 172.21.15.27:52798) 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0354e00700 1 -- 172.21.15.27:0/2100880785 learned_addr learned my addr 172.21.15.27:0/2100880785 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0356200700 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x559426319f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0354e00700 1 -- 172.21.15.27:0/2100880785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559426312a50 msgr2=0x5594263177f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:52.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0356200700 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559426312a50 0x5594263177f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0354e00700 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559426312a50 0x5594263177f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0354e00700 1 -- 172.21.15.27:0/2100880785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 msgr2=0x559426319f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0354e00700 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x559426319f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0354e00700 1 -- 172.21.15.27:0/2100880785 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594262264a0 con 0x55942630ea00 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0356200700 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559426312a50 0x5594263177f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0354e00700 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x5594263172b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f033c0117c0 tx=0x7f033c011b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=422e219b3c895249 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.884+0000 7f0337400700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.885+0000 7f0337400700 1 -- 172.21.15.27:0/2100880785 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c01f060 con 0x55942630ea00 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.885+0000 7f0337400700 1 -- 172.21.15.27:0/2100880785 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f033c004300 con 0x55942630ea00 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.885+0000 7f0362b85b80 1 -- 172.21.15.27:0/2100880785 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03480074e0 con 0x55942630ea00 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.885+0000 7f0337400700 1 -- 172.21.15.27:0/2100880785 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f033c004500 con 0x55942630ea00 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.885+0000 7f0362b85b80 10 client.26819 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.885+0000 7f0362b85b80 1 -- 172.21.15.27:0/2100880785 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55942631c4b0 con 0x55942630ea00 2024-05-10T11:23:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.885+0000 7f0337400700 1 -- 172.21.15.27:0/2100880785 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03480074e0 con 0x55942630ea00 2024-05-10T11:23:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.886+0000 7f0337400700 1 -- 172.21.15.27:0/2100880785 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f033c019030 con 0x55942630ea00 2024-05-10T11:23:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.886+0000 7f0362b85b80 10 client.26819 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.886+0000 7f0362b85b80 10 client.26819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:23:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.886+0000 7f0362b85b80 20 client.26819 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.886+0000 7f0362b85b80 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55942631eaf0 0x55942631ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.886+0000 7f0362b85b80 4 client.26819 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:23:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.886+0000 7f0362b85b80 1 -- 172.21.15.27:0/2100880785 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594262db120 con 0x55942631eaf0 2024-05-10T11:23:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.886+0000 7f0356200700 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55942631eaf0 0x55942631ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.887+0000 7f0356200700 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55942631eaf0 0x55942631ef20 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1331e70415cceea server_cookie=369379d8bdfbff02 in_seq=0 out_seq=0 2024-05-10T11:23:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.888+0000 7f0337400700 10 client.26819 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.888+0000 7f0337400700 1 -- 172.21.15.27:0/2100880785 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5594262db120 con 0x55942631eaf0 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.888+0000 7f0337400700 10 client.26819 handle_command_reply: tid=0 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.888+0000 7f0362b85b80 1 client.26819 shutdown 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.888+0000 7f0362b85b80 1 -- 172.21.15.27:0/2100880785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 msgr2=0x5594263172b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.888+0000 7f0362b85b80 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x5594263172b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f033c0117c0 tx=0x7f033c011b80 comp rx=0 tx=0).stop 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.889+0000 7f0362b85b80 1 -- 172.21.15.27:0/2100880785 shutdown_connections 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.889+0000 7f0362b85b80 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55942631eaf0 0x55942631ef20 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.889+0000 7f0362b85b80 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559426312a50 0x5594263177f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.889+0000 7f0362b85b80 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x559426319f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.889+0000 7f0337400700 0 client.26819 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.889+0000 7f0362b85b80 1 --2- 172.21.15.27:0/2100880785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x5594263172b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.889+0000 7f0362b85b80 1 -- 172.21.15.27:0/2100880785 >> 172.21.15.27:0/2100880785 conn(0x55942623cbe0 msgr2=0x559426303330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.889+0000 7f0362b85b80 1 -- 172.21.15.27:0/2100880785 shutdown_connections 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.889+0000 7f0362b85b80 1 -- 172.21.15.27:0/2100880785 wait complete. 2024-05-10T11:23:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.889+0000 7f0362b85b80 20 client.26819 trim_cache size 0 max 16384 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.914+0000 7f0362b85b80 1 Processor -- start 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.914+0000 7f0362b85b80 1 -- start start 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.914+0000 7f0362b85b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55942630e0e0 0x559426313e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.914+0000 7f0362b85b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x5594263031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.914+0000 7f0362b85b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559426312a50 0x559426327b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.914+0000 7f0362b85b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f03501ad520 con 0x559426312a50 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.914+0000 7f0362b85b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f03501af060 con 0x55942630ea00 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.914+0000 7f0362b85b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f03501adbf0 con 0x55942630e0e0 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.914+0000 7f0354e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x5594263031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0354e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x5594263031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52802/0 (socket says 172.21.15.27:52802) 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0354e00700 1 -- 172.21.15.27:0/2381987786 learned_addr learned my addr 172.21.15.27:0/2381987786 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0356200700 1 --2- 172.21.15.27:0/2381987786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55942630e0e0 0x559426313e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0356200700 1 --2- 172.21.15.27:0/2381987786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559426312a50 0x559426327b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0354e00700 1 -- 172.21.15.27:0/2381987786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55942630e0e0 msgr2=0x559426313e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0354e00700 1 --2- 172.21.15.27:0/2381987786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55942630e0e0 0x559426313e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0354e00700 1 -- 172.21.15.27:0/2381987786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559426312a50 msgr2=0x559426327b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0354e00700 1 --2- 172.21.15.27:0/2381987786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559426312a50 0x559426327b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0354e00700 1 -- 172.21.15.27:0/2381987786 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559426317080 con 0x55942630ea00 2024-05-10T11:23:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0356200700 1 --2- 172.21.15.27:0/2381987786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559426312a50 0x559426327b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:23:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0354e00700 1 --2- 172.21.15.27:0/2381987786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x5594263031b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f033c000c00 tx=0x7f033c0058c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80d597458a289a49 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0342000700 1 -- 172.21.15.27:0/2381987786 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c00f640 con 0x55942630ea00 2024-05-10T11:23:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0342000700 1 -- 172.21.15.27:0/2381987786 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f033c00fc20 con 0x55942630ea00 2024-05-10T11:23:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.915+0000 7f0342000700 1 -- 172.21.15.27:0/2381987786 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c01a4c0 con 0x55942630ea00 2024-05-10T11:23:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.916+0000 7f0362b85b80 1 -- 172.21.15.27:0/2381987786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 msgr2=0x5594263031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.916+0000 7f0362b85b80 1 --2- 172.21.15.27:0/2381987786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x5594263031b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f033c000c00 tx=0x7f033c0058c0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.916+0000 7f0362b85b80 1 -- 172.21.15.27:0/2381987786 shutdown_connections 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.916+0000 7f0362b85b80 1 --2- 172.21.15.27:0/2381987786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55942630e0e0 0x559426313e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.916+0000 7f0362b85b80 1 --2- 172.21.15.27:0/2381987786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559426312a50 0x559426327b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.916+0000 7f0362b85b80 1 --2- 172.21.15.27:0/2381987786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x5594263031b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.916+0000 7f0362b85b80 1 -- 172.21.15.27:0/2381987786 >> 172.21.15.27:0/2381987786 conn(0x55942623cbe0 msgr2=0x559426302aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.917+0000 7f0362b85b80 1 -- 172.21.15.27:0/2381987786 shutdown_connections 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.917+0000 7f0362b85b80 1 -- 172.21.15.27:0/2381987786 wait complete. 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.917+0000 7f0362b85b80 1 -- start start 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.917+0000 7f0362b85b80 1 Processor -- start 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.918+0000 7f0362b85b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x559426324550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.918+0000 7f0362b85b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x559426320ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.918+0000 7f0362b85b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559426312a50 0x5594263214f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.918+0000 7f0362b85b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f03501ae990 con 0x55942630e0e0 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.918+0000 7f0362b85b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f03501ae2c0 con 0x55942630ea00 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.918+0000 7f0362b85b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f03501ab480 con 0x559426312a50 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.918+0000 7f0362b85b80 10 client.0 _ll_register_callbacks cb 0x559426207cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.918+0000 7f0356200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x559426324550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.919+0000 7f0356200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x559426324550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42802/0 (socket says 172.21.15.27:42802) 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.919+0000 7f0356200700 1 -- 172.21.15.27:0/4233832574 learned_addr learned my addr 172.21.15.27:0/4233832574 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.919+0000 7f0356200700 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559426312a50 0x5594263214f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.919+0000 7f0354e00700 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x559426320ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.919+0000 7f0356200700 1 -- 172.21.15.27:0/4233832574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559426312a50 msgr2=0x5594263214f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.919+0000 7f0356200700 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559426312a50 0x5594263214f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.919+0000 7f0356200700 1 -- 172.21.15.27:0/4233832574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 msgr2=0x559426320ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.919+0000 7f0356200700 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x559426320ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.919+0000 7f0356200700 1 -- 172.21.15.27:0/4233832574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55942639a9c0 con 0x55942630e0e0 2024-05-10T11:23:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.920+0000 7f0356200700 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x559426324550 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0348007ee0 tx=0x7f03480094f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd071db348443f79 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.920+0000 7f0337400700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:23:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.920+0000 7f0337400700 1 -- 172.21.15.27:0/4233832574 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0348030390 con 0x55942630e0e0 2024-05-10T11:23:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.920+0000 7f0337400700 1 -- 172.21.15.27:0/4233832574 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f0348030530 con 0x55942630e0e0 2024-05-10T11:23:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.920+0000 7f0362b85b80 1 -- 172.21.15.27:0/4233832574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f033c00fc20 con 0x55942630e0e0 2024-05-10T11:23:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.920+0000 7f0337400700 1 -- 172.21.15.27:0/4233832574 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0348030730 con 0x55942630e0e0 2024-05-10T11:23:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.920+0000 7f0362b85b80 10 client.16590 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.920+0000 7f0362b85b80 1 -- 172.21.15.27:0/4233832574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55942639bb50 con 0x55942630e0e0 2024-05-10T11:23:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.921+0000 7f0337400700 1 -- 172.21.15.27:0/4233832574 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f033c00fc20 con 0x55942630e0e0 2024-05-10T11:23:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.921+0000 7f0337400700 1 -- 172.21.15.27:0/4233832574 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f034802d040 con 0x55942630e0e0 2024-05-10T11:23:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.921+0000 7f0362b85b80 10 client.16590 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.921+0000 7f0362b85b80 10 client.16590 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.921+0000 7f0362b85b80 20 client.16590 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.921+0000 7f0362b85b80 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55942639bd80 0x55942639e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.921+0000 7f0362b85b80 4 client.16590 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.921+0000 7f0362b85b80 1 -- 172.21.15.27:0/4233832574 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55942626e480 con 0x55942639bd80 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.921+0000 7f0354e00700 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55942639bd80 0x55942639e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.922+0000 7f0337400700 10 client.16590 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.922+0000 7f0354e00700 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55942639bd80 0x55942639e170 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8800b53eacf5b69d server_cookie=5cf3adb59055be8e in_seq=0 out_seq=0 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.923+0000 7f0337400700 1 -- 172.21.15.27:0/4233832574 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55942626e480 con 0x55942639bd80 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.923+0000 7f0337400700 10 client.16590 handle_command_reply: tid=0 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.923+0000 7f0362b85b80 1 client.16590 shutdown 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.923+0000 7f0362b85b80 1 -- 172.21.15.27:0/4233832574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 msgr2=0x559426324550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.923+0000 7f0362b85b80 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x559426324550 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0348007ee0 tx=0x7f03480094f0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.924+0000 7f0362b85b80 1 -- 172.21.15.27:0/4233832574 shutdown_connections 2024-05-10T11:23:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.924+0000 7f0362b85b80 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55942639bd80 0x55942639e170 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.924+0000 7f0362b85b80 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559426312a50 0x5594263214f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.924+0000 7f0362b85b80 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55942630e0e0 0x559426324550 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.924+0000 7f0362b85b80 1 --2- 172.21.15.27:0/4233832574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55942630ea00 0x559426320ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.287 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:23:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:23:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "dfd1d9dd-92a0-4fb2-8e0a-e7fd433014d1", 2024-05-10T11:23:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:23:52.288 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:23:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.924+0000 7f0337400700 0 client.16590 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.924+0000 7f0362b85b80 1 -- 172.21.15.27:0/4233832574 >> 172.21.15.27:0/4233832574 conn(0x55942623cbe0 msgr2=0x559426303330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.924+0000 7f0362b85b80 1 -- 172.21.15.27:0/4233832574 shutdown_connections 2024-05-10T11:23:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.924+0000 7f0362b85b80 1 -- 172.21.15.27:0/4233832574 wait complete. 2024-05-10T11:23:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.924+0000 7f0362b85b80 20 client.16590 trim_cache size 0 max 16384 2024-05-10T11:23:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.925+0000 7f0358200700 1 -- 172.21.15.27:0/2613199594 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f03380b2a70 msgr2=0x7f03380b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.925+0000 7f0358200700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f03380b2a70 0x7f03380b4f30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f03440a5b60 tx=0x7f03440e6740 comp rx=0 tx=0).stop 2024-05-10T11:23:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.925+0000 7f0358200700 1 -- 172.21.15.27:0/2613199594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f03501034f0 msgr2=0x7f035019f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.925+0000 7f0358200700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f03501034f0 0x7f035019f730 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f034400e9a0 tx=0x7f034400ecb0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.925+0000 7f0358200700 1 -- 172.21.15.27:0/2613199594 shutdown_connections 2024-05-10T11:23:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.925+0000 7f0358200700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f03380b2a70 0x7f03380b4f30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.925+0000 7f0358200700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350109600 0x7f0350199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.925+0000 7f0358200700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f03501034f0 0x7f035019f730 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.925+0000 7f0358200700 1 --2- 172.21.15.27:0/2613199594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0350103eb0 0x7f035019fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.925+0000 7f0358200700 1 -- 172.21.15.27:0/2613199594 >> 172.21.15.27:0/2613199594 conn(0x7f035007a650 msgr2=0x7f0350101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.925+0000 7f0358200700 1 -- 172.21.15.27:0/2613199594 shutdown_connections 2024-05-10T11:23:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:51.925+0000 7f0358200700 1 -- 172.21.15.27:0/2613199594 wait complete. 2024-05-10T11:23:52.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.327 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.327 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.327 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.327 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:51 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:52.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 644210 6.52 MB/sec execute 397 sec latency 25.866 ms 2024-05-10T11:23:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: pgmap v330: 129 pgs: 129 active+clean; 494 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-05-10T11:23:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-05-10T11:23:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: Unable to set osd_memory_target on smithi189 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-10T11:23:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-05-10T11:23:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: Unable to set osd_memory_target on smithi076 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-05-10T11:23:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:23:53.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:23:53.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:23:53.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:23:53.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:23:53.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:52 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: pgmap v330: 129 pgs: 129 active+clean; 494 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: Unable to set osd_memory_target on smithi189 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: Unable to set osd_memory_target on smithi076 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:52 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: pgmap v330: 129 pgs: 129 active+clean; 494 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: Unable to set osd_memory_target on smithi189 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: Unable to set osd_memory_target on smithi076 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:23:53.336 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.337 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.337 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:52 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:23:53.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:23:53.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 645673 6.52 MB/sec execute 398 sec latency 65.528 ms 2024-05-10T11:23:53.925 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:23:54.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 646737 6.52 MB/sec execute 399 sec latency 26.276 ms 2024-05-10T11:23:55.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:54 smithi076 ceph-mon[34152]: pgmap v331: 129 pgs: 129 active+clean; 494 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-05-10T11:23:55.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:54 smithi189 ceph-mon[34842]: pgmap v331: 129 pgs: 129 active+clean; 494 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-05-10T11:23:55.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:54 smithi027 ceph-mon[30054]: pgmap v331: 129 pgs: 129 active+clean; 494 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-05-10T11:23:55.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 647880 6.52 MB/sec execute 400 sec latency 31.034 ms 2024-05-10T11:23:56.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:56 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:56.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:56 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.392+0000 7f849bc00700 1 -- 172.21.15.27:0/299136381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8494103550 msgr2=0x7f84941039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.392+0000 7f849bc00700 1 --2- 172.21.15.27:0/299136381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8494103550 0x7f84941039b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8488009990 tx=0x7f8488009ca0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.392+0000 7f849bc00700 1 -- 172.21.15.27:0/299136381 shutdown_connections 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.392+0000 7f849bc00700 1 --2- 172.21.15.27:0/299136381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8494103ef0 0x7f84941103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.392+0000 7f849bc00700 1 --2- 172.21.15.27:0/299136381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8494108460 0x7f8494108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.392+0000 7f849bc00700 1 --2- 172.21.15.27:0/299136381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8494103550 0x7f84941039b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.392+0000 7f849bc00700 1 -- 172.21.15.27:0/299136381 >> 172.21.15.27:0/299136381 conn(0x7f849407a600 msgr2=0x7f849407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.392+0000 7f849bc00700 1 -- 172.21.15.27:0/299136381 shutdown_connections 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.393+0000 7f849bc00700 1 -- 172.21.15.27:0/299136381 wait complete. 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.394+0000 7f849bc00700 1 Processor -- start 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.394+0000 7f849bc00700 1 -- start start 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.395+0000 7f849bc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8494103550 0x7f849410a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.395+0000 7f849bc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8494103ef0 0x7f849410a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.395+0000 7f849bc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8494108460 0x7f849410ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.395+0000 7f849bc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f84941136e0 con 0x7f8494108460 2024-05-10T11:23:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.395+0000 7f849bc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8494113560 con 0x7f8494103ef0 2024-05-10T11:23:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.395+0000 7f849bc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8494113860 con 0x7f8494103550 2024-05-10T11:23:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.395+0000 7f8493e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8494103ef0 0x7f849410a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.395+0000 7f8493e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8494103ef0 0x7f849410a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54442/0 (socket says 172.21.15.27:54442) 2024-05-10T11:23:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.395+0000 7f8493e00700 1 -- 172.21.15.27:0/3748031809 learned_addr learned my addr 172.21.15.27:0/3748031809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.395+0000 7f8499c00700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8494108460 0x7f849410ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.395+0000 7f8499200700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8494103550 0x7f849410a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.396+0000 7f8493e00700 1 -- 172.21.15.27:0/3748031809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8494103550 msgr2=0x7f849410a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.396+0000 7f8493e00700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8494103550 0x7f849410a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.396+0000 7f8493e00700 1 -- 172.21.15.27:0/3748031809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8494108460 msgr2=0x7f849410ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.396+0000 7f8493e00700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8494108460 0x7f849410ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.396+0000 7f8493e00700 1 -- 172.21.15.27:0/3748031809 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84941ac500 con 0x7f8494103ef0 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.396+0000 7f8493e00700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8494103ef0 0x7f849410a920 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f848800c010 tx=0x7f848800b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.397+0000 7f8491600700 1 -- 172.21.15.27:0/3748031809 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488027070 con 0x7f8494103ef0 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.397+0000 7f8491600700 1 -- 172.21.15.27:0/3748031809 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8488022470 con 0x7f8494103ef0 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.397+0000 7f8491600700 1 -- 172.21.15.27:0/3748031809 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488017650 con 0x7f8494103ef0 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.397+0000 7f849bc00700 1 -- 172.21.15.27:0/3748031809 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84941ac790 con 0x7f8494103ef0 2024-05-10T11:23:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.397+0000 7f849bc00700 1 -- 172.21.15.27:0/3748031809 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84941acbe0 con 0x7f8494103ef0 2024-05-10T11:23:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.401+0000 7f8491600700 1 -- 172.21.15.27:0/3748031809 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f848802a070 con 0x7f8494103ef0 2024-05-10T11:23:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.403+0000 7f8491600700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f847c0b2a70 0x7f847c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.404+0000 7f8491600700 1 -- 172.21.15.27:0/3748031809 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f84880e37f0 con 0x7f8494103ef0 2024-05-10T11:23:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.404+0000 7f8499200700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f847c0b2a70 0x7f847c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.404+0000 7f84a65b6b80 1 Processor -- start 2024-05-10T11:23:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.404+0000 7f84a65b6b80 1 -- start start 2024-05-10T11:23:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.405+0000 7f84a65b6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae93bc0 0x55d45ae93fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.405+0000 7f84a65b6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae944e0 0x55d45ae8e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.405+0000 7f84a65b6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 0x55d45ae8ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.405+0000 7f84a65b6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d45ade0a20 con 0x55d45ae944e0 2024-05-10T11:23:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.405+0000 7f84a65b6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d45ad7e760 con 0x55d45ae8e8f0 2024-05-10T11:23:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.405+0000 7f84a65b6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d45addc9e0 con 0x55d45ae93bc0 2024-05-10T11:23:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.405+0000 7f8499200700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f847c0b2a70 0x7f847c0b4f30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8494104730 tx=0x7f8484009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.405+0000 7f8499200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae93bc0 0x55d45ae93fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.405+0000 7f8499200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae93bc0 0x55d45ae93fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:47628/0 (socket says 172.21.15.27:47628) 2024-05-10T11:23:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.406+0000 7f8499c00700 1 --2- 172.21.15.27:0/811506240 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 0x55d45ae8ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.406+0000 7f8499c00700 1 -- 172.21.15.27:0/811506240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae93bc0 msgr2=0x55d45ae93fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.406+0000 7f8499c00700 1 --2- 172.21.15.27:0/811506240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae93bc0 0x55d45ae93fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.406+0000 7f8499c00700 1 -- 172.21.15.27:0/811506240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae944e0 msgr2=0x55d45ae8e2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:23:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.407+0000 7f8499c00700 1 --2- 172.21.15.27:0/811506240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae944e0 0x55d45ae8e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.407+0000 7f8499c00700 1 -- 172.21.15.27:0/811506240 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d45ad2d8d0 con 0x55d45ae8e8f0 2024-05-10T11:23:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.405+0000 7f8499200700 1 -- 172.21.15.27:0/811506240 learned_addr learned my addr 172.21.15.27:0/811506240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.407+0000 7f8499200700 1 --2- 172.21.15.27:0/811506240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae93bc0 0x55d45ae93fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-05-10T11:23:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.407+0000 7f8499c00700 1 --2- 172.21.15.27:0/811506240 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 0x55d45ae8ed70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f848c00f410 tx=0x7f848c007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d98d36a825e655c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.407+0000 7f8482000700 1 -- 172.21.15.27:0/811506240 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848c018030 con 0x55d45ae8e8f0 2024-05-10T11:23:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.408+0000 7f8482000700 1 -- 172.21.15.27:0/811506240 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f848c004bb0 con 0x55d45ae8e8f0 2024-05-10T11:23:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.408+0000 7f8482000700 1 -- 172.21.15.27:0/811506240 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848c005020 con 0x55d45ae8e8f0 2024-05-10T11:23:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.409+0000 7f84a65b6b80 1 -- 172.21.15.27:0/811506240 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 msgr2=0x55d45ae8ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.409+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/811506240 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 0x55d45ae8ed70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f848c00f410 tx=0x7f848c007f90 comp rx=0 tx=0).stop 2024-05-10T11:23:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.410+0000 7f84a65b6b80 1 -- 172.21.15.27:0/811506240 shutdown_connections 2024-05-10T11:23:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.410+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/811506240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae93bc0 0x55d45ae93fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.410+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/811506240 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 0x55d45ae8ed70 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.410+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/811506240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae944e0 0x55d45ae8e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.410+0000 7f84a65b6b80 1 -- 172.21.15.27:0/811506240 >> 172.21.15.27:0/811506240 conn(0x55d45ae066d0 msgr2=0x55d45ae88b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.411+0000 7f84a65b6b80 1 -- 172.21.15.27:0/811506240 shutdown_connections 2024-05-10T11:23:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.411+0000 7f84a65b6b80 1 -- 172.21.15.27:0/811506240 wait complete. 2024-05-10T11:23:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.412+0000 7f84a65b6b80 1 -- start start 2024-05-10T11:23:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.412+0000 7f84a65b6b80 1 Processor -- start 2024-05-10T11:23:56.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.413+0000 7f84a65b6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 0x55d45ae9a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.413+0000 7f84a65b6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae93bc0 0x55d45ae975f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.413+0000 7f84a65b6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae944e0 0x55d45ae97b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.413+0000 7f84a65b6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d45ae9af20 con 0x55d45ae944e0 2024-05-10T11:23:56.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.413+0000 7f84a65b6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d45ad46b20 con 0x55d45ae93bc0 2024-05-10T11:23:56.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.413+0000 7f84a65b6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d45ae9b0a0 con 0x55d45ae8e8f0 2024-05-10T11:23:56.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.413+0000 7f8499200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 0x55d45ae9a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.413+0000 7f84a65b6b80 10 client.0 _ll_register_callbacks cb 0x55d45ad87c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:56.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.413+0000 7f8499200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 0x55d45ae9a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:47642/0 (socket says 172.21.15.27:47642) 2024-05-10T11:23:56.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.413+0000 7f8499200700 1 -- 172.21.15.27:0/3065037523 learned_addr learned my addr 172.21.15.27:0/3065037523 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:56.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.414+0000 7f8493e00700 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae93bc0 0x55d45ae975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.414+0000 7f8499c00700 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae944e0 0x55d45ae97b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.414+0000 7f8493e00700 1 -- 172.21.15.27:0/3065037523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 msgr2=0x55d45ae9a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.414+0000 7f8493e00700 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 0x55d45ae9a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.414+0000 7f8493e00700 1 -- 172.21.15.27:0/3065037523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae944e0 msgr2=0x55d45ae97b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.414+0000 7f8493e00700 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae944e0 0x55d45ae97b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.414+0000 7f8493e00700 1 -- 172.21.15.27:0/3065037523 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d45ada6420 con 0x55d45ae93bc0 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.414+0000 7f8499200700 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 0x55d45ae9a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.415+0000 7f8499c00700 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae944e0 0x55d45ae97b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.415+0000 7f8493e00700 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae93bc0 0x55d45ae975f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f84880e0ab0 tx=0x7f84880e1010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d57bf071a1e100c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.415+0000 7f8476a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.415+0000 7f8476a00700 1 -- 172.21.15.27:0/3065037523 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84880aa1e0 con 0x55d45ae93bc0 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.415+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3065037523 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f848c004bb0 con 0x55d45ae93bc0 2024-05-10T11:23:56.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.415+0000 7f8476a00700 1 -- 172.21.15.27:0/3065037523 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f8488092110 con 0x55d45ae93bc0 2024-05-10T11:23:56.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.416+0000 7f8476a00700 1 -- 172.21.15.27:0/3065037523 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f84880aadd0 con 0x55d45ae93bc0 2024-05-10T11:23:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.416+0000 7f84a65b6b80 10 client.26837 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.416+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3065037523 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d45aea5770 con 0x55d45ae93bc0 2024-05-10T11:23:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.416+0000 7f8476a00700 1 -- 172.21.15.27:0/3065037523 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84880aa6d0 con 0x55d45ae93bc0 2024-05-10T11:23:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.416+0000 7f8476a00700 1 -- 172.21.15.27:0/3065037523 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f84880e3300 con 0x55d45ae93bc0 2024-05-10T11:23:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.416+0000 7f84a65b6b80 10 client.26837 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.416+0000 7f84a65b6b80 10 client.26837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:23:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.416+0000 7f84a65b6b80 20 client.26837 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.416+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d45aea7ce0 0x55d45aeaa100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.416+0000 7f84a65b6b80 4 client.26837 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:23:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.416+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3065037523 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f848c004bb0 con 0x55d45aea7ce0 2024-05-10T11:23:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.417+0000 7f8499200700 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d45aea7ce0 0x55d45aeaa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.417+0000 7f8499200700 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d45aea7ce0 0x55d45aeaa100 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=311c8354c9732b3f server_cookie=b6aa4e3ff348270f in_seq=0 out_seq=0 2024-05-10T11:23:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.417+0000 7f8476a00700 10 client.26837 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.418+0000 7f8476a00700 1 -- 172.21.15.27:0/3065037523 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f848c004bb0 con 0x55d45aea7ce0 2024-05-10T11:23:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.418+0000 7f8476a00700 10 client.26837 handle_command_reply: tid=0 2024-05-10T11:23:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.418+0000 7f84a65b6b80 1 client.26837 shutdown 2024-05-10T11:23:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.419+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3065037523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae93bc0 msgr2=0x55d45ae975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.419+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae93bc0 0x55d45ae975f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f84880e0ab0 tx=0x7f84880e1010 comp rx=0 tx=0).stop 2024-05-10T11:23:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.419+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3065037523 shutdown_connections 2024-05-10T11:23:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.419+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d45aea7ce0 0x55d45aeaa100 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.419+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 0x55d45ae9a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.419+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae944e0 0x55d45ae97b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.419+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/3065037523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae93bc0 0x55d45ae975f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.419+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3065037523 >> 172.21.15.27:0/3065037523 conn(0x55d45ae066d0 msgr2=0x55d45ae88190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.419+0000 7f8476a00700 0 client.26837 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.419+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3065037523 shutdown_connections 2024-05-10T11:23:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.419+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3065037523 wait complete. 2024-05-10T11:23:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.419+0000 7f84a65b6b80 20 client.26837 trim_cache size 0 max 16384 2024-05-10T11:23:56.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:56 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:56.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:56 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.444+0000 7f84a65b6b80 1 Processor -- start 2024-05-10T11:23:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.444+0000 7f84a65b6b80 1 -- start start 2024-05-10T11:23:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.445+0000 7f84a65b6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 0x55d45ae8e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.445+0000 7f84a65b6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 0x55d45aea3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.445+0000 7f84a65b6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae944e0 0x55d45ae88910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.445+0000 7f84a65b6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f84941b5ba0 con 0x55d45ae93bc0 2024-05-10T11:23:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.445+0000 7f84a65b6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f84941b76e0 con 0x55d45ae944e0 2024-05-10T11:23:56.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.445+0000 7f84a65b6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f84941b6270 con 0x55d45ae8e8f0 2024-05-10T11:23:56.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.445+0000 7f8499c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae944e0 0x55d45ae88910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.445+0000 7f8493e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 0x55d45aea3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.445+0000 7f8499c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae944e0 0x55d45ae88910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54470/0 (socket says 172.21.15.27:54470) 2024-05-10T11:23:56.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.445+0000 7f8499200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 0x55d45ae8e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.445+0000 7f8493e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 0x55d45aea3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58412/0 (socket says 172.21.15.27:58412) 2024-05-10T11:23:56.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.445+0000 7f8499c00700 1 -- 172.21.15.27:0/3377239372 learned_addr learned my addr 172.21.15.27:0/3377239372 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:56.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.446+0000 7f8499200700 1 -- 172.21.15.27:0/3377239372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae944e0 msgr2=0x55d45ae88910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:56.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.446+0000 7f8499200700 1 --2- 172.21.15.27:0/3377239372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae944e0 0x55d45ae88910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.446+0000 7f8499200700 1 -- 172.21.15.27:0/3377239372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 msgr2=0x55d45aea3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:56.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.446+0000 7f8499200700 1 --2- 172.21.15.27:0/3377239372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 0x55d45aea3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.446+0000 7f8499200700 1 -- 172.21.15.27:0/3377239372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d45ae8f0b0 con 0x55d45ae8e8f0 2024-05-10T11:23:56.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.446+0000 7f8499c00700 1 --2- 172.21.15.27:0/3377239372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae944e0 0x55d45ae88910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:56.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.446+0000 7f8493e00700 1 --2- 172.21.15.27:0/3377239372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 0x55d45aea3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:56.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.446+0000 7f8499200700 1 --2- 172.21.15.27:0/3377239372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 0x55d45ae8e2d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x55d45aeaa910 tx=0x7f8484014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ece2b52afc41248d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:56.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.446+0000 7f8481600700 1 -- 172.21.15.27:0/3377239372 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848401de50 con 0x55d45ae8e8f0 2024-05-10T11:23:56.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.446+0000 7f8481600700 1 -- 172.21.15.27:0/3377239372 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f848402e070 con 0x55d45ae8e8f0 2024-05-10T11:23:56.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.447+0000 7f8481600700 1 -- 172.21.15.27:0/3377239372 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484027a20 con 0x55d45ae8e8f0 2024-05-10T11:23:56.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.447+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3377239372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 msgr2=0x55d45ae8e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.447+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/3377239372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 0x55d45ae8e2d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x55d45aeaa910 tx=0x7f8484014000 comp rx=0 tx=0).stop 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.447+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3377239372 shutdown_connections 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.447+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/3377239372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae8e8f0 0x55d45ae8e2d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.447+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/3377239372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae944e0 0x55d45ae88910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.447+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/3377239372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 0x55d45aea3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.447+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3377239372 >> 172.21.15.27:0/3377239372 conn(0x55d45ae066d0 msgr2=0x55d45ae92f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.448+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3377239372 shutdown_connections 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.448+0000 7f84a65b6b80 1 -- 172.21.15.27:0/3377239372 wait complete. 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.448+0000 7f84a65b6b80 1 -- start start 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.448+0000 7f84a65b6b80 1 Processor -- start 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.449+0000 7f84a65b6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 0x55d45aea0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.449+0000 7f84a65b6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 0x55d45ae9d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.449+0000 7f84a65b6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae944e0 0x55d45ae9d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.449+0000 7f84a65b6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f84941b7010 con 0x55d45ae93bc0 2024-05-10T11:23:56.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.449+0000 7f84a65b6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f84941b6940 con 0x55d45ae8e8f0 2024-05-10T11:23:56.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.449+0000 7f84a65b6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f84941b3b60 con 0x55d45ae944e0 2024-05-10T11:23:56.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.449+0000 7f84a65b6b80 10 client.0 _ll_register_callbacks cb 0x55d45ad87c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:23:56.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.449+0000 7f8499200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 0x55d45aea0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.449+0000 7f8499200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 0x55d45aea0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54472/0 (socket says 172.21.15.27:54472) 2024-05-10T11:23:56.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.449+0000 7f8499200700 1 -- 172.21.15.27:0/362882514 learned_addr learned my addr 172.21.15.27:0/362882514 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:23:56.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.449+0000 7f8493e00700 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 0x55d45ae9d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.449+0000 7f8499c00700 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae944e0 0x55d45ae9d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.450+0000 7f8499200700 1 -- 172.21.15.27:0/362882514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae944e0 msgr2=0x55d45ae9d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:56.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.450+0000 7f8499200700 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae944e0 0x55d45ae9d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.450+0000 7f8499200700 1 -- 172.21.15.27:0/362882514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 msgr2=0x55d45ae9d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:23:56.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.450+0000 7f8499200700 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 0x55d45ae9d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.450+0000 7f8499200700 1 -- 172.21.15.27:0/362882514 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d45ae9df20 con 0x55d45ae8e8f0 2024-05-10T11:23:56.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.450+0000 7f8493e00700 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 0x55d45ae9d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:56.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.450+0000 7f8499c00700 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae944e0 0x55d45ae9d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:23:56.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.450+0000 7f8499200700 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 0x55d45aea0740 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8484005cb0 tx=0x7f8484009180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41812c708915b3cf server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.450+0000 7f8476a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.450+0000 7f8476a00700 1 -- 172.21.15.27:0/362882514 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484037070 con 0x55d45ae8e8f0 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.451+0000 7f84a65b6b80 1 -- 172.21.15.27:0/362882514 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f848402e070 con 0x55d45ae8e8f0 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.451+0000 7f8476a00700 1 -- 172.21.15.27:0/362882514 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9591+0+0 (secure 0 0 0) 0x7f8484005450 con 0x55d45ae8e8f0 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.451+0000 7f8476a00700 1 -- 172.21.15.27:0/362882514 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8484040050 con 0x55d45ae8e8f0 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.451+0000 7f84a65b6b80 10 client.26843 fetch_fsmap learned FSMap version 36 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.451+0000 7f84a65b6b80 1 -- 172.21.15.27:0/362882514 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d45af13480 con 0x55d45ae8e8f0 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.451+0000 7f8476a00700 1 -- 172.21.15.27:0/362882514 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f848402e070 con 0x55d45ae8e8f0 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.451+0000 7f8476a00700 1 -- 172.21.15.27:0/362882514 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f848403a020 con 0x55d45ae8e8f0 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.451+0000 7f84a65b6b80 10 client.26843 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.451+0000 7f84a65b6b80 10 client.26843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.451+0000 7f84a65b6b80 20 client.26843 populate_metadata read hostname 'smithi027' 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.452+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d45af13aa0 0x55d45af15ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:23:56.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.452+0000 7f84a65b6b80 4 client.26843 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:23:56.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.452+0000 7f84a65b6b80 1 -- 172.21.15.27:0/362882514 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d45ae5b140 con 0x55d45af13aa0 2024-05-10T11:23:56.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.452+0000 7f8493e00700 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d45af13aa0 0x55d45af15ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:23:56.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.452+0000 7f8493e00700 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d45af13aa0 0x55d45af15ec0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42eb4435bbea6334 server_cookie=7f93e07a8c9f9faa in_seq=0 out_seq=0 2024-05-10T11:23:56.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.452+0000 7f8476a00700 10 client.26843 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.453+0000 7f8476a00700 1 -- 172.21.15.27:0/362882514 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d45ae5b140 con 0x55d45af13aa0 2024-05-10T11:23:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.453+0000 7f8476a00700 10 client.26843 handle_command_reply: tid=0 2024-05-10T11:23:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.453+0000 7f84a65b6b80 1 client.26843 shutdown 2024-05-10T11:23:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.453+0000 7f84a65b6b80 1 -- 172.21.15.27:0/362882514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 msgr2=0x55d45aea0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.453+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 0x55d45aea0740 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8484005cb0 tx=0x7f8484009180 comp rx=0 tx=0).stop 2024-05-10T11:23:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.453+0000 7f84a65b6b80 1 -- 172.21.15.27:0/362882514 shutdown_connections 2024-05-10T11:23:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.453+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d45af13aa0 0x55d45af15ec0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.453+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d45ae944e0 0x55d45ae9d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.453+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d45ae8e8f0 0x55d45aea0740 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.453+0000 7f84a65b6b80 1 --2- 172.21.15.27:0/362882514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d45ae93bc0 0x55d45ae9d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.453+0000 7f8476a00700 0 client.26843 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:23:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.453+0000 7f84a65b6b80 1 -- 172.21.15.27:0/362882514 >> 172.21.15.27:0/362882514 conn(0x55d45ae066d0 msgr2=0x55d45ae96e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.454+0000 7f84a65b6b80 1 -- 172.21.15.27:0/362882514 shutdown_connections 2024-05-10T11:23:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.454+0000 7f84a65b6b80 1 -- 172.21.15.27:0/362882514 wait complete. 2024-05-10T11:23:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.454+0000 7f84a65b6b80 20 client.26843 trim_cache size 0 max 16384 2024-05-10T11:23:56.457 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:23:56.457 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:23:56.457 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:23:56.457 INFO:teuthology.orchestra.run.smithi027.stdout: "dfd1d9dd-92a0-4fb2-8e0a-e7fd433014d1": { 2024-05-10T11:23:56.457 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:23:56.457 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "dfd1d9dd-92a0-4fb2-8e0a-e7fd433014d1", 2024-05-10T11:23:56.458 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:23:56.458 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:23:56.458 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:23:56.458 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:23:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.455+0000 7f849bc00700 1 -- 172.21.15.27:0/3748031809 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f847c0b2a70 msgr2=0x7f847c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.455+0000 7f849bc00700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f847c0b2a70 0x7f847c0b4f30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8494104730 tx=0x7f8484009260 comp rx=0 tx=0).stop 2024-05-10T11:23:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.455+0000 7f849bc00700 1 -- 172.21.15.27:0/3748031809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8494103ef0 msgr2=0x7f849410a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:23:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.455+0000 7f849bc00700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8494103ef0 0x7f849410a920 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f848800c010 tx=0x7f848800b670 comp rx=0 tx=0).stop 2024-05-10T11:23:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.455+0000 7f849bc00700 1 -- 172.21.15.27:0/3748031809 shutdown_connections 2024-05-10T11:23:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.455+0000 7f849bc00700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f847c0b2a70 0x7f847c0b4f30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.455+0000 7f849bc00700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8494103550 0x7f849410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.455+0000 7f849bc00700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8494108460 0x7f849410ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.455+0000 7f849bc00700 1 --2- 172.21.15.27:0/3748031809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8494103ef0 0x7f849410a920 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:23:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.455+0000 7f849bc00700 1 -- 172.21.15.27:0/3748031809 >> 172.21.15.27:0/3748031809 conn(0x7f849407a600 msgr2=0x7f84941013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:23:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.455+0000 7f849bc00700 1 -- 172.21.15.27:0/3748031809 shutdown_connections 2024-05-10T11:23:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:23:56.455+0000 7f849bc00700 1 -- 172.21.15.27:0/3748031809 wait complete. 2024-05-10T11:23:56.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:56 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:56.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:56 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:56.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 649724 6.53 MB/sec execute 401 sec latency 23.275 ms 2024-05-10T11:23:57.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:57 smithi027 ceph-mon[30054]: pgmap v332: 129 pgs: 129 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 207 op/s 2024-05-10T11:23:57.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:57 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:57 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:57 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:57 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:57 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:57 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:57 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:57 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:57 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:23:57.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:57 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:23:57.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:57 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:57.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:57 smithi076 ceph-mon[34152]: pgmap v332: 129 pgs: 129 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 207 op/s 2024-05-10T11:23:57.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:57 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:57 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:57 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:57 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:57 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:57 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:57 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:57 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:57 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:23:57.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:57 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:23:57.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:57 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:57.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:57 smithi189 ceph-mon[34842]: pgmap v332: 129 pgs: 129 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 207 op/s 2024-05-10T11:23:57.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:57 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:57 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:57 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:57 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:57 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:57 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:57 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:57 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-10T11:23:57.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:57 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:23:57.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:57 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:23:57.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:57 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:23:57.512 INFO:tasks.cephfs.filesystem:scrub status for tag:dfd1d9dd-92a0-4fb2-8e0a-e7fd433014d1 - {'path': '/', 'tag': 'dfd1d9dd-92a0-4fb2-8e0a-e7fd433014d1', 'options': 'recursive,force'} 2024-05-10T11:23:57.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 650358 6.52 MB/sec execute 402 sec latency 23.080 ms 2024-05-10T11:23:58.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:58 smithi027 ceph-mon[30054]: Adjusting osd_memory_target on smithi027 to 176.4M 2024-05-10T11:23:58.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:58 smithi027 ceph-mon[30054]: Unable to set osd_memory_target on smithi027 to 185039769: error parsing value: Value '185039769' is below minimum 939524096 2024-05-10T11:23:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:58 smithi076 ceph-mon[34152]: Adjusting osd_memory_target on smithi027 to 176.4M 2024-05-10T11:23:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:58 smithi076 ceph-mon[34152]: Unable to set osd_memory_target on smithi027 to 185039769: error parsing value: Value '185039769' is below minimum 939524096 2024-05-10T11:23:58.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:58 smithi189 ceph-mon[34842]: Adjusting osd_memory_target on smithi027 to 176.4M 2024-05-10T11:23:58.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:58 smithi189 ceph-mon[34842]: Unable to set osd_memory_target on smithi027 to 185039769: error parsing value: Value '185039769' is below minimum 939524096 2024-05-10T11:23:58.721 INFO:tasks.workunit.client.0.smithi027.stdout: 1 651384 6.52 MB/sec execute 403 sec latency 119.283 ms 2024-05-10T11:23:59.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:23:59 smithi027 ceph-mon[30054]: pgmap v333: 129 pgs: 129 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 138 op/s 2024-05-10T11:23:59.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:23:59 smithi076 ceph-mon[34152]: pgmap v333: 129 pgs: 129 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 138 op/s 2024-05-10T11:23:59.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:23:59 smithi189 ceph-mon[34842]: pgmap v333: 129 pgs: 129 active+clean; 506 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 138 op/s 2024-05-10T11:23:59.722 INFO:tasks.workunit.client.0.smithi027.stdout: 1 652889 6.52 MB/sec execute 404 sec latency 29.411 ms 2024-05-10T11:24:00.722 INFO:tasks.workunit.client.0.smithi027.stdout: 1 653629 6.52 MB/sec execute 405 sec latency 108.190 ms 2024-05-10T11:24:01.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:01 smithi027 ceph-mon[30054]: pgmap v334: 129 pgs: 129 active+clean; 512 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 207 op/s 2024-05-10T11:24:01.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:01 smithi076 ceph-mon[34152]: pgmap v334: 129 pgs: 129 active+clean; 512 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 207 op/s 2024-05-10T11:24:01.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:01 smithi189 ceph-mon[34842]: pgmap v334: 129 pgs: 129 active+clean; 512 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 207 op/s 2024-05-10T11:24:01.722 INFO:tasks.workunit.client.0.smithi027.stdout: 1 653860 6.50 MB/sec execute 406 sec latency 65.105 ms 2024-05-10T11:24:02.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:02 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:24:02.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:02 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:24:02.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:02 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:24:02.722 INFO:tasks.workunit.client.0.smithi027.stdout: 1 654576 6.49 MB/sec execute 407 sec latency 149.511 ms 2024-05-10T11:24:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:03 smithi027 ceph-mon[30054]: pgmap v335: 129 pgs: 129 active+clean; 512 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-05-10T11:24:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:03 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:24:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:03 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:24:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:03 smithi027 ceph-mon[30054]: osdmap e100: 12 total, 12 up, 12 in 2024-05-10T11:24:03.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:03 smithi076 ceph-mon[34152]: pgmap v335: 129 pgs: 129 active+clean; 512 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-05-10T11:24:03.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:03 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:24:03.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:03 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:24:03.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:03 smithi076 ceph-mon[34152]: osdmap e100: 12 total, 12 up, 12 in 2024-05-10T11:24:03.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:03 smithi189 ceph-mon[34842]: pgmap v335: 129 pgs: 129 active+clean; 512 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-05-10T11:24:03.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:03 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:24:03.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:03 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:24:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:03 smithi189 ceph-mon[34842]: osdmap e100: 12 total, 12 up, 12 in 2024-05-10T11:24:03.724 INFO:tasks.workunit.client.0.smithi027.stdout: 1 655737 6.49 MB/sec execute 408 sec latency 30.753 ms 2024-05-10T11:24:04.724 INFO:tasks.workunit.client.0.smithi027.stdout: 1 657193 6.50 MB/sec execute 409 sec latency 39.422 ms 2024-05-10T11:24:05.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:05 smithi076 ceph-mon[34152]: pgmap v337: 129 pgs: 129 active+clean; 512 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-05-10T11:24:05.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:05 smithi189 ceph-mon[34842]: pgmap v337: 129 pgs: 129 active+clean; 512 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-05-10T11:24:05.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:05 smithi027 ceph-mon[30054]: pgmap v337: 129 pgs: 129 active+clean; 512 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-05-10T11:24:05.724 INFO:tasks.workunit.client.0.smithi027.stdout: 1 657704 6.49 MB/sec execute 410 sec latency 39.774 ms 2024-05-10T11:24:06.724 INFO:tasks.workunit.client.0.smithi027.stdout: 1 658759 6.48 MB/sec execute 411 sec latency 73.613 ms 2024-05-10T11:24:07.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:07 smithi027 ceph-mon[30054]: pgmap v338: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 202 op/s 2024-05-10T11:24:07.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:07 smithi076 ceph-mon[34152]: pgmap v338: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 202 op/s 2024-05-10T11:24:07.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:07 smithi189 ceph-mon[34842]: pgmap v338: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 202 op/s 2024-05-10T11:24:07.724 INFO:tasks.workunit.client.0.smithi027.stdout: 1 660545 6.50 MB/sec execute 412 sec latency 23.503 ms 2024-05-10T11:24:08.725 INFO:tasks.workunit.client.0.smithi027.stdout: 1 661013 6.49 MB/sec execute 413 sec latency 23.437 ms 2024-05-10T11:24:09.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:09 smithi076 ceph-mon[34152]: pgmap v339: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 201 op/s 2024-05-10T11:24:09.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:09 smithi076 ceph-mon[34152]: osdmap e101: 12 total, 12 up, 12 in 2024-05-10T11:24:09.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:09 smithi189 ceph-mon[34842]: pgmap v339: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 201 op/s 2024-05-10T11:24:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:09 smithi189 ceph-mon[34842]: osdmap e101: 12 total, 12 up, 12 in 2024-05-10T11:24:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:09 smithi027 ceph-mon[30054]: pgmap v339: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 201 op/s 2024-05-10T11:24:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:09 smithi027 ceph-mon[30054]: osdmap e101: 12 total, 12 up, 12 in 2024-05-10T11:24:09.725 INFO:tasks.workunit.client.0.smithi027.stdout: 1 662089 6.48 MB/sec execute 414 sec latency 28.092 ms 2024-05-10T11:24:10.725 INFO:tasks.workunit.client.0.smithi027.stdout: 1 663529 6.48 MB/sec execute 415 sec latency 24.620 ms 2024-05-10T11:24:11.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:11 smithi076 ceph-mon[34152]: pgmap v341: 129 pgs: 129 active+clean; 519 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 248 op/s 2024-05-10T11:24:11.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:11 smithi189 ceph-mon[34842]: pgmap v341: 129 pgs: 129 active+clean; 519 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 248 op/s 2024-05-10T11:24:11.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:11 smithi027 ceph-mon[30054]: pgmap v341: 129 pgs: 129 active+clean; 519 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 248 op/s 2024-05-10T11:24:11.725 INFO:tasks.workunit.client.0.smithi027.stdout: 1 664019 6.48 MB/sec execute 416 sec latency 731.016 ms 2024-05-10T11:24:12.725 INFO:tasks.workunit.client.0.smithi027.stdout: 1 664019 6.47 MB/sec execute 417 sec latency 1731.115 ms 2024-05-10T11:24:13.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:13 smithi076 ceph-mon[34152]: pgmap v342: 129 pgs: 129 active+clean; 519 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-05-10T11:24:13.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:13 smithi189 ceph-mon[34842]: pgmap v342: 129 pgs: 129 active+clean; 519 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-05-10T11:24:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:13 smithi027 ceph-mon[30054]: pgmap v342: 129 pgs: 129 active+clean; 519 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-05-10T11:24:13.725 INFO:tasks.workunit.client.0.smithi027.stdout: 1 664019 6.45 MB/sec execute 418 sec latency 2731.220 ms 2024-05-10T11:24:14.725 INFO:tasks.workunit.client.0.smithi027.stdout: 1 664019 6.43 MB/sec execute 419 sec latency 3731.391 ms 2024-05-10T11:24:15.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:15 smithi076 ceph-mon[34152]: pgmap v343: 129 pgs: 129 active+clean; 519 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 198 op/s 2024-05-10T11:24:15.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:15 smithi189 ceph-mon[34842]: pgmap v343: 129 pgs: 129 active+clean; 519 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 198 op/s 2024-05-10T11:24:15.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:15 smithi027 ceph-mon[30054]: pgmap v343: 129 pgs: 129 active+clean; 519 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 198 op/s 2024-05-10T11:24:15.726 INFO:tasks.workunit.client.0.smithi027.stdout: 1 664019 6.42 MB/sec execute 420 sec latency 4731.593 ms 2024-05-10T11:24:16.726 INFO:tasks.workunit.client.0.smithi027.stdout: 1 664019 6.40 MB/sec execute 421 sec latency 5731.704 ms 2024-05-10T11:24:17.419 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:17 smithi027 ceph-mon[30054]: pgmap v344: 129 pgs: 129 active+clean; 522 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-05-10T11:24:17.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:17 smithi076 ceph-mon[34152]: pgmap v344: 129 pgs: 129 active+clean; 522 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-05-10T11:24:17.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:17 smithi189 ceph-mon[34842]: pgmap v344: 129 pgs: 129 active+clean; 522 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-05-10T11:24:17.726 INFO:tasks.workunit.client.0.smithi027.stdout: 1 664019 6.39 MB/sec execute 422 sec latency 6731.834 ms 2024-05-10T11:24:18.726 INFO:tasks.workunit.client.0.smithi027.stdout: 1 664146 6.37 MB/sec execute 423 sec latency 7479.695 ms 2024-05-10T11:24:19.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:19 smithi076 ceph-mon[34152]: pgmap v345: 129 pgs: 129 active+clean; 522 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-05-10T11:24:19.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:19 smithi189 ceph-mon[34842]: pgmap v345: 129 pgs: 129 active+clean; 522 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-05-10T11:24:19.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:19 smithi027 ceph-mon[30054]: pgmap v345: 129 pgs: 129 active+clean; 522 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-05-10T11:24:19.726 INFO:tasks.workunit.client.0.smithi027.stdout: 1 664469 6.36 MB/sec execute 424 sec latency 100.500 ms 2024-05-10T11:24:20.726 INFO:tasks.workunit.client.0.smithi027.stdout: 1 665498 6.36 MB/sec execute 425 sec latency 30.676 ms 2024-05-10T11:24:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:21 smithi076 ceph-mon[34152]: pgmap v346: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 148 op/s 2024-05-10T11:24:21.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:21 smithi189 ceph-mon[34842]: pgmap v346: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 148 op/s 2024-05-10T11:24:21.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:21 smithi027 ceph-mon[30054]: pgmap v346: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 148 op/s 2024-05-10T11:24:21.726 INFO:tasks.workunit.client.0.smithi027.stdout: 1 666514 6.35 MB/sec execute 426 sec latency 24.881 ms 2024-05-10T11:24:22.726 INFO:tasks.workunit.client.0.smithi027.stdout: 1 667817 6.36 MB/sec execute 427 sec latency 22.819 ms 2024-05-10T11:24:23.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:23 smithi076 ceph-mon[34152]: pgmap v347: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.5 MiB/s wr, 67 op/s 2024-05-10T11:24:23.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:23 smithi189 ceph-mon[34842]: pgmap v347: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.5 MiB/s wr, 67 op/s 2024-05-10T11:24:23.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:23 smithi027 ceph-mon[30054]: pgmap v347: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.5 MiB/s wr, 67 op/s 2024-05-10T11:24:23.726 INFO:tasks.workunit.client.0.smithi027.stdout: 1 668304 6.35 MB/sec execute 428 sec latency 54.813 ms 2024-05-10T11:24:24.727 INFO:tasks.workunit.client.0.smithi027.stdout: 1 669483 6.35 MB/sec execute 429 sec latency 101.151 ms 2024-05-10T11:24:25.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:25 smithi189 ceph-mon[34842]: pgmap v348: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.5 MiB/s wr, 67 op/s 2024-05-10T11:24:25.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:25 smithi027 ceph-mon[30054]: pgmap v348: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.5 MiB/s wr, 67 op/s 2024-05-10T11:24:25.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:25 smithi076 ceph-mon[34152]: pgmap v348: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.5 MiB/s wr, 67 op/s 2024-05-10T11:24:25.727 INFO:tasks.workunit.client.0.smithi027.stdout: 1 671282 6.36 MB/sec execute 430 sec latency 19.755 ms 2024-05-10T11:24:26.727 INFO:tasks.workunit.client.0.smithi027.stdout: 1 671848 6.35 MB/sec execute 431 sec latency 25.612 ms 2024-05-10T11:24:27.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:27 smithi189 ceph-mon[34842]: pgmap v349: 129 pgs: 129 active+clean; 531 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 132 op/s 2024-05-10T11:24:27.512 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:24:27.599 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:27 smithi027 ceph-mon[30054]: pgmap v349: 129 pgs: 129 active+clean; 531 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 132 op/s 2024-05-10T11:24:27.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:27 smithi076 ceph-mon[34152]: pgmap v349: 129 pgs: 129 active+clean; 531 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 132 op/s 2024-05-10T11:24:27.727 INFO:tasks.workunit.client.0.smithi027.stdout: 1 673037 6.35 MB/sec execute 432 sec latency 27.945 ms 2024-05-10T11:24:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:24:28.727 INFO:tasks.workunit.client.0.smithi027.stdout: 1 674773 6.36 MB/sec execute 433 sec latency 24.507 ms 2024-05-10T11:24:29.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:29 smithi027 ceph-mon[30054]: pgmap v350: 129 pgs: 129 active+clean; 531 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-05-10T11:24:29.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:29 smithi189 ceph-mon[34842]: pgmap v350: 129 pgs: 129 active+clean; 531 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-05-10T11:24:29.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:29 smithi076 ceph-mon[34152]: pgmap v350: 129 pgs: 129 active+clean; 531 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-05-10T11:24:29.727 INFO:tasks.workunit.client.0.smithi027.stdout: 1 675104 6.35 MB/sec execute 434 sec latency 88.684 ms 2024-05-10T11:24:30.727 INFO:tasks.workunit.client.0.smithi027.stdout: 1 676034 6.34 MB/sec execute 435 sec latency 39.034 ms 2024-05-10T11:24:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.731+0000 7f3df2600700 1 -- 172.21.15.27:0/1350527508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dec10ff00 msgr2=0x7f3dec1122f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.731+0000 7f3df2600700 1 --2- 172.21.15.27:0/1350527508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dec10ff00 0x7f3dec1122f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3de400b0a0 tx=0x7f3de400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.731+0000 7f3df2600700 1 -- 172.21.15.27:0/1350527508 shutdown_connections 2024-05-10T11:24:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.731+0000 7f3df2600700 1 --2- 172.21.15.27:0/1350527508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dec0756d0 0x7f3dec075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.731+0000 7f3df2600700 1 --2- 172.21.15.27:0/1350527508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dec10ff00 0x7f3dec1122f0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.731+0000 7f3df2600700 1 --2- 172.21.15.27:0/1350527508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec075ff0 0x7f3dec10f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.731+0000 7f3df2600700 1 -- 172.21.15.27:0/1350527508 >> 172.21.15.27:0/1350527508 conn(0x7f3dec07a600 msgr2=0x7f3dec07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.731+0000 7f3df2600700 1 -- 172.21.15.27:0/1350527508 shutdown_connections 2024-05-10T11:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.731+0000 7f3df2600700 1 -- 172.21.15.27:0/1350527508 wait complete. 2024-05-10T11:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.732+0000 7f3df2600700 1 Processor -- start 2024-05-10T11:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.733+0000 7f3df2600700 1 -- start start 2024-05-10T11:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.733+0000 7f3df2600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec0756d0 0x7f3dec19bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.733+0000 7f3df2600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dec075ff0 0x7f3dec19c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.733+0000 7f3df2600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dec10ff00 0x7f3dec1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.733+0000 7f3df2600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec115940 con 0x7f3dec0756d0 2024-05-10T11:24:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.733+0000 7f3df2600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec1157c0 con 0x7f3dec075ff0 2024-05-10T11:24:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.733+0000 7f3df2600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec115ac0 con 0x7f3dec10ff00 2024-05-10T11:24:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.733+0000 7f3deaa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dec075ff0 0x7f3dec19c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.733+0000 7f3deb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec0756d0 0x7f3dec19bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.733+0000 7f3deaa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dec075ff0 0x7f3dec19c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60732/0 (socket says 172.21.15.27:60732) 2024-05-10T11:24:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.733+0000 7f3deaa00700 1 -- 172.21.15.27:0/4054448179 learned_addr learned my addr 172.21.15.27:0/4054448179 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.733+0000 7f3debe00700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dec10ff00 0x7f3dec1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.734+0000 7f3deaa00700 1 -- 172.21.15.27:0/4054448179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dec10ff00 msgr2=0x7f3dec1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.734+0000 7f3deaa00700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dec10ff00 0x7f3dec1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.734+0000 7f3deaa00700 1 -- 172.21.15.27:0/4054448179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec0756d0 msgr2=0x7f3dec19bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.734+0000 7f3deaa00700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec0756d0 0x7f3dec19bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.734+0000 7f3deaa00700 1 -- 172.21.15.27:0/4054448179 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dec1a0d70 con 0x7f3dec075ff0 2024-05-10T11:24:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.734+0000 7f3deaa00700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dec075ff0 0x7f3dec19c2f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800e9a0 tx=0x7f3dd800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.735+0000 7f3de3e00700 1 -- 172.21.15.27:0/4054448179 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd800ca10 con 0x7f3dec075ff0 2024-05-10T11:24:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.735+0000 7f3de3e00700 1 -- 172.21.15.27:0/4054448179 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dd800cbb0 con 0x7f3dec075ff0 2024-05-10T11:24:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.735+0000 7f3de3e00700 1 -- 172.21.15.27:0/4054448179 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8018680 con 0x7f3dec075ff0 2024-05-10T11:24:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.735+0000 7f3df2600700 1 -- 172.21.15.27:0/4054448179 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dec1a1060 con 0x7f3dec075ff0 2024-05-10T11:24:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.735+0000 7f3df2600700 1 -- 172.21.15.27:0/4054448179 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dec1a88e0 con 0x7f3dec075ff0 2024-05-10T11:24:30.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.740+0000 7f3dfcf50b80 1 Processor -- start 2024-05-10T11:24:30.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.740+0000 7f3dfcf50b80 1 -- start start 2024-05-10T11:24:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.740+0000 7f3dfcf50b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 0x560a6b071440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.740+0000 7f3dfcf50b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 0x560a6b075490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.740+0000 7f3dfcf50b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b0759d0 0x560a6b075e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.740+0000 7f3dfcf50b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560a6afc3a20 con 0x560a6b071060 2024-05-10T11:24:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.740+0000 7f3dfcf50b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x560a6af61760 con 0x560a6b071980 2024-05-10T11:24:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.740+0000 7f3dfcf50b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560a6afbf9e0 con 0x560a6b0759d0 2024-05-10T11:24:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.741+0000 7f3deb400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 0x560a6b075490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.741+0000 7f3debe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 0x560a6b071440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.741+0000 7f3deb400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 0x560a6b075490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60734/0 (socket says 172.21.15.27:60734) 2024-05-10T11:24:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.741+0000 7f3deb400700 1 -- 172.21.15.27:0/4241306309 learned_addr learned my addr 172.21.15.27:0/4241306309 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.741+0000 7f3deb400700 1 -- 172.21.15.27:0/4241306309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b0759d0 msgr2=0x560a6b075e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:24:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.741+0000 7f3deb400700 1 --2- 172.21.15.27:0/4241306309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b0759d0 0x560a6b075e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.741+0000 7f3deb400700 1 -- 172.21.15.27:0/4241306309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 msgr2=0x560a6b071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.741+0000 7f3deb400700 1 --2- 172.21.15.27:0/4241306309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 0x560a6b071440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.741+0000 7f3deb400700 1 -- 172.21.15.27:0/4241306309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a6af108d0 con 0x560a6b071980 2024-05-10T11:24:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.742+0000 7f3de3e00700 1 -- 172.21.15.27:0/4054448179 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3dd8010430 con 0x7f3dec075ff0 2024-05-10T11:24:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.742+0000 7f3deb400700 1 --2- 172.21.15.27:0/4241306309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 0x560a6b075490 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009af0 tx=0x7f3ddc009e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10edeb5fa4383b1f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.742+0000 7f3de0c00700 1 -- 172.21.15.27:0/4241306309 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc017070 con 0x560a6b071980 2024-05-10T11:24:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.742+0000 7f3de0c00700 1 -- 172.21.15.27:0/4241306309 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ddc00ed80 con 0x560a6b071980 2024-05-10T11:24:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.742+0000 7f3de0c00700 1 -- 172.21.15.27:0/4241306309 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc0123d0 con 0x560a6b071980 2024-05-10T11:24:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.743+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/4241306309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 msgr2=0x560a6b075490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.743+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/4241306309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 0x560a6b075490 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009af0 tx=0x7f3ddc009e00 comp rx=0 tx=0).stop 2024-05-10T11:24:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.743+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/4241306309 shutdown_connections 2024-05-10T11:24:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.743+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/4241306309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b0759d0 0x560a6b075e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.743+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/4241306309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 0x560a6b071440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.743+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/4241306309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 0x560a6b075490 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.743+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/4241306309 >> 172.21.15.27:0/4241306309 conn(0x560a6af9fb60 msgr2=0x560a6b06b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:30.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.743+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/4241306309 shutdown_connections 2024-05-10T11:24:30.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.743+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/4241306309 wait complete. 2024-05-10T11:24:30.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.744+0000 7f3de3e00700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3dd00b2900 0x7f3dd00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.744+0000 7f3de3e00700 1 -- 172.21.15.27:0/4054448179 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f3dd80e2c00 con 0x7f3dec075ff0 2024-05-10T11:24:30.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.744+0000 7f3dfcf50b80 1 -- start start 2024-05-10T11:24:30.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.744+0000 7f3deb400700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3dd00b2900 0x7f3dd00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.744+0000 7f3dfcf50b80 1 Processor -- start 2024-05-10T11:24:30.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.745+0000 7f3dfcf50b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071060 0x560a6b07ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3dfcf50b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b071980 0x560a6b07a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3dfcf50b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b0759d0 0x560a6b07a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3dfcf50b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560a6b07dbe0 con 0x560a6b0759d0 2024-05-10T11:24:30.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3dfcf50b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x560a6af29b20 con 0x560a6b071060 2024-05-10T11:24:30.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3dfcf50b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560a6b07dd60 con 0x560a6b071980 2024-05-10T11:24:30.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3dfcf50b80 10 client.0 _ll_register_callbacks cb 0x560a6af6ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:30.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3deb400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071060 0x560a6b07ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3deb400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071060 0x560a6b07ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60740/0 (socket says 172.21.15.27:60740) 2024-05-10T11:24:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3deb400700 1 -- 172.21.15.27:0/203859717 learned_addr learned my addr 172.21.15.27:0/203859717 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3deb400700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3dd00b2900 0x7f3dd00b4dc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc000c00 tx=0x7f3ddc004750 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3deaa00700 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b071980 0x560a6b07a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3deb400700 1 -- 172.21.15.27:0/203859717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b071980 msgr2=0x560a6b07a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3debe00700 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b0759d0 0x560a6b07a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3deb400700 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b071980 0x560a6b07a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.746+0000 7f3deb400700 1 -- 172.21.15.27:0/203859717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b0759d0 msgr2=0x560a6b07a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.747+0000 7f3deb400700 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b0759d0 0x560a6b07a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.747+0000 7f3deb400700 1 -- 172.21.15.27:0/203859717 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a6af89420 con 0x560a6b071060 2024-05-10T11:24:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.747+0000 7f3debe00700 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b0759d0 0x560a6b07a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:24:30.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.747+0000 7f3deb400700 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071060 0x560a6b07ce90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc0136c0 tx=0x7f3ddc013940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c449692c4e11e24 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:30.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.747+0000 7f3dd5600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.747+0000 7f3dd5600700 1 -- 172.21.15.27:0/203859717 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc02e070 con 0x560a6b071060 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.747+0000 7f3dd5600700 1 -- 172.21.15.27:0/203859717 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f3ddc01c860 con 0x560a6b071060 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.747+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/203859717 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ddc00ed80 con 0x560a6b071060 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.748+0000 7f3dd5600700 1 -- 172.21.15.27:0/203859717 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3ddc039050 con 0x560a6b071060 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.748+0000 7f3dfcf50b80 10 client.26855 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.748+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/203859717 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560a6b088500 con 0x560a6b071060 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.748+0000 7f3dd5600700 1 -- 172.21.15.27:0/203859717 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ddc00ed80 con 0x560a6b071060 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.748+0000 7f3dd5600700 1 -- 172.21.15.27:0/203859717 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f3ddc026b50 con 0x560a6b071060 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.748+0000 7f3dfcf50b80 10 client.26855 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.748+0000 7f3dfcf50b80 10 client.26855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.748+0000 7f3dfcf50b80 20 client.26855 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.748+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560a6b08a9d0 0x560a6b08cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.748+0000 7f3dfcf50b80 4 client.26855 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:24:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.748+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/203859717 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a6b03e0a0 con 0x560a6b08a9d0 2024-05-10T11:24:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.749+0000 7f3deaa00700 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560a6b08a9d0 0x560a6b08cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.749+0000 7f3deaa00700 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560a6b08a9d0 0x560a6b08cdc0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2f8e1a4d76c1197 server_cookie=e94a166b58e6fd5c in_seq=0 out_seq=0 2024-05-10T11:24:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.749+0000 7f3dd5600700 10 client.26855 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:24:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.750+0000 7f3dd5600700 1 -- 172.21.15.27:0/203859717 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560a6b03e0a0 con 0x560a6b08a9d0 2024-05-10T11:24:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.750+0000 7f3dd5600700 10 client.26855 handle_command_reply: tid=0 2024-05-10T11:24:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.751+0000 7f3dfcf50b80 1 client.26855 shutdown 2024-05-10T11:24:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.751+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/203859717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071060 msgr2=0x560a6b07ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.751+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071060 0x560a6b07ce90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc0136c0 tx=0x7f3ddc013940 comp rx=0 tx=0).stop 2024-05-10T11:24:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.751+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/203859717 shutdown_connections 2024-05-10T11:24:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.751+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560a6b08a9d0 0x560a6b08cdc0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.751+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b0759d0 0x560a6b07a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.751+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071060 0x560a6b07ce90 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.751+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/203859717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b071980 0x560a6b07a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.751+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/203859717 >> 172.21.15.27:0/203859717 conn(0x560a6af9fb60 msgr2=0x560a6b0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.751+0000 7f3dd5600700 0 client.26855 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:24:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.752+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/203859717 shutdown_connections 2024-05-10T11:24:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.752+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/203859717 wait complete. 2024-05-10T11:24:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.752+0000 7f3dfcf50b80 20 client.26855 trim_cache size 0 max 16384 2024-05-10T11:24:30.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.776+0000 7f3dfcf50b80 1 Processor -- start 2024-05-10T11:24:30.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.776+0000 7f3dfcf50b80 1 -- start start 2024-05-10T11:24:30.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.776+0000 7f3dfcf50b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 0x560a6b092500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.776+0000 7f3dfcf50b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 0x560a6b092a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.776+0000 7f3dfcf50b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b0759d0 0x560a6b076580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.776+0000 7f3dfcf50b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec1b1a90 con 0x560a6b071060 2024-05-10T11:24:30.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.776+0000 7f3dfcf50b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec1b35d0 con 0x560a6b071980 2024-05-10T11:24:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.776+0000 7f3dfcf50b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec1b2160 con 0x560a6b0759d0 2024-05-10T11:24:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.777+0000 7f3deaa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 0x560a6b092a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.777+0000 7f3deaa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 0x560a6b092a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60748/0 (socket says 172.21.15.27:60748) 2024-05-10T11:24:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.777+0000 7f3deaa00700 1 -- 172.21.15.27:0/1364001407 learned_addr learned my addr 172.21.15.27:0/1364001407 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.777+0000 7f3deb400700 1 --2- 172.21.15.27:0/1364001407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 0x560a6b092500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.777+0000 7f3debe00700 1 --2- 172.21.15.27:0/1364001407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b0759d0 0x560a6b076580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.777+0000 7f3deaa00700 1 -- 172.21.15.27:0/1364001407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b0759d0 msgr2=0x560a6b076580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.777+0000 7f3deaa00700 1 --2- 172.21.15.27:0/1364001407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b0759d0 0x560a6b076580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.777+0000 7f3deaa00700 1 -- 172.21.15.27:0/1364001407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 msgr2=0x560a6b092500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.777+0000 7f3deaa00700 1 --2- 172.21.15.27:0/1364001407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 0x560a6b092500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.777+0000 7f3deaa00700 1 -- 172.21.15.27:0/1364001407 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a6b067f90 con 0x560a6b071980 2024-05-10T11:24:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.777+0000 7f3debe00700 1 --2- 172.21.15.27:0/1364001407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b0759d0 0x560a6b076580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:24:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.777+0000 7f3deaa00700 1 --2- 172.21.15.27:0/1364001407 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 0x560a6b092a40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80a9750 tx=0x7f3dd80cc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52c8ce6f2fafd499 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.778+0000 7f3dd7e00700 1 -- 172.21.15.27:0/1364001407 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80a5040 con 0x560a6b071980 2024-05-10T11:24:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.778+0000 7f3dd7e00700 1 -- 172.21.15.27:0/1364001407 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dd80b2040 con 0x560a6b071980 2024-05-10T11:24:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.778+0000 7f3dd7e00700 1 -- 172.21.15.27:0/1364001407 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80a9f10 con 0x560a6b071980 2024-05-10T11:24:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.778+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/1364001407 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 msgr2=0x560a6b092a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.778+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/1364001407 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 0x560a6b092a40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80a9750 tx=0x7f3dd80cc040 comp rx=0 tx=0).stop 2024-05-10T11:24:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.779+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/1364001407 shutdown_connections 2024-05-10T11:24:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.779+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/1364001407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b0759d0 0x560a6b076580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.779+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/1364001407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 0x560a6b092500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.779+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/1364001407 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b071980 0x560a6b092a40 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.779+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/1364001407 >> 172.21.15.27:0/1364001407 conn(0x560a6af9fb60 msgr2=0x560a6b078bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.779+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/1364001407 shutdown_connections 2024-05-10T11:24:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.779+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/1364001407 wait complete. 2024-05-10T11:24:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.779+0000 7f3dfcf50b80 1 -- start start 2024-05-10T11:24:30.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.780+0000 7f3dfcf50b80 1 Processor -- start 2024-05-10T11:24:30.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.780+0000 7f3dfcf50b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 0x560a6b0832e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.780+0000 7f3dfcf50b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b071980 0x560a6b076b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.780+0000 7f3dfcf50b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b0759d0 0x560a6b0770a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.780+0000 7f3dfcf50b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec1b2f00 con 0x560a6b071060 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.780+0000 7f3dfcf50b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec1b2830 con 0x560a6b0759d0 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.780+0000 7f3dfcf50b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec1afa50 con 0x560a6b071980 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.780+0000 7f3dfcf50b80 10 client.0 _ll_register_callbacks cb 0x560a6af6ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.780+0000 7f3deaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b071980 0x560a6b076b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.780+0000 7f3deaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b071980 0x560a6b076b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:47280/0 (socket says 172.21.15.27:47280) 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.781+0000 7f3deaa00700 1 -- 172.21.15.27:0/690365308 learned_addr learned my addr 172.21.15.27:0/690365308 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.781+0000 7f3debe00700 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b0759d0 0x560a6b0770a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.781+0000 7f3deb400700 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 0x560a6b0832e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.781+0000 7f3deaa00700 1 -- 172.21.15.27:0/690365308 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b0759d0 msgr2=0x560a6b0770a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.781+0000 7f3deaa00700 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b0759d0 0x560a6b0770a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.781+0000 7f3deaa00700 1 -- 172.21.15.27:0/690365308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 msgr2=0x560a6b0832e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:30.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.781+0000 7f3deaa00700 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 0x560a6b0832e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.781+0000 7f3deaa00700 1 -- 172.21.15.27:0/690365308 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a6b077660 con 0x560a6b071980 2024-05-10T11:24:30.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.781+0000 7f3deaa00700 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b071980 0x560a6b076b60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80a6cc0 tx=0x7f3dd80a9680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3173de589646451b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:30.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.781+0000 7f3dd5600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:24:30.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.781+0000 7f3dd5600700 1 -- 172.21.15.27:0/690365308 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80c3060 con 0x560a6b071980 2024-05-10T11:24:30.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.782+0000 7f3dd5600700 1 -- 172.21.15.27:0/690365308 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f3dd80a9ec0 con 0x560a6b071980 2024-05-10T11:24:30.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.782+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/690365308 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dd80b2040 con 0x560a6b071980 2024-05-10T11:24:30.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.782+0000 7f3dd5600700 1 -- 172.21.15.27:0/690365308 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3dd80a9a50 con 0x560a6b071980 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.782+0000 7f3dfcf50b80 10 client.26839 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.782+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/690365308 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560a6b0fd680 con 0x560a6b071980 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.782+0000 7f3dd5600700 1 -- 172.21.15.27:0/690365308 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dd80b2040 con 0x560a6b071980 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.782+0000 7f3dd5600700 1 -- 172.21.15.27:0/690365308 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f3dd806f1a0 con 0x560a6b071980 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.782+0000 7f3dfcf50b80 10 client.26839 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.782+0000 7f3dfcf50b80 10 client.26839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.782+0000 7f3dfcf50b80 20 client.26839 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.783+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560a6b0fdc70 0x560a6b100090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.783+0000 7f3dfcf50b80 4 client.26839 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.783+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/690365308 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560a6afd1400 con 0x560a6b0fdc70 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.783+0000 7f3deb400700 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560a6b0fdc70 0x560a6b100090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.784+0000 7f3deb400700 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560a6b0fdc70 0x560a6b100090 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcec6eb39900eb1 server_cookie=71926f21c3344292 in_seq=0 out_seq=0 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.784+0000 7f3dd5600700 10 client.26839 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.784+0000 7f3dd5600700 1 -- 172.21.15.27:0/690365308 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560a6afd1400 con 0x560a6b0fdc70 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.784+0000 7f3dd5600700 10 client.26839 handle_command_reply: tid=0 2024-05-10T11:24:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.784+0000 7f3dfcf50b80 1 client.26839 shutdown 2024-05-10T11:24:30.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.784+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/690365308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b071980 msgr2=0x560a6b076b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:30.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.784+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b071980 0x560a6b076b60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80a6cc0 tx=0x7f3dd80a9680 comp rx=0 tx=0).stop 2024-05-10T11:24:30.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.785+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/690365308 shutdown_connections 2024-05-10T11:24:30.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.785+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x560a6b0fdc70 0x560a6b100090 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.785+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560a6b0759d0 0x560a6b0770a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.785+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6b071060 0x560a6b0832e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.785+0000 7f3dfcf50b80 1 --2- 172.21.15.27:0/690365308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a6b071980 0x560a6b076b60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.785+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/690365308 >> 172.21.15.27:0/690365308 conn(0x560a6af9fb60 msgr2=0x560a6b0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:30.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.785+0000 7f3dd5600700 0 client.26839 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:24:30.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.785+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/690365308 shutdown_connections 2024-05-10T11:24:30.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.785+0000 7f3dfcf50b80 1 -- 172.21.15.27:0/690365308 wait complete. 2024-05-10T11:24:30.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.785+0000 7f3dfcf50b80 20 client.26839 trim_cache size 0 max 16384 2024-05-10T11:24:30.787 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:24:30.788 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:24:30.788 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:24:30.788 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:24:30.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.786+0000 7f3df2600700 1 -- 172.21.15.27:0/4054448179 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3dd00b2900 msgr2=0x7f3dd00b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:30.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.786+0000 7f3df2600700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3dd00b2900 0x7f3dd00b4dc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc000c00 tx=0x7f3ddc004750 comp rx=0 tx=0).stop 2024-05-10T11:24:30.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.786+0000 7f3df2600700 1 -- 172.21.15.27:0/4054448179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dec075ff0 msgr2=0x7f3dec19c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:30.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.786+0000 7f3df2600700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dec075ff0 0x7f3dec19c2f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800e9a0 tx=0x7f3dd800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.786+0000 7f3df2600700 1 -- 172.21.15.27:0/4054448179 shutdown_connections 2024-05-10T11:24:30.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.786+0000 7f3df2600700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3dd00b2900 0x7f3dd00b4dc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.786+0000 7f3df2600700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dec10ff00 0x7f3dec1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.786+0000 7f3df2600700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec0756d0 0x7f3dec19bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.786+0000 7f3df2600700 1 --2- 172.21.15.27:0/4054448179 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dec075ff0 0x7f3dec19c2f0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.786+0000 7f3df2600700 1 -- 172.21.15.27:0/4054448179 >> 172.21.15.27:0/4054448179 conn(0x7f3dec07a600 msgr2=0x7f3dec100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.787+0000 7f3df2600700 1 -- 172.21.15.27:0/4054448179 shutdown_connections 2024-05-10T11:24:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:30.787+0000 7f3df2600700 1 -- 172.21.15.27:0/4054448179 wait complete. 2024-05-10T11:24:31.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:31 smithi189 ceph-mon[34842]: pgmap v351: 129 pgs: 129 active+clean; 532 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 154 op/s 2024-05-10T11:24:31.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:31 smithi027 ceph-mon[30054]: pgmap v351: 129 pgs: 129 active+clean; 532 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 154 op/s 2024-05-10T11:24:31.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:31 smithi076 ceph-mon[34152]: pgmap v351: 129 pgs: 129 active+clean; 532 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 154 op/s 2024-05-10T11:24:31.727 INFO:tasks.workunit.client.0.smithi027.stdout: 1 677686 6.35 MB/sec execute 436 sec latency 39.955 ms 2024-05-10T11:24:31.793 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:24:31.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:24:32.244 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:24:32.727 INFO:tasks.workunit.client.0.smithi027.stdout: 1 678672 6.35 MB/sec execute 437 sec latency 18.779 ms 2024-05-10T11:24:33.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:33 smithi027 ceph-mon[30054]: pgmap v352: 129 pgs: 129 active+clean; 532 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 121 op/s 2024-05-10T11:24:33.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:33 smithi189 ceph-mon[34842]: pgmap v352: 129 pgs: 129 active+clean; 532 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 121 op/s 2024-05-10T11:24:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:33 smithi076 ceph-mon[34152]: pgmap v352: 129 pgs: 129 active+clean; 532 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 121 op/s 2024-05-10T11:24:33.728 INFO:tasks.workunit.client.0.smithi027.stdout: 1 679744 6.35 MB/sec execute 438 sec latency 34.097 ms 2024-05-10T11:24:34.728 INFO:tasks.workunit.client.0.smithi027.stdout: 1 681302 6.35 MB/sec execute 439 sec latency 31.906 ms 2024-05-10T11:24:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.891+0000 7f13e1c00700 1 -- 172.21.15.27:0/649902562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13dc103d90 msgr2=0x7f13dc1102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.892+0000 7f13e1c00700 1 --2- 172.21.15.27:0/649902562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13dc103d90 0x7f13dc1102a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f13d400b0a0 tx=0x7f13d400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:24:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.892+0000 7f13e1c00700 1 -- 172.21.15.27:0/649902562 shutdown_connections 2024-05-10T11:24:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.892+0000 7f13e1c00700 1 --2- 172.21.15.27:0/649902562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc109460 0x7f13dc109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.892+0000 7f13e1c00700 1 --2- 172.21.15.27:0/649902562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13dc103d90 0x7f13dc1102a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.892+0000 7f13e1c00700 1 --2- 172.21.15.27:0/649902562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13dc1033d0 0x7f13dc103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.892+0000 7f13e1c00700 1 -- 172.21.15.27:0/649902562 >> 172.21.15.27:0/649902562 conn(0x7f13dc0fef20 msgr2=0x7f13dc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.892+0000 7f13e1c00700 1 -- 172.21.15.27:0/649902562 shutdown_connections 2024-05-10T11:24:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.892+0000 7f13e1c00700 1 -- 172.21.15.27:0/649902562 wait complete. 2024-05-10T11:24:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.893+0000 7f13e1c00700 1 Processor -- start 2024-05-10T11:24:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.893+0000 7f13e1c00700 1 -- start start 2024-05-10T11:24:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.894+0000 7f13e1c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc1033d0 0x7f13dc1a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.894+0000 7f13e1c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13dc103d90 0x7f13dc1a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.894+0000 7f13e1c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13dc109460 0x7f13dc19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.894+0000 7f13e1c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc1135c0 con 0x7f13dc103d90 2024-05-10T11:24:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.894+0000 7f13e1c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc113440 con 0x7f13dc109460 2024-05-10T11:24:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.894+0000 7f13e1c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc113740 con 0x7f13dc1033d0 2024-05-10T11:24:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.894+0000 7f13dbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc1033d0 0x7f13dc1a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.894+0000 7f13e1200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13dc109460 0x7f13dc19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:34.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.894+0000 7f13e1200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13dc109460 0x7f13dc19ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33694/0 (socket says 172.21.15.27:33694) 2024-05-10T11:24:34.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.894+0000 7f13e1200700 1 -- 172.21.15.27:0/2575550889 learned_addr learned my addr 172.21.15.27:0/2575550889 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:34.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.895+0000 7f13db400700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13dc103d90 0x7f13dc1a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.895+0000 7f13e1200700 1 -- 172.21.15.27:0/2575550889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc1033d0 msgr2=0x7f13dc1a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.895+0000 7f13e1200700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc1033d0 0x7f13dc1a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.895+0000 7f13e1200700 1 -- 172.21.15.27:0/2575550889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13dc103d90 msgr2=0x7f13dc1a1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.895+0000 7f13e1200700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13dc103d90 0x7f13dc1a1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.895+0000 7f13e1200700 1 -- 172.21.15.27:0/2575550889 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13dc19c180 con 0x7f13dc109460 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.895+0000 7f13db400700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13dc103d90 0x7f13dc1a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.895+0000 7f13e1200700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13dc109460 0x7f13dc19ba20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f13d400b070 tx=0x7f13d4004cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.895+0000 7f13d8c00700 1 -- 172.21.15.27:0/2575550889 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d4021070 con 0x7f13dc109460 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.895+0000 7f13d8c00700 1 -- 172.21.15.27:0/2575550889 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13d4007e30 con 0x7f13dc109460 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.895+0000 7f13e1c00700 1 -- 172.21.15.27:0/2575550889 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13dc19c410 con 0x7f13dc109460 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.896+0000 7f13d8c00700 1 -- 172.21.15.27:0/2575550889 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d4004780 con 0x7f13dc109460 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.896+0000 7f13e1c00700 1 -- 172.21.15.27:0/2575550889 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13dc1a8730 con 0x7f13dc109460 2024-05-10T11:24:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.897+0000 7f13e1c00700 1 -- 172.21.15.27:0/2575550889 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13dc06a190 con 0x7f13dc109460 2024-05-10T11:24:34.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.899+0000 7f13d8c00700 1 -- 172.21.15.27:0/2575550889 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f13d4007980 con 0x7f13dc109460 2024-05-10T11:24:34.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.901+0000 7f13d8c00700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f13c00b2a70 0x7f13c00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:34.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.901+0000 7f13d8c00700 1 -- 172.21.15.27:0/2575550889 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f13d400f020 con 0x7f13dc109460 2024-05-10T11:24:34.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.901+0000 7f13dbe00700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f13c00b2a70 0x7f13c00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:34.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.902+0000 7f13dbe00700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f13c00b2a70 0x7f13c00b4f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f13dc19cb60 tx=0x7f13cc0073d0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:34.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:34.905+0000 7f13d8c00700 1 -- 172.21.15.27:0/2575550889 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f13d402a020 con 0x7f13dc109460 2024-05-10T11:24:35.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.210+0000 7f13e1c00700 1 -- 172.21.15.27:0/2575550889 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13dc1a8900 con 0x7f13dc109460 2024-05-10T11:24:35.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.211+0000 7f13d8c00700 1 -- 172.21.15.27:0/2575550889 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11936 (secure 0 0 0) 0x7f13d40ab470 con 0x7f13dc109460 2024-05-10T11:24:35.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:24:35.212 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:23:14.174773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24484,24527,24472,24488,24467]},"id":1}]} 2024-05-10T11:24:35.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.215+0000 7f13e1c00700 1 -- 172.21.15.27:0/2575550889 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f13c00b2a70 msgr2=0x7f13c00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:35.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.215+0000 7f13e1c00700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f13c00b2a70 0x7f13c00b4f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f13dc19cb60 tx=0x7f13cc0073d0 comp rx=0 tx=0).stop 2024-05-10T11:24:35.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.215+0000 7f13e1c00700 1 -- 172.21.15.27:0/2575550889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13dc109460 msgr2=0x7f13dc19ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:35.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.215+0000 7f13e1c00700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13dc109460 0x7f13dc19ba20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f13d400b070 tx=0x7f13d4004cf0 comp rx=0 tx=0).stop 2024-05-10T11:24:35.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.215+0000 7f13e1c00700 1 -- 172.21.15.27:0/2575550889 shutdown_connections 2024-05-10T11:24:35.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.215+0000 7f13e1c00700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f13c00b2a70 0x7f13c00b4f30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:35.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.215+0000 7f13e1c00700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc1033d0 0x7f13dc1a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:35.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.215+0000 7f13e1c00700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13dc109460 0x7f13dc19ba20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:35.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.215+0000 7f13e1c00700 1 --2- 172.21.15.27:0/2575550889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13dc103d90 0x7f13dc1a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:35.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.215+0000 7f13e1c00700 1 -- 172.21.15.27:0/2575550889 >> 172.21.15.27:0/2575550889 conn(0x7f13dc0fef20 msgr2=0x7f13dc10e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:35.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.215+0000 7f13e1c00700 1 -- 172.21.15.27:0/2575550889 shutdown_connections 2024-05-10T11:24:35.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:35.215+0000 7f13e1c00700 1 -- 172.21.15.27:0/2575550889 wait complete. 2024-05-10T11:24:35.217 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-05-10T11:24:35.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:35 smithi189 ceph-mon[34842]: pgmap v353: 129 pgs: 129 active+clean; 532 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 121 op/s 2024-05-10T11:24:35.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:35 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2575550889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:24:35.564 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:35 smithi027 ceph-mon[30054]: pgmap v353: 129 pgs: 129 active+clean; 532 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 121 op/s 2024-05-10T11:24:35.564 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:35 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2575550889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:24:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:35 smithi076 ceph-mon[34152]: pgmap v353: 129 pgs: 129 active+clean; 532 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 121 op/s 2024-05-10T11:24:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:35 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2575550889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:24:35.728 INFO:tasks.workunit.client.0.smithi027.stdout: 1 682245 6.35 MB/sec execute 440 sec latency 33.846 ms 2024-05-10T11:24:36.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:24:36.499 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:24:36.728 INFO:tasks.workunit.client.0.smithi027.stdout: 1 683363 6.35 MB/sec execute 441 sec latency 51.126 ms 2024-05-10T11:24:37.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:37 smithi027 ceph-mon[30054]: pgmap v354: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 183 op/s 2024-05-10T11:24:37.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:37 smithi189 ceph-mon[34842]: pgmap v354: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 183 op/s 2024-05-10T11:24:37.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:37 smithi076 ceph-mon[34152]: pgmap v354: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 183 op/s 2024-05-10T11:24:37.728 INFO:tasks.workunit.client.0.smithi027.stdout: 1 685029 6.35 MB/sec execute 442 sec latency 22.644 ms 2024-05-10T11:24:38.729 INFO:tasks.workunit.client.0.smithi027.stdout: 1 685763 6.35 MB/sec execute 443 sec latency 20.981 ms 2024-05-10T11:24:38.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.882+0000 7f8d85c00700 1 -- 172.21.15.27:0/2626406274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8010ff50 msgr2=0x7f8d80112340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:38.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.882+0000 7f8d85c00700 1 --2- 172.21.15.27:0/2626406274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8010ff50 0x7f8d80112340 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800b0a0 tx=0x7f8d7800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.882+0000 7f8d85c00700 1 -- 172.21.15.27:0/2626406274 shutdown_connections 2024-05-10T11:24:38.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.882+0000 7f8d85c00700 1 --2- 172.21.15.27:0/2626406274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8006b1b0 0x7f8d8006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.882+0000 7f8d85c00700 1 --2- 172.21.15.27:0/2626406274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8010ff50 0x7f8d80112340 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.882+0000 7f8d85c00700 1 --2- 172.21.15.27:0/2626406274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d8006bad0 0x7f8d8010fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.882+0000 7f8d85c00700 1 -- 172.21.15.27:0/2626406274 >> 172.21.15.27:0/2626406274 conn(0x7f8d80079350 msgr2=0x7f8d80079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:38.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.882+0000 7f8d85c00700 1 -- 172.21.15.27:0/2626406274 shutdown_connections 2024-05-10T11:24:38.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.882+0000 7f8d85c00700 1 -- 172.21.15.27:0/2626406274 wait complete. 2024-05-10T11:24:38.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.884+0000 7f8d85c00700 1 Processor -- start 2024-05-10T11:24:38.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.884+0000 7f8d85c00700 1 -- start start 2024-05-10T11:24:38.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.884+0000 7f8d85c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d8006b1b0 0x7f8d8019be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.884+0000 7f8d85c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8006bad0 0x7f8d8019c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.884+0000 7f8d85c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8010ff50 0x7f8d801a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.884+0000 7f8d85c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d801158c0 con 0x7f8d8006b1b0 2024-05-10T11:24:38.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.884+0000 7f8d85c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80115740 con 0x7f8d8006bad0 2024-05-10T11:24:38.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.884+0000 7f8d85c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80115a40 con 0x7f8d8010ff50 2024-05-10T11:24:38.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.885+0000 7f8d7ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d8006b1b0 0x7f8d8019be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.885+0000 7f8d7ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d8006b1b0 0x7f8d8019be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34930/0 (socket says 172.21.15.27:34930) 2024-05-10T11:24:38.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.885+0000 7f8d7ea00700 1 -- 172.21.15.27:0/649296426 learned_addr learned my addr 172.21.15.27:0/649296426 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:38.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.885+0000 7f8d7f400700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8010ff50 0x7f8d801a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.885+0000 7f8d7e000700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8006bad0 0x7f8d8019c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.885+0000 7f8d7f400700 1 -- 172.21.15.27:0/649296426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8006bad0 msgr2=0x7f8d8019c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:38.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.885+0000 7f8d7f400700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8006bad0 0x7f8d8019c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.885+0000 7f8d7f400700 1 -- 172.21.15.27:0/649296426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d8006b1b0 msgr2=0x7f8d8019be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:38.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.885+0000 7f8d7f400700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d8006b1b0 0x7f8d8019be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.885+0000 7f8d7f400700 1 -- 172.21.15.27:0/649296426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d801a0dd0 con 0x7f8d8010ff50 2024-05-10T11:24:38.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.886+0000 7f8d7f400700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8010ff50 0x7f8d801a0690 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800b070 tx=0x7f8d78004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:38.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.886+0000 7f8d77400700 1 -- 172.21.15.27:0/649296426 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d78021070 con 0x7f8d8010ff50 2024-05-10T11:24:38.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.886+0000 7f8d85c00700 1 -- 172.21.15.27:0/649296426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d801a1060 con 0x7f8d8010ff50 2024-05-10T11:24:38.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.886+0000 7f8d77400700 1 -- 172.21.15.27:0/649296426 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d78007e30 con 0x7f8d8010ff50 2024-05-10T11:24:38.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.886+0000 7f8d77400700 1 -- 172.21.15.27:0/649296426 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d780045f0 con 0x7f8d8010ff50 2024-05-10T11:24:38.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.886+0000 7f8d85c00700 1 -- 172.21.15.27:0/649296426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d801a8940 con 0x7f8d8010ff50 2024-05-10T11:24:38.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.890+0000 7f8d77400700 1 -- 172.21.15.27:0/649296426 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f8d78007980 con 0x7f8d8010ff50 2024-05-10T11:24:38.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.892+0000 7f8d905b3b80 1 Processor -- start 2024-05-10T11:24:38.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.892+0000 7f8d905b3b80 1 -- start start 2024-05-10T11:24:38.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.892+0000 7f8d77400700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8d640b2a70 0x7f8d640b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.892+0000 7f8d905b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df15bc0 0x55720df15fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.892+0000 7f8d905b3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df164e0 0x55720df102d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.892+0000 7f8d905b3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df108f0 0x55720df10d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.892+0000 7f8d905b3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55720de62a20 con 0x55720df108f0 2024-05-10T11:24:38.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.892+0000 7f8d905b3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55720de00760 con 0x55720df164e0 2024-05-10T11:24:38.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.892+0000 7f8d905b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55720de5e9e0 con 0x55720df15bc0 2024-05-10T11:24:38.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.892+0000 7f8d77400700 1 -- 172.21.15.27:0/649296426 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f8d780e3820 con 0x7f8d8010ff50 2024-05-10T11:24:38.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.893+0000 7f8d7ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df15bc0 0x55720df15fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.893+0000 7f8d7e000700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8d640b2a70 0x7f8d640b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.893+0000 7f8d7ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df15bc0 0x55720df15fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60412/0 (socket says 172.21.15.27:60412) 2024-05-10T11:24:38.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.893+0000 7f8d7ea00700 1 -- 172.21.15.27:0/1644786078 learned_addr learned my addr 172.21.15.27:0/1644786078 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:38.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.893+0000 7f8d7ea00700 1 -- 172.21.15.27:0/1644786078 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df164e0 msgr2=0x55720df102d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:24:38.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.893+0000 7f8d7ea00700 1 --2- 172.21.15.27:0/1644786078 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df164e0 0x55720df102d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.893+0000 7f8d7ea00700 1 -- 172.21.15.27:0/1644786078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df108f0 msgr2=0x55720df10d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:24:38.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.893+0000 7f8d7ea00700 1 --2- 172.21.15.27:0/1644786078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df108f0 0x55720df10d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.893+0000 7f8d7ea00700 1 -- 172.21.15.27:0/1644786078 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55720ddaf8d0 con 0x55720df15bc0 2024-05-10T11:24:38.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.894+0000 7f8d7ea00700 1 --2- 172.21.15.27:0/1644786078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df15bc0 0x55720df15fa0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009ea0 tx=0x7f8d70011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30673d0bd2157e6b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:38.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.894+0000 7f8d6be00700 1 -- 172.21.15.27:0/1644786078 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7001e070 con 0x55720df15bc0 2024-05-10T11:24:38.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.894+0000 7f8d7e000700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8d640b2a70 0x7f8d640b4f30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x55720df168e0 tx=0x7f8d6c009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:38.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.894+0000 7f8d6be00700 1 -- 172.21.15.27:0/1644786078 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d7000fb30 con 0x55720df15bc0 2024-05-10T11:24:38.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.894+0000 7f8d6be00700 1 -- 172.21.15.27:0/1644786078 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d70019c50 con 0x55720df15bc0 2024-05-10T11:24:38.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.895+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1644786078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df15bc0 msgr2=0x55720df15fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:38.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.895+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1644786078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df15bc0 0x55720df15fa0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009ea0 tx=0x7f8d70011720 comp rx=0 tx=0).stop 2024-05-10T11:24:38.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.895+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1644786078 shutdown_connections 2024-05-10T11:24:38.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.895+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1644786078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df15bc0 0x55720df15fa0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.895+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1644786078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df108f0 0x55720df10d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.895+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1644786078 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df164e0 0x55720df102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.895+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1644786078 >> 172.21.15.27:0/1644786078 conn(0x55720de886d0 msgr2=0x55720df0ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:38.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.895+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1644786078 shutdown_connections 2024-05-10T11:24:38.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.895+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1644786078 wait complete. 2024-05-10T11:24:38.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.896+0000 7f8d905b3b80 1 -- start start 2024-05-10T11:24:38.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.896+0000 7f8d905b3b80 1 Processor -- start 2024-05-10T11:24:38.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.897+0000 7f8d905b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 0x55720df1c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.897+0000 7f8d905b3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df15bc0 0x55720df195f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.897+0000 7f8d905b3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df164e0 0x55720df19b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.897+0000 7f8d905b3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55720df1cf20 con 0x55720df15bc0 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.897+0000 7f8d905b3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55720ddc8b20 con 0x55720df164e0 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.897+0000 7f8d905b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55720df1d0a0 con 0x55720df108f0 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.897+0000 7f8d905b3b80 10 client.0 _ll_register_callbacks cb 0x55720de09c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.897+0000 7f8d7ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 0x55720df1c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.897+0000 7f8d7f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df164e0 0x55720df19b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.897+0000 7f8d7ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 0x55720df1c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60420/0 (socket says 172.21.15.27:60420) 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.897+0000 7f8d7ea00700 1 -- 172.21.15.27:0/1928192099 learned_addr learned my addr 172.21.15.27:0/1928192099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.897+0000 7f8d7e000700 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df15bc0 0x55720df195f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.898+0000 7f8d7f400700 1 -- 172.21.15.27:0/1928192099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 msgr2=0x55720df1c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.898+0000 7f8d7f400700 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 0x55720df1c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.898+0000 7f8d7f400700 1 -- 172.21.15.27:0/1928192099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df15bc0 msgr2=0x55720df195f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.898+0000 7f8d7f400700 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df15bc0 0x55720df195f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.898+0000 7f8d7f400700 1 -- 172.21.15.27:0/1928192099 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55720de28420 con 0x55720df164e0 2024-05-10T11:24:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.898+0000 7f8d7ea00700 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 0x55720df1c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:24:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.898+0000 7f8d7e000700 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df15bc0 0x55720df195f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:24:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.898+0000 7f8d7f400700 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df164e0 0x55720df19b30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8d780e0ae0 tx=0x7f8d780a65c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46b421351d110771 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.898+0000 7f8d68c00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:24:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.898+0000 7f8d68c00700 1 -- 172.21.15.27:0/1928192099 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7802a070 con 0x55720df164e0 2024-05-10T11:24:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.898+0000 7f8d68c00700 1 -- 172.21.15.27:0/1928192099 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f8d78004790 con 0x55720df164e0 2024-05-10T11:24:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.898+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1928192099 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d7000fb30 con 0x55720df164e0 2024-05-10T11:24:38.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.899+0000 7f8d68c00700 1 -- 172.21.15.27:0/1928192099 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8d780a9e70 con 0x55720df164e0 2024-05-10T11:24:38.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.899+0000 7f8d905b3b80 10 client.26876 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:38.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.899+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1928192099 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55720df27820 con 0x55720df164e0 2024-05-10T11:24:38.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.899+0000 7f8d68c00700 1 -- 172.21.15.27:0/1928192099 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d7000fb30 con 0x55720df164e0 2024-05-10T11:24:38.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.899+0000 7f8d68c00700 1 -- 172.21.15.27:0/1928192099 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f8d78011070 con 0x55720df164e0 2024-05-10T11:24:38.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.899+0000 7f8d905b3b80 10 client.26876 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:38.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.899+0000 7f8d905b3b80 10 client.26876 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:24:38.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.899+0000 7f8d905b3b80 20 client.26876 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:38.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.899+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55720df29e80 0x55720df2c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.899+0000 7f8d905b3b80 4 client.26876 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:24:38.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.899+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1928192099 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55720dedd140 con 0x55720df29e80 2024-05-10T11:24:38.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.900+0000 7f8d7ea00700 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55720df29e80 0x55720df2c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.900+0000 7f8d7ea00700 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55720df29e80 0x55720df2c270 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9f8bf1103ad18f78 server_cookie=a7df76f7607b81c8 in_seq=0 out_seq=0 2024-05-10T11:24:38.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.900+0000 7f8d68c00700 10 client.26876 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:24:38.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.901+0000 7f8d68c00700 1 -- 172.21.15.27:0/1928192099 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55720dedd140 con 0x55720df29e80 2024-05-10T11:24:38.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.901+0000 7f8d68c00700 10 client.26876 handle_command_reply: tid=0 2024-05-10T11:24:38.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.901+0000 7f8d905b3b80 1 client.26876 shutdown 2024-05-10T11:24:38.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.901+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1928192099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df164e0 msgr2=0x55720df19b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:38.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.901+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df164e0 0x55720df19b30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8d780e0ae0 tx=0x7f8d780a65c0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.902+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1928192099 shutdown_connections 2024-05-10T11:24:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.902+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55720df29e80 0x55720df2c270 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.902+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 0x55720df1c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.902+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df164e0 0x55720df19b30 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.902+0000 7f8d68c00700 0 client.26876 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:24:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.902+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1928192099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df15bc0 0x55720df195f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.902+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1928192099 >> 172.21.15.27:0/1928192099 conn(0x55720de886d0 msgr2=0x55720df0a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.902+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1928192099 shutdown_connections 2024-05-10T11:24:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.902+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1928192099 wait complete. 2024-05-10T11:24:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.902+0000 7f8d905b3b80 20 client.26876 trim_cache size 0 max 16384 2024-05-10T11:24:38.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.920+0000 7f8d905b3b80 1 Processor -- start 2024-05-10T11:24:38.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.920+0000 7f8d905b3b80 1 -- start start 2024-05-10T11:24:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.920+0000 7f8d905b3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df108f0 0x55720df102d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.920+0000 7f8d905b3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df15bc0 0x55720df25fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.920+0000 7f8d905b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df164e0 0x55720df17020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.920+0000 7f8d905b3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d801b1780 con 0x55720df15bc0 2024-05-10T11:24:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.920+0000 7f8d905b3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8d801b32c0 con 0x55720df108f0 2024-05-10T11:24:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.920+0000 7f8d905b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8d801b1e50 con 0x55720df164e0 2024-05-10T11:24:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.921+0000 7f8d7ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df108f0 0x55720df102d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.921+0000 7f8d7ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df108f0 0x55720df102d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33744/0 (socket says 172.21.15.27:33744) 2024-05-10T11:24:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.921+0000 7f8d7ea00700 1 -- 172.21.15.27:0/1899948734 learned_addr learned my addr 172.21.15.27:0/1899948734 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:38.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.921+0000 7f8d7f400700 1 --2- 172.21.15.27:0/1899948734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df164e0 0x55720df17020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.921+0000 7f8d7e000700 1 --2- 172.21.15.27:0/1899948734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df15bc0 0x55720df25fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.921+0000 7f8d7ea00700 1 -- 172.21.15.27:0/1899948734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df164e0 msgr2=0x55720df17020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:38.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.921+0000 7f8d7ea00700 1 --2- 172.21.15.27:0/1899948734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df164e0 0x55720df17020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.921+0000 7f8d7ea00700 1 -- 172.21.15.27:0/1899948734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df15bc0 msgr2=0x55720df25fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:38.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.921+0000 7f8d7ea00700 1 --2- 172.21.15.27:0/1899948734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df15bc0 0x55720df25fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.921+0000 7f8d7ea00700 1 -- 172.21.15.27:0/1899948734 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55720df2c8d0 con 0x55720df108f0 2024-05-10T11:24:38.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.921+0000 7f8d7e000700 1 --2- 172.21.15.27:0/1899948734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df15bc0 0x55720df25fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:24:38.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.921+0000 7f8d7ea00700 1 --2- 172.21.15.27:0/1899948734 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df108f0 0x55720df102d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8d70013010 tx=0x7f8d7000f7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a06879d3d49dce9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:38.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.922+0000 7f8d6b400700 1 -- 172.21.15.27:0/1899948734 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7002d070 con 0x55720df108f0 2024-05-10T11:24:38.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.922+0000 7f8d6b400700 1 -- 172.21.15.27:0/1899948734 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d7001f040 con 0x55720df108f0 2024-05-10T11:24:38.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.922+0000 7f8d6b400700 1 -- 172.21.15.27:0/1899948734 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7001e040 con 0x55720df108f0 2024-05-10T11:24:38.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.922+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1899948734 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df108f0 msgr2=0x55720df102d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:38.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.922+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1899948734 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df108f0 0x55720df102d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8d70013010 tx=0x7f8d7000f7b0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.923+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1899948734 shutdown_connections 2024-05-10T11:24:38.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.923+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1899948734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df164e0 0x55720df17020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.923+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1899948734 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df108f0 0x55720df102d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.923+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/1899948734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df15bc0 0x55720df25fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.923+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1899948734 >> 172.21.15.27:0/1899948734 conn(0x55720de886d0 msgr2=0x55720df16910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:38.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.923+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1899948734 shutdown_connections 2024-05-10T11:24:38.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.923+0000 7f8d905b3b80 1 -- 172.21.15.27:0/1899948734 wait complete. 2024-05-10T11:24:38.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.923+0000 7f8d905b3b80 1 -- start start 2024-05-10T11:24:38.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.923+0000 7f8d905b3b80 1 Processor -- start 2024-05-10T11:24:38.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.924+0000 7f8d905b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 0x55720df22510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.924+0000 7f8d905b3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df15bc0 0x55720df22a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.924+0000 7f8d905b3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df164e0 0x55720df30c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.924+0000 7f8d905b3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d801b2bf0 con 0x55720df164e0 2024-05-10T11:24:38.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.924+0000 7f8d905b3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8d801b2520 con 0x55720df15bc0 2024-05-10T11:24:38.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.924+0000 7f8d905b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8d801af740 con 0x55720df108f0 2024-05-10T11:24:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.924+0000 7f8d905b3b80 10 client.0 _ll_register_callbacks cb 0x55720de09c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.924+0000 7f8d7ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 0x55720df22510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.924+0000 7f8d7ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 0x55720df22510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60436/0 (socket says 172.21.15.27:60436) 2024-05-10T11:24:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.924+0000 7f8d7ea00700 1 -- 172.21.15.27:0/3858823764 learned_addr learned my addr 172.21.15.27:0/3858823764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.924+0000 7f8d7e000700 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df15bc0 0x55720df22a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.924+0000 7f8d7f400700 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df164e0 0x55720df30c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.925+0000 7f8d7ea00700 1 -- 172.21.15.27:0/3858823764 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df15bc0 msgr2=0x55720df22a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.925+0000 7f8d7ea00700 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df15bc0 0x55720df22a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.925+0000 7f8d7ea00700 1 -- 172.21.15.27:0/3858823764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df164e0 msgr2=0x55720df30c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.925+0000 7f8d7ea00700 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df164e0 0x55720df30c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.925+0000 7f8d7ea00700 1 -- 172.21.15.27:0/3858823764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55720df31440 con 0x55720df108f0 2024-05-10T11:24:38.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.925+0000 7f8d7ea00700 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 0x55720df22510 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8d70011760 tx=0x7f8d700098f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8686beb3d68b4feb server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.925+0000 7f8d68c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.926+0000 7f8d68c00700 1 -- 172.21.15.27:0/3858823764 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d70036060 con 0x55720df108f0 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.926+0000 7f8d68c00700 1 -- 172.21.15.27:0/3858823764 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f8d70005240 con 0x55720df108f0 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.926+0000 7f8d905b3b80 1 -- 172.21.15.27:0/3858823764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d7001f040 con 0x55720df108f0 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.926+0000 7f8d68c00700 1 -- 172.21.15.27:0/3858823764 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8d7002d040 con 0x55720df108f0 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.926+0000 7f8d905b3b80 10 client.26863 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.926+0000 7f8d905b3b80 1 -- 172.21.15.27:0/3858823764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55720df942a0 con 0x55720df108f0 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.927+0000 7f8d68c00700 1 -- 172.21.15.27:0/3858823764 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d7001f040 con 0x55720df108f0 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.927+0000 7f8d68c00700 1 -- 172.21.15.27:0/3858823764 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f8d70005440 con 0x55720df108f0 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.927+0000 7f8d905b3b80 10 client.26863 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.927+0000 7f8d905b3b80 10 client.26863 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.927+0000 7f8d905b3b80 20 client.26863 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.927+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55720df946d0 0x55720df96af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.927+0000 7f8d905b3b80 4 client.26863 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:24:38.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.927+0000 7f8d905b3b80 1 -- 172.21.15.27:0/3858823764 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55720de70400 con 0x55720df946d0 2024-05-10T11:24:38.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.927+0000 7f8d7e000700 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55720df946d0 0x55720df96af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.928+0000 7f8d7e000700 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55720df946d0 0x55720df96af0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5d2639fc7cba5ea2 server_cookie=9ca1fc282ade096b in_seq=0 out_seq=0 2024-05-10T11:24:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.928+0000 7f8d68c00700 10 client.26863 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:24:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.928+0000 7f8d68c00700 1 -- 172.21.15.27:0/3858823764 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55720de70400 con 0x55720df946d0 2024-05-10T11:24:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.928+0000 7f8d68c00700 10 client.26863 handle_command_reply: tid=0 2024-05-10T11:24:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.928+0000 7f8d905b3b80 1 client.26863 shutdown 2024-05-10T11:24:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.928+0000 7f8d905b3b80 1 -- 172.21.15.27:0/3858823764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 msgr2=0x55720df22510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.928+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 0x55720df22510 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8d70011760 tx=0x7f8d700098f0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.929+0000 7f8d905b3b80 1 -- 172.21.15.27:0/3858823764 shutdown_connections 2024-05-10T11:24:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.929+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55720df946d0 0x55720df96af0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.929+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720df108f0 0x55720df22510 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.929+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55720df164e0 0x55720df30c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.929+0000 7f8d905b3b80 1 --2- 172.21.15.27:0/3858823764 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55720df15bc0 0x55720df22a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.929+0000 7f8d68c00700 0 client.26863 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:24:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.929+0000 7f8d905b3b80 1 -- 172.21.15.27:0/3858823764 >> 172.21.15.27:0/3858823764 conn(0x55720de886d0 msgr2=0x55720df14760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.929+0000 7f8d905b3b80 1 -- 172.21.15.27:0/3858823764 shutdown_connections 2024-05-10T11:24:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.929+0000 7f8d905b3b80 1 -- 172.21.15.27:0/3858823764 wait complete. 2024-05-10T11:24:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.929+0000 7f8d905b3b80 20 client.26863 trim_cache size 0 max 16384 2024-05-10T11:24:38.931 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:24:38.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.930+0000 7f8d85c00700 1 -- 172.21.15.27:0/649296426 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8d640b2a70 msgr2=0x7f8d640b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:38.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.930+0000 7f8d85c00700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8d640b2a70 0x7f8d640b4f30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x55720df168e0 tx=0x7f8d6c009260 comp rx=0 tx=0).stop 2024-05-10T11:24:38.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.930+0000 7f8d85c00700 1 -- 172.21.15.27:0/649296426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8010ff50 msgr2=0x7f8d801a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:38.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.930+0000 7f8d85c00700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8010ff50 0x7f8d801a0690 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800b070 tx=0x7f8d78004c80 comp rx=0 tx=0).stop 2024-05-10T11:24:38.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.930+0000 7f8d85c00700 1 -- 172.21.15.27:0/649296426 shutdown_connections 2024-05-10T11:24:38.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.930+0000 7f8d85c00700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8d640b2a70 0x7f8d640b4f30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.930+0000 7f8d85c00700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8010ff50 0x7f8d801a0690 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.930+0000 7f8d85c00700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d8006b1b0 0x7f8d8019be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.930+0000 7f8d85c00700 1 --2- 172.21.15.27:0/649296426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8006bad0 0x7f8d8019c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:38.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.930+0000 7f8d85c00700 1 -- 172.21.15.27:0/649296426 >> 172.21.15.27:0/649296426 conn(0x7f8d80079350 msgr2=0x7f8d80111440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:38.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.930+0000 7f8d85c00700 1 -- 172.21.15.27:0/649296426 shutdown_connections 2024-05-10T11:24:38.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:38.930+0000 7f8d85c00700 1 -- 172.21.15.27:0/649296426 wait complete. 2024-05-10T11:24:39.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:39 smithi027 ceph-mon[30054]: pgmap v355: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-05-10T11:24:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:39 smithi076 ceph-mon[34152]: pgmap v355: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-05-10T11:24:39.729 INFO:tasks.workunit.client.0.smithi027.stdout: 1 686650 6.34 MB/sec execute 444 sec latency 54.340 ms 2024-05-10T11:24:39.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:39 smithi189 ceph-mon[34842]: pgmap v355: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-05-10T11:24:39.854 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:24:40.395 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:24:40.729 INFO:tasks.workunit.client.0.smithi027.stdout: 1 688379 6.35 MB/sec execute 445 sec latency 26.680 ms 2024-05-10T11:24:41.586 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:41 smithi027 ceph-mon[30054]: pgmap v356: 129 pgs: 129 active+clean; 543 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 178 op/s 2024-05-10T11:24:41.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:41 smithi076 ceph-mon[34152]: pgmap v356: 129 pgs: 129 active+clean; 543 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 178 op/s 2024-05-10T11:24:41.729 INFO:tasks.workunit.client.0.smithi027.stdout: 1 689169 6.35 MB/sec execute 446 sec latency 36.048 ms 2024-05-10T11:24:41.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:41 smithi189 ceph-mon[34842]: pgmap v356: 129 pgs: 129 active+clean; 543 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 178 op/s 2024-05-10T11:24:42.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.650+0000 7ff21a600700 1 -- 172.21.15.27:0/43115220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff214103550 msgr2=0x7ff2141039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:42.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.650+0000 7ff21a600700 1 --2- 172.21.15.27:0/43115220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff214103550 0x7ff2141039b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009990 tx=0x7ff1fc009ca0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.650+0000 7ff21a600700 1 -- 172.21.15.27:0/43115220 shutdown_connections 2024-05-10T11:24:42.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.650+0000 7ff21a600700 1 --2- 172.21.15.27:0/43115220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214103ef0 0x7ff2141103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.650+0000 7ff21a600700 1 --2- 172.21.15.27:0/43115220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff214108460 0x7ff214108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.650+0000 7ff21a600700 1 --2- 172.21.15.27:0/43115220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff214103550 0x7ff2141039b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.650+0000 7ff21a600700 1 -- 172.21.15.27:0/43115220 >> 172.21.15.27:0/43115220 conn(0x7ff21407a600 msgr2=0x7ff21407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:42.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.650+0000 7ff21a600700 1 -- 172.21.15.27:0/43115220 shutdown_connections 2024-05-10T11:24:42.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.650+0000 7ff21a600700 1 -- 172.21.15.27:0/43115220 wait complete. 2024-05-10T11:24:42.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.652+0000 7ff21a600700 1 Processor -- start 2024-05-10T11:24:42.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.652+0000 7ff21a600700 1 -- start start 2024-05-10T11:24:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.652+0000 7ff21a600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff214103550 0x7ff2141a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.652+0000 7ff21a600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff214103ef0 0x7ff2141a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff21a600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214108460 0x7ff21410f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff21a600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2141136f0 con 0x7ff214103ef0 2024-05-10T11:24:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff21a600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff214113570 con 0x7ff214108460 2024-05-10T11:24:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff21a600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff214113870 con 0x7ff214103550 2024-05-10T11:24:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff213400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff214103550 0x7ff2141a5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff20be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff214103ef0 0x7ff2141a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff213e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214108460 0x7ff21410f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff213400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff214103550 0x7ff2141a5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60462/0 (socket says 172.21.15.27:60462) 2024-05-10T11:24:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff213400700 1 -- 172.21.15.27:0/2839285921 learned_addr learned my addr 172.21.15.27:0/2839285921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff213400700 1 -- 172.21.15.27:0/2839285921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214108460 msgr2=0x7ff21410f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff213400700 1 --2- 172.21.15.27:0/2839285921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214108460 0x7ff21410f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff213400700 1 -- 172.21.15.27:0/2839285921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff214103ef0 msgr2=0x7ff2141a63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff213400700 1 --2- 172.21.15.27:0/2839285921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff214103ef0 0x7ff2141a63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.653+0000 7ff213400700 1 -- 172.21.15.27:0/2839285921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21410fdc0 con 0x7ff214103550 2024-05-10T11:24:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.654+0000 7ff213400700 1 --2- 172.21.15.27:0/2839285921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff214103550 0x7ff2141a5e90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff20400e970 tx=0x7ff20400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.654+0000 7ff210c00700 1 -- 172.21.15.27:0/2839285921 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff204004220 con 0x7ff214103550 2024-05-10T11:24:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.654+0000 7ff210c00700 1 -- 172.21.15.27:0/2839285921 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2040043c0 con 0x7ff214103550 2024-05-10T11:24:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.654+0000 7ff21a600700 1 -- 172.21.15.27:0/2839285921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2141100b0 con 0x7ff214103550 2024-05-10T11:24:42.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.654+0000 7ff210c00700 1 -- 172.21.15.27:0/2839285921 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff204003ce0 con 0x7ff214103550 2024-05-10T11:24:42.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.654+0000 7ff21a600700 1 -- 172.21.15.27:0/2839285921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2141acec0 con 0x7ff214103550 2024-05-10T11:24:42.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.658+0000 7ff210c00700 1 -- 172.21.15.27:0/2839285921 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff20400c7e0 con 0x7ff214103550 2024-05-10T11:24:42.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.660+0000 7ff224fe9b80 1 Processor -- start 2024-05-10T11:24:42.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.660+0000 7ff224fe9b80 1 -- start start 2024-05-10T11:24:42.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.661+0000 7ff210c00700 1 --2- 172.21.15.27:0/2839285921 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff1f80b2a70 0x7ff1f80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.661+0000 7ff224fe9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984fbc0 0x55e7c984ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.661+0000 7ff224fe9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c98504e0 0x55e7c984a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.661+0000 7ff224fe9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984a8f0 0x55e7c984ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.661+0000 7ff224fe9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e7c979ca20 con 0x55e7c98504e0 2024-05-10T11:24:42.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.661+0000 7ff224fe9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55e7c973a760 con 0x55e7c984fbc0 2024-05-10T11:24:42.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.661+0000 7ff224fe9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e7c97989e0 con 0x55e7c984a8f0 2024-05-10T11:24:42.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.661+0000 7ff210c00700 1 -- 172.21.15.27:0/2839285921 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7ff2040e2bd0 con 0x7ff214103550 2024-05-10T11:24:42.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.662+0000 7ff213e00700 1 --2- 172.21.15.27:0/2839285921 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff1f80b2a70 0x7ff1f80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.662+0000 7ff213e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984a8f0 0x55e7c984ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.662+0000 7ff213e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984a8f0 0x55e7c984ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60474/0 (socket says 172.21.15.27:60474) 2024-05-10T11:24:42.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.662+0000 7ff213e00700 1 -- 172.21.15.27:0/2518151477 learned_addr learned my addr 172.21.15.27:0/2518151477 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:42.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.662+0000 7ff213400700 1 --2- 172.21.15.27:0/2518151477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984fbc0 0x55e7c984ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.662+0000 7ff20be00700 1 --2- 172.21.15.27:0/2518151477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c98504e0 0x55e7c984a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.663+0000 7ff213400700 1 -- 172.21.15.27:0/2518151477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984a8f0 msgr2=0x55e7c984ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:42.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.663+0000 7ff213400700 1 --2- 172.21.15.27:0/2518151477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984a8f0 0x55e7c984ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.663+0000 7ff213400700 1 -- 172.21.15.27:0/2518151477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c98504e0 msgr2=0x55e7c984a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:42.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.663+0000 7ff213400700 1 --2- 172.21.15.27:0/2518151477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c98504e0 0x55e7c984a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.663+0000 7ff213400700 1 -- 172.21.15.27:0/2518151477 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7c96e98d0 con 0x55e7c984fbc0 2024-05-10T11:24:42.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.663+0000 7ff213e00700 1 --2- 172.21.15.27:0/2518151477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984a8f0 0x55e7c984ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:24:42.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.663+0000 7ff20be00700 1 --2- 172.21.15.27:0/2518151477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c98504e0 0x55e7c984a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:24:42.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.663+0000 7ff213400700 1 --2- 172.21.15.27:0/2518151477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984fbc0 0x55e7c984ffa0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff2040dfe90 tx=0x7ff2040a9260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5de2842f0d1feb22 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:42.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.664+0000 7ff208c00700 1 -- 172.21.15.27:0/2518151477 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2040a9530 con 0x55e7c984fbc0 2024-05-10T11:24:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.664+0000 7ff208c00700 1 -- 172.21.15.27:0/2518151477 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2040aa120 con 0x55e7c984fbc0 2024-05-10T11:24:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.664+0000 7ff208c00700 1 -- 172.21.15.27:0/2518151477 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2040a5cb0 con 0x55e7c984fbc0 2024-05-10T11:24:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.664+0000 7ff224fe9b80 1 -- 172.21.15.27:0/2518151477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984fbc0 msgr2=0x55e7c984ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:42.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.664+0000 7ff213e00700 1 --2- 172.21.15.27:0/2839285921 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff1f80b2a70 0x7ff1f80b4f30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55e7c984b2b0 tx=0x7ff20c009920 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:42.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.664+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/2518151477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984fbc0 0x55e7c984ffa0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff2040dfe90 tx=0x7ff2040a9260 comp rx=0 tx=0).stop 2024-05-10T11:24:42.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.665+0000 7ff224fe9b80 1 -- 172.21.15.27:0/2518151477 shutdown_connections 2024-05-10T11:24:42.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.665+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/2518151477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984a8f0 0x55e7c984ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.665+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/2518151477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984fbc0 0x55e7c984ffa0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.665+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/2518151477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c98504e0 0x55e7c984a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.665+0000 7ff224fe9b80 1 -- 172.21.15.27:0/2518151477 >> 172.21.15.27:0/2518151477 conn(0x55e7c97c26d0 msgr2=0x55e7c9844b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:42.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.665+0000 7ff224fe9b80 1 -- 172.21.15.27:0/2518151477 shutdown_connections 2024-05-10T11:24:42.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.665+0000 7ff224fe9b80 1 -- 172.21.15.27:0/2518151477 wait complete. 2024-05-10T11:24:42.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.666+0000 7ff224fe9b80 1 -- start start 2024-05-10T11:24:42.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.666+0000 7ff224fe9b80 1 Processor -- start 2024-05-10T11:24:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.667+0000 7ff224fe9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984a8f0 0x55e7c9856250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.667+0000 7ff224fe9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c984fbc0 0x55e7c98535f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.667+0000 7ff224fe9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c98504e0 0x55e7c9853b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.667+0000 7ff224fe9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e7c9856ec0 con 0x55e7c984fbc0 2024-05-10T11:24:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.667+0000 7ff224fe9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55e7c9702b20 con 0x55e7c98504e0 2024-05-10T11:24:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.667+0000 7ff224fe9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e7c9857040 con 0x55e7c984a8f0 2024-05-10T11:24:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.667+0000 7ff224fe9b80 10 client.0 _ll_register_callbacks cb 0x55e7c9743c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.667+0000 7ff213e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c98504e0 0x55e7c9853b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.667+0000 7ff213e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c98504e0 0x55e7c9853b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33774/0 (socket says 172.21.15.27:33774) 2024-05-10T11:24:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.667+0000 7ff213e00700 1 -- 172.21.15.27:0/3823388417 learned_addr learned my addr 172.21.15.27:0/3823388417 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.667+0000 7ff20be00700 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c984fbc0 0x55e7c98535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.667+0000 7ff213400700 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984a8f0 0x55e7c9856250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.668+0000 7ff213e00700 1 -- 172.21.15.27:0/3823388417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984a8f0 msgr2=0x55e7c9856250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.668+0000 7ff213e00700 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984a8f0 0x55e7c9856250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.668+0000 7ff213e00700 1 -- 172.21.15.27:0/3823388417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c984fbc0 msgr2=0x55e7c98535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.668+0000 7ff213e00700 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c984fbc0 0x55e7c98535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.668+0000 7ff213e00700 1 -- 172.21.15.27:0/3823388417 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7c9762420 con 0x55e7c98504e0 2024-05-10T11:24:42.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.668+0000 7ff213e00700 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c98504e0 0x55e7c9853b30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff20c003d80 tx=0x7ff20c004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5894dd494845f535 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:42.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.668+0000 7ff201600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:24:42.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.668+0000 7ff201600700 1 -- 172.21.15.27:0/3823388417 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c004bb0 con 0x55e7c98504e0 2024-05-10T11:24:42.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.669+0000 7ff201600700 1 -- 172.21.15.27:0/3823388417 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7ff20c004d50 con 0x55e7c98504e0 2024-05-10T11:24:42.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.669+0000 7ff224fe9b80 1 -- 172.21.15.27:0/3823388417 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2040aa120 con 0x55e7c98504e0 2024-05-10T11:24:42.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.669+0000 7ff201600700 1 -- 172.21.15.27:0/3823388417 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff20c004250 con 0x55e7c98504e0 2024-05-10T11:24:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.669+0000 7ff224fe9b80 10 client.26894 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.669+0000 7ff224fe9b80 1 -- 172.21.15.27:0/3823388417 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e7c98727d0 con 0x55e7c98504e0 2024-05-10T11:24:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.669+0000 7ff201600700 1 -- 172.21.15.27:0/3823388417 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2040aa120 con 0x55e7c98504e0 2024-05-10T11:24:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.669+0000 7ff201600700 1 -- 172.21.15.27:0/3823388417 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7ff20c01b9d0 con 0x55e7c98504e0 2024-05-10T11:24:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.669+0000 7ff224fe9b80 10 client.26894 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.669+0000 7ff224fe9b80 10 client.26894 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:24:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.669+0000 7ff224fe9b80 20 client.26894 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.670+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55e7c9874e10 0x55e7c9877200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.670+0000 7ff224fe9b80 4 client.26894 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:24:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.670+0000 7ff224fe9b80 1 -- 172.21.15.27:0/3823388417 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7c9817140 con 0x55e7c9874e10 2024-05-10T11:24:42.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.670+0000 7ff213400700 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55e7c9874e10 0x55e7c9877200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.670+0000 7ff213400700 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55e7c9874e10 0x55e7c9877200 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9c266bb6cc3c339d server_cookie=986dbb0936705525 in_seq=0 out_seq=0 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.670+0000 7ff201600700 10 client.26894 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.671+0000 7ff201600700 1 -- 172.21.15.27:0/3823388417 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e7c9817140 con 0x55e7c9874e10 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.671+0000 7ff201600700 10 client.26894 handle_command_reply: tid=0 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.671+0000 7ff224fe9b80 1 client.26894 shutdown 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.671+0000 7ff224fe9b80 1 -- 172.21.15.27:0/3823388417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c98504e0 msgr2=0x55e7c9853b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.671+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c98504e0 0x55e7c9853b30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff20c003d80 tx=0x7ff20c004b40 comp rx=0 tx=0).stop 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.672+0000 7ff224fe9b80 1 -- 172.21.15.27:0/3823388417 shutdown_connections 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.672+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55e7c9874e10 0x55e7c9877200 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.672+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984a8f0 0x55e7c9856250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.672+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c98504e0 0x55e7c9853b30 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.672+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/3823388417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c984fbc0 0x55e7c98535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.672+0000 7ff224fe9b80 1 -- 172.21.15.27:0/3823388417 >> 172.21.15.27:0/3823388417 conn(0x55e7c97c26d0 msgr2=0x55e7c9844190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.672+0000 7ff201600700 0 client.26894 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:24:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.672+0000 7ff224fe9b80 1 -- 172.21.15.27:0/3823388417 shutdown_connections 2024-05-10T11:24:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.672+0000 7ff224fe9b80 1 -- 172.21.15.27:0/3823388417 wait complete. 2024-05-10T11:24:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.672+0000 7ff224fe9b80 20 client.26894 trim_cache size 0 max 16384 2024-05-10T11:24:42.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.691+0000 7ff224fe9b80 1 Processor -- start 2024-05-10T11:24:42.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.692+0000 7ff224fe9b80 1 -- start start 2024-05-10T11:24:42.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.692+0000 7ff224fe9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 0x55e7c984a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.692+0000 7ff224fe9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c984fbc0 0x55e7c984ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.692+0000 7ff224fe9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c98504e0 0x55e7c9853cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.692+0000 7ff224fe9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2141ad650 con 0x55e7c984fbc0 2024-05-10T11:24:42.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.692+0000 7ff224fe9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff2141af190 con 0x55e7c984a8f0 2024-05-10T11:24:42.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.692+0000 7ff224fe9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2141add20 con 0x55e7c98504e0 2024-05-10T11:24:42.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.692+0000 7ff213400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 0x55e7c984a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.692+0000 7ff213400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 0x55e7c984a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33776/0 (socket says 172.21.15.27:33776) 2024-05-10T11:24:42.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.692+0000 7ff213400700 1 -- 172.21.15.27:0/2753446484 learned_addr learned my addr 172.21.15.27:0/2753446484 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:42.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.692+0000 7ff213e00700 1 --2- 172.21.15.27:0/2753446484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c98504e0 0x55e7c9853cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.692+0000 7ff20be00700 1 --2- 172.21.15.27:0/2753446484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c984fbc0 0x55e7c984ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.693+0000 7ff213400700 1 -- 172.21.15.27:0/2753446484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c98504e0 msgr2=0x55e7c9853cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:42.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.693+0000 7ff213400700 1 --2- 172.21.15.27:0/2753446484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c98504e0 0x55e7c9853cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.693+0000 7ff213400700 1 -- 172.21.15.27:0/2753446484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c984fbc0 msgr2=0x55e7c984ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:42.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.693+0000 7ff213400700 1 --2- 172.21.15.27:0/2753446484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c984fbc0 0x55e7c984ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.693+0000 7ff213400700 1 -- 172.21.15.27:0/2753446484 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7c983fab0 con 0x55e7c984a8f0 2024-05-10T11:24:42.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.693+0000 7ff213400700 1 --2- 172.21.15.27:0/2753446484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 0x55e7c984a2d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff2040e0410 tx=0x7ff2040a5990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76be2ad722e62931 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:42.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.693+0000 7ff203e00700 1 -- 172.21.15.27:0/2753446484 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2040a9530 con 0x55e7c984a8f0 2024-05-10T11:24:42.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.693+0000 7ff203e00700 1 -- 172.21.15.27:0/2753446484 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2040aa120 con 0x55e7c984a8f0 2024-05-10T11:24:42.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.693+0000 7ff203e00700 1 -- 172.21.15.27:0/2753446484 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2040a9a20 con 0x55e7c984a8f0 2024-05-10T11:24:42.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.694+0000 7ff224fe9b80 1 -- 172.21.15.27:0/2753446484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 msgr2=0x55e7c984a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:42.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.694+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/2753446484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 0x55e7c984a2d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff2040e0410 tx=0x7ff2040a5990 comp rx=0 tx=0).stop 2024-05-10T11:24:42.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.696+0000 7ff224fe9b80 1 -- 172.21.15.27:0/2753446484 shutdown_connections 2024-05-10T11:24:42.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.696+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/2753446484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c98504e0 0x55e7c9853cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.696+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/2753446484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 0x55e7c984a2d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.696+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/2753446484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c984fbc0 0x55e7c984ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.696+0000 7ff224fe9b80 1 -- 172.21.15.27:0/2753446484 >> 172.21.15.27:0/2753446484 conn(0x55e7c97c26d0 msgr2=0x55e7c98535b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:42.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.696+0000 7ff224fe9b80 1 -- 172.21.15.27:0/2753446484 shutdown_connections 2024-05-10T11:24:42.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.696+0000 7ff224fe9b80 1 -- 172.21.15.27:0/2753446484 wait complete. 2024-05-10T11:24:42.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.696+0000 7ff224fe9b80 1 -- start start 2024-05-10T11:24:42.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.697+0000 7ff224fe9b80 1 Processor -- start 2024-05-10T11:24:42.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.698+0000 7ff224fe9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 0x55e7c985c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.698+0000 7ff224fe9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984fbc0 0x55e7c985c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.698+0000 7ff224fe9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c98504e0 0x55e7c987baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.698+0000 7ff224fe9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2141aeac0 con 0x55e7c98504e0 2024-05-10T11:24:42.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.698+0000 7ff224fe9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff2141ae3f0 con 0x55e7c984a8f0 2024-05-10T11:24:42.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.698+0000 7ff224fe9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff214073c50 con 0x55e7c984fbc0 2024-05-10T11:24:42.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.698+0000 7ff224fe9b80 10 client.0 _ll_register_callbacks cb 0x55e7c9743c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:42.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.698+0000 7ff20be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984fbc0 0x55e7c985c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.698+0000 7ff20be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984fbc0 0x55e7c985c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60492/0 (socket says 172.21.15.27:60492) 2024-05-10T11:24:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.698+0000 7ff20be00700 1 -- 172.21.15.27:0/238643186 learned_addr learned my addr 172.21.15.27:0/238643186 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.698+0000 7ff213400700 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 0x55e7c985c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.699+0000 7ff213400700 1 -- 172.21.15.27:0/238643186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984fbc0 msgr2=0x55e7c985c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.699+0000 7ff213400700 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984fbc0 0x55e7c985c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.699+0000 7ff213400700 1 -- 172.21.15.27:0/238643186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c98504e0 msgr2=0x55e7c987baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.699+0000 7ff213400700 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c98504e0 0x55e7c987baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.699+0000 7ff213400700 1 -- 172.21.15.27:0/238643186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7c987c1b0 con 0x55e7c984a8f0 2024-05-10T11:24:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.699+0000 7ff20be00700 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984fbc0 0x55e7c985c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:24:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.699+0000 7ff213400700 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 0x55e7c985c2d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff2040a6c90 tx=0x7ff2040dff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b65be78fbe1267df server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.699+0000 7ff201600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:24:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.699+0000 7ff201600700 1 -- 172.21.15.27:0/238643186 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2040a9530 con 0x55e7c984a8f0 2024-05-10T11:24:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.699+0000 7ff201600700 1 -- 172.21.15.27:0/238643186 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7ff204016330 con 0x55e7c984a8f0 2024-05-10T11:24:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.700+0000 7ff224fe9b80 1 -- 172.21.15.27:0/238643186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2040aa120 con 0x55e7c984a8f0 2024-05-10T11:24:42.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.700+0000 7ff201600700 1 -- 172.21.15.27:0/238643186 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff2040a9a20 con 0x55e7c984a8f0 2024-05-10T11:24:42.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.700+0000 7ff224fe9b80 10 client.26900 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:42.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.700+0000 7ff224fe9b80 1 -- 172.21.15.27:0/238643186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e7c98df040 con 0x55e7c984a8f0 2024-05-10T11:24:42.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.701+0000 7ff201600700 1 -- 172.21.15.27:0/238643186 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2040aa120 con 0x55e7c984a8f0 2024-05-10T11:24:42.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.701+0000 7ff201600700 1 -- 172.21.15.27:0/238643186 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7ff2040b23f0 con 0x55e7c984a8f0 2024-05-10T11:24:42.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.701+0000 7ff224fe9b80 10 client.26900 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:42.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.701+0000 7ff224fe9b80 10 client.26900 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:24:42.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.701+0000 7ff224fe9b80 20 client.26900 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:42.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.701+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55e7c98df270 0x55e7c98e1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:42.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.702+0000 7ff224fe9b80 4 client.26900 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:24:42.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.702+0000 7ff224fe9b80 1 -- 172.21.15.27:0/238643186 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7c97aa400 con 0x55e7c98df270 2024-05-10T11:24:42.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.702+0000 7ff20be00700 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55e7c98df270 0x55e7c98e1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:42.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff20be00700 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55e7c98df270 0x55e7c98e1660 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=69420cf639bb5552 server_cookie=b82851015d1df906 in_seq=0 out_seq=0 2024-05-10T11:24:42.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff201600700 10 client.26900 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:24:42.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff201600700 1 -- 172.21.15.27:0/238643186 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7c97aa400 con 0x55e7c98df270 2024-05-10T11:24:42.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff201600700 10 client.26900 handle_command_reply: tid=0 2024-05-10T11:24:42.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff224fe9b80 1 client.26900 shutdown 2024-05-10T11:24:42.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff224fe9b80 1 -- 172.21.15.27:0/238643186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 msgr2=0x55e7c985c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:42.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 0x55e7c985c2d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff2040a6c90 tx=0x7ff2040dff70 comp rx=0 tx=0).stop 2024-05-10T11:24:42.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff224fe9b80 1 -- 172.21.15.27:0/238643186 shutdown_connections 2024-05-10T11:24:42.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55e7c98df270 0x55e7c98e1660 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7c98504e0 0x55e7c987baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7c984a8f0 0x55e7c985c2d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff224fe9b80 1 --2- 172.21.15.27:0/238643186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7c984fbc0 0x55e7c985c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff224fe9b80 1 -- 172.21.15.27:0/238643186 >> 172.21.15.27:0/238643186 conn(0x55e7c97c26d0 msgr2=0x55e7c98508c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:42.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.703+0000 7ff201600700 0 client.26900 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:24:42.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.704+0000 7ff224fe9b80 1 -- 172.21.15.27:0/238643186 shutdown_connections 2024-05-10T11:24:42.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.704+0000 7ff224fe9b80 1 -- 172.21.15.27:0/238643186 wait complete. 2024-05-10T11:24:42.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.704+0000 7ff224fe9b80 20 client.26900 trim_cache size 0 max 16384 2024-05-10T11:24:42.705 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:24:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.705+0000 7ff21a600700 1 -- 172.21.15.27:0/2839285921 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff1f80b2a70 msgr2=0x7ff1f80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.705+0000 7ff21a600700 1 --2- 172.21.15.27:0/2839285921 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff1f80b2a70 0x7ff1f80b4f30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55e7c984b2b0 tx=0x7ff20c009920 comp rx=0 tx=0).stop 2024-05-10T11:24:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.705+0000 7ff21a600700 1 -- 172.21.15.27:0/2839285921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff214103550 msgr2=0x7ff2141a5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.705+0000 7ff21a600700 1 --2- 172.21.15.27:0/2839285921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff214103550 0x7ff2141a5e90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff20400e970 tx=0x7ff20400ec80 comp rx=0 tx=0).stop 2024-05-10T11:24:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.705+0000 7ff21a600700 1 -- 172.21.15.27:0/2839285921 shutdown_connections 2024-05-10T11:24:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.705+0000 7ff21a600700 1 --2- 172.21.15.27:0/2839285921 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff1f80b2a70 0x7ff1f80b4f30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.705+0000 7ff21a600700 1 --2- 172.21.15.27:0/2839285921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff214103550 0x7ff2141a5e90 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.705+0000 7ff21a600700 1 --2- 172.21.15.27:0/2839285921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214108460 0x7ff21410f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.705+0000 7ff21a600700 1 --2- 172.21.15.27:0/2839285921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff214103ef0 0x7ff2141a63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.705+0000 7ff21a600700 1 -- 172.21.15.27:0/2839285921 >> 172.21.15.27:0/2839285921 conn(0x7ff21407a600 msgr2=0x7ff214101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.705+0000 7ff21a600700 1 -- 172.21.15.27:0/2839285921 shutdown_connections 2024-05-10T11:24:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:42.705+0000 7ff21a600700 1 -- 172.21.15.27:0/2839285921 wait complete. 2024-05-10T11:24:42.729 INFO:tasks.workunit.client.0.smithi027.stdout: 1 689885 6.34 MB/sec execute 447 sec latency 33.718 ms 2024-05-10T11:24:43.571 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:43 smithi027 ceph-mon[30054]: pgmap v357: 129 pgs: 129 active+clean; 543 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-05-10T11:24:43.651 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:24:43.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:43 smithi076 ceph-mon[34152]: pgmap v357: 129 pgs: 129 active+clean; 543 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-05-10T11:24:43.729 INFO:tasks.workunit.client.0.smithi027.stdout: 1 691326 6.34 MB/sec execute 448 sec latency 46.612 ms 2024-05-10T11:24:43.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:43 smithi189 ceph-mon[34842]: pgmap v357: 129 pgs: 129 active+clean; 543 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-05-10T11:24:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:24:44.729 INFO:tasks.workunit.client.0.smithi027.stdout: 1 692638 6.35 MB/sec execute 449 sec latency 61.418 ms 2024-05-10T11:24:45.538 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:45 smithi189 ceph-mon[34842]: pgmap v358: 129 pgs: 129 active+clean; 543 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-05-10T11:24:45.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:45 smithi027 ceph-mon[30054]: pgmap v358: 129 pgs: 129 active+clean; 543 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-05-10T11:24:45.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:45 smithi076 ceph-mon[34152]: pgmap v358: 129 pgs: 129 active+clean; 543 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-05-10T11:24:45.729 INFO:tasks.workunit.client.0.smithi027.stdout: 1 693208 6.34 MB/sec execute 450 sec latency 38.034 ms 2024-05-10T11:24:46.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.681+0000 7f974f200700 1 -- 172.21.15.27:0/2544453563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9748114250 msgr2=0x7f9748116640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.681+0000 7f974f200700 1 --2- 172.21.15.27:0/2544453563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9748114250 0x7f9748116640 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f974400b0a0 tx=0x7f974400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.681+0000 7f974f200700 1 -- 172.21.15.27:0/2544453563 shutdown_connections 2024-05-10T11:24:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.681+0000 7f974f200700 1 --2- 172.21.15.27:0/2544453563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9748114250 0x7f9748116640 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.681+0000 7f974f200700 1 --2- 172.21.15.27:0/2544453563 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97480779d0 0x7f9748077db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.681+0000 7f974f200700 1 --2- 172.21.15.27:0/2544453563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97480782f0 0x7f9748113d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.681+0000 7f974f200700 1 -- 172.21.15.27:0/2544453563 >> 172.21.15.27:0/2544453563 conn(0x7f97481010e0 msgr2=0x7f9748103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.681+0000 7f974f200700 1 -- 172.21.15.27:0/2544453563 shutdown_connections 2024-05-10T11:24:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.681+0000 7f974f200700 1 -- 172.21.15.27:0/2544453563 wait complete. 2024-05-10T11:24:46.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.682+0000 7f974f200700 1 Processor -- start 2024-05-10T11:24:46.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.682+0000 7f974f200700 1 -- start start 2024-05-10T11:24:46.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974f200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97480779d0 0x7f97481a00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974f200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97480782f0 0x7f97481a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9748114250 0x7f97481a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974f200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9748119bd0 con 0x7f97480779d0 2024-05-10T11:24:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974f200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9748119a50 con 0x7f97480782f0 2024-05-10T11:24:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9748119d50 con 0x7f9748114250 2024-05-10T11:24:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97480779d0 0x7f97481a00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97480782f0 0x7f97481a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9748114250 0x7f97481a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97480782f0 0x7f97481a0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:47206/0 (socket says 172.21.15.27:47206) 2024-05-10T11:24:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974d400700 1 -- 172.21.15.27:0/3752262420 learned_addr learned my addr 172.21.15.27:0/3752262420 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974e800700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9748114250 0x7f97481a4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36514/0 (socket says 172.21.15.27:36514) 2024-05-10T11:24:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97480779d0 0x7f97481a00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52712/0 (socket says 172.21.15.27:52712) 2024-05-10T11:24:46.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974e800700 1 -- 172.21.15.27:0/3752262420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97480782f0 msgr2=0x7f97481a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:46.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974e800700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97480782f0 0x7f97481a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974e800700 1 -- 172.21.15.27:0/3752262420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97480779d0 msgr2=0x7f97481a00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:46.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974e800700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97480779d0 0x7f97481a00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.683+0000 7f974e800700 1 -- 172.21.15.27:0/3752262420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97481a5080 con 0x7f9748114250 2024-05-10T11:24:46.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.684+0000 7f974e800700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9748114250 0x7f97481a4940 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f974400d010 tx=0x7f9744004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:46.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.684+0000 7f9742a00700 1 -- 172.21.15.27:0/3752262420 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744021070 con 0x7f9748114250 2024-05-10T11:24:46.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.684+0000 7f9742a00700 1 -- 172.21.15.27:0/3752262420 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9744007e30 con 0x7f9748114250 2024-05-10T11:24:46.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.684+0000 7f9742a00700 1 -- 172.21.15.27:0/3752262420 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97440045f0 con 0x7f9748114250 2024-05-10T11:24:46.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.684+0000 7f974f200700 1 -- 172.21.15.27:0/3752262420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97481a5310 con 0x7f9748114250 2024-05-10T11:24:46.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.684+0000 7f974f200700 1 -- 172.21.15.27:0/3752262420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97481acbf0 con 0x7f9748114250 2024-05-10T11:24:46.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.688+0000 7f9742a00700 1 -- 172.21.15.27:0/3752262420 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f9744007980 con 0x7f9748114250 2024-05-10T11:24:46.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.689+0000 7f975b0f7b80 1 Processor -- start 2024-05-10T11:24:46.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.689+0000 7f975b0f7b80 1 -- start start 2024-05-10T11:24:46.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.690+0000 7f9742a00700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f97300b29f0 0x7f97300b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.690+0000 7f9742a00700 1 -- 172.21.15.27:0/3752262420 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f974400f020 con 0x7f9748114250 2024-05-10T11:24:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.690+0000 7f975b0f7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953ada0 0x55a62953b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.690+0000 7f975b0f7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953b6c0 0x55a629535430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.690+0000 7f975b0f7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 0x55a629535ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.690+0000 7f975b0f7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a629487a20 con 0x55a629535a50 2024-05-10T11:24:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.690+0000 7f975b0f7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a629425760 con 0x55a62953b6c0 2024-05-10T11:24:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.690+0000 7f975b0f7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a6294839e0 con 0x55a62953ada0 2024-05-10T11:24:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.690+0000 7f974d400700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f97300b29f0 0x7f97300b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.690+0000 7f974e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 0x55a629535ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.690+0000 7f974e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 0x55a629535ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52718/0 (socket says 172.21.15.27:52718) 2024-05-10T11:24:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.691+0000 7f974e800700 1 -- 172.21.15.27:0/2917805875 learned_addr learned my addr 172.21.15.27:0/2917805875 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.691+0000 7f974de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953ada0 0x55a62953b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.691+0000 7f974d400700 1 --2- 172.21.15.27:0/2917805875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953b6c0 0x55a629535430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.691+0000 7f974de00700 1 -- 172.21.15.27:0/2917805875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953b6c0 msgr2=0x55a629535430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.691+0000 7f974de00700 1 --2- 172.21.15.27:0/2917805875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953b6c0 0x55a629535430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.691+0000 7f974de00700 1 -- 172.21.15.27:0/2917805875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 msgr2=0x55a629535ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.691+0000 7f974de00700 1 --2- 172.21.15.27:0/2917805875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 0x55a629535ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.691+0000 7f974de00700 1 -- 172.21.15.27:0/2917805875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6293d48d0 con 0x55a62953ada0 2024-05-10T11:24:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.691+0000 7f974e800700 1 --2- 172.21.15.27:0/2917805875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 0x55a629535ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:24:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.691+0000 7f974d400700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f97300b29f0 0x7f97300b4eb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9738009880 tx=0x7f9738009bc0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.691+0000 7f974de00700 1 --2- 172.21.15.27:0/2917805875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953ada0 0x55a62953b180 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f973c009cb0 tx=0x7f973c011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d92f9bb4a4741097 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:46.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.692+0000 7f9737400700 1 -- 172.21.15.27:0/2917805875 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c011d20 con 0x55a62953ada0 2024-05-10T11:24:46.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.692+0000 7f9737400700 1 -- 172.21.15.27:0/2917805875 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f973c0044e0 con 0x55a62953ada0 2024-05-10T11:24:46.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.692+0000 7f9737400700 1 -- 172.21.15.27:0/2917805875 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c0193e0 con 0x55a62953ada0 2024-05-10T11:24:46.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.693+0000 7f975b0f7b80 1 -- 172.21.15.27:0/2917805875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953ada0 msgr2=0x55a62953b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:46.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.693+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/2917805875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953ada0 0x55a62953b180 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f973c009cb0 tx=0x7f973c011440 comp rx=0 tx=0).stop 2024-05-10T11:24:46.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.693+0000 7f975b0f7b80 1 -- 172.21.15.27:0/2917805875 shutdown_connections 2024-05-10T11:24:46.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.693+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/2917805875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953ada0 0x55a62953b180 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.693+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/2917805875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 0x55a629535ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.693+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/2917805875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953b6c0 0x55a629535430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.693+0000 7f975b0f7b80 1 -- 172.21.15.27:0/2917805875 >> 172.21.15.27:0/2917805875 conn(0x55a6294ad6d0 msgr2=0x55a62952fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.693+0000 7f975b0f7b80 1 -- 172.21.15.27:0/2917805875 shutdown_connections 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.693+0000 7f975b0f7b80 1 -- 172.21.15.27:0/2917805875 wait complete. 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.693+0000 7f975b0f7b80 1 -- start start 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.694+0000 7f975b0f7b80 1 Processor -- start 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.694+0000 7f975b0f7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a629535a50 0x55a629541340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.694+0000 7f975b0f7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 0x55a62953e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.694+0000 7f975b0f7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a62953b6c0 0x55a62953ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.694+0000 7f975b0f7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a629542150 con 0x55a62953b6c0 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.694+0000 7f975b0f7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a6293edb20 con 0x55a62953ada0 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.694+0000 7f975b0f7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a6295422d0 con 0x55a629535a50 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.695+0000 7f975b0f7b80 10 client.0 _ll_register_callbacks cb 0x55a62942ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.695+0000 7f974de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a629535a50 0x55a629541340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.695+0000 7f974d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 0x55a62953e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.695+0000 7f974de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a629535a50 0x55a629541340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36522/0 (socket says 172.21.15.27:36522) 2024-05-10T11:24:46.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.695+0000 7f974de00700 1 -- 172.21.15.27:0/1655119780 learned_addr learned my addr 172.21.15.27:0/1655119780 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:46.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.695+0000 7f974e800700 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a62953b6c0 0x55a62953ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.695+0000 7f974d400700 1 -- 172.21.15.27:0/1655119780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a629535a50 msgr2=0x55a629541340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:46.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.695+0000 7f974d400700 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a629535a50 0x55a629541340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.695+0000 7f974d400700 1 -- 172.21.15.27:0/1655119780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a62953b6c0 msgr2=0x55a62953ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:46.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.695+0000 7f974d400700 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a62953b6c0 0x55a62953ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.695+0000 7f974d400700 1 -- 172.21.15.27:0/1655119780 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a62944d420 con 0x55a62953ada0 2024-05-10T11:24:46.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.695+0000 7f974e800700 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a62953b6c0 0x55a62953ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:24:46.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.716+0000 7f974d400700 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 0x55a62953e6e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55a62953baa0 tx=0x7f97380047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24fe6a2be1df9c51 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:46.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.716+0000 7f972be00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:24:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.717+0000 7f972be00700 1 -- 172.21.15.27:0/1655119780 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973800fbd0 con 0x55a62953ada0 2024-05-10T11:24:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.717+0000 7f972be00700 1 -- 172.21.15.27:0/1655119780 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f973800fd70 con 0x55a62953ada0 2024-05-10T11:24:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.717+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1655119780 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f973c0044e0 con 0x55a62953ada0 2024-05-10T11:24:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.717+0000 7f972be00700 1 -- 172.21.15.27:0/1655119780 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9738003680 con 0x55a62953ada0 2024-05-10T11:24:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.717+0000 7f975b0f7b80 10 client.26912 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.717+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1655119780 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a62954c910 con 0x55a62953ada0 2024-05-10T11:24:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.717+0000 7f972be00700 1 -- 172.21.15.27:0/1655119780 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f973c0044e0 con 0x55a62953ada0 2024-05-10T11:24:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.718+0000 7f972be00700 1 -- 172.21.15.27:0/1655119780 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f9738020050 con 0x55a62953ada0 2024-05-10T11:24:46.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.718+0000 7f975b0f7b80 10 client.26912 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:46.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.718+0000 7f975b0f7b80 10 client.26912 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:24:46.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.718+0000 7f975b0f7b80 20 client.26912 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:46.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.718+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a62954ef50 0x55a629551340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.718+0000 7f975b0f7b80 4 client.26912 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:24:46.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.718+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1655119780 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a629502140 con 0x55a62954ef50 2024-05-10T11:24:46.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.718+0000 7f974de00700 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a62954ef50 0x55a629551340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.719+0000 7f974de00700 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a62954ef50 0x55a629551340 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cc0d54c94e40bee5 server_cookie=15520f33277fd543 in_seq=0 out_seq=0 2024-05-10T11:24:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.719+0000 7f972be00700 10 client.26912 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:24:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f972be00700 1 -- 172.21.15.27:0/1655119780 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a629502140 con 0x55a62954ef50 2024-05-10T11:24:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f972be00700 10 client.26912 handle_command_reply: tid=0 2024-05-10T11:24:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f975b0f7b80 1 client.26912 shutdown 2024-05-10T11:24:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1655119780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 msgr2=0x55a62953e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 0x55a62953e6e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55a62953baa0 tx=0x7f97380047c0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1655119780 shutdown_connections 2024-05-10T11:24:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a62954ef50 0x55a629551340 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a629535a50 0x55a629541340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a62953b6c0 0x55a62953ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/1655119780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 0x55a62953e6e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1655119780 >> 172.21.15.27:0/1655119780 conn(0x55a6294ad6d0 msgr2=0x55a62952a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:46.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f972be00700 0 client.26912 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:24:46.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1655119780 shutdown_connections 2024-05-10T11:24:46.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.720+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1655119780 wait complete. 2024-05-10T11:24:46.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.721+0000 7f975b0f7b80 20 client.26912 trim_cache size 0 max 16384 2024-05-10T11:24:46.729 INFO:tasks.workunit.client.0.smithi027.stdout: 1 694582 6.33 MB/sec execute 451 sec latency 25.436 ms 2024-05-10T11:24:46.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.740+0000 7f975b0f7b80 1 Processor -- start 2024-05-10T11:24:46.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.740+0000 7f975b0f7b80 1 -- start start 2024-05-10T11:24:46.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.740+0000 7f975b0f7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 0x55a629535430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.740+0000 7f975b0f7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 0x55a62954b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.740+0000 7f975b0f7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953b6c0 0x55a62953d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.740+0000 7f975b0f7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f97481b5a50 con 0x55a629535a50 2024-05-10T11:24:46.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.740+0000 7f975b0f7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f97481b7590 con 0x55a62953ada0 2024-05-10T11:24:46.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.740+0000 7f975b0f7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f97481b6120 con 0x55a62953b6c0 2024-05-10T11:24:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.741+0000 7f974de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 0x55a629535430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.741+0000 7f974d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 0x55a62954b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.741+0000 7f974d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 0x55a62954b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:47242/0 (socket says 172.21.15.27:47242) 2024-05-10T11:24:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.741+0000 7f974d400700 1 -- 172.21.15.27:0/1751065305 learned_addr learned my addr 172.21.15.27:0/1751065305 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.741+0000 7f974de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 0x55a629535430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52732/0 (socket says 172.21.15.27:52732) 2024-05-10T11:24:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.741+0000 7f974e800700 1 --2- 172.21.15.27:0/1751065305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953b6c0 0x55a62953d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.741+0000 7f974d400700 1 -- 172.21.15.27:0/1751065305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953b6c0 msgr2=0x55a62953d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.741+0000 7f974d400700 1 --2- 172.21.15.27:0/1751065305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953b6c0 0x55a62953d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.741+0000 7f974d400700 1 -- 172.21.15.27:0/1751065305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 msgr2=0x55a629535430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.741+0000 7f974d400700 1 --2- 172.21.15.27:0/1751065305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 0x55a629535430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.741+0000 7f974d400700 1 -- 172.21.15.27:0/1751065305 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6295309c0 con 0x55a62953ada0 2024-05-10T11:24:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.742+0000 7f974d400700 1 --2- 172.21.15.27:0/1751065305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 0x55a62954b1e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f973800a010 tx=0x7f973800f720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0517a20a90eb0f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.742+0000 7f9736a00700 1 -- 172.21.15.27:0/1751065305 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738013730 con 0x55a62953ada0 2024-05-10T11:24:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.742+0000 7f9736a00700 1 -- 172.21.15.27:0/1751065305 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9738003cd0 con 0x55a62953ada0 2024-05-10T11:24:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.742+0000 7f9736a00700 1 -- 172.21.15.27:0/1751065305 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738010650 con 0x55a62953ada0 2024-05-10T11:24:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.743+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1751065305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 msgr2=0x55a62954b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.743+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/1751065305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 0x55a62954b1e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f973800a010 tx=0x7f973800f720 comp rx=0 tx=0).stop 2024-05-10T11:24:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.743+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1751065305 shutdown_connections 2024-05-10T11:24:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.743+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/1751065305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953b6c0 0x55a62953d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.743+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/1751065305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a629535a50 0x55a629535430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.743+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/1751065305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a62953ada0 0x55a62954b1e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.743+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1751065305 >> 172.21.15.27:0/1751065305 conn(0x55a6294ad6d0 msgr2=0x55a62953fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.743+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1751065305 shutdown_connections 2024-05-10T11:24:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.743+0000 7f975b0f7b80 1 -- 172.21.15.27:0/1751065305 wait complete. 2024-05-10T11:24:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.744+0000 7f975b0f7b80 1 -- start start 2024-05-10T11:24:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.744+0000 7f975b0f7b80 1 Processor -- start 2024-05-10T11:24:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f975b0f7b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a629535a50 0x55a629547740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f975b0f7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953ada0 0x55a629547c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f975b0f7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a62953b6c0 0x55a629555e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f975b0f7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f97481b6ec0 con 0x55a62953b6c0 2024-05-10T11:24:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f975b0f7b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f97481b67f0 con 0x55a629535a50 2024-05-10T11:24:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f975b0f7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f97481b3a10 con 0x55a62953ada0 2024-05-10T11:24:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f975b0f7b80 10 client.0 _ll_register_callbacks cb 0x55a62942ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f974de00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a629535a50 0x55a629547740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f974d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953ada0 0x55a629547c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f974de00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a629535a50 0x55a629547740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:47244/0 (socket says 172.21.15.27:47244) 2024-05-10T11:24:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f974de00700 1 -- 172.21.15.27:0/983243611 learned_addr learned my addr 172.21.15.27:0/983243611 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f974d400700 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953ada0 0x55a629547c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36532/0 (socket says 172.21.15.27:36532) 2024-05-10T11:24:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f974e800700 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a62953b6c0 0x55a629555e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f974de00700 1 -- 172.21.15.27:0/983243611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953ada0 msgr2=0x55a629547c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f974de00700 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953ada0 0x55a629547c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f974de00700 1 -- 172.21.15.27:0/983243611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a62953b6c0 msgr2=0x55a629555e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f974de00700 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a62953b6c0 0x55a629555e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.745+0000 7f974de00700 1 -- 172.21.15.27:0/983243611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a629556670 con 0x55a629535a50 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.746+0000 7f974de00700 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a629535a50 0x55a629547740 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f973c000c00 tx=0x7f973c009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d839ca4d933d912f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.746+0000 7f972be00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.746+0000 7f972be00700 1 -- 172.21.15.27:0/983243611 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c003a00 con 0x55a629535a50 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.746+0000 7f972be00700 1 -- 172.21.15.27:0/983243611 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f973c005040 con 0x55a629535a50 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.746+0000 7f975b0f7b80 1 -- 172.21.15.27:0/983243611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9738003cd0 con 0x55a629535a50 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.746+0000 7f972be00700 1 -- 172.21.15.27:0/983243611 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f973c039050 con 0x55a629535a50 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.746+0000 7f975b0f7b80 10 client.26918 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.746+0000 7f975b0f7b80 1 -- 172.21.15.27:0/983243611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a6295b93e0 con 0x55a629535a50 2024-05-10T11:24:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.747+0000 7f972be00700 1 -- 172.21.15.27:0/983243611 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9738003cd0 con 0x55a629535a50 2024-05-10T11:24:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.747+0000 7f972be00700 1 -- 172.21.15.27:0/983243611 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f973c018070 con 0x55a629535a50 2024-05-10T11:24:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.747+0000 7f975b0f7b80 10 client.26918 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.747+0000 7f975b0f7b80 10 client.26918 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:24:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.747+0000 7f975b0f7b80 20 client.26918 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.747+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a6295b9980 0x55a6295bbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.747+0000 7f975b0f7b80 4 client.26918 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:24:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.747+0000 7f975b0f7b80 1 -- 172.21.15.27:0/983243611 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a629495400 con 0x55a6295b9980 2024-05-10T11:24:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.747+0000 7f974d400700 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a6295b9980 0x55a6295bbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.748+0000 7f974d400700 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a6295b9980 0x55a6295bbda0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a19fcd977e711db server_cookie=72b22026cdf2fe2b in_seq=0 out_seq=0 2024-05-10T11:24:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.748+0000 7f972be00700 10 client.26918 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.748+0000 7f972be00700 1 -- 172.21.15.27:0/983243611 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a629495400 con 0x55a6295b9980 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.748+0000 7f972be00700 10 client.26918 handle_command_reply: tid=0 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.748+0000 7f975b0f7b80 1 client.26918 shutdown 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.749+0000 7f975b0f7b80 1 -- 172.21.15.27:0/983243611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a629535a50 msgr2=0x55a629547740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.749+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a629535a50 0x55a629547740 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f973c000c00 tx=0x7f973c009bb0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.749+0000 7f975b0f7b80 1 -- 172.21.15.27:0/983243611 shutdown_connections 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.749+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a6295b9980 0x55a6295bbda0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.749+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a62953b6c0 0x55a629555e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.749+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a629535a50 0x55a629547740 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.749+0000 7f972be00700 0 client.26918 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.749+0000 7f975b0f7b80 1 --2- 172.21.15.27:0/983243611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a62953ada0 0x55a629547c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.749+0000 7f975b0f7b80 1 -- 172.21.15.27:0/983243611 >> 172.21.15.27:0/983243611 conn(0x55a6294ad6d0 msgr2=0x55a62953c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.749+0000 7f975b0f7b80 1 -- 172.21.15.27:0/983243611 shutdown_connections 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.749+0000 7f975b0f7b80 1 -- 172.21.15.27:0/983243611 wait complete. 2024-05-10T11:24:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.749+0000 7f975b0f7b80 20 client.26918 trim_cache size 0 max 16384 2024-05-10T11:24:46.752 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:24:46.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.750+0000 7f974f200700 1 -- 172.21.15.27:0/3752262420 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f97300b29f0 msgr2=0x7f97300b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:46.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.750+0000 7f974f200700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f97300b29f0 0x7f97300b4eb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9738009880 tx=0x7f9738009bc0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.750+0000 7f974f200700 1 -- 172.21.15.27:0/3752262420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9748114250 msgr2=0x7f97481a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:46.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.750+0000 7f974f200700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9748114250 0x7f97481a4940 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f974400d010 tx=0x7f9744004c80 comp rx=0 tx=0).stop 2024-05-10T11:24:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.750+0000 7f974f200700 1 -- 172.21.15.27:0/3752262420 shutdown_connections 2024-05-10T11:24:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.750+0000 7f974f200700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f97300b29f0 0x7f97300b4eb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.750+0000 7f974f200700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9748114250 0x7f97481a4940 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.750+0000 7f974f200700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97480779d0 0x7f97481a00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.750+0000 7f974f200700 1 --2- 172.21.15.27:0/3752262420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97480782f0 0x7f97481a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.750+0000 7f974f200700 1 -- 172.21.15.27:0/3752262420 >> 172.21.15.27:0/3752262420 conn(0x7f97481010e0 msgr2=0x7f97481034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.750+0000 7f974f200700 1 -- 172.21.15.27:0/3752262420 shutdown_connections 2024-05-10T11:24:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:46.750+0000 7f974f200700 1 -- 172.21.15.27:0/3752262420 wait complete. 2024-05-10T11:24:47.516 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:47 smithi027 ceph-mon[30054]: pgmap v359: 129 pgs: 129 active+clean; 549 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 188 op/s 2024-05-10T11:24:47.585 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:24:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:47 smithi076 ceph-mon[34152]: pgmap v359: 129 pgs: 129 active+clean; 549 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 188 op/s 2024-05-10T11:24:47.729 INFO:tasks.workunit.client.0.smithi027.stdout: 1 696169 6.35 MB/sec execute 452 sec latency 22.041 ms 2024-05-10T11:24:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:47 smithi189 ceph-mon[34842]: pgmap v359: 129 pgs: 129 active+clean; 549 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 188 op/s 2024-05-10T11:24:48.073 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:24:48.730 INFO:tasks.workunit.client.0.smithi027.stdout: 1 696729 6.34 MB/sec execute 453 sec latency 35.075 ms 2024-05-10T11:24:49.617 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:49 smithi027 ceph-mon[30054]: pgmap v360: 129 pgs: 129 active+clean; 549 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 126 op/s 2024-05-10T11:24:49.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:49 smithi076 ceph-mon[34152]: pgmap v360: 129 pgs: 129 active+clean; 549 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 126 op/s 2024-05-10T11:24:49.731 INFO:tasks.workunit.client.0.smithi027.stdout: 1 697955 6.33 MB/sec execute 454 sec latency 22.643 ms 2024-05-10T11:24:49.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:49 smithi189 ceph-mon[34842]: pgmap v360: 129 pgs: 129 active+clean; 549 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 126 op/s 2024-05-10T11:24:50.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.513+0000 7f7f9b600700 1 -- 172.21.15.27:0/2910099848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f94102d60 msgr2=0x7f7f9410f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:50.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.513+0000 7f7f9b600700 1 --2- 172.21.15.27:0/2910099848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f94102d60 0x7f7f9410f250 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400b0a0 tx=0x7f7f8400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.513+0000 7f7f9b600700 1 -- 172.21.15.27:0/2910099848 shutdown_connections 2024-05-10T11:24:50.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.513+0000 7f7f9b600700 1 --2- 172.21.15.27:0/2910099848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f94102d60 0x7f7f9410f250 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.513+0000 7f7f9b600700 1 --2- 172.21.15.27:0/2910099848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f94108410 0x7f7f941087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.513+0000 7f7f9b600700 1 --2- 172.21.15.27:0/2910099848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f941023a0 0x7f7f94102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.513+0000 7f7f9b600700 1 -- 172.21.15.27:0/2910099848 >> 172.21.15.27:0/2910099848 conn(0x7f7f94079350 msgr2=0x7f7f94079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.514+0000 7f7f9b600700 1 -- 172.21.15.27:0/2910099848 shutdown_connections 2024-05-10T11:24:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.514+0000 7f7f9b600700 1 -- 172.21.15.27:0/2910099848 wait complete. 2024-05-10T11:24:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.515+0000 7f7f9b600700 1 Processor -- start 2024-05-10T11:24:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.515+0000 7f7f9b600700 1 -- start start 2024-05-10T11:24:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f9b600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f941023a0 0x7f7f9419be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f9b600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f94102d60 0x7f7f9419c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f9b600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f94108410 0x7f7f941a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f9b600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9406be10 con 0x7f7f94108410 2024-05-10T11:24:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f9b600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9406bc90 con 0x7f7f94102d60 2024-05-10T11:24:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f9b600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9406bf90 con 0x7f7f941023a0 2024-05-10T11:24:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f98c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f941023a0 0x7f7f9419be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f98c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f941023a0 0x7f7f9419be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36552/0 (socket says 172.21.15.27:36552) 2024-05-10T11:24:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f93e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f94102d60 0x7f7f9419c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f98c00700 1 -- 172.21.15.27:0/4200689927 learned_addr learned my addr 172.21.15.27:0/4200689927 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f99600700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f94108410 0x7f7f941a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f98c00700 1 -- 172.21.15.27:0/4200689927 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f94102d60 msgr2=0x7f7f9419c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f98c00700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f94102d60 0x7f7f9419c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f98c00700 1 -- 172.21.15.27:0/4200689927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f94108410 msgr2=0x7f7f941a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f98c00700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f94108410 0x7f7f941a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.516+0000 7f7f98c00700 1 -- 172.21.15.27:0/4200689927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f941a0e40 con 0x7f7f941023a0 2024-05-10T11:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.517+0000 7f7f98c00700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f941023a0 0x7f7f9419be30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00e9a0 tx=0x7f7f8c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.517+0000 7f7f91600700 1 -- 172.21.15.27:0/4200689927 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c00ca10 con 0x7f7f941023a0 2024-05-10T11:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.517+0000 7f7f91600700 1 -- 172.21.15.27:0/4200689927 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f8c00cbb0 con 0x7f7f941023a0 2024-05-10T11:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.517+0000 7f7f91600700 1 -- 172.21.15.27:0/4200689927 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c018680 con 0x7f7f941023a0 2024-05-10T11:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.517+0000 7f7f9b600700 1 -- 172.21.15.27:0/4200689927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f941a1130 con 0x7f7f941023a0 2024-05-10T11:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.517+0000 7f7f9b600700 1 -- 172.21.15.27:0/4200689927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f941a89b0 con 0x7f7f941023a0 2024-05-10T11:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.522+0000 7f7f91600700 1 -- 172.21.15.27:0/4200689927 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f7f8c010430 con 0x7f7f941023a0 2024-05-10T11:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.522+0000 7f7fa5f9ab80 1 Processor -- start 2024-05-10T11:24:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.523+0000 7f7fa5f9ab80 1 -- start start 2024-05-10T11:24:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.523+0000 7f7fa5f9ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e6bc0 0x55757e8e6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.523+0000 7f7fa5f9ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e74e0 0x55757e8e12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.523+0000 7f7fa5f9ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e18f0 0x55757e8e1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.523+0000 7f7fa5f9ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55757e833a20 con 0x55757e8e6bc0 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.523+0000 7f7fa5f9ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55757e7d1760 con 0x55757e8e18f0 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.523+0000 7f7fa5f9ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55757e82f9e0 con 0x55757e8e74e0 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.523+0000 7f7f93e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e18f0 0x55757e8e1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.523+0000 7f7f93e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e18f0 0x55757e8e1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:47252/0 (socket says 172.21.15.27:47252) 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.523+0000 7f7f93e00700 1 -- 172.21.15.27:0/3151643984 learned_addr learned my addr 172.21.15.27:0/3151643984 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.523+0000 7f7f98c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e74e0 0x55757e8e12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.523+0000 7f7f99600700 1 --2- 172.21.15.27:0/3151643984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e6bc0 0x55757e8e6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.524+0000 7f7f93e00700 1 -- 172.21.15.27:0/3151643984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e74e0 msgr2=0x55757e8e12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.524+0000 7f7f93e00700 1 --2- 172.21.15.27:0/3151643984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e74e0 0x55757e8e12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.524+0000 7f7f93e00700 1 -- 172.21.15.27:0/3151643984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e6bc0 msgr2=0x55757e8e6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.524+0000 7f7f93e00700 1 --2- 172.21.15.27:0/3151643984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e6bc0 0x55757e8e6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.524+0000 7f7f93e00700 1 -- 172.21.15.27:0/3151643984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55757e7808d0 con 0x55757e8e18f0 2024-05-10T11:24:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.524+0000 7f7f99600700 1 --2- 172.21.15.27:0/3151643984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e6bc0 0x55757e8e6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:24:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.524+0000 7f7f91600700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7f780b2a70 0x7f7f780b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.524+0000 7f7f93e00700 1 --2- 172.21.15.27:0/3151643984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e18f0 0x55757e8e1d70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009d20 tx=0x7f7f80011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5050cfba91a9d464 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.524+0000 7f7f91600700 1 -- 172.21.15.27:0/4200689927 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f7f8c0e2b40 con 0x7f7f941023a0 2024-05-10T11:24:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.525+0000 7f7f99600700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7f780b2a70 0x7f7f780b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.525+0000 7f7f8a000700 1 -- 172.21.15.27:0/3151643984 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8001e070 con 0x55757e8e18f0 2024-05-10T11:24:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.525+0000 7f7f8a000700 1 -- 172.21.15.27:0/3151643984 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f80019ae0 con 0x55757e8e18f0 2024-05-10T11:24:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.525+0000 7f7f8a000700 1 -- 172.21.15.27:0/3151643984 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8000eb40 con 0x55757e8e18f0 2024-05-10T11:24:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.525+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/3151643984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e18f0 msgr2=0x55757e8e1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.525+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/3151643984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e18f0 0x55757e8e1d70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009d20 tx=0x7f7f80011720 comp rx=0 tx=0).stop 2024-05-10T11:24:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.526+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/3151643984 shutdown_connections 2024-05-10T11:24:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.526+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/3151643984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e18f0 0x55757e8e1d70 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.526+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/3151643984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e6bc0 0x55757e8e6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.526+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/3151643984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e74e0 0x55757e8e12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.526+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/3151643984 >> 172.21.15.27:0/3151643984 conn(0x55757e8596d0 msgr2=0x55757e8dbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.526+0000 7f7f99600700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7f780b2a70 0x7f7f780b4f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400b780 tx=0x7f7f84006210 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.526+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/3151643984 shutdown_connections 2024-05-10T11:24:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.526+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/3151643984 wait complete. 2024-05-10T11:24:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.527+0000 7f7fa5f9ab80 1 -- start start 2024-05-10T11:24:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.527+0000 7f7fa5f9ab80 1 Processor -- start 2024-05-10T11:24:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.527+0000 7f7fa5f9ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e18f0 0x55757e8ed1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.527+0000 7f7fa5f9ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e6bc0 0x55757e8ea560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.527+0000 7f7fa5f9ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e74e0 0x55757e8eaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.527+0000 7f7fa5f9ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55757e8edf20 con 0x55757e8e18f0 2024-05-10T11:24:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.527+0000 7f7fa5f9ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55757e799b20 con 0x55757e8e74e0 2024-05-10T11:24:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.527+0000 7f7fa5f9ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55757e8ee0a0 con 0x55757e8e6bc0 2024-05-10T11:24:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7fa5f9ab80 10 client.0 _ll_register_callbacks cb 0x55757e7dac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7f99600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e74e0 0x55757e8eaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7f99600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e74e0 0x55757e8eaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:47256/0 (socket says 172.21.15.27:47256) 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7f93e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e6bc0 0x55757e8ea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7f99600700 1 -- 172.21.15.27:0/437273908 learned_addr learned my addr 172.21.15.27:0/437273908 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7f98c00700 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e18f0 0x55757e8ed1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7f99600700 1 -- 172.21.15.27:0/437273908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e6bc0 msgr2=0x55757e8ea560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7f99600700 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e6bc0 0x55757e8ea560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7f99600700 1 -- 172.21.15.27:0/437273908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e18f0 msgr2=0x55757e8ed1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7f99600700 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e18f0 0x55757e8ed1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7f99600700 1 -- 172.21.15.27:0/437273908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55757e7f9420 con 0x55757e8e74e0 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7f98c00700 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e18f0 0x55757e8ed1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.528+0000 7f7f93e00700 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e6bc0 0x55757e8ea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.529+0000 7f7f99600700 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e74e0 0x55757e8eaaa0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400d010 tx=0x7f7f840099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa8e4f803fd58fca server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.529+0000 7f7f7ea00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:24:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.529+0000 7f7f7ea00700 1 -- 172.21.15.27:0/437273908 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8402c070 con 0x55757e8e74e0 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.529+0000 7f7f7ea00700 1 -- 172.21.15.27:0/437273908 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f7f840078f0 con 0x55757e8e74e0 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.529+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/437273908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f80019ae0 con 0x55757e8e74e0 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.529+0000 7f7f7ea00700 1 -- 172.21.15.27:0/437273908 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7f84007af0 con 0x55757e8e74e0 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.529+0000 7f7fa5f9ab80 10 client.26930 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.529+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/437273908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55757e8f8780 con 0x55757e8e74e0 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.530+0000 7f7f7ea00700 1 -- 172.21.15.27:0/437273908 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f80019ae0 con 0x55757e8e74e0 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.530+0000 7f7f7ea00700 1 -- 172.21.15.27:0/437273908 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f7f84011070 con 0x55757e8e74e0 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.530+0000 7f7fa5f9ab80 10 client.26930 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.530+0000 7f7fa5f9ab80 10 client.26930 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.530+0000 7f7fa5f9ab80 20 client.26930 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.530+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55757e8fadc0 0x55757e8fd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.530+0000 7f7fa5f9ab80 4 client.26930 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.530+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/437273908 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55757e8ae140 con 0x55757e8fadc0 2024-05-10T11:24:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.531+0000 7f7f98c00700 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55757e8fadc0 0x55757e8fd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.531+0000 7f7f98c00700 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55757e8fadc0 0x55757e8fd1f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4a54b5f6d59b0f98 server_cookie=c8ad0df5102dc627 in_seq=0 out_seq=0 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.531+0000 7f7f7ea00700 10 client.26930 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.532+0000 7f7f7ea00700 1 -- 172.21.15.27:0/437273908 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55757e8ae140 con 0x55757e8fadc0 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.532+0000 7f7f7ea00700 10 client.26930 handle_command_reply: tid=0 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.532+0000 7f7fa5f9ab80 1 client.26930 shutdown 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.532+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/437273908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e74e0 msgr2=0x55757e8eaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.532+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e74e0 0x55757e8eaaa0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400d010 tx=0x7f7f840099e0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.532+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/437273908 shutdown_connections 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.532+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e74e0 0x55757e8eaaa0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.532+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e18f0 0x55757e8ed1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.532+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55757e8fadc0 0x55757e8fd1f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.532+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/437273908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e6bc0 0x55757e8ea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.533+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/437273908 >> 172.21.15.27:0/437273908 conn(0x55757e8596d0 msgr2=0x55757e8db190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.533+0000 7f7f7ea00700 0 client.26930 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:24:50.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.533+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/437273908 shutdown_connections 2024-05-10T11:24:50.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.533+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/437273908 wait complete. 2024-05-10T11:24:50.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.533+0000 7f7fa5f9ab80 20 client.26930 trim_cache size 0 max 16384 2024-05-10T11:24:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.550+0000 7f7fa5f9ab80 1 Processor -- start 2024-05-10T11:24:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.550+0000 7f7fa5f9ab80 1 -- start start 2024-05-10T11:24:50.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.550+0000 7f7fa5f9ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 0x55757e8e12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.550+0000 7f7fa5f9ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 0x55757e8eab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.550+0000 7f7fa5f9ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e74e0 0x55757e8fd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.550+0000 7f7fa5f9ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941b1930 con 0x55757e8e74e0 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.550+0000 7f7fa5f9ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941b3470 con 0x55757e8e6bc0 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.550+0000 7f7fa5f9ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941b2000 con 0x55757e8e18f0 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.551+0000 7f7f93e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 0x55757e8eab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.551+0000 7f7f93e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 0x55757e8eab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:47268/0 (socket says 172.21.15.27:47268) 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.551+0000 7f7f93e00700 1 -- 172.21.15.27:0/781243194 learned_addr learned my addr 172.21.15.27:0/781243194 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.551+0000 7f7f98c00700 1 --2- 172.21.15.27:0/781243194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 0x55757e8e12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.551+0000 7f7f99600700 1 --2- 172.21.15.27:0/781243194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e74e0 0x55757e8fd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.551+0000 7f7f93e00700 1 -- 172.21.15.27:0/781243194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 msgr2=0x55757e8e12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.551+0000 7f7f93e00700 1 --2- 172.21.15.27:0/781243194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 0x55757e8e12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.551+0000 7f7f93e00700 1 -- 172.21.15.27:0/781243194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e74e0 msgr2=0x55757e8fd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.551+0000 7f7f93e00700 1 --2- 172.21.15.27:0/781243194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e74e0 0x55757e8fd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.551+0000 7f7f93e00700 1 -- 172.21.15.27:0/781243194 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55757e8dc7b0 con 0x55757e8e6bc0 2024-05-10T11:24:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.551+0000 7f7f98c00700 1 --2- 172.21.15.27:0/781243194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 0x55757e8e12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:24:50.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.551+0000 7f7f99600700 1 --2- 172.21.15.27:0/781243194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e74e0 0x55757e8fd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.552+0000 7f7f93e00700 1 --2- 172.21.15.27:0/781243194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 0x55757e8eab90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7f80011a70 tx=0x7f7f80009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e09cac331980e12 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.552+0000 7f7f89600700 1 -- 172.21.15.27:0/781243194 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f80027020 con 0x55757e8e6bc0 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.552+0000 7f7f89600700 1 -- 172.21.15.27:0/781243194 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f80003680 con 0x55757e8e6bc0 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.552+0000 7f7f89600700 1 -- 172.21.15.27:0/781243194 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8001e040 con 0x55757e8e6bc0 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.553+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/781243194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 msgr2=0x55757e8eab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.553+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/781243194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 0x55757e8eab90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7f80011a70 tx=0x7f7f80009fd0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.553+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/781243194 shutdown_connections 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.553+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/781243194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 0x55757e8e12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.553+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/781243194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e74e0 0x55757e8fd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.553+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/781243194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 0x55757e8eab90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.553+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/781243194 >> 172.21.15.27:0/781243194 conn(0x55757e8596d0 msgr2=0x55757e8e8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.553+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/781243194 shutdown_connections 2024-05-10T11:24:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.553+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/781243194 wait complete. 2024-05-10T11:24:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.553+0000 7f7fa5f9ab80 1 -- start start 2024-05-10T11:24:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7fa5f9ab80 1 Processor -- start 2024-05-10T11:24:50.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7fa5f9ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 0x55757e8f3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7fa5f9ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 0x55757e901ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7fa5f9ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e74e0 0x55757e902410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7fa5f9ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941b2da0 con 0x55757e8e74e0 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7fa5f9ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941b26d0 con 0x55757e8e6bc0 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7fa5f9ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941af8f0 con 0x55757e8e18f0 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7fa5f9ab80 10 client.0 _ll_register_callbacks cb 0x55757e7dac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7f98c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 0x55757e8f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7f93e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 0x55757e901ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7f99600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e74e0 0x55757e902410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7f98c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 0x55757e8f3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36582/0 (socket says 172.21.15.27:36582) 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7f98c00700 1 -- 172.21.15.27:0/2907692360 learned_addr learned my addr 172.21.15.27:0/2907692360 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.554+0000 7f7f93e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 0x55757e901ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:47282/0 (socket says 172.21.15.27:47282) 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.555+0000 7f7f98c00700 1 -- 172.21.15.27:0/2907692360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 msgr2=0x55757e901ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.555+0000 7f7f98c00700 1 --2- 172.21.15.27:0/2907692360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 0x55757e901ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.555+0000 7f7f98c00700 1 -- 172.21.15.27:0/2907692360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e74e0 msgr2=0x55757e902410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.555+0000 7f7f98c00700 1 --2- 172.21.15.27:0/2907692360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e74e0 0x55757e902410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.555+0000 7f7f98c00700 1 -- 172.21.15.27:0/2907692360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55757e902a20 con 0x55757e8e18f0 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.555+0000 7f7f98c00700 1 --2- 172.21.15.27:0/2907692360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 0x55757e8f3650 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c0a9920 tx=0x7f7f8c0e4f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9f3e208ed2db3b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.555+0000 7f7f7ea00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.555+0000 7f7f7ea00700 1 -- 172.21.15.27:0/2907692360 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c0e60e0 con 0x55757e8e18f0 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.555+0000 7f7f7ea00700 1 -- 172.21.15.27:0/2907692360 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f7f8c0a9bf0 con 0x55757e8e18f0 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.555+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/2907692360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f80003680 con 0x55757e8e18f0 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.556+0000 7f7f7ea00700 1 -- 172.21.15.27:0/2907692360 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7f8c0c3050 con 0x55757e8e18f0 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.556+0000 7f7fa5f9ab80 10 client.26917 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.556+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/2907692360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55757e9654a0 con 0x55757e8e18f0 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.556+0000 7f7f7ea00700 1 -- 172.21.15.27:0/2907692360 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f80003680 con 0x55757e8e18f0 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.556+0000 7f7f7ea00700 1 -- 172.21.15.27:0/2907692360 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f7f8c0b2360 con 0x55757e8e18f0 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.556+0000 7f7fa5f9ab80 10 client.26917 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.556+0000 7f7fa5f9ab80 10 client.26917 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.556+0000 7f7fa5f9ab80 20 client.26917 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.556+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/2907692360 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55757e965b30 0x55757e967f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.556+0000 7f7fa5f9ab80 4 client.26917 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:24:50.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.556+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/2907692360 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55757e841400 con 0x55757e965b30 2024-05-10T11:24:50.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.557+0000 7f7f93e00700 1 --2- 172.21.15.27:0/2907692360 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55757e965b30 0x55757e967f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:50.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.557+0000 7f7f93e00700 1 --2- 172.21.15.27:0/2907692360 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55757e965b30 0x55757e967f50 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a15a7a8970de394c server_cookie=356c91cbfffe4311 in_seq=0 out_seq=0 2024-05-10T11:24:50.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.557+0000 7f7f7ea00700 10 client.26917 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:24:50.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.557+0000 7f7f7ea00700 1 -- 172.21.15.27:0/2907692360 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55757e841400 con 0x55757e965b30 2024-05-10T11:24:50.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.557+0000 7f7f7ea00700 10 client.26917 handle_command_reply: tid=0 2024-05-10T11:24:50.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.557+0000 7f7fa5f9ab80 1 client.26917 shutdown 2024-05-10T11:24:50.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.558+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/2907692360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 msgr2=0x55757e8f3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:50.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.558+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/2907692360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 0x55757e8f3650 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c0a9920 tx=0x7f7f8c0e4f90 comp rx=0 tx=0).stop 2024-05-10T11:24:50.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.558+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/2907692360 shutdown_connections 2024-05-10T11:24:50.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.558+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/2907692360 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55757e965b30 0x55757e967f50 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.558+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/2907692360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55757e8e18f0 0x55757e8f3650 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.558+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/2907692360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55757e8e74e0 0x55757e902410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.558+0000 7f7fa5f9ab80 1 --2- 172.21.15.27:0/2907692360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55757e8e6bc0 0x55757e901ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.558+0000 7f7f7ea00700 0 client.26917 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:24:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.558+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/2907692360 >> 172.21.15.27:0/2907692360 conn(0x55757e8596d0 msgr2=0x55757e8e7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.558+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/2907692360 shutdown_connections 2024-05-10T11:24:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.558+0000 7f7fa5f9ab80 1 -- 172.21.15.27:0/2907692360 wait complete. 2024-05-10T11:24:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.558+0000 7f7fa5f9ab80 20 client.26917 trim_cache size 0 max 16384 2024-05-10T11:24:50.560 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:24:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.559+0000 7f7f9b600700 1 -- 172.21.15.27:0/4200689927 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7f780b2a70 msgr2=0x7f7f780b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.559+0000 7f7f9b600700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7f780b2a70 0x7f7f780b4f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400b780 tx=0x7f7f84006210 comp rx=0 tx=0).stop 2024-05-10T11:24:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.559+0000 7f7f9b600700 1 -- 172.21.15.27:0/4200689927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f941023a0 msgr2=0x7f7f9419be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.559+0000 7f7f9b600700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f941023a0 0x7f7f9419be30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00e9a0 tx=0x7f7f8c00ecb0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.559+0000 7f7f9b600700 1 -- 172.21.15.27:0/4200689927 shutdown_connections 2024-05-10T11:24:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.559+0000 7f7f9b600700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7f780b2a70 0x7f7f780b4f30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.559+0000 7f7f9b600700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f941023a0 0x7f7f9419be30 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.559+0000 7f7f9b600700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f94108410 0x7f7f941a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.559+0000 7f7f9b600700 1 --2- 172.21.15.27:0/4200689927 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f94102d60 0x7f7f9419c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.559+0000 7f7f9b600700 1 -- 172.21.15.27:0/4200689927 >> 172.21.15.27:0/4200689927 conn(0x7f7f94079350 msgr2=0x7f7f9410d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.559+0000 7f7f9b600700 1 -- 172.21.15.27:0/4200689927 shutdown_connections 2024-05-10T11:24:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:50.559+0000 7f7f9b600700 1 -- 172.21.15.27:0/4200689927 wait complete. 2024-05-10T11:24:50.731 INFO:tasks.workunit.client.0.smithi027.stdout: 1 699670 6.35 MB/sec execute 455 sec latency 22.114 ms 2024-05-10T11:24:51.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:51 smithi027 ceph-mon[30054]: pgmap v361: 129 pgs: 129 active+clean; 555 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 181 op/s 2024-05-10T11:24:51.543 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:24:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:51 smithi076 ceph-mon[34152]: pgmap v361: 129 pgs: 129 active+clean; 555 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 181 op/s 2024-05-10T11:24:51.731 INFO:tasks.workunit.client.0.smithi027.stdout: 1 700051 6.34 MB/sec execute 456 sec latency 22.967 ms 2024-05-10T11:24:51.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:51 smithi189 ceph-mon[34842]: pgmap v361: 129 pgs: 129 active+clean; 555 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 181 op/s 2024-05-10T11:24:52.041 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:24:52.731 INFO:tasks.workunit.client.0.smithi027.stdout: 1 701251 6.33 MB/sec execute 457 sec latency 44.976 ms 2024-05-10T11:24:53.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:53 smithi027 ceph-mon[30054]: pgmap v362: 129 pgs: 129 active+clean; 555 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 122 op/s 2024-05-10T11:24:53.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:53 smithi076 ceph-mon[34152]: pgmap v362: 129 pgs: 129 active+clean; 555 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 122 op/s 2024-05-10T11:24:53.731 INFO:tasks.workunit.client.0.smithi027.stdout: 1 702604 6.34 MB/sec execute 458 sec latency 37.680 ms 2024-05-10T11:24:53.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:53 smithi189 ceph-mon[34842]: pgmap v362: 129 pgs: 129 active+clean; 555 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 122 op/s 2024-05-10T11:24:54.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.622+0000 7f28a8400700 1 -- 172.21.15.27:0/3554723254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28a01056c0 msgr2=0x7f28a0105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:54.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.622+0000 7f28a8400700 1 --2- 172.21.15.27:0/3554723254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28a01056c0 0x7f28a0105b40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f288c009990 tx=0x7f288c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.622+0000 7f28a8400700 1 -- 172.21.15.27:0/3554723254 shutdown_connections 2024-05-10T11:24:54.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.622+0000 7f28a8400700 1 --2- 172.21.15.27:0/3554723254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28a010b6c0 0x7f28a010baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.622+0000 7f28a8400700 1 --2- 172.21.15.27:0/3554723254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a0106080 0x7f28a0112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.622+0000 7f28a8400700 1 --2- 172.21.15.27:0/3554723254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28a01056c0 0x7f28a0105b40 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.622+0000 7f28a8400700 1 -- 172.21.15.27:0/3554723254 >> 172.21.15.27:0/3554723254 conn(0x7f28a01011b0 msgr2=0x7f28a01035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:54.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.622+0000 7f28a8400700 1 -- 172.21.15.27:0/3554723254 shutdown_connections 2024-05-10T11:24:54.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.622+0000 7f28a8400700 1 -- 172.21.15.27:0/3554723254 wait complete. 2024-05-10T11:24:54.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.624+0000 7f28a8400700 1 Processor -- start 2024-05-10T11:24:54.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.624+0000 7f28a8400700 1 -- start start 2024-05-10T11:24:54.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.624+0000 7f28a8400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28a01056c0 0x7f28a019bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.624+0000 7f28a8400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a0106080 0x7f28a019c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.624+0000 7f28a8400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28a010b6c0 0x7f28a01a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.624+0000 7f28a8400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f28a0115900 con 0x7f28a0106080 2024-05-10T11:24:54.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.624+0000 7f28a8400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f28a0115780 con 0x7f28a010b6c0 2024-05-10T11:24:54.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.624+0000 7f28a8400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28a0115a80 con 0x7f28a01056c0 2024-05-10T11:24:54.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.625+0000 7f28a5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28a01056c0 0x7f28a019bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.625+0000 7f28a5000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a0106080 0x7f28a019c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.625+0000 7f28a5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28a01056c0 0x7f28a019bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:32898/0 (socket says 172.21.15.27:32898) 2024-05-10T11:24:54.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.625+0000 7f28a6400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28a010b6c0 0x7f28a01a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.625+0000 7f28a5a00700 1 -- 172.21.15.27:0/304161626 learned_addr learned my addr 172.21.15.27:0/304161626 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.625+0000 7f28a6400700 1 -- 172.21.15.27:0/304161626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28a01056c0 msgr2=0x7f28a019bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.625+0000 7f28a6400700 1 --2- 172.21.15.27:0/304161626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28a01056c0 0x7f28a019bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.625+0000 7f28a6400700 1 -- 172.21.15.27:0/304161626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a0106080 msgr2=0x7f28a019c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.625+0000 7f28a6400700 1 --2- 172.21.15.27:0/304161626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a0106080 0x7f28a019c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.625+0000 7f28a6400700 1 -- 172.21.15.27:0/304161626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a01a0da0 con 0x7f28a010b6c0 2024-05-10T11:24:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.625+0000 7f28a6400700 1 --2- 172.21.15.27:0/304161626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28a010b6c0 0x7f28a01a0660 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f289800efc0 tx=0x7f289800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.626+0000 7f289e000700 1 -- 172.21.15.27:0/304161626 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28980092e0 con 0x7f28a010b6c0 2024-05-10T11:24:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.626+0000 7f289e000700 1 -- 172.21.15.27:0/304161626 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2898004bb0 con 0x7f28a010b6c0 2024-05-10T11:24:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.626+0000 7f28a8400700 1 -- 172.21.15.27:0/304161626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28a01a1030 con 0x7f28a010b6c0 2024-05-10T11:24:54.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.626+0000 7f289e000700 1 -- 172.21.15.27:0/304161626 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2898004020 con 0x7f28a010b6c0 2024-05-10T11:24:54.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.626+0000 7f28a8400700 1 -- 172.21.15.27:0/304161626 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28a01a8910 con 0x7f28a010b6c0 2024-05-10T11:24:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.630+0000 7f289e000700 1 -- 172.21.15.27:0/304161626 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f289801b030 con 0x7f28a010b6c0 2024-05-10T11:24:54.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.632+0000 7f28b2d0bb80 1 Processor -- start 2024-05-10T11:24:54.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.632+0000 7f28b2d0bb80 1 -- start start 2024-05-10T11:24:54.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.632+0000 7f289e000700 1 --2- 172.21.15.27:0/304161626 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f28880b2b40 0x7f28880b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.632+0000 7f28b2d0bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe13fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.632+0000 7f289e000700 1 -- 172.21.15.27:0/304161626 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f28980e2420 con 0x7f28a010b6c0 2024-05-10T11:24:54.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.632+0000 7f28b2d0bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe144e0 0x5627dbe0e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.632+0000 7f28b2d0bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe0e8f0 0x5627dbe0ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.632+0000 7f28b2d0bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5627dbd60a20 con 0x5627dbe0e8f0 2024-05-10T11:24:54.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.632+0000 7f28b2d0bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5627dbcfe760 con 0x5627dbe13bc0 2024-05-10T11:24:54.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28b2d0bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5627dbd5c9e0 con 0x5627dbe144e0 2024-05-10T11:24:54.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28a5000700 1 --2- 172.21.15.27:0/304161626 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f28880b2b40 0x7f28880b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28a5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe144e0 0x5627dbe0e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28a5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe144e0 0x5627dbe0e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:32914/0 (socket says 172.21.15.27:32914) 2024-05-10T11:24:54.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28a5000700 1 -- 172.21.15.27:0/2585249430 learned_addr learned my addr 172.21.15.27:0/2585249430 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:54.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28a6400700 1 --2- 172.21.15.27:0/2585249430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe0e8f0 0x5627dbe0ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28a5a00700 1 --2- 172.21.15.27:0/2585249430 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe13fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28a5000700 1 -- 172.21.15.27:0/2585249430 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 msgr2=0x5627dbe13fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28a5000700 1 --2- 172.21.15.27:0/2585249430 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe13fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28a5000700 1 -- 172.21.15.27:0/2585249430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe0e8f0 msgr2=0x5627dbe0ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28a5000700 1 --2- 172.21.15.27:0/2585249430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe0e8f0 0x5627dbe0ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28a5000700 1 -- 172.21.15.27:0/2585249430 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627dbcad8d0 con 0x5627dbe144e0 2024-05-10T11:24:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.633+0000 7f28a6400700 1 --2- 172.21.15.27:0/2585249430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe0e8f0 0x5627dbe0ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:24:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.634+0000 7f28a5000700 1 --2- 172.21.15.27:0/2585249430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe144e0 0x5627dbe0e2d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f288c009990 tx=0x7f288c0057d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fadf061a249b26a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.634+0000 7f28a5000700 1 --2- 172.21.15.27:0/304161626 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f28880b2b40 0x7f28880b5000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f288c015000 tx=0x7f288c023000 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.634+0000 7f2892a00700 1 -- 172.21.15.27:0/2585249430 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c003a40 con 0x5627dbe144e0 2024-05-10T11:24:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.634+0000 7f2892a00700 1 -- 172.21.15.27:0/2585249430 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f288c003be0 con 0x5627dbe144e0 2024-05-10T11:24:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.634+0000 7f2892a00700 1 -- 172.21.15.27:0/2585249430 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c028dc0 con 0x5627dbe144e0 2024-05-10T11:24:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.635+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/2585249430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe144e0 msgr2=0x5627dbe0e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.635+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/2585249430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe144e0 0x5627dbe0e2d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f288c009990 tx=0x7f288c0057d0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.635+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/2585249430 shutdown_connections 2024-05-10T11:24:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.635+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/2585249430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe0e8f0 0x5627dbe0ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.635+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/2585249430 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe13fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.635+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/2585249430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe144e0 0x5627dbe0e2d0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.635+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/2585249430 >> 172.21.15.27:0/2585249430 conn(0x5627dbd866d0 msgr2=0x5627dbe08b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.635+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/2585249430 shutdown_connections 2024-05-10T11:24:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.635+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/2585249430 wait complete. 2024-05-10T11:24:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.636+0000 7f28b2d0bb80 1 -- start start 2024-05-10T11:24:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.636+0000 7f28b2d0bb80 1 Processor -- start 2024-05-10T11:24:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.637+0000 7f28b2d0bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe0e8f0 0x5627dbe1a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.637+0000 7f28b2d0bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe175a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.637+0000 7f28b2d0bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe144e0 0x5627dbe17ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.637+0000 7f28b2d0bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5627dbe1aec0 con 0x5627dbe0e8f0 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.637+0000 7f28b2d0bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5627dbcc6b20 con 0x5627dbe13bc0 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.637+0000 7f28b2d0bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5627dbe1b040 con 0x5627dbe144e0 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.637+0000 7f28b2d0bb80 10 client.0 _ll_register_callbacks cb 0x5627dbd07c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.637+0000 7f28a5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe0e8f0 0x5627dbe1a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.637+0000 7f28a6400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe144e0 0x5627dbe17ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.637+0000 7f28a5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe0e8f0 0x5627dbe1a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45650/0 (socket says 172.21.15.27:45650) 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.637+0000 7f28a5000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe175a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.637+0000 7f28a5a00700 1 -- 172.21.15.27:0/377089907 learned_addr learned my addr 172.21.15.27:0/377089907 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.638+0000 7f28a5000700 1 -- 172.21.15.27:0/377089907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe144e0 msgr2=0x5627dbe17ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.638+0000 7f28a5000700 1 --2- 172.21.15.27:0/377089907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe144e0 0x5627dbe17ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.638+0000 7f28a5000700 1 -- 172.21.15.27:0/377089907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe0e8f0 msgr2=0x5627dbe1a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.638+0000 7f28a5000700 1 --2- 172.21.15.27:0/377089907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe0e8f0 0x5627dbe1a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.638+0000 7f28a5000700 1 -- 172.21.15.27:0/377089907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627dbd26420 con 0x5627dbe13bc0 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.638+0000 7f28a5000700 1 --2- 172.21.15.27:0/377089907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe175a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f288c009350 tx=0x7f288c0050d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=522419703778191 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.638+0000 7f2887400700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.638+0000 7f2887400700 1 -- 172.21.15.27:0/377089907 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c03d020 con 0x5627dbe13bc0 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.638+0000 7f2887400700 1 -- 172.21.15.27:0/377089907 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f288c028460 con 0x5627dbe13bc0 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.638+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/377089907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f288c003be0 con 0x5627dbe13bc0 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.639+0000 7f2887400700 1 -- 172.21.15.27:0/377089907 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f288c045050 con 0x5627dbe13bc0 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.639+0000 7f28b2d0bb80 10 client.26948 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.639+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/377089907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5627dbe1aec0 con 0x5627dbe13bc0 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.639+0000 7f2887400700 1 -- 172.21.15.27:0/377089907 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f288c003be0 con 0x5627dbe13bc0 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.639+0000 7f2887400700 1 -- 172.21.15.27:0/377089907 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f288c02e020 con 0x5627dbe13bc0 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.639+0000 7f28b2d0bb80 10 client.26948 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.639+0000 7f28b2d0bb80 10 client.26948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.639+0000 7f28b2d0bb80 20 client.26948 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.639+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/377089907 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627dbe27a70 0x5627dbe29e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.639+0000 7f28b2d0bb80 4 client.26948 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:24:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.639+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/377089907 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627dbddb140 con 0x5627dbe27a70 2024-05-10T11:24:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.640+0000 7f28a5a00700 1 --2- 172.21.15.27:0/377089907 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627dbe27a70 0x5627dbe29e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.640+0000 7f28a5a00700 1 --2- 172.21.15.27:0/377089907 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627dbe27a70 0x5627dbe29e60 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b01ada1f2563abca server_cookie=60b50801976ec060 in_seq=0 out_seq=0 2024-05-10T11:24:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.640+0000 7f2887400700 10 client.26948 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:24:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.642+0000 7f2887400700 1 -- 172.21.15.27:0/377089907 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5627dbddb140 con 0x5627dbe27a70 2024-05-10T11:24:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.642+0000 7f2887400700 10 client.26948 handle_command_reply: tid=0 2024-05-10T11:24:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.642+0000 7f28b2d0bb80 1 client.26948 shutdown 2024-05-10T11:24:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.642+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/377089907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 msgr2=0x5627dbe175a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.642+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/377089907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe175a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f288c009350 tx=0x7f288c0050d0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.642+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/377089907 shutdown_connections 2024-05-10T11:24:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.642+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/377089907 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627dbe27a70 0x5627dbe29e60 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.642+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/377089907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe144e0 0x5627dbe17ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.642+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/377089907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe0e8f0 0x5627dbe1a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.642+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/377089907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe175a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.642+0000 7f2887400700 0 client.26948 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:24:54.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.642+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/377089907 >> 172.21.15.27:0/377089907 conn(0x5627dbd866d0 msgr2=0x5627dbe08190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:54.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.643+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/377089907 shutdown_connections 2024-05-10T11:24:54.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.643+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/377089907 wait complete. 2024-05-10T11:24:54.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.643+0000 7f28b2d0bb80 20 client.26948 trim_cache size 0 max 16384 2024-05-10T11:24:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.660+0000 7f28b2d0bb80 1 Processor -- start 2024-05-10T11:24:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.660+0000 7f28b2d0bb80 1 -- start start 2024-05-10T11:24:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.660+0000 7f28b2d0bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe0e8f0 0x5627dbe0e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.660+0000 7f28b2d0bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe23f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.660+0000 7f28b2d0bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe144e0 0x5627dbe08910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.660+0000 7f28b2d0bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01b16f0 con 0x5627dbe144e0 2024-05-10T11:24:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.660+0000 7f28b2d0bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01b3230 con 0x5627dbe13bc0 2024-05-10T11:24:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.660+0000 7f28b2d0bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01b1dc0 con 0x5627dbe0e8f0 2024-05-10T11:24:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.661+0000 7f28a5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe0e8f0 0x5627dbe0e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.661+0000 7f28a5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe0e8f0 0x5627dbe0e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:32946/0 (socket says 172.21.15.27:32946) 2024-05-10T11:24:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.661+0000 7f28a5000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe23f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.661+0000 7f28a5a00700 1 -- 172.21.15.27:0/2928247662 learned_addr learned my addr 172.21.15.27:0/2928247662 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.661+0000 7f28a6400700 1 --2- 172.21.15.27:0/2928247662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe144e0 0x5627dbe08910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.661+0000 7f28a5a00700 1 -- 172.21.15.27:0/2928247662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 msgr2=0x5627dbe23f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.661+0000 7f28a5a00700 1 --2- 172.21.15.27:0/2928247662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe23f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.661+0000 7f28a5a00700 1 -- 172.21.15.27:0/2928247662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe144e0 msgr2=0x5627dbe08910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.661+0000 7f28a5a00700 1 --2- 172.21.15.27:0/2928247662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe144e0 0x5627dbe08910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.661+0000 7f28a5a00700 1 -- 172.21.15.27:0/2928247662 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627dbe03ab0 con 0x5627dbe0e8f0 2024-05-10T11:24:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.661+0000 7f28a5000700 1 --2- 172.21.15.27:0/2928247662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe23f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:24:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.662+0000 7f28a5a00700 1 --2- 172.21.15.27:0/2928247662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe0e8f0 0x5627dbe0e2d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2894007fd0 tx=0x7f2894007480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=472bd2117f2afd29 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.662+0000 7f2892000700 1 -- 172.21.15.27:0/2928247662 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2894026070 con 0x5627dbe0e8f0 2024-05-10T11:24:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.662+0000 7f2892000700 1 -- 172.21.15.27:0/2928247662 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2894003680 con 0x5627dbe0e8f0 2024-05-10T11:24:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.662+0000 7f2892000700 1 -- 172.21.15.27:0/2928247662 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28940204a0 con 0x5627dbe0e8f0 2024-05-10T11:24:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.663+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/2928247662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe0e8f0 msgr2=0x5627dbe0e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.663+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/2928247662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe0e8f0 0x5627dbe0e2d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2894007fd0 tx=0x7f2894007480 comp rx=0 tx=0).stop 2024-05-10T11:24:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.663+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/2928247662 shutdown_connections 2024-05-10T11:24:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.663+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/2928247662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe0e8f0 0x5627dbe0e2d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.663+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/2928247662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe144e0 0x5627dbe08910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.663+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/2928247662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe13bc0 0x5627dbe23f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.663+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/2928247662 >> 172.21.15.27:0/2928247662 conn(0x5627dbd866d0 msgr2=0x5627dbe18c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.663+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/2928247662 shutdown_connections 2024-05-10T11:24:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.663+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/2928247662 wait complete. 2024-05-10T11:24:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.663+0000 7f28b2d0bb80 1 -- start start 2024-05-10T11:24:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.664+0000 7f28b2d0bb80 1 Processor -- start 2024-05-10T11:24:54.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.664+0000 7f28b2d0bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe0e8f0 0x5627dbe202d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.664+0000 7f28b2d0bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe13bc0 0x5627dbe20810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.664+0000 7f28b2d0bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe144e0 0x5627dbe1d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.664+0000 7f28b2d0bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01b2b60 con 0x5627dbe144e0 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.664+0000 7f28b2d0bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01b2490 con 0x5627dbe0e8f0 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.664+0000 7f28b2d0bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28a01af6b0 con 0x5627dbe13bc0 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.664+0000 7f28b2d0bb80 10 client.0 _ll_register_callbacks cb 0x5627dbd07c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe13bc0 0x5627dbe20810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a5a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe0e8f0 0x5627dbe202d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a5a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe0e8f0 0x5627dbe202d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57082/0 (socket says 172.21.15.27:57082) 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a5a00700 1 -- 172.21.15.27:0/3866982858 learned_addr learned my addr 172.21.15.27:0/3866982858 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe13bc0 0x5627dbe20810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:32948/0 (socket says 172.21.15.27:32948) 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a6400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe144e0 0x5627dbe1d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a5a00700 1 -- 172.21.15.27:0/3866982858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe13bc0 msgr2=0x5627dbe20810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a5a00700 1 --2- 172.21.15.27:0/3866982858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe13bc0 0x5627dbe20810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a5a00700 1 -- 172.21.15.27:0/3866982858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe144e0 msgr2=0x5627dbe1d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a5a00700 1 --2- 172.21.15.27:0/3866982858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe144e0 0x5627dbe1d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a5a00700 1 -- 172.21.15.27:0/3866982858 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627dbe1d890 con 0x5627dbe0e8f0 2024-05-10T11:24:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a5000700 1 --2- 172.21.15.27:0/3866982858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe13bc0 0x5627dbe20810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:24:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f28a5a00700 1 --2- 172.21.15.27:0/3866982858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe0e8f0 0x5627dbe202d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2894005830 tx=0x7f2894007c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92b1fe3b7c8ed51 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.665+0000 7f2887400700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:24:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.666+0000 7f2887400700 1 -- 172.21.15.27:0/3866982858 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2894026050 con 0x5627dbe0e8f0 2024-05-10T11:24:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.666+0000 7f2887400700 1 -- 172.21.15.27:0/3866982858 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7f2894003830 con 0x5627dbe0e8f0 2024-05-10T11:24:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.666+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/3866982858 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2894003680 con 0x5627dbe0e8f0 2024-05-10T11:24:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.666+0000 7f2887400700 1 -- 172.21.15.27:0/3866982858 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2894003a30 con 0x5627dbe0e8f0 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.666+0000 7f28b2d0bb80 10 client.26954 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.666+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/3866982858 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5627dbe92a70 con 0x5627dbe0e8f0 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.666+0000 7f2887400700 1 -- 172.21.15.27:0/3866982858 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2894003680 con 0x5627dbe0e8f0 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.667+0000 7f2887400700 1 -- 172.21.15.27:0/3866982858 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f2894033030 con 0x5627dbe0e8f0 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.667+0000 7f28b2d0bb80 10 client.26954 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.667+0000 7f28b2d0bb80 10 client.26954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.667+0000 7f28b2d0bb80 20 client.26954 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.667+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/3866982858 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627dbe930c0 0x5627dbe954f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.667+0000 7f28b2d0bb80 4 client.26954 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.667+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/3866982858 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5627dbd6e400 con 0x5627dbe930c0 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.667+0000 7f28a5000700 1 --2- 172.21.15.27:0/3866982858 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627dbe930c0 0x5627dbe954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f28a5000700 1 --2- 172.21.15.27:0/3866982858 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627dbe930c0 0x5627dbe954f0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d4d402160280aab server_cookie=5ec506cd377087b6 in_seq=0 out_seq=0 2024-05-10T11:24:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f2887400700 10 client.26954 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f2887400700 1 -- 172.21.15.27:0/3866982858 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5627dbd6e400 con 0x5627dbe930c0 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f2887400700 10 client.26954 handle_command_reply: tid=0 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f28b2d0bb80 1 client.26954 shutdown 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/3866982858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe0e8f0 msgr2=0x5627dbe202d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/3866982858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe0e8f0 0x5627dbe202d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2894005830 tx=0x7f2894007c40 comp rx=0 tx=0).stop 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/3866982858 shutdown_connections 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/3866982858 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5627dbe930c0 0x5627dbe954f0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/3866982858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5627dbe144e0 0x5627dbe1d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/3866982858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627dbe0e8f0 0x5627dbe202d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f28b2d0bb80 1 --2- 172.21.15.27:0/3866982858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627dbe13bc0 0x5627dbe20810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f2887400700 0 client.26954 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/3866982858 >> 172.21.15.27:0/3866982858 conn(0x5627dbd866d0 msgr2=0x5627dbe12fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/3866982858 shutdown_connections 2024-05-10T11:24:54.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.668+0000 7f28b2d0bb80 1 -- 172.21.15.27:0/3866982858 wait complete. 2024-05-10T11:24:54.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.669+0000 7f28b2d0bb80 20 client.26954 trim_cache size 0 max 16384 2024-05-10T11:24:54.677 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:24:54.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.669+0000 7f28a8400700 1 -- 172.21.15.27:0/304161626 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f28880b2b40 msgr2=0x7f28880b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:54.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.669+0000 7f28a8400700 1 --2- 172.21.15.27:0/304161626 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f28880b2b40 0x7f28880b5000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f288c015000 tx=0x7f288c023000 comp rx=0 tx=0).stop 2024-05-10T11:24:54.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.669+0000 7f28a8400700 1 -- 172.21.15.27:0/304161626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28a010b6c0 msgr2=0x7f28a01a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:54.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.669+0000 7f28a8400700 1 --2- 172.21.15.27:0/304161626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28a010b6c0 0x7f28a01a0660 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f289800efc0 tx=0x7f289800c3a0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.669+0000 7f28a8400700 1 -- 172.21.15.27:0/304161626 shutdown_connections 2024-05-10T11:24:54.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.669+0000 7f28a8400700 1 --2- 172.21.15.27:0/304161626 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f28880b2b40 0x7f28880b5000 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.669+0000 7f28a8400700 1 --2- 172.21.15.27:0/304161626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28a01056c0 0x7f28a019bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.669+0000 7f28a8400700 1 --2- 172.21.15.27:0/304161626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28a010b6c0 0x7f28a01a0660 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.669+0000 7f28a8400700 1 --2- 172.21.15.27:0/304161626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a0106080 0x7f28a019c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:54.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.669+0000 7f28a8400700 1 -- 172.21.15.27:0/304161626 >> 172.21.15.27:0/304161626 conn(0x7f28a01011b0 msgr2=0x7f28a01018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:54.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.670+0000 7f28a8400700 1 -- 172.21.15.27:0/304161626 shutdown_connections 2024-05-10T11:24:54.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:54.670+0000 7f28a8400700 1 -- 172.21.15.27:0/304161626 wait complete. 2024-05-10T11:24:54.734 INFO:tasks.workunit.client.0.smithi027.stdout: 1 703356 6.34 MB/sec execute 459 sec latency 27.666 ms 2024-05-10T11:24:55.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:55 smithi027 ceph-mon[30054]: pgmap v363: 129 pgs: 129 active+clean; 555 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 122 op/s 2024-05-10T11:24:55.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:55 smithi076 ceph-mon[34152]: pgmap v363: 129 pgs: 129 active+clean; 555 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 122 op/s 2024-05-10T11:24:55.731 INFO:tasks.workunit.client.0.smithi027.stdout: 1 703814 6.32 MB/sec execute 460 sec latency 50.861 ms 2024-05-10T11:24:55.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:55 smithi189 ceph-mon[34842]: pgmap v363: 129 pgs: 129 active+clean; 555 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 122 op/s 2024-05-10T11:24:56.731 INFO:tasks.workunit.client.0.smithi027.stdout: 1 704906 6.32 MB/sec execute 461 sec latency 34.806 ms 2024-05-10T11:24:56.864 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:24:56.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:24:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:24:57.575 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:57 smithi027 ceph-mon[30054]: pgmap v364: 129 pgs: 129 active+clean; 560 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 170 op/s 2024-05-10T11:24:57.575 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:57 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:24:57.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:57 smithi076 ceph-mon[34152]: pgmap v364: 129 pgs: 129 active+clean; 560 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 170 op/s 2024-05-10T11:24:57.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:57 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:24:57.731 INFO:tasks.workunit.client.0.smithi027.stdout: 1 706687 6.33 MB/sec execute 462 sec latency 26.612 ms 2024-05-10T11:24:57.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:57 smithi189 ceph-mon[34842]: pgmap v364: 129 pgs: 129 active+clean; 560 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 170 op/s 2024-05-10T11:24:57.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:57 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:24:58.594 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:58 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:24:58.594 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:58 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:24:58.594 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:58 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:24:58.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:58 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:24:58.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:58 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:24:58.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:58 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:24:58.731 INFO:tasks.workunit.client.0.smithi027.stdout: 1 707123 6.32 MB/sec execute 463 sec latency 20.867 ms 2024-05-10T11:24:58.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:58 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:24:58.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:58 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:24:58.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:58 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:24:59.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:24:59 smithi027 ceph-mon[30054]: pgmap v365: 129 pgs: 129 active+clean; 560 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 103 op/s 2024-05-10T11:24:59.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:24:59 smithi076 ceph-mon[34152]: pgmap v365: 129 pgs: 129 active+clean; 560 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 103 op/s 2024-05-10T11:24:59.732 INFO:tasks.workunit.client.0.smithi027.stdout: 1 708232 6.32 MB/sec execute 464 sec latency 39.227 ms 2024-05-10T11:24:59.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:24:59 smithi189 ceph-mon[34842]: pgmap v365: 129 pgs: 129 active+clean; 560 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 103 op/s 2024-05-10T11:24:59.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.845+0000 7fa132600700 1 -- 172.21.15.27:0/3561119840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa12c1056c0 msgr2=0x7fa12c105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:59.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.845+0000 7fa132600700 1 --2- 172.21.15.27:0/3561119840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa12c1056c0 0x7fa12c105b40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa118009990 tx=0x7fa118009ca0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.845+0000 7fa132600700 1 -- 172.21.15.27:0/3561119840 shutdown_connections 2024-05-10T11:24:59.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.845+0000 7fa132600700 1 --2- 172.21.15.27:0/3561119840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa12c106080 0x7fa12c112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.845+0000 7fa132600700 1 --2- 172.21.15.27:0/3561119840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa12c10b6c0 0x7fa12c10baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.845+0000 7fa132600700 1 --2- 172.21.15.27:0/3561119840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa12c1056c0 0x7fa12c105b40 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.845+0000 7fa132600700 1 -- 172.21.15.27:0/3561119840 >> 172.21.15.27:0/3561119840 conn(0x7fa12c1011b0 msgr2=0x7fa12c1035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:59.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.845+0000 7fa132600700 1 -- 172.21.15.27:0/3561119840 shutdown_connections 2024-05-10T11:24:59.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.845+0000 7fa132600700 1 -- 172.21.15.27:0/3561119840 wait complete. 2024-05-10T11:24:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.846+0000 7fa132600700 1 Processor -- start 2024-05-10T11:24:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.846+0000 7fa132600700 1 -- start start 2024-05-10T11:24:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa132600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa12c1056c0 0x7fa12c19bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa132600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa12c106080 0x7fa12c19c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa132600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa12c10b6c0 0x7fa12c1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa132600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c115900 con 0x7fa12c106080 2024-05-10T11:24:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa132600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c115780 con 0x7fa12c1056c0 2024-05-10T11:24:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa132600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c115a80 con 0x7fa12c10b6c0 2024-05-10T11:24:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa12b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa12c1056c0 0x7fa12c19bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa12b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa12c1056c0 0x7fa12c19bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57094/0 (socket says 172.21.15.27:57094) 2024-05-10T11:24:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa12b400700 1 -- 172.21.15.27:0/1121570484 learned_addr learned my addr 172.21.15.27:0/1121570484 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa12aa00700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa12c106080 0x7fa12c19c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa12be00700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa12c10b6c0 0x7fa12c1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa12b400700 1 -- 172.21.15.27:0/1121570484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa12c10b6c0 msgr2=0x7fa12c1a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa12b400700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa12c10b6c0 0x7fa12c1a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa12b400700 1 -- 172.21.15.27:0/1121570484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa12c106080 msgr2=0x7fa12c19c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa12b400700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa12c106080 0x7fa12c19c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa12b400700 1 -- 172.21.15.27:0/1121570484 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12c1a0dc0 con 0x7fa12c1056c0 2024-05-10T11:24:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.847+0000 7fa12aa00700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa12c106080 0x7fa12c19c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:24:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.848+0000 7fa12b400700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa12c1056c0 0x7fa12c19bdb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00eb30 tx=0x7fa11c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:59.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.848+0000 7fa123e00700 1 -- 172.21.15.27:0/1121570484 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c00cc90 con 0x7fa12c1056c0 2024-05-10T11:24:59.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.848+0000 7fa123e00700 1 -- 172.21.15.27:0/1121570484 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11c0044d0 con 0x7fa12c1056c0 2024-05-10T11:24:59.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.848+0000 7fa123e00700 1 -- 172.21.15.27:0/1121570484 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c010430 con 0x7fa12c1056c0 2024-05-10T11:24:59.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.848+0000 7fa132600700 1 -- 172.21.15.27:0/1121570484 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa12c1a10b0 con 0x7fa12c1056c0 2024-05-10T11:24:59.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.848+0000 7fa132600700 1 -- 172.21.15.27:0/1121570484 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa12c1a8930 con 0x7fa12c1056c0 2024-05-10T11:24:59.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.853+0000 7fa123e00700 1 -- 172.21.15.27:0/1121570484 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa11c003ea0 con 0x7fa12c1056c0 2024-05-10T11:24:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.855+0000 7fa13ceceb80 1 Processor -- start 2024-05-10T11:24:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.855+0000 7fa13ceceb80 1 -- start start 2024-05-10T11:24:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.855+0000 7fa13ceceb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22490c40 0x55df22491020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.855+0000 7fa13ceceb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22491560 0x55df2248b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.855+0000 7fa13ceceb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df2248bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.855+0000 7fa13ceceb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55df223ddaa0 con 0x55df22490c40 2024-05-10T11:24:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.855+0000 7fa13ceceb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55df2237b7e0 con 0x55df2248b970 2024-05-10T11:24:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.855+0000 7fa13ceceb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55df223d9a60 con 0x55df22491560 2024-05-10T11:24:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.855+0000 7fa123e00700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa1100b2a70 0x7fa1100b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.856+0000 7fa12be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22490c40 0x55df22491020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.856+0000 7fa12b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22491560 0x55df2248b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.856+0000 7fa12be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22490c40 0x55df22491020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45698/0 (socket says 172.21.15.27:45698) 2024-05-10T11:24:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.856+0000 7fa12be00700 1 -- 172.21.15.27:0/1413002830 learned_addr learned my addr 172.21.15.27:0/1413002830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.856+0000 7fa123e00700 1 -- 172.21.15.27:0/1121570484 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7fa11c0e2c30 con 0x7fa12c1056c0 2024-05-10T11:24:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.856+0000 7fa12aa00700 1 --2- 172.21.15.27:0/1413002830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df2248bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.856+0000 7fa12b400700 1 -- 172.21.15.27:0/1413002830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 msgr2=0x55df2248bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.856+0000 7fa12b400700 1 --2- 172.21.15.27:0/1413002830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df2248bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.856+0000 7fa12b400700 1 -- 172.21.15.27:0/1413002830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22490c40 msgr2=0x55df22491020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.856+0000 7fa12b400700 1 --2- 172.21.15.27:0/1413002830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22490c40 0x55df22491020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.856+0000 7fa12be00700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa1100b2a70 0x7fa1100b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.856+0000 7fa12b400700 1 -- 172.21.15.27:0/1413002830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df2232a950 con 0x55df22491560 2024-05-10T11:24:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.857+0000 7fa12b400700 1 --2- 172.21.15.27:0/1413002830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22491560 0x55df2248b350 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0dfef0 tx=0x7fa11c074250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=710e8ff321e0b174 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.857+0000 7fa120c00700 1 -- 172.21.15.27:0/1413002830 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0a5210 con 0x55df22491560 2024-05-10T11:24:59.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.857+0000 7fa120c00700 1 -- 172.21.15.27:0/1413002830 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11c009500 con 0x55df22491560 2024-05-10T11:24:59.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.857+0000 7fa120c00700 1 -- 172.21.15.27:0/1413002830 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0a9a80 con 0x55df22491560 2024-05-10T11:24:59.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.858+0000 7fa13ceceb80 1 -- 172.21.15.27:0/1413002830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22491560 msgr2=0x55df2248b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:59.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.858+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/1413002830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22491560 0x55df2248b350 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0dfef0 tx=0x7fa11c074250 comp rx=0 tx=0).stop 2024-05-10T11:24:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.858+0000 7fa12be00700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa1100b2a70 0x7fa1100b4f30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa12400fb80 tx=0x7fa124004b40 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.858+0000 7fa13ceceb80 1 -- 172.21.15.27:0/1413002830 shutdown_connections 2024-05-10T11:24:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.858+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/1413002830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df2248bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.858+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/1413002830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22490c40 0x55df22491020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.858+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/1413002830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22491560 0x55df2248b350 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.858+0000 7fa13ceceb80 1 -- 172.21.15.27:0/1413002830 >> 172.21.15.27:0/1413002830 conn(0x55df22403750 msgr2=0x55df22485b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.858+0000 7fa13ceceb80 1 -- 172.21.15.27:0/1413002830 shutdown_connections 2024-05-10T11:24:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.858+0000 7fa13ceceb80 1 -- 172.21.15.27:0/1413002830 wait complete. 2024-05-10T11:24:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.859+0000 7fa13ceceb80 1 -- start start 2024-05-10T11:24:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.859+0000 7fa13ceceb80 1 Processor -- start 2024-05-10T11:24:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.860+0000 7fa13ceceb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df22497250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.860+0000 7fa13ceceb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22490c40 0x55df224945f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.860+0000 7fa13ceceb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22491560 0x55df22494b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.860+0000 7fa13ceceb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55df22497ee0 con 0x55df22490c40 2024-05-10T11:24:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.860+0000 7fa13ceceb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55df22343ba0 con 0x55df2248b970 2024-05-10T11:24:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.860+0000 7fa13ceceb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55df22498060 con 0x55df22491560 2024-05-10T11:24:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.860+0000 7fa13ceceb80 10 client.0 _ll_register_callbacks cb 0x55df22384cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.860+0000 7fa12b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df22497250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.860+0000 7fa12be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22491560 0x55df22494b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.860+0000 7fa12b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df22497250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57116/0 (socket says 172.21.15.27:57116) 2024-05-10T11:24:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.860+0000 7fa12b400700 1 -- 172.21.15.27:0/672130652 learned_addr learned my addr 172.21.15.27:0/672130652 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.860+0000 7fa12aa00700 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22490c40 0x55df224945f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa12b400700 1 -- 172.21.15.27:0/672130652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22491560 msgr2=0x55df22494b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:59.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa12b400700 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22491560 0x55df22494b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa12b400700 1 -- 172.21.15.27:0/672130652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22490c40 msgr2=0x55df224945f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:59.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa12b400700 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22490c40 0x55df224945f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa12b400700 1 -- 172.21.15.27:0/672130652 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df223a34a0 con 0x55df2248b970 2024-05-10T11:24:59.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa12aa00700 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22490c40 0x55df224945f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:24:59.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa12b400700 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df22497250 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0a9780 tx=0x7fa11c0038b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bde0cabeead7505 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:59.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa115600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:24:59.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa115600700 1 -- 172.21.15.27:0/672130652 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0a9590 con 0x55df2248b970 2024-05-10T11:24:59.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa115600700 1 -- 172.21.15.27:0/672130652 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7fa11c0e0170 con 0x55df2248b970 2024-05-10T11:24:59.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa13ceceb80 1 -- 172.21.15.27:0/672130652 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa11c009500 con 0x55df2248b970 2024-05-10T11:24:59.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa115600700 1 -- 172.21.15.27:0/672130652 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fa11c0e0370 con 0x55df2248b970 2024-05-10T11:24:59.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa13ceceb80 10 client.26966 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:59.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.861+0000 7fa13ceceb80 1 -- 172.21.15.27:0/672130652 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55df224a2750 con 0x55df2248b970 2024-05-10T11:24:59.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.862+0000 7fa115600700 1 -- 172.21.15.27:0/672130652 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11c009500 con 0x55df2248b970 2024-05-10T11:24:59.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.862+0000 7fa115600700 1 -- 172.21.15.27:0/672130652 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fa11c0a9ce0 con 0x55df2248b970 2024-05-10T11:24:59.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.862+0000 7fa13ceceb80 10 client.26966 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:59.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.862+0000 7fa13ceceb80 10 client.26966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:24:59.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.862+0000 7fa13ceceb80 20 client.26966 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:59.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.862+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df224a4c20 0x55df224a7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.862+0000 7fa13ceceb80 4 client.26966 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:24:59.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.862+0000 7fa13ceceb80 1 -- 172.21.15.27:0/672130652 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df224581c0 con 0x55df224a4c20 2024-05-10T11:24:59.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.862+0000 7fa12aa00700 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df224a4c20 0x55df224a7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.863+0000 7fa12aa00700 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df224a4c20 0x55df224a7010 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73231076b99165bf server_cookie=6b8672a561898ae1 in_seq=0 out_seq=0 2024-05-10T11:24:59.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.863+0000 7fa115600700 10 client.26966 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:24:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.864+0000 7fa115600700 1 -- 172.21.15.27:0/672130652 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55df224581c0 con 0x55df224a4c20 2024-05-10T11:24:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.864+0000 7fa115600700 10 client.26966 handle_command_reply: tid=0 2024-05-10T11:24:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.864+0000 7fa13ceceb80 1 client.26966 shutdown 2024-05-10T11:24:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.865+0000 7fa13ceceb80 1 -- 172.21.15.27:0/672130652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 msgr2=0x55df22497250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.865+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df22497250 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0a9780 tx=0x7fa11c0038b0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.865+0000 7fa13ceceb80 1 -- 172.21.15.27:0/672130652 shutdown_connections 2024-05-10T11:24:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.865+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df224a4c20 0x55df224a7010 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.865+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22491560 0x55df22494b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.865+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df22497250 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.865+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/672130652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22490c40 0x55df224945f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.865+0000 7fa13ceceb80 1 -- 172.21.15.27:0/672130652 >> 172.21.15.27:0/672130652 conn(0x55df22403750 msgr2=0x55df22485210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.865+0000 7fa115600700 0 client.26966 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:24:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.865+0000 7fa13ceceb80 1 -- 172.21.15.27:0/672130652 shutdown_connections 2024-05-10T11:24:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.865+0000 7fa13ceceb80 1 -- 172.21.15.27:0/672130652 wait complete. 2024-05-10T11:24:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.865+0000 7fa13ceceb80 20 client.26966 trim_cache size 0 max 16384 2024-05-10T11:24:59.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.892+0000 7fa13ceceb80 1 Processor -- start 2024-05-10T11:24:59.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.892+0000 7fa13ceceb80 1 -- start start 2024-05-10T11:24:59.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.893+0000 7fa13ceceb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df2248b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.893+0000 7fa13ceceb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22490c40 0x55df22485990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.893+0000 7fa13ceceb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22491560 0x55df224945b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.893+0000 7fa13ceceb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c1b19b0 con 0x55df22491560 2024-05-10T11:24:59.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.893+0000 7fa13ceceb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c1b34f0 con 0x55df2248b970 2024-05-10T11:24:59.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.893+0000 7fa13ceceb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c1b2080 con 0x55df22490c40 2024-05-10T11:24:59.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.893+0000 7fa12b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df2248b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.893+0000 7fa12b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df2248b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57122/0 (socket says 172.21.15.27:57122) 2024-05-10T11:24:59.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.893+0000 7fa12b400700 1 -- 172.21.15.27:0/1132879064 learned_addr learned my addr 172.21.15.27:0/1132879064 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:59.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.893+0000 7fa12aa00700 1 --2- 172.21.15.27:0/1132879064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22490c40 0x55df22485990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.894+0000 7fa12be00700 1 --2- 172.21.15.27:0/1132879064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22491560 0x55df224945b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.894+0000 7fa12b400700 1 -- 172.21.15.27:0/1132879064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22490c40 msgr2=0x55df22485990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:59.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.894+0000 7fa12b400700 1 --2- 172.21.15.27:0/1132879064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22490c40 0x55df22485990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.894+0000 7fa12b400700 1 -- 172.21.15.27:0/1132879064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22491560 msgr2=0x55df224945b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.894+0000 7fa12b400700 1 --2- 172.21.15.27:0/1132879064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22491560 0x55df224945b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.894+0000 7fa12b400700 1 -- 172.21.15.27:0/1132879064 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df22480b30 con 0x55df2248b970 2024-05-10T11:24:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.894+0000 7fa12b400700 1 --2- 172.21.15.27:0/1132879064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df2248b350 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0a9a10 tx=0x7fa11c0e2ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb0db2e9f48738e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.895+0000 7fa117e00700 1 -- 172.21.15.27:0/1132879064 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0a9ad0 con 0x55df2248b970 2024-05-10T11:24:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.895+0000 7fa117e00700 1 -- 172.21.15.27:0/1132879064 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11c0c2070 con 0x55df2248b970 2024-05-10T11:24:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.895+0000 7fa117e00700 1 -- 172.21.15.27:0/1132879064 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0b28e0 con 0x55df2248b970 2024-05-10T11:24:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.895+0000 7fa13ceceb80 1 -- 172.21.15.27:0/1132879064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 msgr2=0x55df2248b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.895+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/1132879064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df2248b350 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0a9a10 tx=0x7fa11c0e2ed0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.896+0000 7fa13ceceb80 1 -- 172.21.15.27:0/1132879064 shutdown_connections 2024-05-10T11:24:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.896+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/1132879064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df22491560 0x55df224945b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.896+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/1132879064 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df2248b970 0x55df2248b350 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.896+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/1132879064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22490c40 0x55df22485990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.896+0000 7fa13ceceb80 1 -- 172.21.15.27:0/1132879064 >> 172.21.15.27:0/1132879064 conn(0x55df22403750 msgr2=0x55df22493ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:59.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.896+0000 7fa13ceceb80 1 -- 172.21.15.27:0/1132879064 shutdown_connections 2024-05-10T11:24:59.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.896+0000 7fa13ceceb80 1 -- 172.21.15.27:0/1132879064 wait complete. 2024-05-10T11:24:59.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.896+0000 7fa13ceceb80 1 -- start start 2024-05-10T11:24:59.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.896+0000 7fa13ceceb80 1 Processor -- start 2024-05-10T11:24:59.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.897+0000 7fa13ceceb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df2248b970 0x55df224a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.897+0000 7fa13ceceb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22490c40 0x55df224a9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.897+0000 7fa13ceceb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df22491560 0x55df2250f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.897+0000 7fa13ceceb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c1b2e20 con 0x55df2248b970 2024-05-10T11:24:59.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.897+0000 7fa13ceceb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c1b2750 con 0x55df22491560 2024-05-10T11:24:59.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.897+0000 7fa13ceceb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c1af970 con 0x55df22490c40 2024-05-10T11:24:59.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.897+0000 7fa13ceceb80 10 client.0 _ll_register_callbacks cb 0x55df22384cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:24:59.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.897+0000 7fa12be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df22491560 0x55df2250f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.897+0000 7fa12be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df22491560 0x55df2250f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:57130/0 (socket says 172.21.15.27:57130) 2024-05-10T11:24:59.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.897+0000 7fa12be00700 1 -- 172.21.15.27:0/2203883679 learned_addr learned my addr 172.21.15.27:0/2203883679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:24:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.897+0000 7fa12aa00700 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22490c40 0x55df224a9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.897+0000 7fa12b400700 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df2248b970 0x55df224a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.898+0000 7fa12be00700 1 -- 172.21.15.27:0/2203883679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22490c40 msgr2=0x55df224a9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.898+0000 7fa12be00700 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22490c40 0x55df224a9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.898+0000 7fa12be00700 1 -- 172.21.15.27:0/2203883679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df2248b970 msgr2=0x55df224a8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:24:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.898+0000 7fa12be00700 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df2248b970 0x55df224a8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.898+0000 7fa12be00700 1 -- 172.21.15.27:0/2203883679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df2250f690 con 0x55df22491560 2024-05-10T11:24:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.898+0000 7fa12be00700 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df22491560 0x55df2250f150 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa124018640 tx=0x7fa12400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c5422b67e173407 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:24:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.898+0000 7fa115600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:24:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.899+0000 7fa115600700 1 -- 172.21.15.27:0/2203883679 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124003c10 con 0x55df22491560 2024-05-10T11:24:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.899+0000 7fa13ceceb80 1 -- 172.21.15.27:0/2203883679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa11c0c2070 con 0x55df22491560 2024-05-10T11:24:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.899+0000 7fa115600700 1 -- 172.21.15.27:0/2203883679 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9591+0+0 (secure 0 0 0) 0x7fa124023040 con 0x55df22491560 2024-05-10T11:24:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.899+0000 7fa115600700 1 -- 172.21.15.27:0/2203883679 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fa124003db0 con 0x55df22491560 2024-05-10T11:24:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.899+0000 7fa13ceceb80 10 client.26972 fetch_fsmap learned FSMap version 36 2024-05-10T11:24:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.899+0000 7fa13ceceb80 1 -- 172.21.15.27:0/2203883679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55df22510990 con 0x55df22491560 2024-05-10T11:24:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.899+0000 7fa115600700 1 -- 172.21.15.27:0/2203883679 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11c0c2070 con 0x55df22491560 2024-05-10T11:24:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.899+0000 7fa115600700 1 -- 172.21.15.27:0/2203883679 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fa124028470 con 0x55df22491560 2024-05-10T11:24:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.900+0000 7fa13ceceb80 10 client.26972 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:24:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.900+0000 7fa13ceceb80 10 client.26972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:24:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.900+0000 7fa13ceceb80 20 client.26972 populate_metadata read hostname 'smithi027' 2024-05-10T11:24:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.900+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df225110d0 0x55df225134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:24:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.900+0000 7fa13ceceb80 4 client.26972 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:24:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.900+0000 7fa13ceceb80 1 -- 172.21.15.27:0/2203883679 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55df223eb480 con 0x55df225110d0 2024-05-10T11:24:59.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.900+0000 7fa12b400700 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df225110d0 0x55df225134f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:24:59.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.900+0000 7fa12b400700 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df225110d0 0x55df225134f0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=242853a696770a04 server_cookie=415731cf01b29c75 in_seq=0 out_seq=0 2024-05-10T11:24:59.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.900+0000 7fa115600700 10 client.26972 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:24:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa115600700 1 -- 172.21.15.27:0/2203883679 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55df223eb480 con 0x55df225110d0 2024-05-10T11:24:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa115600700 10 client.26972 handle_command_reply: tid=0 2024-05-10T11:24:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa13ceceb80 1 client.26972 shutdown 2024-05-10T11:24:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa13ceceb80 1 -- 172.21.15.27:0/2203883679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df22491560 msgr2=0x55df2250f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df22491560 0x55df2250f150 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa124018640 tx=0x7fa12400b040 comp rx=0 tx=0).stop 2024-05-10T11:24:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa13ceceb80 1 -- 172.21.15.27:0/2203883679 shutdown_connections 2024-05-10T11:24:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55df225110d0 0x55df225134f0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55df22491560 0x55df2250f150 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55df2248b970 0x55df224a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa115600700 0 client.26972 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:24:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa13ceceb80 1 --2- 172.21.15.27:0/2203883679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df22490c40 0x55df224a9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa13ceceb80 1 -- 172.21.15.27:0/2203883679 >> 172.21.15.27:0/2203883679 conn(0x55df22403750 msgr2=0x55df224a0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.903+0000 7fa13ceceb80 1 -- 172.21.15.27:0/2203883679 shutdown_connections 2024-05-10T11:24:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.904+0000 7fa13ceceb80 1 -- 172.21.15.27:0/2203883679 wait complete. 2024-05-10T11:24:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.904+0000 7fa13ceceb80 20 client.26972 trim_cache size 0 max 16384 2024-05-10T11:24:59.907 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:24:59.907 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:24:59.907 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "1f00b7c2-996e-42b0-a421-a3fc44e2c683", 2024-05-10T11:24:59.907 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:24:59.907 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:24:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.904+0000 7fa132600700 1 -- 172.21.15.27:0/1121570484 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa1100b2a70 msgr2=0x7fa1100b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.904+0000 7fa132600700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa1100b2a70 0x7fa1100b4f30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa12400fb80 tx=0x7fa124004b40 comp rx=0 tx=0).stop 2024-05-10T11:24:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.904+0000 7fa132600700 1 -- 172.21.15.27:0/1121570484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa12c1056c0 msgr2=0x7fa12c19bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:24:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.904+0000 7fa132600700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa12c1056c0 0x7fa12c19bdb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00eb30 tx=0x7fa11c00ee40 comp rx=0 tx=0).stop 2024-05-10T11:24:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.905+0000 7fa132600700 1 -- 172.21.15.27:0/1121570484 shutdown_connections 2024-05-10T11:24:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.905+0000 7fa132600700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa1100b2a70 0x7fa1100b4f30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.905+0000 7fa132600700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa12c10b6c0 0x7fa12c1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.905+0000 7fa132600700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa12c1056c0 0x7fa12c19bdb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.905+0000 7fa132600700 1 --2- 172.21.15.27:0/1121570484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa12c106080 0x7fa12c19c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:24:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.905+0000 7fa132600700 1 -- 172.21.15.27:0/1121570484 >> 172.21.15.27:0/1121570484 conn(0x7fa12c1011b0 msgr2=0x7fa12c101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:24:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.905+0000 7fa132600700 1 -- 172.21.15.27:0/1121570484 shutdown_connections 2024-05-10T11:24:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:24:59.905+0000 7fa132600700 1 -- 172.21.15.27:0/1121570484 wait complete. 2024-05-10T11:25:00.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:00 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:25:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:00 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:00 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:25:00.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:00 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:00 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:25:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:00 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:00 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:25:00.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:00 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:00.732 INFO:tasks.workunit.client.0.smithi027.stdout: 1 709859 6.32 MB/sec execute 465 sec latency 34.188 ms 2024-05-10T11:25:00.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:00 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:25:00.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:00 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:00.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:00 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:25:00.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:00 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:01.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:25:01.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:01 smithi027 ceph-mon[30054]: pgmap v366: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 164 op/s 2024-05-10T11:25:01.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:01 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:25:01.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:01 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:01.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:01 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:25:01.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:01 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:25:01.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:01 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:01.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:01 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:01 smithi076 ceph-mon[34152]: pgmap v366: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 164 op/s 2024-05-10T11:25:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:01 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:25:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:01 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:01 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:25:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:01 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:25:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:01 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:01 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:01.732 INFO:tasks.workunit.client.0.smithi027.stdout: 1 709929 6.31 MB/sec execute 466 sec latency 169.361 ms 2024-05-10T11:25:01.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:01 smithi189 ceph-mon[34842]: pgmap v366: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 164 op/s 2024-05-10T11:25:01.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:01 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:25:01.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:01 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:01.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:01 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:25:01.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:01 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:25:01.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:01 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:01.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:01 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:25:02.026 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:25:02.732 INFO:tasks.workunit.client.0.smithi027.stdout: 1 710358 6.31 MB/sec execute 467 sec latency 102.984 ms 2024-05-10T11:25:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:03 smithi027 ceph-mon[30054]: pgmap v367: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-05-10T11:25:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:03 smithi027 ceph-mon[30054]: osdmap e102: 12 total, 12 up, 12 in 2024-05-10T11:25:03.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:03 smithi076 ceph-mon[34152]: pgmap v367: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-05-10T11:25:03.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:03 smithi076 ceph-mon[34152]: osdmap e102: 12 total, 12 up, 12 in 2024-05-10T11:25:03.732 INFO:tasks.workunit.client.0.smithi027.stdout: 1 710823 6.30 MB/sec execute 468 sec latency 37.762 ms 2024-05-10T11:25:03.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:03 smithi189 ceph-mon[34842]: pgmap v367: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-05-10T11:25:03.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:03 smithi189 ceph-mon[34842]: osdmap e102: 12 total, 12 up, 12 in 2024-05-10T11:25:04.733 INFO:tasks.workunit.client.0.smithi027.stdout: 1 711916 6.29 MB/sec execute 469 sec latency 35.495 ms 2024-05-10T11:25:04.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.833+0000 7feee0e00700 1 -- 172.21.15.27:0/1201788863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc10fc10 msgr2=0x7feedc112080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:04.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.833+0000 7feee0e00700 1 --2- 172.21.15.27:0/1201788863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc10fc10 0x7feedc112080 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7feed4009990 tx=0x7feed4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.833+0000 7feee0e00700 1 -- 172.21.15.27:0/1201788863 shutdown_connections 2024-05-10T11:25:04.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.833+0000 7feee0e00700 1 --2- 172.21.15.27:0/1201788863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc10fc10 0x7feedc112080 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.833+0000 7feee0e00700 1 --2- 172.21.15.27:0/1201788863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feedc0758d0 0x7feedc075cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.833+0000 7feee0e00700 1 --2- 172.21.15.27:0/1201788863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feedc0761f0 0x7feedc10f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.833+0000 7feee0e00700 1 -- 172.21.15.27:0/1201788863 >> 172.21.15.27:0/1201788863 conn(0x7feedc0fef20 msgr2=0x7feedc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:04.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.833+0000 7feee0e00700 1 -- 172.21.15.27:0/1201788863 shutdown_connections 2024-05-10T11:25:04.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.833+0000 7feee0e00700 1 -- 172.21.15.27:0/1201788863 wait complete. 2024-05-10T11:25:04.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.834+0000 7feee0e00700 1 Processor -- start 2024-05-10T11:25:04.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.835+0000 7feee0e00700 1 -- start start 2024-05-10T11:25:04.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.835+0000 7feee0e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feedc0758d0 0x7feedc1a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.835+0000 7feee0e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feedc0761f0 0x7feedc1a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.835+0000 7feee0e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc10fc10 0x7feedc19ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.835+0000 7feee0e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1157a0 con 0x7feedc0761f0 2024-05-10T11:25:04.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.835+0000 7feee0e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7feedc115620 con 0x7feedc0758d0 2024-05-10T11:25:04.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.835+0000 7feee0e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feedc115920 con 0x7feedc10fc10 2024-05-10T11:25:04.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.835+0000 7feedb400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feedc0758d0 0x7feedc1a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.835+0000 7feedb400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feedc0758d0 0x7feedc1a18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44908/0 (socket says 172.21.15.27:44908) 2024-05-10T11:25:04.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.835+0000 7feedb400700 1 -- 172.21.15.27:0/470510740 learned_addr learned my addr 172.21.15.27:0/470510740 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:04.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.836+0000 7feed2a00700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feedc0761f0 0x7feedc1a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.836+0000 7feedbe00700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc10fc10 0x7feedc19ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.836+0000 7feedb400700 1 -- 172.21.15.27:0/470510740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc10fc10 msgr2=0x7feedc19ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:04.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.836+0000 7feedb400700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc10fc10 0x7feedc19ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.836+0000 7feedb400700 1 -- 172.21.15.27:0/470510740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feedc0761f0 msgr2=0x7feedc1a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:04.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.836+0000 7feedb400700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feedc0761f0 0x7feedc1a1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.836+0000 7feedb400700 1 -- 172.21.15.27:0/470510740 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feedc19c1c0 con 0x7feedc0758d0 2024-05-10T11:25:04.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.836+0000 7feedb400700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feedc0758d0 0x7feedc1a18a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7feecc00c370 tx=0x7feecc00c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:04.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.837+0000 7feed8c00700 1 -- 172.21.15.27:0/470510740 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc0092e0 con 0x7feedc0758d0 2024-05-10T11:25:04.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.837+0000 7feee0e00700 1 -- 172.21.15.27:0/470510740 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feedc19c450 con 0x7feedc0758d0 2024-05-10T11:25:04.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.837+0000 7feed8c00700 1 -- 172.21.15.27:0/470510740 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feecc00cbe0 con 0x7feedc0758d0 2024-05-10T11:25:04.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.837+0000 7feed8c00700 1 -- 172.21.15.27:0/470510740 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc004730 con 0x7feedc0758d0 2024-05-10T11:25:04.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.838+0000 7feee0e00700 1 -- 172.21.15.27:0/470510740 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feedc1a8770 con 0x7feedc0758d0 2024-05-10T11:25:04.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.841+0000 7feed8c00700 1 -- 172.21.15.27:0/470510740 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7feecc01a020 con 0x7feedc0758d0 2024-05-10T11:25:04.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.843+0000 7feed8c00700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7feec00b2a70 0x7feec00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.843+0000 7feeecd7cb80 1 Processor -- start 2024-05-10T11:25:04.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.843+0000 7feed8c00700 1 -- 172.21.15.27:0/470510740 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9679+0+0 (secure 0 0 0) 0x7feecc0e1f70 con 0x7feedc0758d0 2024-05-10T11:25:04.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.843+0000 7feeecd7cb80 1 -- start start 2024-05-10T11:25:04.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.843+0000 7feed2a00700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7feec00b2a70 0x7feec00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.844+0000 7feeecd7cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cac9e0 0x55c085cacdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.844+0000 7feeecd7cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085ca6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.844+0000 7feeecd7cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085ca7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.844+0000 7feeecd7cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c085bf9a20 con 0x55c085cac9e0 2024-05-10T11:25:04.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.844+0000 7feeecd7cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c085b97760 con 0x55c085cad300 2024-05-10T11:25:04.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.844+0000 7feeecd7cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c085bf59e0 con 0x55c085ca7600 2024-05-10T11:25:04.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.844+0000 7feed2a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085ca6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.844+0000 7feedb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cac9e0 0x55c085cacdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.844+0000 7feedbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085ca7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.844+0000 7feed2a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085ca6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44912/0 (socket says 172.21.15.27:44912) 2024-05-10T11:25:04.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.844+0000 7feed2a00700 1 -- 172.21.15.27:0/3937111822 learned_addr learned my addr 172.21.15.27:0/3937111822 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.844+0000 7feed2a00700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7feec00b2a70 0x7feec00b4f30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feedc1110a0 tx=0x7feec4009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.845+0000 7feed2a00700 1 -- 172.21.15.27:0/3937111822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 msgr2=0x55c085ca7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.845+0000 7feed2a00700 1 --2- 172.21.15.27:0/3937111822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085ca7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.845+0000 7feed2a00700 1 -- 172.21.15.27:0/3937111822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cac9e0 msgr2=0x55c085cacdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.845+0000 7feed2a00700 1 --2- 172.21.15.27:0/3937111822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cac9e0 0x55c085cacdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.845+0000 7feed2a00700 1 -- 172.21.15.27:0/3937111822 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c085b468d0 con 0x55c085cad300 2024-05-10T11:25:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.845+0000 7feedbe00700 1 --2- 172.21.15.27:0/3937111822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085ca7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.845+0000 7feed2a00700 1 --2- 172.21.15.27:0/3937111822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085ca6fe0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feec4011ce0 tx=0x7feec401c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4549ba39c19c8eca server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.845+0000 7feed0c00700 1 -- 172.21.15.27:0/3937111822 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec4004020 con 0x55c085cad300 2024-05-10T11:25:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.845+0000 7feed0c00700 1 -- 172.21.15.27:0/3937111822 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feec401c950 con 0x55c085cad300 2024-05-10T11:25:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.845+0000 7feed0c00700 1 -- 172.21.15.27:0/3937111822 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec4003cc0 con 0x55c085cad300 2024-05-10T11:25:04.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.846+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3937111822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 msgr2=0x55c085ca6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:04.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.846+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/3937111822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085ca6fe0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feec4011ce0 tx=0x7feec401c000 comp rx=0 tx=0).stop 2024-05-10T11:25:04.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.846+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3937111822 shutdown_connections 2024-05-10T11:25:04.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.846+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/3937111822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085ca7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.846+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/3937111822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cac9e0 0x55c085cacdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.846+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/3937111822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085ca6fe0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.846+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3937111822 >> 172.21.15.27:0/3937111822 conn(0x55c085bd5b60 msgr2=0x55c085ca19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:04.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.846+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3937111822 shutdown_connections 2024-05-10T11:25:04.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.846+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3937111822 wait complete. 2024-05-10T11:25:04.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.847+0000 7feeecd7cb80 1 -- start start 2024-05-10T11:25:04.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.847+0000 7feeecd7cb80 1 Processor -- start 2024-05-10T11:25:04.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.848+0000 7feeecd7cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085cb2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.848+0000 7feeecd7cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cac9e0 0x55c085cb02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.848+0000 7feeecd7cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085cb07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.848+0000 7feeecd7cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c085cb3db0 con 0x55c085cac9e0 2024-05-10T11:25:04.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.848+0000 7feeecd7cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c085b5fb20 con 0x55c085cad300 2024-05-10T11:25:04.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.848+0000 7feeecd7cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c085cb3f30 con 0x55c085ca7600 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.848+0000 7feeecd7cb80 10 client.0 _ll_register_callbacks cb 0x55c085ba0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.848+0000 7feedb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085cb2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.848+0000 7feedb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085cb2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:33588/0 (socket says 172.21.15.27:33588) 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.848+0000 7feedbe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085cb07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.848+0000 7feedb400700 1 -- 172.21.15.27:0/529467200 learned_addr learned my addr 172.21.15.27:0/529467200 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.848+0000 7feed2a00700 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cac9e0 0x55c085cb02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.849+0000 7feedb400700 1 -- 172.21.15.27:0/529467200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 msgr2=0x55c085cb07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.849+0000 7feedb400700 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085cb07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.849+0000 7feedb400700 1 -- 172.21.15.27:0/529467200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cac9e0 msgr2=0x55c085cb02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.849+0000 7feedb400700 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cac9e0 0x55c085cb02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.849+0000 7feedb400700 1 -- 172.21.15.27:0/529467200 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c085bbf420 con 0x55c085ca7600 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.849+0000 7feed2a00700 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cac9e0 0x55c085cb02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.849+0000 7feedb400700 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085cb2f10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feecc00c9d0 tx=0x7feecc0a92b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26159f2892b26592 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.849+0000 7feec9600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:25:04.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.849+0000 7feec9600700 1 -- 172.21.15.27:0/529467200 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc0a9470 con 0x55c085ca7600 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.849+0000 7feec9600700 1 -- 172.21.15.27:0/529467200 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9679+0+0 (secure 0 0 0) 0x7feecc0c3400 con 0x55c085ca7600 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.849+0000 7feeecd7cb80 1 -- 172.21.15.27:0/529467200 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feec401c950 con 0x55c085ca7600 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.850+0000 7feec9600700 1 -- 172.21.15.27:0/529467200 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7feecc0a8d70 con 0x55c085ca7600 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.850+0000 7feeecd7cb80 10 client.26965 fetch_fsmap learned FSMap version 36 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.850+0000 7feeecd7cb80 1 -- 172.21.15.27:0/529467200 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c085cbe4d0 con 0x55c085ca7600 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.850+0000 7feec9600700 1 -- 172.21.15.27:0/529467200 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feec401c950 con 0x55c085ca7600 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.850+0000 7feec9600700 1 -- 172.21.15.27:0/529467200 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7feecc0a7b60 con 0x55c085ca7600 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.850+0000 7feeecd7cb80 10 client.26965 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.850+0000 7feeecd7cb80 10 client.26965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.850+0000 7feeecd7cb80 20 client.26965 populate_metadata read hostname 'smithi027' 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.850+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c085cc0b10 0x55c085cc2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.850+0000 7feeecd7cb80 4 client.26965 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.850+0000 7feeecd7cb80 1 -- 172.21.15.27:0/529467200 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c085c740a0 con 0x55c085cc0b10 2024-05-10T11:25:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.851+0000 7feed2a00700 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c085cc0b10 0x55c085cc2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.851+0000 7feed2a00700 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c085cc0b10 0x55c085cc2f00 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9820f100d5045b0 server_cookie=2627ce6a66229313 in_seq=0 out_seq=0 2024-05-10T11:25:04.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.851+0000 7feec9600700 10 client.26965 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:25:04.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.853+0000 7feec9600700 1 -- 172.21.15.27:0/529467200 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c085c740a0 con 0x55c085cc0b10 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.853+0000 7feec9600700 10 client.26965 handle_command_reply: tid=0 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.854+0000 7feeecd7cb80 1 client.26965 shutdown 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.854+0000 7feeecd7cb80 1 -- 172.21.15.27:0/529467200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 msgr2=0x55c085cb2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.854+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085cb2f10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feecc00c9d0 tx=0x7feecc0a92b0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.854+0000 7feeecd7cb80 1 -- 172.21.15.27:0/529467200 shutdown_connections 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.854+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c085cc0b10 0x55c085cc2f00 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.854+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085cb2f10 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.854+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085cb07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.854+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/529467200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cac9e0 0x55c085cb02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.854+0000 7feeecd7cb80 1 -- 172.21.15.27:0/529467200 >> 172.21.15.27:0/529467200 conn(0x55c085bd5b60 msgr2=0x55c085c9c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.854+0000 7feec9600700 0 client.26965 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.854+0000 7feeecd7cb80 1 -- 172.21.15.27:0/529467200 shutdown_connections 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.854+0000 7feeecd7cb80 1 -- 172.21.15.27:0/529467200 wait complete. 2024-05-10T11:25:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.855+0000 7feeecd7cb80 20 client.26965 trim_cache size 0 max 16384 2024-05-10T11:25:04.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.878+0000 7feeecd7cb80 1 Processor -- start 2024-05-10T11:25:04.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.878+0000 7feeecd7cb80 1 -- start start 2024-05-10T11:25:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.878+0000 7feeecd7cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085ca6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feeecd7cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cac9e0 0x55c085caebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feeecd7cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cad300 0x55c085caf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feeecd7cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1b19a0 con 0x55c085cad300 2024-05-10T11:25:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feeecd7cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1b34e0 con 0x55c085cac9e0 2024-05-10T11:25:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feeecd7cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1b2070 con 0x55c085ca7600 2024-05-10T11:25:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feedb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085ca6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feed2a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cac9e0 0x55c085caebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feed2a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cac9e0 0x55c085caebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44930/0 (socket says 172.21.15.27:44930) 2024-05-10T11:25:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feedb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085ca6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:33600/0 (socket says 172.21.15.27:33600) 2024-05-10T11:25:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feed2a00700 1 -- 172.21.15.27:0/2679529297 learned_addr learned my addr 172.21.15.27:0/2679529297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feedbe00700 1 --2- 172.21.15.27:0/2679529297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cad300 0x55c085caf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feed2a00700 1 -- 172.21.15.27:0/2679529297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 msgr2=0x55c085ca6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feed2a00700 1 --2- 172.21.15.27:0/2679529297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085ca6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feed2a00700 1 -- 172.21.15.27:0/2679529297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cad300 msgr2=0x55c085caf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feed2a00700 1 --2- 172.21.15.27:0/2679529297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cad300 0x55c085caf100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feed2a00700 1 -- 172.21.15.27:0/2679529297 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c085cb3280 con 0x55c085cac9e0 2024-05-10T11:25:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.879+0000 7feedb400700 1 --2- 172.21.15.27:0/2679529297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085ca6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-05-10T11:25:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.880+0000 7feedbe00700 1 --2- 172.21.15.27:0/2679529297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cad300 0x55c085caf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:25:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.880+0000 7feed2a00700 1 --2- 172.21.15.27:0/2679529297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cac9e0 0x55c085caebc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7feec4011cb0 tx=0x7feec4005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da03a5066283777f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.880+0000 7feecbe00700 1 -- 172.21.15.27:0/2679529297 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec4028040 con 0x55c085cac9e0 2024-05-10T11:25:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.880+0000 7feecbe00700 1 -- 172.21.15.27:0/2679529297 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feec4003680 con 0x55c085cac9e0 2024-05-10T11:25:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.880+0000 7feecbe00700 1 -- 172.21.15.27:0/2679529297 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec40352b0 con 0x55c085cac9e0 2024-05-10T11:25:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.881+0000 7feeecd7cb80 1 -- 172.21.15.27:0/2679529297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cac9e0 msgr2=0x55c085caebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.881+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/2679529297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cac9e0 0x55c085caebc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7feec4011cb0 tx=0x7feec4005040 comp rx=0 tx=0).stop 2024-05-10T11:25:04.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.882+0000 7feeecd7cb80 1 -- 172.21.15.27:0/2679529297 shutdown_connections 2024-05-10T11:25:04.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.882+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/2679529297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085ca7600 0x55c085ca6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.882+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/2679529297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085cad300 0x55c085caf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.882+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/2679529297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cac9e0 0x55c085caebc0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.882+0000 7feeecd7cb80 1 -- 172.21.15.27:0/2679529297 >> 172.21.15.27:0/2679529297 conn(0x55c085bd5b60 msgr2=0x55c085cb0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:04.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.882+0000 7feeecd7cb80 1 -- 172.21.15.27:0/2679529297 shutdown_connections 2024-05-10T11:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.883+0000 7feeecd7cb80 1 -- 172.21.15.27:0/2679529297 wait complete. 2024-05-10T11:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.883+0000 7feeecd7cb80 1 -- start start 2024-05-10T11:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.883+0000 7feeecd7cb80 1 Processor -- start 2024-05-10T11:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.884+0000 7feeecd7cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085ca7600 0x55c085cb95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.884+0000 7feeecd7cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085cac9e0 0x55c085cc7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.884+0000 7feeecd7cb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085cc81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.884+0000 7feeecd7cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1b2e10 con 0x55c085ca7600 2024-05-10T11:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.884+0000 7feeecd7cb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1b2740 con 0x55c085cad300 2024-05-10T11:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.884+0000 7feeecd7cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1af960 con 0x55c085cac9e0 2024-05-10T11:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.884+0000 7feeecd7cb80 10 client.0 _ll_register_callbacks cb 0x55c085ba0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.884+0000 7feed2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085cac9e0 0x55c085cc7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.884+0000 7feedb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085ca7600 0x55c085cb95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.884+0000 7feed2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085cac9e0 0x55c085cc7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:33614/0 (socket says 172.21.15.27:33614) 2024-05-10T11:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.884+0000 7feed2a00700 1 -- 172.21.15.27:0/3749234371 learned_addr learned my addr 172.21.15.27:0/3749234371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.885+0000 7feed2a00700 1 -- 172.21.15.27:0/3749234371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 msgr2=0x55c085cc81e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:25:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.885+0000 7feed2a00700 1 --2- 172.21.15.27:0/3749234371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085cc81e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.885+0000 7feed2a00700 1 -- 172.21.15.27:0/3749234371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085ca7600 msgr2=0x55c085cb95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.885+0000 7feed2a00700 1 --2- 172.21.15.27:0/3749234371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085ca7600 0x55c085cb95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.885+0000 7feed2a00700 1 -- 172.21.15.27:0/3749234371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c085cc8860 con 0x55c085cac9e0 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.885+0000 7feedb400700 1 --2- 172.21.15.27:0/3749234371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085ca7600 0x55c085cb95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.885+0000 7feed2a00700 1 --2- 172.21.15.27:0/3749234371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085cac9e0 0x55c085cc7ca0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7feec4011920 tx=0x7feec4011950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd6634dd8c9cb83c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.885+0000 7feec9600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.885+0000 7feec9600700 1 -- 172.21.15.27:0/3749234371 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec4046060 con 0x55c085cac9e0 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.886+0000 7feec9600700 1 -- 172.21.15.27:0/3749234371 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9679+0+0 (secure 0 0 0) 0x7feec4003c70 con 0x55c085cac9e0 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.886+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3749234371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feec4003680 con 0x55c085cac9e0 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.886+0000 7feec9600700 1 -- 172.21.15.27:0/3749234371 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7feec4028040 con 0x55c085cac9e0 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.886+0000 7feeecd7cb80 10 client.26968 fetch_fsmap learned FSMap version 36 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.886+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3749234371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c085d33960 con 0x55c085cac9e0 2024-05-10T11:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.886+0000 7feec9600700 1 -- 172.21.15.27:0/3749234371 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feec4003680 con 0x55c085cac9e0 2024-05-10T11:25:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.887+0000 7feec9600700 1 -- 172.21.15.27:0/3749234371 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7feec4023070 con 0x55c085cac9e0 2024-05-10T11:25:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.887+0000 7feeecd7cb80 10 client.26968 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:25:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.887+0000 7feeecd7cb80 10 client.26968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:25:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.887+0000 7feeecd7cb80 20 client.26968 populate_metadata read hostname 'smithi027' 2024-05-10T11:25:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.887+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/3749234371 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c085d33d00 0x55c085d360f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.887+0000 7feeecd7cb80 4 client.26968 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:25:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.887+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3749234371 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c085c07400 con 0x55c085d33d00 2024-05-10T11:25:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.887+0000 7feedb400700 1 --2- 172.21.15.27:0/3749234371 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c085d33d00 0x55c085d360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.888+0000 7feedb400700 1 --2- 172.21.15.27:0/3749234371 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c085d33d00 0x55c085d360f0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7bd74538f4d244c server_cookie=e1bf77ebbbb9e645 in_seq=0 out_seq=0 2024-05-10T11:25:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.888+0000 7feec9600700 10 client.26968 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feec9600700 1 -- 172.21.15.27:0/3749234371 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c085c07400 con 0x55c085d33d00 2024-05-10T11:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feec9600700 10 client.26968 handle_command_reply: tid=0 2024-05-10T11:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feeecd7cb80 1 client.26968 shutdown 2024-05-10T11:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3749234371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085cac9e0 msgr2=0x55c085cc7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/3749234371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085cac9e0 0x55c085cc7ca0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7feec4011920 tx=0x7feec4011950 comp rx=0 tx=0).stop 2024-05-10T11:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3749234371 shutdown_connections 2024-05-10T11:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/3749234371 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55c085d33d00 0x55c085d360f0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/3749234371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c085cad300 0x55c085cc81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/3749234371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c085ca7600 0x55c085cb95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feeecd7cb80 1 --2- 172.21.15.27:0/3749234371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c085cac9e0 0x55c085cc7ca0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feec9600700 0 client.26968 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3749234371 >> 172.21.15.27:0/3749234371 conn(0x55c085bd5b60 msgr2=0x55c085caeb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3749234371 shutdown_connections 2024-05-10T11:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feeecd7cb80 1 -- 172.21.15.27:0/3749234371 wait complete. 2024-05-10T11:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.889+0000 7feeecd7cb80 20 client.26968 trim_cache size 0 max 16384 2024-05-10T11:25:04.891 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:25:04.891 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:25:04.891 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:25:04.891 INFO:teuthology.orchestra.run.smithi027.stdout: "1f00b7c2-996e-42b0-a421-a3fc44e2c683": { 2024-05-10T11:25:04.891 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:25:04.891 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "1f00b7c2-996e-42b0-a421-a3fc44e2c683", 2024-05-10T11:25:04.892 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:25:04.892 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:25:04.892 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:25:04.892 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:25:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.890+0000 7feee0e00700 1 -- 172.21.15.27:0/470510740 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7feec00b2a70 msgr2=0x7feec00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.890+0000 7feee0e00700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7feec00b2a70 0x7feec00b4f30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feedc1110a0 tx=0x7feec4009260 comp rx=0 tx=0).stop 2024-05-10T11:25:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.890+0000 7feee0e00700 1 -- 172.21.15.27:0/470510740 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feedc0758d0 msgr2=0x7feedc1a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.890+0000 7feee0e00700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feedc0758d0 0x7feedc1a18a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7feecc00c370 tx=0x7feecc00c680 comp rx=0 tx=0).stop 2024-05-10T11:25:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.890+0000 7feee0e00700 1 -- 172.21.15.27:0/470510740 shutdown_connections 2024-05-10T11:25:04.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.890+0000 7feee0e00700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7feec00b2a70 0x7feec00b4f30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.890+0000 7feee0e00700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc10fc10 0x7feedc19ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.890+0000 7feee0e00700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feedc0758d0 0x7feedc1a18a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.890+0000 7feee0e00700 1 --2- 172.21.15.27:0/470510740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feedc0761f0 0x7feedc1a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:04.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.890+0000 7feee0e00700 1 -- 172.21.15.27:0/470510740 >> 172.21.15.27:0/470510740 conn(0x7feedc0fef20 msgr2=0x7feedc0ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:04.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.891+0000 7feee0e00700 1 -- 172.21.15.27:0/470510740 shutdown_connections 2024-05-10T11:25:04.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:04.891+0000 7feee0e00700 1 -- 172.21.15.27:0/470510740 wait complete. 2024-05-10T11:25:05.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:05 smithi027 ceph-mon[30054]: pgmap v369: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 131 op/s 2024-05-10T11:25:05.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:05 smithi076 ceph-mon[34152]: pgmap v369: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 131 op/s 2024-05-10T11:25:05.732 INFO:tasks.workunit.client.0.smithi027.stdout: 1 713548 6.30 MB/sec execute 470 sec latency 91.154 ms 2024-05-10T11:25:05.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:05 smithi189 ceph-mon[34842]: pgmap v369: 129 pgs: 129 active+clean; 537 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 131 op/s 2024-05-10T11:25:06.357 INFO:tasks.cephfs.filesystem:scrub status for tag:1f00b7c2-996e-42b0-a421-a3fc44e2c683 - {'path': '/', 'tag': '1f00b7c2-996e-42b0-a421-a3fc44e2c683', 'options': 'recursive,force'} 2024-05-10T11:25:06.733 INFO:tasks.workunit.client.0.smithi027.stdout: 1 714009 6.30 MB/sec execute 471 sec latency 35.473 ms 2024-05-10T11:25:07.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:07 smithi027 ceph-mon[30054]: pgmap v370: 129 pgs: 129 active+clean; 540 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 182 op/s 2024-05-10T11:25:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:07 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:25:07.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:07 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:25:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:07 smithi076 ceph-mon[34152]: pgmap v370: 129 pgs: 129 active+clean; 540 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 182 op/s 2024-05-10T11:25:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:07 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:25:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:07 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:25:07.733 INFO:tasks.workunit.client.0.smithi027.stdout: 1 714648 6.28 MB/sec execute 472 sec latency 34.733 ms 2024-05-10T11:25:07.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:07 smithi189 ceph-mon[34842]: pgmap v370: 129 pgs: 129 active+clean; 540 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 182 op/s 2024-05-10T11:25:07.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:07 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:25:07.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:07 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:25:08.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:08 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:25:08.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:08 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:25:08.733 INFO:tasks.workunit.client.0.smithi027.stdout: 1 716100 6.28 MB/sec execute 473 sec latency 25.711 ms 2024-05-10T11:25:08.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:08 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:25:09.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:09 smithi027 ceph-mon[30054]: pgmap v371: 129 pgs: 129 active+clean; 540 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 182 op/s 2024-05-10T11:25:09.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:09 smithi027 ceph-mon[30054]: osdmap e103: 12 total, 12 up, 12 in 2024-05-10T11:25:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:09 smithi076 ceph-mon[34152]: pgmap v371: 129 pgs: 129 active+clean; 540 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 182 op/s 2024-05-10T11:25:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:09 smithi076 ceph-mon[34152]: osdmap e103: 12 total, 12 up, 12 in 2024-05-10T11:25:09.733 INFO:tasks.workunit.client.0.smithi027.stdout: 1 717522 6.30 MB/sec execute 474 sec latency 20.692 ms 2024-05-10T11:25:09.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:09 smithi189 ceph-mon[34842]: pgmap v371: 129 pgs: 129 active+clean; 540 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 182 op/s 2024-05-10T11:25:09.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:09 smithi189 ceph-mon[34842]: osdmap e103: 12 total, 12 up, 12 in 2024-05-10T11:25:10.733 INFO:tasks.workunit.client.0.smithi027.stdout: 1 718128 6.28 MB/sec execute 475 sec latency 41.260 ms 2024-05-10T11:25:11.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:11 smithi027 ceph-mon[30054]: pgmap v373: 129 pgs: 129 active+clean; 546 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 229 op/s 2024-05-10T11:25:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:11 smithi076 ceph-mon[34152]: pgmap v373: 129 pgs: 129 active+clean; 546 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 229 op/s 2024-05-10T11:25:11.733 INFO:tasks.workunit.client.0.smithi027.stdout: 1 719453 6.28 MB/sec execute 476 sec latency 60.170 ms 2024-05-10T11:25:11.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:11 smithi189 ceph-mon[34842]: pgmap v373: 129 pgs: 129 active+clean; 546 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 229 op/s 2024-05-10T11:25:12.733 INFO:tasks.workunit.client.0.smithi027.stdout: 1 721015 6.29 MB/sec execute 477 sec latency 20.451 ms 2024-05-10T11:25:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:13 smithi027 ceph-mon[30054]: pgmap v374: 129 pgs: 129 active+clean; 546 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-05-10T11:25:13.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:13 smithi076 ceph-mon[34152]: pgmap v374: 129 pgs: 129 active+clean; 546 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-05-10T11:25:13.733 INFO:tasks.workunit.client.0.smithi027.stdout: 1 721601 6.28 MB/sec execute 478 sec latency 25.635 ms 2024-05-10T11:25:13.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:13 smithi189 ceph-mon[34842]: pgmap v374: 129 pgs: 129 active+clean; 546 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-05-10T11:25:14.734 INFO:tasks.workunit.client.0.smithi027.stdout: 1 722682 6.28 MB/sec execute 479 sec latency 31.566 ms 2024-05-10T11:25:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:15 smithi076 ceph-mon[34152]: pgmap v375: 129 pgs: 129 active+clean; 546 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 183 op/s 2024-05-10T11:25:15.734 INFO:tasks.workunit.client.0.smithi027.stdout: 1 724218 6.29 MB/sec execute 480 sec latency 55.214 ms 2024-05-10T11:25:15.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:15 smithi189 ceph-mon[34842]: pgmap v375: 129 pgs: 129 active+clean; 546 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 183 op/s 2024-05-10T11:25:15.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:15 smithi027 ceph-mon[30054]: pgmap v375: 129 pgs: 129 active+clean; 546 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 183 op/s 2024-05-10T11:25:16.734 INFO:tasks.workunit.client.0.smithi027.stdout: 1 724803 6.28 MB/sec execute 481 sec latency 33.553 ms 2024-05-10T11:25:17.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:17 smithi076 ceph-mon[34152]: pgmap v376: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-05-10T11:25:17.734 INFO:tasks.workunit.client.0.smithi027.stdout: 1 725857 6.28 MB/sec execute 482 sec latency 38.880 ms 2024-05-10T11:25:17.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:17 smithi189 ceph-mon[34842]: pgmap v376: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-05-10T11:25:17.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:17 smithi027 ceph-mon[30054]: pgmap v376: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-05-10T11:25:18.734 INFO:tasks.workunit.client.0.smithi027.stdout: 1 727395 6.28 MB/sec execute 483 sec latency 22.348 ms 2024-05-10T11:25:19.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:19 smithi076 ceph-mon[34152]: pgmap v377: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-05-10T11:25:19.734 INFO:tasks.workunit.client.0.smithi027.stdout: 1 728157 6.28 MB/sec execute 484 sec latency 26.935 ms 2024-05-10T11:25:19.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:19 smithi189 ceph-mon[34842]: pgmap v377: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-05-10T11:25:19.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:19 smithi027 ceph-mon[30054]: pgmap v377: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-05-10T11:25:20.734 INFO:tasks.workunit.client.0.smithi027.stdout: 1 728698 6.27 MB/sec execute 485 sec latency 43.313 ms 2024-05-10T11:25:21.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:21 smithi076 ceph-mon[34152]: pgmap v378: 129 pgs: 129 active+clean; 558 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 211 op/s 2024-05-10T11:25:21.734 INFO:tasks.workunit.client.0.smithi027.stdout: 1 730038 6.27 MB/sec execute 486 sec latency 31.965 ms 2024-05-10T11:25:21.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:21 smithi189 ceph-mon[34842]: pgmap v378: 129 pgs: 129 active+clean; 558 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 211 op/s 2024-05-10T11:25:21.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:21 smithi027 ceph-mon[30054]: pgmap v378: 129 pgs: 129 active+clean; 558 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 211 op/s 2024-05-10T11:25:22.734 INFO:tasks.workunit.client.0.smithi027.stdout: 1 731613 6.28 MB/sec execute 487 sec latency 19.893 ms 2024-05-10T11:25:23.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:23 smithi076 ceph-mon[34152]: pgmap v379: 129 pgs: 129 active+clean; 558 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-05-10T11:25:23.735 INFO:tasks.workunit.client.0.smithi027.stdout: 1 732172 6.27 MB/sec execute 488 sec latency 34.172 ms 2024-05-10T11:25:23.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:23 smithi189 ceph-mon[34842]: pgmap v379: 129 pgs: 129 active+clean; 558 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-05-10T11:25:23.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:23 smithi027 ceph-mon[30054]: pgmap v379: 129 pgs: 129 active+clean; 558 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-05-10T11:25:24.735 INFO:tasks.workunit.client.0.smithi027.stdout: 1 733386 6.27 MB/sec execute 489 sec latency 33.425 ms 2024-05-10T11:25:25.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:25 smithi076 ceph-mon[34152]: pgmap v380: 129 pgs: 129 active+clean; 558 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-05-10T11:25:25.735 INFO:tasks.workunit.client.0.smithi027.stdout: 1 735036 6.28 MB/sec execute 490 sec latency 53.899 ms 2024-05-10T11:25:25.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:25 smithi189 ceph-mon[34842]: pgmap v380: 129 pgs: 129 active+clean; 558 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-05-10T11:25:25.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:25 smithi027 ceph-mon[30054]: pgmap v380: 129 pgs: 129 active+clean; 558 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-05-10T11:25:26.735 INFO:tasks.workunit.client.0.smithi027.stdout: 1 735509 6.27 MB/sec execute 491 sec latency 31.972 ms 2024-05-10T11:25:27.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:27 smithi076 ceph-mon[34152]: pgmap v381: 129 pgs: 129 active+clean; 565 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 194 op/s 2024-05-10T11:25:27.735 INFO:tasks.workunit.client.0.smithi027.stdout: 1 736596 6.27 MB/sec execute 492 sec latency 39.447 ms 2024-05-10T11:25:27.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:27 smithi189 ceph-mon[34842]: pgmap v381: 129 pgs: 129 active+clean; 565 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 194 op/s 2024-05-10T11:25:27.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:27 smithi027 ceph-mon[30054]: pgmap v381: 129 pgs: 129 active+clean; 565 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 194 op/s 2024-05-10T11:25:28.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:28 smithi076 ceph-mon[34152]: pgmap v382: 129 pgs: 129 active+clean; 565 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 127 op/s 2024-05-10T11:25:28.735 INFO:tasks.workunit.client.0.smithi027.stdout: 1 738204 6.27 MB/sec execute 493 sec latency 25.496 ms 2024-05-10T11:25:28.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:28 smithi189 ceph-mon[34842]: pgmap v382: 129 pgs: 129 active+clean; 565 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 127 op/s 2024-05-10T11:25:28.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:28 smithi027 ceph-mon[30054]: pgmap v382: 129 pgs: 129 active+clean; 565 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 127 op/s 2024-05-10T11:25:29.735 INFO:tasks.workunit.client.0.smithi027.stdout: 1 738904 6.27 MB/sec execute 494 sec latency 31.353 ms 2024-05-10T11:25:30.737 INFO:tasks.workunit.client.0.smithi027.stdout: 1 739690 6.27 MB/sec execute 495 sec latency 30.937 ms 2024-05-10T11:25:30.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:30 smithi027 ceph-mon[30054]: pgmap v383: 129 pgs: 129 active+clean; 570 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 184 op/s 2024-05-10T11:25:30.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:30 smithi076 ceph-mon[34152]: pgmap v383: 129 pgs: 129 active+clean; 570 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 184 op/s 2024-05-10T11:25:30.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:30 smithi189 ceph-mon[34842]: pgmap v383: 129 pgs: 129 active+clean; 570 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 184 op/s 2024-05-10T11:25:31.736 INFO:tasks.workunit.client.0.smithi027.stdout: 1 740921 6.26 MB/sec execute 496 sec latency 46.039 ms 2024-05-10T11:25:32.736 INFO:tasks.workunit.client.0.smithi027.stdout: 1 742261 6.27 MB/sec execute 497 sec latency 25.824 ms 2024-05-10T11:25:32.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:32 smithi027 ceph-mon[30054]: pgmap v384: 129 pgs: 129 active+clean; 570 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-05-10T11:25:32.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:32 smithi076 ceph-mon[34152]: pgmap v384: 129 pgs: 129 active+clean; 570 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-05-10T11:25:32.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:32 smithi189 ceph-mon[34842]: pgmap v384: 129 pgs: 129 active+clean; 570 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-05-10T11:25:33.736 INFO:tasks.workunit.client.0.smithi027.stdout: 1 742731 6.26 MB/sec execute 498 sec latency 33.724 ms 2024-05-10T11:25:34.736 INFO:tasks.workunit.client.0.smithi027.stdout: 1 743796 6.26 MB/sec execute 499 sec latency 40.897 ms 2024-05-10T11:25:34.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:34 smithi027 ceph-mon[30054]: pgmap v385: 129 pgs: 129 active+clean; 570 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-05-10T11:25:34.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:34 smithi076 ceph-mon[34152]: pgmap v385: 129 pgs: 129 active+clean; 570 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-05-10T11:25:34.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:34 smithi189 ceph-mon[34842]: pgmap v385: 129 pgs: 129 active+clean; 570 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-05-10T11:25:35.736 INFO:tasks.workunit.client.0.smithi027.stdout: 1 745455 6.27 MB/sec execute 500 sec latency 24.859 ms 2024-05-10T11:25:36.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:25:36.737 INFO:tasks.workunit.client.0.smithi027.stdout: 1 745893 6.26 MB/sec execute 501 sec latency 64.969 ms 2024-05-10T11:25:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:25:36.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:36 smithi027 ceph-mon[30054]: pgmap v386: 129 pgs: 129 active+clean; 576 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 187 op/s 2024-05-10T11:25:36.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:36 smithi076 ceph-mon[34152]: pgmap v386: 129 pgs: 129 active+clean; 576 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 187 op/s 2024-05-10T11:25:36.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:36 smithi189 ceph-mon[34842]: pgmap v386: 129 pgs: 129 active+clean; 576 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 187 op/s 2024-05-10T11:25:37.736 INFO:tasks.workunit.client.0.smithi027.stdout: 1 746477 6.25 MB/sec execute 502 sec latency 33.421 ms 2024-05-10T11:25:38.737 INFO:tasks.workunit.client.0.smithi027.stdout: 1 747817 6.25 MB/sec execute 503 sec latency 30.024 ms 2024-05-10T11:25:38.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:38 smithi027 ceph-mon[30054]: pgmap v387: 129 pgs: 129 active+clean; 576 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 120 op/s 2024-05-10T11:25:38.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:38 smithi076 ceph-mon[34152]: pgmap v387: 129 pgs: 129 active+clean; 576 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 120 op/s 2024-05-10T11:25:38.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:38 smithi189 ceph-mon[34842]: pgmap v387: 129 pgs: 129 active+clean; 576 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 120 op/s 2024-05-10T11:25:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.290+0000 7f5885a00700 1 -- 172.21.15.27:0/3231273072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58801056c0 msgr2=0x7f5880105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.290+0000 7f5885a00700 1 --2- 172.21.15.27:0/3231273072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58801056c0 0x7f5880105b40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f586c009990 tx=0x7f586c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.290+0000 7f5885a00700 1 -- 172.21.15.27:0/3231273072 shutdown_connections 2024-05-10T11:25:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.290+0000 7f5885a00700 1 --2- 172.21.15.27:0/3231273072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5880106080 0x7f5880112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.290+0000 7f5885a00700 1 --2- 172.21.15.27:0/3231273072 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f588010b6c0 0x7f588010baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.290+0000 7f5885a00700 1 --2- 172.21.15.27:0/3231273072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58801056c0 0x7f5880105b40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.290+0000 7f5885a00700 1 -- 172.21.15.27:0/3231273072 >> 172.21.15.27:0/3231273072 conn(0x7f58801011b0 msgr2=0x7f58801035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.290+0000 7f5885a00700 1 -- 172.21.15.27:0/3231273072 shutdown_connections 2024-05-10T11:25:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.290+0000 7f5885a00700 1 -- 172.21.15.27:0/3231273072 wait complete. 2024-05-10T11:25:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.292+0000 7f5885a00700 1 Processor -- start 2024-05-10T11:25:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.292+0000 7f5885a00700 1 -- start start 2024-05-10T11:25:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.292+0000 7f5885a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58801056c0 0x7f588019be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.292+0000 7f5885a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5880106080 0x7f588019c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.292+0000 7f5885a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f588010b6c0 0x7f58801a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.292+0000 7f5885a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5880115950 con 0x7f588010b6c0 2024-05-10T11:25:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.292+0000 7f5885a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f58801157d0 con 0x7f58801056c0 2024-05-10T11:25:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.292+0000 7f5885a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5880115ad0 con 0x7f5880106080 2024-05-10T11:25:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.293+0000 7f587e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5880106080 0x7f588019c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.293+0000 7f587ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58801056c0 0x7f588019be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.293+0000 7f587e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5880106080 0x7f588019c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:34312/0 (socket says 172.21.15.27:34312) 2024-05-10T11:25:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.293+0000 7f587e000700 1 -- 172.21.15.27:0/241895955 learned_addr learned my addr 172.21.15.27:0/241895955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.293+0000 7f587f400700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f588010b6c0 0x7f58801a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.293+0000 7f587e000700 1 -- 172.21.15.27:0/241895955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58801056c0 msgr2=0x7f588019be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.293+0000 7f587e000700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58801056c0 0x7f588019be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.293+0000 7f587e000700 1 -- 172.21.15.27:0/241895955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f588010b6c0 msgr2=0x7f58801a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.293+0000 7f587e000700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f588010b6c0 0x7f58801a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.293+0000 7f587e000700 1 -- 172.21.15.27:0/241895955 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58801a0dc0 con 0x7f5880106080 2024-05-10T11:25:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.293+0000 7f587ea00700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58801056c0 0x7f588019be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:39.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.294+0000 7f587e000700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5880106080 0x7f588019c340 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f586c005910 tx=0x7f586c00b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:39.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.294+0000 7f5877400700 1 -- 172.21.15.27:0/241895955 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c027070 con 0x7f5880106080 2024-05-10T11:25:39.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.294+0000 7f5877400700 1 -- 172.21.15.27:0/241895955 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f586c022470 con 0x7f5880106080 2024-05-10T11:25:39.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.294+0000 7f5877400700 1 -- 172.21.15.27:0/241895955 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c017650 con 0x7f5880106080 2024-05-10T11:25:39.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.294+0000 7f5885a00700 1 -- 172.21.15.27:0/241895955 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58801a1050 con 0x7f5880106080 2024-05-10T11:25:39.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.294+0000 7f5885a00700 1 -- 172.21.15.27:0/241895955 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58801a8930 con 0x7f5880106080 2024-05-10T11:25:39.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.299+0000 7f5877400700 1 -- 172.21.15.27:0/241895955 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f586c02a070 con 0x7f5880106080 2024-05-10T11:25:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.302+0000 7f5890299b80 1 Processor -- start 2024-05-10T11:25:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.302+0000 7f5877400700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f58640b29f0 0x7f58640b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.302+0000 7f5890299b80 1 -- start start 2024-05-10T11:25:39.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.303+0000 7f587f400700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f58640b29f0 0x7f58640b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.303+0000 7f5890299b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e731bc0 0x55d78e731fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.303+0000 7f5890299b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e7324e0 0x55d78e72c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.303+0000 7f5890299b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e72c8f0 0x55d78e72cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.303+0000 7f5890299b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d78e67ea20 con 0x55d78e7324e0 2024-05-10T11:25:39.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.303+0000 7f5890299b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d78e61c760 con 0x55d78e731bc0 2024-05-10T11:25:39.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.303+0000 7f5890299b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d78e67a9e0 con 0x55d78e72c8f0 2024-05-10T11:25:39.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.303+0000 7f5877400700 1 -- 172.21.15.27:0/241895955 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7f586c0e3750 con 0x7f5880106080 2024-05-10T11:25:39.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.303+0000 7f587ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e731bc0 0x55d78e731fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.303+0000 7f587ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e731bc0 0x55d78e731fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44618/0 (socket says 172.21.15.27:44618) 2024-05-10T11:25:39.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.303+0000 7f587ea00700 1 -- 172.21.15.27:0/805205679 learned_addr learned my addr 172.21.15.27:0/805205679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:39.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.304+0000 7f587e000700 1 --2- 172.21.15.27:0/805205679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e7324e0 0x55d78e72c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.304+0000 7f587f400700 1 --2- 172.21.15.27:0/805205679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e72c8f0 0x55d78e72cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.304+0000 7f587ea00700 1 -- 172.21.15.27:0/805205679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e72c8f0 msgr2=0x55d78e72cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:39.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.304+0000 7f587ea00700 1 --2- 172.21.15.27:0/805205679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e72c8f0 0x55d78e72cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.304+0000 7f587ea00700 1 -- 172.21.15.27:0/805205679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e7324e0 msgr2=0x55d78e72c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:39.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.304+0000 7f587ea00700 1 --2- 172.21.15.27:0/805205679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e7324e0 0x55d78e72c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.304+0000 7f587ea00700 1 -- 172.21.15.27:0/805205679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d78e5cb8d0 con 0x55d78e731bc0 2024-05-10T11:25:39.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.304+0000 7f587f400700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f58640b29f0 0x7f58640b4eb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f587800f590 tx=0x7f587800b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:39.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.304+0000 7f587ea00700 1 --2- 172.21.15.27:0/805205679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e731bc0 0x55d78e731fa0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f58700115f0 tx=0x7f5870011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9b85939e4f5340c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:39.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.305+0000 7f586be00700 1 -- 172.21.15.27:0/805205679 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587001e070 con 0x55d78e731bc0 2024-05-10T11:25:39.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.305+0000 7f586be00700 1 -- 172.21.15.27:0/805205679 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58700198c0 con 0x55d78e731bc0 2024-05-10T11:25:39.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.305+0000 7f586be00700 1 -- 172.21.15.27:0/805205679 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870018920 con 0x55d78e731bc0 2024-05-10T11:25:39.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.306+0000 7f5890299b80 1 -- 172.21.15.27:0/805205679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e731bc0 msgr2=0x55d78e731fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:39.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.306+0000 7f5890299b80 1 --2- 172.21.15.27:0/805205679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e731bc0 0x55d78e731fa0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f58700115f0 tx=0x7f5870011900 comp rx=0 tx=0).stop 2024-05-10T11:25:39.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.306+0000 7f5890299b80 1 -- 172.21.15.27:0/805205679 shutdown_connections 2024-05-10T11:25:39.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.306+0000 7f5890299b80 1 --2- 172.21.15.27:0/805205679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e72c8f0 0x55d78e72cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.306+0000 7f5890299b80 1 --2- 172.21.15.27:0/805205679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e731bc0 0x55d78e731fa0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.306+0000 7f5890299b80 1 --2- 172.21.15.27:0/805205679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e7324e0 0x55d78e72c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.306+0000 7f5890299b80 1 -- 172.21.15.27:0/805205679 >> 172.21.15.27:0/805205679 conn(0x55d78e6a46d0 msgr2=0x55d78e726b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:39.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.306+0000 7f5890299b80 1 -- 172.21.15.27:0/805205679 shutdown_connections 2024-05-10T11:25:39.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.306+0000 7f5890299b80 1 -- 172.21.15.27:0/805205679 wait complete. 2024-05-10T11:25:39.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.307+0000 7f5890299b80 1 -- start start 2024-05-10T11:25:39.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.307+0000 7f5890299b80 1 Processor -- start 2024-05-10T11:25:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.307+0000 7f5890299b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e72c8f0 0x55d78e7381d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.307+0000 7f5890299b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 0x55d78e735570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.307+0000 7f5890299b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e7324e0 0x55d78e735ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.307+0000 7f5890299b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d78e738f20 con 0x55d78e7324e0 2024-05-10T11:25:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f5890299b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d78e5e4b20 con 0x55d78e72c8f0 2024-05-10T11:25:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f5890299b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d78e7390a0 con 0x55d78e731bc0 2024-05-10T11:25:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f5890299b80 10 client.0 _ll_register_callbacks cb 0x55d78e625c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:25:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f587e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 0x55d78e735570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f587e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 0x55d78e735570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:34330/0 (socket says 172.21.15.27:34330) 2024-05-10T11:25:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f587e000700 1 -- 172.21.15.27:0/2678447980 learned_addr learned my addr 172.21.15.27:0/2678447980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f587ea00700 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e72c8f0 0x55d78e7381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f587f400700 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e7324e0 0x55d78e735ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f587e000700 1 -- 172.21.15.27:0/2678447980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e72c8f0 msgr2=0x55d78e7381d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f587e000700 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e72c8f0 0x55d78e7381d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f587e000700 1 -- 172.21.15.27:0/2678447980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e7324e0 msgr2=0x55d78e735ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f587e000700 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e7324e0 0x55d78e735ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f587e000700 1 -- 172.21.15.27:0/2678447980 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d78e644420 con 0x55d78e731bc0 2024-05-10T11:25:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.308+0000 7f587f400700 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e7324e0 0x55d78e735ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:25:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.309+0000 7f587e000700 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 0x55d78e735570 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f586c0e0a70 tx=0x7f586c022c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88004911f57bac40 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.309+0000 7f5868c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:25:39.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.309+0000 7f5868c00700 1 -- 172.21.15.27:0/2678447980 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c0aad30 con 0x55d78e731bc0 2024-05-10T11:25:39.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.309+0000 7f5868c00700 1 -- 172.21.15.27:0/2678447980 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7f586c0e0c90 con 0x55d78e731bc0 2024-05-10T11:25:39.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.309+0000 7f5890299b80 1 -- 172.21.15.27:0/2678447980 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58700198c0 con 0x55d78e731bc0 2024-05-10T11:25:39.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.309+0000 7f5868c00700 1 -- 172.21.15.27:0/2678447980 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f586c0e0e90 con 0x55d78e731bc0 2024-05-10T11:25:39.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.309+0000 7f5890299b80 10 client.26980 fetch_fsmap learned FSMap version 36 2024-05-10T11:25:39.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.309+0000 7f5890299b80 1 -- 172.21.15.27:0/2678447980 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d78e738f80 con 0x55d78e731bc0 2024-05-10T11:25:39.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.310+0000 7f5868c00700 1 -- 172.21.15.27:0/2678447980 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58700198c0 con 0x55d78e731bc0 2024-05-10T11:25:39.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.310+0000 7f5868c00700 1 -- 172.21.15.27:0/2678447980 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f586c0b2f70 con 0x55d78e731bc0 2024-05-10T11:25:39.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.310+0000 7f5890299b80 10 client.26980 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:25:39.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.310+0000 7f5890299b80 10 client.26980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:25:39.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.310+0000 7f5890299b80 20 client.26980 populate_metadata read hostname 'smithi027' 2024-05-10T11:25:39.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.310+0000 7f5890299b80 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d78e745dc0 0x55d78e7481b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.310+0000 7f5890299b80 4 client.26980 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:25:39.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.310+0000 7f5890299b80 1 -- 172.21.15.27:0/2678447980 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d78e6f9140 con 0x55d78e745dc0 2024-05-10T11:25:39.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.310+0000 7f587ea00700 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d78e745dc0 0x55d78e7481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.311+0000 7f587ea00700 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d78e745dc0 0x55d78e7481b0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98a1eeac7bf3ab78 server_cookie=47bd21d204fb8ce3 in_seq=0 out_seq=0 2024-05-10T11:25:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.311+0000 7f5868c00700 10 client.26980 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:25:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.312+0000 7f5868c00700 1 -- 172.21.15.27:0/2678447980 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d78e6f9140 con 0x55d78e745dc0 2024-05-10T11:25:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.312+0000 7f5868c00700 10 client.26980 handle_command_reply: tid=0 2024-05-10T11:25:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.312+0000 7f5890299b80 1 client.26980 shutdown 2024-05-10T11:25:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.313+0000 7f5890299b80 1 -- 172.21.15.27:0/2678447980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 msgr2=0x55d78e735570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.313+0000 7f5890299b80 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 0x55d78e735570 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f586c0e0a70 tx=0x7f586c022c00 comp rx=0 tx=0).stop 2024-05-10T11:25:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.313+0000 7f5890299b80 1 -- 172.21.15.27:0/2678447980 shutdown_connections 2024-05-10T11:25:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.313+0000 7f5890299b80 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d78e745dc0 0x55d78e7481b0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.313+0000 7f5890299b80 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e7324e0 0x55d78e735ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.313+0000 7f5890299b80 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e72c8f0 0x55d78e7381d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.313+0000 7f5890299b80 1 --2- 172.21.15.27:0/2678447980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 0x55d78e735570 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.313+0000 7f5890299b80 1 -- 172.21.15.27:0/2678447980 >> 172.21.15.27:0/2678447980 conn(0x55d78e6a46d0 msgr2=0x55d78e726190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:39.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.313+0000 7f5868c00700 0 client.26980 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:25:39.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.313+0000 7f5890299b80 1 -- 172.21.15.27:0/2678447980 shutdown_connections 2024-05-10T11:25:39.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.313+0000 7f5890299b80 1 -- 172.21.15.27:0/2678447980 wait complete. 2024-05-10T11:25:39.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.314+0000 7f5890299b80 20 client.26980 trim_cache size 0 max 16384 2024-05-10T11:25:39.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.340+0000 7f5890299b80 1 Processor -- start 2024-05-10T11:25:39.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.340+0000 7f5890299b80 1 -- start start 2024-05-10T11:25:39.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.340+0000 7f5890299b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e72c8f0 0x55d78e741fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.340+0000 7f5890299b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e731bc0 0x55d78e726910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.340+0000 7f5890299b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e7324e0 0x55d78e730e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.340+0000 7f5890299b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f58801b18f0 con 0x55d78e731bc0 2024-05-10T11:25:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.340+0000 7f5890299b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f58801b3430 con 0x55d78e72c8f0 2024-05-10T11:25:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.340+0000 7f5890299b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f58801b1fc0 con 0x55d78e7324e0 2024-05-10T11:25:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.341+0000 7f587ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e72c8f0 0x55d78e741fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.341+0000 7f587f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e7324e0 0x55d78e730e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.341+0000 7f587ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e72c8f0 0x55d78e741fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44638/0 (socket says 172.21.15.27:44638) 2024-05-10T11:25:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.341+0000 7f587f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e7324e0 0x55d78e730e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:34332/0 (socket says 172.21.15.27:34332) 2024-05-10T11:25:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.341+0000 7f587ea00700 1 -- 172.21.15.27:0/1382378062 learned_addr learned my addr 172.21.15.27:0/1382378062 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.341+0000 7f587e000700 1 --2- 172.21.15.27:0/1382378062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e731bc0 0x55d78e726910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.341+0000 7f587ea00700 1 -- 172.21.15.27:0/1382378062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e7324e0 msgr2=0x55d78e730e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.341+0000 7f587ea00700 1 --2- 172.21.15.27:0/1382378062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e7324e0 0x55d78e730e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.341+0000 7f587ea00700 1 -- 172.21.15.27:0/1382378062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e731bc0 msgr2=0x55d78e726910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.341+0000 7f587ea00700 1 --2- 172.21.15.27:0/1382378062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e731bc0 0x55d78e726910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.341+0000 7f587ea00700 1 -- 172.21.15.27:0/1382378062 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d78e738540 con 0x55d78e72c8f0 2024-05-10T11:25:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.341+0000 7f587ea00700 1 --2- 172.21.15.27:0/1382378062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e72c8f0 0x55d78e741fb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5870005950 tx=0x7f58700184c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed1cec96d7716e01 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.342+0000 7f586b400700 1 -- 172.21.15.27:0/1382378062 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870032070 con 0x55d78e72c8f0 2024-05-10T11:25:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.342+0000 7f586b400700 1 -- 172.21.15.27:0/1382378062 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f587002a070 con 0x55d78e72c8f0 2024-05-10T11:25:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.342+0000 7f586b400700 1 -- 172.21.15.27:0/1382378062 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587001e040 con 0x55d78e72c8f0 2024-05-10T11:25:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.343+0000 7f5890299b80 1 -- 172.21.15.27:0/1382378062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e72c8f0 msgr2=0x55d78e741fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.343+0000 7f5890299b80 1 --2- 172.21.15.27:0/1382378062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e72c8f0 0x55d78e741fb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5870005950 tx=0x7f58700184c0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.343+0000 7f5890299b80 1 -- 172.21.15.27:0/1382378062 shutdown_connections 2024-05-10T11:25:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.343+0000 7f5890299b80 1 --2- 172.21.15.27:0/1382378062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e7324e0 0x55d78e730e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.343+0000 7f5890299b80 1 --2- 172.21.15.27:0/1382378062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e72c8f0 0x55d78e741fb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.343+0000 7f5890299b80 1 --2- 172.21.15.27:0/1382378062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e731bc0 0x55d78e726910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.343+0000 7f5890299b80 1 -- 172.21.15.27:0/1382378062 >> 172.21.15.27:0/1382378062 conn(0x55d78e6a46d0 msgr2=0x55d78e730700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.343+0000 7f5890299b80 1 -- 172.21.15.27:0/1382378062 shutdown_connections 2024-05-10T11:25:39.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.343+0000 7f5890299b80 1 -- 172.21.15.27:0/1382378062 wait complete. 2024-05-10T11:25:39.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.344+0000 7f5890299b80 1 -- start start 2024-05-10T11:25:39.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.344+0000 7f5890299b80 1 Processor -- start 2024-05-10T11:25:39.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.344+0000 7f5890299b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e72c8f0 0x55d78e73e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.344+0000 7f5890299b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 0x55d78e73e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f5890299b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e7324e0 0x55d78e73b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f5890299b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f58801b2d60 con 0x55d78e72c8f0 2024-05-10T11:25:39.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f5890299b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f58801b2690 con 0x55d78e7324e0 2024-05-10T11:25:39.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f5890299b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f58801af8b0 con 0x55d78e731bc0 2024-05-10T11:25:39.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f5890299b80 10 client.0 _ll_register_callbacks cb 0x55d78e625c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:25:39.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f587ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e72c8f0 0x55d78e73e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f587f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e7324e0 0x55d78e73b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f587ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e72c8f0 0x55d78e73e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53610/0 (socket says 172.21.15.27:53610) 2024-05-10T11:25:39.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f587ea00700 1 -- 172.21.15.27:0/1430309275 learned_addr learned my addr 172.21.15.27:0/1430309275 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:39.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f587e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 0x55d78e73e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f587e000700 1 -- 172.21.15.27:0/1430309275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e7324e0 msgr2=0x55d78e73b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f587e000700 1 --2- 172.21.15.27:0/1430309275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e7324e0 0x55d78e73b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f587e000700 1 -- 172.21.15.27:0/1430309275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e72c8f0 msgr2=0x55d78e73e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f587e000700 1 --2- 172.21.15.27:0/1430309275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e72c8f0 0x55d78e73e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.345+0000 7f587e000700 1 -- 172.21.15.27:0/1430309275 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d78e73b840 con 0x55d78e731bc0 2024-05-10T11:25:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.346+0000 7f587f400700 1 --2- 172.21.15.27:0/1430309275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e7324e0 0x55d78e73b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.346+0000 7f587e000700 1 --2- 172.21.15.27:0/1430309275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 0x55d78e73e870 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f586c000c00 tx=0x7f586c0aac10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a869510b92ed09c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.346+0000 7f5868c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:25:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.346+0000 7f5868c00700 1 -- 172.21.15.27:0/1430309275 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c0e0ea0 con 0x55d78e731bc0 2024-05-10T11:25:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.346+0000 7f5890299b80 1 -- 172.21.15.27:0/1430309275 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f587002a070 con 0x55d78e731bc0 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.346+0000 7f5868c00700 1 -- 172.21.15.27:0/1430309275 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7f586c067b90 con 0x55d78e731bc0 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.346+0000 7f5868c00700 1 -- 172.21.15.27:0/1430309275 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f586c0aa630 con 0x55d78e731bc0 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.346+0000 7f5890299b80 10 client.26986 fetch_fsmap learned FSMap version 36 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.346+0000 7f5890299b80 1 -- 172.21.15.27:0/1430309275 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d78e7b0e40 con 0x55d78e731bc0 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.347+0000 7f5868c00700 1 -- 172.21.15.27:0/1430309275 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f587002a070 con 0x55d78e731bc0 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.347+0000 7f5868c00700 1 -- 172.21.15.27:0/1430309275 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f586c0e3250 con 0x55d78e731bc0 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.347+0000 7f5890299b80 10 client.26986 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.347+0000 7f5890299b80 10 client.26986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.347+0000 7f5890299b80 20 client.26986 populate_metadata read hostname 'smithi027' 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.347+0000 7f5890299b80 1 --2- 172.21.15.27:0/1430309275 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d78e7b12e0 0x55d78e7b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.347+0000 7f5890299b80 4 client.26986 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.347+0000 7f5890299b80 1 -- 172.21.15.27:0/1430309275 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d78e68c400 con 0x55d78e7b12e0 2024-05-10T11:25:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.347+0000 7f587ea00700 1 --2- 172.21.15.27:0/1430309275 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d78e7b12e0 0x55d78e7b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:39.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.348+0000 7f587ea00700 1 --2- 172.21.15.27:0/1430309275 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d78e7b12e0 0x55d78e7b3700 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2fa7afc495a3bcd server_cookie=94dc1de3aaee727f in_seq=0 out_seq=0 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.348+0000 7f5868c00700 10 client.26986 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.348+0000 7f5868c00700 1 -- 172.21.15.27:0/1430309275 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d78e68c400 con 0x55d78e7b12e0 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.348+0000 7f5868c00700 10 client.26986 handle_command_reply: tid=0 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.348+0000 7f5890299b80 1 client.26986 shutdown 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.348+0000 7f5890299b80 1 -- 172.21.15.27:0/1430309275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 msgr2=0x55d78e73e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.348+0000 7f5890299b80 1 --2- 172.21.15.27:0/1430309275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 0x55d78e73e870 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f586c000c00 tx=0x7f586c0aac10 comp rx=0 tx=0).stop 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.348+0000 7f5890299b80 1 -- 172.21.15.27:0/1430309275 shutdown_connections 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.348+0000 7f5890299b80 1 --2- 172.21.15.27:0/1430309275 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55d78e7b12e0 0x55d78e7b3700 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.349+0000 7f5868c00700 0 client.26986 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.348+0000 7f5890299b80 1 --2- 172.21.15.27:0/1430309275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d78e7324e0 0x55d78e73b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.349+0000 7f5890299b80 1 --2- 172.21.15.27:0/1430309275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d78e72c8f0 0x55d78e73e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.349+0000 7f5890299b80 1 --2- 172.21.15.27:0/1430309275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d78e731bc0 0x55d78e73e870 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.349+0000 7f5890299b80 1 -- 172.21.15.27:0/1430309275 >> 172.21.15.27:0/1430309275 conn(0x55d78e6a46d0 msgr2=0x55d78e730f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:39.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.349+0000 7f5890299b80 1 -- 172.21.15.27:0/1430309275 shutdown_connections 2024-05-10T11:25:39.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.349+0000 7f5890299b80 1 -- 172.21.15.27:0/1430309275 wait complete. 2024-05-10T11:25:39.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.349+0000 7f5890299b80 20 client.26986 trim_cache size 0 max 16384 2024-05-10T11:25:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:25:39.352 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:25:39.352 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:25:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:25:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.350+0000 7f5885a00700 1 -- 172.21.15.27:0/241895955 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f58640b29f0 msgr2=0x7f58640b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.350+0000 7f5885a00700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f58640b29f0 0x7f58640b4eb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f587800f590 tx=0x7f587800b040 comp rx=0 tx=0).stop 2024-05-10T11:25:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.350+0000 7f5885a00700 1 -- 172.21.15.27:0/241895955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5880106080 msgr2=0x7f588019c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.350+0000 7f5885a00700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5880106080 0x7f588019c340 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f586c005910 tx=0x7f586c00b670 comp rx=0 tx=0).stop 2024-05-10T11:25:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.350+0000 7f5885a00700 1 -- 172.21.15.27:0/241895955 shutdown_connections 2024-05-10T11:25:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.350+0000 7f5885a00700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f58640b29f0 0x7f58640b4eb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.350+0000 7f5885a00700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f588010b6c0 0x7f58801a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.350+0000 7f5885a00700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58801056c0 0x7f588019be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.350+0000 7f5885a00700 1 --2- 172.21.15.27:0/241895955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5880106080 0x7f588019c340 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:39.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.350+0000 7f5885a00700 1 -- 172.21.15.27:0/241895955 >> 172.21.15.27:0/241895955 conn(0x7f58801011b0 msgr2=0x7f58801017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:39.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.350+0000 7f5885a00700 1 -- 172.21.15.27:0/241895955 shutdown_connections 2024-05-10T11:25:39.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:39.350+0000 7f5885a00700 1 -- 172.21.15.27:0/241895955 wait complete. 2024-05-10T11:25:39.737 INFO:tasks.workunit.client.0.smithi027.stdout: 1 749292 6.26 MB/sec execute 504 sec latency 21.188 ms 2024-05-10T11:25:40.355 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:25:40.356 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:25:40.737 INFO:tasks.workunit.client.0.smithi027.stdout: 1 749650 6.25 MB/sec execute 505 sec latency 36.428 ms 2024-05-10T11:25:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:25:40.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:40 smithi027 ceph-mon[30054]: pgmap v388: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-05-10T11:25:40.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:40 smithi076 ceph-mon[34152]: pgmap v388: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-05-10T11:25:40.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:40 smithi189 ceph-mon[34842]: pgmap v388: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-05-10T11:25:41.737 INFO:tasks.workunit.client.0.smithi027.stdout: 1 750738 6.25 MB/sec execute 506 sec latency 111.408 ms 2024-05-10T11:25:42.737 INFO:tasks.workunit.client.0.smithi027.stdout: 1 752228 6.25 MB/sec execute 507 sec latency 50.669 ms 2024-05-10T11:25:42.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:42 smithi027 ceph-mon[30054]: pgmap v389: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 115 op/s 2024-05-10T11:25:42.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:42 smithi076 ceph-mon[34152]: pgmap v389: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 115 op/s 2024-05-10T11:25:42.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:42 smithi189 ceph-mon[34842]: pgmap v389: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 115 op/s 2024-05-10T11:25:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.308+0000 7fc985200700 1 -- 172.21.15.27:0/4275414526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc980106070 msgr2=0x7fc980112580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.308+0000 7fc985200700 1 --2- 172.21.15.27:0/4275414526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc980106070 0x7fc980112580 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc97800b0a0 tx=0x7fc97800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:25:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.309+0000 7fc985200700 1 -- 172.21.15.27:0/4275414526 shutdown_connections 2024-05-10T11:25:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.309+0000 7fc985200700 1 --2- 172.21.15.27:0/4275414526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc980106070 0x7fc980112580 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.309+0000 7fc985200700 1 --2- 172.21.15.27:0/4275414526 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc98010b6b0 0x7fc98010ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.309+0000 7fc985200700 1 --2- 172.21.15.27:0/4275414526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9801056b0 0x7fc980105b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.309+0000 7fc985200700 1 -- 172.21.15.27:0/4275414526 >> 172.21.15.27:0/4275414526 conn(0x7fc980101180 msgr2=0x7fc9801035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.309+0000 7fc985200700 1 -- 172.21.15.27:0/4275414526 shutdown_connections 2024-05-10T11:25:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.309+0000 7fc985200700 1 -- 172.21.15.27:0/4275414526 wait complete. 2024-05-10T11:25:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.310+0000 7fc985200700 1 Processor -- start 2024-05-10T11:25:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.310+0000 7fc985200700 1 -- start start 2024-05-10T11:25:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc985200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9801056b0 0x7fc98019bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc985200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc980106070 0x7fc98019c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc985200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc98010b6b0 0x7fc9801a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc985200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc9801158f0 con 0x7fc98010b6b0 2024-05-10T11:25:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc985200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc980115770 con 0x7fc9801056b0 2024-05-10T11:25:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc985200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc980115a70 con 0x7fc980106070 2024-05-10T11:25:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc97de00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9801056b0 0x7fc98019bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc97de00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9801056b0 0x7fc98019bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44670/0 (socket says 172.21.15.27:44670) 2024-05-10T11:25:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc97de00700 1 -- 172.21.15.27:0/3034682110 learned_addr learned my addr 172.21.15.27:0/3034682110 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc975400700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc980106070 0x7fc98019c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc97e800700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc98010b6b0 0x7fc9801a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc97de00700 1 -- 172.21.15.27:0/3034682110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc980106070 msgr2=0x7fc98019c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc97de00700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc980106070 0x7fc98019c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc97de00700 1 -- 172.21.15.27:0/3034682110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc98010b6b0 msgr2=0x7fc9801a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.311+0000 7fc97de00700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc98010b6b0 0x7fc9801a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.312+0000 7fc97de00700 1 -- 172.21.15.27:0/3034682110 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9801a0d90 con 0x7fc9801056b0 2024-05-10T11:25:43.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.312+0000 7fc97de00700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9801056b0 0x7fc98019bd80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc97000e9a0 tx=0x7fc97000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:43.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.312+0000 7fc977400700 1 -- 172.21.15.27:0/3034682110 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97000cbd0 con 0x7fc9801056b0 2024-05-10T11:25:43.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.312+0000 7fc977400700 1 -- 172.21.15.27:0/3034682110 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc97000cd70 con 0x7fc9801056b0 2024-05-10T11:25:43.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.312+0000 7fc985200700 1 -- 172.21.15.27:0/3034682110 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9801a1080 con 0x7fc9801056b0 2024-05-10T11:25:43.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.312+0000 7fc977400700 1 -- 172.21.15.27:0/3034682110 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970018680 con 0x7fc9801056b0 2024-05-10T11:25:43.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.312+0000 7fc985200700 1 -- 172.21.15.27:0/3034682110 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9801a8900 con 0x7fc9801056b0 2024-05-10T11:25:43.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.314+0000 7fc985200700 1 -- 172.21.15.27:0/3034682110 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc98006a190 con 0x7fc9801056b0 2024-05-10T11:25:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.316+0000 7fc977400700 1 -- 172.21.15.27:0/3034682110 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc970010430 con 0x7fc9801056b0 2024-05-10T11:25:43.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.318+0000 7fc977400700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9640b2a70 0x7fc9640b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:43.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.318+0000 7fc977400700 1 -- 172.21.15.27:0/3034682110 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7fc9700e2c50 con 0x7fc9801056b0 2024-05-10T11:25:43.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.318+0000 7fc975400700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9640b2a70 0x7fc9640b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:43.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.319+0000 7fc975400700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9640b2a70 0x7fc9640b4f30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc98010acb0 tx=0x7fc968009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:43.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.323+0000 7fc977400700 1 -- 172.21.15.27:0/3034682110 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fc9700aa1a0 con 0x7fc9801056b0 2024-05-10T11:25:43.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.654+0000 7fc985200700 1 -- 172.21.15.27:0/3034682110 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc98002e510 con 0x7fc9801056b0 2024-05-10T11:25:43.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.655+0000 7fc977400700 1 -- 172.21.15.27:0/3034682110 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11936 (secure 0 0 0) 0x7fc9700a9aa0 con 0x7fc9801056b0 2024-05-10T11:25:43.656 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:25:43.657 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:23:14.174773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24484,24527,24472,24488,24467]},"id":1}]} 2024-05-10T11:25:43.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.658+0000 7fc985200700 1 -- 172.21.15.27:0/3034682110 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9640b2a70 msgr2=0x7fc9640b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:43.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.658+0000 7fc985200700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9640b2a70 0x7fc9640b4f30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc98010acb0 tx=0x7fc968009260 comp rx=0 tx=0).stop 2024-05-10T11:25:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.658+0000 7fc985200700 1 -- 172.21.15.27:0/3034682110 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9801056b0 msgr2=0x7fc98019bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.658+0000 7fc985200700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9801056b0 0x7fc98019bd80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc97000e9a0 tx=0x7fc97000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:25:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.659+0000 7fc985200700 1 -- 172.21.15.27:0/3034682110 shutdown_connections 2024-05-10T11:25:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.659+0000 7fc985200700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc9640b2a70 0x7fc9640b4f30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.659+0000 7fc985200700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc98010b6b0 0x7fc9801a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.659+0000 7fc985200700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc9801056b0 0x7fc98019bd80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.659+0000 7fc985200700 1 --2- 172.21.15.27:0/3034682110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc980106070 0x7fc98019c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.659+0000 7fc985200700 1 -- 172.21.15.27:0/3034682110 >> 172.21.15.27:0/3034682110 conn(0x7fc980101180 msgr2=0x7fc980101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.659+0000 7fc985200700 1 -- 172.21.15.27:0/3034682110 shutdown_connections 2024-05-10T11:25:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:43.659+0000 7fc985200700 1 -- 172.21.15.27:0/3034682110 wait complete. 2024-05-10T11:25:43.662 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-05-10T11:25:43.737 INFO:tasks.workunit.client.0.smithi027.stdout: 1 753091 6.25 MB/sec execute 508 sec latency 25.380 ms 2024-05-10T11:25:44.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:25:44.738 INFO:tasks.workunit.client.0.smithi027.stdout: 1 753932 6.24 MB/sec execute 509 sec latency 27.194 ms 2024-05-10T11:25:44.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:44 smithi027 ceph-mon[30054]: pgmap v390: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 115 op/s 2024-05-10T11:25:44.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:44 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3034682110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:25:44.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:44 smithi076 ceph-mon[34152]: pgmap v390: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 115 op/s 2024-05-10T11:25:44.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:44 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3034682110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:25:44.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:44 smithi189 ceph-mon[34842]: pgmap v390: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 115 op/s 2024-05-10T11:25:44.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:44 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3034682110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:25:45.002 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:25:45.738 INFO:tasks.workunit.client.0.smithi027.stdout: 1 755571 6.25 MB/sec execute 510 sec latency 23.843 ms 2024-05-10T11:25:46.708 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:46 smithi027 ceph-mon[30054]: pgmap v391: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 173 op/s 2024-05-10T11:25:46.738 INFO:tasks.workunit.client.0.smithi027.stdout: 1 756546 6.25 MB/sec execute 511 sec latency 61.907 ms 2024-05-10T11:25:46.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:46 smithi076 ceph-mon[34152]: pgmap v391: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 173 op/s 2024-05-10T11:25:46.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:46 smithi189 ceph-mon[34842]: pgmap v391: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 173 op/s 2024-05-10T11:25:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.436+0000 7fe7ff200700 1 -- 172.21.15.27:0/4216270988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe800103d90 msgr2=0x7fe8001102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.436+0000 7fe7ff200700 1 --2- 172.21.15.27:0/4216270988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe800103d90 0x7fe8001102a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800b0a0 tx=0x7fe7f800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.436+0000 7fe7ff200700 1 -- 172.21.15.27:0/4216270988 shutdown_connections 2024-05-10T11:25:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.436+0000 7fe7ff200700 1 --2- 172.21.15.27:0/4216270988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe800109460 0x7fe800109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.436+0000 7fe7ff200700 1 --2- 172.21.15.27:0/4216270988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe800103d90 0x7fe8001102a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.436+0000 7fe7ff200700 1 --2- 172.21.15.27:0/4216270988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8001033d0 0x7fe800103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.436+0000 7fe7ff200700 1 -- 172.21.15.27:0/4216270988 >> 172.21.15.27:0/4216270988 conn(0x7fe8000fef20 msgr2=0x7fe800101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.436+0000 7fe7ff200700 1 -- 172.21.15.27:0/4216270988 shutdown_connections 2024-05-10T11:25:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.436+0000 7fe7ff200700 1 -- 172.21.15.27:0/4216270988 wait complete. 2024-05-10T11:25:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.437+0000 7fe7ff200700 1 Processor -- start 2024-05-10T11:25:47.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.437+0000 7fe7ff200700 1 -- start start 2024-05-10T11:25:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7ff200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8001033d0 0x7fe8001a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7ff200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe800103d90 0x7fe8001a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7ff200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe800109460 0x7fe80019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7ff200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001135c0 con 0x7fe800103d90 2024-05-10T11:25:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7ff200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe800113440 con 0x7fe800109460 2024-05-10T11:25:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7ff200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe800113740 con 0x7fe8001033d0 2024-05-10T11:25:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8001033d0 0x7fe8001a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8001033d0 0x7fe8001a1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55386/0 (socket says 172.21.15.27:55386) 2024-05-10T11:25:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7fde00700 1 -- 172.21.15.27:0/605691767 learned_addr learned my addr 172.21.15.27:0/605691767 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7fd400700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe800103d90 0x7fe8001a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7fe800700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe800109460 0x7fe80019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7fde00700 1 -- 172.21.15.27:0/605691767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe800109460 msgr2=0x7fe80019ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7fde00700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe800109460 0x7fe80019ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7fde00700 1 -- 172.21.15.27:0/605691767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe800103d90 msgr2=0x7fe8001a1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7fde00700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe800103d90 0x7fe8001a1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.438+0000 7fe7fde00700 1 -- 172.21.15.27:0/605691767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe80019c1a0 con 0x7fe8001033d0 2024-05-10T11:25:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.439+0000 7fe7fde00700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8001033d0 0x7fe8001a1880 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe7f000e9a0 tx=0x7fe7f000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.439+0000 7fe7f6a00700 1 -- 172.21.15.27:0/605691767 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f000ca10 con 0x7fe8001033d0 2024-05-10T11:25:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.439+0000 7fe7f6a00700 1 -- 172.21.15.27:0/605691767 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7f000cbb0 con 0x7fe8001033d0 2024-05-10T11:25:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.439+0000 7fe7f6a00700 1 -- 172.21.15.27:0/605691767 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0018680 con 0x7fe8001033d0 2024-05-10T11:25:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.439+0000 7fe7ff200700 1 -- 172.21.15.27:0/605691767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe80019c490 con 0x7fe8001033d0 2024-05-10T11:25:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.439+0000 7fe7ff200700 1 -- 172.21.15.27:0/605691767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8001a8750 con 0x7fe8001033d0 2024-05-10T11:25:47.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.443+0000 7fe7f6a00700 1 -- 172.21.15.27:0/605691767 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fe7f0010430 con 0x7fe8001033d0 2024-05-10T11:25:47.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.444+0000 7fe80fa7ab80 1 Processor -- start 2024-05-10T11:25:47.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe80fa7ab80 1 -- start start 2024-05-10T11:25:47.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe7f6a00700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe7e40b29f0 0x7fe7e40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe80fa7ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 0x55c2ec761180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe80fa7ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec7616c0 0x55c2ec75b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe80fa7ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec75ba50 0x55c2ec75bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe80fa7ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c2ec6ada20 con 0x55c2ec75ba50 2024-05-10T11:25:47.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe80fa7ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c2ec64b760 con 0x55c2ec760da0 2024-05-10T11:25:47.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe80fa7ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c2ec6a99e0 con 0x55c2ec7616c0 2024-05-10T11:25:47.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe7f6a00700 1 -- 172.21.15.27:0/605691767 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7fe7f00e2bc0 con 0x7fe8001033d0 2024-05-10T11:25:47.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 0x55c2ec761180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe7fe800700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe7e40b29f0 0x7fe7e40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 0x55c2ec761180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:34634/0 (socket says 172.21.15.27:34634) 2024-05-10T11:25:47.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.445+0000 7fe7fde00700 1 -- 172.21.15.27:0/2327506568 learned_addr learned my addr 172.21.15.27:0/2327506568 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:47.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.446+0000 7fe7fd400700 1 --2- 172.21.15.27:0/2327506568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec7616c0 0x55c2ec75b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.446+0000 7fe7fe800700 1 --2- 172.21.15.27:0/2327506568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec75ba50 0x55c2ec75bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.446+0000 7fe7fde00700 1 -- 172.21.15.27:0/2327506568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec7616c0 msgr2=0x55c2ec75b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:47.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.446+0000 7fe7fde00700 1 --2- 172.21.15.27:0/2327506568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec7616c0 0x55c2ec75b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.446+0000 7fe7fde00700 1 -- 172.21.15.27:0/2327506568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec75ba50 msgr2=0x55c2ec75bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:47.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.446+0000 7fe7fde00700 1 --2- 172.21.15.27:0/2327506568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec75ba50 0x55c2ec75bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.446+0000 7fe7fde00700 1 -- 172.21.15.27:0/2327506568 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2ec5fa8d0 con 0x55c2ec760da0 2024-05-10T11:25:47.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.446+0000 7fe7fd400700 1 --2- 172.21.15.27:0/2327506568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec7616c0 0x55c2ec75b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:25:47.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.446+0000 7fe7fde00700 1 --2- 172.21.15.27:0/2327506568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 0x55c2ec761180 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00a5a90 tx=0x7fe7f00b21b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1855ff1f8a693838 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:47.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.446+0000 7fe7fe800700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe7e40b29f0 0x7fe7e40b4eb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800b780 tx=0x7fe7f800bb90 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:47.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.447+0000 7fe7ef400700 1 -- 172.21.15.27:0/2327506568 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00aa110 con 0x55c2ec760da0 2024-05-10T11:25:47.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.447+0000 7fe7ef400700 1 -- 172.21.15.27:0/2327506568 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7f00a9a10 con 0x55c2ec760da0 2024-05-10T11:25:47.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.447+0000 7fe7ef400700 1 -- 172.21.15.27:0/2327506568 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00a9c70 con 0x55c2ec760da0 2024-05-10T11:25:47.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.447+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/2327506568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 msgr2=0x55c2ec761180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:47.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.447+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/2327506568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 0x55c2ec761180 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00a5a90 tx=0x7fe7f00b21b0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.447+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/2327506568 shutdown_connections 2024-05-10T11:25:47.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.448+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/2327506568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec75ba50 0x55c2ec75bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.448+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/2327506568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 0x55c2ec761180 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.448+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/2327506568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec7616c0 0x55c2ec75b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.448+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/2327506568 >> 172.21.15.27:0/2327506568 conn(0x55c2ec6d36d0 msgr2=0x55c2ec755ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:47.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.448+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/2327506568 shutdown_connections 2024-05-10T11:25:47.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.448+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/2327506568 wait complete. 2024-05-10T11:25:47.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.448+0000 7fe80fa7ab80 1 -- start start 2024-05-10T11:25:47.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.449+0000 7fe80fa7ab80 1 Processor -- start 2024-05-10T11:25:47.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.449+0000 7fe80fa7ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec75ba50 0x55c2ec767440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.449+0000 7fe80fa7ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec760da0 0x55c2ec7647e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.449+0000 7fe80fa7ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec7616c0 0x55c2ec764d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.449+0000 7fe80fa7ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c2ec768150 con 0x55c2ec75ba50 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.449+0000 7fe80fa7ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c2ec613b20 con 0x55c2ec7616c0 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.449+0000 7fe80fa7ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c2ec7682d0 con 0x55c2ec760da0 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.449+0000 7fe80fa7ab80 10 client.0 _ll_register_callbacks cb 0x55c2ec654c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.450+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec75ba50 0x55c2ec767440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.450+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec75ba50 0x55c2ec767440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35552/0 (socket says 172.21.15.27:35552) 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.450+0000 7fe7fde00700 1 -- 172.21.15.27:0/1208321460 learned_addr learned my addr 172.21.15.27:0/1208321460 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.450+0000 7fe7fe800700 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec7616c0 0x55c2ec764d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.450+0000 7fe7fd400700 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec760da0 0x55c2ec7647e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.450+0000 7fe7fde00700 1 -- 172.21.15.27:0/1208321460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec760da0 msgr2=0x55c2ec7647e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.450+0000 7fe7fde00700 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec760da0 0x55c2ec7647e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.450+0000 7fe7fde00700 1 -- 172.21.15.27:0/1208321460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec7616c0 msgr2=0x55c2ec764d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:47.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.450+0000 7fe7fde00700 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec7616c0 0x55c2ec764d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.450+0000 7fe7fde00700 1 -- 172.21.15.27:0/1208321460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2ec673420 con 0x55c2ec75ba50 2024-05-10T11:25:47.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.450+0000 7fe7fe800700 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec7616c0 0x55c2ec764d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:47.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.451+0000 7fe7fde00700 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec75ba50 0x55c2ec767440 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00a9710 tx=0x7fe7f00a9f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91195ecb6cd9e9b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:47.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.451+0000 7fe7e3e00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:25:47.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.451+0000 7fe7e3e00700 1 -- 172.21.15.27:0/1208321460 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00bee60 con 0x55c2ec75ba50 2024-05-10T11:25:47.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.451+0000 7fe7e3e00700 1 -- 172.21.15.27:0/1208321460 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7fe7f00cc300 con 0x55c2ec75ba50 2024-05-10T11:25:47.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.451+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/1208321460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7f00a9a10 con 0x55c2ec75ba50 2024-05-10T11:25:47.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.451+0000 7fe7e3e00700 1 -- 172.21.15.27:0/1208321460 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fe7f0086020 con 0x55c2ec75ba50 2024-05-10T11:25:47.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.451+0000 7fe80fa7ab80 10 client.16767 fetch_fsmap learned FSMap version 36 2024-05-10T11:25:47.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.451+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/1208321460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c2ec772a00 con 0x55c2ec75ba50 2024-05-10T11:25:47.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.452+0000 7fe7e3e00700 1 -- 172.21.15.27:0/1208321460 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7f00a9a10 con 0x55c2ec75ba50 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.452+0000 7fe7e3e00700 1 -- 172.21.15.27:0/1208321460 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fe7f00892f0 con 0x55c2ec75ba50 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.452+0000 7fe80fa7ab80 10 client.16767 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.452+0000 7fe80fa7ab80 10 client.16767 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.452+0000 7fe80fa7ab80 20 client.16767 populate_metadata read hostname 'smithi027' 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.452+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55c2ec774ed0 0x55c2ec7772c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.452+0000 7fe80fa7ab80 4 client.16767 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.452+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/1208321460 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2ec728140 con 0x55c2ec774ed0 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.452+0000 7fe7fd400700 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55c2ec774ed0 0x55c2ec7772c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.453+0000 7fe7fd400700 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55c2ec774ed0 0x55c2ec7772c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6f3b2fb5f087a6f3 server_cookie=148a9186d8867cbf in_seq=0 out_seq=0 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.453+0000 7fe7e3e00700 10 client.16767 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.454+0000 7fe7e3e00700 1 -- 172.21.15.27:0/1208321460 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c2ec728140 con 0x55c2ec774ed0 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.454+0000 7fe7e3e00700 10 client.16767 handle_command_reply: tid=0 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.454+0000 7fe80fa7ab80 1 client.16767 shutdown 2024-05-10T11:25:47.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.454+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/1208321460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec75ba50 msgr2=0x55c2ec767440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:47.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.454+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec75ba50 0x55c2ec767440 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00a9710 tx=0x7fe7f00a9f20 comp rx=0 tx=0).stop 2024-05-10T11:25:47.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.454+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/1208321460 shutdown_connections 2024-05-10T11:25:47.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.454+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec7616c0 0x55c2ec764d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.454+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec75ba50 0x55c2ec767440 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.454+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55c2ec774ed0 0x55c2ec7772c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.455+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/1208321460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec760da0 0x55c2ec7647e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.455+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/1208321460 >> 172.21.15.27:0/1208321460 conn(0x55c2ec6d36d0 msgr2=0x55c2ec750750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:47.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.455+0000 7fe7e3e00700 0 client.16767 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:25:47.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.455+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/1208321460 shutdown_connections 2024-05-10T11:25:47.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.455+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/1208321460 wait complete. 2024-05-10T11:25:47.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.455+0000 7fe80fa7ab80 20 client.16767 trim_cache size 0 max 16384 2024-05-10T11:25:47.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.474+0000 7fe80fa7ab80 1 Processor -- start 2024-05-10T11:25:47.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.475+0000 7fe80fa7ab80 1 -- start start 2024-05-10T11:25:47.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.475+0000 7fe80fa7ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec75ba50 0x55c2ec75b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.475+0000 7fe80fa7ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec760da0 0x55c2ec762360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.475+0000 7fe80fa7ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec7616c0 0x55c2ec750860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.475+0000 7fe80fa7ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001b1770 con 0x55c2ec760da0 2024-05-10T11:25:47.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.475+0000 7fe80fa7ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001b32b0 con 0x55c2ec7616c0 2024-05-10T11:25:47.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.475+0000 7fe80fa7ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001b1e40 con 0x55c2ec75ba50 2024-05-10T11:25:47.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fd400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec760da0 0x55c2ec762360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fd400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec760da0 0x55c2ec762360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35566/0 (socket says 172.21.15.27:35566) 2024-05-10T11:25:47.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fd400700 1 -- 172.21.15.27:0/3312321165 learned_addr learned my addr 172.21.15.27:0/3312321165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:47.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fe800700 1 --2- 172.21.15.27:0/3312321165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec7616c0 0x55c2ec750860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fde00700 1 --2- 172.21.15.27:0/3312321165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec75ba50 0x55c2ec75b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fde00700 1 -- 172.21.15.27:0/3312321165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec7616c0 msgr2=0x55c2ec750860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fde00700 1 --2- 172.21.15.27:0/3312321165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec7616c0 0x55c2ec750860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fde00700 1 -- 172.21.15.27:0/3312321165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec760da0 msgr2=0x55c2ec762360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fde00700 1 --2- 172.21.15.27:0/3312321165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec760da0 0x55c2ec762360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fde00700 1 -- 172.21.15.27:0/3312321165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2ec7613d0 con 0x55c2ec75ba50 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fd400700 1 --2- 172.21.15.27:0/3312321165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec760da0 0x55c2ec762360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fe800700 1 --2- 172.21.15.27:0/3312321165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec7616c0 0x55c2ec750860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.476+0000 7fe7fde00700 1 --2- 172.21.15.27:0/3312321165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec75ba50 0x55c2ec75b430 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe7f000c2d0 tx=0x7fe7f00b21b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=332da9a6a903f56c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.477+0000 7fe7eea00700 1 -- 172.21.15.27:0/3312321165 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00bee60 con 0x55c2ec75ba50 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.477+0000 7fe7eea00700 1 -- 172.21.15.27:0/3312321165 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7f00a9a10 con 0x55c2ec75ba50 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.477+0000 7fe7eea00700 1 -- 172.21.15.27:0/3312321165 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00892f0 con 0x55c2ec75ba50 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.477+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3312321165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec75ba50 msgr2=0x55c2ec75b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.477+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/3312321165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec75ba50 0x55c2ec75b430 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe7f000c2d0 tx=0x7fe7f00b21b0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.478+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3312321165 shutdown_connections 2024-05-10T11:25:47.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.478+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/3312321165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec75ba50 0x55c2ec75b430 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.478+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/3312321165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec7616c0 0x55c2ec750860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.478+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/3312321165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec760da0 0x55c2ec762360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.478+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3312321165 >> 172.21.15.27:0/3312321165 conn(0x55c2ec6d36d0 msgr2=0x55c2ec762fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:47.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.478+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3312321165 shutdown_connections 2024-05-10T11:25:47.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.478+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3312321165 wait complete. 2024-05-10T11:25:47.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.478+0000 7fe80fa7ab80 1 -- start start 2024-05-10T11:25:47.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.478+0000 7fe80fa7ab80 1 Processor -- start 2024-05-10T11:25:47.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.479+0000 7fe80fa7ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec75ba50 0x55c2ec7774c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.479+0000 7fe80fa7ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 0x55c2ec777a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.479+0000 7fe80fa7ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec7616c0 0x55c2ec7d7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.479+0000 7fe80fa7ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001b2be0 con 0x55c2ec7616c0 2024-05-10T11:25:47.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.479+0000 7fe80fa7ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001b2510 con 0x55c2ec760da0 2024-05-10T11:25:47.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.479+0000 7fe80fa7ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001af730 con 0x55c2ec75ba50 2024-05-10T11:25:47.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.479+0000 7fe80fa7ab80 10 client.0 _ll_register_callbacks cb 0x55c2ec654c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:25:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.479+0000 7fe7fd400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 0x55c2ec777a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.479+0000 7fe7fd400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 0x55c2ec777a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:34664/0 (socket says 172.21.15.27:34664) 2024-05-10T11:25:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.479+0000 7fe7fd400700 1 -- 172.21.15.27:0/3252961647 learned_addr learned my addr 172.21.15.27:0/3252961647 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.479+0000 7fe7fde00700 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec75ba50 0x55c2ec7774c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.479+0000 7fe7fe800700 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec7616c0 0x55c2ec7d7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.480+0000 7fe7fd400700 1 -- 172.21.15.27:0/3252961647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec75ba50 msgr2=0x55c2ec7774c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.480+0000 7fe7fd400700 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec75ba50 0x55c2ec7774c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.480+0000 7fe7fd400700 1 -- 172.21.15.27:0/3252961647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec7616c0 msgr2=0x55c2ec7d7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.480+0000 7fe7fd400700 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec7616c0 0x55c2ec7d7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.480+0000 7fe7fd400700 1 -- 172.21.15.27:0/3252961647 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2ec75f340 con 0x55c2ec760da0 2024-05-10T11:25:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.480+0000 7fe7fde00700 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec75ba50 0x55c2ec7774c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:47.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.480+0000 7fe7fd400700 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 0x55c2ec777a00 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe800110750 tx=0x7fe7e8020780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccf0f4420aa7c3c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:47.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.480+0000 7fe7e3e00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:25:47.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.480+0000 7fe7e3e00700 1 -- 172.21.15.27:0/3252961647 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8004500 con 0x55c2ec760da0 2024-05-10T11:25:47.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.480+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3252961647 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7f00a9a10 con 0x55c2ec760da0 2024-05-10T11:25:47.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.480+0000 7fe7e3e00700 1 -- 172.21.15.27:0/3252961647 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7fe7e8004830 con 0x55c2ec760da0 2024-05-10T11:25:47.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.481+0000 7fe7e3e00700 1 -- 172.21.15.27:0/3252961647 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fe7e802f050 con 0x55c2ec760da0 2024-05-10T11:25:47.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.481+0000 7fe80fa7ab80 10 client.27029 fetch_fsmap learned FSMap version 36 2024-05-10T11:25:47.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.481+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3252961647 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c2ec7dec90 con 0x55c2ec760da0 2024-05-10T11:25:47.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.481+0000 7fe7e3e00700 1 -- 172.21.15.27:0/3252961647 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7f00a9a10 con 0x55c2ec760da0 2024-05-10T11:25:47.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.481+0000 7fe7e3e00700 1 -- 172.21.15.27:0/3252961647 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fe7e8003680 con 0x55c2ec760da0 2024-05-10T11:25:47.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.481+0000 7fe80fa7ab80 10 client.27029 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:25:47.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.481+0000 7fe80fa7ab80 10 client.27029 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:25:47.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.481+0000 7fe80fa7ab80 20 client.27029 populate_metadata read hostname 'smithi027' 2024-05-10T11:25:47.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.481+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55c2ec7df3b0 0x55c2ec7e17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:47.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.481+0000 7fe80fa7ab80 4 client.27029 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.481+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3252961647 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c2ec6bb400 con 0x55c2ec7df3b0 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.482+0000 7fe7fde00700 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55c2ec7df3b0 0x55c2ec7e17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.482+0000 7fe7fde00700 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55c2ec7df3b0 0x55c2ec7e17d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cd8f95dbe533c5b server_cookie=aa9e4cd55315f0c5 in_seq=0 out_seq=0 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.482+0000 7fe7e3e00700 10 client.27029 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.482+0000 7fe7e3e00700 1 -- 172.21.15.27:0/3252961647 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c2ec6bb400 con 0x55c2ec7df3b0 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.482+0000 7fe7e3e00700 10 client.27029 handle_command_reply: tid=0 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.482+0000 7fe80fa7ab80 1 client.27029 shutdown 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.483+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3252961647 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 msgr2=0x55c2ec777a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.483+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 0x55c2ec777a00 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe800110750 tx=0x7fe7e8020780 comp rx=0 tx=0).stop 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.483+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3252961647 shutdown_connections 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.483+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55c2ec7df3b0 0x55c2ec7e17d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.483+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2ec75ba50 0x55c2ec7774c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.483+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2ec7616c0 0x55c2ec7d7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.483+0000 7fe80fa7ab80 1 --2- 172.21.15.27:0/3252961647 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c2ec760da0 0x55c2ec777a00 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.483+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3252961647 >> 172.21.15.27:0/3252961647 conn(0x55c2ec6d36d0 msgr2=0x55c2ec7621e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:47.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.483+0000 7fe7e3e00700 0 client.27029 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:25:47.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.483+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3252961647 shutdown_connections 2024-05-10T11:25:47.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.483+0000 7fe80fa7ab80 1 -- 172.21.15.27:0/3252961647 wait complete. 2024-05-10T11:25:47.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.483+0000 7fe80fa7ab80 20 client.27029 trim_cache size 0 max 16384 2024-05-10T11:25:47.489 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:25:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.484+0000 7fe7ff200700 1 -- 172.21.15.27:0/605691767 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe7e40b29f0 msgr2=0x7fe7e40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.484+0000 7fe7ff200700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe7e40b29f0 0x7fe7e40b4eb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800b780 tx=0x7fe7f800bb90 comp rx=0 tx=0).stop 2024-05-10T11:25:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.484+0000 7fe7ff200700 1 -- 172.21.15.27:0/605691767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8001033d0 msgr2=0x7fe8001a1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.484+0000 7fe7ff200700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8001033d0 0x7fe8001a1880 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe7f000e9a0 tx=0x7fe7f000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.484+0000 7fe7ff200700 1 -- 172.21.15.27:0/605691767 shutdown_connections 2024-05-10T11:25:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.484+0000 7fe7ff200700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe7e40b29f0 0x7fe7e40b4eb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.484+0000 7fe7ff200700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8001033d0 0x7fe8001a1880 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.484+0000 7fe7ff200700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe800109460 0x7fe80019ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.484+0000 7fe7ff200700 1 --2- 172.21.15.27:0/605691767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe800103d90 0x7fe8001a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.484+0000 7fe7ff200700 1 -- 172.21.15.27:0/605691767 >> 172.21.15.27:0/605691767 conn(0x7fe8000fef20 msgr2=0x7fe8000ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:47.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.484+0000 7fe7ff200700 1 -- 172.21.15.27:0/605691767 shutdown_connections 2024-05-10T11:25:47.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:47.484+0000 7fe7ff200700 1 -- 172.21.15.27:0/605691767 wait complete. 2024-05-10T11:25:47.738 INFO:tasks.workunit.client.0.smithi027.stdout: 1 757244 6.24 MB/sec execute 512 sec latency 30.149 ms 2024-05-10T11:25:48.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:25:48.738 INFO:tasks.workunit.client.0.smithi027.stdout: 1 758716 6.24 MB/sec execute 513 sec latency 25.652 ms 2024-05-10T11:25:48.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:48 smithi027 ceph-mon[30054]: pgmap v392: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 109 op/s 2024-05-10T11:25:48.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:48 smithi076 ceph-mon[34152]: pgmap v392: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 109 op/s 2024-05-10T11:25:48.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:48 smithi189 ceph-mon[34842]: pgmap v392: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 109 op/s 2024-05-10T11:25:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:25:49.739 INFO:tasks.workunit.client.0.smithi027.stdout: 1 759984 6.25 MB/sec execute 514 sec latency 22.364 ms 2024-05-10T11:25:50.739 INFO:tasks.workunit.client.0.smithi027.stdout: 1 760460 6.24 MB/sec execute 515 sec latency 47.106 ms 2024-05-10T11:25:50.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:50 smithi027 ceph-mon[30054]: pgmap v393: 129 pgs: 129 active+clean; 593 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-05-10T11:25:50.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:50 smithi076 ceph-mon[34152]: pgmap v393: 129 pgs: 129 active+clean; 593 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-05-10T11:25:50.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:50 smithi189 ceph-mon[34842]: pgmap v393: 129 pgs: 129 active+clean; 593 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-05-10T11:25:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.617+0000 7fb9a8c00700 1 -- 172.21.15.27:0/3996252359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9a41060a0 msgr2=0x7fb9a41125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.617+0000 7fb9a8c00700 1 --2- 172.21.15.27:0/3996252359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9a41060a0 0x7fb9a41125b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb99c00b0a0 tx=0x7fb99c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.617+0000 7fb9a8c00700 1 -- 172.21.15.27:0/3996252359 shutdown_connections 2024-05-10T11:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.617+0000 7fb9a8c00700 1 --2- 172.21.15.27:0/3996252359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9a41060a0 0x7fb9a41125b0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.617+0000 7fb9a8c00700 1 --2- 172.21.15.27:0/3996252359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9a410b6e0 0x7fb9a410bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.617+0000 7fb9a8c00700 1 --2- 172.21.15.27:0/3996252359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a41056e0 0x7fb9a4105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.617+0000 7fb9a8c00700 1 -- 172.21.15.27:0/3996252359 >> 172.21.15.27:0/3996252359 conn(0x7fb9a41011d0 msgr2=0x7fb9a41035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.617+0000 7fb9a8c00700 1 -- 172.21.15.27:0/3996252359 shutdown_connections 2024-05-10T11:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.618+0000 7fb9a8c00700 1 -- 172.21.15.27:0/3996252359 wait complete. 2024-05-10T11:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.619+0000 7fb9a8c00700 1 Processor -- start 2024-05-10T11:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.619+0000 7fb9a8c00700 1 -- start start 2024-05-10T11:25:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.620+0000 7fb9a8c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9a41056e0 0x7fb9a419bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.620+0000 7fb9a8c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9a41060a0 0x7fb9a419c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.620+0000 7fb9a8c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a410b6e0 0x7fb9a41a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.620+0000 7fb9a8c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a4115930 con 0x7fb9a41060a0 2024-05-10T11:25:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.620+0000 7fb9a8c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41157b0 con 0x7fb9a41056e0 2024-05-10T11:25:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.620+0000 7fb9a8c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a4115ab0 con 0x7fb9a410b6e0 2024-05-10T11:25:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.620+0000 7fb999e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9a41056e0 0x7fb9a419bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.620+0000 7fb9a2800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a410b6e0 0x7fb9a41a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.620+0000 7fb999e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9a41056e0 0x7fb9a419bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:34670/0 (socket says 172.21.15.27:34670) 2024-05-10T11:25:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.620+0000 7fb999e00700 1 -- 172.21.15.27:0/488751660 learned_addr learned my addr 172.21.15.27:0/488751660 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.620+0000 7fb9a1e00700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9a41060a0 0x7fb9a419c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.621+0000 7fb9a2800700 1 -- 172.21.15.27:0/488751660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9a41056e0 msgr2=0x7fb9a419bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.621+0000 7fb9a2800700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9a41056e0 0x7fb9a419bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.621+0000 7fb9a2800700 1 -- 172.21.15.27:0/488751660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9a41060a0 msgr2=0x7fb9a419c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.621+0000 7fb9a2800700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9a41060a0 0x7fb9a419c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.621+0000 7fb9a2800700 1 -- 172.21.15.27:0/488751660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a41a0d50 con 0x7fb9a410b6e0 2024-05-10T11:25:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.621+0000 7fb999e00700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9a41056e0 0x7fb9a419bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.621+0000 7fb9a1e00700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9a41060a0 0x7fb9a419c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:25:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.621+0000 7fb9a2800700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a410b6e0 0x7fb9a41a0610 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0062a0 tx=0x7fb99c004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.621+0000 7fb99b400700 1 -- 172.21.15.27:0/488751660 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c021070 con 0x7fb9a410b6e0 2024-05-10T11:25:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.621+0000 7fb99b400700 1 -- 172.21.15.27:0/488751660 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb99c007e30 con 0x7fb9a410b6e0 2024-05-10T11:25:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.621+0000 7fb99b400700 1 -- 172.21.15.27:0/488751660 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0046d0 con 0x7fb9a410b6e0 2024-05-10T11:25:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.621+0000 7fb9a8c00700 1 -- 172.21.15.27:0/488751660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a41a0fe0 con 0x7fb9a410b6e0 2024-05-10T11:25:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.622+0000 7fb9a8c00700 1 -- 172.21.15.27:0/488751660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a41a88c0 con 0x7fb9a410b6e0 2024-05-10T11:25:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.625+0000 7fb99b400700 1 -- 172.21.15.27:0/488751660 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb99c007980 con 0x7fb9a410b6e0 2024-05-10T11:25:51.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.627+0000 7fb9b35c4b80 1 Processor -- start 2024-05-10T11:25:51.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.627+0000 7fb9b35c4b80 1 -- start start 2024-05-10T11:25:51.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb9b35c4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f40bc0 0x557362f40fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb99b400700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb9880b29f0 0x7fb9880b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb9b35c4b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f414e0 0x557362f3b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb9b35c4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f3b8f0 0x557362f3bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb9b35c4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557362e8da20 con 0x557362f3b8f0 2024-05-10T11:25:51.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb9b35c4b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557362e2b760 con 0x557362f414e0 2024-05-10T11:25:51.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb9b35c4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557362e899e0 con 0x557362f40bc0 2024-05-10T11:25:51.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb99b400700 1 -- 172.21.15.27:0/488751660 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7fb99c00f020 con 0x7fb9a410b6e0 2024-05-10T11:25:51.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb9a1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f40bc0 0x557362f40fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb9a1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f40bc0 0x557362f40fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55446/0 (socket says 172.21.15.27:55446) 2024-05-10T11:25:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb9a1e00700 1 -- 172.21.15.27:0/3397485969 learned_addr learned my addr 172.21.15.27:0/3397485969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb9a2800700 1 --2- 172.21.15.27:0/3397485969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f3b8f0 0x557362f3bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb9a1e00700 1 --2- 172.21.15.27:0/3397485969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f414e0 0x557362f3b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.628+0000 7fb999e00700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb9880b29f0 0x7fb9880b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.629+0000 7fb9a1e00700 1 -- 172.21.15.27:0/3397485969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f414e0 msgr2=0x557362f3b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:51.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.629+0000 7fb9a1e00700 1 --2- 172.21.15.27:0/3397485969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f414e0 0x557362f3b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.629+0000 7fb9a1e00700 1 -- 172.21.15.27:0/3397485969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f3b8f0 msgr2=0x557362f3bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:51.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.629+0000 7fb9a1e00700 1 --2- 172.21.15.27:0/3397485969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f3b8f0 0x557362f3bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.629+0000 7fb9a1e00700 1 -- 172.21.15.27:0/3397485969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557362dda8d0 con 0x557362f40bc0 2024-05-10T11:25:51.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.629+0000 7fb9a1e00700 1 --2- 172.21.15.27:0/3397485969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f40bc0 0x557362f40fa0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb98c005d90 tx=0x7fb98c00b550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2f7bd67818b7b2f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:51.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.629+0000 7fb999e00700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb9880b29f0 0x7fb9880b4eb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb994009680 tx=0x7fb994009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:51.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.630+0000 7fb993400700 1 -- 172.21.15.27:0/3397485969 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c014e10 con 0x557362f40bc0 2024-05-10T11:25:51.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.630+0000 7fb993400700 1 -- 172.21.15.27:0/3397485969 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb98c004ce0 con 0x557362f40bc0 2024-05-10T11:25:51.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.630+0000 7fb993400700 1 -- 172.21.15.27:0/3397485969 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c0056c0 con 0x557362f40bc0 2024-05-10T11:25:51.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.630+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3397485969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f40bc0 msgr2=0x557362f40fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:51.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.630+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/3397485969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f40bc0 0x557362f40fa0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb98c005d90 tx=0x7fb98c00b550 comp rx=0 tx=0).stop 2024-05-10T11:25:51.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.631+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3397485969 shutdown_connections 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.631+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/3397485969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f40bc0 0x557362f40fa0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.631+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/3397485969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f3b8f0 0x557362f3bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.631+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/3397485969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f414e0 0x557362f3b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.631+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3397485969 >> 172.21.15.27:0/3397485969 conn(0x557362eb36d0 msgr2=0x557362f35b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.631+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3397485969 shutdown_connections 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.631+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3397485969 wait complete. 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.631+0000 7fb9b35c4b80 1 -- start start 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.631+0000 7fb9b35c4b80 1 Processor -- start 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.632+0000 7fb9b35c4b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f3b8f0 0x557362f471d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.632+0000 7fb9b35c4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 0x557362f44570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.632+0000 7fb9b35c4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f414e0 0x557362f44ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.632+0000 7fb9b35c4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557362f47f20 con 0x557362f40bc0 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.632+0000 7fb9b35c4b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557362df3b20 con 0x557362f3b8f0 2024-05-10T11:25:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.632+0000 7fb9b35c4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557362f480a0 con 0x557362f414e0 2024-05-10T11:25:51.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.632+0000 7fb9b35c4b80 10 client.0 _ll_register_callbacks cb 0x557362e34c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:25:51.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.632+0000 7fb9a1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 0x557362f44570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.632+0000 7fb9a1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 0x557362f44570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35616/0 (socket says 172.21.15.27:35616) 2024-05-10T11:25:51.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.632+0000 7fb9a1e00700 1 -- 172.21.15.27:0/2656283485 learned_addr learned my addr 172.21.15.27:0/2656283485 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.632+0000 7fb999e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f3b8f0 0x557362f471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.632+0000 7fb9a2800700 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f414e0 0x557362f44ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.633+0000 7fb9a2800700 1 -- 172.21.15.27:0/2656283485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f3b8f0 msgr2=0x557362f471d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.633+0000 7fb9a2800700 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f3b8f0 0x557362f471d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.633+0000 7fb9a2800700 1 -- 172.21.15.27:0/2656283485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 msgr2=0x557362f44570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.633+0000 7fb9a2800700 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 0x557362f44570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.633+0000 7fb9a2800700 1 -- 172.21.15.27:0/2656283485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557362e53420 con 0x557362f414e0 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.633+0000 7fb9a1e00700 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 0x557362f44570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.633+0000 7fb9a2800700 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f414e0 0x557362f44ab0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0e1170 tx=0x7fb99c0aa760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65232d843c776424 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.633+0000 7fb987e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.633+0000 7fb987e00700 1 -- 172.21.15.27:0/2656283485 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0aa8a0 con 0x557362f414e0 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.633+0000 7fb987e00700 1 -- 172.21.15.27:0/2656283485 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7fb99c0074e0 con 0x557362f414e0 2024-05-10T11:25:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.633+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/2656283485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb98c004ce0 con 0x557362f414e0 2024-05-10T11:25:51.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.634+0000 7fb987e00700 1 -- 172.21.15.27:0/2656283485 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb99c0ab490 con 0x557362f414e0 2024-05-10T11:25:51.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.634+0000 7fb9b35c4b80 10 client.27022 fetch_fsmap learned FSMap version 36 2024-05-10T11:25:51.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.634+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/2656283485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x557362f47fd0 con 0x557362f414e0 2024-05-10T11:25:51.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.634+0000 7fb987e00700 1 -- 172.21.15.27:0/2656283485 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb98c004ce0 con 0x557362f414e0 2024-05-10T11:25:51.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.634+0000 7fb987e00700 1 -- 172.21.15.27:0/2656283485 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fb99c011070 con 0x557362f414e0 2024-05-10T11:25:51.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.635+0000 7fb9b35c4b80 10 client.27022 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:25:51.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.635+0000 7fb9b35c4b80 10 client.27022 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:25:51.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.635+0000 7fb9b35c4b80 20 client.27022 populate_metadata read hostname 'smithi027' 2024-05-10T11:25:51.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.635+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x557362f54c70 0x557362f57060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.635+0000 7fb9b35c4b80 4 client.27022 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:25:51.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.635+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/2656283485 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557362f08140 con 0x557362f54c70 2024-05-10T11:25:51.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.635+0000 7fb999e00700 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x557362f54c70 0x557362f57060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.636+0000 7fb999e00700 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x557362f54c70 0x557362f57060 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ab18ad8495efb27e server_cookie=f6b15eb992c21d4d in_seq=0 out_seq=0 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.636+0000 7fb987e00700 10 client.27022 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.636+0000 7fb987e00700 1 -- 172.21.15.27:0/2656283485 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557362f08140 con 0x557362f54c70 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.636+0000 7fb987e00700 10 client.27022 handle_command_reply: tid=0 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.636+0000 7fb9b35c4b80 1 client.27022 shutdown 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.637+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/2656283485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f414e0 msgr2=0x557362f44ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.637+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f414e0 0x557362f44ab0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0e1170 tx=0x7fb99c0aa760 comp rx=0 tx=0).stop 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.637+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/2656283485 shutdown_connections 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.637+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x557362f54c70 0x557362f57060 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.637+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f414e0 0x557362f44ab0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.637+0000 7fb987e00700 0 client.27022 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.637+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f3b8f0 0x557362f471d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.637+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/2656283485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 0x557362f44570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.637+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/2656283485 >> 172.21.15.27:0/2656283485 conn(0x557362eb36d0 msgr2=0x557362f35190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:51.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.637+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/2656283485 shutdown_connections 2024-05-10T11:25:51.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.637+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/2656283485 wait complete. 2024-05-10T11:25:51.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.637+0000 7fb9b35c4b80 20 client.27022 trim_cache size 0 max 16384 2024-05-10T11:25:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.654+0000 7fb9b35c4b80 1 Processor -- start 2024-05-10T11:25:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.655+0000 7fb9b35c4b80 1 -- start start 2024-05-10T11:25:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.655+0000 7fb9b35c4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f3b8f0 0x557362f3b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.655+0000 7fb9b35c4b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f40bc0 0x557362f50fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.655+0000 7fb9b35c4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f414e0 0x557362f35910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.655+0000 7fb9b35c4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41b17b0 con 0x557362f3b8f0 2024-05-10T11:25:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.655+0000 7fb9b35c4b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41b32f0 con 0x557362f40bc0 2024-05-10T11:25:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.655+0000 7fb9b35c4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41b1e80 con 0x557362f414e0 2024-05-10T11:25:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.655+0000 7fb9a1e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f40bc0 0x557362f50fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.655+0000 7fb9a1e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f40bc0 0x557362f50fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:34712/0 (socket says 172.21.15.27:34712) 2024-05-10T11:25:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.655+0000 7fb9a1e00700 1 -- 172.21.15.27:0/982488476 learned_addr learned my addr 172.21.15.27:0/982488476 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.655+0000 7fb9a2800700 1 --2- 172.21.15.27:0/982488476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f414e0 0x557362f35910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.656+0000 7fb999e00700 1 --2- 172.21.15.27:0/982488476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f3b8f0 0x557362f3b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.656+0000 7fb9a2800700 1 -- 172.21.15.27:0/982488476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f40bc0 msgr2=0x557362f50fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.656+0000 7fb9a2800700 1 --2- 172.21.15.27:0/982488476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f40bc0 0x557362f50fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.656+0000 7fb9a2800700 1 -- 172.21.15.27:0/982488476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f3b8f0 msgr2=0x557362f3b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.656+0000 7fb9a2800700 1 --2- 172.21.15.27:0/982488476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f3b8f0 0x557362f3b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.656+0000 7fb9a2800700 1 -- 172.21.15.27:0/982488476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557362f47e00 con 0x557362f414e0 2024-05-10T11:25:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.656+0000 7fb9a1e00700 1 --2- 172.21.15.27:0/982488476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f40bc0 0x557362f50fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.656+0000 7fb999e00700 1 --2- 172.21.15.27:0/982488476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f3b8f0 0x557362f3b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:25:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.656+0000 7fb9a2800700 1 --2- 172.21.15.27:0/982488476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f414e0 0x557362f35910 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0a8000 tx=0x7fb99c004190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d58c3686fa72438f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.657+0000 7fb992a00700 1 -- 172.21.15.27:0/982488476 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0aa8a0 con 0x557362f414e0 2024-05-10T11:25:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.657+0000 7fb992a00700 1 -- 172.21.15.27:0/982488476 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb99c089040 con 0x557362f414e0 2024-05-10T11:25:51.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.657+0000 7fb992a00700 1 -- 172.21.15.27:0/982488476 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0aad90 con 0x557362f414e0 2024-05-10T11:25:51.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.657+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/982488476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f414e0 msgr2=0x557362f35910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:51.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.657+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/982488476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f414e0 0x557362f35910 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0a8000 tx=0x7fb99c004190 comp rx=0 tx=0).stop 2024-05-10T11:25:51.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.657+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/982488476 shutdown_connections 2024-05-10T11:25:51.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.657+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/982488476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f414e0 0x557362f35910 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.657+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/982488476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f3b8f0 0x557362f3b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.657+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/982488476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f40bc0 0x557362f50fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.658+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/982488476 >> 172.21.15.27:0/982488476 conn(0x557362eb36d0 msgr2=0x557362f42ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:51.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.658+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/982488476 shutdown_connections 2024-05-10T11:25:51.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.658+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/982488476 wait complete. 2024-05-10T11:25:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.658+0000 7fb9b35c4b80 1 -- start start 2024-05-10T11:25:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.659+0000 7fb9b35c4b80 1 Processor -- start 2024-05-10T11:25:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.659+0000 7fb9b35c4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f3b8f0 0x557362f4d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.659+0000 7fb9b35c4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 0x557362f4da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.659+0000 7fb9b35c4b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f414e0 0x557362f5ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.659+0000 7fb9b35c4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41b2c20 con 0x557362f40bc0 2024-05-10T11:25:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.659+0000 7fb9b35c4b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41b2550 con 0x557362f414e0 2024-05-10T11:25:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.659+0000 7fb9b35c4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41af770 con 0x557362f3b8f0 2024-05-10T11:25:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.659+0000 7fb9b35c4b80 10 client.0 _ll_register_callbacks cb 0x557362e34c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:25:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.659+0000 7fb9a2800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f414e0 0x557362f5ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.660+0000 7fb9a2800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f414e0 0x557362f5ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:34716/0 (socket says 172.21.15.27:34716) 2024-05-10T11:25:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.660+0000 7fb9a1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 0x557362f4da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.660+0000 7fb9a2800700 1 -- 172.21.15.27:0/3714559472 learned_addr learned my addr 172.21.15.27:0/3714559472 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.660+0000 7fb999e00700 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f3b8f0 0x557362f4d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.660+0000 7fb9a2800700 1 -- 172.21.15.27:0/3714559472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f3b8f0 msgr2=0x557362f4d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.660+0000 7fb9a2800700 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f3b8f0 0x557362f4d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.660+0000 7fb9a2800700 1 -- 172.21.15.27:0/3714559472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 msgr2=0x557362f4da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.660+0000 7fb9a2800700 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 0x557362f4da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.660+0000 7fb9a2800700 1 -- 172.21.15.27:0/3714559472 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557362f47540 con 0x557362f414e0 2024-05-10T11:25:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.660+0000 7fb9a1e00700 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 0x557362f4da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.660+0000 7fb999e00700 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f3b8f0 0x557362f4d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:25:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.661+0000 7fb9a2800700 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f414e0 0x557362f5ba60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0aad20 tx=0x7fb99c0ab200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=733d5d10b65e585e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.661+0000 7fb987e00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.661+0000 7fb987e00700 1 -- 172.21.15.27:0/3714559472 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0aa8a0 con 0x557362f414e0 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.661+0000 7fb987e00700 1 -- 172.21.15.27:0/3714559472 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7fb99c089de0 con 0x557362f414e0 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.661+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3714559472 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb99c089040 con 0x557362f414e0 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.661+0000 7fb987e00700 1 -- 172.21.15.27:0/3714559472 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb99c004870 con 0x557362f414e0 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.661+0000 7fb9b35c4b80 10 client.27047 fetch_fsmap learned FSMap version 36 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.661+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3714559472 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x557362f30ab0 con 0x557362f414e0 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.661+0000 7fb987e00700 1 -- 172.21.15.27:0/3714559472 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb99c089040 con 0x557362f414e0 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.662+0000 7fb987e00700 1 -- 172.21.15.27:0/3714559472 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fb99c011070 con 0x557362f414e0 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.662+0000 7fb9b35c4b80 10 client.27047 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.662+0000 7fb9b35c4b80 10 client.27047 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.662+0000 7fb9b35c4b80 20 client.27047 populate_metadata read hostname 'smithi027' 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.662+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x557362f54c70 0x557362fc0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:51.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.662+0000 7fb9b35c4b80 4 client.27047 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:25:51.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.662+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3714559472 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557362e9b400 con 0x557362f54c70 2024-05-10T11:25:51.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.663+0000 7fb999e00700 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x557362f54c70 0x557362fc0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:51.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.663+0000 7fb999e00700 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x557362f54c70 0x557362fc0f90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5a5e552c64dbc7a8 server_cookie=dcf1a9a707aef265 in_seq=0 out_seq=0 2024-05-10T11:25:51.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.663+0000 7fb987e00700 10 client.27047 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:25:51.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.663+0000 7fb987e00700 1 -- 172.21.15.27:0/3714559472 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557362e9b400 con 0x557362f54c70 2024-05-10T11:25:51.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.663+0000 7fb987e00700 10 client.27047 handle_command_reply: tid=0 2024-05-10T11:25:51.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.663+0000 7fb9b35c4b80 1 client.27047 shutdown 2024-05-10T11:25:51.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.664+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3714559472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f414e0 msgr2=0x557362f5ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:51.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.664+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f414e0 0x557362f5ba60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0aad20 tx=0x7fb99c0ab200 comp rx=0 tx=0).stop 2024-05-10T11:25:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.664+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3714559472 shutdown_connections 2024-05-10T11:25:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.664+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x557362f54c70 0x557362fc0f90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.664+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557362f3b8f0 0x557362f4d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.664+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557362f414e0 0x557362f5ba60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.664+0000 7fb9b35c4b80 1 --2- 172.21.15.27:0/3714559472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557362f40bc0 0x557362f4da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.664+0000 7fb987e00700 0 client.27047 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:25:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.664+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3714559472 >> 172.21.15.27:0/3714559472 conn(0x557362eb36d0 msgr2=0x557362f3f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.664+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3714559472 shutdown_connections 2024-05-10T11:25:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.664+0000 7fb9b35c4b80 1 -- 172.21.15.27:0/3714559472 wait complete. 2024-05-10T11:25:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.664+0000 7fb9b35c4b80 20 client.27047 trim_cache size 0 max 16384 2024-05-10T11:25:51.667 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:25:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.665+0000 7fb9a8c00700 1 -- 172.21.15.27:0/488751660 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb9880b29f0 msgr2=0x7fb9880b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.665+0000 7fb9a8c00700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb9880b29f0 0x7fb9880b4eb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb994009680 tx=0x7fb994009260 comp rx=0 tx=0).stop 2024-05-10T11:25:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.665+0000 7fb9a8c00700 1 -- 172.21.15.27:0/488751660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a410b6e0 msgr2=0x7fb9a41a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.665+0000 7fb9a8c00700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a410b6e0 0x7fb9a41a0610 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0062a0 tx=0x7fb99c004c80 comp rx=0 tx=0).stop 2024-05-10T11:25:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.665+0000 7fb9a8c00700 1 -- 172.21.15.27:0/488751660 shutdown_connections 2024-05-10T11:25:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.665+0000 7fb9a8c00700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb9880b29f0 0x7fb9880b4eb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.665+0000 7fb9a8c00700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a410b6e0 0x7fb9a41a0610 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.665+0000 7fb9a8c00700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9a41056e0 0x7fb9a419bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.665+0000 7fb9a8c00700 1 --2- 172.21.15.27:0/488751660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9a41060a0 0x7fb9a419c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.665+0000 7fb9a8c00700 1 -- 172.21.15.27:0/488751660 >> 172.21.15.27:0/488751660 conn(0x7fb9a41011d0 msgr2=0x7fb9a4101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.666+0000 7fb9a8c00700 1 -- 172.21.15.27:0/488751660 shutdown_connections 2024-05-10T11:25:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:51.666+0000 7fb9a8c00700 1 -- 172.21.15.27:0/488751660 wait complete. 2024-05-10T11:25:51.738 INFO:tasks.workunit.client.0.smithi027.stdout: 1 761641 6.24 MB/sec execute 516 sec latency 36.152 ms 2024-05-10T11:25:52.739 INFO:tasks.workunit.client.0.smithi027.stdout: 1 763384 6.25 MB/sec execute 517 sec latency 26.702 ms 2024-05-10T11:25:52.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:52 smithi027 ceph-mon[30054]: pgmap v394: 129 pgs: 129 active+clean; 593 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-05-10T11:25:52.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:52 smithi076 ceph-mon[34152]: pgmap v394: 129 pgs: 129 active+clean; 593 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-05-10T11:25:52.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:52 smithi189 ceph-mon[34842]: pgmap v394: 129 pgs: 129 active+clean; 593 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-05-10T11:25:53.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:25:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:25:53.739 INFO:tasks.workunit.client.0.smithi027.stdout: 1 763983 6.24 MB/sec execute 518 sec latency 22.983 ms 2024-05-10T11:25:54.739 INFO:tasks.workunit.client.0.smithi027.stdout: 1 765077 6.24 MB/sec execute 519 sec latency 41.017 ms 2024-05-10T11:25:54.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:54 smithi076 ceph-mon[34152]: pgmap v395: 129 pgs: 129 active+clean; 593 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-05-10T11:25:54.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:54 smithi189 ceph-mon[34842]: pgmap v395: 129 pgs: 129 active+clean; 593 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-05-10T11:25:55.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:54 smithi027 ceph-mon[30054]: pgmap v395: 129 pgs: 129 active+clean; 593 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-05-10T11:25:55.739 INFO:tasks.workunit.client.0.smithi027.stdout: 1 766767 6.25 MB/sec execute 520 sec latency 25.324 ms 2024-05-10T11:25:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.250+0000 7f07a7c00700 1 -- 172.21.15.27:0/1686871368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07a006bad0 msgr2=0x7f07a010fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.250+0000 7f07a7c00700 1 --2- 172.21.15.27:0/1686871368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07a006bad0 0x7f07a010fa10 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f078c009990 tx=0x7f078c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.250+0000 7f07a7c00700 1 -- 172.21.15.27:0/1686871368 shutdown_connections 2024-05-10T11:25:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.250+0000 7f07a7c00700 1 --2- 172.21.15.27:0/1686871368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a010ff50 0x7f07a0112340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.250+0000 7f07a7c00700 1 --2- 172.21.15.27:0/1686871368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a006b1b0 0x7f07a006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.250+0000 7f07a7c00700 1 --2- 172.21.15.27:0/1686871368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07a006bad0 0x7f07a010fa10 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.250+0000 7f07a7c00700 1 -- 172.21.15.27:0/1686871368 >> 172.21.15.27:0/1686871368 conn(0x7f07a0079350 msgr2=0x7f07a0079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.250+0000 7f07a7c00700 1 -- 172.21.15.27:0/1686871368 shutdown_connections 2024-05-10T11:25:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.250+0000 7f07a7c00700 1 -- 172.21.15.27:0/1686871368 wait complete. 2024-05-10T11:25:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.251+0000 7f07a7c00700 1 Processor -- start 2024-05-10T11:25:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f07a7c00700 1 -- start start 2024-05-10T11:25:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f07a7c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a006b1b0 0x7f07a0104680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f07a7c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07a006bad0 0x7f07a0104bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f07a7c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a010ff50 0x7f07a0105100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f07a7c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f07a01158e0 con 0x7f07a006b1b0 2024-05-10T11:25:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f07a7c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f07a0115760 con 0x7f07a006bad0 2024-05-10T11:25:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f07a7c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f07a0115a60 con 0x7f07a010ff50 2024-05-10T11:25:56.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f079fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07a006bad0 0x7f07a0104bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f07a5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a006b1b0 0x7f07a0104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f07a5c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a010ff50 0x7f07a0105100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f079fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07a006bad0 0x7f07a0104bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50196/0 (socket says 172.21.15.27:50196) 2024-05-10T11:25:56.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f07a5c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a010ff50 0x7f07a0105100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37148/0 (socket says 172.21.15.27:37148) 2024-05-10T11:25:56.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.252+0000 7f079fe00700 1 -- 172.21.15.27:0/2068377319 learned_addr learned my addr 172.21.15.27:0/2068377319 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:56.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.253+0000 7f07a5c00700 1 -- 172.21.15.27:0/2068377319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07a006bad0 msgr2=0x7f07a0104bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:56.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.253+0000 7f07a5c00700 1 --2- 172.21.15.27:0/2068377319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07a006bad0 0x7f07a0104bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.253+0000 7f07a5c00700 1 -- 172.21.15.27:0/2068377319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a006b1b0 msgr2=0x7f07a0104680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:56.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.253+0000 7f07a5c00700 1 --2- 172.21.15.27:0/2068377319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a006b1b0 0x7f07a0104680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.253+0000 7f07a5c00700 1 -- 172.21.15.27:0/2068377319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a01ac480 con 0x7f07a010ff50 2024-05-10T11:25:56.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.253+0000 7f07a5c00700 1 --2- 172.21.15.27:0/2068377319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a010ff50 0x7f07a0105100 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f079800efc0 tx=0x7f079800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:56.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.253+0000 7f079d600700 1 -- 172.21.15.27:0/2068377319 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07980092e0 con 0x7f07a010ff50 2024-05-10T11:25:56.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.253+0000 7f079d600700 1 -- 172.21.15.27:0/2068377319 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0798004bb0 con 0x7f07a010ff50 2024-05-10T11:25:56.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.253+0000 7f079d600700 1 -- 172.21.15.27:0/2068377319 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0798004020 con 0x7f07a010ff50 2024-05-10T11:25:56.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.254+0000 7f07a7c00700 1 -- 172.21.15.27:0/2068377319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a01ac710 con 0x7f07a010ff50 2024-05-10T11:25:56.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.254+0000 7f07a7c00700 1 -- 172.21.15.27:0/2068377319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a01acc00 con 0x7f07a010ff50 2024-05-10T11:25:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.257+0000 7f079d600700 1 -- 172.21.15.27:0/2068377319 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f079801b030 con 0x7f07a010ff50 2024-05-10T11:25:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.258+0000 7f07b267db80 1 Processor -- start 2024-05-10T11:25:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.259+0000 7f07b267db80 1 -- start start 2024-05-10T11:25:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.259+0000 7f07b267db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 0x5640b7e1d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.259+0000 7f07b267db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d980 0x5640b7e21490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.259+0000 7f07b267db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e219d0 0x5640b7e21e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.259+0000 7f079d600700 1 --2- 172.21.15.27:0/2068377319 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f07880b2aa0 0x7f07880b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.259+0000 7f07b267db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5640b7d6fa20 con 0x5640b7e219d0 2024-05-10T11:25:56.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.259+0000 7f07b267db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5640b7d0d760 con 0x5640b7e1d060 2024-05-10T11:25:56.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.259+0000 7f07b267db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5640b7d6b9e0 con 0x5640b7e1d980 2024-05-10T11:25:56.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.259+0000 7f079d600700 1 -- 172.21.15.27:0/2068377319 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7f07980e23c0 con 0x7f07a010ff50 2024-05-10T11:25:56.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.259+0000 7f079fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 0x5640b7e1d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.260+0000 7f079fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 0x5640b7e1d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50198/0 (socket says 172.21.15.27:50198) 2024-05-10T11:25:56.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.260+0000 7f07a5200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d980 0x5640b7e21490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.260+0000 7f079fe00700 1 -- 172.21.15.27:0/536005232 learned_addr learned my addr 172.21.15.27:0/536005232 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:56.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.260+0000 7f079fe00700 1 --2- 172.21.15.27:0/2068377319 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f07880b2aa0 0x7f07880b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.260+0000 7f07a5c00700 1 --2- 172.21.15.27:0/536005232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e219d0 0x5640b7e21e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.260+0000 7f079fe00700 1 -- 172.21.15.27:0/536005232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d980 msgr2=0x5640b7e21490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:56.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.260+0000 7f079fe00700 1 --2- 172.21.15.27:0/536005232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d980 0x5640b7e21490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.260+0000 7f079fe00700 1 -- 172.21.15.27:0/536005232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e219d0 msgr2=0x5640b7e21e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.260+0000 7f079fe00700 1 --2- 172.21.15.27:0/536005232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e219d0 0x5640b7e21e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.260+0000 7f079fe00700 1 -- 172.21.15.27:0/536005232 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640b7cbc8d0 con 0x5640b7e1d060 2024-05-10T11:25:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.260+0000 7f07a5200700 1 --2- 172.21.15.27:0/536005232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d980 0x5640b7e21490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.260+0000 7f07a5c00700 1 --2- 172.21.15.27:0/536005232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e219d0 0x5640b7e21e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:25:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.261+0000 7f079fe00700 1 --2- 172.21.15.27:0/536005232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 0x5640b7e1d440 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f078c005f50 tx=0x7f078c0057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6c97294cbdfd4b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:56.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.261+0000 7f079fe00700 1 --2- 172.21.15.27:0/2068377319 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f07880b2aa0 0x7f07880b4f60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f078c00ba00 tx=0x7f078c023000 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:56.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.261+0000 7f0792000700 1 -- 172.21.15.27:0/536005232 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c0052f0 con 0x5640b7e1d060 2024-05-10T11:25:56.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.261+0000 7f0792000700 1 -- 172.21.15.27:0/536005232 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f078c028910 con 0x5640b7e1d060 2024-05-10T11:25:56.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.261+0000 7f0792000700 1 -- 172.21.15.27:0/536005232 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c00f750 con 0x5640b7e1d060 2024-05-10T11:25:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.262+0000 7f07b267db80 1 -- 172.21.15.27:0/536005232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 msgr2=0x5640b7e1d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.262+0000 7f07b267db80 1 --2- 172.21.15.27:0/536005232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 0x5640b7e1d440 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f078c005f50 tx=0x7f078c0057a0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.262+0000 7f07b267db80 1 -- 172.21.15.27:0/536005232 shutdown_connections 2024-05-10T11:25:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.262+0000 7f07b267db80 1 --2- 172.21.15.27:0/536005232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e219d0 0x5640b7e21e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.262+0000 7f07b267db80 1 --2- 172.21.15.27:0/536005232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 0x5640b7e1d440 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.262+0000 7f07b267db80 1 --2- 172.21.15.27:0/536005232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d980 0x5640b7e21490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.262+0000 7f07b267db80 1 -- 172.21.15.27:0/536005232 >> 172.21.15.27:0/536005232 conn(0x5640b7d4bb60 msgr2=0x5640b7e177a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.262+0000 7f07b267db80 1 -- 172.21.15.27:0/536005232 shutdown_connections 2024-05-10T11:25:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.262+0000 7f07b267db80 1 -- 172.21.15.27:0/536005232 wait complete. 2024-05-10T11:25:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.263+0000 7f07b267db80 1 -- start start 2024-05-10T11:25:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.263+0000 7f07b267db80 1 Processor -- start 2024-05-10T11:25:56.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f07b267db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d060 0x5640b7e28e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f07b267db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d980 0x5640b7e26220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f07b267db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e219d0 0x5640b7e26760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f07b267db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5640b7e29b80 con 0x5640b7e1d060 2024-05-10T11:25:56.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f07b267db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5640b7cd5b20 con 0x5640b7e1d980 2024-05-10T11:25:56.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f07b267db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5640b7e29d00 con 0x5640b7e219d0 2024-05-10T11:25:56.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f07b267db80 10 client.0 _ll_register_callbacks cb 0x5640b7d16c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:25:56.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f079fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d980 0x5640b7e26220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f079fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d980 0x5640b7e26220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50210/0 (socket says 172.21.15.27:50210) 2024-05-10T11:25:56.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f07a5c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e219d0 0x5640b7e26760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f079fe00700 1 -- 172.21.15.27:0/431030788 learned_addr learned my addr 172.21.15.27:0/431030788 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:56.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f07a5200700 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d060 0x5640b7e28e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f079fe00700 1 -- 172.21.15.27:0/431030788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e219d0 msgr2=0x5640b7e26760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f079fe00700 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e219d0 0x5640b7e26760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f079fe00700 1 -- 172.21.15.27:0/431030788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d060 msgr2=0x5640b7e28e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.264+0000 7f079fe00700 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d060 0x5640b7e28e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.265+0000 7f079fe00700 1 -- 172.21.15.27:0/431030788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640b7d35420 con 0x5640b7e1d980 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.265+0000 7f07a5c00700 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e219d0 0x5640b7e26760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.265+0000 7f079fe00700 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d980 0x5640b7e26220 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f078c005850 tx=0x7f078c005a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f8b5a36ad7403f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.265+0000 7f0786a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.265+0000 7f0786a00700 1 -- 172.21.15.27:0/431030788 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c028cc0 con 0x5640b7e1d980 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.265+0000 7f07b267db80 1 -- 172.21.15.27:0/431030788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f078c028910 con 0x5640b7e1d980 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.265+0000 7f0786a00700 1 -- 172.21.15.27:0/431030788 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7f078c003a10 con 0x5640b7e1d980 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.266+0000 7f0786a00700 1 -- 172.21.15.27:0/431030788 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f078c003c10 con 0x5640b7e1d980 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.266+0000 7f07b267db80 10 client.27059 fetch_fsmap learned FSMap version 36 2024-05-10T11:25:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.266+0000 7f07b267db80 1 -- 172.21.15.27:0/431030788 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5640b7e29b80 con 0x5640b7e1d980 2024-05-10T11:25:56.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.266+0000 7f0786a00700 1 -- 172.21.15.27:0/431030788 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f078c028910 con 0x5640b7e1d980 2024-05-10T11:25:56.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.266+0000 7f0786a00700 1 -- 172.21.15.27:0/431030788 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f078c00fca0 con 0x5640b7e1d980 2024-05-10T11:25:56.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.266+0000 7f07b267db80 10 client.27059 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:25:56.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.266+0000 7f07b267db80 10 client.27059 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:25:56.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.266+0000 7f07b267db80 20 client.27059 populate_metadata read hostname 'smithi027' 2024-05-10T11:25:56.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.266+0000 7f07b267db80 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5640b7e36710 0x5640b7e38b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.266+0000 7f07b267db80 4 client.27059 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:25:56.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.266+0000 7f07b267db80 1 -- 172.21.15.27:0/431030788 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640b7dea0a0 con 0x5640b7e36710 2024-05-10T11:25:56.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.267+0000 7f07a5200700 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5640b7e36710 0x5640b7e38b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.267+0000 7f07a5200700 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5640b7e36710 0x5640b7e38b00 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c5026c68302fe8a2 server_cookie=8aedd2d45ba408fa in_seq=0 out_seq=0 2024-05-10T11:25:56.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.267+0000 7f0786a00700 10 client.27059 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:25:56.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.268+0000 7f0786a00700 1 -- 172.21.15.27:0/431030788 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5640b7dea0a0 con 0x5640b7e36710 2024-05-10T11:25:56.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.268+0000 7f0786a00700 10 client.27059 handle_command_reply: tid=0 2024-05-10T11:25:56.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.268+0000 7f07b267db80 1 client.27059 shutdown 2024-05-10T11:25:56.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.268+0000 7f07b267db80 1 -- 172.21.15.27:0/431030788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d980 msgr2=0x5640b7e26220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:56.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.268+0000 7f07b267db80 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d980 0x5640b7e26220 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f078c005850 tx=0x7f078c005a00 comp rx=0 tx=0).stop 2024-05-10T11:25:56.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.268+0000 7f07b267db80 1 -- 172.21.15.27:0/431030788 shutdown_connections 2024-05-10T11:25:56.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.268+0000 7f07b267db80 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5640b7e36710 0x5640b7e38b00 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.268+0000 7f07b267db80 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e219d0 0x5640b7e26760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.268+0000 7f07b267db80 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d060 0x5640b7e28e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.268+0000 7f07b267db80 1 --2- 172.21.15.27:0/431030788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d980 0x5640b7e26220 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.268+0000 7f07b267db80 1 -- 172.21.15.27:0/431030788 >> 172.21.15.27:0/431030788 conn(0x5640b7d4bb60 msgr2=0x5640b7e122b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.268+0000 7f0786a00700 0 client.27059 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:25:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.269+0000 7f07b267db80 1 -- 172.21.15.27:0/431030788 shutdown_connections 2024-05-10T11:25:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.269+0000 7f07b267db80 1 -- 172.21.15.27:0/431030788 wait complete. 2024-05-10T11:25:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.269+0000 7f07b267db80 20 client.27059 trim_cache size 0 max 16384 2024-05-10T11:25:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.288+0000 7f07b267db80 1 Processor -- start 2024-05-10T11:25:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.288+0000 7f07b267db80 1 -- start start 2024-05-10T11:25:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.289+0000 7f07b267db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d060 0x5640b7e1d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.289+0000 7f07b267db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d980 0x5640b7e32c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.289+0000 7f07b267db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e219d0 0x5640b7e3e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.289+0000 7f07b267db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f07a01b5c90 con 0x5640b7e1d980 2024-05-10T11:25:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.289+0000 7f07b267db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f07a01b77d0 con 0x5640b7e219d0 2024-05-10T11:25:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.289+0000 7f07b267db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f07a01b6360 con 0x5640b7e1d060 2024-05-10T11:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.289+0000 7f07a5200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d060 0x5640b7e1d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.289+0000 7f07a5200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d060 0x5640b7e1d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37192/0 (socket says 172.21.15.27:37192) 2024-05-10T11:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.289+0000 7f07a5200700 1 -- 172.21.15.27:0/1331117476 learned_addr learned my addr 172.21.15.27:0/1331117476 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.290+0000 7f07a5c00700 1 --2- 172.21.15.27:0/1331117476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e219d0 0x5640b7e3e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.290+0000 7f079fe00700 1 --2- 172.21.15.27:0/1331117476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d980 0x5640b7e32c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.290+0000 7f07a5200700 1 -- 172.21.15.27:0/1331117476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e219d0 msgr2=0x5640b7e3e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.290+0000 7f07a5200700 1 --2- 172.21.15.27:0/1331117476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e219d0 0x5640b7e3e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.290+0000 7f07a5200700 1 -- 172.21.15.27:0/1331117476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d980 msgr2=0x5640b7e32c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.290+0000 7f07a5200700 1 --2- 172.21.15.27:0/1331117476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d980 0x5640b7e32c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.290+0000 7f07a5200700 1 -- 172.21.15.27:0/1331117476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640b7e143f0 con 0x5640b7e1d060 2024-05-10T11:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.291+0000 7f07a5c00700 1 --2- 172.21.15.27:0/1331117476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e219d0 0x5640b7e3e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.291+0000 7f079fe00700 1 --2- 172.21.15.27:0/1331117476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d980 0x5640b7e32c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.291+0000 7f07a5200700 1 --2- 172.21.15.27:0/1331117476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d060 0x5640b7e1d440 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f07940058e0 tx=0x7f0794009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bea9bb41f80cb27 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.292+0000 7f0791600700 1 -- 172.21.15.27:0/1331117476 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794026070 con 0x5640b7e1d060 2024-05-10T11:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.292+0000 7f0791600700 1 -- 172.21.15.27:0/1331117476 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0794004500 con 0x5640b7e1d060 2024-05-10T11:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.292+0000 7f0791600700 1 -- 172.21.15.27:0/1331117476 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794020370 con 0x5640b7e1d060 2024-05-10T11:25:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.293+0000 7f07b267db80 1 -- 172.21.15.27:0/1331117476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d060 msgr2=0x5640b7e1d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.293+0000 7f07b267db80 1 --2- 172.21.15.27:0/1331117476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d060 0x5640b7e1d440 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f07940058e0 tx=0x7f0794009260 comp rx=0 tx=0).stop 2024-05-10T11:25:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.293+0000 7f07b267db80 1 -- 172.21.15.27:0/1331117476 shutdown_connections 2024-05-10T11:25:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.293+0000 7f07b267db80 1 --2- 172.21.15.27:0/1331117476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e1d060 0x5640b7e1d440 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.293+0000 7f07b267db80 1 --2- 172.21.15.27:0/1331117476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e219d0 0x5640b7e3e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.293+0000 7f07b267db80 1 --2- 172.21.15.27:0/1331117476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d980 0x5640b7e32c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.294+0000 7f07b267db80 1 -- 172.21.15.27:0/1331117476 >> 172.21.15.27:0/1331117476 conn(0x5640b7d4bb60 msgr2=0x5640b7e24b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.294+0000 7f07b267db80 1 -- 172.21.15.27:0/1331117476 shutdown_connections 2024-05-10T11:25:56.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.296+0000 7f07b267db80 1 -- 172.21.15.27:0/1331117476 wait complete. 2024-05-10T11:25:56.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.296+0000 7f07b267db80 1 -- start start 2024-05-10T11:25:56.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.297+0000 7f07b267db80 1 Processor -- start 2024-05-10T11:25:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.298+0000 7f07b267db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 0x5640b7e29f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.298+0000 7f07b267db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d980 0x5640b7e2a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.298+0000 7f07b267db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e219d0 0x5640b7e2a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.298+0000 7f07b267db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f07a01b7100 con 0x5640b7e1d980 2024-05-10T11:25:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.298+0000 7f07b267db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f07a01b6a30 con 0x5640b7e1d060 2024-05-10T11:25:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.298+0000 7f07b267db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f07a01b3c50 con 0x5640b7e219d0 2024-05-10T11:25:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.298+0000 7f07b267db80 10 client.0 _ll_register_callbacks cb 0x5640b7d16c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:25:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.298+0000 7f07a5200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 0x5640b7e29f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.298+0000 7f07a5200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 0x5640b7e29f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50236/0 (socket says 172.21.15.27:50236) 2024-05-10T11:25:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.298+0000 7f07a5200700 1 -- 172.21.15.27:0/1667233701 learned_addr learned my addr 172.21.15.27:0/1667233701 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:25:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.299+0000 7f079fe00700 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d980 0x5640b7e2a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.300+0000 7f07a5c00700 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e219d0 0x5640b7e2a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.300+0000 7f07a5200700 1 -- 172.21.15.27:0/1667233701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e219d0 msgr2=0x5640b7e2a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.300+0000 7f07a5200700 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e219d0 0x5640b7e2a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.300+0000 7f07a5200700 1 -- 172.21.15.27:0/1667233701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d980 msgr2=0x5640b7e2a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:25:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.300+0000 7f07a5200700 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d980 0x5640b7e2a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.300+0000 7f07a5200700 1 -- 172.21.15.27:0/1667233701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640b7e9ee90 con 0x5640b7e1d060 2024-05-10T11:25:56.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.301+0000 7f07a5200700 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 0x5640b7e29f50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0794009ca0 tx=0x7f0794003de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8058034aa322623a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:25:56.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.301+0000 7f0786a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:25:56.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.301+0000 7f0786a00700 1 -- 172.21.15.27:0/1667233701 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794026040 con 0x5640b7e1d060 2024-05-10T11:25:56.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.301+0000 7f0786a00700 1 -- 172.21.15.27:0/1667233701 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7f0794003880 con 0x5640b7e1d060 2024-05-10T11:25:56.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.302+0000 7f07b267db80 1 -- 172.21.15.27:0/1667233701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0794004500 con 0x5640b7e1d060 2024-05-10T11:25:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.302+0000 7f0786a00700 1 -- 172.21.15.27:0/1667233701 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0794034050 con 0x5640b7e1d060 2024-05-10T11:25:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.302+0000 7f07b267db80 10 client.27062 fetch_fsmap learned FSMap version 36 2024-05-10T11:25:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.302+0000 7f07b267db80 1 -- 172.21.15.27:0/1667233701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5640b7ea0100 con 0x5640b7e1d060 2024-05-10T11:25:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.303+0000 7f0786a00700 1 -- 172.21.15.27:0/1667233701 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0794004500 con 0x5640b7e1d060 2024-05-10T11:25:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.305+0000 7f0786a00700 1 -- 172.21.15.27:0/1667233701 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f07940047c0 con 0x5640b7e1d060 2024-05-10T11:25:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.305+0000 7f07b267db80 10 client.27062 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:25:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.305+0000 7f07b267db80 10 client.27062 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:25:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.305+0000 7f07b267db80 20 client.27062 populate_metadata read hostname 'smithi027' 2024-05-10T11:25:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.305+0000 7f07b267db80 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5640b7ea0630 0x5640b7ea2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:25:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.305+0000 7f07b267db80 4 client.27062 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:25:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.305+0000 7f07b267db80 1 -- 172.21.15.27:0/1667233701 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5640b7d7d400 con 0x5640b7ea0630 2024-05-10T11:25:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.305+0000 7f079fe00700 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5640b7ea0630 0x5640b7ea2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:25:56.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.311+0000 7f079fe00700 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5640b7ea0630 0x5640b7ea2a50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f74e46fc8afa9e2 server_cookie=c1bab5380ab76165 in_seq=0 out_seq=0 2024-05-10T11:25:56.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.312+0000 7f0786a00700 10 client.27062 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:25:56.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.314+0000 7f0786a00700 1 -- 172.21.15.27:0/1667233701 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5640b7d7d400 con 0x5640b7ea0630 2024-05-10T11:25:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.314+0000 7f0786a00700 10 client.27062 handle_command_reply: tid=0 2024-05-10T11:25:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.315+0000 7f07b267db80 1 client.27062 shutdown 2024-05-10T11:25:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.316+0000 7f07b267db80 1 -- 172.21.15.27:0/1667233701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 msgr2=0x5640b7e29f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.316+0000 7f07b267db80 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 0x5640b7e29f50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0794009ca0 tx=0x7f0794003de0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.317+0000 7f07b267db80 1 -- 172.21.15.27:0/1667233701 shutdown_connections 2024-05-10T11:25:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.317+0000 7f07b267db80 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x5640b7ea0630 0x5640b7ea2a50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.317+0000 7f07b267db80 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640b7e219d0 0x5640b7e2a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.317+0000 7f07b267db80 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5640b7e1d060 0x5640b7e29f50 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.317+0000 7f07b267db80 1 --2- 172.21.15.27:0/1667233701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5640b7e1d980 0x5640b7e2a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.317+0000 7f07b267db80 1 -- 172.21.15.27:0/1667233701 >> 172.21.15.27:0/1667233701 conn(0x5640b7d4bb60 msgr2=0x5640b7e11a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.317+0000 7f0786a00700 0 client.27062 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:25:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.319+0000 7f07b267db80 1 -- 172.21.15.27:0/1667233701 shutdown_connections 2024-05-10T11:25:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.319+0000 7f07b267db80 1 -- 172.21.15.27:0/1667233701 wait complete. 2024-05-10T11:25:56.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.320+0000 7f07b267db80 20 client.27062 trim_cache size 0 max 16384 2024-05-10T11:25:56.322 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:25:56.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.323+0000 7f07a7c00700 1 -- 172.21.15.27:0/2068377319 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f07880b2aa0 msgr2=0x7f07880b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:56.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.323+0000 7f07a7c00700 1 --2- 172.21.15.27:0/2068377319 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f07880b2aa0 0x7f07880b4f60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f078c00ba00 tx=0x7f078c023000 comp rx=0 tx=0).stop 2024-05-10T11:25:56.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.323+0000 7f07a7c00700 1 -- 172.21.15.27:0/2068377319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a010ff50 msgr2=0x7f07a0105100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:25:56.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.323+0000 7f07a7c00700 1 --2- 172.21.15.27:0/2068377319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a010ff50 0x7f07a0105100 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f079800efc0 tx=0x7f079800c3a0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.323+0000 7f07a7c00700 1 -- 172.21.15.27:0/2068377319 shutdown_connections 2024-05-10T11:25:56.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.323+0000 7f07a7c00700 1 --2- 172.21.15.27:0/2068377319 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f07880b2aa0 0x7f07880b4f60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.323+0000 7f07a7c00700 1 --2- 172.21.15.27:0/2068377319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a010ff50 0x7f07a0105100 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.323+0000 7f07a7c00700 1 --2- 172.21.15.27:0/2068377319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a006b1b0 0x7f07a0104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.323+0000 7f07a7c00700 1 --2- 172.21.15.27:0/2068377319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07a006bad0 0x7f07a0104bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:25:56.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.323+0000 7f07a7c00700 1 -- 172.21.15.27:0/2068377319 >> 172.21.15.27:0/2068377319 conn(0x7f07a0079350 msgr2=0x7f07a0111440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:25:56.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.324+0000 7f07a7c00700 1 -- 172.21.15.27:0/2068377319 shutdown_connections 2024-05-10T11:25:56.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:25:56.324+0000 7f07a7c00700 1 -- 172.21.15.27:0/2068377319 wait complete. 2024-05-10T11:25:56.739 INFO:tasks.workunit.client.0.smithi027.stdout: 1 767240 6.24 MB/sec execute 521 sec latency 75.073 ms 2024-05-10T11:25:56.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:56 smithi076 ceph-mon[34152]: pgmap v396: 129 pgs: 129 active+clean; 599 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-05-10T11:25:56.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:56 smithi189 ceph-mon[34842]: pgmap v396: 129 pgs: 129 active+clean; 599 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-05-10T11:25:57.043 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:56 smithi027 ceph-mon[30054]: pgmap v396: 129 pgs: 129 active+clean; 599 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-05-10T11:25:57.478 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:25:57.739 INFO:tasks.workunit.client.0.smithi027.stdout: 1 768065 6.23 MB/sec execute 522 sec latency 68.940 ms 2024-05-10T11:25:57.957 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:25:58.739 INFO:tasks.workunit.client.0.smithi027.stdout: 1 769539 6.23 MB/sec execute 523 sec latency 28.238 ms 2024-05-10T11:25:58.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:58 smithi027 ceph-mon[30054]: pgmap v397: 129 pgs: 129 active+clean; 599 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 128 op/s 2024-05-10T11:25:58.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:25:58 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:25:58.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:58 smithi076 ceph-mon[34152]: pgmap v397: 129 pgs: 129 active+clean; 599 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 128 op/s 2024-05-10T11:25:58.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:25:58 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:25:58.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:58 smithi189 ceph-mon[34842]: pgmap v397: 129 pgs: 129 active+clean; 599 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 128 op/s 2024-05-10T11:25:58.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:25:58 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:25:59.739 INFO:tasks.workunit.client.0.smithi027.stdout: 1 770639 6.24 MB/sec execute 524 sec latency 52.921 ms 2024-05-10T11:26:00.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.185+0000 7f8560e00700 1 -- 172.21.15.27:0/2322136669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1033d0 msgr2=0x7f855c103850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:00.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.185+0000 7f8560e00700 1 --2- 172.21.15.27:0/2322136669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1033d0 0x7f855c103850 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8548009990 tx=0x7f8548009ca0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.185+0000 7f8560e00700 1 -- 172.21.15.27:0/2322136669 shutdown_connections 2024-05-10T11:26:00.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.185+0000 7f8560e00700 1 --2- 172.21.15.27:0/2322136669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f855c103d90 0x7f855c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.185+0000 7f8560e00700 1 --2- 172.21.15.27:0/2322136669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f855c109460 0x7f855c109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.185+0000 7f8560e00700 1 --2- 172.21.15.27:0/2322136669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1033d0 0x7f855c103850 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.185+0000 7f8560e00700 1 -- 172.21.15.27:0/2322136669 >> 172.21.15.27:0/2322136669 conn(0x7f855c0fef20 msgr2=0x7f855c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:00.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.185+0000 7f8560e00700 1 -- 172.21.15.27:0/2322136669 shutdown_connections 2024-05-10T11:26:00.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.185+0000 7f8560e00700 1 -- 172.21.15.27:0/2322136669 wait complete. 2024-05-10T11:26:00.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.186+0000 7f8560e00700 1 Processor -- start 2024-05-10T11:26:00.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.187+0000 7f8560e00700 1 -- start start 2024-05-10T11:26:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.187+0000 7f8560e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1033d0 0x7f855c0757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.187+0000 7f8560e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f855c103d90 0x7f855c075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.187+0000 7f8560e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f855c109460 0x7f855c076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.187+0000 7f8560e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1135c0 con 0x7f855c109460 2024-05-10T11:26:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.187+0000 7f8560e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f855c113440 con 0x7f855c1033d0 2024-05-10T11:26:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.187+0000 7f8560e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f855c113740 con 0x7f855c103d90 2024-05-10T11:26:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.188+0000 7f855aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f855c103d90 0x7f855c075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.188+0000 7f855aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f855c103d90 0x7f855c075ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37212/0 (socket says 172.21.15.27:37212) 2024-05-10T11:26:00.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.188+0000 7f855aa00700 1 -- 172.21.15.27:0/868856862 learned_addr learned my addr 172.21.15.27:0/868856862 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:00.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.188+0000 7f855b400700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1033d0 0x7f855c0757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.188+0000 7f855be00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f855c109460 0x7f855c076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.188+0000 7f855aa00700 1 -- 172.21.15.27:0/868856862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1033d0 msgr2=0x7f855c0757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:00.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.188+0000 7f855aa00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1033d0 0x7f855c0757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.188+0000 7f855aa00700 1 -- 172.21.15.27:0/868856862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f855c109460 msgr2=0x7f855c076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:00.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.188+0000 7f855aa00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f855c109460 0x7f855c076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.188+0000 7f855aa00700 1 -- 172.21.15.27:0/868856862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855c1a8020 con 0x7f855c103d90 2024-05-10T11:26:00.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.188+0000 7f855b400700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1033d0 0x7f855c0757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:26:00.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.189+0000 7f855aa00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f855c103d90 0x7f855c075ce0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8548000c00 tx=0x7f85480057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:00.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.189+0000 7f8553e00700 1 -- 172.21.15.27:0/868856862 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548027070 con 0x7f855c103d90 2024-05-10T11:26:00.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.189+0000 7f8553e00700 1 -- 172.21.15.27:0/868856862 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8548022470 con 0x7f855c103d90 2024-05-10T11:26:00.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.189+0000 7f8553e00700 1 -- 172.21.15.27:0/868856862 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548017650 con 0x7f855c103d90 2024-05-10T11:26:00.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.189+0000 7f8560e00700 1 -- 172.21.15.27:0/868856862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f855c1a82b0 con 0x7f855c103d90 2024-05-10T11:26:00.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.189+0000 7f8560e00700 1 -- 172.21.15.27:0/868856862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f855c1a8760 con 0x7f855c103d90 2024-05-10T11:26:00.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.193+0000 7f8553e00700 1 -- 172.21.15.27:0/868856862 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f854802a070 con 0x7f855c103d90 2024-05-10T11:26:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.194+0000 7f856ccaab80 1 Processor -- start 2024-05-10T11:26:00.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.194+0000 7f856ccaab80 1 -- start start 2024-05-10T11:26:00.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f8553e00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f85400b2a70 0x7f85400b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f856ccaab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe3da0 0x560bb2fe4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f856ccaab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fe46c0 0x560bb2fde430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f856ccaab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fdea50 0x560bb2fdeed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f856ccaab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560bb2f30a20 con 0x560bb2fe3da0 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f856ccaab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x560bb2ece760 con 0x560bb2fe46c0 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f856ccaab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560bb2f2c9e0 con 0x560bb2fdea50 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f8553e00700 1 -- 172.21.15.27:0/868856862 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7f85480e3800 con 0x7f855c103d90 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f855b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe3da0 0x560bb2fe4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f855be00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f85400b2a70 0x7f85400b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f855b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe3da0 0x560bb2fe4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55974/0 (socket says 172.21.15.27:55974) 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f855b400700 1 -- 172.21.15.27:0/3606549906 learned_addr learned my addr 172.21.15.27:0/3606549906 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f855be00700 1 --2- 172.21.15.27:0/3606549906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fdea50 0x560bb2fdeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.195+0000 7f855aa00700 1 --2- 172.21.15.27:0/3606549906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fe46c0 0x560bb2fde430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.196+0000 7f855aa00700 1 -- 172.21.15.27:0/3606549906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fdea50 msgr2=0x560bb2fdeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.196+0000 7f855aa00700 1 --2- 172.21.15.27:0/3606549906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fdea50 0x560bb2fdeed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.196+0000 7f855aa00700 1 -- 172.21.15.27:0/3606549906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe3da0 msgr2=0x560bb2fe4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.196+0000 7f855aa00700 1 --2- 172.21.15.27:0/3606549906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe3da0 0x560bb2fe4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.196+0000 7f855aa00700 1 -- 172.21.15.27:0/3606549906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bb2e7d8d0 con 0x560bb2fe46c0 2024-05-10T11:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.196+0000 7f855be00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f85400b2a70 0x7f85400b4f30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8554015da0 tx=0x7f8554015420 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.196+0000 7f855aa00700 1 --2- 172.21.15.27:0/3606549906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fe46c0 0x560bb2fde430 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f85480e0ac0 tx=0x7f85480aa4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cccd125223c0b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.196+0000 7f8550c00700 1 -- 172.21.15.27:0/3606549906 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85480a9e50 con 0x560bb2fe46c0 2024-05-10T11:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.196+0000 7f8550c00700 1 -- 172.21.15.27:0/3606549906 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85480aa1f0 con 0x560bb2fe46c0 2024-05-10T11:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.196+0000 7f8550c00700 1 -- 172.21.15.27:0/3606549906 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548067960 con 0x560bb2fe46c0 2024-05-10T11:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.197+0000 7f856ccaab80 1 -- 172.21.15.27:0/3606549906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fe46c0 msgr2=0x560bb2fde430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.197+0000 7f856ccaab80 1 --2- 172.21.15.27:0/3606549906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fe46c0 0x560bb2fde430 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f85480e0ac0 tx=0x7f85480aa4b0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.197+0000 7f856ccaab80 1 -- 172.21.15.27:0/3606549906 shutdown_connections 2024-05-10T11:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.197+0000 7f856ccaab80 1 --2- 172.21.15.27:0/3606549906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fdea50 0x560bb2fdeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.197+0000 7f856ccaab80 1 --2- 172.21.15.27:0/3606549906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe3da0 0x560bb2fe4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.197+0000 7f856ccaab80 1 --2- 172.21.15.27:0/3606549906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fe46c0 0x560bb2fde430 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.197+0000 7f856ccaab80 1 -- 172.21.15.27:0/3606549906 >> 172.21.15.27:0/3606549906 conn(0x560bb2f566d0 msgr2=0x560bb2fd8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.198+0000 7f856ccaab80 1 -- 172.21.15.27:0/3606549906 shutdown_connections 2024-05-10T11:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.198+0000 7f856ccaab80 1 -- 172.21.15.27:0/3606549906 wait complete. 2024-05-10T11:26:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.198+0000 7f856ccaab80 1 -- start start 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.198+0000 7f856ccaab80 1 Processor -- start 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.199+0000 7f856ccaab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fdea50 0x560bb2fea360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.199+0000 7f856ccaab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fe3da0 0x560bb2fe7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.199+0000 7f856ccaab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2fe7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.199+0000 7f856ccaab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560bb2feaf10 con 0x560bb2fe46c0 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.199+0000 7f856ccaab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x560bb2e96b20 con 0x560bb2fe3da0 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.199+0000 7f856ccaab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560bb2feb090 con 0x560bb2fdea50 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.199+0000 7f856ccaab80 10 client.0 _ll_register_callbacks cb 0x560bb2ed7c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.199+0000 7f855b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fdea50 0x560bb2fea360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.199+0000 7f855be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2fe7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.199+0000 7f855b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fdea50 0x560bb2fea360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37236/0 (socket says 172.21.15.27:37236) 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.199+0000 7f855b400700 1 -- 172.21.15.27:0/1006699181 learned_addr learned my addr 172.21.15.27:0/1006699181 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.200+0000 7f855aa00700 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fe3da0 0x560bb2fe7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.200+0000 7f855b400700 1 -- 172.21.15.27:0/1006699181 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fe3da0 msgr2=0x560bb2fe7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.200+0000 7f855b400700 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fe3da0 0x560bb2fe7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.200+0000 7f855b400700 1 -- 172.21.15.27:0/1006699181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 msgr2=0x560bb2fe7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.200+0000 7f855b400700 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2fe7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.200+0000 7f855b400700 1 -- 172.21.15.27:0/1006699181 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bb2ef6420 con 0x560bb2fdea50 2024-05-10T11:26:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.200+0000 7f855be00700 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2fe7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:26:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.200+0000 7f855b400700 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fdea50 0x560bb2fea360 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f854c0118a0 tx=0x7f854c011bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c4af3bee8668711 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.200+0000 7f8545600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:26:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.201+0000 7f8545600700 1 -- 172.21.15.27:0/1006699181 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c004830 con 0x560bb2fdea50 2024-05-10T11:26:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.201+0000 7f856ccaab80 1 -- 172.21.15.27:0/1006699181 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85480aa1f0 con 0x560bb2fdea50 2024-05-10T11:26:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.201+0000 7f8545600700 1 -- 172.21.15.27:0/1006699181 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7f854c0049d0 con 0x560bb2fdea50 2024-05-10T11:26:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.201+0000 7f8545600700 1 -- 172.21.15.27:0/1006699181 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f854c025050 con 0x560bb2fdea50 2024-05-10T11:26:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.201+0000 7f856ccaab80 10 client.27058 fetch_fsmap learned FSMap version 36 2024-05-10T11:26:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.201+0000 7f856ccaab80 1 -- 172.21.15.27:0/1006699181 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560bb2ff5680 con 0x560bb2fdea50 2024-05-10T11:26:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.201+0000 7f8545600700 1 -- 172.21.15.27:0/1006699181 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85480aa1f0 con 0x560bb2fdea50 2024-05-10T11:26:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.202+0000 7f8545600700 1 -- 172.21.15.27:0/1006699181 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f854c01aa60 con 0x560bb2fdea50 2024-05-10T11:26:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.202+0000 7f856ccaab80 10 client.27058 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:26:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.202+0000 7f856ccaab80 10 client.27058 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:26:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.202+0000 7f856ccaab80 20 client.27058 populate_metadata read hostname 'smithi027' 2024-05-10T11:26:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.202+0000 7f856ccaab80 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x560bb2ff7cc0 0x560bb2ffa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.202+0000 7f856ccaab80 4 client.27058 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:26:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.202+0000 7f856ccaab80 1 -- 172.21.15.27:0/1006699181 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bb2fab140 con 0x560bb2ff7cc0 2024-05-10T11:26:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.203+0000 7f855aa00700 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x560bb2ff7cc0 0x560bb2ffa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.203+0000 7f855aa00700 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x560bb2ff7cc0 0x560bb2ffa0b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=662e712b4d57687c server_cookie=bce669768294eeb5 in_seq=0 out_seq=0 2024-05-10T11:26:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.203+0000 7f8545600700 10 client.27058 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:26:00.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.204+0000 7f8545600700 1 -- 172.21.15.27:0/1006699181 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560bb2fab140 con 0x560bb2ff7cc0 2024-05-10T11:26:00.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.204+0000 7f8545600700 10 client.27058 handle_command_reply: tid=0 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.204+0000 7f856ccaab80 1 client.27058 shutdown 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.205+0000 7f856ccaab80 1 -- 172.21.15.27:0/1006699181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fdea50 msgr2=0x560bb2fea360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.205+0000 7f856ccaab80 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fdea50 0x560bb2fea360 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f854c0118a0 tx=0x7f854c011bb0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.205+0000 7f856ccaab80 1 -- 172.21.15.27:0/1006699181 shutdown_connections 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.205+0000 7f856ccaab80 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x560bb2ff7cc0 0x560bb2ffa0b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.205+0000 7f856ccaab80 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fdea50 0x560bb2fea360 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.205+0000 7f856ccaab80 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2fe7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.205+0000 7f856ccaab80 1 --2- 172.21.15.27:0/1006699181 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fe3da0 0x560bb2fe7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.205+0000 7f856ccaab80 1 -- 172.21.15.27:0/1006699181 >> 172.21.15.27:0/1006699181 conn(0x560bb2f566d0 msgr2=0x560bb2fd3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.205+0000 7f8545600700 0 client.27058 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.205+0000 7f856ccaab80 1 -- 172.21.15.27:0/1006699181 shutdown_connections 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.205+0000 7f856ccaab80 1 -- 172.21.15.27:0/1006699181 wait complete. 2024-05-10T11:26:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.205+0000 7f856ccaab80 20 client.27058 trim_cache size 0 max 16384 2024-05-10T11:26:00.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f856ccaab80 1 Processor -- start 2024-05-10T11:26:00.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f856ccaab80 1 -- start start 2024-05-10T11:26:00.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f856ccaab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 0x560bb2fde430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f856ccaab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fe3da0 0x560bb2fe5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f856ccaab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2fd3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f856ccaab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1b1620 con 0x560bb2fe46c0 2024-05-10T11:26:00.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f856ccaab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1b3160 con 0x560bb2fdea50 2024-05-10T11:26:00.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f856ccaab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1b1cf0 con 0x560bb2fe3da0 2024-05-10T11:26:00.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f855be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2fd3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f855be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2fd3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55990/0 (socket says 172.21.15.27:55990) 2024-05-10T11:26:00.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f855aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fe3da0 0x560bb2fe5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f855be00700 1 -- 172.21.15.27:0/456411518 learned_addr learned my addr 172.21.15.27:0/456411518 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:00.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.223+0000 7f855b400700 1 --2- 172.21.15.27:0/456411518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 0x560bb2fde430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.224+0000 7f855aa00700 1 -- 172.21.15.27:0/456411518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 msgr2=0x560bb2fde430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:00.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.224+0000 7f855aa00700 1 --2- 172.21.15.27:0/456411518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 0x560bb2fde430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.224+0000 7f855aa00700 1 -- 172.21.15.27:0/456411518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 msgr2=0x560bb2fd3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:00.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.224+0000 7f855aa00700 1 --2- 172.21.15.27:0/456411518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2fd3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.224+0000 7f855aa00700 1 -- 172.21.15.27:0/456411518 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bb2fd3da0 con 0x560bb2fe3da0 2024-05-10T11:26:00.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.224+0000 7f855b400700 1 --2- 172.21.15.27:0/456411518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 0x560bb2fde430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:26:00.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.224+0000 7f855aa00700 1 --2- 172.21.15.27:0/456411518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fe3da0 0x560bb2fe5330 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f854800b680 tx=0x7f8548005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ce4dcd6b72a9ce9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:00.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.224+0000 7f8547e00700 1 -- 172.21.15.27:0/456411518 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85480aade0 con 0x560bb2fe3da0 2024-05-10T11:26:00.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.225+0000 7f8547e00700 1 -- 172.21.15.27:0/456411518 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85480b34b0 con 0x560bb2fe3da0 2024-05-10T11:26:00.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.225+0000 7f8547e00700 1 -- 172.21.15.27:0/456411518 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85480e0d40 con 0x560bb2fe3da0 2024-05-10T11:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.225+0000 7f856ccaab80 1 -- 172.21.15.27:0/456411518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fe3da0 msgr2=0x560bb2fe5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.225+0000 7f856ccaab80 1 --2- 172.21.15.27:0/456411518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fe3da0 0x560bb2fe5330 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f854800b680 tx=0x7f8548005040 comp rx=0 tx=0).stop 2024-05-10T11:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.225+0000 7f856ccaab80 1 -- 172.21.15.27:0/456411518 shutdown_connections 2024-05-10T11:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.225+0000 7f856ccaab80 1 --2- 172.21.15.27:0/456411518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2fd3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.225+0000 7f856ccaab80 1 --2- 172.21.15.27:0/456411518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 0x560bb2fde430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.225+0000 7f856ccaab80 1 --2- 172.21.15.27:0/456411518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fe3da0 0x560bb2fe5330 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.225+0000 7f856ccaab80 1 -- 172.21.15.27:0/456411518 >> 172.21.15.27:0/456411518 conn(0x560bb2f566d0 msgr2=0x560bb2fe5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.226+0000 7f856ccaab80 1 -- 172.21.15.27:0/456411518 shutdown_connections 2024-05-10T11:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.226+0000 7f856ccaab80 1 -- 172.21.15.27:0/456411518 wait complete. 2024-05-10T11:26:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.226+0000 7f856ccaab80 1 -- start start 2024-05-10T11:26:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.226+0000 7f856ccaab80 1 Processor -- start 2024-05-10T11:26:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.227+0000 7f856ccaab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 0x560bb2ff02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.227+0000 7f856ccaab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fe3da0 0x560bb2ff0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.227+0000 7f856ccaab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2ffe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.227+0000 7f856ccaab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1b2a90 con 0x560bb2fe46c0 2024-05-10T11:26:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.227+0000 7f856ccaab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1b23c0 con 0x560bb2fdea50 2024-05-10T11:26:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.227+0000 7f856ccaab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1af5e0 con 0x560bb2fe3da0 2024-05-10T11:26:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.227+0000 7f856ccaab80 10 client.0 _ll_register_callbacks cb 0x560bb2ed7c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:26:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.227+0000 7f855b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 0x560bb2ff02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.227+0000 7f855b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 0x560bb2ff02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50274/0 (socket says 172.21.15.27:50274) 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.227+0000 7f855b400700 1 -- 172.21.15.27:0/897462054 learned_addr learned my addr 172.21.15.27:0/897462054 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.227+0000 7f855aa00700 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fe3da0 0x560bb2ff0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.227+0000 7f855be00700 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2ffe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.228+0000 7f855b400700 1 -- 172.21.15.27:0/897462054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fe3da0 msgr2=0x560bb2ff0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.228+0000 7f855b400700 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fe3da0 0x560bb2ff0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.228+0000 7f855b400700 1 -- 172.21.15.27:0/897462054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 msgr2=0x560bb2ffe920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.228+0000 7f855b400700 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2ffe920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.228+0000 7f855b400700 1 -- 172.21.15.27:0/897462054 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bb2ffeee0 con 0x560bb2fdea50 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.228+0000 7f855be00700 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2ffe920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.228+0000 7f855b400700 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 0x560bb2ff02e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f854c01afd0 tx=0x7f854c010a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e46488b7f7096999 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.228+0000 7f8545600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:26:00.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.228+0000 7f8545600700 1 -- 172.21.15.27:0/897462054 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c005030 con 0x560bb2fdea50 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.228+0000 7f856ccaab80 1 -- 172.21.15.27:0/897462054 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85480b34b0 con 0x560bb2fdea50 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.228+0000 7f8545600700 1 -- 172.21.15.27:0/897462054 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9591+0+0 (secure 0 0 0) 0x7f854c0051d0 con 0x560bb2fdea50 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.229+0000 7f8545600700 1 -- 172.21.15.27:0/897462054 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f854c03b050 con 0x560bb2fdea50 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.229+0000 7f856ccaab80 10 client.27080 fetch_fsmap learned FSMap version 36 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.229+0000 7f856ccaab80 1 -- 172.21.15.27:0/897462054 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560bb3061d20 con 0x560bb2fdea50 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.229+0000 7f8545600700 1 -- 172.21.15.27:0/897462054 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85480b34b0 con 0x560bb2fdea50 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.229+0000 7f8545600700 1 -- 172.21.15.27:0/897462054 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f854c004c90 con 0x560bb2fdea50 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.229+0000 7f856ccaab80 10 client.27080 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.229+0000 7f856ccaab80 10 client.27080 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.229+0000 7f856ccaab80 20 client.27080 populate_metadata read hostname 'smithi027' 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.229+0000 7f856ccaab80 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x560bb3062460 0x560bb3064880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.229+0000 7f856ccaab80 4 client.27080 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.229+0000 7f856ccaab80 1 -- 172.21.15.27:0/897462054 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560bb2f3e400 con 0x560bb3062460 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.230+0000 7f855aa00700 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x560bb3062460 0x560bb3064880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.230+0000 7f855aa00700 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x560bb3062460 0x560bb3064880 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1141cbde2857db86 server_cookie=68327318ef7cc35a in_seq=0 out_seq=0 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.230+0000 7f8545600700 10 client.27080 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.230+0000 7f8545600700 1 -- 172.21.15.27:0/897462054 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560bb2f3e400 con 0x560bb3062460 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.230+0000 7f8545600700 10 client.27080 handle_command_reply: tid=0 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.230+0000 7f856ccaab80 1 client.27080 shutdown 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.230+0000 7f856ccaab80 1 -- 172.21.15.27:0/897462054 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 msgr2=0x560bb2ff02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.230+0000 7f856ccaab80 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 0x560bb2ff02e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f854c01afd0 tx=0x7f854c010a70 comp rx=0 tx=0).stop 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.231+0000 7f856ccaab80 1 -- 172.21.15.27:0/897462054 shutdown_connections 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.231+0000 7f856ccaab80 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bb2fe46c0 0x560bb2ffe920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.231+0000 7f856ccaab80 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560bb2fdea50 0x560bb2ff02e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.231+0000 7f856ccaab80 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x560bb3062460 0x560bb3064880 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.231+0000 7f856ccaab80 1 --2- 172.21.15.27:0/897462054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560bb2fe3da0 0x560bb2ff0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.231+0000 7f8545600700 0 client.27080 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.231+0000 7f856ccaab80 1 -- 172.21.15.27:0/897462054 >> 172.21.15.27:0/897462054 conn(0x560bb2f566d0 msgr2=0x560bb2fe6770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.231+0000 7f856ccaab80 1 -- 172.21.15.27:0/897462054 shutdown_connections 2024-05-10T11:26:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.231+0000 7f856ccaab80 1 -- 172.21.15.27:0/897462054 wait complete. 2024-05-10T11:26:00.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.231+0000 7f856ccaab80 20 client.27080 trim_cache size 0 max 16384 2024-05-10T11:26:00.237 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:26:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.232+0000 7f8560e00700 1 -- 172.21.15.27:0/868856862 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f85400b2a70 msgr2=0x7f85400b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.232+0000 7f8560e00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f85400b2a70 0x7f85400b4f30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8554015da0 tx=0x7f8554015420 comp rx=0 tx=0).stop 2024-05-10T11:26:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.232+0000 7f8560e00700 1 -- 172.21.15.27:0/868856862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f855c103d90 msgr2=0x7f855c075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.232+0000 7f8560e00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f855c103d90 0x7f855c075ce0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8548000c00 tx=0x7f85480057c0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.232+0000 7f8560e00700 1 -- 172.21.15.27:0/868856862 shutdown_connections 2024-05-10T11:26:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.232+0000 7f8560e00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f85400b2a70 0x7f85400b4f30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.232+0000 7f8560e00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f855c109460 0x7f855c076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.232+0000 7f8560e00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1033d0 0x7f855c0757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.232+0000 7f8560e00700 1 --2- 172.21.15.27:0/868856862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f855c103d90 0x7f855c075ce0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.232+0000 7f8560e00700 1 -- 172.21.15.27:0/868856862 >> 172.21.15.27:0/868856862 conn(0x7f855c0fef20 msgr2=0x7f855c0ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.232+0000 7f8560e00700 1 -- 172.21.15.27:0/868856862 shutdown_connections 2024-05-10T11:26:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:00.232+0000 7f8560e00700 1 -- 172.21.15.27:0/868856862 wait complete. 2024-05-10T11:26:00.740 INFO:tasks.workunit.client.0.smithi027.stdout: 1 771102 6.23 MB/sec execute 525 sec latency 36.718 ms 2024-05-10T11:26:00.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:00 smithi076 ceph-mon[34152]: pgmap v398: 129 pgs: 129 active+clean; 604 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 188 op/s 2024-05-10T11:26:00.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:00 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:26:00.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:00 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:26:00.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:00 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:26:00.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:00 smithi189 ceph-mon[34842]: pgmap v398: 129 pgs: 129 active+clean; 604 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 188 op/s 2024-05-10T11:26:00.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:00 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:26:00.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:00 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:26:00.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:00 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:26:01.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:00 smithi027 ceph-mon[30054]: pgmap v398: 129 pgs: 129 active+clean; 604 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 188 op/s 2024-05-10T11:26:01.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:00 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:26:01.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:00 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:26:01.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:00 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:26:01.246 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:26:01.740 INFO:tasks.workunit.client.0.smithi027.stdout: 1 771517 6.22 MB/sec execute 526 sec latency 164.492 ms 2024-05-10T11:26:01.911 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:26:02.740 INFO:tasks.workunit.client.0.smithi027.stdout: 1 771952 6.22 MB/sec execute 527 sec latency 219.236 ms 2024-05-10T11:26:02.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:02 smithi027 ceph-mon[30054]: pgmap v399: 129 pgs: 129 active+clean; 604 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 122 op/s 2024-05-10T11:26:02.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:02 smithi027 ceph-mon[30054]: osdmap e104: 12 total, 12 up, 12 in 2024-05-10T11:26:02.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:02 smithi076 ceph-mon[34152]: pgmap v399: 129 pgs: 129 active+clean; 604 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 122 op/s 2024-05-10T11:26:02.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:02 smithi076 ceph-mon[34152]: osdmap e104: 12 total, 12 up, 12 in 2024-05-10T11:26:02.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:02 smithi189 ceph-mon[34842]: pgmap v399: 129 pgs: 129 active+clean; 604 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 122 op/s 2024-05-10T11:26:02.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:02 smithi189 ceph-mon[34842]: osdmap e104: 12 total, 12 up, 12 in 2024-05-10T11:26:03.740 INFO:tasks.workunit.client.0.smithi027.stdout: 1 773419 6.22 MB/sec execute 528 sec latency 30.799 ms 2024-05-10T11:26:04.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.299+0000 7ff1dc800700 1 -- 172.21.15.27:0/112403855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1d4104100 msgr2=0x7ff1d4108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:04.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.299+0000 7ff1dc800700 1 --2- 172.21.15.27:0/112403855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1d4104100 0x7ff1d4108120 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4009990 tx=0x7ff1c4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.299+0000 7ff1dc800700 1 -- 172.21.15.27:0/112403855 shutdown_connections 2024-05-10T11:26:04.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.299+0000 7ff1dc800700 1 --2- 172.21.15.27:0/112403855 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d4108850 0x7ff1d41103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.299+0000 7ff1dc800700 1 --2- 172.21.15.27:0/112403855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d4103750 0x7ff1d4103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.299+0000 7ff1dc800700 1 --2- 172.21.15.27:0/112403855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1d4104100 0x7ff1d4108120 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.299+0000 7ff1dc800700 1 -- 172.21.15.27:0/112403855 >> 172.21.15.27:0/112403855 conn(0x7ff1d407a600 msgr2=0x7ff1d407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.299+0000 7ff1dc800700 1 -- 172.21.15.27:0/112403855 shutdown_connections 2024-05-10T11:26:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.299+0000 7ff1dc800700 1 -- 172.21.15.27:0/112403855 wait complete. 2024-05-10T11:26:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.301+0000 7ff1dc800700 1 Processor -- start 2024-05-10T11:26:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.301+0000 7ff1dc800700 1 -- start start 2024-05-10T11:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.301+0000 7ff1dc800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d4103750 0x7ff1d419bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.301+0000 7ff1dc800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d4104100 0x7ff1d419c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.301+0000 7ff1dc800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1d4108850 0x7ff1d41a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.301+0000 7ff1dc800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d4113680 con 0x7ff1d4103750 2024-05-10T11:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.301+0000 7ff1dc800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d4113500 con 0x7ff1d4104100 2024-05-10T11:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.301+0000 7ff1dc800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d4113800 con 0x7ff1d4108850 2024-05-10T11:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.302+0000 7ff1d9400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d4104100 0x7ff1d419c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.302+0000 7ff1d9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d4103750 0x7ff1d419bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.302+0000 7ff1d9400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d4104100 0x7ff1d419c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48576/0 (socket says 172.21.15.27:48576) 2024-05-10T11:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.302+0000 7ff1d9400700 1 -- 172.21.15.27:0/2829596350 learned_addr learned my addr 172.21.15.27:0/2829596350 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.302+0000 7ff1da800700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1d4108850 0x7ff1d41a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.302+0000 7ff1d9400700 1 -- 172.21.15.27:0/2829596350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1d4108850 msgr2=0x7ff1d41a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.302+0000 7ff1d9400700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1d4108850 0x7ff1d41a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.302+0000 7ff1d9400700 1 -- 172.21.15.27:0/2829596350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d4103750 msgr2=0x7ff1d419bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.302+0000 7ff1d9400700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d4103750 0x7ff1d419bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.302+0000 7ff1d9400700 1 -- 172.21.15.27:0/2829596350 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d41a0d60 con 0x7ff1d4104100 2024-05-10T11:26:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.303+0000 7ff1d9400700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d4104100 0x7ff1d419c290 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4000c00 tx=0x7ff1c40057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.303+0000 7ff1cea00700 1 -- 172.21.15.27:0/2829596350 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4027070 con 0x7ff1d4104100 2024-05-10T11:26:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.303+0000 7ff1cea00700 1 -- 172.21.15.27:0/2829596350 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1c400b610 con 0x7ff1d4104100 2024-05-10T11:26:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.303+0000 7ff1dc800700 1 -- 172.21.15.27:0/2829596350 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d41a0ff0 con 0x7ff1d4104100 2024-05-10T11:26:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.303+0000 7ff1cea00700 1 -- 172.21.15.27:0/2829596350 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40173f0 con 0x7ff1d4104100 2024-05-10T11:26:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.303+0000 7ff1dc800700 1 -- 172.21.15.27:0/2829596350 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d41a88d0 con 0x7ff1d4104100 2024-05-10T11:26:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.307+0000 7ff1cea00700 1 -- 172.21.15.27:0/2829596350 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff1c402a070 con 0x7ff1d4104100 2024-05-10T11:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.308+0000 7ff1e7262b80 1 Processor -- start 2024-05-10T11:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.308+0000 7ff1e7262b80 1 -- start start 2024-05-10T11:26:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.308+0000 7ff1e7262b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 0x55e7ecf94440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.308+0000 7ff1e7262b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf98490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.308+0000 7ff1e7262b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf989d0 0x55e7ecf98e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.308+0000 7ff1e7262b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e7ecee6a20 con 0x55e7ecf94060 2024-05-10T11:26:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.308+0000 7ff1e7262b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55e7ece84760 con 0x55e7ecf94980 2024-05-10T11:26:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.308+0000 7ff1e7262b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e7ecee29e0 con 0x55e7ecf989d0 2024-05-10T11:26:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1da800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 0x55e7ecf94440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1d9e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf98490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1d9e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf98490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48580/0 (socket says 172.21.15.27:48580) 2024-05-10T11:26:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1d9e00700 1 -- 172.21.15.27:0/3174486967 learned_addr learned my addr 172.21.15.27:0/3174486967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1da800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 0x55e7ecf94440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34020/0 (socket says 172.21.15.27:34020) 2024-05-10T11:26:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1d9400700 1 --2- 172.21.15.27:0/3174486967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf989d0 0x55e7ecf98e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1cea00700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff1bc0b2a70 0x7ff1bc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1cea00700 1 -- 172.21.15.27:0/2829596350 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9679+0+0 (secure 0 0 0) 0x7ff1c40e38f0 con 0x7ff1d4104100 2024-05-10T11:26:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1da800700 1 -- 172.21.15.27:0/3174486967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf989d0 msgr2=0x55e7ecf98e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1da800700 1 --2- 172.21.15.27:0/3174486967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf989d0 0x55e7ecf98e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1da800700 1 -- 172.21.15.27:0/3174486967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 msgr2=0x55e7ecf98490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1da800700 1 --2- 172.21.15.27:0/3174486967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf98490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.309+0000 7ff1da800700 1 -- 172.21.15.27:0/3174486967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7ece338d0 con 0x55e7ecf94060 2024-05-10T11:26:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.310+0000 7ff1da800700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff1bc0b2a70 0x7ff1bc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.310+0000 7ff1d9e00700 1 --2- 172.21.15.27:0/3174486967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf98490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:26:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.310+0000 7ff1da800700 1 --2- 172.21.15.27:0/3174486967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 0x55e7ecf94440 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0011650 tx=0x7ff1d0011960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64803ad52b2dafe7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:04.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.310+0000 7ff1c3400700 1 -- 172.21.15.27:0/3174486967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d001b070 con 0x55e7ecf94060 2024-05-10T11:26:04.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.310+0000 7ff1c3400700 1 -- 172.21.15.27:0/3174486967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d0017b00 con 0x55e7ecf94060 2024-05-10T11:26:04.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.310+0000 7ff1c3400700 1 -- 172.21.15.27:0/3174486967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00207d0 con 0x55e7ecf94060 2024-05-10T11:26:04.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.311+0000 7ff1da800700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff1bc0b2a70 0x7ff1bc0b4f30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0022000 tx=0x7ff1d001f000 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:04.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.311+0000 7ff1e7262b80 1 -- 172.21.15.27:0/3174486967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 msgr2=0x55e7ecf94440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:04.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.311+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/3174486967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 0x55e7ecf94440 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0011650 tx=0x7ff1d0011960 comp rx=0 tx=0).stop 2024-05-10T11:26:04.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.311+0000 7ff1e7262b80 1 -- 172.21.15.27:0/3174486967 shutdown_connections 2024-05-10T11:26:04.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.311+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/3174486967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf989d0 0x55e7ecf98e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.312+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/3174486967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 0x55e7ecf94440 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.312+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/3174486967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf98490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.312+0000 7ff1e7262b80 1 -- 172.21.15.27:0/3174486967 >> 172.21.15.27:0/3174486967 conn(0x55e7ecec2b60 msgr2=0x55e7ecf8e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:04.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.312+0000 7ff1e7262b80 1 -- 172.21.15.27:0/3174486967 shutdown_connections 2024-05-10T11:26:04.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.312+0000 7ff1e7262b80 1 -- 172.21.15.27:0/3174486967 wait complete. 2024-05-10T11:26:04.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.312+0000 7ff1e7262b80 1 -- start start 2024-05-10T11:26:04.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.313+0000 7ff1e7262b80 1 Processor -- start 2024-05-10T11:26:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.313+0000 7ff1e7262b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf94060 0x55e7ecf9fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.313+0000 7ff1e7262b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf9d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.313+0000 7ff1e7262b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf989d0 0x55e7ecf9d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.313+0000 7ff1e7262b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e7ecfa0be0 con 0x55e7ecf989d0 2024-05-10T11:26:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.313+0000 7ff1e7262b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55e7ece4cb20 con 0x55e7ecf94980 2024-05-10T11:26:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.313+0000 7ff1e7262b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e7ecfa0d60 con 0x55e7ecf94060 2024-05-10T11:26:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.313+0000 7ff1e7262b80 10 client.0 _ll_register_callbacks cb 0x55e7ece8dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:26:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.313+0000 7ff1d9400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf9d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1d9400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf9d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48582/0 (socket says 172.21.15.27:48582) 2024-05-10T11:26:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1d9400700 1 -- 172.21.15.27:0/1168212907 learned_addr learned my addr 172.21.15.27:0/1168212907 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1d9e00700 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf94060 0x55e7ecf9fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1da800700 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf989d0 0x55e7ecf9d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1d9400700 1 -- 172.21.15.27:0/1168212907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf94060 msgr2=0x55e7ecf9fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1d9400700 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf94060 0x55e7ecf9fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1d9400700 1 -- 172.21.15.27:0/1168212907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf989d0 msgr2=0x55e7ecf9d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1d9400700 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf989d0 0x55e7ecf9d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1d9400700 1 -- 172.21.15.27:0/1168212907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7eceac420 con 0x55e7ecf94980 2024-05-10T11:26:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1da800700 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf989d0 0x55e7ecf9d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:26:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1d9e00700 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf94060 0x55e7ecf9fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:26:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1d9400700 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf9d1c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40aa480 tx=0x7ff1c40228e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3ef930fbb84ca73 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.314+0000 7ff1b7e00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:26:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.315+0000 7ff1b7e00700 1 -- 172.21.15.27:0/1168212907 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40aa550 con 0x55e7ecf94980 2024-05-10T11:26:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.315+0000 7ff1b7e00700 1 -- 172.21.15.27:0/1168212907 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9679+0+0 (secure 0 0 0) 0x7ff1c40aa9e0 con 0x55e7ecf94980 2024-05-10T11:26:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.315+0000 7ff1e7262b80 1 -- 172.21.15.27:0/1168212907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d0017b00 con 0x55e7ecf94980 2024-05-10T11:26:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.315+0000 7ff1b7e00700 1 -- 172.21.15.27:0/1168212907 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff1c40a9ef0 con 0x55e7ecf94980 2024-05-10T11:26:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.315+0000 7ff1e7262b80 10 client.27092 fetch_fsmap learned FSMap version 36 2024-05-10T11:26:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.315+0000 7ff1e7262b80 1 -- 172.21.15.27:0/1168212907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e7ecfab420 con 0x55e7ecf94980 2024-05-10T11:26:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.315+0000 7ff1b7e00700 1 -- 172.21.15.27:0/1168212907 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d0017b00 con 0x55e7ecf94980 2024-05-10T11:26:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.315+0000 7ff1b7e00700 1 -- 172.21.15.27:0/1168212907 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7ff1c40e3ce0 con 0x55e7ecf94980 2024-05-10T11:26:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.316+0000 7ff1e7262b80 10 client.27092 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:26:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.316+0000 7ff1e7262b80 10 client.27092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:26:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.316+0000 7ff1e7262b80 20 client.27092 populate_metadata read hostname 'smithi027' 2024-05-10T11:26:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.316+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e7ecfad9e0 0x55e7ecfafe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.316+0000 7ff1e7262b80 4 client.27092 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:26:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.316+0000 7ff1e7262b80 1 -- 172.21.15.27:0/1168212907 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7ecf610a0 con 0x55e7ecfad9e0 2024-05-10T11:26:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.316+0000 7ff1d9e00700 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e7ecfad9e0 0x55e7ecfafe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.316+0000 7ff1d9e00700 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e7ecfad9e0 0x55e7ecfafe00 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c568da36b7fc523c server_cookie=3483ef3eb1911e7 in_seq=0 out_seq=0 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.316+0000 7ff1b7e00700 10 client.27092 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.317+0000 7ff1b7e00700 1 -- 172.21.15.27:0/1168212907 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e7ecf610a0 con 0x55e7ecfad9e0 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1b7e00700 10 client.27092 handle_command_reply: tid=0 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1e7262b80 1 client.27092 shutdown 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1e7262b80 1 -- 172.21.15.27:0/1168212907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 msgr2=0x55e7ecf9d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf9d1c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40aa480 tx=0x7ff1c40228e0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1e7262b80 1 -- 172.21.15.27:0/1168212907 shutdown_connections 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e7ecfad9e0 0x55e7ecfafe00 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf94060 0x55e7ecf9fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf989d0 0x55e7ecf9d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/1168212907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf9d1c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1e7262b80 1 -- 172.21.15.27:0/1168212907 >> 172.21.15.27:0/1168212907 conn(0x55e7ecec2b60 msgr2=0x55e7ecf892b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:04.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1b7e00700 0 client.27092 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:04.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1e7262b80 1 -- 172.21.15.27:0/1168212907 shutdown_connections 2024-05-10T11:26:04.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1e7262b80 1 -- 172.21.15.27:0/1168212907 wait complete. 2024-05-10T11:26:04.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.318+0000 7ff1e7262b80 20 client.27092 trim_cache size 0 max 16384 2024-05-10T11:26:04.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.338+0000 7ff1e7262b80 1 Processor -- start 2024-05-10T11:26:04.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.338+0000 7ff1e7262b80 1 -- start start 2024-05-10T11:26:04.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.338+0000 7ff1e7262b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf94060 0x55e7ecfa9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.338+0000 7ff1e7262b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecfb5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.338+0000 7ff1e7262b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf989d0 0x55e7ecfb5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.338+0000 7ff1e7262b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41b1910 con 0x55e7ecf989d0 2024-05-10T11:26:04.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.338+0000 7ff1e7262b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41b3450 con 0x55e7ecf94980 2024-05-10T11:26:04.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.338+0000 7ff1e7262b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41b1fe0 con 0x55e7ecf94060 2024-05-10T11:26:04.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.339+0000 7ff1d9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf94060 0x55e7ecfa9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.339+0000 7ff1d9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf94060 0x55e7ecfa9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60286/0 (socket says 172.21.15.27:60286) 2024-05-10T11:26:04.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.339+0000 7ff1d9400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecfb5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.339+0000 7ff1d9e00700 1 -- 172.21.15.27:0/3644052406 learned_addr learned my addr 172.21.15.27:0/3644052406 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.339+0000 7ff1da800700 1 --2- 172.21.15.27:0/3644052406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf989d0 0x55e7ecfb5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.339+0000 7ff1d9400700 1 -- 172.21.15.27:0/3644052406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf94060 msgr2=0x55e7ecfa9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.339+0000 7ff1d9400700 1 --2- 172.21.15.27:0/3644052406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf94060 0x55e7ecfa9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.339+0000 7ff1d9400700 1 -- 172.21.15.27:0/3644052406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf989d0 msgr2=0x55e7ecfb5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.339+0000 7ff1d9400700 1 --2- 172.21.15.27:0/3644052406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf989d0 0x55e7ecfb5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.339+0000 7ff1d9400700 1 -- 172.21.15.27:0/3644052406 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7ecf8b3f0 con 0x55e7ecf94980 2024-05-10T11:26:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.339+0000 7ff1d9400700 1 --2- 172.21.15.27:0/3644052406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecfb5710 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40b3710 tx=0x7ff1c4009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31cc43506b7e5d26 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.340+0000 7ff1c2a00700 1 -- 172.21.15.27:0/3644052406 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40aa0d0 con 0x55e7ecf94980 2024-05-10T11:26:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.340+0000 7ff1c2a00700 1 -- 172.21.15.27:0/3644052406 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1c40b30c0 con 0x55e7ecf94980 2024-05-10T11:26:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.340+0000 7ff1c2a00700 1 -- 172.21.15.27:0/3644052406 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40aa550 con 0x55e7ecf94980 2024-05-10T11:26:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.341+0000 7ff1e7262b80 1 -- 172.21.15.27:0/3644052406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 msgr2=0x55e7ecfb5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.341+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/3644052406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecfb5710 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40b3710 tx=0x7ff1c4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.341+0000 7ff1e7262b80 1 -- 172.21.15.27:0/3644052406 shutdown_connections 2024-05-10T11:26:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.341+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/3644052406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf94060 0x55e7ecfa9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.341+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/3644052406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf989d0 0x55e7ecfb5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.341+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/3644052406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecfb5710 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.341+0000 7ff1e7262b80 1 -- 172.21.15.27:0/3644052406 >> 172.21.15.27:0/3644052406 conn(0x55e7ecec2b60 msgr2=0x55e7ecf9bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.341+0000 7ff1e7262b80 1 -- 172.21.15.27:0/3644052406 shutdown_connections 2024-05-10T11:26:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.341+0000 7ff1e7262b80 1 -- 172.21.15.27:0/3644052406 wait complete. 2024-05-10T11:26:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.341+0000 7ff1e7262b80 1 -- start start 2024-05-10T11:26:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.341+0000 7ff1e7262b80 1 Processor -- start 2024-05-10T11:26:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1e7262b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 0x55e7ecfa64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1e7262b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf96400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1e7262b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf989d0 0x55e7ecf96940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1e7262b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41b2d80 con 0x55e7ecf94060 2024-05-10T11:26:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1e7262b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41b26b0 con 0x55e7ecf94980 2024-05-10T11:26:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1e7262b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41af8d0 con 0x55e7ecf989d0 2024-05-10T11:26:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1e7262b80 10 client.0 _ll_register_callbacks cb 0x55e7ece8dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:26:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1d9400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf96400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1da800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf989d0 0x55e7ecf96940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1d9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 0x55e7ecfa64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1d9400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf96400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48590/0 (socket says 172.21.15.27:48590) 2024-05-10T11:26:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1d9400700 1 -- 172.21.15.27:0/90148748 learned_addr learned my addr 172.21.15.27:0/90148748 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.342+0000 7ff1da800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf989d0 0x55e7ecf96940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60294/0 (socket says 172.21.15.27:60294) 2024-05-10T11:26:04.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.343+0000 7ff1d9400700 1 -- 172.21.15.27:0/90148748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf989d0 msgr2=0x55e7ecf96940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:04.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.343+0000 7ff1d9400700 1 --2- 172.21.15.27:0/90148748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf989d0 0x55e7ecf96940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.343+0000 7ff1d9400700 1 -- 172.21.15.27:0/90148748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 msgr2=0x55e7ecfa64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:04.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.343+0000 7ff1d9400700 1 --2- 172.21.15.27:0/90148748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 0x55e7ecfa64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.343+0000 7ff1d9400700 1 -- 172.21.15.27:0/90148748 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7ecf97180 con 0x55e7ecf94980 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.343+0000 7ff1d9e00700 1 --2- 172.21.15.27:0/90148748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 0x55e7ecfa64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.343+0000 7ff1d9400700 1 --2- 172.21.15.27:0/90148748 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf96400 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40a6800 tx=0x7ff1c40e0c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1b8580f945d1ca6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.343+0000 7ff1b7e00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.343+0000 7ff1b7e00700 1 -- 172.21.15.27:0/90148748 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40aa550 con 0x55e7ecf94980 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.343+0000 7ff1e7262b80 1 -- 172.21.15.27:0/90148748 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1c40b30c0 con 0x55e7ecf94980 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.343+0000 7ff1b7e00700 1 -- 172.21.15.27:0/90148748 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9679+0+0 (secure 0 0 0) 0x7ff1c4005320 con 0x55e7ecf94980 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.344+0000 7ff1b7e00700 1 -- 172.21.15.27:0/90148748 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff1c40aa0d0 con 0x55e7ecf94980 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.344+0000 7ff1e7262b80 10 client.27098 fetch_fsmap learned FSMap version 36 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.344+0000 7ff1e7262b80 1 -- 172.21.15.27:0/90148748 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e7ecf97180 con 0x55e7ecf94980 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.344+0000 7ff1b7e00700 1 -- 172.21.15.27:0/90148748 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1c40b30c0 con 0x55e7ecf94980 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.344+0000 7ff1b7e00700 1 -- 172.21.15.27:0/90148748 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7ff1c40a99e0 con 0x55e7ecf94980 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.344+0000 7ff1e7262b80 10 client.27098 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:26:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.344+0000 7ff1e7262b80 10 client.27098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:26:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.344+0000 7ff1e7262b80 20 client.27098 populate_metadata read hostname 'smithi027' 2024-05-10T11:26:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.344+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/90148748 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e7ed0184e0 0x55e7ed01a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.344+0000 7ff1e7262b80 4 client.27098 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:26:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.344+0000 7ff1e7262b80 1 -- 172.21.15.27:0/90148748 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7ecef4400 con 0x55e7ed0184e0 2024-05-10T11:26:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.345+0000 7ff1d9e00700 1 --2- 172.21.15.27:0/90148748 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e7ed0184e0 0x55e7ed01a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.345+0000 7ff1d9e00700 1 --2- 172.21.15.27:0/90148748 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e7ed0184e0 0x55e7ed01a900 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14160aa515d1d8f3 server_cookie=557031c11be58306 in_seq=0 out_seq=0 2024-05-10T11:26:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.345+0000 7ff1b7e00700 10 client.27098 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.347+0000 7ff1b7e00700 1 -- 172.21.15.27:0/90148748 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7ecef4400 con 0x55e7ed0184e0 2024-05-10T11:26:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.347+0000 7ff1b7e00700 10 client.27098 handle_command_reply: tid=0 2024-05-10T11:26:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.347+0000 7ff1e7262b80 1 client.27098 shutdown 2024-05-10T11:26:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.347+0000 7ff1e7262b80 1 -- 172.21.15.27:0/90148748 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 msgr2=0x55e7ecf96400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.347+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/90148748 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf96400 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40a6800 tx=0x7ff1c40e0c40 comp rx=0 tx=0).stop 2024-05-10T11:26:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.347+0000 7ff1e7262b80 1 -- 172.21.15.27:0/90148748 shutdown_connections 2024-05-10T11:26:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.347+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/90148748 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e7ed0184e0 0x55e7ed01a900 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.347+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/90148748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7ecf989d0 0x55e7ecf96940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.347+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/90148748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7ecf94060 0x55e7ecfa64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.347+0000 7ff1e7262b80 1 --2- 172.21.15.27:0/90148748 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7ecf94980 0x55e7ecf96400 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.347+0000 7ff1b7e00700 0 client.27098 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.347+0000 7ff1e7262b80 1 -- 172.21.15.27:0/90148748 >> 172.21.15.27:0/90148748 conn(0x55e7ecec2b60 msgr2=0x55e7ecf88a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.348+0000 7ff1e7262b80 1 -- 172.21.15.27:0/90148748 shutdown_connections 2024-05-10T11:26:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.348+0000 7ff1e7262b80 1 -- 172.21.15.27:0/90148748 wait complete. 2024-05-10T11:26:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.348+0000 7ff1e7262b80 20 client.27098 trim_cache size 0 max 16384 2024-05-10T11:26:04.350 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:26:04.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.348+0000 7ff1dc800700 1 -- 172.21.15.27:0/2829596350 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff1bc0b2a70 msgr2=0x7ff1bc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:04.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.348+0000 7ff1dc800700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff1bc0b2a70 0x7ff1bc0b4f30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0022000 tx=0x7ff1d001f000 comp rx=0 tx=0).stop 2024-05-10T11:26:04.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.348+0000 7ff1dc800700 1 -- 172.21.15.27:0/2829596350 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d4104100 msgr2=0x7ff1d419c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.348+0000 7ff1dc800700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d4104100 0x7ff1d419c290 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4000c00 tx=0x7ff1c40057c0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.349+0000 7ff1dc800700 1 -- 172.21.15.27:0/2829596350 shutdown_connections 2024-05-10T11:26:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.349+0000 7ff1dc800700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff1bc0b2a70 0x7ff1bc0b4f30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.349+0000 7ff1dc800700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1d4108850 0x7ff1d41a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.349+0000 7ff1dc800700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d4103750 0x7ff1d419bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.349+0000 7ff1dc800700 1 --2- 172.21.15.27:0/2829596350 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d4104100 0x7ff1d419c290 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.349+0000 7ff1dc800700 1 -- 172.21.15.27:0/2829596350 >> 172.21.15.27:0/2829596350 conn(0x7ff1d407a600 msgr2=0x7ff1d41018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.349+0000 7ff1dc800700 1 -- 172.21.15.27:0/2829596350 shutdown_connections 2024-05-10T11:26:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:04.349+0000 7ff1dc800700 1 -- 172.21.15.27:0/2829596350 wait complete. 2024-05-10T11:26:04.740 INFO:tasks.workunit.client.0.smithi027.stdout: 1 774261 6.22 MB/sec execute 529 sec latency 19.860 ms 2024-05-10T11:26:04.897 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:04 smithi027 ceph-mon[30054]: pgmap v401: 129 pgs: 129 active+clean; 604 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 146 op/s 2024-05-10T11:26:04.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:04 smithi076 ceph-mon[34152]: pgmap v401: 129 pgs: 129 active+clean; 604 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 146 op/s 2024-05-10T11:26:04.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:04 smithi189 ceph-mon[34842]: pgmap v401: 129 pgs: 129 active+clean; 604 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 146 op/s 2024-05-10T11:26:05.740 INFO:tasks.workunit.client.0.smithi027.stdout: 1 774868 6.21 MB/sec execute 530 sec latency 37.393 ms 2024-05-10T11:26:06.445 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:26:06.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:26:06.740 INFO:tasks.workunit.client.0.smithi027.stdout: 1 775943 6.21 MB/sec execute 531 sec latency 110.165 ms 2024-05-10T11:26:06.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:06 smithi027 ceph-mon[30054]: pgmap v402: 129 pgs: 129 active+clean; 606 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-05-10T11:26:06.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:06 smithi076 ceph-mon[34152]: pgmap v402: 129 pgs: 129 active+clean; 606 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-05-10T11:26:06.984 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:26:06.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:06 smithi189 ceph-mon[34842]: pgmap v402: 129 pgs: 129 active+clean; 606 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-05-10T11:26:07.740 INFO:tasks.workunit.client.0.smithi027.stdout: 1 777665 6.22 MB/sec execute 532 sec latency 23.905 ms 2024-05-10T11:26:08.740 INFO:tasks.workunit.client.0.smithi027.stdout: 1 777908 6.21 MB/sec execute 533 sec latency 336.446 ms 2024-05-10T11:26:08.833 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:08 smithi027 ceph-mon[30054]: pgmap v403: 129 pgs: 129 active+clean; 606 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-05-10T11:26:08.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:08 smithi076 ceph-mon[34152]: pgmap v403: 129 pgs: 129 active+clean; 606 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-05-10T11:26:08.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:08 smithi189 ceph-mon[34842]: pgmap v403: 129 pgs: 129 active+clean; 606 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-05-10T11:26:09.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.342+0000 7f61aa800700 1 -- 172.21.15.27:0/824144168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a410d9c0 msgr2=0x7f61a410fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:09.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.342+0000 7f61aa800700 1 --2- 172.21.15.27:0/824144168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a410d9c0 0x7f61a410fe80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f619c00b0a0 tx=0x7f619c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.342+0000 7f61aa800700 1 -- 172.21.15.27:0/824144168 shutdown_connections 2024-05-10T11:26:09.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.342+0000 7f61aa800700 1 --2- 172.21.15.27:0/824144168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61a4104590 0x7f61a4104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.342+0000 7f61aa800700 1 --2- 172.21.15.27:0/824144168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a410d9c0 0x7f61a410fe80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.342+0000 7f61aa800700 1 --2- 172.21.15.27:0/824144168 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61a4104eb0 0x7f61a410d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.342+0000 7f61aa800700 1 -- 172.21.15.27:0/824144168 >> 172.21.15.27:0/824144168 conn(0x7f61a40fddc0 msgr2=0x7f61a41001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:09.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.342+0000 7f61aa800700 1 -- 172.21.15.27:0/824144168 shutdown_connections 2024-05-10T11:26:09.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.342+0000 7f61aa800700 1 -- 172.21.15.27:0/824144168 wait complete. 2024-05-10T11:26:09.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.344+0000 7f61aa800700 1 Processor -- start 2024-05-10T11:26:09.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.344+0000 7f61aa800700 1 -- start start 2024-05-10T11:26:09.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.345+0000 7f61aa800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61a4104590 0x7f61a41a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.345+0000 7f61aa800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a4104eb0 0x7f61a41a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.345+0000 7f61aa800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61a410d9c0 0x7f61a419be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.345+0000 7f61aa800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f61a4102a20 con 0x7f61a4104eb0 2024-05-10T11:26:09.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.345+0000 7f61aa800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f61a41028a0 con 0x7f61a4104590 2024-05-10T11:26:09.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.345+0000 7f61aa800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61a4102ba0 con 0x7f61a410d9c0 2024-05-10T11:26:09.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.345+0000 7f61a9400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61a4104590 0x7f61a41a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.345+0000 7f61a8a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a4104eb0 0x7f61a41a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.345+0000 7f61a9400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61a4104590 0x7f61a41a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48610/0 (socket says 172.21.15.27:48610) 2024-05-10T11:26:09.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.345+0000 7f61a9400700 1 -- 172.21.15.27:0/2283130883 learned_addr learned my addr 172.21.15.27:0/2283130883 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:09.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.345+0000 7f61a9e00700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61a410d9c0 0x7f61a419be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.345+0000 7f61a9400700 1 -- 172.21.15.27:0/2283130883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61a410d9c0 msgr2=0x7f61a419be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:09.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.346+0000 7f61a9400700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61a410d9c0 0x7f61a419be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.346+0000 7f61a9400700 1 -- 172.21.15.27:0/2283130883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a4104eb0 msgr2=0x7f61a41a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:09.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.346+0000 7f61a9400700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a4104eb0 0x7f61a41a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.346+0000 7f61a9400700 1 -- 172.21.15.27:0/2283130883 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a419c5c0 con 0x7f61a4104590 2024-05-10T11:26:09.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.346+0000 7f61a9e00700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61a410d9c0 0x7f61a419be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:26:09.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.346+0000 7f61a9400700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61a4104590 0x7f61a41a0850 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f619800e9a0 tx=0x7f619800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:09.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.346+0000 7f61a2000700 1 -- 172.21.15.27:0/2283130883 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619800ca10 con 0x7f61a4104590 2024-05-10T11:26:09.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.346+0000 7f61aa800700 1 -- 172.21.15.27:0/2283130883 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a419c8b0 con 0x7f61a4104590 2024-05-10T11:26:09.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.346+0000 7f61a2000700 1 -- 172.21.15.27:0/2283130883 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f619800cbb0 con 0x7f61a4104590 2024-05-10T11:26:09.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.346+0000 7f61a2000700 1 -- 172.21.15.27:0/2283130883 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198018680 con 0x7f61a4104590 2024-05-10T11:26:09.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.346+0000 7f61aa800700 1 -- 172.21.15.27:0/2283130883 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a41a8780 con 0x7f61a4104590 2024-05-10T11:26:09.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.350+0000 7f61a2000700 1 -- 172.21.15.27:0/2283130883 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f6198010430 con 0x7f61a4104590 2024-05-10T11:26:09.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.351+0000 7f61b6882b80 1 Processor -- start 2024-05-10T11:26:09.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.351+0000 7f61b6882b80 1 -- start start 2024-05-10T11:26:09.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.351+0000 7f61b6882b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae37e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.351+0000 7f61b6882b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae38380 0x55ca9ae32060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.352+0000 7f61b6882b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae32680 0x55ca9ae32b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.352+0000 7f61b6882b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ca9ad84aa0 con 0x55ca9ae37a60 2024-05-10T11:26:09.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.352+0000 7f61b6882b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ca9ad227e0 con 0x55ca9ae32680 2024-05-10T11:26:09.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.352+0000 7f61b6882b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ca9ad80a60 con 0x55ca9ae38380 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.352+0000 7f61a9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae37e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.352+0000 7f61a9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae37e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34084/0 (socket says 172.21.15.27:34084) 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.352+0000 7f61a9e00700 1 -- 172.21.15.27:0/857361771 learned_addr learned my addr 172.21.15.27:0/857361771 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.352+0000 7f61a8a00700 1 --2- 172.21.15.27:0/857361771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae32680 0x55ca9ae32b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.352+0000 7f61a2000700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f618c0b29f0 0x7f618c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.352+0000 7f61a9400700 1 --2- 172.21.15.27:0/857361771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae38380 0x55ca9ae32060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.353+0000 7f61a2000700 1 -- 172.21.15.27:0/2283130883 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7f61980e2c00 con 0x7f61a4104590 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.353+0000 7f61a9e00700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f618c0b29f0 0x7f618c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.353+0000 7f61a8a00700 1 -- 172.21.15.27:0/857361771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae38380 msgr2=0x55ca9ae32060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.353+0000 7f61a8a00700 1 --2- 172.21.15.27:0/857361771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae38380 0x55ca9ae32060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.353+0000 7f61a8a00700 1 -- 172.21.15.27:0/857361771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 msgr2=0x55ca9ae37e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.353+0000 7f61a8a00700 1 --2- 172.21.15.27:0/857361771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae37e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.353+0000 7f61a8a00700 1 -- 172.21.15.27:0/857361771 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca9acd1950 con 0x55ca9ae32680 2024-05-10T11:26:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.353+0000 7f61a9400700 1 --2- 172.21.15.27:0/857361771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae38380 0x55ca9ae32060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:26:09.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.353+0000 7f61a8a00700 1 --2- 172.21.15.27:0/857361771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae32680 0x55ca9ae32b00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f61a410d6c0 tx=0x7f6190009f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e2ce371115eef0b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:09.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.353+0000 7f6196a00700 1 -- 172.21.15.27:0/857361771 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6190011c80 con 0x55ca9ae32680 2024-05-10T11:26:09.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.353+0000 7f6196a00700 1 -- 172.21.15.27:0/857361771 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6190011e20 con 0x55ca9ae32680 2024-05-10T11:26:09.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.354+0000 7f6196a00700 1 -- 172.21.15.27:0/857361771 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6190005280 con 0x55ca9ae32680 2024-05-10T11:26:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.354+0000 7f61b6882b80 1 -- 172.21.15.27:0/857361771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae32680 msgr2=0x55ca9ae32b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.354+0000 7f61b6882b80 1 --2- 172.21.15.27:0/857361771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae32680 0x55ca9ae32b00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f61a410d6c0 tx=0x7f6190009f50 comp rx=0 tx=0).stop 2024-05-10T11:26:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.354+0000 7f61a9e00700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f618c0b29f0 0x7f618c0b4eb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f619c00d010 tx=0x7f619c00b3b0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.354+0000 7f61b6882b80 1 -- 172.21.15.27:0/857361771 shutdown_connections 2024-05-10T11:26:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.354+0000 7f61b6882b80 1 --2- 172.21.15.27:0/857361771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae32680 0x55ca9ae32b00 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.354+0000 7f61b6882b80 1 --2- 172.21.15.27:0/857361771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae37e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.354+0000 7f61b6882b80 1 --2- 172.21.15.27:0/857361771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae38380 0x55ca9ae32060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.354+0000 7f61b6882b80 1 -- 172.21.15.27:0/857361771 >> 172.21.15.27:0/857361771 conn(0x55ca9ad60be0 msgr2=0x55ca9ae2ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:09.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.354+0000 7f61b6882b80 1 -- 172.21.15.27:0/857361771 shutdown_connections 2024-05-10T11:26:09.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.355+0000 7f61b6882b80 1 -- 172.21.15.27:0/857361771 wait complete. 2024-05-10T11:26:09.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.355+0000 7f61b6882b80 1 -- start start 2024-05-10T11:26:09.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.355+0000 7f61b6882b80 1 Processor -- start 2024-05-10T11:26:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.356+0000 7f61b6882b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae32680 0x55ca9ae3df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.356+0000 7f61b6882b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae37a60 0x55ca9ae3b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.356+0000 7f61b6882b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae38380 0x55ca9ae3b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.356+0000 7f61b6882b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ca9ae3ee30 con 0x55ca9ae32680 2024-05-10T11:26:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.356+0000 7f61b6882b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ca9aceaba0 con 0x55ca9ae37a60 2024-05-10T11:26:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.356+0000 7f61b6882b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ca9ae3efb0 con 0x55ca9ae38380 2024-05-10T11:26:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.356+0000 7f61b6882b80 10 client.0 _ll_register_callbacks cb 0x55ca9ad2bcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:26:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.356+0000 7f61a9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae32680 0x55ca9ae3df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.356+0000 7f61a8a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae37a60 0x55ca9ae3b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.356+0000 7f61a9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae32680 0x55ca9ae3df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34094/0 (socket says 172.21.15.27:34094) 2024-05-10T11:26:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.356+0000 7f61a9400700 1 -- 172.21.15.27:0/4181887957 learned_addr learned my addr 172.21.15.27:0/4181887957 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.356+0000 7f61a9e00700 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae38380 0x55ca9ae3b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.357+0000 7f61a9e00700 1 -- 172.21.15.27:0/4181887957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae37a60 msgr2=0x55ca9ae3b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.357+0000 7f61a9e00700 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae37a60 0x55ca9ae3b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.357+0000 7f61a9e00700 1 -- 172.21.15.27:0/4181887957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae32680 msgr2=0x55ca9ae3df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.357+0000 7f61a9e00700 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae32680 0x55ca9ae3df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.357+0000 7f61a9e00700 1 -- 172.21.15.27:0/4181887957 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca9ad4a4a0 con 0x55ca9ae38380 2024-05-10T11:26:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.357+0000 7f61a9400700 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae32680 0x55ca9ae3df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:26:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.357+0000 7f61a9e00700 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae38380 0x55ca9ae3b840 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f619c015000 tx=0x7f619c0099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c901262930df6185 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.357+0000 7f618b400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:26:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.357+0000 7f618b400700 1 -- 172.21.15.27:0/4181887957 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c02e070 con 0x55ca9ae38380 2024-05-10T11:26:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.357+0000 7f61b6882b80 1 -- 172.21.15.27:0/4181887957 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6190011e20 con 0x55ca9ae38380 2024-05-10T11:26:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.357+0000 7f618b400700 1 -- 172.21.15.27:0/4181887957 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7f619c008d90 con 0x55ca9ae38380 2024-05-10T11:26:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.358+0000 7f618b400700 1 -- 172.21.15.27:0/4181887957 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f619c03c050 con 0x55ca9ae38380 2024-05-10T11:26:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.358+0000 7f61b6882b80 10 client.27094 fetch_fsmap learned FSMap version 36 2024-05-10T11:26:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.358+0000 7f61b6882b80 1 -- 172.21.15.27:0/4181887957 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ca9ae49580 con 0x55ca9ae38380 2024-05-10T11:26:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.358+0000 7f618b400700 1 -- 172.21.15.27:0/4181887957 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6190011e20 con 0x55ca9ae38380 2024-05-10T11:26:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.358+0000 7f618b400700 1 -- 172.21.15.27:0/4181887957 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f619c011070 con 0x55ca9ae38380 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.358+0000 7f61b6882b80 10 client.27094 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.358+0000 7f61b6882b80 10 client.27094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.358+0000 7f61b6882b80 20 client.27094 populate_metadata read hostname 'smithi027' 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.358+0000 7f61b6882b80 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ca9ae4bb10 0x55ca9ae4df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.358+0000 7f61b6882b80 4 client.27094 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.358+0000 7f61b6882b80 1 -- 172.21.15.27:0/4181887957 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca9adff120 con 0x55ca9ae4bb10 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.359+0000 7f61a9400700 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ca9ae4bb10 0x55ca9ae4df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.359+0000 7f61a9400700 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ca9ae4bb10 0x55ca9ae4df00 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc60cb6fbc1890eb server_cookie=c07cf15e13a4351b in_seq=0 out_seq=0 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.359+0000 7f618b400700 10 client.27094 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.360+0000 7f618b400700 1 -- 172.21.15.27:0/4181887957 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ca9adff120 con 0x55ca9ae4bb10 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.360+0000 7f618b400700 10 client.27094 handle_command_reply: tid=0 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.360+0000 7f61b6882b80 1 client.27094 shutdown 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.360+0000 7f61b6882b80 1 -- 172.21.15.27:0/4181887957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae38380 msgr2=0x55ca9ae3b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.360+0000 7f61b6882b80 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae38380 0x55ca9ae3b840 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f619c015000 tx=0x7f619c0099e0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.360+0000 7f61b6882b80 1 -- 172.21.15.27:0/4181887957 shutdown_connections 2024-05-10T11:26:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.360+0000 7f61b6882b80 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ca9ae4bb10 0x55ca9ae4df00 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.360+0000 7f61b6882b80 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae38380 0x55ca9ae3b840 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.360+0000 7f61b6882b80 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae32680 0x55ca9ae3df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.360+0000 7f61b6882b80 1 --2- 172.21.15.27:0/4181887957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae37a60 0x55ca9ae3b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.360+0000 7f618b400700 0 client.27094 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.360+0000 7f61b6882b80 1 -- 172.21.15.27:0/4181887957 >> 172.21.15.27:0/4181887957 conn(0x55ca9ad60be0 msgr2=0x55ca9ae274e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.361+0000 7f61b6882b80 1 -- 172.21.15.27:0/4181887957 shutdown_connections 2024-05-10T11:26:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.361+0000 7f61b6882b80 1 -- 172.21.15.27:0/4181887957 wait complete. 2024-05-10T11:26:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.361+0000 7f61b6882b80 20 client.27094 trim_cache size 0 max 16384 2024-05-10T11:26:09.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.383+0000 7f61b6882b80 1 Processor -- start 2024-05-10T11:26:09.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.383+0000 7f61b6882b80 1 -- start start 2024-05-10T11:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.383+0000 7f61b6882b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae32680 0x55ca9ae32060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.383+0000 7f61b6882b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae3a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.383+0000 7f61b6882b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae38380 0x55ca9ae27360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.383+0000 7f61b6882b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f61a41b1810 con 0x55ca9ae37a60 2024-05-10T11:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.383+0000 7f61b6882b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f61a41b3350 con 0x55ca9ae38380 2024-05-10T11:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.383+0000 7f61b6882b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61a41b1ee0 con 0x55ca9ae32680 2024-05-10T11:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.384+0000 7f61a9e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae38380 0x55ca9ae27360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.384+0000 7f61a9400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae32680 0x55ca9ae32060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.384+0000 7f61a9e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae38380 0x55ca9ae27360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48640/0 (socket says 172.21.15.27:48640) 2024-05-10T11:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.384+0000 7f61a9e00700 1 -- 172.21.15.27:0/2383906513 learned_addr learned my addr 172.21.15.27:0/2383906513 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.384+0000 7f61a8a00700 1 --2- 172.21.15.27:0/2383906513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae3a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.384+0000 7f61a9e00700 1 -- 172.21.15.27:0/2383906513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae32680 msgr2=0x55ca9ae32060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.384+0000 7f61a9e00700 1 --2- 172.21.15.27:0/2383906513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae32680 0x55ca9ae32060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.384+0000 7f61a9e00700 1 -- 172.21.15.27:0/2383906513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 msgr2=0x55ca9ae3a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.384+0000 7f61a9e00700 1 --2- 172.21.15.27:0/2383906513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae3a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.384+0000 7f61a9e00700 1 -- 172.21.15.27:0/2383906513 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca9ae2d890 con 0x55ca9ae38380 2024-05-10T11:26:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.384+0000 7f61a9400700 1 --2- 172.21.15.27:0/2383906513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae32680 0x55ca9ae32060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:26:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.385+0000 7f61a9e00700 1 --2- 172.21.15.27:0/2383906513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae38380 0x55ca9ae27360 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f619c00b0a0 tx=0x7f619c00bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c8003e53f00f652 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.385+0000 7f6196000700 1 -- 172.21.15.27:0/2383906513 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c02e040 con 0x55ca9ae38380 2024-05-10T11:26:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.385+0000 7f6196000700 1 -- 172.21.15.27:0/2383906513 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f619c005c60 con 0x55ca9ae38380 2024-05-10T11:26:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.385+0000 7f6196000700 1 -- 172.21.15.27:0/2383906513 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c01dc20 con 0x55ca9ae38380 2024-05-10T11:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.386+0000 7f61b6882b80 1 -- 172.21.15.27:0/2383906513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae38380 msgr2=0x55ca9ae27360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.386+0000 7f61b6882b80 1 --2- 172.21.15.27:0/2383906513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae38380 0x55ca9ae27360 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f619c00b0a0 tx=0x7f619c00bf50 comp rx=0 tx=0).stop 2024-05-10T11:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.386+0000 7f61b6882b80 1 -- 172.21.15.27:0/2383906513 shutdown_connections 2024-05-10T11:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.386+0000 7f61b6882b80 1 --2- 172.21.15.27:0/2383906513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae32680 0x55ca9ae32060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.386+0000 7f61b6882b80 1 --2- 172.21.15.27:0/2383906513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae38380 0x55ca9ae27360 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.386+0000 7f61b6882b80 1 --2- 172.21.15.27:0/2383906513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae3a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.386+0000 7f61b6882b80 1 -- 172.21.15.27:0/2383906513 >> 172.21.15.27:0/2383906513 conn(0x55ca9ad60be0 msgr2=0x55ca9ae26c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.386+0000 7f61b6882b80 1 -- 172.21.15.27:0/2383906513 shutdown_connections 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.386+0000 7f61b6882b80 1 -- 172.21.15.27:0/2383906513 wait complete. 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.386+0000 7f61b6882b80 1 -- start start 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.386+0000 7f61b6882b80 1 Processor -- start 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.387+0000 7f61b6882b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae32680 0x55ca9ae414f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.387+0000 7f61b6882b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae41a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.387+0000 7f61b6882b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae38380 0x55ca9aeb6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.387+0000 7f61b6882b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f61a41b2c80 con 0x55ca9ae37a60 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.387+0000 7f61b6882b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f61a41b25b0 con 0x55ca9ae38380 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.387+0000 7f61b6882b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61a41af7d0 con 0x55ca9ae32680 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.387+0000 7f61b6882b80 10 client.0 _ll_register_callbacks cb 0x55ca9ad2bcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.387+0000 7f61a8a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae41a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.387+0000 7f61a9400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae32680 0x55ca9ae414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.387+0000 7f61a8a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae41a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34100/0 (socket says 172.21.15.27:34100) 2024-05-10T11:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.387+0000 7f61a8a00700 1 -- 172.21.15.27:0/1082596617 learned_addr learned my addr 172.21.15.27:0/1082596617 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.387+0000 7f61a9e00700 1 --2- 172.21.15.27:0/1082596617 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae38380 0x55ca9aeb6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.388+0000 7f61a9400700 1 -- 172.21.15.27:0/1082596617 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae38380 msgr2=0x55ca9aeb6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.388+0000 7f61a9400700 1 --2- 172.21.15.27:0/1082596617 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae38380 0x55ca9aeb6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.388+0000 7f61a9400700 1 -- 172.21.15.27:0/1082596617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 msgr2=0x55ca9ae41a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.388+0000 7f61a9400700 1 --2- 172.21.15.27:0/1082596617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae41a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.388+0000 7f61a9400700 1 -- 172.21.15.27:0/1082596617 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca9aeb68d0 con 0x55ca9ae32680 2024-05-10T11:26:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.388+0000 7f61a9400700 1 --2- 172.21.15.27:0/1082596617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae32680 0x55ca9ae414f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6198000c00 tx=0x7f61980a60a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f672746c4eacfbcb server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.388+0000 7f618b400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:26:09.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.388+0000 7f618b400700 1 -- 172.21.15.27:0/1082596617 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61980a5040 con 0x55ca9ae32680 2024-05-10T11:26:09.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.388+0000 7f618b400700 1 -- 172.21.15.27:0/1082596617 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7f6198010720 con 0x55ca9ae32680 2024-05-10T11:26:09.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.388+0000 7f61b6882b80 1 -- 172.21.15.27:0/1082596617 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f619c005c60 con 0x55ca9ae32680 2024-05-10T11:26:09.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.389+0000 7f618b400700 1 -- 172.21.15.27:0/1082596617 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f61980a9560 con 0x55ca9ae32680 2024-05-10T11:26:09.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.389+0000 7f61b6882b80 10 client.27100 fetch_fsmap learned FSMap version 36 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.389+0000 7f61b6882b80 1 -- 172.21.15.27:0/1082596617 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ca9aeb7c50 con 0x55ca9ae32680 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.389+0000 7f618b400700 1 -- 172.21.15.27:0/1082596617 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f619c005c60 con 0x55ca9ae32680 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.389+0000 7f618b400700 1 -- 172.21.15.27:0/1082596617 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f619806f260 con 0x55ca9ae32680 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.389+0000 7f61b6882b80 10 client.27100 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.389+0000 7f61b6882b80 10 client.27100 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.389+0000 7f61b6882b80 20 client.27100 populate_metadata read hostname 'smithi027' 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.389+0000 7f61b6882b80 1 --2- 172.21.15.27:0/1082596617 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ca9aeb82e0 0x55ca9aeba700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.389+0000 7f61b6882b80 4 client.27100 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.389+0000 7f61b6882b80 1 -- 172.21.15.27:0/1082596617 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ca9ad92480 con 0x55ca9aeb82e0 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.390+0000 7f61a8a00700 1 --2- 172.21.15.27:0/1082596617 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ca9aeb82e0 0x55ca9aeba700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.390+0000 7f61a8a00700 1 --2- 172.21.15.27:0/1082596617 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ca9aeb82e0 0x55ca9aeba700 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9eabb18239798586 server_cookie=5b4e0c82d4f80b1f in_seq=0 out_seq=0 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.390+0000 7f618b400700 10 client.27100 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.391+0000 7f618b400700 1 -- 172.21.15.27:0/1082596617 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ca9ad92480 con 0x55ca9aeb82e0 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.391+0000 7f618b400700 10 client.27100 handle_command_reply: tid=0 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.391+0000 7f61b6882b80 1 client.27100 shutdown 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.391+0000 7f61b6882b80 1 -- 172.21.15.27:0/1082596617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae32680 msgr2=0x55ca9ae414f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.391+0000 7f61b6882b80 1 --2- 172.21.15.27:0/1082596617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae32680 0x55ca9ae414f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6198000c00 tx=0x7f61980a60a0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.392+0000 7f61b6882b80 1 -- 172.21.15.27:0/1082596617 shutdown_connections 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.392+0000 7f61b6882b80 1 --2- 172.21.15.27:0/1082596617 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55ca9aeb82e0 0x55ca9aeba700 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.392+0000 7f61b6882b80 1 --2- 172.21.15.27:0/1082596617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca9ae32680 0x55ca9ae414f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.392+0000 7f61b6882b80 1 --2- 172.21.15.27:0/1082596617 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ca9ae38380 0x55ca9aeb6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.392+0000 7f61b6882b80 1 --2- 172.21.15.27:0/1082596617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca9ae37a60 0x55ca9ae41a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.392+0000 7f61b6882b80 1 -- 172.21.15.27:0/1082596617 >> 172.21.15.27:0/1082596617 conn(0x55ca9ad60be0 msgr2=0x55ca9ae274e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.392+0000 7f618b400700 0 client.27100 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.392+0000 7f61b6882b80 1 -- 172.21.15.27:0/1082596617 shutdown_connections 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.392+0000 7f61b6882b80 1 -- 172.21.15.27:0/1082596617 wait complete. 2024-05-10T11:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.392+0000 7f61b6882b80 20 client.27100 trim_cache size 0 max 16384 2024-05-10T11:26:09.394 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:26:09.394 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:26:09.394 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "ac946942-0245-4d99-af0f-21494991a8b9", 2024-05-10T11:26:09.394 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:26:09.394 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:26:09.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.393+0000 7f61aa800700 1 -- 172.21.15.27:0/2283130883 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f618c0b29f0 msgr2=0x7f618c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:09.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.393+0000 7f61aa800700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f618c0b29f0 0x7f618c0b4eb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f619c00d010 tx=0x7f619c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.393+0000 7f61aa800700 1 -- 172.21.15.27:0/2283130883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61a4104590 msgr2=0x7f61a41a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:09.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.393+0000 7f61aa800700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61a4104590 0x7f61a41a0850 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f619800e9a0 tx=0x7f619800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.393+0000 7f61aa800700 1 -- 172.21.15.27:0/2283130883 shutdown_connections 2024-05-10T11:26:09.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.393+0000 7f61aa800700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f618c0b29f0 0x7f618c0b4eb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.393+0000 7f61aa800700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61a410d9c0 0x7f61a419be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.393+0000 7f61aa800700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61a4104590 0x7f61a41a0850 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.393+0000 7f61aa800700 1 --2- 172.21.15.27:0/2283130883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a4104eb0 0x7f61a41a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:09.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.393+0000 7f61aa800700 1 -- 172.21.15.27:0/2283130883 >> 172.21.15.27:0/2283130883 conn(0x7f61a40fddc0 msgr2=0x7f61a410eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:09.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.393+0000 7f61aa800700 1 -- 172.21.15.27:0/2283130883 shutdown_connections 2024-05-10T11:26:09.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:09.393+0000 7f61aa800700 1 -- 172.21.15.27:0/2283130883 wait complete. 2024-05-10T11:26:09.740 INFO:tasks.workunit.client.0.smithi027.stdout: 1 777912 6.20 MB/sec execute 534 sec latency 1010.241 ms 2024-05-10T11:26:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:09 smithi189 ceph-mon[34842]: osdmap e105: 12 total, 12 up, 12 in 2024-05-10T11:26:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:09 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:26:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:09 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:09 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:26:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:09 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:26:09.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:09 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:10.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:09 smithi027 ceph-mon[30054]: osdmap e105: 12 total, 12 up, 12 in 2024-05-10T11:26:10.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:09 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:26:10.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:09 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:10.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:09 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:26:10.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:09 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:26:10.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:09 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:10.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:09 smithi076 ceph-mon[34152]: osdmap e105: 12 total, 12 up, 12 in 2024-05-10T11:26:10.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:09 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:26:10.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:09 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:10.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:09 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:26:10.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:09 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:26:10.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:09 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:10.457 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:26:10.740 INFO:tasks.workunit.client.0.smithi027.stdout: 1 777912 6.19 MB/sec execute 535 sec latency 1325.171 ms 2024-05-10T11:26:10.942 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:26:10.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:10 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:26:10.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:10 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:10.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:10 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:26:10.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:10 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:10.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:10 smithi189 ceph-mon[34842]: pgmap v405: 129 pgs: 129 active+clean; 610 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 207 op/s 2024-05-10T11:26:11.101 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:10 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:26:11.101 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:10 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:11.102 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:10 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:26:11.102 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:10 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:11.102 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:10 smithi027 ceph-mon[30054]: pgmap v405: 129 pgs: 129 active+clean; 610 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 207 op/s 2024-05-10T11:26:11.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:10 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:26:11.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:10 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:11.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:10 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:26:11.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:10 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:11.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:10 smithi076 ceph-mon[34152]: pgmap v405: 129 pgs: 129 active+clean; 610 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 207 op/s 2024-05-10T11:26:11.741 INFO:tasks.workunit.client.0.smithi027.stdout: 1 777912 6.17 MB/sec execute 536 sec latency 2325.287 ms 2024-05-10T11:26:12.741 INFO:tasks.workunit.client.0.smithi027.stdout: 1 777912 6.16 MB/sec execute 537 sec latency 3325.422 ms 2024-05-10T11:26:12.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:12 smithi189 ceph-mon[34842]: pgmap v406: 129 pgs: 129 active+clean; 610 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 168 op/s 2024-05-10T11:26:13.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:12 smithi027 ceph-mon[30054]: pgmap v406: 129 pgs: 129 active+clean; 610 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 168 op/s 2024-05-10T11:26:13.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:12 smithi076 ceph-mon[34152]: pgmap v406: 129 pgs: 129 active+clean; 610 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 168 op/s 2024-05-10T11:26:13.741 INFO:tasks.workunit.client.0.smithi027.stdout: 1 778044 6.15 MB/sec execute 538 sec latency 4038.264 ms 2024-05-10T11:26:13.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.822+0000 7fd66fc00700 1 -- 172.21.15.27:0/3324162070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd66810b6c0 msgr2=0x7fd66810baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:13.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.822+0000 7fd66fc00700 1 --2- 172.21.15.27:0/3324162070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd66810b6c0 0x7fd66810baa0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd65c009990 tx=0x7fd65c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.822+0000 7fd66fc00700 1 -- 172.21.15.27:0/3324162070 shutdown_connections 2024-05-10T11:26:13.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.822+0000 7fd66fc00700 1 --2- 172.21.15.27:0/3324162070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd66810b6c0 0x7fd66810baa0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.822+0000 7fd66fc00700 1 --2- 172.21.15.27:0/3324162070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd668106080 0x7fd668112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.822+0000 7fd66fc00700 1 --2- 172.21.15.27:0/3324162070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6681056c0 0x7fd668105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.823+0000 7fd66fc00700 1 -- 172.21.15.27:0/3324162070 >> 172.21.15.27:0/3324162070 conn(0x7fd6681011b0 msgr2=0x7fd6681035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:13.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.823+0000 7fd66fc00700 1 -- 172.21.15.27:0/3324162070 shutdown_connections 2024-05-10T11:26:13.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.823+0000 7fd66fc00700 1 -- 172.21.15.27:0/3324162070 wait complete. 2024-05-10T11:26:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.824+0000 7fd66fc00700 1 Processor -- start 2024-05-10T11:26:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.824+0000 7fd66fc00700 1 -- start start 2024-05-10T11:26:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.825+0000 7fd66fc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6681056c0 0x7fd66819bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.825+0000 7fd66fc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd668106080 0x7fd66819c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.825+0000 7fd66fc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd66810b6c0 0x7fd6681a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.825+0000 7fd66fc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd668115910 con 0x7fd66810b6c0 2024-05-10T11:26:13.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.825+0000 7fd66fc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd668115790 con 0x7fd668106080 2024-05-10T11:26:13.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.825+0000 7fd66fc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd668115a90 con 0x7fd6681056c0 2024-05-10T11:26:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.825+0000 7fd667e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd668106080 0x7fd66819c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.825+0000 7fd66d200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6681056c0 0x7fd66819bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.825+0000 7fd667e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd668106080 0x7fd66819c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48668/0 (socket says 172.21.15.27:48668) 2024-05-10T11:26:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.825+0000 7fd667e00700 1 -- 172.21.15.27:0/853037777 learned_addr learned my addr 172.21.15.27:0/853037777 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.825+0000 7fd66dc00700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd66810b6c0 0x7fd6681a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.826+0000 7fd66d200700 1 -- 172.21.15.27:0/853037777 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd668106080 msgr2=0x7fd66819c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.826+0000 7fd66d200700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd668106080 0x7fd66819c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.826+0000 7fd66d200700 1 -- 172.21.15.27:0/853037777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd66810b6c0 msgr2=0x7fd6681a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.826+0000 7fd66d200700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd66810b6c0 0x7fd6681a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.826+0000 7fd66d200700 1 -- 172.21.15.27:0/853037777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6681a0da0 con 0x7fd6681056c0 2024-05-10T11:26:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.826+0000 7fd667e00700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd668106080 0x7fd66819c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:26:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.826+0000 7fd66dc00700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd66810b6c0 0x7fd6681a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:26:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.826+0000 7fd66d200700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6681056c0 0x7fd66819bde0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd65c000c00 tx=0x7fd65c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.826+0000 7fd665600700 1 -- 172.21.15.27:0/853037777 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c026070 con 0x7fd6681056c0 2024-05-10T11:26:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.827+0000 7fd66fc00700 1 -- 172.21.15.27:0/853037777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6681a1030 con 0x7fd6681056c0 2024-05-10T11:26:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.827+0000 7fd665600700 1 -- 172.21.15.27:0/853037777 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd65c021410 con 0x7fd6681056c0 2024-05-10T11:26:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.827+0000 7fd665600700 1 -- 172.21.15.27:0/853037777 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c017650 con 0x7fd6681056c0 2024-05-10T11:26:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.827+0000 7fd66fc00700 1 -- 172.21.15.27:0/853037777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6681a8910 con 0x7fd6681056c0 2024-05-10T11:26:13.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.830+0000 7fd665600700 1 -- 172.21.15.27:0/853037777 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd65c029070 con 0x7fd6681056c0 2024-05-10T11:26:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.831+0000 7fd67a543b80 1 Processor -- start 2024-05-10T11:26:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.832+0000 7fd67a543b80 1 -- start start 2024-05-10T11:26:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.832+0000 7fd67a543b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 0x558dde658fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.832+0000 7fd67a543b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde6594e0 0x558dde6532d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.832+0000 7fd67a543b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6538f0 0x558dde653d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.832+0000 7fd67a543b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558dde5a5a20 con 0x558dde6538f0 2024-05-10T11:26:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.832+0000 7fd67a543b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558dde543760 con 0x558dde658bc0 2024-05-10T11:26:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.832+0000 7fd67a543b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558dde5a19e0 con 0x558dde6594e0 2024-05-10T11:26:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.832+0000 7fd665600700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd6500b2a70 0x7fd6500b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.832+0000 7fd66dc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 0x558dde658fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.833+0000 7fd66dc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 0x558dde658fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48674/0 (socket says 172.21.15.27:48674) 2024-05-10T11:26:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.833+0000 7fd66dc00700 1 -- 172.21.15.27:0/3049220573 learned_addr learned my addr 172.21.15.27:0/3049220573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.833+0000 7fd66d200700 1 --2- 172.21.15.27:0/3049220573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde6594e0 0x558dde6532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.833+0000 7fd665600700 1 -- 172.21.15.27:0/853037777 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7fd65c0e3910 con 0x7fd6681056c0 2024-05-10T11:26:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.833+0000 7fd667e00700 1 --2- 172.21.15.27:0/3049220573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6538f0 0x558dde653d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.833+0000 7fd66dc00700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd6500b2a70 0x7fd6500b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.833+0000 7fd66d200700 1 -- 172.21.15.27:0/3049220573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 msgr2=0x558dde658fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.833+0000 7fd66d200700 1 --2- 172.21.15.27:0/3049220573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 0x558dde658fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.833+0000 7fd66d200700 1 -- 172.21.15.27:0/3049220573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6538f0 msgr2=0x558dde653d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.833+0000 7fd66d200700 1 --2- 172.21.15.27:0/3049220573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6538f0 0x558dde653d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.833+0000 7fd66d200700 1 -- 172.21.15.27:0/3049220573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558dde4f28d0 con 0x558dde6594e0 2024-05-10T11:26:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.833+0000 7fd66d200700 1 --2- 172.21.15.27:0/3049220573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde6594e0 0x558dde6532d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0e0bd0 tx=0x7fd65c0d9d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=453ecdff27925656 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.834+0000 7fd65a000700 1 -- 172.21.15.27:0/3049220573 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c0a9f60 con 0x558dde6594e0 2024-05-10T11:26:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.834+0000 7fd65a000700 1 -- 172.21.15.27:0/3049220573 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd65c0aa300 con 0x558dde6594e0 2024-05-10T11:26:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.834+0000 7fd65a000700 1 -- 172.21.15.27:0/3049220573 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c0aa7f0 con 0x558dde6594e0 2024-05-10T11:26:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.834+0000 7fd67a543b80 1 -- 172.21.15.27:0/3049220573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde6594e0 msgr2=0x558dde6532d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.834+0000 7fd66dc00700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd6500b2a70 0x7fd6500b4f30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd6600094f0 tx=0x7fd660004b40 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.834+0000 7fd67a543b80 1 --2- 172.21.15.27:0/3049220573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde6594e0 0x558dde6532d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0e0bd0 tx=0x7fd65c0d9d70 comp rx=0 tx=0).stop 2024-05-10T11:26:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.835+0000 7fd67a543b80 1 -- 172.21.15.27:0/3049220573 shutdown_connections 2024-05-10T11:26:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.835+0000 7fd67a543b80 1 --2- 172.21.15.27:0/3049220573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6538f0 0x558dde653d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.835+0000 7fd67a543b80 1 --2- 172.21.15.27:0/3049220573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 0x558dde658fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.835+0000 7fd67a543b80 1 --2- 172.21.15.27:0/3049220573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde6594e0 0x558dde6532d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.835+0000 7fd67a543b80 1 -- 172.21.15.27:0/3049220573 >> 172.21.15.27:0/3049220573 conn(0x558dde5cb6d0 msgr2=0x558dde64db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.835+0000 7fd67a543b80 1 -- 172.21.15.27:0/3049220573 shutdown_connections 2024-05-10T11:26:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.835+0000 7fd67a543b80 1 -- 172.21.15.27:0/3049220573 wait complete. 2024-05-10T11:26:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.835+0000 7fd67a543b80 1 -- start start 2024-05-10T11:26:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.836+0000 7fd67a543b80 1 Processor -- start 2024-05-10T11:26:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.836+0000 7fd67a543b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 0x558dde65f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.836+0000 7fd67a543b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde658bc0 0x558dde65c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.836+0000 7fd67a543b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde65cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.836+0000 7fd67a543b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558dde65fce0 con 0x558dde6594e0 2024-05-10T11:26:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.836+0000 7fd67a543b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558dde50bb20 con 0x558dde6538f0 2024-05-10T11:26:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.836+0000 7fd67a543b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558dde65fe60 con 0x558dde658bc0 2024-05-10T11:26:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.836+0000 7fd67a543b80 10 client.0 _ll_register_callbacks cb 0x558dde54cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:26:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.837+0000 7fd66d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 0x558dde65f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.837+0000 7fd667e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde658bc0 0x558dde65c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.837+0000 7fd66d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 0x558dde65f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48682/0 (socket says 172.21.15.27:48682) 2024-05-10T11:26:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.837+0000 7fd66d200700 1 -- 172.21.15.27:0/2515366930 learned_addr learned my addr 172.21.15.27:0/2515366930 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.837+0000 7fd66dc00700 1 --2- 172.21.15.27:0/2515366930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde65cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.837+0000 7fd667e00700 1 -- 172.21.15.27:0/2515366930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 msgr2=0x558dde65f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:13.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.837+0000 7fd667e00700 1 --2- 172.21.15.27:0/2515366930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 0x558dde65f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.837+0000 7fd667e00700 1 -- 172.21.15.27:0/2515366930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 msgr2=0x558dde65cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:13.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.837+0000 7fd667e00700 1 --2- 172.21.15.27:0/2515366930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde65cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.837+0000 7fd667e00700 1 -- 172.21.15.27:0/2515366930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558dde56b420 con 0x558dde658bc0 2024-05-10T11:26:13.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.838+0000 7fd667e00700 1 --2- 172.21.15.27:0/2515366930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde658bc0 0x558dde65c600 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd668103870 tx=0x7fd654012a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eb24226f7637230 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:13.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.838+0000 7fd64ea00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:26:13.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.838+0000 7fd64ea00700 1 -- 172.21.15.27:0/2515366930 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654004230 con 0x558dde658bc0 2024-05-10T11:26:13.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.838+0000 7fd64ea00700 1 -- 172.21.15.27:0/2515366930 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7fd6540043d0 con 0x558dde658bc0 2024-05-10T11:26:13.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.838+0000 7fd67a543b80 1 -- 172.21.15.27:0/2515366930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd65c0aa300 con 0x558dde658bc0 2024-05-10T11:26:13.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.838+0000 7fd64ea00700 1 -- 172.21.15.27:0/2515366930 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd654025050 con 0x558dde658bc0 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.838+0000 7fd67a543b80 10 client.27112 fetch_fsmap learned FSMap version 36 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.838+0000 7fd67a543b80 1 -- 172.21.15.27:0/2515366930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558dde66a5d0 con 0x558dde658bc0 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.839+0000 7fd64ea00700 1 -- 172.21.15.27:0/2515366930 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd65c0aa300 con 0x558dde658bc0 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.839+0000 7fd64ea00700 1 -- 172.21.15.27:0/2515366930 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fd654003b40 con 0x558dde658bc0 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.839+0000 7fd67a543b80 10 client.27112 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.839+0000 7fd67a543b80 10 client.27112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.839+0000 7fd67a543b80 20 client.27112 populate_metadata read hostname 'smithi027' 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.839+0000 7fd67a543b80 1 --2- 172.21.15.27:0/2515366930 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558dde66cc10 0x558dde66f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.839+0000 7fd67a543b80 4 client.27112 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.839+0000 7fd67a543b80 1 -- 172.21.15.27:0/2515366930 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558dde620140 con 0x558dde66cc10 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.839+0000 7fd66d200700 1 --2- 172.21.15.27:0/2515366930 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558dde66cc10 0x558dde66f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.840+0000 7fd66d200700 1 --2- 172.21.15.27:0/2515366930 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558dde66cc10 0x558dde66f000 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7575882818a7944a server_cookie=12af5bf7a2c090dc in_seq=0 out_seq=0 2024-05-10T11:26:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.840+0000 7fd64ea00700 10 client.27112 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd64ea00700 1 -- 172.21.15.27:0/2515366930 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558dde620140 con 0x558dde66cc10 2024-05-10T11:26:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd64ea00700 10 client.27112 handle_command_reply: tid=0 2024-05-10T11:26:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd67a543b80 1 client.27112 shutdown 2024-05-10T11:26:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd67a543b80 1 -- 172.21.15.27:0/2515366930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde658bc0 msgr2=0x558dde65c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd67a543b80 1 --2- 172.21.15.27:0/2515366930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde658bc0 0x558dde65c600 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd668103870 tx=0x7fd654012a00 comp rx=0 tx=0).stop 2024-05-10T11:26:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd67a543b80 1 -- 172.21.15.27:0/2515366930 shutdown_connections 2024-05-10T11:26:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd67a543b80 1 --2- 172.21.15.27:0/2515366930 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558dde66cc10 0x558dde66f000 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd67a543b80 1 --2- 172.21.15.27:0/2515366930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde65cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd67a543b80 1 --2- 172.21.15.27:0/2515366930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 0x558dde65f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd67a543b80 1 --2- 172.21.15.27:0/2515366930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde658bc0 0x558dde65c600 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd64ea00700 0 client.27112 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:13.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd67a543b80 1 -- 172.21.15.27:0/2515366930 >> 172.21.15.27:0/2515366930 conn(0x558dde5cb6d0 msgr2=0x558dde64d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:13.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd67a543b80 1 -- 172.21.15.27:0/2515366930 shutdown_connections 2024-05-10T11:26:13.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.842+0000 7fd67a543b80 1 -- 172.21.15.27:0/2515366930 wait complete. 2024-05-10T11:26:13.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.843+0000 7fd67a543b80 20 client.27112 trim_cache size 0 max 16384 2024-05-10T11:26:13.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.865+0000 7fd67a543b80 1 Processor -- start 2024-05-10T11:26:13.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.866+0000 7fd67a543b80 1 -- start start 2024-05-10T11:26:13.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.866+0000 7fd67a543b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde6538f0 0x558dde65cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.866+0000 7fd67a543b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 0x558dde668d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.866+0000 7fd67a543b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde64d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.866+0000 7fd67a543b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd6681b17b0 con 0x558dde6594e0 2024-05-10T11:26:13.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.866+0000 7fd67a543b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd6681b32f0 con 0x558dde658bc0 2024-05-10T11:26:13.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.866+0000 7fd67a543b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd6681b1e80 con 0x558dde6538f0 2024-05-10T11:26:13.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.866+0000 7fd667e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 0x558dde668d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.867+0000 7fd667e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 0x558dde668d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48690/0 (socket says 172.21.15.27:48690) 2024-05-10T11:26:13.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.867+0000 7fd667e00700 1 -- 172.21.15.27:0/330247921 learned_addr learned my addr 172.21.15.27:0/330247921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:13.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.867+0000 7fd66d200700 1 --2- 172.21.15.27:0/330247921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde6538f0 0x558dde65cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.867+0000 7fd66dc00700 1 --2- 172.21.15.27:0/330247921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde64d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.867+0000 7fd667e00700 1 -- 172.21.15.27:0/330247921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde6538f0 msgr2=0x558dde65cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.867+0000 7fd667e00700 1 --2- 172.21.15.27:0/330247921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde6538f0 0x558dde65cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.867+0000 7fd667e00700 1 -- 172.21.15.27:0/330247921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 msgr2=0x558dde64d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.867+0000 7fd667e00700 1 --2- 172.21.15.27:0/330247921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde64d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.867+0000 7fd667e00700 1 -- 172.21.15.27:0/330247921 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558dde66f660 con 0x558dde658bc0 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.867+0000 7fd66dc00700 1 --2- 172.21.15.27:0/330247921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde64d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.867+0000 7fd667e00700 1 --2- 172.21.15.27:0/330247921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 0x558dde668d30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd654001eb0 tx=0x7fd654005a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99d984ea47e6d4a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.868+0000 7fd659600700 1 -- 172.21.15.27:0/330247921 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65401c990 con 0x558dde658bc0 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.868+0000 7fd659600700 1 -- 172.21.15.27:0/330247921 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd65402e070 con 0x558dde658bc0 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.868+0000 7fd659600700 1 -- 172.21.15.27:0/330247921 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654010820 con 0x558dde658bc0 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.868+0000 7fd67a543b80 1 -- 172.21.15.27:0/330247921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 msgr2=0x558dde668d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.868+0000 7fd67a543b80 1 --2- 172.21.15.27:0/330247921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 0x558dde668d30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd654001eb0 tx=0x7fd654005a40 comp rx=0 tx=0).stop 2024-05-10T11:26:13.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.868+0000 7fd67a543b80 1 -- 172.21.15.27:0/330247921 shutdown_connections 2024-05-10T11:26:13.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.868+0000 7fd67a543b80 1 --2- 172.21.15.27:0/330247921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde6538f0 0x558dde65cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.868+0000 7fd67a543b80 1 --2- 172.21.15.27:0/330247921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde64d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.868+0000 7fd67a543b80 1 --2- 172.21.15.27:0/330247921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde658bc0 0x558dde668d30 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.868+0000 7fd67a543b80 1 -- 172.21.15.27:0/330247921 >> 172.21.15.27:0/330247921 conn(0x558dde5cb6d0 msgr2=0x558dde65e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:13.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.869+0000 7fd67a543b80 1 -- 172.21.15.27:0/330247921 shutdown_connections 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.869+0000 7fd67a543b80 1 -- 172.21.15.27:0/330247921 wait complete. 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.869+0000 7fd67a543b80 1 -- start start 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.869+0000 7fd67a543b80 1 Processor -- start 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd67a543b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 0x558dde6650b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd67a543b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde658bc0 0x558dde6655f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd67a543b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde662080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd67a543b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd6681b2c20 con 0x558dde6594e0 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd67a543b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd6681b2550 con 0x558dde6538f0 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd67a543b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd6681af770 con 0x558dde658bc0 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd67a543b80 10 client.0 _ll_register_callbacks cb 0x558dde54cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd66d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 0x558dde6650b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd66d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 0x558dde6650b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:48704/0 (socket says 172.21.15.27:48704) 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd66d200700 1 -- 172.21.15.27:0/896284627 learned_addr learned my addr 172.21.15.27:0/896284627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd667e00700 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde658bc0 0x558dde6655f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd66dc00700 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde662080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd66d200700 1 -- 172.21.15.27:0/896284627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde658bc0 msgr2=0x558dde6655f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd66d200700 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde658bc0 0x558dde6655f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd66d200700 1 -- 172.21.15.27:0/896284627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 msgr2=0x558dde662080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd66d200700 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde662080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.870+0000 7fd66d200700 1 -- 172.21.15.27:0/896284627 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558dde6625c0 con 0x558dde6538f0 2024-05-10T11:26:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.871+0000 7fd66d200700 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 0x558dde6650b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0aa4f0 tx=0x7fd65c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1e4b16472e57ecf server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.871+0000 7fd64ea00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:26:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.871+0000 7fd64ea00700 1 -- 172.21.15.27:0/896284627 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c0aa300 con 0x558dde6538f0 2024-05-10T11:26:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.871+0000 7fd64ea00700 1 -- 172.21.15.27:0/896284627 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7fd65c0e0e50 con 0x558dde6538f0 2024-05-10T11:26:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.871+0000 7fd67a543b80 1 -- 172.21.15.27:0/896284627 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd65402e070 con 0x558dde6538f0 2024-05-10T11:26:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.871+0000 7fd64ea00700 1 -- 172.21.15.27:0/896284627 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd65c0e1050 con 0x558dde6538f0 2024-05-10T11:26:13.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.871+0000 7fd67a543b80 10 client.27134 fetch_fsmap learned FSMap version 36 2024-05-10T11:26:13.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.871+0000 7fd67a543b80 1 -- 172.21.15.27:0/896284627 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558dde6e03c0 con 0x558dde6538f0 2024-05-10T11:26:13.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.871+0000 7fd64ea00700 1 -- 172.21.15.27:0/896284627 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd65402e070 con 0x558dde6538f0 2024-05-10T11:26:13.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.872+0000 7fd64ea00700 1 -- 172.21.15.27:0/896284627 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fd65c0aaa50 con 0x558dde6538f0 2024-05-10T11:26:13.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.872+0000 7fd67a543b80 10 client.27134 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:26:13.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.872+0000 7fd67a543b80 10 client.27134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:26:13.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.872+0000 7fd67a543b80 20 client.27134 populate_metadata read hostname 'smithi027' 2024-05-10T11:26:13.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.872+0000 7fd67a543b80 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558dde6e0750 0x558dde6e2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:13.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.872+0000 7fd67a543b80 4 client.27134 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:26:13.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.872+0000 7fd67a543b80 1 -- 172.21.15.27:0/896284627 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558dde5b3400 con 0x558dde6e0750 2024-05-10T11:26:13.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.872+0000 7fd667e00700 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558dde6e0750 0x558dde6e2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.873+0000 7fd667e00700 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558dde6e0750 0x558dde6e2b40 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9137e2819344e4e server_cookie=1c3766b601933134 in_seq=0 out_seq=0 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.873+0000 7fd64ea00700 10 client.27134 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd64ea00700 1 -- 172.21.15.27:0/896284627 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558dde5b3400 con 0x558dde6e0750 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd64ea00700 10 client.27134 handle_command_reply: tid=0 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd67a543b80 1 client.27134 shutdown 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd67a543b80 1 -- 172.21.15.27:0/896284627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 msgr2=0x558dde6650b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd67a543b80 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 0x558dde6650b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0aa4f0 tx=0x7fd65c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd67a543b80 1 -- 172.21.15.27:0/896284627 shutdown_connections 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd67a543b80 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558dde6e0750 0x558dde6e2b40 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd67a543b80 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558dde6594e0 0x558dde662080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd67a543b80 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558dde6538f0 0x558dde6650b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd64ea00700 0 client.27134 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd67a543b80 1 --2- 172.21.15.27:0/896284627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558dde658bc0 0x558dde6655f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd67a543b80 1 -- 172.21.15.27:0/896284627 >> 172.21.15.27:0/896284627 conn(0x558dde5cb6d0 msgr2=0x558dde6522c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:13.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd67a543b80 1 -- 172.21.15.27:0/896284627 shutdown_connections 2024-05-10T11:26:13.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd67a543b80 1 -- 172.21.15.27:0/896284627 wait complete. 2024-05-10T11:26:13.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.876+0000 7fd67a543b80 20 client.27134 trim_cache size 0 max 16384 2024-05-10T11:26:13.877 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:26:13.877 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:26:13.877 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:26:13.877 INFO:teuthology.orchestra.run.smithi027.stdout: "ac946942-0245-4d99-af0f-21494991a8b9": { 2024-05-10T11:26:13.877 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:26:13.877 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "ac946942-0245-4d99-af0f-21494991a8b9", 2024-05-10T11:26:13.878 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:26:13.878 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:26:13.878 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:26:13.878 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:26:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.877+0000 7fd66fc00700 1 -- 172.21.15.27:0/853037777 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd6500b2a70 msgr2=0x7fd6500b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.877+0000 7fd66fc00700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd6500b2a70 0x7fd6500b4f30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd6600094f0 tx=0x7fd660004b40 comp rx=0 tx=0).stop 2024-05-10T11:26:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.877+0000 7fd66fc00700 1 -- 172.21.15.27:0/853037777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6681056c0 msgr2=0x7fd66819bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.877+0000 7fd66fc00700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6681056c0 0x7fd66819bde0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd65c000c00 tx=0x7fd65c0047b0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.877+0000 7fd66fc00700 1 -- 172.21.15.27:0/853037777 shutdown_connections 2024-05-10T11:26:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.877+0000 7fd66fc00700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd6500b2a70 0x7fd6500b4f30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.877+0000 7fd66fc00700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6681056c0 0x7fd66819bde0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.877+0000 7fd66fc00700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd66810b6c0 0x7fd6681a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.877+0000 7fd66fc00700 1 --2- 172.21.15.27:0/853037777 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd668106080 0x7fd66819c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.877+0000 7fd66fc00700 1 -- 172.21.15.27:0/853037777 >> 172.21.15.27:0/853037777 conn(0x7fd6681011b0 msgr2=0x7fd6681019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.877+0000 7fd66fc00700 1 -- 172.21.15.27:0/853037777 shutdown_connections 2024-05-10T11:26:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:13.878+0000 7fd66fc00700 1 -- 172.21.15.27:0/853037777 wait complete. 2024-05-10T11:26:13.880 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:13 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:13.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:13 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:14.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:13 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:26:14.741 INFO:tasks.workunit.client.0.smithi027.stdout: 1 779120 6.15 MB/sec execute 539 sec latency 37.723 ms 2024-05-10T11:26:14.907 INFO:tasks.cephfs.filesystem:scrub status for tag:ac946942-0245-4d99-af0f-21494991a8b9 - {'path': '/', 'tag': 'ac946942-0245-4d99-af0f-21494991a8b9', 'options': 'recursive,force'} 2024-05-10T11:26:14.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:14 smithi189 ceph-mon[34842]: pgmap v407: 129 pgs: 129 active+clean; 610 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 166 op/s 2024-05-10T11:26:15.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:14 smithi027 ceph-mon[30054]: pgmap v407: 129 pgs: 129 active+clean; 610 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 166 op/s 2024-05-10T11:26:15.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:14 smithi076 ceph-mon[34152]: pgmap v407: 129 pgs: 129 active+clean; 610 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 166 op/s 2024-05-10T11:26:15.741 INFO:tasks.workunit.client.0.smithi027.stdout: 1 780584 6.15 MB/sec execute 540 sec latency 25.110 ms 2024-05-10T11:26:16.741 INFO:tasks.workunit.client.0.smithi027.stdout: 1 781489 6.15 MB/sec execute 541 sec latency 37.162 ms 2024-05-10T11:26:16.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:16 smithi189 ceph-mon[34842]: pgmap v408: 129 pgs: 129 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-05-10T11:26:16.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:16 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:26:17.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:16 smithi027 ceph-mon[30054]: pgmap v408: 129 pgs: 129 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-05-10T11:26:17.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:16 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:26:17.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:16 smithi076 ceph-mon[34152]: pgmap v408: 129 pgs: 129 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-05-10T11:26:17.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:16 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:26:17.741 INFO:tasks.workunit.client.0.smithi027.stdout: 1 782524 6.15 MB/sec execute 542 sec latency 50.229 ms 2024-05-10T11:26:18.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:17 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:26:18.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:17 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:26:18.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:17 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:26:18.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:17 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:26:18.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:17 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:26:18.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:17 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:26:18.741 INFO:tasks.workunit.client.0.smithi027.stdout: 1 783898 6.15 MB/sec execute 543 sec latency 27.666 ms 2024-05-10T11:26:18.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:18 smithi027 ceph-mon[30054]: pgmap v409: 129 pgs: 129 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-05-10T11:26:19.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:18 smithi076 ceph-mon[34152]: pgmap v409: 129 pgs: 129 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-05-10T11:26:19.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:18 smithi189 ceph-mon[34842]: pgmap v409: 129 pgs: 129 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-05-10T11:26:19.741 INFO:tasks.workunit.client.0.smithi027.stdout: 1 785038 6.15 MB/sec execute 544 sec latency 22.427 ms 2024-05-10T11:26:20.742 INFO:tasks.workunit.client.0.smithi027.stdout: 1 786137 6.15 MB/sec execute 545 sec latency 26.215 ms 2024-05-10T11:26:21.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:20 smithi027 ceph-mon[30054]: pgmap v410: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 169 op/s 2024-05-10T11:26:21.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:20 smithi076 ceph-mon[34152]: pgmap v410: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 169 op/s 2024-05-10T11:26:21.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:20 smithi189 ceph-mon[34842]: pgmap v410: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 169 op/s 2024-05-10T11:26:21.742 INFO:tasks.workunit.client.0.smithi027.stdout: 1 787672 6.15 MB/sec execute 546 sec latency 24.380 ms 2024-05-10T11:26:22.742 INFO:tasks.workunit.client.0.smithi027.stdout: 1 788377 6.15 MB/sec execute 547 sec latency 51.275 ms 2024-05-10T11:26:23.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:22 smithi027 ceph-mon[30054]: pgmap v411: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-05-10T11:26:23.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:22 smithi076 ceph-mon[34152]: pgmap v411: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-05-10T11:26:23.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:22 smithi189 ceph-mon[34842]: pgmap v411: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-05-10T11:26:23.742 INFO:tasks.workunit.client.0.smithi027.stdout: 1 789113 6.15 MB/sec execute 548 sec latency 43.270 ms 2024-05-10T11:26:24.742 INFO:tasks.workunit.client.0.smithi027.stdout: 1 790568 6.15 MB/sec execute 549 sec latency 22.382 ms 2024-05-10T11:26:25.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:24 smithi027 ceph-mon[30054]: pgmap v412: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-05-10T11:26:25.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:24 smithi076 ceph-mon[34152]: pgmap v412: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-05-10T11:26:25.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:24 smithi189 ceph-mon[34842]: pgmap v412: 129 pgs: 129 active+clean; 581 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-05-10T11:26:25.742 INFO:tasks.workunit.client.0.smithi027.stdout: 1 791852 6.15 MB/sec execute 550 sec latency 26.142 ms 2024-05-10T11:26:26.743 INFO:tasks.workunit.client.0.smithi027.stdout: 1 792230 6.15 MB/sec execute 551 sec latency 23.585 ms 2024-05-10T11:26:27.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:26 smithi027 ceph-mon[30054]: pgmap v413: 129 pgs: 129 active+clean; 589 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-05-10T11:26:27.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:26 smithi076 ceph-mon[34152]: pgmap v413: 129 pgs: 129 active+clean; 589 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-05-10T11:26:27.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:26 smithi189 ceph-mon[34842]: pgmap v413: 129 pgs: 129 active+clean; 589 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-05-10T11:26:27.743 INFO:tasks.workunit.client.0.smithi027.stdout: 1 793302 6.14 MB/sec execute 552 sec latency 79.686 ms 2024-05-10T11:26:28.743 INFO:tasks.workunit.client.0.smithi027.stdout: 1 794784 6.15 MB/sec execute 553 sec latency 25.877 ms 2024-05-10T11:26:29.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:28 smithi027 ceph-mon[30054]: pgmap v414: 129 pgs: 129 active+clean; 589 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 132 op/s 2024-05-10T11:26:29.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:28 smithi076 ceph-mon[34152]: pgmap v414: 129 pgs: 129 active+clean; 589 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 132 op/s 2024-05-10T11:26:29.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:28 smithi189 ceph-mon[34842]: pgmap v414: 129 pgs: 129 active+clean; 589 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 132 op/s 2024-05-10T11:26:29.743 INFO:tasks.workunit.client.0.smithi027.stdout: 1 795518 6.15 MB/sec execute 554 sec latency 30.695 ms 2024-05-10T11:26:30.743 INFO:tasks.workunit.client.0.smithi027.stdout: 1 796065 6.14 MB/sec execute 555 sec latency 33.103 ms 2024-05-10T11:26:31.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:30 smithi027 ceph-mon[30054]: pgmap v415: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 192 op/s 2024-05-10T11:26:31.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:30 smithi076 ceph-mon[34152]: pgmap v415: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 192 op/s 2024-05-10T11:26:31.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:30 smithi189 ceph-mon[34842]: pgmap v415: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 192 op/s 2024-05-10T11:26:31.744 INFO:tasks.workunit.client.0.smithi027.stdout: 1 797278 6.13 MB/sec execute 556 sec latency 25.101 ms 2024-05-10T11:26:32.744 INFO:tasks.workunit.client.0.smithi027.stdout: 1 798871 6.14 MB/sec execute 557 sec latency 23.112 ms 2024-05-10T11:26:33.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:32 smithi027 ceph-mon[30054]: pgmap v416: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-05-10T11:26:33.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:32 smithi076 ceph-mon[34152]: pgmap v416: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-05-10T11:26:33.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:32 smithi189 ceph-mon[34842]: pgmap v416: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-05-10T11:26:33.744 INFO:tasks.workunit.client.0.smithi027.stdout: 1 799539 6.14 MB/sec execute 558 sec latency 35.007 ms 2024-05-10T11:26:34.744 INFO:tasks.workunit.client.0.smithi027.stdout: 1 800777 6.14 MB/sec execute 559 sec latency 24.301 ms 2024-05-10T11:26:35.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:34 smithi027 ceph-mon[30054]: pgmap v417: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-05-10T11:26:35.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:34 smithi076 ceph-mon[34152]: pgmap v417: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-05-10T11:26:35.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:34 smithi189 ceph-mon[34842]: pgmap v417: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-05-10T11:26:35.744 INFO:tasks.workunit.client.0.smithi027.stdout: 1 802519 6.15 MB/sec execute 560 sec latency 23.851 ms 2024-05-10T11:26:36.744 INFO:tasks.workunit.client.0.smithi027.stdout: 1 803090 6.14 MB/sec execute 561 sec latency 51.485 ms 2024-05-10T11:26:37.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:36 smithi027 ceph-mon[30054]: pgmap v418: 129 pgs: 129 active+clean; 602 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 178 op/s 2024-05-10T11:26:37.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:36 smithi076 ceph-mon[34152]: pgmap v418: 129 pgs: 129 active+clean; 602 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 178 op/s 2024-05-10T11:26:37.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:36 smithi189 ceph-mon[34842]: pgmap v418: 129 pgs: 129 active+clean; 602 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 178 op/s 2024-05-10T11:26:37.744 INFO:tasks.workunit.client.0.smithi027.stdout: 1 804389 6.14 MB/sec execute 562 sec latency 49.321 ms 2024-05-10T11:26:38.745 INFO:tasks.workunit.client.0.smithi027.stdout: 1 806037 6.15 MB/sec execute 563 sec latency 26.246 ms 2024-05-10T11:26:39.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:38 smithi027 ceph-mon[30054]: pgmap v419: 129 pgs: 129 active+clean; 602 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 115 op/s 2024-05-10T11:26:39.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:38 smithi076 ceph-mon[34152]: pgmap v419: 129 pgs: 129 active+clean; 602 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 115 op/s 2024-05-10T11:26:39.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:38 smithi189 ceph-mon[34842]: pgmap v419: 129 pgs: 129 active+clean; 602 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 115 op/s 2024-05-10T11:26:39.745 INFO:tasks.workunit.client.0.smithi027.stdout: 1 806517 6.14 MB/sec execute 564 sec latency 30.710 ms 2024-05-10T11:26:40.745 INFO:tasks.workunit.client.0.smithi027.stdout: 1 807658 6.14 MB/sec execute 565 sec latency 26.932 ms 2024-05-10T11:26:41.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:40 smithi027 ceph-mon[30054]: pgmap v420: 129 pgs: 129 active+clean; 609 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 180 op/s 2024-05-10T11:26:41.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:40 smithi076 ceph-mon[34152]: pgmap v420: 129 pgs: 129 active+clean; 609 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 180 op/s 2024-05-10T11:26:41.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:40 smithi189 ceph-mon[34842]: pgmap v420: 129 pgs: 129 active+clean; 609 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 180 op/s 2024-05-10T11:26:41.746 INFO:tasks.workunit.client.0.smithi027.stdout: 1 809496 6.15 MB/sec execute 566 sec latency 22.246 ms 2024-05-10T11:26:42.746 INFO:tasks.workunit.client.0.smithi027.stdout: 1 809894 6.14 MB/sec execute 567 sec latency 25.673 ms 2024-05-10T11:26:43.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:42 smithi027 ceph-mon[30054]: pgmap v421: 129 pgs: 129 active+clean; 609 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 121 op/s 2024-05-10T11:26:43.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:42 smithi076 ceph-mon[34152]: pgmap v421: 129 pgs: 129 active+clean; 609 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 121 op/s 2024-05-10T11:26:43.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:42 smithi189 ceph-mon[34842]: pgmap v421: 129 pgs: 129 active+clean; 609 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 121 op/s 2024-05-10T11:26:43.746 INFO:tasks.workunit.client.0.smithi027.stdout: 1 810950 6.14 MB/sec execute 568 sec latency 40.411 ms 2024-05-10T11:26:44.747 INFO:tasks.workunit.client.0.smithi027.stdout: 1 812269 6.14 MB/sec execute 569 sec latency 24.376 ms 2024-05-10T11:26:44.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:26:45.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:44 smithi027 ceph-mon[30054]: pgmap v422: 129 pgs: 129 active+clean; 609 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 121 op/s 2024-05-10T11:26:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:44 smithi076 ceph-mon[34152]: pgmap v422: 129 pgs: 129 active+clean; 609 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 121 op/s 2024-05-10T11:26:45.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:44 smithi189 ceph-mon[34842]: pgmap v422: 129 pgs: 129 active+clean; 609 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 121 op/s 2024-05-10T11:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:26:45.747 INFO:tasks.workunit.client.0.smithi027.stdout: 1 813340 6.14 MB/sec execute 570 sec latency 21.408 ms 2024-05-10T11:26:46.747 INFO:tasks.workunit.client.0.smithi027.stdout: 1 814233 6.13 MB/sec execute 571 sec latency 33.016 ms 2024-05-10T11:26:47.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:46 smithi027 ceph-mon[30054]: pgmap v423: 129 pgs: 129 active+clean; 616 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 173 op/s 2024-05-10T11:26:47.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:46 smithi076 ceph-mon[34152]: pgmap v423: 129 pgs: 129 active+clean; 616 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 173 op/s 2024-05-10T11:26:47.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:46 smithi189 ceph-mon[34842]: pgmap v423: 129 pgs: 129 active+clean; 616 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 173 op/s 2024-05-10T11:26:47.750 INFO:tasks.workunit.client.0.smithi027.stdout: 1 815811 6.14 MB/sec execute 572 sec latency 24.800 ms 2024-05-10T11:26:48.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.367+0000 7f82f5200700 1 -- 172.21.15.27:0/468682644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f006bb40 msgr2=0x7f82f0110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:48.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.367+0000 7f82f5200700 1 --2- 172.21.15.27:0/468682644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f006bb40 0x7f82f0110330 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f82e800b0a0 tx=0x7f82e800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.367+0000 7f82f5200700 1 -- 172.21.15.27:0/468682644 shutdown_connections 2024-05-10T11:26:48.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.367+0000 7f82f5200700 1 --2- 172.21.15.27:0/468682644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f006bb40 0x7f82f0110330 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.367+0000 7f82f5200700 1 --2- 172.21.15.27:0/468682644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f0109510 0x7f82f01098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.367+0000 7f82f5200700 1 --2- 172.21.15.27:0/468682644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f006b180 0x7f82f006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.367+0000 7f82f5200700 1 -- 172.21.15.27:0/468682644 >> 172.21.15.27:0/468682644 conn(0x7f82f0079350 msgr2=0x7f82f0079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:48.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.367+0000 7f82f5200700 1 -- 172.21.15.27:0/468682644 shutdown_connections 2024-05-10T11:26:48.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.367+0000 7f82f5200700 1 -- 172.21.15.27:0/468682644 wait complete. 2024-05-10T11:26:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.369+0000 7f82f5200700 1 Processor -- start 2024-05-10T11:26:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.369+0000 7f82f5200700 1 -- start start 2024-05-10T11:26:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82f5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f006b180 0x7f82f019bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82f5200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f006bb40 0x7f82f019c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82f5200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f0109510 0x7f82f01a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82f5200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82f0113690 con 0x7f82f006b180 2024-05-10T11:26:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82f5200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f82f0113510 con 0x7f82f006bb40 2024-05-10T11:26:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82f5200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f82f0113810 con 0x7f82f0109510 2024-05-10T11:26:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82e5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f006bb40 0x7f82f019c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82ee800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f0109510 0x7f82f01a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82e5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f006bb40 0x7f82f019c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45836/0 (socket says 172.21.15.27:45836) 2024-05-10T11:26:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82e5400700 1 -- 172.21.15.27:0/1724590441 learned_addr learned my addr 172.21.15.27:0/1724590441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82ede00700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f006b180 0x7f82f019bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82e5400700 1 -- 172.21.15.27:0/1724590441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f0109510 msgr2=0x7f82f01a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82e5400700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f0109510 0x7f82f01a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82e5400700 1 -- 172.21.15.27:0/1724590441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f006b180 msgr2=0x7f82f019bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82e5400700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f006b180 0x7f82f019bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82e5400700 1 -- 172.21.15.27:0/1724590441 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f01a0d60 con 0x7f82f006bb40 2024-05-10T11:26:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.370+0000 7f82ee800700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f0109510 0x7f82f01a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:26:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.371+0000 7f82e5400700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f006bb40 0x7f82f019c290 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f82d800e9a0 tx=0x7f82d800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:48.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.371+0000 7f82e7400700 1 -- 172.21.15.27:0/1724590441 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d800ca10 con 0x7f82f006bb40 2024-05-10T11:26:48.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.371+0000 7f82e7400700 1 -- 172.21.15.27:0/1724590441 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d800cbb0 con 0x7f82f006bb40 2024-05-10T11:26:48.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.371+0000 7f82e7400700 1 -- 172.21.15.27:0/1724590441 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8018680 con 0x7f82f006bb40 2024-05-10T11:26:48.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.371+0000 7f82f5200700 1 -- 172.21.15.27:0/1724590441 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f01a1050 con 0x7f82f006bb40 2024-05-10T11:26:48.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.371+0000 7f82f5200700 1 -- 172.21.15.27:0/1724590441 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f01a88d0 con 0x7f82f006bb40 2024-05-10T11:26:48.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.380+0000 7f82ffb6eb80 1 Processor -- start 2024-05-10T11:26:48.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.381+0000 7f82ffb6eb80 1 -- start start 2024-05-10T11:26:48.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.381+0000 7f82e7400700 1 -- 172.21.15.27:0/1724590441 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f82d8010430 con 0x7f82f006bb40 2024-05-10T11:26:48.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.382+0000 7f82ffb6eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c773bc0 0x56004c773fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.382+0000 7f82ffb6eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c7744e0 0x56004c76e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.382+0000 7f82ffb6eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c76e8f0 0x56004c76ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.382+0000 7f82ffb6eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56004c6c0a20 con 0x56004c7744e0 2024-05-10T11:26:48.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.382+0000 7f82ffb6eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56004c65e760 con 0x56004c773bc0 2024-05-10T11:26:48.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.382+0000 7f82ffb6eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56004c6bc9e0 con 0x56004c76e8f0 2024-05-10T11:26:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.383+0000 7f82e5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c76e8f0 0x56004c76ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.383+0000 7f82ee800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c773bc0 0x56004c773fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.383+0000 7f82e5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c76e8f0 0x56004c76ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:52242/0 (socket says 172.21.15.27:52242) 2024-05-10T11:26:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.383+0000 7f82e5400700 1 -- 172.21.15.27:0/2952443478 learned_addr learned my addr 172.21.15.27:0/2952443478 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.383+0000 7f82e5400700 1 -- 172.21.15.27:0/2952443478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c773bc0 msgr2=0x56004c773fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.383+0000 7f82e5400700 1 --2- 172.21.15.27:0/2952443478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c773bc0 0x56004c773fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.383+0000 7f82e5400700 1 -- 172.21.15.27:0/2952443478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c7744e0 msgr2=0x56004c76e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.383+0000 7f82e5400700 1 --2- 172.21.15.27:0/2952443478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c7744e0 0x56004c76e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.383+0000 7f82e5400700 1 -- 172.21.15.27:0/2952443478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56004c60d8d0 con 0x56004c76e8f0 2024-05-10T11:26:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.384+0000 7f82e5400700 1 --2- 172.21.15.27:0/2952443478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c76e8f0 0x56004c76ed70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f82d80dfda0 tx=0x7f82d80e6f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93f38d7265cadde7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.384+0000 7f82df400700 1 -- 172.21.15.27:0/2952443478 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d80e8050 con 0x56004c76e8f0 2024-05-10T11:26:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.384+0000 7f82df400700 1 -- 172.21.15.27:0/2952443478 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d80f4070 con 0x56004c76e8f0 2024-05-10T11:26:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.384+0000 7f82df400700 1 -- 172.21.15.27:0/2952443478 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d80e8050 con 0x56004c76e8f0 2024-05-10T11:26:48.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.385+0000 7f82e7400700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f82d40b2af0 0x7f82d40b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.385+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2952443478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c76e8f0 msgr2=0x56004c76ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:48.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.385+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/2952443478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c76e8f0 0x56004c76ed70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f82d80dfda0 tx=0x7f82d80e6f70 comp rx=0 tx=0).stop 2024-05-10T11:26:48.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.385+0000 7f82e7400700 1 -- 172.21.15.27:0/1724590441 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7f82d80e2ba0 con 0x7f82f006bb40 2024-05-10T11:26:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.386+0000 7f82ee800700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f82d40b2af0 0x7f82d40b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.386+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2952443478 shutdown_connections 2024-05-10T11:26:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.386+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/2952443478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c76e8f0 0x56004c76ed70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.386+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/2952443478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c773bc0 0x56004c773fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.386+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/2952443478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c7744e0 0x56004c76e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.386+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2952443478 >> 172.21.15.27:0/2952443478 conn(0x56004c6e66d0 msgr2=0x56004c768b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.386+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2952443478 shutdown_connections 2024-05-10T11:26:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.387+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2952443478 wait complete. 2024-05-10T11:26:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.387+0000 7f82ee800700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f82d40b2af0 0x7f82d40b4fb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f82e800b780 tx=0x7f82e801d040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.388+0000 7f82ffb6eb80 1 -- start start 2024-05-10T11:26:48.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.388+0000 7f82ffb6eb80 1 Processor -- start 2024-05-10T11:26:48.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.389+0000 7f82ffb6eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c76e8f0 0x56004c782900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.389+0000 7f82ffb6eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c77fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.389+0000 7f82ffb6eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c7744e0 0x56004c780250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.389+0000 7f82ffb6eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56004c7836d0 con 0x56004c76e8f0 2024-05-10T11:26:48.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.389+0000 7f82ffb6eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56004c626b20 con 0x56004c7744e0 2024-05-10T11:26:48.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.389+0000 7f82ffb6eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56004c783850 con 0x56004c773bc0 2024-05-10T11:26:48.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.389+0000 7f82ffb6eb80 10 client.0 _ll_register_callbacks cb 0x56004c667c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:26:48.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.390+0000 7f82e5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c77fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.390+0000 7f82e5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c77fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:52246/0 (socket says 172.21.15.27:52246) 2024-05-10T11:26:48.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.390+0000 7f82e5400700 1 -- 172.21.15.27:0/2984037263 learned_addr learned my addr 172.21.15.27:0/2984037263 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:48.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.390+0000 7f82ede00700 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c76e8f0 0x56004c782900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.390+0000 7f82ee800700 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c7744e0 0x56004c780250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.391+0000 7f82ede00700 1 -- 172.21.15.27:0/2984037263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 msgr2=0x56004c77fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:48.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.391+0000 7f82ede00700 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c77fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.391+0000 7f82ede00700 1 -- 172.21.15.27:0/2984037263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c7744e0 msgr2=0x56004c780250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:48.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.391+0000 7f82ede00700 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c7744e0 0x56004c780250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.391+0000 7f82ede00700 1 -- 172.21.15.27:0/2984037263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56004c686420 con 0x56004c76e8f0 2024-05-10T11:26:48.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.393+0000 7f82ede00700 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c76e8f0 0x56004c782900 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x56004c7748e0 tx=0x7f82e0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188cfc113dbe2c65 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:48.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.393+0000 7f82d3e00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:26:48.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.394+0000 7f82d3e00700 1 -- 172.21.15.27:0/2984037263 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e0004830 con 0x56004c76e8f0 2024-05-10T11:26:48.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.394+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2984037263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82d80f4070 con 0x56004c76e8f0 2024-05-10T11:26:48.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.395+0000 7f82d3e00700 1 -- 172.21.15.27:0/2984037263 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7f82e00049d0 con 0x56004c76e8f0 2024-05-10T11:26:48.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.395+0000 7f82d3e00700 1 -- 172.21.15.27:0/2984037263 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f82e0024050 con 0x56004c76e8f0 2024-05-10T11:26:48.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.395+0000 7f82ffb6eb80 10 client.16884 fetch_fsmap learned FSMap version 36 2024-05-10T11:26:48.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.396+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2984037263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56004c78ddf0 con 0x56004c76e8f0 2024-05-10T11:26:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.397+0000 7f82d3e00700 1 -- 172.21.15.27:0/2984037263 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d80f4070 con 0x56004c76e8f0 2024-05-10T11:26:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.397+0000 7f82d3e00700 1 -- 172.21.15.27:0/2984037263 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f82e0019760 con 0x56004c76e8f0 2024-05-10T11:26:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.398+0000 7f82ffb6eb80 10 client.16884 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:26:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.398+0000 7f82ffb6eb80 10 client.16884 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:26:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.398+0000 7f82ffb6eb80 20 client.16884 populate_metadata read hostname 'smithi027' 2024-05-10T11:26:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.398+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x56004c790430 0x56004c792820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.398+0000 7f82ffb6eb80 4 client.16884 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:26:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.398+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2984037263 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56004c73b140 con 0x56004c790430 2024-05-10T11:26:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.398+0000 7f82e5400700 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x56004c790430 0x56004c792820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.399+0000 7f82d3e00700 10 client.16884 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.400+0000 7f82e5400700 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x56004c790430 0x56004c792820 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dba32916fb34176e server_cookie=95c0390626c0f2e2 in_seq=0 out_seq=0 2024-05-10T11:26:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.402+0000 7f82d3e00700 1 -- 172.21.15.27:0/2984037263 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56004c73b140 con 0x56004c790430 2024-05-10T11:26:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.402+0000 7f82d3e00700 10 client.16884 handle_command_reply: tid=0 2024-05-10T11:26:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.402+0000 7f82ffb6eb80 1 client.16884 shutdown 2024-05-10T11:26:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.402+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2984037263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c76e8f0 msgr2=0x56004c782900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.402+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c76e8f0 0x56004c782900 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x56004c7748e0 tx=0x7f82e0011ac0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.403+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2984037263 shutdown_connections 2024-05-10T11:26:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.403+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x56004c790430 0x56004c792820 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.403+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c7744e0 0x56004c780250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.403+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c76e8f0 0x56004c782900 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.403+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/2984037263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c77fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.403+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2984037263 >> 172.21.15.27:0/2984037263 conn(0x56004c6e66d0 msgr2=0x56004c768190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.403+0000 7f82d3e00700 0 client.16884 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.403+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2984037263 shutdown_connections 2024-05-10T11:26:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.403+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/2984037263 wait complete. 2024-05-10T11:26:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.404+0000 7f82ffb6eb80 20 client.16884 trim_cache size 0 max 16384 2024-05-10T11:26:48.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.448+0000 7f82ffb6eb80 1 Processor -- start 2024-05-10T11:26:48.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.448+0000 7f82ffb6eb80 1 -- start start 2024-05-10T11:26:48.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.448+0000 7f82ffb6eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c76e8f0 0x56004c780280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.448+0000 7f82ffb6eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c78c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.448+0000 7f82ffb6eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c7744e0 0x56004c768910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.448+0000 7f82ffb6eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82f01ad5c0 con 0x56004c7744e0 2024-05-10T11:26:48.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.448+0000 7f82ffb6eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f82f01af100 con 0x56004c76e8f0 2024-05-10T11:26:48.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.448+0000 7f82ffb6eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f82f01adc90 con 0x56004c773bc0 2024-05-10T11:26:48.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.449+0000 7f82ede00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c76e8f0 0x56004c780280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.449+0000 7f82e5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c78c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.449+0000 7f82ede00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c76e8f0 0x56004c780280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45850/0 (socket says 172.21.15.27:45850) 2024-05-10T11:26:48.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.449+0000 7f82ede00700 1 -- 172.21.15.27:0/4284330556 learned_addr learned my addr 172.21.15.27:0/4284330556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:48.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.449+0000 7f82ee800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c7744e0 0x56004c768910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.449+0000 7f82ede00700 1 -- 172.21.15.27:0/4284330556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 msgr2=0x56004c78c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:48.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.449+0000 7f82ede00700 1 --2- 172.21.15.27:0/4284330556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c78c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.449+0000 7f82ede00700 1 -- 172.21.15.27:0/4284330556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c7744e0 msgr2=0x56004c768910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:48.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.449+0000 7f82ede00700 1 --2- 172.21.15.27:0/4284330556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c7744e0 0x56004c768910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.449+0000 7f82ede00700 1 -- 172.21.15.27:0/4284330556 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56004c77ec70 con 0x56004c76e8f0 2024-05-10T11:26:48.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.450+0000 7f82ede00700 1 --2- 172.21.15.27:0/4284330556 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c76e8f0 0x56004c780280 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f82e0005950 tx=0x7f82e0005d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=629869c2fe2f08bf server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:48.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.450+0000 7f82dea00700 1 -- 172.21.15.27:0/4284330556 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e0000e60 con 0x56004c76e8f0 2024-05-10T11:26:48.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.450+0000 7f82dea00700 1 -- 172.21.15.27:0/4284330556 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82e000f5a0 con 0x56004c76e8f0 2024-05-10T11:26:48.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.450+0000 7f82dea00700 1 -- 172.21.15.27:0/4284330556 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e0028650 con 0x56004c76e8f0 2024-05-10T11:26:48.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.451+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4284330556 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c76e8f0 msgr2=0x56004c780280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:48.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.451+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/4284330556 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c76e8f0 0x56004c780280 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f82e0005950 tx=0x7f82e0005d60 comp rx=0 tx=0).stop 2024-05-10T11:26:48.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.451+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4284330556 shutdown_connections 2024-05-10T11:26:48.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.451+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/4284330556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c7744e0 0x56004c768910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.451+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/4284330556 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c76e8f0 0x56004c780280 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.451+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/4284330556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c78c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.451+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4284330556 >> 172.21.15.27:0/4284330556 conn(0x56004c6e66d0 msgr2=0x56004c7725c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:48.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.451+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4284330556 shutdown_connections 2024-05-10T11:26:48.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.451+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4284330556 wait complete. 2024-05-10T11:26:48.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.452+0000 7f82ffb6eb80 1 -- start start 2024-05-10T11:26:48.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.452+0000 7f82ffb6eb80 1 Processor -- start 2024-05-10T11:26:48.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ffb6eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c76e8f0 0x56004c797b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ffb6eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c7980c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ffb6eb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c7744e0 0x56004c798640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ffb6eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82f01aea30 con 0x56004c76e8f0 2024-05-10T11:26:48.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ffb6eb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f82f01ae360 con 0x56004c7744e0 2024-05-10T11:26:48.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ffb6eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f82f01ab580 con 0x56004c773bc0 2024-05-10T11:26:48.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ffb6eb80 10 client.0 _ll_register_callbacks cb 0x56004c76acd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:26:48.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82e5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c7980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ede00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c76e8f0 0x56004c797b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ee800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c7744e0 0x56004c798640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82e5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c7980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:52256/0 (socket says 172.21.15.27:52256) 2024-05-10T11:26:48.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82e5400700 1 -- 172.21.15.27:0/4057863383 learned_addr learned my addr 172.21.15.27:0/4057863383 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:48.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ee800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c7744e0 0x56004c798640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45864/0 (socket says 172.21.15.27:45864) 2024-05-10T11:26:48.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ee800700 1 -- 172.21.15.27:0/4057863383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 msgr2=0x56004c7980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:48.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ee800700 1 --2- 172.21.15.27:0/4057863383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c7980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ee800700 1 -- 172.21.15.27:0/4057863383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c76e8f0 msgr2=0x56004c797b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:48.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.453+0000 7f82ee800700 1 --2- 172.21.15.27:0/4057863383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c76e8f0 0x56004c797b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.454+0000 7f82ee800700 1 -- 172.21.15.27:0/4057863383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56004c7f9f30 con 0x56004c7744e0 2024-05-10T11:26:48.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.454+0000 7f82ee800700 1 --2- 172.21.15.27:0/4057863383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c7744e0 0x56004c798640 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f82e80062a0 tx=0x7f82e80099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57f69ef1a0a128c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:48.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.454+0000 7f82d3e00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:26:48.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.454+0000 7f82d3e00700 1 -- 172.21.15.27:0/4057863383 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8036060 con 0x56004c7744e0 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.454+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4057863383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82e000f5a0 con 0x56004c7744e0 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.454+0000 7f82d3e00700 1 -- 172.21.15.27:0/4057863383 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7f82e8022070 con 0x56004c7744e0 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.454+0000 7f82d3e00700 1 -- 172.21.15.27:0/4057863383 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f82e803d050 con 0x56004c7744e0 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.454+0000 7f82ffb6eb80 10 client.27149 fetch_fsmap learned FSMap version 36 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.454+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4057863383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56004c7fb290 con 0x56004c7744e0 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.455+0000 7f82d3e00700 1 -- 172.21.15.27:0/4057863383 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82e000f5a0 con 0x56004c7744e0 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.455+0000 7f82d3e00700 1 -- 172.21.15.27:0/4057863383 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f82e8011070 con 0x56004c7744e0 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.455+0000 7f82ffb6eb80 10 client.27149 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.455+0000 7f82ffb6eb80 10 client.27149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.455+0000 7f82ffb6eb80 20 client.27149 populate_metadata read hostname 'smithi027' 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.455+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/4057863383 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x56004c7fb8f0 0x56004c7fdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.455+0000 7f82ffb6eb80 4 client.27149 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:26:48.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.455+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4057863383 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56004c6ce400 con 0x56004c7fb8f0 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.455+0000 7f82ede00700 1 --2- 172.21.15.27:0/4057863383 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x56004c7fb8f0 0x56004c7fdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82d3e00700 10 client.27149 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82ede00700 1 --2- 172.21.15.27:0/4057863383 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x56004c7fb8f0 0x56004c7fdd10 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb5c70127615b0d1 server_cookie=9d3448da32674b75 in_seq=0 out_seq=0 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82d3e00700 1 -- 172.21.15.27:0/4057863383 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56004c6ce400 con 0x56004c7fb8f0 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82d3e00700 10 client.27149 handle_command_reply: tid=0 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82ffb6eb80 1 client.27149 shutdown 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4057863383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c7744e0 msgr2=0x56004c798640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/4057863383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c7744e0 0x56004c798640 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f82e80062a0 tx=0x7f82e80099e0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4057863383 shutdown_connections 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/4057863383 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x56004c7fb8f0 0x56004c7fdd10 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/4057863383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56004c7744e0 0x56004c798640 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/4057863383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56004c76e8f0 0x56004c797b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82ffb6eb80 1 --2- 172.21.15.27:0/4057863383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56004c773bc0 0x56004c7980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.456+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4057863383 >> 172.21.15.27:0/4057863383 conn(0x56004c6e66d0 msgr2=0x56004c781ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:48.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.457+0000 7f82d3e00700 0 client.27149 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:26:48.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.457+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4057863383 shutdown_connections 2024-05-10T11:26:48.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.457+0000 7f82ffb6eb80 1 -- 172.21.15.27:0/4057863383 wait complete. 2024-05-10T11:26:48.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.457+0000 7f82ffb6eb80 20 client.27149 trim_cache size 0 max 16384 2024-05-10T11:26:48.459 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:26:48.459 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:26:48.459 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:26:48.459 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:26:48.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.458+0000 7f82f5200700 1 -- 172.21.15.27:0/1724590441 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f82d40b2af0 msgr2=0x7f82d40b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:48.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.458+0000 7f82f5200700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f82d40b2af0 0x7f82d40b4fb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f82e800b780 tx=0x7f82e801d040 comp rx=0 tx=0).stop 2024-05-10T11:26:48.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.458+0000 7f82f5200700 1 -- 172.21.15.27:0/1724590441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f006bb40 msgr2=0x7f82f019c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:48.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.458+0000 7f82f5200700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f006bb40 0x7f82f019c290 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f82d800e9a0 tx=0x7f82d800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.458+0000 7f82f5200700 1 -- 172.21.15.27:0/1724590441 shutdown_connections 2024-05-10T11:26:48.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.458+0000 7f82f5200700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f82d40b2af0 0x7f82d40b4fb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.458+0000 7f82f5200700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f0109510 0x7f82f01a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.458+0000 7f82f5200700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f006b180 0x7f82f019bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.458+0000 7f82f5200700 1 --2- 172.21.15.27:0/1724590441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f006bb40 0x7f82f019c290 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:48.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.458+0000 7f82f5200700 1 -- 172.21.15.27:0/1724590441 >> 172.21.15.27:0/1724590441 conn(0x7f82f0079350 msgr2=0x7f82f010e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:48.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.458+0000 7f82f5200700 1 -- 172.21.15.27:0/1724590441 shutdown_connections 2024-05-10T11:26:48.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:48.458+0000 7f82f5200700 1 -- 172.21.15.27:0/1724590441 wait complete. 2024-05-10T11:26:48.749 INFO:tasks.workunit.client.0.smithi027.stdout: 1 816935 6.14 MB/sec execute 573 sec latency 32.290 ms 2024-05-10T11:26:49.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:48 smithi027 ceph-mon[30054]: pgmap v424: 129 pgs: 129 active+clean; 616 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 118 op/s 2024-05-10T11:26:49.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:48 smithi076 ceph-mon[34152]: pgmap v424: 129 pgs: 129 active+clean; 616 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 118 op/s 2024-05-10T11:26:49.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:48 smithi189 ceph-mon[34842]: pgmap v424: 129 pgs: 129 active+clean; 616 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 118 op/s 2024-05-10T11:26:49.336 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:26:49.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:26:49.749 INFO:tasks.workunit.client.0.smithi027.stdout: 1 818028 6.14 MB/sec execute 574 sec latency 34.181 ms 2024-05-10T11:26:49.815 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:26:50.750 INFO:tasks.workunit.client.0.smithi027.stdout: 1 819524 6.14 MB/sec execute 575 sec latency 24.174 ms 2024-05-10T11:26:51.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:50 smithi027 ceph-mon[30054]: pgmap v425: 129 pgs: 129 active+clean; 623 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 179 op/s 2024-05-10T11:26:51.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:50 smithi076 ceph-mon[34152]: pgmap v425: 129 pgs: 129 active+clean; 623 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 179 op/s 2024-05-10T11:26:51.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:50 smithi189 ceph-mon[34842]: pgmap v425: 129 pgs: 129 active+clean; 623 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 179 op/s 2024-05-10T11:26:51.750 INFO:tasks.workunit.client.0.smithi027.stdout: 1 820457 6.14 MB/sec execute 576 sec latency 30.087 ms 2024-05-10T11:26:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.354+0000 7f7f57200700 1 -- 172.21.15.27:0/3736352005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f58104eb0 msgr2=0x7f7f5810d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.354+0000 7f7f57200700 1 --2- 172.21.15.27:0/3736352005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f58104eb0 0x7f7f5810d3a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7f40009990 tx=0x7f7f40009ca0 comp rx=0 tx=0).stop 2024-05-10T11:26:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.354+0000 7f7f57200700 1 -- 172.21.15.27:0/3736352005 shutdown_connections 2024-05-10T11:26:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.354+0000 7f7f57200700 1 --2- 172.21.15.27:0/3736352005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f58104590 0x7f7f58104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.354+0000 7f7f57200700 1 --2- 172.21.15.27:0/3736352005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5810d9c0 0x7f7f5810fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.354+0000 7f7f57200700 1 --2- 172.21.15.27:0/3736352005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f58104eb0 0x7f7f5810d3a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.354+0000 7f7f57200700 1 -- 172.21.15.27:0/3736352005 >> 172.21.15.27:0/3736352005 conn(0x7f7f580fddc0 msgr2=0x7f7f581001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.354+0000 7f7f57200700 1 -- 172.21.15.27:0/3736352005 shutdown_connections 2024-05-10T11:26:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.354+0000 7f7f57200700 1 -- 172.21.15.27:0/3736352005 wait complete. 2024-05-10T11:26:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.355+0000 7f7f57200700 1 Processor -- start 2024-05-10T11:26:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.356+0000 7f7f57200700 1 -- start start 2024-05-10T11:26:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.356+0000 7f7f57200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f58104590 0x7f7f581021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.356+0000 7f7f57200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f58104eb0 0x7f7f58102730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.356+0000 7f7f57200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5810d9c0 0x7f7f58102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.356+0000 7f7f57200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f58113600 con 0x7f7f58104eb0 2024-05-10T11:26:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.356+0000 7f7f57200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7f58113480 con 0x7f7f5810d9c0 2024-05-10T11:26:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.356+0000 7f7f57200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7f58113780 con 0x7f7f58104590 2024-05-10T11:26:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.356+0000 7f7f56800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5810d9c0 0x7f7f58102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.356+0000 7f7f4d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f58104eb0 0x7f7f58102730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.356+0000 7f7f55e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f58104590 0x7f7f581021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.356+0000 7f7f56800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5810d9c0 0x7f7f58102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:45884/0 (socket says 172.21.15.27:45884) 2024-05-10T11:26:52.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.356+0000 7f7f56800700 1 -- 172.21.15.27:0/3183425994 learned_addr learned my addr 172.21.15.27:0/3183425994 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:52.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.357+0000 7f7f56800700 1 -- 172.21.15.27:0/3183425994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f58104590 msgr2=0x7f7f581021f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:52.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.357+0000 7f7f56800700 1 --2- 172.21.15.27:0/3183425994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f58104590 0x7f7f581021f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:52.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.357+0000 7f7f56800700 1 -- 172.21.15.27:0/3183425994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f58104eb0 msgr2=0x7f7f58102730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:52.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.357+0000 7f7f56800700 1 --2- 172.21.15.27:0/3183425994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f58104eb0 0x7f7f58102730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:52.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.357+0000 7f7f56800700 1 -- 172.21.15.27:0/3183425994 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f581a8060 con 0x7f7f5810d9c0 2024-05-10T11:26:52.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.357+0000 7f7f56800700 1 --2- 172.21.15.27:0/3183425994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5810d9c0 0x7f7f58102c70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7f5000ef30 tx=0x7f7f5000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:52.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.357+0000 7f7f4f400700 1 -- 172.21.15.27:0/3183425994 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f500092e0 con 0x7f7f5810d9c0 2024-05-10T11:26:52.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.358+0000 7f7f4f400700 1 -- 172.21.15.27:0/3183425994 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f50004bb0 con 0x7f7f5810d9c0 2024-05-10T11:26:52.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.358+0000 7f7f4f400700 1 -- 172.21.15.27:0/3183425994 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f50004020 con 0x7f7f5810d9c0 2024-05-10T11:26:52.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.358+0000 7f7f57200700 1 -- 172.21.15.27:0/3183425994 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f581a82f0 con 0x7f7f5810d9c0 2024-05-10T11:26:52.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.358+0000 7f7f57200700 1 -- 172.21.15.27:0/3183425994 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f581a87b0 con 0x7f7f5810d9c0 2024-05-10T11:26:52.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.359+0000 7f7f57200700 1 -- 172.21.15.27:0/3183425994 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f5806a190 con 0x7f7f5810d9c0 2024-05-10T11:26:52.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.362+0000 7f7f4f400700 1 -- 172.21.15.27:0/3183425994 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f7f5001b030 con 0x7f7f5810d9c0 2024-05-10T11:26:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.364+0000 7f7f4f400700 1 --2- 172.21.15.27:0/3183425994 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7f3c0b2a70 0x7f7f3c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.364+0000 7f7f4f400700 1 -- 172.21.15.27:0/3183425994 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7f7f500e2490 con 0x7f7f5810d9c0 2024-05-10T11:26:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.365+0000 7f7f55e00700 1 --2- 172.21.15.27:0/3183425994 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7f3c0b2a70 0x7f7f3c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.367+0000 7f7f55e00700 1 --2- 172.21.15.27:0/3183425994 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7f3c0b2a70 0x7f7f3c0b4f30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009c60 tx=0x7f7f48009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:52.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.369+0000 7f7f4f400700 1 -- 172.21.15.27:0/3183425994 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7f50022020 con 0x7f7f5810d9c0 2024-05-10T11:26:52.750 INFO:tasks.workunit.client.0.smithi027.stdout: 1 821271 6.14 MB/sec execute 577 sec latency 38.372 ms 2024-05-10T11:26:52.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.803+0000 7f7f57200700 1 -- 172.21.15.27:0/3183425994 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f5802e510 con 0x7f7f5810d9c0 2024-05-10T11:26:52.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.804+0000 7f7f4f400700 1 -- 172.21.15.27:0/3183425994 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11936 (secure 0 0 0) 0x7f7f50010040 con 0x7f7f5810d9c0 2024-05-10T11:26:52.804 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:26:52.804 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:23:14.174773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24484,24527,24472,24488,24467]},"id":1}]} 2024-05-10T11:26:52.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.807+0000 7f7f57200700 1 -- 172.21.15.27:0/3183425994 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7f3c0b2a70 msgr2=0x7f7f3c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:52.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.807+0000 7f7f57200700 1 --2- 172.21.15.27:0/3183425994 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7f3c0b2a70 0x7f7f3c0b4f30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009c60 tx=0x7f7f48009260 comp rx=0 tx=0).stop 2024-05-10T11:26:52.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.807+0000 7f7f57200700 1 -- 172.21.15.27:0/3183425994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5810d9c0 msgr2=0x7f7f58102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:52.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.807+0000 7f7f57200700 1 --2- 172.21.15.27:0/3183425994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5810d9c0 0x7f7f58102c70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7f5000ef30 tx=0x7f7f5000c3a0 comp rx=0 tx=0).stop 2024-05-10T11:26:52.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.807+0000 7f7f57200700 1 -- 172.21.15.27:0/3183425994 shutdown_connections 2024-05-10T11:26:52.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.807+0000 7f7f57200700 1 --2- 172.21.15.27:0/3183425994 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7f3c0b2a70 0x7f7f3c0b4f30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:52.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.807+0000 7f7f57200700 1 --2- 172.21.15.27:0/3183425994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f58104590 0x7f7f581021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:52.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.807+0000 7f7f57200700 1 --2- 172.21.15.27:0/3183425994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5810d9c0 0x7f7f58102c70 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:52.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.807+0000 7f7f57200700 1 --2- 172.21.15.27:0/3183425994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f58104eb0 0x7f7f58102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:52.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.808+0000 7f7f57200700 1 -- 172.21.15.27:0/3183425994 >> 172.21.15.27:0/3183425994 conn(0x7f7f580fddc0 msgr2=0x7f7f580feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:52.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.808+0000 7f7f57200700 1 -- 172.21.15.27:0/3183425994 shutdown_connections 2024-05-10T11:26:52.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:52.808+0000 7f7f57200700 1 -- 172.21.15.27:0/3183425994 wait complete. 2024-05-10T11:26:52.808 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-05-10T11:26:53.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:52 smithi027 ceph-mon[30054]: pgmap v426: 129 pgs: 129 active+clean; 623 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-05-10T11:26:53.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:52 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3183425994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:26:53.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:52 smithi076 ceph-mon[34152]: pgmap v426: 129 pgs: 129 active+clean; 623 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-05-10T11:26:53.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:52 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3183425994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:26:53.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:52 smithi189 ceph-mon[34842]: pgmap v426: 129 pgs: 129 active+clean; 623 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-05-10T11:26:53.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:52 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3183425994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:26:53.690 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:26:53.750 INFO:tasks.workunit.client.0.smithi027.stdout: 1 822589 6.14 MB/sec execute 578 sec latency 43.858 ms 2024-05-10T11:26:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:26:54.751 INFO:tasks.workunit.client.0.smithi027.stdout: 1 823925 6.14 MB/sec execute 579 sec latency 29.930 ms 2024-05-10T11:26:55.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:54 smithi027 ceph-mon[30054]: pgmap v427: 129 pgs: 129 active+clean; 623 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-05-10T11:26:55.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:54 smithi076 ceph-mon[34152]: pgmap v427: 129 pgs: 129 active+clean; 623 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-05-10T11:26:55.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:54 smithi189 ceph-mon[34842]: pgmap v427: 129 pgs: 129 active+clean; 623 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-05-10T11:26:55.751 INFO:tasks.workunit.client.0.smithi027.stdout: 1 824570 6.14 MB/sec execute 580 sec latency 39.287 ms 2024-05-10T11:26:56.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.481+0000 7f0bf8800700 1 -- 172.21.15.27:0/1137894281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bf010a490 msgr2=0x7f0bf01169a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:56.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.481+0000 7f0bf8800700 1 --2- 172.21.15.27:0/1137894281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bf010a490 0x7f0bf01169a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00b0a0 tx=0x7f0bec00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.481+0000 7f0bf8800700 1 -- 172.21.15.27:0/1137894281 shutdown_connections 2024-05-10T11:26:56.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.481+0000 7f0bf8800700 1 --2- 172.21.15.27:0/1137894281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bf010a490 0x7f0bf01169a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.481+0000 7f0bf8800700 1 --2- 172.21.15.27:0/1137894281 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bf010fb60 0x7f0bf010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.481+0000 7f0bf8800700 1 --2- 172.21.15.27:0/1137894281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf0109ad0 0x7f0bf0109f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.481+0000 7f0bf8800700 1 -- 172.21.15.27:0/1137894281 >> 172.21.15.27:0/1137894281 conn(0x7f0bf0076c70 msgr2=0x7f0bf01082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:56.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.482+0000 7f0bf8800700 1 -- 172.21.15.27:0/1137894281 shutdown_connections 2024-05-10T11:26:56.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.482+0000 7f0bf8800700 1 -- 172.21.15.27:0/1137894281 wait complete. 2024-05-10T11:26:56.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.483+0000 7f0bf8800700 1 Processor -- start 2024-05-10T11:26:56.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.483+0000 7f0bf8800700 1 -- start start 2024-05-10T11:26:56.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.484+0000 7f0bf8800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf0109ad0 0x7f0bf01a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.484+0000 7f0bf8800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bf010a490 0x7f0bf01a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.484+0000 7f0bf8800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bf010fb60 0x7f0bf01a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.484+0000 7f0bf8800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf0119d20 con 0x7f0bf0109ad0 2024-05-10T11:26:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.484+0000 7f0bf8800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf0119ba0 con 0x7f0bf010a490 2024-05-10T11:26:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.484+0000 7f0bf8800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf0119ea0 con 0x7f0bf010fb60 2024-05-10T11:26:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.484+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf0109ad0 0x7f0bf01a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.484+0000 7f0bf5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bf010a490 0x7f0bf01a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.484+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf0109ad0 0x7f0bf01a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51124/0 (socket says 172.21.15.27:51124) 2024-05-10T11:26:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.484+0000 7f0bf5e00700 1 -- 172.21.15.27:0/3780798433 learned_addr learned my addr 172.21.15.27:0/3780798433 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.485+0000 7f0bf6800700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bf010fb60 0x7f0bf01a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.485+0000 7f0bf5e00700 1 -- 172.21.15.27:0/3780798433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bf010fb60 msgr2=0x7f0bf01a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.485+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bf010fb60 0x7f0bf01a4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.485+0000 7f0bf5e00700 1 -- 172.21.15.27:0/3780798433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bf010a490 msgr2=0x7f0bf01a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:56.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.485+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bf010a490 0x7f0bf01a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.485+0000 7f0bf5e00700 1 -- 172.21.15.27:0/3780798433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf01a5150 con 0x7f0bf0109ad0 2024-05-10T11:26:56.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.485+0000 7f0bf6800700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bf010fb60 0x7f0bf01a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:26:56.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.485+0000 7f0bf5400700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bf010a490 0x7f0bf01a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:26:56.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.485+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf0109ad0 0x7f0bf01a0190 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0be400e9c0 tx=0x7f0be400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:56.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.486+0000 7f0beaa00700 1 -- 172.21.15.27:0/3780798433 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be400cbd0 con 0x7f0bf0109ad0 2024-05-10T11:26:56.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.486+0000 7f0beaa00700 1 -- 172.21.15.27:0/3780798433 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0be400cd70 con 0x7f0bf0109ad0 2024-05-10T11:26:56.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.486+0000 7f0beaa00700 1 -- 172.21.15.27:0/3780798433 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be4005610 con 0x7f0bf0109ad0 2024-05-10T11:26:56.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.486+0000 7f0bf8800700 1 -- 172.21.15.27:0/3780798433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf01a5440 con 0x7f0bf0109ad0 2024-05-10T11:26:56.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.486+0000 7f0bf8800700 1 -- 172.21.15.27:0/3780798433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf01accc0 con 0x7f0bf0109ad0 2024-05-10T11:26:56.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.490+0000 7f0beaa00700 1 -- 172.21.15.27:0/3780798433 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0be4004750 con 0x7f0bf0109ad0 2024-05-10T11:26:56.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.491+0000 7f0c03192b80 1 Processor -- start 2024-05-10T11:26:56.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.491+0000 7f0c03192b80 1 -- start start 2024-05-10T11:26:56.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.492+0000 7f0c03192b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 0x56256ee8e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.492+0000 7f0c03192b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256ee92490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.492+0000 7f0c03192b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee92e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.492+0000 7f0c03192b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56256ede0a20 con 0x56256ee8e060 2024-05-10T11:26:56.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.492+0000 7f0c03192b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56256ed7e760 con 0x56256ee8e980 2024-05-10T11:26:56.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.492+0000 7f0c03192b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56256eddc9e0 con 0x56256ee929d0 2024-05-10T11:26:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.492+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256ee92490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.492+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256ee92490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36354/0 (socket says 172.21.15.27:36354) 2024-05-10T11:26:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.492+0000 7f0bf5e00700 1 -- 172.21.15.27:0/213054457 learned_addr learned my addr 172.21.15.27:0/213054457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.492+0000 7f0bf6800700 1 --2- 172.21.15.27:0/213054457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 0x56256ee8e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.492+0000 7f0beaa00700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0bd80b2a70 0x7f0bd80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.493+0000 7f0bf5400700 1 --2- 172.21.15.27:0/213054457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee92e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.493+0000 7f0beaa00700 1 -- 172.21.15.27:0/3780798433 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7f0be40e2d30 con 0x7f0bf0109ad0 2024-05-10T11:26:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.493+0000 7f0bf5e00700 1 -- 172.21.15.27:0/213054457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 msgr2=0x56256ee92e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.493+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/213054457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee92e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.493+0000 7f0bf5e00700 1 -- 172.21.15.27:0/213054457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 msgr2=0x56256ee8e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.493+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/213054457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 0x56256ee8e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.493+0000 7f0bf5e00700 1 -- 172.21.15.27:0/213054457 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56256ed2d8d0 con 0x56256ee8e980 2024-05-10T11:26:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.493+0000 7f0bf6800700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0bd80b2a70 0x7f0bd80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.493+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/213054457 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256ee92490 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0be40a6df0 tx=0x7f0be40050b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fe832810af846e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.494+0000 7f0bdf400700 1 -- 172.21.15.27:0/213054457 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be40a94d0 con 0x56256ee8e980 2024-05-10T11:26:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.494+0000 7f0bdf400700 1 -- 172.21.15.27:0/213054457 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0be40b2320 con 0x56256ee8e980 2024-05-10T11:26:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.494+0000 7f0bdf400700 1 -- 172.21.15.27:0/213054457 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be40a92f0 con 0x56256ee8e980 2024-05-10T11:26:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.494+0000 7f0c03192b80 1 -- 172.21.15.27:0/213054457 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 msgr2=0x56256ee92490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.494+0000 7f0c03192b80 1 --2- 172.21.15.27:0/213054457 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256ee92490 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0be40a6df0 tx=0x7f0be40050b0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.494+0000 7f0bf6800700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0bd80b2a70 0x7f0bd80b4f30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0bec009850 tx=0x7f0bec00b3b0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.495+0000 7f0c03192b80 1 -- 172.21.15.27:0/213054457 shutdown_connections 2024-05-10T11:26:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.495+0000 7f0c03192b80 1 --2- 172.21.15.27:0/213054457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee92e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.495+0000 7f0c03192b80 1 --2- 172.21.15.27:0/213054457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 0x56256ee8e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.495+0000 7f0c03192b80 1 --2- 172.21.15.27:0/213054457 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256ee92490 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.495+0000 7f0c03192b80 1 -- 172.21.15.27:0/213054457 >> 172.21.15.27:0/213054457 conn(0x56256edbcb60 msgr2=0x56256ee887a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.495+0000 7f0c03192b80 1 -- 172.21.15.27:0/213054457 shutdown_connections 2024-05-10T11:26:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.495+0000 7f0c03192b80 1 -- 172.21.15.27:0/213054457 wait complete. 2024-05-10T11:26:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.495+0000 7f0c03192b80 1 -- start start 2024-05-10T11:26:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0c03192b80 1 Processor -- start 2024-05-10T11:26:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0c03192b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 0x56256ee99e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0c03192b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256ee97270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0c03192b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee977b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0c03192b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56256ee9ab80 con 0x56256ee8e060 2024-05-10T11:26:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0c03192b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56256ed46b20 con 0x56256ee8e980 2024-05-10T11:26:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0c03192b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56256ee9ad00 con 0x56256ee929d0 2024-05-10T11:26:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0c03192b80 10 client.0 _ll_register_callbacks cb 0x56256ed87c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:26:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0bf6800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0bf5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256ee97270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0bf6800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee977b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48018/0 (socket says 172.21.15.27:48018) 2024-05-10T11:26:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0bf6800700 1 -- 172.21.15.27:0/2823756989 learned_addr learned my addr 172.21.15.27:0/2823756989 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.496+0000 7f0bf5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256ee97270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36370/0 (socket says 172.21.15.27:36370) 2024-05-10T11:26:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.497+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/2823756989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 0x56256ee99e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.497+0000 7f0bf6800700 1 -- 172.21.15.27:0/2823756989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 msgr2=0x56256ee97270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.497+0000 7f0bf6800700 1 --2- 172.21.15.27:0/2823756989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256ee97270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.497+0000 7f0bf6800700 1 -- 172.21.15.27:0/2823756989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 msgr2=0x56256ee99e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.497+0000 7f0bf6800700 1 --2- 172.21.15.27:0/2823756989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 0x56256ee99e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.497+0000 7f0bf6800700 1 -- 172.21.15.27:0/2823756989 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56256eda6420 con 0x56256ee929d0 2024-05-10T11:26:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.497+0000 7f0bf6800700 1 --2- 172.21.15.27:0/2823756989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee977b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0bec015000 tx=0x7f0bec0099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7225123377095ab7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.497+0000 7f0bd3e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.498+0000 7f0bd3e00700 1 -- 172.21.15.27:0/2823756989 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec02a650 con 0x56256ee929d0 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.498+0000 7f0bd3e00700 1 -- 172.21.15.27:0/2823756989 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7f0bec02a7f0 con 0x56256ee929d0 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.498+0000 7f0c03192b80 1 -- 172.21.15.27:0/2823756989 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56256ee5b0a0 con 0x56256ee929d0 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.498+0000 7f0bd3e00700 1 -- 172.21.15.27:0/2823756989 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0bec038050 con 0x56256ee929d0 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.498+0000 7f0c03192b80 10 client.27154 fetch_fsmap learned FSMap version 36 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.498+0000 7f0c03192b80 1 -- 172.21.15.27:0/2823756989 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56256eea5410 con 0x56256ee929d0 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.498+0000 7f0bd3e00700 1 -- 172.21.15.27:0/2823756989 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56256ee5b0a0 con 0x56256ee929d0 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.498+0000 7f0bd3e00700 1 -- 172.21.15.27:0/2823756989 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f0bec011070 con 0x56256ee929d0 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.499+0000 7f0c03192b80 10 client.27154 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.499+0000 7f0c03192b80 10 client.27154 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.499+0000 7f0c03192b80 20 client.27154 populate_metadata read hostname 'smithi027' 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.499+0000 7f0c03192b80 1 --2- 172.21.15.27:0/2823756989 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56256eea7aa0 0x56256eea9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.499+0000 7f0c03192b80 4 client.27154 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:26:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.499+0000 7f0c03192b80 1 -- 172.21.15.27:0/2823756989 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56256edee400 con 0x56256eea7aa0 2024-05-10T11:26:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.499+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/2823756989 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56256eea7aa0 0x56256eea9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.499+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/2823756989 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56256eea7aa0 0x56256eea9f00 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=823b70d10febe08b server_cookie=8bdb7e483f13b3d0 in_seq=0 out_seq=0 2024-05-10T11:26:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.499+0000 7f0bd3e00700 10 client.27154 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:26:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.500+0000 7f0bd3e00700 1 -- 172.21.15.27:0/2823756989 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56256edee400 con 0x56256eea7aa0 2024-05-10T11:26:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.500+0000 7f0bd3e00700 10 client.27154 handle_command_reply: tid=0 2024-05-10T11:26:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.500+0000 7f0c03192b80 1 client.27154 shutdown 2024-05-10T11:26:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.501+0000 7f0c03192b80 1 -- 172.21.15.27:0/2823756989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 msgr2=0x56256ee977b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.501+0000 7f0c03192b80 1 --2- 172.21.15.27:0/2823756989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee977b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0bec015000 tx=0x7f0bec0099e0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.501+0000 7f0c03192b80 1 -- 172.21.15.27:0/2823756989 shutdown_connections 2024-05-10T11:26:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.501+0000 7f0c03192b80 1 --2- 172.21.15.27:0/2823756989 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56256eea7aa0 0x56256eea9f00 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.501+0000 7f0c03192b80 1 --2- 172.21.15.27:0/2823756989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee977b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.501+0000 7f0c03192b80 1 --2- 172.21.15.27:0/2823756989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 0x56256ee99e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.501+0000 7f0bd3e00700 0 client.27154 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:26:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.501+0000 7f0c03192b80 1 --2- 172.21.15.27:0/2823756989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256ee97270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.501+0000 7f0c03192b80 1 -- 172.21.15.27:0/2823756989 >> 172.21.15.27:0/2823756989 conn(0x56256edbcb60 msgr2=0x56256ee832b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:56.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.501+0000 7f0c03192b80 1 -- 172.21.15.27:0/2823756989 shutdown_connections 2024-05-10T11:26:56.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.501+0000 7f0c03192b80 1 -- 172.21.15.27:0/2823756989 wait complete. 2024-05-10T11:26:56.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.501+0000 7f0c03192b80 20 client.27154 trim_cache size 0 max 16384 2024-05-10T11:26:56.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.520+0000 7f0c03192b80 1 Processor -- start 2024-05-10T11:26:56.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.520+0000 7f0c03192b80 1 -- start start 2024-05-10T11:26:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.521+0000 7f0c03192b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 0x56256ee94ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.521+0000 7f0c03192b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256eea3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.521+0000 7f0c03192b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee96270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.521+0000 7f0c03192b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf01b5cf0 con 0x56256ee8e060 2024-05-10T11:26:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.521+0000 7f0c03192b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf01b7830 con 0x56256ee8e980 2024-05-10T11:26:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.521+0000 7f0c03192b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf01b63c0 con 0x56256ee929d0 2024-05-10T11:26:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.521+0000 7f0bf5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256eea3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.521+0000 7f0bf6800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee96270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.521+0000 7f0bf5400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256eea3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36372/0 (socket says 172.21.15.27:36372) 2024-05-10T11:26:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.521+0000 7f0bf6800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee96270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48032/0 (socket says 172.21.15.27:48032) 2024-05-10T11:26:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.521+0000 7f0bf5400700 1 -- 172.21.15.27:0/2447577458 learned_addr learned my addr 172.21.15.27:0/2447577458 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.521+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/2447577458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 0x56256ee94ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.522+0000 7f0bf6800700 1 -- 172.21.15.27:0/2447577458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 msgr2=0x56256eea3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.522+0000 7f0bf6800700 1 --2- 172.21.15.27:0/2447577458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256eea3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.522+0000 7f0bf6800700 1 -- 172.21.15.27:0/2447577458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 msgr2=0x56256ee94ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.522+0000 7f0bf6800700 1 --2- 172.21.15.27:0/2447577458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 0x56256ee94ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.522+0000 7f0bf6800700 1 -- 172.21.15.27:0/2447577458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56256ee957e0 con 0x56256ee929d0 2024-05-10T11:26:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.522+0000 7f0bf6800700 1 --2- 172.21.15.27:0/2447577458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee96270 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00d010 tx=0x7f0bec0094b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cba6f00d4b24a78 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.522+0000 7f0bdea00700 1 -- 172.21.15.27:0/2447577458 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec01f440 con 0x56256ee929d0 2024-05-10T11:26:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.522+0000 7f0bdea00700 1 -- 172.21.15.27:0/2447577458 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bec02f070 con 0x56256ee929d0 2024-05-10T11:26:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.522+0000 7f0bdea00700 1 -- 172.21.15.27:0/2447577458 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec034900 con 0x56256ee929d0 2024-05-10T11:26:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.523+0000 7f0c03192b80 1 -- 172.21.15.27:0/2447577458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 msgr2=0x56256ee96270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.523+0000 7f0c03192b80 1 --2- 172.21.15.27:0/2447577458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee96270 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00d010 tx=0x7f0bec0094b0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.523+0000 7f0c03192b80 1 -- 172.21.15.27:0/2447577458 shutdown_connections 2024-05-10T11:26:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.523+0000 7f0c03192b80 1 --2- 172.21.15.27:0/2447577458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee929d0 0x56256ee96270 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.523+0000 7f0c03192b80 1 --2- 172.21.15.27:0/2447577458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee8e060 0x56256ee94ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.523+0000 7f0c03192b80 1 --2- 172.21.15.27:0/2447577458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e980 0x56256eea3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.523+0000 7f0c03192b80 1 -- 172.21.15.27:0/2447577458 >> 172.21.15.27:0/2447577458 conn(0x56256edbcb60 msgr2=0x56256ee95b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.523+0000 7f0c03192b80 1 -- 172.21.15.27:0/2447577458 shutdown_connections 2024-05-10T11:26:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.523+0000 7f0c03192b80 1 -- 172.21.15.27:0/2447577458 wait complete. 2024-05-10T11:26:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.524+0000 7f0c03192b80 1 -- start start 2024-05-10T11:26:56.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.524+0000 7f0c03192b80 1 Processor -- start 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0c03192b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e060 0x56256eea03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0c03192b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee8e980 0x56256ee93bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0c03192b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee929d0 0x56256ee94110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0c03192b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf01b7160 con 0x56256ee929d0 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0c03192b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf01b6a90 con 0x56256ee8e060 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0c03192b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf01b3cb0 con 0x56256ee8e980 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0c03192b80 10 client.0 _ll_register_callbacks cb 0x56256ed87c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0bf5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee8e980 0x56256ee93bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e060 0x56256eea03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0bf5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee8e980 0x56256ee93bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48034/0 (socket says 172.21.15.27:48034) 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0bf5400700 1 -- 172.21.15.27:0/891872671 learned_addr learned my addr 172.21.15.27:0/891872671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0bf6800700 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee929d0 0x56256ee94110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0bf5400700 1 -- 172.21.15.27:0/891872671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e060 msgr2=0x56256eea03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0bf5400700 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e060 0x56256eea03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0bf5400700 1 -- 172.21.15.27:0/891872671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee929d0 msgr2=0x56256ee94110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0bf5400700 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee929d0 0x56256ee94110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0bf5400700 1 -- 172.21.15.27:0/891872671 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56256ee94840 con 0x56256ee8e980 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.525+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e060 0x56256eea03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.526+0000 7f0bf5400700 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee8e980 0x56256ee93bd0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0be0003900 tx=0x7f0be0003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f36750be3a35b23 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.526+0000 7f0bd3e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.526+0000 7f0bd3e00700 1 -- 172.21.15.27:0/891872671 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be00118b0 con 0x56256ee8e980 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.526+0000 7f0bd3e00700 1 -- 172.21.15.27:0/891872671 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7f0be0011a50 con 0x56256ee8e980 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.526+0000 7f0c03192b80 1 -- 172.21.15.27:0/891872671 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bec02f070 con 0x56256ee8e980 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.526+0000 7f0bd3e00700 1 -- 172.21.15.27:0/891872671 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0be002b050 con 0x56256ee8e980 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.526+0000 7f0c03192b80 10 client.27160 fetch_fsmap learned FSMap version 36 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.526+0000 7f0c03192b80 1 -- 172.21.15.27:0/891872671 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56256ef12380 con 0x56256ee8e980 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.527+0000 7f0bd3e00700 1 -- 172.21.15.27:0/891872671 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bec02f070 con 0x56256ee8e980 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.527+0000 7f0bd3e00700 1 -- 172.21.15.27:0/891872671 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f0be0005220 con 0x56256ee8e980 2024-05-10T11:26:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.527+0000 7f0c03192b80 10 client.27160 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:26:56.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.527+0000 7f0c03192b80 10 client.27160 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:26:56.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.527+0000 7f0c03192b80 20 client.27160 populate_metadata read hostname 'smithi027' 2024-05-10T11:26:56.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.527+0000 7f0c03192b80 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56256ef12af0 0x56256ef14f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:26:56.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.527+0000 7f0c03192b80 4 client.27160 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:26:56.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.527+0000 7f0c03192b80 1 -- 172.21.15.27:0/891872671 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56256ede84a0 con 0x56256ef12af0 2024-05-10T11:26:56.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.527+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56256ef12af0 0x56256ef14f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:26:56.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0bf5e00700 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56256ef12af0 0x56256ef14f10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7770e3a54b21411a server_cookie=78e976326afb505 in_seq=0 out_seq=0 2024-05-10T11:26:56.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0bd3e00700 10 client.27160 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:26:56.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0bd3e00700 1 -- 172.21.15.27:0/891872671 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56256ede84a0 con 0x56256ef12af0 2024-05-10T11:26:56.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0bd3e00700 10 client.27160 handle_command_reply: tid=0 2024-05-10T11:26:56.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0c03192b80 1 client.27160 shutdown 2024-05-10T11:26:56.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0c03192b80 1 -- 172.21.15.27:0/891872671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee8e980 msgr2=0x56256ee93bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0c03192b80 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee8e980 0x56256ee93bd0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0be0003900 tx=0x7f0be0003c10 comp rx=0 tx=0).stop 2024-05-10T11:26:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0c03192b80 1 -- 172.21.15.27:0/891872671 shutdown_connections 2024-05-10T11:26:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0c03192b80 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56256ee929d0 0x56256ee94110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0c03192b80 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56256ee8e060 0x56256eea03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0c03192b80 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56256ef12af0 0x56256ef14f10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0c03192b80 1 --2- 172.21.15.27:0/891872671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56256ee8e980 0x56256ee93bd0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0c03192b80 1 -- 172.21.15.27:0/891872671 >> 172.21.15.27:0/891872671 conn(0x56256edbcb60 msgr2=0x56256ee963f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.528+0000 7f0bd3e00700 0 client.27160 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:26:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.529+0000 7f0c03192b80 1 -- 172.21.15.27:0/891872671 shutdown_connections 2024-05-10T11:26:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.529+0000 7f0c03192b80 1 -- 172.21.15.27:0/891872671 wait complete. 2024-05-10T11:26:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.529+0000 7f0c03192b80 20 client.27160 trim_cache size 0 max 16384 2024-05-10T11:26:56.532 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:26:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.530+0000 7f0bf8800700 1 -- 172.21.15.27:0/3780798433 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0bd80b2a70 msgr2=0x7f0bd80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.530+0000 7f0bf8800700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0bd80b2a70 0x7f0bd80b4f30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0bec009850 tx=0x7f0bec00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.530+0000 7f0bf8800700 1 -- 172.21.15.27:0/3780798433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf0109ad0 msgr2=0x7f0bf01a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:26:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.530+0000 7f0bf8800700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf0109ad0 0x7f0bf01a0190 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0be400e9c0 tx=0x7f0be400ecd0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.530+0000 7f0bf8800700 1 -- 172.21.15.27:0/3780798433 shutdown_connections 2024-05-10T11:26:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.530+0000 7f0bf8800700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0bd80b2a70 0x7f0bd80b4f30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.530+0000 7f0bf8800700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bf010fb60 0x7f0bf01a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.530+0000 7f0bf8800700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf0109ad0 0x7f0bf01a0190 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.530+0000 7f0bf8800700 1 --2- 172.21.15.27:0/3780798433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bf010a490 0x7f0bf01a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:26:56.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.530+0000 7f0bf8800700 1 -- 172.21.15.27:0/3780798433 >> 172.21.15.27:0/3780798433 conn(0x7f0bf0076c70 msgr2=0x7f0bf0077230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:26:56.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.530+0000 7f0bf8800700 1 -- 172.21.15.27:0/3780798433 shutdown_connections 2024-05-10T11:26:56.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:26:56.530+0000 7f0bf8800700 1 -- 172.21.15.27:0/3780798433 wait complete. 2024-05-10T11:26:56.751 INFO:tasks.workunit.client.0.smithi027.stdout: 1 826101 6.14 MB/sec execute 581 sec latency 26.193 ms 2024-05-10T11:26:57.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:56 smithi027 ceph-mon[30054]: pgmap v428: 129 pgs: 129 active+clean; 630 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 179 op/s 2024-05-10T11:26:57.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:56 smithi076 ceph-mon[34152]: pgmap v428: 129 pgs: 129 active+clean; 630 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 179 op/s 2024-05-10T11:26:57.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:56 smithi189 ceph-mon[34842]: pgmap v428: 129 pgs: 129 active+clean; 630 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 179 op/s 2024-05-10T11:26:57.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:26:57.751 INFO:tasks.workunit.client.0.smithi027.stdout: 1 827337 6.14 MB/sec execute 582 sec latency 33.282 ms 2024-05-10T11:26:57.957 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:26:58.751 INFO:tasks.workunit.client.0.smithi027.stdout: 1 827847 6.14 MB/sec execute 583 sec latency 55.432 ms 2024-05-10T11:26:59.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:26:58 smithi027 ceph-mon[30054]: pgmap v429: 129 pgs: 129 active+clean; 630 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-05-10T11:26:59.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:26:58 smithi076 ceph-mon[34152]: pgmap v429: 129 pgs: 129 active+clean; 630 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-05-10T11:26:59.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:26:58 smithi189 ceph-mon[34842]: pgmap v429: 129 pgs: 129 active+clean; 630 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-05-10T11:26:59.751 INFO:tasks.workunit.client.0.smithi027.stdout: 1 829074 6.13 MB/sec execute 584 sec latency 25.957 ms 2024-05-10T11:27:00.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.538+0000 7ff262c00700 1 -- 172.21.15.27:0/1887053486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c06c530 msgr2=0x7ff25c110350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.538+0000 7ff262c00700 1 --2- 172.21.15.27:0/1887053486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c06c530 0x7ff25c110350 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff25400b0a0 tx=0x7ff25400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.538+0000 7ff262c00700 1 -- 172.21.15.27:0/1887053486 shutdown_connections 2024-05-10T11:27:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.538+0000 7ff262c00700 1 --2- 172.21.15.27:0/1887053486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c06c530 0x7ff25c110350 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.538+0000 7ff262c00700 1 --2- 172.21.15.27:0/1887053486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff25c109530 0x7ff25c109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.538+0000 7ff262c00700 1 --2- 172.21.15.27:0/1887053486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff25c06bb70 0x7ff25c06bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.538+0000 7ff262c00700 1 -- 172.21.15.27:0/1887053486 >> 172.21.15.27:0/1887053486 conn(0x7ff25c079350 msgr2=0x7ff25c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.539+0000 7ff262c00700 1 -- 172.21.15.27:0/1887053486 shutdown_connections 2024-05-10T11:27:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.539+0000 7ff262c00700 1 -- 172.21.15.27:0/1887053486 wait complete. 2024-05-10T11:27:00.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.540+0000 7ff262c00700 1 Processor -- start 2024-05-10T11:27:00.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.540+0000 7ff262c00700 1 -- start start 2024-05-10T11:27:00.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.541+0000 7ff262c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff25c06bb70 0x7ff25c19bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.541+0000 7ff262c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c06c530 0x7ff25c19c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.541+0000 7ff262c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff25c109530 0x7ff25c1a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.541+0000 7ff262c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c113690 con 0x7ff25c06bb70 2024-05-10T11:27:00.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.541+0000 7ff262c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c113510 con 0x7ff25c109530 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.541+0000 7ff262c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c113810 con 0x7ff25c06c530 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.541+0000 7ff25b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c06c530 0x7ff25c19c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.541+0000 7ff25be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff25c06bb70 0x7ff25c19bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.541+0000 7ff25b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c06c530 0x7ff25c19c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48042/0 (socket says 172.21.15.27:48042) 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.541+0000 7ff25b400700 1 -- 172.21.15.27:0/2982039010 learned_addr learned my addr 172.21.15.27:0/2982039010 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.541+0000 7ff260c00700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff25c109530 0x7ff25c1a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.542+0000 7ff25b400700 1 -- 172.21.15.27:0/2982039010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff25c109530 msgr2=0x7ff25c1a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.542+0000 7ff25b400700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff25c109530 0x7ff25c1a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.542+0000 7ff25b400700 1 -- 172.21.15.27:0/2982039010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff25c06bb70 msgr2=0x7ff25c19bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.542+0000 7ff25b400700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff25c06bb70 0x7ff25c19bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.542+0000 7ff25b400700 1 -- 172.21.15.27:0/2982039010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff25c1a0d60 con 0x7ff25c06c530 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.542+0000 7ff260c00700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff25c109530 0x7ff25c1a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:27:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.542+0000 7ff25b400700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c06c530 0x7ff25c19c290 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff24800e9a0 tx=0x7ff24800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:00.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.543+0000 7ff258c00700 1 -- 172.21.15.27:0/2982039010 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24800cbd0 con 0x7ff25c06c530 2024-05-10T11:27:00.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.543+0000 7ff258c00700 1 -- 172.21.15.27:0/2982039010 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff24800cd70 con 0x7ff25c06c530 2024-05-10T11:27:00.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.543+0000 7ff262c00700 1 -- 172.21.15.27:0/2982039010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff25c1a1050 con 0x7ff25c06c530 2024-05-10T11:27:00.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.543+0000 7ff258c00700 1 -- 172.21.15.27:0/2982039010 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248018680 con 0x7ff25c06c530 2024-05-10T11:27:00.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.543+0000 7ff262c00700 1 -- 172.21.15.27:0/2982039010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff25c1a88d0 con 0x7ff25c06c530 2024-05-10T11:27:00.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.547+0000 7ff258c00700 1 -- 172.21.15.27:0/2982039010 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff248010430 con 0x7ff25c06c530 2024-05-10T11:27:00.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.549+0000 7ff26d595b80 1 Processor -- start 2024-05-10T11:27:00.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.549+0000 7ff26d595b80 1 -- start start 2024-05-10T11:27:00.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.550+0000 7ff26d595b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49060 0x5616d9a49440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.550+0000 7ff26d595b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49980 0x5616d9a4d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.550+0000 7ff26d595b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a4d9d0 0x5616d9a4de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.550+0000 7ff26d595b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5616d999ba20 con 0x5616d9a4d9d0 2024-05-10T11:27:00.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.550+0000 7ff26d595b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5616d9939760 con 0x5616d9a49980 2024-05-10T11:27:00.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.550+0000 7ff26d595b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5616d99979e0 con 0x5616d9a49060 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.550+0000 7ff260c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49060 0x5616d9a49440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.550+0000 7ff25be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49980 0x5616d9a4d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.550+0000 7ff260c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49060 0x5616d9a49440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48048/0 (socket says 172.21.15.27:48048) 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.550+0000 7ff260c00700 1 -- 172.21.15.27:0/3487099435 learned_addr learned my addr 172.21.15.27:0/3487099435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.550+0000 7ff25b400700 1 --2- 172.21.15.27:0/3487099435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a4d9d0 0x5616d9a4de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.550+0000 7ff258c00700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff2400b2a70 0x7ff2400b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.551+0000 7ff260c00700 1 -- 172.21.15.27:0/3487099435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49980 msgr2=0x5616d9a4d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.551+0000 7ff260c00700 1 --2- 172.21.15.27:0/3487099435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49980 0x5616d9a4d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.551+0000 7ff260c00700 1 -- 172.21.15.27:0/3487099435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a4d9d0 msgr2=0x5616d9a4de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.551+0000 7ff260c00700 1 --2- 172.21.15.27:0/3487099435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a4d9d0 0x5616d9a4de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.551+0000 7ff260c00700 1 -- 172.21.15.27:0/3487099435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616d98e88d0 con 0x5616d9a49060 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.551+0000 7ff258c00700 1 -- 172.21.15.27:0/2982039010 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7ff2480e2c50 con 0x7ff25c06c530 2024-05-10T11:27:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.551+0000 7ff25be00700 1 --2- 172.21.15.27:0/3487099435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49980 0x5616d9a4d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:27:00.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.551+0000 7ff260c00700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff2400b2a70 0x7ff2400b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.551+0000 7ff260c00700 1 --2- 172.21.15.27:0/3487099435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49060 0x5616d9a49440 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff25400b0a0 tx=0x7ff254009410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4c4ec1425bdd41b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.552+0000 7ff251600700 1 -- 172.21.15.27:0/3487099435 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2540076e0 con 0x5616d9a49060 2024-05-10T11:27:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.552+0000 7ff251600700 1 -- 172.21.15.27:0/3487099435 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff254004e30 con 0x5616d9a49060 2024-05-10T11:27:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.552+0000 7ff251600700 1 -- 172.21.15.27:0/3487099435 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254005610 con 0x5616d9a49060 2024-05-10T11:27:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.553+0000 7ff26d595b80 1 -- 172.21.15.27:0/3487099435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49060 msgr2=0x5616d9a49440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.553+0000 7ff260c00700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff2400b2a70 0x7ff2400b4f30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff25402a000 tx=0x7ff25402b000 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.553+0000 7ff26d595b80 1 --2- 172.21.15.27:0/3487099435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49060 0x5616d9a49440 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff25400b0a0 tx=0x7ff254009410 comp rx=0 tx=0).stop 2024-05-10T11:27:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.553+0000 7ff26d595b80 1 -- 172.21.15.27:0/3487099435 shutdown_connections 2024-05-10T11:27:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.553+0000 7ff26d595b80 1 --2- 172.21.15.27:0/3487099435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49060 0x5616d9a49440 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.553+0000 7ff26d595b80 1 --2- 172.21.15.27:0/3487099435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a4d9d0 0x5616d9a4de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.553+0000 7ff26d595b80 1 --2- 172.21.15.27:0/3487099435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49980 0x5616d9a4d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.553+0000 7ff26d595b80 1 -- 172.21.15.27:0/3487099435 >> 172.21.15.27:0/3487099435 conn(0x5616d9977b60 msgr2=0x5616d9a437a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.553+0000 7ff26d595b80 1 -- 172.21.15.27:0/3487099435 shutdown_connections 2024-05-10T11:27:00.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.553+0000 7ff26d595b80 1 -- 172.21.15.27:0/3487099435 wait complete. 2024-05-10T11:27:00.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.554+0000 7ff26d595b80 1 -- start start 2024-05-10T11:27:00.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.554+0000 7ff26d595b80 1 Processor -- start 2024-05-10T11:27:00.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.555+0000 7ff26d595b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49060 0x5616d9a54e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.555+0000 7ff26d595b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49980 0x5616d9a52220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.555+0000 7ff26d595b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 0x5616d9a52760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.555+0000 7ff26d595b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5616d9a55be0 con 0x5616d9a49980 2024-05-10T11:27:00.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.555+0000 7ff26d595b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5616d9901b20 con 0x5616d9a49060 2024-05-10T11:27:00.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.555+0000 7ff26d595b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5616d9a55d60 con 0x5616d9a4d9d0 2024-05-10T11:27:00.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.555+0000 7ff26d595b80 10 client.0 _ll_register_callbacks cb 0x5616d9942c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.555+0000 7ff260c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 0x5616d9a52760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.555+0000 7ff25be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49060 0x5616d9a54e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.555+0000 7ff260c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 0x5616d9a52760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48052/0 (socket says 172.21.15.27:48052) 2024-05-10T11:27:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.555+0000 7ff260c00700 1 -- 172.21.15.27:0/2801066556 learned_addr learned my addr 172.21.15.27:0/2801066556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.555+0000 7ff25b400700 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49980 0x5616d9a52220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.556+0000 7ff260c00700 1 -- 172.21.15.27:0/2801066556 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49060 msgr2=0x5616d9a54e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.556+0000 7ff260c00700 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49060 0x5616d9a54e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.556+0000 7ff260c00700 1 -- 172.21.15.27:0/2801066556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49980 msgr2=0x5616d9a52220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.556+0000 7ff260c00700 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49980 0x5616d9a52220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.556+0000 7ff260c00700 1 -- 172.21.15.27:0/2801066556 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616d9961420 con 0x5616d9a4d9d0 2024-05-10T11:27:00.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.556+0000 7ff25be00700 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49060 0x5616d9a54e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:27:00.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.556+0000 7ff25b400700 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49980 0x5616d9a52220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:27:00.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.557+0000 7ff260c00700 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 0x5616d9a52760 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff25400bf70 tx=0x7ff2540051f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da915ab50f724801 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:00.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.557+0000 7ff246000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:27:00.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.557+0000 7ff246000700 1 -- 172.21.15.27:0/2801066556 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2540074a0 con 0x5616d9a4d9d0 2024-05-10T11:27:00.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.557+0000 7ff246000700 1 -- 172.21.15.27:0/2801066556 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7ff254007640 con 0x5616d9a4d9d0 2024-05-10T11:27:00.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.557+0000 7ff26d595b80 1 -- 172.21.15.27:0/2801066556 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff254004e30 con 0x5616d9a4d9d0 2024-05-10T11:27:00.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.558+0000 7ff246000700 1 -- 172.21.15.27:0/2801066556 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff254007840 con 0x5616d9a4d9d0 2024-05-10T11:27:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.558+0000 7ff26d595b80 10 client.27172 fetch_fsmap learned FSMap version 36 2024-05-10T11:27:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.558+0000 7ff26d595b80 1 -- 172.21.15.27:0/2801066556 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5616d9a55be0 con 0x5616d9a4d9d0 2024-05-10T11:27:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.558+0000 7ff246000700 1 -- 172.21.15.27:0/2801066556 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff254004e30 con 0x5616d9a4d9d0 2024-05-10T11:27:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.558+0000 7ff246000700 1 -- 172.21.15.27:0/2801066556 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7ff254011070 con 0x5616d9a4d9d0 2024-05-10T11:27:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.558+0000 7ff26d595b80 10 client.27172 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:27:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.558+0000 7ff26d595b80 10 client.27172 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:27:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.558+0000 7ff26d595b80 20 client.27172 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.558+0000 7ff26d595b80 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5616d9a62770 0x5616d9a64b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.558+0000 7ff26d595b80 4 client.27172 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:27:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.558+0000 7ff26d595b80 1 -- 172.21.15.27:0/2801066556 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616d9a160a0 con 0x5616d9a62770 2024-05-10T11:27:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.559+0000 7ff25be00700 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5616d9a62770 0x5616d9a64b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.559+0000 7ff25be00700 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5616d9a62770 0x5616d9a64b60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8f7e32160e1e1b17 server_cookie=d7beade3701d1c0d in_seq=0 out_seq=0 2024-05-10T11:27:00.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.559+0000 7ff246000700 10 client.27172 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:27:00.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.560+0000 7ff246000700 1 -- 172.21.15.27:0/2801066556 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5616d9a160a0 con 0x5616d9a62770 2024-05-10T11:27:00.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.560+0000 7ff246000700 10 client.27172 handle_command_reply: tid=0 2024-05-10T11:27:00.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.560+0000 7ff26d595b80 1 client.27172 shutdown 2024-05-10T11:27:00.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.560+0000 7ff26d595b80 1 -- 172.21.15.27:0/2801066556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 msgr2=0x5616d9a52760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:00.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.560+0000 7ff26d595b80 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 0x5616d9a52760 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff25400bf70 tx=0x7ff2540051f0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.560+0000 7ff26d595b80 1 -- 172.21.15.27:0/2801066556 shutdown_connections 2024-05-10T11:27:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.560+0000 7ff26d595b80 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5616d9a62770 0x5616d9a64b60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.561+0000 7ff26d595b80 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 0x5616d9a52760 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.561+0000 7ff26d595b80 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49060 0x5616d9a54e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.561+0000 7ff26d595b80 1 --2- 172.21.15.27:0/2801066556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49980 0x5616d9a52220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.561+0000 7ff246000700 0 client.27172 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:27:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.561+0000 7ff26d595b80 1 -- 172.21.15.27:0/2801066556 >> 172.21.15.27:0/2801066556 conn(0x5616d9977b60 msgr2=0x5616d9a3e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.561+0000 7ff26d595b80 1 -- 172.21.15.27:0/2801066556 shutdown_connections 2024-05-10T11:27:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.561+0000 7ff26d595b80 1 -- 172.21.15.27:0/2801066556 wait complete. 2024-05-10T11:27:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.561+0000 7ff26d595b80 20 client.27172 trim_cache size 0 max 16384 2024-05-10T11:27:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.578+0000 7ff26d595b80 1 Processor -- start 2024-05-10T11:27:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.578+0000 7ff26d595b80 1 -- start start 2024-05-10T11:27:00.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.578+0000 7ff26d595b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 0x5616d9a4edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.578+0000 7ff26d595b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49980 0x5616d9a5ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.578+0000 7ff26d595b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a4d9d0 0x5616d9a3da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff26d595b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1b1840 con 0x5616d9a49060 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff26d595b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1b3380 con 0x5616d9a4d9d0 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff26d595b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1b1f10 con 0x5616d9a49980 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff25b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49980 0x5616d9a5ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff25b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49980 0x5616d9a5ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48068/0 (socket says 172.21.15.27:48068) 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff25b400700 1 -- 172.21.15.27:0/3986224931 learned_addr learned my addr 172.21.15.27:0/3986224931 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff260c00700 1 --2- 172.21.15.27:0/3986224931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a4d9d0 0x5616d9a3da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff25be00700 1 --2- 172.21.15.27:0/3986224931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 0x5616d9a4edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff25b400700 1 -- 172.21.15.27:0/3986224931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a4d9d0 msgr2=0x5616d9a3da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff25b400700 1 --2- 172.21.15.27:0/3986224931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a4d9d0 0x5616d9a3da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff25b400700 1 -- 172.21.15.27:0/3986224931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 msgr2=0x5616d9a4edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff25b400700 1 --2- 172.21.15.27:0/3986224931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 0x5616d9a4edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff25b400700 1 -- 172.21.15.27:0/3986224931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616d9a54ac0 con 0x5616d9a49980 2024-05-10T11:27:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.579+0000 7ff25be00700 1 --2- 172.21.15.27:0/3986224931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 0x5616d9a4edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:27:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.580+0000 7ff25b400700 1 --2- 172.21.15.27:0/3986224931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49980 0x5616d9a5ec70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff2480a97a0 tx=0x7ff248010cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f890827a4c8e3803 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.580+0000 7ff250c00700 1 -- 172.21.15.27:0/3986224931 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2480a93f0 con 0x5616d9a49980 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.580+0000 7ff250c00700 1 -- 172.21.15.27:0/3986224931 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2480e6020 con 0x5616d9a49980 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.580+0000 7ff250c00700 1 -- 172.21.15.27:0/3986224931 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2480a5230 con 0x5616d9a49980 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.581+0000 7ff26d595b80 1 -- 172.21.15.27:0/3986224931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49980 msgr2=0x5616d9a5ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.581+0000 7ff26d595b80 1 --2- 172.21.15.27:0/3986224931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49980 0x5616d9a5ec70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff2480a97a0 tx=0x7ff248010cb0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.581+0000 7ff26d595b80 1 -- 172.21.15.27:0/3986224931 shutdown_connections 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.581+0000 7ff26d595b80 1 --2- 172.21.15.27:0/3986224931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a4d9d0 0x5616d9a3da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.581+0000 7ff26d595b80 1 --2- 172.21.15.27:0/3986224931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 0x5616d9a4edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.581+0000 7ff26d595b80 1 --2- 172.21.15.27:0/3986224931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a49980 0x5616d9a5ec70 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.581+0000 7ff26d595b80 1 -- 172.21.15.27:0/3986224931 >> 172.21.15.27:0/3986224931 conn(0x5616d9977b60 msgr2=0x5616d9a4fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.581+0000 7ff26d595b80 1 -- 172.21.15.27:0/3986224931 shutdown_connections 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.581+0000 7ff26d595b80 1 -- 172.21.15.27:0/3986224931 wait complete. 2024-05-10T11:27:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.581+0000 7ff26d595b80 1 -- start start 2024-05-10T11:27:00.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.582+0000 7ff26d595b80 1 Processor -- start 2024-05-10T11:27:00.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.582+0000 7ff26d595b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 0x5616d9a5b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.582+0000 7ff26d595b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49980 0x5616d9a696e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.582+0000 7ff26d595b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 0x5616d9a69c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.582+0000 7ff26d595b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1b2cb0 con 0x5616d9a49060 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.582+0000 7ff26d595b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1b25e0 con 0x5616d9a49980 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.582+0000 7ff26d595b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1af800 con 0x5616d9a4d9d0 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.582+0000 7ff26d595b80 10 client.0 _ll_register_callbacks cb 0x5616d9942c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff260c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 0x5616d9a69c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff25be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 0x5616d9a5b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff25be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 0x5616d9a5b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51224/0 (socket says 172.21.15.27:51224) 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff260c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 0x5616d9a69c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:48070/0 (socket says 172.21.15.27:48070) 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff25be00700 1 -- 172.21.15.27:0/2760131785 learned_addr learned my addr 172.21.15.27:0/2760131785 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff25b400700 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49980 0x5616d9a696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff25be00700 1 -- 172.21.15.27:0/2760131785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 msgr2=0x5616d9a69c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff25be00700 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 0x5616d9a69c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff25be00700 1 -- 172.21.15.27:0/2760131785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49980 msgr2=0x5616d9a696e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff25be00700 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49980 0x5616d9a696e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff25be00700 1 -- 172.21.15.27:0/2760131785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616d9a6a330 con 0x5616d9a49060 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff25b400700 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49980 0x5616d9a696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.583+0000 7ff260c00700 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 0x5616d9a69c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.584+0000 7ff25be00700 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 0x5616d9a5b4a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x5616d9a65210 tx=0x7ff24c012a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d231a769ab08c917 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.584+0000 7ff246000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.584+0000 7ff246000700 1 -- 172.21.15.27:0/2760131785 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c004220 con 0x5616d9a49060 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.584+0000 7ff26d595b80 1 -- 172.21.15.27:0/2760131785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2480e6020 con 0x5616d9a49060 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.584+0000 7ff246000700 1 -- 172.21.15.27:0/2760131785 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9591+0+0 (secure 0 0 0) 0x7ff24c0043c0 con 0x5616d9a49060 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.584+0000 7ff246000700 1 -- 172.21.15.27:0/2760131785 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff24c031050 con 0x5616d9a49060 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.584+0000 7ff26d595b80 10 client.16932 fetch_fsmap learned FSMap version 36 2024-05-10T11:27:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.584+0000 7ff26d595b80 1 -- 172.21.15.27:0/2760131785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5616d9ad55f0 con 0x5616d9a49060 2024-05-10T11:27:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.585+0000 7ff246000700 1 -- 172.21.15.27:0/2760131785 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2480e6020 con 0x5616d9a49060 2024-05-10T11:27:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.585+0000 7ff246000700 1 -- 172.21.15.27:0/2760131785 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7ff24c0221f0 con 0x5616d9a49060 2024-05-10T11:27:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.585+0000 7ff26d595b80 10 client.16932 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:27:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.585+0000 7ff26d595b80 10 client.16932 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:27:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.585+0000 7ff26d595b80 20 client.16932 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.585+0000 7ff26d595b80 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5616d9ad5cd0 0x5616d9ad80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.585+0000 7ff26d595b80 4 client.16932 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.585+0000 7ff26d595b80 1 -- 172.21.15.27:0/2760131785 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5616d99a9400 con 0x5616d9ad5cd0 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.586+0000 7ff25b400700 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5616d9ad5cd0 0x5616d9ad80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.586+0000 7ff25b400700 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5616d9ad5cd0 0x5616d9ad80f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=87d61b51d3e0f0ef server_cookie=61b1b873aec41857 in_seq=0 out_seq=0 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.586+0000 7ff246000700 10 client.16932 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff246000700 1 -- 172.21.15.27:0/2760131785 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5616d99a9400 con 0x5616d9ad5cd0 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff246000700 10 client.16932 handle_command_reply: tid=0 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff26d595b80 1 client.16932 shutdown 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff26d595b80 1 -- 172.21.15.27:0/2760131785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 msgr2=0x5616d9a5b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff26d595b80 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 0x5616d9a5b4a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x5616d9a65210 tx=0x7ff24c012a10 comp rx=0 tx=0).stop 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff26d595b80 1 -- 172.21.15.27:0/2760131785 shutdown_connections 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff26d595b80 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5616d9ad5cd0 0x5616d9ad80f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff26d595b80 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616d9a4d9d0 0x5616d9a69c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff26d595b80 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616d9a49060 0x5616d9a5b4a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff26d595b80 1 --2- 172.21.15.27:0/2760131785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616d9a49980 0x5616d9a696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff26d595b80 1 -- 172.21.15.27:0/2760131785 >> 172.21.15.27:0/2760131785 conn(0x5616d9977b60 msgr2=0x5616d9a3da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff246000700 0 client.16932 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:27:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff26d595b80 1 -- 172.21.15.27:0/2760131785 shutdown_connections 2024-05-10T11:27:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.587+0000 7ff26d595b80 1 -- 172.21.15.27:0/2760131785 wait complete. 2024-05-10T11:27:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.588+0000 7ff26d595b80 20 client.16932 trim_cache size 0 max 16384 2024-05-10T11:27:00.590 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:27:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.588+0000 7ff262c00700 1 -- 172.21.15.27:0/2982039010 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff2400b2a70 msgr2=0x7ff2400b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.588+0000 7ff262c00700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff2400b2a70 0x7ff2400b4f30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff25402a000 tx=0x7ff25402b000 comp rx=0 tx=0).stop 2024-05-10T11:27:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.588+0000 7ff262c00700 1 -- 172.21.15.27:0/2982039010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c06c530 msgr2=0x7ff25c19c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.588+0000 7ff262c00700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c06c530 0x7ff25c19c290 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff24800e9a0 tx=0x7ff24800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.588+0000 7ff262c00700 1 -- 172.21.15.27:0/2982039010 shutdown_connections 2024-05-10T11:27:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.588+0000 7ff262c00700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff2400b2a70 0x7ff2400b4f30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.588+0000 7ff262c00700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff25c109530 0x7ff25c1a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.588+0000 7ff262c00700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff25c06bb70 0x7ff25c19bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.589+0000 7ff262c00700 1 --2- 172.21.15.27:0/2982039010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c06c530 0x7ff25c19c290 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:00.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.589+0000 7ff262c00700 1 -- 172.21.15.27:0/2982039010 >> 172.21.15.27:0/2982039010 conn(0x7ff25c079350 msgr2=0x7ff25c10e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.589+0000 7ff262c00700 1 -- 172.21.15.27:0/2982039010 shutdown_connections 2024-05-10T11:27:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:00.589+0000 7ff262c00700 1 -- 172.21.15.27:0/2982039010 wait complete. 2024-05-10T11:27:00.751 INFO:tasks.workunit.client.0.smithi027.stdout: 1 830845 6.15 MB/sec execute 585 sec latency 32.229 ms 2024-05-10T11:27:00.843 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:00 smithi027 ceph-mon[30054]: pgmap v430: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 189 op/s 2024-05-10T11:27:00.843 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:00 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:27:01.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:00 smithi076 ceph-mon[34152]: pgmap v430: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 189 op/s 2024-05-10T11:27:01.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:00 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:27:01.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:00 smithi189 ceph-mon[34842]: pgmap v430: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 189 op/s 2024-05-10T11:27:01.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:00 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:27:01.526 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:27:01.752 INFO:tasks.workunit.client.0.smithi027.stdout: 1 830964 6.14 MB/sec execute 586 sec latency 113.050 ms 2024-05-10T11:27:01.854 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:01 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:27:01.854 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:01 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:27:01.854 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:01 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:27:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:27:02.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:01 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:27:02.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:01 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:27:02.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:01 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:27:02.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:01 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:27:02.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:01 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:27:02.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:01 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:27:02.751 INFO:tasks.workunit.client.0.smithi027.stdout: 1 831155 6.13 MB/sec execute 587 sec latency 187.530 ms 2024-05-10T11:27:03.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:02 smithi027 ceph-mon[30054]: pgmap v431: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 128 op/s 2024-05-10T11:27:03.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:02 smithi027 ceph-mon[30054]: osdmap e106: 12 total, 12 up, 12 in 2024-05-10T11:27:03.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:02 smithi076 ceph-mon[34152]: pgmap v431: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 128 op/s 2024-05-10T11:27:03.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:02 smithi076 ceph-mon[34152]: osdmap e106: 12 total, 12 up, 12 in 2024-05-10T11:27:03.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:02 smithi189 ceph-mon[34842]: pgmap v431: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 128 op/s 2024-05-10T11:27:03.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:02 smithi189 ceph-mon[34842]: osdmap e106: 12 total, 12 up, 12 in 2024-05-10T11:27:03.752 INFO:tasks.workunit.client.0.smithi027.stdout: 1 831868 6.12 MB/sec execute 588 sec latency 78.212 ms 2024-05-10T11:27:04.752 INFO:tasks.workunit.client.0.smithi027.stdout: 1 832975 6.12 MB/sec execute 589 sec latency 26.427 ms 2024-05-10T11:27:04.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.867+0000 7f7887600700 1 -- 172.21.15.27:0/2374036734 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7880103750 msgr2=0x7f7880103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:04.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.867+0000 7f7887600700 1 --2- 172.21.15.27:0/2374036734 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7880103750 0x7f7880103b30 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f78740099c0 tx=0x7f7874009cd0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.867+0000 7f7887600700 1 -- 172.21.15.27:0/2374036734 shutdown_connections 2024-05-10T11:27:04.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.867+0000 7f7887600700 1 --2- 172.21.15.27:0/2374036734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7880108850 0x7f78801103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.867+0000 7f7887600700 1 --2- 172.21.15.27:0/2374036734 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7880103750 0x7f7880103b30 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.867+0000 7f7887600700 1 --2- 172.21.15.27:0/2374036734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7880104100 0x7f7880108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.867+0000 7f7887600700 1 -- 172.21.15.27:0/2374036734 >> 172.21.15.27:0/2374036734 conn(0x7f788007a600 msgr2=0x7f788007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:04.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.867+0000 7f7887600700 1 -- 172.21.15.27:0/2374036734 shutdown_connections 2024-05-10T11:27:04.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.867+0000 7f7887600700 1 -- 172.21.15.27:0/2374036734 wait complete. 2024-05-10T11:27:04.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.868+0000 7f7887600700 1 Processor -- start 2024-05-10T11:27:04.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.869+0000 7f7887600700 1 -- start start 2024-05-10T11:27:04.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.869+0000 7f7887600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7880103750 0x7f788019bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.869+0000 7f7887600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7880104100 0x7f788019c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.869+0000 7f7887600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7880108850 0x7f78801a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.869+0000 7f7887600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7880113670 con 0x7f7880104100 2024-05-10T11:27:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.869+0000 7f7887600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f78801134f0 con 0x7f7880108850 2024-05-10T11:27:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.869+0000 7f7887600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f78801137f0 con 0x7f7880103750 2024-05-10T11:27:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.869+0000 7f7884c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7880103750 0x7f788019bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.869+0000 7f7885600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7880108850 0x7f78801a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.869+0000 7f7884c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7880103750 0x7f788019bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54622/0 (socket says 172.21.15.27:54622) 2024-05-10T11:27:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.869+0000 7f7885600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7880108850 0x7f78801a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33364/0 (socket says 172.21.15.27:33364) 2024-05-10T11:27:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.870+0000 7f7884c00700 1 -- 172.21.15.27:0/1590987257 learned_addr learned my addr 172.21.15.27:0/1590987257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.870+0000 7f787fe00700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7880104100 0x7f788019c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.870+0000 7f7885600700 1 -- 172.21.15.27:0/1590987257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7880103750 msgr2=0x7f788019bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.870+0000 7f7885600700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7880103750 0x7f788019bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.870+0000 7f7885600700 1 -- 172.21.15.27:0/1590987257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7880104100 msgr2=0x7f788019c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.870+0000 7f7885600700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7880104100 0x7f788019c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.870+0000 7f7885600700 1 -- 172.21.15.27:0/1590987257 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78801a0d50 con 0x7f7880108850 2024-05-10T11:27:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.870+0000 7f787fe00700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7880104100 0x7f788019c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:27:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.870+0000 7f7884c00700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7880103750 0x7f788019bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:27:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.870+0000 7f7885600700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7880108850 0x7f78801a0610 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f787800efc0 tx=0x7f787800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.871+0000 7f787d600700 1 -- 172.21.15.27:0/1590987257 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78780092e0 con 0x7f7880108850 2024-05-10T11:27:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.871+0000 7f7887600700 1 -- 172.21.15.27:0/1590987257 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78801a0fe0 con 0x7f7880108850 2024-05-10T11:27:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.871+0000 7f787d600700 1 -- 172.21.15.27:0/1590987257 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7878004bb0 con 0x7f7880108850 2024-05-10T11:27:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.871+0000 7f787d600700 1 -- 172.21.15.27:0/1590987257 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7878004080 con 0x7f7880108850 2024-05-10T11:27:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.871+0000 7f7887600700 1 -- 172.21.15.27:0/1590987257 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78801a88c0 con 0x7f7880108850 2024-05-10T11:27:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.874+0000 7f787d600700 1 -- 172.21.15.27:0/1590987257 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f787801b030 con 0x7f7880108850 2024-05-10T11:27:04.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.876+0000 7f787d600700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f78680b29f0 0x7f78680b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.876+0000 7f787d600700 1 -- 172.21.15.27:0/1590987257 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9679+0+0 (secure 0 0 0) 0x7f78780e26d0 con 0x7f7880108850 2024-05-10T11:27:04.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.876+0000 7f7884c00700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f78680b29f0 0x7f78680b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.876+0000 7f7891ed8b80 1 Processor -- start 2024-05-10T11:27:04.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.877+0000 7f7891ed8b80 1 -- start start 2024-05-10T11:27:04.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.877+0000 7f7891ed8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 0x558a3d534fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.877+0000 7f7891ed8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d5354e0 0x558a3d52f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.877+0000 7f7891ed8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d52fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.877+0000 7f7891ed8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558a3d481a20 con 0x558a3d5354e0 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.877+0000 7f7891ed8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558a3d41f760 con 0x558a3d52f8f0 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.877+0000 7f7891ed8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558a3d47d9e0 con 0x558a3d534bc0 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.877+0000 7f7884c00700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f78680b29f0 0x7f78680b4eb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7874009990 tx=0x7f7874005ab0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.878+0000 7f7884c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 0x558a3d534fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.878+0000 7f7884c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 0x558a3d534fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54638/0 (socket says 172.21.15.27:54638) 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.878+0000 7f7884c00700 1 -- 172.21.15.27:0/2718594531 learned_addr learned my addr 172.21.15.27:0/2718594531 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.878+0000 7f7885600700 1 --2- 172.21.15.27:0/2718594531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d52fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.878+0000 7f787fe00700 1 --2- 172.21.15.27:0/2718594531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d5354e0 0x558a3d52f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.878+0000 7f7884c00700 1 -- 172.21.15.27:0/2718594531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 msgr2=0x558a3d52fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.878+0000 7f7884c00700 1 --2- 172.21.15.27:0/2718594531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d52fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.878+0000 7f7884c00700 1 -- 172.21.15.27:0/2718594531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d5354e0 msgr2=0x558a3d52f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.878+0000 7f7884c00700 1 --2- 172.21.15.27:0/2718594531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d5354e0 0x558a3d52f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.878+0000 7f7884c00700 1 -- 172.21.15.27:0/2718594531 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a3d3ce8d0 con 0x558a3d534bc0 2024-05-10T11:27:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.878+0000 7f7884c00700 1 --2- 172.21.15.27:0/2718594531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 0x558a3d534fa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7874000c00 tx=0x7f787401a070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb9c3d80cea5e9fc server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.879+0000 7f7872000700 1 -- 172.21.15.27:0/2718594531 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78740039c0 con 0x558a3d534bc0 2024-05-10T11:27:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.879+0000 7f7872000700 1 -- 172.21.15.27:0/2718594531 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7874009590 con 0x558a3d534bc0 2024-05-10T11:27:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.879+0000 7f7872000700 1 -- 172.21.15.27:0/2718594531 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874010470 con 0x558a3d534bc0 2024-05-10T11:27:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.880+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2718594531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 msgr2=0x558a3d534fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.880+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2718594531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 0x558a3d534fa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7874000c00 tx=0x7f787401a070 comp rx=0 tx=0).stop 2024-05-10T11:27:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.880+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2718594531 shutdown_connections 2024-05-10T11:27:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.880+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2718594531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 0x558a3d534fa0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.880+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2718594531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d52fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.880+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2718594531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d5354e0 0x558a3d52f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.880+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2718594531 >> 172.21.15.27:0/2718594531 conn(0x558a3d4a76d0 msgr2=0x558a3d529b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.880+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2718594531 shutdown_connections 2024-05-10T11:27:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.880+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2718594531 wait complete. 2024-05-10T11:27:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.881+0000 7f7891ed8b80 1 -- start start 2024-05-10T11:27:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.881+0000 7f7891ed8b80 1 Processor -- start 2024-05-10T11:27:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.881+0000 7f7891ed8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d52f8f0 0x558a3d53b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.881+0000 7f7891ed8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d534bc0 0x558a3d538590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.881+0000 7f7891ed8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d5354e0 0x558a3d538ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7891ed8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558a3d53bec0 con 0x558a3d534bc0 2024-05-10T11:27:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7891ed8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558a3d3e7b20 con 0x558a3d5354e0 2024-05-10T11:27:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7891ed8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558a3d53c040 con 0x558a3d52f8f0 2024-05-10T11:27:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7891ed8b80 10 client.0 _ll_register_callbacks cb 0x558a3d428c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7884c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d52f8f0 0x558a3d53b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7884c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d52f8f0 0x558a3d53b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54654/0 (socket says 172.21.15.27:54654) 2024-05-10T11:27:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7884c00700 1 -- 172.21.15.27:0/2153281014 learned_addr learned my addr 172.21.15.27:0/2153281014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f787fe00700 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d534bc0 0x558a3d538590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7885600700 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d5354e0 0x558a3d538ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7884c00700 1 -- 172.21.15.27:0/2153281014 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d5354e0 msgr2=0x558a3d538ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7884c00700 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d5354e0 0x558a3d538ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7884c00700 1 -- 172.21.15.27:0/2153281014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d534bc0 msgr2=0x558a3d538590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7884c00700 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d534bc0 0x558a3d538590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f7884c00700 1 -- 172.21.15.27:0/2153281014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a3d447420 con 0x558a3d52f8f0 2024-05-10T11:27:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.882+0000 7f787fe00700 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d534bc0 0x558a3d538590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:27:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.883+0000 7f7884c00700 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d52f8f0 0x558a3d53b1f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f78740239f0 tx=0x7f7874023710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ed38ab34921ac50 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.883+0000 7f7866a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:27:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.883+0000 7f7866a00700 1 -- 172.21.15.27:0/2153281014 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78740039c0 con 0x558a3d52f8f0 2024-05-10T11:27:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.883+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2153281014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7874009590 con 0x558a3d52f8f0 2024-05-10T11:27:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.883+0000 7f7866a00700 1 -- 172.21.15.27:0/2153281014 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9679+0+0 (secure 0 0 0) 0x7f7874003b60 con 0x558a3d52f8f0 2024-05-10T11:27:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.884+0000 7f7866a00700 1 -- 172.21.15.27:0/2153281014 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7874010470 con 0x558a3d52f8f0 2024-05-10T11:27:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.884+0000 7f7891ed8b80 10 client.27190 fetch_fsmap learned FSMap version 36 2024-05-10T11:27:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.884+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2153281014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558a3d53bec0 con 0x558a3d52f8f0 2024-05-10T11:27:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.884+0000 7f7866a00700 1 -- 172.21.15.27:0/2153281014 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7874009590 con 0x558a3d52f8f0 2024-05-10T11:27:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.884+0000 7f7866a00700 1 -- 172.21.15.27:0/2153281014 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f787402f450 con 0x558a3d52f8f0 2024-05-10T11:27:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.884+0000 7f7891ed8b80 10 client.27190 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:27:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.884+0000 7f7891ed8b80 10 client.27190 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:27:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.884+0000 7f7891ed8b80 20 client.27190 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.884+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x558a3d548a50 0x558a3d54ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.884+0000 7f7891ed8b80 4 client.27190 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:27:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.884+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2153281014 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a3d4fc140 con 0x558a3d548a50 2024-05-10T11:27:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.885+0000 7f787fe00700 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x558a3d548a50 0x558a3d54ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.885+0000 7f787fe00700 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x558a3d548a50 0x558a3d54ae40 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6161346ae3ed526a server_cookie=6a05a32b0821e06d in_seq=0 out_seq=0 2024-05-10T11:27:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.885+0000 7f7866a00700 10 client.27190 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:27:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.886+0000 7f7866a00700 1 -- 172.21.15.27:0/2153281014 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558a3d4fc140 con 0x558a3d548a50 2024-05-10T11:27:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.886+0000 7f7866a00700 10 client.27190 handle_command_reply: tid=0 2024-05-10T11:27:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.886+0000 7f7891ed8b80 1 client.27190 shutdown 2024-05-10T11:27:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.886+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2153281014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d52f8f0 msgr2=0x558a3d53b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.886+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d52f8f0 0x558a3d53b1f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f78740239f0 tx=0x7f7874023710 comp rx=0 tx=0).stop 2024-05-10T11:27:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.887+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2153281014 shutdown_connections 2024-05-10T11:27:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.887+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x558a3d548a50 0x558a3d54ae40 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.887+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d52f8f0 0x558a3d53b1f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.887+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d5354e0 0x558a3d538ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.887+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2153281014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d534bc0 0x558a3d538590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.887+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2153281014 >> 172.21.15.27:0/2153281014 conn(0x558a3d4a76d0 msgr2=0x558a3d529190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.887+0000 7f7866a00700 0 client.27190 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:27:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.887+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2153281014 shutdown_connections 2024-05-10T11:27:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.887+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2153281014 wait complete. 2024-05-10T11:27:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.887+0000 7f7891ed8b80 20 client.27190 trim_cache size 0 max 16384 2024-05-10T11:27:04.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.908+0000 7f7891ed8b80 1 Processor -- start 2024-05-10T11:27:04.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.908+0000 7f7891ed8b80 1 -- start start 2024-05-10T11:27:04.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.908+0000 7f7891ed8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d52f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.908+0000 7f7891ed8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d534bc0 0x558a3d544f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.908+0000 7f7891ed8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d5354e0 0x558a3d537710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7891ed8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f78801b19c0 con 0x558a3d534bc0 2024-05-10T11:27:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7891ed8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f78801b3500 con 0x558a3d52f8f0 2024-05-10T11:27:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7891ed8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f78801b2090 con 0x558a3d5354e0 2024-05-10T11:27:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7884c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d52f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7884c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d52f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33374/0 (socket says 172.21.15.27:33374) 2024-05-10T11:27:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7885600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d5354e0 0x558a3d537710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7884c00700 1 -- 172.21.15.27:0/2202970384 learned_addr learned my addr 172.21.15.27:0/2202970384 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f787fe00700 1 --2- 172.21.15.27:0/2202970384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d534bc0 0x558a3d544f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7885600700 1 -- 172.21.15.27:0/2202970384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 msgr2=0x558a3d52f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7885600700 1 --2- 172.21.15.27:0/2202970384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d52f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7885600700 1 -- 172.21.15.27:0/2202970384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d534bc0 msgr2=0x558a3d544f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7885600700 1 --2- 172.21.15.27:0/2202970384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d534bc0 0x558a3d544f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7885600700 1 -- 172.21.15.27:0/2202970384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a3d52b4f0 con 0x558a3d5354e0 2024-05-10T11:27:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f7884c00700 1 --2- 172.21.15.27:0/2202970384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d52f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:27:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.909+0000 7f787fe00700 1 --2- 172.21.15.27:0/2202970384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d534bc0 0x558a3d544f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:27:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.910+0000 7f7885600700 1 --2- 172.21.15.27:0/2202970384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d5354e0 0x558a3d537710 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f78780df940 tx=0x7f78780a9070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0a78f6e2ed771c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.910+0000 7f7871600700 1 -- 172.21.15.27:0/2202970384 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78780c2020 con 0x558a3d5354e0 2024-05-10T11:27:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.910+0000 7f7871600700 1 -- 172.21.15.27:0/2202970384 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78780a9c60 con 0x558a3d5354e0 2024-05-10T11:27:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.910+0000 7f7871600700 1 -- 172.21.15.27:0/2202970384 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78780b1f90 con 0x558a3d5354e0 2024-05-10T11:27:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.911+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2202970384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d5354e0 msgr2=0x558a3d537710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.911+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2202970384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d5354e0 0x558a3d537710 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f78780df940 tx=0x7f78780a9070 comp rx=0 tx=0).stop 2024-05-10T11:27:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.911+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2202970384 shutdown_connections 2024-05-10T11:27:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.911+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2202970384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d5354e0 0x558a3d537710 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.911+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2202970384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d52f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.911+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/2202970384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d534bc0 0x558a3d544f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.911+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2202970384 >> 172.21.15.27:0/2202970384 conn(0x558a3d4a76d0 msgr2=0x558a3d537e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.911+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2202970384 shutdown_connections 2024-05-10T11:27:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.911+0000 7f7891ed8b80 1 -- 172.21.15.27:0/2202970384 wait complete. 2024-05-10T11:27:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.912+0000 7f7891ed8b80 1 -- start start 2024-05-10T11:27:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.912+0000 7f7891ed8b80 1 Processor -- start 2024-05-10T11:27:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.912+0000 7f7891ed8b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d5412d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.912+0000 7f7891ed8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 0x558a3d541810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.912+0000 7f7891ed8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d5354e0 0x558a3d54f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.912+0000 7f7891ed8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f78801b2e30 con 0x558a3d5354e0 2024-05-10T11:27:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.912+0000 7f7891ed8b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f78801b2760 con 0x558a3d52f8f0 2024-05-10T11:27:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.912+0000 7f7891ed8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f78801af980 con 0x558a3d534bc0 2024-05-10T11:27:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.913+0000 7f7891ed8b80 10 client.0 _ll_register_callbacks cb 0x558a3d428c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.913+0000 7f787fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 0x558a3d541810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.913+0000 7f7885600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d5354e0 0x558a3d54f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.913+0000 7f787fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 0x558a3d541810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54668/0 (socket says 172.21.15.27:54668) 2024-05-10T11:27:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.913+0000 7f787fe00700 1 -- 172.21.15.27:0/3171355593 learned_addr learned my addr 172.21.15.27:0/3171355593 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.913+0000 7f7884c00700 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d5412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.913+0000 7f7884c00700 1 -- 172.21.15.27:0/3171355593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 msgr2=0x558a3d541810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.913+0000 7f7884c00700 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 0x558a3d541810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.913+0000 7f7884c00700 1 -- 172.21.15.27:0/3171355593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d5354e0 msgr2=0x558a3d54f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.913+0000 7f7884c00700 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d5354e0 0x558a3d54f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.913+0000 7f7884c00700 1 -- 172.21.15.27:0/3171355593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a3d54fbe0 con 0x558a3d52f8f0 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.914+0000 7f7885600700 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d5354e0 0x558a3d54f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.914+0000 7f7884c00700 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d5412d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7874009d10 tx=0x7f787402f700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1df67523c76df96 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.914+0000 7f7866a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.914+0000 7f7866a00700 1 -- 172.21.15.27:0/3171355593 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78740039c0 con 0x558a3d52f8f0 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.914+0000 7f7891ed8b80 1 -- 172.21.15.27:0/3171355593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78780a9c60 con 0x558a3d52f8f0 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.914+0000 7f7866a00700 1 -- 172.21.15.27:0/3171355593 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9679+0+0 (secure 0 0 0) 0x7f7874003b60 con 0x558a3d52f8f0 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.914+0000 7f7866a00700 1 -- 172.21.15.27:0/3171355593 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f787402fc50 con 0x558a3d52f8f0 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.914+0000 7f7891ed8b80 10 client.27209 fetch_fsmap learned FSMap version 36 2024-05-10T11:27:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.914+0000 7f7891ed8b80 1 -- 172.21.15.27:0/3171355593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558a3d5b2a50 con 0x558a3d52f8f0 2024-05-10T11:27:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.915+0000 7f7866a00700 1 -- 172.21.15.27:0/3171355593 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78780a9c60 con 0x558a3d52f8f0 2024-05-10T11:27:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.915+0000 7f7866a00700 1 -- 172.21.15.27:0/3171355593 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f7874010d40 con 0x558a3d52f8f0 2024-05-10T11:27:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.915+0000 7f7891ed8b80 10 client.27209 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:27:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.915+0000 7f7891ed8b80 10 client.27209 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:27:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.915+0000 7f7891ed8b80 20 client.27209 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.915+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x558a3d5b30a0 0x558a3d5b5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.915+0000 7f7891ed8b80 4 client.27209 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:27:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.915+0000 7f7891ed8b80 1 -- 172.21.15.27:0/3171355593 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558a3d48f400 con 0x558a3d5b30a0 2024-05-10T11:27:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.915+0000 7f787fe00700 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x558a3d5b30a0 0x558a3d5b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.916+0000 7f787fe00700 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x558a3d5b30a0 0x558a3d5b5490 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9676419b81c3ef54 server_cookie=ae0cc129568ad238 in_seq=0 out_seq=0 2024-05-10T11:27:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.916+0000 7f7866a00700 10 client.27209 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:27:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.916+0000 7f7866a00700 1 -- 172.21.15.27:0/3171355593 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558a3d48f400 con 0x558a3d5b30a0 2024-05-10T11:27:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.916+0000 7f7866a00700 10 client.27209 handle_command_reply: tid=0 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.916+0000 7f7891ed8b80 1 client.27209 shutdown 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.916+0000 7f7891ed8b80 1 -- 172.21.15.27:0/3171355593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 msgr2=0x558a3d5412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.916+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d5412d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7874009d10 tx=0x7f787402f700 comp rx=0 tx=0).stop 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.917+0000 7f7891ed8b80 1 -- 172.21.15.27:0/3171355593 shutdown_connections 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.917+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x558a3d5b30a0 0x558a3d5b5490 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.917+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a3d5354e0 0x558a3d54f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.917+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558a3d52f8f0 0x558a3d5412d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.917+0000 7f7866a00700 0 client.27209 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.917+0000 7f7891ed8b80 1 --2- 172.21.15.27:0/3171355593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3d534bc0 0x558a3d541810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.917+0000 7f7891ed8b80 1 -- 172.21.15.27:0/3171355593 >> 172.21.15.27:0/3171355593 conn(0x558a3d4a76d0 msgr2=0x558a3d537590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.917+0000 7f7891ed8b80 1 -- 172.21.15.27:0/3171355593 shutdown_connections 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.917+0000 7f7891ed8b80 1 -- 172.21.15.27:0/3171355593 wait complete. 2024-05-10T11:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.917+0000 7f7891ed8b80 20 client.27209 trim_cache size 0 max 16384 2024-05-10T11:27:04.921 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.918+0000 7f7887600700 1 -- 172.21.15.27:0/1590987257 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f78680b29f0 msgr2=0x7f78680b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.918+0000 7f7887600700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f78680b29f0 0x7f78680b4eb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7874009990 tx=0x7f7874005ab0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.918+0000 7f7887600700 1 -- 172.21.15.27:0/1590987257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7880108850 msgr2=0x7f78801a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.918+0000 7f7887600700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7880108850 0x7f78801a0610 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f787800efc0 tx=0x7f787800c3a0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.918+0000 7f7887600700 1 -- 172.21.15.27:0/1590987257 shutdown_connections 2024-05-10T11:27:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.918+0000 7f7887600700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f78680b29f0 0x7f78680b4eb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.918+0000 7f7887600700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7880103750 0x7f788019bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.918+0000 7f7887600700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7880108850 0x7f78801a0610 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.918+0000 7f7887600700 1 --2- 172.21.15.27:0/1590987257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7880104100 0x7f788019c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.918+0000 7f7887600700 1 -- 172.21.15.27:0/1590987257 >> 172.21.15.27:0/1590987257 conn(0x7f788007a600 msgr2=0x7f7880101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.918+0000 7f7887600700 1 -- 172.21.15.27:0/1590987257 shutdown_connections 2024-05-10T11:27:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:04.918+0000 7f7887600700 1 -- 172.21.15.27:0/1590987257 wait complete. 2024-05-10T11:27:05.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:04 smithi027 ceph-mon[30054]: pgmap v433: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 154 op/s 2024-05-10T11:27:05.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:04 smithi076 ceph-mon[34152]: pgmap v433: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 154 op/s 2024-05-10T11:27:05.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:04 smithi189 ceph-mon[34842]: pgmap v433: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 154 op/s 2024-05-10T11:27:05.752 INFO:tasks.workunit.client.0.smithi027.stdout: 1 834452 6.13 MB/sec execute 590 sec latency 23.254 ms 2024-05-10T11:27:05.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:27:06.456 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:27:06.752 INFO:tasks.workunit.client.0.smithi027.stdout: 1 835078 6.12 MB/sec execute 591 sec latency 30.535 ms 2024-05-10T11:27:06.982 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:06 smithi027 ceph-mon[30054]: pgmap v434: 129 pgs: 129 active+clean; 622 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 180 op/s 2024-05-10T11:27:07.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:06 smithi076 ceph-mon[34152]: pgmap v434: 129 pgs: 129 active+clean; 622 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 180 op/s 2024-05-10T11:27:07.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:06 smithi189 ceph-mon[34842]: pgmap v434: 129 pgs: 129 active+clean; 622 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 180 op/s 2024-05-10T11:27:07.752 INFO:tasks.workunit.client.0.smithi027.stdout: 1 836340 6.12 MB/sec execute 592 sec latency 28.213 ms 2024-05-10T11:27:08.755 INFO:tasks.workunit.client.0.smithi027.stdout: 1 837734 6.12 MB/sec execute 593 sec latency 102.678 ms 2024-05-10T11:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.981+0000 7f061aa00700 1 -- 172.21.15.27:0/3399149830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06141056c0 msgr2=0x7f0614105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.981+0000 7f061aa00700 1 --2- 172.21.15.27:0/3399149830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06141056c0 0x7f0614105b40 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0600009990 tx=0x7f0600009ca0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.981+0000 7f061aa00700 1 -- 172.21.15.27:0/3399149830 shutdown_connections 2024-05-10T11:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.981+0000 7f061aa00700 1 --2- 172.21.15.27:0/3399149830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0614106080 0x7f0614112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.981+0000 7f061aa00700 1 --2- 172.21.15.27:0/3399149830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061410b6c0 0x7f061410baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.981+0000 7f061aa00700 1 --2- 172.21.15.27:0/3399149830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06141056c0 0x7f0614105b40 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.981+0000 7f061aa00700 1 -- 172.21.15.27:0/3399149830 >> 172.21.15.27:0/3399149830 conn(0x7f06141011b0 msgr2=0x7f06141035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.981+0000 7f061aa00700 1 -- 172.21.15.27:0/3399149830 shutdown_connections 2024-05-10T11:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.981+0000 7f061aa00700 1 -- 172.21.15.27:0/3399149830 wait complete. 2024-05-10T11:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.982+0000 7f061aa00700 1 Processor -- start 2024-05-10T11:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.982+0000 7f061aa00700 1 -- start start 2024-05-10T11:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f061aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06141056c0 0x7f061419bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f061aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0614106080 0x7f061419c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f061aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f061410b6c0 0x7f06141a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f061aa00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0614115910 con 0x7f06141056c0 2024-05-10T11:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f061aa00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0614115790 con 0x7f061410b6c0 2024-05-10T11:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f061aa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0614115a90 con 0x7f0614106080 2024-05-10T11:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f0613400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0614106080 0x7f061419c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f0613400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0614106080 0x7f061419c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54690/0 (socket says 172.21.15.27:54690) 2024-05-10T11:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f0613400700 1 -- 172.21.15.27:0/471429472 learned_addr learned my addr 172.21.15.27:0/471429472 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f0618a00700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f061410b6c0 0x7f06141a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f0613e00700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06141056c0 0x7f061419bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f0613400700 1 -- 172.21.15.27:0/471429472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f061410b6c0 msgr2=0x7f06141a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.983+0000 7f0613400700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f061410b6c0 0x7f06141a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.984+0000 7f0613400700 1 -- 172.21.15.27:0/471429472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06141056c0 msgr2=0x7f061419bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.984+0000 7f0613400700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06141056c0 0x7f061419bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.984+0000 7f0613400700 1 -- 172.21.15.27:0/471429472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06141a0dd0 con 0x7f0614106080 2024-05-10T11:27:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.984+0000 7f0613400700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0614106080 0x7f061419c300 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0600005910 tx=0x7f060000b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.984+0000 7f0610c00700 1 -- 172.21.15.27:0/471429472 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600027070 con 0x7f0614106080 2024-05-10T11:27:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.984+0000 7f0610c00700 1 -- 172.21.15.27:0/471429472 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0600022470 con 0x7f0614106080 2024-05-10T11:27:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.984+0000 7f061aa00700 1 -- 172.21.15.27:0/471429472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06141a1060 con 0x7f0614106080 2024-05-10T11:27:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.984+0000 7f0610c00700 1 -- 172.21.15.27:0/471429472 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600017650 con 0x7f0614106080 2024-05-10T11:27:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.985+0000 7f061aa00700 1 -- 172.21.15.27:0/471429472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06141a8940 con 0x7f0614106080 2024-05-10T11:27:08.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.989+0000 7f0610c00700 1 -- 172.21.15.27:0/471429472 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f060002a070 con 0x7f0614106080 2024-05-10T11:27:08.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.990+0000 7f0625451b80 1 Processor -- start 2024-05-10T11:27:08.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.990+0000 7f0625451b80 1 -- start start 2024-05-10T11:27:08.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.990+0000 7f0625451b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87aafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:08.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.990+0000 7f0625451b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:08.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.990+0000 7f0625451b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87a58f0 0x5594a87a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:08.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.990+0000 7f0625451b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5594a86f7a20 con 0x5594a87a58f0 2024-05-10T11:27:08.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.990+0000 7f0625451b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5594a8695760 con 0x5594a87ab4e0 2024-05-10T11:27:08.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.990+0000 7f0625451b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5594a86f39e0 con 0x5594a87aabc0 2024-05-10T11:27:08.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0613e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:08.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0618a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87aafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:08.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0613e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87a52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33396/0 (socket says 172.21.15.27:33396) 2024-05-10T11:27:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0613e00700 1 -- 172.21.15.27:0/2599412969 learned_addr learned my addr 172.21.15.27:0/2599412969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0610c00700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f05f80b29f0 0x7f05f80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0613400700 1 --2- 172.21.15.27:0/2599412969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87a58f0 0x5594a87a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0613e00700 1 -- 172.21.15.27:0/2599412969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 msgr2=0x5594a87aafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0613e00700 1 --2- 172.21.15.27:0/2599412969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87aafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0613e00700 1 -- 172.21.15.27:0/2599412969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87a58f0 msgr2=0x5594a87a5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0613e00700 1 --2- 172.21.15.27:0/2599412969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87a58f0 0x5594a87a5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0610c00700 1 -- 172.21.15.27:0/471429472 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7f06000e36e0 con 0x7f0614106080 2024-05-10T11:27:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0613e00700 1 -- 172.21.15.27:0/2599412969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594a86448d0 con 0x5594a87ab4e0 2024-05-10T11:27:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0618a00700 1 --2- 172.21.15.27:0/2599412969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87aafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:27:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.991+0000 7f0618a00700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f05f80b29f0 0x7f05f80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.992+0000 7f0613e00700 1 --2- 172.21.15.27:0/2599412969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87a52d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0604009d00 tx=0x7f0604011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c9ad8b661a95d6e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.992+0000 7f0609600700 1 -- 172.21.15.27:0/2599412969 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604004230 con 0x5594a87ab4e0 2024-05-10T11:27:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.992+0000 7f0609600700 1 -- 172.21.15.27:0/2599412969 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06040043d0 con 0x5594a87ab4e0 2024-05-10T11:27:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.992+0000 7f0609600700 1 -- 172.21.15.27:0/2599412969 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604019920 con 0x5594a87ab4e0 2024-05-10T11:27:08.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.993+0000 7f0625451b80 1 -- 172.21.15.27:0/2599412969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 msgr2=0x5594a87a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:08.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.993+0000 7f0625451b80 1 --2- 172.21.15.27:0/2599412969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87a52d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0604009d00 tx=0x7f0604011720 comp rx=0 tx=0).stop 2024-05-10T11:27:08.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.993+0000 7f0618a00700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f05f80b29f0 0x7f05f80b4eb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f060c009ec0 tx=0x7f060c009040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.993+0000 7f0625451b80 1 -- 172.21.15.27:0/2599412969 shutdown_connections 2024-05-10T11:27:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.993+0000 7f0625451b80 1 --2- 172.21.15.27:0/2599412969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87aafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.993+0000 7f0625451b80 1 --2- 172.21.15.27:0/2599412969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87a58f0 0x5594a87a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.993+0000 7f0625451b80 1 --2- 172.21.15.27:0/2599412969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87a52d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.993+0000 7f0625451b80 1 -- 172.21.15.27:0/2599412969 >> 172.21.15.27:0/2599412969 conn(0x5594a871d6d0 msgr2=0x5594a879fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.993+0000 7f0625451b80 1 -- 172.21.15.27:0/2599412969 shutdown_connections 2024-05-10T11:27:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.993+0000 7f0625451b80 1 -- 172.21.15.27:0/2599412969 wait complete. 2024-05-10T11:27:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.994+0000 7f0625451b80 1 -- start start 2024-05-10T11:27:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.994+0000 7f0625451b80 1 Processor -- start 2024-05-10T11:27:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0625451b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87a58f0 0x5594a87b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0625451b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87ae570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0625451b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0625451b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5594a87b1f20 con 0x5594a87a58f0 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0625451b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5594a865db20 con 0x5594a87ab4e0 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0625451b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5594a87b20a0 con 0x5594a87aabc0 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0625451b80 10 client.0 _ll_register_callbacks cb 0x5594a869ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0613400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0613400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87ae570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54708/0 (socket says 172.21.15.27:54708) 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0613400700 1 -- 172.21.15.27:0/3586202483 learned_addr learned my addr 172.21.15.27:0/3586202483 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0618a00700 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0613e00700 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87a58f0 0x5594a87b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0613400700 1 -- 172.21.15.27:0/3586202483 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 msgr2=0x5594a87aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.995+0000 7f0613400700 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87aeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.996+0000 7f0613400700 1 -- 172.21.15.27:0/3586202483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87a58f0 msgr2=0x5594a87b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.996+0000 7f0613400700 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87a58f0 0x5594a87b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:08.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.996+0000 7f0613400700 1 -- 172.21.15.27:0/3586202483 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594a86bd420 con 0x5594a87aabc0 2024-05-10T11:27:08.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.996+0000 7f0613400700 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87ae570 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f06000aa2c0 tx=0x7f0600009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f71b56f56b6a9312 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:08.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.996+0000 7f05fe000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:27:08.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.996+0000 7f05fe000700 1 -- 172.21.15.27:0/3586202483 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06000a9d30 con 0x5594a87aabc0 2024-05-10T11:27:08.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.996+0000 7f05fe000700 1 -- 172.21.15.27:0/3586202483 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7f06000ae5f0 con 0x5594a87aabc0 2024-05-10T11:27:08.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.996+0000 7f0625451b80 1 -- 172.21.15.27:0/3586202483 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06040043d0 con 0x5594a87aabc0 2024-05-10T11:27:08.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.997+0000 7f05fe000700 1 -- 172.21.15.27:0/3586202483 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f06000aa0d0 con 0x5594a87aabc0 2024-05-10T11:27:08.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.997+0000 7f0625451b80 10 client.27208 fetch_fsmap learned FSMap version 36 2024-05-10T11:27:08.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.997+0000 7f0625451b80 1 -- 172.21.15.27:0/3586202483 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5594a87b1f20 con 0x5594a87aabc0 2024-05-10T11:27:08.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.997+0000 7f05fe000700 1 -- 172.21.15.27:0/3586202483 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06040043d0 con 0x5594a87aabc0 2024-05-10T11:27:08.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.997+0000 7f05fe000700 1 -- 172.21.15.27:0/3586202483 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f06000a4ce0 con 0x5594a87aabc0 2024-05-10T11:27:08.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.997+0000 7f0625451b80 10 client.27208 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:27:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.997+0000 7f0625451b80 10 client.27208 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:27:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.997+0000 7f0625451b80 20 client.27208 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.997+0000 7f0625451b80 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5594a87becc0 0x5594a87c10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.997+0000 7f0625451b80 4 client.27208 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:27:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.997+0000 7f0625451b80 1 -- 172.21.15.27:0/3586202483 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594a8772140 con 0x5594a87becc0 2024-05-10T11:27:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.998+0000 7f0613e00700 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5594a87becc0 0x5594a87c10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.998+0000 7f0613e00700 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5594a87becc0 0x5594a87c10f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cea339bcc1bd8a6f server_cookie=1f7de77887d835ef in_seq=0 out_seq=0 2024-05-10T11:27:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.998+0000 7f05fe000700 10 client.27208 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:27:09.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.999+0000 7f05fe000700 1 -- 172.21.15.27:0/3586202483 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5594a8772140 con 0x5594a87becc0 2024-05-10T11:27:09.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.999+0000 7f05fe000700 10 client.27208 handle_command_reply: tid=0 2024-05-10T11:27:09.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:08.999+0000 7f0625451b80 1 client.27208 shutdown 2024-05-10T11:27:09.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.000+0000 7f0625451b80 1 -- 172.21.15.27:0/3586202483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 msgr2=0x5594a87ae570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:09.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.000+0000 7f0625451b80 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87ae570 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f06000aa2c0 tx=0x7f0600009ca0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.000+0000 7f0625451b80 1 -- 172.21.15.27:0/3586202483 shutdown_connections 2024-05-10T11:27:09.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.000+0000 7f0625451b80 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.000+0000 7f0625451b80 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87a58f0 0x5594a87b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.000+0000 7f0625451b80 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5594a87becc0 0x5594a87c10f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.000+0000 7f0625451b80 1 --2- 172.21.15.27:0/3586202483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87ae570 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.000+0000 7f0625451b80 1 -- 172.21.15.27:0/3586202483 >> 172.21.15.27:0/3586202483 conn(0x5594a871d6d0 msgr2=0x5594a879f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:09.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.000+0000 7f05fe000700 0 client.27208 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:27:09.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.000+0000 7f0625451b80 1 -- 172.21.15.27:0/3586202483 shutdown_connections 2024-05-10T11:27:09.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.000+0000 7f0625451b80 1 -- 172.21.15.27:0/3586202483 wait complete. 2024-05-10T11:27:09.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.000+0000 7f0625451b80 20 client.27208 trim_cache size 0 max 16384 2024-05-10T11:27:09.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.019+0000 7f0625451b80 1 Processor -- start 2024-05-10T11:27:09.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.019+0000 7f0625451b80 1 -- start start 2024-05-10T11:27:09.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.019+0000 7f0625451b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87a58f0 0x5594a87a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:09.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.019+0000 7f0625451b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87c10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:09.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0625451b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87ab4e0 0x5594a87bafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:09.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0625451b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f06141b1880 con 0x5594a87ab4e0 2024-05-10T11:27:09.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0625451b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f06141b33c0 con 0x5594a87a58f0 2024-05-10T11:27:09.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0625451b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f06141b1f50 con 0x5594a87aabc0 2024-05-10T11:27:09.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0613e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87a58f0 0x5594a87a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:09.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0613e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87a58f0 0x5594a87a52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33410/0 (socket says 172.21.15.27:33410) 2024-05-10T11:27:09.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0613e00700 1 -- 172.21.15.27:0/2341210813 learned_addr learned my addr 172.21.15.27:0/2341210813 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:09.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0613400700 1 --2- 172.21.15.27:0/2341210813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87c10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0618a00700 1 --2- 172.21.15.27:0/2341210813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87ab4e0 0x5594a87bafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0613e00700 1 -- 172.21.15.27:0/2341210813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 msgr2=0x5594a87c10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0613e00700 1 --2- 172.21.15.27:0/2341210813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87c10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0613e00700 1 -- 172.21.15.27:0/2341210813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87ab4e0 msgr2=0x5594a87bafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0613e00700 1 --2- 172.21.15.27:0/2341210813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87ab4e0 0x5594a87bafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.020+0000 7f0613e00700 1 -- 172.21.15.27:0/2341210813 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594a87a5f20 con 0x5594a87a58f0 2024-05-10T11:27:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.021+0000 7f0613e00700 1 --2- 172.21.15.27:0/2341210813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87a58f0 0x5594a87a52d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x5594a87ab8e0 tx=0x7f0604011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a5a066f6f52b07a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.021+0000 7f0608c00700 1 -- 172.21.15.27:0/2341210813 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604031070 con 0x5594a87a58f0 2024-05-10T11:27:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.021+0000 7f0608c00700 1 -- 172.21.15.27:0/2341210813 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06040054d0 con 0x5594a87a58f0 2024-05-10T11:27:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.021+0000 7f0608c00700 1 -- 172.21.15.27:0/2341210813 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604004900 con 0x5594a87a58f0 2024-05-10T11:27:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.022+0000 7f0625451b80 1 -- 172.21.15.27:0/2341210813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87a58f0 msgr2=0x5594a87a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.022+0000 7f0625451b80 1 --2- 172.21.15.27:0/2341210813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87a58f0 0x5594a87a52d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x5594a87ab8e0 tx=0x7f0604011720 comp rx=0 tx=0).stop 2024-05-10T11:27:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.022+0000 7f0625451b80 1 -- 172.21.15.27:0/2341210813 shutdown_connections 2024-05-10T11:27:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.022+0000 7f0625451b80 1 --2- 172.21.15.27:0/2341210813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87ab4e0 0x5594a87bafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.022+0000 7f0625451b80 1 --2- 172.21.15.27:0/2341210813 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87a58f0 0x5594a87a52d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.022+0000 7f0625451b80 1 --2- 172.21.15.27:0/2341210813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87aabc0 0x5594a87c10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.022+0000 7f0625451b80 1 -- 172.21.15.27:0/2341210813 >> 172.21.15.27:0/2341210813 conn(0x5594a871d6d0 msgr2=0x5594a87ace60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.022+0000 7f0625451b80 1 -- 172.21.15.27:0/2341210813 shutdown_connections 2024-05-10T11:27:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.022+0000 7f0625451b80 1 -- 172.21.15.27:0/2341210813 wait complete. 2024-05-10T11:27:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.022+0000 7f0625451b80 1 -- start start 2024-05-10T11:27:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.023+0000 7f0625451b80 1 Processor -- start 2024-05-10T11:27:09.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.023+0000 7f0625451b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87a58f0 0x5594a87b76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:09.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.023+0000 7f0625451b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87aabc0 0x5594a87c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:09.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.023+0000 7f0625451b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87c6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:09.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.023+0000 7f0625451b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f06141b2cf0 con 0x5594a87aabc0 2024-05-10T11:27:09.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.023+0000 7f0625451b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f06141b2620 con 0x5594a87ab4e0 2024-05-10T11:27:09.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.023+0000 7f0625451b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f06141af840 con 0x5594a87a58f0 2024-05-10T11:27:09.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.023+0000 7f0625451b80 10 client.0 _ll_register_callbacks cb 0x5594a869ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:09.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f0613e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87a58f0 0x5594a87b76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:09.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f0613e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87a58f0 0x5594a87b76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54732/0 (socket says 172.21.15.27:54732) 2024-05-10T11:27:09.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f0613e00700 1 -- 172.21.15.27:0/2945707930 learned_addr learned my addr 172.21.15.27:0/2945707930 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:09.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f0618a00700 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87c6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:09.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f0613400700 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87aabc0 0x5594a87c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:09.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f0613e00700 1 -- 172.21.15.27:0/2945707930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 msgr2=0x5594a87c6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:09.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f0613e00700 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87c6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f0613e00700 1 -- 172.21.15.27:0/2945707930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87aabc0 msgr2=0x5594a87c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:09.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f0613e00700 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87aabc0 0x5594a87c5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f0613e00700 1 -- 172.21.15.27:0/2945707930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594a87c6970 con 0x5594a87a58f0 2024-05-10T11:27:09.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f0613400700 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87aabc0 0x5594a87c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:27:09.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f0613e00700 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87a58f0 0x5594a87b76d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f06040094f0 tx=0x7f0604009760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39485bd095e6f9d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:09.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.024+0000 7f05fe000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:27:09.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.025+0000 7f05fe000700 1 -- 172.21.15.27:0/2945707930 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060403d060 con 0x5594a87a58f0 2024-05-10T11:27:09.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.025+0000 7f05fe000700 1 -- 172.21.15.27:0/2945707930 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7f0604005680 con 0x5594a87a58f0 2024-05-10T11:27:09.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.025+0000 7f0625451b80 1 -- 172.21.15.27:0/2945707930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06040054d0 con 0x5594a87a58f0 2024-05-10T11:27:09.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.025+0000 7f05fe000700 1 -- 172.21.15.27:0/2945707930 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0604031050 con 0x5594a87a58f0 2024-05-10T11:27:09.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.025+0000 7f0625451b80 10 client.27214 fetch_fsmap learned FSMap version 36 2024-05-10T11:27:09.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.025+0000 7f0625451b80 1 -- 172.21.15.27:0/2945707930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5594a8829400 con 0x5594a87a58f0 2024-05-10T11:27:09.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.026+0000 7f05fe000700 1 -- 172.21.15.27:0/2945707930 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06040054d0 con 0x5594a87a58f0 2024-05-10T11:27:09.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.026+0000 7f05fe000700 1 -- 172.21.15.27:0/2945707930 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f060402bce0 con 0x5594a87a58f0 2024-05-10T11:27:09.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.026+0000 7f0625451b80 10 client.27214 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:27:09.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.026+0000 7f0625451b80 10 client.27214 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:27:09.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.026+0000 7f0625451b80 20 client.27214 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:09.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.026+0000 7f0625451b80 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5594a88297b0 0x5594a882bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:09.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.026+0000 7f0625451b80 4 client.27214 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:27:09.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.026+0000 7f0625451b80 1 -- 172.21.15.27:0/2945707930 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5594a8705400 con 0x5594a88297b0 2024-05-10T11:27:09.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.026+0000 7f0613400700 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5594a88297b0 0x5594a882bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:09.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f0613400700 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5594a88297b0 0x5594a882bba0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=896ae8bf6e039e5b server_cookie=28e4f61efc21af35 in_seq=0 out_seq=0 2024-05-10T11:27:09.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f05fe000700 10 client.27214 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:27:09.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f05fe000700 1 -- 172.21.15.27:0/2945707930 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5594a8705400 con 0x5594a88297b0 2024-05-10T11:27:09.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f05fe000700 10 client.27214 handle_command_reply: tid=0 2024-05-10T11:27:09.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f0625451b80 1 client.27214 shutdown 2024-05-10T11:27:09.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f0625451b80 1 -- 172.21.15.27:0/2945707930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87a58f0 msgr2=0x5594a87b76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:09.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f0625451b80 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87a58f0 0x5594a87b76d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f06040094f0 tx=0x7f0604009760 comp rx=0 tx=0).stop 2024-05-10T11:27:09.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f0625451b80 1 -- 172.21.15.27:0/2945707930 shutdown_connections 2024-05-10T11:27:09.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f0625451b80 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x5594a88297b0 0x5594a882bba0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f0625451b80 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a87a58f0 0x5594a87b76d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f0625451b80 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a87ab4e0 0x5594a87c6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f0625451b80 1 --2- 172.21.15.27:0/2945707930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a87aabc0 0x5594a87c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f05fe000700 0 client.27214 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:27:09.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f0625451b80 1 -- 172.21.15.27:0/2945707930 >> 172.21.15.27:0/2945707930 conn(0x5594a871d6d0 msgr2=0x5594a87aca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:09.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f0625451b80 1 -- 172.21.15.27:0/2945707930 shutdown_connections 2024-05-10T11:27:09.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.027+0000 7f0625451b80 1 -- 172.21.15.27:0/2945707930 wait complete. 2024-05-10T11:27:09.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.028+0000 7f0625451b80 20 client.27214 trim_cache size 0 max 16384 2024-05-10T11:27:09.033 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:27:09.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.028+0000 7f061aa00700 1 -- 172.21.15.27:0/471429472 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f05f80b29f0 msgr2=0x7f05f80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:09.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.028+0000 7f061aa00700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f05f80b29f0 0x7f05f80b4eb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f060c009ec0 tx=0x7f060c009040 comp rx=0 tx=0).stop 2024-05-10T11:27:09.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.028+0000 7f061aa00700 1 -- 172.21.15.27:0/471429472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0614106080 msgr2=0x7f061419c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:09.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.028+0000 7f061aa00700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0614106080 0x7f061419c300 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0600005910 tx=0x7f060000b670 comp rx=0 tx=0).stop 2024-05-10T11:27:09.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.028+0000 7f061aa00700 1 -- 172.21.15.27:0/471429472 shutdown_connections 2024-05-10T11:27:09.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.028+0000 7f061aa00700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f05f80b29f0 0x7f05f80b4eb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.028+0000 7f061aa00700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f061410b6c0 0x7f06141a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.028+0000 7f061aa00700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06141056c0 0x7f061419bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.028+0000 7f061aa00700 1 --2- 172.21.15.27:0/471429472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0614106080 0x7f061419c300 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:09.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.028+0000 7f061aa00700 1 -- 172.21.15.27:0/471429472 >> 172.21.15.27:0/471429472 conn(0x7f06141011b0 msgr2=0x7f06141018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:09.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.029+0000 7f061aa00700 1 -- 172.21.15.27:0/471429472 shutdown_connections 2024-05-10T11:27:09.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:09.029+0000 7f061aa00700 1 -- 172.21.15.27:0/471429472 wait complete. 2024-05-10T11:27:09.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:08 smithi027 ceph-mon[30054]: pgmap v435: 129 pgs: 129 active+clean; 622 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 180 op/s 2024-05-10T11:27:09.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:08 smithi027 ceph-mon[30054]: osdmap e107: 12 total, 12 up, 12 in 2024-05-10T11:27:09.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:08 smithi076 ceph-mon[34152]: pgmap v435: 129 pgs: 129 active+clean; 622 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 180 op/s 2024-05-10T11:27:09.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:08 smithi076 ceph-mon[34152]: osdmap e107: 12 total, 12 up, 12 in 2024-05-10T11:27:09.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:08 smithi189 ceph-mon[34842]: pgmap v435: 129 pgs: 129 active+clean; 622 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 180 op/s 2024-05-10T11:27:09.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:08 smithi189 ceph-mon[34842]: osdmap e107: 12 total, 12 up, 12 in 2024-05-10T11:27:09.755 INFO:tasks.workunit.client.0.smithi027.stdout: 1 838273 6.12 MB/sec execute 594 sec latency 23.031 ms 2024-05-10T11:27:10.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:27:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:27:10.755 INFO:tasks.workunit.client.0.smithi027.stdout: 1 839379 6.12 MB/sec execute 595 sec latency 34.163 ms 2024-05-10T11:27:11.013 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:10 smithi027 ceph-mon[30054]: pgmap v437: 129 pgs: 129 active+clean; 629 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 224 op/s 2024-05-10T11:27:11.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:10 smithi076 ceph-mon[34152]: pgmap v437: 129 pgs: 129 active+clean; 629 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 224 op/s 2024-05-10T11:27:11.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:10 smithi189 ceph-mon[34842]: pgmap v437: 129 pgs: 129 active+clean; 629 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 224 op/s 2024-05-10T11:27:11.755 INFO:tasks.workunit.client.0.smithi027.stdout: 1 840697 6.12 MB/sec execute 596 sec latency 46.259 ms 2024-05-10T11:27:12.755 INFO:tasks.workunit.client.0.smithi027.stdout: 1 841677 6.12 MB/sec execute 597 sec latency 27.655 ms 2024-05-10T11:27:13.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:12 smithi027 ceph-mon[30054]: pgmap v438: 129 pgs: 129 active+clean; 629 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 184 op/s 2024-05-10T11:27:13.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:12 smithi076 ceph-mon[34152]: pgmap v438: 129 pgs: 129 active+clean; 629 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 184 op/s 2024-05-10T11:27:13.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:12 smithi189 ceph-mon[34842]: pgmap v438: 129 pgs: 129 active+clean; 629 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 184 op/s 2024-05-10T11:27:13.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.324+0000 7f1ff2400700 1 -- 172.21.15.27:0/1804236515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec1060a0 msgr2=0x7f1fec1125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:13.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.324+0000 7f1ff2400700 1 --2- 172.21.15.27:0/1804236515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec1060a0 0x7f1fec1125b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400b0a0 tx=0x7f1fe400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.325+0000 7f1ff2400700 1 -- 172.21.15.27:0/1804236515 shutdown_connections 2024-05-10T11:27:13.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.325+0000 7f1ff2400700 1 --2- 172.21.15.27:0/1804236515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec1060a0 0x7f1fec1125b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.325+0000 7f1ff2400700 1 --2- 172.21.15.27:0/1804236515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec10b6e0 0x7f1fec10bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.325+0000 7f1ff2400700 1 --2- 172.21.15.27:0/1804236515 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fec1056e0 0x7f1fec105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.325+0000 7f1ff2400700 1 -- 172.21.15.27:0/1804236515 >> 172.21.15.27:0/1804236515 conn(0x7f1fec1011d0 msgr2=0x7f1fec1035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:13.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.325+0000 7f1ff2400700 1 -- 172.21.15.27:0/1804236515 shutdown_connections 2024-05-10T11:27:13.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.325+0000 7f1ff2400700 1 -- 172.21.15.27:0/1804236515 wait complete. 2024-05-10T11:27:13.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.326+0000 7f1ff2400700 1 Processor -- start 2024-05-10T11:27:13.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.326+0000 7f1ff2400700 1 -- start start 2024-05-10T11:27:13.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1ff2400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fec1056e0 0x7f1fec19bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1ff2400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec1060a0 0x7f1fec19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1ff2400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec10b6e0 0x7f1fec1a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1ff2400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec115970 con 0x7f1fec10b6e0 2024-05-10T11:27:13.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1ff2400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1157f0 con 0x7f1fec1056e0 2024-05-10T11:27:13.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1ff2400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec115af0 con 0x7f1fec1060a0 2024-05-10T11:27:13.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1feb400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fec1056e0 0x7f1fec19bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1feb400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fec1056e0 0x7f1fec19bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33434/0 (socket says 172.21.15.27:33434) 2024-05-10T11:27:13.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1feb400700 1 -- 172.21.15.27:0/2740121475 learned_addr learned my addr 172.21.15.27:0/2740121475 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:13.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1febe00700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec10b6e0 0x7f1fec1a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1feaa00700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec1060a0 0x7f1fec19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1feb400700 1 -- 172.21.15.27:0/2740121475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec1060a0 msgr2=0x7f1fec19c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:13.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1feb400700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec1060a0 0x7f1fec19c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.327+0000 7f1feb400700 1 -- 172.21.15.27:0/2740121475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec10b6e0 msgr2=0x7f1fec1a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:13.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.328+0000 7f1feb400700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec10b6e0 0x7f1fec1a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.328+0000 7f1feb400700 1 -- 172.21.15.27:0/2740121475 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fec1a0d40 con 0x7f1fec1056e0 2024-05-10T11:27:13.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.328+0000 7f1feb400700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fec1056e0 0x7f1fec19bd80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00e9a0 tx=0x7f1fdc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:13.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.328+0000 7f1fe3e00700 1 -- 172.21.15.27:0/2740121475 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc00ca10 con 0x7f1fec1056e0 2024-05-10T11:27:13.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.328+0000 7f1fe3e00700 1 -- 172.21.15.27:0/2740121475 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fdc00cbb0 con 0x7f1fec1056e0 2024-05-10T11:27:13.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.328+0000 7f1ff2400700 1 -- 172.21.15.27:0/2740121475 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fec1a1030 con 0x7f1fec1056e0 2024-05-10T11:27:13.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.328+0000 7f1fe3e00700 1 -- 172.21.15.27:0/2740121475 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc018680 con 0x7f1fec1056e0 2024-05-10T11:27:13.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.328+0000 7f1ff2400700 1 -- 172.21.15.27:0/2740121475 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fec1a88b0 con 0x7f1fec1056e0 2024-05-10T11:27:13.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.332+0000 7f1fe3e00700 1 -- 172.21.15.27:0/2740121475 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f1fdc010430 con 0x7f1fec1056e0 2024-05-10T11:27:13.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.333+0000 7f1ffce60b80 1 Processor -- start 2024-05-10T11:27:13.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.334+0000 7f1ffce60b80 1 -- start start 2024-05-10T11:27:13.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.334+0000 7f1ffce60b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e638fc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.334+0000 7f1fe3e00700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f1fd00b2a70 0x7f1fd00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.334+0000 7f1ffce60b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e638fc980 0x564e63900490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.334+0000 7f1ffce60b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e639009d0 0x564e63900e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.334+0000 7f1ffce60b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564e6384ea20 con 0x564e639009d0 2024-05-10T11:27:13.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.334+0000 7f1ffce60b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564e637ec760 con 0x564e638fc060 2024-05-10T11:27:13.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.334+0000 7f1ffce60b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564e6384a9e0 con 0x564e638fc980 2024-05-10T11:27:13.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.334+0000 7f1feaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e638fc980 0x564e63900490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.334+0000 7f1fe3e00700 1 -- 172.21.15.27:0/2740121475 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7f1fdc0e2b30 con 0x7f1fec1056e0 2024-05-10T11:27:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.334+0000 7f1feaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e638fc980 0x564e63900490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54750/0 (socket says 172.21.15.27:54750) 2024-05-10T11:27:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.334+0000 7f1feaa00700 1 -- 172.21.15.27:0/2542601202 learned_addr learned my addr 172.21.15.27:0/2542601202 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.335+0000 7f1febe00700 1 --2- 172.21.15.27:0/2542601202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e638fc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.335+0000 7f1febe00700 1 --2- 172.21.15.27:0/2542601202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e639009d0 0x564e63900e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.335+0000 7f1feb400700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f1fd00b2a70 0x7f1fd00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.335+0000 7f1feaa00700 1 -- 172.21.15.27:0/2542601202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 msgr2=0x564e638fc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.335+0000 7f1feaa00700 1 --2- 172.21.15.27:0/2542601202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e638fc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.335+0000 7f1feaa00700 1 -- 172.21.15.27:0/2542601202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e639009d0 msgr2=0x564e63900e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.335+0000 7f1feaa00700 1 --2- 172.21.15.27:0/2542601202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e639009d0 0x564e63900e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.335+0000 7f1feaa00700 1 -- 172.21.15.27:0/2542601202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e6379b8d0 con 0x564e638fc980 2024-05-10T11:27:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.335+0000 7f1febe00700 1 --2- 172.21.15.27:0/2542601202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e639009d0 0x564e63900e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:27:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.335+0000 7f1feaa00700 1 --2- 172.21.15.27:0/2542601202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e638fc980 0x564e63900490 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8009d30 tx=0x7f1fd8011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ea306dc27a3e6da server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.335+0000 7f1fe0c00700 1 -- 172.21.15.27:0/2542601202 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd801e070 con 0x564e638fc980 2024-05-10T11:27:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.336+0000 7f1feb400700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f1fd00b2a70 0x7f1fd00b4f30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0dfe50 tx=0x7f1fdc010c10 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.336+0000 7f1fe0c00700 1 -- 172.21.15.27:0/2542601202 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fd8019ae0 con 0x564e638fc980 2024-05-10T11:27:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.336+0000 7f1fe0c00700 1 -- 172.21.15.27:0/2542601202 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd800ea90 con 0x564e638fc980 2024-05-10T11:27:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.336+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2542601202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e638fc980 msgr2=0x564e63900490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.336+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2542601202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e638fc980 0x564e63900490 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8009d30 tx=0x7f1fd8011720 comp rx=0 tx=0).stop 2024-05-10T11:27:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.336+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2542601202 shutdown_connections 2024-05-10T11:27:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.336+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2542601202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e639009d0 0x564e63900e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.336+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2542601202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e638fc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.336+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2542601202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e638fc980 0x564e63900490 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.336+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2542601202 >> 172.21.15.27:0/2542601202 conn(0x564e6382ab60 msgr2=0x564e638f67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.337+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2542601202 shutdown_connections 2024-05-10T11:27:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.337+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2542601202 wait complete. 2024-05-10T11:27:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.337+0000 7f1ffce60b80 1 -- start start 2024-05-10T11:27:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.337+0000 7f1ffce60b80 1 Processor -- start 2024-05-10T11:27:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.338+0000 7f1ffce60b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e63907e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.338+0000 7f1ffce60b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc980 0x564e639051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.338+0000 7f1ffce60b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e63905720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.338+0000 7f1ffce60b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564e63908be0 con 0x564e638fc980 2024-05-10T11:27:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.338+0000 7f1ffce60b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564e637b4b20 con 0x564e638fc060 2024-05-10T11:27:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.338+0000 7f1ffce60b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564e63908d60 con 0x564e639009d0 2024-05-10T11:27:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.338+0000 7f1ffce60b80 10 client.0 _ll_register_callbacks cb 0x564e637f5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.338+0000 7f1febe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e63907e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.338+0000 7f1feaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc980 0x564e639051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.338+0000 7f1febe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e63907e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33462/0 (socket says 172.21.15.27:33462) 2024-05-10T11:27:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.338+0000 7f1febe00700 1 -- 172.21.15.27:0/2494198132 learned_addr learned my addr 172.21.15.27:0/2494198132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.338+0000 7f1febe00700 1 --2- 172.21.15.27:0/2494198132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e63905720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.339+0000 7f1febe00700 1 -- 172.21.15.27:0/2494198132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 msgr2=0x564e63905720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.339+0000 7f1febe00700 1 --2- 172.21.15.27:0/2494198132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e63905720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.339+0000 7f1febe00700 1 -- 172.21.15.27:0/2494198132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc980 msgr2=0x564e639051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.339+0000 7f1febe00700 1 --2- 172.21.15.27:0/2494198132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc980 0x564e639051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.339+0000 7f1febe00700 1 -- 172.21.15.27:0/2494198132 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e63814420 con 0x564e638fc060 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.339+0000 7f1febe00700 1 --2- 172.21.15.27:0/2494198132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e63907e40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400d010 tx=0x7f1fe400b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af016514cd23b7c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.339+0000 7f1fd5600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.339+0000 7f1fd5600700 1 -- 172.21.15.27:0/2494198132 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe40078f0 con 0x564e638fc060 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.339+0000 7f1fd5600700 1 -- 172.21.15.27:0/2494198132 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7f1fe4007a90 con 0x564e638fc060 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.339+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2494198132 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fd8019ae0 con 0x564e638fc060 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.340+0000 7f1fd5600700 1 -- 172.21.15.27:0/2494198132 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1fe4007c90 con 0x564e638fc060 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.340+0000 7f1ffce60b80 10 client.27239 fetch_fsmap learned FSMap version 36 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.340+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2494198132 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564e639133e0 con 0x564e638fc060 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.340+0000 7f1fd5600700 1 -- 172.21.15.27:0/2494198132 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fd8019ae0 con 0x564e638fc060 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.340+0000 7f1fd5600700 1 -- 172.21.15.27:0/2494198132 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f1fe4011070 con 0x564e638fc060 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.341+0000 7f1ffce60b80 10 client.27239 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.341+0000 7f1ffce60b80 10 client.27239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.341+0000 7f1ffce60b80 20 client.27239 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.341+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2494198132 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564e63915a20 0x564e63917e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.341+0000 7f1ffce60b80 4 client.27239 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:27:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.341+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2494198132 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e638c90a0 con 0x564e63915a20 2024-05-10T11:27:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.341+0000 7f1feaa00700 1 --2- 172.21.15.27:0/2494198132 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564e63915a20 0x564e63917e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.342+0000 7f1feaa00700 1 --2- 172.21.15.27:0/2494198132 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564e63915a20 0x564e63917e10 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5a248f72c42d79c server_cookie=72e288d3a18185b9 in_seq=0 out_seq=0 2024-05-10T11:27:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.342+0000 7f1fd5600700 10 client.27239 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1fd5600700 1 -- 172.21.15.27:0/2494198132 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564e638c90a0 con 0x564e63915a20 2024-05-10T11:27:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1fd5600700 10 client.27239 handle_command_reply: tid=0 2024-05-10T11:27:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1ffce60b80 1 client.27239 shutdown 2024-05-10T11:27:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2494198132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 msgr2=0x564e63907e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2494198132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e63907e40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400d010 tx=0x7f1fe400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2494198132 shutdown_connections 2024-05-10T11:27:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2494198132 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564e63915a20 0x564e63917e10 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2494198132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e63905720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2494198132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e63907e40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2494198132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc980 0x564e639051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1fd5600700 0 client.27239 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:13.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2494198132 >> 172.21.15.27:0/2494198132 conn(0x564e6382ab60 msgr2=0x564e638f12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:13.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.343+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2494198132 shutdown_connections 2024-05-10T11:27:13.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.344+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2494198132 wait complete. 2024-05-10T11:27:13.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.344+0000 7f1ffce60b80 20 client.27239 trim_cache size 0 max 16384 2024-05-10T11:27:13.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.363+0000 7f1ffce60b80 1 Processor -- start 2024-05-10T11:27:13.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.363+0000 7f1ffce60b80 1 -- start start 2024-05-10T11:27:13.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.363+0000 7f1ffce60b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e63911ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1ffce60b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc980 0x564e63904360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1ffce60b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e638f1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1ffce60b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1b1870 con 0x564e638fc980 2024-05-10T11:27:13.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1ffce60b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1b33b0 con 0x564e638fc060 2024-05-10T11:27:13.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1ffce60b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1b1f40 con 0x564e639009d0 2024-05-10T11:27:13.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1febe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e63911ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1febe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e63911ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33470/0 (socket says 172.21.15.27:33470) 2024-05-10T11:27:13.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1febe00700 1 -- 172.21.15.27:0/2904953593 learned_addr learned my addr 172.21.15.27:0/2904953593 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1feaa00700 1 --2- 172.21.15.27:0/2904953593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc980 0x564e63904360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1febe00700 1 --2- 172.21.15.27:0/2904953593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e638f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1febe00700 1 -- 172.21.15.27:0/2904953593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 msgr2=0x564e638f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1febe00700 1 --2- 172.21.15.27:0/2904953593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e638f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1febe00700 1 -- 172.21.15.27:0/2904953593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc980 msgr2=0x564e63904360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1febe00700 1 --2- 172.21.15.27:0/2904953593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc980 0x564e63904360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.364+0000 7f1febe00700 1 -- 172.21.15.27:0/2904953593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e63904fb0 con 0x564e638fc060 2024-05-10T11:27:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.365+0000 7f1febe00700 1 --2- 172.21.15.27:0/2904953593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e63911ca0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4005a10 tx=0x7f1fe4007e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0c8c5eb1753dda server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.365+0000 7f1fd7e00700 1 -- 172.21.15.27:0/2904953593 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe401e430 con 0x564e638fc060 2024-05-10T11:27:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.365+0000 7f1fd7e00700 1 -- 172.21.15.27:0/2904953593 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fe401e5d0 con 0x564e638fc060 2024-05-10T11:27:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.365+0000 7f1fd7e00700 1 -- 172.21.15.27:0/2904953593 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4005040 con 0x564e638fc060 2024-05-10T11:27:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.366+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2904953593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 msgr2=0x564e63911ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.366+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2904953593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e63911ca0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4005a10 tx=0x7f1fe4007e30 comp rx=0 tx=0).stop 2024-05-10T11:27:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.366+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2904953593 shutdown_connections 2024-05-10T11:27:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.366+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2904953593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e638f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.366+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2904953593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc060 0x564e63911ca0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.366+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/2904953593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc980 0x564e63904360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.366+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2904953593 >> 172.21.15.27:0/2904953593 conn(0x564e6382ab60 msgr2=0x564e638f0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.366+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2904953593 shutdown_connections 2024-05-10T11:27:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.366+0000 7f1ffce60b80 1 -- 172.21.15.27:0/2904953593 wait complete. 2024-05-10T11:27:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.367+0000 7f1ffce60b80 1 -- start start 2024-05-10T11:27:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.367+0000 7f1ffce60b80 1 Processor -- start 2024-05-10T11:27:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1ffce60b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc060 0x564e6390e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1ffce60b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc980 0x564e63909690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1ffce60b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e63909bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1ffce60b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1b2ce0 con 0x564e638fc060 2024-05-10T11:27:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1ffce60b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1b2610 con 0x564e638fc980 2024-05-10T11:27:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1ffce60b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1af830 con 0x564e639009d0 2024-05-10T11:27:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1ffce60b80 10 client.0 _ll_register_callbacks cb 0x564e637f5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1feaa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc980 0x564e63909690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1feaa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc980 0x564e63909690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:33482/0 (socket says 172.21.15.27:33482) 2024-05-10T11:27:13.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1feaa00700 1 -- 172.21.15.27:0/1096718330 learned_addr learned my addr 172.21.15.27:0/1096718330 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:13.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1febe00700 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc060 0x564e6390e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1febe00700 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e63909bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1feaa00700 1 -- 172.21.15.27:0/1096718330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 msgr2=0x564e63909bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:13.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1feaa00700 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e63909bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1feaa00700 1 -- 172.21.15.27:0/1096718330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc060 msgr2=0x564e6390e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:13.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.368+0000 7f1feaa00700 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc060 0x564e6390e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.369+0000 7f1feaa00700 1 -- 172.21.15.27:0/1096718330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e6390a190 con 0x564e638fc980 2024-05-10T11:27:13.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.369+0000 7f1febe00700 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e63909bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:27:13.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.369+0000 7f1feaa00700 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc980 0x564e63909690 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1fd801c040 tx=0x7f1fd8005ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a53e20c48ec7c3e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:13.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.369+0000 7f1fd5600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:27:13.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.369+0000 7f1fd5600700 1 -- 172.21.15.27:0/1096718330 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd801e040 con 0x564e638fc980 2024-05-10T11:27:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.369+0000 7f1ffce60b80 1 -- 172.21.15.27:0/1096718330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fe401e5d0 con 0x564e638fc980 2024-05-10T11:27:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.369+0000 7f1fd5600700 1 -- 172.21.15.27:0/1096718330 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7f1fd8019950 con 0x564e638fc980 2024-05-10T11:27:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.370+0000 7f1fd5600700 1 -- 172.21.15.27:0/1096718330 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1fd80050b0 con 0x564e638fc980 2024-05-10T11:27:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.370+0000 7f1ffce60b80 10 client.27245 fetch_fsmap learned FSMap version 36 2024-05-10T11:27:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.370+0000 7f1ffce60b80 1 -- 172.21.15.27:0/1096718330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564e6397fff0 con 0x564e638fc980 2024-05-10T11:27:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.370+0000 7f1fd5600700 1 -- 172.21.15.27:0/1096718330 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fe401e5d0 con 0x564e638fc980 2024-05-10T11:27:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.370+0000 7f1fd5600700 1 -- 172.21.15.27:0/1096718330 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f1fd8003cc0 con 0x564e638fc980 2024-05-10T11:27:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.370+0000 7f1ffce60b80 10 client.27245 fetch_fsmap finished waiting for FSMap version 36 2024-05-10T11:27:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.370+0000 7f1ffce60b80 10 client.27245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:27:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.370+0000 7f1ffce60b80 20 client.27245 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:13.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.370+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564e63980510 0x564e63982930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:13.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.370+0000 7f1ffce60b80 4 client.27245 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:27:13.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.370+0000 7f1ffce60b80 1 -- 172.21.15.27:0/1096718330 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e6385c400 con 0x564e63980510 2024-05-10T11:27:13.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.371+0000 7f1febe00700 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564e63980510 0x564e63982930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:13.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.371+0000 7f1febe00700 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564e63980510 0x564e63982930 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffb7f3560cb572f2 server_cookie=3dbe8708582d2497 in_seq=0 out_seq=0 2024-05-10T11:27:13.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.371+0000 7f1fd5600700 10 client.27245 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:13.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.371+0000 7f1fd5600700 1 -- 172.21.15.27:0/1096718330 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e6385c400 con 0x564e63980510 2024-05-10T11:27:13.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.371+0000 7f1fd5600700 10 client.27245 handle_command_reply: tid=0 2024-05-10T11:27:13.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.371+0000 7f1ffce60b80 1 client.27245 shutdown 2024-05-10T11:27:13.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.372+0000 7f1ffce60b80 1 -- 172.21.15.27:0/1096718330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc980 msgr2=0x564e63909690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:13.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.372+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc980 0x564e63909690 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1fd801c040 tx=0x7f1fd8005ce0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.372+0000 7f1ffce60b80 1 -- 172.21.15.27:0/1096718330 shutdown_connections 2024-05-10T11:27:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.372+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564e63980510 0x564e63982930 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.372+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e639009d0 0x564e63909bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.372+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e638fc060 0x564e6390e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.372+0000 7f1ffce60b80 1 --2- 172.21.15.27:0/1096718330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564e638fc980 0x564e63909690 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.372+0000 7f1fd5600700 0 client.27245 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.372+0000 7f1ffce60b80 1 -- 172.21.15.27:0/1096718330 >> 172.21.15.27:0/1096718330 conn(0x564e6382ab60 msgr2=0x564e638f12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.372+0000 7f1ffce60b80 1 -- 172.21.15.27:0/1096718330 shutdown_connections 2024-05-10T11:27:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.372+0000 7f1ffce60b80 1 -- 172.21.15.27:0/1096718330 wait complete. 2024-05-10T11:27:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.372+0000 7f1ffce60b80 20 client.27245 trim_cache size 0 max 16384 2024-05-10T11:27:13.376 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:27:13.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.373+0000 7f1ff2400700 1 -- 172.21.15.27:0/2740121475 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f1fd00b2a70 msgr2=0x7f1fd00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:13.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.373+0000 7f1ff2400700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f1fd00b2a70 0x7f1fd00b4f30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0dfe50 tx=0x7f1fdc010c10 comp rx=0 tx=0).stop 2024-05-10T11:27:13.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.373+0000 7f1ff2400700 1 -- 172.21.15.27:0/2740121475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fec1056e0 msgr2=0x7f1fec19bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:13.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.373+0000 7f1ff2400700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fec1056e0 0x7f1fec19bd80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00e9a0 tx=0x7f1fdc00ecb0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.373+0000 7f1ff2400700 1 -- 172.21.15.27:0/2740121475 shutdown_connections 2024-05-10T11:27:13.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.373+0000 7f1ff2400700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f1fd00b2a70 0x7f1fd00b4f30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.373+0000 7f1ff2400700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec10b6e0 0x7f1fec1a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.373+0000 7f1ff2400700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fec1056e0 0x7f1fec19bd80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.373+0000 7f1ff2400700 1 --2- 172.21.15.27:0/2740121475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec1060a0 0x7f1fec19c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:13.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.373+0000 7f1ff2400700 1 -- 172.21.15.27:0/2740121475 >> 172.21.15.27:0/2740121475 conn(0x7f1fec1011d0 msgr2=0x7f1fec101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:13.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.374+0000 7f1ff2400700 1 -- 172.21.15.27:0/2740121475 shutdown_connections 2024-05-10T11:27:13.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:13.374+0000 7f1ff2400700 1 -- 172.21.15.27:0/2740121475 wait complete. 2024-05-10T11:27:13.755 INFO:tasks.workunit.client.0.smithi027.stdout: 1 842258 6.11 MB/sec execute 598 sec latency 53.609 ms 2024-05-10T11:27:14.756 INFO:tasks.workunit.client.0.smithi027.stdout: 1 843342 6.11 MB/sec execute 599 sec latency 73.770 ms 2024-05-10T11:27:15.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:14 smithi076 ceph-mon[34152]: pgmap v439: 129 pgs: 129 active+clean; 629 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 179 op/s 2024-05-10T11:27:15.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:14 smithi189 ceph-mon[34842]: pgmap v439: 129 pgs: 129 active+clean; 629 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 179 op/s 2024-05-10T11:27:15.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:14 smithi027 ceph-mon[30054]: pgmap v439: 129 pgs: 129 active+clean; 629 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 179 op/s 2024-05-10T11:27:15.616 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:27:15.617 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:27:15.756 INFO:tasks.workunit.client.0.smithi027.stdout: 1 cleanup 600 sec 2024-05-10T11:27:16.033 INFO:tasks.workunit.client.0.smithi027.stdout: 0 cleanup 600 sec 2024-05-10T11:27:16.033 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-05-10T11:27:16.033 INFO:tasks.workunit.client.0.smithi027.stdout: Operation Count AvgLat MaxLat 2024-05-10T11:27:16.033 INFO:tasks.workunit.client.0.smithi027.stdout: ---------------------------------------- 2024-05-10T11:27:16.033 INFO:tasks.workunit.client.0.smithi027.stdout: NTCreateX 117090 1.897 4223.150 2024-05-10T11:27:16.033 INFO:tasks.workunit.client.0.smithi027.stdout: Close 86045 0.010 2.405 2024-05-10T11:27:16.033 INFO:tasks.workunit.client.0.smithi027.stdout: Rename 4963 5.682 83.643 2024-05-10T11:27:16.033 INFO:tasks.workunit.client.0.smithi027.stdout: Unlink 23618 2.324 5010.208 2024-05-10T11:27:16.033 INFO:tasks.workunit.client.0.smithi027.stdout: Deltree 2 108.192 117.826 2024-05-10T11:27:16.033 INFO:tasks.workunit.client.0.smithi027.stdout: Mkdir 1 0.040 0.040 2024-05-10T11:27:16.034 INFO:tasks.workunit.client.0.smithi027.stdout: Qpathinfo 106162 1.534 7479.671 2024-05-10T11:27:16.034 INFO:tasks.workunit.client.0.smithi027.stdout: Qfileinfo 18593 0.005 0.245 2024-05-10T11:27:16.034 INFO:tasks.workunit.client.0.smithi027.stdout: Qfsinfo 19467 0.532 32.389 2024-05-10T11:27:16.034 INFO:tasks.workunit.client.0.smithi027.stdout: Sfileinfo 9552 0.079 21.774 2024-05-10T11:27:16.034 INFO:tasks.workunit.client.0.smithi027.stdout: Find 41060 0.724 8540.691 2024-05-10T11:27:16.034 INFO:tasks.workunit.client.0.smithi027.stdout: WriteX 58253 0.269 219.218 2024-05-10T11:27:16.034 INFO:tasks.workunit.client.0.smithi027.stdout: ReadX 183804 0.038 58.715 2024-05-10T11:27:16.034 INFO:tasks.workunit.client.0.smithi027.stdout: LockX 382 0.930 20.183 2024-05-10T11:27:16.034 INFO:tasks.workunit.client.0.smithi027.stdout: UnlockX 382 0.739 7.142 2024-05-10T11:27:16.034 INFO:tasks.workunit.client.0.smithi027.stdout: Flush 8219 7.400 107.366 2024-05-10T11:27:16.034 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-05-10T11:27:16.034 INFO:tasks.workunit.client.0.smithi027.stdout:Throughput 6.10883 MB/sec 1 clients 1 procs max_latency=8540.702 ms 2024-05-10T11:27:16.035 INFO:tasks.workunit.client.0.smithi027.stderr:+ dbench 10 2024-05-10T11:27:16.035 INFO:tasks.workunit.client.0.smithi027.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-05-10T11:27:16.035 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-05-10T11:27:16.035 INFO:tasks.workunit.client.0.smithi027.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-05-10T11:27:16.038 INFO:tasks.workunit.client.0.smithi027.stdout:9 of 10 processes prepared for launch 0 sec 2024-05-10T11:27:16.096 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:27:16.138 INFO:tasks.workunit.client.0.smithi027.stdout:10 of 10 processes prepared for launch 0 sec 2024-05-10T11:27:16.138 INFO:tasks.workunit.client.0.smithi027.stdout:releasing clients 2024-05-10T11:27:17.138 INFO:tasks.workunit.client.0.smithi027.stdout: 10 347 124.98 MB/sec warmup 1 sec latency 163.873 ms 2024-05-10T11:27:17.158 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:16 smithi027 ceph-mon[30054]: pgmap v440: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 150 op/s 2024-05-10T11:27:17.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:16 smithi076 ceph-mon[34152]: pgmap v440: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 150 op/s 2024-05-10T11:27:17.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:16 smithi189 ceph-mon[34842]: pgmap v440: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 150 op/s 2024-05-10T11:27:18.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:17 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:27:18.138 INFO:tasks.workunit.client.0.smithi027.stdout: 10 882 104.05 MB/sec warmup 2 sec latency 229.448 ms 2024-05-10T11:27:18.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:17 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:27:18.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:17 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:27:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.795+0000 7fcbd4800700 1 -- 172.21.15.27:0/967616558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbcc106070 msgr2=0x7fcbcc112580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:18.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.795+0000 7fcbd4800700 1 --2- 172.21.15.27:0/967616558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbcc106070 0x7fcbcc112580 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800b0a0 tx=0x7fcbc800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.796+0000 7fcbd4800700 1 -- 172.21.15.27:0/967616558 shutdown_connections 2024-05-10T11:27:18.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.796+0000 7fcbd4800700 1 --2- 172.21.15.27:0/967616558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbcc106070 0x7fcbcc112580 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.796+0000 7fcbd4800700 1 --2- 172.21.15.27:0/967616558 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcbcc10b6b0 0x7fcbcc10ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.796+0000 7fcbd4800700 1 --2- 172.21.15.27:0/967616558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbcc1056b0 0x7fcbcc105b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.796+0000 7fcbd4800700 1 -- 172.21.15.27:0/967616558 >> 172.21.15.27:0/967616558 conn(0x7fcbcc101180 msgr2=0x7fcbcc1035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:18.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.796+0000 7fcbd4800700 1 -- 172.21.15.27:0/967616558 shutdown_connections 2024-05-10T11:27:18.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.797+0000 7fcbd4800700 1 -- 172.21.15.27:0/967616558 wait complete. 2024-05-10T11:27:18.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.802+0000 7fcbd4800700 1 Processor -- start 2024-05-10T11:27:18.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.802+0000 7fcbd4800700 1 -- start start 2024-05-10T11:27:18.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.804+0000 7fcbd4800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcbcc1056b0 0x7fcbcc1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.804+0000 7fcbd4800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbcc106070 0x7fcbcc1a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.804+0000 7fcbd4800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbcc10b6b0 0x7fcbcc19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.804+0000 7fcbd4800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1158f0 con 0x7fcbcc10b6b0 2024-05-10T11:27:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.804+0000 7fcbd4800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc115770 con 0x7fcbcc1056b0 2024-05-10T11:27:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.804+0000 7fcbd4800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc115a70 con 0x7fcbcc106070 2024-05-10T11:27:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.804+0000 7fcbd1e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcbcc1056b0 0x7fcbcc1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.805+0000 7fcbd2800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbcc10b6b0 0x7fcbcc19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.805+0000 7fcbd2800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbcc10b6b0 0x7fcbcc19bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32822/0 (socket says 172.21.15.27:32822) 2024-05-10T11:27:18.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.805+0000 7fcbd2800700 1 -- 172.21.15.27:0/1197497898 learned_addr learned my addr 172.21.15.27:0/1197497898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:18.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.805+0000 7fcbd1400700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbcc106070 0x7fcbcc1a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.805+0000 7fcbd1e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcbcc1056b0 0x7fcbcc1a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60674/0 (socket says 172.21.15.27:60674) 2024-05-10T11:27:18.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.805+0000 7fcbd2800700 1 -- 172.21.15.27:0/1197497898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbcc106070 msgr2=0x7fcbcc1a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:18.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.805+0000 7fcbd2800700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbcc106070 0x7fcbcc1a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.805+0000 7fcbd2800700 1 -- 172.21.15.27:0/1197497898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcbcc1056b0 msgr2=0x7fcbcc1a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:18.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.805+0000 7fcbd2800700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcbcc1056b0 0x7fcbcc1a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.805+0000 7fcbd2800700 1 -- 172.21.15.27:0/1197497898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbcc19c290 con 0x7fcbcc10b6b0 2024-05-10T11:27:18.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.805+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcbcc1056b0 0x7fcbcc1a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:27:18.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.806+0000 7fcbd2800700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbcc10b6b0 0x7fcbcc19bb30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800b780 tx=0x7fcbc8004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:18.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.806+0000 7fcbc6a00700 1 -- 172.21.15.27:0/1197497898 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8026070 con 0x7fcbcc10b6b0 2024-05-10T11:27:18.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.806+0000 7fcbc6a00700 1 -- 172.21.15.27:0/1197497898 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbc80077f0 con 0x7fcbcc10b6b0 2024-05-10T11:27:18.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.806+0000 7fcbc6a00700 1 -- 172.21.15.27:0/1197497898 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8004260 con 0x7fcbcc10b6b0 2024-05-10T11:27:18.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.806+0000 7fcbd4800700 1 -- 172.21.15.27:0/1197497898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbcc19c520 con 0x7fcbcc10b6b0 2024-05-10T11:27:18.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.806+0000 7fcbd4800700 1 -- 172.21.15.27:0/1197497898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbcc0778e0 con 0x7fcbcc10b6b0 2024-05-10T11:27:18.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.813+0000 7fcbc6a00700 1 -- 172.21.15.27:0/1197497898 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fcbc8007c70 con 0x7fcbcc10b6b0 2024-05-10T11:27:18.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.815+0000 7fcbc6a00700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcbb40b2a70 0x7fcbb40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.815+0000 7fcbc6a00700 1 -- 172.21.15.27:0/1197497898 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7fcbc800f030 con 0x7fcbcc10b6b0 2024-05-10T11:27:18.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.815+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcbb40b2a70 0x7fcbb40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.817+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcbb40b2a70 0x7fcbb40b4f30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009c90 tx=0x7fcbc0009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:18.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.817+0000 7fcbdf275b80 1 Processor -- start 2024-05-10T11:27:18.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.818+0000 7fcbdf275b80 1 -- start start 2024-05-10T11:27:18.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.819+0000 7fcbdf275b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef392c40 0x562cef393020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.819+0000 7fcbdf275b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef393560 0x562cef38d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.819+0000 7fcbdf275b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef38d970 0x562cef38ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.819+0000 7fcbdf275b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562cef2dfaa0 con 0x562cef38d970 2024-05-10T11:27:18.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.819+0000 7fcbdf275b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562cef27d7e0 con 0x562cef392c40 2024-05-10T11:27:18.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.819+0000 7fcbdf275b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562cef2dba60 con 0x562cef393560 2024-05-10T11:27:18.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.820+0000 7fcbd2800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef38d970 0x562cef38ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.820+0000 7fcbd1e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef392c40 0x562cef393020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.820+0000 7fcbd2800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef38d970 0x562cef38ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32836/0 (socket says 172.21.15.27:32836) 2024-05-10T11:27:18.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.820+0000 7fcbd1e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef392c40 0x562cef393020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60676/0 (socket says 172.21.15.27:60676) 2024-05-10T11:27:18.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.820+0000 7fcbd2800700 1 -- 172.21.15.27:0/2660507536 learned_addr learned my addr 172.21.15.27:0/2660507536 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:18.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.820+0000 7fcbd1400700 1 --2- 172.21.15.27:0/2660507536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef393560 0x562cef38d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.820+0000 7fcbd1e00700 1 -- 172.21.15.27:0/2660507536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef393560 msgr2=0x562cef38d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:18.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.820+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/2660507536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef393560 0x562cef38d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.820+0000 7fcbd1e00700 1 -- 172.21.15.27:0/2660507536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef38d970 msgr2=0x562cef38ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:18.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.820+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/2660507536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef38d970 0x562cef38ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.820+0000 7fcbd1e00700 1 -- 172.21.15.27:0/2660507536 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cef22c950 con 0x562cef392c40 2024-05-10T11:27:18.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.820+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/2660507536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef392c40 0x562cef393020 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0010a40 tx=0x7fcbc0013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e139b9ef24d8c53f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:18.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.821+0000 7fcbbb400700 1 -- 172.21.15.27:0/2660507536 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0011bb0 con 0x562cef392c40 2024-05-10T11:27:18.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.821+0000 7fcbbb400700 1 -- 172.21.15.27:0/2660507536 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbc0011d50 con 0x562cef392c40 2024-05-10T11:27:18.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.821+0000 7fcbbb400700 1 -- 172.21.15.27:0/2660507536 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc001c6e0 con 0x562cef392c40 2024-05-10T11:27:18.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.822+0000 7fcbdf275b80 1 -- 172.21.15.27:0/2660507536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef392c40 msgr2=0x562cef393020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:18.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.822+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/2660507536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef392c40 0x562cef393020 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0010a40 tx=0x7fcbc0013000 comp rx=0 tx=0).stop 2024-05-10T11:27:18.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.822+0000 7fcbdf275b80 1 -- 172.21.15.27:0/2660507536 shutdown_connections 2024-05-10T11:27:18.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.822+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/2660507536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef38d970 0x562cef38ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.822+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/2660507536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef392c40 0x562cef393020 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.822+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/2660507536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef393560 0x562cef38d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.822+0000 7fcbdf275b80 1 -- 172.21.15.27:0/2660507536 >> 172.21.15.27:0/2660507536 conn(0x562cef305750 msgr2=0x562cef387b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:18.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.823+0000 7fcbdf275b80 1 -- 172.21.15.27:0/2660507536 shutdown_connections 2024-05-10T11:27:18.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.823+0000 7fcbdf275b80 1 -- 172.21.15.27:0/2660507536 wait complete. 2024-05-10T11:27:18.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.823+0000 7fcbdf275b80 1 -- start start 2024-05-10T11:27:18.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.824+0000 7fcbdf275b80 1 Processor -- start 2024-05-10T11:27:18.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.824+0000 7fcbdf275b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 0x562cef399260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.824+0000 7fcbdf275b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef392c40 0x562cef396600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.825+0000 7fcbdf275b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef393560 0x562cef396b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.825+0000 7fcbd1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 0x562cef399260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.825+0000 7fcbd1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 0x562cef399260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:56956/0 (socket says 172.21.15.27:56956) 2024-05-10T11:27:18.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.825+0000 7fcbdf275b80 1 -- 172.21.15.27:0/166362969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562cef399fa0 con 0x562cef393560 2024-05-10T11:27:18.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.825+0000 7fcbd1e00700 1 -- 172.21.15.27:0/166362969 learned_addr learned my addr 172.21.15.27:0/166362969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:18.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.825+0000 7fcbdf275b80 1 -- 172.21.15.27:0/166362969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562cef245ba0 con 0x562cef392c40 2024-05-10T11:27:18.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.825+0000 7fcbdf275b80 1 -- 172.21.15.27:0/166362969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562cef39a120 con 0x562cef38d970 2024-05-10T11:27:18.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.825+0000 7fcbdf275b80 10 client.0 _ll_register_callbacks cb 0x562cef286cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:18.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.825+0000 7fcbd2800700 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef393560 0x562cef396b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.825+0000 7fcbd1400700 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef392c40 0x562cef396600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.826+0000 7fcbd1e00700 1 -- 172.21.15.27:0/166362969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef392c40 msgr2=0x562cef396600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:18.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.826+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef392c40 0x562cef396600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.826+0000 7fcbd1e00700 1 -- 172.21.15.27:0/166362969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef393560 msgr2=0x562cef396b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:18.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.826+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef393560 0x562cef396b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.826+0000 7fcbd1e00700 1 -- 172.21.15.27:0/166362969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cef2a54a0 con 0x562cef38d970 2024-05-10T11:27:18.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.826+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 0x562cef399260 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcbc001eb30 tx=0x7fcbc001eb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3feb04f33d78cdc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:18.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.826+0000 7fcbafe00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:27:18.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.827+0000 7fcbafe00700 1 -- 172.21.15.27:0/166362969 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0035060 con 0x562cef38d970 2024-05-10T11:27:18.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.827+0000 7fcbafe00700 1 -- 172.21.15.27:0/166362969 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7fcbc0009dd0 con 0x562cef38d970 2024-05-10T11:27:18.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.827+0000 7fcbdf275b80 1 -- 172.21.15.27:0/166362969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbc0011d50 con 0x562cef38d970 2024-05-10T11:27:18.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.827+0000 7fcbafe00700 1 -- 172.21.15.27:0/166362969 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fcbc0011bb0 con 0x562cef38d970 2024-05-10T11:27:18.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.827+0000 7fcbdf275b80 10 client.27241 fetch_fsmap learned FSMap version 37 2024-05-10T11:27:18.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.827+0000 7fcbdf275b80 1 -- 172.21.15.27:0/166362969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562cef3a4820 con 0x562cef38d970 2024-05-10T11:27:18.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.827+0000 7fcbafe00700 1 -- 172.21.15.27:0/166362969 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbc0011d50 con 0x562cef38d970 2024-05-10T11:27:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.828+0000 7fcbafe00700 1 -- 172.21.15.27:0/166362969 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fcbc0005260 con 0x562cef38d970 2024-05-10T11:27:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.828+0000 7fcbdf275b80 10 client.27241 fetch_fsmap finished waiting for FSMap version 37 2024-05-10T11:27:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.828+0000 7fcbdf275b80 10 client.27241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:27:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.828+0000 7fcbdf275b80 20 client.27241 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.828+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562cef3a6e00 0x562cef3a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.828+0000 7fcbdf275b80 4 client.27241 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:27:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.828+0000 7fcbdf275b80 1 -- 172.21.15.27:0/166362969 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562cef35a1c0 con 0x562cef3a6e00 2024-05-10T11:27:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.829+0000 7fcbd1400700 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562cef3a6e00 0x562cef3a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.830+0000 7fcbd1400700 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562cef3a6e00 0x562cef3a91f0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f865b557d93db43 server_cookie=5facd8179a8c27ad in_seq=0 out_seq=0 2024-05-10T11:27:18.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.830+0000 7fcbafe00700 10 client.27241 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:18.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.832+0000 7fcbafe00700 1 -- 172.21.15.27:0/166362969 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562cef35a1c0 con 0x562cef3a6e00 2024-05-10T11:27:18.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.832+0000 7fcbafe00700 10 client.27241 handle_command_reply: tid=0 2024-05-10T11:27:18.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.833+0000 7fcbdf275b80 1 client.27241 shutdown 2024-05-10T11:27:18.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.833+0000 7fcbdf275b80 1 -- 172.21.15.27:0/166362969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 msgr2=0x562cef399260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:18.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.833+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 0x562cef399260 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcbc001eb30 tx=0x7fcbc001eb60 comp rx=0 tx=0).stop 2024-05-10T11:27:18.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.833+0000 7fcbdf275b80 1 -- 172.21.15.27:0/166362969 shutdown_connections 2024-05-10T11:27:18.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.833+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562cef3a6e00 0x562cef3a91f0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.833+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 0x562cef399260 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.833+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef393560 0x562cef396b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.833+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/166362969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef392c40 0x562cef396600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.833+0000 7fcbafe00700 0 client.27241 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:18.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.833+0000 7fcbdf275b80 1 -- 172.21.15.27:0/166362969 >> 172.21.15.27:0/166362969 conn(0x562cef305750 msgr2=0x562cef387210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:18.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.833+0000 7fcbdf275b80 1 -- 172.21.15.27:0/166362969 shutdown_connections 2024-05-10T11:27:18.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.834+0000 7fcbdf275b80 1 -- 172.21.15.27:0/166362969 wait complete. 2024-05-10T11:27:18.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.834+0000 7fcbdf275b80 20 client.27241 trim_cache size 0 max 16384 2024-05-10T11:27:18.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.860+0000 7fcbdf275b80 1 Processor -- start 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.860+0000 7fcbdf275b80 1 -- start start 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.860+0000 7fcbdf275b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 0x562cef38d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.860+0000 7fcbdf275b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef392c40 0x562cef3a3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.860+0000 7fcbdf275b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef393560 0x562cef387990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.860+0000 7fcbdf275b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1ad3f0 con 0x562cef392c40 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.860+0000 7fcbdf275b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1aef90 con 0x562cef393560 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.860+0000 7fcbdf275b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1adac0 con 0x562cef38d970 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.861+0000 7fcbd1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 0x562cef38d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.861+0000 7fcbd1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 0x562cef38d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:56966/0 (socket says 172.21.15.27:56966) 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.861+0000 7fcbd1e00700 1 -- 172.21.15.27:0/2082410896 learned_addr learned my addr 172.21.15.27:0/2082410896 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.861+0000 7fcbd1400700 1 --2- 172.21.15.27:0/2082410896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef392c40 0x562cef3a3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.861+0000 7fcbd2800700 1 --2- 172.21.15.27:0/2082410896 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef393560 0x562cef387990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.861+0000 7fcbd1e00700 1 -- 172.21.15.27:0/2082410896 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef393560 msgr2=0x562cef387990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:18.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.861+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/2082410896 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef393560 0x562cef387990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.861+0000 7fcbd1e00700 1 -- 172.21.15.27:0/2082410896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef392c40 msgr2=0x562cef3a3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:18.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.861+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/2082410896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef392c40 0x562cef3a3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.861+0000 7fcbd1e00700 1 -- 172.21.15.27:0/2082410896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cef382b30 con 0x562cef38d970 2024-05-10T11:27:18.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.862+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/2082410896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 0x562cef38d350 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0010a40 tx=0x7fcbc0009a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab7649da97fee5ba server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:18.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.862+0000 7fcbbaa00700 1 -- 172.21.15.27:0/2082410896 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0035030 con 0x562cef38d970 2024-05-10T11:27:18.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.862+0000 7fcbbaa00700 1 -- 172.21.15.27:0/2082410896 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbc001ee50 con 0x562cef38d970 2024-05-10T11:27:18.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.862+0000 7fcbbaa00700 1 -- 172.21.15.27:0/2082410896 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00306d0 con 0x562cef38d970 2024-05-10T11:27:18.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.863+0000 7fcbdf275b80 1 -- 172.21.15.27:0/2082410896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 msgr2=0x562cef38d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:18.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.863+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/2082410896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 0x562cef38d350 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0010a40 tx=0x7fcbc0009a00 comp rx=0 tx=0).stop 2024-05-10T11:27:18.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.863+0000 7fcbdf275b80 1 -- 172.21.15.27:0/2082410896 shutdown_connections 2024-05-10T11:27:18.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.863+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/2082410896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef38d970 0x562cef38d350 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.863+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/2082410896 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef393560 0x562cef387990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.863+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/2082410896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef392c40 0x562cef3a3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.863+0000 7fcbdf275b80 1 -- 172.21.15.27:0/2082410896 >> 172.21.15.27:0/2082410896 conn(0x562cef305750 msgr2=0x562cef392020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:18.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.864+0000 7fcbdf275b80 1 -- 172.21.15.27:0/2082410896 shutdown_connections 2024-05-10T11:27:18.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.864+0000 7fcbdf275b80 1 -- 172.21.15.27:0/2082410896 wait complete. 2024-05-10T11:27:18.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.865+0000 7fcbdf275b80 1 -- start start 2024-05-10T11:27:18.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.865+0000 7fcbdf275b80 1 Processor -- start 2024-05-10T11:27:18.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.866+0000 7fcbdf275b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef38d970 0x562cef39f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.866+0000 7fcbdf275b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef392c40 0x562cef39c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.866+0000 7fcbdf275b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef393560 0x562cef39c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.866+0000 7fcbdf275b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1ae860 con 0x562cef392c40 2024-05-10T11:27:18.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.866+0000 7fcbdf275b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1ae190 con 0x562cef38d970 2024-05-10T11:27:18.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.866+0000 7fcbdf275b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1ab3b0 con 0x562cef393560 2024-05-10T11:27:18.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.866+0000 7fcbdf275b80 10 client.0 _ll_register_callbacks cb 0x562cef286cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.866+0000 7fcbd2800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef393560 0x562cef39c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.866+0000 7fcbd2800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef393560 0x562cef39c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:56968/0 (socket says 172.21.15.27:56968) 2024-05-10T11:27:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.866+0000 7fcbd2800700 1 -- 172.21.15.27:0/1032716088 learned_addr learned my addr 172.21.15.27:0/1032716088 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.867+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef38d970 0x562cef39f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.867+0000 7fcbd2800700 1 -- 172.21.15.27:0/1032716088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef38d970 msgr2=0x562cef39f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.867+0000 7fcbd2800700 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef38d970 0x562cef39f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.867+0000 7fcbd2800700 1 -- 172.21.15.27:0/1032716088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef392c40 msgr2=0x562cef39c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.867+0000 7fcbd1400700 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef392c40 0x562cef39c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.867+0000 7fcbd2800700 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef392c40 0x562cef39c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.867+0000 7fcbd2800700 1 -- 172.21.15.27:0/1032716088 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cef422420 con 0x562cef393560 2024-05-10T11:27:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.867+0000 7fcbd1400700 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef392c40 0x562cef39c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:27:18.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.868+0000 7fcbd2800700 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef393560 0x562cef39c990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80e9080 tx=0x7fcbc80bb300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed2fb177f79d7a1e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:18.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.868+0000 7fcbafe00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:27:18.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.868+0000 7fcbafe00700 1 -- 172.21.15.27:0/1032716088 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc80b2bf0 con 0x562cef393560 2024-05-10T11:27:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.868+0000 7fcbafe00700 1 -- 172.21.15.27:0/1032716088 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7fcbc80b2e50 con 0x562cef393560 2024-05-10T11:27:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.868+0000 7fcbdf275b80 1 -- 172.21.15.27:0/1032716088 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbc001ee50 con 0x562cef393560 2024-05-10T11:27:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.869+0000 7fcbafe00700 1 -- 172.21.15.27:0/1032716088 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fcbc80b6fa0 con 0x562cef393560 2024-05-10T11:27:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.869+0000 7fcbdf275b80 10 client.27247 fetch_fsmap learned FSMap version 37 2024-05-10T11:27:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.869+0000 7fcbdf275b80 1 -- 172.21.15.27:0/1032716088 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562cef423600 con 0x562cef393560 2024-05-10T11:27:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.869+0000 7fcbafe00700 1 -- 172.21.15.27:0/1032716088 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbc001ee50 con 0x562cef393560 2024-05-10T11:27:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.870+0000 7fcbafe00700 1 -- 172.21.15.27:0/1032716088 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fcbc8011070 con 0x562cef393560 2024-05-10T11:27:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.870+0000 7fcbdf275b80 10 client.27247 fetch_fsmap finished waiting for FSMap version 37 2024-05-10T11:27:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.870+0000 7fcbdf275b80 10 client.27247 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:27:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.870+0000 7fcbdf275b80 20 client.27247 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.870+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562cef423cd0 0x562cef4260f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.870+0000 7fcbdf275b80 4 client.27247 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:27:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.870+0000 7fcbdf275b80 1 -- 172.21.15.27:0/1032716088 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562cef2ed480 con 0x562cef423cd0 2024-05-10T11:27:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.870+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562cef423cd0 0x562cef4260f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.871+0000 7fcbd1e00700 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562cef423cd0 0x562cef4260f0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9cf80cfe4e877e9 server_cookie=7d959269d72a59b4 in_seq=0 out_seq=0 2024-05-10T11:27:18.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.871+0000 7fcbafe00700 10 client.27247 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:18.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.877+0000 7fcbafe00700 1 -- 172.21.15.27:0/1032716088 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562cef2ed480 con 0x562cef423cd0 2024-05-10T11:27:18.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.877+0000 7fcbafe00700 10 client.27247 handle_command_reply: tid=0 2024-05-10T11:27:18.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.877+0000 7fcbdf275b80 1 client.27247 shutdown 2024-05-10T11:27:18.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.877+0000 7fcbdf275b80 1 -- 172.21.15.27:0/1032716088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef393560 msgr2=0x562cef39c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:18.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.877+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef393560 0x562cef39c990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80e9080 tx=0x7fcbc80bb300 comp rx=0 tx=0).stop 2024-05-10T11:27:18.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.878+0000 7fcbdf275b80 1 -- 172.21.15.27:0/1032716088 shutdown_connections 2024-05-10T11:27:18.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.878+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x562cef423cd0 0x562cef4260f0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.878+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562cef393560 0x562cef39c990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.878+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562cef38d970 0x562cef39f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.878+0000 7fcbdf275b80 1 --2- 172.21.15.27:0/1032716088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562cef392c40 0x562cef39c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.878+0000 7fcbdf275b80 1 -- 172.21.15.27:0/1032716088 >> 172.21.15.27:0/1032716088 conn(0x562cef305750 msgr2=0x562cef395e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:18.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.878+0000 7fcbafe00700 0 client.27247 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:18.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.878+0000 7fcbdf275b80 1 -- 172.21.15.27:0/1032716088 shutdown_connections 2024-05-10T11:27:18.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.878+0000 7fcbdf275b80 1 -- 172.21.15.27:0/1032716088 wait complete. 2024-05-10T11:27:18.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.879+0000 7fcbdf275b80 20 client.27247 trim_cache size 0 max 16384 2024-05-10T11:27:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:27:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:27:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "dbd5f1a7-8424-4564-b72c-5b50829150dc", 2024-05-10T11:27:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:27:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:27:18.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.880+0000 7fcbd4800700 1 -- 172.21.15.27:0/1197497898 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcbb40b2a70 msgr2=0x7fcbb40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:18.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.880+0000 7fcbd4800700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcbb40b2a70 0x7fcbb40b4f30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009c90 tx=0x7fcbc0009260 comp rx=0 tx=0).stop 2024-05-10T11:27:18.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.880+0000 7fcbd4800700 1 -- 172.21.15.27:0/1197497898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbcc10b6b0 msgr2=0x7fcbcc19bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:18.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.880+0000 7fcbd4800700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbcc10b6b0 0x7fcbcc19bb30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800b780 tx=0x7fcbc8004d30 comp rx=0 tx=0).stop 2024-05-10T11:27:18.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.880+0000 7fcbd4800700 1 -- 172.21.15.27:0/1197497898 shutdown_connections 2024-05-10T11:27:18.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.880+0000 7fcbd4800700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcbb40b2a70 0x7fcbb40b4f30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.880+0000 7fcbd4800700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbcc10b6b0 0x7fcbcc19bb30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.880+0000 7fcbd4800700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcbcc1056b0 0x7fcbcc1a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.880+0000 7fcbd4800700 1 --2- 172.21.15.27:0/1197497898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbcc106070 0x7fcbcc1a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:18.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.880+0000 7fcbd4800700 1 -- 172.21.15.27:0/1197497898 >> 172.21.15.27:0/1197497898 conn(0x7fcbcc101180 msgr2=0x7fcbcc101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:18.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.880+0000 7fcbd4800700 1 -- 172.21.15.27:0/1197497898 shutdown_connections 2024-05-10T11:27:18.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:18.880+0000 7fcbd4800700 1 -- 172.21.15.27:0/1197497898 wait complete. 2024-05-10T11:27:19.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:18 smithi027 ceph-mon[30054]: pgmap v441: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 150 op/s 2024-05-10T11:27:19.138 INFO:tasks.workunit.client.0.smithi027.stdout: 10 1352 78.79 MB/sec warmup 3 sec latency 258.868 ms 2024-05-10T11:27:19.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:18 smithi076 ceph-mon[34152]: pgmap v441: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 150 op/s 2024-05-10T11:27:19.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:18 smithi189 ceph-mon[34842]: pgmap v441: 129 pgs: 129 active+clean; 637 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 150 op/s 2024-05-10T11:27:20.138 INFO:tasks.workunit.client.0.smithi027.stdout: 10 1962 67.42 MB/sec warmup 4 sec latency 270.822 ms 2024-05-10T11:27:20.143 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:27:20.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:27:20.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:27:20.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:27:20.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:27:20.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:27:20.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:27:20.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:27:20.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:27:20.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:27:20.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:19 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:27:20.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:27:20.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:27:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:27:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:27:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:27:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:27:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:27:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:27:20.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:19 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:27:20.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:27:20.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:27:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:27:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:27:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:27:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:27:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:27:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:27:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:19 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:20.701 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:27:21.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:20 smithi027 ceph-mon[30054]: pgmap v442: 129 pgs: 129 active+clean; 609 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 281 op/s 2024-05-10T11:27:21.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:20 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:27:21.138 INFO:tasks.workunit.client.0.smithi027.stdout: 10 2835 61.37 MB/sec warmup 5 sec latency 213.657 ms 2024-05-10T11:27:21.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:20 smithi189 ceph-mon[34842]: pgmap v442: 129 pgs: 129 active+clean; 609 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 281 op/s 2024-05-10T11:27:21.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:20 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:27:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:20 smithi076 ceph-mon[34152]: pgmap v442: 129 pgs: 129 active+clean; 609 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 281 op/s 2024-05-10T11:27:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:20 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:27:22.138 INFO:tasks.workunit.client.0.smithi027.stdout: 10 3468 59.34 MB/sec warmup 6 sec latency 186.266 ms 2024-05-10T11:27:23.139 INFO:tasks.workunit.client.0.smithi027.stdout: 10 4136 58.23 MB/sec warmup 7 sec latency 926.250 ms 2024-05-10T11:27:23.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.204+0000 7ff245e00700 1 -- 172.21.15.27:0/4009162795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff240109ad0 msgr2=0x7ff240109f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:23.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.204+0000 7ff245e00700 1 --2- 172.21.15.27:0/4009162795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff240109ad0 0x7ff240109f50 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0098f0 tx=0x7ff22c009c00 comp rx=0 tx=0).stop 2024-05-10T11:27:23.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:22 smithi027 ceph-mon[30054]: pgmap v443: 129 pgs: 129 active+clean; 609 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 191 op/s 2024-05-10T11:27:23.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:22 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:27:23.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:22 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:23.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.204+0000 7ff245e00700 1 -- 172.21.15.27:0/4009162795 shutdown_connections 2024-05-10T11:27:23.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.204+0000 7ff245e00700 1 --2- 172.21.15.27:0/4009162795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff24010a490 0x7ff2401169a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.204+0000 7ff245e00700 1 --2- 172.21.15.27:0/4009162795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff24010fb60 0x7ff24010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.204+0000 7ff245e00700 1 --2- 172.21.15.27:0/4009162795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff240109ad0 0x7ff240109f50 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.204+0000 7ff245e00700 1 -- 172.21.15.27:0/4009162795 >> 172.21.15.27:0/4009162795 conn(0x7ff240076c70 msgr2=0x7ff2401082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:23.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.204+0000 7ff245e00700 1 -- 172.21.15.27:0/4009162795 shutdown_connections 2024-05-10T11:27:23.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.204+0000 7ff245e00700 1 -- 172.21.15.27:0/4009162795 wait complete. 2024-05-10T11:27:23.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.206+0000 7ff245e00700 1 Processor -- start 2024-05-10T11:27:23.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.206+0000 7ff245e00700 1 -- start start 2024-05-10T11:27:23.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.206+0000 7ff245e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff240109ad0 0x7ff2401a0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.206+0000 7ff245e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff24010a490 0x7ff2401a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.206+0000 7ff245e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff24010fb60 0x7ff2401a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.206+0000 7ff245e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff240119d20 con 0x7ff240109ad0 2024-05-10T11:27:23.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.206+0000 7ff245e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff240119ba0 con 0x7ff24010a490 2024-05-10T11:27:23.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.206+0000 7ff245e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff240119ea0 con 0x7ff24010fb60 2024-05-10T11:27:23.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.206+0000 7ff23e000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff24010a490 0x7ff2401a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.206+0000 7ff23ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff240109ad0 0x7ff2401a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.206+0000 7ff23e000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff24010a490 0x7ff2401a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60716/0 (socket says 172.21.15.27:60716) 2024-05-10T11:27:23.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.206+0000 7ff23e000700 1 -- 172.21.15.27:0/341616758 learned_addr learned my addr 172.21.15.27:0/341616758 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:23.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.207+0000 7ff23f400700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff24010fb60 0x7ff2401a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.207+0000 7ff23e000700 1 -- 172.21.15.27:0/341616758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff24010fb60 msgr2=0x7ff2401a4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:23.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.207+0000 7ff23e000700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff24010fb60 0x7ff2401a4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.207+0000 7ff23e000700 1 -- 172.21.15.27:0/341616758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff240109ad0 msgr2=0x7ff2401a0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:23.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.207+0000 7ff23e000700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff240109ad0 0x7ff2401a0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.207+0000 7ff23e000700 1 -- 172.21.15.27:0/341616758 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2401a5210 con 0x7ff24010a490 2024-05-10T11:27:23.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.207+0000 7ff23f400700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff24010fb60 0x7ff2401a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:27:23.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.207+0000 7ff23e000700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff24010a490 0x7ff2401a0790 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009230 tx=0x7ff22c0094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:23.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.208+0000 7ff244a00700 1 -- 172.21.15.27:0/341616758 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c005220 con 0x7ff24010a490 2024-05-10T11:27:23.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.208+0000 7ff244a00700 1 -- 172.21.15.27:0/341616758 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff22c00b560 con 0x7ff24010a490 2024-05-10T11:27:23.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.208+0000 7ff245e00700 1 -- 172.21.15.27:0/341616758 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2401a54a0 con 0x7ff24010a490 2024-05-10T11:27:23.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.208+0000 7ff244a00700 1 -- 172.21.15.27:0/341616758 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c003b50 con 0x7ff24010a490 2024-05-10T11:27:23.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.208+0000 7ff245e00700 1 -- 172.21.15.27:0/341616758 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2401acd80 con 0x7ff24010a490 2024-05-10T11:27:23.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.212+0000 7ff244a00700 1 -- 172.21.15.27:0/341616758 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff22c003d10 con 0x7ff24010a490 2024-05-10T11:27:23.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.213+0000 7ff2507edb80 1 Processor -- start 2024-05-10T11:27:23.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.214+0000 7ff2507edb80 1 -- start start 2024-05-10T11:27:23.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.214+0000 7ff244a00700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff2240b2a70 0x7ff2240b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.214+0000 7ff2507edb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f878060 0x564a6f878440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.214+0000 7ff2507edb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f87c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.214+0000 7ff244a00700 1 -- 172.21.15.27:0/341616758 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7ff22c0e3a50 con 0x7ff24010a490 2024-05-10T11:27:23.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.214+0000 7ff2507edb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f87c9d0 0x564a6f87ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.214+0000 7ff2507edb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564a6f7caa20 con 0x564a6f878060 2024-05-10T11:27:23.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.214+0000 7ff2507edb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564a6f768760 con 0x564a6f878980 2024-05-10T11:27:23.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.214+0000 7ff2507edb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564a6f7c69e0 con 0x564a6f87c9d0 2024-05-10T11:27:23.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.214+0000 7ff23f400700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff2240b2a70 0x7ff2240b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23e000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f87c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23e000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f87c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60732/0 (socket says 172.21.15.27:60732) 2024-05-10T11:27:23.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23e000700 1 -- 172.21.15.27:0/969011294 learned_addr learned my addr 172.21.15.27:0/969011294 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:23.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23ea00700 1 --2- 172.21.15.27:0/969011294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f878060 0x564a6f878440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23f400700 1 --2- 172.21.15.27:0/969011294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f87c9d0 0x564a6f87ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23e000700 1 -- 172.21.15.27:0/969011294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f87c9d0 msgr2=0x564a6f87ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:23.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23e000700 1 --2- 172.21.15.27:0/969011294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f87c9d0 0x564a6f87ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23e000700 1 -- 172.21.15.27:0/969011294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f878060 msgr2=0x564a6f878440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:23.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23e000700 1 --2- 172.21.15.27:0/969011294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f878060 0x564a6f878440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23e000700 1 -- 172.21.15.27:0/969011294 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a6f7178d0 con 0x564a6f878980 2024-05-10T11:27:23.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23f400700 1 --2- 172.21.15.27:0/969011294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f87c9d0 0x564a6f87ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:27:23.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23ea00700 1 --2- 172.21.15.27:0/969011294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f878060 0x564a6f878440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:27:23.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.215+0000 7ff23e000700 1 --2- 172.21.15.27:0/969011294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f87c490 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0a69b0 tx=0x7ff22c0aa1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2af65166770807b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:23.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.216+0000 7ff234c00700 1 -- 172.21.15.27:0/969011294 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0aa930 con 0x564a6f878980 2024-05-10T11:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.216+0000 7ff23f400700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff2240b2a70 0x7ff2240b4f30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff2380095b0 tx=0x7ff2380099e0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.216+0000 7ff234c00700 1 -- 172.21.15.27:0/969011294 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff22c090e50 con 0x564a6f878980 2024-05-10T11:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.216+0000 7ff234c00700 1 -- 172.21.15.27:0/969011294 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0b10a0 con 0x564a6f878980 2024-05-10T11:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.217+0000 7ff2507edb80 1 -- 172.21.15.27:0/969011294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 msgr2=0x564a6f87c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.217+0000 7ff2507edb80 1 --2- 172.21.15.27:0/969011294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f87c490 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0a69b0 tx=0x7ff22c0aa1a0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.217+0000 7ff2507edb80 1 -- 172.21.15.27:0/969011294 shutdown_connections 2024-05-10T11:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.217+0000 7ff2507edb80 1 --2- 172.21.15.27:0/969011294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f87c9d0 0x564a6f87ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.217+0000 7ff2507edb80 1 --2- 172.21.15.27:0/969011294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f878060 0x564a6f878440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.217+0000 7ff2507edb80 1 --2- 172.21.15.27:0/969011294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f87c490 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.217+0000 7ff2507edb80 1 -- 172.21.15.27:0/969011294 >> 172.21.15.27:0/969011294 conn(0x564a6f7a6b60 msgr2=0x564a6f8727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:23.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.217+0000 7ff2507edb80 1 -- 172.21.15.27:0/969011294 shutdown_connections 2024-05-10T11:27:23.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.218+0000 7ff2507edb80 1 -- 172.21.15.27:0/969011294 wait complete. 2024-05-10T11:27:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.218+0000 7ff2507edb80 1 -- start start 2024-05-10T11:27:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.218+0000 7ff2507edb80 1 Processor -- start 2024-05-10T11:27:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.219+0000 7ff2507edb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878060 0x564a6f883eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.219+0000 7ff2507edb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f881250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.219+0000 7ff2507edb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f87c9d0 0x564a6f881790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.219+0000 7ff2507edb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564a6f8849a0 con 0x564a6f87c9d0 2024-05-10T11:27:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.219+0000 7ff2507edb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564a6f730b20 con 0x564a6f878980 2024-05-10T11:27:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.219+0000 7ff2507edb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564a6f884b20 con 0x564a6f878060 2024-05-10T11:27:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.219+0000 7ff2507edb80 10 client.0 _ll_register_callbacks cb 0x564a6f771c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.219+0000 7ff23ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878060 0x564a6f883eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.219+0000 7ff23ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878060 0x564a6f883eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:57016/0 (socket says 172.21.15.27:57016) 2024-05-10T11:27:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.219+0000 7ff23ea00700 1 -- 172.21.15.27:0/2556851756 learned_addr learned my addr 172.21.15.27:0/2556851756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.220+0000 7ff23e000700 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f881250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.220+0000 7ff23f400700 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f87c9d0 0x564a6f881790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.220+0000 7ff23ea00700 1 -- 172.21.15.27:0/2556851756 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 msgr2=0x564a6f881250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.220+0000 7ff23ea00700 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f881250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.220+0000 7ff23ea00700 1 -- 172.21.15.27:0/2556851756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f87c9d0 msgr2=0x564a6f881790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.220+0000 7ff23ea00700 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f87c9d0 0x564a6f881790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.220+0000 7ff23ea00700 1 -- 172.21.15.27:0/2556851756 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a6f790420 con 0x564a6f878060 2024-05-10T11:27:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.220+0000 7ff23ea00700 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878060 0x564a6f883eb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff230007eb0 tx=0x7ff230012730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7e7a41dd1f3e691 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.220+0000 7ff229600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:27:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.221+0000 7ff229600700 1 -- 172.21.15.27:0/2556851756 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff230010870 con 0x564a6f878060 2024-05-10T11:27:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.221+0000 7ff229600700 1 -- 172.21.15.27:0/2556851756 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7ff230010a10 con 0x564a6f878060 2024-05-10T11:27:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.221+0000 7ff2507edb80 1 -- 172.21.15.27:0/2556851756 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff22c090e50 con 0x564a6f878060 2024-05-10T11:27:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.221+0000 7ff229600700 1 -- 172.21.15.27:0/2556851756 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7ff230010c50 con 0x564a6f878060 2024-05-10T11:27:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.221+0000 7ff2507edb80 10 client.27259 fetch_fsmap learned FSMap version 40 2024-05-10T11:27:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.221+0000 7ff2507edb80 1 -- 172.21.15.27:0/2556851756 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x564a6f88f210 con 0x564a6f878060 2024-05-10T11:27:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.221+0000 7ff229600700 1 -- 172.21.15.27:0/2556851756 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff22c090e50 con 0x564a6f878060 2024-05-10T11:27:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.222+0000 7ff229600700 1 -- 172.21.15.27:0/2556851756 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7ff230019070 con 0x564a6f878060 2024-05-10T11:27:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.222+0000 7ff2507edb80 10 client.27259 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:27:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.222+0000 7ff2507edb80 10 client.27259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:27:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.222+0000 7ff2507edb80 20 client.27259 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.222+0000 7ff2507edb80 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564a6f8918a0 0x564a6f893c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.222+0000 7ff2507edb80 4 client.27259 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:27:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.222+0000 7ff2507edb80 1 -- 172.21.15.27:0/2556851756 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a6f8450a0 con 0x564a6f8918a0 2024-05-10T11:27:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.222+0000 7ff23e000700 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564a6f8918a0 0x564a6f893c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.222+0000 7ff23e000700 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564a6f8918a0 0x564a6f893c90 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c596527109a702d4 server_cookie=ac9de732b2f66a59 in_seq=0 out_seq=0 2024-05-10T11:27:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.222+0000 7ff229600700 10 client.27259 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.227+0000 7ff229600700 1 -- 172.21.15.27:0/2556851756 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564a6f8450a0 con 0x564a6f8918a0 2024-05-10T11:27:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.227+0000 7ff229600700 10 client.27259 handle_command_reply: tid=0 2024-05-10T11:27:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.227+0000 7ff2507edb80 1 client.27259 shutdown 2024-05-10T11:27:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.227+0000 7ff2507edb80 1 -- 172.21.15.27:0/2556851756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878060 msgr2=0x564a6f883eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.227+0000 7ff2507edb80 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878060 0x564a6f883eb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff230007eb0 tx=0x7ff230012730 comp rx=0 tx=0).stop 2024-05-10T11:27:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.228+0000 7ff2507edb80 1 -- 172.21.15.27:0/2556851756 shutdown_connections 2024-05-10T11:27:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.228+0000 7ff2507edb80 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564a6f8918a0 0x564a6f893c90 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.228+0000 7ff2507edb80 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878060 0x564a6f883eb0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.228+0000 7ff229600700 0 client.27259 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.228+0000 7ff2507edb80 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f87c9d0 0x564a6f881790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.228+0000 7ff2507edb80 1 --2- 172.21.15.27:0/2556851756 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f881250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.228+0000 7ff2507edb80 1 -- 172.21.15.27:0/2556851756 >> 172.21.15.27:0/2556851756 conn(0x564a6f7a6b60 msgr2=0x564a6f86d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.228+0000 7ff2507edb80 1 -- 172.21.15.27:0/2556851756 shutdown_connections 2024-05-10T11:27:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.228+0000 7ff2507edb80 1 -- 172.21.15.27:0/2556851756 wait complete. 2024-05-10T11:27:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.228+0000 7ff2507edb80 20 client.27259 trim_cache size 0 max 16384 2024-05-10T11:27:23.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:22 smithi189 ceph-mon[34842]: pgmap v443: 129 pgs: 129 active+clean; 609 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 191 op/s 2024-05-10T11:27:23.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:22 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:27:23.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:22 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.255+0000 7ff2507edb80 1 Processor -- start 2024-05-10T11:27:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.256+0000 7ff2507edb80 1 -- start start 2024-05-10T11:27:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.256+0000 7ff2507edb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f878060 0x564a6f87ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.256+0000 7ff2507edb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f88d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.256+0000 7ff2507edb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f87c9d0 0x564a6f86d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.256+0000 7ff2507edb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401b5cb0 con 0x564a6f878060 2024-05-10T11:27:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.256+0000 7ff2507edb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401b78e0 con 0x564a6f878980 2024-05-10T11:27:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.256+0000 7ff2507edb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401b6380 con 0x564a6f87c9d0 2024-05-10T11:27:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.257+0000 7ff23e000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f88d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.257+0000 7ff23ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f878060 0x564a6f87ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.257+0000 7ff23f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f87c9d0 0x564a6f86d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.257+0000 7ff23e000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f88d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60756/0 (socket says 172.21.15.27:60756) 2024-05-10T11:27:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.257+0000 7ff23e000700 1 -- 172.21.15.27:0/3501473195 learned_addr learned my addr 172.21.15.27:0/3501473195 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.257+0000 7ff23e000700 1 -- 172.21.15.27:0/3501473195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f87c9d0 msgr2=0x564a6f86d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.257+0000 7ff23e000700 1 --2- 172.21.15.27:0/3501473195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f87c9d0 0x564a6f86d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.257+0000 7ff23e000700 1 -- 172.21.15.27:0/3501473195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f878060 msgr2=0x564a6f87ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.257+0000 7ff23e000700 1 --2- 172.21.15.27:0/3501473195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f878060 0x564a6f87ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.257+0000 7ff23e000700 1 -- 172.21.15.27:0/3501473195 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a6f883af0 con 0x564a6f878980 2024-05-10T11:27:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.257+0000 7ff23ea00700 1 --2- 172.21.15.27:0/3501473195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f878060 0x564a6f87ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:27:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.258+0000 7ff23e000700 1 --2- 172.21.15.27:0/3501473195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f88d9f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0a69b0 tx=0x7ff22c0a6870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bc84d8787931de server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.258+0000 7ff22be00700 1 -- 172.21.15.27:0/3501473195 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0c4070 con 0x564a6f878980 2024-05-10T11:27:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.258+0000 7ff22be00700 1 -- 172.21.15.27:0/3501473195 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff22c090e50 con 0x564a6f878980 2024-05-10T11:27:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.258+0000 7ff22be00700 1 -- 172.21.15.27:0/3501473195 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0b10b0 con 0x564a6f878980 2024-05-10T11:27:23.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.259+0000 7ff2507edb80 1 -- 172.21.15.27:0/3501473195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 msgr2=0x564a6f88d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:23.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.259+0000 7ff2507edb80 1 --2- 172.21.15.27:0/3501473195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f88d9f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0a69b0 tx=0x7ff22c0a6870 comp rx=0 tx=0).stop 2024-05-10T11:27:23.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.259+0000 7ff2507edb80 1 -- 172.21.15.27:0/3501473195 shutdown_connections 2024-05-10T11:27:23.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.259+0000 7ff2507edb80 1 --2- 172.21.15.27:0/3501473195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f87c9d0 0x564a6f86d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.259+0000 7ff2507edb80 1 --2- 172.21.15.27:0/3501473195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f878060 0x564a6f87ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.259+0000 7ff2507edb80 1 --2- 172.21.15.27:0/3501473195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878980 0x564a6f88d9f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.259+0000 7ff2507edb80 1 -- 172.21.15.27:0/3501473195 >> 172.21.15.27:0/3501473195 conn(0x564a6f7a6b60 msgr2=0x564a6f86ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:23.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.260+0000 7ff2507edb80 1 -- 172.21.15.27:0/3501473195 shutdown_connections 2024-05-10T11:27:23.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.260+0000 7ff2507edb80 1 -- 172.21.15.27:0/3501473195 wait complete. 2024-05-10T11:27:23.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.260+0000 7ff2507edb80 1 -- start start 2024-05-10T11:27:23.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.261+0000 7ff2507edb80 1 Processor -- start 2024-05-10T11:27:23.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.261+0000 7ff2507edb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878060 0x564a6f88a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.261+0000 7ff2507edb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878980 0x564a6f87df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.261+0000 7ff2507edb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f87c9d0 0x564a6f87e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff2507edb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401b7150 con 0x564a6f87c9d0 2024-05-10T11:27:23.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff2507edb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401b6a80 con 0x564a6f878060 2024-05-10T11:27:23.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff2507edb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401b3c70 con 0x564a6f878980 2024-05-10T11:27:23.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff2507edb80 10 client.0 _ll_register_callbacks cb 0x564a6f771c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:23.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff23e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878980 0x564a6f87df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff23ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878060 0x564a6f88a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff23e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878980 0x564a6f87df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:57044/0 (socket says 172.21.15.27:57044) 2024-05-10T11:27:23.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff23e000700 1 -- 172.21.15.27:0/855625004 learned_addr learned my addr 172.21.15.27:0/855625004 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:23.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff23f400700 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f87c9d0 0x564a6f87e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff23ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878060 0x564a6f88a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:60762/0 (socket says 172.21.15.27:60762) 2024-05-10T11:27:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff23e000700 1 -- 172.21.15.27:0/855625004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878060 msgr2=0x564a6f88a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff23e000700 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878060 0x564a6f88a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff23e000700 1 -- 172.21.15.27:0/855625004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f87c9d0 msgr2=0x564a6f87e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff23e000700 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f87c9d0 0x564a6f87e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff23e000700 1 -- 172.21.15.27:0/855625004 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a6f887f90 con 0x564a6f878980 2024-05-10T11:27:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.262+0000 7ff23f400700 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f87c9d0 0x564a6f87e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:27:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.263+0000 7ff23e000700 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878980 0x564a6f87df50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff22c092010 tx=0x7ff22c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fdc7afe28ad4268 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.263+0000 7ff229600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:27:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.263+0000 7ff229600700 1 -- 172.21.15.27:0/855625004 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0c4070 con 0x564a6f878980 2024-05-10T11:27:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.263+0000 7ff229600700 1 -- 172.21.15.27:0/855625004 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7ff22c0a6bc0 con 0x564a6f878980 2024-05-10T11:27:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.263+0000 7ff2507edb80 1 -- 172.21.15.27:0/855625004 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff22c090e50 con 0x564a6f878980 2024-05-10T11:27:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.264+0000 7ff229600700 1 -- 172.21.15.27:0/855625004 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7ff22c0cb900 con 0x564a6f878980 2024-05-10T11:27:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.264+0000 7ff2507edb80 10 client.27265 fetch_fsmap learned FSMap version 40 2024-05-10T11:27:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.264+0000 7ff2507edb80 1 -- 172.21.15.27:0/855625004 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x564a6f8fc290 con 0x564a6f878980 2024-05-10T11:27:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.264+0000 7ff229600700 1 -- 172.21.15.27:0/855625004 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff22c090e50 con 0x564a6f878980 2024-05-10T11:27:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.264+0000 7ff229600700 1 -- 172.21.15.27:0/855625004 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7ff22c0e0f90 con 0x564a6f878980 2024-05-10T11:27:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.264+0000 7ff2507edb80 10 client.27265 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:27:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.264+0000 7ff2507edb80 10 client.27265 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:27:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.264+0000 7ff2507edb80 20 client.27265 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.264+0000 7ff2507edb80 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564a6f8fc6f0 0x564a6f8feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.264+0000 7ff2507edb80 4 client.27265 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:27:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.264+0000 7ff2507edb80 1 -- 172.21.15.27:0/855625004 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564a6f7d8400 con 0x564a6f8fc6f0 2024-05-10T11:27:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.265+0000 7ff23ea00700 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564a6f8fc6f0 0x564a6f8feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.266+0000 7ff23ea00700 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564a6f8fc6f0 0x564a6f8feb10 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7951fc6305db624e server_cookie=63953b72a69074fd in_seq=0 out_seq=0 2024-05-10T11:27:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.266+0000 7ff229600700 10 client.27265 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.272+0000 7ff229600700 1 -- 172.21.15.27:0/855625004 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564a6f7d8400 con 0x564a6f8fc6f0 2024-05-10T11:27:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.272+0000 7ff229600700 10 client.27265 handle_command_reply: tid=0 2024-05-10T11:27:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.272+0000 7ff2507edb80 1 client.27265 shutdown 2024-05-10T11:27:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.272+0000 7ff2507edb80 1 -- 172.21.15.27:0/855625004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878980 msgr2=0x564a6f87df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.272+0000 7ff2507edb80 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878980 0x564a6f87df50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff22c092010 tx=0x7ff22c009260 comp rx=0 tx=0).stop 2024-05-10T11:27:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.272+0000 7ff2507edb80 1 -- 172.21.15.27:0/855625004 shutdown_connections 2024-05-10T11:27:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.272+0000 7ff2507edb80 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564a6f8fc6f0 0x564a6f8feb10 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.272+0000 7ff2507edb80 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a6f87c9d0 0x564a6f87e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.272+0000 7ff2507edb80 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564a6f878060 0x564a6f88a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.272+0000 7ff2507edb80 1 --2- 172.21.15.27:0/855625004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a6f878980 0x564a6f87df50 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.272+0000 7ff2507edb80 1 -- 172.21.15.27:0/855625004 >> 172.21.15.27:0/855625004 conn(0x564a6f7a6b60 msgr2=0x564a6f86d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.272+0000 7ff229600700 0 client.27265 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:23.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.273+0000 7ff2507edb80 1 -- 172.21.15.27:0/855625004 shutdown_connections 2024-05-10T11:27:23.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.273+0000 7ff2507edb80 1 -- 172.21.15.27:0/855625004 wait complete. 2024-05-10T11:27:23.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.273+0000 7ff2507edb80 20 client.27265 trim_cache size 0 max 16384 2024-05-10T11:27:23.274 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:27:23.274 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:27:23.274 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:27:23.274 INFO:teuthology.orchestra.run.smithi027.stdout: "dbd5f1a7-8424-4564-b72c-5b50829150dc": { 2024-05-10T11:27:23.274 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:27:23.275 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "dbd5f1a7-8424-4564-b72c-5b50829150dc", 2024-05-10T11:27:23.275 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:27:23.275 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:27:23.275 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:27:23.275 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:27:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.274+0000 7ff245e00700 1 -- 172.21.15.27:0/341616758 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff2240b2a70 msgr2=0x7ff2240b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.274+0000 7ff245e00700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff2240b2a70 0x7ff2240b4f30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff2380095b0 tx=0x7ff2380099e0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.274+0000 7ff245e00700 1 -- 172.21.15.27:0/341616758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff24010a490 msgr2=0x7ff2401a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.274+0000 7ff245e00700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff24010a490 0x7ff2401a0790 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009230 tx=0x7ff22c0094a0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.274+0000 7ff245e00700 1 -- 172.21.15.27:0/341616758 shutdown_connections 2024-05-10T11:27:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.274+0000 7ff245e00700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff2240b2a70 0x7ff2240b4f30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.274+0000 7ff245e00700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff24010fb60 0x7ff2401a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.274+0000 7ff245e00700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff240109ad0 0x7ff2401a0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.274+0000 7ff245e00700 1 --2- 172.21.15.27:0/341616758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff24010a490 0x7ff2401a0790 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.274+0000 7ff245e00700 1 -- 172.21.15.27:0/341616758 >> 172.21.15.27:0/341616758 conn(0x7ff240076c70 msgr2=0x7ff2400772d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.275+0000 7ff245e00700 1 -- 172.21.15.27:0/341616758 shutdown_connections 2024-05-10T11:27:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:23.275+0000 7ff245e00700 1 -- 172.21.15.27:0/341616758 wait complete. 2024-05-10T11:27:23.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:22 smithi076 ceph-mon[34152]: pgmap v443: 129 pgs: 129 active+clean; 609 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 191 op/s 2024-05-10T11:27:23.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:22 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:27:23.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:22 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:27:24.139 INFO:tasks.workunit.client.0.smithi027.stdout: 10 4418 52.53 MB/sec warmup 8 sec latency 1926.374 ms 2024-05-10T11:27:24.686 INFO:tasks.cephfs.filesystem:scrub status for tag:dbd5f1a7-8424-4564-b72c-5b50829150dc - {'path': '/', 'tag': 'dbd5f1a7-8424-4564-b72c-5b50829150dc', 'options': 'recursive,force'} 2024-05-10T11:27:25.139 INFO:tasks.workunit.client.0.smithi027.stdout: 10 4828 48.13 MB/sec warmup 9 sec latency 2926.669 ms 2024-05-10T11:27:25.226 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:24 smithi076 ceph-mon[34152]: pgmap v444: 129 pgs: 129 active+clean; 609 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 191 op/s 2024-05-10T11:27:25.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:24 smithi189 ceph-mon[34842]: pgmap v444: 129 pgs: 129 active+clean; 609 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 191 op/s 2024-05-10T11:27:25.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:24 smithi027 ceph-mon[30054]: pgmap v444: 129 pgs: 129 active+clean; 609 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 191 op/s 2024-05-10T11:27:26.139 INFO:tasks.workunit.client.0.smithi027.stdout: 10 5293 45.72 MB/sec warmup 10 sec latency 3926.772 ms 2024-05-10T11:27:27.139 INFO:tasks.workunit.client.0.smithi027.stdout: 10 6093 44.39 MB/sec warmup 11 sec latency 4926.827 ms 2024-05-10T11:27:27.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:26 smithi189 ceph-mon[34842]: pgmap v445: 129 pgs: 129 active+clean; 630 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 337 op/s 2024-05-10T11:27:27.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:26 smithi027 ceph-mon[30054]: pgmap v445: 129 pgs: 129 active+clean; 630 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 337 op/s 2024-05-10T11:27:27.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:26 smithi076 ceph-mon[34152]: pgmap v445: 129 pgs: 129 active+clean; 630 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 337 op/s 2024-05-10T11:27:28.139 INFO:tasks.workunit.client.0.smithi027.stdout: 10 6809 44.20 MB/sec warmup 12 sec latency 5926.959 ms 2024-05-10T11:27:28.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:28 smithi027 ceph-mon[30054]: pgmap v446: 129 pgs: 129 active+clean; 630 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 274 op/s 2024-05-10T11:27:29.139 INFO:tasks.workunit.client.0.smithi027.stdout: 10 7553 44.56 MB/sec warmup 13 sec latency 6927.094 ms 2024-05-10T11:27:29.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:28 smithi189 ceph-mon[34842]: pgmap v446: 129 pgs: 129 active+clean; 630 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 274 op/s 2024-05-10T11:27:29.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:28 smithi076 ceph-mon[34152]: pgmap v446: 129 pgs: 129 active+clean; 630 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 274 op/s 2024-05-10T11:27:30.139 INFO:tasks.workunit.client.0.smithi027.stdout: 10 8062 44.37 MB/sec warmup 14 sec latency 6983.557 ms 2024-05-10T11:27:31.140 INFO:tasks.workunit.client.0.smithi027.stdout: 10 8371 42.30 MB/sec warmup 15 sec latency 181.206 ms 2024-05-10T11:27:31.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:30 smithi189 ceph-mon[34842]: pgmap v447: 129 pgs: 129 active+clean; 654 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.5 MiB/s wr, 452 op/s 2024-05-10T11:27:31.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:30 smithi027 ceph-mon[30054]: pgmap v447: 129 pgs: 129 active+clean; 654 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.5 MiB/s wr, 452 op/s 2024-05-10T11:27:31.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:30 smithi076 ceph-mon[34152]: pgmap v447: 129 pgs: 129 active+clean; 654 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.5 MiB/s wr, 452 op/s 2024-05-10T11:27:32.140 INFO:tasks.workunit.client.0.smithi027.stdout: 10 8727 40.51 MB/sec warmup 16 sec latency 254.200 ms 2024-05-10T11:27:32.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:31 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:27:32.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:31 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:27:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:31 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:27:33.140 INFO:tasks.workunit.client.0.smithi027.stdout: 10 9155 39.19 MB/sec warmup 17 sec latency 256.190 ms 2024-05-10T11:27:33.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:32 smithi189 ceph-mon[34842]: pgmap v448: 129 pgs: 129 active+clean; 654 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 324 op/s 2024-05-10T11:27:33.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:32 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:27:33.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:32 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:27:33.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:32 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:27:33.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:32 smithi027 ceph-mon[30054]: pgmap v448: 129 pgs: 129 active+clean; 654 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 324 op/s 2024-05-10T11:27:33.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:32 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:27:33.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:32 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:27:33.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:32 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:27:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:32 smithi076 ceph-mon[34152]: pgmap v448: 129 pgs: 129 active+clean; 654 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 324 op/s 2024-05-10T11:27:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:32 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:27:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:32 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:27:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:32 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:27:34.140 INFO:tasks.workunit.client.0.smithi027.stdout: 10 9621 38.11 MB/sec warmup 18 sec latency 343.565 ms 2024-05-10T11:27:35.140 INFO:tasks.workunit.client.0.smithi027.stdout: 10 10324 37.78 MB/sec warmup 19 sec latency 190.604 ms 2024-05-10T11:27:35.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:34 smithi027 ceph-mon[30054]: pgmap v449: 129 pgs: 129 active+clean; 654 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 324 op/s 2024-05-10T11:27:35.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:34 smithi076 ceph-mon[34152]: pgmap v449: 129 pgs: 129 active+clean; 654 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 324 op/s 2024-05-10T11:27:35.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:34 smithi189 ceph-mon[34842]: pgmap v449: 129 pgs: 129 active+clean; 654 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 324 op/s 2024-05-10T11:27:36.140 INFO:tasks.workunit.client.0.smithi027.stdout: 10 11015 37.95 MB/sec warmup 20 sec latency 169.273 ms 2024-05-10T11:27:37.140 INFO:tasks.workunit.client.0.smithi027.stdout: 10 11711 38.55 MB/sec warmup 21 sec latency 61.412 ms 2024-05-10T11:27:37.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:36 smithi027 ceph-mon[30054]: pgmap v450: 129 pgs: 129 active+clean; 707 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 10 MiB/s wr, 503 op/s 2024-05-10T11:27:37.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:36 smithi076 ceph-mon[34152]: pgmap v450: 129 pgs: 129 active+clean; 707 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 10 MiB/s wr, 503 op/s 2024-05-10T11:27:37.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:36 smithi189 ceph-mon[34842]: pgmap v450: 129 pgs: 129 active+clean; 707 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 10 MiB/s wr, 503 op/s 2024-05-10T11:27:38.140 INFO:tasks.workunit.client.0.smithi027.stdout: 10 12242 38.36 MB/sec warmup 22 sec latency 79.718 ms 2024-05-10T11:27:39.142 INFO:tasks.workunit.client.0.smithi027.stdout: 10 12446 37.15 MB/sec warmup 23 sec latency 487.042 ms 2024-05-10T11:27:39.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:38 smithi027 ceph-mon[30054]: pgmap v451: 129 pgs: 129 active+clean; 707 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 8.0 MiB/s wr, 357 op/s 2024-05-10T11:27:39.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:39 smithi076 ceph-mon[34152]: pgmap v451: 129 pgs: 129 active+clean; 707 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 8.0 MiB/s wr, 357 op/s 2024-05-10T11:27:39.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:39 smithi189 ceph-mon[34842]: pgmap v451: 129 pgs: 129 active+clean; 707 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 8.0 MiB/s wr, 357 op/s 2024-05-10T11:27:40.140 INFO:tasks.workunit.client.0.smithi027.stdout: 10 12498 35.65 MB/sec warmup 24 sec latency 529.870 ms 2024-05-10T11:27:41.140 INFO:tasks.workunit.client.0.smithi027.stdout: 10 12584 34.35 MB/sec warmup 25 sec latency 1093.920 ms 2024-05-10T11:27:41.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:41 smithi027 ceph-mon[30054]: pgmap v452: 129 pgs: 129 active+clean; 814 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 18 MiB/s wr, 537 op/s 2024-05-10T11:27:41.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:41 smithi076 ceph-mon[34152]: pgmap v452: 129 pgs: 129 active+clean; 814 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 18 MiB/s wr, 537 op/s 2024-05-10T11:27:41.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:41 smithi189 ceph-mon[34842]: pgmap v452: 129 pgs: 129 active+clean; 814 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 18 MiB/s wr, 537 op/s 2024-05-10T11:27:42.140 INFO:tasks.workunit.client.0.smithi027.stdout: 10 12873 33.54 MB/sec warmup 26 sec latency 302.911 ms 2024-05-10T11:27:43.140 INFO:tasks.workunit.client.0.smithi027.stdout: 10 13209 32.88 MB/sec warmup 27 sec latency 258.566 ms 2024-05-10T11:27:43.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:43 smithi027 ceph-mon[30054]: pgmap v453: 129 pgs: 129 active+clean; 814 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 358 op/s 2024-05-10T11:27:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:43 smithi076 ceph-mon[34152]: pgmap v453: 129 pgs: 129 active+clean; 814 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 358 op/s 2024-05-10T11:27:43.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:43 smithi189 ceph-mon[34842]: pgmap v453: 129 pgs: 129 active+clean; 814 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 358 op/s 2024-05-10T11:27:44.141 INFO:tasks.workunit.client.0.smithi027.stdout: 10 13696 32.49 MB/sec warmup 28 sec latency 186.914 ms 2024-05-10T11:27:45.141 INFO:tasks.workunit.client.0.smithi027.stdout: 10 14415 32.58 MB/sec warmup 29 sec latency 268.532 ms 2024-05-10T11:27:45.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:45 smithi027 ceph-mon[30054]: pgmap v454: 129 pgs: 129 active+clean; 814 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 358 op/s 2024-05-10T11:27:45.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:45 smithi076 ceph-mon[34152]: pgmap v454: 129 pgs: 129 active+clean; 814 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 358 op/s 2024-05-10T11:27:45.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:45 smithi189 ceph-mon[34842]: pgmap v454: 129 pgs: 129 active+clean; 814 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 358 op/s 2024-05-10T11:27:46.141 INFO:tasks.workunit.client.0.smithi027.stdout: 10 15018 32.56 MB/sec warmup 30 sec latency 167.887 ms 2024-05-10T11:27:47.141 INFO:tasks.workunit.client.0.smithi027.stdout: 10 15675 32.99 MB/sec warmup 31 sec latency 76.844 ms 2024-05-10T11:27:47.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:47 smithi189 ceph-mon[34842]: pgmap v455: 129 pgs: 129 active+clean; 825 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 28 MiB/s wr, 697 op/s 2024-05-10T11:27:47.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:47 smithi027 ceph-mon[30054]: pgmap v455: 129 pgs: 129 active+clean; 825 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 28 MiB/s wr, 697 op/s 2024-05-10T11:27:47.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:47 smithi076 ceph-mon[34152]: pgmap v455: 129 pgs: 129 active+clean; 825 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 28 MiB/s wr, 697 op/s 2024-05-10T11:27:48.141 INFO:tasks.workunit.client.0.smithi027.stdout: 10 16128 32.97 MB/sec warmup 32 sec latency 275.576 ms 2024-05-10T11:27:49.141 INFO:tasks.workunit.client.0.smithi027.stdout: 10 16512 32.40 MB/sec warmup 33 sec latency 172.396 ms 2024-05-10T11:27:49.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:49 smithi027 ceph-mon[30054]: pgmap v456: 129 pgs: 129 active+clean; 825 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 23 MiB/s wr, 518 op/s 2024-05-10T11:27:49.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:49 smithi076 ceph-mon[34152]: pgmap v456: 129 pgs: 129 active+clean; 825 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 23 MiB/s wr, 518 op/s 2024-05-10T11:27:49.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:49 smithi189 ceph-mon[34842]: pgmap v456: 129 pgs: 129 active+clean; 825 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 23 MiB/s wr, 518 op/s 2024-05-10T11:27:50.141 INFO:tasks.workunit.client.0.smithi027.stdout: 10 16893 31.84 MB/sec warmup 34 sec latency 204.639 ms 2024-05-10T11:27:51.142 INFO:tasks.workunit.client.0.smithi027.stdout: 10 17354 31.51 MB/sec warmup 35 sec latency 191.984 ms 2024-05-10T11:27:51.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:51 smithi027 ceph-mon[30054]: pgmap v457: 129 pgs: 129 active+clean; 843 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 24 MiB/s wr, 664 op/s 2024-05-10T11:27:51.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:51 smithi076 ceph-mon[34152]: pgmap v457: 129 pgs: 129 active+clean; 843 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 24 MiB/s wr, 664 op/s 2024-05-10T11:27:51.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:51 smithi189 ceph-mon[34842]: pgmap v457: 129 pgs: 129 active+clean; 843 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 24 MiB/s wr, 664 op/s 2024-05-10T11:27:52.141 INFO:tasks.workunit.client.0.smithi027.stdout: 10 17948 31.49 MB/sec warmup 36 sec latency 246.556 ms 2024-05-10T11:27:53.142 INFO:tasks.workunit.client.0.smithi027.stdout: 10 18800 32.06 MB/sec warmup 37 sec latency 59.787 ms 2024-05-10T11:27:53.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:53 smithi027 ceph-mon[30054]: pgmap v458: 129 pgs: 129 active+clean; 843 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 14 MiB/s wr, 485 op/s 2024-05-10T11:27:53.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:53 smithi076 ceph-mon[34152]: pgmap v458: 129 pgs: 129 active+clean; 843 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 14 MiB/s wr, 485 op/s 2024-05-10T11:27:53.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:53 smithi189 ceph-mon[34842]: pgmap v458: 129 pgs: 129 active+clean; 843 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 14 MiB/s wr, 485 op/s 2024-05-10T11:27:54.142 INFO:tasks.workunit.client.0.smithi027.stdout: 10 19338 32.04 MB/sec warmup 38 sec latency 192.560 ms 2024-05-10T11:27:54.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:27:55.142 INFO:tasks.workunit.client.0.smithi027.stdout: 10 19884 32.09 MB/sec warmup 39 sec latency 166.580 ms 2024-05-10T11:27:55.197 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:27:55.356 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:55 smithi027 ceph-mon[30054]: pgmap v459: 129 pgs: 129 active+clean; 843 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 14 MiB/s wr, 485 op/s 2024-05-10T11:27:55.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:55 smithi076 ceph-mon[34152]: pgmap v459: 129 pgs: 129 active+clean; 843 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 14 MiB/s wr, 485 op/s 2024-05-10T11:27:55.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:55 smithi189 ceph-mon[34842]: pgmap v459: 129 pgs: 129 active+clean; 843 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 14 MiB/s wr, 485 op/s 2024-05-10T11:27:56.142 INFO:tasks.workunit.client.0.smithi027.stdout: 10 20271 31.90 MB/sec warmup 40 sec latency 204.951 ms 2024-05-10T11:27:57.142 INFO:tasks.workunit.client.0.smithi027.stdout: 10 20628 31.57 MB/sec warmup 41 sec latency 274.693 ms 2024-05-10T11:27:57.327 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:57 smithi027 ceph-mon[30054]: pgmap v460: 129 pgs: 129 active+clean; 861 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 16 MiB/s wr, 655 op/s 2024-05-10T11:27:57.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:57 smithi076 ceph-mon[34152]: pgmap v460: 129 pgs: 129 active+clean; 861 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 16 MiB/s wr, 655 op/s 2024-05-10T11:27:57.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:57 smithi189 ceph-mon[34842]: pgmap v460: 129 pgs: 129 active+clean; 861 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 16 MiB/s wr, 655 op/s 2024-05-10T11:27:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.727+0000 7f0da0a00700 1 -- 172.21.15.27:0/1015370464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c102d60 msgr2=0x7f0d9c10f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.727+0000 7f0da0a00700 1 --2- 172.21.15.27:0/1015370464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c102d60 0x7f0d9c10f250 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0d9400b0a0 tx=0x7f0d9400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.728+0000 7f0da0a00700 1 -- 172.21.15.27:0/1015370464 shutdown_connections 2024-05-10T11:27:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.728+0000 7f0da0a00700 1 --2- 172.21.15.27:0/1015370464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c102d60 0x7f0d9c10f250 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.728+0000 7f0da0a00700 1 --2- 172.21.15.27:0/1015370464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d9c108410 0x7f0d9c1087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.728+0000 7f0da0a00700 1 --2- 172.21.15.27:0/1015370464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9c1023a0 0x7f0d9c102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.728+0000 7f0da0a00700 1 -- 172.21.15.27:0/1015370464 >> 172.21.15.27:0/1015370464 conn(0x7f0d9c079350 msgr2=0x7f0d9c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.728+0000 7f0da0a00700 1 -- 172.21.15.27:0/1015370464 shutdown_connections 2024-05-10T11:27:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.728+0000 7f0da0a00700 1 -- 172.21.15.27:0/1015370464 wait complete. 2024-05-10T11:27:57.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.729+0000 7f0da0a00700 1 Processor -- start 2024-05-10T11:27:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.729+0000 7f0da0a00700 1 -- start start 2024-05-10T11:27:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.730+0000 7f0da0a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c1023a0 0x7f0d9c19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.730+0000 7f0da0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d9c102d60 0x7f0d9c19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.730+0000 7f0da0a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9c108410 0x7f0d9c199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.730+0000 7f0da0a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c113680 con 0x7f0d9c102d60 2024-05-10T11:27:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.730+0000 7f0da0a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c113500 con 0x7f0d9c1023a0 2024-05-10T11:27:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.730+0000 7f0da0a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c113800 con 0x7f0d9c108410 2024-05-10T11:27:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.730+0000 7f0d99e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c1023a0 0x7f0d9c19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.730+0000 7f0d99e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c1023a0 0x7f0d9c19f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51606/0 (socket says 172.21.15.27:51606) 2024-05-10T11:27:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.730+0000 7f0d99e00700 1 -- 172.21.15.27:0/138316102 learned_addr learned my addr 172.21.15.27:0/138316102 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:57.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.730+0000 7f0d99400700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d9c102d60 0x7f0d9c19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.731+0000 7f0d99e00700 1 -- 172.21.15.27:0/138316102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9c108410 msgr2=0x7f0d9c199900 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-10T11:27:57.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.731+0000 7f0d99e00700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9c108410 0x7f0d9c199900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.731+0000 7f0d99e00700 1 -- 172.21.15.27:0/138316102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d9c102d60 msgr2=0x7f0d9c19fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:57.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.731+0000 7f0d99e00700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d9c102d60 0x7f0d9c19fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.731+0000 7f0d99e00700 1 -- 172.21.15.27:0/138316102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9c19a060 con 0x7f0d9c1023a0 2024-05-10T11:27:57.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.731+0000 7f0d99e00700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c1023a0 0x7f0d9c19f740 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00eb30 tx=0x7f0d8c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:57.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.731+0000 7f0d92a00700 1 -- 172.21.15.27:0/138316102 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c00cc90 con 0x7f0d9c1023a0 2024-05-10T11:27:57.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.731+0000 7f0da0a00700 1 -- 172.21.15.27:0/138316102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d9c19a350 con 0x7f0d9c1023a0 2024-05-10T11:27:57.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.731+0000 7f0d92a00700 1 -- 172.21.15.27:0/138316102 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d8c0044d0 con 0x7f0d9c1023a0 2024-05-10T11:27:57.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.731+0000 7f0d92a00700 1 -- 172.21.15.27:0/138316102 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c010430 con 0x7f0d9c1023a0 2024-05-10T11:27:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.732+0000 7f0da0a00700 1 -- 172.21.15.27:0/138316102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9c10fc10 con 0x7f0d9c1023a0 2024-05-10T11:27:57.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.735+0000 7f0d92a00700 1 -- 172.21.15.27:0/138316102 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0d8c003ed0 con 0x7f0d9c1023a0 2024-05-10T11:27:57.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.737+0000 7f0dab3fab80 1 Processor -- start 2024-05-10T11:27:57.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.737+0000 7f0dab3fab80 1 -- start start 2024-05-10T11:27:57.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.737+0000 7f0d92a00700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0d800b29f0 0x7f0d800b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.737+0000 7f0d92a00700 1 -- 172.21.15.27:0/138316102 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7f0d8c0e2d00 con 0x7f0d9c1023a0 2024-05-10T11:27:57.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.737+0000 7f0dab3fab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c60fdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.737+0000 7f0dab3fab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c6101ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.737+0000 7f0dab3fab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c61021e0 0x55e5c6102640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.737+0000 7f0dab3fab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e5c6047a20 con 0x55e5c60fe190 2024-05-10T11:27:57.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.737+0000 7f0dab3fab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55e5c5fe5760 con 0x55e5c61021e0 2024-05-10T11:27:57.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.737+0000 7f0dab3fab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e5c60439e0 con 0x55e5c60fd870 2024-05-10T11:27:57.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.738+0000 7f0d99e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c60fdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.738+0000 7f0d99e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c60fdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:43134/0 (socket says 172.21.15.27:43134) 2024-05-10T11:27:57.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.738+0000 7f0d99e00700 1 -- 172.21.15.27:0/562486823 learned_addr learned my addr 172.21.15.27:0/562486823 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:57.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.738+0000 7f0d9a800700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0d800b29f0 0x7f0d800b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.738+0000 7f0d99e00700 1 -- 172.21.15.27:0/562486823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c61021e0 msgr2=0x55e5c6102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:57.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.738+0000 7f0d99e00700 1 --2- 172.21.15.27:0/562486823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c61021e0 0x55e5c6102640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.738+0000 7f0d99e00700 1 -- 172.21.15.27:0/562486823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 msgr2=0x55e5c6101ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:57.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.738+0000 7f0d99e00700 1 --2- 172.21.15.27:0/562486823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c6101ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.738+0000 7f0d99e00700 1 -- 172.21.15.27:0/562486823 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5c5f948d0 con 0x55e5c60fd870 2024-05-10T11:27:57.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.739+0000 7f0d99e00700 1 --2- 172.21.15.27:0/562486823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c60fdc50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c0a5bd0 tx=0x7f0d8c0a95d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf600fd619e27b10 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:57.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.739+0000 7f0d87400700 1 -- 172.21.15.27:0/562486823 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0a9660 con 0x55e5c60fd870 2024-05-10T11:27:57.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.739+0000 7f0d87400700 1 -- 172.21.15.27:0/562486823 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d8c0aa250 con 0x55e5c60fd870 2024-05-10T11:27:57.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.739+0000 7f0d87400700 1 -- 172.21.15.27:0/562486823 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0a5de0 con 0x55e5c60fd870 2024-05-10T11:27:57.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.740+0000 7f0dab3fab80 1 -- 172.21.15.27:0/562486823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 msgr2=0x55e5c60fdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:57.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.740+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/562486823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c60fdc50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c0a5bd0 tx=0x7f0d8c0a95d0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.740+0000 7f0d9a800700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0d800b29f0 0x7f0d800b4eb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0d940062a0 tx=0x7f0d94006230 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.740+0000 7f0dab3fab80 1 -- 172.21.15.27:0/562486823 shutdown_connections 2024-05-10T11:27:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.740+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/562486823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c60fdc50 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.740+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/562486823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c61021e0 0x55e5c6102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.740+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/562486823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c6101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.740+0000 7f0dab3fab80 1 -- 172.21.15.27:0/562486823 >> 172.21.15.27:0/562486823 conn(0x55e5c6023b60 msgr2=0x55e5c60ef7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.741+0000 7f0dab3fab80 1 -- 172.21.15.27:0/562486823 shutdown_connections 2024-05-10T11:27:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.741+0000 7f0dab3fab80 1 -- 172.21.15.27:0/562486823 wait complete. 2024-05-10T11:27:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.741+0000 7f0dab3fab80 1 -- start start 2024-05-10T11:27:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.742+0000 7f0dab3fab80 1 Processor -- start 2024-05-10T11:27:57.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.742+0000 7f0dab3fab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c6109710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.743+0000 7f0dab3fab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c6106ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.743+0000 7f0dab3fab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c61021e0 0x55e5c6106ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.743+0000 7f0dab3fab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e5c610a3f0 con 0x55e5c60fe190 2024-05-10T11:27:57.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.743+0000 7f0dab3fab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55e5c5fadb20 con 0x55e5c61021e0 2024-05-10T11:27:57.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.743+0000 7f0dab3fab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e5c610a570 con 0x55e5c60fd870 2024-05-10T11:27:57.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.743+0000 7f0dab3fab80 10 client.0 _ll_register_callbacks cb 0x55e5c5feec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:57.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.743+0000 7f0d9a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c61021e0 0x55e5c6106ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.743+0000 7f0d9a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c61021e0 0x55e5c6106ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51624/0 (socket says 172.21.15.27:51624) 2024-05-10T11:27:57.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.743+0000 7f0d9a800700 1 -- 172.21.15.27:0/1314614295 learned_addr learned my addr 172.21.15.27:0/1314614295 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:57.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.743+0000 7f0d99e00700 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c6109710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0d99400700 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c6106ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0d99e00700 1 -- 172.21.15.27:0/1314614295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c61021e0 msgr2=0x55e5c6106ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0d99e00700 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c61021e0 0x55e5c6106ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0d99e00700 1 -- 172.21.15.27:0/1314614295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 msgr2=0x55e5c6106ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0d99e00700 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c6106ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0d99e00700 1 -- 172.21.15.27:0/1314614295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5c600d420 con 0x55e5c60fd870 2024-05-10T11:27:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0d9a800700 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c61021e0 0x55e5c6106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:27:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0d99400700 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c6106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:27:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0d99e00700 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c6109710 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c0a5bd0 tx=0x7f0d8c00cf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9116ff803566bb5e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0d7be00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:27:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0d7be00700 1 -- 172.21.15.27:0/1314614295 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0a9660 con 0x55e5c60fd870 2024-05-10T11:27:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0d7be00700 1 -- 172.21.15.27:0/1314614295 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7f0d8c0cae00 con 0x55e5c60fd870 2024-05-10T11:27:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.744+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1314614295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d8c0aa250 con 0x55e5c60fd870 2024-05-10T11:27:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.745+0000 7f0d7be00700 1 -- 172.21.15.27:0/1314614295 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f0d8c0c4240 con 0x55e5c60fd870 2024-05-10T11:27:57.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.745+0000 7f0dab3fab80 10 client.27274 fetch_fsmap learned FSMap version 40 2024-05-10T11:27:57.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.745+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1314614295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e5c610a3f0 con 0x55e5c60fd870 2024-05-10T11:27:57.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.745+0000 7f0d7be00700 1 -- 172.21.15.27:0/1314614295 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d8c0aa250 con 0x55e5c60fd870 2024-05-10T11:27:57.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.746+0000 7f0d7be00700 1 -- 172.21.15.27:0/1314614295 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f0d8c06f2b0 con 0x55e5c60fd870 2024-05-10T11:27:57.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.747+0000 7f0dab3fab80 10 client.27274 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:27:57.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.747+0000 7f0dab3fab80 10 client.27274 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:27:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.747+0000 7f0dab3fab80 20 client.27274 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.747+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5c6117000 0x55e5c61193f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.747+0000 7f0dab3fab80 4 client.27274 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:27:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.747+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1314614295 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5c60c20a0 con 0x55e5c6117000 2024-05-10T11:27:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.747+0000 7f0d99400700 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5c6117000 0x55e5c61193f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.748+0000 7f0d99400700 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5c6117000 0x55e5c61193f0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5faf6ce86d71dd74 server_cookie=9953d494b8061ae4 in_seq=0 out_seq=0 2024-05-10T11:27:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.748+0000 7f0d7be00700 10 client.27274 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:57.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.750+0000 7f0d7be00700 1 -- 172.21.15.27:0/1314614295 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e5c60c20a0 con 0x55e5c6117000 2024-05-10T11:27:57.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.750+0000 7f0d7be00700 10 client.27274 handle_command_reply: tid=0 2024-05-10T11:27:57.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.750+0000 7f0dab3fab80 1 client.27274 shutdown 2024-05-10T11:27:57.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.751+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1314614295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 msgr2=0x55e5c6109710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:57.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.751+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c6109710 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c0a5bd0 tx=0x7f0d8c00cf50 comp rx=0 tx=0).stop 2024-05-10T11:27:57.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.751+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1314614295 shutdown_connections 2024-05-10T11:27:57.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.751+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5c6117000 0x55e5c61193f0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.751+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c6109710 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.751+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c61021e0 0x55e5c6106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.751+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/1314614295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c6106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.751+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1314614295 >> 172.21.15.27:0/1314614295 conn(0x55e5c6023b60 msgr2=0x55e5c60ea2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:57.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.751+0000 7f0d7be00700 0 client.27274 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:57.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.751+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1314614295 shutdown_connections 2024-05-10T11:27:57.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.751+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1314614295 wait complete. 2024-05-10T11:27:57.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.752+0000 7f0dab3fab80 20 client.27274 trim_cache size 0 max 16384 2024-05-10T11:27:57.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.784+0000 7f0dab3fab80 1 Processor -- start 2024-05-10T11:27:57.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.784+0000 7f0dab3fab80 1 -- start start 2024-05-10T11:27:57.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.784+0000 7f0dab3fab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c60fd870 0x55e5c6107020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.784+0000 7f0dab3fab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c61035d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.784+0000 7f0dab3fab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c61021e0 0x55e5c6113480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.784+0000 7f0dab3fab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c1ad4d0 con 0x55e5c60fe190 2024-05-10T11:27:57.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.784+0000 7f0dab3fab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c1af100 con 0x55e5c60fd870 2024-05-10T11:27:57.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.785+0000 7f0dab3fab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c1adba0 con 0x55e5c61021e0 2024-05-10T11:27:57.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.785+0000 7f0d99e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c60fd870 0x55e5c6107020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.785+0000 7f0d99e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c60fd870 0x55e5c6107020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51640/0 (socket says 172.21.15.27:51640) 2024-05-10T11:27:57.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.785+0000 7f0d99e00700 1 -- 172.21.15.27:0/1388206890 learned_addr learned my addr 172.21.15.27:0/1388206890 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.785+0000 7f0d99400700 1 --2- 172.21.15.27:0/1388206890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c61035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.786+0000 7f0d99e00700 1 -- 172.21.15.27:0/1388206890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c61021e0 msgr2=0x55e5c6113480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.785+0000 7f0d9a800700 1 --2- 172.21.15.27:0/1388206890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c61021e0 0x55e5c6113480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.786+0000 7f0d99e00700 1 --2- 172.21.15.27:0/1388206890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c61021e0 0x55e5c6113480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.786+0000 7f0d99e00700 1 -- 172.21.15.27:0/1388206890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 msgr2=0x55e5c61035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.786+0000 7f0d99e00700 1 --2- 172.21.15.27:0/1388206890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c61035d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.786+0000 7f0d99e00700 1 -- 172.21.15.27:0/1388206890 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5c60f0610 con 0x55e5c60fd870 2024-05-10T11:27:57.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.786+0000 7f0d99400700 1 --2- 172.21.15.27:0/1388206890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c61035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:27:57.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.786+0000 7f0d99e00700 1 --2- 172.21.15.27:0/1388206890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c60fd870 0x55e5c6107020 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c0a9850 tx=0x7f0d8c0a5b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e6766b1537136b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:57.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.786+0000 7f0d9a800700 1 --2- 172.21.15.27:0/1388206890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c61021e0 0x55e5c6113480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:27:57.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.787+0000 7f0d86a00700 1 -- 172.21.15.27:0/1388206890 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0a9660 con 0x55e5c60fd870 2024-05-10T11:27:57.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.787+0000 7f0d86a00700 1 -- 172.21.15.27:0/1388206890 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d8c0aa250 con 0x55e5c60fd870 2024-05-10T11:27:57.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.787+0000 7f0d86a00700 1 -- 172.21.15.27:0/1388206890 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0b2730 con 0x55e5c60fd870 2024-05-10T11:27:57.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.788+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1388206890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c60fd870 msgr2=0x55e5c6107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:57.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.788+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/1388206890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c60fd870 0x55e5c6107020 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c0a9850 tx=0x7f0d8c0a5b40 comp rx=0 tx=0).stop 2024-05-10T11:27:57.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.788+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1388206890 shutdown_connections 2024-05-10T11:27:57.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.788+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/1388206890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c61021e0 0x55e5c6113480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.788+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/1388206890 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c60fd870 0x55e5c6107020 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.788+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/1388206890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c60fe190 0x55e5c61035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.788+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1388206890 >> 172.21.15.27:0/1388206890 conn(0x55e5c6023b60 msgr2=0x55e5c61010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:57.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.789+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1388206890 shutdown_connections 2024-05-10T11:27:57.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.789+0000 7f0dab3fab80 1 -- 172.21.15.27:0/1388206890 wait complete. 2024-05-10T11:27:57.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.790+0000 7f0dab3fab80 1 -- start start 2024-05-10T11:27:57.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.790+0000 7f0dab3fab80 1 Processor -- start 2024-05-10T11:27:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.791+0000 7f0dab3fab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c610f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.791+0000 7f0dab3fab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c60fe190 0x55e5c610fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.791+0000 7f0dab3fab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c61021e0 0x55e5c610c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.791+0000 7f0dab3fab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c1ae970 con 0x55e5c61021e0 2024-05-10T11:27:57.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.791+0000 7f0dab3fab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c1ae2a0 con 0x55e5c60fe190 2024-05-10T11:27:57.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.791+0000 7f0dab3fab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c1ab490 con 0x55e5c60fd870 2024-05-10T11:27:57.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.791+0000 7f0dab3fab80 10 client.0 _ll_register_callbacks cb 0x55e5c5feec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:27:57.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.791+0000 7f0d99e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c610f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.791+0000 7f0d99e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c610f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:43164/0 (socket says 172.21.15.27:43164) 2024-05-10T11:27:57.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.791+0000 7f0d99e00700 1 -- 172.21.15.27:0/3942268513 learned_addr learned my addr 172.21.15.27:0/3942268513 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:27:57.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.792+0000 7f0d99e00700 1 -- 172.21.15.27:0/3942268513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c60fe190 msgr2=0x55e5c610fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:27:57.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.792+0000 7f0d99400700 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c60fe190 0x55e5c610fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.792+0000 7f0d99e00700 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c60fe190 0x55e5c610fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.792+0000 7f0d99e00700 1 -- 172.21.15.27:0/3942268513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c61021e0 msgr2=0x55e5c610c7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:27:57.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.793+0000 7f0d9a800700 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c61021e0 0x55e5c610c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.793+0000 7f0d99e00700 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c61021e0 0x55e5c610c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.793+0000 7f0d99e00700 1 -- 172.21.15.27:0/3942268513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5c610ce80 con 0x55e5c60fd870 2024-05-10T11:27:57.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.793+0000 7f0d9a800700 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c61021e0 0x55e5c610c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:27:57.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.793+0000 7f0d99e00700 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c610f800 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009230 tx=0x7f0d8c005b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=869457104f70b036 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:27:57.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.793+0000 7f0d7be00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:27:57.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.793+0000 7f0d7be00700 1 -- 172.21.15.27:0/3942268513 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0adf00 con 0x55e5c60fd870 2024-05-10T11:27:57.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.794+0000 7f0d7be00700 1 -- 172.21.15.27:0/3942268513 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7f0d8c0cae00 con 0x55e5c60fd870 2024-05-10T11:27:57.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.794+0000 7f0dab3fab80 1 -- 172.21.15.27:0/3942268513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d8c0aa250 con 0x55e5c60fd870 2024-05-10T11:27:57.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.794+0000 7f0d7be00700 1 -- 172.21.15.27:0/3942268513 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f0d8c0b29b0 con 0x55e5c60fd870 2024-05-10T11:27:57.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.795+0000 7f0dab3fab80 10 client.27277 fetch_fsmap learned FSMap version 40 2024-05-10T11:27:57.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.795+0000 7f0dab3fab80 1 -- 172.21.15.27:0/3942268513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e5c610ce80 con 0x55e5c60fd870 2024-05-10T11:27:57.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.796+0000 7f0d7be00700 1 -- 172.21.15.27:0/3942268513 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d8c0a6140 con 0x55e5c60fd870 2024-05-10T11:27:57.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.796+0000 7f0d7be00700 1 -- 172.21.15.27:0/3942268513 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f0d8c0af090 con 0x55e5c60fd870 2024-05-10T11:27:57.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.796+0000 7f0dab3fab80 10 client.27277 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:27:57.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.796+0000 7f0dab3fab80 10 client.27277 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:27:57.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.796+0000 7f0dab3fab80 20 client.27277 populate_metadata read hostname 'smithi027' 2024-05-10T11:27:57.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.796+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5c6182140 0x55e5c6184560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:27:57.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.796+0000 7f0dab3fab80 4 client.27277 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:27:57.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.796+0000 7f0dab3fab80 1 -- 172.21.15.27:0/3942268513 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f0d8c0aa250 con 0x55e5c6182140 2024-05-10T11:27:57.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.797+0000 7f0d99400700 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5c6182140 0x55e5c6184560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:27:57.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.799+0000 7f0d7be00700 10 client.27277 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:57.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.800+0000 7f0d99400700 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5c6182140 0x55e5c6184560 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ccf407841e76012 server_cookie=e7c40f9752aa8684 in_seq=0 out_seq=0 2024-05-10T11:27:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.803+0000 7f0d7be00700 1 -- 172.21.15.27:0/3942268513 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f0d8c0aa250 con 0x55e5c6182140 2024-05-10T11:27:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.803+0000 7f0d7be00700 10 client.27277 handle_command_reply: tid=0 2024-05-10T11:27:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.803+0000 7f0dab3fab80 1 client.27277 shutdown 2024-05-10T11:27:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.804+0000 7f0dab3fab80 1 -- 172.21.15.27:0/3942268513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 msgr2=0x55e5c610f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.804+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c610f800 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009230 tx=0x7f0d8c005b40 comp rx=0 tx=0).stop 2024-05-10T11:27:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.804+0000 7f0dab3fab80 1 -- 172.21.15.27:0/3942268513 shutdown_connections 2024-05-10T11:27:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.804+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55e5c6182140 0x55e5c6184560 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.804+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5c60fd870 0x55e5c610f800 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.804+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e5c61021e0 0x55e5c610c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.804+0000 7f0dab3fab80 1 --2- 172.21.15.27:0/3942268513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e5c60fe190 0x55e5c610fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.804+0000 7f0dab3fab80 1 -- 172.21.15.27:0/3942268513 >> 172.21.15.27:0/3942268513 conn(0x55e5c6023b60 msgr2=0x55e5c6104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.805+0000 7f0d7be00700 0 client.27277 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:27:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.805+0000 7f0dab3fab80 1 -- 172.21.15.27:0/3942268513 shutdown_connections 2024-05-10T11:27:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.805+0000 7f0dab3fab80 1 -- 172.21.15.27:0/3942268513 wait complete. 2024-05-10T11:27:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.805+0000 7f0dab3fab80 20 client.27277 trim_cache size 0 max 16384 2024-05-10T11:27:57.806 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:27:57.807 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:27:57.807 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:27:57.807 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:27:57.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.806+0000 7f0da0a00700 1 -- 172.21.15.27:0/138316102 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0d800b29f0 msgr2=0x7f0d800b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:57.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.806+0000 7f0da0a00700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0d800b29f0 0x7f0d800b4eb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0d940062a0 tx=0x7f0d94006230 comp rx=0 tx=0).stop 2024-05-10T11:27:57.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.806+0000 7f0da0a00700 1 -- 172.21.15.27:0/138316102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c1023a0 msgr2=0x7f0d9c19f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:27:57.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.806+0000 7f0da0a00700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c1023a0 0x7f0d9c19f740 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00eb30 tx=0x7f0d8c00ee40 comp rx=0 tx=0).stop 2024-05-10T11:27:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.808+0000 7f0da0a00700 1 -- 172.21.15.27:0/138316102 shutdown_connections 2024-05-10T11:27:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.808+0000 7f0da0a00700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0d800b29f0 0x7f0d800b4eb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.808+0000 7f0da0a00700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9c108410 0x7f0d9c199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.808+0000 7f0da0a00700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c1023a0 0x7f0d9c19f740 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.809+0000 7f0da0a00700 1 --2- 172.21.15.27:0/138316102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d9c102d60 0x7f0d9c19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:27:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.809+0000 7f0da0a00700 1 -- 172.21.15.27:0/138316102 >> 172.21.15.27:0/138316102 conn(0x7f0d9c079350 msgr2=0x7f0d9c100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:27:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.809+0000 7f0da0a00700 1 -- 172.21.15.27:0/138316102 shutdown_connections 2024-05-10T11:27:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:27:57.809+0000 7f0da0a00700 1 -- 172.21.15.27:0/138316102 wait complete. 2024-05-10T11:27:58.142 INFO:tasks.workunit.client.0.smithi027.stdout: 10 21140 31.21 MB/sec warmup 42 sec latency 202.253 ms 2024-05-10T11:27:58.757 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:27:58.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:27:59.142 INFO:tasks.workunit.client.0.smithi027.stdout: 10 21583 31.03 MB/sec warmup 43 sec latency 84.239 ms 2024-05-10T11:27:59.321 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:27:59.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:27:59 smithi027 ceph-mon[30054]: pgmap v461: 129 pgs: 129 active+clean; 861 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 315 op/s 2024-05-10T11:27:59.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:27:59 smithi076 ceph-mon[34152]: pgmap v461: 129 pgs: 129 active+clean; 861 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 315 op/s 2024-05-10T11:27:59.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:27:59 smithi189 ceph-mon[34842]: pgmap v461: 129 pgs: 129 active+clean; 861 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 315 op/s 2024-05-10T11:28:00.142 INFO:tasks.workunit.client.0.smithi027.stdout: 10 22332 31.20 MB/sec warmup 44 sec latency 73.178 ms 2024-05-10T11:28:01.142 INFO:tasks.workunit.client.0.smithi027.stdout: 10 22957 31.26 MB/sec warmup 45 sec latency 137.189 ms 2024-05-10T11:28:01.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:01 smithi027 ceph-mon[30054]: pgmap v462: 129 pgs: 129 active+clean; 879 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.3 MiB/s wr, 463 op/s 2024-05-10T11:28:01.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:01 smithi076 ceph-mon[34152]: pgmap v462: 129 pgs: 129 active+clean; 879 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.3 MiB/s wr, 463 op/s 2024-05-10T11:28:01.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:01 smithi189 ceph-mon[34842]: pgmap v462: 129 pgs: 129 active+clean; 879 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.3 MiB/s wr, 463 op/s 2024-05-10T11:28:01.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.580+0000 7fdc38600700 1 -- 172.21.15.27:0/1294984604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc30118530 msgr2=0x7fdc3011a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:01.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.580+0000 7fdc38600700 1 --2- 172.21.15.27:0/1294984604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc30118530 0x7fdc3011a920 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800b0a0 tx=0x7fdc2800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:28:01.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.581+0000 7fdc38600700 1 -- 172.21.15.27:0/1294984604 shutdown_connections 2024-05-10T11:28:01.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.581+0000 7fdc38600700 1 --2- 172.21.15.27:0/1294984604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc30118530 0x7fdc3011a920 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:01.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.581+0000 7fdc38600700 1 --2- 172.21.15.27:0/1294984604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc3010db90 0x7fdc3010a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:01.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.581+0000 7fdc38600700 1 --2- 172.21.15.27:0/1294984604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc3010a9e0 0x7fdc3010ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:01.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.581+0000 7fdc38600700 1 -- 172.21.15.27:0/1294984604 >> 172.21.15.27:0/1294984604 conn(0x7fdc3006e9f0 msgr2=0x7fdc3006ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:01.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.583+0000 7fdc38600700 1 -- 172.21.15.27:0/1294984604 shutdown_connections 2024-05-10T11:28:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.583+0000 7fdc38600700 1 -- 172.21.15.27:0/1294984604 wait complete. 2024-05-10T11:28:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.585+0000 7fdc38600700 1 Processor -- start 2024-05-10T11:28:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.585+0000 7fdc38600700 1 -- start start 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.586+0000 7fdc38600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc3010a9e0 0x7fdc301aa580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.586+0000 7fdc38600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc3010db90 0x7fdc301aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.586+0000 7fdc38600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc30118530 0x7fdc30111190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.586+0000 7fdc38600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3011e030 con 0x7fdc3010a9e0 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.586+0000 7fdc38600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3011deb0 con 0x7fdc30118530 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.586+0000 7fdc38600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3011e1b0 con 0x7fdc3010db90 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.586+0000 7fdc36600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc30118530 0x7fdc30111190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.586+0000 7fdc36600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc30118530 0x7fdc30111190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51670/0 (socket says 172.21.15.27:51670) 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.586+0000 7fdc36600700 1 -- 172.21.15.27:0/2652145070 learned_addr learned my addr 172.21.15.27:0/2652145070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.586+0000 7fdc35200700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc3010db90 0x7fdc301aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.588+0000 7fdc36600700 1 -- 172.21.15.27:0/2652145070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc3010db90 msgr2=0x7fdc301aaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.588+0000 7fdc36600700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc3010db90 0x7fdc301aaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.588+0000 7fdc36600700 1 -- 172.21.15.27:0/2652145070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc3010a9e0 msgr2=0x7fdc301aa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.588+0000 7fdc36600700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc3010a9e0 0x7fdc301aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.588+0000 7fdc36600700 1 -- 172.21.15.27:0/2652145070 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc301118d0 con 0x7fdc30118530 2024-05-10T11:28:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.588+0000 7fdc36600700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc30118530 0x7fdc30111190 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800b070 tx=0x7fdc28004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.588+0000 7fdc2e000700 1 -- 172.21.15.27:0/2652145070 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc28021070 con 0x7fdc30118530 2024-05-10T11:28:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.588+0000 7fdc38600700 1 -- 172.21.15.27:0/2652145070 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc30111b60 con 0x7fdc30118530 2024-05-10T11:28:01.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.589+0000 7fdc38600700 1 -- 172.21.15.27:0/2652145070 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc30117470 con 0x7fdc30118530 2024-05-10T11:28:01.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.589+0000 7fdc2e000700 1 -- 172.21.15.27:0/2652145070 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc280074e0 con 0x7fdc30118530 2024-05-10T11:28:01.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.589+0000 7fdc2e000700 1 -- 172.21.15.27:0/2652145070 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2801dd20 con 0x7fdc30118530 2024-05-10T11:28:01.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.591+0000 7fdc38600700 1 -- 172.21.15.27:0/2652145070 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc10005330 con 0x7fdc30118530 2024-05-10T11:28:01.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.596+0000 7fdc2e000700 1 -- 172.21.15.27:0/2652145070 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fdc28053d10 con 0x7fdc30118530 2024-05-10T11:28:01.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.599+0000 7fdc2e000700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fdc180b2a70 0x7fdc180b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:01.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.599+0000 7fdc2e000700 1 -- 172.21.15.27:0/2652145070 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7fdc2800f020 con 0x7fdc30118530 2024-05-10T11:28:01.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.600+0000 7fdc35c00700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fdc180b2a70 0x7fdc180b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:01.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.601+0000 7fdc2e000700 1 -- 172.21.15.27:0/2652145070 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fdc2802b020 con 0x7fdc30118530 2024-05-10T11:28:01.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.602+0000 7fdc35c00700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fdc180b2a70 0x7fdc180b4f30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400a8f0 tx=0x7fdc24005bf0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:01.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.952+0000 7fdc38600700 1 -- 172.21.15.27:0/2652145070 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc10005620 con 0x7fdc30118530 2024-05-10T11:28:01.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.953+0000 7fdc2e000700 1 -- 172.21.15.27:0/2652145070 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11945 (secure 0 0 0) 0x7fdc280ab790 con 0x7fdc30118530 2024-05-10T11:28:01.954 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:28:01.955 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:27:21.585392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24484,24527,24472,24488,24467]},"id":1}]} 2024-05-10T11:28:01.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.957+0000 7fdc38600700 1 -- 172.21.15.27:0/2652145070 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fdc180b2a70 msgr2=0x7fdc180b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:01.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.957+0000 7fdc38600700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fdc180b2a70 0x7fdc180b4f30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400a8f0 tx=0x7fdc24005bf0 comp rx=0 tx=0).stop 2024-05-10T11:28:01.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.957+0000 7fdc38600700 1 -- 172.21.15.27:0/2652145070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc30118530 msgr2=0x7fdc30111190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:01.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.957+0000 7fdc38600700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc30118530 0x7fdc30111190 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800b070 tx=0x7fdc28004ce0 comp rx=0 tx=0).stop 2024-05-10T11:28:01.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.957+0000 7fdc38600700 1 -- 172.21.15.27:0/2652145070 shutdown_connections 2024-05-10T11:28:01.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.957+0000 7fdc38600700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fdc180b2a70 0x7fdc180b4f30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:01.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.957+0000 7fdc38600700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc30118530 0x7fdc30111190 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:01.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.957+0000 7fdc38600700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc3010a9e0 0x7fdc301aa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:01.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.957+0000 7fdc38600700 1 --2- 172.21.15.27:0/2652145070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc3010db90 0x7fdc301aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:01.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.957+0000 7fdc38600700 1 -- 172.21.15.27:0/2652145070 >> 172.21.15.27:0/2652145070 conn(0x7fdc3006e9f0 msgr2=0x7fdc30119c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:01.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.957+0000 7fdc38600700 1 -- 172.21.15.27:0/2652145070 shutdown_connections 2024-05-10T11:28:01.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:01.957+0000 7fdc38600700 1 -- 172.21.15.27:0/2652145070 wait complete. 2024-05-10T11:28:01.959 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 40 2024-05-10T11:28:02.143 INFO:tasks.workunit.client.0.smithi027.stdout: 10 22957 30.58 MB/sec warmup 46 sec latency 1064.972 ms 2024-05-10T11:28:02.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:02 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2652145070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:28:02.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:02 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2652145070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:28:02.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:02 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2652145070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:28:02.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:28:03.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:03 smithi027 ceph-mon[30054]: pgmap v463: 129 pgs: 129 active+clean; 879 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 317 op/s 2024-05-10T11:28:03.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:03 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:28:03.143 INFO:tasks.workunit.client.0.smithi027.stdout: 10 22957 29.93 MB/sec warmup 47 sec latency 2065.029 ms 2024-05-10T11:28:03.344 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:28:03.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:03 smithi076 ceph-mon[34152]: pgmap v463: 129 pgs: 129 active+clean; 879 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 317 op/s 2024-05-10T11:28:03.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:03 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:28:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:03 smithi189 ceph-mon[34842]: pgmap v463: 129 pgs: 129 active+clean; 879 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 317 op/s 2024-05-10T11:28:03.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:03 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:28:04.143 INFO:tasks.workunit.client.0.smithi027.stdout: 10 22957 29.30 MB/sec warmup 48 sec latency 3065.143 ms 2024-05-10T11:28:05.143 INFO:tasks.workunit.client.0.smithi027.stdout: 10 23360 29.10 MB/sec warmup 49 sec latency 3145.574 ms 2024-05-10T11:28:05.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:05 smithi027 ceph-mon[30054]: pgmap v464: 129 pgs: 129 active+clean; 879 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 317 op/s 2024-05-10T11:28:05.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:05 smithi076 ceph-mon[34152]: pgmap v464: 129 pgs: 129 active+clean; 879 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 317 op/s 2024-05-10T11:28:05.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:05 smithi189 ceph-mon[34842]: pgmap v464: 129 pgs: 129 active+clean; 879 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 317 op/s 2024-05-10T11:28:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.806+0000 7f0ea5200700 1 -- 172.21.15.27:0/82059192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ea010b6e0 msgr2=0x7f0ea010bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.806+0000 7f0ea5200700 1 --2- 172.21.15.27:0/82059192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ea010b6e0 0x7f0ea010bac0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800b0a0 tx=0x7f0e9800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.807+0000 7f0ea5200700 1 -- 172.21.15.27:0/82059192 shutdown_connections 2024-05-10T11:28:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.807+0000 7f0ea5200700 1 --2- 172.21.15.27:0/82059192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01060a0 0x7f0ea01125b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.807+0000 7f0ea5200700 1 --2- 172.21.15.27:0/82059192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ea010b6e0 0x7f0ea010bac0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.807+0000 7f0ea5200700 1 --2- 172.21.15.27:0/82059192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea01056e0 0x7f0ea0105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.807+0000 7f0ea5200700 1 -- 172.21.15.27:0/82059192 >> 172.21.15.27:0/82059192 conn(0x7f0ea01011d0 msgr2=0x7f0ea01035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.807+0000 7f0ea5200700 1 -- 172.21.15.27:0/82059192 shutdown_connections 2024-05-10T11:28:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.807+0000 7f0ea5200700 1 -- 172.21.15.27:0/82059192 wait complete. 2024-05-10T11:28:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.808+0000 7f0ea5200700 1 Processor -- start 2024-05-10T11:28:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.808+0000 7f0ea5200700 1 -- start start 2024-05-10T11:28:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.809+0000 7f0ea5200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01056e0 0x7f0ea0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.809+0000 7f0ea5200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ea01060a0 0x7f0ea0075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.809+0000 7f0ea5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea010b6e0 0x7f0ea006ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.809+0000 7f0e9de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01056e0 0x7f0ea0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.809+0000 7f0e9de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01056e0 0x7f0ea0074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54692/0 (socket says 172.21.15.27:54692) 2024-05-10T11:28:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.809+0000 7f0e9de00700 1 -- 172.21.15.27:0/3552304911 learned_addr learned my addr 172.21.15.27:0/3552304911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.809+0000 7f0ea5200700 1 -- 172.21.15.27:0/3552304911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea006d780 con 0x7f0ea010b6e0 2024-05-10T11:28:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.809+0000 7f0ea5200700 1 -- 172.21.15.27:0/3552304911 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea006d600 con 0x7f0ea01060a0 2024-05-10T11:28:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.809+0000 7f0ea5200700 1 -- 172.21.15.27:0/3552304911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea006d900 con 0x7f0ea01056e0 2024-05-10T11:28:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.809+0000 7f0e9e800700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea010b6e0 0x7f0ea006ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.810+0000 7f0e9de00700 1 -- 172.21.15.27:0/3552304911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ea01060a0 msgr2=0x7f0ea0075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.810+0000 7f0e9d400700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ea01060a0 0x7f0ea0075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.810+0000 7f0e9de00700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ea01060a0 0x7f0ea0075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.810+0000 7f0e9de00700 1 -- 172.21.15.27:0/3552304911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea010b6e0 msgr2=0x7f0ea006ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.810+0000 7f0e9de00700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea010b6e0 0x7f0ea006ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.810+0000 7f0e9de00700 1 -- 172.21.15.27:0/3552304911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea006f530 con 0x7f0ea01056e0 2024-05-10T11:28:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.811+0000 7f0e9e800700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea010b6e0 0x7f0ea006ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:28:05.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.811+0000 7f0e9de00700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01056e0 0x7f0ea0074b10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800b070 tx=0x7f0e98009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:05.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.811+0000 7f0e9d400700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ea01060a0 0x7f0ea0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:28:05.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.811+0000 7f0e96a00700 1 -- 172.21.15.27:0/3552304911 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e98021070 con 0x7f0ea01056e0 2024-05-10T11:28:05.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.811+0000 7f0ea5200700 1 -- 172.21.15.27:0/3552304911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea006f7c0 con 0x7f0ea01056e0 2024-05-10T11:28:05.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.811+0000 7f0ea5200700 1 -- 172.21.15.27:0/3552304911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea01b0fd0 con 0x7f0ea01056e0 2024-05-10T11:28:05.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.811+0000 7f0e96a00700 1 -- 172.21.15.27:0/3552304911 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e98007e20 con 0x7f0ea01056e0 2024-05-10T11:28:05.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.812+0000 7f0e96a00700 1 -- 172.21.15.27:0/3552304911 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e98021070 con 0x7f0ea01056e0 2024-05-10T11:28:05.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.819+0000 7f0e96a00700 1 -- 172.21.15.27:0/3552304911 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0e98024070 con 0x7f0ea01056e0 2024-05-10T11:28:05.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.822+0000 7f0e96a00700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0e840b2df0 0x7f0e840b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.824+0000 7f0e96a00700 1 -- 172.21.15.27:0/3552304911 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7f0e98007c20 con 0x7f0ea01056e0 2024-05-10T11:28:05.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.825+0000 7f0e9d400700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0e840b2df0 0x7f0e840b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.830+0000 7f0e9d400700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0e840b2df0 0x7f0e840b52b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0070350 tx=0x7f0e90009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.831+0000 7f0eafc25b80 1 Processor -- start 2024-05-10T11:28:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.834+0000 7f0eafc25b80 1 -- start start 2024-05-10T11:28:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.834+0000 7f0eafc25b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d060 0x55ad6416d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.834+0000 7f0eafc25b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d980 0x55ad64171490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.834+0000 7f0eafc25b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad641719d0 0x55ad64171e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.835+0000 7f0eafc25b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ad640bfa20 con 0x55ad641719d0 2024-05-10T11:28:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.835+0000 7f0eafc25b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ad6405d760 con 0x55ad6416d980 2024-05-10T11:28:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.835+0000 7f0eafc25b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ad640bb9e0 con 0x55ad6416d060 2024-05-10T11:28:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.835+0000 7f0e9de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d060 0x55ad6416d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.835+0000 7f0e9de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d060 0x55ad6416d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54698/0 (socket says 172.21.15.27:54698) 2024-05-10T11:28:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.835+0000 7f0e9de00700 1 -- 172.21.15.27:0/157674469 learned_addr learned my addr 172.21.15.27:0/157674469 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.835+0000 7f0e9d400700 1 --2- 172.21.15.27:0/157674469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d980 0x55ad64171490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.836+0000 7f0e9de00700 1 -- 172.21.15.27:0/157674469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d980 msgr2=0x55ad64171490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.836+0000 7f0e9de00700 1 --2- 172.21.15.27:0/157674469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d980 0x55ad64171490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.836+0000 7f0e9de00700 1 -- 172.21.15.27:0/157674469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad641719d0 msgr2=0x55ad64171e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.836+0000 7f0e9de00700 1 --2- 172.21.15.27:0/157674469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad641719d0 0x55ad64171e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.836+0000 7f0e9de00700 1 -- 172.21.15.27:0/157674469 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad6400c8d0 con 0x55ad6416d060 2024-05-10T11:28:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.837+0000 7f0e9de00700 1 --2- 172.21.15.27:0/157674469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d060 0x55ad6416d440 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0e980a77c0 tx=0x7f0e980aac50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20002d11572071e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.837+0000 7f0e8b400700 1 -- 172.21.15.27:0/157674469 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9802b070 con 0x55ad6416d060 2024-05-10T11:28:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.837+0000 7f0e8b400700 1 -- 172.21.15.27:0/157674469 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e980b3320 con 0x55ad6416d060 2024-05-10T11:28:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.837+0000 7f0e8b400700 1 -- 172.21.15.27:0/157674469 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e98021070 con 0x55ad6416d060 2024-05-10T11:28:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.839+0000 7f0eafc25b80 1 -- 172.21.15.27:0/157674469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d060 msgr2=0x55ad6416d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.839+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/157674469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d060 0x55ad6416d440 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0e980a77c0 tx=0x7f0e980aac50 comp rx=0 tx=0).stop 2024-05-10T11:28:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.839+0000 7f0eafc25b80 1 -- 172.21.15.27:0/157674469 shutdown_connections 2024-05-10T11:28:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.839+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/157674469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d060 0x55ad6416d440 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.839+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/157674469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad641719d0 0x55ad64171e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.839+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/157674469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d980 0x55ad64171490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.839+0000 7f0eafc25b80 1 -- 172.21.15.27:0/157674469 >> 172.21.15.27:0/157674469 conn(0x55ad6409bb60 msgr2=0x55ad641677a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.839+0000 7f0eafc25b80 1 -- 172.21.15.27:0/157674469 shutdown_connections 2024-05-10T11:28:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.840+0000 7f0eafc25b80 1 -- 172.21.15.27:0/157674469 wait complete. 2024-05-10T11:28:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.840+0000 7f0eafc25b80 1 -- start start 2024-05-10T11:28:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.841+0000 7f0eafc25b80 1 Processor -- start 2024-05-10T11:28:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.842+0000 7f0eafc25b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad6416d060 0x55ad64178ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.842+0000 7f0eafc25b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 0x55ad64176240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.843+0000 7f0eafc25b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad641719d0 0x55ad64176780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.843+0000 7f0e9d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 0x55ad64176240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.843+0000 7f0e9d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 0x55ad64176240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54710/0 (socket says 172.21.15.27:54710) 2024-05-10T11:28:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.843+0000 7f0eafc25b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ad641799a0 con 0x55ad6416d060 2024-05-10T11:28:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.843+0000 7f0eafc25b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ad64025b20 con 0x55ad641719d0 2024-05-10T11:28:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.843+0000 7f0eafc25b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ad64179b20 con 0x55ad6416d980 2024-05-10T11:28:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.844+0000 7f0eafc25b80 10 client.0 _ll_register_callbacks cb 0x55ad64066c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.844+0000 7f0e9d400700 1 -- 172.21.15.27:0/2421946219 learned_addr learned my addr 172.21.15.27:0/2421946219 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:05.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.844+0000 7f0e9de00700 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad6416d060 0x55ad64178ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.844+0000 7f0e9e800700 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad641719d0 0x55ad64176780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.845+0000 7f0e9d400700 1 -- 172.21.15.27:0/2421946219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad641719d0 msgr2=0x55ad64176780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:05.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.846+0000 7f0e9d400700 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad641719d0 0x55ad64176780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.846+0000 7f0e9d400700 1 -- 172.21.15.27:0/2421946219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad6416d060 msgr2=0x55ad64178ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:05.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.846+0000 7f0e9d400700 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad6416d060 0x55ad64178ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.846+0000 7f0e9d400700 1 -- 172.21.15.27:0/2421946219 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad64085420 con 0x55ad6416d980 2024-05-10T11:28:05.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.847+0000 7f0e9d400700 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 0x55ad64176240 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0e90010fb0 tx=0x7f0e90004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e283fc3a843c92e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:05.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.847+0000 7f0e7fe00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:28:05.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.851+0000 7f0e7fe00700 1 -- 172.21.15.27:0/2421946219 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e90029670 con 0x55ad6416d980 2024-05-10T11:28:05.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.851+0000 7f0e7fe00700 1 -- 172.21.15.27:0/2421946219 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7f0e90029810 con 0x55ad6416d980 2024-05-10T11:28:05.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.851+0000 7f0eafc25b80 1 -- 172.21.15.27:0/2421946219 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e980b3320 con 0x55ad6416d980 2024-05-10T11:28:05.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.854+0000 7f0e7fe00700 1 -- 172.21.15.27:0/2421946219 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f0e90029a50 con 0x55ad6416d980 2024-05-10T11:28:05.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.854+0000 7f0eafc25b80 10 client.27292 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:05.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.854+0000 7f0eafc25b80 1 -- 172.21.15.27:0/2421946219 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ad6418ca70 con 0x55ad6416d980 2024-05-10T11:28:05.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.855+0000 7f0e7fe00700 1 -- 172.21.15.27:0/2421946219 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e980b3320 con 0x55ad6416d980 2024-05-10T11:28:05.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.855+0000 7f0e7fe00700 1 -- 172.21.15.27:0/2421946219 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f0e9000f040 con 0x55ad6416d980 2024-05-10T11:28:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.855+0000 7f0eafc25b80 10 client.27292 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.855+0000 7f0eafc25b80 10 client.27292 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:28:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.855+0000 7f0eafc25b80 20 client.27292 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.855+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ad6418f000 0x55ad641913f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.855+0000 7f0eafc25b80 4 client.27292 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:28:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.855+0000 7f0eafc25b80 1 -- 172.21.15.27:0/2421946219 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad6413a0a0 con 0x55ad6418f000 2024-05-10T11:28:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.856+0000 7f0e9de00700 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ad6418f000 0x55ad641913f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.858+0000 7f0e9de00700 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ad6418f000 0x55ad641913f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=492c86676fa72213 server_cookie=c2f0aaca80fa567c in_seq=0 out_seq=0 2024-05-10T11:28:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.858+0000 7f0e7fe00700 10 client.27292 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:28:05.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.860+0000 7f0e7fe00700 1 -- 172.21.15.27:0/2421946219 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ad6413a0a0 con 0x55ad6418f000 2024-05-10T11:28:05.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.860+0000 7f0e7fe00700 10 client.27292 handle_command_reply: tid=0 2024-05-10T11:28:05.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.860+0000 7f0eafc25b80 1 client.27292 shutdown 2024-05-10T11:28:05.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.860+0000 7f0eafc25b80 1 -- 172.21.15.27:0/2421946219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 msgr2=0x55ad64176240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:05.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.860+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 0x55ad64176240 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0e90010fb0 tx=0x7f0e90004750 comp rx=0 tx=0).stop 2024-05-10T11:28:05.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.861+0000 7f0eafc25b80 1 -- 172.21.15.27:0/2421946219 shutdown_connections 2024-05-10T11:28:05.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.861+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad641719d0 0x55ad64176780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.861+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad6416d060 0x55ad64178ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.861+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ad6418f000 0x55ad641913f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.861+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/2421946219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 0x55ad64176240 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.861+0000 7f0eafc25b80 1 -- 172.21.15.27:0/2421946219 >> 172.21.15.27:0/2421946219 conn(0x55ad6409bb60 msgr2=0x55ad641622b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:05.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.861+0000 7f0e7fe00700 0 client.27292 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:28:05.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.861+0000 7f0eafc25b80 1 -- 172.21.15.27:0/2421946219 shutdown_connections 2024-05-10T11:28:05.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.861+0000 7f0eafc25b80 1 -- 172.21.15.27:0/2421946219 wait complete. 2024-05-10T11:28:05.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.862+0000 7f0eafc25b80 20 client.27292 trim_cache size 0 max 16384 2024-05-10T11:28:05.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.902+0000 7f0eafc25b80 1 Processor -- start 2024-05-10T11:28:05.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.902+0000 7f0eafc25b80 1 -- start start 2024-05-10T11:28:05.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.902+0000 7f0eafc25b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d060 0x55ad6418b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.902+0000 7f0eafc25b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 0x55ad64169500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.903+0000 7f0eafc25b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad641719d0 0x55ad64172c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.903+0000 7f0eafc25b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8000ef40 con 0x55ad641719d0 2024-05-10T11:28:05.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.903+0000 7f0eafc25b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0e80010b70 con 0x55ad6416d060 2024-05-10T11:28:05.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.903+0000 7f0eafc25b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8000f610 con 0x55ad6416d980 2024-05-10T11:28:05.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.903+0000 7f0e9d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 0x55ad64169500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.903+0000 7f0e9e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad641719d0 0x55ad64172c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.903+0000 7f0e9d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 0x55ad64169500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54726/0 (socket says 172.21.15.27:54726) 2024-05-10T11:28:05.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.903+0000 7f0e9d400700 1 -- 172.21.15.27:0/3415539000 learned_addr learned my addr 172.21.15.27:0/3415539000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:05.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.904+0000 7f0e9de00700 1 --2- 172.21.15.27:0/3415539000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d060 0x55ad6418b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.904+0000 7f0e9d400700 1 -- 172.21.15.27:0/3415539000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d060 msgr2=0x55ad6418b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:05.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.904+0000 7f0e9d400700 1 --2- 172.21.15.27:0/3415539000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d060 0x55ad6418b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.904+0000 7f0e9d400700 1 -- 172.21.15.27:0/3415539000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad641719d0 msgr2=0x55ad64172c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:05.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.904+0000 7f0e9d400700 1 --2- 172.21.15.27:0/3415539000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad641719d0 0x55ad64172c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.904+0000 7f0e9d400700 1 -- 172.21.15.27:0/3415539000 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad64191a20 con 0x55ad6416d980 2024-05-10T11:28:05.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.904+0000 7f0e9e800700 1 --2- 172.21.15.27:0/3415539000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad641719d0 0x55ad64172c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:28:05.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.906+0000 7f0e9d400700 1 --2- 172.21.15.27:0/3415539000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 0x55ad64169500 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0e90006010 tx=0x7f0e90009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b70492833e4e6d6f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:05.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.907+0000 7f0e8aa00700 1 -- 172.21.15.27:0/3415539000 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e90029670 con 0x55ad6416d980 2024-05-10T11:28:05.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.907+0000 7f0e8aa00700 1 -- 172.21.15.27:0/3415539000 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e90024070 con 0x55ad6416d980 2024-05-10T11:28:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.907+0000 7f0e8aa00700 1 -- 172.21.15.27:0/3415539000 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e90020960 con 0x55ad6416d980 2024-05-10T11:28:05.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.910+0000 7f0eafc25b80 1 -- 172.21.15.27:0/3415539000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 msgr2=0x55ad64169500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:05.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.910+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/3415539000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 0x55ad64169500 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0e90006010 tx=0x7f0e90009580 comp rx=0 tx=0).stop 2024-05-10T11:28:05.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.910+0000 7f0eafc25b80 1 -- 172.21.15.27:0/3415539000 shutdown_connections 2024-05-10T11:28:05.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.910+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/3415539000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad641719d0 0x55ad64172c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.910+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/3415539000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d060 0x55ad6418b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.910+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/3415539000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad6416d980 0x55ad64169500 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.910+0000 7f0eafc25b80 1 -- 172.21.15.27:0/3415539000 >> 172.21.15.27:0/3415539000 conn(0x55ad6409bb60 msgr2=0x55ad64172520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:05.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.910+0000 7f0eafc25b80 1 -- 172.21.15.27:0/3415539000 shutdown_connections 2024-05-10T11:28:05.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.910+0000 7f0eafc25b80 1 -- 172.21.15.27:0/3415539000 wait complete. 2024-05-10T11:28:05.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.910+0000 7f0eafc25b80 1 -- start start 2024-05-10T11:28:05.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.911+0000 7f0eafc25b80 1 Processor -- start 2024-05-10T11:28:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.912+0000 7f0eafc25b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d060 0x55ad64183180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.912+0000 7f0eafc25b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad6416d980 0x55ad641836c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.912+0000 7f0eafc25b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad641719d0 0x55ad64183c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.913+0000 7f0e9d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad6416d980 0x55ad641836c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.913+0000 7f0e9de00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d060 0x55ad64183180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.913+0000 7f0e9de00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d060 0x55ad64183180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37272/0 (socket says 172.21.15.27:37272) 2024-05-10T11:28:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.913+0000 7f0e9de00700 1 -- 172.21.15.27:0/951514446 learned_addr learned my addr 172.21.15.27:0/951514446 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.913+0000 7f0e9e800700 1 --2- 172.21.15.27:0/951514446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad641719d0 0x55ad64183c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.913+0000 7f0eafc25b80 1 -- 172.21.15.27:0/951514446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0e800103e0 con 0x55ad6416d980 2024-05-10T11:28:05.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.913+0000 7f0eafc25b80 1 -- 172.21.15.27:0/951514446 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8000fd10 con 0x55ad6416d060 2024-05-10T11:28:05.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.913+0000 7f0eafc25b80 1 -- 172.21.15.27:0/951514446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8000cf00 con 0x55ad641719d0 2024-05-10T11:28:05.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.913+0000 7f0eafc25b80 10 client.0 _ll_register_callbacks cb 0x55ad64066c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:05.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.915+0000 7f0e9e800700 1 -- 172.21.15.27:0/951514446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d060 msgr2=0x55ad64183180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:05.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.915+0000 7f0e9e800700 1 --2- 172.21.15.27:0/951514446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d060 0x55ad64183180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.915+0000 7f0e9e800700 1 -- 172.21.15.27:0/951514446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad6416d980 msgr2=0x55ad641836c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:05.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.915+0000 7f0e9e800700 1 --2- 172.21.15.27:0/951514446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad6416d980 0x55ad641836c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.915+0000 7f0e9e800700 1 -- 172.21.15.27:0/951514446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad641f8890 con 0x55ad641719d0 2024-05-10T11:28:05.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.917+0000 7f0e7fe00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:28:05.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.917+0000 7f0e9e800700 1 --2- 172.21.15.27:0/951514446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad641719d0 0x55ad64183c00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x55ad641703d0 tx=0x7f0e8c011540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8789f32cc55266ef server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:05.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.918+0000 7f0e7fe00700 1 -- 172.21.15.27:0/951514446 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c011d60 con 0x55ad641719d0 2024-05-10T11:28:05.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.919+0000 7f0e7fe00700 1 -- 172.21.15.27:0/951514446 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9591+0+0 (secure 0 0 0) 0x7f0e8c015440 con 0x55ad641719d0 2024-05-10T11:28:05.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.919+0000 7f0eafc25b80 1 -- 172.21.15.27:0/951514446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e90024070 con 0x55ad641719d0 2024-05-10T11:28:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.920+0000 7f0e7fe00700 1 -- 172.21.15.27:0/951514446 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f0e8c015640 con 0x55ad641719d0 2024-05-10T11:28:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.920+0000 7f0eafc25b80 10 client.27298 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.920+0000 7f0eafc25b80 1 -- 172.21.15.27:0/951514446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ad641f9bd0 con 0x55ad641719d0 2024-05-10T11:28:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.921+0000 7f0e7fe00700 1 -- 172.21.15.27:0/951514446 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e90024070 con 0x55ad641719d0 2024-05-10T11:28:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.921+0000 7f0e7fe00700 1 -- 172.21.15.27:0/951514446 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f0e8c01f7f0 con 0x55ad641719d0 2024-05-10T11:28:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.922+0000 7f0eafc25b80 10 client.27298 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.922+0000 7f0eafc25b80 10 client.27298 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:28:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.922+0000 7f0eafc25b80 20 client.27298 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.922+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/951514446 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ad641fa2b0 0x55ad641fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.922+0000 7f0eafc25b80 4 client.27298 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:28:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.922+0000 7f0eafc25b80 1 -- 172.21.15.27:0/951514446 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad640cd400 con 0x55ad641fa2b0 2024-05-10T11:28:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.922+0000 7f0e9de00700 1 --2- 172.21.15.27:0/951514446 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ad641fa2b0 0x55ad641fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:05.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.923+0000 7f0e7fe00700 10 client.27298 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:28:05.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.923+0000 7f0e9de00700 1 --2- 172.21.15.27:0/951514446 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ad641fa2b0 0x55ad641fc6d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bcdf68f5eb4d0809 server_cookie=ae64ef4d0fa45355 in_seq=0 out_seq=0 2024-05-10T11:28:05.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.924+0000 7f0e7fe00700 1 -- 172.21.15.27:0/951514446 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad640cd400 con 0x55ad641fa2b0 2024-05-10T11:28:05.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.924+0000 7f0e7fe00700 10 client.27298 handle_command_reply: tid=0 2024-05-10T11:28:05.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.924+0000 7f0eafc25b80 1 client.27298 shutdown 2024-05-10T11:28:05.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.925+0000 7f0eafc25b80 1 -- 172.21.15.27:0/951514446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad641719d0 msgr2=0x55ad64183c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:05.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.925+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/951514446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad641719d0 0x55ad64183c00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x55ad641703d0 tx=0x7f0e8c011540 comp rx=0 tx=0).stop 2024-05-10T11:28:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.925+0000 7f0eafc25b80 1 -- 172.21.15.27:0/951514446 shutdown_connections 2024-05-10T11:28:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.925+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/951514446 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55ad641fa2b0 0x55ad641fc6d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.925+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/951514446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad641719d0 0x55ad64183c00 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.925+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/951514446 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6416d060 0x55ad64183180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.925+0000 7f0eafc25b80 1 --2- 172.21.15.27:0/951514446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad6416d980 0x55ad641836c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.925+0000 7f0eafc25b80 1 -- 172.21.15.27:0/951514446 >> 172.21.15.27:0/951514446 conn(0x55ad6409bb60 msgr2=0x55ad641622b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.925+0000 7f0e7fe00700 0 client.27298 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:28:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.926+0000 7f0eafc25b80 1 -- 172.21.15.27:0/951514446 shutdown_connections 2024-05-10T11:28:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.926+0000 7f0eafc25b80 1 -- 172.21.15.27:0/951514446 wait complete. 2024-05-10T11:28:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.926+0000 7f0eafc25b80 20 client.27298 trim_cache size 0 max 16384 2024-05-10T11:28:05.927 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:28:05.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.928+0000 7f0ea5200700 1 -- 172.21.15.27:0/3552304911 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0e840b2df0 msgr2=0x7f0e840b52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:05.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.928+0000 7f0ea5200700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0e840b2df0 0x7f0e840b52b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0070350 tx=0x7f0e90009260 comp rx=0 tx=0).stop 2024-05-10T11:28:05.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.928+0000 7f0ea5200700 1 -- 172.21.15.27:0/3552304911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01056e0 msgr2=0x7f0ea0074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:05.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.928+0000 7f0ea5200700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01056e0 0x7f0ea0074b10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800b070 tx=0x7f0e98009c30 comp rx=0 tx=0).stop 2024-05-10T11:28:05.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.928+0000 7f0ea5200700 1 -- 172.21.15.27:0/3552304911 shutdown_connections 2024-05-10T11:28:05.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.928+0000 7f0ea5200700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f0e840b2df0 0x7f0e840b52b0 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0070350 tx=0x7f0e90009260 comp rx=0 tx=0).stop 2024-05-10T11:28:05.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.928+0000 7f0ea5200700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01056e0 0x7f0ea0074b10 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.928+0000 7f0ea5200700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea010b6e0 0x7f0ea006ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.928+0000 7f0ea5200700 1 --2- 172.21.15.27:0/3552304911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ea01060a0 0x7f0ea0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:05.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.928+0000 7f0ea5200700 1 -- 172.21.15.27:0/3552304911 >> 172.21.15.27:0/3552304911 conn(0x7f0ea01011d0 msgr2=0x7f0ea01015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:05.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.928+0000 7f0ea5200700 1 -- 172.21.15.27:0/3552304911 shutdown_connections 2024-05-10T11:28:05.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:05.929+0000 7f0ea5200700 1 -- 172.21.15.27:0/3552304911 wait complete. 2024-05-10T11:28:06.143 INFO:tasks.workunit.client.0.smithi027.stdout: 10 23453 28.58 MB/sec warmup 50 sec latency 534.418 ms 2024-05-10T11:28:06.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:06 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:28:06.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:06 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:28:06.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:06 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:28:06.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:06 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:28:06.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:06 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:28:06.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:06 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:28:06.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:06 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:28:06.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:06 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:28:06.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:06 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:28:06.901 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:28:07.143 INFO:tasks.workunit.client.0.smithi027.stdout: 10 23529 28.14 MB/sec warmup 51 sec latency 564.733 ms 2024-05-10T11:28:07.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:07 smithi076 ceph-mon[34152]: pgmap v465: 129 pgs: 129 active+clean; 887 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 4.4 MiB/s wr, 414 op/s 2024-05-10T11:28:07.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:07 smithi189 ceph-mon[34842]: pgmap v465: 129 pgs: 129 active+clean; 887 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 4.4 MiB/s wr, 414 op/s 2024-05-10T11:28:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:28:07.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:07 smithi027 ceph-mon[30054]: pgmap v465: 129 pgs: 129 active+clean; 887 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 4.4 MiB/s wr, 414 op/s 2024-05-10T11:28:08.144 INFO:tasks.workunit.client.0.smithi027.stdout: 10 23808 27.92 MB/sec warmup 52 sec latency 471.542 ms 2024-05-10T11:28:08.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:08 smithi027 ceph-mon[30054]: osdmap e108: 12 total, 12 up, 12 in 2024-05-10T11:28:08.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:08 smithi076 ceph-mon[34152]: osdmap e108: 12 total, 12 up, 12 in 2024-05-10T11:28:08.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:08 smithi189 ceph-mon[34842]: osdmap e108: 12 total, 12 up, 12 in 2024-05-10T11:28:09.144 INFO:tasks.workunit.client.0.smithi027.stdout: 10 24211 27.96 MB/sec warmup 53 sec latency 235.233 ms 2024-05-10T11:28:09.318 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:09 smithi027 ceph-mon[30054]: pgmap v467: 129 pgs: 129 active+clean; 887 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 293 op/s 2024-05-10T11:28:09.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:09 smithi189 ceph-mon[34842]: pgmap v467: 129 pgs: 129 active+clean; 887 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 293 op/s 2024-05-10T11:28:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:09 smithi076 ceph-mon[34152]: pgmap v467: 129 pgs: 129 active+clean; 887 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 293 op/s 2024-05-10T11:28:10.144 INFO:tasks.workunit.client.0.smithi027.stdout: 10 24624 27.88 MB/sec warmup 54 sec latency 209.641 ms 2024-05-10T11:28:10.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.422+0000 7f2ae7a00700 1 -- 172.21.15.27:0/3453352111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae006ba10 msgr2=0x7f2ae006be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.422+0000 7f2ae7a00700 1 --2- 172.21.15.27:0/3453352111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae006ba10 0x7f2ae006be90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0098f0 tx=0x7f2acc009c00 comp rx=0 tx=0).stop 2024-05-10T11:28:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.423+0000 7f2ae7a00700 1 -- 172.21.15.27:0/3453352111 shutdown_connections 2024-05-10T11:28:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.423+0000 7f2ae7a00700 1 --2- 172.21.15.27:0/3453352111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae006c3d0 0x7f2ae0110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.423+0000 7f2ae7a00700 1 --2- 172.21.15.27:0/3453352111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ae0109500 0x7f2ae01098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.423+0000 7f2ae7a00700 1 --2- 172.21.15.27:0/3453352111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae006ba10 0x7f2ae006be90 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.423+0000 7f2ae7a00700 1 -- 172.21.15.27:0/3453352111 >> 172.21.15.27:0/3453352111 conn(0x7f2ae0079310 msgr2=0x7f2ae0079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.423+0000 7f2ae7a00700 1 -- 172.21.15.27:0/3453352111 shutdown_connections 2024-05-10T11:28:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.423+0000 7f2ae7a00700 1 -- 172.21.15.27:0/3453352111 wait complete. 2024-05-10T11:28:10.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.425+0000 7f2ae7a00700 1 Processor -- start 2024-05-10T11:28:10.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.425+0000 7f2ae7a00700 1 -- start start 2024-05-10T11:28:10.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.426+0000 7f2ae7a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae006ba10 0x7f2ae010a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.426+0000 7f2ae7a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae006c3d0 0x7f2ae010a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.426+0000 7f2ae7a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ae0109500 0x7f2ae010adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.426+0000 7f2ae7a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae0113630 con 0x7f2ae006ba10 2024-05-10T11:28:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.426+0000 7f2ae7a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae01134b0 con 0x7f2ae0109500 2024-05-10T11:28:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.426+0000 7f2ae7a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae01137b0 con 0x7f2ae006c3d0 2024-05-10T11:28:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.426+0000 7f2adfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae006c3d0 0x7f2ae010a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.426+0000 7f2adfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae006c3d0 0x7f2ae010a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54756/0 (socket says 172.21.15.27:54756) 2024-05-10T11:28:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.427+0000 7f2ae5a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ae0109500 0x7f2ae010adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.427+0000 7f2adfe00700 1 -- 172.21.15.27:0/420511443 learned_addr learned my addr 172.21.15.27:0/420511443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.427+0000 7f2adfe00700 1 -- 172.21.15.27:0/420511443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ae0109500 msgr2=0x7f2ae010adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.427+0000 7f2adfe00700 1 --2- 172.21.15.27:0/420511443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ae0109500 0x7f2ae010adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.427+0000 7f2adfe00700 1 -- 172.21.15.27:0/420511443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae006ba10 msgr2=0x7f2ae010a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:10.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.427+0000 7f2adfe00700 1 --2- 172.21.15.27:0/420511443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae006ba10 0x7f2ae010a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.427+0000 7f2adfe00700 1 -- 172.21.15.27:0/420511443 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae01ac4f0 con 0x7f2ae006c3d0 2024-05-10T11:28:10.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.427+0000 7f2adfe00700 1 --2- 172.21.15.27:0/420511443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae006c3d0 0x7f2ae010a870 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00fac0 tx=0x7f2acc00faf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:10.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.428+0000 7f2add600700 1 -- 172.21.15.27:0/420511443 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0054b0 con 0x7f2ae006c3d0 2024-05-10T11:28:10.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.428+0000 7f2add600700 1 -- 172.21.15.27:0/420511443 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2acc005650 con 0x7f2ae006c3d0 2024-05-10T11:28:10.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.428+0000 7f2ae7a00700 1 -- 172.21.15.27:0/420511443 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae01ac780 con 0x7f2ae006c3d0 2024-05-10T11:28:10.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.428+0000 7f2add600700 1 -- 172.21.15.27:0/420511443 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc004dc0 con 0x7f2ae006c3d0 2024-05-10T11:28:10.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.428+0000 7f2ae7a00700 1 -- 172.21.15.27:0/420511443 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae01acbd0 con 0x7f2ae006c3d0 2024-05-10T11:28:10.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.434+0000 7f2af246ab80 1 Processor -- start 2024-05-10T11:28:10.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.434+0000 7f2add600700 1 -- 172.21.15.27:0/420511443 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2acc00b560 con 0x7f2ae006c3d0 2024-05-10T11:28:10.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.434+0000 7f2af246ab80 1 -- start start 2024-05-10T11:28:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.434+0000 7f2af246ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 0x56165ac32440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.434+0000 7f2af246ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 0x56165ac36490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.435+0000 7f2af246ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac36e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.435+0000 7f2af246ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56165ab84a20 con 0x56165ac32980 2024-05-10T11:28:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.435+0000 7f2af246ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56165ab22760 con 0x56165ac369d0 2024-05-10T11:28:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.435+0000 7f2af246ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56165ab809e0 con 0x56165ac32060 2024-05-10T11:28:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.435+0000 7f2ae5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 0x56165ac32440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.435+0000 7f2ae5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 0x56165ac32440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54758/0 (socket says 172.21.15.27:54758) 2024-05-10T11:28:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.435+0000 7f2ae5a00700 1 -- 172.21.15.27:0/3696239280 learned_addr learned my addr 172.21.15.27:0/3696239280 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.435+0000 7f2ae5a00700 1 -- 172.21.15.27:0/3696239280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 msgr2=0x56165ac36e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.435+0000 7f2adfe00700 1 --2- 172.21.15.27:0/3696239280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac36e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.436+0000 7f2ae5a00700 1 --2- 172.21.15.27:0/3696239280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac36e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.436+0000 7f2ae5a00700 1 -- 172.21.15.27:0/3696239280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 msgr2=0x56165ac36490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.436+0000 7f2ae5a00700 1 --2- 172.21.15.27:0/3696239280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 0x56165ac36490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.436+0000 7f2ae5a00700 1 -- 172.21.15.27:0/3696239280 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56165aad18d0 con 0x56165ac32060 2024-05-10T11:28:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.436+0000 7f2adfe00700 1 --2- 172.21.15.27:0/3696239280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac36e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:28:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.436+0000 7f2ae5a00700 1 --2- 172.21.15.27:0/3696239280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 0x56165ac32440 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8011670 tx=0x7f2ad80099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69700a4c4fa9925d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.437+0000 7f2ad2000700 1 -- 172.21.15.27:0/3696239280 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8019070 con 0x56165ac32060 2024-05-10T11:28:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.437+0000 7f2add600700 1 --2- 172.21.15.27:0/420511443 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2ac80b2b40 0x7f2ac80b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.437+0000 7f2add600700 1 -- 172.21.15.27:0/420511443 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9679+0+0 (secure 0 0 0) 0x7f2acc0e3b90 con 0x7f2ae006c3d0 2024-05-10T11:28:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.437+0000 7f2ae5000700 1 --2- 172.21.15.27:0/420511443 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2ac80b2b40 0x7f2ac80b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.437+0000 7f2ad2000700 1 -- 172.21.15.27:0/3696239280 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ad8004020 con 0x56165ac32060 2024-05-10T11:28:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.437+0000 7f2ad2000700 1 -- 172.21.15.27:0/3696239280 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8007820 con 0x56165ac32060 2024-05-10T11:28:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.438+0000 7f2af246ab80 1 -- 172.21.15.27:0/3696239280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 msgr2=0x56165ac32440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.438+0000 7f2ae5000700 1 --2- 172.21.15.27:0/420511443 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2ac80b2b40 0x7f2ac80b5000 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400b9e0 tx=0x7f2ad4005f50 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.438+0000 7f2af246ab80 1 --2- 172.21.15.27:0/3696239280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 0x56165ac32440 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8011670 tx=0x7f2ad80099e0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.439+0000 7f2af246ab80 1 -- 172.21.15.27:0/3696239280 shutdown_connections 2024-05-10T11:28:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.439+0000 7f2af246ab80 1 --2- 172.21.15.27:0/3696239280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 0x56165ac32440 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.439+0000 7f2af246ab80 1 --2- 172.21.15.27:0/3696239280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac36e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.439+0000 7f2af246ab80 1 --2- 172.21.15.27:0/3696239280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 0x56165ac36490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.439+0000 7f2af246ab80 1 -- 172.21.15.27:0/3696239280 >> 172.21.15.27:0/3696239280 conn(0x56165ab60b60 msgr2=0x56165ac2c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.439+0000 7f2af246ab80 1 -- 172.21.15.27:0/3696239280 shutdown_connections 2024-05-10T11:28:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.439+0000 7f2af246ab80 1 -- 172.21.15.27:0/3696239280 wait complete. 2024-05-10T11:28:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.440+0000 7f2af246ab80 1 -- start start 2024-05-10T11:28:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.440+0000 7f2af246ab80 1 Processor -- start 2024-05-10T11:28:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.441+0000 7f2af246ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac32060 0x56165ac3deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.441+0000 7f2af246ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 0x56165ac3b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.441+0000 7f2af246ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac369d0 0x56165ac3b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.441+0000 7f2ae5000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac32060 0x56165ac3deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.442+0000 7f2ae5000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac32060 0x56165ac3deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37300/0 (socket says 172.21.15.27:37300) 2024-05-10T11:28:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.442+0000 7f2adfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 0x56165ac3b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.442+0000 7f2adfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 0x56165ac3b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36638/0 (socket says 172.21.15.27:36638) 2024-05-10T11:28:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.442+0000 7f2ae5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac369d0 0x56165ac3b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.442+0000 7f2ae5000700 1 -- 172.21.15.27:0/1685990230 learned_addr learned my addr 172.21.15.27:0/1685990230 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.442+0000 7f2af246ab80 1 -- 172.21.15.27:0/1685990230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56165ac3ebe0 con 0x56165ac32980 2024-05-10T11:28:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.442+0000 7f2af246ab80 1 -- 172.21.15.27:0/1685990230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56165aaeab20 con 0x56165ac32060 2024-05-10T11:28:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.442+0000 7f2af246ab80 1 -- 172.21.15.27:0/1685990230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56165ac3ed60 con 0x56165ac369d0 2024-05-10T11:28:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.442+0000 7f2af246ab80 10 client.0 _ll_register_callbacks cb 0x56165ab2bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.443+0000 7f2ae5000700 1 -- 172.21.15.27:0/1685990230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac369d0 msgr2=0x56165ac3b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:10.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.443+0000 7f2ae5000700 1 --2- 172.21.15.27:0/1685990230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac369d0 0x56165ac3b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.443+0000 7f2ae5000700 1 -- 172.21.15.27:0/1685990230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 msgr2=0x56165ac3b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:10.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.443+0000 7f2ae5000700 1 --2- 172.21.15.27:0/1685990230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 0x56165ac3b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.443+0000 7f2ae5000700 1 -- 172.21.15.27:0/1685990230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56165ab4a420 con 0x56165ac32060 2024-05-10T11:28:10.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.443+0000 7f2ae5a00700 1 --2- 172.21.15.27:0/1685990230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac369d0 0x56165ac3b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:28:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.444+0000 7f2ae5000700 1 --2- 172.21.15.27:0/1685990230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac32060 0x56165ac3deb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400dcf0 tx=0x7f2ad40058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62e916f6de227474 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.444+0000 7f2ac6a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:28:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.444+0000 7f2ac6a00700 1 -- 172.21.15.27:0/1685990230 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4021020 con 0x56165ac32060 2024-05-10T11:28:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.444+0000 7f2ac6a00700 1 -- 172.21.15.27:0/1685990230 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9679+0+0 (secure 0 0 0) 0x7f2ad4010b30 con 0x56165ac32060 2024-05-10T11:28:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.444+0000 7f2af246ab80 1 -- 172.21.15.27:0/1685990230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ad8004020 con 0x56165ac32060 2024-05-10T11:28:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.445+0000 7f2ac6a00700 1 -- 172.21.15.27:0/1685990230 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f2ad402c050 con 0x56165ac32060 2024-05-10T11:28:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.445+0000 7f2af246ab80 10 client.27320 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.445+0000 7f2af246ab80 1 -- 172.21.15.27:0/1685990230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56165ac494b0 con 0x56165ac32060 2024-05-10T11:28:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.445+0000 7f2ac6a00700 1 -- 172.21.15.27:0/1685990230 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ad8004020 con 0x56165ac32060 2024-05-10T11:28:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.445+0000 7f2ac6a00700 1 -- 172.21.15.27:0/1685990230 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f2ad400f3c0 con 0x56165ac32060 2024-05-10T11:28:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.445+0000 7f2af246ab80 10 client.27320 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.445+0000 7f2af246ab80 10 client.27320 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:28:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.445+0000 7f2af246ab80 20 client.27320 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.445+0000 7f2af246ab80 1 --2- 172.21.15.27:0/1685990230 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56165ac4bb40 0x56165ac4df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.445+0000 7f2af246ab80 4 client.27320 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:28:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.445+0000 7f2af246ab80 1 -- 172.21.15.27:0/1685990230 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56165abff0a0 con 0x56165ac4bb40 2024-05-10T11:28:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.446+0000 7f2adfe00700 1 --2- 172.21.15.27:0/1685990230 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56165ac4bb40 0x56165ac4df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.446+0000 7f2ac6a00700 10 client.27320 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:28:10.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.447+0000 7f2adfe00700 1 --2- 172.21.15.27:0/1685990230 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56165ac4bb40 0x56165ac4df90 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=506362a19834803d server_cookie=9dc01e5c69148998 in_seq=0 out_seq=0 2024-05-10T11:28:10.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.448+0000 7f2ac6a00700 1 -- 172.21.15.27:0/1685990230 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56165abff0a0 con 0x56165ac4bb40 2024-05-10T11:28:10.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.448+0000 7f2ac6a00700 10 client.27320 handle_command_reply: tid=0 2024-05-10T11:28:10.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.448+0000 7f2af246ab80 1 client.27320 shutdown 2024-05-10T11:28:10.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.448+0000 7f2af246ab80 1 -- 172.21.15.27:0/1685990230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac32060 msgr2=0x56165ac3deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:10.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.448+0000 7f2af246ab80 1 --2- 172.21.15.27:0/1685990230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac32060 0x56165ac3deb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400dcf0 tx=0x7f2ad40058e0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.448+0000 7f2af246ab80 1 -- 172.21.15.27:0/1685990230 shutdown_connections 2024-05-10T11:28:10.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.448+0000 7f2af246ab80 1 --2- 172.21.15.27:0/1685990230 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56165ac4bb40 0x56165ac4df90 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.448+0000 7f2ac6a00700 0 client.27320 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:28:10.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.448+0000 7f2af246ab80 1 --2- 172.21.15.27:0/1685990230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac369d0 0x56165ac3b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.448+0000 7f2af246ab80 1 --2- 172.21.15.27:0/1685990230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac32060 0x56165ac3deb0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.449+0000 7f2af246ab80 1 --2- 172.21.15.27:0/1685990230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 0x56165ac3b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.449+0000 7f2af246ab80 1 -- 172.21.15.27:0/1685990230 >> 172.21.15.27:0/1685990230 conn(0x56165ab60b60 msgr2=0x56165ac272b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:10.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.449+0000 7f2af246ab80 1 -- 172.21.15.27:0/1685990230 shutdown_connections 2024-05-10T11:28:10.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.449+0000 7f2af246ab80 1 -- 172.21.15.27:0/1685990230 wait complete. 2024-05-10T11:28:10.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.449+0000 7f2af246ab80 20 client.27320 trim_cache size 0 max 16384 2024-05-10T11:28:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2af246ab80 1 Processor -- start 2024-05-10T11:28:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2af246ab80 1 -- start start 2024-05-10T11:28:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2af246ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32060 0x56165ac47c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2af246ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32980 0x56165ac37dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2af246ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac27130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2af246ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae01b5d80 con 0x56165ac32060 2024-05-10T11:28:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2af246ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae01b79b0 con 0x56165ac369d0 2024-05-10T11:28:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2af246ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae01b6450 con 0x56165ac32980 2024-05-10T11:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2ae5000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32060 0x56165ac47c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2ae5a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac27130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2ae5000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32060 0x56165ac47c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36654/0 (socket says 172.21.15.27:36654) 2024-05-10T11:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2adfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32980 0x56165ac37dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.469+0000 7f2ae5a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac27130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37302/0 (socket says 172.21.15.27:37302) 2024-05-10T11:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.470+0000 7f2adfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32980 0x56165ac37dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54782/0 (socket says 172.21.15.27:54782) 2024-05-10T11:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.470+0000 7f2ae5000700 1 -- 172.21.15.27:0/3629273593 learned_addr learned my addr 172.21.15.27:0/3629273593 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.470+0000 7f2ae5a00700 1 -- 172.21.15.27:0/3629273593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32980 msgr2=0x56165ac37dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.470+0000 7f2ae5a00700 1 --2- 172.21.15.27:0/3629273593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32980 0x56165ac37dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.470+0000 7f2ae5a00700 1 -- 172.21.15.27:0/3629273593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32060 msgr2=0x56165ac47c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.470+0000 7f2ae5a00700 1 --2- 172.21.15.27:0/3629273593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32060 0x56165ac47c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.470+0000 7f2ae5a00700 1 -- 172.21.15.27:0/3629273593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56165ac492e0 con 0x56165ac369d0 2024-05-10T11:28:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.471+0000 7f2ae5a00700 1 --2- 172.21.15.27:0/3629273593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac27130 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8011d10 tx=0x7f2ad8004f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2533b7b2382ed41 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.471+0000 7f2ad1600700 1 -- 172.21.15.27:0/3629273593 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8019040 con 0x56165ac369d0 2024-05-10T11:28:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.471+0000 7f2ad1600700 1 -- 172.21.15.27:0/3629273593 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ad8009460 con 0x56165ac369d0 2024-05-10T11:28:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.471+0000 7f2ad1600700 1 -- 172.21.15.27:0/3629273593 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8015a80 con 0x56165ac369d0 2024-05-10T11:28:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.472+0000 7f2af246ab80 1 -- 172.21.15.27:0/3629273593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 msgr2=0x56165ac27130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.472+0000 7f2af246ab80 1 --2- 172.21.15.27:0/3629273593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac27130 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8011d10 tx=0x7f2ad8004f90 comp rx=0 tx=0).stop 2024-05-10T11:28:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.472+0000 7f2af246ab80 1 -- 172.21.15.27:0/3629273593 shutdown_connections 2024-05-10T11:28:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.472+0000 7f2af246ab80 1 --2- 172.21.15.27:0/3629273593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac27130 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.472+0000 7f2af246ab80 1 --2- 172.21.15.27:0/3629273593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32060 0x56165ac47c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.472+0000 7f2af246ab80 1 --2- 172.21.15.27:0/3629273593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32980 0x56165ac37dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.472+0000 7f2af246ab80 1 -- 172.21.15.27:0/3629273593 >> 172.21.15.27:0/3629273593 conn(0x56165ab60b60 msgr2=0x56165ac26a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.472+0000 7f2af246ab80 1 -- 172.21.15.27:0/3629273593 shutdown_connections 2024-05-10T11:28:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.472+0000 7f2af246ab80 1 -- 172.21.15.27:0/3629273593 wait complete. 2024-05-10T11:28:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.473+0000 7f2af246ab80 1 -- start start 2024-05-10T11:28:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.473+0000 7f2af246ab80 1 Processor -- start 2024-05-10T11:28:10.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.474+0000 7f2af246ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 0x56165ac44240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.474+0000 7f2af246ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 0x56165ac52a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.474+0000 7f2af246ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac52f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.474+0000 7f2af246ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae01b7220 con 0x56165ac32980 2024-05-10T11:28:10.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.474+0000 7f2af246ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae01b6b50 con 0x56165ac369d0 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.474+0000 7f2af246ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae01b3d40 con 0x56165ac32060 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.474+0000 7f2af246ab80 10 client.0 _ll_register_callbacks cb 0x56165ab2bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.474+0000 7f2ae5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 0x56165ac44240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.474+0000 7f2ae5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 0x56165ac44240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54796/0 (socket says 172.21.15.27:54796) 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.474+0000 7f2ae5000700 1 -- 172.21.15.27:0/560562701 learned_addr learned my addr 172.21.15.27:0/560562701 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.474+0000 7f2adfe00700 1 --2- 172.21.15.27:0/560562701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 0x56165ac52a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.475+0000 7f2ae5000700 1 -- 172.21.15.27:0/560562701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 msgr2=0x56165ac52f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.475+0000 7f2ae5000700 1 --2- 172.21.15.27:0/560562701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac52f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.475+0000 7f2ae5000700 1 -- 172.21.15.27:0/560562701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 msgr2=0x56165ac52a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.475+0000 7f2ae5000700 1 --2- 172.21.15.27:0/560562701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 0x56165ac52a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.475+0000 7f2ae5000700 1 -- 172.21.15.27:0/560562701 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56165ac534d0 con 0x56165ac32060 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.475+0000 7f2ae5000700 1 --2- 172.21.15.27:0/560562701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 0x56165ac44240 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400dcf0 tx=0x7f2ad400f720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5669f90cb51503ff server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.475+0000 7f2ac6a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.475+0000 7f2ac6a00700 1 -- 172.21.15.27:0/560562701 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4021040 con 0x56165ac32060 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.475+0000 7f2ac6a00700 1 -- 172.21.15.27:0/560562701 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9679+0+0 (secure 0 0 0) 0x7f2ad400fd50 con 0x56165ac32060 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.475+0000 7f2af246ab80 1 -- 172.21.15.27:0/560562701 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ad8009460 con 0x56165ac32060 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.476+0000 7f2ac6a00700 1 -- 172.21.15.27:0/560562701 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f2ad4043050 con 0x56165ac32060 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.476+0000 7f2af246ab80 10 client.27316 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.476+0000 7f2af246ab80 1 -- 172.21.15.27:0/560562701 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56165acb6100 con 0x56165ac32060 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.476+0000 7f2ac6a00700 1 -- 172.21.15.27:0/560562701 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ad8009460 con 0x56165ac32060 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.476+0000 7f2ac6a00700 1 -- 172.21.15.27:0/560562701 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f2ad401da10 con 0x56165ac32060 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.477+0000 7f2af246ab80 10 client.27316 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.477+0000 7f2af246ab80 10 client.27316 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.477+0000 7f2af246ab80 20 client.27316 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.477+0000 7f2af246ab80 1 --2- 172.21.15.27:0/560562701 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56165ac4bb40 0x56165acb8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.477+0000 7f2af246ab80 4 client.27316 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:28:10.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.477+0000 7f2af246ab80 1 -- 172.21.15.27:0/560562701 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56165ab92400 con 0x56165ac4bb40 2024-05-10T11:28:10.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.477+0000 7f2adfe00700 1 --2- 172.21.15.27:0/560562701 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56165ac4bb40 0x56165acb8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:10.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2adfe00700 1 --2- 172.21.15.27:0/560562701 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56165ac4bb40 0x56165acb8800 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7313565c28c9c8e0 server_cookie=3a14b3d8620484f5 in_seq=0 out_seq=0 2024-05-10T11:28:10.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2ac6a00700 10 client.27316 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:28:10.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2ac6a00700 1 -- 172.21.15.27:0/560562701 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56165ab92400 con 0x56165ac4bb40 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2ac6a00700 10 client.27316 handle_command_reply: tid=0 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2af246ab80 1 client.27316 shutdown 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2af246ab80 1 -- 172.21.15.27:0/560562701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 msgr2=0x56165ac44240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2af246ab80 1 --2- 172.21.15.27:0/560562701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 0x56165ac44240 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400dcf0 tx=0x7f2ad400f720 comp rx=0 tx=0).stop 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2af246ab80 1 -- 172.21.15.27:0/560562701 shutdown_connections 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2af246ab80 1 --2- 172.21.15.27:0/560562701 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56165ac4bb40 0x56165acb8800 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2af246ab80 1 --2- 172.21.15.27:0/560562701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56165ac32060 0x56165ac44240 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2af246ab80 1 --2- 172.21.15.27:0/560562701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56165ac369d0 0x56165ac52f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2af246ab80 1 --2- 172.21.15.27:0/560562701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56165ac32980 0x56165ac52a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2ac6a00700 0 client.27316 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.478+0000 7f2af246ab80 1 -- 172.21.15.27:0/560562701 >> 172.21.15.27:0/560562701 conn(0x56165ab60b60 msgr2=0x56165ac272b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.479+0000 7f2af246ab80 1 -- 172.21.15.27:0/560562701 shutdown_connections 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.479+0000 7f2af246ab80 1 -- 172.21.15.27:0/560562701 wait complete. 2024-05-10T11:28:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.479+0000 7f2af246ab80 20 client.27316 trim_cache size 0 max 16384 2024-05-10T11:28:10.482 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:28:10.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.480+0000 7f2ae7a00700 1 -- 172.21.15.27:0/420511443 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2ac80b2b40 msgr2=0x7f2ac80b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:10.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.480+0000 7f2ae7a00700 1 --2- 172.21.15.27:0/420511443 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2ac80b2b40 0x7f2ac80b5000 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400b9e0 tx=0x7f2ad4005f50 comp rx=0 tx=0).stop 2024-05-10T11:28:10.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.480+0000 7f2ae7a00700 1 -- 172.21.15.27:0/420511443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae006c3d0 msgr2=0x7f2ae010a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:10.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.480+0000 7f2ae7a00700 1 --2- 172.21.15.27:0/420511443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae006c3d0 0x7f2ae010a870 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00fac0 tx=0x7f2acc00faf0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.480+0000 7f2ae7a00700 1 -- 172.21.15.27:0/420511443 shutdown_connections 2024-05-10T11:28:10.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.480+0000 7f2ae7a00700 1 --2- 172.21.15.27:0/420511443 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f2ac80b2b40 0x7f2ac80b5000 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.480+0000 7f2ae7a00700 1 --2- 172.21.15.27:0/420511443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ae0109500 0x7f2ae010adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.480+0000 7f2ae7a00700 1 --2- 172.21.15.27:0/420511443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae006ba10 0x7f2ae010a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.480+0000 7f2ae7a00700 1 --2- 172.21.15.27:0/420511443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae006c3d0 0x7f2ae010a870 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:10.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.480+0000 7f2ae7a00700 1 -- 172.21.15.27:0/420511443 >> 172.21.15.27:0/420511443 conn(0x7f2ae0079310 msgr2=0x7f2ae010e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.480+0000 7f2ae7a00700 1 -- 172.21.15.27:0/420511443 shutdown_connections 2024-05-10T11:28:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:10.480+0000 7f2ae7a00700 1 -- 172.21.15.27:0/420511443 wait complete. 2024-05-10T11:28:11.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 25094 27.69 MB/sec warmup 55 sec latency 213.727 ms 2024-05-10T11:28:11.456 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:11 smithi027 ceph-mon[30054]: pgmap v468: 129 pgs: 129 active+clean; 894 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 487 op/s 2024-05-10T11:28:11.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:11 smithi189 ceph-mon[34842]: pgmap v468: 129 pgs: 129 active+clean; 894 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 487 op/s 2024-05-10T11:28:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:11 smithi076 ceph-mon[34152]: pgmap v468: 129 pgs: 129 active+clean; 894 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 487 op/s 2024-05-10T11:28:12.144 INFO:tasks.workunit.client.0.smithi027.stdout: 10 25354 27.53 MB/sec warmup 56 sec latency 445.362 ms 2024-05-10T11:28:12.962 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:28:13.144 INFO:tasks.workunit.client.0.smithi027.stdout: 10 25762 27.41 MB/sec warmup 57 sec latency 193.013 ms 2024-05-10T11:28:13.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:13 smithi027 ceph-mon[30054]: pgmap v469: 129 pgs: 129 active+clean; 894 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 487 op/s 2024-05-10T11:28:13.443 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:28:13.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:13 smithi189 ceph-mon[34842]: pgmap v469: 129 pgs: 129 active+clean; 894 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 487 op/s 2024-05-10T11:28:13.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:13 smithi076 ceph-mon[34152]: pgmap v469: 129 pgs: 129 active+clean; 894 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 487 op/s 2024-05-10T11:28:14.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 26369 27.52 MB/sec warmup 58 sec latency 197.552 ms 2024-05-10T11:28:14.751 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:14 smithi027 ceph-mon[30054]: pgmap v470: 129 pgs: 129 active+clean; 894 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 487 op/s 2024-05-10T11:28:14.751 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:14 smithi027 ceph-mon[30054]: osdmap e109: 12 total, 12 up, 12 in 2024-05-10T11:28:15.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 26831 27.45 MB/sec warmup 59 sec latency 274.624 ms 2024-05-10T11:28:15.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:14 smithi076 ceph-mon[34152]: pgmap v470: 129 pgs: 129 active+clean; 894 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 487 op/s 2024-05-10T11:28:15.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:14 smithi076 ceph-mon[34152]: osdmap e109: 12 total, 12 up, 12 in 2024-05-10T11:28:15.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:14 smithi189 ceph-mon[34842]: pgmap v470: 129 pgs: 129 active+clean; 894 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.1 MiB/s wr, 487 op/s 2024-05-10T11:28:15.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:14 smithi189 ceph-mon[34842]: osdmap e109: 12 total, 12 up, 12 in 2024-05-10T11:28:16.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 27402 27.38 MB/sec warmup 60 sec latency 192.680 ms 2024-05-10T11:28:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.399+0000 7f16c3000700 1 -- 172.21.15.27:0/1221382908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc1165f0 msgr2=0x7f16bc1189e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.399+0000 7f16c3000700 1 --2- 172.21.15.27:0/1221382908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc1165f0 0x7f16bc1189e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00b0d0 tx=0x7f16ac00b3e0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.399+0000 7f16c3000700 1 -- 172.21.15.27:0/1221382908 shutdown_connections 2024-05-10T11:28:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.399+0000 7f16c3000700 1 --2- 172.21.15.27:0/1221382908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc1165f0 0x7f16bc1189e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.399+0000 7f16c3000700 1 --2- 172.21.15.27:0/1221382908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16bc10a4a0 0x7f16bc10a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.399+0000 7f16c3000700 1 --2- 172.21.15.27:0/1221382908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16bc10adc0 0x7f16bc1160b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.399+0000 7f16c3000700 1 -- 172.21.15.27:0/1221382908 >> 172.21.15.27:0/1221382908 conn(0x7f16bc06e9f0 msgr2=0x7f16bc06ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.403+0000 7f16c3000700 1 -- 172.21.15.27:0/1221382908 shutdown_connections 2024-05-10T11:28:16.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.404+0000 7f16c3000700 1 -- 172.21.15.27:0/1221382908 wait complete. 2024-05-10T11:28:16.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.405+0000 7f16c3000700 1 Processor -- start 2024-05-10T11:28:16.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.406+0000 7f16c3000700 1 -- start start 2024-05-10T11:28:16.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.406+0000 7f16c3000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16bc10a4a0 0x7f16bc1a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.406+0000 7f16c3000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc10adc0 0x7f16bc1a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.406+0000 7f16c3000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16bc1165f0 0x7f16bc1a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.406+0000 7f16c3000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc11e150 con 0x7f16bc1165f0 2024-05-10T11:28:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.406+0000 7f16c3000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc11dfd0 con 0x7f16bc10a4a0 2024-05-10T11:28:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.406+0000 7f16c3000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc11e2d0 con 0x7f16bc10adc0 2024-05-10T11:28:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.407+0000 7f16c1000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16bc1165f0 0x7f16bc1a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.407+0000 7f16bbe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16bc10a4a0 0x7f16bc1a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.407+0000 7f16c1000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16bc1165f0 0x7f16bc1a8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42624/0 (socket says 172.21.15.27:42624) 2024-05-10T11:28:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.407+0000 7f16c1000700 1 -- 172.21.15.27:0/2457520412 learned_addr learned my addr 172.21.15.27:0/2457520412 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.407+0000 7f16bb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc10adc0 0x7f16bc1a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.409+0000 7f16c1000700 1 -- 172.21.15.27:0/2457520412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc10adc0 msgr2=0x7f16bc1a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.409+0000 7f16c1000700 1 --2- 172.21.15.27:0/2457520412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc10adc0 0x7f16bc1a4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.409+0000 7f16c1000700 1 -- 172.21.15.27:0/2457520412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16bc10a4a0 msgr2=0x7f16bc1a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.409+0000 7f16c1000700 1 --2- 172.21.15.27:0/2457520412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16bc10a4a0 0x7f16bc1a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.409+0000 7f16c1000700 1 -- 172.21.15.27:0/2457520412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16bc1a95f0 con 0x7f16bc1165f0 2024-05-10T11:28:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.410+0000 7f16c1000700 1 --2- 172.21.15.27:0/2457520412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16bc1165f0 0x7f16bc1a8eb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00b820 tx=0x7f16ac0079a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.411+0000 7f16b8c00700 1 -- 172.21.15.27:0/2457520412 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac021070 con 0x7f16bc1165f0 2024-05-10T11:28:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.412+0000 7f16b8c00700 1 -- 172.21.15.27:0/2457520412 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16ac007e50 con 0x7f16bc1165f0 2024-05-10T11:28:16.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.412+0000 7f16b8c00700 1 -- 172.21.15.27:0/2457520412 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac01ba10 con 0x7f16bc1165f0 2024-05-10T11:28:16.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.414+0000 7f16c3000700 1 -- 172.21.15.27:0/2457520412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16bc1a9880 con 0x7f16bc1165f0 2024-05-10T11:28:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.414+0000 7f16c3000700 1 -- 172.21.15.27:0/2457520412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16bc119730 con 0x7f16bc1165f0 2024-05-10T11:28:16.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.427+0000 7f16cda82b80 1 Processor -- start 2024-05-10T11:28:16.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.430+0000 7f16b8c00700 1 -- 172.21.15.27:0/2457520412 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f16ac024070 con 0x7f16bc1165f0 2024-05-10T11:28:16.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.432+0000 7f16cda82b80 1 -- start start 2024-05-10T11:28:16.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.432+0000 7f16cda82b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c37013bc0 0x557c37013fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.432+0000 7f16cda82b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c370144e0 0x557c3700e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.432+0000 7f16cda82b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c3700ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.433+0000 7f16cda82b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557c36f60a20 con 0x557c37013bc0 2024-05-10T11:28:16.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.433+0000 7f16cda82b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557c36efe760 con 0x557c370144e0 2024-05-10T11:28:16.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.433+0000 7f16cda82b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557c36f5c9e0 con 0x557c3700e8f0 2024-05-10T11:28:16.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.433+0000 7f16bbe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c370144e0 0x557c3700e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.433+0000 7f16bbe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c370144e0 0x557c3700e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37848/0 (socket says 172.21.15.27:37848) 2024-05-10T11:28:16.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.433+0000 7f16bbe00700 1 -- 172.21.15.27:0/3607170937 learned_addr learned my addr 172.21.15.27:0/3607170937 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:16.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.435+0000 7f16b8c00700 1 --2- 172.21.15.27:0/2457520412 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f16a00b2b40 0x7f16a00b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.436+0000 7f16bbe00700 1 -- 172.21.15.27:0/3607170937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 msgr2=0x557c3700ed70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:28:16.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.436+0000 7f16bbe00700 1 --2- 172.21.15.27:0/3607170937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c3700ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.436+0000 7f16bbe00700 1 -- 172.21.15.27:0/3607170937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c37013bc0 msgr2=0x557c37013fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:16.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.436+0000 7f16bbe00700 1 --2- 172.21.15.27:0/3607170937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c37013bc0 0x557c37013fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.436+0000 7f16bbe00700 1 -- 172.21.15.27:0/3607170937 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c36ead8d0 con 0x557c370144e0 2024-05-10T11:28:16.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.436+0000 7f16bbe00700 1 --2- 172.21.15.27:0/3607170937 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c370144e0 0x557c3700e2d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f16b400a950 tx=0x7f16b400ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a623dca660497197 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:16.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.437+0000 7f16c1000700 1 --2- 172.21.15.27:0/2457520412 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f16a00b2b40 0x7f16a00b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.437+0000 7f16b1600700 1 -- 172.21.15.27:0/3607170937 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b4015070 con 0x557c370144e0 2024-05-10T11:28:16.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.437+0000 7f16b1600700 1 -- 172.21.15.27:0/3607170937 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16b40113f0 con 0x557c370144e0 2024-05-10T11:28:16.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.437+0000 7f16b1600700 1 -- 172.21.15.27:0/3607170937 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b40105b0 con 0x557c370144e0 2024-05-10T11:28:16.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.438+0000 7f16b8c00700 1 -- 172.21.15.27:0/2457520412 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7f16ac00f070 con 0x7f16bc1165f0 2024-05-10T11:28:16.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.439+0000 7f16cda82b80 1 -- 172.21.15.27:0/3607170937 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c370144e0 msgr2=0x557c3700e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:16.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.439+0000 7f16cda82b80 1 --2- 172.21.15.27:0/3607170937 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c370144e0 0x557c3700e2d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f16b400a950 tx=0x7f16b400ac60 comp rx=0 tx=0).stop 2024-05-10T11:28:16.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.439+0000 7f16cda82b80 1 -- 172.21.15.27:0/3607170937 shutdown_connections 2024-05-10T11:28:16.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.439+0000 7f16cda82b80 1 --2- 172.21.15.27:0/3607170937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c3700ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.439+0000 7f16cda82b80 1 --2- 172.21.15.27:0/3607170937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c37013bc0 0x557c37013fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.439+0000 7f16cda82b80 1 --2- 172.21.15.27:0/3607170937 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c370144e0 0x557c3700e2d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.439+0000 7f16cda82b80 1 -- 172.21.15.27:0/3607170937 >> 172.21.15.27:0/3607170937 conn(0x557c36f866d0 msgr2=0x557c37008b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:16.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.439+0000 7f16cda82b80 1 -- 172.21.15.27:0/3607170937 shutdown_connections 2024-05-10T11:28:16.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.439+0000 7f16cda82b80 1 -- 172.21.15.27:0/3607170937 wait complete. 2024-05-10T11:28:16.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.441+0000 7f16cda82b80 1 -- start start 2024-05-10T11:28:16.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.441+0000 7f16cda82b80 1 Processor -- start 2024-05-10T11:28:16.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.443+0000 7f16c1000700 1 --2- 172.21.15.27:0/2457520412 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f16a00b2b40 0x7f16a00b5000 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0e0af0 tx=0x7f16ac01b4b0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:16.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.443+0000 7f16cda82b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c3701a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.443+0000 7f16bb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c3701a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.443+0000 7f16bb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c3701a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54110/0 (socket says 172.21.15.27:54110) 2024-05-10T11:28:16.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.444+0000 7f16bb400700 1 -- 172.21.15.27:0/42095678 learned_addr learned my addr 172.21.15.27:0/42095678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:16.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.444+0000 7f16cda82b80 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c37013bc0 0x557c37017590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.444+0000 7f16cda82b80 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c370144e0 0x557c37017ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.444+0000 7f16cda82b80 1 -- 172.21.15.27:0/42095678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557c3701af20 con 0x557c370144e0 2024-05-10T11:28:16.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.444+0000 7f16cda82b80 1 -- 172.21.15.27:0/42095678 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557c36ec6b20 con 0x557c37013bc0 2024-05-10T11:28:16.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.445+0000 7f16bbe00700 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c37013bc0 0x557c37017590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.445+0000 7f16cda82b80 1 -- 172.21.15.27:0/42095678 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557c3701b0a0 con 0x557c3700e8f0 2024-05-10T11:28:16.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.445+0000 7f16cda82b80 10 client.0 _ll_register_callbacks cb 0x557c36f07c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:16.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.446+0000 7f16bbe00700 1 -- 172.21.15.27:0/42095678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 msgr2=0x557c3701a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:16.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.446+0000 7f16bbe00700 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c3701a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.446+0000 7f16bbe00700 1 -- 172.21.15.27:0/42095678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c370144e0 msgr2=0x557c37017ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:16.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.446+0000 7f16bbe00700 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c370144e0 0x557c37017ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.446+0000 7f16bbe00700 1 -- 172.21.15.27:0/42095678 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c36f26420 con 0x557c37013bc0 2024-05-10T11:28:16.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.447+0000 7f16bbe00700 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c37013bc0 0x557c37017590 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f16b4011d50 tx=0x7f16b4006ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b324c0b79763826 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:16.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.447+0000 7f16aa000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:28:16.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.448+0000 7f16aa000700 1 -- 172.21.15.27:0/42095678 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b4015040 con 0x557c37013bc0 2024-05-10T11:28:16.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.448+0000 7f16aa000700 1 -- 172.21.15.27:0/42095678 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7f16b400cca0 con 0x557c37013bc0 2024-05-10T11:28:16.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.448+0000 7f16cda82b80 1 -- 172.21.15.27:0/42095678 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16b40113f0 con 0x557c37013bc0 2024-05-10T11:28:16.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.449+0000 7f16aa000700 1 -- 172.21.15.27:0/42095678 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f16b4004040 con 0x557c37013bc0 2024-05-10T11:28:16.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.449+0000 7f16cda82b80 10 client.27329 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:16.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.449+0000 7f16cda82b80 1 -- 172.21.15.27:0/42095678 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x557c37025790 con 0x557c37013bc0 2024-05-10T11:28:16.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.449+0000 7f16aa000700 1 -- 172.21.15.27:0/42095678 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16b40113f0 con 0x557c37013bc0 2024-05-10T11:28:16.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.449+0000 7f16aa000700 1 -- 172.21.15.27:0/42095678 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f16b4003a80 con 0x557c37013bc0 2024-05-10T11:28:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.451+0000 7f16cda82b80 10 client.27329 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.451+0000 7f16cda82b80 10 client.27329 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:28:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.451+0000 7f16cda82b80 20 client.27329 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.451+0000 7f16cda82b80 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x557c37027c60 0x557c3702a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.451+0000 7f16cda82b80 4 client.27329 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:28:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.451+0000 7f16cda82b80 1 -- 172.21.15.27:0/42095678 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c36fdb140 con 0x557c37027c60 2024-05-10T11:28:16.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.451+0000 7f16bb400700 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x557c37027c60 0x557c3702a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.452+0000 7f16aa000700 10 client.27329 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:28:16.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.453+0000 7f16bb400700 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x557c37027c60 0x557c3702a050 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d52a2689afeb2d09 server_cookie=976b738a4131ac66 in_seq=0 out_seq=0 2024-05-10T11:28:16.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.454+0000 7f16aa000700 1 -- 172.21.15.27:0/42095678 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557c36fdb140 con 0x557c37027c60 2024-05-10T11:28:16.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.454+0000 7f16aa000700 10 client.27329 handle_command_reply: tid=0 2024-05-10T11:28:16.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.454+0000 7f16cda82b80 1 client.27329 shutdown 2024-05-10T11:28:16.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.454+0000 7f16cda82b80 1 -- 172.21.15.27:0/42095678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c37013bc0 msgr2=0x557c37017590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:16.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.454+0000 7f16cda82b80 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c37013bc0 0x557c37017590 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f16b4011d50 tx=0x7f16b4006ec0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.455+0000 7f16cda82b80 1 -- 172.21.15.27:0/42095678 shutdown_connections 2024-05-10T11:28:16.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.455+0000 7f16cda82b80 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x557c37027c60 0x557c3702a050 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.455+0000 7f16cda82b80 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c3701a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.455+0000 7f16cda82b80 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c370144e0 0x557c37017ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.455+0000 7f16cda82b80 1 --2- 172.21.15.27:0/42095678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c37013bc0 0x557c37017590 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.455+0000 7f16aa000700 0 client.27329 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:28:16.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.455+0000 7f16cda82b80 1 -- 172.21.15.27:0/42095678 >> 172.21.15.27:0/42095678 conn(0x557c36f866d0 msgr2=0x557c37008190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:16.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.455+0000 7f16cda82b80 1 -- 172.21.15.27:0/42095678 shutdown_connections 2024-05-10T11:28:16.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.455+0000 7f16cda82b80 1 -- 172.21.15.27:0/42095678 wait complete. 2024-05-10T11:28:16.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.456+0000 7f16cda82b80 20 client.27329 trim_cache size 0 max 16384 2024-05-10T11:28:16.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.497+0000 7f16cda82b80 1 Processor -- start 2024-05-10T11:28:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.497+0000 7f16cda82b80 1 -- start start 2024-05-10T11:28:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.498+0000 7f16cda82b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c3700e8f0 0x557c3700e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.498+0000 7f16cda82b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c37013bc0 0x557c37014f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.498+0000 7f16cda82b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c370144e0 0x557c37023fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.498+0000 7f16cda82b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1be620 con 0x557c3700e8f0 2024-05-10T11:28:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.498+0000 7f16cda82b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1c0250 con 0x557c370144e0 2024-05-10T11:28:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.498+0000 7f16cda82b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1becf0 con 0x557c37013bc0 2024-05-10T11:28:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.499+0000 7f16bbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c37013bc0 0x557c37014f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.499+0000 7f16bb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c3700e8f0 0x557c3700e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.499+0000 7f16bbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c37013bc0 0x557c37014f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54112/0 (socket says 172.21.15.27:54112) 2024-05-10T11:28:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.499+0000 7f16bbe00700 1 -- 172.21.15.27:0/3350464832 learned_addr learned my addr 172.21.15.27:0/3350464832 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.500+0000 7f16bb400700 1 --2- 172.21.15.27:0/3350464832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c370144e0 0x557c37023fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.500+0000 7f16bbe00700 1 -- 172.21.15.27:0/3350464832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c370144e0 msgr2=0x557c37023fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.500+0000 7f16bbe00700 1 --2- 172.21.15.27:0/3350464832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c370144e0 0x557c37023fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.500+0000 7f16bbe00700 1 -- 172.21.15.27:0/3350464832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c3700e8f0 msgr2=0x557c3700e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.500+0000 7f16bbe00700 1 --2- 172.21.15.27:0/3350464832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c3700e8f0 0x557c3700e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.500+0000 7f16bbe00700 1 -- 172.21.15.27:0/3350464832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c37003ab0 con 0x557c37013bc0 2024-05-10T11:28:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.501+0000 7f16bbe00700 1 --2- 172.21.15.27:0/3350464832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c37013bc0 0x557c37014f30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f16b402f570 tx=0x7f16b4006d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c083c57d142d7d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.502+0000 7f16b0c00700 1 -- 172.21.15.27:0/3350464832 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b4011a30 con 0x557c37013bc0 2024-05-10T11:28:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.502+0000 7f16b0c00700 1 -- 172.21.15.27:0/3350464832 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16b4006ab0 con 0x557c37013bc0 2024-05-10T11:28:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.502+0000 7f16b0c00700 1 -- 172.21.15.27:0/3350464832 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b402f970 con 0x557c37013bc0 2024-05-10T11:28:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.504+0000 7f16cda82b80 1 -- 172.21.15.27:0/3350464832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c37013bc0 msgr2=0x557c37014f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.504+0000 7f16cda82b80 1 --2- 172.21.15.27:0/3350464832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c37013bc0 0x557c37014f30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f16b402f570 tx=0x7f16b4006d70 comp rx=0 tx=0).stop 2024-05-10T11:28:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.504+0000 7f16cda82b80 1 -- 172.21.15.27:0/3350464832 shutdown_connections 2024-05-10T11:28:16.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.504+0000 7f16cda82b80 1 --2- 172.21.15.27:0/3350464832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c370144e0 0x557c37023fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.504+0000 7f16cda82b80 1 --2- 172.21.15.27:0/3350464832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c3700e8f0 0x557c3700e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.505+0000 7f16cda82b80 1 --2- 172.21.15.27:0/3350464832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c37013bc0 0x557c37014f30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.505+0000 7f16cda82b80 1 -- 172.21.15.27:0/3350464832 >> 172.21.15.27:0/3350464832 conn(0x557c36f866d0 msgr2=0x557c37012680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:16.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.506+0000 7f16cda82b80 1 -- 172.21.15.27:0/3350464832 shutdown_connections 2024-05-10T11:28:16.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.506+0000 7f16cda82b80 1 -- 172.21.15.27:0/3350464832 wait complete. 2024-05-10T11:28:16.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.507+0000 7f16cda82b80 1 -- start start 2024-05-10T11:28:16.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.507+0000 7f16cda82b80 1 Processor -- start 2024-05-10T11:28:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.509+0000 7f16cda82b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c37020410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.509+0000 7f16cda82b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c370144e0 0x557c37020950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.510+0000 7f16cda82b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c3701b740 0x557c3701bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.510+0000 7f16cda82b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1bfac0 con 0x557c370144e0 2024-05-10T11:28:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.510+0000 7f16cda82b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1bf3f0 con 0x557c3701b740 2024-05-10T11:28:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.510+0000 7f16cda82b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1bc5e0 con 0x557c3700e8f0 2024-05-10T11:28:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.510+0000 7f16cda82b80 10 client.0 _ll_register_callbacks cb 0x557c3700acd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.511+0000 7f16bb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c37020410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.511+0000 7f16bb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c37020410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54126/0 (socket says 172.21.15.27:54126) 2024-05-10T11:28:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.511+0000 7f16bb400700 1 -- 172.21.15.27:0/2432081799 learned_addr learned my addr 172.21.15.27:0/2432081799 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.512+0000 7f16bbe00700 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c370144e0 0x557c37020950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.513+0000 7f16bb400700 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c3701b740 0x557c3701bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.513+0000 7f16bb400700 1 -- 172.21.15.27:0/2432081799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c3701b740 msgr2=0x557c3701bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.514+0000 7f16bb400700 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c3701b740 0x557c3701bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.514+0000 7f16bb400700 1 -- 172.21.15.27:0/2432081799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c370144e0 msgr2=0x557c37020950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.514+0000 7f16bb400700 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c370144e0 0x557c37020950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.514+0000 7f16bb400700 1 -- 172.21.15.27:0/2432081799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c3701c3a0 con 0x557c3700e8f0 2024-05-10T11:28:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.515+0000 7f16bbe00700 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c370144e0 0x557c37020950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:28:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.515+0000 7f16bb400700 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c37020410 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f16a4005850 tx=0x7f16a400a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7ca887324620a23 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.515+0000 7f16aa000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:28:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.517+0000 7f16aa000700 1 -- 172.21.15.27:0/2432081799 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a4024030 con 0x557c3700e8f0 2024-05-10T11:28:16.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.518+0000 7f16aa000700 1 -- 172.21.15.27:0/2432081799 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7f16a4013a50 con 0x557c3700e8f0 2024-05-10T11:28:16.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.518+0000 7f16cda82b80 1 -- 172.21.15.27:0/2432081799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16b4006ab0 con 0x557c3700e8f0 2024-05-10T11:28:16.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.518+0000 7f16aa000700 1 -- 172.21.15.27:0/2432081799 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f16a4004020 con 0x557c3700e8f0 2024-05-10T11:28:16.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.518+0000 7f16cda82b80 10 client.27331 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:16.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.518+0000 7f16cda82b80 1 -- 172.21.15.27:0/2432081799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x557c37092230 con 0x557c3700e8f0 2024-05-10T11:28:16.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.519+0000 7f16aa000700 1 -- 172.21.15.27:0/2432081799 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16b4006ab0 con 0x557c3700e8f0 2024-05-10T11:28:16.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.519+0000 7f16aa000700 1 -- 172.21.15.27:0/2432081799 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f16a4004830 con 0x557c3700e8f0 2024-05-10T11:28:16.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.519+0000 7f16cda82b80 10 client.27331 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:16.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.519+0000 7f16cda82b80 10 client.27331 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:28:16.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.519+0000 7f16cda82b80 20 client.27331 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.519+0000 7f16cda82b80 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x557c37092970 0x557c37094d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.519+0000 7f16cda82b80 4 client.27331 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:28:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.519+0000 7f16cda82b80 1 -- 172.21.15.27:0/2432081799 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557c36f6e400 con 0x557c37092970 2024-05-10T11:28:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.519+0000 7f16bbe00700 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x557c37092970 0x557c37094d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:16.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.521+0000 7f16aa000700 10 client.27331 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:28:16.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.523+0000 7f16bbe00700 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x557c37092970 0x557c37094d90 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d31abd59b4aefee server_cookie=d8a43a1aa9380b84 in_seq=0 out_seq=0 2024-05-10T11:28:16.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.524+0000 7f16aa000700 1 -- 172.21.15.27:0/2432081799 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557c36f6e400 con 0x557c37092970 2024-05-10T11:28:16.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.524+0000 7f16aa000700 10 client.27331 handle_command_reply: tid=0 2024-05-10T11:28:16.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.527+0000 7f16cda82b80 1 client.27331 shutdown 2024-05-10T11:28:16.528 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:28:16.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.527+0000 7f16cda82b80 1 -- 172.21.15.27:0/2432081799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 msgr2=0x557c37020410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:16.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.527+0000 7f16cda82b80 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c37020410 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f16a4005850 tx=0x7f16a400a300 comp rx=0 tx=0).stop 2024-05-10T11:28:16.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.527+0000 7f16cda82b80 1 -- 172.21.15.27:0/2432081799 shutdown_connections 2024-05-10T11:28:16.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.527+0000 7f16cda82b80 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x557c37092970 0x557c37094d90 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.527+0000 7f16cda82b80 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c3700e8f0 0x557c37020410 secure :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f16a4005850 tx=0x7f16a400a300 comp rx=0 tx=0).stop 2024-05-10T11:28:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.527+0000 7f16cda82b80 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557c3701b740 0x557c3701bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.527+0000 7f16cda82b80 1 --2- 172.21.15.27:0/2432081799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557c370144e0 0x557c37020950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.527+0000 7f16cda82b80 1 -- 172.21.15.27:0/2432081799 >> 172.21.15.27:0/2432081799 conn(0x557c36f866d0 msgr2=0x557c37018b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.527+0000 7f16aa000700 0 client.27331 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:28:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.528+0000 7f16cda82b80 1 -- 172.21.15.27:0/2432081799 shutdown_connections 2024-05-10T11:28:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.528+0000 7f16cda82b80 1 -- 172.21.15.27:0/2432081799 wait complete. 2024-05-10T11:28:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.528+0000 7f16cda82b80 20 client.27331 trim_cache size 0 max 16384 2024-05-10T11:28:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.529+0000 7f16c3000700 1 -- 172.21.15.27:0/2457520412 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f16a00b2b40 msgr2=0x7f16a00b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.529+0000 7f16c3000700 1 --2- 172.21.15.27:0/2457520412 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f16a00b2b40 0x7f16a00b5000 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0e0af0 tx=0x7f16ac01b4b0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.529+0000 7f16c3000700 1 -- 172.21.15.27:0/2457520412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16bc1165f0 msgr2=0x7f16bc1a8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.529+0000 7f16c3000700 1 --2- 172.21.15.27:0/2457520412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16bc1165f0 0x7f16bc1a8eb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00b820 tx=0x7f16ac0079a0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.530+0000 7f16c3000700 1 -- 172.21.15.27:0/2457520412 shutdown_connections 2024-05-10T11:28:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.530+0000 7f16c3000700 1 --2- 172.21.15.27:0/2457520412 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f16a00b2b40 0x7f16a00b5000 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.530+0000 7f16c3000700 1 --2- 172.21.15.27:0/2457520412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16bc1165f0 0x7f16bc1a8eb0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.530+0000 7f16c3000700 1 --2- 172.21.15.27:0/2457520412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16bc10a4a0 0x7f16bc1a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.530+0000 7f16c3000700 1 --2- 172.21.15.27:0/2457520412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc10adc0 0x7f16bc1a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.530+0000 7f16c3000700 1 -- 172.21.15.27:0/2457520412 >> 172.21.15.27:0/2457520412 conn(0x7f16bc06e9f0 msgr2=0x7f16bc0722c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:16.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.530+0000 7f16c3000700 1 -- 172.21.15.27:0/2457520412 shutdown_connections 2024-05-10T11:28:16.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:16.530+0000 7f16c3000700 1 -- 172.21.15.27:0/2457520412 wait complete. 2024-05-10T11:28:16.757 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:16 smithi027 ceph-mon[30054]: pgmap v472: 129 pgs: 129 active+clean; 916 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 9.0 MiB/s wr, 709 op/s 2024-05-10T11:28:17.039 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:16 smithi189 ceph-mon[34842]: pgmap v472: 129 pgs: 129 active+clean; 916 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 9.0 MiB/s wr, 709 op/s 2024-05-10T11:28:17.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 27579 27.17 MB/sec warmup 61 sec latency 378.266 ms 2024-05-10T11:28:17.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:16 smithi076 ceph-mon[34152]: pgmap v472: 129 pgs: 129 active+clean; 916 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 9.0 MiB/s wr, 709 op/s 2024-05-10T11:28:17.572 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:28:18.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 28025 27.18 MB/sec warmup 62 sec latency 267.284 ms 2024-05-10T11:28:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:28:19.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:18 smithi027 ceph-mon[30054]: pgmap v473: 129 pgs: 129 active+clean; 916 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.5 MiB/s wr, 598 op/s 2024-05-10T11:28:19.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 28574 27.29 MB/sec warmup 63 sec latency 74.602 ms 2024-05-10T11:28:19.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:18 smithi076 ceph-mon[34152]: pgmap v473: 129 pgs: 129 active+clean; 916 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.5 MiB/s wr, 598 op/s 2024-05-10T11:28:19.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:18 smithi189 ceph-mon[34842]: pgmap v473: 129 pgs: 129 active+clean; 916 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.5 MiB/s wr, 598 op/s 2024-05-10T11:28:20.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 29068 27.26 MB/sec warmup 64 sec latency 82.438 ms 2024-05-10T11:28:21.063 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:20 smithi027 ceph-mon[30054]: pgmap v474: 129 pgs: 129 active+clean; 877 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.9 MiB/s wr, 493 op/s 2024-05-10T11:28:21.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 29416 27.02 MB/sec warmup 65 sec latency 176.028 ms 2024-05-10T11:28:21.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:20 smithi076 ceph-mon[34152]: pgmap v474: 129 pgs: 129 active+clean; 877 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.9 MiB/s wr, 493 op/s 2024-05-10T11:28:21.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:20 smithi189 ceph-mon[34842]: pgmap v474: 129 pgs: 129 active+clean; 877 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.9 MiB/s wr, 493 op/s 2024-05-10T11:28:21.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.431+0000 7fadc7000700 1 -- 172.21.15.27:0/4145858384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fadc00756d0 msgr2=0x7fadc0075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.431+0000 7fadc7000700 1 --2- 172.21.15.27:0/4145858384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fadc00756d0 0x7fadc0075ab0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fadb0009990 tx=0x7fadb0009ca0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.431+0000 7fadc7000700 1 -- 172.21.15.27:0/4145858384 shutdown_connections 2024-05-10T11:28:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.431+0000 7fadc7000700 1 --2- 172.21.15.27:0/4145858384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadc010ff00 0x7fadc01122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.431+0000 7fadc7000700 1 --2- 172.21.15.27:0/4145858384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fadc00756d0 0x7fadc0075ab0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.431+0000 7fadc7000700 1 --2- 172.21.15.27:0/4145858384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc0075ff0 0x7fadc010f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.431+0000 7fadc7000700 1 -- 172.21.15.27:0/4145858384 >> 172.21.15.27:0/4145858384 conn(0x7fadc007a600 msgr2=0x7fadc007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.431+0000 7fadc7000700 1 -- 172.21.15.27:0/4145858384 shutdown_connections 2024-05-10T11:28:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.431+0000 7fadc7000700 1 -- 172.21.15.27:0/4145858384 wait complete. 2024-05-10T11:28:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.433+0000 7fadc7000700 1 Processor -- start 2024-05-10T11:28:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.433+0000 7fadc7000700 1 -- start start 2024-05-10T11:28:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.433+0000 7fadc7000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadc00756d0 0x7fadc019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.433+0000 7fadc7000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fadc0075ff0 0x7fadc019c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.433+0000 7fadc7000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc010ff00 0x7fadc01a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.433+0000 7fadc7000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01158e0 con 0x7fadc010ff00 2024-05-10T11:28:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.433+0000 7fadc7000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fadc0115760 con 0x7fadc0075ff0 2024-05-10T11:28:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.433+0000 7fadc7000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fadc0115a60 con 0x7fadc00756d0 2024-05-10T11:28:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.434+0000 7fadbfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadc00756d0 0x7fadc019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.434+0000 7fadbf400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fadc0075ff0 0x7fadc019c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.434+0000 7fadbf400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fadc0075ff0 0x7fadc019c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37900/0 (socket says 172.21.15.27:37900) 2024-05-10T11:28:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.434+0000 7fadbf400700 1 -- 172.21.15.27:0/1467203692 learned_addr learned my addr 172.21.15.27:0/1467203692 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.434+0000 7fadc5000700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc010ff00 0x7fadc01a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.434+0000 7fadbfe00700 1 -- 172.21.15.27:0/1467203692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fadc0075ff0 msgr2=0x7fadc019c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.434+0000 7fadbfe00700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fadc0075ff0 0x7fadc019c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.434+0000 7fadbfe00700 1 -- 172.21.15.27:0/1467203692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc010ff00 msgr2=0x7fadc01a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.434+0000 7fadbfe00700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc010ff00 0x7fadc01a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.434+0000 7fadbfe00700 1 -- 172.21.15.27:0/1467203692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc01a0d40 con 0x7fadc00756d0 2024-05-10T11:28:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.435+0000 7fadbfe00700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadc00756d0 0x7fadc019bd30 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fadb00098e0 tx=0x7fadb00057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.435+0000 7fadbcc00700 1 -- 172.21.15.27:0/1467203692 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb0027070 con 0x7fadc00756d0 2024-05-10T11:28:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.435+0000 7fadbcc00700 1 -- 172.21.15.27:0/1467203692 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadb0022470 con 0x7fadc00756d0 2024-05-10T11:28:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.435+0000 7fadbcc00700 1 -- 172.21.15.27:0/1467203692 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb0017650 con 0x7fadc00756d0 2024-05-10T11:28:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.435+0000 7fadc7000700 1 -- 172.21.15.27:0/1467203692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc01a0fd0 con 0x7fadc00756d0 2024-05-10T11:28:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.435+0000 7fadc7000700 1 -- 172.21.15.27:0/1467203692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc01a88b0 con 0x7fadc00756d0 2024-05-10T11:28:21.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.439+0000 7fadbcc00700 1 -- 172.21.15.27:0/1467203692 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fadb0009d10 con 0x7fadc00756d0 2024-05-10T11:28:21.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.440+0000 7fadd1920b80 1 Processor -- start 2024-05-10T11:28:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.440+0000 7fadd1920b80 1 -- start start 2024-05-10T11:28:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadbcc00700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fada40b2a70 0x7fada40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadd1920b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f844bc0 0x55a28f844fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadd1920b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f8454e0 0x55a28f83f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadd1920b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f83f8f0 0x55a28f83fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadd1920b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a28f791a20 con 0x55a28f83f8f0 2024-05-10T11:28:21.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadd1920b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a28f72f760 con 0x55a28f844bc0 2024-05-10T11:28:21.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadd1920b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a28f78d9e0 con 0x55a28f8454e0 2024-05-10T11:28:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadbcc00700 1 -- 172.21.15.27:0/1467203692 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7fadb00e3160 con 0x7fadc00756d0 2024-05-10T11:28:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadbfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f844bc0 0x55a28f844fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadbfe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f844bc0 0x55a28f844fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37904/0 (socket says 172.21.15.27:37904) 2024-05-10T11:28:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadbfe00700 1 -- 172.21.15.27:0/1503154772 learned_addr learned my addr 172.21.15.27:0/1503154772 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadc5000700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fada40b2a70 0x7fada40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.441+0000 7fadbf400700 1 --2- 172.21.15.27:0/1503154772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f8454e0 0x55a28f83f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.442+0000 7fadc5000700 1 --2- 172.21.15.27:0/1503154772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f83f8f0 0x55a28f83fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.442+0000 7fadbf400700 1 -- 172.21.15.27:0/1503154772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f844bc0 msgr2=0x55a28f844fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.442+0000 7fadbf400700 1 --2- 172.21.15.27:0/1503154772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f844bc0 0x55a28f844fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.442+0000 7fadbf400700 1 -- 172.21.15.27:0/1503154772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f83f8f0 msgr2=0x55a28f83fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.442+0000 7fadbf400700 1 --2- 172.21.15.27:0/1503154772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f83f8f0 0x55a28f83fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.442+0000 7fadbf400700 1 -- 172.21.15.27:0/1503154772 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a28f6de8d0 con 0x55a28f8454e0 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.442+0000 7fadc5000700 1 --2- 172.21.15.27:0/1503154772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f83f8f0 0x55a28f83fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.442+0000 7fadbfe00700 1 --2- 172.21.15.27:0/1503154772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f844bc0 0x55a28f844fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.442+0000 7fadbf400700 1 --2- 172.21.15.27:0/1503154772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f8454e0 0x55a28f83f2d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fadc0111ac0 tx=0x7fadac011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9267613f38dcd03 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.443+0000 7fadb5600700 1 -- 172.21.15.27:0/1503154772 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadac011d80 con 0x55a28f8454e0 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.443+0000 7fadc5000700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fada40b2a70 0x7fada40b4f30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55a28f8402b0 tx=0x7fadb8009920 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.443+0000 7fadb5600700 1 -- 172.21.15.27:0/1503154772 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadac004500 con 0x55a28f8454e0 2024-05-10T11:28:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.443+0000 7fadb5600700 1 -- 172.21.15.27:0/1503154772 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadac005260 con 0x55a28f8454e0 2024-05-10T11:28:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.444+0000 7fadd1920b80 1 -- 172.21.15.27:0/1503154772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f8454e0 msgr2=0x55a28f83f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.444+0000 7fadd1920b80 1 --2- 172.21.15.27:0/1503154772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f8454e0 0x55a28f83f2d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fadc0111ac0 tx=0x7fadac011440 comp rx=0 tx=0).stop 2024-05-10T11:28:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.444+0000 7fadd1920b80 1 -- 172.21.15.27:0/1503154772 shutdown_connections 2024-05-10T11:28:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.444+0000 7fadd1920b80 1 --2- 172.21.15.27:0/1503154772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f83f8f0 0x55a28f83fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.444+0000 7fadd1920b80 1 --2- 172.21.15.27:0/1503154772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f844bc0 0x55a28f844fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.444+0000 7fadd1920b80 1 --2- 172.21.15.27:0/1503154772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f8454e0 0x55a28f83f2d0 secure :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fadc0111ac0 tx=0x7fadac011440 comp rx=0 tx=0).stop 2024-05-10T11:28:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.444+0000 7fadd1920b80 1 -- 172.21.15.27:0/1503154772 >> 172.21.15.27:0/1503154772 conn(0x55a28f7b76d0 msgr2=0x55a28f839b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:21.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.445+0000 7fadd1920b80 1 -- 172.21.15.27:0/1503154772 shutdown_connections 2024-05-10T11:28:21.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.445+0000 7fadd1920b80 1 -- 172.21.15.27:0/1503154772 wait complete. 2024-05-10T11:28:21.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.445+0000 7fadd1920b80 1 -- start start 2024-05-10T11:28:21.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.446+0000 7fadd1920b80 1 Processor -- start 2024-05-10T11:28:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.446+0000 7fadd1920b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f83f8f0 0x55a28f84b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.446+0000 7fadd1920b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 0x55a28f8486d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.446+0000 7fadd1920b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f848c10 0x55a28f849090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.446+0000 7fadd1920b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a28f84bf20 con 0x55a28f83f8f0 2024-05-10T11:28:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.446+0000 7fadd1920b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a28f6f7b20 con 0x55a28f848c10 2024-05-10T11:28:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.446+0000 7fadd1920b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a28f84c0a0 con 0x55a28f844bc0 2024-05-10T11:28:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.446+0000 7fadd1920b80 10 client.0 _ll_register_callbacks cb 0x55a28f738c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:21.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.446+0000 7fadbf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 0x55a28f8486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.447+0000 7fadbf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 0x55a28f8486d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54158/0 (socket says 172.21.15.27:54158) 2024-05-10T11:28:21.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.447+0000 7fadbf400700 1 -- 172.21.15.27:0/2780671492 learned_addr learned my addr 172.21.15.27:0/2780671492 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:21.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.447+0000 7fadbfe00700 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f83f8f0 0x55a28f84b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.447+0000 7fadc5000700 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f848c10 0x55a28f849090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.447+0000 7fadbf400700 1 -- 172.21.15.27:0/2780671492 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f848c10 msgr2=0x55a28f849090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.447+0000 7fadbf400700 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f848c10 0x55a28f849090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.447+0000 7fadbf400700 1 -- 172.21.15.27:0/2780671492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f83f8f0 msgr2=0x55a28f84b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.447+0000 7fadbf400700 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f83f8f0 0x55a28f84b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.447+0000 7fadbf400700 1 -- 172.21.15.27:0/2780671492 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a28f757420 con 0x55a28f844bc0 2024-05-10T11:28:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.447+0000 7fadbf400700 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 0x55a28f8486d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fadac0096b0 tx=0x7fadac005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3459aace0453828b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.447+0000 7fadaa000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:28:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.448+0000 7fadaa000700 1 -- 172.21.15.27:0/2780671492 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadac0042e0 con 0x55a28f844bc0 2024-05-10T11:28:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.448+0000 7fadd1920b80 1 -- 172.21.15.27:0/2780671492 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadac004500 con 0x55a28f844bc0 2024-05-10T11:28:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.448+0000 7fadaa000700 1 -- 172.21.15.27:0/2780671492 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7fadac003680 con 0x55a28f844bc0 2024-05-10T11:28:21.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.448+0000 7fadaa000700 1 -- 172.21.15.27:0/2780671492 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fadac030050 con 0x55a28f844bc0 2024-05-10T11:28:21.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.448+0000 7fadd1920b80 10 client.27343 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:21.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.448+0000 7fadd1920b80 1 -- 172.21.15.27:0/2780671492 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a28f856c90 con 0x55a28f844bc0 2024-05-10T11:28:21.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.449+0000 7fadaa000700 1 -- 172.21.15.27:0/2780671492 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadac004500 con 0x55a28f844bc0 2024-05-10T11:28:21.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.449+0000 7fadaa000700 1 -- 172.21.15.27:0/2780671492 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fadac018070 con 0x55a28f844bc0 2024-05-10T11:28:21.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.449+0000 7fadd1920b80 10 client.27343 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:21.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.449+0000 7fadd1920b80 10 client.27343 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:28:21.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.449+0000 7fadd1920b80 20 client.27343 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:21.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.449+0000 7fadd1920b80 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a28f859160 0x55a28f85b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.449+0000 7fadd1920b80 4 client.27343 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:28:21.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.449+0000 7fadd1920b80 1 -- 172.21.15.27:0/2780671492 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a28f80c140 con 0x55a28f859160 2024-05-10T11:28:21.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.449+0000 7fadbfe00700 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a28f859160 0x55a28f85b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.450+0000 7fadbfe00700 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a28f859160 0x55a28f85b550 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=779e893db41028de server_cookie=51370bb024e5cac7 in_seq=0 out_seq=0 2024-05-10T11:28:21.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.450+0000 7fadaa000700 10 client.27343 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:28:21.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.451+0000 7fadaa000700 1 -- 172.21.15.27:0/2780671492 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a28f80c140 con 0x55a28f859160 2024-05-10T11:28:21.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.451+0000 7fadaa000700 10 client.27343 handle_command_reply: tid=0 2024-05-10T11:28:21.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.452+0000 7fadd1920b80 1 client.27343 shutdown 2024-05-10T11:28:21.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.452+0000 7fadd1920b80 1 -- 172.21.15.27:0/2780671492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 msgr2=0x55a28f8486d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:21.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.452+0000 7fadd1920b80 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 0x55a28f8486d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fadac0096b0 tx=0x7fadac005dd0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.452+0000 7fadd1920b80 1 -- 172.21.15.27:0/2780671492 shutdown_connections 2024-05-10T11:28:21.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.452+0000 7fadd1920b80 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f848c10 0x55a28f849090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.452+0000 7fadd1920b80 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f83f8f0 0x55a28f84b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.452+0000 7fadd1920b80 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a28f859160 0x55a28f85b550 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.452+0000 7fadd1920b80 1 --2- 172.21.15.27:0/2780671492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 0x55a28f8486d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.452+0000 7fadd1920b80 1 -- 172.21.15.27:0/2780671492 >> 172.21.15.27:0/2780671492 conn(0x55a28f7b76d0 msgr2=0x55a28f839190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:21.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.452+0000 7fadaa000700 0 client.27343 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:28:21.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.452+0000 7fadd1920b80 1 -- 172.21.15.27:0/2780671492 shutdown_connections 2024-05-10T11:28:21.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.452+0000 7fadd1920b80 1 -- 172.21.15.27:0/2780671492 wait complete. 2024-05-10T11:28:21.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.453+0000 7fadd1920b80 20 client.27343 trim_cache size 0 max 16384 2024-05-10T11:28:21.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.472+0000 7fadd1920b80 1 Processor -- start 2024-05-10T11:28:21.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.472+0000 7fadd1920b80 1 -- start start 2024-05-10T11:28:21.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.473+0000 7fadd1920b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f83f8f0 0x55a28f83f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.473+0000 7fadd1920b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 0x55a28f848690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.473+0000 7fadd1920b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f848c10 0x55a28f854fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.473+0000 7fadd1920b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01b1810 con 0x55a28f848c10 2024-05-10T11:28:21.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.473+0000 7fadd1920b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01b3440 con 0x55a28f83f8f0 2024-05-10T11:28:21.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.473+0000 7fadd1920b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01b1ee0 con 0x55a28f844bc0 2024-05-10T11:28:21.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.473+0000 7fadbf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 0x55a28f848690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.473+0000 7fadc5000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f848c10 0x55a28f854fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.473+0000 7fadbf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 0x55a28f848690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54172/0 (socket says 172.21.15.27:54172) 2024-05-10T11:28:21.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.473+0000 7fadbf400700 1 -- 172.21.15.27:0/782516834 learned_addr learned my addr 172.21.15.27:0/782516834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:21.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.474+0000 7fadbf400700 1 -- 172.21.15.27:0/782516834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f83f8f0 msgr2=0x55a28f83f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:21.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.474+0000 7fadbf400700 1 --2- 172.21.15.27:0/782516834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f83f8f0 0x55a28f83f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.474+0000 7fadbf400700 1 -- 172.21.15.27:0/782516834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f848c10 msgr2=0x55a28f854fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:21.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.474+0000 7fadbf400700 1 --2- 172.21.15.27:0/782516834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f848c10 0x55a28f854fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.474+0000 7fadbf400700 1 -- 172.21.15.27:0/782516834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a28f83a840 con 0x55a28f844bc0 2024-05-10T11:28:21.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.474+0000 7fadbf400700 1 --2- 172.21.15.27:0/782516834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 0x55a28f848690 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fadac000c00 tx=0x7fadac001e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90adaba2b10a246b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:21.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.477+0000 7fadb4c00700 1 -- 172.21.15.27:0/782516834 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadac039070 con 0x55a28f844bc0 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.477+0000 7fadb4c00700 1 -- 172.21.15.27:0/782516834 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadac027070 con 0x55a28f844bc0 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.477+0000 7fadb4c00700 1 -- 172.21.15.27:0/782516834 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadac004520 con 0x55a28f844bc0 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.478+0000 7fadd1920b80 1 -- 172.21.15.27:0/782516834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 msgr2=0x55a28f848690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.478+0000 7fadd1920b80 1 --2- 172.21.15.27:0/782516834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 0x55a28f848690 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fadac000c00 tx=0x7fadac001e50 comp rx=0 tx=0).stop 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.478+0000 7fadd1920b80 1 -- 172.21.15.27:0/782516834 shutdown_connections 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.478+0000 7fadd1920b80 1 --2- 172.21.15.27:0/782516834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f848c10 0x55a28f854fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.478+0000 7fadd1920b80 1 --2- 172.21.15.27:0/782516834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f83f8f0 0x55a28f83f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.478+0000 7fadd1920b80 1 --2- 172.21.15.27:0/782516834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f844bc0 0x55a28f848690 secure :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fadac000c00 tx=0x7fadac001e50 comp rx=0 tx=0).stop 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.478+0000 7fadd1920b80 1 -- 172.21.15.27:0/782516834 >> 172.21.15.27:0/782516834 conn(0x55a28f7b76d0 msgr2=0x55a28f843800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.478+0000 7fadd1920b80 1 -- 172.21.15.27:0/782516834 shutdown_connections 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.478+0000 7fadd1920b80 1 -- 172.21.15.27:0/782516834 wait complete. 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.479+0000 7fadd1920b80 1 -- start start 2024-05-10T11:28:21.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.479+0000 7fadd1920b80 1 Processor -- start 2024-05-10T11:28:21.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.480+0000 7fadd1920b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f83f8f0 0x55a28f851510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.480+0000 7fadd1920b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f848c10 0x55a28f851a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.480+0000 7fadd1920b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f85fed0 0x55a28f860350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.480+0000 7fadd1920b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01b2cb0 con 0x55a28f848c10 2024-05-10T11:28:21.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.480+0000 7fadd1920b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01b25e0 con 0x55a28f85fed0 2024-05-10T11:28:21.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.480+0000 7fadd1920b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01af7d0 con 0x55a28f83f8f0 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.480+0000 7fadd1920b80 10 client.0 _ll_register_callbacks cb 0x55a28f83bcd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.481+0000 7fadc5000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f85fed0 0x55a28f860350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.481+0000 7fadc5000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f85fed0 0x55a28f860350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:37938/0 (socket says 172.21.15.27:37938) 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.481+0000 7fadc5000700 1 -- 172.21.15.27:0/3601672456 learned_addr learned my addr 172.21.15.27:0/3601672456 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.481+0000 7fadbfe00700 1 --2- 172.21.15.27:0/3601672456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f83f8f0 0x55a28f851510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.482+0000 7fadbfe00700 1 -- 172.21.15.27:0/3601672456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f85fed0 msgr2=0x55a28f860350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.482+0000 7fadbfe00700 1 --2- 172.21.15.27:0/3601672456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f85fed0 0x55a28f860350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.482+0000 7fadbfe00700 1 -- 172.21.15.27:0/3601672456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f848c10 msgr2=0x55a28f851a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.482+0000 7fadbfe00700 1 --2- 172.21.15.27:0/3601672456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f848c10 0x55a28f851a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.482+0000 7fadbfe00700 1 -- 172.21.15.27:0/3601672456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a28f860b10 con 0x55a28f83f8f0 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.482+0000 7fadbfe00700 1 --2- 172.21.15.27:0/3601672456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f83f8f0 0x55a28f851510 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fadb00a60c0 tx=0x7fadb0022850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=450681acc83e0ec3 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.482+0000 7fadaa000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:28:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.482+0000 7fadaa000700 1 -- 172.21.15.27:0/3601672456 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb00aa740 con 0x55a28f83f8f0 2024-05-10T11:28:21.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.482+0000 7fadaa000700 1 -- 172.21.15.27:0/3601672456 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7fadb00228e0 con 0x55a28f83f8f0 2024-05-10T11:28:21.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.482+0000 7fadd1920b80 1 -- 172.21.15.27:0/3601672456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadac027070 con 0x55a28f83f8f0 2024-05-10T11:28:21.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.483+0000 7fadaa000700 1 -- 172.21.15.27:0/3601672456 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fadb00aa040 con 0x55a28f83f8f0 2024-05-10T11:28:21.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.483+0000 7fadd1920b80 10 client.27349 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:21.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.483+0000 7fadd1920b80 1 -- 172.21.15.27:0/3601672456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a28f8cc0f0 con 0x55a28f83f8f0 2024-05-10T11:28:21.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.485+0000 7fadaa000700 1 -- 172.21.15.27:0/3601672456 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadac027070 con 0x55a28f83f8f0 2024-05-10T11:28:21.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.485+0000 7fadaa000700 1 -- 172.21.15.27:0/3601672456 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fadb00aa2a0 con 0x55a28f83f8f0 2024-05-10T11:28:21.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.485+0000 7fadd1920b80 10 client.27349 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:21.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.485+0000 7fadd1920b80 10 client.27349 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:28:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.485+0000 7fadd1920b80 20 client.27349 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.485+0000 7fadd1920b80 1 --2- 172.21.15.27:0/3601672456 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a28f8cc7a0 0x55a28f8cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.485+0000 7fadd1920b80 4 client.27349 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:28:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.485+0000 7fadd1920b80 1 -- 172.21.15.27:0/3601672456 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a28f79f400 con 0x55a28f8cc7a0 2024-05-10T11:28:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.486+0000 7fadbf400700 1 --2- 172.21.15.27:0/3601672456 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a28f8cc7a0 0x55a28f8cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.489+0000 7fadbf400700 1 --2- 172.21.15.27:0/3601672456 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a28f8cc7a0 0x55a28f8cebc0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9dbf495efbd93fd0 server_cookie=ad20550116dad410 in_seq=0 out_seq=0 2024-05-10T11:28:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.490+0000 7fadaa000700 10 client.27349 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:28:21.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.491+0000 7fadaa000700 1 -- 172.21.15.27:0/3601672456 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a28f79f400 con 0x55a28f8cc7a0 2024-05-10T11:28:21.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.491+0000 7fadaa000700 10 client.27349 handle_command_reply: tid=0 2024-05-10T11:28:21.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.491+0000 7fadd1920b80 1 client.27349 shutdown 2024-05-10T11:28:21.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.492+0000 7fadd1920b80 1 -- 172.21.15.27:0/3601672456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f83f8f0 msgr2=0x55a28f851510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:21.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.492+0000 7fadd1920b80 1 --2- 172.21.15.27:0/3601672456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f83f8f0 0x55a28f851510 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fadb00a60c0 tx=0x7fadb0022850 comp rx=0 tx=0).stop 2024-05-10T11:28:21.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.492+0000 7fadd1920b80 1 -- 172.21.15.27:0/3601672456 shutdown_connections 2024-05-10T11:28:21.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.492+0000 7fadd1920b80 1 --2- 172.21.15.27:0/3601672456 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55a28f8cc7a0 0x55a28f8cebc0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.492+0000 7fadd1920b80 1 --2- 172.21.15.27:0/3601672456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a28f83f8f0 0x55a28f851510 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.492+0000 7fadd1920b80 1 --2- 172.21.15.27:0/3601672456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a28f85fed0 0x55a28f860350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.492+0000 7fadd1920b80 1 --2- 172.21.15.27:0/3601672456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a28f848c10 0x55a28f851a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.492+0000 7fadd1920b80 1 -- 172.21.15.27:0/3601672456 >> 172.21.15.27:0/3601672456 conn(0x55a28f7b76d0 msgr2=0x55a28f849eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:21.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.492+0000 7fadaa000700 0 client.27349 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:28:21.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.493+0000 7fadd1920b80 1 -- 172.21.15.27:0/3601672456 shutdown_connections 2024-05-10T11:28:21.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.493+0000 7fadd1920b80 1 -- 172.21.15.27:0/3601672456 wait complete. 2024-05-10T11:28:21.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.493+0000 7fadd1920b80 20 client.27349 trim_cache size 0 max 16384 2024-05-10T11:28:21.494 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:28:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.495+0000 7fadc7000700 1 -- 172.21.15.27:0/1467203692 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fada40b2a70 msgr2=0x7fada40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.495+0000 7fadc7000700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fada40b2a70 0x7fada40b4f30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55a28f8402b0 tx=0x7fadb8009920 comp rx=0 tx=0).stop 2024-05-10T11:28:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.495+0000 7fadc7000700 1 -- 172.21.15.27:0/1467203692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadc00756d0 msgr2=0x7fadc019bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.495+0000 7fadc7000700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadc00756d0 0x7fadc019bd30 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fadb00098e0 tx=0x7fadb00057c0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.495+0000 7fadc7000700 1 -- 172.21.15.27:0/1467203692 shutdown_connections 2024-05-10T11:28:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.495+0000 7fadc7000700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fada40b2a70 0x7fada40b4f30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.495+0000 7fadc7000700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadc00756d0 0x7fadc019bd30 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.495+0000 7fadc7000700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc010ff00 0x7fadc01a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.495+0000 7fadc7000700 1 --2- 172.21.15.27:0/1467203692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fadc0075ff0 0x7fadc019c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.496+0000 7fadc7000700 1 -- 172.21.15.27:0/1467203692 >> 172.21.15.27:0/1467203692 conn(0x7fadc007a600 msgr2=0x7fadc01006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.496+0000 7fadc7000700 1 -- 172.21.15.27:0/1467203692 shutdown_connections 2024-05-10T11:28:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:21.496+0000 7fadc7000700 1 -- 172.21.15.27:0/1467203692 wait complete. 2024-05-10T11:28:22.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 29686 26.85 MB/sec warmup 66 sec latency 198.445 ms 2024-05-10T11:28:22.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:28:22.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:22 smithi027 ceph-mon[30054]: pgmap v475: 129 pgs: 129 active+clean; 877 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.9 MiB/s wr, 493 op/s 2024-05-10T11:28:23.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 30131 26.73 MB/sec warmup 67 sec latency 110.996 ms 2024-05-10T11:28:23.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:22 smithi076 ceph-mon[34152]: pgmap v475: 129 pgs: 129 active+clean; 877 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.9 MiB/s wr, 493 op/s 2024-05-10T11:28:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:28:23.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:22 smithi189 ceph-mon[34842]: pgmap v475: 129 pgs: 129 active+clean; 877 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.9 MiB/s wr, 493 op/s 2024-05-10T11:28:24.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 30654 26.91 MB/sec warmup 68 sec latency 204.229 ms 2024-05-10T11:28:25.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:24 smithi027 ceph-mon[30054]: pgmap v476: 129 pgs: 129 active+clean; 877 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.9 MiB/s wr, 493 op/s 2024-05-10T11:28:25.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 31226 26.80 MB/sec warmup 69 sec latency 184.757 ms 2024-05-10T11:28:25.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:24 smithi076 ceph-mon[34152]: pgmap v476: 129 pgs: 129 active+clean; 877 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.9 MiB/s wr, 493 op/s 2024-05-10T11:28:25.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:24 smithi189 ceph-mon[34842]: pgmap v476: 129 pgs: 129 active+clean; 877 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.9 MiB/s wr, 493 op/s 2024-05-10T11:28:26.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 31746 26.92 MB/sec warmup 70 sec latency 193.060 ms 2024-05-10T11:28:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.225+0000 7f198a800700 1 -- 172.21.15.27:0/2920837564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19841060a0 msgr2=0x7f19841125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.225+0000 7f198a800700 1 --2- 172.21.15.27:0/2920837564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19841060a0 0x7f19841125b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1974009990 tx=0x7f1974009ca0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.225+0000 7f198a800700 1 -- 172.21.15.27:0/2920837564 shutdown_connections 2024-05-10T11:28:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.225+0000 7f198a800700 1 --2- 172.21.15.27:0/2920837564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19841060a0 0x7f19841125b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.225+0000 7f198a800700 1 --2- 172.21.15.27:0/2920837564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f198410b6e0 0x7f198410bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.225+0000 7f198a800700 1 --2- 172.21.15.27:0/2920837564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19841056e0 0x7f1984105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.225+0000 7f198a800700 1 -- 172.21.15.27:0/2920837564 >> 172.21.15.27:0/2920837564 conn(0x7f19841011d0 msgr2=0x7f19841035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.225+0000 7f198a800700 1 -- 172.21.15.27:0/2920837564 shutdown_connections 2024-05-10T11:28:26.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.226+0000 7f198a800700 1 -- 172.21.15.27:0/2920837564 wait complete. 2024-05-10T11:28:26.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.227+0000 7f198a800700 1 Processor -- start 2024-05-10T11:28:26.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.227+0000 7f198a800700 1 -- start start 2024-05-10T11:28:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.228+0000 7f198a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19841056e0 0x7f198419bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.228+0000 7f198a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19841060a0 0x7f198419c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.228+0000 7f198a800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f198410b6e0 0x7f19841a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.228+0000 7f198a800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1984115920 con 0x7f19841056e0 2024-05-10T11:28:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.228+0000 7f198a800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f19841157a0 con 0x7f19841060a0 2024-05-10T11:28:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.228+0000 7f198a800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1984115aa0 con 0x7f198410b6e0 2024-05-10T11:28:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.228+0000 7f1983400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19841056e0 0x7f198419bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.228+0000 7f1982a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19841060a0 0x7f198419c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.228+0000 7f1982a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19841060a0 0x7f198419c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41600/0 (socket says 172.21.15.27:41600) 2024-05-10T11:28:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.228+0000 7f1982a00700 1 -- 172.21.15.27:0/1233294792 learned_addr learned my addr 172.21.15.27:0/1233294792 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.228+0000 7f1983e00700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f198410b6e0 0x7f19841a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.229+0000 7f1983e00700 1 -- 172.21.15.27:0/1233294792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19841060a0 msgr2=0x7f198419c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.229+0000 7f1983e00700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19841060a0 0x7f198419c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.229+0000 7f1983e00700 1 -- 172.21.15.27:0/1233294792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19841056e0 msgr2=0x7f198419bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.229+0000 7f1983e00700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19841056e0 0x7f198419bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.229+0000 7f1983e00700 1 -- 172.21.15.27:0/1233294792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19841a0d40 con 0x7f198410b6e0 2024-05-10T11:28:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.229+0000 7f1982a00700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19841060a0 0x7f198419c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:28:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.229+0000 7f1983e00700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f198410b6e0 0x7f19841a0600 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f19740098e0 tx=0x7f19740047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.230+0000 7f197be00700 1 -- 172.21.15.27:0/1233294792 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974027070 con 0x7f198410b6e0 2024-05-10T11:28:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.230+0000 7f198a800700 1 -- 172.21.15.27:0/1233294792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19841a0fd0 con 0x7f198410b6e0 2024-05-10T11:28:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.230+0000 7f198a800700 1 -- 172.21.15.27:0/1233294792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19841a88b0 con 0x7f198410b6e0 2024-05-10T11:28:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.230+0000 7f197be00700 1 -- 172.21.15.27:0/1233294792 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1974022470 con 0x7f198410b6e0 2024-05-10T11:28:26.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.230+0000 7f197be00700 1 -- 172.21.15.27:0/1233294792 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974017650 con 0x7f198410b6e0 2024-05-10T11:28:26.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.235+0000 7f197be00700 1 -- 172.21.15.27:0/1233294792 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f197402a070 con 0x7f198410b6e0 2024-05-10T11:28:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.236+0000 7f1995110b80 1 Processor -- start 2024-05-10T11:28:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.237+0000 7f1995110b80 1 -- start start 2024-05-10T11:28:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.237+0000 7f197be00700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f19680b2b40 0x7f19680b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.237+0000 7f197be00700 1 -- 172.21.15.27:0/1233294792 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7f19740e34e0 con 0x7f198410b6e0 2024-05-10T11:28:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.237+0000 7f1995110b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce381ebc0 0x556ce381efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.237+0000 7f1995110b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce381f4e0 0x556ce38192d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.237+0000 7f1995110b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce38198f0 0x556ce3819d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.237+0000 7f1995110b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556ce376ba20 con 0x556ce38198f0 2024-05-10T11:28:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.237+0000 7f1995110b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x556ce3709760 con 0x556ce381ebc0 2024-05-10T11:28:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.237+0000 7f1995110b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556ce37679e0 con 0x556ce381f4e0 2024-05-10T11:28:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.237+0000 7f1982a00700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f19680b2b40 0x7f19680b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.238+0000 7f1983400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce381ebc0 0x556ce381efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.238+0000 7f1983400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce381ebc0 0x556ce381efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41602/0 (socket says 172.21.15.27:41602) 2024-05-10T11:28:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.238+0000 7f1983400700 1 -- 172.21.15.27:0/1621791997 learned_addr learned my addr 172.21.15.27:0/1621791997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.238+0000 7f1983e00700 1 --2- 172.21.15.27:0/1621791997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce38198f0 0x556ce3819d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.238+0000 7f1982a00700 1 --2- 172.21.15.27:0/1621791997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce381f4e0 0x556ce38192d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.239+0000 7f1983400700 1 -- 172.21.15.27:0/1621791997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce381f4e0 msgr2=0x556ce38192d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:26.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.239+0000 7f1983400700 1 --2- 172.21.15.27:0/1621791997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce381f4e0 0x556ce38192d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.239+0000 7f1983400700 1 -- 172.21.15.27:0/1621791997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce38198f0 msgr2=0x556ce3819d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:26.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.239+0000 7f1983400700 1 --2- 172.21.15.27:0/1621791997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce38198f0 0x556ce3819d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.239+0000 7f1983400700 1 -- 172.21.15.27:0/1621791997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ce36b88d0 con 0x556ce381ebc0 2024-05-10T11:28:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.239+0000 7f1982a00700 1 --2- 172.21.15.27:0/1621791997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce381f4e0 0x556ce38192d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:28:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.239+0000 7f1983400700 1 --2- 172.21.15.27:0/1621791997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce381ebc0 0x556ce381efa0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f197c009d00 tx=0x7f197c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d5c0a4775dee51e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.239+0000 7f1982a00700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f19680b2b40 0x7f19680b5000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1970009fd0 tx=0x7f1970009410 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.239+0000 7f1978c00700 1 -- 172.21.15.27:0/1621791997 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c01e070 con 0x556ce381ebc0 2024-05-10T11:28:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.239+0000 7f1978c00700 1 -- 172.21.15.27:0/1621791997 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f197c019ae0 con 0x556ce381ebc0 2024-05-10T11:28:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.239+0000 7f1978c00700 1 -- 172.21.15.27:0/1621791997 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c00e9a0 con 0x556ce381ebc0 2024-05-10T11:28:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.240+0000 7f1995110b80 1 -- 172.21.15.27:0/1621791997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce381ebc0 msgr2=0x556ce381efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.240+0000 7f1995110b80 1 --2- 172.21.15.27:0/1621791997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce381ebc0 0x556ce381efa0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f197c009d00 tx=0x7f197c011720 comp rx=0 tx=0).stop 2024-05-10T11:28:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.240+0000 7f1995110b80 1 -- 172.21.15.27:0/1621791997 shutdown_connections 2024-05-10T11:28:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.240+0000 7f1995110b80 1 --2- 172.21.15.27:0/1621791997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce38198f0 0x556ce3819d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.240+0000 7f1995110b80 1 --2- 172.21.15.27:0/1621791997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce381ebc0 0x556ce381efa0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.240+0000 7f1995110b80 1 --2- 172.21.15.27:0/1621791997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce381f4e0 0x556ce38192d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.240+0000 7f1995110b80 1 -- 172.21.15.27:0/1621791997 >> 172.21.15.27:0/1621791997 conn(0x556ce37916d0 msgr2=0x556ce3813b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.241+0000 7f1995110b80 1 -- 172.21.15.27:0/1621791997 shutdown_connections 2024-05-10T11:28:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.241+0000 7f1995110b80 1 -- 172.21.15.27:0/1621791997 wait complete. 2024-05-10T11:28:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.241+0000 7f1995110b80 1 -- start start 2024-05-10T11:28:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.241+0000 7f1995110b80 1 Processor -- start 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.242+0000 7f1995110b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce38198f0 0x556ce38250e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.242+0000 7f1995110b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce381ebc0 0x556ce3822480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.242+0000 7f1995110b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381f4e0 0x556ce38229c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.242+0000 7f1995110b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556ce3825f20 con 0x556ce381f4e0 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.242+0000 7f1995110b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x556ce36d1b20 con 0x556ce38198f0 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.242+0000 7f1995110b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556ce38260a0 con 0x556ce381ebc0 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.242+0000 7f1995110b80 10 client.0 _ll_register_callbacks cb 0x556ce3712c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.242+0000 7f1983400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce38198f0 0x556ce38250e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.243+0000 7f1983400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce38198f0 0x556ce38250e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41618/0 (socket says 172.21.15.27:41618) 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.243+0000 7f1983400700 1 -- 172.21.15.27:0/246644548 learned_addr learned my addr 172.21.15.27:0/246644548 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.243+0000 7f1983e00700 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381f4e0 0x556ce38229c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.243+0000 7f1983400700 1 -- 172.21.15.27:0/246644548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce381ebc0 msgr2=0x556ce3822480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:28:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.243+0000 7f1983400700 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce381ebc0 0x556ce3822480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.243+0000 7f1983400700 1 -- 172.21.15.27:0/246644548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381f4e0 msgr2=0x556ce38229c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.243+0000 7f1983400700 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381f4e0 0x556ce38229c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.243+0000 7f1983400700 1 -- 172.21.15.27:0/246644548 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ce3731420 con 0x556ce38198f0 2024-05-10T11:28:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.243+0000 7f1983e00700 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381f4e0 0x556ce38229c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:28:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.243+0000 7f1983400700 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce38198f0 0x556ce38250e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f197c009310 tx=0x7f197c019660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16f28e0a40ccc7ba server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.243+0000 7f196d600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:28:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.244+0000 7f196d600700 1 -- 172.21.15.27:0/246644548 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c01e050 con 0x556ce38198f0 2024-05-10T11:28:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.244+0000 7f196d600700 1 -- 172.21.15.27:0/246644548 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7f197c004020 con 0x556ce38198f0 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.244+0000 7f1995110b80 1 -- 172.21.15.27:0/246644548 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f197c019ae0 con 0x556ce38198f0 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.244+0000 7f196d600700 1 -- 172.21.15.27:0/246644548 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f197c032050 con 0x556ce38198f0 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.244+0000 7f1995110b80 10 client.27362 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.244+0000 7f1995110b80 1 -- 172.21.15.27:0/246644548 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x556ce3838f10 con 0x556ce38198f0 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.245+0000 7f196d600700 1 -- 172.21.15.27:0/246644548 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f197c019ae0 con 0x556ce38198f0 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.246+0000 7f196d600700 1 -- 172.21.15.27:0/246644548 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f197c004750 con 0x556ce38198f0 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.246+0000 7f1995110b80 10 client.27362 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.246+0000 7f1995110b80 10 client.27362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.246+0000 7f1995110b80 20 client.27362 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.246+0000 7f1995110b80 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556ce383b490 0x556ce383d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.246+0000 7f1995110b80 4 client.27362 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.246+0000 7f1995110b80 1 -- 172.21.15.27:0/246644548 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ce37e6140 con 0x556ce383b490 2024-05-10T11:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.246+0000 7f1982a00700 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556ce383b490 0x556ce383d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.247+0000 7f1982a00700 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556ce383b490 0x556ce383d880 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adefd65e0208d0c7 server_cookie=77ad80a260fc197a in_seq=0 out_seq=0 2024-05-10T11:28:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.247+0000 7f196d600700 10 client.27362 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:28:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.249+0000 7f196d600700 1 -- 172.21.15.27:0/246644548 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556ce37e6140 con 0x556ce383b490 2024-05-10T11:28:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.249+0000 7f196d600700 10 client.27362 handle_command_reply: tid=0 2024-05-10T11:28:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.249+0000 7f1995110b80 1 client.27362 shutdown 2024-05-10T11:28:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.250+0000 7f1995110b80 1 -- 172.21.15.27:0/246644548 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce38198f0 msgr2=0x556ce38250e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.250+0000 7f1995110b80 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce38198f0 0x556ce38250e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f197c009310 tx=0x7f197c019660 comp rx=0 tx=0).stop 2024-05-10T11:28:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.250+0000 7f1995110b80 1 -- 172.21.15.27:0/246644548 shutdown_connections 2024-05-10T11:28:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.250+0000 7f1995110b80 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556ce383b490 0x556ce383d880 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.250+0000 7f1995110b80 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381f4e0 0x556ce38229c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.250+0000 7f196d600700 0 client.27362 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.250+0000 7f1995110b80 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce38198f0 0x556ce38250e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.250+0000 7f1995110b80 1 --2- 172.21.15.27:0/246644548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce381ebc0 0x556ce3822480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.250+0000 7f1995110b80 1 -- 172.21.15.27:0/246644548 >> 172.21.15.27:0/246644548 conn(0x556ce37916d0 msgr2=0x556ce3813190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.250+0000 7f1995110b80 1 -- 172.21.15.27:0/246644548 shutdown_connections 2024-05-10T11:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.250+0000 7f1995110b80 1 -- 172.21.15.27:0/246644548 wait complete. 2024-05-10T11:28:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.251+0000 7f1995110b80 20 client.27362 trim_cache size 0 max 16384 2024-05-10T11:28:26.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.275+0000 7f1995110b80 1 Processor -- start 2024-05-10T11:28:26.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.275+0000 7f1995110b80 1 -- start start 2024-05-10T11:28:26.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.275+0000 7f1995110b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 0x556ce38213e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.275+0000 7f1995110b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce38198f0 0x556ce3813910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.275+0000 7f1995110b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381ebc0 0x556ce3824190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.275+0000 7f1995110b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f196400ef40 con 0x556ce381ebc0 2024-05-10T11:28:26.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.275+0000 7f1995110b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1964010b70 con 0x556ce37916d0 2024-05-10T11:28:26.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.275+0000 7f1995110b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f196400f610 con 0x556ce38198f0 2024-05-10T11:28:26.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.276+0000 7f1983400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 0x556ce38213e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.276+0000 7f1982a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce38198f0 0x556ce3813910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.276+0000 7f1983400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 0x556ce38213e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41634/0 (socket says 172.21.15.27:41634) 2024-05-10T11:28:26.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.276+0000 7f1983e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381ebc0 0x556ce3824190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.276+0000 7f1983400700 1 -- 172.21.15.27:0/3539005163 learned_addr learned my addr 172.21.15.27:0/3539005163 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.276+0000 7f1983400700 1 -- 172.21.15.27:0/3539005163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce38198f0 msgr2=0x556ce3813910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.276+0000 7f1983400700 1 --2- 172.21.15.27:0/3539005163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce38198f0 0x556ce3813910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.276+0000 7f1983400700 1 -- 172.21.15.27:0/3539005163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381ebc0 msgr2=0x556ce3824190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.276+0000 7f1983400700 1 --2- 172.21.15.27:0/3539005163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381ebc0 0x556ce3824190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.276+0000 7f1983400700 1 -- 172.21.15.27:0/3539005163 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ce3825e00 con 0x556ce37916d0 2024-05-10T11:28:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.277+0000 7f1983400700 1 --2- 172.21.15.27:0/3539005163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 0x556ce38213e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f197c004ba0 tx=0x7f197c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd12590f4ae4203c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.277+0000 7f196fe00700 1 -- 172.21.15.27:0/3539005163 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c01e040 con 0x556ce37916d0 2024-05-10T11:28:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.277+0000 7f196fe00700 1 -- 172.21.15.27:0/3539005163 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f197c01f040 con 0x556ce37916d0 2024-05-10T11:28:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.277+0000 7f196fe00700 1 -- 172.21.15.27:0/3539005163 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c0046d0 con 0x556ce37916d0 2024-05-10T11:28:26.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.278+0000 7f1995110b80 1 -- 172.21.15.27:0/3539005163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 msgr2=0x556ce38213e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:26.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.278+0000 7f1995110b80 1 --2- 172.21.15.27:0/3539005163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 0x556ce38213e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f197c004ba0 tx=0x7f197c011720 comp rx=0 tx=0).stop 2024-05-10T11:28:26.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.278+0000 7f1995110b80 1 -- 172.21.15.27:0/3539005163 shutdown_connections 2024-05-10T11:28:26.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.278+0000 7f1995110b80 1 --2- 172.21.15.27:0/3539005163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381ebc0 0x556ce3824190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.278+0000 7f1995110b80 1 --2- 172.21.15.27:0/3539005163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 0x556ce38213e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.278+0000 7f1995110b80 1 --2- 172.21.15.27:0/3539005163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce38198f0 0x556ce3813910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.278+0000 7f1995110b80 1 -- 172.21.15.27:0/3539005163 >> 172.21.15.27:0/3539005163 conn(0x556ce3810750 msgr2=0x556ce3823a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:26.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.278+0000 7f1995110b80 1 -- 172.21.15.27:0/3539005163 shutdown_connections 2024-05-10T11:28:26.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.278+0000 7f1995110b80 1 -- 172.21.15.27:0/3539005163 wait complete. 2024-05-10T11:28:26.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.279+0000 7f1995110b80 1 -- start start 2024-05-10T11:28:26.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.279+0000 7f1995110b80 1 Processor -- start 2024-05-10T11:28:26.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.279+0000 7f1995110b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 0x556ce38429b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.279+0000 7f1995110b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce38198f0 0x556ce3842ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1995110b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381ebc0 0x556ce3843430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1995110b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f19640103e0 con 0x556ce381ebc0 2024-05-10T11:28:26.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1995110b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f196400fd10 con 0x556ce37916d0 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1995110b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f196400cf00 con 0x556ce38198f0 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1995110b80 10 client.0 _ll_register_callbacks cb 0x556ce3712c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1983400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 0x556ce38429b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1983400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 0x556ce38429b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41638/0 (socket says 172.21.15.27:41638) 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1983400700 1 -- 172.21.15.27:0/2371726582 learned_addr learned my addr 172.21.15.27:0/2371726582 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1983e00700 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381ebc0 0x556ce3843430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1982a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce38198f0 0x556ce3842ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1983400700 1 -- 172.21.15.27:0/2371726582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce38198f0 msgr2=0x556ce3842ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1983400700 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce38198f0 0x556ce3842ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1983400700 1 -- 172.21.15.27:0/2371726582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381ebc0 msgr2=0x556ce3843430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1983400700 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381ebc0 0x556ce3843430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1983400700 1 -- 172.21.15.27:0/2371726582 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ce38a4d30 con 0x556ce37916d0 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.280+0000 7f1982a00700 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce38198f0 0x556ce3842ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:28:26.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.281+0000 7f1983400700 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 0x556ce38429b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f197c004c90 tx=0x7f197c019780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b13e73703135eeb server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:26.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.281+0000 7f196d600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:28:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.281+0000 7f196d600700 1 -- 172.21.15.27:0/2371726582 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c043060 con 0x556ce37916d0 2024-05-10T11:28:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.281+0000 7f1995110b80 1 -- 172.21.15.27:0/2371726582 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f197c01f040 con 0x556ce37916d0 2024-05-10T11:28:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.281+0000 7f196d600700 1 -- 172.21.15.27:0/2371726582 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7f197c0046d0 con 0x556ce37916d0 2024-05-10T11:28:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.282+0000 7f196d600700 1 -- 172.21.15.27:0/2371726582 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f197c04a050 con 0x556ce37916d0 2024-05-10T11:28:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.282+0000 7f1995110b80 10 client.27368 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.282+0000 7f1995110b80 1 -- 172.21.15.27:0/2371726582 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x556ce38a6010 con 0x556ce37916d0 2024-05-10T11:28:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.282+0000 7f196d600700 1 -- 172.21.15.27:0/2371726582 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f197c01f040 con 0x556ce37916d0 2024-05-10T11:28:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.282+0000 7f196d600700 1 -- 172.21.15.27:0/2371726582 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f197c0054b0 con 0x556ce37916d0 2024-05-10T11:28:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.283+0000 7f1995110b80 10 client.27368 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.283+0000 7f1995110b80 10 client.27368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:28:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.283+0000 7f1995110b80 20 client.27368 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.283+0000 7f1995110b80 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556ce381f4e0 0x556ce38a86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:26.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.283+0000 7f1995110b80 4 client.27368 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:28:26.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.283+0000 7f1995110b80 1 -- 172.21.15.27:0/2371726582 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556ce3779400 con 0x556ce381f4e0 2024-05-10T11:28:26.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.283+0000 7f1982a00700 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556ce381f4e0 0x556ce38a86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:26.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.284+0000 7f1982a00700 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556ce381f4e0 0x556ce38a86a0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a3e87ed35a346bb server_cookie=b4e1c3fdd0906827 in_seq=0 out_seq=0 2024-05-10T11:28:26.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.284+0000 7f196d600700 10 client.27368 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:28:26.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.287+0000 7f196d600700 1 -- 172.21.15.27:0/2371726582 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556ce3779400 con 0x556ce381f4e0 2024-05-10T11:28:26.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.287+0000 7f196d600700 10 client.27368 handle_command_reply: tid=0 2024-05-10T11:28:26.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.287+0000 7f1995110b80 1 client.27368 shutdown 2024-05-10T11:28:26.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.287+0000 7f1995110b80 1 -- 172.21.15.27:0/2371726582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 msgr2=0x556ce38429b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:26.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.287+0000 7f1995110b80 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 0x556ce38429b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f197c004c90 tx=0x7f197c019780 comp rx=0 tx=0).stop 2024-05-10T11:28:26.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.287+0000 7f1995110b80 1 -- 172.21.15.27:0/2371726582 shutdown_connections 2024-05-10T11:28:26.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.287+0000 7f1995110b80 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556ce381f4e0 0x556ce38a86a0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.287+0000 7f1995110b80 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ce381ebc0 0x556ce3843430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.287+0000 7f1995110b80 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556ce37916d0 0x556ce38429b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.287+0000 7f1995110b80 1 --2- 172.21.15.27:0/2371726582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ce38198f0 0x556ce3842ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.287+0000 7f196d600700 0 client.27368 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:28:26.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.287+0000 7f1995110b80 1 -- 172.21.15.27:0/2371726582 >> 172.21.15.27:0/2371726582 conn(0x556ce3810750 msgr2=0x556ce38225c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:26.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.288+0000 7f1995110b80 1 -- 172.21.15.27:0/2371726582 shutdown_connections 2024-05-10T11:28:26.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.288+0000 7f1995110b80 1 -- 172.21.15.27:0/2371726582 wait complete. 2024-05-10T11:28:26.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.288+0000 7f1995110b80 20 client.27368 trim_cache size 0 max 16384 2024-05-10T11:28:26.289 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:28:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.289+0000 7f198a800700 1 -- 172.21.15.27:0/1233294792 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f19680b2b40 msgr2=0x7f19680b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.290+0000 7f198a800700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f19680b2b40 0x7f19680b5000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1970009fd0 tx=0x7f1970009410 comp rx=0 tx=0).stop 2024-05-10T11:28:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.290+0000 7f198a800700 1 -- 172.21.15.27:0/1233294792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f198410b6e0 msgr2=0x7f19841a0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.290+0000 7f198a800700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f198410b6e0 0x7f19841a0600 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f19740098e0 tx=0x7f19740047b0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.290+0000 7f198a800700 1 -- 172.21.15.27:0/1233294792 shutdown_connections 2024-05-10T11:28:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.290+0000 7f198a800700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f19680b2b40 0x7f19680b5000 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.290+0000 7f198a800700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f198410b6e0 0x7f19841a0600 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.290+0000 7f198a800700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19841056e0 0x7f198419bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.290+0000 7f198a800700 1 --2- 172.21.15.27:0/1233294792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19841060a0 0x7f198419c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.290+0000 7f198a800700 1 -- 172.21.15.27:0/1233294792 >> 172.21.15.27:0/1233294792 conn(0x7f19841011d0 msgr2=0x7f1984101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.291+0000 7f198a800700 1 -- 172.21.15.27:0/1233294792 shutdown_connections 2024-05-10T11:28:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:26.291+0000 7f198a800700 1 -- 172.21.15.27:0/1233294792 wait complete. 2024-05-10T11:28:27.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:26 smithi027 ceph-mon[30054]: pgmap v477: 129 pgs: 129 active+clean; 896 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 12 MiB/s wr, 627 op/s 2024-05-10T11:28:27.145 INFO:tasks.workunit.client.0.smithi027.stdout: 10 32094 26.84 MB/sec warmup 71 sec latency 610.165 ms 2024-05-10T11:28:27.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:26 smithi076 ceph-mon[34152]: pgmap v477: 129 pgs: 129 active+clean; 896 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 12 MiB/s wr, 627 op/s 2024-05-10T11:28:27.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:26 smithi189 ceph-mon[34842]: pgmap v477: 129 pgs: 129 active+clean; 896 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 12 MiB/s wr, 627 op/s 2024-05-10T11:28:28.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 32437 26.70 MB/sec warmup 72 sec latency 696.530 ms 2024-05-10T11:28:28.619 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:28:28.620 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:28:29.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:28 smithi027 ceph-mon[30054]: pgmap v478: 129 pgs: 129 active+clean; 896 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 8.3 MiB/s wr, 428 op/s 2024-05-10T11:28:29.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 32839 26.61 MB/sec warmup 73 sec latency 242.058 ms 2024-05-10T11:28:29.164 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:28:29.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:28 smithi076 ceph-mon[34152]: pgmap v478: 129 pgs: 129 active+clean; 896 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 8.3 MiB/s wr, 428 op/s 2024-05-10T11:28:29.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:28 smithi189 ceph-mon[34842]: pgmap v478: 129 pgs: 129 active+clean; 896 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 8.3 MiB/s wr, 428 op/s 2024-05-10T11:28:30.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 33398 26.70 MB/sec warmup 74 sec latency 60.106 ms 2024-05-10T11:28:31.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:30 smithi027 ceph-mon[30054]: pgmap v479: 129 pgs: 129 active+clean; 914 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 13 MiB/s wr, 651 op/s 2024-05-10T11:28:31.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 33852 26.61 MB/sec warmup 75 sec latency 259.686 ms 2024-05-10T11:28:31.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:30 smithi076 ceph-mon[34152]: pgmap v479: 129 pgs: 129 active+clean; 914 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 13 MiB/s wr, 651 op/s 2024-05-10T11:28:31.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:30 smithi189 ceph-mon[34842]: pgmap v479: 129 pgs: 129 active+clean; 914 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 13 MiB/s wr, 651 op/s 2024-05-10T11:28:32.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 34216 26.48 MB/sec warmup 76 sec latency 272.423 ms 2024-05-10T11:28:32.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.872+0000 7f5690200700 1 -- 172.21.15.27:0/2408012687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5688103750 msgr2=0x7f5688103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:32.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.872+0000 7f5690200700 1 --2- 172.21.15.27:0/2408012687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5688103750 0x7f5688103b30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f567c009990 tx=0x7f567c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.872+0000 7f5690200700 1 -- 172.21.15.27:0/2408012687 shutdown_connections 2024-05-10T11:28:32.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.872+0000 7f5690200700 1 --2- 172.21.15.27:0/2408012687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5688103750 0x7f5688103b30 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.872+0000 7f5690200700 1 --2- 172.21.15.27:0/2408012687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5688108850 0x7f56881103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.872+0000 7f5690200700 1 --2- 172.21.15.27:0/2408012687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5688104100 0x7f5688108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.872+0000 7f5690200700 1 -- 172.21.15.27:0/2408012687 >> 172.21.15.27:0/2408012687 conn(0x7f568807a600 msgr2=0x7f568807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:32.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.873+0000 7f5690200700 1 -- 172.21.15.27:0/2408012687 shutdown_connections 2024-05-10T11:28:32.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.873+0000 7f5690200700 1 -- 172.21.15.27:0/2408012687 wait complete. 2024-05-10T11:28:32.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.874+0000 7f5690200700 1 Processor -- start 2024-05-10T11:28:32.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.874+0000 7f5690200700 1 -- start start 2024-05-10T11:28:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f5690200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5688103750 0x7f56881a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f5690200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5688104100 0x7f56881a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f5690200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5688108850 0x7f568810f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f5690200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5688113670 con 0x7f5688103750 2024-05-10T11:28:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f5690200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f56881134f0 con 0x7f5688108850 2024-05-10T11:28:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f5690200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f56881137f0 con 0x7f5688104100 2024-05-10T11:28:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f568d800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5688103750 0x7f56881a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f568ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5688104100 0x7f56881a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f568d800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5688103750 0x7f56881a5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54702/0 (socket says 172.21.15.27:54702) 2024-05-10T11:28:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f568e200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5688108850 0x7f568810f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f568d800700 1 -- 172.21.15.27:0/4281780161 learned_addr learned my addr 172.21.15.27:0/4281780161 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f568e200700 1 -- 172.21.15.27:0/4281780161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5688104100 msgr2=0x7f56881a6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:32.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f568e200700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5688104100 0x7f56881a6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f568e200700 1 -- 172.21.15.27:0/4281780161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5688103750 msgr2=0x7f56881a5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:32.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f568e200700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5688103750 0x7f56881a5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f568e200700 1 -- 172.21.15.27:0/4281780161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568810fc60 con 0x7f5688108850 2024-05-10T11:28:32.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.875+0000 7f568ce00700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5688104100 0x7f56881a6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:28:32.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.876+0000 7f568d800700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5688103750 0x7f56881a5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:28:32.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.876+0000 7f568e200700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5688108850 0x7f568810f490 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f568000efc0 tx=0x7f568000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:32.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.876+0000 7f5686000700 1 -- 172.21.15.27:0/4281780161 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56800092e0 con 0x7f5688108850 2024-05-10T11:28:32.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.876+0000 7f5686000700 1 -- 172.21.15.27:0/4281780161 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5680004bb0 con 0x7f5688108850 2024-05-10T11:28:32.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.876+0000 7f5690200700 1 -- 172.21.15.27:0/4281780161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f568810fef0 con 0x7f5688108850 2024-05-10T11:28:32.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.876+0000 7f5686000700 1 -- 172.21.15.27:0/4281780161 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5680004020 con 0x7f5688108850 2024-05-10T11:28:32.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.876+0000 7f5690200700 1 -- 172.21.15.27:0/4281780161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56881acc20 con 0x7f5688108850 2024-05-10T11:28:32.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.881+0000 7f5686000700 1 -- 172.21.15.27:0/4281780161 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f568001b030 con 0x7f5688108850 2024-05-10T11:28:32.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.881+0000 7f569abf6b80 1 Processor -- start 2024-05-10T11:28:32.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.881+0000 7f569abf6b80 1 -- start start 2024-05-10T11:28:32.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.882+0000 7f569abf6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296af54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.882+0000 7f569abf6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 0x558296af9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.882+0000 7f569abf6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296af9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.882+0000 7f569abf6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558296a47aa0 con 0x558296af9a50 2024-05-10T11:28:32.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.882+0000 7f569abf6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5582969e57e0 con 0x558296af50e0 2024-05-10T11:28:32.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.882+0000 7f569abf6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558296a43a60 con 0x558296af5a00 2024-05-10T11:28:32.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.882+0000 7f568d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 0x558296af9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.882+0000 7f568d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 0x558296af9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:44664/0 (socket says 172.21.15.27:44664) 2024-05-10T11:28:32.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.882+0000 7f568d800700 1 -- 172.21.15.27:0/4104792911 learned_addr learned my addr 172.21.15.27:0/4104792911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:32.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.882+0000 7f568ce00700 1 --2- 172.21.15.27:0/4104792911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296af54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.882+0000 7f568ce00700 1 --2- 172.21.15.27:0/4104792911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296af9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.883+0000 7f568d800700 1 -- 172.21.15.27:0/4104792911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 msgr2=0x558296af54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:32.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.883+0000 7f568d800700 1 --2- 172.21.15.27:0/4104792911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296af54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.883+0000 7f568d800700 1 -- 172.21.15.27:0/4104792911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 msgr2=0x558296af9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:32.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.883+0000 7f568d800700 1 --2- 172.21.15.27:0/4104792911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296af9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.883+0000 7f568d800700 1 -- 172.21.15.27:0/4104792911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558296994950 con 0x558296af5a00 2024-05-10T11:28:32.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.883+0000 7f5686000700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f56700b29f0 0x7f56700b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.883+0000 7f568d800700 1 --2- 172.21.15.27:0/4104792911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 0x558296af9510 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f567c005c70 tx=0x7f567c009590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1aa58a1de987a3a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.883+0000 7f5686000700 1 -- 172.21.15.27:0/4281780161 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7f56800e2500 con 0x7f5688108850 2024-05-10T11:28:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.883+0000 7f567aa00700 1 -- 172.21.15.27:0/4104792911 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c027070 con 0x558296af5a00 2024-05-10T11:28:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.883+0000 7f567aa00700 1 -- 172.21.15.27:0/4104792911 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f567c003680 con 0x558296af5a00 2024-05-10T11:28:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.883+0000 7f567aa00700 1 -- 172.21.15.27:0/4104792911 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c00f6c0 con 0x558296af5a00 2024-05-10T11:28:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.884+0000 7f568ce00700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f56700b29f0 0x7f56700b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.884+0000 7f569abf6b80 1 -- 172.21.15.27:0/4104792911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 msgr2=0x558296af9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.884+0000 7f568ce00700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f56700b29f0 0x7f56700b4eb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5674009fd0 tx=0x7f5674009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.884+0000 7f569abf6b80 1 --2- 172.21.15.27:0/4104792911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 0x558296af9510 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f567c005c70 tx=0x7f567c009590 comp rx=0 tx=0).stop 2024-05-10T11:28:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.885+0000 7f569abf6b80 1 -- 172.21.15.27:0/4104792911 shutdown_connections 2024-05-10T11:28:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.885+0000 7f569abf6b80 1 --2- 172.21.15.27:0/4104792911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296af9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.885+0000 7f569abf6b80 1 --2- 172.21.15.27:0/4104792911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296af54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.885+0000 7f569abf6b80 1 --2- 172.21.15.27:0/4104792911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 0x558296af9510 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.885+0000 7f569abf6b80 1 -- 172.21.15.27:0/4104792911 >> 172.21.15.27:0/4104792911 conn(0x558296a23be0 msgr2=0x558296aef820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.885+0000 7f569abf6b80 1 -- 172.21.15.27:0/4104792911 shutdown_connections 2024-05-10T11:28:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.885+0000 7f569abf6b80 1 -- 172.21.15.27:0/4104792911 wait complete. 2024-05-10T11:28:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.885+0000 7f569abf6b80 1 -- start start 2024-05-10T11:28:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.886+0000 7f569abf6b80 1 Processor -- start 2024-05-10T11:28:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.887+0000 7f569abf6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296b00eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.887+0000 7f569abf6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 0x558296afe250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.887+0000 7f569abf6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296afe790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.887+0000 7f569abf6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558296b01c00 con 0x558296af9a50 2024-05-10T11:28:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.887+0000 7f569abf6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5582969adba0 con 0x558296af50e0 2024-05-10T11:28:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.887+0000 7f569abf6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558296b01d80 con 0x558296af5a00 2024-05-10T11:28:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.887+0000 7f569abf6b80 10 client.0 _ll_register_callbacks cb 0x5582969eecf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.888+0000 7f568d800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296b00eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.888+0000 7f568d800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296b00eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41680/0 (socket says 172.21.15.27:41680) 2024-05-10T11:28:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.888+0000 7f568d800700 1 -- 172.21.15.27:0/2131977710 learned_addr learned my addr 172.21.15.27:0/2131977710 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.889+0000 7f568e200700 1 --2- 172.21.15.27:0/2131977710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296afe790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.889+0000 7f568e200700 1 -- 172.21.15.27:0/2131977710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 msgr2=0x558296afe250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.889+0000 7f568e200700 1 --2- 172.21.15.27:0/2131977710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 0x558296afe250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.889+0000 7f568e200700 1 -- 172.21.15.27:0/2131977710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 msgr2=0x558296b00eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.889+0000 7f568e200700 1 --2- 172.21.15.27:0/2131977710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296b00eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.889+0000 7f568e200700 1 -- 172.21.15.27:0/2131977710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558296a0d4a0 con 0x558296af9a50 2024-05-10T11:28:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.890+0000 7f568e200700 1 --2- 172.21.15.27:0/2131977710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296afe790 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f56800df820 tx=0x7f56800aa830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76cb0b1bbd624c31 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.890+0000 7f566f400700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:28:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.890+0000 7f566f400700 1 -- 172.21.15.27:0/2131977710 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56800aa2f0 con 0x558296af9a50 2024-05-10T11:28:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.890+0000 7f569abf6b80 1 -- 172.21.15.27:0/2131977710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f567c003680 con 0x558296af9a50 2024-05-10T11:28:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.890+0000 7f566f400700 1 -- 172.21.15.27:0/2131977710 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7f56800e2160 con 0x558296af9a50 2024-05-10T11:28:32.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.891+0000 7f566f400700 1 -- 172.21.15.27:0/2131977710 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f56800e2360 con 0x558296af9a50 2024-05-10T11:28:32.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.891+0000 7f569abf6b80 10 client.17094 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:32.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.891+0000 7f569abf6b80 1 -- 172.21.15.27:0/2131977710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x558296b1d500 con 0x558296af9a50 2024-05-10T11:28:32.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.892+0000 7f566f400700 1 -- 172.21.15.27:0/2131977710 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f567c003680 con 0x558296af9a50 2024-05-10T11:28:32.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.892+0000 7f566f400700 1 -- 172.21.15.27:0/2131977710 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f56800a5c60 con 0x558296af9a50 2024-05-10T11:28:32.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.893+0000 7f569abf6b80 10 client.17094 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:32.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.893+0000 7f569abf6b80 10 client.17094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:28:32.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.893+0000 7f569abf6b80 20 client.17094 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:32.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.893+0000 7f569abf6b80 1 --2- 172.21.15.27:0/2131977710 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558296b1fa90 0x558296b21e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.893+0000 7f569abf6b80 4 client.17094 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:28:32.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.893+0000 7f569abf6b80 1 -- 172.21.15.27:0/2131977710 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558296ac2120 con 0x558296b1fa90 2024-05-10T11:28:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.893+0000 7f568d800700 1 --2- 172.21.15.27:0/2131977710 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558296b1fa90 0x558296b21e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.894+0000 7f566f400700 10 client.17094 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:28:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.895+0000 7f568d800700 1 --2- 172.21.15.27:0/2131977710 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558296b1fa90 0x558296b21e80 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1be5109247b2201d server_cookie=292b5fbb98874c3a in_seq=0 out_seq=0 2024-05-10T11:28:32.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.900+0000 7f566f400700 1 -- 172.21.15.27:0/2131977710 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558296ac2120 con 0x558296b1fa90 2024-05-10T11:28:32.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.900+0000 7f566f400700 10 client.17094 handle_command_reply: tid=0 2024-05-10T11:28:32.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:32 smithi027 ceph-mon[30054]: pgmap v480: 129 pgs: 129 active+clean; 914 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.6 MiB/s wr, 430 op/s 2024-05-10T11:28:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.901+0000 7f569abf6b80 1 client.17094 shutdown 2024-05-10T11:28:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.902+0000 7f569abf6b80 1 -- 172.21.15.27:0/2131977710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 msgr2=0x558296afe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.902+0000 7f569abf6b80 1 --2- 172.21.15.27:0/2131977710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296afe790 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f56800df820 tx=0x7f56800aa830 comp rx=0 tx=0).stop 2024-05-10T11:28:32.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.902+0000 7f569abf6b80 1 -- 172.21.15.27:0/2131977710 shutdown_connections 2024-05-10T11:28:32.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.902+0000 7f569abf6b80 1 --2- 172.21.15.27:0/2131977710 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558296b1fa90 0x558296b21e80 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.902+0000 7f569abf6b80 1 --2- 172.21.15.27:0/2131977710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296afe790 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.902+0000 7f569abf6b80 1 --2- 172.21.15.27:0/2131977710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296b00eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.902+0000 7f569abf6b80 1 --2- 172.21.15.27:0/2131977710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 0x558296afe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.902+0000 7f569abf6b80 1 -- 172.21.15.27:0/2131977710 >> 172.21.15.27:0/2131977710 conn(0x558296a23be0 msgr2=0x558296aef7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:32.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.902+0000 7f566f400700 0 client.17094 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:28:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.904+0000 7f569abf6b80 1 -- 172.21.15.27:0/2131977710 shutdown_connections 2024-05-10T11:28:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.904+0000 7f569abf6b80 1 -- 172.21.15.27:0/2131977710 wait complete. 2024-05-10T11:28:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.905+0000 7f569abf6b80 20 client.17094 trim_cache size 0 max 16384 2024-05-10T11:28:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.943+0000 7f569abf6b80 1 Processor -- start 2024-05-10T11:28:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.943+0000 7f569abf6b80 1 -- start start 2024-05-10T11:28:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.944+0000 7f569abf6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296afd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.944+0000 7f569abf6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 0x558296b27790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.944+0000 7f569abf6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296b27cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.944+0000 7f569abf6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f56881ad4e0 con 0x558296af9a50 2024-05-10T11:28:32.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.944+0000 7f569abf6b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f56881af110 con 0x558296af50e0 2024-05-10T11:28:32.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.944+0000 7f569abf6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f56881adbb0 con 0x558296af5a00 2024-05-10T11:28:32.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.944+0000 7f568d800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296afd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.944+0000 7f568d800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296afd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:41692/0 (socket says 172.21.15.27:41692) 2024-05-10T11:28:32.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.944+0000 7f568d800700 1 -- 172.21.15.27:0/451751418 learned_addr learned my addr 172.21.15.27:0/451751418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:32.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.945+0000 7f568d800700 1 -- 172.21.15.27:0/451751418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 msgr2=0x558296b27790 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:28:32.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.945+0000 7f568d800700 1 --2- 172.21.15.27:0/451751418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 0x558296b27790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.945+0000 7f568d800700 1 -- 172.21.15.27:0/451751418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 msgr2=0x558296b27cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:28:32.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.945+0000 7f568d800700 1 --2- 172.21.15.27:0/451751418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296b27cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.945+0000 7f568d800700 1 -- 172.21.15.27:0/451751418 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558296af0a40 con 0x558296af50e0 2024-05-10T11:28:32.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.946+0000 7f568d800700 1 --2- 172.21.15.27:0/451751418 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296afd1f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f567c00be80 tx=0x7f567c01a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=834a6df589cabbae server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:32.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.946+0000 7f567a000700 1 -- 172.21.15.27:0/451751418 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c027040 con 0x558296af50e0 2024-05-10T11:28:32.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.946+0000 7f567a000700 1 -- 172.21.15.27:0/451751418 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f567c028040 con 0x558296af50e0 2024-05-10T11:28:32.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.946+0000 7f567a000700 1 -- 172.21.15.27:0/451751418 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c0315b0 con 0x558296af50e0 2024-05-10T11:28:32.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.947+0000 7f569abf6b80 1 -- 172.21.15.27:0/451751418 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 msgr2=0x558296afd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:32.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.947+0000 7f569abf6b80 1 --2- 172.21.15.27:0/451751418 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296afd1f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f567c00be80 tx=0x7f567c01a040 comp rx=0 tx=0).stop 2024-05-10T11:28:32.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.947+0000 7f569abf6b80 1 -- 172.21.15.27:0/451751418 shutdown_connections 2024-05-10T11:28:32.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.947+0000 7f569abf6b80 1 --2- 172.21.15.27:0/451751418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296b27cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.947+0000 7f569abf6b80 1 --2- 172.21.15.27:0/451751418 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af50e0 0x558296afd1f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.947+0000 7f569abf6b80 1 --2- 172.21.15.27:0/451751418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af5a00 0x558296b27790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.947+0000 7f569abf6b80 1 -- 172.21.15.27:0/451751418 >> 172.21.15.27:0/451751418 conn(0x558296a23be0 msgr2=0x558296aea330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.947+0000 7f569abf6b80 1 -- 172.21.15.27:0/451751418 shutdown_connections 2024-05-10T11:28:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.948+0000 7f569abf6b80 1 -- 172.21.15.27:0/451751418 wait complete. 2024-05-10T11:28:32.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.949+0000 7f569abf6b80 1 -- start start 2024-05-10T11:28:32.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.949+0000 7f569abf6b80 1 Processor -- start 2024-05-10T11:28:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.951+0000 7f569abf6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af50e0 0x558296b03d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.951+0000 7f569abf6b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af5a00 0x558296b042a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.951+0000 7f569abf6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296b047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.951+0000 7f568d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af50e0 0x558296b03d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.951+0000 7f568d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af50e0 0x558296b03d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:44694/0 (socket says 172.21.15.27:44694) 2024-05-10T11:28:32.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.951+0000 7f568d800700 1 -- 172.21.15.27:0/2135345670 learned_addr learned my addr 172.21.15.27:0/2135345670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:32.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.951+0000 7f569abf6b80 1 -- 172.21.15.27:0/2135345670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f56881ae980 con 0x558296af9a50 2024-05-10T11:28:32.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.951+0000 7f569abf6b80 1 -- 172.21.15.27:0/2135345670 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f56881ae2b0 con 0x558296af5a00 2024-05-10T11:28:32.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.951+0000 7f569abf6b80 1 -- 172.21.15.27:0/2135345670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f568819f980 con 0x558296af50e0 2024-05-10T11:28:32.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.952+0000 7f569abf6b80 10 client.0 _ll_register_callbacks cb 0x5582969eecf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:32.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.952+0000 7f568ce00700 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af5a00 0x558296b042a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.952+0000 7f568d800700 1 -- 172.21.15.27:0/2135345670 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af5a00 msgr2=0x558296b042a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:32.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.952+0000 7f568d800700 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af5a00 0x558296b042a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.952+0000 7f568d800700 1 -- 172.21.15.27:0/2135345670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 msgr2=0x558296b047e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:32.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.952+0000 7f568e200700 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296b047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.952+0000 7f568d800700 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296b047e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.952+0000 7f568d800700 1 -- 172.21.15.27:0/2135345670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558296b8a510 con 0x558296af50e0 2024-05-10T11:28:32.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.952+0000 7f568e200700 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296b047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:28:32.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.953+0000 7f568d800700 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af50e0 0x558296b03d60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f567c001eb0 tx=0x7f567c003d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60614478a2291f70 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:32.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.953+0000 7f566f400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:28:32.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.953+0000 7f566f400700 1 -- 172.21.15.27:0/2135345670 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c0054e0 con 0x558296af50e0 2024-05-10T11:28:32.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.954+0000 7f566f400700 1 -- 172.21.15.27:0/2135345670 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7f567c031230 con 0x558296af50e0 2024-05-10T11:28:32.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.954+0000 7f569abf6b80 1 -- 172.21.15.27:0/2135345670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f567c028040 con 0x558296af50e0 2024-05-10T11:28:32.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.955+0000 7f566f400700 1 -- 172.21.15.27:0/2135345670 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f567c040050 con 0x558296af50e0 2024-05-10T11:28:32.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.956+0000 7f569abf6b80 10 client.27373 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:32.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.956+0000 7f569abf6b80 1 -- 172.21.15.27:0/2135345670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x558296b8b810 con 0x558296af50e0 2024-05-10T11:28:32.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.956+0000 7f566f400700 1 -- 172.21.15.27:0/2135345670 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f567c028040 con 0x558296af50e0 2024-05-10T11:28:32.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.957+0000 7f566f400700 1 -- 172.21.15.27:0/2135345670 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f567c021030 con 0x558296af50e0 2024-05-10T11:28:32.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.957+0000 7f569abf6b80 10 client.27373 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:32.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.957+0000 7f569abf6b80 10 client.27373 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:28:32.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.957+0000 7f569abf6b80 20 client.27373 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:32.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.957+0000 7f569abf6b80 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558296b8ba90 0x558296b8de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:32.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.957+0000 7f569abf6b80 4 client.27373 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:28:32.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.957+0000 7f569abf6b80 1 -- 172.21.15.27:0/2135345670 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558296a55480 con 0x558296b8ba90 2024-05-10T11:28:32.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.958+0000 7f568ce00700 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558296b8ba90 0x558296b8de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:32.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.960+0000 7f568ce00700 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558296b8ba90 0x558296b8de80 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60fa2104073507e server_cookie=c8f1e82a5fbe3a65 in_seq=0 out_seq=0 2024-05-10T11:28:32.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.960+0000 7f566f400700 10 client.27373 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:28:32.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.969+0000 7f566f400700 1 -- 172.21.15.27:0/2135345670 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558296a55480 con 0x558296b8ba90 2024-05-10T11:28:32.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.969+0000 7f566f400700 10 client.27373 handle_command_reply: tid=0 2024-05-10T11:28:32.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.969+0000 7f569abf6b80 1 client.27373 shutdown 2024-05-10T11:28:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.971+0000 7f569abf6b80 1 -- 172.21.15.27:0/2135345670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af50e0 msgr2=0x558296b03d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.971+0000 7f569abf6b80 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af50e0 0x558296b03d60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f567c001eb0 tx=0x7f567c003d20 comp rx=0 tx=0).stop 2024-05-10T11:28:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.971+0000 7f569abf6b80 1 -- 172.21.15.27:0/2135345670 shutdown_connections 2024-05-10T11:28:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.971+0000 7f569abf6b80 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x558296b8ba90 0x558296b8de80 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.971+0000 7f569abf6b80 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558296af50e0 0x558296b03d60 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.971+0000 7f569abf6b80 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558296af9a50 0x558296b047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.971+0000 7f569abf6b80 1 --2- 172.21.15.27:0/2135345670 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558296af5a00 0x558296b042a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.971+0000 7f569abf6b80 1 -- 172.21.15.27:0/2135345670 >> 172.21.15.27:0/2135345670 conn(0x558296a23be0 msgr2=0x558296af3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.972+0000 7f566f400700 0 client.27373 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:28:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.972+0000 7f569abf6b80 1 -- 172.21.15.27:0/2135345670 shutdown_connections 2024-05-10T11:28:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.972+0000 7f569abf6b80 1 -- 172.21.15.27:0/2135345670 wait complete. 2024-05-10T11:28:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.972+0000 7f569abf6b80 20 client.27373 trim_cache size 0 max 16384 2024-05-10T11:28:32.974 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:28:32.974 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:28:32.974 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "337f5dbe-3d1c-47cb-8ba5-a5f0b4324c13", 2024-05-10T11:28:32.974 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:28:32.974 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:28:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.974+0000 7f5690200700 1 -- 172.21.15.27:0/4281780161 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f56700b29f0 msgr2=0x7f56700b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.974+0000 7f5690200700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f56700b29f0 0x7f56700b4eb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5674009fd0 tx=0x7f5674009260 comp rx=0 tx=0).stop 2024-05-10T11:28:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.974+0000 7f5690200700 1 -- 172.21.15.27:0/4281780161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5688108850 msgr2=0x7f568810f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.974+0000 7f5690200700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5688108850 0x7f568810f490 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f568000efc0 tx=0x7f568000c3a0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.975+0000 7f5690200700 1 -- 172.21.15.27:0/4281780161 shutdown_connections 2024-05-10T11:28:32.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.975+0000 7f5690200700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f56700b29f0 0x7f56700b4eb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.975+0000 7f5690200700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5688108850 0x7f568810f490 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.975+0000 7f5690200700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5688103750 0x7f56881a5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.975+0000 7f5690200700 1 --2- 172.21.15.27:0/4281780161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5688104100 0x7f56881a6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:32.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.975+0000 7f5690200700 1 -- 172.21.15.27:0/4281780161 >> 172.21.15.27:0/4281780161 conn(0x7f568807a600 msgr2=0x7f56881018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:32.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.975+0000 7f5690200700 1 -- 172.21.15.27:0/4281780161 shutdown_connections 2024-05-10T11:28:32.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:32.975+0000 7f5690200700 1 -- 172.21.15.27:0/4281780161 wait complete. 2024-05-10T11:28:33.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 34600 26.51 MB/sec warmup 77 sec latency 349.516 ms 2024-05-10T11:28:33.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:32 smithi076 ceph-mon[34152]: pgmap v480: 129 pgs: 129 active+clean; 914 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.6 MiB/s wr, 430 op/s 2024-05-10T11:28:33.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:32 smithi189 ceph-mon[34842]: pgmap v480: 129 pgs: 129 active+clean; 914 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.6 MiB/s wr, 430 op/s 2024-05-10T11:28:33.960 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:28:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:28:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:28:34.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:33 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 35043 26.47 MB/sec warmup 78 sec latency 313.786 ms 2024-05-10T11:28:34.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:28:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:28:34.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:33 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:28:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.239 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.239 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.239 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.239 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.239 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.239 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.239 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.239 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:28:34.239 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:33 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:34.439 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:28:34.856 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:34 smithi027 ceph-mon[30054]: pgmap v481: 129 pgs: 129 active+clean; 914 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.6 MiB/s wr, 430 op/s 2024-05-10T11:28:34.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:34 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:35.147 INFO:tasks.workunit.client.0.smithi027.stdout: 10 35634 26.45 MB/sec warmup 79 sec latency 212.018 ms 2024-05-10T11:28:35.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:34 smithi076 ceph-mon[34152]: pgmap v481: 129 pgs: 129 active+clean; 914 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.6 MiB/s wr, 430 op/s 2024-05-10T11:28:35.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:34 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:35.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:34 smithi189 ceph-mon[34842]: pgmap v481: 129 pgs: 129 active+clean; 914 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.6 MiB/s wr, 430 op/s 2024-05-10T11:28:35.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:34 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:28:36.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 36148 26.57 MB/sec warmup 80 sec latency 95.842 ms 2024-05-10T11:28:37.102 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:36 smithi027 ceph-mon[30054]: pgmap v482: 129 pgs: 129 active+clean; 929 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 613 op/s 2024-05-10T11:28:37.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 36639 26.61 MB/sec warmup 81 sec latency 102.690 ms 2024-05-10T11:28:37.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:36 smithi076 ceph-mon[34152]: pgmap v482: 129 pgs: 129 active+clean; 929 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 613 op/s 2024-05-10T11:28:37.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:36 smithi189 ceph-mon[34842]: pgmap v482: 129 pgs: 129 active+clean; 929 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 613 op/s 2024-05-10T11:28:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.474+0000 7f474ba00700 1 -- 172.21.15.27:0/3051875224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4744108a80 msgr2=0x7f474410f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.474+0000 7f474ba00700 1 --2- 172.21.15.27:0/3051875224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4744108a80 0x7f474410f240 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f473c00b0d0 tx=0x7f473c00b3e0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.474+0000 7f474ba00700 1 -- 172.21.15.27:0/3051875224 shutdown_connections 2024-05-10T11:28:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.474+0000 7f474ba00700 1 --2- 172.21.15.27:0/3051875224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47441073d0 0x7f47441077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.474+0000 7f474ba00700 1 --2- 172.21.15.27:0/3051875224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4744108a80 0x7f474410f240 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.474+0000 7f474ba00700 1 --2- 172.21.15.27:0/3051875224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47441080c0 0x7f4744108540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.474+0000 7f474ba00700 1 -- 172.21.15.27:0/3051875224 >> 172.21.15.27:0/3051875224 conn(0x7f4744077740 msgr2=0x7f4744077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:37.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.474+0000 7f474ba00700 1 -- 172.21.15.27:0/3051875224 shutdown_connections 2024-05-10T11:28:37.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.475+0000 7f474ba00700 1 -- 172.21.15.27:0/3051875224 wait complete. 2024-05-10T11:28:37.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.476+0000 7f474ba00700 1 Processor -- start 2024-05-10T11:28:37.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.476+0000 7f474ba00700 1 -- start start 2024-05-10T11:28:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.476+0000 7f474ba00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47441073d0 0x7f47441a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.476+0000 7f474ba00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47441080c0 0x7f47441a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.476+0000 7f474ba00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4744108a80 0x7f474419bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.476+0000 7f474ba00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f47441125a0 con 0x7f47441073d0 2024-05-10T11:28:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.476+0000 7f474ba00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4744112420 con 0x7f4744108a80 2024-05-10T11:28:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.476+0000 7f474ba00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4744112720 con 0x7f47441080c0 2024-05-10T11:28:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.477+0000 7f4743e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47441080c0 0x7f47441a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.477+0000 7f4749000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47441073d0 0x7f47441a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.477+0000 7f4743e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47441080c0 0x7f47441a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54754/0 (socket says 172.21.15.27:54754) 2024-05-10T11:28:37.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.477+0000 7f4743e00700 1 -- 172.21.15.27:0/850247344 learned_addr learned my addr 172.21.15.27:0/850247344 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:37.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.477+0000 7f4749a00700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4744108a80 0x7f474419bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.477+0000 7f4743e00700 1 -- 172.21.15.27:0/850247344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4744108a80 msgr2=0x7f474419bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.477+0000 7f4743e00700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4744108a80 0x7f474419bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.477+0000 7f4743e00700 1 -- 172.21.15.27:0/850247344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47441073d0 msgr2=0x7f47441a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.477+0000 7f4743e00700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47441073d0 0x7f47441a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.477+0000 7f4743e00700 1 -- 172.21.15.27:0/850247344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474419c2b0 con 0x7f47441080c0 2024-05-10T11:28:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.477+0000 7f4749000700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47441073d0 0x7f47441a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:28:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.478+0000 7f4743e00700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47441080c0 0x7f47441a1e80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f473000e9a0 tx=0x7f473000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.478+0000 7f4741600700 1 -- 172.21.15.27:0/850247344 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473000ca10 con 0x7f47441080c0 2024-05-10T11:28:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.478+0000 7f4741600700 1 -- 172.21.15.27:0/850247344 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f473000cbb0 con 0x7f47441080c0 2024-05-10T11:28:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.478+0000 7f4741600700 1 -- 172.21.15.27:0/850247344 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4730018680 con 0x7f47441080c0 2024-05-10T11:28:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.478+0000 7f474ba00700 1 -- 172.21.15.27:0/850247344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f474419c5a0 con 0x7f47441080c0 2024-05-10T11:28:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.478+0000 7f474ba00700 1 -- 172.21.15.27:0/850247344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47441a8810 con 0x7f47441080c0 2024-05-10T11:28:37.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.482+0000 7f4741600700 1 -- 172.21.15.27:0/850247344 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f4730010430 con 0x7f47441080c0 2024-05-10T11:28:37.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.483+0000 7f47563d3b80 1 Processor -- start 2024-05-10T11:28:37.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.484+0000 7f47563d3b80 1 -- start start 2024-05-10T11:28:37.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.484+0000 7f47563d3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653eebc0 0x55f4653eefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.484+0000 7f47563d3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653ef4e0 0x55f4653e92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.484+0000 7f47563d3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653e98f0 0x55f4653e9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.485+0000 7f47563d3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f46533ba20 con 0x55f4653e98f0 2024-05-10T11:28:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.485+0000 7f47563d3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f4652d9760 con 0x55f4653eebc0 2024-05-10T11:28:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.485+0000 7f47563d3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f4653379e0 con 0x55f4653ef4e0 2024-05-10T11:28:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.485+0000 7f4741600700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f472c0b2a70 0x7f472c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.485+0000 7f4741600700 1 -- 172.21.15.27:0/850247344 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7f47300e2ba0 con 0x7f47441080c0 2024-05-10T11:28:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.486+0000 7f4749000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653ef4e0 0x55f4653e92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.486+0000 7f4749000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653ef4e0 0x55f4653e92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54768/0 (socket says 172.21.15.27:54768) 2024-05-10T11:28:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.486+0000 7f4749000700 1 -- 172.21.15.27:0/2276434074 learned_addr learned my addr 172.21.15.27:0/2276434074 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.486+0000 7f4749a00700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f472c0b2a70 0x7f472c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.487+0000 7f4749a00700 1 --2- 172.21.15.27:0/2276434074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653eebc0 0x55f4653eefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.488+0000 7f4743e00700 1 --2- 172.21.15.27:0/2276434074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653e98f0 0x55f4653e9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.488+0000 7f4749a00700 1 -- 172.21.15.27:0/2276434074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653ef4e0 msgr2=0x55f4653e92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.488+0000 7f4749a00700 1 --2- 172.21.15.27:0/2276434074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653ef4e0 0x55f4653e92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.488+0000 7f4749a00700 1 -- 172.21.15.27:0/2276434074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653e98f0 msgr2=0x55f4653e9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.488+0000 7f4749a00700 1 --2- 172.21.15.27:0/2276434074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653e98f0 0x55f4653e9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.488+0000 7f4749a00700 1 -- 172.21.15.27:0/2276434074 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4652888d0 con 0x55f4653eebc0 2024-05-10T11:28:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.488+0000 7f4743e00700 1 --2- 172.21.15.27:0/2276434074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653e98f0 0x55f4653e9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:28:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.488+0000 7f4749a00700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f472c0b2a70 0x7f472c0b4f30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f473c006260 tx=0x7f473c009e20 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.488+0000 7f4749a00700 1 --2- 172.21.15.27:0/2276434074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653eebc0 0x55f4653eefa0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f473c00b770 tx=0x7f473c027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57f142b3e0e0c4d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.489+0000 7f4736000700 1 -- 172.21.15.27:0/2276434074 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c0074e0 con 0x55f4653eebc0 2024-05-10T11:28:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.489+0000 7f4736000700 1 -- 172.21.15.27:0/2276434074 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f473c030070 con 0x55f4653eebc0 2024-05-10T11:28:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.489+0000 7f4736000700 1 -- 172.21.15.27:0/2276434074 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c02b940 con 0x55f4653eebc0 2024-05-10T11:28:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.490+0000 7f47563d3b80 1 -- 172.21.15.27:0/2276434074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653eebc0 msgr2=0x55f4653eefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.490+0000 7f47563d3b80 1 --2- 172.21.15.27:0/2276434074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653eebc0 0x55f4653eefa0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f473c00b770 tx=0x7f473c027000 comp rx=0 tx=0).stop 2024-05-10T11:28:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.490+0000 7f47563d3b80 1 -- 172.21.15.27:0/2276434074 shutdown_connections 2024-05-10T11:28:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.490+0000 7f47563d3b80 1 --2- 172.21.15.27:0/2276434074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653e98f0 0x55f4653e9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.490+0000 7f47563d3b80 1 --2- 172.21.15.27:0/2276434074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653eebc0 0x55f4653eefa0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.490+0000 7f47563d3b80 1 --2- 172.21.15.27:0/2276434074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653ef4e0 0x55f4653e92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.490+0000 7f47563d3b80 1 -- 172.21.15.27:0/2276434074 >> 172.21.15.27:0/2276434074 conn(0x55f4653616d0 msgr2=0x55f4653e3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.490+0000 7f47563d3b80 1 -- 172.21.15.27:0/2276434074 shutdown_connections 2024-05-10T11:28:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.491+0000 7f47563d3b80 1 -- 172.21.15.27:0/2276434074 wait complete. 2024-05-10T11:28:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.491+0000 7f47563d3b80 1 -- start start 2024-05-10T11:28:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.492+0000 7f47563d3b80 1 Processor -- start 2024-05-10T11:28:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.492+0000 7f47563d3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 0x55f4653f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.492+0000 7f47563d3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653eebc0 0x55f4653f2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.492+0000 7f47563d3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 0x55f4653f2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.492+0000 7f47563d3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f4653f5ec0 con 0x55f4653eebc0 2024-05-10T11:28:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.492+0000 7f47563d3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f4652a1b20 con 0x55f4653ef4e0 2024-05-10T11:28:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.492+0000 7f47563d3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f4653f6040 con 0x55f4653e98f0 2024-05-10T11:28:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.492+0000 7f47563d3b80 10 client.0 _ll_register_callbacks cb 0x55f4652e2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.493+0000 7f4749000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 0x55f4653f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.493+0000 7f4749000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 0x55f4653f51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54776/0 (socket says 172.21.15.27:54776) 2024-05-10T11:28:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.493+0000 7f4749000700 1 -- 172.21.15.27:0/1446772936 learned_addr learned my addr 172.21.15.27:0/1446772936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.493+0000 7f4743e00700 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653eebc0 0x55f4653f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.493+0000 7f4749a00700 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 0x55f4653f2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.493+0000 7f4749000700 1 -- 172.21.15.27:0/1446772936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 msgr2=0x55f4653f2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.493+0000 7f4749000700 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 0x55f4653f2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.493+0000 7f4749000700 1 -- 172.21.15.27:0/1446772936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653eebc0 msgr2=0x55f4653f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.493+0000 7f4749000700 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653eebc0 0x55f4653f2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.493+0000 7f4749000700 1 -- 172.21.15.27:0/1446772936 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f465301420 con 0x55f4653e98f0 2024-05-10T11:28:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.494+0000 7f4749a00700 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 0x55f4653f2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:28:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.494+0000 7f4749000700 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 0x55f4653f51e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4738012d20 tx=0x7f473800f6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edc4179012c28df server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.494+0000 7f472aa00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:28:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.494+0000 7f472aa00700 1 -- 172.21.15.27:0/1446772936 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738009c90 con 0x55f4653e98f0 2024-05-10T11:28:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.494+0000 7f47563d3b80 1 -- 172.21.15.27:0/1446772936 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f473c030070 con 0x55f4653e98f0 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.494+0000 7f472aa00700 1 -- 172.21.15.27:0/1446772936 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7f473800fc30 con 0x55f4653e98f0 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.494+0000 7f472aa00700 1 -- 172.21.15.27:0/1446772936 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f4738025050 con 0x55f4653e98f0 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.494+0000 7f47563d3b80 10 client.27385 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.494+0000 7f47563d3b80 1 -- 172.21.15.27:0/1446772936 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f465400710 con 0x55f4653e98f0 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.495+0000 7f472aa00700 1 -- 172.21.15.27:0/1446772936 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f473c030070 con 0x55f4653e98f0 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.495+0000 7f472aa00700 1 -- 172.21.15.27:0/1446772936 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f4738003990 con 0x55f4653e98f0 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.495+0000 7f47563d3b80 10 client.27385 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.495+0000 7f47563d3b80 10 client.27385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.495+0000 7f47563d3b80 20 client.27385 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.495+0000 7f47563d3b80 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f465402d50 0x55f4654051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.495+0000 7f47563d3b80 4 client.27385 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.495+0000 7f47563d3b80 1 -- 172.21.15.27:0/1446772936 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4653b6140 con 0x55f465402d50 2024-05-10T11:28:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.496+0000 7f4743e00700 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f465402d50 0x55f4654051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.496+0000 7f4743e00700 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f465402d50 0x55f4654051a0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb1eb1336b8fb2a server_cookie=bed80196334349f4 in_seq=0 out_seq=0 2024-05-10T11:28:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.496+0000 7f472aa00700 10 client.27385 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:28:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.499+0000 7f472aa00700 1 -- 172.21.15.27:0/1446772936 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f4653b6140 con 0x55f465402d50 2024-05-10T11:28:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.499+0000 7f472aa00700 10 client.27385 handle_command_reply: tid=0 2024-05-10T11:28:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.499+0000 7f47563d3b80 1 client.27385 shutdown 2024-05-10T11:28:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.500+0000 7f47563d3b80 1 -- 172.21.15.27:0/1446772936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 msgr2=0x55f4653f51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.500+0000 7f47563d3b80 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 0x55f4653f51e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4738012d20 tx=0x7f473800f6a0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.500+0000 7f47563d3b80 1 -- 172.21.15.27:0/1446772936 shutdown_connections 2024-05-10T11:28:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.500+0000 7f47563d3b80 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f465402d50 0x55f4654051a0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.500+0000 7f47563d3b80 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 0x55f4653f51e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.500+0000 7f47563d3b80 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 0x55f4653f2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.500+0000 7f47563d3b80 1 --2- 172.21.15.27:0/1446772936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653eebc0 0x55f4653f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.500+0000 7f47563d3b80 1 -- 172.21.15.27:0/1446772936 >> 172.21.15.27:0/1446772936 conn(0x55f4653616d0 msgr2=0x55f4653e3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.500+0000 7f472aa00700 0 client.27385 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:28:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.500+0000 7f47563d3b80 1 -- 172.21.15.27:0/1446772936 shutdown_connections 2024-05-10T11:28:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.500+0000 7f47563d3b80 1 -- 172.21.15.27:0/1446772936 wait complete. 2024-05-10T11:28:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.500+0000 7f47563d3b80 20 client.27385 trim_cache size 0 max 16384 2024-05-10T11:28:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.527+0000 7f47563d3b80 1 Processor -- start 2024-05-10T11:28:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.527+0000 7f47563d3b80 1 -- start start 2024-05-10T11:28:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.527+0000 7f47563d3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653e98f0 0x55f4653e92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f47563d3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653eebc0 0x55f4653f1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f47563d3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653ef4e0 0x55f4653f0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f47563d3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f47441ad510 con 0x55f4653ef4e0 2024-05-10T11:28:37.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f47563d3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f47441af140 con 0x55f4653e98f0 2024-05-10T11:28:37.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f47563d3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f47441adbe0 con 0x55f4653eebc0 2024-05-10T11:28:37.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f4749000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653e98f0 0x55f4653e92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f4749000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653e98f0 0x55f4653e92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44098/0 (socket says 172.21.15.27:44098) 2024-05-10T11:28:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f4749000700 1 -- 172.21.15.27:0/4113939265 learned_addr learned my addr 172.21.15.27:0/4113939265 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f4743e00700 1 --2- 172.21.15.27:0/4113939265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653eebc0 0x55f4653f1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f4749a00700 1 --2- 172.21.15.27:0/4113939265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653ef4e0 0x55f4653f0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f4749000700 1 -- 172.21.15.27:0/4113939265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653eebc0 msgr2=0x55f4653f1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f4749000700 1 --2- 172.21.15.27:0/4113939265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653eebc0 0x55f4653f1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f4749000700 1 -- 172.21.15.27:0/4113939265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653ef4e0 msgr2=0x55f4653f0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f4749000700 1 --2- 172.21.15.27:0/4113939265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653ef4e0 0x55f4653f0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.528+0000 7f4749000700 1 -- 172.21.15.27:0/4113939265 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4653e4820 con 0x55f4653e98f0 2024-05-10T11:28:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.529+0000 7f4749000700 1 --2- 172.21.15.27:0/4113939265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653e98f0 0x55f4653e92d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4738012a40 tx=0x7f47380092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=699c4f3d83bd14da server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.529+0000 7f4735600700 1 -- 172.21.15.27:0/4113939265 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473802d020 con 0x55f4653e98f0 2024-05-10T11:28:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.529+0000 7f4735600700 1 -- 172.21.15.27:0/4113939265 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4738003b30 con 0x55f4653e98f0 2024-05-10T11:28:37.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.529+0000 7f4735600700 1 -- 172.21.15.27:0/4113939265 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738005220 con 0x55f4653e98f0 2024-05-10T11:28:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.530+0000 7f47563d3b80 1 -- 172.21.15.27:0/4113939265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653e98f0 msgr2=0x55f4653e92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.530+0000 7f47563d3b80 1 --2- 172.21.15.27:0/4113939265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653e98f0 0x55f4653e92d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4738012a40 tx=0x7f47380092d0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.530+0000 7f47563d3b80 1 -- 172.21.15.27:0/4113939265 shutdown_connections 2024-05-10T11:28:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.530+0000 7f47563d3b80 1 --2- 172.21.15.27:0/4113939265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653ef4e0 0x55f4653f0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.530+0000 7f47563d3b80 1 --2- 172.21.15.27:0/4113939265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653e98f0 0x55f4653e92d0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.530+0000 7f47563d3b80 1 --2- 172.21.15.27:0/4113939265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653eebc0 0x55f4653f1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.530+0000 7f47563d3b80 1 -- 172.21.15.27:0/4113939265 >> 172.21.15.27:0/4113939265 conn(0x55f4653616d0 msgr2=0x55f4653ed740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.530+0000 7f47563d3b80 1 -- 172.21.15.27:0/4113939265 shutdown_connections 2024-05-10T11:28:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.530+0000 7f47563d3b80 1 -- 172.21.15.27:0/4113939265 wait complete. 2024-05-10T11:28:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.531+0000 7f47563d3b80 1 -- start start 2024-05-10T11:28:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.531+0000 7f47563d3b80 1 Processor -- start 2024-05-10T11:28:37.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f47563d3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 0x55f4653fb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f47563d3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653eebc0 0x55f465409ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f47563d3b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 0x55f46540a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f47563d3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f47441ae9b0 con 0x55f4653eebc0 2024-05-10T11:28:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f47563d3b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f47441ae2e0 con 0x55f4653ef4e0 2024-05-10T11:28:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f47563d3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f47441ab530 con 0x55f4653e98f0 2024-05-10T11:28:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f47563d3b80 10 client.0 _ll_register_callbacks cb 0x55f4652e2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:28:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f4749a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 0x55f46540a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f4749000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 0x55f4653fb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f4749a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 0x55f46540a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44102/0 (socket says 172.21.15.27:44102) 2024-05-10T11:28:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f4749000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 0x55f4653fb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54786/0 (socket says 172.21.15.27:54786) 2024-05-10T11:28:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f4749a00700 1 -- 172.21.15.27:0/3337849329 learned_addr learned my addr 172.21.15.27:0/3337849329 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:28:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.532+0000 7f4743e00700 1 --2- 172.21.15.27:0/3337849329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653eebc0 0x55f465409ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.533+0000 7f4749a00700 1 -- 172.21.15.27:0/3337849329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 msgr2=0x55f4653fb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:37.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.533+0000 7f4749a00700 1 --2- 172.21.15.27:0/3337849329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 0x55f4653fb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.533+0000 7f4749a00700 1 -- 172.21.15.27:0/3337849329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653eebc0 msgr2=0x55f465409ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:28:37.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.533+0000 7f4749a00700 1 --2- 172.21.15.27:0/3337849329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653eebc0 0x55f465409ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.533+0000 7f4749a00700 1 -- 172.21.15.27:0/3337849329 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f46540ab10 con 0x55f4653ef4e0 2024-05-10T11:28:37.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.533+0000 7f4749a00700 1 --2- 172.21.15.27:0/3337849329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 0x55f46540a410 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f473c006230 tx=0x7f473c007d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8938ff93d57c9ad server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:28:37.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.533+0000 7f472aa00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:28:37.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.533+0000 7f472aa00700 1 -- 172.21.15.27:0/3337849329 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c02b4d0 con 0x55f4653ef4e0 2024-05-10T11:28:37.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.533+0000 7f47563d3b80 1 -- 172.21.15.27:0/3337849329 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4738003b30 con 0x55f4653ef4e0 2024-05-10T11:28:37.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.533+0000 7f472aa00700 1 -- 172.21.15.27:0/3337849329 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9591+0+0 (secure 0 0 0) 0x7f473c02b670 con 0x55f4653ef4e0 2024-05-10T11:28:37.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.534+0000 7f472aa00700 1 -- 172.21.15.27:0/3337849329 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f473c02b870 con 0x55f4653ef4e0 2024-05-10T11:28:37.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.534+0000 7f47563d3b80 10 client.27404 fetch_fsmap learned FSMap version 40 2024-05-10T11:28:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.534+0000 7f47563d3b80 1 -- 172.21.15.27:0/3337849329 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f46546d5f0 con 0x55f4653ef4e0 2024-05-10T11:28:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.534+0000 7f472aa00700 1 -- 172.21.15.27:0/3337849329 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4738003b30 con 0x55f4653ef4e0 2024-05-10T11:28:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.534+0000 7f472aa00700 1 -- 172.21.15.27:0/3337849329 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f473c011070 con 0x55f4653ef4e0 2024-05-10T11:28:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.534+0000 7f47563d3b80 10 client.27404 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:28:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.534+0000 7f47563d3b80 10 client.27404 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:28:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.534+0000 7f47563d3b80 20 client.27404 populate_metadata read hostname 'smithi027' 2024-05-10T11:28:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.534+0000 7f47563d3b80 1 --2- 172.21.15.27:0/3337849329 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f46546d8d0 0x55f46546fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:28:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.534+0000 7f47563d3b80 4 client.27404 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:28:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.534+0000 7f47563d3b80 1 -- 172.21.15.27:0/3337849329 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f465349400 con 0x55f46546d8d0 2024-05-10T11:28:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.535+0000 7f4749000700 1 --2- 172.21.15.27:0/3337849329 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f46546d8d0 0x55f46546fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:28:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.535+0000 7f4749000700 1 --2- 172.21.15.27:0/3337849329 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f46546d8d0 0x55f46546fcc0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=103a432c813782a9 server_cookie=cbd285ad25458c91 in_seq=0 out_seq=0 2024-05-10T11:28:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.535+0000 7f472aa00700 10 client.27404 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:28:37.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.538+0000 7f472aa00700 1 -- 172.21.15.27:0/3337849329 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f465349400 con 0x55f46546d8d0 2024-05-10T11:28:37.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.538+0000 7f472aa00700 10 client.27404 handle_command_reply: tid=0 2024-05-10T11:28:37.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.538+0000 7f47563d3b80 1 client.27404 shutdown 2024-05-10T11:28:37.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.539+0000 7f47563d3b80 1 -- 172.21.15.27:0/3337849329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 msgr2=0x55f46540a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:37.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.539+0000 7f47563d3b80 1 --2- 172.21.15.27:0/3337849329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 0x55f46540a410 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f473c006230 tx=0x7f473c007d40 comp rx=0 tx=0).stop 2024-05-10T11:28:37.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.539+0000 7f47563d3b80 1 -- 172.21.15.27:0/3337849329 shutdown_connections 2024-05-10T11:28:37.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.539+0000 7f47563d3b80 1 --2- 172.21.15.27:0/3337849329 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f46546d8d0 0x55f46546fcc0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.539+0000 7f47563d3b80 1 --2- 172.21.15.27:0/3337849329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4653e98f0 0x55f4653fb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.539+0000 7f47563d3b80 1 --2- 172.21.15.27:0/3337849329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f4653ef4e0 0x55f46540a410 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.539+0000 7f47563d3b80 1 --2- 172.21.15.27:0/3337849329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f4653eebc0 0x55f465409ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.539+0000 7f472aa00700 0 client.27404 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:28:37.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.539+0000 7f47563d3b80 1 -- 172.21.15.27:0/3337849329 >> 172.21.15.27:0/3337849329 conn(0x55f4653616d0 msgr2=0x55f4653edfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:37.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.539+0000 7f47563d3b80 1 -- 172.21.15.27:0/3337849329 shutdown_connections 2024-05-10T11:28:37.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.539+0000 7f47563d3b80 1 -- 172.21.15.27:0/3337849329 wait complete. 2024-05-10T11:28:37.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.539+0000 7f47563d3b80 20 client.27404 trim_cache size 0 max 16384 2024-05-10T11:28:37.541 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:28:37.541 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:28:37.541 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:28:37.541 INFO:teuthology.orchestra.run.smithi027.stdout: "337f5dbe-3d1c-47cb-8ba5-a5f0b4324c13": { 2024-05-10T11:28:37.541 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:28:37.541 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "337f5dbe-3d1c-47cb-8ba5-a5f0b4324c13", 2024-05-10T11:28:37.541 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:28:37.541 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:28:37.541 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:28:37.541 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:28:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.540+0000 7f474ba00700 1 -- 172.21.15.27:0/850247344 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f472c0b2a70 msgr2=0x7f472c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.540+0000 7f474ba00700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f472c0b2a70 0x7f472c0b4f30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f473c006260 tx=0x7f473c009e20 comp rx=0 tx=0).stop 2024-05-10T11:28:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.540+0000 7f474ba00700 1 -- 172.21.15.27:0/850247344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47441080c0 msgr2=0x7f47441a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:28:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.540+0000 7f474ba00700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47441080c0 0x7f47441a1e80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f473000e9a0 tx=0x7f473000ecb0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.540+0000 7f474ba00700 1 -- 172.21.15.27:0/850247344 shutdown_connections 2024-05-10T11:28:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.540+0000 7f474ba00700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f472c0b2a70 0x7f472c0b4f30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.540+0000 7f474ba00700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4744108a80 0x7f474419bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.540+0000 7f474ba00700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47441073d0 0x7f47441a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.540+0000 7f474ba00700 1 --2- 172.21.15.27:0/850247344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47441080c0 0x7f47441a1e80 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:28:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.540+0000 7f474ba00700 1 -- 172.21.15.27:0/850247344 >> 172.21.15.27:0/850247344 conn(0x7f4744077740 msgr2=0x7f4744105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:28:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.540+0000 7f474ba00700 1 -- 172.21.15.27:0/850247344 shutdown_connections 2024-05-10T11:28:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:28:37.540+0000 7f474ba00700 1 -- 172.21.15.27:0/850247344 wait complete. 2024-05-10T11:28:38.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 37102 26.52 MB/sec warmup 82 sec latency 161.001 ms 2024-05-10T11:28:38.568 INFO:tasks.cephfs.filesystem:scrub status for tag:337f5dbe-3d1c-47cb-8ba5-a5f0b4324c13 - {'path': '/', 'tag': '337f5dbe-3d1c-47cb-8ba5-a5f0b4324c13', 'options': 'recursive,force'} 2024-05-10T11:28:39.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:38 smithi027 ceph-mon[30054]: pgmap v483: 129 pgs: 129 active+clean; 929 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.1 MiB/s wr, 405 op/s 2024-05-10T11:28:39.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 37446 26.45 MB/sec warmup 83 sec latency 269.141 ms 2024-05-10T11:28:39.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:38 smithi076 ceph-mon[34152]: pgmap v483: 129 pgs: 129 active+clean; 929 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.1 MiB/s wr, 405 op/s 2024-05-10T11:28:39.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:38 smithi189 ceph-mon[34842]: pgmap v483: 129 pgs: 129 active+clean; 929 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.1 MiB/s wr, 405 op/s 2024-05-10T11:28:40.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 37884 26.39 MB/sec warmup 84 sec latency 213.697 ms 2024-05-10T11:28:41.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:40 smithi027 ceph-mon[30054]: pgmap v484: 129 pgs: 129 active+clean; 949 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 9.6 MiB/s wr, 561 op/s 2024-05-10T11:28:41.146 INFO:tasks.workunit.client.0.smithi027.stdout: 10 38422 26.41 MB/sec warmup 85 sec latency 224.618 ms 2024-05-10T11:28:41.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:40 smithi076 ceph-mon[34152]: pgmap v484: 129 pgs: 129 active+clean; 949 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 9.6 MiB/s wr, 561 op/s 2024-05-10T11:28:41.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:40 smithi189 ceph-mon[34842]: pgmap v484: 129 pgs: 129 active+clean; 949 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 9.6 MiB/s wr, 561 op/s 2024-05-10T11:28:42.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:41 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:28:42.147 INFO:tasks.workunit.client.0.smithi027.stdout: 10 38914 26.40 MB/sec warmup 86 sec latency 245.780 ms 2024-05-10T11:28:42.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:41 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:28:42.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:41 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:28:43.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:42 smithi027 ceph-mon[30054]: pgmap v485: 129 pgs: 129 active+clean; 949 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.7 MiB/s wr, 338 op/s 2024-05-10T11:28:43.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:42 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:28:43.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:42 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:28:43.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:42 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:28:43.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:42 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:28:43.147 INFO:tasks.workunit.client.0.smithi027.stdout: 10 39254 26.24 MB/sec warmup 87 sec latency 614.068 ms 2024-05-10T11:28:43.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:42 smithi076 ceph-mon[34152]: pgmap v485: 129 pgs: 129 active+clean; 949 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.7 MiB/s wr, 338 op/s 2024-05-10T11:28:43.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:42 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:28:43.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:42 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:28:43.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:42 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:28:43.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:42 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:28:43.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:42 smithi189 ceph-mon[34842]: pgmap v485: 129 pgs: 129 active+clean; 949 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.7 MiB/s wr, 338 op/s 2024-05-10T11:28:43.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:42 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:28:43.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:42 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:28:43.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:42 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:28:43.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:42 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:28:44.147 INFO:tasks.workunit.client.0.smithi027.stdout: 10 39670 26.14 MB/sec warmup 88 sec latency 374.123 ms 2024-05-10T11:28:45.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:44 smithi027 ceph-mon[30054]: pgmap v486: 129 pgs: 129 active+clean; 949 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.7 MiB/s wr, 338 op/s 2024-05-10T11:28:45.147 INFO:tasks.workunit.client.0.smithi027.stdout: 10 40291 26.36 MB/sec warmup 89 sec latency 166.851 ms 2024-05-10T11:28:45.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:44 smithi076 ceph-mon[34152]: pgmap v486: 129 pgs: 129 active+clean; 949 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.7 MiB/s wr, 338 op/s 2024-05-10T11:28:45.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:44 smithi189 ceph-mon[34842]: pgmap v486: 129 pgs: 129 active+clean; 949 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.7 MiB/s wr, 338 op/s 2024-05-10T11:28:46.147 INFO:tasks.workunit.client.0.smithi027.stdout: 10 40807 26.36 MB/sec warmup 90 sec latency 184.795 ms 2024-05-10T11:28:47.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:46 smithi027 ceph-mon[30054]: pgmap v487: 129 pgs: 129 active+clean; 966 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 8.0 MiB/s wr, 525 op/s 2024-05-10T11:28:47.147 INFO:tasks.workunit.client.0.smithi027.stdout: 10 41279 26.40 MB/sec warmup 91 sec latency 61.780 ms 2024-05-10T11:28:47.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:46 smithi076 ceph-mon[34152]: pgmap v487: 129 pgs: 129 active+clean; 966 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 8.0 MiB/s wr, 525 op/s 2024-05-10T11:28:47.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:46 smithi189 ceph-mon[34842]: pgmap v487: 129 pgs: 129 active+clean; 966 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 8.0 MiB/s wr, 525 op/s 2024-05-10T11:28:48.147 INFO:tasks.workunit.client.0.smithi027.stdout: 10 41551 26.29 MB/sec warmup 92 sec latency 239.623 ms 2024-05-10T11:28:49.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:48 smithi027 ceph-mon[30054]: pgmap v488: 129 pgs: 129 active+clean; 966 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.7 MiB/s wr, 342 op/s 2024-05-10T11:28:49.147 INFO:tasks.workunit.client.0.smithi027.stdout: 10 41894 26.17 MB/sec warmup 93 sec latency 138.359 ms 2024-05-10T11:28:49.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:48 smithi076 ceph-mon[34152]: pgmap v488: 129 pgs: 129 active+clean; 966 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.7 MiB/s wr, 342 op/s 2024-05-10T11:28:49.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:48 smithi189 ceph-mon[34842]: pgmap v488: 129 pgs: 129 active+clean; 966 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.7 MiB/s wr, 342 op/s 2024-05-10T11:28:50.148 INFO:tasks.workunit.client.0.smithi027.stdout: 10 42331 26.15 MB/sec warmup 94 sec latency 139.052 ms 2024-05-10T11:28:51.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:50 smithi027 ceph-mon[30054]: pgmap v489: 129 pgs: 129 active+clean; 901 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 9.1 MiB/s wr, 541 op/s 2024-05-10T11:28:51.148 INFO:tasks.workunit.client.0.smithi027.stdout: 10 42984 26.20 MB/sec warmup 95 sec latency 58.672 ms 2024-05-10T11:28:51.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:50 smithi076 ceph-mon[34152]: pgmap v489: 129 pgs: 129 active+clean; 901 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 9.1 MiB/s wr, 541 op/s 2024-05-10T11:28:51.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:50 smithi189 ceph-mon[34842]: pgmap v489: 129 pgs: 129 active+clean; 901 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 9.1 MiB/s wr, 541 op/s 2024-05-10T11:28:52.148 INFO:tasks.workunit.client.0.smithi027.stdout: 10 43442 26.12 MB/sec warmup 96 sec latency 84.571 ms 2024-05-10T11:28:53.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:52 smithi027 ceph-mon[30054]: pgmap v490: 129 pgs: 129 active+clean; 901 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 6.6 MiB/s wr, 385 op/s 2024-05-10T11:28:53.148 INFO:tasks.workunit.client.0.smithi027.stdout: 10 43778 26.09 MB/sec warmup 97 sec latency 209.077 ms 2024-05-10T11:28:53.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:52 smithi076 ceph-mon[34152]: pgmap v490: 129 pgs: 129 active+clean; 901 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 6.6 MiB/s wr, 385 op/s 2024-05-10T11:28:53.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:52 smithi189 ceph-mon[34842]: pgmap v490: 129 pgs: 129 active+clean; 901 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 6.6 MiB/s wr, 385 op/s 2024-05-10T11:28:54.148 INFO:tasks.workunit.client.0.smithi027.stdout: 10 44353 26.22 MB/sec warmup 98 sec latency 127.864 ms 2024-05-10T11:28:55.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:54 smithi027 ceph-mon[30054]: pgmap v491: 129 pgs: 129 active+clean; 901 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 6.6 MiB/s wr, 385 op/s 2024-05-10T11:28:55.148 INFO:tasks.workunit.client.0.smithi027.stdout: 10 44858 26.22 MB/sec warmup 99 sec latency 188.677 ms 2024-05-10T11:28:55.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:54 smithi076 ceph-mon[34152]: pgmap v491: 129 pgs: 129 active+clean; 901 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 6.6 MiB/s wr, 385 op/s 2024-05-10T11:28:55.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:54 smithi189 ceph-mon[34842]: pgmap v491: 129 pgs: 129 active+clean; 901 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 6.6 MiB/s wr, 385 op/s 2024-05-10T11:28:56.148 INFO:tasks.workunit.client.0.smithi027.stdout: 10 45298 26.17 MB/sec warmup 100 sec latency 213.182 ms 2024-05-10T11:28:57.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:56 smithi027 ceph-mon[30054]: pgmap v492: 129 pgs: 129 active+clean; 920 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 11 MiB/s wr, 577 op/s 2024-05-10T11:28:57.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 45805 26.19 MB/sec warmup 101 sec latency 229.973 ms 2024-05-10T11:28:57.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:56 smithi076 ceph-mon[34152]: pgmap v492: 129 pgs: 129 active+clean; 920 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 11 MiB/s wr, 577 op/s 2024-05-10T11:28:57.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:56 smithi189 ceph-mon[34842]: pgmap v492: 129 pgs: 129 active+clean; 920 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 11 MiB/s wr, 577 op/s 2024-05-10T11:28:58.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 46126 26.11 MB/sec warmup 102 sec latency 237.051 ms 2024-05-10T11:28:59.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 46333 25.91 MB/sec warmup 103 sec latency 395.550 ms 2024-05-10T11:28:59.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:28:58 smithi076 ceph-mon[34152]: pgmap v493: 129 pgs: 129 active+clean; 920 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 7.4 MiB/s wr, 390 op/s 2024-05-10T11:28:59.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:28:58 smithi189 ceph-mon[34842]: pgmap v493: 129 pgs: 129 active+clean; 920 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 7.4 MiB/s wr, 390 op/s 2024-05-10T11:28:59.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:28:58 smithi027 ceph-mon[30054]: pgmap v493: 129 pgs: 129 active+clean; 920 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 7.4 MiB/s wr, 390 op/s 2024-05-10T11:29:00.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 46846 25.88 MB/sec warmup 104 sec latency 265.853 ms 2024-05-10T11:29:01.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 47325 25.88 MB/sec warmup 105 sec latency 229.278 ms 2024-05-10T11:29:01.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:00 smithi076 ceph-mon[34152]: pgmap v494: 129 pgs: 129 active+clean; 948 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 11 MiB/s wr, 586 op/s 2024-05-10T11:29:01.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:00 smithi189 ceph-mon[34842]: pgmap v494: 129 pgs: 129 active+clean; 948 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 11 MiB/s wr, 586 op/s 2024-05-10T11:29:01.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:00 smithi027 ceph-mon[30054]: pgmap v494: 129 pgs: 129 active+clean; 948 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 11 MiB/s wr, 586 op/s 2024-05-10T11:29:02.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 47547 25.71 MB/sec warmup 106 sec latency 326.542 ms 2024-05-10T11:29:03.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 47629 25.53 MB/sec warmup 107 sec latency 358.871 ms 2024-05-10T11:29:03.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:02 smithi076 ceph-mon[34152]: pgmap v495: 129 pgs: 129 active+clean; 948 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 7.7 MiB/s wr, 386 op/s 2024-05-10T11:29:03.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:02 smithi076 ceph-mon[34152]: osdmap e110: 12 total, 12 up, 12 in 2024-05-10T11:29:03.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:02 smithi189 ceph-mon[34842]: pgmap v495: 129 pgs: 129 active+clean; 948 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 7.7 MiB/s wr, 386 op/s 2024-05-10T11:29:03.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:02 smithi189 ceph-mon[34842]: osdmap e110: 12 total, 12 up, 12 in 2024-05-10T11:29:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:02 smithi027 ceph-mon[30054]: pgmap v495: 129 pgs: 129 active+clean; 948 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 7.7 MiB/s wr, 386 op/s 2024-05-10T11:29:03.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:02 smithi027 ceph-mon[30054]: osdmap e110: 12 total, 12 up, 12 in 2024-05-10T11:29:04.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 47725 25.36 MB/sec warmup 108 sec latency 789.159 ms 2024-05-10T11:29:05.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 47911 25.20 MB/sec warmup 109 sec latency 457.584 ms 2024-05-10T11:29:05.172 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:04 smithi027 ceph-mon[30054]: pgmap v497: 129 pgs: 129 active+clean; 948 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.2 MiB/s wr, 464 op/s 2024-05-10T11:29:05.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:04 smithi076 ceph-mon[34152]: pgmap v497: 129 pgs: 129 active+clean; 948 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.2 MiB/s wr, 464 op/s 2024-05-10T11:29:05.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:04 smithi189 ceph-mon[34842]: pgmap v497: 129 pgs: 129 active+clean; 948 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.2 MiB/s wr, 464 op/s 2024-05-10T11:29:06.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 48435 25.33 MB/sec warmup 110 sec latency 307.461 ms 2024-05-10T11:29:06.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:05 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:29:06.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:05 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:29:06.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:05 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:29:07.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 49007 25.42 MB/sec warmup 111 sec latency 125.473 ms 2024-05-10T11:29:07.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:06 smithi076 ceph-mon[34152]: pgmap v498: 129 pgs: 129 active+clean; 958 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.5 MiB/s wr, 604 op/s 2024-05-10T11:29:07.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:06 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:29:07.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:06 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:29:07.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:06 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:29:07.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:06 smithi189 ceph-mon[34842]: pgmap v498: 129 pgs: 129 active+clean; 958 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.5 MiB/s wr, 604 op/s 2024-05-10T11:29:07.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:06 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:29:07.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:06 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:29:07.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:06 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:29:07.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:06 smithi027 ceph-mon[30054]: pgmap v498: 129 pgs: 129 active+clean; 958 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.5 MiB/s wr, 604 op/s 2024-05-10T11:29:07.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:06 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:29:07.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:06 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:29:07.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:06 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:29:08.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 49414 25.36 MB/sec warmup 112 sec latency 116.201 ms 2024-05-10T11:29:08.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:29:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:08 smithi027 ceph-mon[30054]: pgmap v499: 129 pgs: 129 active+clean; 958 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.5 MiB/s wr, 604 op/s 2024-05-10T11:29:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:08 smithi027 ceph-mon[30054]: osdmap e111: 12 total, 12 up, 12 in 2024-05-10T11:29:09.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 49654 25.25 MB/sec warmup 113 sec latency 310.548 ms 2024-05-10T11:29:09.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:08 smithi076 ceph-mon[34152]: pgmap v499: 129 pgs: 129 active+clean; 958 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.5 MiB/s wr, 604 op/s 2024-05-10T11:29:09.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:08 smithi076 ceph-mon[34152]: osdmap e111: 12 total, 12 up, 12 in 2024-05-10T11:29:09.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:08 smithi189 ceph-mon[34842]: pgmap v499: 129 pgs: 129 active+clean; 958 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.5 MiB/s wr, 604 op/s 2024-05-10T11:29:09.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:08 smithi189 ceph-mon[34842]: osdmap e111: 12 total, 12 up, 12 in 2024-05-10T11:29:09.281 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:29:10.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 50034 25.19 MB/sec warmup 114 sec latency 132.291 ms 2024-05-10T11:29:11.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:10 smithi027 ceph-mon[30054]: pgmap v501: 129 pgs: 129 active+clean; 976 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 10 MiB/s wr, 790 op/s 2024-05-10T11:29:11.149 INFO:tasks.workunit.client.0.smithi027.stdout: 10 50467 25.23 MB/sec warmup 115 sec latency 165.825 ms 2024-05-10T11:29:11.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:10 smithi076 ceph-mon[34152]: pgmap v501: 129 pgs: 129 active+clean; 976 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 10 MiB/s wr, 790 op/s 2024-05-10T11:29:11.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:10 smithi189 ceph-mon[34842]: pgmap v501: 129 pgs: 129 active+clean; 976 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 10 MiB/s wr, 790 op/s 2024-05-10T11:29:12.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 50890 25.11 MB/sec warmup 116 sec latency 107.226 ms 2024-05-10T11:29:12.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.412+0000 7f6e08600700 1 -- 172.21.15.27:0/651772204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00108850 msgr2=0x7f6e001103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.412+0000 7f6e08600700 1 --2- 172.21.15.27:0/651772204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00108850 0x7f6e001103d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6df80099c0 tx=0x7f6df8009cd0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.413+0000 7f6e08600700 1 -- 172.21.15.27:0/651772204 shutdown_connections 2024-05-10T11:29:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.413+0000 7f6e08600700 1 --2- 172.21.15.27:0/651772204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e00103750 0x7f6e00103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.413+0000 7f6e08600700 1 --2- 172.21.15.27:0/651772204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00108850 0x7f6e001103d0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.413+0000 7f6e08600700 1 --2- 172.21.15.27:0/651772204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e00104100 0x7f6e00108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.413+0000 7f6e08600700 1 -- 172.21.15.27:0/651772204 >> 172.21.15.27:0/651772204 conn(0x7f6e0007a600 msgr2=0x7f6e0007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.413+0000 7f6e08600700 1 -- 172.21.15.27:0/651772204 shutdown_connections 2024-05-10T11:29:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.413+0000 7f6e08600700 1 -- 172.21.15.27:0/651772204 wait complete. 2024-05-10T11:29:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.414+0000 7f6e08600700 1 Processor -- start 2024-05-10T11:29:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.414+0000 7f6e08600700 1 -- start start 2024-05-10T11:29:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.415+0000 7f6e08600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00103750 0x7f6e001a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.415+0000 7f6e08600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e00104100 0x7f6e001a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.415+0000 7f6e08600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e00108850 0x7f6e0019baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.415+0000 7f6e08600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e00113670 con 0x7f6e00104100 2024-05-10T11:29:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.415+0000 7f6e08600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001134f0 con 0x7f6e00103750 2024-05-10T11:29:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.415+0000 7f6e08600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001137f0 con 0x7f6e00108850 2024-05-10T11:29:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.415+0000 7f6e05c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00103750 0x7f6e001a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.415+0000 7f6e05200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e00104100 0x7f6e001a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.415+0000 7f6e05c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00103750 0x7f6e001a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52130/0 (socket says 172.21.15.27:52130) 2024-05-10T11:29:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.415+0000 7f6e05c00700 1 -- 172.21.15.27:0/4044322902 learned_addr learned my addr 172.21.15.27:0/4044322902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.415+0000 7f6e06600700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e00108850 0x7f6e0019baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.416+0000 7f6e05c00700 1 -- 172.21.15.27:0/4044322902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e00108850 msgr2=0x7f6e0019baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.416+0000 7f6e05c00700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e00108850 0x7f6e0019baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.416+0000 7f6e05c00700 1 -- 172.21.15.27:0/4044322902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e00104100 msgr2=0x7f6e001a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.416+0000 7f6e05c00700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e00104100 0x7f6e001a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.416+0000 7f6e05c00700 1 -- 172.21.15.27:0/4044322902 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0019c200 con 0x7f6e00103750 2024-05-10T11:29:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.416+0000 7f6e05200700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e00104100 0x7f6e001a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:29:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.416+0000 7f6e05c00700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00103750 0x7f6e001a18e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6df400ef10 tx=0x7f6df400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.416+0000 7f6dfe000700 1 -- 172.21.15.27:0/4044322902 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df400c9c0 con 0x7f6e00103750 2024-05-10T11:29:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.416+0000 7f6e08600700 1 -- 172.21.15.27:0/4044322902 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e0019c490 con 0x7f6e00103750 2024-05-10T11:29:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.417+0000 7f6dfe000700 1 -- 172.21.15.27:0/4044322902 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6df4019070 con 0x7f6e00103750 2024-05-10T11:29:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.417+0000 7f6dfe000700 1 -- 172.21.15.27:0/4044322902 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4014480 con 0x7f6e00103750 2024-05-10T11:29:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.417+0000 7f6e08600700 1 -- 172.21.15.27:0/4044322902 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e001a87b0 con 0x7f6e00103750 2024-05-10T11:29:12.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.421+0000 7f6dfe000700 1 -- 172.21.15.27:0/4044322902 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f6df4012050 con 0x7f6e00103750 2024-05-10T11:29:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.423+0000 7f6e12f46b80 1 Processor -- start 2024-05-10T11:29:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.423+0000 7f6dfe000700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6de80b2a70 0x7f6de80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.423+0000 7f6dfe000700 1 -- 172.21.15.27:0/4044322902 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f6df40e2960 con 0x7f6e00103750 2024-05-10T11:29:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.424+0000 7f6e12f46b80 1 -- start start 2024-05-10T11:29:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.424+0000 7f6e06600700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6de80b2a70 0x7f6de80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.424+0000 7f6e12f46b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 0x55f9590bd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.424+0000 7f6e12f46b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590c1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.424+0000 7f6e12f46b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590c19d0 0x55f9590c1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.424+0000 7f6e12f46b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f95900fa20 con 0x55f9590c19d0 2024-05-10T11:29:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.424+0000 7f6e12f46b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f958fad760 con 0x55f9590bd980 2024-05-10T11:29:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.424+0000 7f6e12f46b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f95900b9e0 con 0x55f9590bd060 2024-05-10T11:29:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.424+0000 7f6e05c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 0x55f9590bd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.424+0000 7f6e05c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 0x55f9590bd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37382/0 (socket says 172.21.15.27:37382) 2024-05-10T11:29:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.425+0000 7f6e05c00700 1 -- 172.21.15.27:0/1123406766 learned_addr learned my addr 172.21.15.27:0/1123406766 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.425+0000 7f6e05200700 1 --2- 172.21.15.27:0/1123406766 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590c1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.425+0000 7f6e06600700 1 --2- 172.21.15.27:0/1123406766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590c19d0 0x55f9590c1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.425+0000 7f6e05c00700 1 -- 172.21.15.27:0/1123406766 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 msgr2=0x55f9590c1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.425+0000 7f6e05c00700 1 --2- 172.21.15.27:0/1123406766 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590c1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.425+0000 7f6e05c00700 1 -- 172.21.15.27:0/1123406766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590c19d0 msgr2=0x55f9590c1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.425+0000 7f6e05c00700 1 --2- 172.21.15.27:0/1123406766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590c19d0 0x55f9590c1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.425+0000 7f6e05c00700 1 -- 172.21.15.27:0/1123406766 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f958f5c8d0 con 0x55f9590bd060 2024-05-10T11:29:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.425+0000 7f6e05200700 1 --2- 172.21.15.27:0/1123406766 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590c1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:29:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.425+0000 7f6e06600700 1 --2- 172.21.15.27:0/1123406766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590c19d0 0x55f9590c1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:29:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.425+0000 7f6e06600700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6de80b2a70 0x7f6de80b4f30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6df800b580 tx=0x7f6df8009f20 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.426+0000 7f6e05c00700 1 --2- 172.21.15.27:0/1123406766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 0x55f9590bd440 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6df40a94b0 tx=0x7f6df40a91d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9f20257d72b257c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.426+0000 7f6df2a00700 1 -- 172.21.15.27:0/1123406766 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df40a97b0 con 0x55f9590bd060 2024-05-10T11:29:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.426+0000 7f6df2a00700 1 -- 172.21.15.27:0/1123406766 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6df40adb60 con 0x55f9590bd060 2024-05-10T11:29:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.426+0000 7f6df2a00700 1 -- 172.21.15.27:0/1123406766 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df40a9a10 con 0x55f9590bd060 2024-05-10T11:29:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.427+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1123406766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 msgr2=0x55f9590bd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.427+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/1123406766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 0x55f9590bd440 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6df40a94b0 tx=0x7f6df40a91d0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.427+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1123406766 shutdown_connections 2024-05-10T11:29:12.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.427+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/1123406766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 0x55f9590bd440 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.427+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/1123406766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590c19d0 0x55f9590c1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.427+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/1123406766 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590c1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.427+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1123406766 >> 172.21.15.27:0/1123406766 conn(0x55f958febb60 msgr2=0x55f9590b77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:12.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.427+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1123406766 shutdown_connections 2024-05-10T11:29:12.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.427+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1123406766 wait complete. 2024-05-10T11:29:12.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.428+0000 7f6e12f46b80 1 -- start start 2024-05-10T11:29:12.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.428+0000 7f6e12f46b80 1 Processor -- start 2024-05-10T11:29:12.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.428+0000 7f6e12f46b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd060 0x55f9590c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e12f46b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590bd980 0x55f9590c6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e12f46b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590c19d0 0x55f9590c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e12f46b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f9590c9be0 con 0x55f9590bd980 2024-05-10T11:29:12.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e12f46b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f958f75b20 con 0x55f9590bd060 2024-05-10T11:29:12.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e12f46b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f9590c9d60 con 0x55f9590c19d0 2024-05-10T11:29:12.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e12f46b80 10 client.0 _ll_register_callbacks cb 0x55f958fb6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:12.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e05c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd060 0x55f9590c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e05c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd060 0x55f9590c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:52146/0 (socket says 172.21.15.27:52146) 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e05c00700 1 -- 172.21.15.27:0/2341343542 learned_addr learned my addr 172.21.15.27:0/2341343542 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e05200700 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590bd980 0x55f9590c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e06600700 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590c19d0 0x55f9590c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e05c00700 1 -- 172.21.15.27:0/2341343542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590c19d0 msgr2=0x55f9590c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e05c00700 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590c19d0 0x55f9590c67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e05c00700 1 -- 172.21.15.27:0/2341343542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590bd980 msgr2=0x55f9590c6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e05c00700 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590bd980 0x55f9590c6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.429+0000 7f6e05c00700 1 -- 172.21.15.27:0/2341343542 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f958fd5420 con 0x55f9590bd060 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.430+0000 7f6e05c00700 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd060 0x55f9590c8ee0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6df40e01a0 tx=0x7f6df4009eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fba70937382a5d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.430+0000 7f6de7400700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.430+0000 7f6de7400700 1 -- 172.21.15.27:0/2341343542 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df40c3060 con 0x55f9590bd060 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.430+0000 7f6de7400700 1 -- 172.21.15.27:0/2341343542 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f6df40a9c20 con 0x55f9590bd060 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.430+0000 7f6e12f46b80 1 -- 172.21.15.27:0/2341343542 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6df40adb60 con 0x55f9590bd060 2024-05-10T11:29:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.430+0000 7f6de7400700 1 -- 172.21.15.27:0/2341343542 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6df40b2610 con 0x55f9590bd060 2024-05-10T11:29:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.430+0000 7f6e12f46b80 10 client.27416 fetch_fsmap learned FSMap version 40 2024-05-10T11:29:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.430+0000 7f6e12f46b80 1 -- 172.21.15.27:0/2341343542 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f9590d4450 con 0x55f9590bd060 2024-05-10T11:29:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.431+0000 7f6de7400700 1 -- 172.21.15.27:0/2341343542 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6df40adb60 con 0x55f9590bd060 2024-05-10T11:29:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.431+0000 7f6de7400700 1 -- 172.21.15.27:0/2341343542 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f6df40a5120 con 0x55f9590bd060 2024-05-10T11:29:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.431+0000 7f6e12f46b80 10 client.27416 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:29:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.431+0000 7f6e12f46b80 10 client.27416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:29:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.431+0000 7f6e12f46b80 20 client.27416 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.431+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f9590d6920 0x55f9590d8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.431+0000 7f6e12f46b80 4 client.27416 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:29:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.431+0000 7f6e12f46b80 1 -- 172.21.15.27:0/2341343542 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f95908a0a0 con 0x55f9590d6920 2024-05-10T11:29:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.431+0000 7f6e05200700 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f9590d6920 0x55f9590d8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.432+0000 7f6e05200700 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f9590d6920 0x55f9590d8d10 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99081ca1630153b3 server_cookie=63874a5d002ac438 in_seq=0 out_seq=0 2024-05-10T11:29:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.432+0000 7f6de7400700 10 client.27416 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:12.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.435+0000 7f6de7400700 1 -- 172.21.15.27:0/2341343542 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f95908a0a0 con 0x55f9590d6920 2024-05-10T11:29:12.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.435+0000 7f6de7400700 10 client.27416 handle_command_reply: tid=0 2024-05-10T11:29:12.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.435+0000 7f6e12f46b80 1 client.27416 shutdown 2024-05-10T11:29:12.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.436+0000 7f6e12f46b80 1 -- 172.21.15.27:0/2341343542 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd060 msgr2=0x55f9590c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.436+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd060 0x55f9590c8ee0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6df40e01a0 tx=0x7f6df4009eb0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.436+0000 7f6e12f46b80 1 -- 172.21.15.27:0/2341343542 shutdown_connections 2024-05-10T11:29:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.436+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f9590d6920 0x55f9590d8d10 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.436+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590c19d0 0x55f9590c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.436+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd060 0x55f9590c8ee0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.436+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/2341343542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590bd980 0x55f9590c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.436+0000 7f6e12f46b80 1 -- 172.21.15.27:0/2341343542 >> 172.21.15.27:0/2341343542 conn(0x55f958febb60 msgr2=0x55f9590b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:12.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.436+0000 7f6de7400700 0 client.27416 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:12.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.437+0000 7f6e12f46b80 1 -- 172.21.15.27:0/2341343542 shutdown_connections 2024-05-10T11:29:12.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.437+0000 7f6e12f46b80 1 -- 172.21.15.27:0/2341343542 wait complete. 2024-05-10T11:29:12.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.437+0000 7f6e12f46b80 20 client.27416 trim_cache size 0 max 16384 2024-05-10T11:29:12.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.466+0000 7f6e12f46b80 1 Processor -- start 2024-05-10T11:29:12.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.466+0000 7f6e12f46b80 1 -- start start 2024-05-10T11:29:12.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.467+0000 7f6e12f46b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590bd060 0x55f9590c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.467+0000 7f6e12f46b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590db950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.467+0000 7f6e12f46b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590c19d0 0x55f9590dbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.467+0000 7f6e12f46b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001b1870 con 0x55f9590bd060 2024-05-10T11:29:12.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.467+0000 7f6e12f46b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001b34a0 con 0x55f9590bd980 2024-05-10T11:29:12.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.467+0000 7f6e12f46b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001b1f40 con 0x55f9590c19d0 2024-05-10T11:29:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.467+0000 7f6e06600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590c19d0 0x55f9590dbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.467+0000 7f6e05200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590db950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.467+0000 7f6e06600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590c19d0 0x55f9590dbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37394/0 (socket says 172.21.15.27:37394) 2024-05-10T11:29:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.467+0000 7f6e06600700 1 -- 172.21.15.27:0/4007455944 learned_addr learned my addr 172.21.15.27:0/4007455944 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.468+0000 7f6e05200700 1 -- 172.21.15.27:0/4007455944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590c19d0 msgr2=0x55f9590dbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.468+0000 7f6e05200700 1 --2- 172.21.15.27:0/4007455944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590c19d0 0x55f9590dbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.468+0000 7f6e05200700 1 -- 172.21.15.27:0/4007455944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590bd060 msgr2=0x55f9590c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.468+0000 7f6e05200700 1 --2- 172.21.15.27:0/4007455944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590bd060 0x55f9590c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.468+0000 7f6e05200700 1 -- 172.21.15.27:0/4007455944 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9590b43f0 con 0x55f9590bd980 2024-05-10T11:29:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.468+0000 7f6e05200700 1 --2- 172.21.15.27:0/4007455944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590db950 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x55f9590c9e30 tx=0x7f6dec009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=548a80684fbff506 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.469+0000 7f6df2000700 1 -- 172.21.15.27:0/4007455944 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec004310 con 0x55f9590bd980 2024-05-10T11:29:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.469+0000 7f6df2000700 1 -- 172.21.15.27:0/4007455944 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dec0044b0 con 0x55f9590bd980 2024-05-10T11:29:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.469+0000 7f6df2000700 1 -- 172.21.15.27:0/4007455944 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec0056d0 con 0x55f9590bd980 2024-05-10T11:29:12.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.470+0000 7f6e12f46b80 1 -- 172.21.15.27:0/4007455944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 msgr2=0x55f9590db950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:12.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.470+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/4007455944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590db950 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x55f9590c9e30 tx=0x7f6dec009260 comp rx=0 tx=0).stop 2024-05-10T11:29:12.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.471+0000 7f6e12f46b80 1 -- 172.21.15.27:0/4007455944 shutdown_connections 2024-05-10T11:29:12.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.471+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/4007455944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590c19d0 0x55f9590dbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.471+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/4007455944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590bd060 0x55f9590c2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.471+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/4007455944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590db950 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.471+0000 7f6e12f46b80 1 -- 172.21.15.27:0/4007455944 >> 172.21.15.27:0/4007455944 conn(0x55f958febb60 msgr2=0x55f9590c2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:12.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.471+0000 7f6e12f46b80 1 -- 172.21.15.27:0/4007455944 shutdown_connections 2024-05-10T11:29:12.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.473+0000 7f6e12f46b80 1 -- 172.21.15.27:0/4007455944 wait complete. 2024-05-10T11:29:12.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.474+0000 7f6e12f46b80 1 -- start start 2024-05-10T11:29:12.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.475+0000 7f6e12f46b80 1 Processor -- start 2024-05-10T11:29:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.476+0000 7f6e12f46b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 0x55f9590d97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.476+0000 7f6e12f46b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590d9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.476+0000 7f6e12f46b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590c19d0 0x55f9590da230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.476+0000 7f6e12f46b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001b2d10 con 0x55f9590c19d0 2024-05-10T11:29:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.476+0000 7f6e12f46b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001b2640 con 0x55f9590bd980 2024-05-10T11:29:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.476+0000 7f6e12f46b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001af830 con 0x55f9590bd060 2024-05-10T11:29:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.476+0000 7f6e12f46b80 10 client.0 _ll_register_callbacks cb 0x55f958fb6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.476+0000 7f6e05c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 0x55f9590d97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.476+0000 7f6e05c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 0x55f9590d97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:37398/0 (socket says 172.21.15.27:37398) 2024-05-10T11:29:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.476+0000 7f6e05c00700 1 -- 172.21.15.27:0/1462988633 learned_addr learned my addr 172.21.15.27:0/1462988633 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.477+0000 7f6e05c00700 1 -- 172.21.15.27:0/1462988633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 msgr2=0x55f9590d9cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:29:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.477+0000 7f6e05c00700 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590d9cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.477+0000 7f6e05c00700 1 -- 172.21.15.27:0/1462988633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590c19d0 msgr2=0x55f9590da230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:29:12.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.478+0000 7f6e06600700 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590c19d0 0x55f9590da230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.478+0000 7f6e05c00700 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590c19d0 0x55f9590da230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.478+0000 7f6e05c00700 1 -- 172.21.15.27:0/1462988633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9591400f0 con 0x55f9590bd060 2024-05-10T11:29:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.478+0000 7f6e06600700 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590c19d0 0x55f9590da230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:29:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.479+0000 7f6e05c00700 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 0x55f9590d97b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6df40a9740 tx=0x7f6df400efa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e628e73ef62bd36f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.479+0000 7f6de7400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:29:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.479+0000 7f6de7400700 1 -- 172.21.15.27:0/1462988633 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df40c3070 con 0x55f9590bd060 2024-05-10T11:29:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.479+0000 7f6de7400700 1 -- 172.21.15.27:0/1462988633 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f6df40b2180 con 0x55f9590bd060 2024-05-10T11:29:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.479+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1462988633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dec0044b0 con 0x55f9590bd060 2024-05-10T11:29:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.480+0000 7f6de7400700 1 -- 172.21.15.27:0/1462988633 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6df4087030 con 0x55f9590bd060 2024-05-10T11:29:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.480+0000 7f6e12f46b80 10 client.27409 fetch_fsmap learned FSMap version 40 2024-05-10T11:29:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.480+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1462988633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f959149c10 con 0x55f9590bd060 2024-05-10T11:29:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.480+0000 7f6de7400700 1 -- 172.21.15.27:0/1462988633 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dec0044b0 con 0x55f9590bd060 2024-05-10T11:29:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.481+0000 7f6de7400700 1 -- 172.21.15.27:0/1462988633 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f6df40e2470 con 0x55f9590bd060 2024-05-10T11:29:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.481+0000 7f6e12f46b80 10 client.27409 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:29:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.481+0000 7f6e12f46b80 10 client.27409 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:29:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.481+0000 7f6e12f46b80 20 client.27409 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.481+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f95914a1c0 0x55f95914c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.481+0000 7f6e12f46b80 4 client.27409 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:29:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.481+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1462988633 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f95901d400 con 0x55f95914a1c0 2024-05-10T11:29:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.481+0000 7f6e05200700 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f95914a1c0 0x55f95914c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:12.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.483+0000 7f6e05200700 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f95914a1c0 0x55f95914c600 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33b118c290dd680b server_cookie=91efe8dff388a0e in_seq=0 out_seq=0 2024-05-10T11:29:12.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.483+0000 7f6de7400700 10 client.27409 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:12.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.489+0000 7f6de7400700 1 -- 172.21.15.27:0/1462988633 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f95901d400 con 0x55f95914a1c0 2024-05-10T11:29:12.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.489+0000 7f6de7400700 10 client.27409 handle_command_reply: tid=0 2024-05-10T11:29:12.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.489+0000 7f6e12f46b80 1 client.27409 shutdown 2024-05-10T11:29:12.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.490+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1462988633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 msgr2=0x55f9590d97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:12.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.490+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 0x55f9590d97b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6df40a9740 tx=0x7f6df400efa0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.490+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1462988633 shutdown_connections 2024-05-10T11:29:12.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.490+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f95914a1c0 0x55f95914c600 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.490+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9590bd060 0x55f9590d97b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.490+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f9590c19d0 0x55f9590da230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.490+0000 7f6e12f46b80 1 --2- 172.21.15.27:0/1462988633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f9590bd980 0x55f9590d9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.490+0000 7f6de7400700 0 client.27409 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.490+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1462988633 >> 172.21.15.27:0/1462988633 conn(0x55f958febb60 msgr2=0x55f9590b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.490+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1462988633 shutdown_connections 2024-05-10T11:29:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.491+0000 7f6e12f46b80 1 -- 172.21.15.27:0/1462988633 wait complete. 2024-05-10T11:29:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.491+0000 7f6e12f46b80 20 client.27409 trim_cache size 0 max 16384 2024-05-10T11:29:12.492 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:29:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:29:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:29:12.493 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:29:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.492+0000 7f6e08600700 1 -- 172.21.15.27:0/4044322902 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6de80b2a70 msgr2=0x7f6de80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.492+0000 7f6e08600700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6de80b2a70 0x7f6de80b4f30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6df800b580 tx=0x7f6df8009f20 comp rx=0 tx=0).stop 2024-05-10T11:29:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.492+0000 7f6e08600700 1 -- 172.21.15.27:0/4044322902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00103750 msgr2=0x7f6e001a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.492+0000 7f6e08600700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00103750 0x7f6e001a18e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6df400ef10 tx=0x7f6df400c3a0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.492+0000 7f6e08600700 1 -- 172.21.15.27:0/4044322902 shutdown_connections 2024-05-10T11:29:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.492+0000 7f6e08600700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6de80b2a70 0x7f6de80b4f30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.492+0000 7f6e08600700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e00108850 0x7f6e0019baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.492+0000 7f6e08600700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e00103750 0x7f6e001a18e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.492+0000 7f6e08600700 1 --2- 172.21.15.27:0/4044322902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e00104100 0x7f6e001a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.492+0000 7f6e08600700 1 -- 172.21.15.27:0/4044322902 >> 172.21.15.27:0/4044322902 conn(0x7f6e0007a600 msgr2=0x7f6e00101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.493+0000 7f6e08600700 1 -- 172.21.15.27:0/4044322902 shutdown_connections 2024-05-10T11:29:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:12.493+0000 7f6e08600700 1 -- 172.21.15.27:0/4044322902 wait complete. 2024-05-10T11:29:13.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 51422 25.14 MB/sec warmup 117 sec latency 161.137 ms 2024-05-10T11:29:13.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:12 smithi189 ceph-mon[34842]: pgmap v502: 129 pgs: 129 active+clean; 976 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 8.6 MiB/s wr, 650 op/s 2024-05-10T11:29:13.311 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:12 smithi027 ceph-mon[30054]: pgmap v502: 129 pgs: 129 active+clean; 976 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 8.6 MiB/s wr, 650 op/s 2024-05-10T11:29:13.371 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:29:13.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:29:13.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:12 smithi076 ceph-mon[34152]: pgmap v502: 129 pgs: 129 active+clean; 976 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 8.6 MiB/s wr, 650 op/s 2024-05-10T11:29:14.056 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:29:14.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 51799 25.09 MB/sec warmup 118 sec latency 316.511 ms 2024-05-10T11:29:15.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:14 smithi027 ceph-mon[30054]: pgmap v503: 129 pgs: 129 active+clean; 976 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 8.3 MiB/s wr, 632 op/s 2024-05-10T11:29:15.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 52263 25.10 MB/sec warmup 119 sec latency 187.589 ms 2024-05-10T11:29:15.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:14 smithi189 ceph-mon[34842]: pgmap v503: 129 pgs: 129 active+clean; 976 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 8.3 MiB/s wr, 632 op/s 2024-05-10T11:29:15.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:14 smithi076 ceph-mon[34152]: pgmap v503: 129 pgs: 129 active+clean; 976 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 8.3 MiB/s wr, 632 op/s 2024-05-10T11:29:16.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.554+0000 7efd3ac00700 1 -- 172.21.15.27:0/609641387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd34104590 msgr2=0x7efd34104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:16.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.554+0000 7efd3ac00700 1 --2- 172.21.15.27:0/609641387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd34104590 0x7efd34104970 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7efd28009990 tx=0x7efd28009ca0 comp rx=0 tx=0).stop 2024-05-10T11:29:16.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.554+0000 7efd3ac00700 1 -- 172.21.15.27:0/609641387 shutdown_connections 2024-05-10T11:29:16.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.554+0000 7efd3ac00700 1 --2- 172.21.15.27:0/609641387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3410d9c0 0x7efd3410fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:16.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.554+0000 7efd3ac00700 1 --2- 172.21.15.27:0/609641387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd34104590 0x7efd34104970 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:16.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.554+0000 7efd3ac00700 1 --2- 172.21.15.27:0/609641387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd34104eb0 0x7efd3410d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:16.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.554+0000 7efd3ac00700 1 -- 172.21.15.27:0/609641387 >> 172.21.15.27:0/609641387 conn(0x7efd340fddc0 msgr2=0x7efd341001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:16.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.554+0000 7efd3ac00700 1 -- 172.21.15.27:0/609641387 shutdown_connections 2024-05-10T11:29:16.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.554+0000 7efd3ac00700 1 -- 172.21.15.27:0/609641387 wait complete. 2024-05-10T11:29:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.556+0000 7efd3ac00700 1 Processor -- start 2024-05-10T11:29:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.556+0000 7efd3ac00700 1 -- start start 2024-05-10T11:29:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.556+0000 7efd3ac00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd34104590 0x7efd3419bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.556+0000 7efd3ac00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd34104eb0 0x7efd3419c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.556+0000 7efd3ac00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3410d9c0 0x7efd341a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.556+0000 7efd3ac00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efd34113600 con 0x7efd3410d9c0 2024-05-10T11:29:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.556+0000 7efd3ac00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efd34113480 con 0x7efd34104590 2024-05-10T11:29:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.556+0000 7efd3ac00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efd34113780 con 0x7efd34104eb0 2024-05-10T11:29:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.556+0000 7efd39800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd34104590 0x7efd3419bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.556+0000 7efd39800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd34104590 0x7efd3419bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:59376/0 (socket says 172.21.15.27:59376) 2024-05-10T11:29:16.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.556+0000 7efd39800700 1 -- 172.21.15.27:0/1409781073 learned_addr learned my addr 172.21.15.27:0/1409781073 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:16.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.557+0000 7efd3a200700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3410d9c0 0x7efd341a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:16.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.557+0000 7efd38e00700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd34104eb0 0x7efd3419c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:16.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.557+0000 7efd39800700 1 -- 172.21.15.27:0/1409781073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd34104eb0 msgr2=0x7efd3419c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.557+0000 7efd39800700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd34104eb0 0x7efd3419c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.557+0000 7efd39800700 1 -- 172.21.15.27:0/1409781073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3410d9c0 msgr2=0x7efd341a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.557+0000 7efd39800700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3410d9c0 0x7efd341a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.557+0000 7efd39800700 1 -- 172.21.15.27:0/1409781073 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd341a0d60 con 0x7efd34104590 2024-05-10T11:29:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.557+0000 7efd38e00700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd34104eb0 0x7efd3419c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:29:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.557+0000 7efd39800700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd34104590 0x7efd3419bd50 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7efd28006010 tx=0x7efd280057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.558+0000 7efd32000700 1 -- 172.21.15.27:0/1409781073 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28027070 con 0x7efd34104590 2024-05-10T11:29:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.558+0000 7efd32000700 1 -- 172.21.15.27:0/1409781073 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd28005da0 con 0x7efd34104590 2024-05-10T11:29:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.558+0000 7efd32000700 1 -- 172.21.15.27:0/1409781073 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd280173f0 con 0x7efd34104590 2024-05-10T11:29:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.558+0000 7efd3ac00700 1 -- 172.21.15.27:0/1409781073 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd341a0ff0 con 0x7efd34104590 2024-05-10T11:29:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.558+0000 7efd3ac00700 1 -- 172.21.15.27:0/1409781073 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd341a88d0 con 0x7efd34104590 2024-05-10T11:29:16.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.559+0000 7efd3ac00700 1 -- 172.21.15.27:0/1409781073 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd3406a190 con 0x7efd34104590 2024-05-10T11:29:16.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.561+0000 7efd32000700 1 -- 172.21.15.27:0/1409781073 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7efd2802a070 con 0x7efd34104590 2024-05-10T11:29:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.563+0000 7efd32000700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd1c0b2a70 0x7efd1c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.563+0000 7efd32000700 1 -- 172.21.15.27:0/1409781073 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7efd280e38f0 con 0x7efd34104590 2024-05-10T11:29:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.563+0000 7efd38e00700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd1c0b2a70 0x7efd1c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:16.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.564+0000 7efd38e00700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd1c0b2a70 0x7efd1c0b4f30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efd3410f220 tx=0x7efd20009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:16.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.571+0000 7efd32000700 1 -- 172.21.15.27:0/1409781073 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7efd28013070 con 0x7efd34104590 2024-05-10T11:29:16.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.953+0000 7efd3ac00700 1 -- 172.21.15.27:0/1409781073 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd341a8aa0 con 0x7efd34104590 2024-05-10T11:29:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.954+0000 7efd32000700 1 -- 172.21.15.27:0/1409781073 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11945 (secure 0 0 0) 0x7efd280aaed0 con 0x7efd34104590 2024-05-10T11:29:16.955 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:29:16.955 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:27:21.585392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24484,24527,24472,24488,24467]},"id":1}]} 2024-05-10T11:29:16.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.957+0000 7efd3ac00700 1 -- 172.21.15.27:0/1409781073 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd1c0b2a70 msgr2=0x7efd1c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:16.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.957+0000 7efd3ac00700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd1c0b2a70 0x7efd1c0b4f30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efd3410f220 tx=0x7efd20009260 comp rx=0 tx=0).stop 2024-05-10T11:29:16.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.957+0000 7efd3ac00700 1 -- 172.21.15.27:0/1409781073 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd34104590 msgr2=0x7efd3419bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:16.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.957+0000 7efd3ac00700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd34104590 0x7efd3419bd50 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7efd28006010 tx=0x7efd280057c0 comp rx=0 tx=0).stop 2024-05-10T11:29:16.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.957+0000 7efd3ac00700 1 -- 172.21.15.27:0/1409781073 shutdown_connections 2024-05-10T11:29:16.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.957+0000 7efd3ac00700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7efd1c0b2a70 0x7efd1c0b4f30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:16.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.957+0000 7efd3ac00700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd3410d9c0 0x7efd341a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:16.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.958+0000 7efd3ac00700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd34104590 0x7efd3419bd50 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:16.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.958+0000 7efd3ac00700 1 --2- 172.21.15.27:0/1409781073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd34104eb0 0x7efd3419c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:16.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.958+0000 7efd3ac00700 1 -- 172.21.15.27:0/1409781073 >> 172.21.15.27:0/1409781073 conn(0x7efd340fddc0 msgr2=0x7efd341001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:16.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.958+0000 7efd3ac00700 1 -- 172.21.15.27:0/1409781073 shutdown_connections 2024-05-10T11:29:16.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:16.958+0000 7efd3ac00700 1 -- 172.21.15.27:0/1409781073 wait complete. 2024-05-10T11:29:16.960 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 40 2024-05-10T11:29:17.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 53327 26.90 MB/sec execute 1 sec latency 260.821 ms 2024-05-10T11:29:17.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:16 smithi189 ceph-mon[34842]: pgmap v504: 129 pgs: 129 active+clean; 998 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.6 MiB/s wr, 517 op/s 2024-05-10T11:29:17.332 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:16 smithi027 ceph-mon[30054]: pgmap v504: 129 pgs: 129 active+clean; 998 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.6 MiB/s wr, 517 op/s 2024-05-10T11:29:17.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:16 smithi076 ceph-mon[34152]: pgmap v504: 129 pgs: 129 active+clean; 998 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.6 MiB/s wr, 517 op/s 2024-05-10T11:29:17.824 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:29:18.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:17 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1409781073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:29:18.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 53765 23.84 MB/sec execute 2 sec latency 136.383 ms 2024-05-10T11:29:18.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:17 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1409781073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:29:18.409 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:29:18.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:17 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1409781073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:29:19.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 54117 23.31 MB/sec execute 3 sec latency 294.352 ms 2024-05-10T11:29:19.162 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:18 smithi027 ceph-mon[30054]: pgmap v505: 129 pgs: 129 active+clean; 998 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.6 MiB/s wr, 517 op/s 2024-05-10T11:29:19.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:18 smithi189 ceph-mon[34842]: pgmap v505: 129 pgs: 129 active+clean; 998 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.6 MiB/s wr, 517 op/s 2024-05-10T11:29:19.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:18 smithi076 ceph-mon[34152]: pgmap v505: 129 pgs: 129 active+clean; 998 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.6 MiB/s wr, 517 op/s 2024-05-10T11:29:20.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 54437 21.65 MB/sec execute 4 sec latency 153.468 ms 2024-05-10T11:29:21.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 54876 21.71 MB/sec execute 5 sec latency 110.155 ms 2024-05-10T11:29:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.224+0000 7fa2c3400700 1 -- 172.21.15.27:0/3960893006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2bc109510 msgr2=0x7fa2bc1098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.224+0000 7fa2c3400700 1 --2- 172.21.15.27:0/3960893006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2bc109510 0x7fa2bc1098f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00098f0 tx=0x7fa2b0009c00 comp rx=0 tx=0).stop 2024-05-10T11:29:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.224+0000 7fa2c3400700 1 -- 172.21.15.27:0/3960893006 shutdown_connections 2024-05-10T11:29:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.224+0000 7fa2c3400700 1 --2- 172.21.15.27:0/3960893006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2bc109510 0x7fa2bc1098f0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.224+0000 7fa2c3400700 1 --2- 172.21.15.27:0/3960893006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2bc06bb40 0x7fa2bc110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.224+0000 7fa2c3400700 1 --2- 172.21.15.27:0/3960893006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc06b180 0x7fa2bc06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.224+0000 7fa2c3400700 1 -- 172.21.15.27:0/3960893006 >> 172.21.15.27:0/3960893006 conn(0x7fa2bc079350 msgr2=0x7fa2bc079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.225+0000 7fa2c3400700 1 -- 172.21.15.27:0/3960893006 shutdown_connections 2024-05-10T11:29:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.225+0000 7fa2c3400700 1 -- 172.21.15.27:0/3960893006 wait complete. 2024-05-10T11:29:21.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:20 smithi027 ceph-mon[30054]: pgmap v506: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 13 MiB/s wr, 698 op/s 2024-05-10T11:29:21.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.226+0000 7fa2c3400700 1 Processor -- start 2024-05-10T11:29:21.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.226+0000 7fa2c3400700 1 -- start start 2024-05-10T11:29:21.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.227+0000 7fa2c3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2bc06b180 0x7fa2bc102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.227+0000 7fa2c3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc06bb40 0x7fa2bc1029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.227+0000 7fa2c0a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2bc06b180 0x7fa2bc102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.227+0000 7fa2c3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2bc109510 0x7fa2bc102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.227+0000 7fa2c3400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc113670 con 0x7fa2bc06bb40 2024-05-10T11:29:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.227+0000 7fa2c3400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc1134f0 con 0x7fa2bc06b180 2024-05-10T11:29:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.227+0000 7fa2c3400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc1137f0 con 0x7fa2bc109510 2024-05-10T11:29:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.228+0000 7fa2bbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc06bb40 0x7fa2bc1029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.228+0000 7fa2bbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc06bb40 0x7fa2bc1029b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35982/0 (socket says 172.21.15.27:35982) 2024-05-10T11:29:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.228+0000 7fa2bbe00700 1 -- 172.21.15.27:0/3579909224 learned_addr learned my addr 172.21.15.27:0/3579909224 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:21.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.227+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2bc06b180 0x7fa2bc102470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:59404/0 (socket says 172.21.15.27:59404) 2024-05-10T11:29:21.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.228+0000 7fa2c1400700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2bc109510 0x7fa2bc102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.228+0000 7fa2bbe00700 1 -- 172.21.15.27:0/3579909224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2bc109510 msgr2=0x7fa2bc102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.228+0000 7fa2bbe00700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2bc109510 0x7fa2bc102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.228+0000 7fa2bbe00700 1 -- 172.21.15.27:0/3579909224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2bc06b180 msgr2=0x7fa2bc102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.228+0000 7fa2bbe00700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2bc06b180 0x7fa2bc102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.228+0000 7fa2bbe00700 1 -- 172.21.15.27:0/3579909224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2bc1a8130 con 0x7fa2bc06bb40 2024-05-10T11:29:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.228+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2bc06b180 0x7fa2bc102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:29:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.229+0000 7fa2bbe00700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc06bb40 0x7fa2bc1029b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800e9a0 tx=0x7fa2a800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.229+0000 7fa2b9600700 1 -- 172.21.15.27:0/3579909224 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a800ca10 con 0x7fa2bc06bb40 2024-05-10T11:29:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.229+0000 7fa2c3400700 1 -- 172.21.15.27:0/3579909224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2bc1a8420 con 0x7fa2bc06bb40 2024-05-10T11:29:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.229+0000 7fa2b9600700 1 -- 172.21.15.27:0/3579909224 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2a800cbb0 con 0x7fa2bc06bb40 2024-05-10T11:29:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.229+0000 7fa2b9600700 1 -- 172.21.15.27:0/3579909224 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8018680 con 0x7fa2bc06bb40 2024-05-10T11:29:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.229+0000 7fa2c3400700 1 -- 172.21.15.27:0/3579909224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2bc1a8960 con 0x7fa2bc06bb40 2024-05-10T11:29:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.233+0000 7fa2b9600700 1 -- 172.21.15.27:0/3579909224 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa2a8010430 con 0x7fa2bc06bb40 2024-05-10T11:29:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.235+0000 7fa2cdd49b80 1 Processor -- start 2024-05-10T11:29:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.235+0000 7fa2cdd49b80 1 -- start start 2024-05-10T11:29:21.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.236+0000 7fa2cdd49b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562695082060 0x562695082440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.236+0000 7fa2cdd49b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x562695086490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.236+0000 7fa2b9600700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa2a40b2a70 0x7fa2a40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.236+0000 7fa2cdd49b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5626950869d0 0x562695086e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.236+0000 7fa2cdd49b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562694fd4a20 con 0x562695082060 2024-05-10T11:29:21.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.236+0000 7fa2cdd49b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562694f72760 con 0x562695082980 2024-05-10T11:29:21.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.236+0000 7fa2cdd49b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562694fd09e0 con 0x5626950869d0 2024-05-10T11:29:21.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:20 smithi189 ceph-mon[34842]: pgmap v506: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 13 MiB/s wr, 698 op/s 2024-05-10T11:29:21.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.236+0000 7fa2bbe00700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa2a40b2a70 0x7fa2a40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.236+0000 7fa2c1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562695082060 0x562695082440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.237+0000 7fa2c1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562695082060 0x562695082440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35984/0 (socket says 172.21.15.27:35984) 2024-05-10T11:29:21.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.237+0000 7fa2c1400700 1 -- 172.21.15.27:0/626868182 learned_addr learned my addr 172.21.15.27:0/626868182 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:21.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.237+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/626868182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x562695086490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.237+0000 7fa2c1400700 1 --2- 172.21.15.27:0/626868182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5626950869d0 0x562695086e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.237+0000 7fa2b9600700 1 -- 172.21.15.27:0/3579909224 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7fa2a80e2c00 con 0x7fa2bc06bb40 2024-05-10T11:29:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.237+0000 7fa2bbe00700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa2a40b2a70 0x7fa2a40b4f30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa2a80a6cc0 tx=0x7fa2a80a98e0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.238+0000 7fa2c0a00700 1 -- 172.21.15.27:0/626868182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5626950869d0 msgr2=0x562695086e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.238+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/626868182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5626950869d0 0x562695086e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.238+0000 7fa2c0a00700 1 -- 172.21.15.27:0/626868182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562695082060 msgr2=0x562695082440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.238+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/626868182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562695082060 0x562695082440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.238+0000 7fa2c0a00700 1 -- 172.21.15.27:0/626868182 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562694f218d0 con 0x562695082980 2024-05-10T11:29:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.238+0000 7fa2c1400700 1 --2- 172.21.15.27:0/626868182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5626950869d0 0x562695086e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:29:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.238+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/626868182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x562695086490 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa2b000c010 tx=0x7fa2b000bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44c820eecab4dd97 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:21.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.239+0000 7fa2ae000700 1 -- 172.21.15.27:0/626868182 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b000f720 con 0x562695082980 2024-05-10T11:29:21.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.239+0000 7fa2ae000700 1 -- 172.21.15.27:0/626868182 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b000f8c0 con 0x562695082980 2024-05-10T11:29:21.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.239+0000 7fa2ae000700 1 -- 172.21.15.27:0/626868182 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0004ba0 con 0x562695082980 2024-05-10T11:29:21.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.239+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/626868182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 msgr2=0x562695086490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:21.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.239+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/626868182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x562695086490 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa2b000c010 tx=0x7fa2b000bf60 comp rx=0 tx=0).stop 2024-05-10T11:29:21.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.239+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/626868182 shutdown_connections 2024-05-10T11:29:21.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.239+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/626868182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5626950869d0 0x562695086e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.239+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/626868182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562695082060 0x562695082440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.239+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/626868182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x562695086490 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.239+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/626868182 >> 172.21.15.27:0/626868182 conn(0x562694fb0b60 msgr2=0x56269507c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.240+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/626868182 shutdown_connections 2024-05-10T11:29:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.240+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/626868182 wait complete. 2024-05-10T11:29:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.240+0000 7fa2cdd49b80 1 -- start start 2024-05-10T11:29:21.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.240+0000 7fa2cdd49b80 1 Processor -- start 2024-05-10T11:29:21.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.241+0000 7fa2cdd49b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562695082060 0x56269508dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.241+0000 7fa2cdd49b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x56269508b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.241+0000 7fa2cdd49b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5626950869d0 0x56269508b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.241+0000 7fa2cdd49b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56269508eb80 con 0x562695082060 2024-05-10T11:29:21.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.241+0000 7fa2cdd49b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x562694f3ab20 con 0x562695082980 2024-05-10T11:29:21.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.241+0000 7fa2cdd49b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56269508ed00 con 0x5626950869d0 2024-05-10T11:29:21.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.241+0000 7fa2cdd49b80 10 client.0 _ll_register_callbacks cb 0x562694f7bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:21.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.241+0000 7fa2c0a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x56269508b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.242+0000 7fa2c1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562695082060 0x56269508dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.242+0000 7fa2c1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562695082060 0x56269508dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35986/0 (socket says 172.21.15.27:35986) 2024-05-10T11:29:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.242+0000 7fa2c1400700 1 -- 172.21.15.27:0/2314711448 learned_addr learned my addr 172.21.15.27:0/2314711448 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.242+0000 7fa2c1400700 1 --2- 172.21.15.27:0/2314711448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5626950869d0 0x56269508b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.242+0000 7fa2c1400700 1 -- 172.21.15.27:0/2314711448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 msgr2=0x56269508b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.242+0000 7fa2c1400700 1 --2- 172.21.15.27:0/2314711448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x56269508b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.242+0000 7fa2c1400700 1 -- 172.21.15.27:0/2314711448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562695082060 msgr2=0x56269508dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.242+0000 7fa2c1400700 1 --2- 172.21.15.27:0/2314711448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562695082060 0x56269508dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.242+0000 7fa2c1400700 1 -- 172.21.15.27:0/2314711448 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562694f9a420 con 0x5626950869d0 2024-05-10T11:29:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.243+0000 7fa2c1400700 1 --2- 172.21.15.27:0/2314711448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5626950869d0 0x56269508b6c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009e60 tx=0x7fa2b4004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9794ba74ec69ea4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.243+0000 7fa2a2a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:29:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.243+0000 7fa2a2a00700 1 -- 172.21.15.27:0/2314711448 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4019660 con 0x5626950869d0 2024-05-10T11:29:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.243+0000 7fa2a2a00700 1 -- 172.21.15.27:0/2314711448 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7fa2b4019800 con 0x5626950869d0 2024-05-10T11:29:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.243+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/2314711448 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2b000f8c0 con 0x5626950869d0 2024-05-10T11:29:21.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.244+0000 7fa2a2a00700 1 -- 172.21.15.27:0/2314711448 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fa2b4003e10 con 0x5626950869d0 2024-05-10T11:29:21.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.244+0000 7fa2cdd49b80 10 client.27424 fetch_fsmap learned FSMap version 40 2024-05-10T11:29:21.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.244+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/2314711448 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5626950a1b00 con 0x5626950869d0 2024-05-10T11:29:21.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.244+0000 7fa2a2a00700 1 -- 172.21.15.27:0/2314711448 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b401b450 con 0x5626950869d0 2024-05-10T11:29:21.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.244+0000 7fa2a2a00700 1 -- 172.21.15.27:0/2314711448 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fa2b4015400 con 0x5626950869d0 2024-05-10T11:29:21.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.244+0000 7fa2cdd49b80 10 client.27424 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:29:21.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.244+0000 7fa2cdd49b80 10 client.27424 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:29:21.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.244+0000 7fa2cdd49b80 20 client.27424 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:21.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.244+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/2314711448 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5626950a4140 0x5626950a6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.244+0000 7fa2cdd49b80 4 client.27424 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:29:21.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.244+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/2314711448 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa2b000f8c0 con 0x5626950a4140 2024-05-10T11:29:21.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.245+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/2314711448 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5626950a4140 0x5626950a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.246+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/2314711448 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5626950a4140 0x5626950a6530 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4b18ff07ffbc23f1 server_cookie=700a46b5a5290c22 in_seq=0 out_seq=0 2024-05-10T11:29:21.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.246+0000 7fa2a2a00700 10 client.27424 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:29:21.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.247+0000 7fa2a2a00700 1 -- 172.21.15.27:0/2314711448 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fa2b000f8c0 con 0x5626950a4140 2024-05-10T11:29:21.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.247+0000 7fa2a2a00700 10 client.27424 handle_command_reply: tid=0 2024-05-10T11:29:21.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.247+0000 7fa2cdd49b80 1 client.27424 shutdown 2024-05-10T11:29:21.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.247+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/2314711448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5626950869d0 msgr2=0x56269508b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:21.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.247+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/2314711448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5626950869d0 0x56269508b6c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009e60 tx=0x7fa2b4004ad0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.248+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/2314711448 shutdown_connections 2024-05-10T11:29:21.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.248+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/2314711448 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x5626950a4140 0x5626950a6530 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.248+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/2314711448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5626950869d0 0x56269508b6c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.248+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/2314711448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562695082060 0x56269508dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.248+0000 7fa2a2a00700 0 client.27424 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:29:21.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.248+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/2314711448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x56269508b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.248+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/2314711448 >> 172.21.15.27:0/2314711448 conn(0x562694fb0b60 msgr2=0x5626950772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:21.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.248+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/2314711448 shutdown_connections 2024-05-10T11:29:21.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.248+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/2314711448 wait complete. 2024-05-10T11:29:21.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.248+0000 7fa2cdd49b80 20 client.27424 trim_cache size 0 max 16384 2024-05-10T11:29:21.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.272+0000 7fa2cdd49b80 1 Processor -- start 2024-05-10T11:29:21.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.272+0000 7fa2cdd49b80 1 -- start start 2024-05-10T11:29:21.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.273+0000 7fa2cdd49b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562695082060 0x562695097c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.273+0000 7fa2cdd49b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x56269508a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.273+0000 7fa2cdd49b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626950869d0 0x562695077130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.273+0000 7fa2cdd49b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc1ad4f0 con 0x5626950869d0 2024-05-10T11:29:21.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.273+0000 7fa2cdd49b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc1af120 con 0x562695082980 2024-05-10T11:29:21.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.273+0000 7fa2cdd49b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc1adbc0 con 0x562695082060 2024-05-10T11:29:21.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.273+0000 7fa2c1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562695082060 0x562695097c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.273+0000 7fa2c0a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x56269508a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.273+0000 7fa2c1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562695082060 0x562695097c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60952/0 (socket says 172.21.15.27:60952) 2024-05-10T11:29:21.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.273+0000 7fa2c1400700 1 -- 172.21.15.27:0/4028888471 learned_addr learned my addr 172.21.15.27:0/4028888471 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:21.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.273+0000 7fa2c1400700 1 --2- 172.21.15.27:0/4028888471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626950869d0 0x562695077130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.274+0000 7fa2c0a00700 1 -- 172.21.15.27:0/4028888471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562695082060 msgr2=0x562695097c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:21.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.274+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/4028888471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562695082060 0x562695097c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.274+0000 7fa2c0a00700 1 -- 172.21.15.27:0/4028888471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626950869d0 msgr2=0x562695077130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:21.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.274+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/4028888471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626950869d0 0x562695077130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.274+0000 7fa2c0a00700 1 -- 172.21.15.27:0/4028888471 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56269507d270 con 0x562695082980 2024-05-10T11:29:21.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.274+0000 7fa2c1400700 1 --2- 172.21.15.27:0/4028888471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626950869d0 0x562695077130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:29:21.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.274+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/4028888471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x56269508a0e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0000c00 tx=0x7fa2b0005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=891a55a61b730c29 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:21.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.274+0000 7fa2ad600700 1 -- 172.21.15.27:0/4028888471 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0005130 con 0x562695082980 2024-05-10T11:29:21.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.274+0000 7fa2ad600700 1 -- 172.21.15.27:0/4028888471 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b0003680 con 0x562695082980 2024-05-10T11:29:21.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.274+0000 7fa2ad600700 1 -- 172.21.15.27:0/4028888471 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b000b7c0 con 0x562695082980 2024-05-10T11:29:21.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.275+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/4028888471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 msgr2=0x56269508a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:21.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.275+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/4028888471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x56269508a0e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0000c00 tx=0x7fa2b0005040 comp rx=0 tx=0).stop 2024-05-10T11:29:21.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.275+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/4028888471 shutdown_connections 2024-05-10T11:29:21.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.275+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/4028888471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562695082060 0x562695097c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.275+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/4028888471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626950869d0 0x562695077130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.275+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/4028888471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x56269508a0e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.275+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/4028888471 >> 172.21.15.27:0/4028888471 conn(0x562694fb0b60 msgr2=0x562695076a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:21.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.275+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/4028888471 shutdown_connections 2024-05-10T11:29:21.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.275+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/4028888471 wait complete. 2024-05-10T11:29:21.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.276+0000 7fa2cdd49b80 1 -- start start 2024-05-10T11:29:21.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.276+0000 7fa2cdd49b80 1 Processor -- start 2024-05-10T11:29:21.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.277+0000 7fa2cdd49b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562695082060 0x5626950944a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.277+0000 7fa2cdd49b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x562695084480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.277+0000 7fa2cdd49b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626950869d0 0x5626950849c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.277+0000 7fa2cdd49b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc1ae990 con 0x5626950869d0 2024-05-10T11:29:21.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.277+0000 7fa2cdd49b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc1ae2c0 con 0x562695082980 2024-05-10T11:29:21.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.277+0000 7fa2cdd49b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc1ab4d0 con 0x562695082060 2024-05-10T11:29:21.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.277+0000 7fa2cdd49b80 10 client.0 _ll_register_callbacks cb 0x562694f7bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:21.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.277+0000 7fa2c0a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x562695084480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.277+0000 7fa2c0a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x562695084480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:59438/0 (socket says 172.21.15.27:59438) 2024-05-10T11:29:21.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.277+0000 7fa2c0a00700 1 -- 172.21.15.27:0/371444246 learned_addr learned my addr 172.21.15.27:0/371444246 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:21.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.277+0000 7fa2c1400700 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562695082060 0x5626950944a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.278+0000 7fa2c1400700 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626950869d0 0x5626950849c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.278+0000 7fa2c0a00700 1 -- 172.21.15.27:0/371444246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562695082060 msgr2=0x5626950944a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:21.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.278+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562695082060 0x5626950944a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.278+0000 7fa2c0a00700 1 -- 172.21.15.27:0/371444246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626950869d0 msgr2=0x5626950849c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.278+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626950869d0 0x5626950849c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.278+0000 7fa2c0a00700 1 -- 172.21.15.27:0/371444246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562695087e30 con 0x562695082980 2024-05-10T11:29:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.278+0000 7fa2c0a00700 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x562695084480 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0005a80 tx=0x7fa2b00051e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3835db51cf0813d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.278+0000 7fa2a2a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:29:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.279+0000 7fa2a2a00700 1 -- 172.21.15.27:0/371444246 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0003a30 con 0x562695082980 2024-05-10T11:29:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.279+0000 7fa2a2a00700 1 -- 172.21.15.27:0/371444246 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7fa2b0003bd0 con 0x562695082980 2024-05-10T11:29:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.279+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/371444246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2b0003680 con 0x562695082980 2024-05-10T11:29:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.279+0000 7fa2a2a00700 1 -- 172.21.15.27:0/371444246 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fa2b0047050 con 0x562695082980 2024-05-10T11:29:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.279+0000 7fa2cdd49b80 10 client.27443 fetch_fsmap learned FSMap version 40 2024-05-10T11:29:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.279+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/371444246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56269510eaf0 con 0x562695082980 2024-05-10T11:29:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.279+0000 7fa2a2a00700 1 -- 172.21.15.27:0/371444246 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b0003680 con 0x562695082980 2024-05-10T11:29:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.280+0000 7fa2a2a00700 1 -- 172.21.15.27:0/371444246 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fa2b003e600 con 0x562695082980 2024-05-10T11:29:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.280+0000 7fa2cdd49b80 10 client.27443 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:29:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.280+0000 7fa2cdd49b80 10 client.27443 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:29:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.280+0000 7fa2cdd49b80 20 client.27443 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.280+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56269510ed70 0x562695111160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.280+0000 7fa2cdd49b80 4 client.27443 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:29:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.280+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/371444246 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56269504f0a0 con 0x56269510ed70 2024-05-10T11:29:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.280+0000 7fa2c1400700 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56269510ed70 0x562695111160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.281+0000 7fa2c1400700 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56269510ed70 0x562695111160 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b2fc8492f4d8d3a server_cookie=22ae6b90c776b024 in_seq=0 out_seq=0 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.281+0000 7fa2a2a00700 10 client.27443 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.281+0000 7fa2a2a00700 1 -- 172.21.15.27:0/371444246 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56269504f0a0 con 0x56269510ed70 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.281+0000 7fa2a2a00700 10 client.27443 handle_command_reply: tid=0 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.281+0000 7fa2cdd49b80 1 client.27443 shutdown 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.281+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/371444246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 msgr2=0x562695084480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.281+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x562695084480 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0005a80 tx=0x7fa2b00051e0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.282+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/371444246 shutdown_connections 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.282+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x56269510ed70 0x562695111160 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.282+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562695082060 0x5626950944a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.282+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626950869d0 0x5626950849c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.282+0000 7fa2cdd49b80 1 --2- 172.21.15.27:0/371444246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562695082980 0x562695084480 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.282+0000 7fa2a2a00700 0 client.27443 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.282+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/371444246 >> 172.21.15.27:0/371444246 conn(0x562694fb0b60 msgr2=0x562695080e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.282+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/371444246 shutdown_connections 2024-05-10T11:29:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.282+0000 7fa2cdd49b80 1 -- 172.21.15.27:0/371444246 wait complete. 2024-05-10T11:29:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.282+0000 7fa2cdd49b80 20 client.27443 trim_cache size 0 max 16384 2024-05-10T11:29:21.285 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:29:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.283+0000 7fa2c3400700 1 -- 172.21.15.27:0/3579909224 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa2a40b2a70 msgr2=0x7fa2a40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.283+0000 7fa2c3400700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa2a40b2a70 0x7fa2a40b4f30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa2a80a6cc0 tx=0x7fa2a80a98e0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.283+0000 7fa2c3400700 1 -- 172.21.15.27:0/3579909224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc06bb40 msgr2=0x7fa2bc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.283+0000 7fa2c3400700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc06bb40 0x7fa2bc1029b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800e9a0 tx=0x7fa2a800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.283+0000 7fa2c3400700 1 -- 172.21.15.27:0/3579909224 shutdown_connections 2024-05-10T11:29:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.283+0000 7fa2c3400700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa2a40b2a70 0x7fa2a40b4f30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.283+0000 7fa2c3400700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2bc109510 0x7fa2bc102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.283+0000 7fa2c3400700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2bc06b180 0x7fa2bc102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.283+0000 7fa2c3400700 1 --2- 172.21.15.27:0/3579909224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc06bb40 0x7fa2bc1029b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.283+0000 7fa2c3400700 1 -- 172.21.15.27:0/3579909224 >> 172.21.15.27:0/3579909224 conn(0x7fa2bc079350 msgr2=0x7fa2bc101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.284+0000 7fa2c3400700 1 -- 172.21.15.27:0/3579909224 shutdown_connections 2024-05-10T11:29:21.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:21.284+0000 7fa2c3400700 1 -- 172.21.15.27:0/3579909224 wait complete. 2024-05-10T11:29:21.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:20 smithi076 ceph-mon[34152]: pgmap v506: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 13 MiB/s wr, 698 op/s 2024-05-10T11:29:22.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 55311 21.89 MB/sec execute 6 sec latency 160.359 ms 2024-05-10T11:29:22.179 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:29:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:29:23.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:22 smithi027 ceph-mon[30054]: pgmap v507: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 8.0 MiB/s wr, 412 op/s 2024-05-10T11:29:23.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 55925 22.16 MB/sec execute 7 sec latency 87.737 ms 2024-05-10T11:29:23.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:22 smithi189 ceph-mon[34842]: pgmap v507: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 8.0 MiB/s wr, 412 op/s 2024-05-10T11:29:23.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:22 smithi076 ceph-mon[34152]: pgmap v507: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 8.0 MiB/s wr, 412 op/s 2024-05-10T11:29:24.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 56215 22.34 MB/sec execute 8 sec latency 435.500 ms 2024-05-10T11:29:25.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:24 smithi027 ceph-mon[30054]: pgmap v508: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 8.0 MiB/s wr, 412 op/s 2024-05-10T11:29:25.150 INFO:tasks.workunit.client.0.smithi027.stdout: 10 56534 21.83 MB/sec execute 9 sec latency 847.710 ms 2024-05-10T11:29:25.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:24 smithi189 ceph-mon[34842]: pgmap v508: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 8.0 MiB/s wr, 412 op/s 2024-05-10T11:29:25.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:24 smithi076 ceph-mon[34152]: pgmap v508: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 8.0 MiB/s wr, 412 op/s 2024-05-10T11:29:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.441+0000 7f28f2600700 1 -- 172.21.15.27:0/1232447882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ec075ff0 msgr2=0x7f28ec114130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.441+0000 7f28f2600700 1 --2- 172.21.15.27:0/1232447882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ec075ff0 0x7f28ec114130 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f28d8009990 tx=0x7f28d8009ca0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.442+0000 7f28f2600700 1 -- 172.21.15.27:0/1232447882 shutdown_connections 2024-05-10T11:29:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.442+0000 7f28f2600700 1 --2- 172.21.15.27:0/1232447882 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28ec114670 0x7f28ec116a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.442+0000 7f28f2600700 1 --2- 172.21.15.27:0/1232447882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28ec0756d0 0x7f28ec075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.442+0000 7f28f2600700 1 --2- 172.21.15.27:0/1232447882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ec075ff0 0x7f28ec114130 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.442+0000 7f28f2600700 1 -- 172.21.15.27:0/1232447882 >> 172.21.15.27:0/1232447882 conn(0x7f28ec07a650 msgr2=0x7f28ec07aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:25.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.442+0000 7f28f2600700 1 -- 172.21.15.27:0/1232447882 shutdown_connections 2024-05-10T11:29:25.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.443+0000 7f28f2600700 1 -- 172.21.15.27:0/1232447882 wait complete. 2024-05-10T11:29:25.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.444+0000 7f28f2600700 1 Processor -- start 2024-05-10T11:29:25.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.444+0000 7f28f2600700 1 -- start start 2024-05-10T11:29:25.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.445+0000 7f28f2600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28ec0756d0 0x7f28ec1106a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.445+0000 7f28f2600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ec075ff0 0x7f28ec110be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.445+0000 7f28f2600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28ec114670 0x7f28ec109980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.445+0000 7f28f2600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec119d20 con 0x7f28ec114670 2024-05-10T11:29:25.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.445+0000 7f28f2600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec119ba0 con 0x7f28ec0756d0 2024-05-10T11:29:25.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.445+0000 7f28f2600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec119ea0 con 0x7f28ec075ff0 2024-05-10T11:29:25.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.445+0000 7f28ebe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28ec114670 0x7f28ec109980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.445+0000 7f28ebe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28ec114670 0x7f28ec109980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41598/0 (socket says 172.21.15.27:41598) 2024-05-10T11:29:25.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.445+0000 7f28ebe00700 1 -- 172.21.15.27:0/2505573100 learned_addr learned my addr 172.21.15.27:0/2505573100 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:25.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.446+0000 7f28ebe00700 1 -- 172.21.15.27:0/2505573100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ec075ff0 msgr2=0x7f28ec110be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T11:29:25.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.446+0000 7f28ebe00700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ec075ff0 0x7f28ec110be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.446+0000 7f28eb400700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28ec0756d0 0x7f28ec1106a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.446+0000 7f28ebe00700 1 -- 172.21.15.27:0/2505573100 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28ec0756d0 msgr2=0x7f28ec1106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:25.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.446+0000 7f28ebe00700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28ec0756d0 0x7f28ec1106a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.446+0000 7f28ebe00700 1 -- 172.21.15.27:0/2505573100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ec10a280 con 0x7f28ec114670 2024-05-10T11:29:25.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.447+0000 7f28ebe00700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28ec114670 0x7f28ec109980 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f28e400b9e0 tx=0x7f28e400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.449+0000 7f28e3e00700 1 -- 172.21.15.27:0/2505573100 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4019070 con 0x7f28ec114670 2024-05-10T11:29:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.449+0000 7f28e3e00700 1 -- 172.21.15.27:0/2505573100 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28e400ce00 con 0x7f28ec114670 2024-05-10T11:29:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.449+0000 7f28e3e00700 1 -- 172.21.15.27:0/2505573100 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4007370 con 0x7f28ec114670 2024-05-10T11:29:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.449+0000 7f28f2600700 1 -- 172.21.15.27:0/2505573100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28ec10a570 con 0x7f28ec114670 2024-05-10T11:29:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.449+0000 7f28f2600700 1 -- 172.21.15.27:0/2505573100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28ec10e4f0 con 0x7f28ec114670 2024-05-10T11:29:25.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.459+0000 7f28fceccb80 1 Processor -- start 2024-05-10T11:29:25.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.460+0000 7f28e3e00700 1 -- 172.21.15.27:0/2505573100 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f28e401c070 con 0x7f28ec114670 2024-05-10T11:29:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.461+0000 7f28fceccb80 1 -- start start 2024-05-10T11:29:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.461+0000 7f28fceccb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fa9060 0x561fd7fa9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.462+0000 7f28fceccb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 0x561fd7fad490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.462+0000 7f28fceccb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 0x561fd7fade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.462+0000 7f28fceccb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561fd7efba20 con 0x561fd7fa9060 2024-05-10T11:29:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.462+0000 7f28fceccb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561fd7e99760 con 0x561fd7fa9980 2024-05-10T11:29:25.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.462+0000 7f28fceccb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561fd7ef79e0 con 0x561fd7fad9d0 2024-05-10T11:29:25.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.462+0000 7f28eaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fa9060 0x561fd7fa9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.462+0000 7f28eaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 0x561fd7fade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.462+0000 7f28eaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 0x561fd7fade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60346/0 (socket says 172.21.15.27:60346) 2024-05-10T11:29:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.462+0000 7f28eaa00700 1 -- 172.21.15.27:0/1454353591 learned_addr learned my addr 172.21.15.27:0/1454353591 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.463+0000 7f28eaa00700 1 -- 172.21.15.27:0/1454353591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 msgr2=0x561fd7fad490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.463+0000 7f28eaa00700 1 --2- 172.21.15.27:0/1454353591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 0x561fd7fad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.463+0000 7f28eaa00700 1 -- 172.21.15.27:0/1454353591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fa9060 msgr2=0x561fd7fa9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.463+0000 7f28eaa00700 1 --2- 172.21.15.27:0/1454353591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fa9060 0x561fd7fa9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.463+0000 7f28eaa00700 1 -- 172.21.15.27:0/1454353591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fd7e488d0 con 0x561fd7fad9d0 2024-05-10T11:29:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.465+0000 7f28eaa00700 1 --2- 172.21.15.27:0/1454353591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 0x561fd7fade30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f28d800b5c0 tx=0x7f28d8009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e95101cce55a2db8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.466+0000 7f28e0c00700 1 -- 172.21.15.27:0/1454353591 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d801f030 con 0x561fd7fad9d0 2024-05-10T11:29:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.466+0000 7f28e0c00700 1 -- 172.21.15.27:0/1454353591 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28d8003d10 con 0x561fd7fad9d0 2024-05-10T11:29:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.466+0000 7f28e0c00700 1 -- 172.21.15.27:0/1454353591 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d8022040 con 0x561fd7fad9d0 2024-05-10T11:29:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.466+0000 7f28e3e00700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f28d00b2a70 0x7f28d00b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.467+0000 7f28fceccb80 1 -- 172.21.15.27:0/1454353591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 msgr2=0x561fd7fade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.467+0000 7f28fceccb80 1 --2- 172.21.15.27:0/1454353591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 0x561fd7fade30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f28d800b5c0 tx=0x7f28d8009310 comp rx=0 tx=0).stop 2024-05-10T11:29:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.467+0000 7f28e3e00700 1 -- 172.21.15.27:0/2505573100 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f28e40e1d40 con 0x7f28ec114670 2024-05-10T11:29:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.467+0000 7f28fceccb80 1 -- 172.21.15.27:0/1454353591 shutdown_connections 2024-05-10T11:29:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.467+0000 7f28fceccb80 1 --2- 172.21.15.27:0/1454353591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 0x561fd7fade30 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.467+0000 7f28fceccb80 1 --2- 172.21.15.27:0/1454353591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fa9060 0x561fd7fa9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.467+0000 7f28fceccb80 1 --2- 172.21.15.27:0/1454353591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 0x561fd7fad490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.467+0000 7f28fceccb80 1 -- 172.21.15.27:0/1454353591 >> 172.21.15.27:0/1454353591 conn(0x561fd7ed7b60 msgr2=0x561fd7fa37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.467+0000 7f28fceccb80 1 -- 172.21.15.27:0/1454353591 shutdown_connections 2024-05-10T11:29:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.468+0000 7f28fceccb80 1 -- 172.21.15.27:0/1454353591 wait complete. 2024-05-10T11:29:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.469+0000 7f28fceccb80 1 -- start start 2024-05-10T11:29:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.469+0000 7f28eb400700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f28d00b2a70 0x7f28d00b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.470+0000 7f28fceccb80 1 Processor -- start 2024-05-10T11:29:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.471+0000 7f28fceccb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 0x561fd7fb4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.471+0000 7f28fceccb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 0x561fd7fb21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.471+0000 7f28fceccb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fad9d0 0x561fd7fb2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.471+0000 7f28eb400700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f28d00b2a70 0x7f28d00b4f30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f28dc005950 tx=0x7f28dc005bd0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.471+0000 7f28eb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 0x561fd7fb4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.471+0000 7f28eb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 0x561fd7fb4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60358/0 (socket says 172.21.15.27:60358) 2024-05-10T11:29:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.471+0000 7f28fceccb80 1 -- 172.21.15.27:0/337166440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561fd7fb5b80 con 0x561fd7fad9d0 2024-05-10T11:29:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.471+0000 7f28fceccb80 1 -- 172.21.15.27:0/337166440 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x561fd7e61b20 con 0x561fd7fa9980 2024-05-10T11:29:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.471+0000 7f28fceccb80 1 -- 172.21.15.27:0/337166440 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561fd7fb5d00 con 0x561fd7fa9060 2024-05-10T11:29:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.471+0000 7f28eb400700 1 -- 172.21.15.27:0/337166440 learned_addr learned my addr 172.21.15.27:0/337166440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.471+0000 7f28eaa00700 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 0x561fd7fb21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.472+0000 7f28fceccb80 10 client.0 _ll_register_callbacks cb 0x561fd7ea2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.472+0000 7f28eaa00700 1 -- 172.21.15.27:0/337166440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 msgr2=0x561fd7fb4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.472+0000 7f28eaa00700 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 0x561fd7fb4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.472+0000 7f28eaa00700 1 -- 172.21.15.27:0/337166440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fad9d0 msgr2=0x561fd7fb2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:25.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.472+0000 7f28ebe00700 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fad9d0 0x561fd7fb2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.473+0000 7f28eaa00700 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fad9d0 0x561fd7fb2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.473+0000 7f28eaa00700 1 -- 172.21.15.27:0/337166440 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fd7ec1420 con 0x561fd7fa9980 2024-05-10T11:29:25.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.473+0000 7f28eb400700 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 0x561fd7fb4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:29:25.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.473+0000 7f28ebe00700 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fad9d0 0x561fd7fb2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:29:25.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.473+0000 7f28fceccb80 1 -- 172.21.15.27:0/337166440 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28d8003d10 con 0x561fd7fa9980 2024-05-10T11:29:25.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.474+0000 7f28eaa00700 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 0x561fd7fb21e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f28d8009990 tx=0x7f28d800bdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86dbe46c8175e2bd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:25.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.474+0000 7f28d5600700 10 client.27446 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:29:25.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.475+0000 7f28d5600700 1 -- 172.21.15.27:0/337166440 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d801f040 con 0x561fd7fa9980 2024-05-10T11:29:25.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.475+0000 7f28d5600700 1 -- 172.21.15.27:0/337166440 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f28d8005280 con 0x561fd7fa9980 2024-05-10T11:29:25.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.476+0000 7f28d5600700 1 -- 172.21.15.27:0/337166440 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f28d8005480 con 0x561fd7fa9980 2024-05-10T11:29:25.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.476+0000 7f28fceccb80 10 client.27446 fetch_fsmap learned FSMap version 40 2024-05-10T11:29:25.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.476+0000 7f28fceccb80 1 -- 172.21.15.27:0/337166440 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({fsmap=40}) -- 0x561fd7fc0310 con 0x561fd7fa9980 2024-05-10T11:29:25.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.477+0000 7f28d5600700 1 -- 172.21.15.27:0/337166440 <== mon.2 v2:172.21.15.76:3300/0 4 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f28d8031460 con 0x561fd7fa9980 2024-05-10T11:29:25.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.477+0000 7f28fceccb80 10 client.27446 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:29:25.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.477+0000 7f28fceccb80 10 client.27446 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:29:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.477+0000 7f28fceccb80 20 client.27446 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.477+0000 7f28fceccb80 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x561fd7fc27e0 0x561fd7fc4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.477+0000 7f28fceccb80 4 client.27446 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:29:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.477+0000 7f28fceccb80 1 -- 172.21.15.27:0/337166440 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fd7f760a0 con 0x561fd7fc27e0 2024-05-10T11:29:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.478+0000 7f28eb400700 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x561fd7fc27e0 0x561fd7fc4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.478+0000 7f28eb400700 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x561fd7fc27e0 0x561fd7fc4bd0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d31fdf6cd68ec58f server_cookie=70bff444d17da6d8 in_seq=0 out_seq=0 2024-05-10T11:29:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.478+0000 7f28d5600700 10 client.27446 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:29:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.479+0000 7f28d5600700 1 -- 172.21.15.27:0/337166440 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561fd7f760a0 con 0x561fd7fc27e0 2024-05-10T11:29:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.479+0000 7f28d5600700 10 client.27446 handle_command_reply: tid=0 2024-05-10T11:29:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.480+0000 7f28fceccb80 1 client.27446 shutdown 2024-05-10T11:29:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.480+0000 7f28fceccb80 1 -- 172.21.15.27:0/337166440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 msgr2=0x561fd7fb21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.480+0000 7f28fceccb80 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 0x561fd7fb21e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f28d8009990 tx=0x7f28d800bdf0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.480+0000 7f28fceccb80 1 -- 172.21.15.27:0/337166440 shutdown_connections 2024-05-10T11:29:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.480+0000 7f28fceccb80 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x561fd7fc27e0 0x561fd7fc4bd0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.480+0000 7f28fceccb80 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 0x561fd7fb4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.480+0000 7f28fceccb80 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fad9d0 0x561fd7fb2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.480+0000 7f28d5600700 0 client.27446 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:29:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.480+0000 7f28fceccb80 1 --2- 172.21.15.27:0/337166440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 0x561fd7fb21e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.480+0000 7f28fceccb80 1 -- 172.21.15.27:0/337166440 >> 172.21.15.27:0/337166440 conn(0x561fd7ed7b60 msgr2=0x561fd7f9e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:25.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.480+0000 7f28fceccb80 1 -- 172.21.15.27:0/337166440 shutdown_connections 2024-05-10T11:29:25.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.481+0000 7f28fceccb80 1 -- 172.21.15.27:0/337166440 wait complete. 2024-05-10T11:29:25.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.481+0000 7f28fceccb80 20 client.27446 trim_cache size 0 max 16384 2024-05-10T11:29:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.515+0000 7f28fceccb80 1 Processor -- start 2024-05-10T11:29:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.516+0000 7f28fceccb80 1 -- start start 2024-05-10T11:29:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.516+0000 7f28fceccb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9060 0x561fd7f9e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.516+0000 7f28fceccb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fa9980 0x561fd7faddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.516+0000 7f28fceccb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 0x561fd7fae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.516+0000 7f28fceccb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec1be230 con 0x561fd7fa9980 2024-05-10T11:29:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.516+0000 7f28fceccb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec1bfe60 con 0x561fd7fa9060 2024-05-10T11:29:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.516+0000 7f28fceccb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec1be900 con 0x561fd7fad9d0 2024-05-10T11:29:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.517+0000 7f28ebe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 0x561fd7fae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.517+0000 7f28ebe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 0x561fd7fae2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60374/0 (socket says 172.21.15.27:60374) 2024-05-10T11:29:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.517+0000 7f28ebe00700 1 -- 172.21.15.27:0/3713223523 learned_addr learned my addr 172.21.15.27:0/3713223523 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.517+0000 7f28eb400700 1 --2- 172.21.15.27:0/3713223523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9060 0x561fd7f9e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.517+0000 7f28eaa00700 1 --2- 172.21.15.27:0/3713223523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fa9980 0x561fd7faddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.518+0000 7f28ebe00700 1 -- 172.21.15.27:0/3713223523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9060 msgr2=0x561fd7f9e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.518+0000 7f28ebe00700 1 --2- 172.21.15.27:0/3713223523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9060 0x561fd7f9e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.518+0000 7f28ebe00700 1 -- 172.21.15.27:0/3713223523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fa9980 msgr2=0x561fd7faddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.518+0000 7f28ebe00700 1 --2- 172.21.15.27:0/3713223523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fa9980 0x561fd7faddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.518+0000 7f28ebe00700 1 -- 172.21.15.27:0/3713223523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fd7fa5950 con 0x561fd7fad9d0 2024-05-10T11:29:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.518+0000 7f28ebe00700 1 --2- 172.21.15.27:0/3713223523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 0x561fd7fae2f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f28e400cfd0 tx=0x7f28e4004be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1935f561e62a6fd7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.518+0000 7f28d7e00700 1 -- 172.21.15.27:0/3713223523 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e40b1410 con 0x561fd7fad9d0 2024-05-10T11:29:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.518+0000 7f28d7e00700 1 -- 172.21.15.27:0/3713223523 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28e4090040 con 0x561fd7fad9d0 2024-05-10T11:29:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.519+0000 7f28d7e00700 1 -- 172.21.15.27:0/3713223523 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e40a9120 con 0x561fd7fad9d0 2024-05-10T11:29:25.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.520+0000 7f28fceccb80 1 -- 172.21.15.27:0/3713223523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 msgr2=0x561fd7fae2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.520+0000 7f28fceccb80 1 --2- 172.21.15.27:0/3713223523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 0x561fd7fae2f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f28e400cfd0 tx=0x7f28e4004be0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.520+0000 7f28fceccb80 1 -- 172.21.15.27:0/3713223523 shutdown_connections 2024-05-10T11:29:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.520+0000 7f28fceccb80 1 --2- 172.21.15.27:0/3713223523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fad9d0 0x561fd7fae2f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.520+0000 7f28fceccb80 1 --2- 172.21.15.27:0/3713223523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9060 0x561fd7f9e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.520+0000 7f28fceccb80 1 --2- 172.21.15.27:0/3713223523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fa9980 0x561fd7faddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.520+0000 7f28fceccb80 1 -- 172.21.15.27:0/3713223523 >> 172.21.15.27:0/3713223523 conn(0x561fd7ed7b60 msgr2=0x561fd7f9da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.520+0000 7f28fceccb80 1 -- 172.21.15.27:0/3713223523 shutdown_connections 2024-05-10T11:29:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.520+0000 7f28fceccb80 1 -- 172.21.15.27:0/3713223523 wait complete. 2024-05-10T11:29:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.521+0000 7f28fceccb80 1 -- start start 2024-05-10T11:29:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.522+0000 7f28fceccb80 1 Processor -- start 2024-05-10T11:29:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.523+0000 7f28fceccb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 0x561fd7fc8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.524+0000 7f28eb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 0x561fd7fc8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.524+0000 7f28eb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 0x561fd7fc8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60388/0 (socket says 172.21.15.27:60388) 2024-05-10T11:29:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.524+0000 7f28fceccb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 0x561fd80256f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.524+0000 7f28fceccb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fad9d0 0x561fd8025d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.524+0000 7f28fceccb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec1bf6d0 con 0x561fd7fad9d0 2024-05-10T11:29:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.524+0000 7f28fceccb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec1bf000 con 0x561fd7fa9980 2024-05-10T11:29:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.524+0000 7f28fceccb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec1bc1f0 con 0x561fd7fa9060 2024-05-10T11:29:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.524+0000 7f28fceccb80 10 client.0 _ll_register_callbacks cb 0x561fd7ea2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.524+0000 7f28eb400700 1 -- 172.21.15.27:0/835169749 learned_addr learned my addr 172.21.15.27:0/835169749 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.524+0000 7f28eaa00700 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 0x561fd80256f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.524+0000 7f28ebe00700 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fad9d0 0x561fd8025d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.525+0000 7f28eb400700 1 -- 172.21.15.27:0/835169749 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 msgr2=0x561fd80256f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.525+0000 7f28eb400700 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 0x561fd80256f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.525+0000 7f28eb400700 1 -- 172.21.15.27:0/835169749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fad9d0 msgr2=0x561fd8025d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.525+0000 7f28eb400700 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fad9d0 0x561fd8025d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.525+0000 7f28eb400700 1 -- 172.21.15.27:0/835169749 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fd7fc5db0 con 0x561fd7fa9060 2024-05-10T11:29:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.526+0000 7f28eb400700 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 0x561fd7fc8260 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00d620 tx=0x7f28dc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de5bd45e443b3364 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.526+0000 7f28d5600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:29:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.526+0000 7f28d5600700 1 -- 172.21.15.27:0/835169749 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc005450 con 0x561fd7fa9060 2024-05-10T11:29:25.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.526+0000 7f28d5600700 1 -- 172.21.15.27:0/835169749 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f28dc0055f0 con 0x561fd7fa9060 2024-05-10T11:29:25.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.526+0000 7f28fceccb80 1 -- 172.21.15.27:0/835169749 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28e4090040 con 0x561fd7fa9060 2024-05-10T11:29:25.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.527+0000 7f28d5600700 1 -- 172.21.15.27:0/835169749 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f28dc036050 con 0x561fd7fa9060 2024-05-10T11:29:25.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.527+0000 7f28fceccb80 10 client.27445 fetch_fsmap learned FSMap version 40 2024-05-10T11:29:25.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.527+0000 7f28fceccb80 1 -- 172.21.15.27:0/835169749 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x561fd803dee0 con 0x561fd7fa9060 2024-05-10T11:29:25.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.527+0000 7f28d5600700 1 -- 172.21.15.27:0/835169749 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28dc024740 con 0x561fd7fa9060 2024-05-10T11:29:25.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.527+0000 7f28d5600700 1 -- 172.21.15.27:0/835169749 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f28dc02d7d0 con 0x561fd7fa9060 2024-05-10T11:29:25.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.528+0000 7f28fceccb80 10 client.27445 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:29:25.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.528+0000 7f28fceccb80 10 client.27445 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:29:25.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.528+0000 7f28fceccb80 20 client.27445 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:25.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.528+0000 7f28fceccb80 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x561fd803e590 0x561fd80409b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:25.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.528+0000 7f28fceccb80 4 client.27445 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:29:25.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.528+0000 7f28fceccb80 1 -- 172.21.15.27:0/835169749 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f28e4090040 con 0x561fd803e590 2024-05-10T11:29:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.529+0000 7f28eaa00700 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x561fd803e590 0x561fd80409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.529+0000 7f28eaa00700 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x561fd803e590 0x561fd80409b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7fd652bda04cf04c server_cookie=8f245c7d7bd95326 in_seq=0 out_seq=0 2024-05-10T11:29:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.529+0000 7f28d5600700 10 client.27445 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:29:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.530+0000 7f28d5600700 1 -- 172.21.15.27:0/835169749 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f28e4090040 con 0x561fd803e590 2024-05-10T11:29:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.530+0000 7f28d5600700 10 client.27445 handle_command_reply: tid=0 2024-05-10T11:29:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.530+0000 7f28fceccb80 1 client.27445 shutdown 2024-05-10T11:29:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.530+0000 7f28fceccb80 1 -- 172.21.15.27:0/835169749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 msgr2=0x561fd7fc8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.530+0000 7f28fceccb80 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 0x561fd7fc8260 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00d620 tx=0x7f28dc004750 comp rx=0 tx=0).stop 2024-05-10T11:29:25.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.534+0000 7f28fceccb80 1 -- 172.21.15.27:0/835169749 shutdown_connections 2024-05-10T11:29:25.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.534+0000 7f28fceccb80 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x561fd803e590 0x561fd80409b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.534+0000 7f28fceccb80 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fd7fa9060 0x561fd7fc8260 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.534+0000 7f28fceccb80 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd7fad9d0 0x561fd8025d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.534+0000 7f28fceccb80 1 --2- 172.21.15.27:0/835169749 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x561fd7fa9980 0x561fd80256f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.534+0000 7f28d5600700 0 client.27445 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:29:25.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.534+0000 7f28fceccb80 1 -- 172.21.15.27:0/835169749 >> 172.21.15.27:0/835169749 conn(0x561fd7ed7b60 msgr2=0x561fd7f9e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:25.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.535+0000 7f28fceccb80 1 -- 172.21.15.27:0/835169749 shutdown_connections 2024-05-10T11:29:25.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.535+0000 7f28fceccb80 1 -- 172.21.15.27:0/835169749 wait complete. 2024-05-10T11:29:25.536 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:29:25.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.535+0000 7f28fceccb80 20 client.27445 trim_cache size 0 max 16384 2024-05-10T11:29:25.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.537+0000 7f28f2600700 1 -- 172.21.15.27:0/2505573100 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f28d00b2a70 msgr2=0x7f28d00b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:25.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.537+0000 7f28f2600700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f28d00b2a70 0x7f28d00b4f30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f28dc005950 tx=0x7f28dc005bd0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.537+0000 7f28f2600700 1 -- 172.21.15.27:0/2505573100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28ec114670 msgr2=0x7f28ec109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:25.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.537+0000 7f28f2600700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28ec114670 0x7f28ec109980 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f28e400b9e0 tx=0x7f28e400bda0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.537+0000 7f28f2600700 1 -- 172.21.15.27:0/2505573100 shutdown_connections 2024-05-10T11:29:25.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.537+0000 7f28f2600700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f28d00b2a70 0x7f28d00b4f30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.537+0000 7f28f2600700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28ec114670 0x7f28ec109980 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.537+0000 7f28f2600700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28ec0756d0 0x7f28ec1106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.537+0000 7f28f2600700 1 --2- 172.21.15.27:0/2505573100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ec075ff0 0x7f28ec110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:25.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.537+0000 7f28f2600700 1 -- 172.21.15.27:0/2505573100 >> 172.21.15.27:0/2505573100 conn(0x7f28ec07a650 msgr2=0x7f28ec111490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:25.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.538+0000 7f28f2600700 1 -- 172.21.15.27:0/2505573100 shutdown_connections 2024-05-10T11:29:25.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:25.538+0000 7f28f2600700 1 -- 172.21.15.27:0/2505573100 wait complete. 2024-05-10T11:29:26.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 57007 22.45 MB/sec execute 10 sec latency 69.062 ms 2024-05-10T11:29:26.502 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:29:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:29:27.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 57491 23.30 MB/sec execute 11 sec latency 209.898 ms 2024-05-10T11:29:27.225 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:26 smithi076 ceph-mon[34152]: pgmap v509: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 11 MiB/s wr, 600 op/s 2024-05-10T11:29:27.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:26 smithi189 ceph-mon[34842]: pgmap v509: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 11 MiB/s wr, 600 op/s 2024-05-10T11:29:27.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:26 smithi027 ceph-mon[30054]: pgmap v509: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 11 MiB/s wr, 600 op/s 2024-05-10T11:29:28.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 57854 22.97 MB/sec execute 12 sec latency 257.480 ms 2024-05-10T11:29:29.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:28 smithi027 ceph-mon[30054]: pgmap v510: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 7.7 MiB/s wr, 387 op/s 2024-05-10T11:29:29.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 58108 21.97 MB/sec execute 13 sec latency 293.936 ms 2024-05-10T11:29:29.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:28 smithi189 ceph-mon[34842]: pgmap v510: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 7.7 MiB/s wr, 387 op/s 2024-05-10T11:29:29.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:28 smithi076 ceph-mon[34152]: pgmap v510: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 7.7 MiB/s wr, 387 op/s 2024-05-10T11:29:30.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 58481 21.42 MB/sec execute 14 sec latency 393.587 ms 2024-05-10T11:29:31.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 59004 21.70 MB/sec execute 15 sec latency 235.716 ms 2024-05-10T11:29:31.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:30 smithi189 ceph-mon[34842]: pgmap v511: 129 pgs: 129 active+clean; 971 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 582 op/s 2024-05-10T11:29:31.299 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:30 smithi027 ceph-mon[30054]: pgmap v511: 129 pgs: 129 active+clean; 971 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 582 op/s 2024-05-10T11:29:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.356+0000 7fa24fe00700 1 -- 172.21.15.27:0/627112340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa25010fc10 msgr2=0x7fa250112080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.356+0000 7fa24fe00700 1 --2- 172.21.15.27:0/627112340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa25010fc10 0x7fa250112080 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa24800b0a0 tx=0x7fa24800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.356+0000 7fa24fe00700 1 -- 172.21.15.27:0/627112340 shutdown_connections 2024-05-10T11:29:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.356+0000 7fa24fe00700 1 --2- 172.21.15.27:0/627112340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2500758d0 0x7fa250075cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.356+0000 7fa24fe00700 1 --2- 172.21.15.27:0/627112340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa25010fc10 0x7fa250112080 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.356+0000 7fa24fe00700 1 --2- 172.21.15.27:0/627112340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2500761f0 0x7fa25010f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.356+0000 7fa24fe00700 1 -- 172.21.15.27:0/627112340 >> 172.21.15.27:0/627112340 conn(0x7fa2500fef20 msgr2=0x7fa250101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.356+0000 7fa24fe00700 1 -- 172.21.15.27:0/627112340 shutdown_connections 2024-05-10T11:29:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.356+0000 7fa24fe00700 1 -- 172.21.15.27:0/627112340 wait complete. 2024-05-10T11:29:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.358+0000 7fa24fe00700 1 Processor -- start 2024-05-10T11:29:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.358+0000 7fa24fe00700 1 -- start start 2024-05-10T11:29:31.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.358+0000 7fa24fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2500758d0 0x7fa2501a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.358+0000 7fa24fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2500761f0 0x7fa2501a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.358+0000 7fa24fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25010fc10 0x7fa25019baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.358+0000 7fa24fe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501157e0 con 0x7fa25010fc10 2024-05-10T11:29:31.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.358+0000 7fa24fe00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa250115660 con 0x7fa2500761f0 2024-05-10T11:29:31.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.358+0000 7fa24fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa250115960 con 0x7fa2500758d0 2024-05-10T11:29:31.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.359+0000 7fa24ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2500758d0 0x7fa2501a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.359+0000 7fa24ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2500758d0 0x7fa2501a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60412/0 (socket says 172.21.15.27:60412) 2024-05-10T11:29:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.359+0000 7fa24ea00700 1 -- 172.21.15.27:0/1572554096 learned_addr learned my addr 172.21.15.27:0/1572554096 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.359+0000 7fa24f400700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25010fc10 0x7fa25019baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.359+0000 7fa24e000700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2500761f0 0x7fa2501a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.359+0000 7fa24ea00700 1 -- 172.21.15.27:0/1572554096 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2500761f0 msgr2=0x7fa2501a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.359+0000 7fa24ea00700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2500761f0 0x7fa2501a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.359+0000 7fa24ea00700 1 -- 172.21.15.27:0/1572554096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25010fc10 msgr2=0x7fa25019baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.359+0000 7fa24ea00700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25010fc10 0x7fa25019baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.359+0000 7fa24ea00700 1 -- 172.21.15.27:0/1572554096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25019c200 con 0x7fa2500758d0 2024-05-10T11:29:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.359+0000 7fa24e000700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2500761f0 0x7fa2501a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:29:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.360+0000 7fa24ea00700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2500758d0 0x7fa2501a18e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa24000eb30 tx=0x7fa24000eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.360+0000 7fa247400700 1 -- 172.21.15.27:0/1572554096 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24000cd40 con 0x7fa2500758d0 2024-05-10T11:29:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.360+0000 7fa247400700 1 -- 172.21.15.27:0/1572554096 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2400044d0 con 0x7fa2500758d0 2024-05-10T11:29:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.360+0000 7fa247400700 1 -- 172.21.15.27:0/1572554096 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa240010430 con 0x7fa2500758d0 2024-05-10T11:29:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.360+0000 7fa24fe00700 1 -- 172.21.15.27:0/1572554096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25019c4f0 con 0x7fa2500758d0 2024-05-10T11:29:31.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.360+0000 7fa24fe00700 1 -- 172.21.15.27:0/1572554096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2501a87b0 con 0x7fa2500758d0 2024-05-10T11:29:31.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.366+0000 7fa247400700 1 -- 172.21.15.27:0/1572554096 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa240004000 con 0x7fa2500758d0 2024-05-10T11:29:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.367+0000 7fa260589b80 1 Processor -- start 2024-05-10T11:29:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.367+0000 7fa260589b80 1 -- start start 2024-05-10T11:29:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.367+0000 7fa260589b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364b0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.367+0000 7fa260589b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364b06c0 0x556f364aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.367+0000 7fa260589b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364aaa50 0x556f364aaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.367+0000 7fa260589b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556f363fca20 con 0x556f364afda0 2024-05-10T11:29:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.367+0000 7fa260589b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x556f3639a760 con 0x556f364b06c0 2024-05-10T11:29:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.367+0000 7fa260589b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556f363f89e0 con 0x556f364aaa50 2024-05-10T11:29:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.368+0000 7fa247400700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa2340b2b40 0x7fa2340b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.368+0000 7fa24ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364b06c0 0x556f364aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.368+0000 7fa24f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.368+0000 7fa24f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364b0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41650/0 (socket says 172.21.15.27:41650) 2024-05-10T11:29:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.368+0000 7fa24f400700 1 -- 172.21.15.27:0/1823780106 learned_addr learned my addr 172.21.15.27:0/1823780106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.368+0000 7fa24f400700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa2340b2b40 0x7fa2340b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.368+0000 7fa24e000700 1 --2- 172.21.15.27:0/1823780106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364aaa50 0x556f364aaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.368+0000 7fa247400700 1 -- 172.21.15.27:0/1572554096 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7fa2400e2cf0 con 0x7fa2500758d0 2024-05-10T11:29:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.369+0000 7fa24f400700 1 -- 172.21.15.27:0/1823780106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364aaa50 msgr2=0x556f364aaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.369+0000 7fa24f400700 1 --2- 172.21.15.27:0/1823780106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364aaa50 0x556f364aaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.369+0000 7fa24f400700 1 -- 172.21.15.27:0/1823780106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364b06c0 msgr2=0x556f364aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.369+0000 7fa24f400700 1 --2- 172.21.15.27:0/1823780106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364b06c0 0x556f364aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.369+0000 7fa24f400700 1 -- 172.21.15.27:0/1823780106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f363498d0 con 0x556f364afda0 2024-05-10T11:29:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.369+0000 7fa24ea00700 1 --2- 172.21.15.27:0/1823780106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364b06c0 0x556f364aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:29:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.369+0000 7fa24f400700 1 --2- 172.21.15.27:0/1823780106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364b0180 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa24800b0a0 tx=0x7fa24800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7658b43899dec378 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.369+0000 7fa24f400700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa2340b2b40 0x7fa2340b5000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa248015000 tx=0x7fa248019000 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.370+0000 7fa23be00700 1 -- 172.21.15.27:0/1823780106 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24802d030 con 0x556f364afda0 2024-05-10T11:29:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.370+0000 7fa23be00700 1 -- 172.21.15.27:0/1823780106 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2480092e0 con 0x556f364afda0 2024-05-10T11:29:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.370+0000 7fa23be00700 1 -- 172.21.15.27:0/1823780106 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24801f440 con 0x556f364afda0 2024-05-10T11:29:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.371+0000 7fa260589b80 1 -- 172.21.15.27:0/1823780106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 msgr2=0x556f364b0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.371+0000 7fa260589b80 1 --2- 172.21.15.27:0/1823780106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364b0180 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa24800b0a0 tx=0x7fa24800bf60 comp rx=0 tx=0).stop 2024-05-10T11:29:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.371+0000 7fa260589b80 1 -- 172.21.15.27:0/1823780106 shutdown_connections 2024-05-10T11:29:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.371+0000 7fa260589b80 1 --2- 172.21.15.27:0/1823780106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364aaa50 0x556f364aaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.371+0000 7fa260589b80 1 --2- 172.21.15.27:0/1823780106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364b0180 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.371+0000 7fa260589b80 1 --2- 172.21.15.27:0/1823780106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364b06c0 0x556f364aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.371+0000 7fa260589b80 1 -- 172.21.15.27:0/1823780106 >> 172.21.15.27:0/1823780106 conn(0x556f364226d0 msgr2=0x556f364a4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.371+0000 7fa260589b80 1 -- 172.21.15.27:0/1823780106 shutdown_connections 2024-05-10T11:29:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.371+0000 7fa260589b80 1 -- 172.21.15.27:0/1823780106 wait complete. 2024-05-10T11:29:31.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.372+0000 7fa260589b80 1 -- start start 2024-05-10T11:29:31.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.372+0000 7fa260589b80 1 Processor -- start 2024-05-10T11:29:31.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.373+0000 7fa260589b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364aaa50 0x556f364b6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.373+0000 7fa260589b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364afda0 0x556f364b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.373+0000 7fa260589b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364b06c0 0x556f364b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.373+0000 7fa260589b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556f364b70f0 con 0x556f364b06c0 2024-05-10T11:29:31.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.373+0000 7fa260589b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x556f36362b20 con 0x556f364aaa50 2024-05-10T11:29:31.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.373+0000 7fa260589b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556f364b7270 con 0x556f364afda0 2024-05-10T11:29:31.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.373+0000 7fa260589b80 10 client.0 _ll_register_callbacks cb 0x556f363a3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:31.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.373+0000 7fa24ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364aaa50 0x556f364b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.373+0000 7fa24ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364aaa50 0x556f364b6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58024/0 (socket says 172.21.15.27:58024) 2024-05-10T11:29:31.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.373+0000 7fa24ea00700 1 -- 172.21.15.27:0/2353236662 learned_addr learned my addr 172.21.15.27:0/2353236662 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:31.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.373+0000 7fa24f400700 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364b06c0 0x556f364b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.373+0000 7fa24e000700 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364afda0 0x556f364b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.374+0000 7fa24f400700 1 -- 172.21.15.27:0/2353236662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364afda0 msgr2=0x556f364b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:31.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.374+0000 7fa24f400700 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364afda0 0x556f364b3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.374+0000 7fa24f400700 1 -- 172.21.15.27:0/2353236662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364aaa50 msgr2=0x556f364b6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:31.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.374+0000 7fa24f400700 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364aaa50 0x556f364b6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.374+0000 7fa24f400700 1 -- 172.21.15.27:0/2353236662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f363c2420 con 0x556f364b06c0 2024-05-10T11:29:31.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.374+0000 7fa24f400700 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364b06c0 0x556f364b3dd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa24800b070 tx=0x7fa248007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c612c9ef5ec707bf server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:31.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.374+0000 7fa238c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:29:31.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.374+0000 7fa238c00700 1 -- 172.21.15.27:0/2353236662 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa248035020 con 0x556f364b06c0 2024-05-10T11:29:31.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.374+0000 7fa238c00700 1 -- 172.21.15.27:0/2353236662 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7fa248026040 con 0x556f364b06c0 2024-05-10T11:29:31.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.375+0000 7fa260589b80 1 -- 172.21.15.27:0/2353236662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556f36477140 con 0x556f364b06c0 2024-05-10T11:29:31.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.375+0000 7fa238c00700 1 -- 172.21.15.27:0/2353236662 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fa2480079d0 con 0x556f364b06c0 2024-05-10T11:29:31.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.375+0000 7fa260589b80 10 client.17166 fetch_fsmap learned FSMap version 40 2024-05-10T11:29:31.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.375+0000 7fa260589b80 1 -- 172.21.15.27:0/2353236662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x556f364c1a80 con 0x556f364b06c0 2024-05-10T11:29:31.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.375+0000 7fa238c00700 1 -- 172.21.15.27:0/2353236662 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556f36477140 con 0x556f364b06c0 2024-05-10T11:29:31.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.376+0000 7fa238c00700 1 -- 172.21.15.27:0/2353236662 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fa248011070 con 0x556f364b06c0 2024-05-10T11:29:31.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.376+0000 7fa260589b80 10 client.17166 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:29:31.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.376+0000 7fa260589b80 10 client.17166 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:29:31.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.376+0000 7fa260589b80 20 client.17166 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:31.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.376+0000 7fa260589b80 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x556f364c3f50 0x556f364c6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.376+0000 7fa260589b80 4 client.17166 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:29:31.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.376+0000 7fa260589b80 1 -- 172.21.15.27:0/2353236662 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f3640a400 con 0x556f364c3f50 2024-05-10T11:29:31.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.376+0000 7fa24ea00700 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x556f364c3f50 0x556f364c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.377+0000 7fa24ea00700 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x556f364c3f50 0x556f364c6340 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d44ba60593d63857 server_cookie=dc1375cd12eba823 in_seq=0 out_seq=0 2024-05-10T11:29:31.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.377+0000 7fa238c00700 10 client.17166 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:29:31.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.377+0000 7fa238c00700 1 -- 172.21.15.27:0/2353236662 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556f3640a400 con 0x556f364c3f50 2024-05-10T11:29:31.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.377+0000 7fa238c00700 10 client.17166 handle_command_reply: tid=0 2024-05-10T11:29:31.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.377+0000 7fa260589b80 1 client.17166 shutdown 2024-05-10T11:29:31.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.378+0000 7fa260589b80 1 -- 172.21.15.27:0/2353236662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364b06c0 msgr2=0x556f364b3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:31.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.378+0000 7fa260589b80 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364b06c0 0x556f364b3dd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa24800b070 tx=0x7fa248007f00 comp rx=0 tx=0).stop 2024-05-10T11:29:31.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.378+0000 7fa260589b80 1 -- 172.21.15.27:0/2353236662 shutdown_connections 2024-05-10T11:29:31.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.378+0000 7fa260589b80 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x556f364c3f50 0x556f364c6340 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.378+0000 7fa260589b80 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364b06c0 0x556f364b3dd0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.378+0000 7fa260589b80 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364aaa50 0x556f364b6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.378+0000 7fa238c00700 0 client.17166 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:29:31.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.378+0000 7fa260589b80 1 --2- 172.21.15.27:0/2353236662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364afda0 0x556f364b3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.378+0000 7fa260589b80 1 -- 172.21.15.27:0/2353236662 >> 172.21.15.27:0/2353236662 conn(0x556f364226d0 msgr2=0x556f3649f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:31.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.378+0000 7fa260589b80 1 -- 172.21.15.27:0/2353236662 shutdown_connections 2024-05-10T11:29:31.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.378+0000 7fa260589b80 1 -- 172.21.15.27:0/2353236662 wait complete. 2024-05-10T11:29:31.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.378+0000 7fa260589b80 20 client.17166 trim_cache size 0 max 16384 2024-05-10T11:29:31.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.396+0000 7fa260589b80 1 Processor -- start 2024-05-10T11:29:31.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.396+0000 7fa260589b80 1 -- start start 2024-05-10T11:29:31.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.396+0000 7fa260589b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364aaa50 0x556f364aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.396+0000 7fa260589b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.396+0000 7fa260589b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364b06c0 0x556f3649f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.396+0000 7fa260589b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b1700 con 0x556f364afda0 2024-05-10T11:29:31.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.396+0000 7fa260589b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b3330 con 0x556f364aaa50 2024-05-10T11:29:31.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.396+0000 7fa260589b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b1dd0 con 0x556f364b06c0 2024-05-10T11:29:31.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.396+0000 7fa24ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364aaa50 0x556f364aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.397+0000 7fa24ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364aaa50 0x556f364aa430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:58030/0 (socket says 172.21.15.27:58030) 2024-05-10T11:29:31.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.397+0000 7fa24ea00700 1 -- 172.21.15.27:0/1316836965 learned_addr learned my addr 172.21.15.27:0/1316836965 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:31.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.397+0000 7fa24e000700 1 --2- 172.21.15.27:0/1316836965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.397+0000 7fa24f400700 1 --2- 172.21.15.27:0/1316836965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364b06c0 0x556f3649f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.397+0000 7fa24f400700 1 -- 172.21.15.27:0/1316836965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364aaa50 msgr2=0x556f364aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:31.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.397+0000 7fa24f400700 1 --2- 172.21.15.27:0/1316836965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364aaa50 0x556f364aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.397+0000 7fa24f400700 1 -- 172.21.15.27:0/1316836965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 msgr2=0x556f364b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:31.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.397+0000 7fa24f400700 1 --2- 172.21.15.27:0/1316836965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364b1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.397+0000 7fa24f400700 1 -- 172.21.15.27:0/1316836965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f364a18a0 con 0x556f364b06c0 2024-05-10T11:29:31.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.397+0000 7fa24f400700 1 --2- 172.21.15.27:0/1316836965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364b06c0 0x556f3649f860 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa248009fd0 tx=0x7fa248006120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95be463826f15954 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:31.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.398+0000 7fa23b400700 1 -- 172.21.15.27:0/1316836965 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa248035020 con 0x556f364b06c0 2024-05-10T11:29:31.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.398+0000 7fa23b400700 1 -- 172.21.15.27:0/1316836965 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa248029660 con 0x556f364b06c0 2024-05-10T11:29:31.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.398+0000 7fa23b400700 1 -- 172.21.15.27:0/1316836965 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24801f930 con 0x556f364b06c0 2024-05-10T11:29:31.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.399+0000 7fa260589b80 1 -- 172.21.15.27:0/1316836965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364b06c0 msgr2=0x556f3649f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:31.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.399+0000 7fa260589b80 1 --2- 172.21.15.27:0/1316836965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364b06c0 0x556f3649f860 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa248009fd0 tx=0x7fa248006120 comp rx=0 tx=0).stop 2024-05-10T11:29:31.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.399+0000 7fa260589b80 1 -- 172.21.15.27:0/1316836965 shutdown_connections 2024-05-10T11:29:31.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.399+0000 7fa260589b80 1 --2- 172.21.15.27:0/1316836965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364b06c0 0x556f3649f860 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.399+0000 7fa260589b80 1 --2- 172.21.15.27:0/1316836965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364aaa50 0x556f364aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.399+0000 7fa260589b80 1 --2- 172.21.15.27:0/1316836965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.399+0000 7fa260589b80 1 -- 172.21.15.27:0/1316836965 >> 172.21.15.27:0/1316836965 conn(0x556f364226d0 msgr2=0x556f364b57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:31.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.399+0000 7fa260589b80 1 -- 172.21.15.27:0/1316836965 shutdown_connections 2024-05-10T11:29:31.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.399+0000 7fa260589b80 1 -- 172.21.15.27:0/1316836965 wait complete. 2024-05-10T11:29:31.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.400+0000 7fa260589b80 1 -- start start 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.400+0000 7fa260589b80 1 Processor -- start 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.401+0000 7fa260589b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364aaa50 0x556f364bcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.401+0000 7fa260589b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.401+0000 7fa260589b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364b06c0 0x556f364cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.401+0000 7fa260589b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b2ba0 con 0x556f364afda0 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.401+0000 7fa260589b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b24d0 con 0x556f364b06c0 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.401+0000 7fa260589b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501af6a0 con 0x556f364aaa50 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.401+0000 7fa260589b80 10 client.0 _ll_register_callbacks cb 0x556f363a3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.401+0000 7fa24ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364aaa50 0x556f364bcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.401+0000 7fa24ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364aaa50 0x556f364bcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:60436/0 (socket says 172.21.15.27:60436) 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.401+0000 7fa24ea00700 1 -- 172.21.15.27:0/3615337608 learned_addr learned my addr 172.21.15.27:0/3615337608 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.401+0000 7fa24f400700 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364b06c0 0x556f364cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.401+0000 7fa24e000700 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.402+0000 7fa24ea00700 1 -- 172.21.15.27:0/3615337608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364b06c0 msgr2=0x556f364cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.402+0000 7fa24ea00700 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364b06c0 0x556f364cb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.402+0000 7fa24ea00700 1 -- 172.21.15.27:0/3615337608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 msgr2=0x556f364cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.402+0000 7fa24ea00700 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.402+0000 7fa24ea00700 1 -- 172.21.15.27:0/3615337608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f3652d4c0 con 0x556f364aaa50 2024-05-10T11:29:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.402+0000 7fa24e000700 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:29:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.402+0000 7fa24ea00700 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364aaa50 0x556f364bcc10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa2400a5bc0 tx=0x7fa240074390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6760bc8862abffd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.402+0000 7fa238c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:29:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.402+0000 7fa238c00700 1 -- 172.21.15.27:0/3615337608 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2400aa240 con 0x556f364aaa50 2024-05-10T11:29:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.402+0000 7fa238c00700 1 -- 172.21.15.27:0/3615337608 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7fa2400730d0 con 0x556f364aaa50 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.402+0000 7fa260589b80 1 -- 172.21.15.27:0/3615337608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa248029660 con 0x556f364aaa50 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.403+0000 7fa238c00700 1 -- 172.21.15.27:0/3615337608 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fa24006f5c0 con 0x556f364aaa50 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.403+0000 7fa260589b80 10 client.27463 fetch_fsmap learned FSMap version 40 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.403+0000 7fa260589b80 1 -- 172.21.15.27:0/3615337608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x556f3652e740 con 0x556f364aaa50 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.403+0000 7fa238c00700 1 -- 172.21.15.27:0/3615337608 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa248029660 con 0x556f364aaa50 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.403+0000 7fa238c00700 1 -- 172.21.15.27:0/3615337608 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fa2400e0230 con 0x556f364aaa50 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.403+0000 7fa260589b80 10 client.27463 fetch_fsmap finished waiting for FSMap version 40 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.403+0000 7fa260589b80 10 client.27463 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.403+0000 7fa260589b80 20 client.27463 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.403+0000 7fa260589b80 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x556f3652edd0 0x556f365311f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.403+0000 7fa260589b80 4 client.27463 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.403+0000 7fa260589b80 1 -- 172.21.15.27:0/3615337608 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556f364044a0 con 0x556f3652edd0 2024-05-10T11:29:31.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.404+0000 7fa24e000700 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x556f3652edd0 0x556f365311f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:31.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.404+0000 7fa24e000700 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x556f3652edd0 0x556f365311f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7632c0e6236335f8 server_cookie=af6c32ca1346cd85 in_seq=0 out_seq=0 2024-05-10T11:29:31.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.404+0000 7fa238c00700 10 client.27463 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:29:31.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.404+0000 7fa238c00700 1 -- 172.21.15.27:0/3615337608 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556f364044a0 con 0x556f3652edd0 2024-05-10T11:29:31.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.404+0000 7fa238c00700 10 client.27463 handle_command_reply: tid=0 2024-05-10T11:29:31.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.404+0000 7fa260589b80 1 client.27463 shutdown 2024-05-10T11:29:31.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.405+0000 7fa260589b80 1 -- 172.21.15.27:0/3615337608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364aaa50 msgr2=0x556f364bcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:31.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.405+0000 7fa260589b80 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364aaa50 0x556f364bcc10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa2400a5bc0 tx=0x7fa240074390 comp rx=0 tx=0).stop 2024-05-10T11:29:31.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.405+0000 7fa260589b80 1 -- 172.21.15.27:0/3615337608 shutdown_connections 2024-05-10T11:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.405+0000 7fa260589b80 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x556f3652edd0 0x556f365311f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.405+0000 7fa260589b80 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f364aaa50 0x556f364bcc10 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.405+0000 7fa260589b80 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556f364b06c0 0x556f364cb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.405+0000 7fa260589b80 1 --2- 172.21.15.27:0/3615337608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556f364afda0 0x556f364cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.406+0000 7fa260589b80 1 -- 172.21.15.27:0/3615337608 >> 172.21.15.27:0/3615337608 conn(0x556f364226d0 msgr2=0x556f364b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.406+0000 7fa238c00700 0 client.27463 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.406+0000 7fa260589b80 1 -- 172.21.15.27:0/3615337608 shutdown_connections 2024-05-10T11:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.406+0000 7fa260589b80 1 -- 172.21.15.27:0/3615337608 wait complete. 2024-05-10T11:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.406+0000 7fa260589b80 20 client.27463 trim_cache size 0 max 16384 2024-05-10T11:29:31.409 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:29:31.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.411+0000 7fa24fe00700 1 -- 172.21.15.27:0/1572554096 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa2340b2b40 msgr2=0x7fa2340b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:31.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.411+0000 7fa24fe00700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa2340b2b40 0x7fa2340b5000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa248015000 tx=0x7fa248019000 comp rx=0 tx=0).stop 2024-05-10T11:29:31.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.411+0000 7fa24fe00700 1 -- 172.21.15.27:0/1572554096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2500758d0 msgr2=0x7fa2501a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:31.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.411+0000 7fa24fe00700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2500758d0 0x7fa2501a18e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa24000eb30 tx=0x7fa24000eef0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.411+0000 7fa24fe00700 1 -- 172.21.15.27:0/1572554096 shutdown_connections 2024-05-10T11:29:31.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.411+0000 7fa24fe00700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa2340b2b40 0x7fa2340b5000 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.411+0000 7fa24fe00700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2500758d0 0x7fa2501a18e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.411+0000 7fa24fe00700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25010fc10 0x7fa25019baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.411+0000 7fa24fe00700 1 --2- 172.21.15.27:0/1572554096 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2500761f0 0x7fa2501a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:31.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.411+0000 7fa24fe00700 1 -- 172.21.15.27:0/1572554096 >> 172.21.15.27:0/1572554096 conn(0x7fa2500fef20 msgr2=0x7fa2500ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:31.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.411+0000 7fa24fe00700 1 -- 172.21.15.27:0/1572554096 shutdown_connections 2024-05-10T11:29:31.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:31.411+0000 7fa24fe00700 1 -- 172.21.15.27:0/1572554096 wait complete. 2024-05-10T11:29:31.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:30 smithi076 ceph-mon[34152]: pgmap v511: 129 pgs: 129 active+clean; 971 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 582 op/s 2024-05-10T11:29:32.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 59473 22.13 MB/sec execute 16 sec latency 223.519 ms 2024-05-10T11:29:32.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:29:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:29:33.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 60149 22.42 MB/sec execute 17 sec latency 121.835 ms 2024-05-10T11:29:33.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:32 smithi189 ceph-mon[34842]: pgmap v512: 129 pgs: 129 active+clean; 971 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.0 MiB/s wr, 383 op/s 2024-05-10T11:29:33.238 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:32 smithi027 ceph-mon[30054]: pgmap v512: 129 pgs: 129 active+clean; 971 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.0 MiB/s wr, 383 op/s 2024-05-10T11:29:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:32 smithi076 ceph-mon[34152]: pgmap v512: 129 pgs: 129 active+clean; 971 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.0 MiB/s wr, 383 op/s 2024-05-10T11:29:34.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 60534 22.64 MB/sec execute 18 sec latency 241.051 ms 2024-05-10T11:29:35.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 61054 23.33 MB/sec execute 19 sec latency 137.807 ms 2024-05-10T11:29:35.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:34 smithi027 ceph-mon[30054]: pgmap v513: 129 pgs: 129 active+clean; 971 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.0 MiB/s wr, 383 op/s 2024-05-10T11:29:35.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:34 smithi076 ceph-mon[34152]: pgmap v513: 129 pgs: 129 active+clean; 971 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.0 MiB/s wr, 383 op/s 2024-05-10T11:29:35.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:34 smithi189 ceph-mon[34842]: pgmap v513: 129 pgs: 129 active+clean; 971 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.0 MiB/s wr, 383 op/s 2024-05-10T11:29:36.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.015+0000 7f49e1000700 1 -- 172.21.15.27:0/1946328930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc102180 msgr2=0x7f49dc102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:36.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.015+0000 7f49e1000700 1 --2- 172.21.15.27:0/1946328930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc102180 0x7f49dc102560 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f49cc009990 tx=0x7f49cc009ca0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.015+0000 7f49e1000700 1 -- 172.21.15.27:0/1946328930 shutdown_connections 2024-05-10T11:29:36.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.015+0000 7f49e1000700 1 --2- 172.21.15.27:0/1946328930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49dc10fd30 0x7f49dc112120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.015+0000 7f49e1000700 1 --2- 172.21.15.27:0/1946328930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc102180 0x7f49dc102560 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.015+0000 7f49e1000700 1 --2- 172.21.15.27:0/1946328930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49dc102b30 0x7f49dc10f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.015+0000 7f49e1000700 1 -- 172.21.15.27:0/1946328930 >> 172.21.15.27:0/1946328930 conn(0x7f49dc0fddc0 msgr2=0x7f49dc1001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:36.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.015+0000 7f49e1000700 1 -- 172.21.15.27:0/1946328930 shutdown_connections 2024-05-10T11:29:36.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.016+0000 7f49e1000700 1 -- 172.21.15.27:0/1946328930 wait complete. 2024-05-10T11:29:36.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.017+0000 7f49e1000700 1 Processor -- start 2024-05-10T11:29:36.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.017+0000 7f49e1000700 1 -- start start 2024-05-10T11:29:36.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49e1000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc102180 0x7f49dc1a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49e1000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49dc102b30 0x7f49dc1a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49e1000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49dc10fd30 0x7f49dc19bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49e1000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1157e0 con 0x7f49dc10fd30 2024-05-10T11:29:36.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49e1000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc115660 con 0x7f49dc102180 2024-05-10T11:29:36.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49e1000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc115960 con 0x7f49dc102b30 2024-05-10T11:29:36.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49d3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc102180 0x7f49dc1a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49d3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc102180 0x7f49dc1a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:56482/0 (socket says 172.21.15.27:56482) 2024-05-10T11:29:36.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49d3400700 1 -- 172.21.15.27:0/1538741487 learned_addr learned my addr 172.21.15.27:0/1538741487 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49db400700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49dc102b30 0x7f49dc1a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49dbe00700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49dc10fd30 0x7f49dc19bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49d3400700 1 -- 172.21.15.27:0/1538741487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49dc102b30 msgr2=0x7f49dc1a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49d3400700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49dc102b30 0x7f49dc1a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49d3400700 1 -- 172.21.15.27:0/1538741487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49dc10fd30 msgr2=0x7f49dc19bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49d3400700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49dc10fd30 0x7f49dc19bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.018+0000 7f49d3400700 1 -- 172.21.15.27:0/1538741487 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49dc19c280 con 0x7f49dc102180 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.019+0000 7f49d3400700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc102180 0x7f49dc1a1910 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f49cc000c00 tx=0x7f49cc0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.019+0000 7f49d8c00700 1 -- 172.21.15.27:0/1538741487 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc027070 con 0x7f49dc102180 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.019+0000 7f49d8c00700 1 -- 172.21.15.27:0/1538741487 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49cc022470 con 0x7f49dc102180 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.019+0000 7f49e1000700 1 -- 172.21.15.27:0/1538741487 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49dc19c510 con 0x7f49dc102180 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.019+0000 7f49d8c00700 1 -- 172.21.15.27:0/1538741487 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc017630 con 0x7f49dc102180 2024-05-10T11:29:36.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.019+0000 7f49e1000700 1 -- 172.21.15.27:0/1538741487 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49dc1a87e0 con 0x7f49dc102180 2024-05-10T11:29:36.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.023+0000 7f49d8c00700 1 -- 172.21.15.27:0/1538741487 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f49cc02a070 con 0x7f49dc102180 2024-05-10T11:29:36.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.025+0000 7f49ecf5db80 1 Processor -- start 2024-05-10T11:29:36.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.026+0000 7f49d8c00700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f49c00b29f0 0x7f49c00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.026+0000 7f49ecf5db80 1 -- start start 2024-05-10T11:29:36.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.026+0000 7f49d8c00700 1 -- 172.21.15.27:0/1538741487 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f49cc0e37f0 con 0x7f49dc102180 2024-05-10T11:29:36.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.026+0000 7f49dbe00700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f49c00b29f0 0x7f49c00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.026+0000 7f49ecf5db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3c0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.026+0000 7f49ecf5db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c1300 0x55ce9e3bafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.026+0000 7f49ecf5db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3bb600 0x55ce9e3bba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.026+0000 7f49ecf5db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ce9e30da20 con 0x55ce9e3bb600 2024-05-10T11:29:36.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.026+0000 7f49ecf5db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ce9e2ab760 con 0x55ce9e3c09e0 2024-05-10T11:29:36.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.026+0000 7f49ecf5db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ce9e3099e0 con 0x55ce9e3c1300 2024-05-10T11:29:36.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.027+0000 7f49d3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3c0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.027+0000 7f49d3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3c0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:56486/0 (socket says 172.21.15.27:56486) 2024-05-10T11:29:36.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.027+0000 7f49d3400700 1 -- 172.21.15.27:0/2292625969 learned_addr learned my addr 172.21.15.27:0/2292625969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:36.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.027+0000 7f49db400700 1 --2- 172.21.15.27:0/2292625969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c1300 0x55ce9e3bafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.027+0000 7f49dbe00700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f49c00b29f0 0x7f49c00b4eb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f49dc06ada0 tx=0x7f49d400b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:36.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.027+0000 7f49dbe00700 1 --2- 172.21.15.27:0/2292625969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3bb600 0x55ce9e3bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.027+0000 7f49d3400700 1 -- 172.21.15.27:0/2292625969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c1300 msgr2=0x55ce9e3bafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:36.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.027+0000 7f49d3400700 1 --2- 172.21.15.27:0/2292625969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c1300 0x55ce9e3bafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.027+0000 7f49d3400700 1 -- 172.21.15.27:0/2292625969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3bb600 msgr2=0x55ce9e3bba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:36.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.027+0000 7f49d3400700 1 --2- 172.21.15.27:0/2292625969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3bb600 0x55ce9e3bba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.027+0000 7f49d3400700 1 -- 172.21.15.27:0/2292625969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce9e25a8d0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.028+0000 7f49d3400700 1 --2- 172.21.15.27:0/2292625969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3c0dc0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0aa3d0 tx=0x7f49cc0aa020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b45ccd86a0020293 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:36.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.028+0000 7f49d0c00700 1 -- 172.21.15.27:0/2292625969 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc0aadd0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.028+0000 7f49d0c00700 1 -- 172.21.15.27:0/2292625969 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49cc0aa6d0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.028+0000 7f49d0c00700 1 -- 172.21.15.27:0/2292625969 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc022940 con 0x55ce9e3c09e0 2024-05-10T11:29:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.029+0000 7f49ecf5db80 1 -- 172.21.15.27:0/2292625969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 msgr2=0x55ce9e3c0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.029+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/2292625969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3c0dc0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0aa3d0 tx=0x7f49cc0aa020 comp rx=0 tx=0).stop 2024-05-10T11:29:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.029+0000 7f49ecf5db80 1 -- 172.21.15.27:0/2292625969 shutdown_connections 2024-05-10T11:29:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.029+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/2292625969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3bb600 0x55ce9e3bba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.029+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/2292625969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3c0dc0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.029+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/2292625969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c1300 0x55ce9e3bafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.029+0000 7f49ecf5db80 1 -- 172.21.15.27:0/2292625969 >> 172.21.15.27:0/2292625969 conn(0x55ce9e2e9b60 msgr2=0x55ce9e3b59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.029+0000 7f49ecf5db80 1 -- 172.21.15.27:0/2292625969 shutdown_connections 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.029+0000 7f49ecf5db80 1 -- 172.21.15.27:0/2292625969 wait complete. 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.030+0000 7f49ecf5db80 1 -- start start 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.030+0000 7f49ecf5db80 1 Processor -- start 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.030+0000 7f49ecf5db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3bb600 0x55ce9e3c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.030+0000 7f49ecf5db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3c4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.030+0000 7f49ecf5db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3c1300 0x55ce9e3c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.030+0000 7f49ecf5db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ce9e3c7b70 con 0x55ce9e3c1300 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.031+0000 7f49ecf5db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ce9e273b20 con 0x55ce9e3bb600 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.031+0000 7f49ecf5db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ce9e3c7cf0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.031+0000 7f49db400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.031+0000 7f49db400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3c4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54122/0 (socket says 172.21.15.27:54122) 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.031+0000 7f49db400700 1 -- 172.21.15.27:0/3676769672 learned_addr learned my addr 172.21.15.27:0/3676769672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.031+0000 7f49ecf5db80 10 client.0 _ll_register_callbacks cb 0x55ce9e2b4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.031+0000 7f49d3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3bb600 0x55ce9e3c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.031+0000 7f49db400700 1 -- 172.21.15.27:0/3676769672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3bb600 msgr2=0x55ce9e3c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.031+0000 7f49db400700 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3bb600 0x55ce9e3c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.031+0000 7f49db400700 1 -- 172.21.15.27:0/3676769672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3c1300 msgr2=0x55ce9e3c48d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.031+0000 7f49dbe00700 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3c1300 0x55ce9e3c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.032+0000 7f49db400700 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3c1300 0x55ce9e3c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.032+0000 7f49db400700 1 -- 172.21.15.27:0/3676769672 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce9e2d3420 con 0x55ce9e3c09e0 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.032+0000 7f49dbe00700 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3c1300 0x55ce9e3c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.032+0000 7f49db400700 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3c4390 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f49c40127f0 tx=0x7f49c4012bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2d21d1b15a3ef02 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.032+0000 7f49c9600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.032+0000 7f49c9600700 1 -- 172.21.15.27:0/3676769672 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c400f640 con 0x55ce9e3c09e0 2024-05-10T11:29:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.032+0000 7f49c9600700 1 -- 172.21.15.27:0/3676769672 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f49c400f7e0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.032+0000 7f49ecf5db80 1 -- 172.21.15.27:0/3676769672 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49cc0aa6d0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.033+0000 7f49c9600700 1 -- 172.21.15.27:0/3676769672 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f49c400f9e0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.033+0000 7f49ecf5db80 10 client.27475 fetch_fsmap learned FSMap version 41 2024-05-10T11:29:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.033+0000 7f49ecf5db80 1 -- 172.21.15.27:0/3676769672 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55ce9e3c92e0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.033+0000 7f49c9600700 1 -- 172.21.15.27:0/3676769672 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49cc0aa6d0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.033+0000 7f49c9600700 1 -- 172.21.15.27:0/3676769672 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 41) ==== 8382+0+0 (secure 0 0 0) 0x7f49c4010bb0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.033+0000 7f49ecf5db80 10 client.27475 fetch_fsmap finished waiting for FSMap version 41 2024-05-10T11:29:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.033+0000 7f49ecf5db80 10 client.27475 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:29:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.033+0000 7f49ecf5db80 20 client.27475 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.033+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce9e3cb970 0x55ce9e3cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.033+0000 7f49ecf5db80 4 client.27475 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.033+0000 7f49ecf5db80 1 -- 172.21.15.27:0/3676769672 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce9e3880a0 con 0x55ce9e3cb970 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.034+0000 7f49d3400700 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce9e3cb970 0x55ce9e3cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.034+0000 7f49c9600700 10 client.27475 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.034+0000 7f49d3400700 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce9e3cb970 0x55ce9e3cdd90 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a30d32c6e2cef551 server_cookie=22eb1b0366c8fefb in_seq=0 out_seq=0 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.035+0000 7f49c9600700 1 -- 172.21.15.27:0/3676769672 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ce9e3880a0 con 0x55ce9e3cb970 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.035+0000 7f49c9600700 10 client.27475 handle_command_reply: tid=0 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.036+0000 7f49ecf5db80 1 client.27475 shutdown 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.036+0000 7f49ecf5db80 1 -- 172.21.15.27:0/3676769672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 msgr2=0x55ce9e3c4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.036+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3c4390 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f49c40127f0 tx=0x7f49c4012bb0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.036+0000 7f49ecf5db80 1 -- 172.21.15.27:0/3676769672 shutdown_connections 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.036+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3c1300 0x55ce9e3c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.036+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3bb600 0x55ce9e3c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.036+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce9e3cb970 0x55ce9e3cdd90 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.036+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/3676769672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3c4390 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.036+0000 7f49ecf5db80 1 -- 172.21.15.27:0/3676769672 >> 172.21.15.27:0/3676769672 conn(0x55ce9e2e9b60 msgr2=0x55ce9e3b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.036+0000 7f49c9600700 0 client.27475 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:29:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.036+0000 7f49ecf5db80 1 -- 172.21.15.27:0/3676769672 shutdown_connections 2024-05-10T11:29:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.036+0000 7f49ecf5db80 1 -- 172.21.15.27:0/3676769672 wait complete. 2024-05-10T11:29:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.037+0000 7f49ecf5db80 20 client.27475 trim_cache size 0 max 16384 2024-05-10T11:29:36.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.055+0000 7f49ecf5db80 1 Processor -- start 2024-05-10T11:29:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.055+0000 7f49ecf5db80 1 -- start start 2024-05-10T11:29:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49ecf5db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3bb600 0x55ce9e3c1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49ecf5db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3e1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49ecf5db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3c1300 0x55ce9e3e1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49ecf5db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1a8fc0 con 0x55ce9e3c1300 2024-05-10T11:29:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49ecf5db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1aabc0 con 0x55ce9e3bb600 2024-05-10T11:29:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49ecf5db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1a9690 con 0x55ce9e3c09e0 2024-05-10T11:29:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49d3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3bb600 0x55ce9e3c1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49d3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3bb600 0x55ce9e3c1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:56494/0 (socket says 172.21.15.27:56494) 2024-05-10T11:29:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49d3400700 1 -- 172.21.15.27:0/150349905 learned_addr learned my addr 172.21.15.27:0/150349905 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49db400700 1 --2- 172.21.15.27:0/150349905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3e1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49dbe00700 1 --2- 172.21.15.27:0/150349905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3c1300 0x55ce9e3e1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49d3400700 1 -- 172.21.15.27:0/150349905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 msgr2=0x55ce9e3e1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49d3400700 1 --2- 172.21.15.27:0/150349905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3e1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49d3400700 1 -- 172.21.15.27:0/150349905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3c1300 msgr2=0x55ce9e3e1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.056+0000 7f49d3400700 1 --2- 172.21.15.27:0/150349905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3c1300 0x55ce9e3e1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.057+0000 7f49d3400700 1 -- 172.21.15.27:0/150349905 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce9e3ce410 con 0x55ce9e3bb600 2024-05-10T11:29:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.057+0000 7f49db400700 1 --2- 172.21.15.27:0/150349905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3e1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:29:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.057+0000 7f49d3400700 1 --2- 172.21.15.27:0/150349905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3bb600 0x55ce9e3c1df0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0058b0 tx=0x7f49cc0cd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a02bd9d6bd3caf4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.057+0000 7f49cbe00700 1 -- 172.21.15.27:0/150349905 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc0aadd0 con 0x55ce9e3bb600 2024-05-10T11:29:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.057+0000 7f49cbe00700 1 -- 172.21.15.27:0/150349905 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49cc0a6960 con 0x55ce9e3bb600 2024-05-10T11:29:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.057+0000 7f49cbe00700 1 -- 172.21.15.27:0/150349905 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc022940 con 0x55ce9e3bb600 2024-05-10T11:29:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.058+0000 7f49ecf5db80 1 -- 172.21.15.27:0/150349905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3bb600 msgr2=0x55ce9e3c1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.058+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/150349905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3bb600 0x55ce9e3c1df0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0058b0 tx=0x7f49cc0cd040 comp rx=0 tx=0).stop 2024-05-10T11:29:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.058+0000 7f49ecf5db80 1 -- 172.21.15.27:0/150349905 shutdown_connections 2024-05-10T11:29:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.058+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/150349905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3c1300 0x55ce9e3e1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.058+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/150349905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3bb600 0x55ce9e3c1df0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.058+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/150349905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3e1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.058+0000 7f49ecf5db80 1 -- 172.21.15.27:0/150349905 >> 172.21.15.27:0/150349905 conn(0x55ce9e2e9b60 msgr2=0x55ce9e3c16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.058+0000 7f49ecf5db80 1 -- 172.21.15.27:0/150349905 shutdown_connections 2024-05-10T11:29:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.058+0000 7f49ecf5db80 1 -- 172.21.15.27:0/150349905 wait complete. 2024-05-10T11:29:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.059+0000 7f49ecf5db80 1 -- start start 2024-05-10T11:29:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.059+0000 7f49ecf5db80 1 Processor -- start 2024-05-10T11:29:36.060 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:36 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:36.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.060+0000 7f49ecf5db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3bb600 0x55ce9e3d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.060+0000 7f49ecf5db80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3d1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.060+0000 7f49ecf5db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c1300 0x55ce9e45fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.060+0000 7f49ecf5db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1aa430 con 0x55ce9e3bb600 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.060+0000 7f49ecf5db80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1a9d60 con 0x55ce9e3c09e0 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.060+0000 7f49ecf5db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc073ca0 con 0x55ce9e3c1300 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.060+0000 7f49ecf5db80 10 client.0 _ll_register_callbacks cb 0x55ce9e2b4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.060+0000 7f49db400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.060+0000 7f49db400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3d1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:56508/0 (socket says 172.21.15.27:56508) 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.060+0000 7f49db400700 1 -- 172.21.15.27:0/1356116659 learned_addr learned my addr 172.21.15.27:0/1356116659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.061+0000 7f49d3400700 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3bb600 0x55ce9e3d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.061+0000 7f49dbe00700 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c1300 0x55ce9e45fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.061+0000 7f49db400700 1 -- 172.21.15.27:0/1356116659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c1300 msgr2=0x55ce9e45fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.061+0000 7f49db400700 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c1300 0x55ce9e45fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.061+0000 7f49db400700 1 -- 172.21.15.27:0/1356116659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3bb600 msgr2=0x55ce9e3d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.061+0000 7f49db400700 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3bb600 0x55ce9e3d0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.061+0000 7f49db400700 1 -- 172.21.15.27:0/1356116659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce9e4604e0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.061+0000 7f49db400700 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3d1130 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f49c4005a80 tx=0x7f49c401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10c56d86601dcf3c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.061+0000 7f49c9600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:29:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.062+0000 7f49c9600700 1 -- 172.21.15.27:0/1356116659 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c402f060 con 0x55ce9e3c09e0 2024-05-10T11:29:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.062+0000 7f49c9600700 1 -- 172.21.15.27:0/1356116659 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f49c401c9b0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.062+0000 7f49ecf5db80 1 -- 172.21.15.27:0/1356116659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49cc0a6960 con 0x55ce9e3c09e0 2024-05-10T11:29:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.062+0000 7f49c9600700 1 -- 172.21.15.27:0/1356116659 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f49c401cbd0 con 0x55ce9e3c09e0 2024-05-10T11:29:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.062+0000 7f49ecf5db80 10 client.27485 fetch_fsmap learned FSMap version 41 2024-05-10T11:29:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.062+0000 7f49ecf5db80 1 -- 172.21.15.27:0/1356116659 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55ce9e461850 con 0x55ce9e3c09e0 2024-05-10T11:29:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.063+0000 7f49c9600700 1 -- 172.21.15.27:0/1356116659 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49cc0a6960 con 0x55ce9e3c09e0 2024-05-10T11:29:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.063+0000 7f49c9600700 1 -- 172.21.15.27:0/1356116659 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 41) ==== 8382+0+0 (secure 0 0 0) 0x7f49c4003840 con 0x55ce9e3c09e0 2024-05-10T11:29:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.063+0000 7f49ecf5db80 10 client.27485 fetch_fsmap finished waiting for FSMap version 41 2024-05-10T11:29:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.063+0000 7f49ecf5db80 10 client.27485 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:29:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.063+0000 7f49ecf5db80 20 client.27485 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.063+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce9e461f90 0x55ce9e4643f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.063+0000 7f49ecf5db80 4 client.27485 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:29:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.063+0000 7f49ecf5db80 1 -- 172.21.15.27:0/1356116659 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ce9e31b400 con 0x55ce9e461f90 2024-05-10T11:29:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.063+0000 7f49d3400700 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3bb600 0x55ce9e3d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:29:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.064+0000 7f49d3400700 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce9e461f90 0x55ce9e4643f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.065+0000 7f49d3400700 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce9e461f90 0x55ce9e4643f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=84c19dde73c2d189 server_cookie=5a120a6f448f75b8 in_seq=0 out_seq=0 2024-05-10T11:29:36.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.065+0000 7f49c9600700 10 client.27485 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:29:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49c9600700 1 -- 172.21.15.27:0/1356116659 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ce9e31b400 con 0x55ce9e461f90 2024-05-10T11:29:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49c9600700 10 client.27485 handle_command_reply: tid=0 2024-05-10T11:29:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49ecf5db80 1 client.27485 shutdown 2024-05-10T11:29:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49ecf5db80 1 -- 172.21.15.27:0/1356116659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 msgr2=0x55ce9e3d1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3d1130 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f49c4005a80 tx=0x7f49c401c940 comp rx=0 tx=0).stop 2024-05-10T11:29:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49ecf5db80 1 -- 172.21.15.27:0/1356116659 shutdown_connections 2024-05-10T11:29:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55ce9e461f90 0x55ce9e4643f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce9e3c1300 0x55ce9e45fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce9e3bb600 0x55ce9e3d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49ecf5db80 1 --2- 172.21.15.27:0/1356116659 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce9e3c09e0 0x55ce9e3d1130 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49c9600700 0 client.27485 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:29:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49ecf5db80 1 -- 172.21.15.27:0/1356116659 >> 172.21.15.27:0/1356116659 conn(0x55ce9e2e9b60 msgr2=0x55ce9e3afbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.067+0000 7f49ecf5db80 1 -- 172.21.15.27:0/1356116659 shutdown_connections 2024-05-10T11:29:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.068+0000 7f49ecf5db80 1 -- 172.21.15.27:0/1356116659 wait complete. 2024-05-10T11:29:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.068+0000 7f49ecf5db80 20 client.27485 trim_cache size 0 max 16384 2024-05-10T11:29:36.068 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:29:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.069+0000 7f49e1000700 1 -- 172.21.15.27:0/1538741487 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f49c00b29f0 msgr2=0x7f49c00b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.069+0000 7f49e1000700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f49c00b29f0 0x7f49c00b4eb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f49dc06ada0 tx=0x7f49d400b040 comp rx=0 tx=0).stop 2024-05-10T11:29:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.069+0000 7f49e1000700 1 -- 172.21.15.27:0/1538741487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc102180 msgr2=0x7f49dc1a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.069+0000 7f49e1000700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc102180 0x7f49dc1a1910 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f49cc000c00 tx=0x7f49cc0057c0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.069+0000 7f49e1000700 1 -- 172.21.15.27:0/1538741487 shutdown_connections 2024-05-10T11:29:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.069+0000 7f49e1000700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f49c00b29f0 0x7f49c00b4eb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.069+0000 7f49e1000700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49dc10fd30 0x7f49dc19bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.069+0000 7f49e1000700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc102180 0x7f49dc1a1910 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.069+0000 7f49e1000700 1 --2- 172.21.15.27:0/1538741487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49dc102b30 0x7f49dc1a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.069+0000 7f49e1000700 1 -- 172.21.15.27:0/1538741487 >> 172.21.15.27:0/1538741487 conn(0x7f49dc0fddc0 msgr2=0x7f49dc1001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.070+0000 7f49e1000700 1 -- 172.21.15.27:0/1538741487 shutdown_connections 2024-05-10T11:29:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:36.070+0000 7f49e1000700 1 -- 172.21.15.27:0/1538741487 wait complete. 2024-05-10T11:29:36.151 INFO:tasks.workunit.client.0.smithi027.stdout: 10 61512 23.80 MB/sec execute 20 sec latency 166.555 ms 2024-05-10T11:29:36.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:36 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:36.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:36 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:37.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:29:37.152 INFO:tasks.workunit.client.0.smithi027.stdout: 10 61968 23.67 MB/sec execute 21 sec latency 126.081 ms 2024-05-10T11:29:37.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:37 smithi027 ceph-mon[30054]: pgmap v514: 129 pgs: 129 active+clean; 988 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 9.8 MiB/s wr, 558 op/s 2024-05-10T11:29:37.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:37 smithi076 ceph-mon[34152]: pgmap v514: 129 pgs: 129 active+clean; 988 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 9.8 MiB/s wr, 558 op/s 2024-05-10T11:29:37.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:37 smithi189 ceph-mon[34842]: pgmap v514: 129 pgs: 129 active+clean; 988 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 9.8 MiB/s wr, 558 op/s 2024-05-10T11:29:37.646 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:29:38.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:38 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:38.152 INFO:tasks.workunit.client.0.smithi027.stdout: 10 62373 23.53 MB/sec execute 22 sec latency 91.279 ms 2024-05-10T11:29:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:38 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:38.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:38 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:39.153 INFO:tasks.workunit.client.0.smithi027.stdout: 10 62683 22.82 MB/sec execute 23 sec latency 139.132 ms 2024-05-10T11:29:39.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:39 smithi027 ceph-mon[30054]: pgmap v515: 129 pgs: 129 active+clean; 988 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.4 MiB/s wr, 370 op/s 2024-05-10T11:29:39.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:39 smithi076 ceph-mon[34152]: pgmap v515: 129 pgs: 129 active+clean; 988 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.4 MiB/s wr, 370 op/s 2024-05-10T11:29:39.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:39 smithi189 ceph-mon[34842]: pgmap v515: 129 pgs: 129 active+clean; 988 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.4 MiB/s wr, 370 op/s 2024-05-10T11:29:40.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:40 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:40.152 INFO:tasks.workunit.client.0.smithi027.stdout: 10 63029 22.60 MB/sec execute 24 sec latency 399.883 ms 2024-05-10T11:29:40.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:40 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:40.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:40 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:40.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.810+0000 7fd793200700 1 -- 172.21.15.27:0/1393911744 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd78c1060a0 msgr2=0x7fd78c1125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:40.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.810+0000 7fd793200700 1 --2- 172.21.15.27:0/1393911744 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd78c1060a0 0x7fd78c1125b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd78400b0a0 tx=0x7fd78400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.810+0000 7fd793200700 1 -- 172.21.15.27:0/1393911744 shutdown_connections 2024-05-10T11:29:40.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.810+0000 7fd793200700 1 --2- 172.21.15.27:0/1393911744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78c10b6e0 0x7fd78c10bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.810+0000 7fd793200700 1 --2- 172.21.15.27:0/1393911744 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd78c1060a0 0x7fd78c1125b0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.810+0000 7fd793200700 1 --2- 172.21.15.27:0/1393911744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd78c1056e0 0x7fd78c105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.810+0000 7fd793200700 1 -- 172.21.15.27:0/1393911744 >> 172.21.15.27:0/1393911744 conn(0x7fd78c1011d0 msgr2=0x7fd78c1035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:40.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.811+0000 7fd793200700 1 -- 172.21.15.27:0/1393911744 shutdown_connections 2024-05-10T11:29:40.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.812+0000 7fd793200700 1 -- 172.21.15.27:0/1393911744 wait complete. 2024-05-10T11:29:40.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.813+0000 7fd793200700 1 Processor -- start 2024-05-10T11:29:40.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.813+0000 7fd793200700 1 -- start start 2024-05-10T11:29:40.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd793200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd78c1056e0 0x7fd78c19bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd793200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78c1060a0 0x7fd78c19c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd793200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd78c10b6e0 0x7fd78c1a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd793200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c115920 con 0x7fd78c1056e0 2024-05-10T11:29:40.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd793200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c1157a0 con 0x7fd78c10b6e0 2024-05-10T11:29:40.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd793200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c115aa0 con 0x7fd78c1060a0 2024-05-10T11:29:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd791200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd78c10b6e0 0x7fd78c1a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd78b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78c1060a0 0x7fd78c19c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd78be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd78c1056e0 0x7fd78c19bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd78b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78c1060a0 0x7fd78c19c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54172/0 (socket says 172.21.15.27:54172) 2024-05-10T11:29:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd791200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd78c10b6e0 0x7fd78c1a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:56532/0 (socket says 172.21.15.27:56532) 2024-05-10T11:29:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd78b400700 1 -- 172.21.15.27:0/1890875369 learned_addr learned my addr 172.21.15.27:0/1890875369 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd791200700 1 -- 172.21.15.27:0/1890875369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78c1060a0 msgr2=0x7fd78c19c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd791200700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78c1060a0 0x7fd78c19c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd791200700 1 -- 172.21.15.27:0/1890875369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd78c1056e0 msgr2=0x7fd78c19bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd791200700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd78c1056e0 0x7fd78c19bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd791200700 1 -- 172.21.15.27:0/1890875369 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78c1a0d20 con 0x7fd78c10b6e0 2024-05-10T11:29:40.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.814+0000 7fd78b400700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78c1060a0 0x7fd78c19c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:29:40.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.815+0000 7fd791200700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd78c10b6e0 0x7fd78c1a05e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd78400b070 tx=0x7fd784004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:40.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.815+0000 7fd788c00700 1 -- 172.21.15.27:0/1890875369 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784021070 con 0x7fd78c10b6e0 2024-05-10T11:29:40.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.815+0000 7fd788c00700 1 -- 172.21.15.27:0/1890875369 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd784007e30 con 0x7fd78c10b6e0 2024-05-10T11:29:40.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.815+0000 7fd788c00700 1 -- 172.21.15.27:0/1890875369 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7840045f0 con 0x7fd78c10b6e0 2024-05-10T11:29:40.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.815+0000 7fd793200700 1 -- 172.21.15.27:0/1890875369 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78c1a0fb0 con 0x7fd78c10b6e0 2024-05-10T11:29:40.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.815+0000 7fd793200700 1 -- 172.21.15.27:0/1890875369 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd78c1a8890 con 0x7fd78c10b6e0 2024-05-10T11:29:40.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.819+0000 7fd788c00700 1 -- 172.21.15.27:0/1890875369 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd784007980 con 0x7fd78c10b6e0 2024-05-10T11:29:40.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.821+0000 7fd79dc52b80 1 Processor -- start 2024-05-10T11:29:40.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.821+0000 7fd79dc52b80 1 -- start start 2024-05-10T11:29:40.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.821+0000 7fd79dc52b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1060 0x5595a4ec1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.821+0000 7fd788c00700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd7700b2a70 0x7fd7700b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd79dc52b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1980 0x5595a4ec5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd79dc52b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec59d0 0x5595a4ec5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd79dc52b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5595a4e13a20 con 0x5595a4ec1060 2024-05-10T11:29:40.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd79dc52b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5595a4db1760 con 0x5595a4ec1980 2024-05-10T11:29:40.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd79dc52b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5595a4e0f9e0 con 0x5595a4ec59d0 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd788c00700 1 -- 172.21.15.27:0/1890875369 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7fd78400f020 con 0x7fd78c10b6e0 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd791200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec59d0 0x5595a4ec5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd78b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1060 0x5595a4ec1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd791200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec59d0 0x5595a4ec5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54186/0 (socket says 172.21.15.27:54186) 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd78b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1060 0x5595a4ec1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57144/0 (socket says 172.21.15.27:57144) 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd791200700 1 -- 172.21.15.27:0/599152464 learned_addr learned my addr 172.21.15.27:0/599152464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd78be00700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd7700b2a70 0x7fd7700b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd791200700 1 -- 172.21.15.27:0/599152464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1980 msgr2=0x5595a4ec5490 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd791200700 1 --2- 172.21.15.27:0/599152464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1980 0x5595a4ec5490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd791200700 1 -- 172.21.15.27:0/599152464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1060 msgr2=0x5595a4ec1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd791200700 1 --2- 172.21.15.27:0/599152464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1060 0x5595a4ec1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.822+0000 7fd791200700 1 -- 172.21.15.27:0/599152464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595a4d608d0 con 0x5595a4ec59d0 2024-05-10T11:29:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.823+0000 7fd791200700 1 --2- 172.21.15.27:0/599152464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec59d0 0x5595a4ec5e30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd7840aab80 tx=0x7fd784004120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4edff84b68cdbfd4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.823+0000 7fd78be00700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd7700b2a70 0x7fd7700b4f30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd77c009be0 tx=0x7fd77c009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.823+0000 7fd781600700 1 -- 172.21.15.27:0/599152464 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78408a930 con 0x5595a4ec59d0 2024-05-10T11:29:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.823+0000 7fd781600700 1 -- 172.21.15.27:0/599152464 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd784007c70 con 0x5595a4ec59d0 2024-05-10T11:29:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.823+0000 7fd781600700 1 -- 172.21.15.27:0/599152464 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78400d130 con 0x5595a4ec59d0 2024-05-10T11:29:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.824+0000 7fd79dc52b80 1 -- 172.21.15.27:0/599152464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec59d0 msgr2=0x5595a4ec5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.824+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/599152464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec59d0 0x5595a4ec5e30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd7840aab80 tx=0x7fd784004120 comp rx=0 tx=0).stop 2024-05-10T11:29:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.824+0000 7fd79dc52b80 1 -- 172.21.15.27:0/599152464 shutdown_connections 2024-05-10T11:29:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.824+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/599152464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec59d0 0x5595a4ec5e30 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.824+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/599152464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1060 0x5595a4ec1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.824+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/599152464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1980 0x5595a4ec5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.824+0000 7fd79dc52b80 1 -- 172.21.15.27:0/599152464 >> 172.21.15.27:0/599152464 conn(0x5595a4defb60 msgr2=0x5595a4ebb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.824+0000 7fd79dc52b80 1 -- 172.21.15.27:0/599152464 shutdown_connections 2024-05-10T11:29:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.824+0000 7fd79dc52b80 1 -- 172.21.15.27:0/599152464 wait complete. 2024-05-10T11:29:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.825+0000 7fd79dc52b80 1 -- start start 2024-05-10T11:29:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.825+0000 7fd79dc52b80 1 Processor -- start 2024-05-10T11:29:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.825+0000 7fd79dc52b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 0x5595a4ecce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd79dc52b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1980 0x5595a4eca210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd79dc52b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec59d0 0x5595a4eca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd79dc52b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5595a4ecd9a0 con 0x5595a4ec59d0 2024-05-10T11:29:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd79dc52b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5595a4d79b20 con 0x5595a4ec1060 2024-05-10T11:29:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd79dc52b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5595a4ecdb20 con 0x5595a4ec1980 2024-05-10T11:29:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd79dc52b80 10 client.0 _ll_register_callbacks cb 0x5595a4dbac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd78be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 0x5595a4ecce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd78be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 0x5595a4ecce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:56550/0 (socket says 172.21.15.27:56550) 2024-05-10T11:29:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd78b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1980 0x5595a4eca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd78be00700 1 -- 172.21.15.27:0/513396846 learned_addr learned my addr 172.21.15.27:0/513396846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd791200700 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec59d0 0x5595a4eca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd78b400700 1 -- 172.21.15.27:0/513396846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 msgr2=0x5595a4ecce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd78b400700 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 0x5595a4ecce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd78b400700 1 -- 172.21.15.27:0/513396846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec59d0 msgr2=0x5595a4eca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd78b400700 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec59d0 0x5595a4eca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.826+0000 7fd78b400700 1 -- 172.21.15.27:0/513396846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595a4dd9420 con 0x5595a4ec1980 2024-05-10T11:29:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.827+0000 7fd791200700 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec59d0 0x5595a4eca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:29:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.827+0000 7fd78b400700 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1980 0x5595a4eca210 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd778012ca0 tx=0x7fd778009f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e83116403c9e5ea server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.827+0000 7fd776000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:29:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.827+0000 7fd776000700 1 -- 172.21.15.27:0/513396846 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77800f980 con 0x5595a4ec1980 2024-05-10T11:29:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.827+0000 7fd79dc52b80 1 -- 172.21.15.27:0/513396846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5595a4e8e0a0 con 0x5595a4ec1980 2024-05-10T11:29:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.827+0000 7fd776000700 1 -- 172.21.15.27:0/513396846 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7fd77800fb20 con 0x5595a4ec1980 2024-05-10T11:29:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.828+0000 7fd776000700 1 -- 172.21.15.27:0/513396846 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd778025050 con 0x5595a4ec1980 2024-05-10T11:29:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.828+0000 7fd79dc52b80 10 client.27493 fetch_fsmap learned FSMap version 43 2024-05-10T11:29:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.828+0000 7fd79dc52b80 1 -- 172.21.15.27:0/513396846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5595a4ed8210 con 0x5595a4ec1980 2024-05-10T11:29:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.828+0000 7fd776000700 1 -- 172.21.15.27:0/513396846 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5595a4e8e0a0 con 0x5595a4ec1980 2024-05-10T11:29:40.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.828+0000 7fd776000700 1 -- 172.21.15.27:0/513396846 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8374+0+0 (secure 0 0 0) 0x7fd778003ac0 con 0x5595a4ec1980 2024-05-10T11:29:40.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.829+0000 7fd79dc52b80 10 client.27493 fetch_fsmap finished waiting for FSMap version 43 2024-05-10T11:29:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.829+0000 7fd79dc52b80 10 client.27493 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:29:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.829+0000 7fd79dc52b80 20 client.27493 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.829+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5595a4eda7a0 0x5595a4edcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.829+0000 7fd79dc52b80 4 client.27493 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:29:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.829+0000 7fd79dc52b80 1 -- 172.21.15.27:0/513396846 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595a4e21400 con 0x5595a4eda7a0 2024-05-10T11:29:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.829+0000 7fd78be00700 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5595a4eda7a0 0x5595a4edcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.830+0000 7fd78be00700 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5595a4eda7a0 0x5595a4edcbc0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f879f4263b3417a4 server_cookie=7d0f5a5301aa5d84 in_seq=0 out_seq=0 2024-05-10T11:29:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.830+0000 7fd776000700 10 client.27493 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:40.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.831+0000 7fd776000700 1 -- 172.21.15.27:0/513396846 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5595a4e21400 con 0x5595a4eda7a0 2024-05-10T11:29:40.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.831+0000 7fd776000700 10 client.27493 handle_command_reply: tid=0 2024-05-10T11:29:40.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.831+0000 7fd79dc52b80 1 client.27493 shutdown 2024-05-10T11:29:40.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.831+0000 7fd79dc52b80 1 -- 172.21.15.27:0/513396846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1980 msgr2=0x5595a4eca210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:40.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.831+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1980 0x5595a4eca210 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd778012ca0 tx=0x7fd778009f10 comp rx=0 tx=0).stop 2024-05-10T11:29:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.831+0000 7fd79dc52b80 1 -- 172.21.15.27:0/513396846 shutdown_connections 2024-05-10T11:29:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.831+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5595a4eda7a0 0x5595a4edcbc0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.831+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec59d0 0x5595a4eca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.831+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 0x5595a4ecce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.831+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/513396846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1980 0x5595a4eca210 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.831+0000 7fd776000700 0 client.27493 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.831+0000 7fd79dc52b80 1 -- 172.21.15.27:0/513396846 >> 172.21.15.27:0/513396846 conn(0x5595a4defb60 msgr2=0x5595a4eb62b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.832+0000 7fd79dc52b80 1 -- 172.21.15.27:0/513396846 shutdown_connections 2024-05-10T11:29:40.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.833+0000 7fd79dc52b80 1 -- 172.21.15.27:0/513396846 wait complete. 2024-05-10T11:29:40.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.834+0000 7fd79dc52b80 20 client.27493 trim_cache size 0 max 16384 2024-05-10T11:29:40.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.855+0000 7fd79dc52b80 1 Processor -- start 2024-05-10T11:29:40.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.855+0000 7fd79dc52b80 1 -- start start 2024-05-10T11:29:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.855+0000 7fd79dc52b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 0x5595a4ec7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.855+0000 7fd79dc52b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1980 0x5595a4ecce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.855+0000 7fd79dc52b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec59d0 0x5595a4ed69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.855+0000 7fd79dc52b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c1b1810 con 0x5595a4ec1980 2024-05-10T11:29:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.855+0000 7fd79dc52b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c1b33e0 con 0x5595a4ec1060 2024-05-10T11:29:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.855+0000 7fd79dc52b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c1b1ee0 con 0x5595a4ec59d0 2024-05-10T11:29:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.856+0000 7fd78be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 0x5595a4ec7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.856+0000 7fd78be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 0x5595a4ec7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:56556/0 (socket says 172.21.15.27:56556) 2024-05-10T11:29:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.856+0000 7fd78be00700 1 -- 172.21.15.27:0/2735907936 learned_addr learned my addr 172.21.15.27:0/2735907936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.856+0000 7fd78b400700 1 --2- 172.21.15.27:0/2735907936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1980 0x5595a4ecce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.856+0000 7fd791200700 1 --2- 172.21.15.27:0/2735907936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec59d0 0x5595a4ed69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.856+0000 7fd78be00700 1 -- 172.21.15.27:0/2735907936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec59d0 msgr2=0x5595a4ed69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:40.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.856+0000 7fd78be00700 1 --2- 172.21.15.27:0/2735907936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec59d0 0x5595a4ed69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.856+0000 7fd78be00700 1 -- 172.21.15.27:0/2735907936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1980 msgr2=0x5595a4ecce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:40.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.856+0000 7fd78be00700 1 --2- 172.21.15.27:0/2735907936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1980 0x5595a4ecce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.856+0000 7fd78be00700 1 -- 172.21.15.27:0/2735907936 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595a4edd240 con 0x5595a4ec1060 2024-05-10T11:29:40.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.857+0000 7fd78b400700 1 --2- 172.21.15.27:0/2735907936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1980 0x5595a4ecce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:29:40.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.857+0000 7fd78be00700 1 --2- 172.21.15.27:0/2735907936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 0x5595a4ec7e80 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x5595a4edd4c0 tx=0x7fd77c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38a43633e735046e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:40.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.857+0000 7fd780c00700 1 -- 172.21.15.27:0/2735907936 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c02c070 con 0x5595a4ec1060 2024-05-10T11:29:40.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.857+0000 7fd780c00700 1 -- 172.21.15.27:0/2735907936 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd77c0044e0 con 0x5595a4ec1060 2024-05-10T11:29:40.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.857+0000 7fd780c00700 1 -- 172.21.15.27:0/2735907936 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c020b70 con 0x5595a4ec1060 2024-05-10T11:29:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.858+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2735907936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 msgr2=0x5595a4ec7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.858+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/2735907936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 0x5595a4ec7e80 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x5595a4edd4c0 tx=0x7fd77c014000 comp rx=0 tx=0).stop 2024-05-10T11:29:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.858+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2735907936 shutdown_connections 2024-05-10T11:29:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.858+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/2735907936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec59d0 0x5595a4ed69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.858+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/2735907936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec1060 0x5595a4ec7e80 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.858+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/2735907936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1980 0x5595a4ecce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.858+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2735907936 >> 172.21.15.27:0/2735907936 conn(0x5595a4defb60 msgr2=0x5595a4eb5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.858+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2735907936 shutdown_connections 2024-05-10T11:29:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.858+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2735907936 wait complete. 2024-05-10T11:29:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.859+0000 7fd79dc52b80 1 -- start start 2024-05-10T11:29:40.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.860+0000 7fd79dc52b80 1 Processor -- start 2024-05-10T11:29:40.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.860+0000 7fd79dc52b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1060 0x5595a4f42320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.860+0000 7fd79dc52b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1980 0x5595a4f42860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.860+0000 7fd79dc52b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec59d0 0x5595a4ed2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.860+0000 7fd79dc52b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c1b2c80 con 0x5595a4ec1980 2024-05-10T11:29:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.860+0000 7fd79dc52b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c1b25b0 con 0x5595a4ec59d0 2024-05-10T11:29:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.860+0000 7fd79dc52b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c1af7d0 con 0x5595a4ec1060 2024-05-10T11:29:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.861+0000 7fd78be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1060 0x5595a4f42320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.861+0000 7fd78be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1060 0x5595a4f42320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:54210/0 (socket says 172.21.15.27:54210) 2024-05-10T11:29:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.861+0000 7fd78be00700 1 -- 172.21.15.27:0/2454892754 learned_addr learned my addr 172.21.15.27:0/2454892754 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.861+0000 7fd79dc52b80 10 client.0 _ll_register_callbacks cb 0x5595a4dbac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.861+0000 7fd78b400700 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1980 0x5595a4f42860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.861+0000 7fd78be00700 1 -- 172.21.15.27:0/2454892754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec59d0 msgr2=0x5595a4ed2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.861+0000 7fd791200700 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec59d0 0x5595a4ed2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.861+0000 7fd78be00700 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec59d0 0x5595a4ed2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.861+0000 7fd78be00700 1 -- 172.21.15.27:0/2454892754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1980 msgr2=0x5595a4f42860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:40.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.861+0000 7fd78be00700 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1980 0x5595a4f42860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.861+0000 7fd78be00700 1 -- 172.21.15.27:0/2454892754 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595a4ed33e0 con 0x5595a4ec1060 2024-05-10T11:29:40.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.861+0000 7fd791200700 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec59d0 0x5595a4ed2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:29:40.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.862+0000 7fd78be00700 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1060 0x5595a4f42320 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0203e0 tx=0x7fd77c005760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eab2542f98a4671b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:40.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.862+0000 7fd776000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:29:40.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.862+0000 7fd776000700 1 -- 172.21.15.27:0/2454892754 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c02c040 con 0x5595a4ec1060 2024-05-10T11:29:40.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.862+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2454892754 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd77c0044e0 con 0x5595a4ec1060 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.862+0000 7fd776000700 1 -- 172.21.15.27:0/2454892754 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7fd77c026050 con 0x5595a4ec1060 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.862+0000 7fd776000700 1 -- 172.21.15.27:0/2454892754 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd77c0200d0 con 0x5595a4ec1060 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.863+0000 7fd79dc52b80 10 client.27499 fetch_fsmap learned FSMap version 43 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.863+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2454892754 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5595a4ec77b0 con 0x5595a4ec1060 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.863+0000 7fd776000700 1 -- 172.21.15.27:0/2454892754 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd77c0044e0 con 0x5595a4ec1060 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.863+0000 7fd776000700 1 -- 172.21.15.27:0/2454892754 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8374+0+0 (secure 0 0 0) 0x7fd77c038030 con 0x5595a4ec1060 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.863+0000 7fd79dc52b80 10 client.27499 fetch_fsmap finished waiting for FSMap version 43 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.863+0000 7fd79dc52b80 10 client.27499 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.863+0000 7fd79dc52b80 20 client.27499 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.863+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5595a4f4dd70 0x5595a4f50190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.863+0000 7fd79dc52b80 4 client.27499 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.863+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2454892754 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5595a4e1b4a0 con 0x5595a4f4dd70 2024-05-10T11:29:40.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.864+0000 7fd78b400700 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5595a4f4dd70 0x5595a4f50190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:40.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.865+0000 7fd78b400700 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5595a4f4dd70 0x5595a4f50190 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cf68619e5e46da7 server_cookie=94527d0995d664ee in_seq=0 out_seq=0 2024-05-10T11:29:40.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.865+0000 7fd776000700 10 client.27499 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:40.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd776000700 1 -- 172.21.15.27:0/2454892754 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5595a4e1b4a0 con 0x5595a4f4dd70 2024-05-10T11:29:40.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd776000700 10 client.27499 handle_command_reply: tid=0 2024-05-10T11:29:40.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd79dc52b80 1 client.27499 shutdown 2024-05-10T11:29:40.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2454892754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1060 msgr2=0x5595a4f42320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:40.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1060 0x5595a4f42320 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0203e0 tx=0x7fd77c005760 comp rx=0 tx=0).stop 2024-05-10T11:29:40.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2454892754 shutdown_connections 2024-05-10T11:29:40.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5595a4f4dd70 0x5595a4f50190 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd776000700 0 client.27499 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:40.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595a4ec1060 0x5595a4f42320 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595a4ec59d0 0x5595a4ed2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd79dc52b80 1 --2- 172.21.15.27:0/2454892754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595a4ec1980 0x5595a4f42860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2454892754 >> 172.21.15.27:0/2454892754 conn(0x5595a4defb60 msgr2=0x5595a4ecb810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:40.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2454892754 shutdown_connections 2024-05-10T11:29:40.869 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:29:40.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.867+0000 7fd79dc52b80 1 -- 172.21.15.27:0/2454892754 wait complete. 2024-05-10T11:29:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.868+0000 7fd79dc52b80 20 client.27499 trim_cache size 0 max 16384 2024-05-10T11:29:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.869+0000 7fd793200700 1 -- 172.21.15.27:0/1890875369 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd7700b2a70 msgr2=0x7fd7700b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.869+0000 7fd793200700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd7700b2a70 0x7fd7700b4f30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd77c009be0 tx=0x7fd77c009260 comp rx=0 tx=0).stop 2024-05-10T11:29:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.869+0000 7fd793200700 1 -- 172.21.15.27:0/1890875369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd78c10b6e0 msgr2=0x7fd78c1a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.869+0000 7fd793200700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd78c10b6e0 0x7fd78c1a05e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd78400b070 tx=0x7fd784004c80 comp rx=0 tx=0).stop 2024-05-10T11:29:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.869+0000 7fd793200700 1 -- 172.21.15.27:0/1890875369 shutdown_connections 2024-05-10T11:29:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.869+0000 7fd793200700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd7700b2a70 0x7fd7700b4f30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.869+0000 7fd793200700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd78c10b6e0 0x7fd78c1a05e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.869+0000 7fd793200700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd78c1056e0 0x7fd78c19bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.869+0000 7fd793200700 1 --2- 172.21.15.27:0/1890875369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78c1060a0 0x7fd78c19c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.869+0000 7fd793200700 1 -- 172.21.15.27:0/1890875369 >> 172.21.15.27:0/1890875369 conn(0x7fd78c1011d0 msgr2=0x7fd78c1016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.871+0000 7fd793200700 1 -- 172.21.15.27:0/1890875369 shutdown_connections 2024-05-10T11:29:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:40.871+0000 7fd793200700 1 -- 172.21.15.27:0/1890875369 wait complete. 2024-05-10T11:29:41.152 INFO:tasks.workunit.client.0.smithi027.stdout: 10 63605 23.25 MB/sec execute 25 sec latency 87.288 ms 2024-05-10T11:29:41.314 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:41 smithi027 ceph-mon[30054]: pgmap v516: 129 pgs: 129 active+clean; 1019 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 11 MiB/s wr, 567 op/s 2024-05-10T11:29:41.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:41 smithi076 ceph-mon[34152]: pgmap v516: 129 pgs: 129 active+clean; 1019 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 11 MiB/s wr, 567 op/s 2024-05-10T11:29:41.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:41 smithi189 ceph-mon[34842]: pgmap v516: 129 pgs: 129 active+clean; 1019 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 11 MiB/s wr, 567 op/s 2024-05-10T11:29:42.153 INFO:tasks.workunit.client.0.smithi027.stdout: 10 64146 23.18 MB/sec execute 26 sec latency 200.460 ms 2024-05-10T11:29:43.026 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:29:43.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:29:43.153 INFO:tasks.workunit.client.0.smithi027.stdout: 10 64690 23.35 MB/sec execute 27 sec latency 182.990 ms 2024-05-10T11:29:43.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:43 smithi027 ceph-mon[30054]: pgmap v517: 129 pgs: 129 active+clean; 1019 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.0 MiB/s wr, 372 op/s 2024-05-10T11:29:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:43 smithi076 ceph-mon[34152]: pgmap v517: 129 pgs: 129 active+clean; 1019 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.0 MiB/s wr, 372 op/s 2024-05-10T11:29:43.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:43 smithi189 ceph-mon[34842]: pgmap v517: 129 pgs: 129 active+clean; 1019 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.0 MiB/s wr, 372 op/s 2024-05-10T11:29:43.659 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:29:44.154 INFO:tasks.workunit.client.0.smithi027.stdout: 10 65210 23.97 MB/sec execute 28 sec latency 210.891 ms 2024-05-10T11:29:45.153 INFO:tasks.workunit.client.0.smithi027.stdout: 10 65513 23.71 MB/sec execute 29 sec latency 357.334 ms 2024-05-10T11:29:45.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:45 smithi027 ceph-mon[30054]: pgmap v518: 129 pgs: 129 active+clean; 1019 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.0 MiB/s wr, 372 op/s 2024-05-10T11:29:45.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:45 smithi076 ceph-mon[34152]: pgmap v518: 129 pgs: 129 active+clean; 1019 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.0 MiB/s wr, 372 op/s 2024-05-10T11:29:45.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:45 smithi189 ceph-mon[34842]: pgmap v518: 129 pgs: 129 active+clean; 1019 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.0 MiB/s wr, 372 op/s 2024-05-10T11:29:46.153 INFO:tasks.workunit.client.0.smithi027.stdout: 10 65887 23.46 MB/sec execute 30 sec latency 222.007 ms 2024-05-10T11:29:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.572+0000 7f46ef400700 1 -- 172.21.15.27:0/2355794062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e8109520 msgr2=0x7f46e8109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.572+0000 7f46ef400700 1 --2- 172.21.15.27:0/2355794062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e8109520 0x7f46e8109900 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f46d800b0a0 tx=0x7f46d800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.573+0000 7f46ef400700 1 -- 172.21.15.27:0/2355794062 shutdown_connections 2024-05-10T11:29:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.573+0000 7f46ef400700 1 --2- 172.21.15.27:0/2355794062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46e8102450 0x7f46e8110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.573+0000 7f46ef400700 1 --2- 172.21.15.27:0/2355794062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e8109520 0x7f46e8109900 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.573+0000 7f46ef400700 1 --2- 172.21.15.27:0/2355794062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e8101a90 0x7f46e8101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.573+0000 7f46ef400700 1 -- 172.21.15.27:0/2355794062 >> 172.21.15.27:0/2355794062 conn(0x7f46e8078a10 msgr2=0x7f46e80ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.573+0000 7f46ef400700 1 -- 172.21.15.27:0/2355794062 shutdown_connections 2024-05-10T11:29:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.573+0000 7f46ef400700 1 -- 172.21.15.27:0/2355794062 wait complete. 2024-05-10T11:29:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.574+0000 7f46ef400700 1 Processor -- start 2024-05-10T11:29:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.574+0000 7f46ef400700 1 -- start start 2024-05-10T11:29:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.574+0000 7f46ef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e8101a90 0x7f46e819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.574+0000 7f46ef400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46e8102450 0x7f46e819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.574+0000 7f46ef400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e8109520 0x7f46e8199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.574+0000 7f46ef400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f46e8113680 con 0x7f46e8101a90 2024-05-10T11:29:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.574+0000 7f46ef400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f46e8113500 con 0x7f46e8109520 2024-05-10T11:29:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.574+0000 7f46ef400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f46e8113800 con 0x7f46e8102450 2024-05-10T11:29:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.575+0000 7f46dfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46e8102450 0x7f46e819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.575+0000 7f46ed400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e8109520 0x7f46e8199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.575+0000 7f46dfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46e8102450 0x7f46e819fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:43996/0 (socket says 172.21.15.27:43996) 2024-05-10T11:29:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.575+0000 7f46eca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e8101a90 0x7f46e819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.575+0000 7f46dfe00700 1 -- 172.21.15.27:0/3718250775 learned_addr learned my addr 172.21.15.27:0/3718250775 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.575+0000 7f46dfe00700 1 -- 172.21.15.27:0/3718250775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e8109520 msgr2=0x7f46e8199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.575+0000 7f46dfe00700 1 --2- 172.21.15.27:0/3718250775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e8109520 0x7f46e8199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.575+0000 7f46dfe00700 1 -- 172.21.15.27:0/3718250775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e8101a90 msgr2=0x7f46e819f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.575+0000 7f46dfe00700 1 --2- 172.21.15.27:0/3718250775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e8101a90 0x7f46e819f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.575+0000 7f46dfe00700 1 -- 172.21.15.27:0/3718250775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e819a0e0 con 0x7f46e8102450 2024-05-10T11:29:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.576+0000 7f46dfe00700 1 --2- 172.21.15.27:0/3718250775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46e8102450 0x7f46e819fd00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f46d000eb30 tx=0x7f46d000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.576+0000 7f46e6000700 1 -- 172.21.15.27:0/3718250775 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d000cbd0 con 0x7f46e8102450 2024-05-10T11:29:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.576+0000 7f46ef400700 1 -- 172.21.15.27:0/3718250775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e819a3d0 con 0x7f46e8102450 2024-05-10T11:29:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.576+0000 7f46e6000700 1 -- 172.21.15.27:0/3718250775 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46d00044d0 con 0x7f46e8102450 2024-05-10T11:29:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.576+0000 7f46ef400700 1 -- 172.21.15.27:0/3718250775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e81a6690 con 0x7f46e8102450 2024-05-10T11:29:46.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.576+0000 7f46e6000700 1 -- 172.21.15.27:0/3718250775 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d0003e60 con 0x7f46e8102450 2024-05-10T11:29:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.580+0000 7f46e6000700 1 -- 172.21.15.27:0/3718250775 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f46d0004000 con 0x7f46e8102450 2024-05-10T11:29:46.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.581+0000 7f46f9e8bb80 1 Processor -- start 2024-05-10T11:29:46.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.581+0000 7f46f9e8bb80 1 -- start start 2024-05-10T11:29:46.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.581+0000 7f46f9e8bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d059160b0 0x556d05916490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.581+0000 7f46f9e8bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059169d0 0x556d05910760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.581+0000 7f46f9e8bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 0x556d059111e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.581+0000 7f46f9e8bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556d0585eaa0 con 0x556d059160b0 2024-05-10T11:29:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.581+0000 7f46f9e8bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x556d057fc7e0 con 0x556d059169d0 2024-05-10T11:29:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.581+0000 7f46f9e8bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556d0585aa60 con 0x556d05910d80 2024-05-10T11:29:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.581+0000 7f46e6000700 1 --2- 172.21.15.27:0/3718250775 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f46cc0b2a70 0x7f46cc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.581+0000 7f46dfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 0x556d059111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.582+0000 7f46eca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059169d0 0x556d05910760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.582+0000 7f46dfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 0x556d059111e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:44006/0 (socket says 172.21.15.27:44006) 2024-05-10T11:29:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.582+0000 7f46dfe00700 1 -- 172.21.15.27:0/1818940334 learned_addr learned my addr 172.21.15.27:0/1818940334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.582+0000 7f46e6000700 1 -- 172.21.15.27:0/3718250775 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f46d00e2d90 con 0x7f46e8102450 2024-05-10T11:29:46.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.582+0000 7f46ed400700 1 --2- 172.21.15.27:0/1818940334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d059160b0 0x556d05916490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.582+0000 7f46ed400700 1 --2- 172.21.15.27:0/3718250775 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f46cc0b2a70 0x7f46cc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.582+0000 7f46eca00700 1 -- 172.21.15.27:0/1818940334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 msgr2=0x556d059111e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.582+0000 7f46eca00700 1 --2- 172.21.15.27:0/1818940334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 0x556d059111e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.582+0000 7f46eca00700 1 -- 172.21.15.27:0/1818940334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d059160b0 msgr2=0x556d05916490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.582+0000 7f46eca00700 1 --2- 172.21.15.27:0/1818940334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d059160b0 0x556d05916490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.582+0000 7f46eca00700 1 -- 172.21.15.27:0/1818940334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d057ab950 con 0x556d059169d0 2024-05-10T11:29:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.583+0000 7f46eca00700 1 --2- 172.21.15.27:0/1818940334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059169d0 0x556d05910760 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f46d800b070 tx=0x7f46d8009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c05a9a53f72cac6e server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.583+0000 7f46ed400700 1 --2- 172.21.15.27:0/3718250775 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f46cc0b2a70 0x7f46cc0b4f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f46e00097a0 tx=0x7f46e0009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.583+0000 7f46de000700 1 -- 172.21.15.27:0/1818940334 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d800bcb0 con 0x556d059169d0 2024-05-10T11:29:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.583+0000 7f46de000700 1 -- 172.21.15.27:0/1818940334 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46d80074e0 con 0x556d059169d0 2024-05-10T11:29:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.583+0000 7f46de000700 1 -- 172.21.15.27:0/1818940334 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8005280 con 0x556d059169d0 2024-05-10T11:29:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.584+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/1818940334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059169d0 msgr2=0x556d05910760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.584+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/1818940334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059169d0 0x556d05910760 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f46d800b070 tx=0x7f46d8009360 comp rx=0 tx=0).stop 2024-05-10T11:29:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.584+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/1818940334 shutdown_connections 2024-05-10T11:29:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.584+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/1818940334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 0x556d059111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.584+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/1818940334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d059160b0 0x556d05916490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.584+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/1818940334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059169d0 0x556d05910760 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.584+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/1818940334 >> 172.21.15.27:0/1818940334 conn(0x556d05884750 msgr2=0x556d05906b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.584+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/1818940334 shutdown_connections 2024-05-10T11:29:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.584+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/1818940334 wait complete. 2024-05-10T11:29:46.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.585+0000 7f46f9e8bb80 1 -- start start 2024-05-10T11:29:46.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.585+0000 7f46f9e8bb80 1 Processor -- start 2024-05-10T11:29:46.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.585+0000 7f46f9e8bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 0x556d0591c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.585+0000 7f46f9e8bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d059160b0 0x556d05919a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46f9e8bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059169d0 0x556d05919f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46f9e8bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556d0591d3d0 con 0x556d05910d80 2024-05-10T11:29:46.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46f9e8bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x556d057c4ba0 con 0x556d059169d0 2024-05-10T11:29:46.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46f9e8bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556d0591d550 con 0x556d059160b0 2024-05-10T11:29:46.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46f9e8bb80 10 client.0 _ll_register_callbacks cb 0x556d05805cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:46.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46eca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 0x556d0591c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46dfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d059160b0 0x556d05919a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46eca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 0x556d0591c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41426/0 (socket says 172.21.15.27:41426) 2024-05-10T11:29:46.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46eca00700 1 -- 172.21.15.27:0/3991732695 learned_addr learned my addr 172.21.15.27:0/3991732695 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:46.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46ed400700 1 --2- 172.21.15.27:0/3991732695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059169d0 0x556d05919f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46eca00700 1 -- 172.21.15.27:0/3991732695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d059160b0 msgr2=0x556d05919a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:46.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46eca00700 1 --2- 172.21.15.27:0/3991732695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d059160b0 0x556d05919a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46eca00700 1 -- 172.21.15.27:0/3991732695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059169d0 msgr2=0x556d05919f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:46.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46eca00700 1 --2- 172.21.15.27:0/3991732695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059169d0 0x556d05919f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.586+0000 7f46eca00700 1 -- 172.21.15.27:0/3991732695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d058244a0 con 0x556d05910d80 2024-05-10T11:29:46.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.587+0000 7f46eca00700 1 --2- 172.21.15.27:0/3991732695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 0x556d0591c620 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f46d800d010 tx=0x7f46d8007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46910e8ca08928a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:46.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.587+0000 7f46d6a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:29:46.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.587+0000 7f46d6a00700 1 -- 172.21.15.27:0/3991732695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8007690 con 0x556d05910d80 2024-05-10T11:29:46.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.587+0000 7f46d6a00700 1 -- 172.21.15.27:0/3991732695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f46d8007830 con 0x556d05910d80 2024-05-10T11:29:46.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.587+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3991732695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46d80074e0 con 0x556d05910d80 2024-05-10T11:29:46.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.587+0000 7f46d6a00700 1 -- 172.21.15.27:0/3991732695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f46d8003a00 con 0x556d05910d80 2024-05-10T11:29:46.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.587+0000 7f46f9e8bb80 10 client.17211 fetch_fsmap learned FSMap version 43 2024-05-10T11:29:46.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.587+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3991732695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x556d05927b40 con 0x556d05910d80 2024-05-10T11:29:46.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.588+0000 7f46d6a00700 1 -- 172.21.15.27:0/3991732695 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46d80074e0 con 0x556d05910d80 2024-05-10T11:29:46.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.588+0000 7f46d6a00700 1 -- 172.21.15.27:0/3991732695 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 43) ==== 8374+0+0 (secure 0 0 0) 0x7f46d8011070 con 0x556d05910d80 2024-05-10T11:29:46.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.588+0000 7f46f9e8bb80 10 client.17211 fetch_fsmap finished waiting for FSMap version 43 2024-05-10T11:29:46.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.588+0000 7f46f9e8bb80 10 client.17211 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:29:46.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.588+0000 7f46f9e8bb80 20 client.17211 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:46.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.588+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3991732695 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556d0592a010 0x556d0592c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.588+0000 7f46f9e8bb80 4 client.17211 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:29:46.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.588+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3991732695 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d058d91c0 con 0x556d0592a010 2024-05-10T11:29:46.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.588+0000 7f46dfe00700 1 --2- 172.21.15.27:0/3991732695 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556d0592a010 0x556d0592c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.589+0000 7f46dfe00700 1 --2- 172.21.15.27:0/3991732695 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556d0592a010 0x556d0592c400 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0166bc5bfa4ef95 server_cookie=7bca3a3e444498ca in_seq=0 out_seq=0 2024-05-10T11:29:46.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.589+0000 7f46d6a00700 10 client.17211 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46d6a00700 1 -- 172.21.15.27:0/3991732695 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556d058d91c0 con 0x556d0592a010 2024-05-10T11:29:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46d6a00700 10 client.17211 handle_command_reply: tid=0 2024-05-10T11:29:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46f9e8bb80 1 client.17211 shutdown 2024-05-10T11:29:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3991732695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 msgr2=0x556d0591c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3991732695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 0x556d0591c620 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f46d800d010 tx=0x7f46d8007f60 comp rx=0 tx=0).stop 2024-05-10T11:29:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3991732695 shutdown_connections 2024-05-10T11:29:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3991732695 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556d0592a010 0x556d0592c400 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3991732695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059169d0 0x556d05919f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3991732695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 0x556d0591c620 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3991732695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d059160b0 0x556d05919a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3991732695 >> 172.21.15.27:0/3991732695 conn(0x556d05884750 msgr2=0x556d05906210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46d6a00700 0 client.17211 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.595+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3991732695 shutdown_connections 2024-05-10T11:29:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.596+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3991732695 wait complete. 2024-05-10T11:29:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.596+0000 7f46f9e8bb80 20 client.17211 trim_cache size 0 max 16384 2024-05-10T11:29:46.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.620+0000 7f46f9e8bb80 1 Processor -- start 2024-05-10T11:29:46.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.620+0000 7f46f9e8bb80 1 -- start start 2024-05-10T11:29:46.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46f9e8bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05884750 0x556d05919f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46f9e8bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 0x556d05917420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46f9e8bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059160b0 0x556d05906990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46f9e8bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f46e81ad3f0 con 0x556d05884750 2024-05-10T11:29:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46f9e8bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f46e81aefc0 con 0x556d059160b0 2024-05-10T11:29:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46f9e8bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f46e81adac0 con 0x556d05910d80 2024-05-10T11:29:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46dfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 0x556d05917420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46eca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05884750 0x556d05919f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46ed400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059160b0 0x556d05906990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46dfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 0x556d05917420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:44022/0 (socket says 172.21.15.27:44022) 2024-05-10T11:29:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46ed400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059160b0 0x556d05906990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50034/0 (socket says 172.21.15.27:50034) 2024-05-10T11:29:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46dfe00700 1 -- 172.21.15.27:0/3875523713 learned_addr learned my addr 172.21.15.27:0/3875523713 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46dfe00700 1 -- 172.21.15.27:0/3875523713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059160b0 msgr2=0x556d05906990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46dfe00700 1 --2- 172.21.15.27:0/3875523713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059160b0 0x556d05906990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46dfe00700 1 -- 172.21.15.27:0/3875523713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05884750 msgr2=0x556d05919f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:46.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46dfe00700 1 --2- 172.21.15.27:0/3875523713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05884750 0x556d05919f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.621+0000 7f46dfe00700 1 -- 172.21.15.27:0/3875523713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d05917f80 con 0x556d05910d80 2024-05-10T11:29:46.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.622+0000 7f46dfe00700 1 --2- 172.21.15.27:0/3875523713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 0x556d05917420 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f46d00a6e50 tx=0x7f46d0076510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5521fc3b7fe6b99f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:46.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.622+0000 7f46dd600700 1 -- 172.21.15.27:0/3875523713 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d000ce90 con 0x556d05910d80 2024-05-10T11:29:46.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.622+0000 7f46dd600700 1 -- 172.21.15.27:0/3875523713 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46d00b2380 con 0x556d05910d80 2024-05-10T11:29:46.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.622+0000 7f46dd600700 1 -- 172.21.15.27:0/3875523713 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d00a9be0 con 0x556d05910d80 2024-05-10T11:29:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.623+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3875523713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 msgr2=0x556d05917420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.623+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3875523713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 0x556d05917420 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f46d00a6e50 tx=0x7f46d0076510 comp rx=0 tx=0).stop 2024-05-10T11:29:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.623+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3875523713 shutdown_connections 2024-05-10T11:29:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.623+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3875523713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059160b0 0x556d05906990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.623+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3875523713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05884750 0x556d05919f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.623+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3875523713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05910d80 0x556d05917420 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.623+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3875523713 >> 172.21.15.27:0/3875523713 conn(0x556d059037d0 msgr2=0x556d0590bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.623+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3875523713 shutdown_connections 2024-05-10T11:29:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.623+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3875523713 wait complete. 2024-05-10T11:29:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.624+0000 7f46f9e8bb80 1 -- start start 2024-05-10T11:29:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.624+0000 7f46f9e8bb80 1 Processor -- start 2024-05-10T11:29:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.625+0000 7f46f9e8bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05884750 0x556d0592f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.625+0000 7f46f9e8bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 0x556d0592db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.625+0000 7f46f9e8bb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059160b0 0x556d0592e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.625+0000 7f46f9e8bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f46e81ae860 con 0x556d05910d80 2024-05-10T11:29:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.625+0000 7f46f9e8bb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f46e81ae190 con 0x556d059160b0 2024-05-10T11:29:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.625+0000 7f46f9e8bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f46e81ab3f0 con 0x556d05884750 2024-05-10T11:29:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.625+0000 7f46f9e8bb80 10 client.0 _ll_register_callbacks cb 0x556d05805cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.625+0000 7f46dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 0x556d0592db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.625+0000 7f46dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 0x556d0592db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41450/0 (socket says 172.21.15.27:41450) 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.625+0000 7f46eca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05884750 0x556d0592f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.625+0000 7f46dfe00700 1 -- 172.21.15.27:0/3798023726 learned_addr learned my addr 172.21.15.27:0/3798023726 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.625+0000 7f46ed400700 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059160b0 0x556d0592e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.626+0000 7f46eca00700 1 -- 172.21.15.27:0/3798023726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059160b0 msgr2=0x556d0592e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.626+0000 7f46eca00700 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059160b0 0x556d0592e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.626+0000 7f46eca00700 1 -- 172.21.15.27:0/3798023726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 msgr2=0x556d0592db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.626+0000 7f46eca00700 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 0x556d0592db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.626+0000 7f46eca00700 1 -- 172.21.15.27:0/3798023726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d05994540 con 0x556d05884750 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.626+0000 7f46dfe00700 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 0x556d0592db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.626+0000 7f46eca00700 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05884750 0x556d0592f770 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f46d8000f80 tx=0x7f46d801c500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81e455d2e3edd720 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.626+0000 7f46d6a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:29:46.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.626+0000 7f46d6a00700 1 -- 172.21.15.27:0/3798023726 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8003ba0 con 0x556d05884750 2024-05-10T11:29:46.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.626+0000 7f46d6a00700 1 -- 172.21.15.27:0/3798023726 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f46d8003d40 con 0x556d05884750 2024-05-10T11:29:46.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.626+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3798023726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556d0586c480 con 0x556d05884750 2024-05-10T11:29:46.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.627+0000 7f46d6a00700 1 -- 172.21.15.27:0/3798023726 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f46d8005110 con 0x556d05884750 2024-05-10T11:29:46.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.627+0000 7f46f9e8bb80 10 client.27517 fetch_fsmap learned FSMap version 43 2024-05-10T11:29:46.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.627+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3798023726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x556d059957f0 con 0x556d05884750 2024-05-10T11:29:46.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.630+0000 7f46d6a00700 1 -- 172.21.15.27:0/3798023726 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556d0586c480 con 0x556d05884750 2024-05-10T11:29:46.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.630+0000 7f46d6a00700 1 -- 172.21.15.27:0/3798023726 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8374+0+0 (secure 0 0 0) 0x7f46d8011040 con 0x556d05884750 2024-05-10T11:29:46.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.630+0000 7f46f9e8bb80 10 client.27517 fetch_fsmap finished waiting for FSMap version 43 2024-05-10T11:29:46.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.630+0000 7f46f9e8bb80 10 client.27517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:29:46.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.630+0000 7f46f9e8bb80 20 client.27517 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:46.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.630+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556d059169d0 0x556d05997d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:46.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.630+0000 7f46f9e8bb80 4 client.27517 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:29:46.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.630+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3798023726 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556d05866520 con 0x556d059169d0 2024-05-10T11:29:46.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.633+0000 7f46dfe00700 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556d059169d0 0x556d05997d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:46.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.637+0000 7f46d6a00700 10 client.27517 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:46.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.637+0000 7f46dfe00700 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556d059169d0 0x556d05997d90 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=696e6d50fbddc0ce server_cookie=ef8f5b0c1b0b3d55 in_seq=0 out_seq=0 2024-05-10T11:29:46.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.643+0000 7f46d6a00700 1 -- 172.21.15.27:0/3798023726 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556d05866520 con 0x556d059169d0 2024-05-10T11:29:46.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.643+0000 7f46d6a00700 10 client.27517 handle_command_reply: tid=0 2024-05-10T11:29:46.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.643+0000 7f46f9e8bb80 1 client.27517 shutdown 2024-05-10T11:29:46.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.645+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3798023726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05884750 msgr2=0x556d0592f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:46.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.645+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05884750 0x556d0592f770 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f46d8000f80 tx=0x7f46d801c500 comp rx=0 tx=0).stop 2024-05-10T11:29:46.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.645+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3798023726 shutdown_connections 2024-05-10T11:29:46.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.645+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x556d059169d0 0x556d05997d90 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.645+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556d05884750 0x556d0592f770 secure :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f46d8000f80 tx=0x7f46d801c500 comp rx=0 tx=0).stop 2024-05-10T11:29:46.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.645+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556d059160b0 0x556d0592e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.645+0000 7f46f9e8bb80 1 --2- 172.21.15.27:0/3798023726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d05910d80 0x556d0592db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.645+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3798023726 >> 172.21.15.27:0/3798023726 conn(0x556d059037d0 msgr2=0x556d0590bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:46.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.646+0000 7f46d6a00700 0 client.27517 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:46.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.648+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3798023726 shutdown_connections 2024-05-10T11:29:46.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.648+0000 7f46f9e8bb80 1 -- 172.21.15.27:0/3798023726 wait complete. 2024-05-10T11:29:46.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.648+0000 7f46f9e8bb80 20 client.27517 trim_cache size 0 max 16384 2024-05-10T11:29:46.649 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:29:46.649 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:29:46.649 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "df3972d2-4611-43ba-ad31-582245af93e1", 2024-05-10T11:29:46.650 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:29:46.650 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:29:46.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.649+0000 7f46ef400700 1 -- 172.21.15.27:0/3718250775 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f46cc0b2a70 msgr2=0x7f46cc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:46.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.649+0000 7f46ef400700 1 --2- 172.21.15.27:0/3718250775 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f46cc0b2a70 0x7f46cc0b4f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f46e00097a0 tx=0x7f46e0009260 comp rx=0 tx=0).stop 2024-05-10T11:29:46.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.649+0000 7f46ef400700 1 -- 172.21.15.27:0/3718250775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46e8102450 msgr2=0x7f46e819fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:46.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.649+0000 7f46ef400700 1 --2- 172.21.15.27:0/3718250775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46e8102450 0x7f46e819fd00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f46d000eb30 tx=0x7f46d000ee40 comp rx=0 tx=0).stop 2024-05-10T11:29:46.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.650+0000 7f46ef400700 1 -- 172.21.15.27:0/3718250775 shutdown_connections 2024-05-10T11:29:46.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.650+0000 7f46ef400700 1 --2- 172.21.15.27:0/3718250775 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f46cc0b2a70 0x7f46cc0b4f30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.650+0000 7f46ef400700 1 --2- 172.21.15.27:0/3718250775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46e8109520 0x7f46e8199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.650+0000 7f46ef400700 1 --2- 172.21.15.27:0/3718250775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e8101a90 0x7f46e819f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.650+0000 7f46ef400700 1 --2- 172.21.15.27:0/3718250775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46e8102450 0x7f46e819fd00 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:46.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.650+0000 7f46ef400700 1 -- 172.21.15.27:0/3718250775 >> 172.21.15.27:0/3718250775 conn(0x7f46e8078a10 msgr2=0x7f46e81002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:46.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.651+0000 7f46ef400700 1 -- 172.21.15.27:0/3718250775 shutdown_connections 2024-05-10T11:29:46.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:46.651+0000 7f46ef400700 1 -- 172.21.15.27:0/3718250775 wait complete. 2024-05-10T11:29:47.153 INFO:tasks.workunit.client.0.smithi027.stdout: 10 66313 23.30 MB/sec execute 31 sec latency 306.984 ms 2024-05-10T11:29:47.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:47 smithi027 ceph-mon[30054]: pgmap v519: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 10 MiB/s wr, 552 op/s 2024-05-10T11:29:47.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:47 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:29:47.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:47 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:47.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:47 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:47.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:47 smithi076 ceph-mon[34152]: pgmap v519: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 10 MiB/s wr, 552 op/s 2024-05-10T11:29:47.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:47 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:29:47.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:47 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:47.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:47 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:47.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:47 smithi189 ceph-mon[34842]: pgmap v519: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 10 MiB/s wr, 552 op/s 2024-05-10T11:29:47.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:47 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:29:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:47 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:47.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:47 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:47.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:29:48.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:29:48.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:48.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:48 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.153 INFO:tasks.workunit.client.0.smithi027.stdout: 10 66785 23.38 MB/sec execute 32 sec latency 658.728 ms 2024-05-10T11:29:48.364 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:29:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:29:48.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:48.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:48 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:29:48.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:48.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:48.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:48 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:29:49.153 INFO:tasks.workunit.client.0.smithi027.stdout: 10 67302 23.54 MB/sec execute 33 sec latency 113.867 ms 2024-05-10T11:29:49.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:49 smithi027 ceph-mon[30054]: pgmap v520: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.1 MiB/s wr, 377 op/s 2024-05-10T11:29:49.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:49 smithi076 ceph-mon[34152]: pgmap v520: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.1 MiB/s wr, 377 op/s 2024-05-10T11:29:49.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:49 smithi189 ceph-mon[34842]: pgmap v520: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 7.1 MiB/s wr, 377 op/s 2024-05-10T11:29:50.153 INFO:tasks.workunit.client.0.smithi027.stdout: 10 67640 23.38 MB/sec execute 34 sec latency 278.719 ms 2024-05-10T11:29:51.154 INFO:tasks.workunit.client.0.smithi027.stdout: 10 68098 23.59 MB/sec execute 35 sec latency 127.981 ms 2024-05-10T11:29:51.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:51 smithi027 ceph-mon[30054]: pgmap v521: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 10 MiB/s wr, 583 op/s 2024-05-10T11:29:51.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:51 smithi076 ceph-mon[34152]: pgmap v521: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 10 MiB/s wr, 583 op/s 2024-05-10T11:29:51.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:51 smithi189 ceph-mon[34842]: pgmap v521: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 10 MiB/s wr, 583 op/s 2024-05-10T11:29:51.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.687+0000 7f4b92a00700 1 -- 172.21.15.27:0/1988628936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b8c10b690 msgr2=0x7f4b8c10ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.687+0000 7f4b92a00700 1 --2- 172.21.15.27:0/1988628936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b8c10b690 0x7f4b8c10ba70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009990 tx=0x7f4b7c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.687+0000 7f4b92a00700 1 -- 172.21.15.27:0/1988628936 shutdown_connections 2024-05-10T11:29:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.687+0000 7f4b92a00700 1 --2- 172.21.15.27:0/1988628936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b8c106050 0x7f4b8c112560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.687+0000 7f4b92a00700 1 --2- 172.21.15.27:0/1988628936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b8c10b690 0x7f4b8c10ba70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.687+0000 7f4b92a00700 1 --2- 172.21.15.27:0/1988628936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b8c105690 0x7f4b8c105b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.687+0000 7f4b92a00700 1 -- 172.21.15.27:0/1988628936 >> 172.21.15.27:0/1988628936 conn(0x7f4b8c101180 msgr2=0x7f4b8c1035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.687+0000 7f4b92a00700 1 -- 172.21.15.27:0/1988628936 shutdown_connections 2024-05-10T11:29:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.687+0000 7f4b92a00700 1 -- 172.21.15.27:0/1988628936 wait complete. 2024-05-10T11:29:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.688+0000 7f4b92a00700 1 Processor -- start 2024-05-10T11:29:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.689+0000 7f4b92a00700 1 -- start start 2024-05-10T11:29:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.689+0000 7f4b92a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b8c105690 0x7f4b8c19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.689+0000 7f4b92a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b8c106050 0x7f4b8c19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.689+0000 7f4b92a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b8c10b690 0x7f4b8c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.689+0000 7f4b92a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c1158e0 con 0x7f4b8c106050 2024-05-10T11:29:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.689+0000 7f4b92a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c115760 con 0x7f4b8c10b690 2024-05-10T11:29:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.689+0000 7f4b92a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c115a60 con 0x7f4b8c105690 2024-05-10T11:29:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.690+0000 7f4b8b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b8c106050 0x7f4b8c19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.690+0000 7f4b8b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b8c106050 0x7f4b8c19c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41460/0 (socket says 172.21.15.27:41460) 2024-05-10T11:29:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.690+0000 7f4b8b400700 1 -- 172.21.15.27:0/1290610979 learned_addr learned my addr 172.21.15.27:0/1290610979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.690+0000 7f4b90a00700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b8c10b690 0x7f4b8c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.690+0000 7f4b8be00700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b8c105690 0x7f4b8c19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.690+0000 7f4b8b400700 1 -- 172.21.15.27:0/1290610979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b8c105690 msgr2=0x7f4b8c19bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:51.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.690+0000 7f4b8b400700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b8c105690 0x7f4b8c19bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.690+0000 7f4b8b400700 1 -- 172.21.15.27:0/1290610979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b8c10b690 msgr2=0x7f4b8c1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:51.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.690+0000 7f4b8b400700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b8c10b690 0x7f4b8c1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.690+0000 7f4b8b400700 1 -- 172.21.15.27:0/1290610979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8c1a0da0 con 0x7f4b8c106050 2024-05-10T11:29:51.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.691+0000 7f4b8b400700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b8c106050 0x7f4b8c19c2d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800e9a0 tx=0x7f4b7800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:51.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.694+0000 7f4b88c00700 1 -- 172.21.15.27:0/1290610979 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7800ca10 con 0x7f4b8c106050 2024-05-10T11:29:51.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.694+0000 7f4b88c00700 1 -- 172.21.15.27:0/1290610979 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b7800cbb0 con 0x7f4b8c106050 2024-05-10T11:29:51.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.694+0000 7f4b88c00700 1 -- 172.21.15.27:0/1290610979 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b78018680 con 0x7f4b8c106050 2024-05-10T11:29:51.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.694+0000 7f4b92a00700 1 -- 172.21.15.27:0/1290610979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8c1a1090 con 0x7f4b8c106050 2024-05-10T11:29:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.694+0000 7f4b92a00700 1 -- 172.21.15.27:0/1290610979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b8c0778e0 con 0x7f4b8c106050 2024-05-10T11:29:51.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.701+0000 7f4b9d363b80 1 Processor -- start 2024-05-10T11:29:51.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.701+0000 7f4b9d363b80 1 -- start start 2024-05-10T11:29:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.702+0000 7f4b9d363b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84cbbc0 0x5585c84cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.702+0000 7f4b9d363b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 0x5585c84c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.702+0000 7f4b9d363b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 0x5585c84c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.702+0000 7f4b9d363b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5585c8418a20 con 0x5585c84c68f0 2024-05-10T11:29:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.702+0000 7f4b9d363b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5585c83b6760 con 0x5585c84cc4e0 2024-05-10T11:29:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.702+0000 7f4b9d363b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5585c84149e0 con 0x5585c84cbbc0 2024-05-10T11:29:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.702+0000 7f4b8be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 0x5585c84c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.702+0000 7f4b90a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84cbbc0 0x5585c84cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.702+0000 7f4b8be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 0x5585c84c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50074/0 (socket says 172.21.15.27:50074) 2024-05-10T11:29:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.702+0000 7f4b8be00700 1 -- 172.21.15.27:0/467201380 learned_addr learned my addr 172.21.15.27:0/467201380 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.703+0000 7f4b90a00700 1 -- 172.21.15.27:0/467201380 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 msgr2=0x5585c84c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.703+0000 7f4b90a00700 1 --2- 172.21.15.27:0/467201380 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 0x5585c84c62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.703+0000 7f4b90a00700 1 -- 172.21.15.27:0/467201380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 msgr2=0x5585c84c6d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:29:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.703+0000 7f4b90a00700 1 --2- 172.21.15.27:0/467201380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 0x5585c84c6d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.703+0000 7f4b90a00700 1 -- 172.21.15.27:0/467201380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585c83658d0 con 0x5585c84cbbc0 2024-05-10T11:29:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.704+0000 7f4b88c00700 1 -- 172.21.15.27:0/1290610979 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f4b78010430 con 0x7f4b8c106050 2024-05-10T11:29:51.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.706+0000 7f4b90a00700 1 --2- 172.21.15.27:0/467201380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84cbbc0 0x5585c84cbfa0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4b84003ea0 tx=0x7f4b84009ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc6348eec309371c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:51.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.706+0000 7f4b81600700 1 -- 172.21.15.27:0/467201380 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b84013c50 con 0x5585c84cbbc0 2024-05-10T11:29:51.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.707+0000 7f4b81600700 1 -- 172.21.15.27:0/467201380 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b84013df0 con 0x5585c84cbbc0 2024-05-10T11:29:51.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.707+0000 7f4b81600700 1 -- 172.21.15.27:0/467201380 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b84007760 con 0x5585c84cbbc0 2024-05-10T11:29:51.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.707+0000 7f4b88c00700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4b700b2a70 0x7f4b700b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.707+0000 7f4b88c00700 1 -- 172.21.15.27:0/1290610979 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f4b780e2ba0 con 0x7f4b8c106050 2024-05-10T11:29:51.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.708+0000 7f4b8be00700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4b700b2a70 0x7f4b700b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.708+0000 7f4b9d363b80 1 -- 172.21.15.27:0/467201380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84cbbc0 msgr2=0x5585c84cbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.708+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/467201380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84cbbc0 0x5585c84cbfa0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4b84003ea0 tx=0x7f4b84009ba0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.709+0000 7f4b9d363b80 1 -- 172.21.15.27:0/467201380 shutdown_connections 2024-05-10T11:29:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.709+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/467201380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84cbbc0 0x5585c84cbfa0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.709+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/467201380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 0x5585c84c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.709+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/467201380 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 0x5585c84c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.709+0000 7f4b8be00700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4b700b2a70 0x7f4b700b4f30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x5585c84cc8e0 tx=0x7f4b7c00b280 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.709+0000 7f4b9d363b80 1 -- 172.21.15.27:0/467201380 >> 172.21.15.27:0/467201380 conn(0x5585c843e6d0 msgr2=0x5585c84c0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:51.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.709+0000 7f4b9d363b80 1 -- 172.21.15.27:0/467201380 shutdown_connections 2024-05-10T11:29:51.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.709+0000 7f4b9d363b80 1 -- 172.21.15.27:0/467201380 wait complete. 2024-05-10T11:29:51.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.710+0000 7f4b9d363b80 1 -- start start 2024-05-10T11:29:51.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.710+0000 7f4b9d363b80 1 Processor -- start 2024-05-10T11:29:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.711+0000 7f4b9d363b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 0x5585c84d2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.711+0000 7f4b9d363b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84cbbc0 0x5585c84cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.711+0000 7f4b9d363b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 0x5585c84cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.711+0000 7f4b9d363b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5585c84d2f20 con 0x5585c84c68f0 2024-05-10T11:29:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.711+0000 7f4b9d363b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5585c837eb20 con 0x5585c84cc4e0 2024-05-10T11:29:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.711+0000 7f4b9d363b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5585c84d30a0 con 0x5585c84cbbc0 2024-05-10T11:29:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.711+0000 7f4b9d363b80 10 client.0 _ll_register_callbacks cb 0x5585c83bfc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.712+0000 7f4b8b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84cbbc0 0x5585c84cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.712+0000 7f4b90a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 0x5585c84cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.712+0000 7f4b90a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 0x5585c84cfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50086/0 (socket says 172.21.15.27:50086) 2024-05-10T11:29:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.712+0000 7f4b8b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84cbbc0 0x5585c84cf5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:44066/0 (socket says 172.21.15.27:44066) 2024-05-10T11:29:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.712+0000 7f4b90a00700 1 -- 172.21.15.27:0/1474010534 learned_addr learned my addr 172.21.15.27:0/1474010534 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.712+0000 7f4b8be00700 1 --2- 172.21.15.27:0/1474010534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 0x5585c84d2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.712+0000 7f4b90a00700 1 -- 172.21.15.27:0/1474010534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84cbbc0 msgr2=0x5585c84cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.712+0000 7f4b90a00700 1 --2- 172.21.15.27:0/1474010534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84cbbc0 0x5585c84cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.712+0000 7f4b90a00700 1 -- 172.21.15.27:0/1474010534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 msgr2=0x5585c84d2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.712+0000 7f4b90a00700 1 --2- 172.21.15.27:0/1474010534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 0x5585c84d2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.712+0000 7f4b90a00700 1 -- 172.21.15.27:0/1474010534 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585c83de420 con 0x5585c84cc4e0 2024-05-10T11:29:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.713+0000 7f4b90a00700 1 --2- 172.21.15.27:0/1474010534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 0x5585c84cfb10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4b840046d0 tx=0x7f4b84004df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=470a790d64fa35df server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.713+0000 7f4b76000700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:29:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.713+0000 7f4b76000700 1 -- 172.21.15.27:0/1474010534 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8402b020 con 0x5585c84cc4e0 2024-05-10T11:29:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.713+0000 7f4b76000700 1 -- 172.21.15.27:0/1474010534 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f4b84004210 con 0x5585c84cc4e0 2024-05-10T11:29:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.713+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1474010534 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b84013df0 con 0x5585c84cc4e0 2024-05-10T11:29:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.714+0000 7f4b76000700 1 -- 172.21.15.27:0/1474010534 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f4b8400f050 con 0x5585c84cc4e0 2024-05-10T11:29:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.714+0000 7f4b9d363b80 10 client.27527 fetch_fsmap learned FSMap version 44 2024-05-10T11:29:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.714+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1474010534 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5585c84ee830 con 0x5585c84cc4e0 2024-05-10T11:29:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.714+0000 7f4b76000700 1 -- 172.21.15.27:0/1474010534 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b84013df0 con 0x5585c84cc4e0 2024-05-10T11:29:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.715+0000 7f4b76000700 1 -- 172.21.15.27:0/1474010534 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 44) ==== 8370+0+0 (secure 0 0 0) 0x7f4b84007b50 con 0x5585c84cc4e0 2024-05-10T11:29:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.715+0000 7f4b9d363b80 10 client.27527 fetch_fsmap finished waiting for FSMap version 44 2024-05-10T11:29:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.715+0000 7f4b9d363b80 10 client.27527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:29:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.715+0000 7f4b9d363b80 20 client.27527 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.715+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/1474010534 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5585c84f0d00 0x5585c84f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.715+0000 7f4b9d363b80 4 client.27527 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:29:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.715+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1474010534 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585c8493140 con 0x5585c84f0d00 2024-05-10T11:29:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.716+0000 7f4b8be00700 1 --2- 172.21.15.27:0/1474010534 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5585c84f0d00 0x5585c84f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.717+0000 7f4b8be00700 1 --2- 172.21.15.27:0/1474010534 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5585c84f0d00 0x5585c84f30f0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72ae4ebe58affc72 server_cookie=4039ba949b2b34e7 in_seq=0 out_seq=0 2024-05-10T11:29:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.717+0000 7f4b76000700 10 client.27527 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:51.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.718+0000 7f4b76000700 1 -- 172.21.15.27:0/1474010534 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5585c8493140 con 0x5585c84f0d00 2024-05-10T11:29:51.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.718+0000 7f4b76000700 10 client.27527 handle_command_reply: tid=0 2024-05-10T11:29:51.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.718+0000 7f4b9d363b80 1 client.27527 shutdown 2024-05-10T11:29:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.718+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1474010534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 msgr2=0x5585c84cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.718+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/1474010534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 0x5585c84cfb10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4b840046d0 tx=0x7f4b84004df0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.718+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1474010534 shutdown_connections 2024-05-10T11:29:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.718+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/1474010534 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5585c84f0d00 0x5585c84f30f0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.718+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/1474010534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cc4e0 0x5585c84cfb10 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.718+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/1474010534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 0x5585c84d2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.718+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/1474010534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84cbbc0 0x5585c84cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.719+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1474010534 >> 172.21.15.27:0/1474010534 conn(0x5585c843e6d0 msgr2=0x5585c84c0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.719+0000 7f4b76000700 0 client.27527 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.719+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1474010534 shutdown_connections 2024-05-10T11:29:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.719+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1474010534 wait complete. 2024-05-10T11:29:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.719+0000 7f4b9d363b80 20 client.27527 trim_cache size 0 max 16384 2024-05-10T11:29:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.755+0000 7f4b9d363b80 1 Processor -- start 2024-05-10T11:29:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.755+0000 7f4b9d363b80 1 -- start start 2024-05-10T11:29:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.756+0000 7f4b9d363b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c843e6d0 0x5585c84ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.756+0000 7f4b9d363b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84c68f0 0x5585c84dbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.756+0000 7f4b9d363b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84cbbc0 0x5585c84cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.756+0000 7f4b9d363b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c1ad660 con 0x5585c84cbbc0 2024-05-10T11:29:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.756+0000 7f4b9d363b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c1af200 con 0x5585c843e6d0 2024-05-10T11:29:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.756+0000 7f4b9d363b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c1add30 con 0x5585c84c68f0 2024-05-10T11:29:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.756+0000 7f4b8b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84c68f0 0x5585c84dbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.756+0000 7f4b8b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84c68f0 0x5585c84dbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:44074/0 (socket says 172.21.15.27:44074) 2024-05-10T11:29:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.756+0000 7f4b8b400700 1 -- 172.21.15.27:0/4189165805 learned_addr learned my addr 172.21.15.27:0/4189165805 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.757+0000 7f4b90a00700 1 --2- 172.21.15.27:0/4189165805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84cbbc0 0x5585c84cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.756+0000 7f4b8be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c843e6d0 0x5585c84ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.757+0000 7f4b8b400700 1 -- 172.21.15.27:0/4189165805 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c843e6d0 msgr2=0x5585c84ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.757+0000 7f4b8b400700 1 --2- 172.21.15.27:0/4189165805 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c843e6d0 0x5585c84ce5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.757+0000 7f4b8b400700 1 -- 172.21.15.27:0/4189165805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84cbbc0 msgr2=0x5585c84cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.757+0000 7f4b8b400700 1 --2- 172.21.15.27:0/4189165805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84cbbc0 0x5585c84cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.757+0000 7f4b8b400700 1 -- 172.21.15.27:0/4189165805 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585c84c6f60 con 0x5585c84c68f0 2024-05-10T11:29:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.757+0000 7f4b8be00700 1 --2- 172.21.15.27:0/4189165805 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c843e6d0 0x5585c84ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:29:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.757+0000 7f4b8b400700 1 --2- 172.21.15.27:0/4189165805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84c68f0 0x5585c84dbfe0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4b780a96f0 tx=0x7f4b78011fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=605efe8af7b81ea1 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.757+0000 7f4b80c00700 1 -- 172.21.15.27:0/4189165805 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b780aa0f0 con 0x5585c84c68f0 2024-05-10T11:29:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.757+0000 7f4b80c00700 1 -- 172.21.15.27:0/4189165805 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b780a99f0 con 0x5585c84c68f0 2024-05-10T11:29:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.757+0000 7f4b80c00700 1 -- 172.21.15.27:0/4189165805 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b780b2620 con 0x5585c84c68f0 2024-05-10T11:29:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.758+0000 7f4b9d363b80 1 -- 172.21.15.27:0/4189165805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84c68f0 msgr2=0x5585c84dbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.758+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/4189165805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84c68f0 0x5585c84dbfe0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4b780a96f0 tx=0x7f4b78011fa0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.758+0000 7f4b9d363b80 1 -- 172.21.15.27:0/4189165805 shutdown_connections 2024-05-10T11:29:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.758+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/4189165805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84cbbc0 0x5585c84cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.758+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/4189165805 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c843e6d0 0x5585c84ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.758+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/4189165805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c84c68f0 0x5585c84dbfe0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.758+0000 7f4b9d363b80 1 -- 172.21.15.27:0/4189165805 >> 172.21.15.27:0/4189165805 conn(0x5585c84cf190 msgr2=0x5585c84ca740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.759+0000 7f4b9d363b80 1 -- 172.21.15.27:0/4189165805 shutdown_connections 2024-05-10T11:29:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.759+0000 7f4b9d363b80 1 -- 172.21.15.27:0/4189165805 wait complete. 2024-05-10T11:29:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.759+0000 7f4b9d363b80 1 -- start start 2024-05-10T11:29:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.759+0000 7f4b9d363b80 1 Processor -- start 2024-05-10T11:29:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.760+0000 7f4b9d363b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c843e6d0 0x5585c84f8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.760+0000 7f4b9d363b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 0x5585c84f8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.760+0000 7f4b9d363b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cbbc0 0x5585c84f8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.760+0000 7f4b9d363b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c1aead0 con 0x5585c84c68f0 2024-05-10T11:29:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.760+0000 7f4b9d363b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c1ae400 con 0x5585c84cbbc0 2024-05-10T11:29:51.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.760+0000 7f4b9d363b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c1ab620 con 0x5585c843e6d0 2024-05-10T11:29:51.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.760+0000 7f4b9d363b80 10 client.0 _ll_register_callbacks cb 0x5585c83bfc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:29:51.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.761+0000 7f4b90a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cbbc0 0x5585c84f8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.761+0000 7f4b90a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cbbc0 0x5585c84f8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50106/0 (socket says 172.21.15.27:50106) 2024-05-10T11:29:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.761+0000 7f4b90a00700 1 -- 172.21.15.27:0/1558548252 learned_addr learned my addr 172.21.15.27:0/1558548252 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:29:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.761+0000 7f4b8b400700 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 0x5585c84f8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.761+0000 7f4b8be00700 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c843e6d0 0x5585c84f8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.762+0000 7f4b8be00700 1 -- 172.21.15.27:0/1558548252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cbbc0 msgr2=0x5585c84f8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.762+0000 7f4b8be00700 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cbbc0 0x5585c84f8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.762+0000 7f4b8be00700 1 -- 172.21.15.27:0/1558548252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 msgr2=0x5585c84f8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:29:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.762+0000 7f4b8be00700 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 0x5585c84f8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.762+0000 7f4b8be00700 1 -- 172.21.15.27:0/1558548252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585c84d25a0 con 0x5585c843e6d0 2024-05-10T11:29:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.762+0000 7f4b8b400700 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 0x5585c84f8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:29:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.762+0000 7f4b8be00700 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c843e6d0 0x5585c84f8210 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x5585c84f3660 tx=0x7f4b7c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4e0262b3c56b5ec server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:29:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.762+0000 7f4b76000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:29:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.762+0000 7f4b76000700 1 -- 172.21.15.27:0/1558548252 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c003b10 con 0x5585c843e6d0 2024-05-10T11:29:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.763+0000 7f4b76000700 1 -- 172.21.15.27:0/1558548252 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9591+0+0 (secure 0 0 0) 0x7f4b7c003cb0 con 0x5585c843e6d0 2024-05-10T11:29:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.763+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1558548252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b780a99f0 con 0x5585c843e6d0 2024-05-10T11:29:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.763+0000 7f4b76000700 1 -- 172.21.15.27:0/1558548252 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f4b7c00f460 con 0x5585c843e6d0 2024-05-10T11:29:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.763+0000 7f4b9d363b80 10 client.27535 fetch_fsmap learned FSMap version 44 2024-05-10T11:29:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.763+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1558548252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5585c855b680 con 0x5585c843e6d0 2024-05-10T11:29:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.764+0000 7f4b76000700 1 -- 172.21.15.27:0/1558548252 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b780a99f0 con 0x5585c843e6d0 2024-05-10T11:29:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.764+0000 7f4b76000700 1 -- 172.21.15.27:0/1558548252 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 44) ==== 8370+0+0 (secure 0 0 0) 0x7f4b7c025440 con 0x5585c843e6d0 2024-05-10T11:29:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.764+0000 7f4b9d363b80 10 client.27535 fetch_fsmap finished waiting for FSMap version 44 2024-05-10T11:29:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.764+0000 7f4b9d363b80 10 client.27535 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:29:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.764+0000 7f4b9d363b80 20 client.27535 populate_metadata read hostname 'smithi027' 2024-05-10T11:29:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.764+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5585c84cc4e0 0x5585c855de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:29:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.764+0000 7f4b9d363b80 4 client.27535 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:29:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.764+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1558548252 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5585c8426400 con 0x5585c84cc4e0 2024-05-10T11:29:51.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.764+0000 7f4b8b400700 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5585c84cc4e0 0x5585c855de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:29:51.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.765+0000 7f4b76000700 10 client.27535 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:51.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.766+0000 7f4b8b400700 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5585c84cc4e0 0x5585c855de20 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c0c9662ec9b41d server_cookie=b4b87335204832e2 in_seq=0 out_seq=0 2024-05-10T11:29:51.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b76000700 1 -- 172.21.15.27:0/1558548252 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5585c8426400 con 0x5585c84cc4e0 2024-05-10T11:29:51.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b76000700 10 client.27535 handle_command_reply: tid=0 2024-05-10T11:29:51.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b9d363b80 1 client.27535 shutdown 2024-05-10T11:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1558548252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c843e6d0 msgr2=0x5585c84f8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c843e6d0 0x5585c84f8210 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x5585c84f3660 tx=0x7f4b7c009310 comp rx=0 tx=0).stop 2024-05-10T11:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1558548252 shutdown_connections 2024-05-10T11:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5585c84cc4e0 0x5585c855de20 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585c843e6d0 0x5585c84f8210 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585c84cbbc0 0x5585c84f8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b9d363b80 1 --2- 172.21.15.27:0/1558548252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585c84c68f0 0x5585c84f8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1558548252 >> 172.21.15.27:0/1558548252 conn(0x5585c84cf190 msgr2=0x5585c84cafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b76000700 0 client.27535 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.770+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1558548252 shutdown_connections 2024-05-10T11:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.771+0000 7f4b9d363b80 1 -- 172.21.15.27:0/1558548252 wait complete. 2024-05-10T11:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.771+0000 7f4b9d363b80 20 client.27535 trim_cache size 0 max 16384 2024-05-10T11:29:51.772 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:29:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:29:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:29:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "df3972d2-4611-43ba-ad31-582245af93e1": { 2024-05-10T11:29:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:29:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "df3972d2-4611-43ba-ad31-582245af93e1", 2024-05-10T11:29:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:29:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:29:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:29:51.773 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.771+0000 7f4b92a00700 1 -- 172.21.15.27:0/1290610979 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4b700b2a70 msgr2=0x7f4b700b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.771+0000 7f4b92a00700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4b700b2a70 0x7f4b700b4f30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x5585c84cc8e0 tx=0x7f4b7c00b280 comp rx=0 tx=0).stop 2024-05-10T11:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.771+0000 7f4b92a00700 1 -- 172.21.15.27:0/1290610979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b8c106050 msgr2=0x7f4b8c19c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.771+0000 7f4b92a00700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b8c106050 0x7f4b8c19c2d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800e9a0 tx=0x7f4b7800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.772+0000 7f4b92a00700 1 -- 172.21.15.27:0/1290610979 shutdown_connections 2024-05-10T11:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.772+0000 7f4b92a00700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f4b700b2a70 0x7f4b700b4f30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.772+0000 7f4b92a00700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b8c105690 0x7f4b8c19bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.772+0000 7f4b92a00700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b8c10b690 0x7f4b8c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.772+0000 7f4b92a00700 1 --2- 172.21.15.27:0/1290610979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b8c106050 0x7f4b8c19c2d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.772+0000 7f4b92a00700 1 -- 172.21.15.27:0/1290610979 >> 172.21.15.27:0/1290610979 conn(0x7f4b8c101180 msgr2=0x7f4b8c1018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.772+0000 7f4b92a00700 1 -- 172.21.15.27:0/1290610979 shutdown_connections 2024-05-10T11:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:29:51.772+0000 7f4b92a00700 1 -- 172.21.15.27:0/1290610979 wait complete. 2024-05-10T11:29:52.153 INFO:tasks.workunit.client.0.smithi027.stdout: 10 68668 23.54 MB/sec execute 36 sec latency 85.846 ms 2024-05-10T11:29:52.991 INFO:tasks.cephfs.filesystem:scrub status for tag:df3972d2-4611-43ba-ad31-582245af93e1 - {'path': '/', 'tag': 'df3972d2-4611-43ba-ad31-582245af93e1', 'options': 'recursive,force'} 2024-05-10T11:29:53.154 INFO:tasks.workunit.client.0.smithi027.stdout: 10 69161 23.74 MB/sec execute 37 sec latency 92.647 ms 2024-05-10T11:29:53.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:53 smithi027 ceph-mon[30054]: pgmap v522: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.0 MiB/s wr, 386 op/s 2024-05-10T11:29:53.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:53 smithi027 ceph-mon[30054]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:53.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:53 smithi076 ceph-mon[34152]: pgmap v522: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.0 MiB/s wr, 386 op/s 2024-05-10T11:29:53.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:53 smithi076 ceph-mon[34152]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:53.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:53 smithi189 ceph-mon[34842]: pgmap v522: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.0 MiB/s wr, 386 op/s 2024-05-10T11:29:53.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:53 smithi189 ceph-mon[34842]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-10T11:29:54.154 INFO:tasks.workunit.client.0.smithi027.stdout: 10 69584 23.87 MB/sec execute 38 sec latency 183.054 ms 2024-05-10T11:29:55.156 INFO:tasks.workunit.client.0.smithi027.stdout: 10 69848 23.60 MB/sec execute 39 sec latency 213.698 ms 2024-05-10T11:29:55.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:55 smithi076 ceph-mon[34152]: pgmap v523: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.0 MiB/s wr, 386 op/s 2024-05-10T11:29:55.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:55 smithi189 ceph-mon[34842]: pgmap v523: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.0 MiB/s wr, 386 op/s 2024-05-10T11:29:55.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:55 smithi027 ceph-mon[30054]: pgmap v523: 129 pgs: 129 active+clean; 1.0 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.0 MiB/s wr, 386 op/s 2024-05-10T11:29:56.156 INFO:tasks.workunit.client.0.smithi027.stdout: 10 70274 23.47 MB/sec execute 40 sec latency 133.225 ms 2024-05-10T11:29:57.156 INFO:tasks.workunit.client.0.smithi027.stdout: 10 70653 23.43 MB/sec execute 41 sec latency 417.887 ms 2024-05-10T11:29:57.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:57 smithi076 ceph-mon[34152]: pgmap v524: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 9.8 MiB/s wr, 572 op/s 2024-05-10T11:29:57.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:57 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:29:57.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:57 smithi189 ceph-mon[34842]: pgmap v524: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 9.8 MiB/s wr, 572 op/s 2024-05-10T11:29:57.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:57 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:29:57.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:57 smithi027 ceph-mon[30054]: pgmap v524: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 9.8 MiB/s wr, 572 op/s 2024-05-10T11:29:57.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:57 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:29:58.156 INFO:tasks.workunit.client.0.smithi027.stdout: 10 71160 23.40 MB/sec execute 42 sec latency 178.567 ms 2024-05-10T11:29:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:58 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:29:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:58 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:29:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:58 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:29:58.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:58 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:29:58.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:58 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:29:58.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:58 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:29:58.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:58 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:29:58.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:58 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:29:58.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:58 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:29:58.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:58 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:29:58.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:58 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:29:58.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:58 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:29:59.156 INFO:tasks.workunit.client.0.smithi027.stdout: 10 71564 23.38 MB/sec execute 43 sec latency 233.932 ms 2024-05-10T11:29:59.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:29:59 smithi076 ceph-mon[34152]: pgmap v525: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.8 MiB/s wr, 393 op/s 2024-05-10T11:29:59.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:29:59 smithi189 ceph-mon[34842]: pgmap v525: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.8 MiB/s wr, 393 op/s 2024-05-10T11:29:59.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:29:59 smithi027 ceph-mon[30054]: pgmap v525: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.8 MiB/s wr, 393 op/s 2024-05-10T11:30:00.161 INFO:tasks.workunit.client.0.smithi027.stdout: 10 71939 23.22 MB/sec execute 44 sec latency 242.508 ms 2024-05-10T11:30:00.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:00 smithi076 ceph-mon[34152]: overall HEALTH_OK 2024-05-10T11:30:00.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:00 smithi189 ceph-mon[34842]: overall HEALTH_OK 2024-05-10T11:30:00.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:00 smithi027 ceph-mon[30054]: overall HEALTH_OK 2024-05-10T11:30:01.159 INFO:tasks.workunit.client.0.smithi027.stdout: 10 72353 23.25 MB/sec execute 45 sec latency 284.528 ms 2024-05-10T11:30:01.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:01 smithi076 ceph-mon[34152]: pgmap v526: 129 pgs: 129 active+clean; 999 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.4 MiB/s wr, 538 op/s 2024-05-10T11:30:01.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:01 smithi189 ceph-mon[34842]: pgmap v526: 129 pgs: 129 active+clean; 999 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.4 MiB/s wr, 538 op/s 2024-05-10T11:30:01.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:01 smithi027 ceph-mon[30054]: pgmap v526: 129 pgs: 129 active+clean; 999 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.4 MiB/s wr, 538 op/s 2024-05-10T11:30:02.160 INFO:tasks.workunit.client.0.smithi027.stdout: 10 72710 23.06 MB/sec execute 46 sec latency 210.171 ms 2024-05-10T11:30:03.160 INFO:tasks.workunit.client.0.smithi027.stdout: 10 72911 22.87 MB/sec execute 47 sec latency 375.211 ms 2024-05-10T11:30:03.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:03 smithi076 ceph-mon[34152]: pgmap v527: 129 pgs: 129 active+clean; 990 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.1 MiB/s wr, 383 op/s 2024-05-10T11:30:03.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:03 smithi076 ceph-mon[34152]: osdmap e112: 12 total, 12 up, 12 in 2024-05-10T11:30:03.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:03 smithi189 ceph-mon[34842]: pgmap v527: 129 pgs: 129 active+clean; 990 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.1 MiB/s wr, 383 op/s 2024-05-10T11:30:03.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:03 smithi189 ceph-mon[34842]: osdmap e112: 12 total, 12 up, 12 in 2024-05-10T11:30:03.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:03 smithi027 ceph-mon[30054]: pgmap v527: 129 pgs: 129 active+clean; 990 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.1 MiB/s wr, 383 op/s 2024-05-10T11:30:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:03 smithi027 ceph-mon[30054]: osdmap e112: 12 total, 12 up, 12 in 2024-05-10T11:30:04.160 INFO:tasks.workunit.client.0.smithi027.stdout: 10 73168 22.66 MB/sec execute 48 sec latency 322.246 ms 2024-05-10T11:30:05.160 INFO:tasks.workunit.client.0.smithi027.stdout: 10 73325 22.31 MB/sec execute 49 sec latency 512.263 ms 2024-05-10T11:30:05.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:05 smithi189 ceph-mon[34842]: pgmap v529: 129 pgs: 129 active+clean; 990 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 8.5 MiB/s wr, 460 op/s 2024-05-10T11:30:05.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:05 smithi027 ceph-mon[30054]: pgmap v529: 129 pgs: 129 active+clean; 990 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 8.5 MiB/s wr, 460 op/s 2024-05-10T11:30:05.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:05 smithi076 ceph-mon[34152]: pgmap v529: 129 pgs: 129 active+clean; 990 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 8.5 MiB/s wr, 460 op/s 2024-05-10T11:30:06.161 INFO:tasks.workunit.client.0.smithi027.stdout: 10 73713 22.44 MB/sec execute 50 sec latency 184.398 ms 2024-05-10T11:30:07.161 INFO:tasks.workunit.client.0.smithi027.stdout: 10 74384 22.82 MB/sec execute 51 sec latency 107.594 ms 2024-05-10T11:30:07.454 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:07 smithi027 ceph-mon[30054]: pgmap v530: 129 pgs: 129 active+clean; 1004 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.5 MiB/s wr, 574 op/s 2024-05-10T11:30:07.454 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:07 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:30:07.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:07 smithi189 ceph-mon[34842]: pgmap v530: 129 pgs: 129 active+clean; 1004 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.5 MiB/s wr, 574 op/s 2024-05-10T11:30:07.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:07 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:30:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:07 smithi076 ceph-mon[34152]: pgmap v530: 129 pgs: 129 active+clean; 1004 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.5 MiB/s wr, 574 op/s 2024-05-10T11:30:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:07 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:30:08.161 INFO:tasks.workunit.client.0.smithi027.stdout: 10 74830 22.82 MB/sec execute 52 sec latency 130.265 ms 2024-05-10T11:30:09.161 INFO:tasks.workunit.client.0.smithi027.stdout: 10 75225 22.89 MB/sec execute 53 sec latency 88.617 ms 2024-05-10T11:30:09.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:09 smithi189 ceph-mon[34842]: pgmap v531: 129 pgs: 129 active+clean; 1004 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.5 MiB/s wr, 574 op/s 2024-05-10T11:30:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:09 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:30:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:09 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:30:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:09 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:30:09.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:09 smithi189 ceph-mon[34842]: osdmap e113: 12 total, 12 up, 12 in 2024-05-10T11:30:09.534 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:09 smithi027 ceph-mon[30054]: pgmap v531: 129 pgs: 129 active+clean; 1004 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.5 MiB/s wr, 574 op/s 2024-05-10T11:30:09.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:09 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:30:09.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:09 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:30:09.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:09 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:30:09.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:09 smithi027 ceph-mon[30054]: osdmap e113: 12 total, 12 up, 12 in 2024-05-10T11:30:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:09 smithi076 ceph-mon[34152]: pgmap v531: 129 pgs: 129 active+clean; 1004 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.5 MiB/s wr, 574 op/s 2024-05-10T11:30:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:09 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:30:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:09 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:30:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:09 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:30:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:09 smithi076 ceph-mon[34152]: osdmap e113: 12 total, 12 up, 12 in 2024-05-10T11:30:10.161 INFO:tasks.workunit.client.0.smithi027.stdout: 10 75641 22.88 MB/sec execute 54 sec latency 221.635 ms 2024-05-10T11:30:11.161 INFO:tasks.workunit.client.0.smithi027.stdout: 10 75919 22.68 MB/sec execute 55 sec latency 226.806 ms 2024-05-10T11:30:11.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:11 smithi189 ceph-mon[34842]: pgmap v533: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 11 MiB/s wr, 694 op/s 2024-05-10T11:30:11.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:11 smithi027 ceph-mon[30054]: pgmap v533: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 11 MiB/s wr, 694 op/s 2024-05-10T11:30:11.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:11 smithi076 ceph-mon[34152]: pgmap v533: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 11 MiB/s wr, 694 op/s 2024-05-10T11:30:12.161 INFO:tasks.workunit.client.0.smithi027.stdout: 10 76231 22.53 MB/sec execute 56 sec latency 133.837 ms 2024-05-10T11:30:13.162 INFO:tasks.workunit.client.0.smithi027.stdout: 10 76721 22.54 MB/sec execute 57 sec latency 141.833 ms 2024-05-10T11:30:13.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:13 smithi189 ceph-mon[34842]: pgmap v534: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 10 MiB/s wr, 633 op/s 2024-05-10T11:30:13.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:13 smithi027 ceph-mon[30054]: pgmap v534: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 10 MiB/s wr, 633 op/s 2024-05-10T11:30:13.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:13 smithi076 ceph-mon[34152]: pgmap v534: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 10 MiB/s wr, 633 op/s 2024-05-10T11:30:14.161 INFO:tasks.workunit.client.0.smithi027.stdout: 10 77251 22.60 MB/sec execute 58 sec latency 227.842 ms 2024-05-10T11:30:15.161 INFO:tasks.workunit.client.0.smithi027.stdout: 10 77572 22.47 MB/sec execute 59 sec latency 231.688 ms 2024-05-10T11:30:15.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:15 smithi189 ceph-mon[34842]: pgmap v535: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 9.6 MiB/s wr, 599 op/s 2024-05-10T11:30:15.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:15 smithi027 ceph-mon[30054]: pgmap v535: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 9.6 MiB/s wr, 599 op/s 2024-05-10T11:30:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:15 smithi076 ceph-mon[34152]: pgmap v535: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 9.6 MiB/s wr, 599 op/s 2024-05-10T11:30:16.161 INFO:tasks.workunit.client.0.smithi027.stdout: 10 77942 22.47 MB/sec execute 60 sec latency 283.160 ms 2024-05-10T11:30:17.161 INFO:tasks.workunit.client.0.smithi027.stdout: 10 78438 22.61 MB/sec execute 61 sec latency 284.193 ms 2024-05-10T11:30:17.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:17 smithi189 ceph-mon[34842]: pgmap v536: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 9.6 MiB/s wr, 487 op/s 2024-05-10T11:30:17.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:17 smithi027 ceph-mon[30054]: pgmap v536: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 9.6 MiB/s wr, 487 op/s 2024-05-10T11:30:17.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:17 smithi076 ceph-mon[34152]: pgmap v536: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 9.6 MiB/s wr, 487 op/s 2024-05-10T11:30:18.162 INFO:tasks.workunit.client.0.smithi027.stdout: 10 78973 22.71 MB/sec execute 62 sec latency 170.967 ms 2024-05-10T11:30:19.162 INFO:tasks.workunit.client.0.smithi027.stdout: 10 79512 22.79 MB/sec execute 63 sec latency 166.712 ms 2024-05-10T11:30:19.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:19 smithi189 ceph-mon[34842]: pgmap v537: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 9.6 MiB/s wr, 486 op/s 2024-05-10T11:30:19.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:19 smithi027 ceph-mon[30054]: pgmap v537: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 9.6 MiB/s wr, 486 op/s 2024-05-10T11:30:19.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:19 smithi076 ceph-mon[34152]: pgmap v537: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 9.6 MiB/s wr, 486 op/s 2024-05-10T11:30:20.162 INFO:tasks.workunit.client.0.smithi027.stdout: 10 79893 22.79 MB/sec execute 64 sec latency 192.266 ms 2024-05-10T11:30:21.162 INFO:tasks.workunit.client.0.smithi027.stdout: 10 80223 22.75 MB/sec execute 65 sec latency 289.256 ms 2024-05-10T11:30:21.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:21 smithi189 ceph-mon[34842]: pgmap v538: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 13 MiB/s wr, 626 op/s 2024-05-10T11:30:21.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:21 smithi027 ceph-mon[30054]: pgmap v538: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 13 MiB/s wr, 626 op/s 2024-05-10T11:30:21.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:21 smithi076 ceph-mon[34152]: pgmap v538: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 13 MiB/s wr, 626 op/s 2024-05-10T11:30:22.162 INFO:tasks.workunit.client.0.smithi027.stdout: 10 80543 22.66 MB/sec execute 66 sec latency 154.285 ms 2024-05-10T11:30:22.992 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:30:23.162 INFO:tasks.workunit.client.0.smithi027.stdout: 10 80956 22.57 MB/sec execute 67 sec latency 195.624 ms 2024-05-10T11:30:23.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:23 smithi027 ceph-mon[30054]: pgmap v539: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 9.0 MiB/s wr, 441 op/s 2024-05-10T11:30:23.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:23 smithi189 ceph-mon[34842]: pgmap v539: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 9.0 MiB/s wr, 441 op/s 2024-05-10T11:30:23.523 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:30:23.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:23 smithi076 ceph-mon[34152]: pgmap v539: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 9.0 MiB/s wr, 441 op/s 2024-05-10T11:30:24.162 INFO:tasks.workunit.client.0.smithi027.stdout: 10 81570 22.69 MB/sec execute 68 sec latency 105.903 ms 2024-05-10T11:30:25.162 INFO:tasks.workunit.client.0.smithi027.stdout: 10 82120 22.79 MB/sec execute 69 sec latency 86.555 ms 2024-05-10T11:30:25.599 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:25 smithi027 ceph-mon[30054]: pgmap v540: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 8.4 MiB/s wr, 403 op/s 2024-05-10T11:30:25.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:25 smithi076 ceph-mon[34152]: pgmap v540: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 8.4 MiB/s wr, 403 op/s 2024-05-10T11:30:25.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:25 smithi189 ceph-mon[34842]: pgmap v540: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 8.4 MiB/s wr, 403 op/s 2024-05-10T11:30:26.162 INFO:tasks.workunit.client.0.smithi027.stdout: 10 82402 22.72 MB/sec execute 70 sec latency 207.796 ms 2024-05-10T11:30:26.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.274+0000 7fa136200700 1 -- 172.21.15.27:0/4035796879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa13006b180 msgr2=0x7fa13006b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:26.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.275+0000 7fa136200700 1 --2- 172.21.15.27:0/4035796879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa13006b180 0x7fa13006b600 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa12000dd20 tx=0x7fa12000f330 comp rx=0 tx=0).stop 2024-05-10T11:30:26.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.276+0000 7fa136200700 1 -- 172.21.15.27:0/4035796879 shutdown_connections 2024-05-10T11:30:26.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.277+0000 7fa136200700 1 --2- 172.21.15.27:0/4035796879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa13006bb40 0x7fa130110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.277+0000 7fa136200700 1 --2- 172.21.15.27:0/4035796879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa130109510 0x7fa1301098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.277+0000 7fa136200700 1 --2- 172.21.15.27:0/4035796879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa13006b180 0x7fa13006b600 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.277+0000 7fa136200700 1 -- 172.21.15.27:0/4035796879 >> 172.21.15.27:0/4035796879 conn(0x7fa130079350 msgr2=0x7fa130079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.278+0000 7fa136200700 1 -- 172.21.15.27:0/4035796879 shutdown_connections 2024-05-10T11:30:26.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.278+0000 7fa136200700 1 -- 172.21.15.27:0/4035796879 wait complete. 2024-05-10T11:30:26.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.280+0000 7fa136200700 1 Processor -- start 2024-05-10T11:30:26.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.281+0000 7fa136200700 1 -- start start 2024-05-10T11:30:26.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.281+0000 7fa136200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa13006b180 0x7fa13019d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.281+0000 7fa136200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa13006bb40 0x7fa13019dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.281+0000 7fa136200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa130109510 0x7fa130197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.281+0000 7fa136200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa1301114f0 con 0x7fa13006bb40 2024-05-10T11:30:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.281+0000 7fa136200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa130111370 con 0x7fa130109510 2024-05-10T11:30:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.281+0000 7fa136200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa130111670 con 0x7fa13006b180 2024-05-10T11:30:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.282+0000 7fa12f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa13006b180 0x7fa13019d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.282+0000 7fa12f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa13006b180 0x7fa13019d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55202/0 (socket says 172.21.15.27:55202) 2024-05-10T11:30:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.282+0000 7fa12f400700 1 -- 172.21.15.27:0/3728943643 learned_addr learned my addr 172.21.15.27:0/3728943643 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.282+0000 7fa12fe00700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa130109510 0x7fa130197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.283+0000 7fa12f400700 1 -- 172.21.15.27:0/3728943643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa130109510 msgr2=0x7fa130197870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.283+0000 7fa12f400700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa130109510 0x7fa130197870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.283+0000 7fa12f400700 1 -- 172.21.15.27:0/3728943643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa13006bb40 msgr2=0x7fa13019dbf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T11:30:26.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.285+0000 7fa12ea00700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa13006bb40 0x7fa13019dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.285+0000 7fa12f400700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa13006bb40 0x7fa13019dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.285+0000 7fa12f400700 1 -- 172.21.15.27:0/3728943643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa130197fd0 con 0x7fa13006b180 2024-05-10T11:30:26.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.287+0000 7fa12f400700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa13006b180 0x7fa13019d6b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00d950 tx=0x7fa11c00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.289+0000 7fa127e00700 1 -- 172.21.15.27:0/3728943643 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c016070 con 0x7fa13006b180 2024-05-10T11:30:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.289+0000 7fa127e00700 1 -- 172.21.15.27:0/3728943643 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11c009df0 con 0x7fa13006b180 2024-05-10T11:30:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.289+0000 7fa136200700 1 -- 172.21.15.27:0/3728943643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1301982c0 con 0x7fa13006b180 2024-05-10T11:30:26.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.290+0000 7fa136200700 1 -- 172.21.15.27:0/3728943643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1301a4580 con 0x7fa13006b180 2024-05-10T11:30:26.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.290+0000 7fa127e00700 1 -- 172.21.15.27:0/3728943643 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c00f3c0 con 0x7fa13006b180 2024-05-10T11:30:26.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.309+0000 7fa140abeb80 1 Processor -- start 2024-05-10T11:30:26.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.311+0000 7fa140abeb80 1 -- start start 2024-05-10T11:30:26.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.312+0000 7fa140abeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1c9fbc0 0x5594a1c9ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.312+0000 7fa127e00700 1 -- 172.21.15.27:0/3728943643 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa11c019070 con 0x7fa13006b180 2024-05-10T11:30:26.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.314+0000 7fa140abeb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1c9a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.314+0000 7fa140abeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9a8f0 0x5594a1c9ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.314+0000 7fa140abeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5594a1beca20 con 0x5594a1c9fbc0 2024-05-10T11:30:26.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.314+0000 7fa140abeb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5594a1b8a760 con 0x5594a1ca04e0 2024-05-10T11:30:26.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.314+0000 7fa140abeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5594a1be89e0 con 0x5594a1c9a8f0 2024-05-10T11:30:26.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.314+0000 7fa12ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9a8f0 0x5594a1c9ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.314+0000 7fa12ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9a8f0 0x5594a1c9ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55210/0 (socket says 172.21.15.27:55210) 2024-05-10T11:30:26.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.314+0000 7fa12ea00700 1 -- 172.21.15.27:0/869139208 learned_addr learned my addr 172.21.15.27:0/869139208 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:26.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.315+0000 7fa12fe00700 1 --2- 172.21.15.27:0/869139208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1c9fbc0 0x5594a1c9ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.316+0000 7fa12ea00700 1 -- 172.21.15.27:0/869139208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 msgr2=0x5594a1c9a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:26.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.317+0000 7fa127e00700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa1140b2af0 0x7fa1140b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.317+0000 7fa127e00700 1 -- 172.21.15.27:0/3728943643 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7fa11c0e2560 con 0x7fa13006b180 2024-05-10T11:30:26.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.317+0000 7fa12f400700 1 --2- 172.21.15.27:0/869139208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1c9a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.317+0000 7fa12ea00700 1 --2- 172.21.15.27:0/869139208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1c9a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.317+0000 7fa12ea00700 1 -- 172.21.15.27:0/869139208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1c9fbc0 msgr2=0x5594a1c9ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:26.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.317+0000 7fa12ea00700 1 --2- 172.21.15.27:0/869139208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1c9fbc0 0x5594a1c9ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.317+0000 7fa12ea00700 1 -- 172.21.15.27:0/869139208 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594a1b398d0 con 0x5594a1c9a8f0 2024-05-10T11:30:26.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.317+0000 7fa12fe00700 1 --2- 172.21.15.27:0/869139208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1c9fbc0 0x5594a1c9ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:30:26.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.318+0000 7fa12fe00700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa1140b2af0 0x7fa1140b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.318+0000 7fa12f400700 1 --2- 172.21.15.27:0/869139208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1c9a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:30:26.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.319+0000 7fa12ea00700 1 --2- 172.21.15.27:0/869139208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9a8f0 0x5594a1c9ad70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa120005b40 tx=0x7fa120003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d45ded1af37ec521 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:26.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.320+0000 7fa124c00700 1 -- 172.21.15.27:0/869139208 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12001b030 con 0x5594a1c9a8f0 2024-05-10T11:30:26.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.320+0000 7fa124c00700 1 -- 172.21.15.27:0/869139208 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa12001d070 con 0x5594a1c9a8f0 2024-05-10T11:30:26.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.321+0000 7fa140abeb80 1 -- 172.21.15.27:0/869139208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9a8f0 msgr2=0x5594a1c9ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.321+0000 7fa12fe00700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa1140b2af0 0x7fa1140b4fb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa128009ec0 tx=0x7fa128009040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.321+0000 7fa140abeb80 1 --2- 172.21.15.27:0/869139208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9a8f0 0x5594a1c9ad70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa120005b40 tx=0x7fa120003680 comp rx=0 tx=0).stop 2024-05-10T11:30:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.322+0000 7fa140abeb80 1 -- 172.21.15.27:0/869139208 shutdown_connections 2024-05-10T11:30:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.322+0000 7fa140abeb80 1 --2- 172.21.15.27:0/869139208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9a8f0 0x5594a1c9ad70 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.322+0000 7fa140abeb80 1 --2- 172.21.15.27:0/869139208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1c9fbc0 0x5594a1c9ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.322+0000 7fa140abeb80 1 --2- 172.21.15.27:0/869139208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1c9a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.322+0000 7fa140abeb80 1 -- 172.21.15.27:0/869139208 >> 172.21.15.27:0/869139208 conn(0x5594a1c126d0 msgr2=0x5594a1c94b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:26.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.323+0000 7fa140abeb80 1 -- 172.21.15.27:0/869139208 shutdown_connections 2024-05-10T11:30:26.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.323+0000 7fa140abeb80 1 -- 172.21.15.27:0/869139208 wait complete. 2024-05-10T11:30:26.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.324+0000 7fa140abeb80 1 -- start start 2024-05-10T11:30:26.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.325+0000 7fa140abeb80 1 Processor -- start 2024-05-10T11:30:26.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.325+0000 7fa140abeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1ca5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.326+0000 7fa140abeb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1ca6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.326+0000 7fa140abeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 0x5594a1ca3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.326+0000 7fa12f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1ca5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.326+0000 7fa12f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1ca5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55224/0 (socket says 172.21.15.27:55224) 2024-05-10T11:30:26.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.326+0000 7fa12fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 0x5594a1ca3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.327+0000 7fa12ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1ca6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.327+0000 7fa12ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1ca6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44928/0 (socket says 172.21.15.27:44928) 2024-05-10T11:30:26.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.327+0000 7fa12fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 0x5594a1ca3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33500/0 (socket says 172.21.15.27:33500) 2024-05-10T11:30:26.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.327+0000 7fa12f400700 1 -- 172.21.15.27:0/3826741466 learned_addr learned my addr 172.21.15.27:0/3826741466 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:26.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.327+0000 7fa140abeb80 1 -- 172.21.15.27:0/3826741466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5594a1ca6f60 con 0x5594a1ca30f0 2024-05-10T11:30:26.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.327+0000 7fa140abeb80 1 -- 172.21.15.27:0/3826741466 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5594a1b52b20 con 0x5594a1ca04e0 2024-05-10T11:30:26.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.327+0000 7fa140abeb80 1 -- 172.21.15.27:0/3826741466 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5594a1ca70e0 con 0x5594a1c9fbc0 2024-05-10T11:30:26.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.327+0000 7fa12f400700 1 -- 172.21.15.27:0/3826741466 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 msgr2=0x5594a1ca6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:26.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.327+0000 7fa12f400700 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1ca6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.327+0000 7fa12f400700 1 -- 172.21.15.27:0/3826741466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 msgr2=0x5594a1ca3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:26.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.327+0000 7fa12f400700 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 0x5594a1ca3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.328+0000 7fa12f400700 1 -- 172.21.15.27:0/3826741466 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594a1bb2420 con 0x5594a1c9fbc0 2024-05-10T11:30:26.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.328+0000 7fa12fe00700 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 0x5594a1ca3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:30:26.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.328+0000 7fa12ea00700 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1ca6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:30:26.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.328+0000 7fa140abeb80 10 client.0 _ll_register_callbacks cb 0x5594a1b93c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:30:26.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.328+0000 7fa140abeb80 1 -- 172.21.15.27:0/3826741466 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5594a1c67140 con 0x5594a1c9fbc0 2024-05-10T11:30:26.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.329+0000 7fa12f400700 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1ca5d10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00b2f0 tx=0x7fa11c0a9be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f840411ea402e3f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:26.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.329+0000 7fa119600700 10 client.27547 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:30:26.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.329+0000 7fa119600700 1 -- 172.21.15.27:0/3826741466 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c020040 con 0x5594a1c9fbc0 2024-05-10T11:30:26.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.331+0000 7fa119600700 1 -- 172.21.15.27:0/3826741466 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7fa11c010710 con 0x5594a1c9fbc0 2024-05-10T11:30:26.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.331+0000 7fa119600700 1 -- 172.21.15.27:0/3826741466 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fa11c0b2210 con 0x5594a1c9fbc0 2024-05-10T11:30:26.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.333+0000 7fa140abeb80 10 client.27547 fetch_fsmap learned FSMap version 45 2024-05-10T11:30:26.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.333+0000 7fa140abeb80 1 -- 172.21.15.27:0/3826741466 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=45}) -- 0x5594a1cb1770 con 0x5594a1c9fbc0 2024-05-10T11:30:26.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.333+0000 7fa119600700 1 -- 172.21.15.27:0/3826741466 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7fa11c0a9610 con 0x5594a1c9fbc0 2024-05-10T11:30:26.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.333+0000 7fa140abeb80 10 client.27547 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:30:26.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.333+0000 7fa140abeb80 10 client.27547 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:30:26.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.333+0000 7fa140abeb80 20 client.27547 populate_metadata read hostname 'smithi027' 2024-05-10T11:30:26.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.333+0000 7fa140abeb80 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5594a1cb3ca0 0x5594a1cb6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.333+0000 7fa140abeb80 4 client.27547 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:30:26.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.333+0000 7fa140abeb80 1 -- 172.21.15.27:0/3826741466 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594a1bfa400 con 0x5594a1cb3ca0 2024-05-10T11:30:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.335+0000 7fa12ea00700 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5594a1cb3ca0 0x5594a1cb6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.340+0000 7fa119600700 10 client.27547 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:30:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.340+0000 7fa12ea00700 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5594a1cb3ca0 0x5594a1cb6160 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=146af0809fff346 server_cookie=a410a176ef71053b in_seq=0 out_seq=0 2024-05-10T11:30:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.346+0000 7fa119600700 1 -- 172.21.15.27:0/3826741466 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5594a1bfa400 con 0x5594a1cb3ca0 2024-05-10T11:30:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.346+0000 7fa119600700 10 client.27547 handle_command_reply: tid=0 2024-05-10T11:30:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.346+0000 7fa140abeb80 1 client.27547 shutdown 2024-05-10T11:30:26.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.346+0000 7fa140abeb80 1 -- 172.21.15.27:0/3826741466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 msgr2=0x5594a1ca5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:26.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.346+0000 7fa140abeb80 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1ca5d10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00b2f0 tx=0x7fa11c0a9be0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.348+0000 7fa140abeb80 1 -- 172.21.15.27:0/3826741466 shutdown_connections 2024-05-10T11:30:26.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.348+0000 7fa140abeb80 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5594a1cb3ca0 0x5594a1cb6160 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.348+0000 7fa140abeb80 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1ca5d10 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.348+0000 7fa140abeb80 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 0x5594a1ca3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.348+0000 7fa140abeb80 1 --2- 172.21.15.27:0/3826741466 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1ca6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.348+0000 7fa119600700 0 client.27547 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:30:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.348+0000 7fa140abeb80 1 -- 172.21.15.27:0/3826741466 >> 172.21.15.27:0/3826741466 conn(0x5594a1c126d0 msgr2=0x5594a1c94190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.348+0000 7fa140abeb80 1 -- 172.21.15.27:0/3826741466 shutdown_connections 2024-05-10T11:30:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.349+0000 7fa140abeb80 1 -- 172.21.15.27:0/3826741466 wait complete. 2024-05-10T11:30:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.349+0000 7fa140abeb80 20 client.27547 trim_cache size 0 max 16384 2024-05-10T11:30:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.403+0000 7fa140abeb80 1 Processor -- start 2024-05-10T11:30:26.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.403+0000 7fa140abeb80 1 -- start start 2024-05-10T11:30:26.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.403+0000 7fa140abeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1cb00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.403+0000 7fa140abeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca04e0 0x5594a1ca22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.404+0000 7fa140abeb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca30f0 0x5594a1c94910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.405+0000 7fa140abeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa11000ef80 con 0x5594a1ca04e0 2024-05-10T11:30:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.405+0000 7fa140abeb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa110010ac0 con 0x5594a1ca30f0 2024-05-10T11:30:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.405+0000 7fa140abeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa11000f650 con 0x5594a1c9fbc0 2024-05-10T11:30:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.405+0000 7fa12fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca30f0 0x5594a1c94910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.405+0000 7fa12f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1cb00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.405+0000 7fa12f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1cb00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55232/0 (socket says 172.21.15.27:55232) 2024-05-10T11:30:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.405+0000 7fa12f400700 1 -- 172.21.15.27:0/755060634 learned_addr learned my addr 172.21.15.27:0/755060634 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.406+0000 7fa12ea00700 1 --2- 172.21.15.27:0/755060634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca04e0 0x5594a1ca22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.407+0000 7fa12ea00700 1 -- 172.21.15.27:0/755060634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 msgr2=0x5594a1cb00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.407+0000 7fa12ea00700 1 --2- 172.21.15.27:0/755060634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1cb00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.408+0000 7fa12ea00700 1 -- 172.21.15.27:0/755060634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca30f0 msgr2=0x5594a1c94910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.408+0000 7fa12ea00700 1 --2- 172.21.15.27:0/755060634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca30f0 0x5594a1c94910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.408+0000 7fa12ea00700 1 -- 172.21.15.27:0/755060634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594a1c95820 con 0x5594a1ca04e0 2024-05-10T11:30:26.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.408+0000 7fa12f400700 1 --2- 172.21.15.27:0/755060634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1cb00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:30:26.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.409+0000 7fa12ea00700 1 --2- 172.21.15.27:0/755060634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca04e0 0x5594a1ca22d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5594a1c8fc30 tx=0x7fa1200058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c08364eae67b7d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:26.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.410+0000 7fa11be00700 1 -- 172.21.15.27:0/755060634 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12001b040 con 0x5594a1ca04e0 2024-05-10T11:30:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.411+0000 7fa11be00700 1 -- 172.21.15.27:0/755060634 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa120003e30 con 0x5594a1ca04e0 2024-05-10T11:30:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.411+0000 7fa11be00700 1 -- 172.21.15.27:0/755060634 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa120005070 con 0x5594a1ca04e0 2024-05-10T11:30:26.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.413+0000 7fa140abeb80 1 -- 172.21.15.27:0/755060634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca04e0 msgr2=0x5594a1ca22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:26.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.413+0000 7fa140abeb80 1 --2- 172.21.15.27:0/755060634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca04e0 0x5594a1ca22d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5594a1c8fc30 tx=0x7fa1200058e0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.414+0000 7fa140abeb80 1 -- 172.21.15.27:0/755060634 shutdown_connections 2024-05-10T11:30:26.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.414+0000 7fa140abeb80 1 --2- 172.21.15.27:0/755060634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1cb00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.414+0000 7fa140abeb80 1 --2- 172.21.15.27:0/755060634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca30f0 0x5594a1c94910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.414+0000 7fa140abeb80 1 --2- 172.21.15.27:0/755060634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca04e0 0x5594a1ca22d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.414+0000 7fa140abeb80 1 -- 172.21.15.27:0/755060634 >> 172.21.15.27:0/755060634 conn(0x5594a1c126d0 msgr2=0x5594a1ca4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:26.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.415+0000 7fa140abeb80 1 -- 172.21.15.27:0/755060634 shutdown_connections 2024-05-10T11:30:26.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.415+0000 7fa140abeb80 1 -- 172.21.15.27:0/755060634 wait complete. 2024-05-10T11:30:26.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.416+0000 7fa140abeb80 1 -- start start 2024-05-10T11:30:26.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.416+0000 7fa140abeb80 1 Processor -- start 2024-05-10T11:30:26.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.418+0000 7fa140abeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1cacab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.418+0000 7fa140abeb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1cbb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.418+0000 7fa140abeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 0x5594a1cbb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.419+0000 7fa12ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1cbb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.419+0000 7fa12ea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1cbb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44944/0 (socket says 172.21.15.27:44944) 2024-05-10T11:30:26.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.419+0000 7fa12fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 0x5594a1cbb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.420+0000 7fa12fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 0x5594a1cbb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33520/0 (socket says 172.21.15.27:33520) 2024-05-10T11:30:26.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.420+0000 7fa140abeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa1100103f0 con 0x5594a1ca30f0 2024-05-10T11:30:26.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.420+0000 7fa140abeb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa11000fd20 con 0x5594a1ca04e0 2024-05-10T11:30:26.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.420+0000 7fa140abeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa11000cf40 con 0x5594a1c9fbc0 2024-05-10T11:30:26.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.421+0000 7fa12ea00700 1 -- 172.21.15.27:0/3881548506 learned_addr learned my addr 172.21.15.27:0/3881548506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:26.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.421+0000 7fa12f400700 1 --2- 172.21.15.27:0/3881548506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1cacab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.421+0000 7fa140abeb80 10 client.0 _ll_register_callbacks cb 0x5594a1b93c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:30:26.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.422+0000 7fa12ea00700 1 -- 172.21.15.27:0/3881548506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 msgr2=0x5594a1cacab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:26.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.422+0000 7fa12ea00700 1 --2- 172.21.15.27:0/3881548506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1cacab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.422+0000 7fa12ea00700 1 -- 172.21.15.27:0/3881548506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 msgr2=0x5594a1cbb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:26.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.422+0000 7fa12ea00700 1 --2- 172.21.15.27:0/3881548506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 0x5594a1cbb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.423+0000 7fa12ea00700 1 -- 172.21.15.27:0/3881548506 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594a1d1d630 con 0x5594a1ca04e0 2024-05-10T11:30:26.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.424+0000 7fa12ea00700 1 --2- 172.21.15.27:0/3881548506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1cbb090 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa12000dd50 tx=0x7fa12000d6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b745cb20b12163b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:26.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.424+0000 7fa119600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:30:26.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.424+0000 7fa119600700 1 -- 172.21.15.27:0/3881548506 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12001b040 con 0x5594a1ca04e0 2024-05-10T11:30:26.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.424+0000 7fa119600700 1 -- 172.21.15.27:0/3881548506 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7fa120003980 con 0x5594a1ca04e0 2024-05-10T11:30:26.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.424+0000 7fa140abeb80 1 -- 172.21.15.27:0/3881548506 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa120003e30 con 0x5594a1ca04e0 2024-05-10T11:30:26.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.425+0000 7fa119600700 1 -- 172.21.15.27:0/3881548506 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fa120003ba0 con 0x5594a1ca04e0 2024-05-10T11:30:26.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.425+0000 7fa140abeb80 10 client.27542 fetch_fsmap learned FSMap version 45 2024-05-10T11:30:26.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.425+0000 7fa140abeb80 1 -- 172.21.15.27:0/3881548506 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5594a1d1e6f0 con 0x5594a1ca04e0 2024-05-10T11:30:26.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.426+0000 7fa119600700 1 -- 172.21.15.27:0/3881548506 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa120003e30 con 0x5594a1ca04e0 2024-05-10T11:30:26.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.426+0000 7fa119600700 1 -- 172.21.15.27:0/3881548506 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7fa120015070 con 0x5594a1ca04e0 2024-05-10T11:30:26.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.426+0000 7fa140abeb80 10 client.27542 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:30:26.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.426+0000 7fa140abeb80 10 client.27542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:30:26.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.426+0000 7fa140abeb80 20 client.27542 populate_metadata read hostname 'smithi027' 2024-05-10T11:30:26.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.426+0000 7fa140abeb80 1 --2- 172.21.15.27:0/3881548506 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5594a1d1e970 0x5594a1d20d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:26.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.426+0000 7fa140abeb80 4 client.27542 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:30:26.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.426+0000 7fa140abeb80 1 -- 172.21.15.27:0/3881548506 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5594a1bf44a0 con 0x5594a1d1e970 2024-05-10T11:30:26.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.426+0000 7fa12f400700 1 --2- 172.21.15.27:0/3881548506 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5594a1d1e970 0x5594a1d20d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:26.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.427+0000 7fa12f400700 1 --2- 172.21.15.27:0/3881548506 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5594a1d1e970 0x5594a1d20d60 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c90823b2c6a207cb server_cookie=d920953fefce53b5 in_seq=0 out_seq=0 2024-05-10T11:30:26.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.429+0000 7fa119600700 10 client.27542 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:30:26.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.429+0000 7fa119600700 1 -- 172.21.15.27:0/3881548506 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5594a1bf44a0 con 0x5594a1d1e970 2024-05-10T11:30:26.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.429+0000 7fa119600700 10 client.27542 handle_command_reply: tid=0 2024-05-10T11:30:26.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.429+0000 7fa140abeb80 1 client.27542 shutdown 2024-05-10T11:30:26.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.431+0000 7fa140abeb80 1 -- 172.21.15.27:0/3881548506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 msgr2=0x5594a1cbb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:26.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.431+0000 7fa140abeb80 1 --2- 172.21.15.27:0/3881548506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1cbb090 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa12000dd50 tx=0x7fa12000d6a0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.432+0000 7fa140abeb80 1 -- 172.21.15.27:0/3881548506 shutdown_connections 2024-05-10T11:30:26.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.432+0000 7fa140abeb80 1 --2- 172.21.15.27:0/3881548506 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5594a1d1e970 0x5594a1d20d60 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.432+0000 7fa140abeb80 1 --2- 172.21.15.27:0/3881548506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a1c9fbc0 0x5594a1cacab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.432+0000 7fa140abeb80 1 --2- 172.21.15.27:0/3881548506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594a1ca30f0 0x5594a1cbb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.432+0000 7fa140abeb80 1 --2- 172.21.15.27:0/3881548506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5594a1ca04e0 0x5594a1cbb090 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.432+0000 7fa140abeb80 1 -- 172.21.15.27:0/3881548506 >> 172.21.15.27:0/3881548506 conn(0x5594a1c126d0 msgr2=0x5594a1ca51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:26.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.432+0000 7fa119600700 0 client.27542 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:30:26.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.432+0000 7fa140abeb80 1 -- 172.21.15.27:0/3881548506 shutdown_connections 2024-05-10T11:30:26.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.433+0000 7fa140abeb80 1 -- 172.21.15.27:0/3881548506 wait complete. 2024-05-10T11:30:26.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.433+0000 7fa140abeb80 20 client.27542 trim_cache size 0 max 16384 2024-05-10T11:30:26.436 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:30:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:30:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:30:26.437 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:30:26.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.436+0000 7fa136200700 1 -- 172.21.15.27:0/3728943643 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa1140b2af0 msgr2=0x7fa1140b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:26.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.436+0000 7fa136200700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa1140b2af0 0x7fa1140b4fb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa128009ec0 tx=0x7fa128009040 comp rx=0 tx=0).stop 2024-05-10T11:30:26.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.436+0000 7fa136200700 1 -- 172.21.15.27:0/3728943643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa13006b180 msgr2=0x7fa13019d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:26.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.436+0000 7fa136200700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa13006b180 0x7fa13019d6b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00d950 tx=0x7fa11c00dc60 comp rx=0 tx=0).stop 2024-05-10T11:30:26.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.436+0000 7fa136200700 1 -- 172.21.15.27:0/3728943643 shutdown_connections 2024-05-10T11:30:26.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.436+0000 7fa136200700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fa1140b2af0 0x7fa1140b4fb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.436+0000 7fa136200700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa13006b180 0x7fa13019d6b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.436+0000 7fa136200700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa130109510 0x7fa130197870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.436+0000 7fa136200700 1 --2- 172.21.15.27:0/3728943643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa13006bb40 0x7fa13019dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:26.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.436+0000 7fa136200700 1 -- 172.21.15.27:0/3728943643 >> 172.21.15.27:0/3728943643 conn(0x7fa130079350 msgr2=0x7fa130101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:26.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.437+0000 7fa136200700 1 -- 172.21.15.27:0/3728943643 shutdown_connections 2024-05-10T11:30:26.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:26.438+0000 7fa136200700 1 -- 172.21.15.27:0/3728943643 wait complete. 2024-05-10T11:30:27.162 INFO:tasks.workunit.client.0.smithi027.stdout: 10 82685 22.54 MB/sec execute 71 sec latency 306.589 ms 2024-05-10T11:30:27.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:27 smithi027 ceph-mon[30054]: pgmap v541: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 12 MiB/s wr, 600 op/s 2024-05-10T11:30:27.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:27 smithi076 ceph-mon[34152]: pgmap v541: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 12 MiB/s wr, 600 op/s 2024-05-10T11:30:27.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:27 smithi189 ceph-mon[34842]: pgmap v541: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 12 MiB/s wr, 600 op/s 2024-05-10T11:30:28.162 INFO:tasks.workunit.client.0.smithi027.stdout: 10 83067 22.63 MB/sec execute 72 sec latency 380.639 ms 2024-05-10T11:30:28.474 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:30:28.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:30:28.956 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:30:29.163 INFO:tasks.workunit.client.0.smithi027.stdout: 10 83569 22.72 MB/sec execute 73 sec latency 123.174 ms 2024-05-10T11:30:29.388 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:29 smithi027 ceph-mon[30054]: pgmap v542: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 8.4 MiB/s wr, 413 op/s 2024-05-10T11:30:29.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:29 smithi076 ceph-mon[34152]: pgmap v542: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 8.4 MiB/s wr, 413 op/s 2024-05-10T11:30:29.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:29 smithi189 ceph-mon[34842]: pgmap v542: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 8.4 MiB/s wr, 413 op/s 2024-05-10T11:30:30.163 INFO:tasks.workunit.client.0.smithi027.stdout: 10 83962 22.70 MB/sec execute 74 sec latency 127.931 ms 2024-05-10T11:30:31.164 INFO:tasks.workunit.client.0.smithi027.stdout: 10 84207 22.56 MB/sec execute 75 sec latency 208.209 ms 2024-05-10T11:30:31.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.361+0000 7f178ee00700 1 -- 172.21.15.27:0/2111837377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1788102d60 msgr2=0x7f178806b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:31.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.361+0000 7f178ee00700 1 --2- 172.21.15.27:0/2111837377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1788102d60 0x7f178806b180 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f178000b0a0 tx=0x7f178000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:30:31.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.362+0000 7f178ee00700 1 -- 172.21.15.27:0/2111837377 shutdown_connections 2024-05-10T11:30:31.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.362+0000 7f178ee00700 1 --2- 172.21.15.27:0/2111837377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1788102d60 0x7f178806b180 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:31.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.362+0000 7f178ee00700 1 --2- 172.21.15.27:0/2111837377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1788108410 0x7f17881087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:31.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.362+0000 7f178ee00700 1 --2- 172.21.15.27:0/2111837377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17881023a0 0x7f1788102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:31.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.362+0000 7f178ee00700 1 -- 172.21.15.27:0/2111837377 >> 172.21.15.27:0/2111837377 conn(0x7f1788079350 msgr2=0x7f1788079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:31.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.363+0000 7f178ee00700 1 -- 172.21.15.27:0/2111837377 shutdown_connections 2024-05-10T11:30:31.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.363+0000 7f178ee00700 1 -- 172.21.15.27:0/2111837377 wait complete. 2024-05-10T11:30:31.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.364+0000 7f178ee00700 1 Processor -- start 2024-05-10T11:30:31.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.365+0000 7f178ee00700 1 -- start start 2024-05-10T11:30:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.365+0000 7f178ee00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17881023a0 0x7f17881178c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.365+0000 7f178ee00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1788102d60 0x7f1788117e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.365+0000 7f178ee00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1788108410 0x7f178811d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.365+0000 7f178ee00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1788111460 con 0x7f1788108410 2024-05-10T11:30:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.365+0000 7f178ee00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f178810f2d0 con 0x7f17881023a0 2024-05-10T11:30:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.365+0000 7f178ee00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f17881115e0 con 0x7f1788102d60 2024-05-10T11:30:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.366+0000 7f1787400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1788102d60 0x7f1788117e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.366+0000 7f1787400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1788102d60 0x7f1788117e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:55260/0 (socket says 172.21.15.27:55260) 2024-05-10T11:30:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.366+0000 7f1787400700 1 -- 172.21.15.27:0/4038083887 learned_addr learned my addr 172.21.15.27:0/4038083887 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.366+0000 7f178ce00700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1788108410 0x7f178811d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.366+0000 7f1787400700 1 -- 172.21.15.27:0/4038083887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17881023a0 msgr2=0x7f17881178c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-10T11:30:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.366+0000 7f1787400700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17881023a0 0x7f17881178c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.366+0000 7f1787400700 1 -- 172.21.15.27:0/4038083887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1788108410 msgr2=0x7f178811d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.366+0000 7f1787400700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1788108410 0x7f178811d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.366+0000 7f1787400700 1 -- 172.21.15.27:0/4038083887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1788112a30 con 0x7f1788102d60 2024-05-10T11:30:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.366+0000 7f1787400700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1788102d60 0x7f1788117e00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f177400e9a0 tx=0x7f177400ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.366+0000 7f1784c00700 1 -- 172.21.15.27:0/4038083887 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177400cbd0 con 0x7f1788102d60 2024-05-10T11:30:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.366+0000 7f1784c00700 1 -- 172.21.15.27:0/4038083887 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f177400cd70 con 0x7f1788102d60 2024-05-10T11:30:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.367+0000 7f1784c00700 1 -- 172.21.15.27:0/4038083887 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1774018680 con 0x7f1788102d60 2024-05-10T11:30:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.367+0000 7f178ee00700 1 -- 172.21.15.27:0/4038083887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1788112d20 con 0x7f1788102d60 2024-05-10T11:30:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.367+0000 7f178ee00700 1 -- 172.21.15.27:0/4038083887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1788113210 con 0x7f1788102d60 2024-05-10T11:30:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.368+0000 7f178ee00700 1 -- 172.21.15.27:0/4038083887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1788103ac0 con 0x7f1788102d60 2024-05-10T11:30:31.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.372+0000 7f1784c00700 1 -- 172.21.15.27:0/4038083887 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f1774010970 con 0x7f1788102d60 2024-05-10T11:30:31.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.373+0000 7f1784c00700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f176c0b29f0 0x7f176c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:31.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.374+0000 7f1784c00700 1 -- 172.21.15.27:0/4038083887 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7f17740e2e70 con 0x7f1788102d60 2024-05-10T11:30:31.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.374+0000 7f1787e00700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f176c0b29f0 0x7f176c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:31.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.375+0000 7f1787e00700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f176c0b29f0 0x7f176c0b4eb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f177800b990 tx=0x7f177800b370 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:31.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.378+0000 7f1784c00700 1 -- 172.21.15.27:0/4038083887 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f17740aa3c0 con 0x7f1788102d60 2024-05-10T11:30:31.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:31 smithi027 ceph-mon[30054]: pgmap v543: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 11 MiB/s wr, 660 op/s 2024-05-10T11:30:31.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:31 smithi076 ceph-mon[34152]: pgmap v543: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 11 MiB/s wr, 660 op/s 2024-05-10T11:30:31.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:31 smithi189 ceph-mon[34842]: pgmap v543: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 11 MiB/s wr, 660 op/s 2024-05-10T11:30:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.780+0000 7f178ee00700 1 -- 172.21.15.27:0/4038083887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1788113f10 con 0x7f1788102d60 2024-05-10T11:30:31.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.781+0000 7f1784c00700 1 -- 172.21.15.27:0/4038083887 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11936 (secure 0 0 0) 0x7f17740a9cc0 con 0x7f1788102d60 2024-05-10T11:30:31.781 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:30:31.782 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:29:51.629207+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:30:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.783+0000 7f178ee00700 1 -- 172.21.15.27:0/4038083887 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f176c0b29f0 msgr2=0x7f176c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.784+0000 7f178ee00700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f176c0b29f0 0x7f176c0b4eb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f177800b990 tx=0x7f177800b370 comp rx=0 tx=0).stop 2024-05-10T11:30:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.784+0000 7f178ee00700 1 -- 172.21.15.27:0/4038083887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1788102d60 msgr2=0x7f1788117e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.784+0000 7f178ee00700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1788102d60 0x7f1788117e00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f177400e9a0 tx=0x7f177400ed60 comp rx=0 tx=0).stop 2024-05-10T11:30:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.784+0000 7f178ee00700 1 -- 172.21.15.27:0/4038083887 shutdown_connections 2024-05-10T11:30:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.784+0000 7f178ee00700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f176c0b29f0 0x7f176c0b4eb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.784+0000 7f178ee00700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1788108410 0x7f178811d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.784+0000 7f178ee00700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17881023a0 0x7f17881178c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.784+0000 7f178ee00700 1 --2- 172.21.15.27:0/4038083887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1788102d60 0x7f1788117e00 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.784+0000 7f178ee00700 1 -- 172.21.15.27:0/4038083887 >> 172.21.15.27:0/4038083887 conn(0x7f1788079350 msgr2=0x7f1788109200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.784+0000 7f178ee00700 1 -- 172.21.15.27:0/4038083887 shutdown_connections 2024-05-10T11:30:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:31.784+0000 7f178ee00700 1 -- 172.21.15.27:0/4038083887 wait complete. 2024-05-10T11:30:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 45 2024-05-10T11:30:32.163 INFO:tasks.workunit.client.0.smithi027.stdout: 10 84538 22.53 MB/sec execute 76 sec latency 279.645 ms 2024-05-10T11:30:32.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:32 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4038083887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:30:32.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:32 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4038083887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:30:32.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:32 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4038083887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:30:33.163 INFO:tasks.workunit.client.0.smithi027.stdout: 10 84816 22.36 MB/sec execute 77 sec latency 259.472 ms 2024-05-10T11:30:33.562 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:33 smithi027 ceph-mon[30054]: pgmap v544: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 8.2 MiB/s wr, 569 op/s 2024-05-10T11:30:33.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:30:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:33 smithi076 ceph-mon[34152]: pgmap v544: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 8.2 MiB/s wr, 569 op/s 2024-05-10T11:30:33.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:33 smithi189 ceph-mon[34842]: pgmap v544: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 8.2 MiB/s wr, 569 op/s 2024-05-10T11:30:34.118 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:30:34.164 INFO:tasks.workunit.client.0.smithi027.stdout: 10 85470 22.40 MB/sec execute 78 sec latency 179.570 ms 2024-05-10T11:30:35.166 INFO:tasks.workunit.client.0.smithi027.stdout: 10 86134 22.60 MB/sec execute 79 sec latency 166.076 ms 2024-05-10T11:30:35.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:35 smithi027 ceph-mon[30054]: pgmap v545: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.2 MiB/s wr, 513 op/s 2024-05-10T11:30:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:35 smithi076 ceph-mon[34152]: pgmap v545: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.2 MiB/s wr, 513 op/s 2024-05-10T11:30:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:35 smithi189 ceph-mon[34842]: pgmap v545: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.2 MiB/s wr, 513 op/s 2024-05-10T11:30:36.166 INFO:tasks.workunit.client.0.smithi027.stdout: 10 86559 22.54 MB/sec execute 80 sec latency 211.258 ms 2024-05-10T11:30:36.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.565+0000 7f8de1200700 1 -- 172.21.15.27:0/2887975381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ddc103fd0 msgr2=0x7f8ddc108020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:36.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.565+0000 7f8de1200700 1 --2- 172.21.15.27:0/2887975381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ddc103fd0 0x7f8ddc108020 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80099c0 tx=0x7f8dc8009cd0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.566+0000 7f8de1200700 1 -- 172.21.15.27:0/2887975381 shutdown_connections 2024-05-10T11:30:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.566+0000 7f8de1200700 1 --2- 172.21.15.27:0/2887975381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ddc108700 0x7f8ddc1102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.566+0000 7f8de1200700 1 --2- 172.21.15.27:0/2887975381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ddc103620 0x7f8ddc103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.566+0000 7f8de1200700 1 --2- 172.21.15.27:0/2887975381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ddc103fd0 0x7f8ddc108020 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.566+0000 7f8de1200700 1 -- 172.21.15.27:0/2887975381 >> 172.21.15.27:0/2887975381 conn(0x7f8ddc0feef0 msgr2=0x7f8ddc101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.566+0000 7f8de1200700 1 -- 172.21.15.27:0/2887975381 shutdown_connections 2024-05-10T11:30:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.566+0000 7f8de1200700 1 -- 172.21.15.27:0/2887975381 wait complete. 2024-05-10T11:30:36.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.567+0000 7f8de1200700 1 Processor -- start 2024-05-10T11:30:36.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.567+0000 7f8de1200700 1 -- start start 2024-05-10T11:30:36.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.568+0000 7f8de1200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ddc103620 0x7f8ddc1a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.568+0000 7f8de1200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ddc103fd0 0x7f8ddc1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.568+0000 7f8de1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ddc108700 0x7f8ddc1a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.568+0000 7f8de1200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc119c50 con 0x7f8ddc103fd0 2024-05-10T11:30:36.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.568+0000 7f8de1200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc119ad0 con 0x7f8ddc103620 2024-05-10T11:30:36.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.568+0000 7f8de1200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc119dd0 con 0x7f8ddc108700 2024-05-10T11:30:36.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.568+0000 7f8ddaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ddc103fd0 0x7f8ddc1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.569+0000 7f8ddaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ddc103fd0 0x7f8ddc1a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35906/0 (socket says 172.21.15.27:35906) 2024-05-10T11:30:36.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.569+0000 7f8ddbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ddc108700 0x7f8ddc1a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.569+0000 7f8ddbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ddc108700 0x7f8ddc1a49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:38498/0 (socket says 172.21.15.27:38498) 2024-05-10T11:30:36.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.569+0000 7f8ddbe00700 1 -- 172.21.15.27:0/1970850988 learned_addr learned my addr 172.21.15.27:0/1970850988 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:36.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.570+0000 7f8ddb400700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ddc103620 0x7f8ddc1a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.570+0000 7f8ddbe00700 1 -- 172.21.15.27:0/1970850988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ddc103620 msgr2=0x7f8ddc1a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:36.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.570+0000 7f8ddbe00700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ddc103620 0x7f8ddc1a0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.570+0000 7f8ddbe00700 1 -- 172.21.15.27:0/1970850988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ddc103fd0 msgr2=0x7f8ddc1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:36.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.570+0000 7f8ddbe00700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ddc103fd0 0x7f8ddc1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.570+0000 7f8ddbe00700 1 -- 172.21.15.27:0/1970850988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ddc1a5130 con 0x7f8ddc108700 2024-05-10T11:30:36.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.570+0000 7f8ddbe00700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ddc108700 0x7f8ddc1a49f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00ada0 tx=0x7f8dcc00c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:36.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.571+0000 7f8dd3e00700 1 -- 172.21.15.27:0/1970850988 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc017020 con 0x7f8ddc108700 2024-05-10T11:30:36.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.571+0000 7f8dd3e00700 1 -- 172.21.15.27:0/1970850988 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dcc00ce30 con 0x7f8ddc108700 2024-05-10T11:30:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.571+0000 7f8dd3e00700 1 -- 172.21.15.27:0/1970850988 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc005440 con 0x7f8ddc108700 2024-05-10T11:30:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.572+0000 7f8de1200700 1 -- 172.21.15.27:0/1970850988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ddc1a5420 con 0x7f8ddc108700 2024-05-10T11:30:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.572+0000 7f8de1200700 1 -- 172.21.15.27:0/1970850988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ddc110e70 con 0x7f8ddc108700 2024-05-10T11:30:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.579+0000 7f8ded286b80 1 Processor -- start 2024-05-10T11:30:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.582+0000 7f8dd3e00700 1 -- 172.21.15.27:0/1970850988 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f8dcc004ad0 con 0x7f8ddc108700 2024-05-10T11:30:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.583+0000 7f8ded286b80 1 -- start start 2024-05-10T11:30:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.585+0000 7f8ded286b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7efda0 0x555abc7f0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.585+0000 7f8ded286b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc7f06c0 0x555abc7ea430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.585+0000 7f8ded286b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7eaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.585+0000 7f8ded286b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555abc73ca20 con 0x555abc7f06c0 2024-05-10T11:30:36.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.585+0000 7f8ded286b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x555abc6da760 con 0x555abc7efda0 2024-05-10T11:30:36.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.585+0000 7f8ded286b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555abc7389e0 con 0x555abc7eaa50 2024-05-10T11:30:36.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.585+0000 7f8ddb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc7f06c0 0x555abc7ea430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.585+0000 7f8ddaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7eaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.585+0000 7f8ddaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7eaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:38506/0 (socket says 172.21.15.27:38506) 2024-05-10T11:30:36.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.585+0000 7f8ddaa00700 1 -- 172.21.15.27:0/2621137508 learned_addr learned my addr 172.21.15.27:0/2621137508 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:36.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.585+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/2621137508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7efda0 0x555abc7f0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.586+0000 7f8dd3e00700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8dc00b2a50 0x7f8dc00b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.586+0000 7f8ddaa00700 1 -- 172.21.15.27:0/2621137508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7efda0 msgr2=0x555abc7f0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:36.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.586+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/2621137508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7efda0 0x555abc7f0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.586+0000 7f8ddaa00700 1 -- 172.21.15.27:0/2621137508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc7f06c0 msgr2=0x555abc7ea430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:36.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.586+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/2621137508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc7f06c0 0x555abc7ea430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.586+0000 7f8ddaa00700 1 -- 172.21.15.27:0/2621137508 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555abc6898d0 con 0x555abc7eaa50 2024-05-10T11:30:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.586+0000 7f8dd3e00700 1 -- 172.21.15.27:0/1970850988 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7f8dcc0e1ec0 con 0x7f8ddc108700 2024-05-10T11:30:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.587+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/2621137508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7eaed0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8003800 tx=0x7f8dc8005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b02907f0eaef608d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.587+0000 7f8ddbe00700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8dc00b2a50 0x7f8dc00b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.588+0000 7f8dd0c00700 1 -- 172.21.15.27:0/2621137508 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8003d20 con 0x555abc7eaa50 2024-05-10T11:30:36.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.588+0000 7f8dd0c00700 1 -- 172.21.15.27:0/2621137508 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dc8005570 con 0x555abc7eaa50 2024-05-10T11:30:36.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.588+0000 7f8dd0c00700 1 -- 172.21.15.27:0/2621137508 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc800f460 con 0x555abc7eaa50 2024-05-10T11:30:36.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.589+0000 7f8ddbe00700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8dc00b2a50 0x7f8dc00b4f10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0df7e0 tx=0x7f8dcc004fb0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:36.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.589+0000 7f8ded286b80 1 -- 172.21.15.27:0/2621137508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 msgr2=0x555abc7eaed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:36.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.589+0000 7f8ded286b80 1 --2- 172.21.15.27:0/2621137508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7eaed0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8003800 tx=0x7f8dc8005bc0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.589+0000 7f8ded286b80 1 -- 172.21.15.27:0/2621137508 shutdown_connections 2024-05-10T11:30:36.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.589+0000 7f8ded286b80 1 --2- 172.21.15.27:0/2621137508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7eaed0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.589+0000 7f8ded286b80 1 --2- 172.21.15.27:0/2621137508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7efda0 0x555abc7f0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.589+0000 7f8ded286b80 1 --2- 172.21.15.27:0/2621137508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc7f06c0 0x555abc7ea430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.589+0000 7f8ded286b80 1 -- 172.21.15.27:0/2621137508 >> 172.21.15.27:0/2621137508 conn(0x555abc7626d0 msgr2=0x555abc7e4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:36.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.591+0000 7f8ded286b80 1 -- 172.21.15.27:0/2621137508 shutdown_connections 2024-05-10T11:30:36.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.592+0000 7f8ded286b80 1 -- 172.21.15.27:0/2621137508 wait complete. 2024-05-10T11:30:36.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.592+0000 7f8ded286b80 1 -- start start 2024-05-10T11:30:36.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.593+0000 7f8ded286b80 1 Processor -- start 2024-05-10T11:30:36.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.594+0000 7f8ded286b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.594+0000 7f8ded286b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc7efda0 0x555abc7f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.594+0000 7f8ded286b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7f06c0 0x555abc7f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.594+0000 7f8ded286b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555abc7f7140 con 0x555abc7efda0 2024-05-10T11:30:36.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.594+0000 7f8ded286b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x555abc6a2b20 con 0x555abc7f06c0 2024-05-10T11:30:36.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.594+0000 7f8ded286b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555abc7f72c0 con 0x555abc7eaa50 2024-05-10T11:30:36.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.594+0000 7f8ded286b80 10 client.0 _ll_register_callbacks cb 0x555abc6e3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:30:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.597+0000 7f8ddaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.597+0000 7f8ddaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7f63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:38514/0 (socket says 172.21.15.27:38514) 2024-05-10T11:30:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.597+0000 7f8ddaa00700 1 -- 172.21.15.27:0/4162183966 learned_addr learned my addr 172.21.15.27:0/4162183966 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.597+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7f06c0 0x555abc7f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.598+0000 7f8ddb400700 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc7efda0 0x555abc7f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.599+0000 7f8ddaa00700 1 -- 172.21.15.27:0/4162183966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 msgr2=0x555abc7f63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:36.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.599+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7f63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.599+0000 7f8ddaa00700 1 -- 172.21.15.27:0/4162183966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc7efda0 msgr2=0x555abc7f37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:36.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.599+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc7efda0 0x555abc7f37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.599+0000 7f8ddaa00700 1 -- 172.21.15.27:0/4162183966 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555abc702420 con 0x555abc7f06c0 2024-05-10T11:30:36.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.599+0000 7f8ddb400700 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc7efda0 0x555abc7f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:30:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.600+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7f06c0 0x555abc7f3cf0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8006010 tx=0x7f8dc8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b66c50632874ef93 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.600+0000 7f8dc5600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:30:36.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.601+0000 7f8dc5600700 1 -- 172.21.15.27:0/4162183966 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc80050c0 con 0x555abc7f06c0 2024-05-10T11:30:36.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.601+0000 7f8dc5600700 1 -- 172.21.15.27:0/4162183966 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7f8dc8005260 con 0x555abc7f06c0 2024-05-10T11:30:36.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.601+0000 7f8ded286b80 1 -- 172.21.15.27:0/4162183966 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dc8005570 con 0x555abc7f06c0 2024-05-10T11:30:36.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.602+0000 7f8dc5600700 1 -- 172.21.15.27:0/4162183966 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f8dc803e050 con 0x555abc7f06c0 2024-05-10T11:30:36.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.603+0000 7f8ded286b80 10 client.27551 fetch_fsmap learned FSMap version 45 2024-05-10T11:30:36.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.603+0000 7f8ded286b80 1 -- 172.21.15.27:0/4162183966 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x555abc7f7140 con 0x555abc7f06c0 2024-05-10T11:30:36.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.604+0000 7f8dc5600700 1 -- 172.21.15.27:0/4162183966 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dc8005570 con 0x555abc7f06c0 2024-05-10T11:30:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.606+0000 7f8dc5600700 1 -- 172.21.15.27:0/4162183966 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f8dc800fad0 con 0x555abc7f06c0 2024-05-10T11:30:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.606+0000 7f8ded286b80 10 client.27551 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:30:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.606+0000 7f8ded286b80 10 client.27551 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:30:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.606+0000 7f8ded286b80 20 client.27551 populate_metadata read hostname 'smithi027' 2024-05-10T11:30:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.606+0000 7f8ded286b80 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x555abc803d30 0x555abc806120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.607+0000 7f8ded286b80 4 client.27551 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:30:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.607+0000 7f8ded286b80 1 -- 172.21.15.27:0/4162183966 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555abc7b7140 con 0x555abc803d30 2024-05-10T11:30:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.607+0000 7f8ddb400700 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x555abc803d30 0x555abc806120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.608+0000 7f8ddb400700 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x555abc803d30 0x555abc806120 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c40412c0bec946b2 server_cookie=e556052ecf880851 in_seq=0 out_seq=0 2024-05-10T11:30:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.608+0000 7f8dc5600700 10 client.27551 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:30:36.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.609+0000 7f8dc5600700 1 -- 172.21.15.27:0/4162183966 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555abc7b7140 con 0x555abc803d30 2024-05-10T11:30:36.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.609+0000 7f8dc5600700 10 client.27551 handle_command_reply: tid=0 2024-05-10T11:30:36.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.609+0000 7f8ded286b80 1 client.27551 shutdown 2024-05-10T11:30:36.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.610+0000 7f8ded286b80 1 -- 172.21.15.27:0/4162183966 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7f06c0 msgr2=0x555abc7f3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:36.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.610+0000 7f8ded286b80 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7f06c0 0x555abc7f3cf0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8006010 tx=0x7f8dc8000bc0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.610+0000 7f8ded286b80 1 -- 172.21.15.27:0/4162183966 shutdown_connections 2024-05-10T11:30:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.610+0000 7f8ded286b80 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x555abc803d30 0x555abc806120 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.610+0000 7f8dc5600700 0 client.27551 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:30:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.610+0000 7f8ded286b80 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7f63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.610+0000 7f8ded286b80 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7f06c0 0x555abc7f3cf0 secure :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8006010 tx=0x7f8dc8000bc0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.610+0000 7f8ded286b80 1 --2- 172.21.15.27:0/4162183966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc7efda0 0x555abc7f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.610+0000 7f8ded286b80 1 -- 172.21.15.27:0/4162183966 >> 172.21.15.27:0/4162183966 conn(0x555abc7626d0 msgr2=0x555abc7df750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.611+0000 7f8ded286b80 1 -- 172.21.15.27:0/4162183966 shutdown_connections 2024-05-10T11:30:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.611+0000 7f8ded286b80 1 -- 172.21.15.27:0/4162183966 wait complete. 2024-05-10T11:30:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.611+0000 7f8ded286b80 20 client.27551 trim_cache size 0 max 16384 2024-05-10T11:30:36.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.643+0000 7f8ded286b80 1 Processor -- start 2024-05-10T11:30:36.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.643+0000 7f8ded286b80 1 -- start start 2024-05-10T11:30:36.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.643+0000 7f8ded286b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc800220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.643+0000 7f8ded286b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7efda0 0x555abc7df860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.644+0000 7f8ded286b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc803d30 0x555abc7f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.644+0000 7f8ded286b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1be390 con 0x555abc803d30 2024-05-10T11:30:36.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.644+0000 7f8ded286b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1bfed0 con 0x555abc7efda0 2024-05-10T11:30:36.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.644+0000 7f8ded286b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1bea60 con 0x555abc7eaa50 2024-05-10T11:30:36.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.644+0000 7f8ddb400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7efda0 0x555abc7df860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.644+0000 7f8ddb400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7efda0 0x555abc7df860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55404/0 (socket says 172.21.15.27:55404) 2024-05-10T11:30:36.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.644+0000 7f8ddb400700 1 -- 172.21.15.27:0/3429890783 learned_addr learned my addr 172.21.15.27:0/3429890783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:36.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.645+0000 7f8ddb400700 1 -- 172.21.15.27:0/3429890783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 msgr2=0x555abc800220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:36.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.645+0000 7f8ddb400700 1 --2- 172.21.15.27:0/3429890783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc800220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.645+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/3429890783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc803d30 0x555abc7f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.645+0000 7f8ddb400700 1 -- 172.21.15.27:0/3429890783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc803d30 msgr2=0x555abc7f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:36.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.645+0000 7f8ddb400700 1 --2- 172.21.15.27:0/3429890783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc803d30 0x555abc7f11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.645+0000 7f8ddb400700 1 -- 172.21.15.27:0/3429890783 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555abc7e18a0 con 0x555abc7efda0 2024-05-10T11:30:36.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.646+0000 7f8ddb400700 1 --2- 172.21.15.27:0/3429890783 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7efda0 0x555abc7df860 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x555abc7e6970 tx=0x7f8dd4014750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0b73c9b18fe3c62 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:36.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.647+0000 7f8dc7e00700 1 -- 172.21.15.27:0/3429890783 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd40140d0 con 0x555abc7efda0 2024-05-10T11:30:36.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.647+0000 7f8dc7e00700 1 -- 172.21.15.27:0/3429890783 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dd4014270 con 0x555abc7efda0 2024-05-10T11:30:36.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.647+0000 7f8dc7e00700 1 -- 172.21.15.27:0/3429890783 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd401d8a0 con 0x555abc7efda0 2024-05-10T11:30:36.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.648+0000 7f8ded286b80 1 -- 172.21.15.27:0/3429890783 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7efda0 msgr2=0x555abc7df860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:36.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.648+0000 7f8ded286b80 1 --2- 172.21.15.27:0/3429890783 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7efda0 0x555abc7df860 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x555abc7e6970 tx=0x7f8dd4014750 comp rx=0 tx=0).stop 2024-05-10T11:30:36.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.648+0000 7f8ded286b80 1 -- 172.21.15.27:0/3429890783 shutdown_connections 2024-05-10T11:30:36.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.648+0000 7f8ded286b80 1 --2- 172.21.15.27:0/3429890783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc800220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.648+0000 7f8ded286b80 1 --2- 172.21.15.27:0/3429890783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc803d30 0x555abc7f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.648+0000 7f8ded286b80 1 --2- 172.21.15.27:0/3429890783 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7efda0 0x555abc7df860 secure :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0x555abc7e6970 tx=0x7f8dd4014750 comp rx=0 tx=0).stop 2024-05-10T11:30:36.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.648+0000 7f8ded286b80 1 -- 172.21.15.27:0/3429890783 >> 172.21.15.27:0/3429890783 conn(0x555abc7626d0 msgr2=0x555abc7f0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:36.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.650+0000 7f8ded286b80 1 -- 172.21.15.27:0/3429890783 shutdown_connections 2024-05-10T11:30:36.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.650+0000 7f8ded286b80 1 -- 172.21.15.27:0/3429890783 wait complete. 2024-05-10T11:30:36.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.652+0000 7f8ded286b80 1 -- start start 2024-05-10T11:30:36.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.652+0000 7f8ded286b80 1 Processor -- start 2024-05-10T11:30:36.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.653+0000 7f8ded286b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7fc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.653+0000 7f8ded286b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc803d30 0x555abc7fcc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.653+0000 7f8ded286b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7fb390 0x555abc7fb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.653+0000 7f8ded286b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1bf800 con 0x555abc803d30 2024-05-10T11:30:36.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.653+0000 7f8ded286b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1bf130 con 0x555abc7fb390 2024-05-10T11:30:36.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.653+0000 7f8ded286b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc117d20 con 0x555abc7eaa50 2024-05-10T11:30:36.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.655+0000 7f8ddb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc803d30 0x555abc7fcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.655+0000 7f8ddaa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7fb390 0x555abc7fb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.656+0000 7f8ded286b80 10 client.0 _ll_register_callbacks cb 0x555abc6e3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:30:36.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.656+0000 7f8ddaa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7fb390 0x555abc7fb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55408/0 (socket says 172.21.15.27:55408) 2024-05-10T11:30:36.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.656+0000 7f8ddaa00700 1 -- 172.21.15.27:0/1966394582 learned_addr learned my addr 172.21.15.27:0/1966394582 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:36.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.658+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/1966394582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7fc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.659+0000 7f8ddaa00700 1 -- 172.21.15.27:0/1966394582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 msgr2=0x555abc7fc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:36.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.659+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/1966394582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7fc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.659+0000 7f8ddaa00700 1 -- 172.21.15.27:0/1966394582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc803d30 msgr2=0x555abc7fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.659+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/1966394582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc803d30 0x555abc7fcc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.659+0000 7f8ddaa00700 1 -- 172.21.15.27:0/1966394582 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555abc7e67a0 con 0x555abc7fb390 2024-05-10T11:30:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.660+0000 7f8ddaa00700 1 --2- 172.21.15.27:0/1966394582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7fb390 0x555abc7fb810 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80274f0 tx=0x7f8dc8009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e67dc506fc44834 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.660+0000 7f8dc5600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:30:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.660+0000 7f8dc5600700 1 -- 172.21.15.27:0/1966394582 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8005bc0 con 0x555abc7fb390 2024-05-10T11:30:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.660+0000 7f8dc5600700 1 -- 172.21.15.27:0/1966394582 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7f8dc8003aa0 con 0x555abc7fb390 2024-05-10T11:30:36.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.661+0000 7f8ded286b80 1 -- 172.21.15.27:0/1966394582 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dd4014270 con 0x555abc7fb390 2024-05-10T11:30:36.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.662+0000 7f8dc5600700 1 -- 172.21.15.27:0/1966394582 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f8dc8003ca0 con 0x555abc7fb390 2024-05-10T11:30:36.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.662+0000 7f8ded286b80 10 client.27557 fetch_fsmap learned FSMap version 45 2024-05-10T11:30:36.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.662+0000 7f8ded286b80 1 -- 172.21.15.27:0/1966394582 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x555abc86e1b0 con 0x555abc7fb390 2024-05-10T11:30:36.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.663+0000 7f8dc5600700 1 -- 172.21.15.27:0/1966394582 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dd4014270 con 0x555abc7fb390 2024-05-10T11:30:36.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.663+0000 7f8dc5600700 1 -- 172.21.15.27:0/1966394582 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f8dc8027c60 con 0x555abc7fb390 2024-05-10T11:30:36.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.663+0000 7f8ded286b80 10 client.27557 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:30:36.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.663+0000 7f8ded286b80 10 client.27557 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:30:36.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.663+0000 7f8ded286b80 20 client.27557 populate_metadata read hostname 'smithi027' 2024-05-10T11:30:36.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.663+0000 7f8ded286b80 1 --2- 172.21.15.27:0/1966394582 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x555abc86e7a0 0x555abc870b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:36.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.663+0000 7f8ded286b80 4 client.27557 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:30:36.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.663+0000 7f8ded286b80 1 -- 172.21.15.27:0/1966394582 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555abc74a400 con 0x555abc86e7a0 2024-05-10T11:30:36.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.663+0000 7f8ddb400700 1 --2- 172.21.15.27:0/1966394582 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x555abc86e7a0 0x555abc870b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:36.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.666+0000 7f8ddb400700 1 --2- 172.21.15.27:0/1966394582 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x555abc86e7a0 0x555abc870b90 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3c22d0e832f81db1 server_cookie=96def3a333217e10 in_seq=0 out_seq=0 2024-05-10T11:30:36.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.666+0000 7f8dc5600700 10 client.27557 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:30:36.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.666+0000 7f8dc5600700 1 -- 172.21.15.27:0/1966394582 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555abc74a400 con 0x555abc86e7a0 2024-05-10T11:30:36.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.666+0000 7f8dc5600700 10 client.27557 handle_command_reply: tid=0 2024-05-10T11:30:36.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.667+0000 7f8ded286b80 1 client.27557 shutdown 2024-05-10T11:30:36.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.667+0000 7f8ded286b80 1 -- 172.21.15.27:0/1966394582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7fb390 msgr2=0x555abc7fb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:36.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.667+0000 7f8ded286b80 1 --2- 172.21.15.27:0/1966394582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7fb390 0x555abc7fb810 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80274f0 tx=0x7f8dc8009cd0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.667+0000 7f8ded286b80 1 -- 172.21.15.27:0/1966394582 shutdown_connections 2024-05-10T11:30:36.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.667+0000 7f8ded286b80 1 --2- 172.21.15.27:0/1966394582 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x555abc86e7a0 0x555abc870b90 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.667+0000 7f8ded286b80 1 --2- 172.21.15.27:0/1966394582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555abc7eaa50 0x555abc7fc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.667+0000 7f8ded286b80 1 --2- 172.21.15.27:0/1966394582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555abc7fb390 0x555abc7fb810 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.667+0000 7f8ded286b80 1 --2- 172.21.15.27:0/1966394582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555abc803d30 0x555abc7fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.667+0000 7f8ded286b80 1 -- 172.21.15.27:0/1966394582 >> 172.21.15.27:0/1966394582 conn(0x555abc7626d0 msgr2=0x555abc7ee8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:36.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.667+0000 7f8dc5600700 0 client.27557 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:30:36.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.668+0000 7f8ded286b80 1 -- 172.21.15.27:0/1966394582 shutdown_connections 2024-05-10T11:30:36.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.669+0000 7f8ded286b80 1 -- 172.21.15.27:0/1966394582 wait complete. 2024-05-10T11:30:36.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.669+0000 7f8ded286b80 20 client.27557 trim_cache size 0 max 16384 2024-05-10T11:30:36.670 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:30:36.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.670+0000 7f8de1200700 1 -- 172.21.15.27:0/1970850988 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8dc00b2a50 msgr2=0x7f8dc00b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:36.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.670+0000 7f8de1200700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8dc00b2a50 0x7f8dc00b4f10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0df7e0 tx=0x7f8dcc004fb0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.670+0000 7f8de1200700 1 -- 172.21.15.27:0/1970850988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ddc108700 msgr2=0x7f8ddc1a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:36.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.670+0000 7f8de1200700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ddc108700 0x7f8ddc1a49f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00ada0 tx=0x7f8dcc00c330 comp rx=0 tx=0).stop 2024-05-10T11:30:36.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.670+0000 7f8de1200700 1 -- 172.21.15.27:0/1970850988 shutdown_connections 2024-05-10T11:30:36.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.670+0000 7f8de1200700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f8dc00b2a50 0x7f8dc00b4f10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.670+0000 7f8de1200700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ddc108700 0x7f8ddc1a49f0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.670+0000 7f8de1200700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ddc103620 0x7f8ddc1a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.670+0000 7f8de1200700 1 --2- 172.21.15.27:0/1970850988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ddc103fd0 0x7f8ddc1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:36.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.670+0000 7f8de1200700 1 -- 172.21.15.27:0/1970850988 >> 172.21.15.27:0/1970850988 conn(0x7f8ddc0feef0 msgr2=0x7f8ddc101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:36.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.671+0000 7f8de1200700 1 -- 172.21.15.27:0/1970850988 shutdown_connections 2024-05-10T11:30:36.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:36.671+0000 7f8de1200700 1 -- 172.21.15.27:0/1970850988 wait complete. 2024-05-10T11:30:37.166 INFO:tasks.workunit.client.0.smithi027.stdout: 10 87082 22.76 MB/sec execute 81 sec latency 77.058 ms 2024-05-10T11:30:37.490 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:37 smithi027 ceph-mon[30054]: pgmap v546: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.9 MiB/s wr, 691 op/s 2024-05-10T11:30:37.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:37 smithi076 ceph-mon[34152]: pgmap v546: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.9 MiB/s wr, 691 op/s 2024-05-10T11:30:37.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:37 smithi189 ceph-mon[34842]: pgmap v546: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.9 MiB/s wr, 691 op/s 2024-05-10T11:30:38.030 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:30:38.166 INFO:tasks.workunit.client.0.smithi027.stdout: 10 87710 23.01 MB/sec execute 82 sec latency 69.813 ms 2024-05-10T11:30:38.589 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:30:39.166 INFO:tasks.workunit.client.0.smithi027.stdout: 10 88016 22.93 MB/sec execute 83 sec latency 154.963 ms 2024-05-10T11:30:39.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:39 smithi027 ceph-mon[30054]: pgmap v547: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.1 MiB/s wr, 494 op/s 2024-05-10T11:30:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:39 smithi076 ceph-mon[34152]: pgmap v547: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.1 MiB/s wr, 494 op/s 2024-05-10T11:30:39.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:39 smithi189 ceph-mon[34842]: pgmap v547: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.1 MiB/s wr, 494 op/s 2024-05-10T11:30:40.166 INFO:tasks.workunit.client.0.smithi027.stdout: 10 88372 22.83 MB/sec execute 84 sec latency 147.088 ms 2024-05-10T11:30:41.166 INFO:tasks.workunit.client.0.smithi027.stdout: 10 88616 22.70 MB/sec execute 85 sec latency 407.651 ms 2024-05-10T11:30:41.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.423+0000 7f2994a00700 1 -- 172.21.15.27:0/487472897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2990101a90 msgr2=0x7f2990101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:41.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.423+0000 7f2994a00700 1 --2- 172.21.15.27:0/487472897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2990101a90 0x7f2990101f10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f297c0098f0 tx=0x7f297c009c00 comp rx=0 tx=0).stop 2024-05-10T11:30:41.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.423+0000 7f2994a00700 1 -- 172.21.15.27:0/487472897 shutdown_connections 2024-05-10T11:30:41.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.423+0000 7f2994a00700 1 --2- 172.21.15.27:0/487472897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2990102450 0x7f2990110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.423+0000 7f2994a00700 1 --2- 172.21.15.27:0/487472897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990109520 0x7f2990109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.423+0000 7f2994a00700 1 --2- 172.21.15.27:0/487472897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2990101a90 0x7f2990101f10 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.423+0000 7f2994a00700 1 -- 172.21.15.27:0/487472897 >> 172.21.15.27:0/487472897 conn(0x7f2990078a10 msgr2=0x7f29900ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:41.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.423+0000 7f2994a00700 1 -- 172.21.15.27:0/487472897 shutdown_connections 2024-05-10T11:30:41.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.423+0000 7f2994a00700 1 -- 172.21.15.27:0/487472897 wait complete. 2024-05-10T11:30:41.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.424+0000 7f2994a00700 1 Processor -- start 2024-05-10T11:30:41.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.424+0000 7f2994a00700 1 -- start start 2024-05-10T11:30:41.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.425+0000 7f2994a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2990101a90 0x7f299019bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.425+0000 7f2994a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990102450 0x7f299019c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.425+0000 7f2994a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2990109520 0x7f29901a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.425+0000 7f2994a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2990113680 con 0x7f2990102450 2024-05-10T11:30:41.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.425+0000 7f2994a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2990113500 con 0x7f2990109520 2024-05-10T11:30:41.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.425+0000 7f2994a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2990113800 con 0x7f2990101a90 2024-05-10T11:30:41.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.425+0000 7f298de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2990101a90 0x7f299019bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.425+0000 7f298e800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2990109520 0x7f29901a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.425+0000 7f298e800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2990109520 0x7f29901a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55416/0 (socket says 172.21.15.27:55416) 2024-05-10T11:30:41.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.425+0000 7f298e800700 1 -- 172.21.15.27:0/1906580234 learned_addr learned my addr 172.21.15.27:0/1906580234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:41.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.425+0000 7f298d400700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990102450 0x7f299019c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.426+0000 7f298de00700 1 -- 172.21.15.27:0/1906580234 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2990109520 msgr2=0x7f29901a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:41.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.426+0000 7f298de00700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2990109520 0x7f29901a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.426+0000 7f298de00700 1 -- 172.21.15.27:0/1906580234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990102450 msgr2=0x7f299019c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:41.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.426+0000 7f298de00700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990102450 0x7f299019c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.426+0000 7f298de00700 1 -- 172.21.15.27:0/1906580234 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29901a0e00 con 0x7f2990101a90 2024-05-10T11:30:41.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.426+0000 7f298d400700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990102450 0x7f299019c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:30:41.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.427+0000 7f298de00700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2990101a90 0x7f299019bdf0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f298000e780 tx=0x7f298000ea90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:41.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.428+0000 7f2986a00700 1 -- 172.21.15.27:0/1906580234 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980004220 con 0x7f2990101a90 2024-05-10T11:30:41.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.429+0000 7f2986a00700 1 -- 172.21.15.27:0/1906580234 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29800043c0 con 0x7f2990101a90 2024-05-10T11:30:41.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.429+0000 7f2994a00700 1 -- 172.21.15.27:0/1906580234 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29901a10f0 con 0x7f2990101a90 2024-05-10T11:30:41.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.429+0000 7f2986a00700 1 -- 172.21.15.27:0/1906580234 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800106a0 con 0x7f2990101a90 2024-05-10T11:30:41.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.429+0000 7f2994a00700 1 -- 172.21.15.27:0/1906580234 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29901a8970 con 0x7f2990101a90 2024-05-10T11:30:41.437 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:41 smithi027 ceph-mon[30054]: pgmap v548: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.6 MiB/s wr, 627 op/s 2024-05-10T11:30:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.435+0000 7f2986a00700 1 -- 172.21.15.27:0/1906580234 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2980010840 con 0x7f2990101a90 2024-05-10T11:30:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.435+0000 7f299f3aab80 1 Processor -- start 2024-05-10T11:30:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.435+0000 7f299f3aab80 1 -- start start 2024-05-10T11:30:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.435+0000 7f299f3aab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd630bc0 0x55eadd630fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f299f3aab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd6314e0 0x55eadd62b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f299f3aab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd62b8f0 0x55eadd62bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f299f3aab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55eadd57da20 con 0x55eadd62b8f0 2024-05-10T11:30:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f299f3aab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55eadd51b760 con 0x55eadd6314e0 2024-05-10T11:30:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f299f3aab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55eadd5799e0 con 0x55eadd630bc0 2024-05-10T11:30:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f298e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd630bc0 0x55eadd630fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f298de00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd6314e0 0x55eadd62b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f298e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd630bc0 0x55eadd630fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:38540/0 (socket says 172.21.15.27:38540) 2024-05-10T11:30:41.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f298e800700 1 -- 172.21.15.27:0/3298428462 learned_addr learned my addr 172.21.15.27:0/3298428462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:41.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f298e800700 1 -- 172.21.15.27:0/3298428462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd6314e0 msgr2=0x55eadd62b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:41.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f298e800700 1 --2- 172.21.15.27:0/3298428462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd6314e0 0x55eadd62b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f298d400700 1 --2- 172.21.15.27:0/3298428462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd62b8f0 0x55eadd62bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.436+0000 7f298e800700 1 -- 172.21.15.27:0/3298428462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd62b8f0 msgr2=0x55eadd62bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:41.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.437+0000 7f298e800700 1 --2- 172.21.15.27:0/3298428462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd62b8f0 0x55eadd62bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.437+0000 7f298e800700 1 -- 172.21.15.27:0/3298428462 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eadd4ca8d0 con 0x55eadd630bc0 2024-05-10T11:30:41.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.437+0000 7f298d400700 1 --2- 172.21.15.27:0/3298428462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd62b8f0 0x55eadd62bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:30:41.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.437+0000 7f298e800700 1 --2- 172.21.15.27:0/3298428462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd630bc0 0x55eadd630fa0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2988009fa0 tx=0x7f29880091b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75733d8aebcfabe6 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:41.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.438+0000 7f297b400700 1 -- 172.21.15.27:0/3298428462 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988004bb0 con 0x55eadd630bc0 2024-05-10T11:30:41.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.438+0000 7f297b400700 1 -- 172.21.15.27:0/3298428462 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2988004d50 con 0x55eadd630bc0 2024-05-10T11:30:41.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.438+0000 7f297b400700 1 -- 172.21.15.27:0/3298428462 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988005350 con 0x55eadd630bc0 2024-05-10T11:30:41.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.438+0000 7f2986a00700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f29740b2b40 0x7f29740b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.439+0000 7f299f3aab80 1 -- 172.21.15.27:0/3298428462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd630bc0 msgr2=0x55eadd630fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:41.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.439+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3298428462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd630bc0 0x55eadd630fa0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2988009fa0 tx=0x7f29880091b0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.439+0000 7f2986a00700 1 -- 172.21.15.27:0/1906580234 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7f29800e25d0 con 0x7f2990101a90 2024-05-10T11:30:41.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.439+0000 7f299f3aab80 1 -- 172.21.15.27:0/3298428462 shutdown_connections 2024-05-10T11:30:41.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.439+0000 7f298de00700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f29740b2b40 0x7f29740b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.439+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3298428462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd630bc0 0x55eadd630fa0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.439+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3298428462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd62b8f0 0x55eadd62bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.439+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3298428462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd6314e0 0x55eadd62b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.439+0000 7f299f3aab80 1 -- 172.21.15.27:0/3298428462 >> 172.21.15.27:0/3298428462 conn(0x55eadd5a36d0 msgr2=0x55eadd625b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:41.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.439+0000 7f299f3aab80 1 -- 172.21.15.27:0/3298428462 shutdown_connections 2024-05-10T11:30:41.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.440+0000 7f299f3aab80 1 -- 172.21.15.27:0/3298428462 wait complete. 2024-05-10T11:30:41.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.440+0000 7f299f3aab80 1 -- start start 2024-05-10T11:30:41.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.440+0000 7f299f3aab80 1 Processor -- start 2024-05-10T11:30:41.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.440+0000 7f298de00700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f29740b2b40 0x7f29740b5000 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55eadd6318e0 tx=0x7f29800e7040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:41.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.441+0000 7f299f3aab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd62b8f0 0x55eadd6371f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.441+0000 7f299f3aab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd630bc0 0x55eadd634590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.442+0000 7f299f3aab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd634ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.442+0000 7f299f3aab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55eadd637f20 con 0x55eadd6314e0 2024-05-10T11:30:41.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.442+0000 7f299f3aab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55eadd4e3b20 con 0x55eadd630bc0 2024-05-10T11:30:41.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.442+0000 7f299f3aab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55eadd6380a0 con 0x55eadd62b8f0 2024-05-10T11:30:41.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.442+0000 7f298e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd62b8f0 0x55eadd6371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.442+0000 7f298e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd62b8f0 0x55eadd6371f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:38548/0 (socket says 172.21.15.27:38548) 2024-05-10T11:30:41.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.442+0000 7f298e800700 1 -- 172.21.15.27:0/977361409 learned_addr learned my addr 172.21.15.27:0/977361409 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:41.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.442+0000 7f299f3aab80 10 client.0 _ll_register_callbacks cb 0x55eadd524c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:30:41.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.442+0000 7f298d400700 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd630bc0 0x55eadd634590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.443+0000 7f298e800700 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd634ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.443+0000 7f298d400700 1 -- 172.21.15.27:0/977361409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd62b8f0 msgr2=0x55eadd6371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:41.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.443+0000 7f298d400700 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd62b8f0 0x55eadd6371f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.443+0000 7f298d400700 1 -- 172.21.15.27:0/977361409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 msgr2=0x55eadd634ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:41.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.443+0000 7f298d400700 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd634ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.443+0000 7f298d400700 1 -- 172.21.15.27:0/977361409 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eadd543420 con 0x55eadd630bc0 2024-05-10T11:30:41.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.443+0000 7f298e800700 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd634ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:30:41.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.444+0000 7f298d400700 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd630bc0 0x55eadd634590 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f29901100e0 tx=0x7f297c005580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4ef366635fffa82 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:41.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.444+0000 7f296fe00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:30:41.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.444+0000 7f296fe00700 1 -- 172.21.15.27:0/977361409 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c0048e0 con 0x55eadd630bc0 2024-05-10T11:30:41.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.444+0000 7f296fe00700 1 -- 172.21.15.27:0/977361409 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7f297c004a80 con 0x55eadd630bc0 2024-05-10T11:30:41.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.444+0000 7f299f3aab80 1 -- 172.21.15.27:0/977361409 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2988004d50 con 0x55eadd630bc0 2024-05-10T11:30:41.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.445+0000 7f296fe00700 1 -- 172.21.15.27:0/977361409 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f297c004ce0 con 0x55eadd630bc0 2024-05-10T11:30:41.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.445+0000 7f299f3aab80 10 client.27569 fetch_fsmap learned FSMap version 45 2024-05-10T11:30:41.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.445+0000 7f299f3aab80 1 -- 172.21.15.27:0/977361409 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55eadd642790 con 0x55eadd630bc0 2024-05-10T11:30:41.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.446+0000 7f296fe00700 1 -- 172.21.15.27:0/977361409 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2988004d50 con 0x55eadd630bc0 2024-05-10T11:30:41.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.447+0000 7f296fe00700 1 -- 172.21.15.27:0/977361409 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f297c00f580 con 0x55eadd630bc0 2024-05-10T11:30:41.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.447+0000 7f299f3aab80 10 client.27569 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:30:41.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.447+0000 7f299f3aab80 10 client.27569 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:30:41.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.447+0000 7f299f3aab80 20 client.27569 populate_metadata read hostname 'smithi027' 2024-05-10T11:30:41.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.447+0000 7f299f3aab80 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55eadd644d70 0x55eadd647180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.447+0000 7f299f3aab80 4 client.27569 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:30:41.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.447+0000 7f299f3aab80 1 -- 172.21.15.27:0/977361409 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eadd5f8140 con 0x55eadd644d70 2024-05-10T11:30:41.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.447+0000 7f298e800700 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55eadd644d70 0x55eadd647180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.448+0000 7f298e800700 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55eadd644d70 0x55eadd647180 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a97b152479688516 server_cookie=738b1ac694eb0fa4 in_seq=0 out_seq=0 2024-05-10T11:30:41.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.448+0000 7f296fe00700 10 client.27569 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:30:41.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.449+0000 7f296fe00700 1 -- 172.21.15.27:0/977361409 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55eadd5f8140 con 0x55eadd644d70 2024-05-10T11:30:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.449+0000 7f296fe00700 10 client.27569 handle_command_reply: tid=0 2024-05-10T11:30:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.449+0000 7f299f3aab80 1 client.27569 shutdown 2024-05-10T11:30:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.449+0000 7f299f3aab80 1 -- 172.21.15.27:0/977361409 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd630bc0 msgr2=0x55eadd634590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.449+0000 7f299f3aab80 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd630bc0 0x55eadd634590 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f29901100e0 tx=0x7f297c005580 comp rx=0 tx=0).stop 2024-05-10T11:30:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.449+0000 7f299f3aab80 1 -- 172.21.15.27:0/977361409 shutdown_connections 2024-05-10T11:30:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.449+0000 7f299f3aab80 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55eadd644d70 0x55eadd647180 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.449+0000 7f299f3aab80 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd62b8f0 0x55eadd6371f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.449+0000 7f299f3aab80 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd634ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.449+0000 7f299f3aab80 1 --2- 172.21.15.27:0/977361409 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd630bc0 0x55eadd634590 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.449+0000 7f299f3aab80 1 -- 172.21.15.27:0/977361409 >> 172.21.15.27:0/977361409 conn(0x55eadd5a36d0 msgr2=0x55eadd625190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.449+0000 7f296fe00700 0 client.27569 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:30:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.450+0000 7f299f3aab80 1 -- 172.21.15.27:0/977361409 shutdown_connections 2024-05-10T11:30:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.450+0000 7f299f3aab80 1 -- 172.21.15.27:0/977361409 wait complete. 2024-05-10T11:30:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.451+0000 7f299f3aab80 20 client.27569 trim_cache size 0 max 16384 2024-05-10T11:30:41.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.470+0000 7f299f3aab80 1 Processor -- start 2024-05-10T11:30:41.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.470+0000 7f299f3aab80 1 -- start start 2024-05-10T11:30:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.470+0000 7f299f3aab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd62b8f0 0x55eadd62b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.470+0000 7f299f3aab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd641040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f299f3aab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd636ba0 0x55eadd632020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f299f3aab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f297000ef40 con 0x55eadd6314e0 2024-05-10T11:30:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f299f3aab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2970010a80 con 0x55eadd636ba0 2024-05-10T11:30:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f299f3aab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f297000f610 con 0x55eadd62b8f0 2024-05-10T11:30:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f298e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd62b8f0 0x55eadd62b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f298d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd641040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f298d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd641040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36004/0 (socket says 172.21.15.27:36004) 2024-05-10T11:30:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f298e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd62b8f0 0x55eadd62b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:38554/0 (socket says 172.21.15.27:38554) 2024-05-10T11:30:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f298d400700 1 -- 172.21.15.27:0/3849114111 learned_addr learned my addr 172.21.15.27:0/3849114111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:41.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f298e800700 1 --2- 172.21.15.27:0/3849114111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd636ba0 0x55eadd632020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f298d400700 1 -- 172.21.15.27:0/3849114111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd62b8f0 msgr2=0x55eadd62b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:41.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f298d400700 1 --2- 172.21.15.27:0/3849114111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd62b8f0 0x55eadd62b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f298d400700 1 -- 172.21.15.27:0/3849114111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd636ba0 msgr2=0x55eadd632020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:41.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f298d400700 1 --2- 172.21.15.27:0/3849114111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd636ba0 0x55eadd632020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.471+0000 7f298d400700 1 -- 172.21.15.27:0/3849114111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eadd632560 con 0x55eadd6314e0 2024-05-10T11:30:41.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.472+0000 7f298d400700 1 --2- 172.21.15.27:0/3849114111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd641040 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f297c00faf0 tx=0x7f297c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eed4dc9df6948f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:41.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.472+0000 7f297aa00700 1 -- 172.21.15.27:0/3849114111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c01b040 con 0x55eadd6314e0 2024-05-10T11:30:41.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.472+0000 7f297aa00700 1 -- 172.21.15.27:0/3849114111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f297c031070 con 0x55eadd6314e0 2024-05-10T11:30:41.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.472+0000 7f297aa00700 1 -- 172.21.15.27:0/3849114111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c00b610 con 0x55eadd6314e0 2024-05-10T11:30:41.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.473+0000 7f299f3aab80 1 -- 172.21.15.27:0/3849114111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 msgr2=0x55eadd641040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:41.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.473+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3849114111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd641040 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f297c00faf0 tx=0x7f297c000bc0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.473+0000 7f299f3aab80 1 -- 172.21.15.27:0/3849114111 shutdown_connections 2024-05-10T11:30:41.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.473+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3849114111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd62b8f0 0x55eadd62b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.473+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3849114111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd636ba0 0x55eadd632020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.473+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3849114111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd641040 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.473+0000 7f299f3aab80 1 -- 172.21.15.27:0/3849114111 >> 172.21.15.27:0/3849114111 conn(0x55eadd5a36d0 msgr2=0x55eadd631910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:41.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.473+0000 7f299f3aab80 1 -- 172.21.15.27:0/3849114111 shutdown_connections 2024-05-10T11:30:41.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.474+0000 7f299f3aab80 1 -- 172.21.15.27:0/3849114111 wait complete. 2024-05-10T11:30:41.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.474+0000 7f299f3aab80 1 -- start start 2024-05-10T11:30:41.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.474+0000 7f299f3aab80 1 Processor -- start 2024-05-10T11:30:41.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f299f3aab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd62b8f0 0x55eadd63d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f299f3aab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd64be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f299f3aab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd636ba0 0x55eadd64c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f299f3aab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f29700103b0 con 0x55eadd6314e0 2024-05-10T11:30:41.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f299f3aab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f297000fce0 con 0x55eadd62b8f0 2024-05-10T11:30:41.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f299f3aab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f297000cf00 con 0x55eadd636ba0 2024-05-10T11:30:41.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f299f3aab80 10 client.0 _ll_register_callbacks cb 0x55eadd524c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:30:41.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f298e800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd62b8f0 0x55eadd63d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f298e800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd62b8f0 0x55eadd63d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:55446/0 (socket says 172.21.15.27:55446) 2024-05-10T11:30:41.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f298e800700 1 -- 172.21.15.27:0/3028293628 learned_addr learned my addr 172.21.15.27:0/3028293628 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:41.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f298e800700 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd636ba0 0x55eadd64c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f298d400700 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd64be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f298e800700 1 -- 172.21.15.27:0/3028293628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd636ba0 msgr2=0x55eadd64c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:41.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f298e800700 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd636ba0 0x55eadd64c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f298e800700 1 -- 172.21.15.27:0/3028293628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 msgr2=0x55eadd64be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:41.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.475+0000 7f298e800700 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd64be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.476+0000 7f298e800700 1 -- 172.21.15.27:0/3028293628 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eadd64ca00 con 0x55eadd62b8f0 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.476+0000 7f298e800700 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd62b8f0 0x55eadd63d7c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f29880093f0 tx=0x7f297c005390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72b821db628ed9de server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.476+0000 7f296fe00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.476+0000 7f296fe00700 1 -- 172.21.15.27:0/3028293628 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298802a2a0 con 0x55eadd62b8f0 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.476+0000 7f299f3aab80 1 -- 172.21.15.27:0/3028293628 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f297c031070 con 0x55eadd62b8f0 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.476+0000 7f296fe00700 1 -- 172.21.15.27:0/3028293628 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7f298802a440 con 0x55eadd62b8f0 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.477+0000 7f296fe00700 1 -- 172.21.15.27:0/3028293628 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f298802a640 con 0x55eadd62b8f0 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.477+0000 7f299f3aab80 10 client.27575 fetch_fsmap learned FSMap version 45 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.477+0000 7f299f3aab80 1 -- 172.21.15.27:0/3028293628 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55eadd6af490 con 0x55eadd62b8f0 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.477+0000 7f296fe00700 1 -- 172.21.15.27:0/3028293628 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f297c031070 con 0x55eadd62b8f0 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.477+0000 7f296fe00700 1 -- 172.21.15.27:0/3028293628 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f2988021a10 con 0x55eadd62b8f0 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.477+0000 7f299f3aab80 10 client.27575 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.477+0000 7f299f3aab80 10 client.27575 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:30:41.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.477+0000 7f299f3aab80 20 client.27575 populate_metadata read hostname 'smithi027' 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.477+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55eadd6af9b0 0x55eadd6b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.477+0000 7f299f3aab80 4 client.27575 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.477+0000 7f299f3aab80 1 -- 172.21.15.27:0/3028293628 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eadd58b400 con 0x55eadd6af9b0 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.478+0000 7f298d400700 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55eadd6af9b0 0x55eadd6b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.478+0000 7f298d400700 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55eadd6af9b0 0x55eadd6b1da0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ad43334ff8b1840a server_cookie=43e5e15d42c70820 in_seq=0 out_seq=0 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.478+0000 7f296fe00700 10 client.27575 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.478+0000 7f296fe00700 1 -- 172.21.15.27:0/3028293628 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eadd58b400 con 0x55eadd6af9b0 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.478+0000 7f296fe00700 10 client.27575 handle_command_reply: tid=0 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.478+0000 7f299f3aab80 1 client.27575 shutdown 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.479+0000 7f299f3aab80 1 -- 172.21.15.27:0/3028293628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd62b8f0 msgr2=0x55eadd63d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.479+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd62b8f0 0x55eadd63d7c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f29880093f0 tx=0x7f297c005390 comp rx=0 tx=0).stop 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.479+0000 7f299f3aab80 1 -- 172.21.15.27:0/3028293628 shutdown_connections 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.479+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x55eadd6af9b0 0x55eadd6b1da0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.479+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eadd636ba0 0x55eadd64c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.479+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eadd62b8f0 0x55eadd63d7c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.479+0000 7f299f3aab80 1 --2- 172.21.15.27:0/3028293628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eadd6314e0 0x55eadd64be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.479+0000 7f296fe00700 0 client.27575 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:30:41.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.479+0000 7f299f3aab80 1 -- 172.21.15.27:0/3028293628 >> 172.21.15.27:0/3028293628 conn(0x55eadd5a36d0 msgr2=0x55eadd635b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:41.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.479+0000 7f299f3aab80 1 -- 172.21.15.27:0/3028293628 shutdown_connections 2024-05-10T11:30:41.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.479+0000 7f299f3aab80 1 -- 172.21.15.27:0/3028293628 wait complete. 2024-05-10T11:30:41.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.479+0000 7f299f3aab80 20 client.27575 trim_cache size 0 max 16384 2024-05-10T11:30:41.482 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:30:41.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.480+0000 7f2994a00700 1 -- 172.21.15.27:0/1906580234 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f29740b2b40 msgr2=0x7f29740b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:41.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.480+0000 7f2994a00700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f29740b2b40 0x7f29740b5000 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55eadd6318e0 tx=0x7f29800e7040 comp rx=0 tx=0).stop 2024-05-10T11:30:41.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.480+0000 7f2994a00700 1 -- 172.21.15.27:0/1906580234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2990101a90 msgr2=0x7f299019bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:41.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.480+0000 7f2994a00700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2990101a90 0x7f299019bdf0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f298000e780 tx=0x7f298000ea90 comp rx=0 tx=0).stop 2024-05-10T11:30:41.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.480+0000 7f2994a00700 1 -- 172.21.15.27:0/1906580234 shutdown_connections 2024-05-10T11:30:41.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.480+0000 7f2994a00700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f29740b2b40 0x7f29740b5000 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.480+0000 7f2994a00700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2990101a90 0x7f299019bdf0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.480+0000 7f2994a00700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2990109520 0x7f29901a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.480+0000 7f2994a00700 1 --2- 172.21.15.27:0/1906580234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2990102450 0x7f299019c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:41.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.480+0000 7f2994a00700 1 -- 172.21.15.27:0/1906580234 >> 172.21.15.27:0/1906580234 conn(0x7f2990078a10 msgr2=0x7f299010f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:41.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.480+0000 7f2994a00700 1 -- 172.21.15.27:0/1906580234 shutdown_connections 2024-05-10T11:30:41.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:41.480+0000 7f2994a00700 1 -- 172.21.15.27:0/1906580234 wait complete. 2024-05-10T11:30:41.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:41 smithi076 ceph-mon[34152]: pgmap v548: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.6 MiB/s wr, 627 op/s 2024-05-10T11:30:41.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:41 smithi189 ceph-mon[34842]: pgmap v548: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.6 MiB/s wr, 627 op/s 2024-05-10T11:30:42.174 INFO:tasks.workunit.client.0.smithi027.stdout: 10 88958 22.63 MB/sec execute 86 sec latency 163.128 ms 2024-05-10T11:30:43.174 INFO:tasks.workunit.client.0.smithi027.stdout: 10 89410 22.55 MB/sec execute 87 sec latency 167.516 ms 2024-05-10T11:30:43.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:43 smithi027 ceph-mon[30054]: pgmap v549: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 6.6 MiB/s wr, 423 op/s 2024-05-10T11:30:43.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:43 smithi076 ceph-mon[34152]: pgmap v549: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 6.6 MiB/s wr, 423 op/s 2024-05-10T11:30:43.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:43 smithi189 ceph-mon[34842]: pgmap v549: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 6.6 MiB/s wr, 423 op/s 2024-05-10T11:30:44.174 INFO:tasks.workunit.client.0.smithi027.stdout: 10 89882 22.59 MB/sec execute 88 sec latency 336.880 ms 2024-05-10T11:30:44.535 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:30:45.082 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:30:45.174 INFO:tasks.workunit.client.0.smithi027.stdout: 10 90585 22.70 MB/sec execute 89 sec latency 133.270 ms 2024-05-10T11:30:45.592 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:45 smithi027 ceph-mon[30054]: pgmap v550: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.9 MiB/s wr, 354 op/s 2024-05-10T11:30:45.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:45 smithi076 ceph-mon[34152]: pgmap v550: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.9 MiB/s wr, 354 op/s 2024-05-10T11:30:45.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:45 smithi189 ceph-mon[34842]: pgmap v550: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.9 MiB/s wr, 354 op/s 2024-05-10T11:30:46.174 INFO:tasks.workunit.client.0.smithi027.stdout: 10 90940 22.75 MB/sec execute 90 sec latency 183.866 ms 2024-05-10T11:30:47.175 INFO:tasks.workunit.client.0.smithi027.stdout: 10 91371 22.88 MB/sec execute 91 sec latency 151.333 ms 2024-05-10T11:30:47.564 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:47 smithi027 ceph-mon[30054]: pgmap v551: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 10 MiB/s wr, 547 op/s 2024-05-10T11:30:47.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.616+0000 7f3aa6000700 1 -- 172.21.15.27:0/4266965871 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3aa0104100 msgr2=0x7f3aa0108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:47.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.616+0000 7f3aa6000700 1 --2- 172.21.15.27:0/4266965871 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3aa0104100 0x7f3aa0108120 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c0099c0 tx=0x7f3a8c009cd0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.616+0000 7f3aa6000700 1 -- 172.21.15.27:0/4266965871 shutdown_connections 2024-05-10T11:30:47.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.616+0000 7f3aa6000700 1 --2- 172.21.15.27:0/4266965871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0108850 0x7f3aa01103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.616+0000 7f3aa6000700 1 --2- 172.21.15.27:0/4266965871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aa0103750 0x7f3aa0103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.616+0000 7f3aa6000700 1 --2- 172.21.15.27:0/4266965871 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3aa0104100 0x7f3aa0108120 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.616+0000 7f3aa6000700 1 -- 172.21.15.27:0/4266965871 >> 172.21.15.27:0/4266965871 conn(0x7f3aa007a600 msgr2=0x7f3aa007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:47.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.616+0000 7f3aa6000700 1 -- 172.21.15.27:0/4266965871 shutdown_connections 2024-05-10T11:30:47.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.616+0000 7f3aa6000700 1 -- 172.21.15.27:0/4266965871 wait complete. 2024-05-10T11:30:47.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.618+0000 7f3aa6000700 1 Processor -- start 2024-05-10T11:30:47.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.618+0000 7f3aa6000700 1 -- start start 2024-05-10T11:30:47.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.618+0000 7f3aa6000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3aa0103750 0x7f3aa01a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.618+0000 7f3aa6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0104100 0x7f3aa01a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.618+0000 7f3aa6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aa0108850 0x7f3aa019bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.618+0000 7f3aa6000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa0113660 con 0x7f3aa0108850 2024-05-10T11:30:47.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.618+0000 7f3aa6000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01134e0 con 0x7f3aa0103750 2024-05-10T11:30:47.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.618+0000 7f3aa6000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01137e0 con 0x7f3aa0104100 2024-05-10T11:30:47.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.619+0000 7f3a9ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0104100 0x7f3aa01a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.619+0000 7f3a9ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0104100 0x7f3aa01a1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51558/0 (socket says 172.21.15.27:51558) 2024-05-10T11:30:47.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.619+0000 7f3a9ea00700 1 -- 172.21.15.27:0/1422665895 learned_addr learned my addr 172.21.15.27:0/1422665895 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:47.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.619+0000 7f3a9fe00700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aa0108850 0x7f3aa019bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.619+0000 7f3a9ea00700 1 -- 172.21.15.27:0/1422665895 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3aa0103750 msgr2=0x7f3aa01a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:47.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.619+0000 7f3a9f400700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3aa0103750 0x7f3aa01a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.619+0000 7f3a9ea00700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3aa0103750 0x7f3aa01a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.619+0000 7f3a9ea00700 1 -- 172.21.15.27:0/1422665895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aa0108850 msgr2=0x7f3aa019bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:47.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.619+0000 7f3a9ea00700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aa0108850 0x7f3aa019bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.619+0000 7f3a9ea00700 1 -- 172.21.15.27:0/1422665895 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa019c400 con 0x7f3aa0104100 2024-05-10T11:30:47.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.620+0000 7f3a9f400700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3aa0103750 0x7f3aa01a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:30:47.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.620+0000 7f3a9ea00700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0104100 0x7f3aa01a1f90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c005910 tx=0x7f3a8c004320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:47.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.620+0000 7f3a97e00700 1 -- 172.21.15.27:0/1422665895 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c026070 con 0x7f3aa0104100 2024-05-10T11:30:47.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.620+0000 7f3aa6000700 1 -- 172.21.15.27:0/1422665895 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa019c690 con 0x7f3aa0104100 2024-05-10T11:30:47.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.620+0000 7f3a97e00700 1 -- 172.21.15.27:0/1422665895 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a8c003c70 con 0x7f3aa0104100 2024-05-10T11:30:47.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.620+0000 7f3a97e00700 1 -- 172.21.15.27:0/1422665895 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c00fb50 con 0x7f3aa0104100 2024-05-10T11:30:47.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.620+0000 7f3aa6000700 1 -- 172.21.15.27:0/1422665895 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa01a8800 con 0x7f3aa0104100 2024-05-10T11:30:47.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.626+0000 7f3a97e00700 1 -- 172.21.15.27:0/1422665895 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3a8c029070 con 0x7f3aa0104100 2024-05-10T11:30:47.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.627+0000 7f3ab0a89b80 1 Processor -- start 2024-05-10T11:30:47.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.627+0000 7f3ab0a89b80 1 -- start start 2024-05-10T11:30:47.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.627+0000 7f3ab0a89b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64dffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.627+0000 7f3ab0a89b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64e04e0 0x55a4e64da2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.627+0000 7f3ab0a89b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64da8f0 0x55a4e64dad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3ab0a89b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a4e642ca20 con 0x55a4e64da8f0 2024-05-10T11:30:47.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3ab0a89b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a4e63ca760 con 0x55a4e64e04e0 2024-05-10T11:30:47.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3ab0a89b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a4e64289e0 con 0x55a4e64dfbc0 2024-05-10T11:30:47.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3a9fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64dffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3a9f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64e04e0 0x55a4e64da2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3a9fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64dffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51562/0 (socket says 172.21.15.27:51562) 2024-05-10T11:30:47.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3a9fe00700 1 -- 172.21.15.27:0/3223335707 learned_addr learned my addr 172.21.15.27:0/3223335707 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:47.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3a9ea00700 1 --2- 172.21.15.27:0/3223335707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64da8f0 0x55a4e64dad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3a9fe00700 1 -- 172.21.15.27:0/3223335707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64e04e0 msgr2=0x55a4e64da2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:47.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3a9fe00700 1 --2- 172.21.15.27:0/3223335707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64e04e0 0x55a4e64da2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3a9fe00700 1 -- 172.21.15.27:0/3223335707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64da8f0 msgr2=0x55a4e64dad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:47.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3a9fe00700 1 --2- 172.21.15.27:0/3223335707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64da8f0 0x55a4e64dad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3a9fe00700 1 -- 172.21.15.27:0/3223335707 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4e63798d0 con 0x55a4e64dfbc0 2024-05-10T11:30:47.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.628+0000 7f3a97e00700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3a840b29f0 0x7f3a840b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.629+0000 7f3a9fe00700 1 --2- 172.21.15.27:0/3223335707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64dffa0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3a98011630 tx=0x7f3a98009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbba98393fd9c4d server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:47.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.629+0000 7f3a97e00700 1 -- 172.21.15.27:0/1422665895 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7f3a8c0e31d0 con 0x7f3aa0104100 2024-05-10T11:30:47.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.629+0000 7f3a9f400700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3a840b29f0 0x7f3a840b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.629+0000 7f3a94c00700 1 -- 172.21.15.27:0/3223335707 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98003a00 con 0x55a4e64dfbc0 2024-05-10T11:30:47.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.629+0000 7f3a94c00700 1 -- 172.21.15.27:0/3223335707 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a9801a070 con 0x55a4e64dfbc0 2024-05-10T11:30:47.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.629+0000 7f3a94c00700 1 -- 172.21.15.27:0/3223335707 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98007760 con 0x55a4e64dfbc0 2024-05-10T11:30:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.630+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3223335707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 msgr2=0x55a4e64dffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.630+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/3223335707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64dffa0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3a98011630 tx=0x7f3a98009920 comp rx=0 tx=0).stop 2024-05-10T11:30:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.630+0000 7f3a9f400700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3a840b29f0 0x7f3a840b4eb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55a4e64e08e0 tx=0x7f3a90009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.631+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3223335707 shutdown_connections 2024-05-10T11:30:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.631+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/3223335707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64dffa0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.631+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/3223335707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64da8f0 0x55a4e64dad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.631+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/3223335707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64e04e0 0x55a4e64da2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.631+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3223335707 >> 172.21.15.27:0/3223335707 conn(0x55a4e64526d0 msgr2=0x55a4e64d4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:47.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.631+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3223335707 shutdown_connections 2024-05-10T11:30:47.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.631+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3223335707 wait complete. 2024-05-10T11:30:47.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.632+0000 7f3ab0a89b80 1 -- start start 2024-05-10T11:30:47.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.632+0000 7f3ab0a89b80 1 Processor -- start 2024-05-10T11:30:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.632+0000 7f3ab0a89b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64da8f0 0x55a4e64e6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.632+0000 7f3ab0a89b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64e35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.632+0000 7f3ab0a89b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64e04e0 0x55a4e64e3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.632+0000 7f3ab0a89b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a4e64e6ec0 con 0x55a4e64e04e0 2024-05-10T11:30:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.632+0000 7f3ab0a89b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a4e6392b20 con 0x55a4e64da8f0 2024-05-10T11:30:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3ab0a89b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a4e64e7040 con 0x55a4e64dfbc0 2024-05-10T11:30:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3ab0a89b80 10 client.0 _ll_register_callbacks cb 0x55a4e63d3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a9ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64e35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a9f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64da8f0 0x55a4e64e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a9ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64e35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51572/0 (socket says 172.21.15.27:51572) 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a9ea00700 1 -- 172.21.15.27:0/1877839163 learned_addr learned my addr 172.21.15.27:0/1877839163 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a9fe00700 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64e04e0 0x55a4e64e3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a9ea00700 1 -- 172.21.15.27:0/1877839163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64da8f0 msgr2=0x55a4e64e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a9ea00700 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64da8f0 0x55a4e64e6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a9ea00700 1 -- 172.21.15.27:0/1877839163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64e04e0 msgr2=0x55a4e64e3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a9ea00700 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64e04e0 0x55a4e64e3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a9ea00700 1 -- 172.21.15.27:0/1877839163 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4e63f2420 con 0x55a4e64dfbc0 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a9fe00700 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64e04e0 0x55a4e64e3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a9ea00700 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64e35e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c0a9e10 tx=0x7f3a8c00f530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3651b8e6053ddf80 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.633+0000 7f3a89600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:30:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.634+0000 7f3a89600700 1 -- 172.21.15.27:0/1877839163 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c0a9a60 con 0x55a4e64dfbc0 2024-05-10T11:30:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.634+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1877839163 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a9801a070 con 0x55a4e64dfbc0 2024-05-10T11:30:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.634+0000 7f3a89600700 1 -- 172.21.15.27:0/1877839163 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7f3a8c0aa580 con 0x55a4e64dfbc0 2024-05-10T11:30:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.636+0000 7f3a89600700 1 -- 172.21.15.27:0/1877839163 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f3a8c009e60 con 0x55a4e64dfbc0 2024-05-10T11:30:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.636+0000 7f3ab0a89b80 10 client.27604 fetch_fsmap learned FSMap version 45 2024-05-10T11:30:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.636+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1877839163 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55a4e64e6f20 con 0x55a4e64dfbc0 2024-05-10T11:30:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.636+0000 7f3a89600700 1 -- 172.21.15.27:0/1877839163 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a9801a070 con 0x55a4e64dfbc0 2024-05-10T11:30:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.636+0000 7f3a89600700 1 -- 172.21.15.27:0/1877839163 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f3a8c0b2a50 con 0x55a4e64dfbc0 2024-05-10T11:30:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.636+0000 7f3ab0a89b80 10 client.27604 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:30:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.636+0000 7f3ab0a89b80 10 client.27604 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:30:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.636+0000 7f3ab0a89b80 20 client.27604 populate_metadata read hostname 'smithi027' 2024-05-10T11:30:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.636+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a4e64f3d90 0x55a4e64f61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.636+0000 7f3ab0a89b80 4 client.27604 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:30:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.636+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1877839163 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4e64a7140 con 0x55a4e64f3d90 2024-05-10T11:30:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.637+0000 7f3a9f400700 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a4e64f3d90 0x55a4e64f61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.637+0000 7f3a89600700 10 client.27604 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:30:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.638+0000 7f3a9f400700 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a4e64f3d90 0x55a4e64f61b0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=15344a97e34d882 server_cookie=7aa8584919d8e7cf in_seq=0 out_seq=0 2024-05-10T11:30:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.639+0000 7f3a89600700 1 -- 172.21.15.27:0/1877839163 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a4e64a7140 con 0x55a4e64f3d90 2024-05-10T11:30:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.639+0000 7f3a89600700 10 client.27604 handle_command_reply: tid=0 2024-05-10T11:30:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.639+0000 7f3ab0a89b80 1 client.27604 shutdown 2024-05-10T11:30:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.639+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1877839163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 msgr2=0x55a4e64e35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.639+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64e35e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c0a9e10 tx=0x7f3a8c00f530 comp rx=0 tx=0).stop 2024-05-10T11:30:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.640+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1877839163 shutdown_connections 2024-05-10T11:30:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.640+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a4e64f3d90 0x55a4e64f61b0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.640+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64e04e0 0x55a4e64e3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.640+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64da8f0 0x55a4e64e6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.640+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/1877839163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64e35e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.640+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1877839163 >> 172.21.15.27:0/1877839163 conn(0x55a4e64526d0 msgr2=0x55a4e64d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.640+0000 7f3a89600700 0 client.27604 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:30:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.640+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1877839163 shutdown_connections 2024-05-10T11:30:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.640+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1877839163 wait complete. 2024-05-10T11:30:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.640+0000 7f3ab0a89b80 20 client.27604 trim_cache size 0 max 16384 2024-05-10T11:30:47.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.660+0000 7f3ab0a89b80 1 Processor -- start 2024-05-10T11:30:47.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.660+0000 7f3ab0a89b80 1 -- start start 2024-05-10T11:30:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.660+0000 7f3ab0a89b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64da8f0 0x55a4e64da2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.660+0000 7f3ab0a89b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64eff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.660+0000 7f3ab0a89b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64e04e0 0x55a4e64d4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.660+0000 7f3ab0a89b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01b1710 con 0x55a4e64e04e0 2024-05-10T11:30:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.660+0000 7f3ab0a89b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01b3250 con 0x55a4e64da8f0 2024-05-10T11:30:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.660+0000 7f3ab0a89b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01b1de0 con 0x55a4e64dfbc0 2024-05-10T11:30:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.661+0000 7f3a9f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64da8f0 0x55a4e64da2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.661+0000 7f3a9f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64da8f0 0x55a4e64da2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51898/0 (socket says 172.21.15.27:51898) 2024-05-10T11:30:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.661+0000 7f3a9f400700 1 -- 172.21.15.27:0/1025225832 learned_addr learned my addr 172.21.15.27:0/1025225832 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.661+0000 7f3a9fe00700 1 --2- 172.21.15.27:0/1025225832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64e04e0 0x55a4e64d4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.661+0000 7f3a9ea00700 1 --2- 172.21.15.27:0/1025225832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64eff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.661+0000 7f3a9f400700 1 -- 172.21.15.27:0/1025225832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 msgr2=0x55a4e64eff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:47.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.661+0000 7f3a9f400700 1 --2- 172.21.15.27:0/1025225832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64eff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.661+0000 7f3a9f400700 1 -- 172.21.15.27:0/1025225832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64e04e0 msgr2=0x55a4e64d4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:47.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.661+0000 7f3a9f400700 1 --2- 172.21.15.27:0/1025225832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64e04e0 0x55a4e64d4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.661+0000 7f3a9f400700 1 -- 172.21.15.27:0/1025225832 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4e64e01f0 con 0x55a4e64da8f0 2024-05-10T11:30:47.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.661+0000 7f3a9ea00700 1 --2- 172.21.15.27:0/1025225832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64eff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:30:47.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.661+0000 7f3a9f400700 1 --2- 172.21.15.27:0/1025225832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64da8f0 0x55a4e64da2d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3a90015010 tx=0x7f3a90016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87006bdd3d539ae7 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:47.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.662+0000 7f3a8be00700 1 -- 172.21.15.27:0/1025225832 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a90030070 con 0x55a4e64da8f0 2024-05-10T11:30:47.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.662+0000 7f3a8be00700 1 -- 172.21.15.27:0/1025225832 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a90003d10 con 0x55a4e64da8f0 2024-05-10T11:30:47.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.662+0000 7f3a8be00700 1 -- 172.21.15.27:0/1025225832 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a90028a90 con 0x55a4e64da8f0 2024-05-10T11:30:47.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.663+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1025225832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64da8f0 msgr2=0x55a4e64da2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:47.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.663+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/1025225832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64da8f0 0x55a4e64da2d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3a90015010 tx=0x7f3a90016000 comp rx=0 tx=0).stop 2024-05-10T11:30:47.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.663+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1025225832 shutdown_connections 2024-05-10T11:30:47.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.663+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/1025225832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64e04e0 0x55a4e64d4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.663+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/1025225832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64da8f0 0x55a4e64da2d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.663+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/1025225832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64dfbc0 0x55a4e64eff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.663+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1025225832 >> 172.21.15.27:0/1025225832 conn(0x55a4e64526d0 msgr2=0x55a4e64de740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:47.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.663+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1025225832 shutdown_connections 2024-05-10T11:30:47.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.663+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/1025225832 wait complete. 2024-05-10T11:30:47.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.663+0000 7f3ab0a89b80 1 -- start start 2024-05-10T11:30:47.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.664+0000 7f3ab0a89b80 1 Processor -- start 2024-05-10T11:30:47.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.664+0000 7f3ab0a89b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64da8f0 0x55a4e64fb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.664+0000 7f3ab0a89b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64dfbc0 0x55a4e64fb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.664+0000 7f3ab0a89b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64e04e0 0x55a4e64fbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.664+0000 7f3ab0a89b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01b2b80 con 0x55a4e64dfbc0 2024-05-10T11:30:47.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.664+0000 7f3ab0a89b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01b24b0 con 0x55a4e64e04e0 2024-05-10T11:30:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.664+0000 7f3ab0a89b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01af6d0 con 0x55a4e64da8f0 2024-05-10T11:30:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3ab0a89b80 10 client.0 _ll_register_callbacks cb 0x55a4e63d3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:30:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3a9f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64da8f0 0x55a4e64fb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3a9fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64e04e0 0x55a4e64fbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3a9f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64da8f0 0x55a4e64fb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51602/0 (socket says 172.21.15.27:51602) 2024-05-10T11:30:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3a9fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64e04e0 0x55a4e64fbc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51910/0 (socket says 172.21.15.27:51910) 2024-05-10T11:30:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3a9f400700 1 -- 172.21.15.27:0/3815973603 learned_addr learned my addr 172.21.15.27:0/3815973603 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3a9ea00700 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64dfbc0 0x55a4e64fb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3a9f400700 1 -- 172.21.15.27:0/3815973603 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64e04e0 msgr2=0x55a4e64fbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3a9f400700 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64e04e0 0x55a4e64fbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3a9f400700 1 -- 172.21.15.27:0/3815973603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64dfbc0 msgr2=0x55a4e64fb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3a9f400700 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64dfbc0 0x55a4e64fb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3a9f400700 1 -- 172.21.15.27:0/3815973603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4e655d610 con 0x55a4e64da8f0 2024-05-10T11:30:47.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.665+0000 7f3a9fe00700 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64e04e0 0x55a4e64fbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:30:47.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.666+0000 7f3a9f400700 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64da8f0 0x55a4e64fb190 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3a900286b0 tx=0x7f3a90003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cae4703b8b854b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:47.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.666+0000 7f3a89600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:30:47.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.666+0000 7f3a89600700 1 -- 172.21.15.27:0/3815973603 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a90030070 con 0x55a4e64da8f0 2024-05-10T11:30:47.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.666+0000 7f3a89600700 1 -- 172.21.15.27:0/3815973603 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7f3a9002a630 con 0x55a4e64da8f0 2024-05-10T11:30:47.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.667+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3815973603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a90003d10 con 0x55a4e64da8f0 2024-05-10T11:30:47.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.667+0000 7f3a89600700 1 -- 172.21.15.27:0/3815973603 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f3a9002a830 con 0x55a4e64da8f0 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.667+0000 7f3ab0a89b80 10 client.27610 fetch_fsmap learned FSMap version 45 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.667+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3815973603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55a4e655e830 con 0x55a4e64da8f0 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.667+0000 7f3a89600700 1 -- 172.21.15.27:0/3815973603 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a90003d10 con 0x55a4e64da8f0 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.668+0000 7f3a89600700 1 -- 172.21.15.27:0/3815973603 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f3a90027070 con 0x55a4e64da8f0 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.668+0000 7f3ab0a89b80 10 client.27610 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.668+0000 7f3ab0a89b80 10 client.27610 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.668+0000 7f3ab0a89b80 20 client.27610 populate_metadata read hostname 'smithi027' 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.668+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a4e655ede0 0x55a4e6561200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.668+0000 7f3ab0a89b80 4 client.27610 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.668+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3815973603 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a4e643a400 con 0x55a4e655ede0 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.669+0000 7f3a9ea00700 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a4e655ede0 0x55a4e6561200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.669+0000 7f3a9ea00700 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a4e655ede0 0x55a4e6561200 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac9bf39232badc72 server_cookie=b8803dc674b13952 in_seq=0 out_seq=0 2024-05-10T11:30:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.669+0000 7f3a89600700 10 client.27610 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:30:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.670+0000 7f3a89600700 1 -- 172.21.15.27:0/3815973603 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a4e643a400 con 0x55a4e655ede0 2024-05-10T11:30:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.670+0000 7f3a89600700 10 client.27610 handle_command_reply: tid=0 2024-05-10T11:30:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.670+0000 7f3ab0a89b80 1 client.27610 shutdown 2024-05-10T11:30:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.670+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3815973603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64da8f0 msgr2=0x55a4e64fb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.670+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64da8f0 0x55a4e64fb190 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3a900286b0 tx=0x7f3a90003f90 comp rx=0 tx=0).stop 2024-05-10T11:30:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.671+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3815973603 shutdown_connections 2024-05-10T11:30:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.671+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x55a4e655ede0 0x55a4e6561200 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.671+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a4e64da8f0 0x55a4e64fb190 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.671+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4e64e04e0 0x55a4e64fbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.671+0000 7f3a89600700 0 client.27610 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:30:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.671+0000 7f3ab0a89b80 1 --2- 172.21.15.27:0/3815973603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4e64dfbc0 0x55a4e64fb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.671+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3815973603 >> 172.21.15.27:0/3815973603 conn(0x55a4e64526d0 msgr2=0x55a4e64defd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:47.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.671+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3815973603 shutdown_connections 2024-05-10T11:30:47.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.671+0000 7f3ab0a89b80 1 -- 172.21.15.27:0/3815973603 wait complete. 2024-05-10T11:30:47.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.672+0000 7f3ab0a89b80 20 client.27610 trim_cache size 0 max 16384 2024-05-10T11:30:47.675 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:30:47.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.673+0000 7f3aa6000700 1 -- 172.21.15.27:0/1422665895 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3a840b29f0 msgr2=0x7f3a840b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:47.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.673+0000 7f3aa6000700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3a840b29f0 0x7f3a840b4eb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55a4e64e08e0 tx=0x7f3a90009260 comp rx=0 tx=0).stop 2024-05-10T11:30:47.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.673+0000 7f3aa6000700 1 -- 172.21.15.27:0/1422665895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0104100 msgr2=0x7f3aa01a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:47.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.673+0000 7f3aa6000700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0104100 0x7f3aa01a1f90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c005910 tx=0x7f3a8c004320 comp rx=0 tx=0).stop 2024-05-10T11:30:47.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.673+0000 7f3aa6000700 1 -- 172.21.15.27:0/1422665895 shutdown_connections 2024-05-10T11:30:47.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.673+0000 7f3aa6000700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3a840b29f0 0x7f3a840b4eb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.673+0000 7f3aa6000700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aa0108850 0x7f3aa019bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.673+0000 7f3aa6000700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3aa0103750 0x7f3aa01a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.673+0000 7f3aa6000700 1 --2- 172.21.15.27:0/1422665895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0104100 0x7f3aa01a1f90 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.673+0000 7f3aa6000700 1 -- 172.21.15.27:0/1422665895 >> 172.21.15.27:0/1422665895 conn(0x7f3aa007a600 msgr2=0x7f3aa0101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.674+0000 7f3aa6000700 1 -- 172.21.15.27:0/1422665895 shutdown_connections 2024-05-10T11:30:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:47.674+0000 7f3aa6000700 1 -- 172.21.15.27:0/1422665895 wait complete. 2024-05-10T11:30:47.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:47 smithi076 ceph-mon[34152]: pgmap v551: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 10 MiB/s wr, 547 op/s 2024-05-10T11:30:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:47 smithi189 ceph-mon[34842]: pgmap v551: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 10 MiB/s wr, 547 op/s 2024-05-10T11:30:48.174 INFO:tasks.workunit.client.0.smithi027.stdout: 10 91893 22.99 MB/sec execute 92 sec latency 163.804 ms 2024-05-10T11:30:48.716 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:30:49.175 INFO:tasks.workunit.client.0.smithi027.stdout: 10 92152 22.85 MB/sec execute 93 sec latency 189.411 ms 2024-05-10T11:30:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:30:49.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:49 smithi027 ceph-mon[30054]: pgmap v552: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 7.6 MiB/s wr, 369 op/s 2024-05-10T11:30:49.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:49 smithi076 ceph-mon[34152]: pgmap v552: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 7.6 MiB/s wr, 369 op/s 2024-05-10T11:30:49.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:49 smithi189 ceph-mon[34842]: pgmap v552: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 7.6 MiB/s wr, 369 op/s 2024-05-10T11:30:50.175 INFO:tasks.workunit.client.0.smithi027.stdout: 10 92460 22.73 MB/sec execute 94 sec latency 232.419 ms 2024-05-10T11:30:51.175 INFO:tasks.workunit.client.0.smithi027.stdout: 10 92737 22.56 MB/sec execute 95 sec latency 339.434 ms 2024-05-10T11:30:51.317 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:51 smithi027 ceph-mon[30054]: pgmap v553: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 10 MiB/s wr, 519 op/s 2024-05-10T11:30:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:51 smithi076 ceph-mon[34152]: pgmap v553: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 10 MiB/s wr, 519 op/s 2024-05-10T11:30:51.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:51 smithi189 ceph-mon[34842]: pgmap v553: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 10 MiB/s wr, 519 op/s 2024-05-10T11:30:51.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.970+0000 7fb25d800700 1 -- 172.21.15.27:0/550637036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb258103550 msgr2=0x7fb2581039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:51.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.970+0000 7fb25d800700 1 --2- 172.21.15.27:0/550637036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb258103550 0x7fb2581039b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb24800b0a0 tx=0x7fb24800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:30:51.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.971+0000 7fb25d800700 1 -- 172.21.15.27:0/550637036 shutdown_connections 2024-05-10T11:30:51.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.971+0000 7fb25d800700 1 --2- 172.21.15.27:0/550637036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb258103ef0 0x7fb2581103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:51.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.971+0000 7fb25d800700 1 --2- 172.21.15.27:0/550637036 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb258108460 0x7fb258108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:51.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.971+0000 7fb25d800700 1 --2- 172.21.15.27:0/550637036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb258103550 0x7fb2581039b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:51.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.971+0000 7fb25d800700 1 -- 172.21.15.27:0/550637036 >> 172.21.15.27:0/550637036 conn(0x7fb25807a600 msgr2=0x7fb25807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:51.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.971+0000 7fb25d800700 1 -- 172.21.15.27:0/550637036 shutdown_connections 2024-05-10T11:30:51.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.971+0000 7fb25d800700 1 -- 172.21.15.27:0/550637036 wait complete. 2024-05-10T11:30:51.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.973+0000 7fb25d800700 1 Processor -- start 2024-05-10T11:30:51.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.973+0000 7fb25d800700 1 -- start start 2024-05-10T11:30:51.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.973+0000 7fb25d800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb258103550 0x7fb2580757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:51.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.973+0000 7fb25d800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb258103ef0 0x7fb258075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:51.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.973+0000 7fb25d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb258108460 0x7fb258076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:51.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.974+0000 7fb25d800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581137c0 con 0x7fb258103550 2024-05-10T11:30:51.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.974+0000 7fb25d800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb258113640 con 0x7fb258103ef0 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.974+0000 7fb25d800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb258113940 con 0x7fb258108460 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.974+0000 7fb256000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb258103ef0 0x7fb258075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.974+0000 7fb256000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb258103ef0 0x7fb258075ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51934/0 (socket says 172.21.15.27:51934) 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.974+0000 7fb256000700 1 -- 172.21.15.27:0/1572993440 learned_addr learned my addr 172.21.15.27:0/1572993440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.974+0000 7fb256000700 1 -- 172.21.15.27:0/1572993440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb258108460 msgr2=0x7fb258076220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.974+0000 7fb256000700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb258108460 0x7fb258076220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.974+0000 7fb256000700 1 -- 172.21.15.27:0/1572993440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb258103550 msgr2=0x7fb2580757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.975+0000 7fb256a00700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb258103550 0x7fb2580757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.975+0000 7fb256000700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb258103550 0x7fb2580757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.975+0000 7fb256000700 1 -- 172.21.15.27:0/1572993440 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2581a8140 con 0x7fb258103ef0 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.975+0000 7fb256000700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb258103ef0 0x7fb258075ce0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb248000f80 tx=0x7fb248004e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.975+0000 7fb24f400700 1 -- 172.21.15.27:0/1572993440 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb248021070 con 0x7fb258103ef0 2024-05-10T11:30:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.976+0000 7fb24f400700 1 -- 172.21.15.27:0/1572993440 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb248007cc0 con 0x7fb258103ef0 2024-05-10T11:30:51.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.976+0000 7fb24f400700 1 -- 172.21.15.27:0/1572993440 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb248004830 con 0x7fb258103ef0 2024-05-10T11:30:51.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.976+0000 7fb25d800700 1 -- 172.21.15.27:0/1572993440 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb258199950 con 0x7fb258103ef0 2024-05-10T11:30:51.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.976+0000 7fb25d800700 1 -- 172.21.15.27:0/1572993440 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb258199e40 con 0x7fb258103ef0 2024-05-10T11:30:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.981+0000 7fb24f400700 1 -- 172.21.15.27:0/1572993440 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb24801d450 con 0x7fb258103ef0 2024-05-10T11:30:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.983+0000 7fb268137b80 1 Processor -- start 2024-05-10T11:30:51.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.984+0000 7fb268137b80 1 -- start start 2024-05-10T11:30:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.985+0000 7fb268137b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e23060 0x55c726e23440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.985+0000 7fb268137b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e27490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.985+0000 7fb268137b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e279d0 0x55c726e27e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.985+0000 7fb268137b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c726d75a20 con 0x55c726e279d0 2024-05-10T11:30:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.985+0000 7fb268137b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c726d13760 con 0x55c726e23980 2024-05-10T11:30:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.987+0000 7fb268137b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c726d719e0 con 0x55c726e23060 2024-05-10T11:30:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.987+0000 7fb256000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e279d0 0x55c726e27e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.985+0000 7fb257400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e23060 0x55c726e23440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.987+0000 7fb257400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e23060 0x55c726e23440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:51640/0 (socket says 172.21.15.27:51640) 2024-05-10T11:30:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.987+0000 7fb257400700 1 -- 172.21.15.27:0/1233451155 learned_addr learned my addr 172.21.15.27:0/1233451155 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:51.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.987+0000 7fb256a00700 1 --2- 172.21.15.27:0/1233451155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e27490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:51.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.988+0000 7fb256a00700 1 -- 172.21.15.27:0/1233451155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e23060 msgr2=0x55c726e23440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:51.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.988+0000 7fb256a00700 1 --2- 172.21.15.27:0/1233451155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e23060 0x55c726e23440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:51.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.988+0000 7fb256a00700 1 -- 172.21.15.27:0/1233451155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e279d0 msgr2=0x55c726e27e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:51.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.988+0000 7fb256a00700 1 --2- 172.21.15.27:0/1233451155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e279d0 0x55c726e27e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:51.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.988+0000 7fb256a00700 1 -- 172.21.15.27:0/1233451155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c726cc28d0 con 0x55c726e23980 2024-05-10T11:30:51.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.988+0000 7fb256a00700 1 --2- 172.21.15.27:0/1233451155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e27490 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb24000a870 tx=0x7fb24000ab80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a4f6ca63f8f7ba server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:51.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.988+0000 7fb24f400700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb23c0b2aa0 0x7fb23c0b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:51.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.989+0000 7fb24f400700 1 -- 172.21.15.27:0/1572993440 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7fb24800f020 con 0x7fb258103ef0 2024-05-10T11:30:51.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.990+0000 7fb257400700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb23c0b2aa0 0x7fb23c0b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:51.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.990+0000 7fb247e00700 1 -- 172.21.15.27:0/1233451155 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb240015070 con 0x55c726e23980 2024-05-10T11:30:51.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.990+0000 7fb247e00700 1 -- 172.21.15.27:0/1233451155 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb240011cb0 con 0x55c726e23980 2024-05-10T11:30:51.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.990+0000 7fb247e00700 1 -- 172.21.15.27:0/1233451155 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24000cdf0 con 0x55c726e23980 2024-05-10T11:30:51.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.991+0000 7fb268137b80 1 -- 172.21.15.27:0/1233451155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 msgr2=0x55c726e27490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.991+0000 7fb268137b80 1 --2- 172.21.15.27:0/1233451155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e27490 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb24000a870 tx=0x7fb24000ab80 comp rx=0 tx=0).stop 2024-05-10T11:30:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.992+0000 7fb268137b80 1 -- 172.21.15.27:0/1233451155 shutdown_connections 2024-05-10T11:30:51.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.992+0000 7fb268137b80 1 --2- 172.21.15.27:0/1233451155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e23060 0x55c726e23440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:51.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.992+0000 7fb268137b80 1 --2- 172.21.15.27:0/1233451155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e279d0 0x55c726e27e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:51.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.992+0000 7fb268137b80 1 --2- 172.21.15.27:0/1233451155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e27490 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:51.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.992+0000 7fb268137b80 1 -- 172.21.15.27:0/1233451155 >> 172.21.15.27:0/1233451155 conn(0x55c726d51b60 msgr2=0x55c726e1d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:51.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.994+0000 7fb268137b80 1 -- 172.21.15.27:0/1233451155 shutdown_connections 2024-05-10T11:30:51.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.994+0000 7fb268137b80 1 -- 172.21.15.27:0/1233451155 wait complete. 2024-05-10T11:30:51.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.994+0000 7fb257400700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb23c0b2aa0 0x7fb23c0b4f60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb250009d50 tx=0x7fb250009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:51.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.995+0000 7fb268137b80 1 -- start start 2024-05-10T11:30:51.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.995+0000 7fb268137b80 1 Processor -- start 2024-05-10T11:30:51.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.996+0000 7fb268137b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e23060 0x55c726e2ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:51.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.996+0000 7fb268137b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e2c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:51.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.996+0000 7fb268137b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e279d0 0x55c726e2c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:51.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.996+0000 7fb256000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e2c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:51.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.996+0000 7fb256000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e2c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51958/0 (socket says 172.21.15.27:51958) 2024-05-10T11:30:51.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.996+0000 7fb256000700 1 -- 172.21.15.27:0/731627870 learned_addr learned my addr 172.21.15.27:0/731627870 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:51.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.996+0000 7fb268137b80 1 -- 172.21.15.27:0/731627870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c726e40bf0 con 0x55c726e23060 2024-05-10T11:30:51.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.996+0000 7fb268137b80 1 -- 172.21.15.27:0/731627870 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55c726cdbb20 con 0x55c726e23980 2024-05-10T11:30:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.996+0000 7fb268137b80 1 -- 172.21.15.27:0/731627870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c726e40d70 con 0x55c726e279d0 2024-05-10T11:30:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.996+0000 7fb268137b80 10 client.0 _ll_register_callbacks cb 0x55c726d1cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:30:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.997+0000 7fb256a00700 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e23060 0x55c726e2ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.998+0000 7fb256000700 1 -- 172.21.15.27:0/731627870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e279d0 msgr2=0x55c726e2c770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:30:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.998+0000 7fb256000700 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e279d0 0x55c726e2c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.998+0000 7fb256000700 1 -- 172.21.15.27:0/731627870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e23060 msgr2=0x55c726e2ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.998+0000 7fb256000700 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e23060 0x55c726e2ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.998+0000 7fb256000700 1 -- 172.21.15.27:0/731627870 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c726d3b420 con 0x55c726e23980 2024-05-10T11:30:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.998+0000 7fb256a00700 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e23060 0x55c726e2ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:30:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.998+0000 7fb256000700 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e2c230 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb2480e1330 tx=0x7fb2480099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eff97eea0a24e12 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.998+0000 7fb244c00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:30:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.998+0000 7fb244c00700 1 -- 172.21.15.27:0/731627870 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2480aacc0 con 0x55c726e23980 2024-05-10T11:30:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.998+0000 7fb244c00700 1 -- 172.21.15.27:0/731627870 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7fb2480e7da0 con 0x55c726e23980 2024-05-10T11:30:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.999+0000 7fb268137b80 1 -- 172.21.15.27:0/731627870 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb240011cb0 con 0x55c726e23980 2024-05-10T11:30:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.999+0000 7fb244c00700 1 -- 172.21.15.27:0/731627870 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb2480aa660 con 0x55c726e23980 2024-05-10T11:30:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.999+0000 7fb268137b80 10 client.27599 fetch_fsmap learned FSMap version 45 2024-05-10T11:30:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:51.999+0000 7fb268137b80 1 -- 172.21.15.27:0/731627870 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55c726e4b530 con 0x55c726e23980 2024-05-10T11:30:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.000+0000 7fb244c00700 1 -- 172.21.15.27:0/731627870 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb240011cb0 con 0x55c726e23980 2024-05-10T11:30:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.000+0000 7fb244c00700 1 -- 172.21.15.27:0/731627870 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7fb248011070 con 0x55c726e23980 2024-05-10T11:30:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.001+0000 7fb268137b80 10 client.27599 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:30:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.001+0000 7fb268137b80 10 client.27599 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:30:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.001+0000 7fb268137b80 20 client.27599 populate_metadata read hostname 'smithi027' 2024-05-10T11:30:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.001+0000 7fb268137b80 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c726e5eb90 0x55c726e61030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.001+0000 7fb268137b80 4 client.27599 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:30:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.001+0000 7fb268137b80 1 -- 172.21.15.27:0/731627870 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c726df00a0 con 0x55c726e5eb90 2024-05-10T11:30:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.001+0000 7fb256a00700 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c726e5eb90 0x55c726e61030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:52.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.003+0000 7fb244c00700 10 client.27599 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:30:52.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.003+0000 7fb256a00700 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c726e5eb90 0x55c726e61030 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=96a4d4cae48bc9bf server_cookie=3806376b602a8ff in_seq=0 out_seq=0 2024-05-10T11:30:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.004+0000 7fb244c00700 1 -- 172.21.15.27:0/731627870 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c726df00a0 con 0x55c726e5eb90 2024-05-10T11:30:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.004+0000 7fb244c00700 10 client.27599 handle_command_reply: tid=0 2024-05-10T11:30:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.005+0000 7fb268137b80 1 client.27599 shutdown 2024-05-10T11:30:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.006+0000 7fb268137b80 1 -- 172.21.15.27:0/731627870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 msgr2=0x55c726e2c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.006+0000 7fb268137b80 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e2c230 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb2480e1330 tx=0x7fb2480099e0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.006+0000 7fb268137b80 1 -- 172.21.15.27:0/731627870 shutdown_connections 2024-05-10T11:30:52.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.006+0000 7fb268137b80 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c726e5eb90 0x55c726e61030 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.006+0000 7fb268137b80 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e279d0 0x55c726e2c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.006+0000 7fb268137b80 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e23060 0x55c726e2ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.006+0000 7fb268137b80 1 --2- 172.21.15.27:0/731627870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e2c230 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.006+0000 7fb268137b80 1 -- 172.21.15.27:0/731627870 >> 172.21.15.27:0/731627870 conn(0x55c726d51b60 msgr2=0x55c726e182b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:52.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.007+0000 7fb244c00700 0 client.27599 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:30:52.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.007+0000 7fb268137b80 1 -- 172.21.15.27:0/731627870 shutdown_connections 2024-05-10T11:30:52.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.007+0000 7fb268137b80 1 -- 172.21.15.27:0/731627870 wait complete. 2024-05-10T11:30:52.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.007+0000 7fb268137b80 20 client.27599 trim_cache size 0 max 16384 2024-05-10T11:30:52.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.043+0000 7fb268137b80 1 Processor -- start 2024-05-10T11:30:52.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.043+0000 7fb268137b80 1 -- start start 2024-05-10T11:30:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.043+0000 7fb268137b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726d51b60 0x55c726e29ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.043+0000 7fb268137b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e23060 0x55c726e49d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.043+0000 7fb268137b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e2ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.043+0000 7fb268137b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb23400ef20 con 0x55c726d51b60 2024-05-10T11:30:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.043+0000 7fb268137b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb234010a60 con 0x55c726e23980 2024-05-10T11:30:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.043+0000 7fb268137b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb23400f5f0 con 0x55c726e23060 2024-05-10T11:30:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.044+0000 7fb257400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e2ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.044+0000 7fb256000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e23060 0x55c726e49d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.044+0000 7fb257400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e2ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51974/0 (socket says 172.21.15.27:51974) 2024-05-10T11:30:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.044+0000 7fb257400700 1 -- 172.21.15.27:0/4138212253 learned_addr learned my addr 172.21.15.27:0/4138212253 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:52.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.044+0000 7fb257400700 1 -- 172.21.15.27:0/4138212253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e23060 msgr2=0x55c726e49d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:52.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.044+0000 7fb257400700 1 --2- 172.21.15.27:0/4138212253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e23060 0x55c726e49d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.044+0000 7fb257400700 1 -- 172.21.15.27:0/4138212253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726d51b60 msgr2=0x55c726e29ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:52.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.044+0000 7fb257400700 1 --2- 172.21.15.27:0/4138212253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726d51b60 0x55c726e29ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.044+0000 7fb257400700 1 -- 172.21.15.27:0/4138212253 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c726e2a7a0 con 0x55c726e23980 2024-05-10T11:30:52.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.045+0000 7fb257400700 1 --2- 172.21.15.27:0/4138212253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e2ab20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb250009930 tx=0x7fb250014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=593962bea81faa55 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.045+0000 7fb247400700 1 -- 172.21.15.27:0/4138212253 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb250003680 con 0x55c726e23980 2024-05-10T11:30:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.045+0000 7fb247400700 1 -- 172.21.15.27:0/4138212253 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb250003cd0 con 0x55c726e23980 2024-05-10T11:30:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.045+0000 7fb247400700 1 -- 172.21.15.27:0/4138212253 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25001ed80 con 0x55c726e23980 2024-05-10T11:30:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.046+0000 7fb268137b80 1 -- 172.21.15.27:0/4138212253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 msgr2=0x55c726e2ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.046+0000 7fb268137b80 1 --2- 172.21.15.27:0/4138212253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e2ab20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb250009930 tx=0x7fb250014000 comp rx=0 tx=0).stop 2024-05-10T11:30:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.047+0000 7fb268137b80 1 -- 172.21.15.27:0/4138212253 shutdown_connections 2024-05-10T11:30:52.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.047+0000 7fb268137b80 1 --2- 172.21.15.27:0/4138212253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e2ab20 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.047+0000 7fb268137b80 1 --2- 172.21.15.27:0/4138212253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726d51b60 0x55c726e29ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.047+0000 7fb268137b80 1 --2- 172.21.15.27:0/4138212253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726e23060 0x55c726e49d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.047+0000 7fb268137b80 1 -- 172.21.15.27:0/4138212253 >> 172.21.15.27:0/4138212253 conn(0x55c726e28080 msgr2=0x55c726e2c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:52.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.049+0000 7fb268137b80 1 -- 172.21.15.27:0/4138212253 shutdown_connections 2024-05-10T11:30:52.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.049+0000 7fb268137b80 1 -- 172.21.15.27:0/4138212253 wait complete. 2024-05-10T11:30:52.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.049+0000 7fb268137b80 1 -- start start 2024-05-10T11:30:52.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.050+0000 7fb268137b80 1 Processor -- start 2024-05-10T11:30:52.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.051+0000 7fb268137b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726d51b60 0x55c726e66160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:52.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.051+0000 7fb268137b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e23060 0x55c726e666a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:52.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.051+0000 7fb268137b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e66be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:52.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.051+0000 7fb268137b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb234010390 con 0x55c726e23060 2024-05-10T11:30:52.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.051+0000 7fb268137b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb23400fcc0 con 0x55c726e23980 2024-05-10T11:30:52.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.051+0000 7fb268137b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb23400cee0 con 0x55c726d51b60 2024-05-10T11:30:52.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.051+0000 7fb268137b80 10 client.0 _ll_register_callbacks cb 0x55c726d1cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:30:52.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.052+0000 7fb257400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e66be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:52.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.052+0000 7fb257400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e66be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:51990/0 (socket says 172.21.15.27:51990) 2024-05-10T11:30:52.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.052+0000 7fb257400700 1 -- 172.21.15.27:0/3591098380 learned_addr learned my addr 172.21.15.27:0/3591098380 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:52.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.052+0000 7fb256000700 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e23060 0x55c726e666a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:52.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.052+0000 7fb256a00700 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726d51b60 0x55c726e66160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:52.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.052+0000 7fb257400700 1 -- 172.21.15.27:0/3591098380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726d51b60 msgr2=0x55c726e66160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:52.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.052+0000 7fb257400700 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726d51b60 0x55c726e66160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.052+0000 7fb257400700 1 -- 172.21.15.27:0/3591098380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e23060 msgr2=0x55c726e666a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:52.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.053+0000 7fb257400700 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e23060 0x55c726e666a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.053+0000 7fb257400700 1 -- 172.21.15.27:0/3591098380 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c726e44420 con 0x55c726e23980 2024-05-10T11:30:52.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.054+0000 7fb256a00700 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726d51b60 0x55c726e66160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:30:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.054+0000 7fb257400700 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e66be0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb25001ece0 tx=0x7fb25001edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2fcd58d5e608c50 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.054+0000 7fb244c00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:30:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.054+0000 7fb244c00700 1 -- 172.21.15.27:0/3591098380 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb250003890 con 0x55c726e23980 2024-05-10T11:30:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.054+0000 7fb244c00700 1 -- 172.21.15.27:0/3591098380 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7fb250003a30 con 0x55c726e23980 2024-05-10T11:30:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.055+0000 7fb268137b80 1 -- 172.21.15.27:0/3591098380 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb250003cd0 con 0x55c726e23980 2024-05-10T11:30:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.055+0000 7fb244c00700 1 -- 172.21.15.27:0/3591098380 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb250005030 con 0x55c726e23980 2024-05-10T11:30:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.055+0000 7fb268137b80 10 client.27605 fetch_fsmap learned FSMap version 45 2024-05-10T11:30:52.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.055+0000 7fb268137b80 1 -- 172.21.15.27:0/3591098380 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55c726ec9830 con 0x55c726e23980 2024-05-10T11:30:52.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.055+0000 7fb244c00700 1 -- 172.21.15.27:0/3591098380 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb250003cd0 con 0x55c726e23980 2024-05-10T11:30:52.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.056+0000 7fb256000700 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e23060 0x55c726e666a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:30:52.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.056+0000 7fb244c00700 1 -- 172.21.15.27:0/3591098380 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7fb250004c00 con 0x55c726e23980 2024-05-10T11:30:52.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.056+0000 7fb268137b80 10 client.27605 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:30:52.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.056+0000 7fb268137b80 10 client.27605 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:30:52.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.056+0000 7fb268137b80 20 client.27605 populate_metadata read hostname 'smithi027' 2024-05-10T11:30:52.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.056+0000 7fb268137b80 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c726e279d0 0x55c726ecbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:52.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.056+0000 7fb268137b80 4 client.27605 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:30:52.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.056+0000 7fb268137b80 1 -- 172.21.15.27:0/3591098380 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c726d83400 con 0x55c726e279d0 2024-05-10T11:30:52.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.057+0000 7fb256a00700 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c726e279d0 0x55c726ecbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:52.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.059+0000 7fb256a00700 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c726e279d0 0x55c726ecbe40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=32246bba21a8087f server_cookie=b5f1246afd5a0fbf in_seq=0 out_seq=0 2024-05-10T11:30:52.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.059+0000 7fb244c00700 10 client.27605 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:30:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.062+0000 7fb244c00700 1 -- 172.21.15.27:0/3591098380 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c726d83400 con 0x55c726e279d0 2024-05-10T11:30:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.062+0000 7fb244c00700 10 client.27605 handle_command_reply: tid=0 2024-05-10T11:30:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.063+0000 7fb268137b80 1 client.27605 shutdown 2024-05-10T11:30:52.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.065+0000 7fb268137b80 1 -- 172.21.15.27:0/3591098380 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 msgr2=0x55c726e66be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:52.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.065+0000 7fb268137b80 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e66be0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb25001ece0 tx=0x7fb25001edc0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.065+0000 7fb268137b80 1 -- 172.21.15.27:0/3591098380 shutdown_connections 2024-05-10T11:30:52.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.065+0000 7fb268137b80 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x55c726e279d0 0x55c726ecbe40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.065+0000 7fb268137b80 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c726d51b60 0x55c726e66160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.065+0000 7fb268137b80 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c726e23980 0x55c726e66be0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.065+0000 7fb268137b80 1 --2- 172.21.15.27:0/3591098380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c726e23060 0x55c726e666a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.065+0000 7fb268137b80 1 -- 172.21.15.27:0/3591098380 >> 172.21.15.27:0/3591098380 conn(0x55c726e28080 msgr2=0x55c726e2aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:52.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.066+0000 7fb244c00700 0 client.27605 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:30:52.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.066+0000 7fb268137b80 1 -- 172.21.15.27:0/3591098380 shutdown_connections 2024-05-10T11:30:52.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.066+0000 7fb268137b80 1 -- 172.21.15.27:0/3591098380 wait complete. 2024-05-10T11:30:52.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.066+0000 7fb268137b80 20 client.27605 trim_cache size 0 max 16384 2024-05-10T11:30:52.066 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:30:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.067+0000 7fb25d800700 1 -- 172.21.15.27:0/1572993440 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb23c0b2aa0 msgr2=0x7fb23c0b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.067+0000 7fb25d800700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb23c0b2aa0 0x7fb23c0b4f60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb250009d50 tx=0x7fb250009260 comp rx=0 tx=0).stop 2024-05-10T11:30:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.067+0000 7fb25d800700 1 -- 172.21.15.27:0/1572993440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb258103ef0 msgr2=0x7fb258075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.067+0000 7fb25d800700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb258103ef0 0x7fb258075ce0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb248000f80 tx=0x7fb248004e70 comp rx=0 tx=0).stop 2024-05-10T11:30:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.067+0000 7fb25d800700 1 -- 172.21.15.27:0/1572993440 shutdown_connections 2024-05-10T11:30:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.067+0000 7fb25d800700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb23c0b2aa0 0x7fb23c0b4f60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.067+0000 7fb25d800700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb258108460 0x7fb258076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.067+0000 7fb25d800700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb258103550 0x7fb2580757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.067+0000 7fb25d800700 1 --2- 172.21.15.27:0/1572993440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb258103ef0 0x7fb258075ce0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.068+0000 7fb25d800700 1 -- 172.21.15.27:0/1572993440 >> 172.21.15.27:0/1572993440 conn(0x7fb25807a600 msgr2=0x7fb258101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.068+0000 7fb25d800700 1 -- 172.21.15.27:0/1572993440 shutdown_connections 2024-05-10T11:30:52.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:52.069+0000 7fb25d800700 1 -- 172.21.15.27:0/1572993440 wait complete. 2024-05-10T11:30:52.175 INFO:tasks.workunit.client.0.smithi027.stdout: 10 93128 22.56 MB/sec execute 96 sec latency 333.275 ms 2024-05-10T11:30:53.175 INFO:tasks.workunit.client.0.smithi027.stdout: 10 93680 22.56 MB/sec execute 97 sec latency 100.062 ms 2024-05-10T11:30:53.469 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:30:53.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:53 smithi027 ceph-mon[30054]: pgmap v554: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.1 MiB/s wr, 408 op/s 2024-05-10T11:30:53.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:53 smithi076 ceph-mon[34152]: pgmap v554: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.1 MiB/s wr, 408 op/s 2024-05-10T11:30:53.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:53 smithi189 ceph-mon[34842]: pgmap v554: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.1 MiB/s wr, 408 op/s 2024-05-10T11:30:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:30:54.175 INFO:tasks.workunit.client.0.smithi027.stdout: 10 94419 22.77 MB/sec execute 98 sec latency 85.945 ms 2024-05-10T11:30:55.175 INFO:tasks.workunit.client.0.smithi027.stdout: 10 95010 22.92 MB/sec execute 99 sec latency 126.850 ms 2024-05-10T11:30:55.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:55 smithi027 ceph-mon[30054]: pgmap v555: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.4 MiB/s wr, 365 op/s 2024-05-10T11:30:55.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:55 smithi076 ceph-mon[34152]: pgmap v555: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.4 MiB/s wr, 365 op/s 2024-05-10T11:30:55.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:55 smithi189 ceph-mon[34842]: pgmap v555: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.4 MiB/s wr, 365 op/s 2024-05-10T11:30:56.175 INFO:tasks.workunit.client.0.smithi027.stdout: 10 95350 22.93 MB/sec execute 100 sec latency 261.364 ms 2024-05-10T11:30:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.578+0000 7fcc07600700 1 -- 172.21.15.27:0/4109150138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc00106070 msgr2=0x7fcc00112580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.578+0000 7fcc07600700 1 --2- 172.21.15.27:0/4109150138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc00106070 0x7fcc00112580 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800b0a0 tx=0x7fcbf800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.578+0000 7fcc07600700 1 -- 172.21.15.27:0/4109150138 shutdown_connections 2024-05-10T11:30:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.578+0000 7fcc07600700 1 --2- 172.21.15.27:0/4109150138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc00106070 0x7fcc00112580 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.578+0000 7fcc07600700 1 --2- 172.21.15.27:0/4109150138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0010b6b0 0x7fcc0010ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.578+0000 7fcc07600700 1 --2- 172.21.15.27:0/4109150138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc001056b0 0x7fcc00105b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.578+0000 7fcc07600700 1 -- 172.21.15.27:0/4109150138 >> 172.21.15.27:0/4109150138 conn(0x7fcc00101180 msgr2=0x7fcc001035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:56.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.578+0000 7fcc07600700 1 -- 172.21.15.27:0/4109150138 shutdown_connections 2024-05-10T11:30:56.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.579+0000 7fcc07600700 1 -- 172.21.15.27:0/4109150138 wait complete. 2024-05-10T11:30:56.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.580+0000 7fcc07600700 1 Processor -- start 2024-05-10T11:30:56.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.580+0000 7fcc07600700 1 -- start start 2024-05-10T11:30:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.581+0000 7fcc07600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc001056b0 0x7fcc001a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.581+0000 7fcc07600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc00106070 0x7fcc001a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.581+0000 7fcc07600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc0010b6b0 0x7fcc0019bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.581+0000 7fcc07600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcc001158f0 con 0x7fcc00106070 2024-05-10T11:30:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.581+0000 7fcc07600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcc00115770 con 0x7fcc001056b0 2024-05-10T11:30:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.581+0000 7fcc07600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc00115a70 con 0x7fcc0010b6b0 2024-05-10T11:30:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.581+0000 7fcc04c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc001056b0 0x7fcc001a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.581+0000 7fcc04c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc001056b0 0x7fcc001a1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50016/0 (socket says 172.21.15.27:50016) 2024-05-10T11:30:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.581+0000 7fcc04c00700 1 -- 172.21.15.27:0/3660255492 learned_addr learned my addr 172.21.15.27:0/3660255492 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.581+0000 7fcc05600700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc0010b6b0 0x7fcc0019bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.582+0000 7fcbffe00700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc00106070 0x7fcc001a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.582+0000 7fcc04c00700 1 -- 172.21.15.27:0/3660255492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc0010b6b0 msgr2=0x7fcc0019bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.582+0000 7fcc04c00700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc0010b6b0 0x7fcc0019bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.582+0000 7fcc04c00700 1 -- 172.21.15.27:0/3660255492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc00106070 msgr2=0x7fcc001a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.582+0000 7fcc04c00700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc00106070 0x7fcc001a1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.582+0000 7fcc04c00700 1 -- 172.21.15.27:0/3660255492 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0019c410 con 0x7fcc001056b0 2024-05-10T11:30:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.582+0000 7fcbffe00700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc00106070 0x7fcc001a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:30:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.582+0000 7fcc05600700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc0010b6b0 0x7fcc0019bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:30:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.582+0000 7fcc04c00700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc001056b0 0x7fcc001a1a60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400eb30 tx=0x7fcbf400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.583+0000 7fcbfd600700 1 -- 172.21.15.27:0/3660255492 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf400cc90 con 0x7fcc001056b0 2024-05-10T11:30:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.583+0000 7fcc07600700 1 -- 172.21.15.27:0/3660255492 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0019c6a0 con 0x7fcc001056b0 2024-05-10T11:30:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.583+0000 7fcc07600700 1 -- 172.21.15.27:0/3660255492 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc00077ab0 con 0x7fcc001056b0 2024-05-10T11:30:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.583+0000 7fcbfd600700 1 -- 172.21.15.27:0/3660255492 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbf40044d0 con 0x7fcc001056b0 2024-05-10T11:30:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.583+0000 7fcbfd600700 1 -- 172.21.15.27:0/3660255492 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4010430 con 0x7fcc001056b0 2024-05-10T11:30:56.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.588+0000 7fcc11f09b80 1 Processor -- start 2024-05-10T11:30:56.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.588+0000 7fcc11f09b80 1 -- start start 2024-05-10T11:30:56.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.588+0000 7fcbfd600700 1 -- 172.21.15.27:0/3660255492 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fcbf4003680 con 0x7fcc001056b0 2024-05-10T11:30:56.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.589+0000 7fcc11f09b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2f7c0 0x564449e2fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.589+0000 7fcc11f09b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e2ae90 0x564449e2b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.589+0000 7fcc11f09b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2b850 0x564449e376e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.589+0000 7fcc11f09b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564449d6ca20 con 0x564449e2f7c0 2024-05-10T11:30:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.589+0000 7fcc11f09b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564449d0a760 con 0x564449e2ae90 2024-05-10T11:30:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.589+0000 7fcc11f09b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564449d689e0 con 0x564449e2b850 2024-05-10T11:30:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.589+0000 7fcc05600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2f7c0 0x564449e2fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.590+0000 7fcbffe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2b850 0x564449e376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.590+0000 7fcbffe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2b850 0x564449e376e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49472/0 (socket says 172.21.15.27:49472) 2024-05-10T11:30:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.590+0000 7fcbffe00700 1 -- 172.21.15.27:0/355388048 learned_addr learned my addr 172.21.15.27:0/355388048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.590+0000 7fcc04c00700 1 --2- 172.21.15.27:0/355388048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e2ae90 0x564449e2b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.591+0000 7fcbffe00700 1 -- 172.21.15.27:0/355388048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e2ae90 msgr2=0x564449e2b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.591+0000 7fcbffe00700 1 --2- 172.21.15.27:0/355388048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e2ae90 0x564449e2b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.591+0000 7fcbffe00700 1 -- 172.21.15.27:0/355388048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2f7c0 msgr2=0x564449e2fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.591+0000 7fcbffe00700 1 --2- 172.21.15.27:0/355388048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2f7c0 0x564449e2fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.592+0000 7fcbffe00700 1 -- 172.21.15.27:0/355388048 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564449cb98d0 con 0x564449e2b850 2024-05-10T11:30:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.592+0000 7fcc05600700 1 --2- 172.21.15.27:0/355388048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2f7c0 0x564449e2fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:30:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.592+0000 7fcbffe00700 1 --2- 172.21.15.27:0/355388048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2b850 0x564449e376e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009c70 tx=0x7fcbec009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27ca2590f1a8bb07 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.593+0000 7fcbfd600700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcbe80b2a70 0x7fcbe80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.593+0000 7fcbf2000700 1 -- 172.21.15.27:0/355388048 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec016070 con 0x564449e2b850 2024-05-10T11:30:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.593+0000 7fcbfd600700 1 -- 172.21.15.27:0/3660255492 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7fcbf40e2b20 con 0x7fcc001056b0 2024-05-10T11:30:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.593+0000 7fcbf2000700 1 -- 172.21.15.27:0/355388048 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbec004830 con 0x564449e2b850 2024-05-10T11:30:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.593+0000 7fcbf2000700 1 -- 172.21.15.27:0/355388048 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0055d0 con 0x564449e2b850 2024-05-10T11:30:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.595+0000 7fcc05600700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcbe80b2a70 0x7fcbe80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.595+0000 7fcc11f09b80 1 -- 172.21.15.27:0/355388048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2b850 msgr2=0x564449e376e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.595+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/355388048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2b850 0x564449e376e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009c70 tx=0x7fcbec009f80 comp rx=0 tx=0).stop 2024-05-10T11:30:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.595+0000 7fcc11f09b80 1 -- 172.21.15.27:0/355388048 shutdown_connections 2024-05-10T11:30:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.595+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/355388048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2b850 0x564449e376e0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.595+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/355388048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2f7c0 0x564449e2fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.595+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/355388048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e2ae90 0x564449e2b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.595+0000 7fcc11f09b80 1 -- 172.21.15.27:0/355388048 >> 172.21.15.27:0/355388048 conn(0x564449d48b60 msgr2=0x564449e147a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:56.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.596+0000 7fcc11f09b80 1 -- 172.21.15.27:0/355388048 shutdown_connections 2024-05-10T11:30:56.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.596+0000 7fcc11f09b80 1 -- 172.21.15.27:0/355388048 wait complete. 2024-05-10T11:30:56.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.597+0000 7fcc11f09b80 1 -- start start 2024-05-10T11:30:56.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.598+0000 7fcc11f09b80 1 Processor -- start 2024-05-10T11:30:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.599+0000 7fcc11f09b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2ae90 0x564449e37580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.599+0000 7fcc11f09b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2f7c0 0x564449e348d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.599+0000 7fcc11f09b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 0x564449e35290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.599+0000 7fcc11f09b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564449e37c80 con 0x564449e2f7c0 2024-05-10T11:30:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.599+0000 7fcc11f09b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564449cd2b20 con 0x564449e34e10 2024-05-10T11:30:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.599+0000 7fcc11f09b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564449e37e00 con 0x564449e2ae90 2024-05-10T11:30:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.599+0000 7fcc05600700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcbe80b2a70 0x7fcbe80b4f30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800b780 tx=0x7fcbf801d040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.599+0000 7fcc11f09b80 10 client.0 _ll_register_callbacks cb 0x564449d13c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:30:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.599+0000 7fcc05600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 0x564449e35290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.599+0000 7fcc05600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 0x564449e35290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50034/0 (socket says 172.21.15.27:50034) 2024-05-10T11:30:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.599+0000 7fcc05600700 1 -- 172.21.15.27:0/4155014558 learned_addr learned my addr 172.21.15.27:0/4155014558 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:56.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.600+0000 7fcc05600700 1 -- 172.21.15.27:0/4155014558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2ae90 msgr2=0x564449e37580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:30:56.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.600+0000 7fcc05600700 1 --2- 172.21.15.27:0/4155014558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2ae90 0x564449e37580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.600+0000 7fcc05600700 1 -- 172.21.15.27:0/4155014558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2f7c0 msgr2=0x564449e348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:56.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.600+0000 7fcc05600700 1 --2- 172.21.15.27:0/4155014558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2f7c0 0x564449e348d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.600+0000 7fcc05600700 1 -- 172.21.15.27:0/4155014558 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564449d32420 con 0x564449e34e10 2024-05-10T11:30:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.600+0000 7fcc05600700 1 --2- 172.21.15.27:0/4155014558 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 0x564449e35290 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8015000 tx=0x7fcbf800b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1129c0c153ef879 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.600+0000 7fcbe6a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:30:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.601+0000 7fcbe6a00700 1 -- 172.21.15.27:0/4155014558 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8005040 con 0x564449e34e10 2024-05-10T11:30:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.601+0000 7fcbe6a00700 1 -- 172.21.15.27:0/4155014558 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7fcbf80051e0 con 0x564449e34e10 2024-05-10T11:30:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.601+0000 7fcc11f09b80 1 -- 172.21.15.27:0/4155014558 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbec004830 con 0x564449e34e10 2024-05-10T11:30:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.601+0000 7fcbe6a00700 1 -- 172.21.15.27:0/4155014558 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fcbf80053e0 con 0x564449e34e10 2024-05-10T11:30:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.601+0000 7fcc11f09b80 10 client.27617 fetch_fsmap learned FSMap version 45 2024-05-10T11:30:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.601+0000 7fcc11f09b80 1 -- 172.21.15.27:0/4155014558 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x564449e42bb0 con 0x564449e34e10 2024-05-10T11:30:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.602+0000 7fcbe6a00700 1 -- 172.21.15.27:0/4155014558 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbec004830 con 0x564449e34e10 2024-05-10T11:30:56.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.602+0000 7fcbe6a00700 1 -- 172.21.15.27:0/4155014558 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7fcbf802e030 con 0x564449e34e10 2024-05-10T11:30:56.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.603+0000 7fcc11f09b80 10 client.27617 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:30:56.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.603+0000 7fcc11f09b80 10 client.27617 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:30:56.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.603+0000 7fcc11f09b80 20 client.27617 populate_metadata read hostname 'smithi027' 2024-05-10T11:30:56.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.603+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/4155014558 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564449e45290 0x564449e47720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.604+0000 7fcc11f09b80 4 client.27617 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:30:56.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.604+0000 7fcc04c00700 1 --2- 172.21.15.27:0/4155014558 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564449e45290 0x564449e47720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.604+0000 7fcc11f09b80 1 -- 172.21.15.27:0/4155014558 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564449de70a0 con 0x564449e45290 2024-05-10T11:30:56.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.606+0000 7fcbe6a00700 10 client.27617 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:30:56.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.606+0000 7fcc04c00700 1 --2- 172.21.15.27:0/4155014558 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564449e45290 0x564449e47720 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39dea870aa8f4253 server_cookie=98da2cc6d3fd2edb in_seq=0 out_seq=0 2024-05-10T11:30:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.614+0000 7fcbe6a00700 1 -- 172.21.15.27:0/4155014558 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564449de70a0 con 0x564449e45290 2024-05-10T11:30:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.614+0000 7fcbe6a00700 10 client.27617 handle_command_reply: tid=0 2024-05-10T11:30:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.614+0000 7fcc11f09b80 1 client.27617 shutdown 2024-05-10T11:30:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.615+0000 7fcc11f09b80 1 -- 172.21.15.27:0/4155014558 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 msgr2=0x564449e35290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.615+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/4155014558 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 0x564449e35290 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8015000 tx=0x7fcbf800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.615+0000 7fcc11f09b80 1 -- 172.21.15.27:0/4155014558 shutdown_connections 2024-05-10T11:30:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.615+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/4155014558 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564449e45290 0x564449e47720 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.615+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/4155014558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2ae90 0x564449e37580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.615+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/4155014558 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 0x564449e35290 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.615+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/4155014558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2f7c0 0x564449e348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.615+0000 7fcc11f09b80 1 -- 172.21.15.27:0/4155014558 >> 172.21.15.27:0/4155014558 conn(0x564449d48b60 msgr2=0x564449e0f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:56.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.615+0000 7fcbe6a00700 0 client.27617 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:30:56.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.616+0000 7fcc11f09b80 1 -- 172.21.15.27:0/4155014558 shutdown_connections 2024-05-10T11:30:56.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.616+0000 7fcc11f09b80 1 -- 172.21.15.27:0/4155014558 wait complete. 2024-05-10T11:30:56.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.617+0000 7fcc11f09b80 20 client.27617 trim_cache size 0 max 16384 2024-05-10T11:30:56.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.648+0000 7fcc11f09b80 1 Processor -- start 2024-05-10T11:30:56.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.648+0000 7fcc11f09b80 1 -- start start 2024-05-10T11:30:56.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.649+0000 7fcc11f09b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2ae90 0x564449e40da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.649+0000 7fcc11f09b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e2f7c0 0x564449e164d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.649+0000 7fcc11f09b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e34e10 0x564449e341f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.649+0000 7fcc11f09b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe000ef50 con 0x564449e34e10 2024-05-10T11:30:56.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.649+0000 7fcc11f09b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe0010a90 con 0x564449e2f7c0 2024-05-10T11:30:56.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.649+0000 7fcc11f09b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe000f620 con 0x564449e2ae90 2024-05-10T11:30:56.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.649+0000 7fcbffe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e2f7c0 0x564449e164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.649+0000 7fcc04c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2ae90 0x564449e40da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.649+0000 7fcbffe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e2f7c0 0x564449e164d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50050/0 (socket says 172.21.15.27:50050) 2024-05-10T11:30:56.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.649+0000 7fcbffe00700 1 -- 172.21.15.27:0/171366003 learned_addr learned my addr 172.21.15.27:0/171366003 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:56.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.649+0000 7fcc05600700 1 --2- 172.21.15.27:0/171366003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e34e10 0x564449e341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.650+0000 7fcc04c00700 1 -- 172.21.15.27:0/171366003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e2f7c0 msgr2=0x564449e164d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:56.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.650+0000 7fcc04c00700 1 --2- 172.21.15.27:0/171366003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e2f7c0 0x564449e164d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.650+0000 7fcc04c00700 1 -- 172.21.15.27:0/171366003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e34e10 msgr2=0x564449e341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.650+0000 7fcc04c00700 1 --2- 172.21.15.27:0/171366003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e34e10 0x564449e341f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.650+0000 7fcc04c00700 1 -- 172.21.15.27:0/171366003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564449e0cdb0 con 0x564449e2ae90 2024-05-10T11:30:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.650+0000 7fcbffe00700 1 --2- 172.21.15.27:0/171366003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e2f7c0 0x564449e164d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:30:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.651+0000 7fcc04c00700 1 --2- 172.21.15.27:0/171366003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2ae90 0x564449e40da0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40dfde0 tx=0x7fcbf40e4fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8bc7b3bf98eb1bd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.651+0000 7fcbf1600700 1 -- 172.21.15.27:0/171366003 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40e8070 con 0x564449e2ae90 2024-05-10T11:30:56.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.651+0000 7fcbf1600700 1 -- 172.21.15.27:0/171366003 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbf40b2110 con 0x564449e2ae90 2024-05-10T11:30:56.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.651+0000 7fcbf1600700 1 -- 172.21.15.27:0/171366003 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40add20 con 0x564449e2ae90 2024-05-10T11:30:56.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.653+0000 7fcc11f09b80 1 -- 172.21.15.27:0/171366003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2ae90 msgr2=0x564449e40da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:56.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.653+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/171366003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2ae90 0x564449e40da0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40dfde0 tx=0x7fcbf40e4fe0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.655+0000 7fcc11f09b80 1 -- 172.21.15.27:0/171366003 shutdown_connections 2024-05-10T11:30:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.655+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/171366003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2ae90 0x564449e40da0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.655+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/171366003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e34e10 0x564449e341f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.655+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/171366003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e2f7c0 0x564449e164d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.655+0000 7fcc11f09b80 1 -- 172.21.15.27:0/171366003 >> 172.21.15.27:0/171366003 conn(0x564449d48b60 msgr2=0x564449e14720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.657+0000 7fcc11f09b80 1 -- 172.21.15.27:0/171366003 shutdown_connections 2024-05-10T11:30:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.658+0000 7fcc11f09b80 1 -- 172.21.15.27:0/171366003 wait complete. 2024-05-10T11:30:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.658+0000 7fcc11f09b80 1 -- start start 2024-05-10T11:30:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.658+0000 7fcc11f09b80 1 Processor -- start 2024-05-10T11:30:56.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.660+0000 7fcc11f09b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2ae90 0x564449e4c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.660+0000 7fcc11f09b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2f7c0 0x564449e4ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.660+0000 7fcc11f09b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 0x564449e4d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc11f09b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe00103c0 con 0x564449e2ae90 2024-05-10T11:30:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc11f09b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe000fcf0 con 0x564449e34e10 2024-05-10T11:30:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc11f09b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe000cf10 con 0x564449e2f7c0 2024-05-10T11:30:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc11f09b80 10 client.0 _ll_register_callbacks cb 0x564449d13c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:30:56.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc05600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 0x564449e4d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc05600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 0x564449e4d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:50058/0 (socket says 172.21.15.27:50058) 2024-05-10T11:30:56.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc05600700 1 -- 172.21.15.27:0/2907771908 learned_addr learned my addr 172.21.15.27:0/2907771908 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:30:56.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcbffe00700 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2f7c0 0x564449e4ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc04c00700 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2ae90 0x564449e4c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc05600700 1 -- 172.21.15.27:0/2907771908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2f7c0 msgr2=0x564449e4ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc05600700 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2f7c0 0x564449e4ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc05600700 1 -- 172.21.15.27:0/2907771908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2ae90 msgr2=0x564449e4c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:30:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc05600700 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2ae90 0x564449e4c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcc05600700 1 -- 172.21.15.27:0/2907771908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564449eaebe0 con 0x564449e34e10 2024-05-10T11:30:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.661+0000 7fcbffe00700 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2f7c0 0x564449e4ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:30:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.662+0000 7fcc04c00700 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2ae90 0x564449e4c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:30:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.663+0000 7fcc05600700 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 0x564449e4d230 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800d010 tx=0x7fcbf8005010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7321f45fa7df4072 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:30:56.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.663+0000 7fcbe6a00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:30:56.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.663+0000 7fcbe6a00700 1 -- 172.21.15.27:0/2907771908 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf803d020 con 0x564449e34e10 2024-05-10T11:30:56.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.664+0000 7fcc11f09b80 1 -- 172.21.15.27:0/2907771908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564449d7a400 con 0x564449e34e10 2024-05-10T11:30:56.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.664+0000 7fcbe6a00700 1 -- 172.21.15.27:0/2907771908 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9591+0+0 (secure 0 0 0) 0x7fcbf8025420 con 0x564449e34e10 2024-05-10T11:30:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.664+0000 7fcbe6a00700 1 -- 172.21.15.27:0/2907771908 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fcbf8048050 con 0x564449e34e10 2024-05-10T11:30:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.665+0000 7fcc11f09b80 10 client.27623 fetch_fsmap learned FSMap version 45 2024-05-10T11:30:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.665+0000 7fcc11f09b80 1 -- 172.21.15.27:0/2907771908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x564449eafe60 con 0x564449e34e10 2024-05-10T11:30:56.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.666+0000 7fcbe6a00700 1 -- 172.21.15.27:0/2907771908 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564449d7a400 con 0x564449e34e10 2024-05-10T11:30:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.666+0000 7fcbe6a00700 1 -- 172.21.15.27:0/2907771908 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7fcbf802e040 con 0x564449e34e10 2024-05-10T11:30:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.666+0000 7fcc11f09b80 10 client.27623 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:30:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.666+0000 7fcc11f09b80 10 client.27623 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:30:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.666+0000 7fcc11f09b80 20 client.27623 populate_metadata read hostname 'smithi027' 2024-05-10T11:30:56.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.666+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564449eb04a0 0x564449eb28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:30:56.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.667+0000 7fcc11f09b80 4 client.27623 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:30:56.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.667+0000 7fcc11f09b80 1 -- 172.21.15.27:0/2907771908 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564449d744a0 con 0x564449eb04a0 2024-05-10T11:30:56.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.667+0000 7fcc04c00700 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564449eb04a0 0x564449eb28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.669+0000 7fcc04c00700 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564449eb04a0 0x564449eb28c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9db88f100890f7f3 server_cookie=f487d930336d6d2 in_seq=0 out_seq=0 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.669+0000 7fcbe6a00700 10 client.27623 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.671+0000 7fcbe6a00700 1 -- 172.21.15.27:0/2907771908 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564449d744a0 con 0x564449eb04a0 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.671+0000 7fcbe6a00700 10 client.27623 handle_command_reply: tid=0 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.671+0000 7fcc11f09b80 1 client.27623 shutdown 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.671+0000 7fcc11f09b80 1 -- 172.21.15.27:0/2907771908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 msgr2=0x564449e4d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.671+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 0x564449e4d230 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800d010 tx=0x7fcbf8005010 comp rx=0 tx=0).stop 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.672+0000 7fcc11f09b80 1 -- 172.21.15.27:0/2907771908 shutdown_connections 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.672+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x564449eb04a0 0x564449eb28c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.672+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564449e34e10 0x564449e4d230 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.672+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564449e2ae90 0x564449e4c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.672+0000 7fcc11f09b80 1 --2- 172.21.15.27:0/2907771908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564449e2f7c0 0x564449e4ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.672+0000 7fcbe6a00700 0 client.27623 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:30:56.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.672+0000 7fcc11f09b80 1 -- 172.21.15.27:0/2907771908 >> 172.21.15.27:0/2907771908 conn(0x564449d48b60 msgr2=0x564449e0ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:56.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.672+0000 7fcc11f09b80 1 -- 172.21.15.27:0/2907771908 shutdown_connections 2024-05-10T11:30:56.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.672+0000 7fcc11f09b80 1 -- 172.21.15.27:0/2907771908 wait complete. 2024-05-10T11:30:56.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.672+0000 7fcc11f09b80 20 client.27623 trim_cache size 0 max 16384 2024-05-10T11:30:56.673 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:30:56.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.675+0000 7fcc07600700 1 -- 172.21.15.27:0/3660255492 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcbe80b2a70 msgr2=0x7fcbe80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:56.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.675+0000 7fcc07600700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcbe80b2a70 0x7fcbe80b4f30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800b780 tx=0x7fcbf801d040 comp rx=0 tx=0).stop 2024-05-10T11:30:56.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.675+0000 7fcc07600700 1 -- 172.21.15.27:0/3660255492 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc001056b0 msgr2=0x7fcc001a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:30:56.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.675+0000 7fcc07600700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc001056b0 0x7fcc001a1a60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400eb30 tx=0x7fcbf400ee40 comp rx=0 tx=0).stop 2024-05-10T11:30:56.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.675+0000 7fcc07600700 1 -- 172.21.15.27:0/3660255492 shutdown_connections 2024-05-10T11:30:56.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.675+0000 7fcc07600700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fcbe80b2a70 0x7fcbe80b4f30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.675+0000 7fcc07600700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc0010b6b0 0x7fcc0019bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.675+0000 7fcc07600700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc001056b0 0x7fcc001a1a60 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.675+0000 7fcc07600700 1 --2- 172.21.15.27:0/3660255492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc00106070 0x7fcc001a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:30:56.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.675+0000 7fcc07600700 1 -- 172.21.15.27:0/3660255492 >> 172.21.15.27:0/3660255492 conn(0x7fcc00101180 msgr2=0x7fcc00101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:30:56.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.676+0000 7fcc07600700 1 -- 172.21.15.27:0/3660255492 shutdown_connections 2024-05-10T11:30:56.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:30:56.676+0000 7fcc07600700 1 -- 172.21.15.27:0/3660255492 wait complete. 2024-05-10T11:30:57.176 INFO:tasks.workunit.client.0.smithi027.stdout: 10 95768 22.99 MB/sec execute 101 sec latency 109.984 ms 2024-05-10T11:30:57.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:57 smithi027 ceph-mon[30054]: pgmap v556: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 12 MiB/s wr, 569 op/s 2024-05-10T11:30:57.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:57 smithi076 ceph-mon[34152]: pgmap v556: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 12 MiB/s wr, 569 op/s 2024-05-10T11:30:57.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:57 smithi189 ceph-mon[34842]: pgmap v556: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 12 MiB/s wr, 569 op/s 2024-05-10T11:30:58.175 INFO:tasks.workunit.client.0.smithi027.stdout: 10 96171 23.00 MB/sec execute 102 sec latency 99.097 ms 2024-05-10T11:30:58.679 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-10T11:30:58.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-10T11:30:59.176 INFO:tasks.workunit.client.0.smithi027.stdout: 10 96430 22.86 MB/sec execute 103 sec latency 184.934 ms 2024-05-10T11:30:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:30:59.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:30:59 smithi027 ceph-mon[30054]: pgmap v557: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 7.3 MiB/s wr, 375 op/s 2024-05-10T11:30:59.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:30:59 smithi076 ceph-mon[34152]: pgmap v557: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 7.3 MiB/s wr, 375 op/s 2024-05-10T11:30:59.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:30:59 smithi189 ceph-mon[34842]: pgmap v557: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 7.3 MiB/s wr, 375 op/s 2024-05-10T11:31:00.176 INFO:tasks.workunit.client.0.smithi027.stdout: 10 96760 22.73 MB/sec execute 104 sec latency 189.151 ms 2024-05-10T11:31:01.176 INFO:tasks.workunit.client.0.smithi027.stdout: 10 97153 22.67 MB/sec execute 105 sec latency 222.605 ms 2024-05-10T11:31:01.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:01 smithi027 ceph-mon[30054]: pgmap v558: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 533 op/s 2024-05-10T11:31:01.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:01 smithi076 ceph-mon[34152]: pgmap v558: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 533 op/s 2024-05-10T11:31:01.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:01 smithi189 ceph-mon[34842]: pgmap v558: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 533 op/s 2024-05-10T11:31:02.176 INFO:tasks.workunit.client.0.smithi027.stdout: 10 97392 22.54 MB/sec execute 106 sec latency 394.344 ms 2024-05-10T11:31:02.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.408+0000 7f387d400700 1 -- 172.21.15.27:0/3195044469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f38780756d0 msgr2=0x7f3878075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:02.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.408+0000 7f387d400700 1 --2- 172.21.15.27:0/3195044469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f38780756d0 0x7f3878075ab0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3868009990 tx=0x7f3868009ca0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.409+0000 7f387d400700 1 -- 172.21.15.27:0/3195044469 shutdown_connections 2024-05-10T11:31:02.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.409+0000 7f387d400700 1 --2- 172.21.15.27:0/3195044469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3878114670 0x7f3878116a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.409+0000 7f387d400700 1 --2- 172.21.15.27:0/3195044469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f38780756d0 0x7f3878075ab0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.409+0000 7f387d400700 1 --2- 172.21.15.27:0/3195044469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3878075ff0 0x7f3878114130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.409+0000 7f387d400700 1 -- 172.21.15.27:0/3195044469 >> 172.21.15.27:0/3195044469 conn(0x7f387807a600 msgr2=0x7f387807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:02.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.409+0000 7f387d400700 1 -- 172.21.15.27:0/3195044469 shutdown_connections 2024-05-10T11:31:02.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.409+0000 7f387d400700 1 -- 172.21.15.27:0/3195044469 wait complete. 2024-05-10T11:31:02.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.411+0000 7f387d400700 1 Processor -- start 2024-05-10T11:31:02.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.411+0000 7f387d400700 1 -- start start 2024-05-10T11:31:02.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.412+0000 7f387d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38780756d0 0x7f3878074cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.413+0000 7f387d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3878075ff0 0x7f387806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.413+0000 7f3876800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38780756d0 0x7f3878074cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.413+0000 7f3876800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38780756d0 0x7f3878074cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49550/0 (socket says 172.21.15.27:49550) 2024-05-10T11:31:02.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.413+0000 7f387d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3878114670 0x7f387806e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.413+0000 7f387d400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3878119ca0 con 0x7f3878114670 2024-05-10T11:31:02.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.413+0000 7f387d400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3878119b20 con 0x7f3878075ff0 2024-05-10T11:31:02.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.413+0000 7f387d400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3878119e20 con 0x7f38780756d0 2024-05-10T11:31:02.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.413+0000 7f3877200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3878114670 0x7f387806e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.413+0000 7f3877200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3878114670 0x7f387806e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60196/0 (socket says 172.21.15.27:60196) 2024-05-10T11:31:02.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.413+0000 7f3877200700 1 -- 172.21.15.27:0/3930944698 learned_addr learned my addr 172.21.15.27:0/3930944698 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:02.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.414+0000 7f3875e00700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3878075ff0 0x7f387806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.415+0000 7f3876800700 1 -- 172.21.15.27:0/3930944698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3878075ff0 msgr2=0x7f387806dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:02.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.415+0000 7f3876800700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3878075ff0 0x7f387806dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.415+0000 7f3876800700 1 -- 172.21.15.27:0/3930944698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3878114670 msgr2=0x7f387806e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:02.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.415+0000 7f3876800700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3878114670 0x7f387806e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.415+0000 7f3876800700 1 -- 172.21.15.27:0/3930944698 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f387806e8d0 con 0x7f38780756d0 2024-05-10T11:31:02.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.415+0000 7f3877200700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3878114670 0x7f387806e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:31:02.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.415+0000 7f3876800700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38780756d0 0x7f3878074cc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3868000c00 tx=0x7f38680047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:02.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.416+0000 7f386f400700 1 -- 172.21.15.27:0/3930944698 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3868027070 con 0x7f38780756d0 2024-05-10T11:31:02.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.416+0000 7f386f400700 1 -- 172.21.15.27:0/3930944698 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3868003680 con 0x7f38780756d0 2024-05-10T11:31:02.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.416+0000 7f386f400700 1 -- 172.21.15.27:0/3930944698 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3868022a80 con 0x7f38780756d0 2024-05-10T11:31:02.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.416+0000 7f387d400700 1 -- 172.21.15.27:0/3930944698 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f387806eb60 con 0x7f38780756d0 2024-05-10T11:31:02.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.416+0000 7f387d400700 1 -- 172.21.15.27:0/3930944698 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3878072a20 con 0x7f38780756d0 2024-05-10T11:31:02.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.426+0000 7f3887dedb80 1 Processor -- start 2024-05-10T11:31:02.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.426+0000 7f386f400700 1 -- 172.21.15.27:0/3930944698 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f386802a070 con 0x7f38780756d0 2024-05-10T11:31:02.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.428+0000 7f386f400700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f385c0b2bc0 0x7f385c0b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.428+0000 7f386f400700 1 -- 172.21.15.27:0/3930944698 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9679+0+0 (secure 0 0 0) 0x7f38680040d0 con 0x7f38780756d0 2024-05-10T11:31:02.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.428+0000 7f3877200700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f385c0b2bc0 0x7f385c0b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.429+0000 7f3887dedb80 1 -- start start 2024-05-10T11:31:02.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.429+0000 7f3887dedb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb2920e0 0x5618cb2924c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.429+0000 7f3887dedb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb292a00 0x5618cb296510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.430+0000 7f3887dedb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb296a50 0x5618cb296eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.430+0000 7f3887dedb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5618cb1e4aa0 con 0x5618cb2920e0 2024-05-10T11:31:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.430+0000 7f3887dedb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5618cb1827e0 con 0x5618cb292a00 2024-05-10T11:31:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.430+0000 7f3887dedb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5618cb1e0a60 con 0x5618cb296a50 2024-05-10T11:31:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.430+0000 7f3876800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb2920e0 0x5618cb2924c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.430+0000 7f3876800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb2920e0 0x5618cb2924c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60204/0 (socket says 172.21.15.27:60204) 2024-05-10T11:31:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.430+0000 7f3876800700 1 -- 172.21.15.27:0/679766525 learned_addr learned my addr 172.21.15.27:0/679766525 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.430+0000 7f3877200700 1 --2- 172.21.15.27:0/679766525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb296a50 0x5618cb296eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.430+0000 7f3875e00700 1 --2- 172.21.15.27:0/679766525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb292a00 0x5618cb296510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.430+0000 7f3877200700 1 -- 172.21.15.27:0/679766525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb292a00 msgr2=0x5618cb296510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.430+0000 7f3877200700 1 --2- 172.21.15.27:0/679766525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb292a00 0x5618cb296510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.431+0000 7f3877200700 1 -- 172.21.15.27:0/679766525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb2920e0 msgr2=0x5618cb2924c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:02.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.431+0000 7f3877200700 1 --2- 172.21.15.27:0/679766525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb2920e0 0x5618cb2924c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.431+0000 7f3877200700 1 -- 172.21.15.27:0/679766525 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618cb131950 con 0x5618cb296a50 2024-05-10T11:31:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.431+0000 7f3876800700 1 --2- 172.21.15.27:0/679766525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb2920e0 0x5618cb2924c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:31:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.431+0000 7f3877200700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f385c0b2bc0 0x7f385c0b5080 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3870007fd0 tx=0x7f3870009040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.431+0000 7f3877200700 1 --2- 172.21.15.27:0/679766525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb296a50 0x5618cb296eb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f38700166e0 tx=0x7f3870004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae7ffaf86e6e7584 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.432+0000 7f3863e00700 1 -- 172.21.15.27:0/679766525 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870016d70 con 0x5618cb296a50 2024-05-10T11:31:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.432+0000 7f3863e00700 1 -- 172.21.15.27:0/679766525 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3870024070 con 0x5618cb296a50 2024-05-10T11:31:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.432+0000 7f3863e00700 1 -- 172.21.15.27:0/679766525 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870004730 con 0x5618cb296a50 2024-05-10T11:31:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.432+0000 7f3887dedb80 1 -- 172.21.15.27:0/679766525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb296a50 msgr2=0x5618cb296eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.432+0000 7f3887dedb80 1 --2- 172.21.15.27:0/679766525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb296a50 0x5618cb296eb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f38700166e0 tx=0x7f3870004ad0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.433+0000 7f3887dedb80 1 -- 172.21.15.27:0/679766525 shutdown_connections 2024-05-10T11:31:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.433+0000 7f3887dedb80 1 --2- 172.21.15.27:0/679766525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb296a50 0x5618cb296eb0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.433+0000 7f3887dedb80 1 --2- 172.21.15.27:0/679766525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb2920e0 0x5618cb2924c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.433+0000 7f3887dedb80 1 --2- 172.21.15.27:0/679766525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb292a00 0x5618cb296510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.433+0000 7f3887dedb80 1 -- 172.21.15.27:0/679766525 >> 172.21.15.27:0/679766525 conn(0x5618cb1c0be0 msgr2=0x5618cb28c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.433+0000 7f3887dedb80 1 -- 172.21.15.27:0/679766525 shutdown_connections 2024-05-10T11:31:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.434+0000 7f3887dedb80 1 -- 172.21.15.27:0/679766525 wait complete. 2024-05-10T11:31:02.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.435+0000 7f3887dedb80 1 -- start start 2024-05-10T11:31:02.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.435+0000 7f3887dedb80 1 Processor -- start 2024-05-10T11:31:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.436+0000 7f3887dedb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb2920e0 0x5618cb29df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.436+0000 7f3887dedb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 0x5618cb29b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.436+0000 7f3887dedb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb296a50 0x5618cb29b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.436+0000 7f3887dedb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5618cb29ec00 con 0x5618cb2920e0 2024-05-10T11:31:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.436+0000 7f3887dedb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5618cb14aba0 con 0x5618cb296a50 2024-05-10T11:31:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.436+0000 7f3887dedb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5618cb29ed80 con 0x5618cb292a00 2024-05-10T11:31:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.436+0000 7f3887dedb80 10 client.0 _ll_register_callbacks cb 0x5618cb18bcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:31:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.437+0000 7f3875e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 0x5618cb29b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.437+0000 7f3875e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 0x5618cb29b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49574/0 (socket says 172.21.15.27:49574) 2024-05-10T11:31:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.437+0000 7f3875e00700 1 -- 172.21.15.27:0/3926383016 learned_addr learned my addr 172.21.15.27:0/3926383016 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.437+0000 7f3877200700 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb296a50 0x5618cb29b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.437+0000 7f3876800700 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb2920e0 0x5618cb29df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.437+0000 7f3877200700 1 -- 172.21.15.27:0/3926383016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 msgr2=0x5618cb29b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.437+0000 7f3877200700 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 0x5618cb29b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.437+0000 7f3877200700 1 -- 172.21.15.27:0/3926383016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb2920e0 msgr2=0x5618cb29df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.437+0000 7f3877200700 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb2920e0 0x5618cb29df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.437+0000 7f3877200700 1 -- 172.21.15.27:0/3926383016 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618cb1aa4a0 con 0x5618cb296a50 2024-05-10T11:31:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.438+0000 7f3877200700 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb296a50 0x5618cb29b810 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f387000ce10 tx=0x7f3870004800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7022a23ca3099641 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.438+0000 7f3860c00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:31:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.438+0000 7f3860c00700 1 -- 172.21.15.27:0/3926383016 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387002f070 con 0x5618cb296a50 2024-05-10T11:31:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.438+0000 7f3860c00700 1 -- 172.21.15.27:0/3926383016 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9679+0+0 (secure 0 0 0) 0x7f3870003d90 con 0x5618cb296a50 2024-05-10T11:31:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.439+0000 7f3887dedb80 1 -- 172.21.15.27:0/3926383016 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3870024070 con 0x5618cb296a50 2024-05-10T11:31:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.439+0000 7f3860c00700 1 -- 172.21.15.27:0/3926383016 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f387003b050 con 0x5618cb296a50 2024-05-10T11:31:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.439+0000 7f3887dedb80 10 client.27635 fetch_fsmap learned FSMap version 45 2024-05-10T11:31:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.439+0000 7f3887dedb80 1 -- 172.21.15.27:0/3926383016 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5618cb2a94d0 con 0x5618cb296a50 2024-05-10T11:31:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.439+0000 7f3860c00700 1 -- 172.21.15.27:0/3926383016 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3870024070 con 0x5618cb296a50 2024-05-10T11:31:02.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.440+0000 7f3860c00700 1 -- 172.21.15.27:0/3926383016 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f387001f640 con 0x5618cb296a50 2024-05-10T11:31:02.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.440+0000 7f3887dedb80 10 client.27635 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:31:02.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.440+0000 7f3887dedb80 10 client.27635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:31:02.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.440+0000 7f3887dedb80 20 client.27635 populate_metadata read hostname 'smithi027' 2024-05-10T11:31:02.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.440+0000 7f3887dedb80 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5618cb2aba60 0x5618cb2ade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.440+0000 7f3887dedb80 4 client.27635 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:31:02.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.440+0000 7f3887dedb80 1 -- 172.21.15.27:0/3926383016 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618cb25f120 con 0x5618cb2aba60 2024-05-10T11:31:02.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.441+0000 7f3876800700 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5618cb2aba60 0x5618cb2ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.444+0000 7f3876800700 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5618cb2aba60 0x5618cb2ade50 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a25346d40953a8fa server_cookie=8ce587ca2ed942c in_seq=0 out_seq=0 2024-05-10T11:31:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.444+0000 7f3860c00700 10 client.27635 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:31:02.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.449+0000 7f3860c00700 1 -- 172.21.15.27:0/3926383016 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5618cb25f120 con 0x5618cb2aba60 2024-05-10T11:31:02.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.449+0000 7f3860c00700 10 client.27635 handle_command_reply: tid=0 2024-05-10T11:31:02.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.449+0000 7f3887dedb80 1 client.27635 shutdown 2024-05-10T11:31:02.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.450+0000 7f3887dedb80 1 -- 172.21.15.27:0/3926383016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb296a50 msgr2=0x5618cb29b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:02.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.450+0000 7f3887dedb80 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb296a50 0x5618cb29b810 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f387000ce10 tx=0x7f3870004800 comp rx=0 tx=0).stop 2024-05-10T11:31:02.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.454+0000 7f3887dedb80 1 -- 172.21.15.27:0/3926383016 shutdown_connections 2024-05-10T11:31:02.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.454+0000 7f3887dedb80 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5618cb2aba60 0x5618cb2ade50 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.454+0000 7f3887dedb80 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb296a50 0x5618cb29b810 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.454+0000 7f3887dedb80 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb2920e0 0x5618cb29df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.454+0000 7f3887dedb80 1 --2- 172.21.15.27:0/3926383016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 0x5618cb29b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.454+0000 7f3887dedb80 1 -- 172.21.15.27:0/3926383016 >> 172.21.15.27:0/3926383016 conn(0x5618cb1c0be0 msgr2=0x5618cb287330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:02.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.454+0000 7f3860c00700 0 client.27635 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:31:02.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.454+0000 7f3887dedb80 1 -- 172.21.15.27:0/3926383016 shutdown_connections 2024-05-10T11:31:02.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.454+0000 7f3887dedb80 1 -- 172.21.15.27:0/3926383016 wait complete. 2024-05-10T11:31:02.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.455+0000 7f3887dedb80 20 client.27635 trim_cache size 0 max 16384 2024-05-10T11:31:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.499+0000 7f3887dedb80 1 Processor -- start 2024-05-10T11:31:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.499+0000 7f3887dedb80 1 -- start start 2024-05-10T11:31:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.499+0000 7f3887dedb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb2920e0 0x5618cb298f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.499+0000 7f3887dedb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb292a00 0x5618cb29b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.500+0000 7f3887dedb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb296a50 0x5618cb2a7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.500+0000 7f3887dedb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f38781be440 con 0x5618cb296a50 2024-05-10T11:31:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.500+0000 7f3887dedb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f38781bff80 con 0x5618cb292a00 2024-05-10T11:31:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.500+0000 7f3887dedb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38781beb10 con 0x5618cb2920e0 2024-05-10T11:31:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.500+0000 7f3876800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb2920e0 0x5618cb298f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.500+0000 7f3876800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb2920e0 0x5618cb298f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49576/0 (socket says 172.21.15.27:49576) 2024-05-10T11:31:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.500+0000 7f3876800700 1 -- 172.21.15.27:0/1714813877 learned_addr learned my addr 172.21.15.27:0/1714813877 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.500+0000 7f3876800700 1 -- 172.21.15.27:0/1714813877 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb292a00 msgr2=0x5618cb29b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.500+0000 7f3876800700 1 --2- 172.21.15.27:0/1714813877 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb292a00 0x5618cb29b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.500+0000 7f3876800700 1 -- 172.21.15.27:0/1714813877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb296a50 msgr2=0x5618cb2a7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.501+0000 7f3877200700 1 --2- 172.21.15.27:0/1714813877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb296a50 0x5618cb2a7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.501+0000 7f3876800700 1 --2- 172.21.15.27:0/1714813877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb296a50 0x5618cb2a7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.501+0000 7f3876800700 1 -- 172.21.15.27:0/1714813877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618cb297cd0 con 0x5618cb2920e0 2024-05-10T11:31:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.501+0000 7f3877200700 1 --2- 172.21.15.27:0/1714813877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb296a50 0x5618cb2a7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:31:02.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.502+0000 7f3876800700 1 --2- 172.21.15.27:0/1714813877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb2920e0 0x5618cb298f10 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3868009750 tx=0x7f3868009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99884b8da011ea14 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:02.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.502+0000 7f3863400700 1 -- 172.21.15.27:0/1714813877 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38680abe60 con 0x5618cb2920e0 2024-05-10T11:31:02.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.502+0000 7f3863400700 1 -- 172.21.15.27:0/1714813877 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38680ab760 con 0x5618cb2920e0 2024-05-10T11:31:02.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.504+0000 7f3887dedb80 1 -- 172.21.15.27:0/1714813877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb2920e0 msgr2=0x5618cb298f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:02.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.504+0000 7f3887dedb80 1 --2- 172.21.15.27:0/1714813877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb2920e0 0x5618cb298f10 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3868009750 tx=0x7f3868009310 comp rx=0 tx=0).stop 2024-05-10T11:31:02.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.504+0000 7f3887dedb80 1 -- 172.21.15.27:0/1714813877 shutdown_connections 2024-05-10T11:31:02.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.504+0000 7f3887dedb80 1 --2- 172.21.15.27:0/1714813877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb2920e0 0x5618cb298f10 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.504+0000 7f3887dedb80 1 --2- 172.21.15.27:0/1714813877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb296a50 0x5618cb2a7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.504+0000 7f3887dedb80 1 --2- 172.21.15.27:0/1714813877 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb292a00 0x5618cb29b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.504+0000 7f3887dedb80 1 -- 172.21.15.27:0/1714813877 >> 172.21.15.27:0/1714813877 conn(0x5618cb1c0be0 msgr2=0x5618cb286aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:02.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.504+0000 7f3887dedb80 1 -- 172.21.15.27:0/1714813877 shutdown_connections 2024-05-10T11:31:02.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.505+0000 7f3887dedb80 1 -- 172.21.15.27:0/1714813877 wait complete. 2024-05-10T11:31:02.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.506+0000 7f3887dedb80 1 -- start start 2024-05-10T11:31:02.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.506+0000 7f3887dedb80 1 Processor -- start 2024-05-10T11:31:02.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.507+0000 7f3887dedb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb2920e0 0x5618cb2b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.508+0000 7f3887dedb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 0x5618cb2b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.508+0000 7f3887dedb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb296a50 0x5618cb2a2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.508+0000 7f3887dedb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f38781bf8b0 con 0x5618cb296a50 2024-05-10T11:31:02.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.509+0000 7f3887dedb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f38781bf1e0 con 0x5618cb2920e0 2024-05-10T11:31:02.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.509+0000 7f3887dedb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38781bc400 con 0x5618cb292a00 2024-05-10T11:31:02.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.509+0000 7f3875e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 0x5618cb2b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.509+0000 7f3876800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb2920e0 0x5618cb2b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.509+0000 7f3875e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 0x5618cb2b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:49578/0 (socket says 172.21.15.27:49578) 2024-05-10T11:31:02.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.509+0000 7f3875e00700 1 -- 172.21.15.27:0/1649875425 learned_addr learned my addr 172.21.15.27:0/1649875425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:02.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.510+0000 7f3887dedb80 10 client.0 _ll_register_callbacks cb 0x5618cb18bcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:31:02.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.511+0000 7f3877200700 1 --2- 172.21.15.27:0/1649875425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb296a50 0x5618cb2a2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.511+0000 7f3875e00700 1 -- 172.21.15.27:0/1649875425 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb2920e0 msgr2=0x5618cb2b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:02.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.511+0000 7f3875e00700 1 --2- 172.21.15.27:0/1649875425 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb2920e0 0x5618cb2b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.511+0000 7f3875e00700 1 -- 172.21.15.27:0/1649875425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb296a50 msgr2=0x5618cb2a2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:02.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.511+0000 7f3875e00700 1 --2- 172.21.15.27:0/1649875425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb296a50 0x5618cb2a2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.512+0000 7f3875e00700 1 -- 172.21.15.27:0/1649875425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618cb2a2c50 con 0x5618cb292a00 2024-05-10T11:31:02.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.512+0000 7f3860c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:31:02.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.512+0000 7f3875e00700 1 --2- 172.21.15.27:0/1649875425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 0x5618cb2b35a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f386400bb40 tx=0x7f386400be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=565d38c02afe4855 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:02.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.513+0000 7f3860c00700 1 -- 172.21.15.27:0/1649875425 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3864004830 con 0x5618cb292a00 2024-05-10T11:31:02.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.513+0000 7f3887dedb80 1 -- 172.21.15.27:0/1649875425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38680ab760 con 0x5618cb292a00 2024-05-10T11:31:02.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.513+0000 7f3860c00700 1 -- 172.21.15.27:0/1649875425 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9679+0+0 (secure 0 0 0) 0x7f3864003d00 con 0x5618cb292a00 2024-05-10T11:31:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.514+0000 7f3860c00700 1 -- 172.21.15.27:0/1649875425 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f3864026050 con 0x5618cb292a00 2024-05-10T11:31:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.514+0000 7f3887dedb80 10 client.27652 fetch_fsmap learned FSMap version 45 2024-05-10T11:31:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.514+0000 7f3887dedb80 1 -- 172.21.15.27:0/1649875425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5618cb316720 con 0x5618cb292a00 2024-05-10T11:31:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.515+0000 7f3860c00700 1 -- 172.21.15.27:0/1649875425 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38680ab760 con 0x5618cb292a00 2024-05-10T11:31:02.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.515+0000 7f3860c00700 1 -- 172.21.15.27:0/1649875425 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f3864011440 con 0x5618cb292a00 2024-05-10T11:31:02.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.515+0000 7f3887dedb80 10 client.27652 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:31:02.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.515+0000 7f3887dedb80 10 client.27652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:31:02.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.515+0000 7f3887dedb80 20 client.27652 populate_metadata read hostname 'smithi027' 2024-05-10T11:31:02.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.515+0000 7f3887dedb80 1 --2- 172.21.15.27:0/1649875425 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5618cb316e60 0x5618cb319280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:02.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.516+0000 7f3887dedb80 4 client.27652 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-10T11:31:02.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.516+0000 7f3887dedb80 1 -- 172.21.15.27:0/1649875425 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5618cb1f2480 con 0x5618cb316e60 2024-05-10T11:31:02.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.516+0000 7f3876800700 1 --2- 172.21.15.27:0/1649875425 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5618cb316e60 0x5618cb319280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:02.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.517+0000 7f3876800700 1 --2- 172.21.15.27:0/1649875425 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5618cb316e60 0x5618cb319280 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53dbfa00ce8623e8 server_cookie=425f6336ff591d94 in_seq=0 out_seq=0 2024-05-10T11:31:02.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.517+0000 7f3860c00700 10 client.27652 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:31:02.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.519+0000 7f3860c00700 1 -- 172.21.15.27:0/1649875425 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5618cb1f2480 con 0x5618cb316e60 2024-05-10T11:31:02.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.519+0000 7f3860c00700 10 client.27652 handle_command_reply: tid=0 2024-05-10T11:31:02.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.520+0000 7f3887dedb80 1 client.27652 shutdown 2024-05-10T11:31:02.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.520+0000 7f3887dedb80 1 -- 172.21.15.27:0/1649875425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 msgr2=0x5618cb2b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:02.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.520+0000 7f3887dedb80 1 --2- 172.21.15.27:0/1649875425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 0x5618cb2b35a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f386400bb40 tx=0x7f386400be50 comp rx=0 tx=0).stop 2024-05-10T11:31:02.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.521+0000 7f3887dedb80 1 -- 172.21.15.27:0/1649875425 shutdown_connections 2024-05-10T11:31:02.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.521+0000 7f3887dedb80 1 --2- 172.21.15.27:0/1649875425 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5618cb316e60 0x5618cb319280 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.521+0000 7f3887dedb80 1 --2- 172.21.15.27:0/1649875425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618cb296a50 0x5618cb2a2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.521+0000 7f3887dedb80 1 --2- 172.21.15.27:0/1649875425 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5618cb2920e0 0x5618cb2b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.521+0000 7f3887dedb80 1 --2- 172.21.15.27:0/1649875425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618cb292a00 0x5618cb2b35a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.521+0000 7f3887dedb80 1 -- 172.21.15.27:0/1649875425 >> 172.21.15.27:0/1649875425 conn(0x5618cb1c0be0 msgr2=0x5618cb287330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:02.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.521+0000 7f3860c00700 0 client.27652 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:31:02.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.521+0000 7f3887dedb80 1 -- 172.21.15.27:0/1649875425 shutdown_connections 2024-05-10T11:31:02.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.522+0000 7f3887dedb80 1 -- 172.21.15.27:0/1649875425 wait complete. 2024-05-10T11:31:02.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.522+0000 7f3887dedb80 20 client.27652 trim_cache size 0 max 16384 2024-05-10T11:31:02.523 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:31:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-05-10T11:31:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "ed3c6d44-d154-495a-888b-c0408db30ced", 2024-05-10T11:31:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-05-10T11:31:02.524 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:31:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.523+0000 7f387d400700 1 -- 172.21.15.27:0/3930944698 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f385c0b2bc0 msgr2=0x7f385c0b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.523+0000 7f387d400700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f385c0b2bc0 0x7f385c0b5080 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3870007fd0 tx=0x7f3870009040 comp rx=0 tx=0).stop 2024-05-10T11:31:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.523+0000 7f387d400700 1 -- 172.21.15.27:0/3930944698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38780756d0 msgr2=0x7f3878074cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.523+0000 7f387d400700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38780756d0 0x7f3878074cc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3868000c00 tx=0x7f38680047b0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.524+0000 7f387d400700 1 -- 172.21.15.27:0/3930944698 shutdown_connections 2024-05-10T11:31:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.524+0000 7f387d400700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f385c0b2bc0 0x7f385c0b5080 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.524+0000 7f387d400700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38780756d0 0x7f3878074cc0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.524+0000 7f387d400700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3878114670 0x7f387806e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.524+0000 7f387d400700 1 --2- 172.21.15.27:0/3930944698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3878075ff0 0x7f387806dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:02.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.524+0000 7f387d400700 1 -- 172.21.15.27:0/3930944698 >> 172.21.15.27:0/3930944698 conn(0x7f387807a600 msgr2=0x7f38781134b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:02.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.524+0000 7f387d400700 1 -- 172.21.15.27:0/3930944698 shutdown_connections 2024-05-10T11:31:02.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:02.524+0000 7f387d400700 1 -- 172.21.15.27:0/3930944698 wait complete. 2024-05-10T11:31:03.176 INFO:tasks.workunit.client.0.smithi027.stdout: 10 97559 22.37 MB/sec execute 107 sec latency 472.490 ms 2024-05-10T11:31:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: pgmap v559: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 8.1 MiB/s wr, 422 op/s 2024-05-10T11:31:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: osdmap e114: 12 total, 12 up, 12 in 2024-05-10T11:31:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub queued for path: / 2024-05-10T11:31:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:03.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:03.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:03 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: pgmap v559: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 8.1 MiB/s wr, 422 op/s 2024-05-10T11:31:03.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: osdmap e114: 12 total, 12 up, 12 in 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub queued for path: / 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:03.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:03.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:03 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: pgmap v559: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 8.1 MiB/s wr, 422 op/s 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: osdmap e114: 12 total, 12 up, 12 in 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub queued for path: / 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:03.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:03.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:03.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:03 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.176 INFO:tasks.workunit.client.0.smithi027.stdout: 10 97568 22.18 MB/sec execute 108 sec latency 432.781 ms 2024-05-10T11:31:04.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:04.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: active paths [/] 2024-05-10T11:31:04.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:04 smithi027 ceph-mon[30054]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:04.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:04.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:04.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:04.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:04.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:04.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: active paths [/] 2024-05-10T11:31:04.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:04 smithi076 ceph-mon[34152]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:04.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: active paths [/] 2024-05-10T11:31:04.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:04.738 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:04 smithi189 ceph-mon[34842]: scrub summary: idle+waiting paths [/] 2024-05-10T11:31:05.163 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:31:05.176 INFO:tasks.workunit.client.0.smithi027.stdout: 10 97601 22.02 MB/sec execute 109 sec latency 693.481 ms 2024-05-10T11:31:05.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:05 smithi027 ceph-mon[30054]: pgmap v561: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 9.3 MiB/s wr, 480 op/s 2024-05-10T11:31:05.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:05 smithi076 ceph-mon[34152]: pgmap v561: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 9.3 MiB/s wr, 480 op/s 2024-05-10T11:31:05.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:05 smithi189 ceph-mon[34842]: pgmap v561: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 9.3 MiB/s wr, 480 op/s 2024-05-10T11:31:05.870 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:31:06.176 INFO:tasks.workunit.client.0.smithi027.stdout: 10 97710 21.89 MB/sec execute 110 sec latency 419.298 ms 2024-05-10T11:31:07.176 INFO:tasks.workunit.client.0.smithi027.stdout: 10 97940 21.83 MB/sec execute 111 sec latency 685.962 ms 2024-05-10T11:31:07.643 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:07 smithi027 ceph-mon[30054]: pgmap v562: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 7.9 MiB/s wr, 590 op/s 2024-05-10T11:31:07.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:07 smithi076 ceph-mon[34152]: pgmap v562: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 7.9 MiB/s wr, 590 op/s 2024-05-10T11:31:07.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:07 smithi189 ceph-mon[34842]: pgmap v562: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 7.9 MiB/s wr, 590 op/s 2024-05-10T11:31:08.177 INFO:tasks.workunit.client.0.smithi027.stdout: 10 98476 21.82 MB/sec execute 112 sec latency 372.161 ms 2024-05-10T11:31:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.543+0000 7f3f5bc00700 1 -- 172.21.15.27:0/2886654540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f541060a0 msgr2=0x7f3f541125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.543+0000 7f3f5bc00700 1 --2- 172.21.15.27:0/2886654540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f541060a0 0x7f3f541125b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00b0a0 tx=0x7f3f4c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.544+0000 7f3f5bc00700 1 -- 172.21.15.27:0/2886654540 shutdown_connections 2024-05-10T11:31:08.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.544+0000 7f3f5bc00700 1 --2- 172.21.15.27:0/2886654540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f541060a0 0x7f3f541125b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.544+0000 7f3f5bc00700 1 --2- 172.21.15.27:0/2886654540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f5410b6e0 0x7f3f5410bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.544+0000 7f3f5bc00700 1 --2- 172.21.15.27:0/2886654540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f541056e0 0x7f3f54105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.544+0000 7f3f5bc00700 1 -- 172.21.15.27:0/2886654540 >> 172.21.15.27:0/2886654540 conn(0x7f3f541011d0 msgr2=0x7f3f541035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:08.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.544+0000 7f3f5bc00700 1 -- 172.21.15.27:0/2886654540 shutdown_connections 2024-05-10T11:31:08.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.544+0000 7f3f5bc00700 1 -- 172.21.15.27:0/2886654540 wait complete. 2024-05-10T11:31:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.546+0000 7f3f5bc00700 1 Processor -- start 2024-05-10T11:31:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.546+0000 7f3f5bc00700 1 -- start start 2024-05-10T11:31:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.546+0000 7f3f5bc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f541056e0 0x7f3f541123f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.546+0000 7f3f5bc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f541060a0 0x7f3f5410f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.546+0000 7f3f5bc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5410b6e0 0x7f3f5410fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.546+0000 7f3f5bc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f54115930 con 0x7f3f541056e0 2024-05-10T11:31:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.546+0000 7f3f5bc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541157b0 con 0x7f3f5410b6e0 2024-05-10T11:31:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.546+0000 7f3f5bc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f54115ab0 con 0x7f3f541060a0 2024-05-10T11:31:08.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.547+0000 7f3f53e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f541060a0 0x7f3f5410f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.547+0000 7f3f53e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f541060a0 0x7f3f5410f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:47278/0 (socket says 172.21.15.27:47278) 2024-05-10T11:31:08.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.547+0000 7f3f53e00700 1 -- 172.21.15.27:0/1444763175 learned_addr learned my addr 172.21.15.27:0/1444763175 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:08.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.547+0000 7f3f59200700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f541056e0 0x7f3f541123f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.547+0000 7f3f59c00700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5410b6e0 0x7f3f5410fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.548+0000 7f3f53e00700 1 -- 172.21.15.27:0/1444763175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5410b6e0 msgr2=0x7f3f5410fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:08.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.548+0000 7f3f53e00700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5410b6e0 0x7f3f5410fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.548+0000 7f3f53e00700 1 -- 172.21.15.27:0/1444763175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f541056e0 msgr2=0x7f3f541123f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:08.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.548+0000 7f3f53e00700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f541056e0 0x7f3f541123f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.548+0000 7f3f53e00700 1 -- 172.21.15.27:0/1444763175 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f541102e0 con 0x7f3f541060a0 2024-05-10T11:31:08.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.549+0000 7f3f53e00700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f541060a0 0x7f3f5410f4f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000e9c0 tx=0x7f3f4000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.549+0000 7f3f59200700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f541056e0 0x7f3f541123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:31:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.549+0000 7f3f51600700 1 -- 172.21.15.27:0/1444763175 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4000cbd0 con 0x7f3f541060a0 2024-05-10T11:31:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.549+0000 7f3f51600700 1 -- 172.21.15.27:0/1444763175 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f4000cd70 con 0x7f3f541060a0 2024-05-10T11:31:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.549+0000 7f3f5bc00700 1 -- 172.21.15.27:0/1444763175 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f541ac750 con 0x7f3f541060a0 2024-05-10T11:31:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.549+0000 7f3f51600700 1 -- 172.21.15.27:0/1444763175 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40005610 con 0x7f3f541060a0 2024-05-10T11:31:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.549+0000 7f3f5bc00700 1 -- 172.21.15.27:0/1444763175 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f541acc60 con 0x7f3f541060a0 2024-05-10T11:31:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.554+0000 7f3f51600700 1 -- 172.21.15.27:0/1444763175 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3f40004750 con 0x7f3f541060a0 2024-05-10T11:31:08.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.556+0000 7f3f6663ab80 1 Processor -- start 2024-05-10T11:31:08.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.557+0000 7f3f6663ab80 1 -- start start 2024-05-10T11:31:08.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.557+0000 7f3f6663ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 0x55712c789fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.558+0000 7f3f6663ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c7842d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.558+0000 7f3f6663ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c784d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.558+0000 7f3f6663ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55712c6d6a20 con 0x55712c789bc0 2024-05-10T11:31:08.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.558+0000 7f3f6663ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55712c674760 con 0x55712c78a4e0 2024-05-10T11:31:08.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.558+0000 7f3f6663ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55712c6d29e0 con 0x55712c7848f0 2024-05-10T11:31:08.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.558+0000 7f3f53e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c784d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.558+0000 7f3f59200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c7842d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.558+0000 7f3f53e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c784d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:47284/0 (socket says 172.21.15.27:47284) 2024-05-10T11:31:08.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.558+0000 7f3f53e00700 1 -- 172.21.15.27:0/496531475 learned_addr learned my addr 172.21.15.27:0/496531475 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.558+0000 7f3f51600700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3f3c0b2af0 0x7f3f3c0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.559+0000 7f3f53e00700 1 -- 172.21.15.27:0/496531475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 msgr2=0x55712c7842d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.559+0000 7f3f53e00700 1 --2- 172.21.15.27:0/496531475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c7842d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.559+0000 7f3f53e00700 1 -- 172.21.15.27:0/496531475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 msgr2=0x55712c789fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.559+0000 7f3f53e00700 1 --2- 172.21.15.27:0/496531475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 0x55712c789fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.559+0000 7f3f53e00700 1 -- 172.21.15.27:0/496531475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55712c6238d0 con 0x55712c7848f0 2024-05-10T11:31:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.559+0000 7f3f59c00700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3f3c0b2af0 0x7f3f3c0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.559+0000 7f3f53e00700 1 --2- 172.21.15.27:0/496531475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c784d70 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3f400dff40 tx=0x7f3f4006f080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=476deba7655e24c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:08.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.560+0000 7f3f51600700 1 -- 172.21.15.27:0/1444763175 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9679+0+0 (secure 0 0 0) 0x7f3f400e2d90 con 0x7f3f541060a0 2024-05-10T11:31:08.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.560+0000 7f3f46000700 1 -- 172.21.15.27:0/496531475 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f400a9960 con 0x55712c7848f0 2024-05-10T11:31:08.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.560+0000 7f3f59c00700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3f3c0b2af0 0x7f3f3c0b4fb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00b780 tx=0x7f3f4c006000 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:08.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.561+0000 7f3f46000700 1 -- 172.21.15.27:0/496531475 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f4007f7a0 con 0x55712c7848f0 2024-05-10T11:31:08.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.561+0000 7f3f46000700 1 -- 172.21.15.27:0/496531475 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f400a5320 con 0x55712c7848f0 2024-05-10T11:31:08.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.561+0000 7f3f6663ab80 1 -- 172.21.15.27:0/496531475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 msgr2=0x55712c784d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:08.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.561+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/496531475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c784d70 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3f400dff40 tx=0x7f3f4006f080 comp rx=0 tx=0).stop 2024-05-10T11:31:08.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.562+0000 7f3f6663ab80 1 -- 172.21.15.27:0/496531475 shutdown_connections 2024-05-10T11:31:08.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.562+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/496531475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c784d70 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.562+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/496531475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 0x55712c789fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.562+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/496531475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c7842d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.562+0000 7f3f6663ab80 1 -- 172.21.15.27:0/496531475 >> 172.21.15.27:0/496531475 conn(0x55712c6fc6d0 msgr2=0x55712c77eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:08.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.562+0000 7f3f6663ab80 1 -- 172.21.15.27:0/496531475 shutdown_connections 2024-05-10T11:31:08.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.562+0000 7f3f6663ab80 1 -- 172.21.15.27:0/496531475 wait complete. 2024-05-10T11:31:08.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.563+0000 7f3f6663ab80 1 -- start start 2024-05-10T11:31:08.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.563+0000 7f3f6663ab80 1 Processor -- start 2024-05-10T11:31:08.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.564+0000 7f3f6663ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c7848f0 0x55712c790140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.564+0000 7f3f6663ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c789bc0 0x55712c78d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.564+0000 7f3f6663ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c78da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.564+0000 7f3f6663ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55712c790ec0 con 0x55712c7848f0 2024-05-10T11:31:08.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.564+0000 7f3f6663ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55712c63cb20 con 0x55712c78a4e0 2024-05-10T11:31:08.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.564+0000 7f3f6663ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55712c791040 con 0x55712c789bc0 2024-05-10T11:31:08.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.564+0000 7f3f6663ab80 10 client.0 _ll_register_callbacks cb 0x55712c67dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:31:08.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.565+0000 7f3f59200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c7848f0 0x55712c790140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.565+0000 7f3f59c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c78da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.565+0000 7f3f59200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c7848f0 0x55712c790140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43206/0 (socket says 172.21.15.27:43206) 2024-05-10T11:31:08.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.565+0000 7f3f59200700 1 -- 172.21.15.27:0/1317949699 learned_addr learned my addr 172.21.15.27:0/1317949699 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:08.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.564+0000 7f3f53e00700 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c789bc0 0x55712c78d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.566+0000 7f3f59c00700 1 -- 172.21.15.27:0/1317949699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c789bc0 msgr2=0x55712c78d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:08.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.566+0000 7f3f59c00700 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c789bc0 0x55712c78d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.566+0000 7f3f59c00700 1 -- 172.21.15.27:0/1317949699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c7848f0 msgr2=0x55712c790140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:08.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.566+0000 7f3f59c00700 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c7848f0 0x55712c790140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.566+0000 7f3f59c00700 1 -- 172.21.15.27:0/1317949699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55712c69c420 con 0x55712c78a4e0 2024-05-10T11:31:08.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.566+0000 7f3f59200700 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c7848f0 0x55712c790140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:31:08.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.566+0000 7f3f53e00700 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c789bc0 0x55712c78d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:31:08.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.566+0000 7f3f59c00700 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c78da20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00b070 tx=0x7f3f4c00b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8bfaed1dde0b8fd server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:08.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.566+0000 7f3f3aa00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:31:08.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.567+0000 7f3f3aa00700 1 -- 172.21.15.27:0/1317949699 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c02d070 con 0x55712c78a4e0 2024-05-10T11:31:08.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.567+0000 7f3f3aa00700 1 -- 172.21.15.27:0/1317949699 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9679+0+0 (secure 0 0 0) 0x7f3f4c008df0 con 0x55712c78a4e0 2024-05-10T11:31:08.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.567+0000 7f3f6663ab80 1 -- 172.21.15.27:0/1317949699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55712c751140 con 0x55712c78a4e0 2024-05-10T11:31:08.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.567+0000 7f3f3aa00700 1 -- 172.21.15.27:0/1317949699 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f3f4c037050 con 0x55712c78a4e0 2024-05-10T11:31:08.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.567+0000 7f3f6663ab80 10 client.27650 fetch_fsmap learned FSMap version 45 2024-05-10T11:31:08.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.567+0000 7f3f6663ab80 1 -- 172.21.15.27:0/1317949699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55712c79b6c0 con 0x55712c78a4e0 2024-05-10T11:31:08.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.568+0000 7f3f3aa00700 1 -- 172.21.15.27:0/1317949699 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55712c751140 con 0x55712c78a4e0 2024-05-10T11:31:08.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.568+0000 7f3f3aa00700 1 -- 172.21.15.27:0/1317949699 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f3f4c011070 con 0x55712c78a4e0 2024-05-10T11:31:08.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.568+0000 7f3f6663ab80 10 client.27650 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:31:08.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.568+0000 7f3f6663ab80 10 client.27650 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:31:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.568+0000 7f3f6663ab80 20 client.27650 populate_metadata read hostname 'smithi027' 2024-05-10T11:31:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.568+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55712c79dd00 0x55712c7a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.568+0000 7f3f6663ab80 4 client.27650 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:31:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.568+0000 7f3f6663ab80 1 -- 172.21.15.27:0/1317949699 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55712c6e4400 con 0x55712c79dd00 2024-05-10T11:31:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.569+0000 7f3f59200700 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55712c79dd00 0x55712c7a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.569+0000 7f3f59200700 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55712c79dd00 0x55712c7a00f0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b99474dd1cb9ec14 server_cookie=996b2fb7b2695c1a in_seq=0 out_seq=0 2024-05-10T11:31:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.569+0000 7f3f3aa00700 10 client.27650 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:31:08.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.578+0000 7f3f3aa00700 1 -- 172.21.15.27:0/1317949699 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55712c6e4400 con 0x55712c79dd00 2024-05-10T11:31:08.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.578+0000 7f3f3aa00700 10 client.27650 handle_command_reply: tid=0 2024-05-10T11:31:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.578+0000 7f3f6663ab80 1 client.27650 shutdown 2024-05-10T11:31:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.579+0000 7f3f6663ab80 1 -- 172.21.15.27:0/1317949699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 msgr2=0x55712c78da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.579+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c78da20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00b070 tx=0x7f3f4c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.579+0000 7f3f6663ab80 1 -- 172.21.15.27:0/1317949699 shutdown_connections 2024-05-10T11:31:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.579+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55712c79dd00 0x55712c7a00f0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.579+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c78da20 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.579+0000 7f3f3aa00700 0 client.27650 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:31:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.579+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c7848f0 0x55712c790140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.579+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/1317949699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c789bc0 0x55712c78d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.579+0000 7f3f6663ab80 1 -- 172.21.15.27:0/1317949699 >> 172.21.15.27:0/1317949699 conn(0x55712c6fc6d0 msgr2=0x55712c77e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.580+0000 7f3f6663ab80 1 -- 172.21.15.27:0/1317949699 shutdown_connections 2024-05-10T11:31:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.580+0000 7f3f6663ab80 1 -- 172.21.15.27:0/1317949699 wait complete. 2024-05-10T11:31:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.580+0000 7f3f6663ab80 20 client.27650 trim_cache size 0 max 16384 2024-05-10T11:31:08.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.607+0000 7f3f6663ab80 1 Processor -- start 2024-05-10T11:31:08.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.607+0000 7f3f6663ab80 1 -- start start 2024-05-10T11:31:08.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.608+0000 7f3f6663ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c7842d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.608+0000 7f3f6663ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 0x55712c799f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.608+0000 7f3f6663ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c78afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.608+0000 7f3f6663ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541b5e90 con 0x55712c789bc0 2024-05-10T11:31:08.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.608+0000 7f3f6663ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541b79d0 con 0x55712c78a4e0 2024-05-10T11:31:08.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.608+0000 7f3f6663ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541b6560 con 0x55712c7848f0 2024-05-10T11:31:08.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.608+0000 7f3f59200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c7842d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.608+0000 7f3f59200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c7842d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:47292/0 (socket says 172.21.15.27:47292) 2024-05-10T11:31:08.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.608+0000 7f3f59200700 1 -- 172.21.15.27:0/941739047 learned_addr learned my addr 172.21.15.27:0/941739047 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:08.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.608+0000 7f3f59c00700 1 --2- 172.21.15.27:0/941739047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c78afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.608+0000 7f3f53e00700 1 --2- 172.21.15.27:0/941739047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 0x55712c799f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.609+0000 7f3f59200700 1 -- 172.21.15.27:0/941739047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 msgr2=0x55712c78afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:08.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.609+0000 7f3f59200700 1 --2- 172.21.15.27:0/941739047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c78afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.609+0000 7f3f59200700 1 -- 172.21.15.27:0/941739047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 msgr2=0x55712c799f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:08.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.609+0000 7f3f59200700 1 --2- 172.21.15.27:0/941739047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 0x55712c799f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.609+0000 7f3f59200700 1 -- 172.21.15.27:0/941739047 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55712c779ab0 con 0x55712c7848f0 2024-05-10T11:31:08.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.609+0000 7f3f59200700 1 --2- 172.21.15.27:0/941739047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c7842d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x55712c7a08e0 tx=0x7f3f48011330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a4e546c0fd8fe09 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:08.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.610+0000 7f3f45600700 1 -- 172.21.15.27:0/941739047 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48012ca0 con 0x55712c7848f0 2024-05-10T11:31:08.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.610+0000 7f3f45600700 1 -- 172.21.15.27:0/941739047 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f48022030 con 0x55712c7848f0 2024-05-10T11:31:08.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.610+0000 7f3f45600700 1 -- 172.21.15.27:0/941739047 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4800fa60 con 0x55712c7848f0 2024-05-10T11:31:08.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.610+0000 7f3f6663ab80 1 -- 172.21.15.27:0/941739047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 msgr2=0x55712c7842d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:08.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.610+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/941739047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c7842d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x55712c7a08e0 tx=0x7f3f48011330 comp rx=0 tx=0).stop 2024-05-10T11:31:08.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.611+0000 7f3f6663ab80 1 -- 172.21.15.27:0/941739047 shutdown_connections 2024-05-10T11:31:08.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.611+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/941739047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c7842d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.611+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/941739047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c78afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.611+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/941739047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 0x55712c799f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.611+0000 7f3f6663ab80 1 -- 172.21.15.27:0/941739047 >> 172.21.15.27:0/941739047 conn(0x55712c6fc6d0 msgr2=0x55712c78a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:08.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.611+0000 7f3f6663ab80 1 -- 172.21.15.27:0/941739047 shutdown_connections 2024-05-10T11:31:08.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.611+0000 7f3f6663ab80 1 -- 172.21.15.27:0/941739047 wait complete. 2024-05-10T11:31:08.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.611+0000 7f3f6663ab80 1 -- start start 2024-05-10T11:31:08.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.612+0000 7f3f6663ab80 1 Processor -- start 2024-05-10T11:31:08.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.612+0000 7f3f6663ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c7a5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.612+0000 7f3f6663ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 0x55712c7a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.612+0000 7f3f6663ab80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c7a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.612+0000 7f3f6663ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541b7300 con 0x55712c789bc0 2024-05-10T11:31:08.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.612+0000 7f3f6663ab80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541b6c30 con 0x55712c78a4e0 2024-05-10T11:31:08.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.612+0000 7f3f6663ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541b3e50 con 0x55712c7848f0 2024-05-10T11:31:08.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.612+0000 7f3f6663ab80 10 client.0 _ll_register_callbacks cb 0x55712c67dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.613+0000 7f3f59200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c7a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.613+0000 7f3f59c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c7a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.613+0000 7f3f59c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c7a5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:53218/0 (socket says 172.21.15.27:53218) 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.613+0000 7f3f59c00700 1 -- 172.21.15.27:0/2741205370 learned_addr learned my addr 172.21.15.27:0/2741205370 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.613+0000 7f3f53e00700 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 0x55712c7a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.613+0000 7f3f59c00700 1 -- 172.21.15.27:0/2741205370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 msgr2=0x55712c7a5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.613+0000 7f3f59c00700 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c7a5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.613+0000 7f3f59c00700 1 -- 172.21.15.27:0/2741205370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 msgr2=0x55712c7a57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.613+0000 7f3f59c00700 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 0x55712c7a57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.613+0000 7f3f59c00700 1 -- 172.21.15.27:0/2741205370 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55712c807690 con 0x55712c78a4e0 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.613+0000 7f3f59200700 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c7a5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.613+0000 7f3f53e00700 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 0x55712c7a57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:31:08.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.614+0000 7f3f59c00700 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c7a5d00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c009350 tx=0x7f3f4c009020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7076fa8164cfdad server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.614+0000 7f3f3aa00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.614+0000 7f3f3aa00700 1 -- 172.21.15.27:0/2741205370 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c02d040 con 0x55712c78a4e0 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.614+0000 7f3f3aa00700 1 -- 172.21.15.27:0/2741205370 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9679+0+0 (secure 0 0 0) 0x7f3f4c008220 con 0x55712c78a4e0 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.614+0000 7f3f6663ab80 1 -- 172.21.15.27:0/2741205370 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f48022030 con 0x55712c78a4e0 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.615+0000 7f3f3aa00700 1 -- 172.21.15.27:0/2741205370 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f3f4c04c050 con 0x55712c78a4e0 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.615+0000 7f3f6663ab80 10 client.27656 fetch_fsmap learned FSMap version 45 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.615+0000 7f3f6663ab80 1 -- 172.21.15.27:0/2741205370 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55712c808940 con 0x55712c78a4e0 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.615+0000 7f3f3aa00700 1 -- 172.21.15.27:0/2741205370 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f48022030 con 0x55712c78a4e0 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.615+0000 7f3f3aa00700 1 -- 172.21.15.27:0/2741205370 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f3f4c011040 con 0x55712c78a4e0 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.615+0000 7f3f6663ab80 10 client.27656 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.615+0000 7f3f6663ab80 10 client.27656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.615+0000 7f3f6663ab80 20 client.27656 populate_metadata read hostname 'smithi027' 2024-05-10T11:31:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.616+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55712c808f90 0x55712c80b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:08.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.616+0000 7f3f6663ab80 4 client.27656 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:31:08.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.616+0000 7f3f6663ab80 1 -- 172.21.15.27:0/2741205370 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55712c6de4a0 con 0x55712c808f90 2024-05-10T11:31:08.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.616+0000 7f3f59200700 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55712c808f90 0x55712c80b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:08.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.617+0000 7f3f59200700 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55712c808f90 0x55712c80b3b0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13f0bf14f55f512d server_cookie=f2e27a2019f97886 in_seq=0 out_seq=0 2024-05-10T11:31:08.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.617+0000 7f3f3aa00700 10 client.27656 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:31:08.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.621+0000 7f3f3aa00700 1 -- 172.21.15.27:0/2741205370 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55712c6de4a0 con 0x55712c808f90 2024-05-10T11:31:08.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.621+0000 7f3f3aa00700 10 client.27656 handle_command_reply: tid=0 2024-05-10T11:31:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.621+0000 7f3f6663ab80 1 client.27656 shutdown 2024-05-10T11:31:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.622+0000 7f3f6663ab80 1 -- 172.21.15.27:0/2741205370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 msgr2=0x55712c7a5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.622+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c7a5d00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c009350 tx=0x7f3f4c009020 comp rx=0 tx=0).stop 2024-05-10T11:31:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.622+0000 7f3f6663ab80 1 -- 172.21.15.27:0/2741205370 shutdown_connections 2024-05-10T11:31:08.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.622+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55712c808f90 0x55712c80b3b0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.622+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55712c7848f0 0x55712c7a5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.622+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55712c78a4e0 0x55712c7a5d00 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.622+0000 7f3f6663ab80 1 --2- 172.21.15.27:0/2741205370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712c789bc0 0x55712c7a57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.622+0000 7f3f6663ab80 1 -- 172.21.15.27:0/2741205370 >> 172.21.15.27:0/2741205370 conn(0x55712c6fc6d0 msgr2=0x55712c788f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:08.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.622+0000 7f3f3aa00700 0 client.27656 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:31:08.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.623+0000 7f3f6663ab80 1 -- 172.21.15.27:0/2741205370 shutdown_connections 2024-05-10T11:31:08.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.623+0000 7f3f6663ab80 1 -- 172.21.15.27:0/2741205370 wait complete. 2024-05-10T11:31:08.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.623+0000 7f3f6663ab80 20 client.27656 trim_cache size 0 max 16384 2024-05-10T11:31:08.625 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:31:08.625 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-10T11:31:08.626 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-05-10T11:31:08.626 INFO:teuthology.orchestra.run.smithi027.stdout: "ed3c6d44-d154-495a-888b-c0408db30ced": { 2024-05-10T11:31:08.626 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-05-10T11:31:08.626 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "ed3c6d44-d154-495a-888b-c0408db30ced", 2024-05-10T11:31:08.626 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-05-10T11:31:08.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:31:08.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:31:08.626 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:31:08.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.624+0000 7f3f5bc00700 1 -- 172.21.15.27:0/1444763175 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3f3c0b2af0 msgr2=0x7f3f3c0b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:08.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.624+0000 7f3f5bc00700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3f3c0b2af0 0x7f3f3c0b4fb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00b780 tx=0x7f3f4c006000 comp rx=0 tx=0).stop 2024-05-10T11:31:08.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.624+0000 7f3f5bc00700 1 -- 172.21.15.27:0/1444763175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f541060a0 msgr2=0x7f3f5410f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:08.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.624+0000 7f3f5bc00700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f541060a0 0x7f3f5410f4f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000e9c0 tx=0x7f3f4000ecd0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.625+0000 7f3f5bc00700 1 -- 172.21.15.27:0/1444763175 shutdown_connections 2024-05-10T11:31:08.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.625+0000 7f3f5bc00700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3f3c0b2af0 0x7f3f3c0b4fb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.625+0000 7f3f5bc00700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5410b6e0 0x7f3f5410fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.625+0000 7f3f5bc00700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f541056e0 0x7f3f541123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.625+0000 7f3f5bc00700 1 --2- 172.21.15.27:0/1444763175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f541060a0 0x7f3f5410f4f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:08.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.625+0000 7f3f5bc00700 1 -- 172.21.15.27:0/1444763175 >> 172.21.15.27:0/1444763175 conn(0x7f3f541011d0 msgr2=0x7f3f541016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:08.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.625+0000 7f3f5bc00700 1 -- 172.21.15.27:0/1444763175 shutdown_connections 2024-05-10T11:31:08.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:08.625+0000 7f3f5bc00700 1 -- 172.21.15.27:0/1444763175 wait complete. 2024-05-10T11:31:09.177 INFO:tasks.workunit.client.0.smithi027.stdout: 10 99100 22.02 MB/sec execute 113 sec latency 210.371 ms 2024-05-10T11:31:09.636 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:09 smithi027 ceph-mon[30054]: pgmap v563: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 7.9 MiB/s wr, 590 op/s 2024-05-10T11:31:09.636 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:09 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:31:09.636 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:09 smithi027 ceph-mon[30054]: osdmap e115: 12 total, 12 up, 12 in 2024-05-10T11:31:09.638 INFO:tasks.cephfs.filesystem:scrub status for tag:ed3c6d44-d154-495a-888b-c0408db30ced - {'path': '/', 'tag': 'ed3c6d44-d154-495a-888b-c0408db30ced', 'options': 'recursive,force'} 2024-05-10T11:31:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:09 smithi076 ceph-mon[34152]: pgmap v563: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 7.9 MiB/s wr, 590 op/s 2024-05-10T11:31:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:09 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:31:09.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:09 smithi076 ceph-mon[34152]: osdmap e115: 12 total, 12 up, 12 in 2024-05-10T11:31:09.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:09 smithi189 ceph-mon[34842]: pgmap v563: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 7.9 MiB/s wr, 590 op/s 2024-05-10T11:31:09.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:09 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:31:09.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:09 smithi189 ceph-mon[34842]: osdmap e115: 12 total, 12 up, 12 in 2024-05-10T11:31:10.177 INFO:tasks.workunit.client.0.smithi027.stdout: 10 99594 22.12 MB/sec execute 114 sec latency 228.693 ms 2024-05-10T11:31:11.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:10 smithi027 ceph-mon[30054]: pgmap v565: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.8 MiB/s wr, 706 op/s 2024-05-10T11:31:11.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:10 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:31:11.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:10 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:31:11.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:10 smithi076 ceph-mon[34152]: pgmap v565: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.8 MiB/s wr, 706 op/s 2024-05-10T11:31:11.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:10 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:31:11.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:10 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:31:11.177 INFO:tasks.workunit.client.0.smithi027.stdout: 10 100074 22.19 MB/sec execute 115 sec latency 535.715 ms 2024-05-10T11:31:11.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:10 smithi189 ceph-mon[34842]: pgmap v565: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.8 MiB/s wr, 706 op/s 2024-05-10T11:31:11.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:10 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:31:11.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:10 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:31:12.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:11 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:31:12.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:11 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:31:12.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:11 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:31:12.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:11 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:31:12.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:11 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:31:12.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:11 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:31:12.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:11 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:31:12.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:11 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:31:12.177 INFO:tasks.workunit.client.0.smithi027.stdout: 10 100234 22.07 MB/sec execute 116 sec latency 1535.807 ms 2024-05-10T11:31:12.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:11 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:31:12.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:11 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:31:12.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:11 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:31:12.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:11 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:31:13.177 INFO:tasks.workunit.client.0.smithi027.stdout: 10 100525 22.00 MB/sec execute 117 sec latency 2535.866 ms 2024-05-10T11:31:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:13 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:31:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:13 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:31:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:13 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:31:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:13 smithi027 ceph-mon[30054]: scrub summary: idle 2024-05-10T11:31:13.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:13 smithi027 ceph-mon[30054]: pgmap v566: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 8.3 MiB/s wr, 667 op/s 2024-05-10T11:31:13.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:13 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:31:13.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:13 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:31:13.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:13 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:31:13.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:13 smithi076 ceph-mon[34152]: scrub summary: idle 2024-05-10T11:31:13.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:13 smithi076 ceph-mon[34152]: pgmap v566: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 8.3 MiB/s wr, 667 op/s 2024-05-10T11:31:13.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:13 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:31:13.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:13 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:31:13.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:13 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:31:13.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:13 smithi189 ceph-mon[34842]: scrub summary: idle 2024-05-10T11:31:13.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:13 smithi189 ceph-mon[34842]: pgmap v566: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 8.3 MiB/s wr, 667 op/s 2024-05-10T11:31:14.177 INFO:tasks.workunit.client.0.smithi027.stdout: 10 100819 21.91 MB/sec execute 118 sec latency 3535.912 ms 2024-05-10T11:31:15.177 INFO:tasks.workunit.client.0.smithi027.stdout: 10 101294 21.93 MB/sec execute 119 sec latency 4535.977 ms 2024-05-10T11:31:15.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:15 smithi027 ceph-mon[30054]: pgmap v567: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.8 MiB/s wr, 621 op/s 2024-05-10T11:31:15.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:15 smithi076 ceph-mon[34152]: pgmap v567: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.8 MiB/s wr, 621 op/s 2024-05-10T11:31:15.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:15 smithi189 ceph-mon[34842]: pgmap v567: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.8 MiB/s wr, 621 op/s 2024-05-10T11:31:16.178 INFO:tasks.workunit.client.0.smithi027.stdout: 10 101889 21.94 MB/sec execute 120 sec latency 4593.670 ms 2024-05-10T11:31:17.178 INFO:tasks.workunit.client.0.smithi027.stdout: 10 102212 21.95 MB/sec execute 121 sec latency 3778.453 ms 2024-05-10T11:31:17.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:17 smithi027 ceph-mon[30054]: pgmap v568: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 9.6 MiB/s wr, 499 op/s 2024-05-10T11:31:17.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:17 smithi076 ceph-mon[34152]: pgmap v568: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 9.6 MiB/s wr, 499 op/s 2024-05-10T11:31:17.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:17 smithi189 ceph-mon[34842]: pgmap v568: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 9.6 MiB/s wr, 499 op/s 2024-05-10T11:31:18.178 INFO:tasks.workunit.client.0.smithi027.stdout: 10 102520 21.92 MB/sec execute 122 sec latency 4778.530 ms 2024-05-10T11:31:19.178 INFO:tasks.workunit.client.0.smithi027.stdout: 10 102983 22.00 MB/sec execute 123 sec latency 5778.587 ms 2024-05-10T11:31:19.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:19 smithi027 ceph-mon[30054]: pgmap v569: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 9.6 MiB/s wr, 499 op/s 2024-05-10T11:31:19.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:19 smithi076 ceph-mon[34152]: pgmap v569: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 9.6 MiB/s wr, 499 op/s 2024-05-10T11:31:19.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:19 smithi189 ceph-mon[34842]: pgmap v569: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 9.6 MiB/s wr, 499 op/s 2024-05-10T11:31:20.178 INFO:tasks.workunit.client.0.smithi027.stdout: 10 103297 22.00 MB/sec execute 124 sec latency 6778.658 ms 2024-05-10T11:31:21.178 INFO:tasks.workunit.client.0.smithi027.stdout: 10 103505 21.87 MB/sec execute 125 sec latency 7778.798 ms 2024-05-10T11:31:21.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:21 smithi027 ceph-mon[30054]: pgmap v570: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 12 MiB/s wr, 602 op/s 2024-05-10T11:31:21.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:21 smithi076 ceph-mon[34152]: pgmap v570: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 12 MiB/s wr, 602 op/s 2024-05-10T11:31:21.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:21 smithi189 ceph-mon[34842]: pgmap v570: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 12 MiB/s wr, 602 op/s 2024-05-10T11:31:22.178 INFO:tasks.workunit.client.0.smithi027.stdout: 10 103682 21.74 MB/sec execute 126 sec latency 8778.898 ms 2024-05-10T11:31:23.178 INFO:tasks.workunit.client.0.smithi027.stdout: 10 104042 21.72 MB/sec execute 127 sec latency 9779.002 ms 2024-05-10T11:31:23.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:23 smithi027 ceph-mon[30054]: pgmap v571: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 8.9 MiB/s wr, 422 op/s 2024-05-10T11:31:23.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:23 smithi076 ceph-mon[34152]: pgmap v571: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 8.9 MiB/s wr, 422 op/s 2024-05-10T11:31:23.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:23 smithi189 ceph-mon[34842]: pgmap v571: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 8.9 MiB/s wr, 422 op/s 2024-05-10T11:31:24.178 INFO:tasks.workunit.client.0.smithi027.stdout: 10 104524 21.69 MB/sec execute 128 sec latency 10263.589 ms 2024-05-10T11:31:25.179 INFO:tasks.workunit.client.0.smithi027.stdout: 10 105090 21.72 MB/sec execute 129 sec latency 237.049 ms 2024-05-10T11:31:25.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:25 smithi027 ceph-mon[30054]: pgmap v572: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 8.3 MiB/s wr, 374 op/s 2024-05-10T11:31:25.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:25 smithi076 ceph-mon[34152]: pgmap v572: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 8.3 MiB/s wr, 374 op/s 2024-05-10T11:31:25.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:25 smithi189 ceph-mon[34842]: pgmap v572: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 8.3 MiB/s wr, 374 op/s 2024-05-10T11:31:26.178 INFO:tasks.workunit.client.0.smithi027.stdout: 10 105782 21.90 MB/sec execute 130 sec latency 163.294 ms 2024-05-10T11:31:27.179 INFO:tasks.workunit.client.0.smithi027.stdout: 10 106093 21.93 MB/sec execute 131 sec latency 323.343 ms 2024-05-10T11:31:27.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:27 smithi027 ceph-mon[30054]: pgmap v573: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 12 MiB/s wr, 534 op/s 2024-05-10T11:31:27.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:27 smithi076 ceph-mon[34152]: pgmap v573: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 12 MiB/s wr, 534 op/s 2024-05-10T11:31:27.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:27 smithi189 ceph-mon[34842]: pgmap v573: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 12 MiB/s wr, 534 op/s 2024-05-10T11:31:28.181 INFO:tasks.workunit.client.0.smithi027.stdout: 10 106417 21.93 MB/sec execute 132 sec latency 127.769 ms 2024-05-10T11:31:29.179 INFO:tasks.workunit.client.0.smithi027.stdout: 10 106784 21.89 MB/sec execute 133 sec latency 99.234 ms 2024-05-10T11:31:29.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:29 smithi027 ceph-mon[30054]: pgmap v574: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.1 MiB/s wr, 340 op/s 2024-05-10T11:31:29.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:29 smithi076 ceph-mon[34152]: pgmap v574: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.1 MiB/s wr, 340 op/s 2024-05-10T11:31:29.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:29 smithi189 ceph-mon[34842]: pgmap v574: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.1 MiB/s wr, 340 op/s 2024-05-10T11:31:30.179 INFO:tasks.workunit.client.0.smithi027.stdout: 10 107177 21.87 MB/sec execute 134 sec latency 78.540 ms 2024-05-10T11:31:31.179 INFO:tasks.workunit.client.0.smithi027.stdout: 10 107619 21.89 MB/sec execute 135 sec latency 82.770 ms 2024-05-10T11:31:31.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:31 smithi027 ceph-mon[30054]: pgmap v575: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 11 MiB/s wr, 503 op/s 2024-05-10T11:31:31.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:31 smithi076 ceph-mon[34152]: pgmap v575: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 11 MiB/s wr, 503 op/s 2024-05-10T11:31:31.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:31 smithi189 ceph-mon[34842]: pgmap v575: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 11 MiB/s wr, 503 op/s 2024-05-10T11:31:32.179 INFO:tasks.workunit.client.0.smithi027.stdout: 10 107991 21.82 MB/sec execute 136 sec latency 228.516 ms 2024-05-10T11:31:33.179 INFO:tasks.workunit.client.0.smithi027.stdout: 10 108471 21.81 MB/sec execute 137 sec latency 193.717 ms 2024-05-10T11:31:33.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:33 smithi027 ceph-mon[30054]: pgmap v576: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.0 MiB/s wr, 428 op/s 2024-05-10T11:31:33.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:33 smithi076 ceph-mon[34152]: pgmap v576: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.0 MiB/s wr, 428 op/s 2024-05-10T11:31:33.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:33 smithi189 ceph-mon[34842]: pgmap v576: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.0 MiB/s wr, 428 op/s 2024-05-10T11:31:34.179 INFO:tasks.workunit.client.0.smithi027.stdout: 10 109064 21.89 MB/sec execute 138 sec latency 124.919 ms 2024-05-10T11:31:35.180 INFO:tasks.workunit.client.0.smithi027.stdout: 10 109602 22.01 MB/sec execute 139 sec latency 117.234 ms 2024-05-10T11:31:35.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:35 smithi027 ceph-mon[30054]: pgmap v577: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.0 MiB/s wr, 374 op/s 2024-05-10T11:31:35.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:35 smithi076 ceph-mon[34152]: pgmap v577: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.0 MiB/s wr, 374 op/s 2024-05-10T11:31:35.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:35 smithi189 ceph-mon[34842]: pgmap v577: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.0 MiB/s wr, 374 op/s 2024-05-10T11:31:36.180 INFO:tasks.workunit.client.0.smithi027.stdout: 10 110170 22.09 MB/sec execute 140 sec latency 129.953 ms 2024-05-10T11:31:37.180 INFO:tasks.workunit.client.0.smithi027.stdout: 10 110489 22.09 MB/sec execute 141 sec latency 215.055 ms 2024-05-10T11:31:37.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:37 smithi027 ceph-mon[30054]: pgmap v578: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 550 op/s 2024-05-10T11:31:37.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:37 smithi076 ceph-mon[34152]: pgmap v578: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 550 op/s 2024-05-10T11:31:37.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:37 smithi189 ceph-mon[34842]: pgmap v578: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 550 op/s 2024-05-10T11:31:38.180 INFO:tasks.workunit.client.0.smithi027.stdout: 10 110747 21.97 MB/sec execute 142 sec latency 162.011 ms 2024-05-10T11:31:39.180 INFO:tasks.workunit.client.0.smithi027.stdout: 10 111162 21.94 MB/sec execute 143 sec latency 209.710 ms 2024-05-10T11:31:39.639 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-10T11:31:39.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:39 smithi076 ceph-mon[34152]: pgmap v579: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.5 MiB/s wr, 391 op/s 2024-05-10T11:31:39.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:39 smithi027 ceph-mon[30054]: pgmap v579: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.5 MiB/s wr, 391 op/s 2024-05-10T11:31:39.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:39 smithi189 ceph-mon[34842]: pgmap v579: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.5 MiB/s wr, 391 op/s 2024-05-10T11:31:40.180 INFO:tasks.workunit.client.0.smithi027.stdout: 10 111704 22.02 MB/sec execute 144 sec latency 184.377 ms 2024-05-10T11:31:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:31:41.181 INFO:tasks.workunit.client.0.smithi027.stdout: 10 112325 22.08 MB/sec execute 145 sec latency 80.066 ms 2024-05-10T11:31:41.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:41 smithi027 ceph-mon[30054]: pgmap v580: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 9.3 MiB/s wr, 506 op/s 2024-05-10T11:31:41.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:41 smithi076 ceph-mon[34152]: pgmap v580: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 9.3 MiB/s wr, 506 op/s 2024-05-10T11:31:41.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:41 smithi189 ceph-mon[34842]: pgmap v580: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 9.3 MiB/s wr, 506 op/s 2024-05-10T11:31:42.181 INFO:tasks.workunit.client.0.smithi027.stdout: 10 112694 22.08 MB/sec execute 146 sec latency 282.829 ms 2024-05-10T11:31:43.181 INFO:tasks.workunit.client.0.smithi027.stdout: 10 112990 22.01 MB/sec execute 147 sec latency 699.298 ms 2024-05-10T11:31:43.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.283+0000 7f7c8a000700 1 -- 172.21.15.27:0/3044903455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c8406bb40 msgr2=0x7f7c84110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:43.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.283+0000 7f7c8a000700 1 --2- 172.21.15.27:0/3044903455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c8406bb40 0x7f7c84110330 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c00b0a0 tx=0x7f7c7c00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.283+0000 7f7c8a000700 1 -- 172.21.15.27:0/3044903455 shutdown_connections 2024-05-10T11:31:43.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.283+0000 7f7c8a000700 1 --2- 172.21.15.27:0/3044903455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c84109510 0x7f7c841098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.283+0000 7f7c8a000700 1 --2- 172.21.15.27:0/3044903455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c8406bb40 0x7f7c84110330 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.283+0000 7f7c8a000700 1 --2- 172.21.15.27:0/3044903455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c8406b180 0x7f7c8406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.283+0000 7f7c8a000700 1 -- 172.21.15.27:0/3044903455 >> 172.21.15.27:0/3044903455 conn(0x7f7c84079350 msgr2=0x7f7c84079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:43.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.283+0000 7f7c8a000700 1 -- 172.21.15.27:0/3044903455 shutdown_connections 2024-05-10T11:31:43.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.283+0000 7f7c8a000700 1 -- 172.21.15.27:0/3044903455 wait complete. 2024-05-10T11:31:43.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.285+0000 7f7c8a000700 1 Processor -- start 2024-05-10T11:31:43.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.285+0000 7f7c8a000700 1 -- start start 2024-05-10T11:31:43.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.286+0000 7f7c8a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c8406b180 0x7f7c84074fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.286+0000 7f7c8a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c8406bb40 0x7f7c8406cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.286+0000 7f7c8a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c84109510 0x7f7c8406d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.286+0000 7f7c8a000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7c84113670 con 0x7f7c8406bb40 2024-05-10T11:31:43.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.286+0000 7f7c8a000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841134f0 con 0x7f7c8406b180 2024-05-10T11:31:43.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.286+0000 7f7c8a000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841137f0 con 0x7f7c84109510 2024-05-10T11:31:43.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.287+0000 7f7c83e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c84109510 0x7f7c8406d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.287+0000 7f7c83e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c84109510 0x7f7c8406d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:40274/0 (socket says 172.21.15.27:40274) 2024-05-10T11:31:43.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.287+0000 7f7c83e00700 1 -- 172.21.15.27:0/1790926255 learned_addr learned my addr 172.21.15.27:0/1790926255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:43.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.287+0000 7f7c83400700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c8406b180 0x7f7c84074fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.287+0000 7f7c82a00700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c8406bb40 0x7f7c8406cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.287+0000 7f7c83e00700 1 -- 172.21.15.27:0/1790926255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c8406b180 msgr2=0x7f7c84074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:43.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.287+0000 7f7c83e00700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c8406b180 0x7f7c84074fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.287+0000 7f7c83e00700 1 -- 172.21.15.27:0/1790926255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c8406bb40 msgr2=0x7f7c8406cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:43.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.287+0000 7f7c83e00700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c8406bb40 0x7f7c8406cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.287+0000 7f7c83e00700 1 -- 172.21.15.27:0/1790926255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8406ddf0 con 0x7f7c84109510 2024-05-10T11:31:43.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.288+0000 7f7c83e00700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c84109510 0x7f7c8406d4f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c004c20 tx=0x7f7c7c004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:43.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.288+0000 7f7c7be00700 1 -- 172.21.15.27:0/1790926255 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c021070 con 0x7f7c84109510 2024-05-10T11:31:43.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.289+0000 7f7c8a000700 1 -- 172.21.15.27:0/1790926255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c841b0af0 con 0x7f7c84109510 2024-05-10T11:31:43.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.289+0000 7f7c7be00700 1 -- 172.21.15.27:0/1790926255 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c7c007b80 con 0x7f7c84109510 2024-05-10T11:31:43.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.289+0000 7f7c7be00700 1 -- 172.21.15.27:0/1790926255 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c0052a0 con 0x7f7c84109510 2024-05-10T11:31:43.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.289+0000 7f7c8a000700 1 -- 172.21.15.27:0/1790926255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c841b1000 con 0x7f7c84109510 2024-05-10T11:31:43.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.298+0000 7f7c94931b80 1 Processor -- start 2024-05-10T11:31:43.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.300+0000 7f7c7be00700 1 -- 172.21.15.27:0/1790926255 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f7c7c024070 con 0x7f7c84109510 2024-05-10T11:31:43.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.300+0000 7f7c94931b80 1 -- start start 2024-05-10T11:31:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.300+0000 7f7c94931b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9490 0x5558b44a9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.300+0000 7f7c94931b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44a9db0 0x5558b44ae940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.301+0000 7f7c94931b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44aee80 0x5558b44af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.301+0000 7f7c94931b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5558b43fea20 con 0x5558b44a9490 2024-05-10T11:31:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.301+0000 7f7c94931b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5558b439c760 con 0x5558b44a9db0 2024-05-10T11:31:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.301+0000 7f7c94931b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5558b43fa9e0 con 0x5558b44aee80 2024-05-10T11:31:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.301+0000 7f7c83400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44a9db0 0x5558b44ae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.301+0000 7f7c82a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9490 0x5558b44a9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.301+0000 7f7c82a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9490 0x5558b44a9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47932/0 (socket says 172.21.15.27:47932) 2024-05-10T11:31:43.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.301+0000 7f7c82a00700 1 -- 172.21.15.27:0/1207583481 learned_addr learned my addr 172.21.15.27:0/1207583481 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:43.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.302+0000 7f7c82a00700 1 --2- 172.21.15.27:0/1207583481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44aee80 0x5558b44af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.302+0000 7f7c82a00700 1 -- 172.21.15.27:0/1207583481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44aee80 msgr2=0x5558b44af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:43.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.302+0000 7f7c82a00700 1 --2- 172.21.15.27:0/1207583481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44aee80 0x5558b44af2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.302+0000 7f7c82a00700 1 -- 172.21.15.27:0/1207583481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44a9db0 msgr2=0x5558b44ae940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:43.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.302+0000 7f7c82a00700 1 --2- 172.21.15.27:0/1207583481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44a9db0 0x5558b44ae940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.302+0000 7f7c82a00700 1 -- 172.21.15.27:0/1207583481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558b434b8d0 con 0x5558b44a9490 2024-05-10T11:31:43.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.303+0000 7f7c7be00700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7c680b29f0 0x7f7c680b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.303+0000 7f7c7be00700 1 -- 172.21.15.27:0/1790926255 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9591+0+0 (secure 0 0 0) 0x7f7c7c0e3ed0 con 0x7f7c84109510 2024-05-10T11:31:43.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.305+0000 7f7c83400700 1 --2- 172.21.15.27:0/1207583481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44a9db0 0x5558b44ae940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:31:43.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.306+0000 7f7c82a00700 1 --2- 172.21.15.27:0/1207583481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9490 0x5558b44a9870 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7c700155d0 tx=0x7f7c70009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb14d3445c60403 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:43.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.306+0000 7f7c83400700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7c680b29f0 0x7f7c680b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.306+0000 7f7c78c00700 1 -- 172.21.15.27:0/1207583481 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c70015120 con 0x5558b44a9490 2024-05-10T11:31:43.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.306+0000 7f7c78c00700 1 -- 172.21.15.27:0/1207583481 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c700152c0 con 0x5558b44a9490 2024-05-10T11:31:43.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.306+0000 7f7c78c00700 1 -- 172.21.15.27:0/1207583481 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c70015120 con 0x5558b44a9490 2024-05-10T11:31:43.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.307+0000 7f7c94931b80 1 -- 172.21.15.27:0/1207583481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9490 msgr2=0x5558b44a9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:43.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.307+0000 7f7c94931b80 1 --2- 172.21.15.27:0/1207583481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9490 0x5558b44a9870 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7c700155d0 tx=0x7f7c70009900 comp rx=0 tx=0).stop 2024-05-10T11:31:43.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.308+0000 7f7c83400700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7c680b29f0 0x7f7c680b4eb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5558b44a85b0 tx=0x7f7c74009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:43.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.308+0000 7f7c94931b80 1 -- 172.21.15.27:0/1207583481 shutdown_connections 2024-05-10T11:31:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.308+0000 7f7c94931b80 1 --2- 172.21.15.27:0/1207583481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44aee80 0x5558b44af2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.308+0000 7f7c94931b80 1 --2- 172.21.15.27:0/1207583481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9490 0x5558b44a9870 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.308+0000 7f7c94931b80 1 --2- 172.21.15.27:0/1207583481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44a9db0 0x5558b44ae940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.308+0000 7f7c94931b80 1 -- 172.21.15.27:0/1207583481 >> 172.21.15.27:0/1207583481 conn(0x5558b44246d0 msgr2=0x5558b44a6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.308+0000 7f7c94931b80 1 -- 172.21.15.27:0/1207583481 shutdown_connections 2024-05-10T11:31:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.308+0000 7f7c94931b80 1 -- 172.21.15.27:0/1207583481 wait complete. 2024-05-10T11:31:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.309+0000 7f7c94931b80 1 -- start start 2024-05-10T11:31:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.310+0000 7f7c94931b80 1 Processor -- start 2024-05-10T11:31:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.311+0000 7f7c94931b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44a9490 0x5558b44ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.311+0000 7f7c94931b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9db0 0x5558b44ab990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.312+0000 7f7c94931b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44aee80 0x5558b44abed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.312+0000 7f7c94931b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5558b44c9f40 con 0x5558b44aee80 2024-05-10T11:31:43.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.312+0000 7f7c94931b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x5558b4364b20 con 0x5558b44a9490 2024-05-10T11:31:43.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.312+0000 7f7c94931b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5558b44ca0c0 con 0x5558b44a9db0 2024-05-10T11:31:43.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.312+0000 7f7c94931b80 10 client.0 _ll_register_callbacks cb 0x5558b43a5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:31:43.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.312+0000 7f7c82a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9db0 0x5558b44ab990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.312+0000 7f7c82a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9db0 0x5558b44ab990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:40282/0 (socket says 172.21.15.27:40282) 2024-05-10T11:31:43.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.312+0000 7f7c82a00700 1 -- 172.21.15.27:0/3039839557 learned_addr learned my addr 172.21.15.27:0/3039839557 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:43.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.312+0000 7f7c83400700 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44a9490 0x5558b44ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.313+0000 7f7c83e00700 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44aee80 0x5558b44abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.314+0000 7f7c82a00700 1 -- 172.21.15.27:0/3039839557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44a9490 msgr2=0x5558b44ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:43.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.314+0000 7f7c82a00700 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44a9490 0x5558b44ae5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.314+0000 7f7c82a00700 1 -- 172.21.15.27:0/3039839557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44aee80 msgr2=0x5558b44abed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:43.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.314+0000 7f7c82a00700 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44aee80 0x5558b44abed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.314+0000 7f7c82a00700 1 -- 172.21.15.27:0/3039839557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558b43c4420 con 0x5558b44a9db0 2024-05-10T11:31:43.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.314+0000 7f7c83e00700 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44aee80 0x5558b44abed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:31:43.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.314+0000 7f7c83400700 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44a9490 0x5558b44ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:31:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.314+0000 7f7c82a00700 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9db0 0x5558b44ab990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7c70015f90 tx=0x7f7c70009600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f80a6d5d091141 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.315+0000 7f7c6d600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:31:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.315+0000 7f7c6d600700 1 -- 172.21.15.27:0/3039839557 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c70025020 con 0x5558b44a9db0 2024-05-10T11:31:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.315+0000 7f7c94931b80 1 -- 172.21.15.27:0/3039839557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5558b4479140 con 0x5558b44a9db0 2024-05-10T11:31:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.316+0000 7f7c6d600700 1 -- 172.21.15.27:0/3039839557 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9591+0+0 (secure 0 0 0) 0x7f7c70005240 con 0x5558b44a9db0 2024-05-10T11:31:43.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.317+0000 7f7c6d600700 1 -- 172.21.15.27:0/3039839557 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f7c70030050 con 0x5558b44a9db0 2024-05-10T11:31:43.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.317+0000 7f7c94931b80 10 client.27679 fetch_fsmap learned FSMap version 45 2024-05-10T11:31:43.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.317+0000 7f7c94931b80 1 -- 172.21.15.27:0/3039839557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5558b44d4510 con 0x5558b44a9db0 2024-05-10T11:31:43.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.317+0000 7f7c6d600700 1 -- 172.21.15.27:0/3039839557 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5558b4479140 con 0x5558b44a9db0 2024-05-10T11:31:43.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.318+0000 7f7c6d600700 1 -- 172.21.15.27:0/3039839557 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f7c70021450 con 0x5558b44a9db0 2024-05-10T11:31:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.318+0000 7f7c94931b80 10 client.27679 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:31:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.318+0000 7f7c94931b80 10 client.27679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:31:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.318+0000 7f7c94931b80 20 client.27679 populate_metadata read hostname 'smithi027' 2024-05-10T11:31:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.319+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5558b44d6aa0 0x5558b44d8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.319+0000 7f7c94931b80 4 client.27679 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:31:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.319+0000 7f7c94931b80 1 -- 172.21.15.27:0/3039839557 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558b440c400 con 0x5558b44d6aa0 2024-05-10T11:31:43.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.320+0000 7f7c83400700 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5558b44d6aa0 0x5558b44d8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.322+0000 7f7c83400700 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5558b44d6aa0 0x5558b44d8e90 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b389e25d74c4b9 server_cookie=245f9f2c721870f9 in_seq=0 out_seq=0 2024-05-10T11:31:43.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.324+0000 7f7c6d600700 10 client.27679 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:31:43.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.324+0000 7f7c6d600700 1 -- 172.21.15.27:0/3039839557 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5558b440c400 con 0x5558b44d6aa0 2024-05-10T11:31:43.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.324+0000 7f7c6d600700 10 client.27679 handle_command_reply: tid=0 2024-05-10T11:31:43.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.324+0000 7f7c94931b80 1 client.27679 shutdown 2024-05-10T11:31:43.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.324+0000 7f7c94931b80 1 -- 172.21.15.27:0/3039839557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9db0 msgr2=0x5558b44ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:43.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.324+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9db0 0x5558b44ab990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7c70015f90 tx=0x7f7c70009600 comp rx=0 tx=0).stop 2024-05-10T11:31:43.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.325+0000 7f7c94931b80 1 -- 172.21.15.27:0/3039839557 shutdown_connections 2024-05-10T11:31:43.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.325+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5558b44d6aa0 0x5558b44d8e90 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.325+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44aee80 0x5558b44abed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.325+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44a9490 0x5558b44ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.325+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3039839557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9db0 0x5558b44ab990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.325+0000 7f7c6d600700 0 client.27679 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:31:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.325+0000 7f7c94931b80 1 -- 172.21.15.27:0/3039839557 >> 172.21.15.27:0/3039839557 conn(0x5558b44246d0 msgr2=0x5558b44a6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.325+0000 7f7c94931b80 1 -- 172.21.15.27:0/3039839557 shutdown_connections 2024-05-10T11:31:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.326+0000 7f7c94931b80 1 -- 172.21.15.27:0/3039839557 wait complete. 2024-05-10T11:31:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.326+0000 7f7c94931b80 20 client.27679 trim_cache size 0 max 16384 2024-05-10T11:31:43.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.371+0000 7f7c94931b80 1 Processor -- start 2024-05-10T11:31:43.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.372+0000 7f7c94931b80 1 -- start start 2024-05-10T11:31:43.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.372+0000 7f7c94931b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9490 0x5558b44c6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.372+0000 7f7c94931b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9db0 0x5558b44a6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.372+0000 7f7c94931b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44aee80 0x5558b44de430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.372+0000 7f7c94931b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841b5bb0 con 0x5558b44a9db0 2024-05-10T11:31:43.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.372+0000 7f7c94931b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841b76f0 con 0x5558b44aee80 2024-05-10T11:31:43.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.372+0000 7f7c94931b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841b6280 con 0x5558b44a9490 2024-05-10T11:31:43.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.373+0000 7f7c83400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9490 0x5558b44c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.373+0000 7f7c83400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9490 0x5558b44c6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:40288/0 (socket says 172.21.15.27:40288) 2024-05-10T11:31:43.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.373+0000 7f7c83400700 1 -- 172.21.15.27:0/3385579983 learned_addr learned my addr 172.21.15.27:0/3385579983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:43.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.373+0000 7f7c83e00700 1 --2- 172.21.15.27:0/3385579983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44aee80 0x5558b44de430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.373+0000 7f7c82a00700 1 --2- 172.21.15.27:0/3385579983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9db0 0x5558b44a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.373+0000 7f7c83400700 1 -- 172.21.15.27:0/3385579983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44aee80 msgr2=0x5558b44de430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:43.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.373+0000 7f7c83400700 1 --2- 172.21.15.27:0/3385579983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44aee80 0x5558b44de430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.373+0000 7f7c83400700 1 -- 172.21.15.27:0/3385579983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9db0 msgr2=0x5558b44a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:43.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.373+0000 7f7c83400700 1 --2- 172.21.15.27:0/3385579983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9db0 0x5558b44a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.373+0000 7f7c83400700 1 -- 172.21.15.27:0/3385579983 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558b44a73f0 con 0x5558b44a9490 2024-05-10T11:31:43.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.374+0000 7f7c83400700 1 --2- 172.21.15.27:0/3385579983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9490 0x5558b44c6200 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7c740099a0 tx=0x7f7c74013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=395cf76ea77e113a server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:43.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.375+0000 7f7c6fe00700 1 -- 172.21.15.27:0/3385579983 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c74011c80 con 0x5558b44a9490 2024-05-10T11:31:43.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.375+0000 7f7c6fe00700 1 -- 172.21.15.27:0/3385579983 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c74011e20 con 0x5558b44a9490 2024-05-10T11:31:43.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.376+0000 7f7c6fe00700 1 -- 172.21.15.27:0/3385579983 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c740272d0 con 0x5558b44a9490 2024-05-10T11:31:43.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.379+0000 7f7c94931b80 1 -- 172.21.15.27:0/3385579983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9490 msgr2=0x5558b44c6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:43.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.379+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3385579983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9490 0x5558b44c6200 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7c740099a0 tx=0x7f7c74013000 comp rx=0 tx=0).stop 2024-05-10T11:31:43.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.379+0000 7f7c94931b80 1 -- 172.21.15.27:0/3385579983 shutdown_connections 2024-05-10T11:31:43.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.379+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3385579983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44a9490 0x5558b44c6200 secure :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7c740099a0 tx=0x7f7c74013000 comp rx=0 tx=0).stop 2024-05-10T11:31:43.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.379+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3385579983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44aee80 0x5558b44de430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.379+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3385579983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9db0 0x5558b44a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.379+0000 7f7c94931b80 1 -- 172.21.15.27:0/3385579983 >> 172.21.15.27:0/3385579983 conn(0x5558b44246d0 msgr2=0x5558b44c5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:43.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.381+0000 7f7c94931b80 1 -- 172.21.15.27:0/3385579983 shutdown_connections 2024-05-10T11:31:43.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.381+0000 7f7c94931b80 1 -- 172.21.15.27:0/3385579983 wait complete. 2024-05-10T11:31:43.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.381+0000 7f7c94931b80 1 -- start start 2024-05-10T11:31:43.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.382+0000 7f7c94931b80 1 Processor -- start 2024-05-10T11:31:43.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.383+0000 7f7c94931b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9db0 0x5558b44cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.384+0000 7f7c94931b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44aee80 0x5558b44c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.384+0000 7f7c94931b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44c7b40 0x5558b44c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.384+0000 7f7c83e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44c7b40 0x5558b44c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.385+0000 7f7c83e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44c7b40 0x5558b44c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:40302/0 (socket says 172.21.15.27:40302) 2024-05-10T11:31:43.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.385+0000 7f7c83e00700 1 -- 172.21.15.27:0/3350314098 learned_addr learned my addr 172.21.15.27:0/3350314098 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:43.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.385+0000 7f7c94931b80 1 -- 172.21.15.27:0/3350314098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841b7020 con 0x5558b44a9db0 2024-05-10T11:31:43.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.385+0000 7f7c94931b80 1 -- 172.21.15.27:0/3350314098 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841b6950 con 0x5558b44aee80 2024-05-10T11:31:43.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.385+0000 7f7c94931b80 1 -- 172.21.15.27:0/3350314098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841b3b70 con 0x5558b44c7b40 2024-05-10T11:31:43.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.385+0000 7f7c94931b80 10 client.0 _ll_register_callbacks cb 0x5558b43a5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:31:43.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.385+0000 7f7c83e00700 1 -- 172.21.15.27:0/3350314098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44aee80 msgr2=0x5558b44c7600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:31:43.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.385+0000 7f7c83e00700 1 --2- 172.21.15.27:0/3350314098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44aee80 0x5558b44c7600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.385+0000 7f7c83e00700 1 -- 172.21.15.27:0/3350314098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9db0 msgr2=0x5558b44cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:43.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.385+0000 7f7c83e00700 1 --2- 172.21.15.27:0/3350314098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9db0 0x5558b44cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.385+0000 7f7c83e00700 1 -- 172.21.15.27:0/3350314098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558b4540060 con 0x5558b44c7b40 2024-05-10T11:31:43.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.388+0000 7f7c83e00700 1 --2- 172.21.15.27:0/3350314098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44c7b40 0x5558b44c7fc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c004fd0 tx=0x7f7c7c0a6dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b7d0626ec4c274f server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:43.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.388+0000 7f7c6d600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:31:43.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.390+0000 7f7c6d600700 1 -- 172.21.15.27:0/3350314098 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c0c1020 con 0x5558b44c7b40 2024-05-10T11:31:43.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.390+0000 7f7c6d600700 1 -- 172.21.15.27:0/3350314098 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9591+0+0 (secure 0 0 0) 0x7f7c7c0ab210 con 0x5558b44c7b40 2024-05-10T11:31:43.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.390+0000 7f7c94931b80 1 -- 172.21.15.27:0/3350314098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c74011e20 con 0x5558b44c7b40 2024-05-10T11:31:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.396+0000 7f7c6d600700 1 -- 172.21.15.27:0/3350314098 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f7c7c02b070 con 0x5558b44c7b40 2024-05-10T11:31:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.397+0000 7f7c94931b80 10 client.27685 fetch_fsmap learned FSMap version 45 2024-05-10T11:31:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.397+0000 7f7c94931b80 1 -- 172.21.15.27:0/3350314098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5558b4541390 con 0x5558b44c7b40 2024-05-10T11:31:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.398+0000 7f7c6d600700 1 -- 172.21.15.27:0/3350314098 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c74011e20 con 0x5558b44c7b40 2024-05-10T11:31:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.398+0000 7f7c6d600700 1 -- 172.21.15.27:0/3350314098 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f7c7c0b1040 con 0x5558b44c7b40 2024-05-10T11:31:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.398+0000 7f7c94931b80 10 client.27685 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:31:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.398+0000 7f7c94931b80 10 client.27685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:31:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.398+0000 7f7c94931b80 20 client.27685 populate_metadata read hostname 'smithi027' 2024-05-10T11:31:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.398+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3350314098 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5558b4541b10 0x5558b4543f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.398+0000 7f7c94931b80 4 client.27685 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-10T11:31:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.398+0000 7f7c94931b80 1 -- 172.21.15.27:0/3350314098 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5558b44064a0 con 0x5558b4541b10 2024-05-10T11:31:43.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.400+0000 7f7c83400700 1 --2- 172.21.15.27:0/3350314098 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5558b4541b10 0x5558b4543f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:43.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.405+0000 7f7c83400700 1 --2- 172.21.15.27:0/3350314098 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5558b4541b10 0x5558b4543f70 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e27507dd58de174e server_cookie=4902e8c517f5696f in_seq=0 out_seq=0 2024-05-10T11:31:43.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.405+0000 7f7c6d600700 10 client.27685 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:31:43.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.413+0000 7f7c6d600700 1 -- 172.21.15.27:0/3350314098 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5558b44064a0 con 0x5558b4541b10 2024-05-10T11:31:43.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.413+0000 7f7c6d600700 10 client.27685 handle_command_reply: tid=0 2024-05-10T11:31:43.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.413+0000 7f7c94931b80 1 client.27685 shutdown 2024-05-10T11:31:43.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.413+0000 7f7c94931b80 1 -- 172.21.15.27:0/3350314098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44c7b40 msgr2=0x5558b44c7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:43.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.413+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3350314098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44c7b40 0x5558b44c7fc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c004fd0 tx=0x7f7c7c0a6dc0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.413+0000 7f7c94931b80 1 -- 172.21.15.27:0/3350314098 shutdown_connections 2024-05-10T11:31:43.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.413+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3350314098 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x5558b4541b10 0x5558b4543f70 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.413+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3350314098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558b44c7b40 0x5558b44c7fc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.413+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3350314098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558b44a9db0 0x5558b44cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.413+0000 7f7c94931b80 1 --2- 172.21.15.27:0/3350314098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558b44aee80 0x5558b44c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.413+0000 7f7c94931b80 1 -- 172.21.15.27:0/3350314098 >> 172.21.15.27:0/3350314098 conn(0x5558b44246d0 msgr2=0x5558b44acf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:43.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.414+0000 7f7c6d600700 0 client.27685 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:31:43.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.414+0000 7f7c94931b80 1 -- 172.21.15.27:0/3350314098 shutdown_connections 2024-05-10T11:31:43.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.414+0000 7f7c94931b80 1 -- 172.21.15.27:0/3350314098 wait complete. 2024-05-10T11:31:43.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.415+0000 7f7c94931b80 20 client.27685 trim_cache size 0 max 16384 2024-05-10T11:31:43.417 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:31:43.417 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-05-10T11:31:43.417 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-05-10T11:31:43.417 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:31:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.417+0000 7f7c8a000700 1 -- 172.21.15.27:0/1790926255 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7c680b29f0 msgr2=0x7f7c680b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.417+0000 7f7c8a000700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7c680b29f0 0x7f7c680b4eb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5558b44a85b0 tx=0x7f7c74009260 comp rx=0 tx=0).stop 2024-05-10T11:31:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.417+0000 7f7c8a000700 1 -- 172.21.15.27:0/1790926255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c84109510 msgr2=0x7f7c8406d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.417+0000 7f7c8a000700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c84109510 0x7f7c8406d4f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c004c20 tx=0x7f7c7c004d00 comp rx=0 tx=0).stop 2024-05-10T11:31:43.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.418+0000 7f7c8a000700 1 -- 172.21.15.27:0/1790926255 shutdown_connections 2024-05-10T11:31:43.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.418+0000 7f7c8a000700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f7c680b29f0 0x7f7c680b4eb0 secure :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5558b44a85b0 tx=0x7f7c74009260 comp rx=0 tx=0).stop 2024-05-10T11:31:43.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.418+0000 7f7c8a000700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c84109510 0x7f7c8406d4f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.418+0000 7f7c8a000700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c8406b180 0x7f7c84074fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.418+0000 7f7c8a000700 1 --2- 172.21.15.27:0/1790926255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c8406bb40 0x7f7c8406cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:43.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.418+0000 7f7c8a000700 1 -- 172.21.15.27:0/1790926255 >> 172.21.15.27:0/1790926255 conn(0x7f7c84079350 msgr2=0x7f7c8410e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:43.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.418+0000 7f7c8a000700 1 -- 172.21.15.27:0/1790926255 shutdown_connections 2024-05-10T11:31:43.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:43.418+0000 7f7c8a000700 1 -- 172.21.15.27:0/1790926255 wait complete. 2024-05-10T11:31:43.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:43 smithi027 ceph-mon[30054]: pgmap v581: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 6.6 MiB/s wr, 402 op/s 2024-05-10T11:31:43.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:43 smithi076 ceph-mon[34152]: pgmap v581: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 6.6 MiB/s wr, 402 op/s 2024-05-10T11:31:43.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:43 smithi189 ceph-mon[34842]: pgmap v581: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 6.6 MiB/s wr, 402 op/s 2024-05-10T11:31:44.181 INFO:tasks.workunit.client.0.smithi027.stdout: 10 113452 22.07 MB/sec execute 148 sec latency 133.763 ms 2024-05-10T11:31:44.351 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-10T11:31:44.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:31:44.975 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:31:45.183 INFO:tasks.workunit.client.0.smithi027.stdout: 10 113813 22.07 MB/sec execute 149 sec latency 106.803 ms 2024-05-10T11:31:45.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:45 smithi027 ceph-mon[30054]: pgmap v582: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.6 MiB/s wr, 350 op/s 2024-05-10T11:31:45.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:45 smithi076 ceph-mon[34152]: pgmap v582: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.6 MiB/s wr, 350 op/s 2024-05-10T11:31:45.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:45 smithi189 ceph-mon[34842]: pgmap v582: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.6 MiB/s wr, 350 op/s 2024-05-10T11:31:46.183 INFO:tasks.workunit.client.0.smithi027.stdout: 10 114336 22.12 MB/sec execute 150 sec latency 65.083 ms 2024-05-10T11:31:47.183 INFO:tasks.workunit.client.0.smithi027.stdout: 10 114648 22.08 MB/sec execute 151 sec latency 214.602 ms 2024-05-10T11:31:47.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:47 smithi027 ceph-mon[30054]: pgmap v583: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 10 MiB/s wr, 548 op/s 2024-05-10T11:31:47.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:47 smithi076 ceph-mon[34152]: pgmap v583: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 10 MiB/s wr, 548 op/s 2024-05-10T11:31:47.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:47 smithi189 ceph-mon[34842]: pgmap v583: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 10 MiB/s wr, 548 op/s 2024-05-10T11:31:48.183 INFO:tasks.workunit.client.0.smithi027.stdout: 10 114802 21.98 MB/sec execute 152 sec latency 614.320 ms 2024-05-10T11:31:48.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.291+0000 7f429ee00700 1 -- 172.21.15.27:0/2117755229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4298075ff0 msgr2=0x7f429810f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:48.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.291+0000 7f429ee00700 1 --2- 172.21.15.27:0/2117755229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4298075ff0 0x7f429810f9c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4284009990 tx=0x7f4284009ca0 comp rx=0 tx=0).stop 2024-05-10T11:31:48.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.291+0000 7f429ee00700 1 -- 172.21.15.27:0/2117755229 shutdown_connections 2024-05-10T11:31:48.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.291+0000 7f429ee00700 1 --2- 172.21.15.27:0/2117755229 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f429810ff00 0x7f42981122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:48.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.291+0000 7f429ee00700 1 --2- 172.21.15.27:0/2117755229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42980756d0 0x7f4298075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:48.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.291+0000 7f429ee00700 1 --2- 172.21.15.27:0/2117755229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4298075ff0 0x7f429810f9c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:48.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.291+0000 7f429ee00700 1 -- 172.21.15.27:0/2117755229 >> 172.21.15.27:0/2117755229 conn(0x7f429807a600 msgr2=0x7f429807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:48.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.292+0000 7f429ee00700 1 -- 172.21.15.27:0/2117755229 shutdown_connections 2024-05-10T11:31:48.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.292+0000 7f429ee00700 1 -- 172.21.15.27:0/2117755229 wait complete. 2024-05-10T11:31:48.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.294+0000 7f429ee00700 1 Processor -- start 2024-05-10T11:31:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.295+0000 7f429ee00700 1 -- start start 2024-05-10T11:31:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.296+0000 7f429ee00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42980756d0 0x7f429810f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.296+0000 7f429ee00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4298075ff0 0x7f429810c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.296+0000 7f429ee00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f429810ff00 0x7f429810cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.296+0000 7f429ee00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4298115940 con 0x7f4298075ff0 2024-05-10T11:31:48.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.296+0000 7f429ee00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f42981157c0 con 0x7f42980756d0 2024-05-10T11:31:48.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.296+0000 7f429ee00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4298115ac0 con 0x7f429810ff00 2024-05-10T11:31:48.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.297+0000 7f429ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f429810ff00 0x7f429810cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:48.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.297+0000 7f429ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f429810ff00 0x7f429810cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:38756/0 (socket says 172.21.15.27:38756) 2024-05-10T11:31:48.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.297+0000 7f429ce00700 1 -- 172.21.15.27:0/1824339167 learned_addr learned my addr 172.21.15.27:0/1824339167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:48.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.297+0000 7f4297e00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42980756d0 0x7f429810f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:48.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.297+0000 7f4297400700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4298075ff0 0x7f429810c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.298+0000 7f429ce00700 1 -- 172.21.15.27:0/1824339167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42980756d0 msgr2=0x7f429810f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.298+0000 7f429ce00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42980756d0 0x7f429810f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.298+0000 7f429ce00700 1 -- 172.21.15.27:0/1824339167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4298075ff0 msgr2=0x7f429810c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.300+0000 7f429ce00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4298075ff0 0x7f429810c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.300+0000 7f429ce00700 1 -- 172.21.15.27:0/1824339167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429810d660 con 0x7f429810ff00 2024-05-10T11:31:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.300+0000 7f4297400700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4298075ff0 0x7f429810c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:31:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.301+0000 7f429ce00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f429810ff00 0x7f429810cd60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f429000efc0 tx=0x7f429000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.302+0000 7f4294c00700 1 -- 172.21.15.27:0/1824339167 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42900092e0 con 0x7f429810ff00 2024-05-10T11:31:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.302+0000 7f4294c00700 1 -- 172.21.15.27:0/1824339167 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4290004bb0 con 0x7f429810ff00 2024-05-10T11:31:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.302+0000 7f4294c00700 1 -- 172.21.15.27:0/1824339167 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4290004280 con 0x7f429810ff00 2024-05-10T11:31:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.303+0000 7f429ee00700 1 -- 172.21.15.27:0/1824339167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42981ac6e0 con 0x7f429810ff00 2024-05-10T11:31:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.303+0000 7f429ee00700 1 -- 172.21.15.27:0/1824339167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42981acbd0 con 0x7f429810ff00 2024-05-10T11:31:48.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.305+0000 7f429ee00700 1 -- 172.21.15.27:0/1824339167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4298076d90 con 0x7f429810ff00 2024-05-10T11:31:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.309+0000 7f4294c00700 1 -- 172.21.15.27:0/1824339167 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f429001b030 con 0x7f429810ff00 2024-05-10T11:31:48.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.312+0000 7f4294c00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f427c0b2b80 0x7f427c0b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:48.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.312+0000 7f4294c00700 1 -- 172.21.15.27:0/1824339167 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9591+0+0 (secure 0 0 0) 0x7f42900e22f0 con 0x7f429810ff00 2024-05-10T11:31:48.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.313+0000 7f4297e00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f427c0b2b80 0x7f427c0b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.316+0000 7f4297e00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f427c0b2b80 0x7f427c0b5040 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f428800ba60 tx=0x7f4288005d50 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.319+0000 7f4294c00700 1 -- 172.21.15.27:0/1824339167 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4290022020 con 0x7f429810ff00 2024-05-10T11:31:48.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.780+0000 7f429ee00700 1 -- 172.21.15.27:0/1824339167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f429802e510 con 0x7f429810ff00 2024-05-10T11:31:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.781+0000 7f4294c00700 1 -- 172.21.15.27:0/1824339167 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11936 (secure 0 0 0) 0x7f4290010040 con 0x7f429810ff00 2024-05-10T11:31:48.782 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:31:48.783 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:29:51.629207+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:31:48.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.786+0000 7f429ee00700 1 -- 172.21.15.27:0/1824339167 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f427c0b2b80 msgr2=0x7f427c0b5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:48.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.786+0000 7f429ee00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f427c0b2b80 0x7f427c0b5040 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f428800ba60 tx=0x7f4288005d50 comp rx=0 tx=0).stop 2024-05-10T11:31:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.786+0000 7f429ee00700 1 -- 172.21.15.27:0/1824339167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f429810ff00 msgr2=0x7f429810cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.786+0000 7f429ee00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f429810ff00 0x7f429810cd60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f429000efc0 tx=0x7f429000c3a0 comp rx=0 tx=0).stop 2024-05-10T11:31:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.786+0000 7f429ee00700 1 -- 172.21.15.27:0/1824339167 shutdown_connections 2024-05-10T11:31:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.786+0000 7f429ee00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f427c0b2b80 0x7f427c0b5040 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.786+0000 7f429ee00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f429810ff00 0x7f429810cd60 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.786+0000 7f429ee00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42980756d0 0x7f429810f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:48.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.786+0000 7f429ee00700 1 --2- 172.21.15.27:0/1824339167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4298075ff0 0x7f429810c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:48.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.786+0000 7f429ee00700 1 -- 172.21.15.27:0/1824339167 >> 172.21.15.27:0/1824339167 conn(0x7f429807a600 msgr2=0x7f4298100970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:48.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.787+0000 7f429ee00700 1 -- 172.21.15.27:0/1824339167 shutdown_connections 2024-05-10T11:31:48.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:48.787+0000 7f429ee00700 1 -- 172.21.15.27:0/1824339167 wait complete. 2024-05-10T11:31:48.790 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 45 2024-05-10T11:31:49.183 INFO:tasks.workunit.client.0.smithi027.stdout: 10 115187 21.94 MB/sec execute 153 sec latency 118.395 ms 2024-05-10T11:31:49.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:49 smithi076 ceph-mon[34152]: pgmap v584: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.4 MiB/s wr, 372 op/s 2024-05-10T11:31:49.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:49 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1824339167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:31:49.705 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:49 smithi027 ceph-mon[30054]: pgmap v584: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.4 MiB/s wr, 372 op/s 2024-05-10T11:31:49.705 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:49 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1824339167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:31:49.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:49 smithi189 ceph-mon[34842]: pgmap v584: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.4 MiB/s wr, 372 op/s 2024-05-10T11:31:49.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:49 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1824339167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:31:49.762 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-10T11:31:50.184 INFO:tasks.workunit.client.0.smithi027.stdout: 10 115648 21.95 MB/sec execute 154 sec latency 157.195 ms 2024-05-10T11:31:50.353 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:31:51.184 INFO:tasks.workunit.client.0.smithi027.stdout: 10 116274 22.03 MB/sec execute 155 sec latency 118.170 ms 2024-05-10T11:31:51.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:51 smithi076 ceph-mon[34152]: pgmap v585: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 10 MiB/s wr, 509 op/s 2024-05-10T11:31:51.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:51 smithi189 ceph-mon[34842]: pgmap v585: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 10 MiB/s wr, 509 op/s 2024-05-10T11:31:51.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:51 smithi027 ceph-mon[30054]: pgmap v585: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 10 MiB/s wr, 509 op/s 2024-05-10T11:31:52.186 INFO:tasks.workunit.client.0.smithi027.stdout: 10 116645 22.05 MB/sec execute 156 sec latency 349.620 ms 2024-05-10T11:31:53.184 INFO:tasks.workunit.client.0.smithi027.stdout: 10 117001 22.01 MB/sec execute 157 sec latency 313.937 ms 2024-05-10T11:31:53.675 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:53 smithi076 ceph-mon[34152]: pgmap v586: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 9.0 MiB/s wr, 438 op/s 2024-05-10T11:31:53.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:53 smithi189 ceph-mon[34842]: pgmap v586: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 9.0 MiB/s wr, 438 op/s 2024-05-10T11:31:53.765 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:53 smithi027 ceph-mon[30054]: pgmap v586: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 9.0 MiB/s wr, 438 op/s 2024-05-10T11:31:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.961+0000 7f9699600700 1 -- 172.21.15.27:0/148630005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9694103750 msgr2=0x7f9694103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.961+0000 7f9699600700 1 --2- 172.21.15.27:0/148630005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9694103750 0x7f9694103b30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9684009990 tx=0x7f9684009ca0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.961+0000 7f9699600700 1 -- 172.21.15.27:0/148630005 shutdown_connections 2024-05-10T11:31:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.961+0000 7f9699600700 1 --2- 172.21.15.27:0/148630005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9694103750 0x7f9694103b30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.961+0000 7f9699600700 1 --2- 172.21.15.27:0/148630005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9694108850 0x7f96941103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.961+0000 7f9699600700 1 --2- 172.21.15.27:0/148630005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9694104100 0x7f9694108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.961+0000 7f9699600700 1 -- 172.21.15.27:0/148630005 >> 172.21.15.27:0/148630005 conn(0x7f969407a600 msgr2=0x7f969407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.961+0000 7f9699600700 1 -- 172.21.15.27:0/148630005 shutdown_connections 2024-05-10T11:31:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.962+0000 7f9699600700 1 -- 172.21.15.27:0/148630005 wait complete. 2024-05-10T11:31:53.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.963+0000 7f9699600700 1 Processor -- start 2024-05-10T11:31:53.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.964+0000 7f9699600700 1 -- start start 2024-05-10T11:31:53.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.964+0000 7f9699600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9694103750 0x7f969419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:53.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.964+0000 7f9699600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9694104100 0x7f969419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:53.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.964+0000 7f9699600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9694108850 0x7f9694199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:53.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.965+0000 7f9692a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9694103750 0x7f969419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.965+0000 7f9692000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9694104100 0x7f969419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.965+0000 7f9692a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9694103750 0x7f969419f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50472/0 (socket says 172.21.15.27:50472) 2024-05-10T11:31:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.965+0000 7f9692a00700 1 -- 172.21.15.27:0/1347845922 learned_addr learned my addr 172.21.15.27:0/1347845922 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.965+0000 7f9693400700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9694108850 0x7f9694199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.965+0000 7f9699600700 1 -- 172.21.15.27:0/1347845922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9694113670 con 0x7f9694104100 2024-05-10T11:31:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.965+0000 7f9699600700 1 -- 172.21.15.27:0/1347845922 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f96941134f0 con 0x7f9694108850 2024-05-10T11:31:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.965+0000 7f9699600700 1 -- 172.21.15.27:0/1347845922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f96941137f0 con 0x7f9694103750 2024-05-10T11:31:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.966+0000 7f9692a00700 1 -- 172.21.15.27:0/1347845922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9694108850 msgr2=0x7f9694199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.966+0000 7f9692a00700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9694108850 0x7f9694199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.966+0000 7f9692a00700 1 -- 172.21.15.27:0/1347845922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9694104100 msgr2=0x7f969419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.966+0000 7f9692a00700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9694104100 0x7f969419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.966+0000 7f9692a00700 1 -- 172.21.15.27:0/1347845922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9694199ee0 con 0x7f9694103750 2024-05-10T11:31:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.966+0000 7f9693400700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9694108850 0x7f9694199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:31:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.966+0000 7f9692000700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9694104100 0x7f969419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:31:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.966+0000 7f9692a00700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9694103750 0x7f969419f760 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9684006010 tx=0x7f968400bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.967+0000 7f968b400700 1 -- 172.21.15.27:0/1347845922 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9684027070 con 0x7f9694103750 2024-05-10T11:31:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.967+0000 7f9699600700 1 -- 172.21.15.27:0/1347845922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f969419a170 con 0x7f9694103750 2024-05-10T11:31:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.967+0000 7f9699600700 1 -- 172.21.15.27:0/1347845922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f969419a640 con 0x7f9694103750 2024-05-10T11:31:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.967+0000 7f968b400700 1 -- 172.21.15.27:0/1347845922 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9684005da0 con 0x7f9694103750 2024-05-10T11:31:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.967+0000 7f968b400700 1 -- 172.21.15.27:0/1347845922 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96840173f0 con 0x7f9694103750 2024-05-10T11:31:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.970+0000 7f968b400700 1 -- 172.21.15.27:0/1347845922 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f968402a070 con 0x7f9694103750 2024-05-10T11:31:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.972+0000 7f96a3f48b80 1 Processor -- start 2024-05-10T11:31:53.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.972+0000 7f968b400700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f96780b2a70 0x7f96780b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:53.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.973+0000 7f96a3f48b80 1 -- start start 2024-05-10T11:31:53.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.973+0000 7f9693400700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f96780b2a70 0x7f96780b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:53.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.973+0000 7f968b400700 1 -- 172.21.15.27:0/1347845922 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9591+0+0 (secure 0 0 0) 0x7f96840e36a0 con 0x7f9694103750 2024-05-10T11:31:53.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.973+0000 7f96a3f48b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e7835bc0 0x55a9e7835fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:53.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.973+0000 7f96a3f48b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e78364e0 0x55a9e78302d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.973+0000 7f96a3f48b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e7830d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.973+0000 7f96a3f48b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a9e7782a20 con 0x55a9e7835bc0 2024-05-10T11:31:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.973+0000 7f96a3f48b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a9e7720760 con 0x55a9e78364e0 2024-05-10T11:31:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.973+0000 7f96a3f48b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a9e777e9e0 con 0x55a9e78308f0 2024-05-10T11:31:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.974+0000 7f9692a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e7835bc0 0x55a9e7835fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.974+0000 7f9692000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e78364e0 0x55a9e78302d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.974+0000 7f9692000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e78364e0 0x55a9e78302d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44456/0 (socket says 172.21.15.27:44456) 2024-05-10T11:31:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.974+0000 7f9692000700 1 -- 172.21.15.27:0/1426008004 learned_addr learned my addr 172.21.15.27:0/1426008004 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.974+0000 7f9692000700 1 -- 172.21.15.27:0/1426008004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 msgr2=0x55a9e7830d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:31:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.974+0000 7f9692000700 1 --2- 172.21.15.27:0/1426008004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e7830d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.974+0000 7f9692000700 1 -- 172.21.15.27:0/1426008004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e7835bc0 msgr2=0x55a9e7835fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.974+0000 7f9692000700 1 --2- 172.21.15.27:0/1426008004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e7835bc0 0x55a9e7835fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.974+0000 7f9692000700 1 -- 172.21.15.27:0/1426008004 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e76cf8d0 con 0x55a9e78364e0 2024-05-10T11:31:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.974+0000 7f9692a00700 1 --2- 172.21.15.27:0/1426008004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e7835bc0 0x55a9e7835fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:31:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.974+0000 7f9693400700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f96780b2a70 0x7f96780b4f30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f968c003e30 tx=0x7f968c00b040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.975+0000 7f9692000700 1 --2- 172.21.15.27:0/1426008004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e78364e0 0x55a9e78302d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9680010550 tx=0x7f9680010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1bba590c5fa86f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.975+0000 7f967fe00700 1 -- 172.21.15.27:0/1426008004 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680011640 con 0x55a9e78364e0 2024-05-10T11:31:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.975+0000 7f967fe00700 1 -- 172.21.15.27:0/1426008004 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9680011c90 con 0x55a9e78364e0 2024-05-10T11:31:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.975+0000 7f967fe00700 1 -- 172.21.15.27:0/1426008004 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680019c90 con 0x55a9e78364e0 2024-05-10T11:31:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.976+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1426008004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e78364e0 msgr2=0x55a9e78302d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.976+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/1426008004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e78364e0 0x55a9e78302d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9680010550 tx=0x7f9680010860 comp rx=0 tx=0).stop 2024-05-10T11:31:53.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.976+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1426008004 shutdown_connections 2024-05-10T11:31:53.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.976+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/1426008004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e7830d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.976+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/1426008004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e7835bc0 0x55a9e7835fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.976+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/1426008004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e78364e0 0x55a9e78302d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.976+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1426008004 >> 172.21.15.27:0/1426008004 conn(0x55a9e77a86d0 msgr2=0x55a9e782ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.977+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1426008004 shutdown_connections 2024-05-10T11:31:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.977+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1426008004 wait complete. 2024-05-10T11:31:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.977+0000 7f96a3f48b80 1 -- start start 2024-05-10T11:31:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.978+0000 7f96a3f48b80 1 Processor -- start 2024-05-10T11:31:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.978+0000 7f96a3f48b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e783c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.978+0000 7f96a3f48b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e7835bc0 0x55a9e7839500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.978+0000 7f96a3f48b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e78364e0 0x55a9e7839a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.978+0000 7f96a3f48b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a9e783cf20 con 0x55a9e78364e0 2024-05-10T11:31:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.978+0000 7f96a3f48b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55a9e76e8b20 con 0x55a9e7835bc0 2024-05-10T11:31:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.978+0000 7f96a3f48b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a9e783d0a0 con 0x55a9e78308f0 2024-05-10T11:31:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.978+0000 7f96a3f48b80 10 client.0 _ll_register_callbacks cb 0x55a9e7729c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:31:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.979+0000 7f9692000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e7835bc0 0x55a9e7839500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.979+0000 7f9692a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e783c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.979+0000 7f9692a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e783c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50482/0 (socket says 172.21.15.27:50482) 2024-05-10T11:31:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.979+0000 7f9692a00700 1 -- 172.21.15.27:0/1630075405 learned_addr learned my addr 172.21.15.27:0/1630075405 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.979+0000 7f9693400700 1 --2- 172.21.15.27:0/1630075405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e78364e0 0x55a9e7839a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.980+0000 7f9692000700 1 -- 172.21.15.27:0/1630075405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 msgr2=0x55a9e783c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.980+0000 7f9692000700 1 --2- 172.21.15.27:0/1630075405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e783c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.980+0000 7f9692000700 1 -- 172.21.15.27:0/1630075405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e78364e0 msgr2=0x55a9e7839a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.980+0000 7f9692000700 1 --2- 172.21.15.27:0/1630075405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e78364e0 0x55a9e7839a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.980+0000 7f9692000700 1 -- 172.21.15.27:0/1630075405 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e7748420 con 0x55a9e7835bc0 2024-05-10T11:31:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.980+0000 7f9692000700 1 --2- 172.21.15.27:0/1630075405 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e7835bc0 0x55a9e7839500 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9680000c00 tx=0x7f9680019700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=208c3c082b4216ff server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.981+0000 7f967cc00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:31:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.981+0000 7f967cc00700 1 -- 172.21.15.27:0/1630075405 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968001f040 con 0x55a9e7835bc0 2024-05-10T11:31:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.981+0000 7f967cc00700 1 -- 172.21.15.27:0/1630075405 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9591+0+0 (secure 0 0 0) 0x7f9680019990 con 0x55a9e7835bc0 2024-05-10T11:31:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.981+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1630075405 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9680011c90 con 0x55a9e7835bc0 2024-05-10T11:31:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.981+0000 7f967cc00700 1 -- 172.21.15.27:0/1630075405 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f9680019b90 con 0x55a9e7835bc0 2024-05-10T11:31:53.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.981+0000 7f96a3f48b80 10 client.27680 fetch_fsmap learned FSMap version 45 2024-05-10T11:31:53.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.981+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1630075405 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55a9e78477b0 con 0x55a9e7835bc0 2024-05-10T11:31:53.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.982+0000 7f967cc00700 1 -- 172.21.15.27:0/1630075405 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9680011c90 con 0x55a9e7835bc0 2024-05-10T11:31:53.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.983+0000 7f967cc00700 1 -- 172.21.15.27:0/1630075405 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f9680022070 con 0x55a9e7835bc0 2024-05-10T11:31:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.983+0000 7f96a3f48b80 10 client.27680 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:31:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.983+0000 7f96a3f48b80 10 client.27680 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:31:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.983+0000 7f96a3f48b80 20 client.27680 populate_metadata read hostname 'smithi027' 2024-05-10T11:31:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.983+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/1630075405 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55a9e7849d30 0x55a9e784c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.983+0000 7f96a3f48b80 4 client.27680 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:31:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.983+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1630075405 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9e77fd140 con 0x55a9e7849d30 2024-05-10T11:31:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.984+0000 7f9692a00700 1 --2- 172.21.15.27:0/1630075405 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55a9e7849d30 0x55a9e784c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:53.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.984+0000 7f9692a00700 1 --2- 172.21.15.27:0/1630075405 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55a9e7849d30 0x55a9e784c120 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bf30f61b521a07c1 server_cookie=4cbc26e1f64a38a3 in_seq=0 out_seq=0 2024-05-10T11:31:53.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.984+0000 7f967cc00700 10 client.27680 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:31:53.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.985+0000 7f967cc00700 1 -- 172.21.15.27:0/1630075405 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a9e77fd140 con 0x55a9e7849d30 2024-05-10T11:31:53.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.985+0000 7f967cc00700 10 client.27680 handle_command_reply: tid=0 2024-05-10T11:31:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.985+0000 7f96a3f48b80 1 client.27680 shutdown 2024-05-10T11:31:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.986+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1630075405 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e7835bc0 msgr2=0x55a9e7839500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.986+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/1630075405 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e7835bc0 0x55a9e7839500 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9680000c00 tx=0x7f9680019700 comp rx=0 tx=0).stop 2024-05-10T11:31:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.986+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1630075405 shutdown_connections 2024-05-10T11:31:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.986+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/1630075405 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55a9e7849d30 0x55a9e784c120 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.986+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/1630075405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e783c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.986+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/1630075405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e78364e0 0x55a9e7839a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.986+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/1630075405 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e7835bc0 0x55a9e7839500 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.986+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1630075405 >> 172.21.15.27:0/1630075405 conn(0x55a9e77a86d0 msgr2=0x55a9e782a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.986+0000 7f967cc00700 0 client.27680 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:31:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.987+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1630075405 shutdown_connections 2024-05-10T11:31:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.987+0000 7f96a3f48b80 1 -- 172.21.15.27:0/1630075405 wait complete. 2024-05-10T11:31:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:53.987+0000 7f96a3f48b80 20 client.27680 trim_cache size 0 max 16384 2024-05-10T11:31:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.017+0000 7f96a3f48b80 1 Processor -- start 2024-05-10T11:31:54.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.017+0000 7f96a3f48b80 1 -- start start 2024-05-10T11:31:54.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.018+0000 7f96a3f48b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e77a86d0 0x55a9e78302d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.018+0000 7f96a3f48b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e78371a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.018+0000 7f96a3f48b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e7835bc0 0x55a9e7846040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.018+0000 7f96a3f48b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f967400f020 con 0x55a9e77a86d0 2024-05-10T11:31:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.018+0000 7f96a3f48b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9674010b60 con 0x55a9e7835bc0 2024-05-10T11:31:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.018+0000 7f96a3f48b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f967400f6f0 con 0x55a9e78308f0 2024-05-10T11:31:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.018+0000 7f9692000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e78371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.018+0000 7f9692000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e78371a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50496/0 (socket says 172.21.15.27:50496) 2024-05-10T11:31:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.018+0000 7f9692000700 1 -- 172.21.15.27:0/2359589475 learned_addr learned my addr 172.21.15.27:0/2359589475 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.018+0000 7f9693400700 1 --2- 172.21.15.27:0/2359589475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e7835bc0 0x55a9e7846040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.019+0000 7f9692000700 1 -- 172.21.15.27:0/2359589475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e7835bc0 msgr2=0x55a9e7846040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.019+0000 7f9692000700 1 --2- 172.21.15.27:0/2359589475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e7835bc0 0x55a9e7846040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.019+0000 7f9692000700 1 -- 172.21.15.27:0/2359589475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e77a86d0 msgr2=0x55a9e78302d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:31:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.021+0000 7f9692a00700 1 --2- 172.21.15.27:0/2359589475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e77a86d0 0x55a9e78302d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.021+0000 7f9692000700 1 --2- 172.21.15.27:0/2359589475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e77a86d0 0x55a9e78302d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.021+0000 7f9692000700 1 -- 172.21.15.27:0/2359589475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e7825ab0 con 0x55a9e78308f0 2024-05-10T11:31:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.021+0000 7f9692000700 1 --2- 172.21.15.27:0/2359589475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e78371a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f96800104a0 tx=0x7f9680019540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=896557ad932b7921 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.022+0000 7f967f400700 1 -- 172.21.15.27:0/2359589475 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680011440 con 0x55a9e78308f0 2024-05-10T11:31:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.022+0000 7f967f400700 1 -- 172.21.15.27:0/2359589475 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9680012e50 con 0x55a9e78308f0 2024-05-10T11:31:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.022+0000 7f967f400700 1 -- 172.21.15.27:0/2359589475 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680011bc0 con 0x55a9e78308f0 2024-05-10T11:31:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.023+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2359589475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 msgr2=0x55a9e78371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.023+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/2359589475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e78371a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f96800104a0 tx=0x7f9680019540 comp rx=0 tx=0).stop 2024-05-10T11:31:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.023+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2359589475 shutdown_connections 2024-05-10T11:31:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.023+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/2359589475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e7835bc0 0x55a9e7846040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.023+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/2359589475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e77a86d0 0x55a9e78302d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.023+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/2359589475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e78308f0 0x55a9e78371a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.023+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2359589475 >> 172.21.15.27:0/2359589475 conn(0x55a9e7827750 msgr2=0x55a9e78346e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.024+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2359589475 shutdown_connections 2024-05-10T11:31:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.024+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2359589475 wait complete. 2024-05-10T11:31:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.025+0000 7f96a3f48b80 1 -- start start 2024-05-10T11:31:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.025+0000 7f96a3f48b80 1 Processor -- start 2024-05-10T11:31:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.026+0000 7f96a3f48b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e77a86d0 0x55a9e78afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.026+0000 7f96a3f48b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e78308f0 0x55a9e78423c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.026+0000 7f96a3f48b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e7835bc0 0x55a9e78429d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.027+0000 7f9692a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e77a86d0 0x55a9e78afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.027+0000 7f9692a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e77a86d0 0x55a9e78afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44484/0 (socket says 172.21.15.27:44484) 2024-05-10T11:31:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.027+0000 7f9693400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e7835bc0 0x55a9e78429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.027+0000 7f9693400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e7835bc0 0x55a9e78429d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50512/0 (socket says 172.21.15.27:50512) 2024-05-10T11:31:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.027+0000 7f96a3f48b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9674010490 con 0x55a9e78308f0 2024-05-10T11:31:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.027+0000 7f96a3f48b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f967400fdc0 con 0x55a9e77a86d0 2024-05-10T11:31:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.027+0000 7f9692a00700 1 -- 172.21.15.27:0/2611655897 learned_addr learned my addr 172.21.15.27:0/2611655897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.028+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2611655897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f967400cfe0 con 0x55a9e7835bc0 2024-05-10T11:31:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.028+0000 7f96a3f48b80 10 client.0 _ll_register_callbacks cb 0x55a9e7729c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:31:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.029+0000 7f9692000700 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e78308f0 0x55a9e78423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.029+0000 7f9693400700 1 -- 172.21.15.27:0/2611655897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e77a86d0 msgr2=0x55a9e78afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.029+0000 7f9693400700 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e77a86d0 0x55a9e78afdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.029+0000 7f9693400700 1 -- 172.21.15.27:0/2611655897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e78308f0 msgr2=0x55a9e78423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.029+0000 7f9693400700 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e78308f0 0x55a9e78423c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.029+0000 7f9693400700 1 -- 172.21.15.27:0/2611655897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e7851490 con 0x55a9e7835bc0 2024-05-10T11:31:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.030+0000 7f9693400700 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e7835bc0 0x55a9e78429d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x55a9e78376e0 tx=0x7f968c004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe1a67b676b83509 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.030+0000 7f9692a00700 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e77a86d0 0x55a9e78afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:31:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.030+0000 7f967cc00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:31:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.031+0000 7f967cc00700 1 -- 172.21.15.27:0/2611655897 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c027070 con 0x55a9e7835bc0 2024-05-10T11:31:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.031+0000 7f967cc00700 1 -- 172.21.15.27:0/2611655897 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9591+0+0 (secure 0 0 0) 0x7f968c017db0 con 0x55a9e7835bc0 2024-05-10T11:31:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.031+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2611655897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9680012e50 con 0x55a9e7835bc0 2024-05-10T11:31:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.031+0000 7f9692000700 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e78308f0 0x55a9e78423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:31:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.032+0000 7f967cc00700 1 -- 172.21.15.27:0/2611655897 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f968c035050 con 0x55a9e7835bc0 2024-05-10T11:31:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.032+0000 7f96a3f48b80 10 client.27706 fetch_fsmap learned FSMap version 45 2024-05-10T11:31:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.032+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2611655897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55a9e78bd110 con 0x55a9e7835bc0 2024-05-10T11:31:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.033+0000 7f967cc00700 1 -- 172.21.15.27:0/2611655897 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f968c01b040 con 0x55a9e7835bc0 2024-05-10T11:31:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.033+0000 7f967cc00700 1 -- 172.21.15.27:0/2611655897 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f968c0158b0 con 0x55a9e7835bc0 2024-05-10T11:31:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.033+0000 7f96a3f48b80 10 client.27706 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:31:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.033+0000 7f96a3f48b80 10 client.27706 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-05-10T11:31:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.033+0000 7f96a3f48b80 20 client.27706 populate_metadata read hostname 'smithi027' 2024-05-10T11:31:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.033+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55a9e78364e0 0x55a9e78bf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.033+0000 7f96a3f48b80 4 client.27706 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:31:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.033+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2611655897 --> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f9680012e50 con 0x55a9e78364e0 2024-05-10T11:31:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.033+0000 7f9692a00700 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55a9e78364e0 0x55a9e78bf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.035+0000 7f9692a00700 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55a9e78364e0 0x55a9e78bf830 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6c33d0a14abe68c5 server_cookie=fc95610673d8349a in_seq=0 out_seq=0 2024-05-10T11:31:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.036+0000 7f967cc00700 10 client.27706 ms_handle_connect on v2:172.21.15.76:6834/3618511593 2024-05-10T11:31:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.036+0000 7f967cc00700 1 -- 172.21.15.27:0/2611655897 <== mds.4 v2:172.21.15.76:6834/3618511593 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f9680012e50 con 0x55a9e78364e0 2024-05-10T11:31:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.036+0000 7f967cc00700 10 client.27706 handle_command_reply: tid=0 2024-05-10T11:31:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.036+0000 7f96a3f48b80 1 client.27706 shutdown 2024-05-10T11:31:54.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.037+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2611655897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e7835bc0 msgr2=0x55a9e78429d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:54.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.037+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e7835bc0 0x55a9e78429d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x55a9e78376e0 tx=0x7f968c004ad0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.037+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2611655897 shutdown_connections 2024-05-10T11:31:54.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.037+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.76:6834/3618511593,v1:172.21.15.76:6835/3618511593] conn(0x55a9e78364e0 0x55a9e78bf830 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.037+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9e7835bc0 0x55a9e78429d0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.037+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a9e77a86d0 0x55a9e78afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.037+0000 7f96a3f48b80 1 --2- 172.21.15.27:0/2611655897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a9e78308f0 0x55a9e78423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.037+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2611655897 >> 172.21.15.27:0/2611655897 conn(0x55a9e7827750 msgr2=0x55a9e7839660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.037+0000 7f967cc00700 0 client.27706 ms_handle_reset on v2:172.21.15.76:6834/3618511593 2024-05-10T11:31:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.038+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2611655897 shutdown_connections 2024-05-10T11:31:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.038+0000 7f96a3f48b80 1 -- 172.21.15.27:0/2611655897 wait complete. 2024-05-10T11:31:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.038+0000 7f96a3f48b80 20 client.27706 trim_cache size 0 max 16384 2024-05-10T11:31:54.040 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:31:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.041+0000 7f9699600700 1 -- 172.21.15.27:0/1347845922 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f96780b2a70 msgr2=0x7f96780b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.041+0000 7f9699600700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f96780b2a70 0x7f96780b4f30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f968c003e30 tx=0x7f968c00b040 comp rx=0 tx=0).stop 2024-05-10T11:31:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.041+0000 7f9699600700 1 -- 172.21.15.27:0/1347845922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9694103750 msgr2=0x7f969419f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.041+0000 7f9699600700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9694103750 0x7f969419f760 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9684006010 tx=0x7f968400bfd0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.042+0000 7f9699600700 1 -- 172.21.15.27:0/1347845922 shutdown_connections 2024-05-10T11:31:54.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.042+0000 7f9699600700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f96780b2a70 0x7f96780b4f30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.042+0000 7f9699600700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9694103750 0x7f969419f760 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.042+0000 7f9699600700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9694108850 0x7f9694199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.042+0000 7f9699600700 1 --2- 172.21.15.27:0/1347845922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9694104100 0x7f969419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:54.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.042+0000 7f9699600700 1 -- 172.21.15.27:0/1347845922 >> 172.21.15.27:0/1347845922 conn(0x7f969407a600 msgr2=0x7f9694101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:54.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.042+0000 7f9699600700 1 -- 172.21.15.27:0/1347845922 shutdown_connections 2024-05-10T11:31:54.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:54.043+0000 7f9699600700 1 -- 172.21.15.27:0/1347845922 wait complete. 2024-05-10T11:31:54.184 INFO:tasks.workunit.client.0.smithi027.stdout: 10 117454 22.04 MB/sec execute 158 sec latency 180.665 ms 2024-05-10T11:31:55.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-10T11:31:55.184 INFO:tasks.workunit.client.0.smithi027.stdout: 10 117844 22.03 MB/sec execute 159 sec latency 184.845 ms 2024-05-10T11:31:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:31:55.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:55 smithi189 ceph-mon[34842]: pgmap v587: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 8.2 MiB/s wr, 378 op/s 2024-05-10T11:31:55.790 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:55 smithi027 ceph-mon[30054]: pgmap v587: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 8.2 MiB/s wr, 378 op/s 2024-05-10T11:31:55.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:55 smithi076 ceph-mon[34152]: pgmap v587: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 8.2 MiB/s wr, 378 op/s 2024-05-10T11:31:56.184 INFO:tasks.workunit.client.0.smithi027.stdout: 10 118247 22.04 MB/sec execute 160 sec latency 146.845 ms 2024-05-10T11:31:57.184 INFO:tasks.workunit.client.0.smithi027.stdout: 10 118523 21.99 MB/sec execute 161 sec latency 398.242 ms 2024-05-10T11:31:57.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:57 smithi027 ceph-mon[30054]: pgmap v588: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 11 MiB/s wr, 530 op/s 2024-05-10T11:31:57.726 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:57 smithi076 ceph-mon[34152]: pgmap v588: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 11 MiB/s wr, 530 op/s 2024-05-10T11:31:57.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:57 smithi189 ceph-mon[34842]: pgmap v588: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 11 MiB/s wr, 530 op/s 2024-05-10T11:31:58.184 INFO:tasks.workunit.client.0.smithi027.stdout: 10 118703 21.90 MB/sec execute 162 sec latency 462.144 ms 2024-05-10T11:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.894+0000 7fbe4a800700 1 -- 172.21.15.27:0/1309140906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe44103750 msgr2=0x7fbe44103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.894+0000 7fbe4a800700 1 --2- 172.21.15.27:0/1309140906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe44103750 0x7fbe44103b30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fbe3400fb90 tx=0x7fbe3400fea0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.896+0000 7fbe4a800700 1 -- 172.21.15.27:0/1309140906 shutdown_connections 2024-05-10T11:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.896+0000 7fbe4a800700 1 --2- 172.21.15.27:0/1309140906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe44108850 0x7fbe441103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.896+0000 7fbe4a800700 1 --2- 172.21.15.27:0/1309140906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe44103750 0x7fbe44103b30 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.896+0000 7fbe4a800700 1 --2- 172.21.15.27:0/1309140906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe44104100 0x7fbe44108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.896+0000 7fbe4a800700 1 -- 172.21.15.27:0/1309140906 >> 172.21.15.27:0/1309140906 conn(0x7fbe4407a600 msgr2=0x7fbe4407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.896+0000 7fbe4a800700 1 -- 172.21.15.27:0/1309140906 shutdown_connections 2024-05-10T11:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.896+0000 7fbe4a800700 1 -- 172.21.15.27:0/1309140906 wait complete. 2024-05-10T11:31:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.899+0000 7fbe4a800700 1 Processor -- start 2024-05-10T11:31:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.899+0000 7fbe4a800700 1 -- start start 2024-05-10T11:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.899+0000 7fbe4a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe44103750 0x7fbe44118a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.899+0000 7fbe4a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe44104100 0x7fbe44118f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.900+0000 7fbe4a800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe44108850 0x7fbe44112ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.900+0000 7fbe4a800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbe44111400 con 0x7fbe44103750 2024-05-10T11:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.900+0000 7fbe4a800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbe44111280 con 0x7fbe44104100 2024-05-10T11:31:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.900+0000 7fbe4a800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbe44111580 con 0x7fbe44108850 2024-05-10T11:31:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.900+0000 7fbe42a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe44104100 0x7fbe44118f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.900+0000 7fbe43e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe44108850 0x7fbe44112ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.900+0000 7fbe43e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe44108850 0x7fbe44112ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50522/0 (socket says 172.21.15.27:50522) 2024-05-10T11:31:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.900+0000 7fbe43e00700 1 -- 172.21.15.27:0/1166504045 learned_addr learned my addr 172.21.15.27:0/1166504045 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.900+0000 7fbe42a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe44104100 0x7fbe44118f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44494/0 (socket says 172.21.15.27:44494) 2024-05-10T11:31:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.901+0000 7fbe42a00700 1 -- 172.21.15.27:0/1166504045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe44108850 msgr2=0x7fbe44112ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:58.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.901+0000 7fbe42a00700 1 --2- 172.21.15.27:0/1166504045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe44108850 0x7fbe44112ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.901+0000 7fbe42a00700 1 -- 172.21.15.27:0/1166504045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe44103750 msgr2=0x7fbe44118a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:58.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.901+0000 7fbe42a00700 1 --2- 172.21.15.27:0/1166504045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe44103750 0x7fbe44118a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.901+0000 7fbe42a00700 1 -- 172.21.15.27:0/1166504045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe44113360 con 0x7fbe44104100 2024-05-10T11:31:58.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.902+0000 7fbe42a00700 1 --2- 172.21.15.27:0/1166504045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe44104100 0x7fbe44118f40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000e9c0 tx=0x7fbe3000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:58.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.903+0000 7fbe3be00700 1 -- 172.21.15.27:0/1166504045 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000c9c0 con 0x7fbe44104100 2024-05-10T11:31:58.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.903+0000 7fbe4a800700 1 -- 172.21.15.27:0/1166504045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe44113650 con 0x7fbe44104100 2024-05-10T11:31:58.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.903+0000 7fbe4a800700 1 -- 172.21.15.27:0/1166504045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe441b5620 con 0x7fbe44104100 2024-05-10T11:31:58.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.904+0000 7fbe3be00700 1 -- 172.21.15.27:0/1166504045 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe3000cb60 con 0x7fbe44104100 2024-05-10T11:31:58.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.904+0000 7fbe3be00700 1 -- 172.21.15.27:0/1166504045 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe30018680 con 0x7fbe44104100 2024-05-10T11:31:58.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.913+0000 7fbe3be00700 1 -- 172.21.15.27:0/1166504045 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fbe30018820 con 0x7fbe44104100 2024-05-10T11:31:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.914+0000 7fbe5515fb80 1 Processor -- start 2024-05-10T11:31:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.914+0000 7fbe5515fb80 1 -- start start 2024-05-10T11:31:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.915+0000 7fbe5515fb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 0x564222568fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.915+0000 7fbe5515fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225694e0 0x5642225632d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.915+0000 7fbe5515fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x564222563d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.915+0000 7fbe5515fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5642224b5a20 con 0x5642225638f0 2024-05-10T11:31:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.915+0000 7fbe5515fb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564222453760 con 0x564222568bc0 2024-05-10T11:31:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.915+0000 7fbe5515fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5642224b19e0 con 0x5642225694e0 2024-05-10T11:31:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.915+0000 7fbe43e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 0x564222568fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.915+0000 7fbe43e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 0x564222568fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44506/0 (socket says 172.21.15.27:44506) 2024-05-10T11:31:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.915+0000 7fbe43e00700 1 -- 172.21.15.27:0/601380447 learned_addr learned my addr 172.21.15.27:0/601380447 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.915+0000 7fbe43400700 1 --2- 172.21.15.27:0/601380447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225694e0 0x5642225632d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.916+0000 7fbe43e00700 1 -- 172.21.15.27:0/601380447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225694e0 msgr2=0x5642225632d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.916+0000 7fbe43e00700 1 --2- 172.21.15.27:0/601380447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225694e0 0x5642225632d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.916+0000 7fbe43e00700 1 -- 172.21.15.27:0/601380447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 msgr2=0x564222563d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.916+0000 7fbe43e00700 1 --2- 172.21.15.27:0/601380447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x564222563d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.916+0000 7fbe43e00700 1 -- 172.21.15.27:0/601380447 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642224028d0 con 0x564222568bc0 2024-05-10T11:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.916+0000 7fbe43400700 1 --2- 172.21.15.27:0/601380447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225694e0 0x5642225632d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.916+0000 7fbe43e00700 1 --2- 172.21.15.27:0/601380447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 0x564222568fa0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0094b0 tx=0x7fbe3c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd73ce52a6113e94 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.916+0000 7fbe38c00700 1 -- 172.21.15.27:0/601380447 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c004db0 con 0x564222568bc0 2024-05-10T11:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.917+0000 7fbe38c00700 1 -- 172.21.15.27:0/601380447 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe3c0054d0 con 0x564222568bc0 2024-05-10T11:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.917+0000 7fbe38c00700 1 -- 172.21.15.27:0/601380447 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c015a00 con 0x564222568bc0 2024-05-10T11:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.918+0000 7fbe5515fb80 1 -- 172.21.15.27:0/601380447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 msgr2=0x564222568fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.918+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/601380447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 0x564222568fa0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0094b0 tx=0x7fbe3c009920 comp rx=0 tx=0).stop 2024-05-10T11:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.919+0000 7fbe5515fb80 1 -- 172.21.15.27:0/601380447 shutdown_connections 2024-05-10T11:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.919+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/601380447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x564222563d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.919+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/601380447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 0x564222568fa0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.919+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/601380447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225694e0 0x5642225632d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.919+0000 7fbe5515fb80 1 -- 172.21.15.27:0/601380447 >> 172.21.15.27:0/601380447 conn(0x5642224db6d0 msgr2=0x56422255db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.919+0000 7fbe5515fb80 1 -- 172.21.15.27:0/601380447 shutdown_connections 2024-05-10T11:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.919+0000 7fbe5515fb80 1 -- 172.21.15.27:0/601380447 wait complete. 2024-05-10T11:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.920+0000 7fbe5515fb80 1 -- start start 2024-05-10T11:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.920+0000 7fbe3be00700 1 --2- 172.21.15.27:0/1166504045 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbe280b2980 0x7fbe280b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.920+0000 7fbe5515fb80 1 Processor -- start 2024-05-10T11:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.921+0000 7fbe5515fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225638f0 0x56422256f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.921+0000 7fbe5515fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564222568bc0 0x56422256c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.921+0000 7fbe5515fb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642225694e0 0x56422256cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.921+0000 7fbe5515fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56422256ff20 con 0x564222568bc0 2024-05-10T11:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.921+0000 7fbe5515fb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56422241bb20 con 0x5642225694e0 2024-05-10T11:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.921+0000 7fbe5515fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5642225700a0 con 0x5642225638f0 2024-05-10T11:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.921+0000 7fbe43e00700 1 --2- 172.21.15.27:0/1166504045 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbe280b2980 0x7fbe280b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.921+0000 7fbe5515fb80 10 client.0 _ll_register_callbacks cb 0x56422245cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.921+0000 7fbe43400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225638f0 0x56422256f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.921+0000 7fbe43400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225638f0 0x56422256f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50534/0 (socket says 172.21.15.27:50534) 2024-05-10T11:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.921+0000 7fbe43400700 1 -- 172.21.15.27:0/1849358978 learned_addr learned my addr 172.21.15.27:0/1849358978 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.922+0000 7fbe3be00700 1 -- 172.21.15.27:0/1166504045 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9591+0+0 (secure 0 0 0) 0x7fbe300e2a60 con 0x7fbe44104100 2024-05-10T11:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.922+0000 7fbe43400700 1 -- 172.21.15.27:0/1849358978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642225694e0 msgr2=0x56422256cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-10T11:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.922+0000 7fbe43400700 1 --2- 172.21.15.27:0/1849358978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642225694e0 0x56422256cab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.922+0000 7fbe43400700 1 -- 172.21.15.27:0/1849358978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564222568bc0 msgr2=0x56422256c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.922+0000 7fbe43400700 1 --2- 172.21.15.27:0/1849358978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564222568bc0 0x56422256c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.922+0000 7fbe43400700 1 -- 172.21.15.27:0/1849358978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56422247b420 con 0x5642225638f0 2024-05-10T11:31:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.923+0000 7fbe43400700 1 --2- 172.21.15.27:0/1849358978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225638f0 0x56422256f1d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbe34019000 tx=0x7fbe340052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3134b14c2ff763b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.923+0000 7fbe2d600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-10T11:31:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.923+0000 7fbe2d600700 1 -- 172.21.15.27:0/1849358978 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe34025060 con 0x5642225638f0 2024-05-10T11:31:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.924+0000 7fbe2d600700 1 -- 172.21.15.27:0/1849358978 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9591+0+0 (secure 0 0 0) 0x7fbe3400e440 con 0x5642225638f0 2024-05-10T11:31:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.924+0000 7fbe5515fb80 1 -- 172.21.15.27:0/1849358978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe3c0054d0 con 0x5642225638f0 2024-05-10T11:31:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.924+0000 7fbe2d600700 1 -- 172.21.15.27:0/1849358978 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fbe3401c050 con 0x5642225638f0 2024-05-10T11:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.924+0000 7fbe5515fb80 10 client.27718 fetch_fsmap learned FSMap version 45 2024-05-10T11:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.924+0000 7fbe5515fb80 1 -- 172.21.15.27:0/1849358978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56422257a820 con 0x5642225638f0 2024-05-10T11:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.925+0000 7fbe43e00700 1 --2- 172.21.15.27:0/1166504045 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbe280b2980 0x7fbe280b4e40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00b010 tx=0x7fbe3c000f40 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.925+0000 7fbe2d600700 1 -- 172.21.15.27:0/1849358978 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe3c0054d0 con 0x5642225638f0 2024-05-10T11:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.925+0000 7fbe2d600700 1 -- 172.21.15.27:0/1849358978 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7fbe3400f510 con 0x5642225638f0 2024-05-10T11:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.925+0000 7fbe5515fb80 10 client.27718 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.925+0000 7fbe5515fb80 10 client.27718 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.925+0000 7fbe5515fb80 20 client.27718 populate_metadata read hostname 'smithi027' 2024-05-10T11:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.925+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/1849358978 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56422257ce60 0x56422257f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.925+0000 7fbe5515fb80 4 client.27718 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:31:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.925+0000 7fbe5515fb80 1 -- 172.21.15.27:0/1849358978 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564222530140 con 0x56422257ce60 2024-05-10T11:31:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.926+0000 7fbe42a00700 1 --2- 172.21.15.27:0/1849358978 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56422257ce60 0x56422257f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.927+0000 7fbe42a00700 1 --2- 172.21.15.27:0/1849358978 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56422257ce60 0x56422257f250 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ea08eb3936f5ebac server_cookie=af7eb4e191326ee3 in_seq=0 out_seq=0 2024-05-10T11:31:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.927+0000 7fbe2d600700 10 client.27718 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:31:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.928+0000 7fbe2d600700 1 -- 172.21.15.27:0/1849358978 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564222530140 con 0x56422257ce60 2024-05-10T11:31:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.928+0000 7fbe2d600700 10 client.27718 handle_command_reply: tid=0 2024-05-10T11:31:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.928+0000 7fbe5515fb80 1 client.27718 shutdown 2024-05-10T11:31:58.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.928+0000 7fbe5515fb80 1 -- 172.21.15.27:0/1849358978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225638f0 msgr2=0x56422256f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:58.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.928+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/1849358978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225638f0 0x56422256f1d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbe34019000 tx=0x7fbe340052f0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.929+0000 7fbe5515fb80 1 -- 172.21.15.27:0/1849358978 shutdown_connections 2024-05-10T11:31:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.929+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/1849358978 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x56422257ce60 0x56422257f250 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.929+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/1849358978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225638f0 0x56422256f1d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.929+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/1849358978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642225694e0 0x56422256cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.929+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/1849358978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564222568bc0 0x56422256c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.929+0000 7fbe5515fb80 1 -- 172.21.15.27:0/1849358978 >> 172.21.15.27:0/1849358978 conn(0x5642224db6d0 msgr2=0x56422255d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.929+0000 7fbe2d600700 0 client.27718 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:31:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.929+0000 7fbe5515fb80 1 -- 172.21.15.27:0/1849358978 shutdown_connections 2024-05-10T11:31:58.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.930+0000 7fbe5515fb80 1 -- 172.21.15.27:0/1849358978 wait complete. 2024-05-10T11:31:58.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.930+0000 7fbe5515fb80 20 client.27718 trim_cache size 0 max 16384 2024-05-10T11:31:58.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.957+0000 7fbe5515fb80 1 Processor -- start 2024-05-10T11:31:58.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.957+0000 7fbe5515fb80 1 -- start start 2024-05-10T11:31:58.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.958+0000 7fbe5515fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x564222579040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.958+0000 7fbe5515fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564222568bc0 0x564222569fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.958+0000 7fbe5515fb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642225694e0 0x56422255d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.958+0000 7fbe5515fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2400f020 con 0x5642225638f0 2024-05-10T11:31:58.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.958+0000 7fbe5515fb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbe24010b60 con 0x5642225694e0 2024-05-10T11:31:58.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.958+0000 7fbe5515fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2400f6f0 con 0x564222568bc0 2024-05-10T11:31:58.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.959+0000 7fbe42a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564222568bc0 0x564222569fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.959+0000 7fbe43400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x564222579040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.959+0000 7fbe43e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642225694e0 0x56422255d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.959+0000 7fbe43e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642225694e0 0x56422255d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44526/0 (socket says 172.21.15.27:44526) 2024-05-10T11:31:58.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.959+0000 7fbe43e00700 1 -- 172.21.15.27:0/476076601 learned_addr learned my addr 172.21.15.27:0/476076601 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:58.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.960+0000 7fbe43400700 1 -- 172.21.15.27:0/476076601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564222568bc0 msgr2=0x564222569fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:58.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.960+0000 7fbe43400700 1 --2- 172.21.15.27:0/476076601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564222568bc0 0x564222569fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.960+0000 7fbe43400700 1 -- 172.21.15.27:0/476076601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642225694e0 msgr2=0x56422255d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:58.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.960+0000 7fbe43400700 1 --2- 172.21.15.27:0/476076601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642225694e0 0x56422255d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.960+0000 7fbe43400700 1 -- 172.21.15.27:0/476076601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56422256ab00 con 0x5642225638f0 2024-05-10T11:31:58.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.960+0000 7fbe42a00700 1 --2- 172.21.15.27:0/476076601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564222568bc0 0x564222569fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:31:58.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.960+0000 7fbe43400700 1 --2- 172.21.15.27:0/476076601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x564222579040 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbe3400ffd0 tx=0x7fbe34003b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642cecfa72425ee9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:58.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.961+0000 7fbe2fe00700 1 -- 172.21.15.27:0/476076601 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe34025070 con 0x5642225638f0 2024-05-10T11:31:58.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.961+0000 7fbe2fe00700 1 -- 172.21.15.27:0/476076601 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe34005b20 con 0x5642225638f0 2024-05-10T11:31:58.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.961+0000 7fbe2fe00700 1 -- 172.21.15.27:0/476076601 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3402f080 con 0x5642225638f0 2024-05-10T11:31:58.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.963+0000 7fbe5515fb80 1 -- 172.21.15.27:0/476076601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 msgr2=0x564222579040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.963+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/476076601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x564222579040 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbe3400ffd0 tx=0x7fbe34003b50 comp rx=0 tx=0).stop 2024-05-10T11:31:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.964+0000 7fbe5515fb80 1 -- 172.21.15.27:0/476076601 shutdown_connections 2024-05-10T11:31:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.964+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/476076601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642225694e0 0x56422255d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.964+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/476076601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x564222579040 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.964+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/476076601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564222568bc0 0x564222569fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.964+0000 7fbe5515fb80 1 -- 172.21.15.27:0/476076601 >> 172.21.15.27:0/476076601 conn(0x5642224db6d0 msgr2=0x5642225698c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.965+0000 7fbe5515fb80 1 -- 172.21.15.27:0/476076601 shutdown_connections 2024-05-10T11:31:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.965+0000 7fbe5515fb80 1 -- 172.21.15.27:0/476076601 wait complete. 2024-05-10T11:31:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.966+0000 7fbe5515fb80 1 -- start start 2024-05-10T11:31:58.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.966+0000 7fbe5515fb80 1 Processor -- start 2024-05-10T11:31:58.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.967+0000 7fbe5515fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x5642225759e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.967+0000 7fbe5515fb80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 0x5642225840e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.967+0000 7fbe5515fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225694e0 0x564222584640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.967+0000 7fbe5515fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbe24010490 con 0x5642225638f0 2024-05-10T11:31:58.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.967+0000 7fbe5515fb80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2400fdc0 con 0x564222568bc0 2024-05-10T11:31:58.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.967+0000 7fbe5515fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2400cfe0 con 0x5642225694e0 2024-05-10T11:31:58.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.967+0000 7fbe5515fb80 10 client.0 _ll_register_callbacks cb 0x56422245cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:31:58.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.967+0000 7fbe43e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225694e0 0x564222584640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.968+0000 7fbe43400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x5642225759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.968+0000 7fbe42a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 0x5642225840e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.969+0000 7fbe43e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225694e0 0x564222584640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50556/0 (socket says 172.21.15.27:50556) 2024-05-10T11:31:58.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.969+0000 7fbe43e00700 1 -- 172.21.15.27:0/2097899616 learned_addr learned my addr 172.21.15.27:0/2097899616 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:31:58.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.970+0000 7fbe42a00700 1 -- 172.21.15.27:0/2097899616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225694e0 msgr2=0x564222584640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:58.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.970+0000 7fbe42a00700 1 --2- 172.21.15.27:0/2097899616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225694e0 0x564222584640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.970+0000 7fbe42a00700 1 -- 172.21.15.27:0/2097899616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 msgr2=0x5642225759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:31:58.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.970+0000 7fbe42a00700 1 --2- 172.21.15.27:0/2097899616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x5642225759e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.970+0000 7fbe42a00700 1 -- 172.21.15.27:0/2097899616 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564222584eb0 con 0x564222568bc0 2024-05-10T11:31:58.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.970+0000 7fbe42a00700 1 --2- 172.21.15.27:0/2097899616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 0x5642225840e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbe300dfd80 tx=0x7fbe300b2040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dd42c4091fb2293 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:31:58.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.970+0000 7fbe2d600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:31:58.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.970+0000 7fbe43400700 1 --2- 172.21.15.27:0/2097899616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x5642225759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-10T11:31:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.972+0000 7fbe2d600700 1 -- 172.21.15.27:0/2097899616 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe300aa010 con 0x564222568bc0 2024-05-10T11:31:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.972+0000 7fbe2d600700 1 -- 172.21.15.27:0/2097899616 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9591+0+0 (secure 0 0 0) 0x7fbe300a5f00 con 0x564222568bc0 2024-05-10T11:31:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.972+0000 7fbe5515fb80 1 -- 172.21.15.27:0/2097899616 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5642224c3400 con 0x564222568bc0 2024-05-10T11:31:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.972+0000 7fbe2d600700 1 -- 172.21.15.27:0/2097899616 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fbe300a9910 con 0x564222568bc0 2024-05-10T11:31:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.973+0000 7fbe5515fb80 10 client.27701 fetch_fsmap learned FSMap version 45 2024-05-10T11:31:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.973+0000 7fbe5515fb80 1 -- 172.21.15.27:0/2097899616 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5642225e7770 con 0x564222568bc0 2024-05-10T11:31:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.973+0000 7fbe2d600700 1 -- 172.21.15.27:0/2097899616 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5642224c3400 con 0x564222568bc0 2024-05-10T11:31:58.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.973+0000 7fbe2d600700 1 -- 172.21.15.27:0/2097899616 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7fbe300b22e0 con 0x564222568bc0 2024-05-10T11:31:58.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.973+0000 7fbe5515fb80 10 client.27701 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:31:58.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.973+0000 7fbe5515fb80 10 client.27701 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-05-10T11:31:58.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.973+0000 7fbe5515fb80 20 client.27701 populate_metadata read hostname 'smithi027' 2024-05-10T11:31:58.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.973+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/2097899616 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5642225e7e80 0x5642225ea2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:31:58.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.973+0000 7fbe5515fb80 4 client.27701 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:31:58.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.973+0000 7fbe5515fb80 1 -- 172.21.15.27:0/2097899616 --> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5642224bd4a0 con 0x5642225e7e80 2024-05-10T11:31:58.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.974+0000 7fbe43400700 1 --2- 172.21.15.27:0/2097899616 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5642225e7e80 0x5642225ea2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:31:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.974+0000 7fbe43400700 1 --2- 172.21.15.27:0/2097899616 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5642225e7e80 0x5642225ea2a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4204fa2f2177a8e7 server_cookie=489ae456bf9f9593 in_seq=0 out_seq=0 2024-05-10T11:31:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.974+0000 7fbe2d600700 10 client.27701 ms_handle_connect on v2:172.21.15.189:6836/3185830081 2024-05-10T11:31:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe2d600700 1 -- 172.21.15.27:0/2097899616 <== mds.3 v2:172.21.15.189:6836/3185830081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5642224bd4a0 con 0x5642225e7e80 2024-05-10T11:31:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe2d600700 10 client.27701 handle_command_reply: tid=0 2024-05-10T11:31:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe5515fb80 1 client.27701 shutdown 2024-05-10T11:31:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe5515fb80 1 -- 172.21.15.27:0/2097899616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 msgr2=0x5642225840e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/2097899616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 0x5642225840e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbe300dfd80 tx=0x7fbe300b2040 comp rx=0 tx=0).stop 2024-05-10T11:31:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe5515fb80 1 -- 172.21.15.27:0/2097899616 shutdown_connections 2024-05-10T11:31:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/2097899616 >> [v2:172.21.15.189:6836/3185830081,v1:172.21.15.189:6837/3185830081] conn(0x5642225e7e80 0x5642225ea2a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/2097899616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642225694e0 0x564222584640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/2097899616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642225638f0 0x5642225759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe5515fb80 1 --2- 172.21.15.27:0/2097899616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564222568bc0 0x5642225840e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe5515fb80 1 -- 172.21.15.27:0/2097899616 >> 172.21.15.27:0/2097899616 conn(0x5642224db6d0 msgr2=0x5642225676a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:58.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe2d600700 0 client.27701 ms_handle_reset on v2:172.21.15.189:6836/3185830081 2024-05-10T11:31:58.977 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:31:58.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.975+0000 7fbe5515fb80 1 -- 172.21.15.27:0/2097899616 shutdown_connections 2024-05-10T11:31:58.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.976+0000 7fbe5515fb80 1 -- 172.21.15.27:0/2097899616 wait complete. 2024-05-10T11:31:58.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.976+0000 7fbe5515fb80 20 client.27701 trim_cache size 0 max 16384 2024-05-10T11:31:58.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.977+0000 7fbe4a800700 1 -- 172.21.15.27:0/1166504045 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbe280b2980 msgr2=0x7fbe280b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:58.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.977+0000 7fbe4a800700 1 --2- 172.21.15.27:0/1166504045 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbe280b2980 0x7fbe280b4e40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00b010 tx=0x7fbe3c000f40 comp rx=0 tx=0).stop 2024-05-10T11:31:58.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.977+0000 7fbe4a800700 1 -- 172.21.15.27:0/1166504045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe44104100 msgr2=0x7fbe44118f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:31:58.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.977+0000 7fbe4a800700 1 --2- 172.21.15.27:0/1166504045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe44104100 0x7fbe44118f40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000e9c0 tx=0x7fbe3000ecd0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.977+0000 7fbe4a800700 1 -- 172.21.15.27:0/1166504045 shutdown_connections 2024-05-10T11:31:58.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.977+0000 7fbe4a800700 1 --2- 172.21.15.27:0/1166504045 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbe280b2980 0x7fbe280b4e40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.977+0000 7fbe4a800700 1 --2- 172.21.15.27:0/1166504045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe44108850 0x7fbe44112ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.977+0000 7fbe4a800700 1 --2- 172.21.15.27:0/1166504045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe44103750 0x7fbe44118a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.977+0000 7fbe4a800700 1 --2- 172.21.15.27:0/1166504045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe44104100 0x7fbe44118f40 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:31:58.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.977+0000 7fbe4a800700 1 -- 172.21.15.27:0/1166504045 >> 172.21.15.27:0/1166504045 conn(0x7fbe4407a600 msgr2=0x7fbe44107450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:31:58.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.980+0000 7fbe4a800700 1 -- 172.21.15.27:0/1166504045 shutdown_connections 2024-05-10T11:31:58.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:31:58.980+0000 7fbe4a800700 1 -- 172.21.15.27:0/1166504045 wait complete. 2024-05-10T11:31:59.185 INFO:tasks.workunit.client.0.smithi027.stdout: 10 119144 21.91 MB/sec execute 163 sec latency 97.059 ms 2024-05-10T11:31:59.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:31:59 smithi189 ceph-mon[34842]: pgmap v589: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.6 MiB/s wr, 331 op/s 2024-05-10T11:31:59.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:31:59 smithi027 ceph-mon[30054]: pgmap v589: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.6 MiB/s wr, 331 op/s 2024-05-10T11:31:59.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:31:59 smithi076 ceph-mon[34152]: pgmap v589: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.6 MiB/s wr, 331 op/s 2024-05-10T11:32:00.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-10T11:32:00.185 INFO:tasks.workunit.client.0.smithi027.stdout: 10 119664 21.95 MB/sec execute 164 sec latency 138.417 ms 2024-05-10T11:32:00.591 INFO:tasks.workunit.client.0.smithi027.stdout:[104961] write failed on handle 10988 (Resource temporarily unavailable) 2024-05-10T11:32:00.591 INFO:tasks.workunit.client.0.smithi027.stdout:Child failed with status 1 2024-05-10T11:32:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T11:32:00.650 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-05-10T11:32:00.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-10T11:32:00.694 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:32:01.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:01 smithi027 ceph-mon[30054]: pgmap v590: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 9.5 MiB/s wr, 465 op/s 2024-05-10T11:32:01.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:01 smithi189 ceph-mon[34842]: pgmap v590: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 9.5 MiB/s wr, 465 op/s 2024-05-10T11:32:01.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:01 smithi076 ceph-mon[34152]: pgmap v590: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 9.5 MiB/s wr, 465 op/s 2024-05-10T11:32:01.943 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d39fa08867836ac070c47cb66c782fa559fd480b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d39fa08867836ac070c47cb66c782fa559fd480b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d39fa08867836ac070c47cb66c782fa559fd480b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39fa08867836ac070c47cb66c782fa559fd480b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-05-10T11:32:02.201 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2d1f3391e3474d42902fb733914c6374 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d39fa08867836ac070c47cb66c782fa559fd480b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d39fa08867836ac070c47cb66c782fa559fd480b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d39fa08867836ac070c47cb66c782fa559fd480b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39fa08867836ac070c47cb66c782fa559fd480b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-05-10T11:32:02.204 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-05-10T11:32:02.215 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-05-10T11:32:02.215 INFO:teuthology.task.exec:Executing custom commands... 2024-05-10T11:32:02.215 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-05-10T11:32:02.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-05-10T11:32:02.324 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-05-10T11:32:02.462 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-10T11:32:02.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.799+0000 7f64db800640 1 -- 172.21.15.27:0/207302697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64d4109210 msgr2=0x7f64d41095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:02.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.799+0000 7f64db800640 1 --2- 172.21.15.27:0/207302697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64d4109210 0x7f64d41095f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f64bc009900 tx=0x7f64bc02a400 comp rx=0 tx=0).stop 2024-05-10T11:32:02.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.799+0000 7f64db800640 1 -- 172.21.15.27:0/207302697 shutdown_connections 2024-05-10T11:32:02.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.799+0000 7f64db800640 1 --2- 172.21.15.27:0/207302697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d410e2e0 0x7f64d4115eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:02.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.799+0000 7f64db800640 1 --2- 172.21.15.27:0/207302697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64d4109bc0 0x7f64d410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:02.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.799+0000 7f64db800640 1 --2- 172.21.15.27:0/207302697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64d4109210 0x7f64d41095f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:02.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.799+0000 7f64db800640 1 -- 172.21.15.27:0/207302697 >> 172.21.15.27:0/207302697 conn(0x7f64d4076f40 msgr2=0x7f64d4079360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:02.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.801+0000 7f64db800640 1 -- 172.21.15.27:0/207302697 shutdown_connections 2024-05-10T11:32:02.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.801+0000 7f64db800640 1 -- 172.21.15.27:0/207302697 wait complete. 2024-05-10T11:32:02.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.802+0000 7f64db800640 1 Processor -- start 2024-05-10T11:32:02.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.806+0000 7f64db800640 1 -- start start 2024-05-10T11:32:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.807+0000 7f64db800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64d410e2e0 0x7f64d4074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.807+0000 7f64db800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64d406d3d0 0x7f64d406d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.808+0000 7f64db800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d406dd70 0x7f64d4071e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.808+0000 7f64db800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f64d4119270 con 0x7f64d410e2e0 2024-05-10T11:32:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.808+0000 7f64db800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41190f0 con 0x7f64d406d3d0 2024-05-10T11:32:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.808+0000 7f64db800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41193f0 con 0x7f64d406dd70 2024-05-10T11:32:02.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.808+0000 7f64d8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64d410e2e0 0x7f64d4074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:02.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.808+0000 7f64d9800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d406dd70 0x7f64d4071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:02.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.808+0000 7f64d9800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d406dd70 0x7f64d4071e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50582/0 (socket says 172.21.15.27:50582) 2024-05-10T11:32:02.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.808+0000 7f64d9800640 1 -- 172.21.15.27:0/3274796156 learned_addr learned my addr 172.21.15.27:0/3274796156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:02.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.808+0000 7f64d3e00640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64d406d3d0 0x7f64d406d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:02.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.809+0000 7f64d9800640 1 -- 172.21.15.27:0/3274796156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64d406d3d0 msgr2=0x7f64d406d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:02.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.809+0000 7f64d9800640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64d406d3d0 0x7f64d406d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:02.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.809+0000 7f64d9800640 1 -- 172.21.15.27:0/3274796156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64d410e2e0 msgr2=0x7f64d4074300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:02.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.809+0000 7f64d9800640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64d410e2e0 0x7f64d4074300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:02.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.809+0000 7f64d9800640 1 -- 172.21.15.27:0/3274796156 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d4072360 con 0x7f64d406dd70 2024-05-10T11:32:02.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.809+0000 7f64d8e00640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64d410e2e0 0x7f64d4074300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:32:02.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.813+0000 7f64d9800640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d406dd70 0x7f64d4071e20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00cce0 tx=0x7f64cc00a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:02.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.813+0000 7f64d1600640 1 -- 172.21.15.27:0/3274796156 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc00f450 con 0x7f64d406dd70 2024-05-10T11:32:02.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.813+0000 7f64d1600640 1 -- 172.21.15.27:0/3274796156 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64cc00faa0 con 0x7f64d406dd70 2024-05-10T11:32:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.813+0000 7f64d1600640 1 -- 172.21.15.27:0/3274796156 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc018b70 con 0x7f64d406dd70 2024-05-10T11:32:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.813+0000 7f64db800640 1 -- 172.21.15.27:0/3274796156 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d40725f0 con 0x7f64d406dd70 2024-05-10T11:32:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.814+0000 7f64db800640 1 -- 172.21.15.27:0/3274796156 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d4072a90 con 0x7f64d406dd70 2024-05-10T11:32:02.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.816+0000 7f64db800640 1 -- 172.21.15.27:0/3274796156 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64d410a6e0 con 0x7f64d406dd70 2024-05-10T11:32:02.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.818+0000 7f64d1600640 1 -- 172.21.15.27:0/3274796156 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f64cc018d10 con 0x7f64d406dd70 2024-05-10T11:32:02.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.820+0000 7f64d1600640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f64b80b2a90 0x7f64b80b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:02.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.821+0000 7f64d1600640 1 -- 172.21.15.27:0/3274796156 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9679+0+0 (secure 0 0 0) 0x7f64cc0e2d60 con 0x7f64d406dd70 2024-05-10T11:32:02.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.822+0000 7f64d8e00640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f64b80b2a90 0x7f64b80b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:02.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.823+0000 7f64d8e00640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f64b80b2a90 0x7f64b80b4f50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f64bc006010 tx=0x7f64bc005a20 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:02.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:02.824+0000 7f64d1600640 1 -- 172.21.15.27:0/3274796156 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f64cc014030 con 0x7f64d406dd70 2024-05-10T11:32:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.105+0000 7f64db800640 1 -- 172.21.15.27:0/3274796156 --> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f64d4109bc0 con 0x7f64b80b2a90 2024-05-10T11:32:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.120+0000 7f64d1600640 1 -- 172.21.15.27:0/3274796156 <== mgr.25220 v2:172.21.15.27:6800/1167199542 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f64d4109bc0 con 0x7f64b80b2a90 2024-05-10T11:32:03.121 INFO:teuthology.orchestra.run.smithi027.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-10T00:00:00", "created": "2024-05-10T11:13:32", "first": "2024-05-10T11:14:00", "last": "2024-05-10T11:32:00", "last_pruned": "2024-05-10T11:32:00", "created_count": 19, "pruned_count": 13, "active": true} 2024-05-10T11:32:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.123+0000 7f64db800640 1 -- 172.21.15.27:0/3274796156 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f64b80b2a90 msgr2=0x7f64b80b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.123+0000 7f64db800640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f64b80b2a90 0x7f64b80b4f50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f64bc006010 tx=0x7f64bc005a20 comp rx=0 tx=0).stop 2024-05-10T11:32:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.123+0000 7f64db800640 1 -- 172.21.15.27:0/3274796156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d406dd70 msgr2=0x7f64d4071e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.123+0000 7f64db800640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d406dd70 0x7f64d4071e20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00cce0 tx=0x7f64cc00a620 comp rx=0 tx=0).stop 2024-05-10T11:32:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.123+0000 7f64db800640 1 -- 172.21.15.27:0/3274796156 shutdown_connections 2024-05-10T11:32:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.123+0000 7f64db800640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f64b80b2a90 0x7f64b80b4f50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.123+0000 7f64db800640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d406dd70 0x7f64d4071e20 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.123+0000 7f64db800640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64d406d3d0 0x7f64d406d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.123+0000 7f64db800640 1 --2- 172.21.15.27:0/3274796156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64d410e2e0 0x7f64d4074300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.123+0000 7f64db800640 1 -- 172.21.15.27:0/3274796156 >> 172.21.15.27:0/3274796156 conn(0x7f64d4076f40 msgr2=0x7f64d40792d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:03.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.123+0000 7f64db800640 1 -- 172.21.15.27:0/3274796156 shutdown_connections 2024-05-10T11:32:03.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.123+0000 7f64db800640 1 -- 172.21.15.27:0/3274796156 wait complete. 2024-05-10T11:32:03.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-10T11:32:03.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.334+0000 7f0615e00640 1 -- 172.21.15.27:0/1859121284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0610104e00 msgr2=0x7f06101051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.334+0000 7f0615e00640 1 --2- 172.21.15.27:0/1859121284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0610104e00 0x7f06101051e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0600009900 tx=0x7f060002a410 comp rx=0 tx=0).stop 2024-05-10T11:32:03.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.334+0000 7f0615e00640 1 -- 172.21.15.27:0/1859121284 shutdown_connections 2024-05-10T11:32:03.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.334+0000 7f0615e00640 1 --2- 172.21.15.27:0/1859121284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0610109ed0 0x7f0610111aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.334+0000 7f0615e00640 1 --2- 172.21.15.27:0/1859121284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06101057b0 0x7f06101097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.334+0000 7f0615e00640 1 --2- 172.21.15.27:0/1859121284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0610104e00 0x7f06101051e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.334+0000 7f0615e00640 1 -- 172.21.15.27:0/1859121284 >> 172.21.15.27:0/1859121284 conn(0x7f06101006a0 msgr2=0x7f0610102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:03.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.334+0000 7f0615e00640 1 -- 172.21.15.27:0/1859121284 shutdown_connections 2024-05-10T11:32:03.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.334+0000 7f0615e00640 1 -- 172.21.15.27:0/1859121284 wait complete. 2024-05-10T11:32:03.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.335+0000 7f0615e00640 1 Processor -- start 2024-05-10T11:32:03.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.335+0000 7f0615e00640 1 -- start start 2024-05-10T11:32:03.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.336+0000 7f0615e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0610104e00 0x7f061019b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.336+0000 7f0615e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0610109ed0 0x7f061019b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.336+0000 7f0615e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f061019fbc0 0x7f06101a0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.336+0000 7f0615e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0610114e30 con 0x7f0610109ed0 2024-05-10T11:32:03.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.336+0000 7f0615e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0610114cb0 con 0x7f061019fbc0 2024-05-10T11:32:03.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.336+0000 7f0615e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0610114fb0 con 0x7f0610104e00 2024-05-10T11:32:03.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.336+0000 7f0607e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0610109ed0 0x7f061019b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.336+0000 7f060f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f061019fbc0 0x7f06101a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.336+0000 7f0607e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0610109ed0 0x7f061019b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59162/0 (socket says 172.21.15.27:59162) 2024-05-10T11:32:03.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.336+0000 7f0607e00640 1 -- 172.21.15.27:0/3236744513 learned_addr learned my addr 172.21.15.27:0/3236744513 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:03.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.336+0000 7f060ea00640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0610104e00 0x7f061019b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.337+0000 7f060f400640 1 -- 172.21.15.27:0/3236744513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0610104e00 msgr2=0x7f061019b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.337+0000 7f060f400640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0610104e00 0x7f061019b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.337+0000 7f060f400640 1 -- 172.21.15.27:0/3236744513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0610109ed0 msgr2=0x7f061019b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.337+0000 7f060f400640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0610109ed0 0x7f061019b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.337+0000 7f060f400640 1 -- 172.21.15.27:0/3236744513 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06101a06b0 con 0x7f061019fbc0 2024-05-10T11:32:03.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.337+0000 7f0607e00640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0610109ed0 0x7f061019b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:32:03.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.337+0000 7f060ea00640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0610104e00 0x7f061019b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:32:03.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.337+0000 7f060f400640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f061019fbc0 0x7f06101a0070 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0608009fd0 tx=0x7f060800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:03.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.337+0000 7f0614a00640 1 -- 172.21.15.27:0/3236744513 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060800ce20 con 0x7f061019fbc0 2024-05-10T11:32:03.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.338+0000 7f0615e00640 1 -- 172.21.15.27:0/3236744513 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06101a09a0 con 0x7f061019fbc0 2024-05-10T11:32:03.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.338+0000 7f0614a00640 1 -- 172.21.15.27:0/3236744513 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0608005020 con 0x7f061019fbc0 2024-05-10T11:32:03.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.338+0000 7f0614a00640 1 -- 172.21.15.27:0/3236744513 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0608010690 con 0x7f061019fbc0 2024-05-10T11:32:03.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.338+0000 7f0615e00640 1 -- 172.21.15.27:0/3236744513 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06101a8190 con 0x7f061019fbc0 2024-05-10T11:32:03.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.339+0000 7f0615e00640 1 -- 172.21.15.27:0/3236744513 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0610050740 con 0x7f061019fbc0 2024-05-10T11:32:03.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.343+0000 7f0614a00640 1 -- 172.21.15.27:0/3236744513 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f06080054d0 con 0x7f061019fbc0 2024-05-10T11:32:03.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.345+0000 7f0614a00640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f05f40b2a10 0x7f05f40b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.345+0000 7f0614a00640 1 -- 172.21.15.27:0/3236744513 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9679+0+0 (secure 0 0 0) 0x7f06080e3340 con 0x7f061019fbc0 2024-05-10T11:32:03.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.345+0000 7f060ea00640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f05f40b2a10 0x7f05f40b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.345+0000 7f0614a00640 1 -- 172.21.15.27:0/3236744513 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f06080aa8a0 con 0x7f061019fbc0 2024-05-10T11:32:03.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.346+0000 7f060ea00640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f05f40b2a10 0x7f05f40b4ed0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0600031440 tx=0x7f0600005b50 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:03.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.440+0000 7f6e4e000700 1 -- 172.21.15.27:0/2683847620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e48104100 msgr2=0x7f6e48108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.440+0000 7f6e4e000700 1 --2- 172.21.15.27:0/2683847620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e48104100 0x7f6e48108120 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f6e340099c0 tx=0x7f6e34009cd0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.440+0000 7f6e4e000700 1 -- 172.21.15.27:0/2683847620 shutdown_connections 2024-05-10T11:32:03.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.440+0000 7f6e4e000700 1 --2- 172.21.15.27:0/2683847620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e48108850 0x7f6e481103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.440+0000 7f6e4e000700 1 --2- 172.21.15.27:0/2683847620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e48103750 0x7f6e48103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.440+0000 7f6e4e000700 1 --2- 172.21.15.27:0/2683847620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e48104100 0x7f6e48108120 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.440+0000 7f6e4e000700 1 -- 172.21.15.27:0/2683847620 >> 172.21.15.27:0/2683847620 conn(0x7f6e4807a600 msgr2=0x7f6e4807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.440+0000 7f6e4e000700 1 -- 172.21.15.27:0/2683847620 shutdown_connections 2024-05-10T11:32:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.440+0000 7f6e4e000700 1 -- 172.21.15.27:0/2683847620 wait complete. 2024-05-10T11:32:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.441+0000 7f6e4e000700 1 Processor -- start 2024-05-10T11:32:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.441+0000 7f6e4e000700 1 -- start start 2024-05-10T11:32:03.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.442+0000 7f6e4e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e48103750 0x7f6e4819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.442+0000 7f6e4e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e48104100 0x7f6e4819c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.442+0000 7f6e4e000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e48108850 0x7f6e481a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.442+0000 7f6e4e000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e48113680 con 0x7f6e48103750 2024-05-10T11:32:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.442+0000 7f6e4e000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6e48113500 con 0x7f6e48108850 2024-05-10T11:32:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.442+0000 7f6e4e000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6e48113800 con 0x7f6e48104100 2024-05-10T11:32:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.442+0000 7f6e47e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e48108850 0x7f6e481a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.442+0000 7f6e47400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e48103750 0x7f6e4819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.442+0000 7f6e47e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e48108850 0x7f6e481a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44592/0 (socket says 172.21.15.27:44592) 2024-05-10T11:32:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.442+0000 7f6e47e00700 1 -- 172.21.15.27:0/257543743 learned_addr learned my addr 172.21.15.27:0/257543743 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.442+0000 7f6e46a00700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e48104100 0x7f6e4819c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.443+0000 7f6e47e00700 1 -- 172.21.15.27:0/257543743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e48104100 msgr2=0x7f6e4819c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.443+0000 7f6e47e00700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e48104100 0x7f6e4819c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.443+0000 7f6e47e00700 1 -- 172.21.15.27:0/257543743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e48103750 msgr2=0x7f6e4819bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.443+0000 7f6e47e00700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e48103750 0x7f6e4819bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.443+0000 7f6e47e00700 1 -- 172.21.15.27:0/257543743 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e481a0d20 con 0x7f6e48108850 2024-05-10T11:32:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.443+0000 7f6e47e00700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e48108850 0x7f6e481a05e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000e940 tx=0x7f6e4000ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.443+0000 7f6e3fe00700 1 -- 172.21.15.27:0/257543743 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e40004a30 con 0x7f6e48108850 2024-05-10T11:32:03.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.443+0000 7f6e4e000700 1 -- 172.21.15.27:0/257543743 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e481a1010 con 0x7f6e48108850 2024-05-10T11:32:03.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.443+0000 7f6e3fe00700 1 -- 172.21.15.27:0/257543743 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e400054d0 con 0x7f6e48108850 2024-05-10T11:32:03.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.443+0000 7f6e3fe00700 1 -- 172.21.15.27:0/257543743 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4000ccf0 con 0x7f6e48108850 2024-05-10T11:32:03.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.444+0000 7f6e4e000700 1 -- 172.21.15.27:0/257543743 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e481a8890 con 0x7f6e48108850 2024-05-10T11:32:03.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.447+0000 7f6e3fe00700 1 -- 172.21.15.27:0/257543743 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f6e40005020 con 0x7f6e48108850 2024-05-10T11:32:03.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.448+0000 7f6e58913b80 1 Processor -- start 2024-05-10T11:32:03.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.448+0000 7f6e58913b80 1 -- start start 2024-05-10T11:32:03.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.448+0000 7f6e58913b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805060 0x563ea8805440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.448+0000 7f6e58913b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea8805980 0x563ea8809490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.448+0000 7f6e58913b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea88099d0 0x563ea8809e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.448+0000 7f6e58913b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563ea8757a20 con 0x563ea88099d0 2024-05-10T11:32:03.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.448+0000 7f6e58913b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563ea86f5760 con 0x563ea8805060 2024-05-10T11:32:03.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.448+0000 7f6e58913b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563ea87539e0 con 0x563ea8805980 2024-05-10T11:32:03.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e47400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea8805980 0x563ea8809490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e3fe00700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6e2c0b2a70 0x7f6e2c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e47400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea8805980 0x563ea8809490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:50624/0 (socket says 172.21.15.27:50624) 2024-05-10T11:32:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e47400700 1 -- 172.21.15.27:0/422229591 learned_addr learned my addr 172.21.15.27:0/422229591 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e46a00700 1 --2- 172.21.15.27:0/422229591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805060 0x563ea8805440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e46a00700 1 --2- 172.21.15.27:0/422229591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea88099d0 0x563ea8809e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e3fe00700 1 -- 172.21.15.27:0/257543743 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9679+0+0 (secure 0 0 0) 0x7f6e400e2d50 con 0x7f6e48108850 2024-05-10T11:32:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e47e00700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6e2c0b2a70 0x7f6e2c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e47400700 1 -- 172.21.15.27:0/422229591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805060 msgr2=0x563ea8805440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e47400700 1 --2- 172.21.15.27:0/422229591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805060 0x563ea8805440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e47400700 1 -- 172.21.15.27:0/422229591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea88099d0 msgr2=0x563ea8809e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e47400700 1 --2- 172.21.15.27:0/422229591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea88099d0 0x563ea8809e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.449+0000 7f6e47400700 1 -- 172.21.15.27:0/422229591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ea86a48d0 con 0x563ea8805980 2024-05-10T11:32:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.450+0000 7f6e47400700 1 --2- 172.21.15.27:0/422229591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea8805980 0x563ea8809490 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6e38011650 tx=0x7f6e38009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b138c29e4602204b server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.450+0000 7f6e3cc00700 1 -- 172.21.15.27:0/422229591 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e38019020 con 0x563ea8805980 2024-05-10T11:32:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.450+0000 7f6e3cc00700 1 -- 172.21.15.27:0/422229591 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e380079b0 con 0x563ea8805980 2024-05-10T11:32:03.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.450+0000 7f6e3cc00700 1 -- 172.21.15.27:0/422229591 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e38015930 con 0x563ea8805980 2024-05-10T11:32:03.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.451+0000 7f6e58913b80 1 -- 172.21.15.27:0/422229591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea8805980 msgr2=0x563ea8809490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.451+0000 7f6e58913b80 1 --2- 172.21.15.27:0/422229591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea8805980 0x563ea8809490 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6e38011650 tx=0x7f6e38009920 comp rx=0 tx=0).stop 2024-05-10T11:32:03.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.451+0000 7f6e47e00700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6e2c0b2a70 0x7f6e2c0b4f30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6e400098e0 tx=0x7f6e40009870 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:03.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.451+0000 7f6e58913b80 1 -- 172.21.15.27:0/422229591 shutdown_connections 2024-05-10T11:32:03.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.451+0000 7f6e58913b80 1 --2- 172.21.15.27:0/422229591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea88099d0 0x563ea8809e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.451+0000 7f6e58913b80 1 --2- 172.21.15.27:0/422229591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805060 0x563ea8805440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.451+0000 7f6e58913b80 1 --2- 172.21.15.27:0/422229591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea8805980 0x563ea8809490 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.451+0000 7f6e58913b80 1 -- 172.21.15.27:0/422229591 >> 172.21.15.27:0/422229591 conn(0x563ea8733b60 msgr2=0x563ea87ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:03.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.451+0000 7f6e58913b80 1 -- 172.21.15.27:0/422229591 shutdown_connections 2024-05-10T11:32:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.451+0000 7f6e58913b80 1 -- 172.21.15.27:0/422229591 wait complete. 2024-05-10T11:32:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.452+0000 7f6e58913b80 1 -- start start 2024-05-10T11:32:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.452+0000 7f6e58913b80 1 Processor -- start 2024-05-10T11:32:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.452+0000 7f6e58913b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea8805060 0x563ea8810de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.452+0000 7f6e58913b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805980 0x563ea880e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e58913b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea88099d0 0x563ea880e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e58913b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563ea8811be0 con 0x563ea88099d0 2024-05-10T11:32:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e58913b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563ea86bdb20 con 0x563ea8805980 2024-05-10T11:32:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e58913b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563ea8811d60 con 0x563ea8805060 2024-05-10T11:32:03.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e58913b80 10 client.0 _ll_register_callbacks cb 0x563ea86fec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:32:03.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e47400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805980 0x563ea880e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e46a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea8805060 0x563ea8810de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e47400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805980 0x563ea880e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44618/0 (socket says 172.21.15.27:44618) 2024-05-10T11:32:03.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e47400700 1 -- 172.21.15.27:0/2615576830 learned_addr learned my addr 172.21.15.27:0/2615576830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:03.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e46a00700 1 --2- 172.21.15.27:0/2615576830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea88099d0 0x563ea880e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e47400700 1 -- 172.21.15.27:0/2615576830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea8805060 msgr2=0x563ea8810de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:03.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e47400700 1 --2- 172.21.15.27:0/2615576830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea8805060 0x563ea8810de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e47400700 1 -- 172.21.15.27:0/2615576830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea88099d0 msgr2=0x563ea880e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e47400700 1 --2- 172.21.15.27:0/2615576830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea88099d0 0x563ea880e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.453+0000 7f6e47400700 1 -- 172.21.15.27:0/2615576830 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ea871d420 con 0x563ea8805980 2024-05-10T11:32:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.454+0000 7f6e47400700 1 --2- 172.21.15.27:0/2615576830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805980 0x563ea880e180 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6e380099d0 tx=0x7f6e380096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86b3581635ed0e12 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.454+0000 7f6e31600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:32:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.457+0000 7f6e31600700 1 -- 172.21.15.27:0/2615576830 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e38019040 con 0x563ea8805980 2024-05-10T11:32:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.457+0000 7f6e31600700 1 -- 172.21.15.27:0/2615576830 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9679+0+0 (secure 0 0 0) 0x7f6e38004ad0 con 0x563ea8805980 2024-05-10T11:32:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.457+0000 7f6e58913b80 1 -- 172.21.15.27:0/2615576830 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e380079b0 con 0x563ea8805980 2024-05-10T11:32:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.458+0000 7f6e31600700 1 -- 172.21.15.27:0/2615576830 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f6e38004cd0 con 0x563ea8805980 2024-05-10T11:32:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.458+0000 7f6e58913b80 10 client.27725 fetch_fsmap learned FSMap version 45 2024-05-10T11:32:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.458+0000 7f6e58913b80 1 -- 172.21.15.27:0/2615576830 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x563ea881c410 con 0x563ea8805980 2024-05-10T11:32:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.458+0000 7f6e31600700 1 -- 172.21.15.27:0/2615576830 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e380079b0 con 0x563ea8805980 2024-05-10T11:32:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.458+0000 7f6e31600700 1 -- 172.21.15.27:0/2615576830 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f6e380154e0 con 0x563ea8805980 2024-05-10T11:32:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.458+0000 7f6e58913b80 10 client.27725 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:32:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.458+0000 7f6e58913b80 10 client.27725 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:32:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.458+0000 7f6e58913b80 20 client.27725 populate_metadata read hostname 'smithi027' 2024-05-10T11:32:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.458+0000 7f6e58913b80 1 --2- 172.21.15.27:0/2615576830 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ea881e9f0 0x563ea8820de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.458+0000 7f6e58913b80 4 client.27725 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:32:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.458+0000 7f6e58913b80 1 -- 172.21.15.27:0/2615576830 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ea87d20a0 con 0x563ea881e9f0 2024-05-10T11:32:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.458+0000 7f6e46a00700 1 --2- 172.21.15.27:0/2615576830 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ea881e9f0 0x563ea8820de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.459+0000 7f6e46a00700 1 --2- 172.21.15.27:0/2615576830 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ea881e9f0 0x563ea8820de0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=889b1ee35da7afee server_cookie=b562dd7785eaf24a in_seq=0 out_seq=0 2024-05-10T11:32:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.459+0000 7f6e31600700 10 client.27725 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:32:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e31600700 1 -- 172.21.15.27:0/2615576830 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563ea87d20a0 con 0x563ea881e9f0 2024-05-10T11:32:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e31600700 10 client.27725 handle_command_reply: tid=0 2024-05-10T11:32:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e58913b80 1 client.27725 shutdown 2024-05-10T11:32:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e58913b80 1 -- 172.21.15.27:0/2615576830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805980 msgr2=0x563ea880e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e58913b80 1 --2- 172.21.15.27:0/2615576830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805980 0x563ea880e180 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6e380099d0 tx=0x7f6e380096a0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e58913b80 1 -- 172.21.15.27:0/2615576830 shutdown_connections 2024-05-10T11:32:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e58913b80 1 --2- 172.21.15.27:0/2615576830 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ea881e9f0 0x563ea8820de0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e58913b80 1 --2- 172.21.15.27:0/2615576830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea8805060 0x563ea8810de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e58913b80 1 --2- 172.21.15.27:0/2615576830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea88099d0 0x563ea880e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e58913b80 1 --2- 172.21.15.27:0/2615576830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805980 0x563ea880e180 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e31600700 0 client.27725 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:32:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e58913b80 1 -- 172.21.15.27:0/2615576830 >> 172.21.15.27:0/2615576830 conn(0x563ea8733b60 msgr2=0x563ea87fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.460+0000 7f6e58913b80 1 -- 172.21.15.27:0/2615576830 shutdown_connections 2024-05-10T11:32:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.461+0000 7f6e58913b80 1 -- 172.21.15.27:0/2615576830 wait complete. 2024-05-10T11:32:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.461+0000 7f6e58913b80 20 client.27725 trim_cache size 0 max 16384 2024-05-10T11:32:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.477+0000 7f6e58913b80 1 Processor -- start 2024-05-10T11:32:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.477+0000 7f6e58913b80 1 -- start start 2024-05-10T11:32:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.477+0000 7f6e58913b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805060 0x563ea881aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.477+0000 7f6e58913b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805980 0x563ea880d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.477+0000 7f6e58913b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea88099d0 0x563ea8808aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.477+0000 7f6e58913b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e481ad6b0 con 0x563ea8805060 2024-05-10T11:32:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.477+0000 7f6e58913b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6e481af1f0 con 0x563ea8805980 2024-05-10T11:32:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.477+0000 7f6e58913b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6e481add80 con 0x563ea88099d0 2024-05-10T11:32:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.477+0000 7f6e46a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805060 0x563ea881aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.477+0000 7f6e46a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805060 0x563ea881aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59210/0 (socket says 172.21.15.27:59210) 2024-05-10T11:32:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.477+0000 7f6e46a00700 1 -- 172.21.15.27:0/147696188 learned_addr learned my addr 172.21.15.27:0/147696188 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:03.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.477+0000 7f6e47400700 1 --2- 172.21.15.27:0/147696188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805980 0x563ea880d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.478+0000 7f6e46a00700 1 --2- 172.21.15.27:0/147696188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea88099d0 0x563ea8808aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.478+0000 7f6e46a00700 1 -- 172.21.15.27:0/147696188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea88099d0 msgr2=0x563ea8808aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:03.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.478+0000 7f6e46a00700 1 --2- 172.21.15.27:0/147696188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea88099d0 0x563ea8808aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.478+0000 7f6e46a00700 1 -- 172.21.15.27:0/147696188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805980 msgr2=0x563ea880d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:03.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.478+0000 7f6e46a00700 1 --2- 172.21.15.27:0/147696188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805980 0x563ea880d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.478+0000 7f6e46a00700 1 -- 172.21.15.27:0/147696188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ea8810a20 con 0x563ea8805060 2024-05-10T11:32:03.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.478+0000 7f6e46a00700 1 --2- 172.21.15.27:0/147696188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805060 0x563ea881aca0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009750 tx=0x7f6e34009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd419fa7e9c69c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:03.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.478+0000 7f6e33e00700 1 -- 172.21.15.27:0/147696188 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e34005690 con 0x563ea8805060 2024-05-10T11:32:03.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.478+0000 7f6e33e00700 1 -- 172.21.15.27:0/147696188 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e3402e020 con 0x563ea8805060 2024-05-10T11:32:03.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.478+0000 7f6e33e00700 1 -- 172.21.15.27:0/147696188 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3400be20 con 0x563ea8805060 2024-05-10T11:32:03.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.479+0000 7f6e58913b80 1 -- 172.21.15.27:0/147696188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805060 msgr2=0x563ea881aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.479+0000 7f6e58913b80 1 --2- 172.21.15.27:0/147696188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805060 0x563ea881aca0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009750 tx=0x7f6e34009310 comp rx=0 tx=0).stop 2024-05-10T11:32:03.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.479+0000 7f6e58913b80 1 -- 172.21.15.27:0/147696188 shutdown_connections 2024-05-10T11:32:03.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.479+0000 7f6e58913b80 1 --2- 172.21.15.27:0/147696188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea88099d0 0x563ea8808aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.479+0000 7f6e58913b80 1 --2- 172.21.15.27:0/147696188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805060 0x563ea881aca0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.479+0000 7f6e58913b80 1 --2- 172.21.15.27:0/147696188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805980 0x563ea880d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.479+0000 7f6e58913b80 1 -- 172.21.15.27:0/147696188 >> 172.21.15.27:0/147696188 conn(0x563ea8733b60 msgr2=0x563ea8808390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:03.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.479+0000 7f6e58913b80 1 -- 172.21.15.27:0/147696188 shutdown_connections 2024-05-10T11:32:03.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.479+0000 7f6e58913b80 1 -- 172.21.15.27:0/147696188 wait complete. 2024-05-10T11:32:03.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.480+0000 7f6e58913b80 1 -- start start 2024-05-10T11:32:03.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.480+0000 7f6e58913b80 1 Processor -- start 2024-05-10T11:32:03.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e58913b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805060 0x563ea88172d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e58913b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805980 0x563ea8814670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e58913b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea88099d0 0x563ea8814bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e58913b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e481aeb20 con 0x563ea8805980 2024-05-10T11:32:03.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e58913b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6e481ae450 con 0x563ea8805060 2024-05-10T11:32:03.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e58913b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6e481ab670 con 0x563ea88099d0 2024-05-10T11:32:03.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e58913b80 10 client.0 _ll_register_callbacks cb 0x563ea86fec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:32:03.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e47400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805980 0x563ea8814670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e46a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805060 0x563ea88172d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e46a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805060 0x563ea88172d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:44646/0 (socket says 172.21.15.27:44646) 2024-05-10T11:32:03.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e46a00700 1 -- 172.21.15.27:0/2024414787 learned_addr learned my addr 172.21.15.27:0/2024414787 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:03.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e47400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805980 0x563ea8814670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59220/0 (socket says 172.21.15.27:59220) 2024-05-10T11:32:03.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e46a00700 1 --2- 172.21.15.27:0/2024414787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea88099d0 0x563ea8814bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e47400700 1 -- 172.21.15.27:0/2024414787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea88099d0 msgr2=0x563ea8814bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:03.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e47400700 1 --2- 172.21.15.27:0/2024414787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea88099d0 0x563ea8814bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e47400700 1 -- 172.21.15.27:0/2024414787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805060 msgr2=0x563ea88172d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:03.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e47400700 1 --2- 172.21.15.27:0/2024414787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805060 0x563ea88172d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.481+0000 7f6e47400700 1 -- 172.21.15.27:0/2024414787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ea8815170 con 0x563ea8805980 2024-05-10T11:32:03.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.482+0000 7f6e47400700 1 --2- 172.21.15.27:0/2024414787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805980 0x563ea8814670 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6e38009ee0 tx=0x7f6e38004c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1f9c1027ff97a18 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:03.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.482+0000 7f6e31600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:32:03.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.482+0000 7f6e31600700 1 -- 172.21.15.27:0/2024414787 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e38019040 con 0x563ea8805980 2024-05-10T11:32:03.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.482+0000 7f6e31600700 1 -- 172.21.15.27:0/2024414787 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9679+0+0 (secure 0 0 0) 0x7f6e38005c60 con 0x563ea8805980 2024-05-10T11:32:03.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.482+0000 7f6e58913b80 1 -- 172.21.15.27:0/2024414787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e3402e020 con 0x563ea8805980 2024-05-10T11:32:03.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.482+0000 7f6e31600700 1 -- 172.21.15.27:0/2024414787 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f6e38005ea0 con 0x563ea8805980 2024-05-10T11:32:03.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.482+0000 7f6e58913b80 10 client.17379 fetch_fsmap learned FSMap version 45 2024-05-10T11:32:03.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.482+0000 7f6e58913b80 1 -- 172.21.15.27:0/2024414787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x563ea8888ea0 con 0x563ea8805980 2024-05-10T11:32:03.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.483+0000 7f6e31600700 1 -- 172.21.15.27:0/2024414787 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e3402e020 con 0x563ea8805980 2024-05-10T11:32:03.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.483+0000 7f6e31600700 1 -- 172.21.15.27:0/2024414787 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7f6e38008070 con 0x563ea8805980 2024-05-10T11:32:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.483+0000 7f6e58913b80 10 client.17379 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:32:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.483+0000 7f6e58913b80 10 client.17379 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-10T11:32:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.483+0000 7f6e58913b80 20 client.17379 populate_metadata read hostname 'smithi027' 2024-05-10T11:32:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.483+0000 7f6e58913b80 1 --2- 172.21.15.27:0/2024414787 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ea8889300 0x563ea888b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.483+0000 7f6e58913b80 4 client.17379 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:32:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.483+0000 7f6e58913b80 1 -- 172.21.15.27:0/2024414787 --> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563ea8765400 con 0x563ea8889300 2024-05-10T11:32:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.483+0000 7f6e46a00700 1 --2- 172.21.15.27:0/2024414787 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ea8889300 0x563ea888b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e46a00700 1 --2- 172.21.15.27:0/2024414787 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ea8889300 0x563ea888b6f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=397f94bce7ff7a9 server_cookie=99f3a6112e6b5f05 in_seq=0 out_seq=0 2024-05-10T11:32:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e31600700 10 client.17379 ms_handle_connect on v2:172.21.15.189:6838/2071082194 2024-05-10T11:32:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e31600700 1 -- 172.21.15.27:0/2024414787 <== mds.2 v2:172.21.15.189:6838/2071082194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563ea8765400 con 0x563ea8889300 2024-05-10T11:32:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e31600700 10 client.17379 handle_command_reply: tid=0 2024-05-10T11:32:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e58913b80 1 client.17379 shutdown 2024-05-10T11:32:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e58913b80 1 -- 172.21.15.27:0/2024414787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805980 msgr2=0x563ea8814670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e58913b80 1 --2- 172.21.15.27:0/2024414787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805980 0x563ea8814670 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6e38009ee0 tx=0x7f6e38004c40 comp rx=0 tx=0).stop 2024-05-10T11:32:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e58913b80 1 -- 172.21.15.27:0/2024414787 shutdown_connections 2024-05-10T11:32:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e58913b80 1 --2- 172.21.15.27:0/2024414787 >> [v2:172.21.15.189:6838/2071082194,v1:172.21.15.189:6839/2071082194] conn(0x563ea8889300 0x563ea888b6f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e58913b80 1 --2- 172.21.15.27:0/2024414787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ea88099d0 0x563ea8814bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e58913b80 1 --2- 172.21.15.27:0/2024414787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ea8805060 0x563ea88172d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e58913b80 1 --2- 172.21.15.27:0/2024414787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ea8805980 0x563ea8814670 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e31600700 0 client.17379 ms_handle_reset on v2:172.21.15.189:6838/2071082194 2024-05-10T11:32:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.484+0000 7f6e58913b80 1 -- 172.21.15.27:0/2024414787 >> 172.21.15.27:0/2024414787 conn(0x563ea8733b60 msgr2=0x563ea87f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.485+0000 7f6e58913b80 1 -- 172.21.15.27:0/2024414787 shutdown_connections 2024-05-10T11:32:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.485+0000 7f6e58913b80 1 -- 172.21.15.27:0/2024414787 wait complete. 2024-05-10T11:32:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.485+0000 7f6e58913b80 20 client.17379 trim_cache size 0 max 16384 2024-05-10T11:32:03.493 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:32:03.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.485+0000 7f6e4e000700 1 -- 172.21.15.27:0/257543743 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6e2c0b2a70 msgr2=0x7f6e2c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.485+0000 7f6e4e000700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6e2c0b2a70 0x7f6e2c0b4f30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6e400098e0 tx=0x7f6e40009870 comp rx=0 tx=0).stop 2024-05-10T11:32:03.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.485+0000 7f6e4e000700 1 -- 172.21.15.27:0/257543743 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e48108850 msgr2=0x7f6e481a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.485+0000 7f6e4e000700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e48108850 0x7f6e481a05e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000e940 tx=0x7f6e4000ec50 comp rx=0 tx=0).stop 2024-05-10T11:32:03.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.486+0000 7f6e4e000700 1 -- 172.21.15.27:0/257543743 shutdown_connections 2024-05-10T11:32:03.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.486+0000 7f6e4e000700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f6e2c0b2a70 0x7f6e2c0b4f30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.486+0000 7f6e4e000700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e48108850 0x7f6e481a05e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.486+0000 7f6e4e000700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e48103750 0x7f6e4819bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.486+0000 7f6e4e000700 1 --2- 172.21.15.27:0/257543743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e48104100 0x7f6e4819c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.486+0000 7f6e4e000700 1 -- 172.21.15.27:0/257543743 >> 172.21.15.27:0/257543743 conn(0x7f6e4807a600 msgr2=0x7f6e48101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:03.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.486+0000 7f6e4e000700 1 -- 172.21.15.27:0/257543743 shutdown_connections 2024-05-10T11:32:03.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.486+0000 7f6e4e000700 1 -- 172.21.15.27:0/257543743 wait complete. 2024-05-10T11:32:03.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.589+0000 7f0615e00640 1 -- 172.21.15.27:0/3236744513 --> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f06101057b0 con 0x7f05f40b2a10 2024-05-10T11:32:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.603+0000 7f0614a00640 1 -- 172.21.15.27:0/3236744513 <== mgr.25220 v2:172.21.15.27:6800/1167199542 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f06101057b0 con 0x7f05f40b2a10 2024-05-10T11:32:03.604 INFO:teuthology.orchestra.run.smithi027.stdout:/ 1m 6m3h 2024-05-10T11:32:03.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.606+0000 7f0615e00640 1 -- 172.21.15.27:0/3236744513 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f05f40b2a10 msgr2=0x7f05f40b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.606+0000 7f0615e00640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f05f40b2a10 0x7f05f40b4ed0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0600031440 tx=0x7f0600005b50 comp rx=0 tx=0).stop 2024-05-10T11:32:03.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.606+0000 7f0615e00640 1 -- 172.21.15.27:0/3236744513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f061019fbc0 msgr2=0x7f06101a0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:03.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.606+0000 7f0615e00640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f061019fbc0 0x7f06101a0070 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0608009fd0 tx=0x7f060800ef10 comp rx=0 tx=0).stop 2024-05-10T11:32:03.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.606+0000 7f0615e00640 1 -- 172.21.15.27:0/3236744513 shutdown_connections 2024-05-10T11:32:03.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.606+0000 7f0615e00640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f05f40b2a10 0x7f05f40b4ed0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.606+0000 7f0615e00640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f061019fbc0 0x7f06101a0070 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.606+0000 7f0615e00640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0610109ed0 0x7f061019b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.606+0000 7f0615e00640 1 --2- 172.21.15.27:0/3236744513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0610104e00 0x7f061019b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.606+0000 7f0615e00640 1 -- 172.21.15.27:0/3236744513 >> 172.21.15.27:0/3236744513 conn(0x7f06101006a0 msgr2=0x7f0610102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.606+0000 7f0615e00640 1 -- 172.21.15.27:0/3236744513 shutdown_connections 2024-05-10T11:32:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:03.606+0000 7f0615e00640 1 -- 172.21.15.27:0/3236744513 wait complete. 2024-05-10T11:32:03.624 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-10T11:32:03.635 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-10T11:32:03.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:03 smithi189 ceph-mon[34842]: pgmap v591: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.9 MiB/s wr, 392 op/s 2024-05-10T11:32:03.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:03 smithi189 ceph-mon[34842]: osdmap e116: 12 total, 12 up, 12 in 2024-05-10T11:32:03.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:03 smithi027 ceph-mon[30054]: pgmap v591: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.9 MiB/s wr, 392 op/s 2024-05-10T11:32:03.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:03 smithi027 ceph-mon[30054]: osdmap e116: 12 total, 12 up, 12 in 2024-05-10T11:32:03.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:03 smithi076 ceph-mon[34152]: pgmap v591: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.9 MiB/s wr, 392 op/s 2024-05-10T11:32:03.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:03 smithi076 ceph-mon[34152]: osdmap e116: 12 total, 12 up, 12 in 2024-05-10T11:32:04.387 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-10T11:32:04.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:04 smithi027 ceph-mon[30054]: from='client.27727 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:32:04.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:04 smithi189 ceph-mon[34842]: from='client.27727 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:32:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:32:04.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:04 smithi076 ceph-mon[34152]: from='client.27727 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:32:05.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:05 smithi189 ceph-mon[34842]: from='client.27713 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:32:05.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:05 smithi189 ceph-mon[34842]: pgmap v593: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 8.5 MiB/s wr, 418 op/s 2024-05-10T11:32:05.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:05 smithi027 ceph-mon[30054]: from='client.27713 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:32:05.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:05 smithi027 ceph-mon[30054]: pgmap v593: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 8.5 MiB/s wr, 418 op/s 2024-05-10T11:32:05.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:05 smithi076 ceph-mon[34152]: from='client.27713 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-10T11:32:05.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:05 smithi076 ceph-mon[34152]: pgmap v593: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 8.5 MiB/s wr, 418 op/s 2024-05-10T11:32:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.447+0000 7fe1e9c00700 1 -- 172.21.15.27:0/1867652851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e4103750 msgr2=0x7fe1e4103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.447+0000 7fe1e9c00700 1 --2- 172.21.15.27:0/1867652851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e4103750 0x7fe1e4103b30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4009990 tx=0x7fe1d4009ca0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.447+0000 7fe1e9c00700 1 -- 172.21.15.27:0/1867652851 shutdown_connections 2024-05-10T11:32:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.447+0000 7fe1e9c00700 1 --2- 172.21.15.27:0/1867652851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e4103750 0x7fe1e4103b30 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.447+0000 7fe1e9c00700 1 --2- 172.21.15.27:0/1867652851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1e4108850 0x7fe1e41103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.447+0000 7fe1e9c00700 1 --2- 172.21.15.27:0/1867652851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1e4104100 0x7fe1e4108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.447+0000 7fe1e9c00700 1 -- 172.21.15.27:0/1867652851 >> 172.21.15.27:0/1867652851 conn(0x7fe1e407a600 msgr2=0x7fe1e407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.447+0000 7fe1e9c00700 1 -- 172.21.15.27:0/1867652851 shutdown_connections 2024-05-10T11:32:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.447+0000 7fe1e9c00700 1 -- 172.21.15.27:0/1867652851 wait complete. 2024-05-10T11:32:07.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.448+0000 7fe1e9c00700 1 Processor -- start 2024-05-10T11:32:07.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.449+0000 7fe1e9c00700 1 -- start start 2024-05-10T11:32:07.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.449+0000 7fe1e9c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1e4103750 0x7fe1e419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.449+0000 7fe1e9c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e4104100 0x7fe1e419fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.449+0000 7fe1e9c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1e4108850 0x7fe1e4199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.449+0000 7fe1e9c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e4113620 con 0x7fe1e4103750 2024-05-10T11:32:07.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.449+0000 7fe1e9c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e41134a0 con 0x7fe1e4108850 2024-05-10T11:32:07.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.449+0000 7fe1e9c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e41137a0 con 0x7fe1e4104100 2024-05-10T11:32:07.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.449+0000 7fe1e2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1e4103750 0x7fe1e419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.450+0000 7fe1e2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1e4103750 0x7fe1e419f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43424/0 (socket says 172.21.15.27:43424) 2024-05-10T11:32:07.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.450+0000 7fe1e2a00700 1 -- 172.21.15.27:0/3316915911 learned_addr learned my addr 172.21.15.27:0/3316915911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:07.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.450+0000 7fe1e3400700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1e4108850 0x7fe1e4199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.450+0000 7fe1dbe00700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e4104100 0x7fe1e419fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.450+0000 7fe1e2a00700 1 -- 172.21.15.27:0/3316915911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e4104100 msgr2=0x7fe1e419fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.450+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e4104100 0x7fe1e419fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.450+0000 7fe1e2a00700 1 -- 172.21.15.27:0/3316915911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1e4108850 msgr2=0x7fe1e4199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.450+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1e4108850 0x7fe1e4199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.450+0000 7fe1e2a00700 1 -- 172.21.15.27:0/3316915911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e419a080 con 0x7fe1e4103750 2024-05-10T11:32:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.450+0000 7fe1e3400700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1e4108850 0x7fe1e4199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:32:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.451+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1e4103750 0x7fe1e419f710 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40098e0 tx=0x7fe1d40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:07.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.451+0000 7fe1db400700 1 -- 172.21.15.27:0/3316915911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4027070 con 0x7fe1e4103750 2024-05-10T11:32:07.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.451+0000 7fe1e9c00700 1 -- 172.21.15.27:0/3316915911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e419a310 con 0x7fe1e4103750 2024-05-10T11:32:07.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.451+0000 7fe1db400700 1 -- 172.21.15.27:0/3316915911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1d400b610 con 0x7fe1e4103750 2024-05-10T11:32:07.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.451+0000 7fe1db400700 1 -- 172.21.15.27:0/3316915911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40173f0 con 0x7fe1e4103750 2024-05-10T11:32:07.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.451+0000 7fe1e9c00700 1 -- 172.21.15.27:0/3316915911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e41a65e0 con 0x7fe1e4103750 2024-05-10T11:32:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.457+0000 7fe1db400700 1 -- 172.21.15.27:0/3316915911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fe1d402a070 con 0x7fe1e4103750 2024-05-10T11:32:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.457+0000 7fe1f44e9b80 1 Processor -- start 2024-05-10T11:32:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.458+0000 7fe1f44e9b80 1 -- start start 2024-05-10T11:32:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.458+0000 7fe1f44e9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a09bd0 0x563e37a09fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.458+0000 7fe1f44e9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 0x563e37a06750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.458+0000 7fe1f44e9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a06c90 0x563e37a1af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.458+0000 7fe1f44e9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563e37958a20 con 0x563e37a09bd0 2024-05-10T11:32:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.458+0000 7fe1f44e9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563e378f6760 con 0x563e37a06c90 2024-05-10T11:32:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.458+0000 7fe1f44e9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563e379549e0 con 0x563e37a062d0 2024-05-10T11:32:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1e2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 0x563e37a06750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1e2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 0x563e37a06750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41288/0 (socket says 172.21.15.27:41288) 2024-05-10T11:32:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1e2a00700 1 -- 172.21.15.27:0/2018484688 learned_addr learned my addr 172.21.15.27:0/2018484688 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1e3400700 1 --2- 172.21.15.27:0/2018484688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a09bd0 0x563e37a09fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1dbe00700 1 --2- 172.21.15.27:0/2018484688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a06c90 0x563e37a1af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1db400700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe1c80b2a70 0x7fe1c80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1e3400700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe1c80b2a70 0x7fe1c80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1e2a00700 1 -- 172.21.15.27:0/2018484688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a06c90 msgr2=0x563e37a1af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/2018484688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a06c90 0x563e37a1af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1e2a00700 1 -- 172.21.15.27:0/2018484688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a09bd0 msgr2=0x563e37a09fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/2018484688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a09bd0 0x563e37a09fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1db400700 1 -- 172.21.15.27:0/3316915911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9679+0+0 (secure 0 0 0) 0x7fe1d40e3860 con 0x7fe1e4103750 2024-05-10T11:32:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.459+0000 7fe1e2a00700 1 -- 172.21.15.27:0/2018484688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e378a58d0 con 0x563e37a062d0 2024-05-10T11:32:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.460+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/2018484688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 0x563e37a06750 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40a79c0 tx=0x7fe1d4009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26e4e6d1e0489840 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.460+0000 7fe1e3400700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe1c80b2a70 0x7fe1c80b4f30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0060b0 tx=0x7fe1dc004b40 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.460+0000 7fe1d3e00700 1 -- 172.21.15.27:0/2018484688 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40a9ec0 con 0x563e37a062d0 2024-05-10T11:32:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.460+0000 7fe1d3e00700 1 -- 172.21.15.27:0/2018484688 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1d40aa260 con 0x563e37a062d0 2024-05-10T11:32:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.460+0000 7fe1d3e00700 1 -- 172.21.15.27:0/2018484688 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4092110 con 0x563e37a062d0 2024-05-10T11:32:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.461+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/2018484688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 msgr2=0x563e37a06750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.461+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/2018484688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 0x563e37a06750 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40a79c0 tx=0x7fe1d4009310 comp rx=0 tx=0).stop 2024-05-10T11:32:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.461+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/2018484688 shutdown_connections 2024-05-10T11:32:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.461+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/2018484688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a06c90 0x563e37a1af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.461+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/2018484688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a09bd0 0x563e37a09fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.461+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/2018484688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 0x563e37a06750 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.461+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/2018484688 >> 172.21.15.27:0/2018484688 conn(0x563e3797e6d0 msgr2=0x563e37a00b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.462+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/2018484688 shutdown_connections 2024-05-10T11:32:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.462+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/2018484688 wait complete. 2024-05-10T11:32:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.462+0000 7fe1f44e9b80 1 -- start start 2024-05-10T11:32:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.462+0000 7fe1f44e9b80 1 Processor -- start 2024-05-10T11:32:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.463+0000 7fe1f44e9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a062d0 0x563e37a1ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.463+0000 7fe1f44e9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a06c90 0x563e37a17eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.463+0000 7fe1f44e9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a09bd0 0x563e37a183f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.463+0000 7fe1f44e9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563e37a1b540 con 0x563e37a062d0 2024-05-10T11:32:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.463+0000 7fe1f44e9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563e378beb20 con 0x563e37a09bd0 2024-05-10T11:32:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.463+0000 7fe1f44e9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563e37a1b6c0 con 0x563e37a06c90 2024-05-10T11:32:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.463+0000 7fe1f44e9b80 10 client.0 _ll_register_callbacks cb 0x563e378ffc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:32:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.463+0000 7fe1dbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a06c90 0x563e37a17eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.463+0000 7fe1dbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a06c90 0x563e37a17eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41298/0 (socket says 172.21.15.27:41298) 2024-05-10T11:32:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.463+0000 7fe1dbe00700 1 -- 172.21.15.27:0/3099906841 learned_addr learned my addr 172.21.15.27:0/3099906841 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.463+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a062d0 0x563e37a1ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.464+0000 7fe1e3400700 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a09bd0 0x563e37a183f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.464+0000 7fe1e2a00700 1 -- 172.21.15.27:0/3099906841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a06c90 msgr2=0x563e37a17eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.464+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a06c90 0x563e37a17eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.464+0000 7fe1e2a00700 1 -- 172.21.15.27:0/3099906841 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a09bd0 msgr2=0x563e37a183f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.464+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a09bd0 0x563e37a183f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.464+0000 7fe1e2a00700 1 -- 172.21.15.27:0/3099906841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e3791e420 con 0x563e37a062d0 2024-05-10T11:32:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.464+0000 7fe1dbe00700 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a06c90 0x563e37a17eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:32:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.464+0000 7fe1e3400700 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a09bd0 0x563e37a183f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:32:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.464+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a062d0 0x563e37a1ab10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40aa210 tx=0x7fe1d40a6db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d6bd255cddeaf4 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.464+0000 7fe1d0c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-05-10T11:32:07.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.465+0000 7fe1d0c00700 1 -- 172.21.15.27:0/3099906841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40aae50 con 0x563e37a062d0 2024-05-10T11:32:07.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.465+0000 7fe1d0c00700 1 -- 172.21.15.27:0/3099906841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9679+0+0 (secure 0 0 0) 0x7fe1d40aabc0 con 0x563e37a062d0 2024-05-10T11:32:07.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.465+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/3099906841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1d40aa260 con 0x563e37a062d0 2024-05-10T11:32:07.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.465+0000 7fe1d0c00700 1 -- 172.21.15.27:0/3099906841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe1d40aeb00 con 0x563e37a062d0 2024-05-10T11:32:07.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.465+0000 7fe1f44e9b80 10 client.17391 fetch_fsmap learned FSMap version 45 2024-05-10T11:32:07.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.465+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/3099906841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x563e37a1b540 con 0x563e37a062d0 2024-05-10T11:32:07.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.465+0000 7fe1d0c00700 1 -- 172.21.15.27:0/3099906841 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1d40aa260 con 0x563e37a062d0 2024-05-10T11:32:07.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.466+0000 7fe1d0c00700 1 -- 172.21.15.27:0/3099906841 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7fe1d40e3380 con 0x563e37a062d0 2024-05-10T11:32:07.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.466+0000 7fe1f44e9b80 10 client.17391 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:32:07.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.466+0000 7fe1f44e9b80 10 client.17391 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:32:07.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.466+0000 7fe1f44e9b80 20 client.17391 populate_metadata read hostname 'smithi027' 2024-05-10T11:32:07.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.466+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x563e37a27ef0 0x563e37a2a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.466+0000 7fe1f44e9b80 4 client.17391 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:32:07.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.466+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/3099906841 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e379d3140 con 0x563e37a27ef0 2024-05-10T11:32:07.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.466+0000 7fe1dbe00700 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x563e37a27ef0 0x563e37a2a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.467+0000 7fe1dbe00700 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x563e37a27ef0 0x563e37a2a2e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c571e9d85d9a7b74 server_cookie=f2b565d67b8ffe99 in_seq=0 out_seq=0 2024-05-10T11:32:07.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.467+0000 7fe1d0c00700 10 client.17391 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:32:07.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.468+0000 7fe1d0c00700 1 -- 172.21.15.27:0/3099906841 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563e379d3140 con 0x563e37a27ef0 2024-05-10T11:32:07.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.468+0000 7fe1d0c00700 10 client.17391 handle_command_reply: tid=0 2024-05-10T11:32:07.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.468+0000 7fe1f44e9b80 1 client.17391 shutdown 2024-05-10T11:32:07.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.468+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/3099906841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a062d0 msgr2=0x563e37a1ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:07.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.468+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a062d0 0x563e37a1ab10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40aa210 tx=0x7fe1d40a6db0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.469+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/3099906841 shutdown_connections 2024-05-10T11:32:07.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.469+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a09bd0 0x563e37a183f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.469+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a062d0 0x563e37a1ab10 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.469+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x563e37a27ef0 0x563e37a2a2e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.469+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/3099906841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a06c90 0x563e37a17eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.469+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/3099906841 >> 172.21.15.27:0/3099906841 conn(0x563e3797e6d0 msgr2=0x563e37a00190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:07.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.469+0000 7fe1d0c00700 0 client.17391 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:32:07.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.469+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/3099906841 shutdown_connections 2024-05-10T11:32:07.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.469+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/3099906841 wait complete. 2024-05-10T11:32:07.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.470+0000 7fe1f44e9b80 20 client.17391 trim_cache size 0 max 16384 2024-05-10T11:32:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.491+0000 7fe1f44e9b80 1 Processor -- start 2024-05-10T11:32:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.491+0000 7fe1f44e9b80 1 -- start start 2024-05-10T11:32:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.491+0000 7fe1f44e9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 0x563e37a171b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.491+0000 7fe1f44e9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a06c90 0x563e37a19c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.491+0000 7fe1f44e9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a09bd0 0x563e37a2fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.492+0000 7fe1f44e9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e41ad4c0 con 0x563e37a06c90 2024-05-10T11:32:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.492+0000 7fe1f44e9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e41af000 con 0x563e37a09bd0 2024-05-10T11:32:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.492+0000 7fe1f44e9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e41adb90 con 0x563e37a062d0 2024-05-10T11:32:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.492+0000 7fe1e2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 0x563e37a171b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.492+0000 7fe1e2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 0x563e37a171b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41306/0 (socket says 172.21.15.27:41306) 2024-05-10T11:32:07.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.492+0000 7fe1e2a00700 1 -- 172.21.15.27:0/1792634967 learned_addr learned my addr 172.21.15.27:0/1792634967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:07.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.493+0000 7fe1e2a00700 1 -- 172.21.15.27:0/1792634967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a09bd0 msgr2=0x563e37a2fb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:32:07.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.493+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/1792634967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a09bd0 0x563e37a2fb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.493+0000 7fe1e2a00700 1 -- 172.21.15.27:0/1792634967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a06c90 msgr2=0x563e37a19c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:07.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.493+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/1792634967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a06c90 0x563e37a19c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.493+0000 7fe1e2a00700 1 -- 172.21.15.27:0/1792634967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e379fbab0 con 0x563e37a062d0 2024-05-10T11:32:07.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.496+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/1792634967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 0x563e37a171b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40aa450 tx=0x7fe1d4022a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f743c0f62557ba86 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:07.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.496+0000 7fe1d3400700 1 -- 172.21.15.27:0/1792634967 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40aa260 con 0x563e37a062d0 2024-05-10T11:32:07.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.496+0000 7fe1d3400700 1 -- 172.21.15.27:0/1792634967 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1d40aae50 con 0x563e37a062d0 2024-05-10T11:32:07.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.496+0000 7fe1d3400700 1 -- 172.21.15.27:0/1792634967 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40b3090 con 0x563e37a062d0 2024-05-10T11:32:07.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.497+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/1792634967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 msgr2=0x563e37a171b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:07.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.497+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/1792634967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 0x563e37a171b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40aa450 tx=0x7fe1d4022a30 comp rx=0 tx=0).stop 2024-05-10T11:32:07.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.498+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/1792634967 shutdown_connections 2024-05-10T11:32:07.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.498+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/1792634967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a062d0 0x563e37a171b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.498+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/1792634967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a09bd0 0x563e37a2fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.498+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/1792634967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a06c90 0x563e37a19c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.498+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/1792634967 >> 172.21.15.27:0/1792634967 conn(0x563e3797e6d0 msgr2=0x563e37a19560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:07.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.499+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/1792634967 shutdown_connections 2024-05-10T11:32:07.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.499+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/1792634967 wait complete. 2024-05-10T11:32:07.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.500+0000 7fe1f44e9b80 1 -- start start 2024-05-10T11:32:07.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.500+0000 7fe1f44e9b80 1 Processor -- start 2024-05-10T11:32:07.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.503+0000 7fe1f44e9b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a062d0 0x563e37a207f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.503+0000 7fe1f44e9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a06c90 0x563e37a20d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.503+0000 7fe1f44e9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a09bd0 0x563e37a1d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.503+0000 7fe1f44e9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e41ae930 con 0x563e37a06c90 2024-05-10T11:32:07.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.503+0000 7fe1f44e9b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e41ae260 con 0x563e37a062d0 2024-05-10T11:32:07.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.503+0000 7fe1f44e9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e41ab480 con 0x563e37a09bd0 2024-05-10T11:32:07.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.503+0000 7fe1f44e9b80 10 client.0 _ll_register_callbacks cb 0x563e378ffc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:32:07.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.503+0000 7fe1e2a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a062d0 0x563e37a207f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.503+0000 7fe1e2a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a062d0 0x563e37a207f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36434/0 (socket says 172.21.15.27:36434) 2024-05-10T11:32:07.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.503+0000 7fe1e2a00700 1 -- 172.21.15.27:0/610928656 learned_addr learned my addr 172.21.15.27:0/610928656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:07.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.504+0000 7fe1e2a00700 1 -- 172.21.15.27:0/610928656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a09bd0 msgr2=0x563e37a1d7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-10T11:32:07.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.505+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/610928656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a09bd0 0x563e37a1d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.505+0000 7fe1e2a00700 1 -- 172.21.15.27:0/610928656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a06c90 msgr2=0x563e37a20d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:07.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.505+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/610928656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a06c90 0x563e37a20d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.505+0000 7fe1e2a00700 1 -- 172.21.15.27:0/610928656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e37a1de30 con 0x563e37a062d0 2024-05-10T11:32:07.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.506+0000 7fe1e2a00700 1 --2- 172.21.15.27:0/610928656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a062d0 0x563e37a207f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40bfd90 tx=0x7fe1d40aa160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab516ce1304358f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:07.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.506+0000 7fe1d0c00700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:32:07.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.507+0000 7fe1d0c00700 1 -- 172.21.15.27:0/610928656 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40b32a0 con 0x563e37a062d0 2024-05-10T11:32:07.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.507+0000 7fe1d0c00700 1 -- 172.21.15.27:0/610928656 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9679+0+0 (secure 0 0 0) 0x7fe1d40a6d40 con 0x563e37a062d0 2024-05-10T11:32:07.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.508+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/610928656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1d40aae50 con 0x563e37a062d0 2024-05-10T11:32:07.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.509+0000 7fe1d0c00700 1 -- 172.21.15.27:0/610928656 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe1d4089050 con 0x563e37a062d0 2024-05-10T11:32:07.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.509+0000 7fe1f44e9b80 10 client.27746 fetch_fsmap learned FSMap version 45 2024-05-10T11:32:07.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.509+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/610928656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x563e37a1de30 con 0x563e37a062d0 2024-05-10T11:32:07.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.510+0000 7fe1d0c00700 1 -- 172.21.15.27:0/610928656 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1d40aae50 con 0x563e37a062d0 2024-05-10T11:32:07.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.511+0000 7fe1d0c00700 1 -- 172.21.15.27:0/610928656 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7fe1d40a60c0 con 0x563e37a062d0 2024-05-10T11:32:07.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.511+0000 7fe1f44e9b80 10 client.27746 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:32:07.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.511+0000 7fe1f44e9b80 10 client.27746 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-10T11:32:07.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.511+0000 7fe1f44e9b80 20 client.27746 populate_metadata read hostname 'smithi027' 2024-05-10T11:32:07.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.512+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/610928656 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x563e37a931f0 0x563e37a955e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:07.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.512+0000 7fe1f44e9b80 4 client.27746 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:32:07.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.512+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/610928656 --> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563e37a02bd0 con 0x563e37a931f0 2024-05-10T11:32:07.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.513+0000 7fe1dbe00700 1 --2- 172.21.15.27:0/610928656 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x563e37a931f0 0x563e37a955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.517+0000 7fe1d0c00700 10 client.27746 ms_handle_connect on v2:172.21.15.76:6836/569330029 2024-05-10T11:32:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.517+0000 7fe1dbe00700 1 --2- 172.21.15.27:0/610928656 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x563e37a931f0 0x563e37a955e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=48db7b355f7f4f4c server_cookie=a3db7af42d5f1bcc in_seq=0 out_seq=0 2024-05-10T11:32:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.518+0000 7fe1d0c00700 1 -- 172.21.15.27:0/610928656 <== mds.1 v2:172.21.15.76:6836/569330029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563e37a02bd0 con 0x563e37a931f0 2024-05-10T11:32:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.518+0000 7fe1d0c00700 10 client.27746 handle_command_reply: tid=0 2024-05-10T11:32:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.518+0000 7fe1f44e9b80 1 client.27746 shutdown 2024-05-10T11:32:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.519+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/610928656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a062d0 msgr2=0x563e37a207f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:07.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.519+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/610928656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a062d0 0x563e37a207f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40bfd90 tx=0x7fe1d40aa160 comp rx=0 tx=0).stop 2024-05-10T11:32:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.521+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/610928656 shutdown_connections 2024-05-10T11:32:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.521+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/610928656 >> [v2:172.21.15.76:6836/569330029,v1:172.21.15.76:6837/569330029] conn(0x563e37a931f0 0x563e37a955e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.521+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/610928656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563e37a09bd0 0x563e37a1d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.521+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/610928656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563e37a062d0 0x563e37a207f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.521+0000 7fe1f44e9b80 1 --2- 172.21.15.27:0/610928656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e37a06c90 0x563e37a20d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.521+0000 7fe1d0c00700 0 client.27746 ms_handle_reset on v2:172.21.15.76:6836/569330029 2024-05-10T11:32:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.521+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/610928656 >> 172.21.15.27:0/610928656 conn(0x563e3797e6d0 msgr2=0x563e37a19d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.521+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/610928656 shutdown_connections 2024-05-10T11:32:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.522+0000 7fe1f44e9b80 1 -- 172.21.15.27:0/610928656 wait complete. 2024-05-10T11:32:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.522+0000 7fe1f44e9b80 20 client.27746 trim_cache size 0 max 16384 2024-05-10T11:32:07.523 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:32:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.524+0000 7fe1e9c00700 1 -- 172.21.15.27:0/3316915911 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe1c80b2a70 msgr2=0x7fe1c80b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.524+0000 7fe1e9c00700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe1c80b2a70 0x7fe1c80b4f30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0060b0 tx=0x7fe1dc004b40 comp rx=0 tx=0).stop 2024-05-10T11:32:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.524+0000 7fe1e9c00700 1 -- 172.21.15.27:0/3316915911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1e4103750 msgr2=0x7fe1e419f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.524+0000 7fe1e9c00700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1e4103750 0x7fe1e419f710 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40098e0 tx=0x7fe1d40047b0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.524+0000 7fe1e9c00700 1 -- 172.21.15.27:0/3316915911 shutdown_connections 2024-05-10T11:32:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.524+0000 7fe1e9c00700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fe1c80b2a70 0x7fe1c80b4f30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.524+0000 7fe1e9c00700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1e4108850 0x7fe1e4199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.524+0000 7fe1e9c00700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1e4103750 0x7fe1e419f710 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.524+0000 7fe1e9c00700 1 --2- 172.21.15.27:0/3316915911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e4104100 0x7fe1e419fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.524+0000 7fe1e9c00700 1 -- 172.21.15.27:0/3316915911 >> 172.21.15.27:0/3316915911 conn(0x7fe1e407a600 msgr2=0x7fe1e4101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:07.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.530+0000 7fe1e9c00700 1 -- 172.21.15.27:0/3316915911 shutdown_connections 2024-05-10T11:32:07.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:07.530+0000 7fe1e9c00700 1 -- 172.21.15.27:0/3316915911 wait complete. 2024-05-10T11:32:07.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:07 smithi027 ceph-mon[30054]: pgmap v594: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 6.7 MiB/s wr, 336 op/s 2024-05-10T11:32:07.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:07 smithi189 ceph-mon[34842]: pgmap v594: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 6.7 MiB/s wr, 336 op/s 2024-05-10T11:32:07.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:07 smithi076 ceph-mon[34152]: pgmap v594: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 6.7 MiB/s wr, 336 op/s 2024-05-10T11:32:08.901 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-10T11:32:09.462 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:32:09.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:09 smithi189 ceph-mon[34842]: pgmap v595: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 6.7 MiB/s wr, 336 op/s 2024-05-10T11:32:09.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:09 smithi189 ceph-mon[34842]: osdmap e117: 12 total, 12 up, 12 in 2024-05-10T11:32:09.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:09 smithi027 ceph-mon[30054]: pgmap v595: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 6.7 MiB/s wr, 336 op/s 2024-05-10T11:32:09.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:09 smithi027 ceph-mon[30054]: osdmap e117: 12 total, 12 up, 12 in 2024-05-10T11:32:09.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:09 smithi076 ceph-mon[34152]: pgmap v595: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 6.7 MiB/s wr, 336 op/s 2024-05-10T11:32:09.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:09 smithi076 ceph-mon[34152]: osdmap e117: 12 total, 12 up, 12 in 2024-05-10T11:32:11.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:11 smithi027 ceph-mon[30054]: pgmap v597: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 216 op/s 2024-05-10T11:32:11.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:11 smithi189 ceph-mon[34842]: pgmap v597: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 216 op/s 2024-05-10T11:32:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.887+0000 7fb4c6400700 1 -- 172.21.15.27:0/3338535751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4c0109510 msgr2=0x7fb4c01098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.887+0000 7fb4c6400700 1 --2- 172.21.15.27:0/3338535751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4c0109510 0x7fb4c01098f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009990 tx=0x7fb4b0009ca0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.887+0000 7fb4c6400700 1 -- 172.21.15.27:0/3338535751 shutdown_connections 2024-05-10T11:32:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.887+0000 7fb4c6400700 1 --2- 172.21.15.27:0/3338535751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4c006bb40 0x7fb4c0110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.887+0000 7fb4c6400700 1 --2- 172.21.15.27:0/3338535751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4c0109510 0x7fb4c01098f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.887+0000 7fb4c6400700 1 --2- 172.21.15.27:0/3338535751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c006b180 0x7fb4c006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.887+0000 7fb4c6400700 1 -- 172.21.15.27:0/3338535751 >> 172.21.15.27:0/3338535751 conn(0x7fb4c0079350 msgr2=0x7fb4c0079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.887+0000 7fb4c6400700 1 -- 172.21.15.27:0/3338535751 shutdown_connections 2024-05-10T11:32:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.887+0000 7fb4c6400700 1 -- 172.21.15.27:0/3338535751 wait complete. 2024-05-10T11:32:11.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.889+0000 7fb4c6400700 1 Processor -- start 2024-05-10T11:32:11.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.889+0000 7fb4c6400700 1 -- start start 2024-05-10T11:32:11.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.890+0000 7fb4c6400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c006b180 0x7fb4c019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.890+0000 7fb4c6400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4c006bb40 0x7fb4c019fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.890+0000 7fb4c6400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4c0109510 0x7fb4c01999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.890+0000 7fb4c6400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c0113620 con 0x7fb4c0109510 2024-05-10T11:32:11.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.890+0000 7fb4c6400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c01134a0 con 0x7fb4c006bb40 2024-05-10T11:32:11.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.890+0000 7fb4c6400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c01137a0 con 0x7fb4c006b180 2024-05-10T11:32:11.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.890+0000 7fb4bf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c006b180 0x7fb4c019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.890+0000 7fb4bf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c006b180 0x7fb4c019f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41328/0 (socket says 172.21.15.27:41328) 2024-05-10T11:32:11.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.890+0000 7fb4bea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4c006bb40 0x7fb4c019fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.890+0000 7fb4bf400700 1 -- 172.21.15.27:0/4146710858 learned_addr learned my addr 172.21.15.27:0/4146710858 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:11.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.890+0000 7fb4bfe00700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4c0109510 0x7fb4c01999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.891+0000 7fb4bf400700 1 -- 172.21.15.27:0/4146710858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4c006bb40 msgr2=0x7fb4c019fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:11.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.891+0000 7fb4bf400700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4c006bb40 0x7fb4c019fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.891+0000 7fb4bf400700 1 -- 172.21.15.27:0/4146710858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4c0109510 msgr2=0x7fb4c01999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:11.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.891+0000 7fb4bf400700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4c0109510 0x7fb4c01999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.891+0000 7fb4bf400700 1 -- 172.21.15.27:0/4146710858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c019a140 con 0x7fb4c006b180 2024-05-10T11:32:11.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.891+0000 7fb4bea00700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4c006bb40 0x7fb4c019fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:32:11.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.891+0000 7fb4bfe00700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4c0109510 0x7fb4c01999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:32:11.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.891+0000 7fb4bf400700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c006b180 0x7fb4c019f820 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00057c0 tx=0x7fb4b0009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:11.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.891+0000 7fb4b7e00700 1 -- 172.21.15.27:0/4146710858 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0026070 con 0x7fb4c006b180 2024-05-10T11:32:11.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.891+0000 7fb4b7e00700 1 -- 172.21.15.27:0/4146710858 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4b0021410 con 0x7fb4c006b180 2024-05-10T11:32:11.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.892+0000 7fb4c6400700 1 -- 172.21.15.27:0/4146710858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c019a3d0 con 0x7fb4c006b180 2024-05-10T11:32:11.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.892+0000 7fb4b7e00700 1 -- 172.21.15.27:0/4146710858 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0017650 con 0x7fb4c006b180 2024-05-10T11:32:11.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.892+0000 7fb4c6400700 1 -- 172.21.15.27:0/4146710858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c01a66f0 con 0x7fb4c006b180 2024-05-10T11:32:11.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.895+0000 7fb4b7e00700 1 -- 172.21.15.27:0/4146710858 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb4b0029070 con 0x7fb4c006b180 2024-05-10T11:32:11.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.897+0000 7fb4b7e00700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb4a40b2a70 0x7fb4a40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.897+0000 7fb4b7e00700 1 -- 172.21.15.27:0/4146710858 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9591+0+0 (secure 0 0 0) 0x7fb4b00e36e0 con 0x7fb4c006b180 2024-05-10T11:32:11.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.897+0000 7fb4bea00700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb4a40b2a70 0x7fb4a40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.898+0000 7fb4bea00700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb4a40b2a70 0x7fb4a40b4f30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0079a70 tx=0x7fb4ac0073d0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.899+0000 7fb4d0dc2b80 1 Processor -- start 2024-05-10T11:32:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.899+0000 7fb4d0dc2b80 1 -- start start 2024-05-10T11:32:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.900+0000 7fb4d0dc2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1408e5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.900+0000 7fb4d0dc2b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1408e9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.900+0000 7fb4d0dc2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f1408e9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.900+0000 7fb4d0dc2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f140837a20 con 0x55f1408e5060 2024-05-10T11:32:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.900+0000 7fb4d0dc2b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f1407d5760 con 0x55f1408e5980 2024-05-10T11:32:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.900+0000 7fb4d0dc2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f1408339e0 con 0x55f1408e99d0 2024-05-10T11:32:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.900+0000 7fb4bea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1408e9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.900+0000 7fb4bea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1408e9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36464/0 (socket says 172.21.15.27:36464) 2024-05-10T11:32:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.900+0000 7fb4bea00700 1 -- 172.21.15.27:0/3843004399 learned_addr learned my addr 172.21.15.27:0/3843004399 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.900+0000 7fb4bf400700 1 --2- 172.21.15.27:0/3843004399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1408e5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.900+0000 7fb4bfe00700 1 --2- 172.21.15.27:0/3843004399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f1408e9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.901+0000 7fb4bea00700 1 -- 172.21.15.27:0/3843004399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 msgr2=0x55f1408e9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:11.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.901+0000 7fb4bea00700 1 --2- 172.21.15.27:0/3843004399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f1408e9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.901+0000 7fb4bea00700 1 -- 172.21.15.27:0/3843004399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 msgr2=0x55f1408e5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:11.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.901+0000 7fb4bea00700 1 --2- 172.21.15.27:0/3843004399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1408e5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.901+0000 7fb4bea00700 1 -- 172.21.15.27:0/3843004399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1407848d0 con 0x55f1408e5980 2024-05-10T11:32:11.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.901+0000 7fb4bea00700 1 --2- 172.21.15.27:0/3843004399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1408e9490 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac004350 tx=0x7fb4ac013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5127b4e7c70e5ce8 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:11.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.901+0000 7fb4b4c00700 1 -- 172.21.15.27:0/3843004399 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac024070 con 0x55f1408e5980 2024-05-10T11:32:11.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.901+0000 7fb4b4c00700 1 -- 172.21.15.27:0/3843004399 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4ac015070 con 0x55f1408e5980 2024-05-10T11:32:11.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.901+0000 7fb4b4c00700 1 -- 172.21.15.27:0/3843004399 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac011c30 con 0x55f1408e5980 2024-05-10T11:32:11.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.902+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/3843004399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 msgr2=0x55f1408e9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.902+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/3843004399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1408e9490 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac004350 tx=0x7fb4ac013000 comp rx=0 tx=0).stop 2024-05-10T11:32:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.902+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/3843004399 shutdown_connections 2024-05-10T11:32:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.902+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/3843004399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f1408e9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.902+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/3843004399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1408e5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.902+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/3843004399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1408e9490 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.902+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/3843004399 >> 172.21.15.27:0/3843004399 conn(0x55f140813b60 msgr2=0x55f1408df7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.903+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/3843004399 shutdown_connections 2024-05-10T11:32:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.903+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/3843004399 wait complete. 2024-05-10T11:32:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.903+0000 7fb4d0dc2b80 1 -- start start 2024-05-10T11:32:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.904+0000 7fb4d0dc2b80 1 Processor -- start 2024-05-10T11:32:11.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4d0dc2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1408f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4d0dc2b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1408ee280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4d0dc2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f1408ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4d0dc2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f1408f1be0 con 0x55f1408e5060 2024-05-10T11:32:11.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4d0dc2b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f14079db20 con 0x55f1408e5980 2024-05-10T11:32:11.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4d0dc2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f1408f1d60 con 0x55f1408e99d0 2024-05-10T11:32:11.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4d0dc2b80 10 client.0 _ll_register_callbacks cb 0x55f1407dec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:32:11.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4bea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1408ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4bea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1408ee280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36466/0 (socket says 172.21.15.27:36466) 2024-05-10T11:32:11.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4bea00700 1 -- 172.21.15.27:0/1796389532 learned_addr learned my addr 172.21.15.27:0/1796389532 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:11.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4bf400700 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1408f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4bfe00700 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f1408ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4bea00700 1 -- 172.21.15.27:0/1796389532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 msgr2=0x55f1408ee7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:11.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4bea00700 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f1408ee7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4bea00700 1 -- 172.21.15.27:0/1796389532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 msgr2=0x55f1408f0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4bea00700 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1408f0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4bea00700 1 -- 172.21.15.27:0/1796389532 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1407fd420 con 0x55f1408e5980 2024-05-10T11:32:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4bf400700 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1408f0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:32:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.905+0000 7fb4bfe00700 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f1408ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-10T11:32:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.906+0000 7fb4bea00700 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1408ee280 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac005e40 tx=0x7fb4ac012970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e55a2f517251c3c server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.906+0000 7fb4a9600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:32:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.906+0000 7fb4a9600700 1 -- 172.21.15.27:0/1796389532 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac030060 con 0x55f1408e5980 2024-05-10T11:32:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.906+0000 7fb4a9600700 1 -- 172.21.15.27:0/1796389532 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9591+0+0 (secure 0 0 0) 0x7fb4ac005230 con 0x55f1408e5980 2024-05-10T11:32:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.906+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1796389532 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4ac015070 con 0x55f1408e5980 2024-05-10T11:32:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.907+0000 7fb4a9600700 1 -- 172.21.15.27:0/1796389532 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb4ac024070 con 0x55f1408e5980 2024-05-10T11:32:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.907+0000 7fb4d0dc2b80 10 client.27758 fetch_fsmap learned FSMap version 45 2024-05-10T11:32:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.907+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1796389532 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55f1408fc510 con 0x55f1408e5980 2024-05-10T11:32:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.907+0000 7fb4a9600700 1 -- 172.21.15.27:0/1796389532 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4ac015070 con 0x55f1408e5980 2024-05-10T11:32:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.907+0000 7fb4a9600700 1 -- 172.21.15.27:0/1796389532 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7fb4ac02a070 con 0x55f1408e5980 2024-05-10T11:32:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.907+0000 7fb4d0dc2b80 10 client.27758 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:32:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.907+0000 7fb4d0dc2b80 10 client.27758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:32:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.907+0000 7fb4d0dc2b80 20 client.27758 populate_metadata read hostname 'smithi027' 2024-05-10T11:32:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.907+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f1408feaf0 0x55f140900ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.907+0000 7fb4d0dc2b80 4 client.27758 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-10T11:32:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.907+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1796389532 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1408b20a0 con 0x55f1408feaf0 2024-05-10T11:32:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.908+0000 7fb4bf400700 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f1408feaf0 0x55f140900ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.908+0000 7fb4bf400700 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f1408feaf0 0x55f140900ee0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b990c649b953faed server_cookie=f8dd5d79b6d644ad in_seq=0 out_seq=0 2024-05-10T11:32:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.908+0000 7fb4a9600700 10 client.27758 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:32:11.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.909+0000 7fb4a9600700 1 -- 172.21.15.27:0/1796389532 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f1408b20a0 con 0x55f1408feaf0 2024-05-10T11:32:11.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.909+0000 7fb4a9600700 10 client.27758 handle_command_reply: tid=0 2024-05-10T11:32:11.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.909+0000 7fb4d0dc2b80 1 client.27758 shutdown 2024-05-10T11:32:11.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.909+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1796389532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 msgr2=0x55f1408ee280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:11.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.909+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1408ee280 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac005e40 tx=0x7fb4ac012970 comp rx=0 tx=0).stop 2024-05-10T11:32:11.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.909+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1796389532 shutdown_connections 2024-05-10T11:32:11.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.909+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f1408feaf0 0x55f140900ee0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.909+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f1408ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.909+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1408f0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.909+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1796389532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1408ee280 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.909+0000 7fb4a9600700 0 client.27758 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:32:11.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.909+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1796389532 >> 172.21.15.27:0/1796389532 conn(0x55f140813b60 msgr2=0x55f1408df760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:11.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.910+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1796389532 shutdown_connections 2024-05-10T11:32:11.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.910+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1796389532 wait complete. 2024-05-10T11:32:11.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.910+0000 7fb4d0dc2b80 20 client.27758 trim_cache size 0 max 16384 2024-05-10T11:32:11.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:11 smithi076 ceph-mon[34152]: pgmap v597: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 216 op/s 2024-05-10T11:32:11.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.927+0000 7fb4d0dc2b80 1 Processor -- start 2024-05-10T11:32:11.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.927+0000 7fb4d0dc2b80 1 -- start start 2024-05-10T11:32:11.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.927+0000 7fb4d0dc2b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5060 0x55f1408ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.927+0000 7fb4d0dc2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e5980 0x55f1408da130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.927+0000 7fb4d0dc2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e99d0 0x55f1408e3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.927+0000 7fb4d0dc2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c01ad400 con 0x55f1408e99d0 2024-05-10T11:32:11.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.927+0000 7fb4d0dc2b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c01aef40 con 0x55f1408e5060 2024-05-10T11:32:11.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.927+0000 7fb4d0dc2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c01adad0 con 0x55f1408e5980 2024-05-10T11:32:11.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.928+0000 7fb4bea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e5980 0x55f1408da130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.928+0000 7fb4bfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e99d0 0x55f1408e3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.928+0000 7fb4bea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e5980 0x55f1408da130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:41348/0 (socket says 172.21.15.27:41348) 2024-05-10T11:32:11.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.928+0000 7fb4bea00700 1 -- 172.21.15.27:0/1295865317 learned_addr learned my addr 172.21.15.27:0/1295865317 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:11.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.928+0000 7fb4bf400700 1 --2- 172.21.15.27:0/1295865317 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5060 0x55f1408ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.928+0000 7fb4bea00700 1 -- 172.21.15.27:0/1295865317 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5060 msgr2=0x55f1408ebee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:11.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.928+0000 7fb4bea00700 1 --2- 172.21.15.27:0/1295865317 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5060 0x55f1408ebee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.928+0000 7fb4bea00700 1 -- 172.21.15.27:0/1295865317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e99d0 msgr2=0x55f1408e3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.928+0000 7fb4bea00700 1 --2- 172.21.15.27:0/1295865317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e99d0 0x55f1408e3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.928+0000 7fb4bea00700 1 -- 172.21.15.27:0/1295865317 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1408e09c0 con 0x55f1408e5980 2024-05-10T11:32:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.929+0000 7fb4bea00700 1 --2- 172.21.15.27:0/1295865317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e5980 0x55f1408da130 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac005e90 tx=0x7fb4ac005f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a58272239835abe server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.929+0000 7fb4abe00700 1 -- 172.21.15.27:0/1295865317 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac030070 con 0x55f1408e5980 2024-05-10T11:32:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.929+0000 7fb4abe00700 1 -- 172.21.15.27:0/1295865317 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4ac015040 con 0x55f1408e5980 2024-05-10T11:32:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.929+0000 7fb4abe00700 1 -- 172.21.15.27:0/1295865317 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac037030 con 0x55f1408e5980 2024-05-10T11:32:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.930+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1295865317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e5980 msgr2=0x55f1408da130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.930+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1295865317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e5980 0x55f1408da130 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac005e90 tx=0x7fb4ac005f60 comp rx=0 tx=0).stop 2024-05-10T11:32:11.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.930+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1295865317 shutdown_connections 2024-05-10T11:32:11.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.930+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1295865317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e99d0 0x55f1408e3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.930+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1295865317 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5060 0x55f1408ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.930+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1295865317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e5980 0x55f1408da130 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.930+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1295865317 >> 172.21.15.27:0/1295865317 conn(0x55f140813b60 msgr2=0x55f1408d9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:11.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.930+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1295865317 shutdown_connections 2024-05-10T11:32:11.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.930+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1295865317 wait complete. 2024-05-10T11:32:11.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.931+0000 7fb4d0dc2b80 1 -- start start 2024-05-10T11:32:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.931+0000 7fb4d0dc2b80 1 Processor -- start 2024-05-10T11:32:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.931+0000 7fb4d0dc2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1409041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.931+0000 7fb4d0dc2b80 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1409015a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.931+0000 7fb4d0dc2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f140901ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.931+0000 7fb4d0dc2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c01ae870 con 0x55f1408e5060 2024-05-10T11:32:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.931+0000 7fb4d0dc2b80 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c01ae1a0 con 0x55f1408e5980 2024-05-10T11:32:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.931+0000 7fb4d0dc2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c01ab3c0 con 0x55f1408e99d0 2024-05-10T11:32:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4d0dc2b80 10 client.0 _ll_register_callbacks cb 0x55f1407dec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-10T11:32:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4bf400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1409041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4bea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1409015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4bf400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1409041d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43514/0 (socket says 172.21.15.27:43514) 2024-05-10T11:32:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4bf400700 1 -- 172.21.15.27:0/1754710284 learned_addr learned my addr 172.21.15.27:0/1754710284 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4bfe00700 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f140901ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4bea00700 1 -- 172.21.15.27:0/1754710284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 msgr2=0x55f140901ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4bea00700 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f140901ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4bea00700 1 -- 172.21.15.27:0/1754710284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 msgr2=0x55f1409041d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-10T11:32:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4bea00700 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1409041d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4bea00700 1 -- 172.21.15.27:0/1754710284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1409020a0 con 0x55f1408e5980 2024-05-10T11:32:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4bf400700 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1409041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:32:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.932+0000 7fb4bfe00700 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f140901ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:32:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.933+0000 7fb4bea00700 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1409015a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x55f1408e0800 tx=0x7fb4ac005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22d6c89ffac08549 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.933+0000 7fb4a9600700 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-05-10T11:32:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.933+0000 7fb4a9600700 1 -- 172.21.15.27:0/1754710284 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac030070 con 0x55f1408e5980 2024-05-10T11:32:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.933+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1754710284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4ac015040 con 0x55f1408e5980 2024-05-10T11:32:11.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.933+0000 7fb4a9600700 1 -- 172.21.15.27:0/1754710284 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9591+0+0 (secure 0 0 0) 0x7fb4ac012020 con 0x55f1408e5980 2024-05-10T11:32:11.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.933+0000 7fb4a9600700 1 -- 172.21.15.27:0/1754710284 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb4ac04b050 con 0x55f1408e5980 2024-05-10T11:32:11.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.933+0000 7fb4d0dc2b80 10 client.27764 fetch_fsmap learned FSMap version 45 2024-05-10T11:32:11.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.933+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1754710284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55f140968dc0 con 0x55f1408e5980 2024-05-10T11:32:11.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.934+0000 7fb4a9600700 1 -- 172.21.15.27:0/1754710284 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4ac015040 con 0x55f1408e5980 2024-05-10T11:32:11.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.934+0000 7fb4a9600700 1 -- 172.21.15.27:0/1754710284 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 45) ==== 8362+0+0 (secure 0 0 0) 0x7fb4ac02a070 con 0x55f1408e5980 2024-05-10T11:32:11.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.934+0000 7fb4d0dc2b80 10 client.27764 fetch_fsmap finished waiting for FSMap version 45 2024-05-10T11:32:11.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.934+0000 7fb4d0dc2b80 10 client.27764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-10T11:32:11.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.934+0000 7fb4d0dc2b80 20 client.27764 populate_metadata read hostname 'smithi027' 2024-05-10T11:32:11.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.934+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f1409693a0 0x55f14096b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:11.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.934+0000 7fb4d0dc2b80 4 client.27764 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-10T11:32:11.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.934+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1754710284 --> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f140845400 con 0x55f1409693a0 2024-05-10T11:32:11.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.934+0000 7fb4bf400700 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f1409693a0 0x55f14096b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:11.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4bf400700 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f1409693a0 0x55f14096b790 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63923c519d24d270 server_cookie=d0a58db9076408fc in_seq=0 out_seq=0 2024-05-10T11:32:11.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4a9600700 10 client.27764 ms_handle_connect on v2:172.21.15.76:6838/1778204420 2024-05-10T11:32:11.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4a9600700 1 -- 172.21.15.27:0/1754710284 <== mds.0 v2:172.21.15.76:6838/1778204420 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f140845400 con 0x55f1409693a0 2024-05-10T11:32:11.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4a9600700 10 client.27764 handle_command_reply: tid=0 2024-05-10T11:32:11.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4d0dc2b80 1 client.27764 shutdown 2024-05-10T11:32:11.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1754710284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 msgr2=0x55f1409015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:11.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1409015a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x55f1408e0800 tx=0x7fb4ac005270 comp rx=0 tx=0).stop 2024-05-10T11:32:11.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1754710284 shutdown_connections 2024-05-10T11:32:11.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.76:6838/1778204420,v1:172.21.15.76:6839/1778204420] conn(0x55f1409693a0 0x55f14096b790 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1408e99d0 0x55f140901ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1408e5060 0x55f1409041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4d0dc2b80 1 --2- 172.21.15.27:0/1754710284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f1408e5980 0x55f1409015a0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4a9600700 0 client.27764 ms_handle_reset on v2:172.21.15.76:6838/1778204420 2024-05-10T11:32:11.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.935+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1754710284 >> 172.21.15.27:0/1754710284 conn(0x55f140813b60 msgr2=0x55f1408da2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:11.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.936+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1754710284 shutdown_connections 2024-05-10T11:32:11.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.936+0000 7fb4d0dc2b80 1 -- 172.21.15.27:0/1754710284 wait complete. 2024-05-10T11:32:11.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.936+0000 7fb4d0dc2b80 20 client.27764 trim_cache size 0 max 16384 2024-05-10T11:32:11.942 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-05-10T11:32:11.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.936+0000 7fb4c6400700 1 -- 172.21.15.27:0/4146710858 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb4a40b2a70 msgr2=0x7fb4a40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:11.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.936+0000 7fb4c6400700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb4a40b2a70 0x7fb4a40b4f30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0079a70 tx=0x7fb4ac0073d0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.936+0000 7fb4c6400700 1 -- 172.21.15.27:0/4146710858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c006b180 msgr2=0x7fb4c019f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:11.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.936+0000 7fb4c6400700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c006b180 0x7fb4c019f820 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00057c0 tx=0x7fb4b0009580 comp rx=0 tx=0).stop 2024-05-10T11:32:11.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.937+0000 7fb4c6400700 1 -- 172.21.15.27:0/4146710858 shutdown_connections 2024-05-10T11:32:11.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.937+0000 7fb4c6400700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fb4a40b2a70 0x7fb4a40b4f30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.937+0000 7fb4c6400700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c006b180 0x7fb4c019f820 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.937+0000 7fb4c6400700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4c0109510 0x7fb4c01999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.937+0000 7fb4c6400700 1 --2- 172.21.15.27:0/4146710858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb4c006bb40 0x7fb4c019fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:11.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.937+0000 7fb4c6400700 1 -- 172.21.15.27:0/4146710858 >> 172.21.15.27:0/4146710858 conn(0x7fb4c0079350 msgr2=0x7fb4c0101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:11.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.937+0000 7fb4c6400700 1 -- 172.21.15.27:0/4146710858 shutdown_connections 2024-05-10T11:32:11.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:11.937+0000 7fb4c6400700 1 -- 172.21.15.27:0/4146710858 wait complete. 2024-05-10T11:32:12.736 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:12 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:32:12.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:12 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:12.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:12 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:12.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:12 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:12.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:12 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:12.865 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:12 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:32:12.865 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:12 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:12.865 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:12 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:12.865 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:12 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:12.865 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:12 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:12.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:12 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:32:12.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:12 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:12.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:12 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:12.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:12 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:12.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:12 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:13.415 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d39fa08867836ac070c47cb66c782fa559fd480b/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_d39fa08867836ac070c47cb66c782fa559fd480b/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph-c_d39fa08867836ac070c47cb66c782fa559fd480b/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-05-10T11:32:13.420 INFO:tasks.fwd_scrub:done joining 2024-05-10T11:32:13.420 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-10T11:32:13.431 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-10T11:32:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.651+0000 7f37ae200640 1 -- 172.21.15.27:0/262637724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f37a8105390 msgr2=0x7f37a81118a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.651+0000 7f37ae200640 1 --2- 172.21.15.27:0/262637724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f37a8105390 0x7f37a81118a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009be0 tx=0x7f37a002a6d0 comp rx=0 tx=0).stop 2024-05-10T11:32:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.651+0000 7f37ae200640 1 -- 172.21.15.27:0/262637724 shutdown_connections 2024-05-10T11:32:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.651+0000 7f37ae200640 1 --2- 172.21.15.27:0/262637724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f37a8105390 0x7f37a81118a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.651+0000 7f37ae200640 1 --2- 172.21.15.27:0/262637724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a81049d0 0x7f37a8104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.651+0000 7f37ae200640 1 --2- 172.21.15.27:0/262637724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a810a9e0 0x7f37a810adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.651+0000 7f37ae200640 1 -- 172.21.15.27:0/262637724 >> 172.21.15.27:0/262637724 conn(0x7f37a8100480 msgr2=0x7f37a81028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:13.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.651+0000 7f37ae200640 1 -- 172.21.15.27:0/262637724 shutdown_connections 2024-05-10T11:32:13.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.651+0000 7f37ae200640 1 -- 172.21.15.27:0/262637724 wait complete. 2024-05-10T11:32:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.652+0000 7f37ae200640 1 Processor -- start 2024-05-10T11:32:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.652+0000 7f37ae200640 1 -- start start 2024-05-10T11:32:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.652+0000 7f37ae200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a81049d0 0x7f37a819b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ae200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f37a8105390 0x7f37a819b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ae200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810a9e0 0x7f37a819fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ae200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f37a8114ca0 con 0x7f37a81049d0 2024-05-10T11:32:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ae200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f37a8114b20 con 0x7f37a8105390 2024-05-10T11:32:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ae200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f37a8114e20 con 0x7f37a810a9e0 2024-05-10T11:32:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ace00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a81049d0 0x7f37a819b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ace00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a81049d0 0x7f37a819b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43540/0 (socket says 172.21.15.27:43540) 2024-05-10T11:32:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ace00640 1 -- 172.21.15.27:0/2760726811 learned_addr learned my addr 172.21.15.27:0/2760726811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ad800640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810a9e0 0x7f37a819fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37a7e00640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f37a8105390 0x7f37a819b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ace00640 1 -- 172.21.15.27:0/2760726811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810a9e0 msgr2=0x7f37a819fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ace00640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810a9e0 0x7f37a819fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ace00640 1 -- 172.21.15.27:0/2760726811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f37a8105390 msgr2=0x7f37a819b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ace00640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f37a8105390 0x7f37a819b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.653+0000 7f37ace00640 1 -- 172.21.15.27:0/2760726811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a81a0160 con 0x7f37a81049d0 2024-05-10T11:32:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.654+0000 7f37ace00640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a81049d0 0x7f37a819b1c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f379c00e9f0 tx=0x7f379c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.654+0000 7f37a5600640 1 -- 172.21.15.27:0/2760726811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379c00ccc0 con 0x7f37a81049d0 2024-05-10T11:32:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.654+0000 7f37a5600640 1 -- 172.21.15.27:0/2760726811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f379c0108e0 con 0x7f37a81049d0 2024-05-10T11:32:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.654+0000 7f37ae200640 1 -- 172.21.15.27:0/2760726811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37a81a0450 con 0x7f37a81049d0 2024-05-10T11:32:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.654+0000 7f37a5600640 1 -- 172.21.15.27:0/2760726811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379c018880 con 0x7f37a81049d0 2024-05-10T11:32:13.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.654+0000 7f37ae200640 1 -- 172.21.15.27:0/2760726811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37a81a7ce0 con 0x7f37a81049d0 2024-05-10T11:32:13.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.655+0000 7f37ae200640 1 -- 172.21.15.27:0/2760726811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37a80696f0 con 0x7f37a81049d0 2024-05-10T11:32:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.659+0000 7f37a5600640 1 -- 172.21.15.27:0/2760726811 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f379c010430 con 0x7f37a81049d0 2024-05-10T11:32:13.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.660+0000 7f37a5600640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f37900b2960 0x7f37900b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:13.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.661+0000 7f37a5600640 1 -- 172.21.15.27:0/2760726811 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9591+0+0 (secure 0 0 0) 0x7f379c0e3300 con 0x7f37a81049d0 2024-05-10T11:32:13.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.661+0000 7f37a7e00640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f37900b2960 0x7f37900b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:13.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.661+0000 7f37a7e00640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f37900b2960 0x7f37900b4e20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f37a8102b40 tx=0x7f3794009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:13.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.662+0000 7f37a5600640 1 -- 172.21.15.27:0/2760726811 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f379c0aa850 con 0x7f37a81049d0 2024-05-10T11:32:13.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:13 smithi027 ceph-mon[30054]: pgmap v598: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 212 op/s 2024-05-10T11:32:13.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:32:13.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:32:13.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:13 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.885+0000 7f37ae200640 1 -- 172.21.15.27:0/2760726811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37a819c4c0 con 0x7f37a81049d0 2024-05-10T11:32:13.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.888+0000 7f37a5600640 1 -- 172.21.15.27:0/2760726811 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) ==== 74+0+25401 (secure 0 0 0) 0x7f379c0aa150 con 0x7f37a81049d0 2024-05-10T11:32:13.888 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:32:13.888 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:32:08.852070+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6},{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1021663132":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6801/1704535643":"2024-05-11T11:13:13.118713+0000","172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/1450791252":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1410017298":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/1704535643":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/2991615845":"2024-05-11T11:13:13.118713+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":20,"length":1},{"begin":33,"length":1}]},{"pool":4,"snaps":[{"begin":20,"length":1},{"begin":33,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:32:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.890+0000 7f37ae200640 1 -- 172.21.15.27:0/2760726811 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f37900b2960 msgr2=0x7f37900b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.890+0000 7f37ae200640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f37900b2960 0x7f37900b4e20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f37a8102b40 tx=0x7f3794009260 comp rx=0 tx=0).stop 2024-05-10T11:32:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.890+0000 7f37ae200640 1 -- 172.21.15.27:0/2760726811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a81049d0 msgr2=0x7f37a819b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.890+0000 7f37ae200640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a81049d0 0x7f37a819b1c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f379c00e9f0 tx=0x7f379c00edf0 comp rx=0 tx=0).stop 2024-05-10T11:32:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.890+0000 7f37ae200640 1 -- 172.21.15.27:0/2760726811 shutdown_connections 2024-05-10T11:32:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.890+0000 7f37ae200640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f37900b2960 0x7f37900b4e20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.890+0000 7f37ae200640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810a9e0 0x7f37a819fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.890+0000 7f37ae200640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f37a8105390 0x7f37a819b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.890+0000 7f37ae200640 1 --2- 172.21.15.27:0/2760726811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a81049d0 0x7f37a819b1c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.891+0000 7f37ae200640 1 -- 172.21.15.27:0/2760726811 >> 172.21.15.27:0/2760726811 conn(0x7f37a8100480 msgr2=0x7f37a8100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.891+0000 7f37ae200640 1 -- 172.21.15.27:0/2760726811 shutdown_connections 2024-05-10T11:32:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:13.891+0000 7f37ae200640 1 -- 172.21.15.27:0/2760726811 wait complete. 2024-05-10T11:32:13.909 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-10T11:02:06.781475+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-10T11:07:40.667104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-10T11:07:45.158839+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 116, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-10T11:07:58.869099+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 116, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-10T11:32:13.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-10T11:32:13.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:13 smithi076 ceph-mon[34152]: pgmap v598: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 212 op/s 2024-05-10T11:32:13.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:32:13.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:32:13.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:13 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:13.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:13 smithi189 ceph-mon[34842]: pgmap v598: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 212 op/s 2024-05-10T11:32:13.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:32:13.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:32:13.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:13 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:32:14.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.111+0000 7fbb65e00640 1 -- 172.21.15.27:0/3550112093 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb60103210 msgr2=0x7fbb6010f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:14.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.111+0000 7fbb65e00640 1 --2- 172.21.15.27:0/3550112093 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb60103210 0x7fbb6010f720 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fbb58009860 tx=0x7fbb5802a340 comp rx=0 tx=0).stop 2024-05-10T11:32:14.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.112+0000 7fbb65e00640 1 -- 172.21.15.27:0/3550112093 shutdown_connections 2024-05-10T11:32:14.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.112+0000 7fbb65e00640 1 --2- 172.21.15.27:0/3550112093 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb60103210 0x7fbb6010f720 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.112+0000 7fbb65e00640 1 --2- 172.21.15.27:0/3550112093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb60102850 0x7fbb60102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.112+0000 7fbb65e00640 1 --2- 172.21.15.27:0/3550112093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb60108860 0x7fbb60108c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.112+0000 7fbb65e00640 1 -- 172.21.15.27:0/3550112093 >> 172.21.15.27:0/3550112093 conn(0x7fbb600fe380 msgr2=0x7fbb601007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:14.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.112+0000 7fbb65e00640 1 -- 172.21.15.27:0/3550112093 shutdown_connections 2024-05-10T11:32:14.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.112+0000 7fbb65e00640 1 -- 172.21.15.27:0/3550112093 wait complete. 2024-05-10T11:32:14.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.113+0000 7fbb65e00640 1 Processor -- start 2024-05-10T11:32:14.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.113+0000 7fbb65e00640 1 -- start start 2024-05-10T11:32:14.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.113+0000 7fbb65e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb60102850 0x7fbb6019b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:14.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.113+0000 7fbb65e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb60103210 0x7fbb6019b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:14.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.113+0000 7fbb65e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb60108860 0x7fbb6019fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:14.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.113+0000 7fbb65e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb60112b20 con 0x7fbb60108860 2024-05-10T11:32:14.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.113+0000 7fbb65e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbb601129a0 con 0x7fbb60102850 2024-05-10T11:32:14.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.113+0000 7fbb65e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbb60112ca0 con 0x7fbb60103210 2024-05-10T11:32:14.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.114+0000 7fbb64a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb60102850 0x7fbb6019b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:14.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.114+0000 7fbb64a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb60102850 0x7fbb6019b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:39136/0 (socket says 172.21.15.27:39136) 2024-05-10T11:32:14.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.114+0000 7fbb64a00640 1 -- 172.21.15.27:0/1653699018 learned_addr learned my addr 172.21.15.27:0/1653699018 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:14.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.114+0000 7fbb65400640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb60108860 0x7fbb6019fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:14.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.114+0000 7fbb5fe00640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb60103210 0x7fbb6019b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:14.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.114+0000 7fbb64a00640 1 -- 172.21.15.27:0/1653699018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb60103210 msgr2=0x7fbb6019b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:14.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.114+0000 7fbb64a00640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb60103210 0x7fbb6019b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.114+0000 7fbb64a00640 1 -- 172.21.15.27:0/1653699018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb60108860 msgr2=0x7fbb6019fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:14.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.114+0000 7fbb64a00640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb60108860 0x7fbb6019fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.114+0000 7fbb64a00640 1 -- 172.21.15.27:0/1653699018 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb601a01f0 con 0x7fbb60102850 2024-05-10T11:32:14.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.114+0000 7fbb64a00640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb60102850 0x7fbb6019b200 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800e9f0 tx=0x7fbb4800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:14.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.115+0000 7fbb5d600640 1 -- 172.21.15.27:0/1653699018 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4800cd80 con 0x7fbb60102850 2024-05-10T11:32:14.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.115+0000 7fbb5d600640 1 -- 172.21.15.27:0/1653699018 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb480108e0 con 0x7fbb60102850 2024-05-10T11:32:14.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.115+0000 7fbb5d600640 1 -- 172.21.15.27:0/1653699018 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48018980 con 0x7fbb60102850 2024-05-10T11:32:14.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.115+0000 7fbb65e00640 1 -- 172.21.15.27:0/1653699018 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb601a04e0 con 0x7fbb60102850 2024-05-10T11:32:14.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.115+0000 7fbb65e00640 1 -- 172.21.15.27:0/1653699018 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb601a7d70 con 0x7fbb60102850 2024-05-10T11:32:14.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.116+0000 7fbb65e00640 1 -- 172.21.15.27:0/1653699018 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb600696f0 con 0x7fbb60102850 2024-05-10T11:32:14.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.118+0000 7fbb5d600640 1 -- 172.21.15.27:0/1653699018 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fbb48010430 con 0x7fbb60102850 2024-05-10T11:32:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.120+0000 7fbb5d600640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbb440b29e0 0x7fbb440b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.120+0000 7fbb5d600640 1 -- 172.21.15.27:0/1653699018 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9591+0+0 (secure 0 0 0) 0x7fbb480e2ce0 con 0x7fbb60102850 2024-05-10T11:32:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.120+0000 7fbb5fe00640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbb440b29e0 0x7fbb440b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:14.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.121+0000 7fbb5fe00640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbb440b29e0 0x7fbb440b4ea0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009a60 tx=0x7fbb50010040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:14.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.124+0000 7fbb5d600640 1 -- 172.21.15.27:0/1653699018 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fbb480a9be0 con 0x7fbb60102850 2024-05-10T11:32:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.351+0000 7fbb65e00640 1 -- 172.21.15.27:0/1653699018 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbb601a7fa0 con 0x7fbb60102850 2024-05-10T11:32:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.352+0000 7fbb5d600640 1 -- 172.21.15.27:0/1653699018 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) ==== 93+0+10 (secure 0 0 0) 0x7fbb480adee0 con 0x7fbb60102850 2024-05-10T11:32:14.352 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-05-10T11:32:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.355+0000 7fbb65e00640 1 -- 172.21.15.27:0/1653699018 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbb440b29e0 msgr2=0x7fbb440b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.355+0000 7fbb65e00640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbb440b29e0 0x7fbb440b4ea0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009a60 tx=0x7fbb50010040 comp rx=0 tx=0).stop 2024-05-10T11:32:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.355+0000 7fbb65e00640 1 -- 172.21.15.27:0/1653699018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb60102850 msgr2=0x7fbb6019b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.355+0000 7fbb65e00640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb60102850 0x7fbb6019b200 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800e9f0 tx=0x7fbb4800eeb0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.355+0000 7fbb65e00640 1 -- 172.21.15.27:0/1653699018 shutdown_connections 2024-05-10T11:32:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.356+0000 7fbb65e00640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbb440b29e0 0x7fbb440b4ea0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.356+0000 7fbb65e00640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb60108860 0x7fbb6019fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.356+0000 7fbb65e00640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb60103210 0x7fbb6019b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.356+0000 7fbb65e00640 1 --2- 172.21.15.27:0/1653699018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb60102850 0x7fbb6019b200 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.356+0000 7fbb65e00640 1 -- 172.21.15.27:0/1653699018 >> 172.21.15.27:0/1653699018 conn(0x7fbb600fe380 msgr2=0x7fbb6010df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.356+0000 7fbb65e00640 1 -- 172.21.15.27:0/1653699018 shutdown_connections 2024-05-10T11:32:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.356+0000 7fbb65e00640 1 -- 172.21.15.27:0/1653699018 wait complete. 2024-05-10T11:32:14.375 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-10T11:32:14.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.601+0000 7fd311800640 1 -- 172.21.15.27:0/613444500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd30c104a60 msgr2=0x7fd30c104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:14.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.601+0000 7fd311800640 1 --2- 172.21.15.27:0/613444500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd30c104a60 0x7fd30c104ee0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009930 tx=0x7fd2f802a440 comp rx=0 tx=0).stop 2024-05-10T11:32:14.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.601+0000 7fd311800640 1 -- 172.21.15.27:0/613444500 shutdown_connections 2024-05-10T11:32:14.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.601+0000 7fd311800640 1 --2- 172.21.15.27:0/613444500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd30c105420 0x7fd30c111930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.601+0000 7fd311800640 1 --2- 172.21.15.27:0/613444500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd30c104a60 0x7fd30c104ee0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.601+0000 7fd311800640 1 --2- 172.21.15.27:0/613444500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd30c10aa70 0x7fd30c10ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.601+0000 7fd311800640 1 -- 172.21.15.27:0/613444500 >> 172.21.15.27:0/613444500 conn(0x7fd30c1004f0 msgr2=0x7fd30c102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:14.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.601+0000 7fd311800640 1 -- 172.21.15.27:0/613444500 shutdown_connections 2024-05-10T11:32:14.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.601+0000 7fd311800640 1 -- 172.21.15.27:0/613444500 wait complete. 2024-05-10T11:32:14.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.602+0000 7fd311800640 1 Processor -- start 2024-05-10T11:32:14.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.603+0000 7fd311800640 1 -- start start 2024-05-10T11:32:14.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.603+0000 7fd311800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd30c104a60 0x7fd30c19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:14.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.603+0000 7fd311800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd30c105420 0x7fd30c19b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:14.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.603+0000 7fd311800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd30c10aa70 0x7fd30c19fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:14.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.603+0000 7fd311800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd30c114d80 con 0x7fd30c104a60 2024-05-10T11:32:14.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.603+0000 7fd311800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd30c114c00 con 0x7fd30c10aa70 2024-05-10T11:32:14.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.603+0000 7fd311800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd30c114f00 con 0x7fd30c105420 2024-05-10T11:32:14.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.603+0000 7fd30be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd30c104a60 0x7fd30c19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:14.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.604+0000 7fd30be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd30c104a60 0x7fd30c19b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59590/0 (socket says 172.21.15.27:59590) 2024-05-10T11:32:14.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.604+0000 7fd30be00640 1 -- 172.21.15.27:0/677887021 learned_addr learned my addr 172.21.15.27:0/677887021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:14.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.604+0000 7fd30b400640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd30c105420 0x7fd30c19b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:14.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.604+0000 7fd310e00640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd30c10aa70 0x7fd30c19fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:14.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.604+0000 7fd30be00640 1 -- 172.21.15.27:0/677887021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd30c105420 msgr2=0x7fd30c19b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:14.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.604+0000 7fd30be00640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd30c105420 0x7fd30c19b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.604+0000 7fd30be00640 1 -- 172.21.15.27:0/677887021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd30c10aa70 msgr2=0x7fd30c19fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:14.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.604+0000 7fd30be00640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd30c10aa70 0x7fd30c19fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.604+0000 7fd30be00640 1 -- 172.21.15.27:0/677887021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd30c1a02f0 con 0x7fd30c104a60 2024-05-10T11:32:14.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.604+0000 7fd310e00640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd30c10aa70 0x7fd30c19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:32:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.604+0000 7fd30be00640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd30c104a60 0x7fd30c19b300 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc00e9f0 tx=0x7fd2fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.605+0000 7fd308c00640 1 -- 172.21.15.27:0/677887021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc00cd80 con 0x7fd30c104a60 2024-05-10T11:32:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.605+0000 7fd308c00640 1 -- 172.21.15.27:0/677887021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2fc0108e0 con 0x7fd30c104a60 2024-05-10T11:32:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.605+0000 7fd311800640 1 -- 172.21.15.27:0/677887021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd30c1a05e0 con 0x7fd30c104a60 2024-05-10T11:32:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.605+0000 7fd308c00640 1 -- 172.21.15.27:0/677887021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc018980 con 0x7fd30c104a60 2024-05-10T11:32:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.605+0000 7fd311800640 1 -- 172.21.15.27:0/677887021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd30c1a7e70 con 0x7fd30c104a60 2024-05-10T11:32:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.606+0000 7fd311800640 1 -- 172.21.15.27:0/677887021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd30c069b40 con 0x7fd30c104a60 2024-05-10T11:32:14.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.610+0000 7fd308c00640 1 -- 172.21.15.27:0/677887021 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd2fc010430 con 0x7fd30c104a60 2024-05-10T11:32:14.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.612+0000 7fd308c00640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd2f00b2990 0x7fd2f00b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:14.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.612+0000 7fd308c00640 1 -- 172.21.15.27:0/677887021 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9591+0+0 (secure 0 0 0) 0x7fd2fc0e3240 con 0x7fd30c104a60 2024-05-10T11:32:14.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.612+0000 7fd30b400640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd2f00b2990 0x7fd2f00b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:14.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.613+0000 7fd30b400640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd2f00b2990 0x7fd2f00b4e50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8031440 tx=0x7fd2f8000bc0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:14.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.615+0000 7fd308c00640 1 -- 172.21.15.27:0/677887021 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fd2fc0aa140 con 0x7fd30c104a60 2024-05-10T11:32:14.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.858+0000 7fd311800640 1 -- 172.21.15.27:0/677887021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd30c10cfd0 con 0x7fd30c104a60 2024-05-10T11:32:14.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.859+0000 7fd308c00640 1 -- 172.21.15.27:0/677887021 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v117) ==== 104+0+11 (secure 0 0 0) 0x7fd2fc0ae440 con 0x7fd30c104a60 2024-05-10T11:32:14.859 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-05-10T11:32:14.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.863+0000 7fd311800640 1 -- 172.21.15.27:0/677887021 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd2f00b2990 msgr2=0x7fd2f00b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:14.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.863+0000 7fd311800640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd2f00b2990 0x7fd2f00b4e50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8031440 tx=0x7fd2f8000bc0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.863+0000 7fd311800640 1 -- 172.21.15.27:0/677887021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd30c104a60 msgr2=0x7fd30c19b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:14.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.863+0000 7fd311800640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd30c104a60 0x7fd30c19b300 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc00e9f0 tx=0x7fd2fc00eeb0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.864+0000 7fd311800640 1 -- 172.21.15.27:0/677887021 shutdown_connections 2024-05-10T11:32:14.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.864+0000 7fd311800640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fd2f00b2990 0x7fd2f00b4e50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.864+0000 7fd311800640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd30c10aa70 0x7fd30c19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.864+0000 7fd311800640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd30c105420 0x7fd30c19b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.864+0000 7fd311800640 1 --2- 172.21.15.27:0/677887021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd30c104a60 0x7fd30c19b300 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:14.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.864+0000 7fd311800640 1 -- 172.21.15.27:0/677887021 >> 172.21.15.27:0/677887021 conn(0x7fd30c1004f0 msgr2=0x7fd30c100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:14.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.864+0000 7fd311800640 1 -- 172.21.15.27:0/677887021 shutdown_connections 2024-05-10T11:32:14.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:14.864+0000 7fd311800640 1 -- 172.21.15.27:0/677887021 wait complete. 2024-05-10T11:32:14.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:14 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2760726811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:32:14.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:14 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1653699018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-10T11:32:14.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-10T11:32:14.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:14 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2760726811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:32:14.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:14 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1653699018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-10T11:32:14.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:14 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2760726811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:32:14.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:14 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1653699018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-10T11:32:15.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.176+0000 7f699d800640 1 -- 172.21.15.27:0/546656674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6998102340 msgr2=0x7f6998106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.176+0000 7f699d800640 1 --2- 172.21.15.27:0/546656674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6998102340 0x7f6998106800 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6988009900 tx=0x7f698802a520 comp rx=0 tx=0).stop 2024-05-10T11:32:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.178+0000 7f699d800640 1 -- 172.21.15.27:0/546656674 shutdown_connections 2024-05-10T11:32:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.178+0000 7f699d800640 1 --2- 172.21.15.27:0/546656674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6998106d40 0x7f69981071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.178+0000 7f699d800640 1 --2- 172.21.15.27:0/546656674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6998102340 0x7f6998106800 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.178+0000 7f699d800640 1 --2- 172.21.15.27:0/546656674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6998101a20 0x7f6998101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.178+0000 7f699d800640 1 -- 172.21.15.27:0/546656674 >> 172.21.15.27:0/546656674 conn(0x7f69980789b0 msgr2=0x7f6998078dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.179+0000 7f699d800640 1 -- 172.21.15.27:0/546656674 shutdown_connections 2024-05-10T11:32:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.179+0000 7f699d800640 1 -- 172.21.15.27:0/546656674 wait complete. 2024-05-10T11:32:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.180+0000 7f699d800640 1 Processor -- start 2024-05-10T11:32:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.180+0000 7f699d800640 1 -- start start 2024-05-10T11:32:15.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.180+0000 7f699d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6998101a20 0x7f69981a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:15.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.181+0000 7f699d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6998102340 0x7f69981a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:15.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.181+0000 7f699d800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6998106d40 0x7f699819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:15.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.181+0000 7f699d800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6998119240 con 0x7f6998101a20 2024-05-10T11:32:15.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.181+0000 7f699d800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f69981190c0 con 0x7f6998106d40 2024-05-10T11:32:15.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.181+0000 7f699d800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f69981193c0 con 0x7f6998102340 2024-05-10T11:32:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.181+0000 7f6996000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6998102340 0x7f69981a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.181+0000 7f6997400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6998106d40 0x7f699819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.181+0000 7f6997400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6998106d40 0x7f699819f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:39164/0 (socket says 172.21.15.27:39164) 2024-05-10T11:32:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.181+0000 7f6997400640 1 -- 172.21.15.27:0/4291956508 learned_addr learned my addr 172.21.15.27:0/4291956508 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.181+0000 7f6996a00640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6998101a20 0x7f69981a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.182+0000 7f6997400640 1 -- 172.21.15.27:0/4291956508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6998102340 msgr2=0x7f69981a5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.182+0000 7f6997400640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6998102340 0x7f69981a5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.182+0000 7f6997400640 1 -- 172.21.15.27:0/4291956508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6998101a20 msgr2=0x7f69981a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.182+0000 7f6997400640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6998101a20 0x7f69981a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.182+0000 7f6997400640 1 -- 172.21.15.27:0/4291956508 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699819ff10 con 0x7f6998106d40 2024-05-10T11:32:15.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.182+0000 7f6997400640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6998106d40 0x7f699819f650 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6990009fd0 tx=0x7f699000b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:15.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.183+0000 7f698f400640 1 -- 172.21.15.27:0/4291956508 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990017070 con 0x7f6998106d40 2024-05-10T11:32:15.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.183+0000 7f698f400640 1 -- 172.21.15.27:0/4291956508 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6990003e20 con 0x7f6998106d40 2024-05-10T11:32:15.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.183+0000 7f698f400640 1 -- 172.21.15.27:0/4291956508 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69900106c0 con 0x7f6998106d40 2024-05-10T11:32:15.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.183+0000 7f699d800640 1 -- 172.21.15.27:0/4291956508 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69981a0200 con 0x7f6998106d40 2024-05-10T11:32:15.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.183+0000 7f699d800640 1 -- 172.21.15.27:0/4291956508 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69981ac510 con 0x7f6998106d40 2024-05-10T11:32:15.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.185+0000 7f699d800640 1 -- 172.21.15.27:0/4291956508 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f699806a340 con 0x7f6998106d40 2024-05-10T11:32:15.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.187+0000 7f698f400640 1 -- 172.21.15.27:0/4291956508 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f69900041d0 con 0x7f6998106d40 2024-05-10T11:32:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.190+0000 7f698f400640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f697c0b2a10 0x7f697c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.190+0000 7f698f400640 1 -- 172.21.15.27:0/4291956508 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9591+0+0 (secure 0 0 0) 0x7f69900e26e0 con 0x7f6998106d40 2024-05-10T11:32:15.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.191+0000 7f6996a00640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f697c0b2a10 0x7f697c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.192+0000 7f6996a00640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f697c0b2a10 0x7f697c0b4ed0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f698400bbe0 tx=0x7f698400b390 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:15.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.194+0000 7f698f400640 1 -- 172.21.15.27:0/4291956508 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f69900a95e0 con 0x7f6998106d40 2024-05-10T11:32:15.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.431+0000 7f699d800640 1 -- 172.21.15.27:0/4291956508 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f69981a0430 con 0x7f6998106d40 2024-05-10T11:32:15.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.432+0000 7f698f400640 1 -- 172.21.15.27:0/4291956508 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v117) ==== 100+0+11 (secure 0 0 0) 0x7f69900ad8e0 con 0x7f6998106d40 2024-05-10T11:32:15.431 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-05-10T11:32:15.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.434+0000 7f699d800640 1 -- 172.21.15.27:0/4291956508 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f697c0b2a10 msgr2=0x7f697c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:15.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.434+0000 7f699d800640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f697c0b2a10 0x7f697c0b4ed0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f698400bbe0 tx=0x7f698400b390 comp rx=0 tx=0).stop 2024-05-10T11:32:15.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.434+0000 7f699d800640 1 -- 172.21.15.27:0/4291956508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6998106d40 msgr2=0x7f699819f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:15.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.434+0000 7f699d800640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6998106d40 0x7f699819f650 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6990009fd0 tx=0x7f699000b260 comp rx=0 tx=0).stop 2024-05-10T11:32:15.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.434+0000 7f699d800640 1 -- 172.21.15.27:0/4291956508 shutdown_connections 2024-05-10T11:32:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.434+0000 7f699d800640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f697c0b2a10 0x7f697c0b4ed0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.434+0000 7f699d800640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6998106d40 0x7f699819f650 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.434+0000 7f699d800640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6998102340 0x7f69981a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.434+0000 7f699d800640 1 --2- 172.21.15.27:0/4291956508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6998101a20 0x7f69981a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.434+0000 7f699d800640 1 -- 172.21.15.27:0/4291956508 >> 172.21.15.27:0/4291956508 conn(0x7f69980789b0 msgr2=0x7f69981047d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.435+0000 7f699d800640 1 -- 172.21.15.27:0/4291956508 shutdown_connections 2024-05-10T11:32:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.435+0000 7f699d800640 1 -- 172.21.15.27:0/4291956508 wait complete. 2024-05-10T11:32:15.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-10T11:32:15.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.655+0000 7f9027000640 1 -- 172.21.15.27:0/1510366347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90201053f0 msgr2=0x7f9020111900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:15.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.655+0000 7f9027000640 1 --2- 172.21.15.27:0/1510366347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90201053f0 0x7f9020111900 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9018009ca0 tx=0x7f901802a8c0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.655+0000 7f9027000640 1 -- 172.21.15.27:0/1510366347 shutdown_connections 2024-05-10T11:32:15.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.655+0000 7f9027000640 1 --2- 172.21.15.27:0/1510366347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90201053f0 0x7f9020111900 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.655+0000 7f9027000640 1 --2- 172.21.15.27:0/1510366347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9020104a30 0x7f9020104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.655+0000 7f9027000640 1 --2- 172.21.15.27:0/1510366347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f902010aa40 0x7f902010ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.655+0000 7f9027000640 1 -- 172.21.15.27:0/1510366347 >> 172.21.15.27:0/1510366347 conn(0x7f90201004c0 msgr2=0x7f90201028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:15.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.655+0000 7f9027000640 1 -- 172.21.15.27:0/1510366347 shutdown_connections 2024-05-10T11:32:15.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.655+0000 7f9027000640 1 -- 172.21.15.27:0/1510366347 wait complete. 2024-05-10T11:32:15.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.656+0000 7f9027000640 1 Processor -- start 2024-05-10T11:32:15.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.656+0000 7f9027000640 1 -- start start 2024-05-10T11:32:15.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.656+0000 7f9027000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9020104a30 0x7f902019b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:15.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9027000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90201053f0 0x7f902019b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:15.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9027000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f902010aa40 0x7f902019fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:15.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9027000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9020114d50 con 0x7f9020104a30 2024-05-10T11:32:15.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9027000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9020114bd0 con 0x7f902010aa40 2024-05-10T11:32:15.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9027000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9020114ed0 con 0x7f90201053f0 2024-05-10T11:32:15.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9025c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9020104a30 0x7f902019b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:15.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9025c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9020104a30 0x7f902019b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59622/0 (socket says 172.21.15.27:59622) 2024-05-10T11:32:15.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9025c00640 1 -- 172.21.15.27:0/2127414105 learned_addr learned my addr 172.21.15.27:0/2127414105 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:15.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9026600640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f902010aa40 0x7f902019fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:15.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9025200640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90201053f0 0x7f902019b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:15.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9025c00640 1 -- 172.21.15.27:0/2127414105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90201053f0 msgr2=0x7f902019b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:15.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9025c00640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90201053f0 0x7f902019b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9025c00640 1 -- 172.21.15.27:0/2127414105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f902010aa40 msgr2=0x7f902019fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:15.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9025c00640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f902010aa40 0x7f902019fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.657+0000 7f9025c00640 1 -- 172.21.15.27:0/2127414105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90201a01e0 con 0x7f9020104a30 2024-05-10T11:32:15.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.658+0000 7f9025c00640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9020104a30 0x7f902019b1f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f901400e9f0 tx=0x7f901400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:15.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.658+0000 7f901e000640 1 -- 172.21.15.27:0/2127414105 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901400cd80 con 0x7f9020104a30 2024-05-10T11:32:15.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.658+0000 7f901e000640 1 -- 172.21.15.27:0/2127414105 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90140054d0 con 0x7f9020104a30 2024-05-10T11:32:15.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.658+0000 7f9027000640 1 -- 172.21.15.27:0/2127414105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90201a04d0 con 0x7f9020104a30 2024-05-10T11:32:15.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.658+0000 7f901e000640 1 -- 172.21.15.27:0/2127414105 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9014010690 con 0x7f9020104a30 2024-05-10T11:32:15.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.658+0000 7f9027000640 1 -- 172.21.15.27:0/2127414105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90201a7d60 con 0x7f9020104a30 2024-05-10T11:32:15.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.659+0000 7f9027000640 1 -- 172.21.15.27:0/2127414105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9020069b40 con 0x7f9020104a30 2024-05-10T11:32:15.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.662+0000 7f901e000640 1 -- 172.21.15.27:0/2127414105 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f9014004e90 con 0x7f9020104a30 2024-05-10T11:32:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.663+0000 7f901e000640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f90080b2960 0x7f90080b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.663+0000 7f901e000640 1 -- 172.21.15.27:0/2127414105 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9591+0+0 (secure 0 0 0) 0x7f90140e2da0 con 0x7f9020104a30 2024-05-10T11:32:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.663+0000 7f9025200640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f90080b2960 0x7f90080b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.664+0000 7f9025200640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f90080b2960 0x7f90080b4e20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9020102ac0 tx=0x7f900c005f90 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:15.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.667+0000 7f901e000640 1 -- 172.21.15.27:0/2127414105 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f90140aa350 con 0x7f9020104a30 2024-05-10T11:32:15.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:15 smithi027 ceph-mon[30054]: pgmap v599: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.8 MiB/s wr, 196 op/s 2024-05-10T11:32:15.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:15 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/677887021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-10T11:32:15.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:15 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4291956508' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-10T11:32:15.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.879+0000 7f9027000640 1 -- 172.21.15.27:0/2127414105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f902010cfa0 con 0x7f9020104a30 2024-05-10T11:32:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.880+0000 7f901e000640 1 -- 172.21.15.27:0/2127414105 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v117) ==== 103+0+11 (secure 0 0 0) 0x7f90140a9c50 con 0x7f9020104a30 2024-05-10T11:32:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-05-10T11:32:15.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.883+0000 7f9027000640 1 -- 172.21.15.27:0/2127414105 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f90080b2960 msgr2=0x7f90080b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:15.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.883+0000 7f9027000640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f90080b2960 0x7f90080b4e20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9020102ac0 tx=0x7f900c005f90 comp rx=0 tx=0).stop 2024-05-10T11:32:15.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.883+0000 7f9027000640 1 -- 172.21.15.27:0/2127414105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9020104a30 msgr2=0x7f902019b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:15.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.883+0000 7f9027000640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9020104a30 0x7f902019b1f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f901400e9f0 tx=0x7f901400edf0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.883+0000 7f9027000640 1 -- 172.21.15.27:0/2127414105 shutdown_connections 2024-05-10T11:32:15.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.883+0000 7f9027000640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f90080b2960 0x7f90080b4e20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.883+0000 7f9027000640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f902010aa40 0x7f902019fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.883+0000 7f9027000640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90201053f0 0x7f902019b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.883+0000 7f9027000640 1 --2- 172.21.15.27:0/2127414105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9020104a30 0x7f902019b1f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:15.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.883+0000 7f9027000640 1 -- 172.21.15.27:0/2127414105 >> 172.21.15.27:0/2127414105 conn(0x7f90201004c0 msgr2=0x7f9020100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:15.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.883+0000 7f9027000640 1 -- 172.21.15.27:0/2127414105 shutdown_connections 2024-05-10T11:32:15.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:15.883+0000 7f9027000640 1 -- 172.21.15.27:0/2127414105 wait complete. 2024-05-10T11:32:15.903 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-10T11:32:15.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:15 smithi076 ceph-mon[34152]: pgmap v599: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.8 MiB/s wr, 196 op/s 2024-05-10T11:32:15.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:15 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/677887021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-10T11:32:15.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:15 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4291956508' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-10T11:32:15.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:15 smithi189 ceph-mon[34842]: pgmap v599: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.8 MiB/s wr, 196 op/s 2024-05-10T11:32:15.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:15 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/677887021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-10T11:32:15.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:15 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4291956508' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-10T11:32:16.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.121+0000 7f25cb400640 1 -- 172.21.15.27:0/3708898960 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25c41057b0 msgr2=0x7f25c41097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:16.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.121+0000 7f25cb400640 1 --2- 172.21.15.27:0/3708898960 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25c41057b0 0x7f25c41097f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009860 tx=0x7f25ac02a340 comp rx=0 tx=0).stop 2024-05-10T11:32:16.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.121+0000 7f25cb400640 1 -- 172.21.15.27:0/3708898960 shutdown_connections 2024-05-10T11:32:16.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.121+0000 7f25cb400640 1 --2- 172.21.15.27:0/3708898960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c4109ed0 0x7f25c4111aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:16.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.121+0000 7f25cb400640 1 --2- 172.21.15.27:0/3708898960 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25c41057b0 0x7f25c41097f0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:16.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.121+0000 7f25cb400640 1 --2- 172.21.15.27:0/3708898960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25c4104e00 0x7f25c41051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:16.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.121+0000 7f25cb400640 1 -- 172.21.15.27:0/3708898960 >> 172.21.15.27:0/3708898960 conn(0x7f25c41006a0 msgr2=0x7f25c4102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:16.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.121+0000 7f25cb400640 1 -- 172.21.15.27:0/3708898960 shutdown_connections 2024-05-10T11:32:16.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.121+0000 7f25cb400640 1 -- 172.21.15.27:0/3708898960 wait complete. 2024-05-10T11:32:16.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.122+0000 7f25cb400640 1 Processor -- start 2024-05-10T11:32:16.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.123+0000 7f25cb400640 1 -- start start 2024-05-10T11:32:16.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.123+0000 7f25cb400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25c4104e00 0x7f25c419b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:16.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.123+0000 7f25cb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c4109ed0 0x7f25c419b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.123+0000 7f25cb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25c419fcd0 0x7f25c41a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.123+0000 7f25cb400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f25c4114e30 con 0x7f25c4109ed0 2024-05-10T11:32:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.123+0000 7f25cb400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f25c4114cb0 con 0x7f25c4104e00 2024-05-10T11:32:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.123+0000 7f25cb400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f25c4114fb0 con 0x7f25c419fcd0 2024-05-10T11:32:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.123+0000 7f25c9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25c419fcd0 0x7f25c41a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.123+0000 7f25c8a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25c4104e00 0x7f25c419b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.123+0000 7f25bbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c4109ed0 0x7f25c419b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.123+0000 7f25c9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25c419fcd0 0x7f25c41a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:40084/0 (socket says 172.21.15.27:40084) 2024-05-10T11:32:16.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.123+0000 7f25c9400640 1 -- 172.21.15.27:0/506358858 learned_addr learned my addr 172.21.15.27:0/506358858 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:32:16.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.124+0000 7f25c8a00640 1 -- 172.21.15.27:0/506358858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25c419fcd0 msgr2=0x7f25c41a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:16.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.124+0000 7f25c8a00640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25c419fcd0 0x7f25c41a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:16.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.124+0000 7f25c8a00640 1 -- 172.21.15.27:0/506358858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c4109ed0 msgr2=0x7f25c419b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:16.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.124+0000 7f25c8a00640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c4109ed0 0x7f25c419b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:16.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.124+0000 7f25c8a00640 1 -- 172.21.15.27:0/506358858 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c41a07c0 con 0x7f25c4104e00 2024-05-10T11:32:16.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.124+0000 7f25c9400640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25c419fcd0 0x7f25c41a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:32:16.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.124+0000 7f25c8a00640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25c4104e00 0x7f25c419b410 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f25b400eb00 tx=0x7f25b400ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:16.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.124+0000 7f25c2000640 1 -- 172.21.15.27:0/506358858 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b400ce40 con 0x7f25c4104e00 2024-05-10T11:32:16.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.124+0000 7f25cb400640 1 -- 172.21.15.27:0/506358858 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25c41a0ab0 con 0x7f25c4104e00 2024-05-10T11:32:16.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.125+0000 7f25c2000640 1 -- 172.21.15.27:0/506358858 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25b40108e0 con 0x7f25c4104e00 2024-05-10T11:32:16.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.125+0000 7f25c2000640 1 -- 172.21.15.27:0/506358858 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b40179f0 con 0x7f25c4104e00 2024-05-10T11:32:16.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.125+0000 7f25cb400640 1 -- 172.21.15.27:0/506358858 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25c41a8260 con 0x7f25c4104e00 2024-05-10T11:32:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.126+0000 7f25cb400640 1 -- 172.21.15.27:0/506358858 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25c4069c20 con 0x7f25c4104e00 2024-05-10T11:32:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.128+0000 7f25c2000640 1 -- 172.21.15.27:0/506358858 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f25b4017b90 con 0x7f25c4104e00 2024-05-10T11:32:16.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.130+0000 7f25c2000640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f25a80b2960 0x7f25a80b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:32:16.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.130+0000 7f25c2000640 1 -- 172.21.15.27:0/506358858 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9591+0+0 (secure 0 0 0) 0x7f25b40e3310 con 0x7f25c4104e00 2024-05-10T11:32:16.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.130+0000 7f25bbe00640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f25a80b2960 0x7f25a80b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:32:16.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.131+0000 7f25bbe00640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f25a80b2960 0x7f25a80b4e20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0303f0 tx=0x7f25ac030340 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:32:16.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.134+0000 7f25c2000640 1 -- 172.21.15.27:0/506358858 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f25b40aa860 con 0x7f25c4104e00 2024-05-10T11:32:16.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.399+0000 7f25cb400640 1 -- 172.21.15.27:0/506358858 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f25c41057b0 con 0x7f25c4104e00 2024-05-10T11:32:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.412+0000 7f25c2000640 1 -- 172.21.15.27:0/506358858 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) ==== 81+0+816173 (secure 0 0 0) 0x7f25b40aa160 con 0x7f25c4104e00 2024-05-10T11:32:16.421 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:32:16.421 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:32:16.421 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 27, 2024-05-10T11:32:16.421 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.422 INFO:teuthology.orchestra.run.smithi027.stdout: "active_gid": 25220, 2024-05-10T11:32:16.422 INFO:teuthology.orchestra.run.smithi027.stdout: "active_name": "x", 2024-05-10T11:32:16.422 INFO:teuthology.orchestra.run.smithi027.stdout: "active_addrs": { 2024-05-10T11:32:16.422 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-10T11:32:16.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.423 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-10T11:32:16.423 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6800", 2024-05-10T11:32:16.423 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1167199542 2024-05-10T11:32:16.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.423 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-10T11:32:16.423 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6801", 2024-05-10T11:32:16.423 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1167199542 2024-05-10T11:32:16.424 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.424 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-10T11:32:16.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.424 INFO:teuthology.orchestra.run.smithi027.stdout: "active_addr": "172.21.15.27:6801/1167199542", 2024-05-10T11:32:16.424 INFO:teuthology.orchestra.run.smithi027.stdout: "active_change": "2024-05-10T11:13:13.119084+0000", 2024-05-10T11:32:16.424 INFO:teuthology.orchestra.run.smithi027.stdout: "active_mgr_features": 4540701547738038271, 2024-05-10T11:32:16.424 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-05-10T11:32:16.424 INFO:teuthology.orchestra.run.smithi027.stdout: "standbys": [ 2024-05-10T11:32:16.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.425 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 25192, 2024-05-10T11:32:16.425 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "z", 2024-05-10T11:32:16.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_features": 4540701547738038271, 2024-05-10T11:32:16.425 INFO:teuthology.orchestra.run.smithi027.stdout: "available_modules": [ 2024-05-10T11:32:16.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.425 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "alerts", 2024-05-10T11:32:16.426 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.426 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.426 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.426 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:16.426 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:16.426 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.426 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.426 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.427 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:16.427 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.427 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.427 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.427 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How frequently to reexamine health status", 2024-05-10T11:32:16.427 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.427 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.428 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.428 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.428 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.428 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.428 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.428 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.428 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.431 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.431 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.431 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.431 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.431 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.431 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.431 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.431 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.431 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.432 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.432 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.432 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.432 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.432 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.432 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.432 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.433 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.433 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.433 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.433 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.433 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.433 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.433 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.433 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.434 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.434 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.434 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.434 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.434 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.434 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.434 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.434 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.435 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.435 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.435 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.435 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.435 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.435 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.435 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.435 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.436 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.436 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.436 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.436 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.436 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.436 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.436 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_destination": { 2024-05-10T11:32:16.437 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_destination", 2024-05-10T11:32:16.437 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.437 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.437 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.437 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.437 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.437 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.437 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.437 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email address to send alerts to", 2024-05-10T11:32:16.438 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.438 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.438 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.438 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_from_name": { 2024-05-10T11:32:16.438 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_from_name", 2024-05-10T11:32:16.438 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.438 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.439 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.439 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "Ceph", 2024-05-10T11:32:16.439 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.439 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.439 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.439 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email From: name", 2024-05-10T11:32:16.439 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.439 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.440 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.440 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_host": { 2024-05-10T11:32:16.440 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_host", 2024-05-10T11:32:16.440 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.440 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.440 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.440 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.441 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.441 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.441 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.441 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP server", 2024-05-10T11:32:16.441 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.441 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.442 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.442 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_password": { 2024-05-10T11:32:16.442 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_password", 2024-05-10T11:32:16.442 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.442 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.442 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.442 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.443 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.443 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.443 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.443 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Password to authenticate with", 2024-05-10T11:32:16.443 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.443 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.443 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.443 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_port": { 2024-05-10T11:32:16.444 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_port", 2024-05-10T11:32:16.444 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.444 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.444 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.444 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "465", 2024-05-10T11:32:16.444 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.444 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.444 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP port", 2024-05-10T11:32:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_sender": { 2024-05-10T11:32:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_sender", 2024-05-10T11:32:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP envelope sender", 2024-05-10T11:32:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_ssl": { 2024-05-10T11:32:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_ssl", 2024-05-10T11:32:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-10T11:32:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_user": { 2024-05-10T11:32:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_user", 2024-05-10T11:32:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "User to authenticate as", 2024-05-10T11:32:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "balancer", 2024-05-10T11:32:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "active": { 2024-05-10T11:32:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "active", 2024-05-10T11:32:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically balance PGs across cluster", 2024-05-10T11:32:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_time": { 2024-05-10T11:32:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_time", 2024-05-10T11:32:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0000", 2024-05-10T11:32:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "beginning time of day to automatically balance", 2024-05-10T11:32:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-10T11:32:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_weekday": { 2024-05-10T11:32:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_weekday", 2024-05-10T11:32:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.458 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-05-10T11:32:16.458 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-05-10T11:32:16.458 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.458 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-10T11:32:16.458 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-10T11:32:16.458 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.458 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.459 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_max_iterations": { 2024-05-10T11:32:16.459 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_max_iterations", 2024-05-10T11:32:16.459 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:16.459 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.459 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.459 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "25", 2024-05-10T11:32:16.459 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-05-10T11:32:16.459 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "250", 2024-05-10T11:32:16.460 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.460 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-10T11:32:16.460 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.460 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.460 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.460 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_metrics": { 2024-05-10T11:32:16.460 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_metrics", 2024-05-10T11:32:16.460 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.461 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.461 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.461 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "pgs,objects,bytes", 2024-05-10T11:32:16.461 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.461 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.461 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.461 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-10T11:32:16.461 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-10T11:32:16.462 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.462 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.462 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_step": { 2024-05-10T11:32:16.462 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_step", 2024-05-10T11:32:16.462 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:16.462 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.462 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.462 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.5", 2024-05-10T11:32:16.463 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0.001", 2024-05-10T11:32:16.463 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "0.999", 2024-05-10T11:32:16.463 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.463 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "aggressiveness of optimization", 2024-05-10T11:32:16.463 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-10T11:32:16.463 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.463 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.464 INFO:teuthology.orchestra.run.smithi027.stdout: "end_time": { 2024-05-10T11:32:16.464 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_time", 2024-05-10T11:32:16.464 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.464 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.464 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.464 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2359", 2024-05-10T11:32:16.464 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.464 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.464 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.465 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ending time of day to automatically balance", 2024-05-10T11:32:16.465 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-10T11:32:16.465 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.465 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.465 INFO:teuthology.orchestra.run.smithi027.stdout: "end_weekday": { 2024-05-10T11:32:16.465 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_weekday", 2024-05-10T11:32:16.465 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:16.466 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.466 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.466 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.466 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-05-10T11:32:16.466 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-05-10T11:32:16.466 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.466 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-10T11:32:16.466 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-10T11:32:16.467 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.467 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.467 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.467 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.467 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.467 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.467 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.468 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.468 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.468 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.468 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.468 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.468 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.468 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.469 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.469 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.469 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.469 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.469 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.469 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.469 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.469 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.470 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.470 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.470 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.470 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.470 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.470 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.470 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.470 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.471 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.471 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.471 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.471 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.471 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.471 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.471 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.472 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.472 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.472 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.472 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.472 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.472 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.472 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.472 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.472 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.473 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.473 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.473 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.473 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.473 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.473 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.473 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.473 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.474 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.474 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.474 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.474 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.474 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.474 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.474 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.474 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.475 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.475 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.475 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.475 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.475 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.475 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.475 INFO:teuthology.orchestra.run.smithi027.stdout: "min_score": { 2024-05-10T11:32:16.476 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_score", 2024-05-10T11:32:16.476 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:16.476 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.476 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.476 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.476 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.476 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.476 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.477 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-10T11:32:16.477 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.477 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.477 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-05-10T11:32:16.477 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-05-10T11:32:16.477 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.477 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.478 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.478 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "upmap", 2024-05-10T11:32:16.478 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.478 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.478 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.478 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-compat", 2024-05-10T11:32:16.478 INFO:teuthology.orchestra.run.smithi027.stdout: "none", 2024-05-10T11:32:16.478 INFO:teuthology.orchestra.run.smithi027.stdout: "read", 2024-05-10T11:32:16.479 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap", 2024-05-10T11:32:16.479 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap-read" 2024-05-10T11:32:16.479 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.479 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Balancer mode", 2024-05-10T11:32:16.479 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.479 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.479 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.480 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ids": { 2024-05-10T11:32:16.480 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_ids", 2024-05-10T11:32:16.480 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.480 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.480 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.480 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.480 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.480 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.481 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.481 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-10T11:32:16.481 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.481 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.481 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.481 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:16.481 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:16.482 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.482 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.482 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.482 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:16.482 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.482 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.482 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.482 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-10T11:32:16.482 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.483 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.483 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.483 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.483 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.483 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.483 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.483 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.484 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.484 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.484 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.484 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.484 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.484 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.484 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.484 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.485 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_deviation": { 2024-05-10T11:32:16.485 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_deviation", 2024-05-10T11:32:16.485 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.485 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.485 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.485 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-05-10T11:32:16.485 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-05-10T11:32:16.486 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.486 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.486 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-10T11:32:16.486 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-10T11:32:16.486 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.486 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.486 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_optimizations": { 2024-05-10T11:32:16.487 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_optimizations", 2024-05-10T11:32:16.487 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:16.487 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.487 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.487 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:16.487 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.487 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.487 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.487 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-10T11:32:16.488 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.488 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.488 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.488 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.488 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.488 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm", 2024-05-10T11:32:16.489 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.489 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.489 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.489 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_down_multiplier": { 2024-05-10T11:32:16.489 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_down_multiplier", 2024-05-10T11:32:16.489 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:16.489 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.489 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.490 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-05-10T11:32:16.490 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.490 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.490 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.490 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-10T11:32:16.490 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.490 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.490 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.491 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_refresh_rate": { 2024-05-10T11:32:16.491 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_refresh_rate", 2024-05-10T11:32:16.491 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.491 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.491 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.491 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "20", 2024-05-10T11:32:16.491 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.492 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.492 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.492 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-10T11:32:16.492 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.492 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.492 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.492 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_starting_port": { 2024-05-10T11:32:16.493 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_starting_port", 2024-05-10T11:32:16.493 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.493 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.493 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.493 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4721", 2024-05-10T11:32:16.493 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.493 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.493 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.493 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-10T11:32:16.494 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.494 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.494 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.494 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_ptrace": { 2024-05-10T11:32:16.494 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_ptrace", 2024-05-10T11:32:16.494 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.494 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.495 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.495 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.495 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.495 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.495 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.495 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-10T11:32:16.495 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-10T11:32:16.495 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.496 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.496 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_interval": { 2024-05-10T11:32:16.496 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_interval", 2024-05-10T11:32:16.496 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.496 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.496 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.496 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:16.496 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.497 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.497 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.497 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-10T11:32:16.497 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.497 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.497 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.497 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_memory_target_ratio": { 2024-05-10T11:32:16.498 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_memory_target_ratio", 2024-05-10T11:32:16.498 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:16.498 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.498 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.498 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.7", 2024-05-10T11:32:16.498 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.498 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.498 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.499 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-10T11:32:16.499 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.499 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.499 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.499 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_log_destination": { 2024-05-10T11:32:16.499 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm_log_destination", 2024-05-10T11:32:16.500 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.500 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.500 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.500 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.500 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.500 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.500 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.500 INFO:teuthology.orchestra.run.smithi027.stdout: "file", 2024-05-10T11:32:16.500 INFO:teuthology.orchestra.run.smithi027.stdout: "file,syslog", 2024-05-10T11:32:16.501 INFO:teuthology.orchestra.run.smithi027.stdout: "syslog" 2024-05-10T11:32:16.501 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.501 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-10T11:32:16.501 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.501 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.501 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.501 INFO:teuthology.orchestra.run.smithi027.stdout: "cgroups_split": { 2024-05-10T11:32:16.502 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cgroups_split", 2024-05-10T11:32:16.502 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.502 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.502 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.502 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.502 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.502 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.502 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.503 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-10T11:32:16.503 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.503 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.503 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.503 INFO:teuthology.orchestra.run.smithi027.stdout: "config_checks_enabled": { 2024-05-10T11:32:16.503 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_checks_enabled", 2024-05-10T11:32:16.503 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.504 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.504 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.504 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.504 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.504 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.504 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.504 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-10T11:32:16.504 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.505 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.505 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.505 INFO:teuthology.orchestra.run.smithi027.stdout: "config_dashboard": { 2024-05-10T11:32:16.505 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_dashboard", 2024-05-10T11:32:16.505 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.505 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.505 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.505 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.506 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.506 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.506 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.506 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-10T11:32:16.506 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.506 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.506 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.507 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_alertmanager": { 2024-05-10T11:32:16.507 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_alertmanager", 2024-05-10T11:32:16.507 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.507 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.507 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.507 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-10T11:32:16.507 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.507 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.508 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.508 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-05-10T11:32:16.508 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.508 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.508 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.508 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_base": { 2024-05-10T11:32:16.508 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_base", 2024-05-10T11:32:16.509 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.509 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.509 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.509 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-10T11:32:16.509 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.509 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.509 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.509 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Container image name, without the tag", 2024-05-10T11:32:16.510 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.510 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.510 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.510 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_elasticsearch": { 2024-05-10T11:32:16.510 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_elasticsearch", 2024-05-10T11:32:16.510 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.510 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.510 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.511 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-10T11:32:16.511 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.511 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.511 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.511 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "elasticsearch container image", 2024-05-10T11:32:16.511 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.511 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.512 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.512 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_grafana": { 2024-05-10T11:32:16.512 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_grafana", 2024-05-10T11:32:16.512 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.512 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.512 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.512 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-10T11:32:16.512 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.513 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.513 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.513 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-05-10T11:32:16.513 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.513 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.513 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.513 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_haproxy": { 2024-05-10T11:32:16.514 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_haproxy", 2024-05-10T11:32:16.514 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.514 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.514 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.514 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-10T11:32:16.514 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.514 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.514 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.514 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "HAproxy container image", 2024-05-10T11:32:16.515 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.515 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.515 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.515 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_agent": { 2024-05-10T11:32:16.515 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_agent", 2024-05-10T11:32:16.515 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.515 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.516 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.516 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-10T11:32:16.516 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.516 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.516 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.516 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger agent container image", 2024-05-10T11:32:16.516 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.516 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.517 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.517 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_collector": { 2024-05-10T11:32:16.517 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_collector", 2024-05-10T11:32:16.517 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.517 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.517 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.517 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-10T11:32:16.517 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.518 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.518 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.518 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger collector container image", 2024-05-10T11:32:16.518 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.518 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.518 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.518 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_query": { 2024-05-10T11:32:16.519 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_query", 2024-05-10T11:32:16.519 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.519 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.519 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.519 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-10T11:32:16.519 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.519 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.519 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.520 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger query container image", 2024-05-10T11:32:16.520 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.520 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.520 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.520 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_keepalived": { 2024-05-10T11:32:16.520 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_keepalived", 2024-05-10T11:32:16.520 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.521 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.521 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.521 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-10T11:32:16.521 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.521 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.521 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.521 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Keepalived container image", 2024-05-10T11:32:16.521 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.521 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.522 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.522 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_loki": { 2024-05-10T11:32:16.522 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_loki", 2024-05-10T11:32:16.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.522 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.522 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.523 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-10T11:32:16.523 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.523 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.523 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.523 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Loki container image", 2024-05-10T11:32:16.523 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.523 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.523 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.524 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_node_exporter": { 2024-05-10T11:32:16.524 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_node_exporter", 2024-05-10T11:32:16.524 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.524 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.524 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.524 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-10T11:32:16.524 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.524 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.525 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.525 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-05-10T11:32:16.525 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.525 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.525 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.525 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_nvmeof": { 2024-05-10T11:32:16.525 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_nvmeof", 2024-05-10T11:32:16.526 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.526 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.526 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.526 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-10T11:32:16.526 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.526 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.526 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.526 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Nvme-of container image", 2024-05-10T11:32:16.527 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.527 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.527 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.527 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_prometheus": { 2024-05-10T11:32:16.527 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_prometheus", 2024-05-10T11:32:16.527 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.527 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.528 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.528 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-10T11:32:16.528 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.528 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.528 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.528 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-05-10T11:32:16.528 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.528 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.529 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.529 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_promtail": { 2024-05-10T11:32:16.529 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_promtail", 2024-05-10T11:32:16.529 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.529 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.529 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.529 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-10T11:32:16.530 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.530 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.530 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.530 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Promtail container image", 2024-05-10T11:32:16.530 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.530 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.530 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.531 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_samba": { 2024-05-10T11:32:16.531 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_samba", 2024-05-10T11:32:16.531 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.531 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.531 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.531 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-10T11:32:16.531 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.531 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.531 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.532 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Samba/SMB container image", 2024-05-10T11:32:16.532 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.532 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.532 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.532 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_snmp_gateway": { 2024-05-10T11:32:16.532 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_snmp_gateway", 2024-05-10T11:32:16.532 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.533 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.533 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.533 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-10T11:32:16.533 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.533 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.533 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.533 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SNMP Gateway container image", 2024-05-10T11:32:16.533 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.534 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.534 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.534 INFO:teuthology.orchestra.run.smithi027.stdout: "container_init": { 2024-05-10T11:32:16.534 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_init", 2024-05-10T11:32:16.534 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.534 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.534 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.535 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.535 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.535 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.535 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.535 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Run podman/docker with `--init`", 2024-05-10T11:32:16.535 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.535 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.535 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.536 INFO:teuthology.orchestra.run.smithi027.stdout: "daemon_cache_timeout": { 2024-05-10T11:32:16.536 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "daemon_cache_timeout", 2024-05-10T11:32:16.536 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.536 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.536 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.536 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:16.536 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.537 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.537 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.537 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-10T11:32:16.537 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.537 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.537 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.537 INFO:teuthology.orchestra.run.smithi027.stdout: "default_cephadm_command_timeout": { 2024-05-10T11:32:16.538 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_cephadm_command_timeout", 2024-05-10T11:32:16.538 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.538 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.538 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.538 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "900", 2024-05-10T11:32:16.538 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.538 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.538 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.539 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-10T11:32:16.539 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.539 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.539 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.539 INFO:teuthology.orchestra.run.smithi027.stdout: "default_registry": { 2024-05-10T11:32:16.539 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_registry", 2024-05-10T11:32:16.539 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.539 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.540 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.540 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io", 2024-05-10T11:32:16.540 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.540 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.540 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.540 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-10T11:32:16.540 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.540 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.541 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.541 INFO:teuthology.orchestra.run.smithi027.stdout: "device_cache_timeout": { 2024-05-10T11:32:16.541 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_cache_timeout", 2024-05-10T11:32:16.541 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.541 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.541 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.541 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1800", 2024-05-10T11:32:16.541 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.542 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.542 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.542 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache device inventory", 2024-05-10T11:32:16.542 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.542 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.542 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.542 INFO:teuthology.orchestra.run.smithi027.stdout: "device_enhanced_scan": { 2024-05-10T11:32:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_enhanced_scan", 2024-05-10T11:32:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-10T11:32:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: "facts_cache_timeout": { 2024-05-10T11:32:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "facts_cache_timeout", 2024-05-10T11:32:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache host facts data", 2024-05-10T11:32:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: "grafana_dashboards_path": { 2024-05-10T11:32:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "grafana_dashboards_path", 2024-05-10T11:32:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-10T11:32:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-10T11:32:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: "host_check_interval": { 2024-05-10T11:32:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "host_check_interval", 2024-05-10T11:32:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to perform a host check", 2024-05-10T11:32:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: "hw_monitoring": { 2024-05-10T11:32:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hw_monitoring", 2024-05-10T11:32:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-10T11:32:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: "inventory_list_all": { 2024-05-10T11:32:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "inventory_list_all", 2024-05-10T11:32:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-10T11:32:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.554 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.554 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.554 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.554 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.554 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.554 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.554 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.554 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.555 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.555 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.555 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.555 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.555 INFO:teuthology.orchestra.run.smithi027.stdout: "log_refresh_metadata": { 2024-05-10T11:32:16.555 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_refresh_metadata", 2024-05-10T11:32:16.555 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.556 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.556 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.556 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.556 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.556 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.556 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.556 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-10T11:32:16.556 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.557 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.557 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.557 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.557 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.557 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.557 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.557 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.557 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.558 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.558 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.558 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.558 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-10T11:32:16.558 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.558 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.558 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.559 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.559 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.559 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.559 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.559 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.559 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.559 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.559 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.560 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.560 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.560 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.560 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.560 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.560 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.560 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.560 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.560 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.561 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.561 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.561 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.561 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.561 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.561 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.561 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.562 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.562 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.562 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.562 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.562 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.562 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.562 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.562 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.563 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.563 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-10T11:32:16.563 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-10T11:32:16.563 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.563 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.563 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.563 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.563 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.564 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.564 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.564 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-10T11:32:16.564 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.564 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.564 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.564 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-10T11:32:16.565 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-10T11:32:16.565 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.565 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.565 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.565 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "*", 2024-05-10T11:32:16.565 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.565 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.565 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.565 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-10T11:32:16.566 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.566 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.566 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.566 INFO:teuthology.orchestra.run.smithi027.stdout: "max_count_per_host": { 2024-05-10T11:32:16.566 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_count_per_host", 2024-05-10T11:32:16.566 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.566 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of daemons per service per host", 2024-05-10T11:32:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.568 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.568 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd_draining_count": { 2024-05-10T11:32:16.568 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_osd_draining_count", 2024-05-10T11:32:16.568 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.568 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.568 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.568 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:16.568 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.569 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.569 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.569 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-10T11:32:16.569 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.569 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.569 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.569 INFO:teuthology.orchestra.run.smithi027.stdout: "migration_current": { 2024-05-10T11:32:16.569 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "migration_current", 2024-05-10T11:32:16.570 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.570 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.570 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.570 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.570 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.570 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.570 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.570 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "internal - do not modify", 2024-05-10T11:32:16.571 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.571 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.571 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.571 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-05-10T11:32:16.571 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-05-10T11:32:16.571 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.571 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.572 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.572 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "root", 2024-05-10T11:32:16.572 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.572 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.572 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.572 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm-package", 2024-05-10T11:32:16.572 INFO:teuthology.orchestra.run.smithi027.stdout: "root" 2024-05-10T11:32:16.572 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.572 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "mode for remote execution of cephadm", 2024-05-10T11:32:16.573 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.573 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.573 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.573 INFO:teuthology.orchestra.run.smithi027.stdout: "oob_default_addr": { 2024-05-10T11:32:16.573 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "oob_default_addr", 2024-05-10T11:32:16.573 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.573 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.574 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.574 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "169.254.1.1", 2024-05-10T11:32:16.574 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.574 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.574 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.574 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-10T11:32:16.574 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.575 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.575 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.575 INFO:teuthology.orchestra.run.smithi027.stdout: "prometheus_alerts_path": { 2024-05-10T11:32:16.575 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus_alerts_path", 2024-05-10T11:32:16.575 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.575 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.575 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.576 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-10T11:32:16.576 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.576 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.576 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.576 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-10T11:32:16.576 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.576 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.576 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.577 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_insecure": { 2024-05-10T11:32:16.577 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_insecure", 2024-05-10T11:32:16.577 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.577 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.577 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.577 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.577 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.577 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.578 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.578 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-10T11:32:16.578 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.578 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.578 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.578 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_password": { 2024-05-10T11:32:16.578 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_password", 2024-05-10T11:32:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-10T11:32:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_url": { 2024-05-10T11:32:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_url", 2024-05-10T11:32:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-10T11:32:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.582 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_username": { 2024-05-10T11:32:16.582 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_username", 2024-05-10T11:32:16.582 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.582 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.582 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.582 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.582 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.583 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.583 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.583 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-10T11:32:16.583 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.583 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.583 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.583 INFO:teuthology.orchestra.run.smithi027.stdout: "secure_monitoring_stack": { 2024-05-10T11:32:16.583 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "secure_monitoring_stack", 2024-05-10T11:32:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-10T11:32:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "service_discovery_port": { 2024-05-10T11:32:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "service_discovery_port", 2024-05-10T11:32:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.586 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.586 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8765", 2024-05-10T11:32:16.586 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.586 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.586 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.586 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "cephadm service discovery port", 2024-05-10T11:32:16.586 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.586 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.587 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.587 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.587 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.587 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.587 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.587 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.587 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.588 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.588 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.588 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.588 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.588 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.588 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.588 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.589 INFO:teuthology.orchestra.run.smithi027.stdout: "ssh_config_file": { 2024-05-10T11:32:16.589 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssh_config_file", 2024-05-10T11:32:16.589 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.589 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.589 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.589 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.589 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.589 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.589 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.590 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-10T11:32:16.590 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.590 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.590 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.590 INFO:teuthology.orchestra.run.smithi027.stdout: "use_agent": { 2024-05-10T11:32:16.590 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_agent", 2024-05-10T11:32:16.590 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.591 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.591 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.591 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.591 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.591 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.591 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.591 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-10T11:32:16.591 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.592 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.592 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.592 INFO:teuthology.orchestra.run.smithi027.stdout: "use_repo_digest": { 2024-05-10T11:32:16.592 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_repo_digest", 2024-05-10T11:32:16.592 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.592 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.592 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.592 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.593 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.593 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.593 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.593 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-10T11:32:16.593 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.593 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.593 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_failed_host_check": { 2024-05-10T11:32:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_failed_host_check", 2024-05-10T11:32:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if the host check fails", 2024-05-10T11:32:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_daemons": { 2024-05-10T11:32:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_daemons", 2024-05-10T11:32:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-10T11:32:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_hosts": { 2024-05-10T11:32:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_hosts", 2024-05-10T11:32:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-10T11:32:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crash", 2024-05-10T11:32:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.607 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.607 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.607 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.607 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.607 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.607 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.607 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.607 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.607 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.608 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.608 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.608 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.608 INFO:teuthology.orchestra.run.smithi027.stdout: "retain_interval": { 2024-05-10T11:32:16.608 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retain_interval", 2024-05-10T11:32:16.608 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.608 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.609 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.609 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "31536000", 2024-05-10T11:32:16.609 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.609 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.609 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.609 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-10T11:32:16.609 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.609 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.610 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.610 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.610 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.610 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.610 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.610 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.610 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.611 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.611 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.611 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.611 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.611 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.611 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.611 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.611 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_recent_interval": { 2024-05-10T11:32:16.612 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_recent_interval", 2024-05-10T11:32:16.612 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.612 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.612 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.612 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1209600", 2024-05-10T11:32:16.612 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.612 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.612 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.613 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-10T11:32:16.613 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.613 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.613 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.613 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.613 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.613 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dashboard", 2024-05-10T11:32:16.614 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.614 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.614 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.614 INFO:teuthology.orchestra.run.smithi027.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-10T11:32:16.614 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-10T11:32:16.614 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.614 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.615 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.615 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:16.615 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.615 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.615 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.615 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.615 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.615 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.615 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.616 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_HOST": { 2024-05-10T11:32:16.616 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-10T11:32:16.616 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.616 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.616 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.616 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.616 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.617 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.617 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.617 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.617 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.617 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.617 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.617 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-10T11:32:16.618 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-10T11:32:16.618 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.618 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.618 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.618 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.618 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.618 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.618 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.619 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.619 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.619 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.619 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.619 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_ENABLED": { 2024-05-10T11:32:16.619 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_ENABLED", 2024-05-10T11:32:16.619 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.621 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.621 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.621 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-10T11:32:16.621 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-10T11:32:16.621 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.621 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.621 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.621 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.622 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.622 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.622 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.622 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.622 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.622 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.622 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.623 INFO:teuthology.orchestra.run.smithi027.stdout: "ENABLE_BROWSABLE_API": { 2024-05-10T11:32:16.623 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-10T11:32:16.623 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.623 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.623 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.623 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.623 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.623 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.624 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.624 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.624 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.624 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.624 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.624 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-10T11:32:16.624 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-10T11:32:16.625 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.625 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.625 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.625 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.625 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.625 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.625 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.625 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.625 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.626 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.626 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.626 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-10T11:32:16.626 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-10T11:32:16.626 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.626 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.626 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.627 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.627 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.627 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.627 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.627 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.627 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.627 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.627 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.628 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-10T11:32:16.628 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-10T11:32:16.628 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.628 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.628 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.628 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.628 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.629 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.629 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.629 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.629 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.629 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.629 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.629 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-10T11:32:16.629 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-10T11:32:16.630 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.630 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.630 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.630 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.630 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.630 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.630 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.630 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.631 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.631 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.631 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.631 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-10T11:32:16.631 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-10T11:32:16.631 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.631 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.632 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.632 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.632 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.632 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.632 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.632 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.632 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.632 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.632 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.633 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-10T11:32:16.633 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-10T11:32:16.633 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.633 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.633 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.633 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.633 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.633 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.634 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.634 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.634 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.634 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.634 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.634 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-10T11:32:16.634 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-10T11:32:16.635 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.635 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.635 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.635 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.635 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.635 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.635 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.635 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.636 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.636 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.636 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.636 INFO:teuthology.orchestra.run.smithi027.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-10T11:32:16.636 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-10T11:32:16.636 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.636 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.637 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.637 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.637 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.637 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.637 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.637 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.637 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.637 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.638 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.638 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_PASSWORD": { 2024-05-10T11:32:16.638 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-10T11:32:16.638 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.638 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.638 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.638 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-05-10T11:32:16.639 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.639 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.639 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.639 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.639 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.639 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.639 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.639 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-10T11:32:16.640 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-10T11:32:16.640 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.640 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.640 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.640 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.640 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.640 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.641 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.641 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.641 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.641 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.641 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.641 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_URL": { 2024-05-10T11:32:16.641 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_URL", 2024-05-10T11:32:16.641 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.642 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.642 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.642 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.642 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.642 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.642 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.642 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.642 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.643 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.643 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.643 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_USERNAME": { 2024-05-10T11:32:16.643 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-10T11:32:16.643 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.643 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.644 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.644 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-05-10T11:32:16.644 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.644 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.644 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.644 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.644 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.644 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.644 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.645 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-10T11:32:16.645 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-10T11:32:16.645 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.645 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.645 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.645 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.645 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.646 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.646 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.646 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.646 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.646 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.646 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.646 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-10T11:32:16.647 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-10T11:32:16.647 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.647 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.647 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.647 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.647 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.647 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.647 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.647 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.648 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.648 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.648 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.650 INFO:teuthology.orchestra.run.smithi027.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-10T11:32:16.650 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-10T11:32:16.650 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.650 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.650 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.650 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.650 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.651 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.651 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.651 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.651 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.651 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.651 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.651 INFO:teuthology.orchestra.run.smithi027.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-10T11:32:16.652 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-10T11:32:16.652 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.652 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.652 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.652 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.652 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.652 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.652 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.652 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.653 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.653 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.653 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.653 INFO:teuthology.orchestra.run.smithi027.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-10T11:32:16.653 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-10T11:32:16.653 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.653 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.654 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.654 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "[]", 2024-05-10T11:32:16.654 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.654 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.654 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.654 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.654 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.654 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.655 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.655 INFO:teuthology.orchestra.run.smithi027.stdout: "MULTICLUSTER_CONFIG": { 2024-05-10T11:32:16.655 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-10T11:32:16.655 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.655 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.655 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.655 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "{}", 2024-05-10T11:32:16.655 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.656 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.656 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.656 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.656 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.656 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.656 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.656 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_HOST": { 2024-05-10T11:32:16.657 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-10T11:32:16.657 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.657 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.657 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.657 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.657 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.657 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.657 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.658 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.658 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.658 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.658 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.658 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-10T11:32:16.658 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-10T11:32:16.658 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.659 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.659 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.659 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.659 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.659 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.659 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.659 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.659 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.660 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.660 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.660 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-10T11:32:16.660 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-10T11:32:16.660 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.660 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.660 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.661 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.661 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.661 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.661 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.661 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.661 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.661 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.661 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.662 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-10T11:32:16.662 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-10T11:32:16.662 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.662 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.662 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.662 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.662 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.662 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.663 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.663 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.663 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.663 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.663 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.663 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-10T11:32:16.663 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-10T11:32:16.664 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.664 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.664 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.664 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.664 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.664 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.664 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.664 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.665 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.665 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.665 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.665 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-10T11:32:16.665 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-10T11:32:16.665 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.665 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.666 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.666 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.666 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.666 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.666 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.666 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.666 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.666 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.667 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.667 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-10T11:32:16.667 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-10T11:32:16.667 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.667 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.667 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.667 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.668 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.668 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.668 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.668 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.668 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.668 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.668 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.669 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-10T11:32:16.669 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-10T11:32:16.669 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.669 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.669 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.669 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.669 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.669 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.669 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.670 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.670 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.670 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.670 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.670 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-10T11:32:16.670 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-10T11:32:16.670 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.671 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.671 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.671 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.671 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.671 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.671 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.671 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.671 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.672 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.672 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.672 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_ENABLED": { 2024-05-10T11:32:16.672 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-10T11:32:16.672 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.672 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.672 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.672 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.673 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.673 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.673 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.673 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.673 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.673 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.673 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.674 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-10T11:32:16.674 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-10T11:32:16.674 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.674 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.674 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.674 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-10T11:32:16.674 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.674 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.675 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.675 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.675 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.675 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.675 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.675 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-10T11:32:16.675 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-10T11:32:16.675 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.676 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.676 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.676 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:16.676 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.676 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.676 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.676 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.676 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.677 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.677 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.677 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-10T11:32:16.677 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-10T11:32:16.677 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.677 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.677 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.677 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8", 2024-05-10T11:32:16.678 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.678 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.678 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.678 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.678 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.678 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.678 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.679 INFO:teuthology.orchestra.run.smithi027.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-10T11:32:16.679 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-10T11:32:16.679 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.679 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.679 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.679 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "45", 2024-05-10T11:32:16.679 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.679 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.680 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.680 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.680 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.680 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.680 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.680 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ACCESS_KEY": { 2024-05-10T11:32:16.680 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-10T11:32:16.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.681 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.681 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.681 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.681 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.681 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.681 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.681 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.681 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.682 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.682 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.682 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-10T11:32:16.682 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-10T11:32:16.682 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.682 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.682 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.683 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-05-10T11:32:16.683 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.683 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.683 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.683 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.683 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.683 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.683 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.684 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SECRET_KEY": { 2024-05-10T11:32:16.684 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-10T11:32:16.684 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.684 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.684 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.684 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.684 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.685 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.685 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.685 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.685 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.685 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.685 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.685 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SSL_VERIFY": { 2024-05-10T11:32:16.686 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-10T11:32:16.686 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.686 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.686 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.686 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.686 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.686 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.686 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.687 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.687 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.687 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.687 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.687 INFO:teuthology.orchestra.run.smithi027.stdout: "UNSAFE_TLS_v1_2": { 2024-05-10T11:32:16.687 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-10T11:32:16.687 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.688 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.688 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.688 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.688 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.688 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.688 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.688 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.688 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.689 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.689 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.689 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-10T11:32:16.689 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-10T11:32:16.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.689 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.689 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.689 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.690 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.690 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.690 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.690 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.690 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.690 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.690 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.691 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-10T11:32:16.691 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-10T11:32:16.691 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.691 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.691 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.691 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:16.691 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.691 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.692 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.692 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.692 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.692 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.692 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.692 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-10T11:32:16.692 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-10T11:32:16.693 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.693 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.693 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.693 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-05-10T11:32:16.693 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.693 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.693 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.693 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.694 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.694 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.694 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.694 INFO:teuthology.orchestra.run.smithi027.stdout: "cross_origin_url": { 2024-05-10T11:32:16.694 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cross_origin_url", 2024-05-10T11:32:16.694 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.694 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.695 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.695 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.695 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.695 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.695 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.695 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.695 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.695 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.695 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.696 INFO:teuthology.orchestra.run.smithi027.stdout: "crt_file": { 2024-05-10T11:32:16.696 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crt_file", 2024-05-10T11:32:16.696 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.696 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.696 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.696 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.696 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.697 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.697 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.697 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.697 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.697 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.697 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.697 INFO:teuthology.orchestra.run.smithi027.stdout: "debug": { 2024-05-10T11:32:16.698 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "debug", 2024-05-10T11:32:16.698 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.698 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.698 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.698 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.698 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.698 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.698 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.699 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable/disable debug options", 2024-05-10T11:32:16.699 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.699 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.699 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.699 INFO:teuthology.orchestra.run.smithi027.stdout: "jwt_token_ttl": { 2024-05-10T11:32:16.699 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "jwt_token_ttl", 2024-05-10T11:32:16.699 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.699 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.700 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.700 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "28800", 2024-05-10T11:32:16.700 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.700 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.700 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.700 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.700 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.700 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.701 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.701 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-05-10T11:32:16.701 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-05-10T11:32:16.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.701 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.701 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.701 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.701 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.702 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.702 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.702 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.702 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.702 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.702 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.702 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.703 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.703 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.703 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.703 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.703 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.703 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.703 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.703 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.704 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.704 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.704 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.704 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.704 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.704 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.704 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.704 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.705 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.705 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.705 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.705 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.705 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.705 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.705 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.705 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.706 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.706 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.706 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.706 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.706 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.706 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.706 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.707 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.707 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.707 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.707 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.707 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.707 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.707 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.707 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.708 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.708 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.708 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.708 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.708 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.708 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.708 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.708 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.709 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.709 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.709 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.709 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.709 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.709 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.709 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.710 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.710 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.710 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.710 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.710 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.710 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.710 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.710 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.711 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.711 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.711 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.711 INFO:teuthology.orchestra.run.smithi027.stdout: "motd": { 2024-05-10T11:32:16.711 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "motd", 2024-05-10T11:32:16.711 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.711 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.712 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.712 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.712 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.712 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.712 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.712 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "The message of the day", 2024-05-10T11:32:16.712 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.712 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.713 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.713 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect_resolve_ip_addr": { 2024-05-10T11:32:16.713 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "redirect_resolve_ip_addr", 2024-05-10T11:32:16.713 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.713 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.713 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.713 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.713 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.714 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.714 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.714 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.714 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.714 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.714 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.714 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-05-10T11:32:16.715 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-05-10T11:32:16.715 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.715 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.715 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.715 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-05-10T11:32:16.715 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.715 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.716 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.716 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.716 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.716 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.716 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.716 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-05-10T11:32:16.716 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-05-10T11:32:16.717 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.717 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.717 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.717 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8080", 2024-05-10T11:32:16.717 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.717 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.717 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.717 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.718 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.718 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.718 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.718 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.718 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.718 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.718 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.719 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.719 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.719 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.719 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.719 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.719 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.719 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.719 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.720 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.720 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-05-10T11:32:16.720 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-05-10T11:32:16.720 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.720 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.720 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.720 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.721 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.721 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.721 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.721 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.721 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.721 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.721 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.722 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl_server_port": { 2024-05-10T11:32:16.722 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl_server_port", 2024-05-10T11:32:16.722 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.722 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.722 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.722 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8443", 2024-05-10T11:32:16.722 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.722 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.723 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.723 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.723 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.723 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.723 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.723 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-05-10T11:32:16.723 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-05-10T11:32:16.724 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.724 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.724 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.724 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "redirect", 2024-05-10T11:32:16.724 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.724 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.724 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.724 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.725 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect" 2024-05-10T11:32:16.725 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.725 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.725 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.725 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.725 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.725 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-05-10T11:32:16.726 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-05-10T11:32:16.726 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.726 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.726 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.726 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-05-10T11:32:16.726 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-05-10T11:32:16.726 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-05-10T11:32:16.726 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.726 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.727 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.727 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.727 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.727 INFO:teuthology.orchestra.run.smithi027.stdout: "url_prefix": { 2024-05-10T11:32:16.727 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url_prefix", 2024-05-10T11:32:16.727 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.727 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.728 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.728 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.728 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.728 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.728 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.728 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.728 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.728 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.728 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.729 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.729 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.729 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.729 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-05-10T11:32:16.729 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.729 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.729 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.729 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_monitoring": { 2024-05-10T11:32:16.730 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_monitoring", 2024-05-10T11:32:16.730 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.730 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.730 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.730 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.730 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.730 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.730 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.731 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "monitor device health metrics", 2024-05-10T11:32:16.731 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.731 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.731 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.731 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.731 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.731 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.732 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.732 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.732 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.732 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.732 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.732 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.732 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.732 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.733 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.733 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.733 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.733 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.733 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.733 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.733 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.733 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.734 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.734 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.734 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.734 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.734 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.734 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.734 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.734 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.735 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.735 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.735 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.735 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.735 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.735 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.735 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.736 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.736 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.736 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.736 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.736 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.736 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.736 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.736 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.736 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.737 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.737 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.737 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.737 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.737 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.737 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.737 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.737 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.738 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.738 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.738 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.738 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.738 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.738 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.738 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.739 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.739 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.739 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.739 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.739 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.739 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.739 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.739 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.740 INFO:teuthology.orchestra.run.smithi027.stdout: "mark_out_threshold": { 2024-05-10T11:32:16.740 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mark_out_threshold", 2024-05-10T11:32:16.740 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.740 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.740 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.740 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2419200", 2024-05-10T11:32:16.740 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.740 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.741 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.741 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-10T11:32:16.741 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.741 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.741 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.741 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": { 2024-05-10T11:32:16.741 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_name", 2024-05-10T11:32:16.742 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.742 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.742 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.742 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "device_health_metrics", 2024-05-10T11:32:16.742 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.742 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.742 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.742 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-10T11:32:16.743 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.743 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.743 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.743 INFO:teuthology.orchestra.run.smithi027.stdout: "retention_period": { 2024-05-10T11:32:16.743 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retention_period", 2024-05-10T11:32:16.743 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.743 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15552000", 2024-05-10T11:32:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain device health metrics", 2024-05-10T11:32:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.745 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_frequency": { 2024-05-10T11:32:16.745 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_frequency", 2024-05-10T11:32:16.745 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.745 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.745 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.745 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-05-10T11:32:16.745 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.746 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.746 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.746 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-10T11:32:16.746 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.746 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.746 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.746 INFO:teuthology.orchestra.run.smithi027.stdout: "self_heal": { 2024-05-10T11:32:16.746 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "self_heal", 2024-05-10T11:32:16.747 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.747 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.747 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.747 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.747 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.747 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.747 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.747 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-10T11:32:16.748 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.748 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.748 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.748 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:16.748 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:16.748 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.748 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.748 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.749 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:16.749 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.749 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.749 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.749 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and check device health", 2024-05-10T11:32:16.749 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.749 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.749 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.750 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.750 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.750 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.750 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.750 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.750 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.750 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.751 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.751 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.751 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.751 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.751 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.751 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.751 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_threshold": { 2024-05-10T11:32:16.751 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_threshold", 2024-05-10T11:32:16.752 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.752 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.752 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.752 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7257600", 2024-05-10T11:32:16.752 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.752 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.752 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.752 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-10T11:32:16.753 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.753 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.753 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.753 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.753 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.753 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "diskprediction_local", 2024-05-10T11:32:16.753 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.754 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.754 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.754 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.754 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.754 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.754 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.754 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.755 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.755 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.755 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.755 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.755 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.755 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.755 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.755 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.756 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.756 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.756 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.756 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.756 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.756 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.756 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.757 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.757 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.757 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.757 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.757 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.757 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.757 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.757 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.758 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.758 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.758 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.758 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.758 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.758 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.758 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.759 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.759 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.759 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.759 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.759 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.759 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.759 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.759 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.760 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.760 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.760 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.760 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.760 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.760 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.760 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.760 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.761 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.761 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.761 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.761 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.761 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.761 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.761 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.761 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.762 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.762 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.762 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.762 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.762 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.762 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.762 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.763 INFO:teuthology.orchestra.run.smithi027.stdout: "predict_interval": { 2024-05-10T11:32:16.763 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predict_interval", 2024-05-10T11:32:16.763 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.763 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.763 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.763 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-05-10T11:32:16.763 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.763 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.764 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.764 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.764 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.764 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.764 INFO:teuthology.orchestra.run.smithi027.stdout: "predictor_model": { 2024-05-10T11:32:16.764 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predictor_model", 2024-05-10T11:32:16.765 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.765 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.765 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.765 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "prophetstor", 2024-05-10T11:32:16.765 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.765 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.765 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.765 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.765 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.766 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.766 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.766 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:16.766 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:16.766 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.766 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.766 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.767 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:16.767 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.767 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.767 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.767 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.767 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.767 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.767 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.768 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.768 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.768 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.768 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.768 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.768 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.768 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.768 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.769 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.769 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.769 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.769 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.769 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.769 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.769 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.770 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "influx", 2024-05-10T11:32:16.770 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": false, 2024-05-10T11:32:16.770 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "influxdb python module not found", 2024-05-10T11:32:16.770 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.770 INFO:teuthology.orchestra.run.smithi027.stdout: "batch_size": { 2024-05-10T11:32:16.770 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "batch_size", 2024-05-10T11:32:16.770 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.770 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.771 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.771 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5000", 2024-05-10T11:32:16.771 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.771 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.771 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.771 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-10T11:32:16.771 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.771 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.772 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.772 INFO:teuthology.orchestra.run.smithi027.stdout: "database": { 2024-05-10T11:32:16.772 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "database", 2024-05-10T11:32:16.772 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.772 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.772 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.772 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "ceph", 2024-05-10T11:32:16.772 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.773 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.773 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.773 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-10T11:32:16.773 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.773 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.773 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.773 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": { 2024-05-10T11:32:16.774 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hostname", 2024-05-10T11:32:16.774 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.774 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.774 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.774 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.774 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.774 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.774 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.774 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server hostname", 2024-05-10T11:32:16.775 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.775 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.775 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.775 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:16.775 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:16.775 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.775 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.776 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.776 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "30", 2024-05-10T11:32:16.776 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "5", 2024-05-10T11:32:16.776 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.776 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.776 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-10T11:32:16.776 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.776 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.776 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.777 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.777 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.777 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.777 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.777 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.777 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.777 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.778 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.778 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.778 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.778 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.778 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.778 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.778 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.778 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.779 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.779 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.779 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.779 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.779 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.779 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.779 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.779 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.780 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.780 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.780 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.780 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.780 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.780 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.780 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.780 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.781 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.781 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.781 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.781 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.781 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.781 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.781 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.782 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.782 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.782 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.782 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.782 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.782 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.782 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.782 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: "password": { 2024-05-10T11:32:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "password", 2024-05-10T11:32:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "password of InfluxDB server user", 2024-05-10T11:32:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: "port": { 2024-05-10T11:32:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "port", 2024-05-10T11:32:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8086", 2024-05-10T11:32:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server port", 2024-05-10T11:32:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-05-10T11:32:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-05-10T11:32:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "false", 2024-05-10T11:32:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-10T11:32:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: "threads": { 2024-05-10T11:32:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threads", 2024-05-10T11:32:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-05-10T11:32:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-05-10T11:32:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "32", 2024-05-10T11:32:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-10T11:32:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: "username": { 2024-05-10T11:32:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "username", 2024-05-10T11:32:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "username of InfluxDB server user", 2024-05-10T11:32:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: "verify_ssl": { 2024-05-10T11:32:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "verify_ssl", 2024-05-10T11:32:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "true", 2024-05-10T11:32:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.797 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.797 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-10T11:32:16.797 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.797 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.797 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.797 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.797 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.798 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "insights", 2024-05-10T11:32:16.798 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.798 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.798 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.798 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.798 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.798 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.798 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.799 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.799 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.799 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.799 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.799 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.799 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.799 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.799 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.799 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.800 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.800 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.800 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.800 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.800 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.800 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.800 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.801 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.801 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.801 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.801 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.801 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.801 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.801 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.801 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.802 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.802 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.802 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.802 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.802 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.802 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.802 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.803 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.803 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.803 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.803 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.803 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.803 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.803 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.803 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.803 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.804 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.804 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.804 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.804 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.804 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.804 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.804 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.804 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.805 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.805 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.805 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.805 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.805 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.805 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.805 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.805 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.806 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.806 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.806 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.806 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.806 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.806 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.806 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.807 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.807 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.807 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.807 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.807 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.807 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.807 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.807 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.808 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.808 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.808 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.808 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.808 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.808 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.808 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "iostat", 2024-05-10T11:32:16.809 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.809 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.809 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.809 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.809 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.809 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.809 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.809 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.810 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.810 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.810 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.810 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.810 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.810 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.810 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.810 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.810 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.811 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.811 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.811 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.811 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.811 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.811 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.811 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.812 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.812 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.812 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.812 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.812 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.812 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.812 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.812 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.812 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.813 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.813 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.813 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.813 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.813 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.813 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.813 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.814 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.814 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.814 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.814 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.814 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.814 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.814 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.814 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.817 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.817 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.817 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.817 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.817 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.817 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.817 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.818 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.818 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.818 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.818 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.818 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.818 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.818 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.818 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.819 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.819 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.819 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.819 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.819 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.819 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "k8sevents", 2024-05-10T11:32:16.820 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.820 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.820 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.820 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_event_retention_days": { 2024-05-10T11:32:16.820 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ceph_event_retention_days", 2024-05-10T11:32:16.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.820 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.820 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.820 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7", 2024-05-10T11:32:16.821 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.821 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.821 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.821 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-10T11:32:16.821 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.821 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.821 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.822 INFO:teuthology.orchestra.run.smithi027.stdout: "config_check_secs": { 2024-05-10T11:32:16.822 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_check_secs", 2024-05-10T11:32:16.822 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.822 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.822 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.822 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:16.822 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "10", 2024-05-10T11:32:16.822 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.822 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.823 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-10T11:32:16.823 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.823 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.823 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.823 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.823 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.824 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.824 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.824 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.824 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.824 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.824 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.824 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.824 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.824 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.825 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.825 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.825 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.825 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.825 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.825 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.825 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.825 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.826 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.826 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.826 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.826 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.826 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.826 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.826 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.826 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.827 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.827 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.827 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.827 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.827 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.827 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.827 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.828 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.828 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.828 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.828 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.828 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.828 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.828 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.828 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.828 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.829 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.829 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.829 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.829 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.829 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.829 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.829 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.829 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.830 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.830 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.830 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.830 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.830 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.830 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.830 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.831 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.831 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.831 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.831 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.831 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.831 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.831 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.831 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.832 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.832 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.832 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.832 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.832 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.832 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.832 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.832 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.833 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.833 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.833 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.833 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.833 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.833 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.833 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.833 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.834 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "localpool", 2024-05-10T11:32:16.834 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.834 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.834 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.834 INFO:teuthology.orchestra.run.smithi027.stdout: "failure_domain": { 2024-05-10T11:32:16.835 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "failure_domain", 2024-05-10T11:32:16.835 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.835 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.835 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.835 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "host", 2024-05-10T11:32:16.835 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.835 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.836 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.836 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "failure domain for any created local pool", 2024-05-10T11:32:16.836 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-10T11:32:16.836 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.836 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.836 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.837 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.837 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.837 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.837 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.837 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.837 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.838 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.838 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.838 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.838 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.838 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.838 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.838 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.839 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.839 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.839 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.839 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.839 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.839 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.839 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.840 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.840 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.840 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.840 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.840 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.840 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.840 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.841 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.841 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.841 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.841 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.841 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.841 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.841 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.842 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.842 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.842 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.842 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.842 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.842 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.842 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.843 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.843 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.843 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.843 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.843 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.843 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.843 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.844 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.844 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.844 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.844 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.844 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.844 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.845 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.845 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.845 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.845 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.845 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.845 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.846 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.846 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.846 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.846 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.846 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.846 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": { 2024-05-10T11:32:16.847 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_size", 2024-05-10T11:32:16.847 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.847 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.847 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.847 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.847 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.847 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.847 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.848 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default min_size for any created local pool", 2024-05-10T11:32:16.848 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-10T11:32:16.848 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.848 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.848 INFO:teuthology.orchestra.run.smithi027.stdout: "num_rep": { 2024-05-10T11:32:16.848 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "num_rep", 2024-05-10T11:32:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3", 2024-05-10T11:32:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default replica count for any created local pool", 2024-05-10T11:32:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": { 2024-05-10T11:32:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_num", 2024-05-10T11:32:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "128", 2024-05-10T11:32:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default pg_num for any created local pool", 2024-05-10T11:32:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "prefix": { 2024-05-10T11:32:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prefix", 2024-05-10T11:32:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.856 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name prefix for any created local pool", 2024-05-10T11:32:16.856 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.856 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.856 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.856 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.856 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.856 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.857 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.857 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.857 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.857 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.857 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.857 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.857 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.858 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.858 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.858 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.858 INFO:teuthology.orchestra.run.smithi027.stdout: "subtree": { 2024-05-10T11:32:16.858 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subtree", 2024-05-10T11:32:16.859 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.859 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.859 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.859 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "rack", 2024-05-10T11:32:16.859 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.859 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.859 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.860 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-10T11:32:16.860 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-10T11:32:16.860 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.860 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.860 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.860 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.861 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mds_autoscaler", 2024-05-10T11:32:16.861 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.861 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.861 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.861 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.861 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.861 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.862 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.862 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.862 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.862 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.862 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.862 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.862 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.863 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.863 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.863 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.863 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.863 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.863 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.863 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.864 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.864 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.864 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.864 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.864 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.864 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.865 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.865 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.865 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.865 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.865 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.865 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.865 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.866 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.866 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.866 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.866 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.866 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.866 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.867 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.867 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.867 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.867 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.867 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.867 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.867 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.868 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.868 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.868 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.868 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.868 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.868 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.868 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.869 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.869 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.869 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.869 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.869 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.869 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.869 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.870 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.870 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.870 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.870 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.870 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.870 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.870 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.871 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.871 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.871 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.871 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.871 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.871 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.872 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.872 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.872 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.872 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.872 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.872 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.872 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.873 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.873 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.873 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.873 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.873 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirroring", 2024-05-10T11:32:16.874 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.874 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.874 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.874 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.874 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.874 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.874 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.875 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.875 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.875 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.875 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.875 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.875 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.875 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.876 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.876 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.876 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.876 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.876 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.876 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.876 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.876 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.877 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.877 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.877 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.877 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.877 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.877 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.878 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.878 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.878 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.878 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.878 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.878 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.878 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.878 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.879 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.879 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.879 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.879 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.879 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.879 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.880 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.880 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.880 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.880 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.880 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.880 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.880 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.880 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.881 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.881 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.881 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.881 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.881 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.881 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.882 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.882 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.882 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.882 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.882 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.882 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.882 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.883 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.883 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.883 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.883 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.883 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.883 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.883 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.884 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.884 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.884 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.884 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.884 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.884 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.884 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.885 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.885 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.885 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.885 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.885 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.885 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.885 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.886 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.886 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "nfs", 2024-05-10T11:32:16.886 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.886 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.886 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.886 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.887 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.887 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.887 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.887 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.887 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.887 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.887 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.888 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.888 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.888 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.888 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.888 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.888 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.888 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.889 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.889 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.889 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.889 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.889 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.889 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.889 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.890 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.890 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.890 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.890 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.890 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.890 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.890 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.891 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.891 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.891 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.891 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.891 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.891 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.892 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.892 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.892 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.892 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.892 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.892 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.892 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.893 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.893 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.893 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.893 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.893 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.893 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.893 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.894 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.894 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.894 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.894 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.894 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.894 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.895 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.895 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.895 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.895 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.895 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.895 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.895 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.896 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.896 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.896 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.896 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.896 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.896 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.897 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.897 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.897 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.897 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.897 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.897 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.897 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.897 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.898 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.898 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.898 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.898 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.898 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.899 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-05-10T11:32:16.899 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.899 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.899 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.899 INFO:teuthology.orchestra.run.smithi027.stdout: "fail_fs": { 2024-05-10T11:32:16.899 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "fail_fs", 2024-05-10T11:32:16.899 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.900 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.900 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.900 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.900 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.900 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.900 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.900 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-10T11:32:16.901 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.901 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.901 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.901 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.901 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.901 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.902 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.902 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.902 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.902 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.902 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.902 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.902 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.903 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.903 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.903 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.903 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.903 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.903 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.903 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.904 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.904 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.904 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.904 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.904 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.905 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.905 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.905 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.905 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.905 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.905 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.905 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.906 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.906 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.906 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.906 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.906 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.906 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.907 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.907 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.907 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.907 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.907 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.907 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.907 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.908 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.908 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.908 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.908 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.908 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.908 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.908 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.909 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.909 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.909 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.909 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.909 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.909 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.910 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.910 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.910 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.910 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.910 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.910 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.911 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.911 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.911 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.911 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.911 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.911 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator": { 2024-05-10T11:32:16.911 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-05-10T11:32:16.912 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.912 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.912 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.912 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.912 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.912 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.912 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.913 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-05-10T11:32:16.913 INFO:teuthology.orchestra.run.smithi027.stdout: "rook", 2024-05-10T11:32:16.913 INFO:teuthology.orchestra.run.smithi027.stdout: "test_orchestrator" 2024-05-10T11:32:16.913 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.913 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Orchestrator backend", 2024-05-10T11:32:16.913 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.913 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.913 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.914 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.914 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.914 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.914 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.914 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.915 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.915 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.915 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.915 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.915 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.915 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.915 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.915 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.916 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.916 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.916 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.916 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_perf_query", 2024-05-10T11:32:16.916 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.916 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.917 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.917 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.917 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.917 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.917 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.917 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.917 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.918 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.918 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.918 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.918 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.918 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.918 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.918 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.919 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.919 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.919 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.919 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.919 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.919 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.919 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.920 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.920 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.920 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.920 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.920 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.920 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.920 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.921 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.921 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.921 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.921 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.921 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.921 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.921 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.922 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.922 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.922 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.922 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.922 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.922 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.922 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.923 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.923 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.923 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.923 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.923 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.923 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.923 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.923 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.924 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.924 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.924 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.924 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.924 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.924 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.925 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.925 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.925 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.925 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.925 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.925 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.925 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.925 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.926 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.926 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.926 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.926 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.926 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.926 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.926 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.927 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.927 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.927 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.927 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.927 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.927 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.927 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.928 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.928 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.928 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.928 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.928 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.928 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_support", 2024-05-10T11:32:16.928 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.929 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.929 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.929 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.929 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.929 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.929 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.929 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.930 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.930 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.930 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.930 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.930 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.930 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.930 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.930 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.931 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.931 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.931 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.931 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.931 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.931 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.931 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.932 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.932 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.932 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.932 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.932 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.932 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.933 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.933 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.933 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.933 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.933 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.933 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.933 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.934 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.934 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.934 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.934 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.934 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.934 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.935 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.935 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.935 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.935 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.935 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.935 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.935 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.935 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.936 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.936 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.936 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.936 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.936 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.936 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.937 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.937 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.937 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.937 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.937 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.937 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.937 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.938 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.938 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.938 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.938 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.938 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.938 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.939 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.939 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.939 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.939 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.939 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.939 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.939 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.940 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.940 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.940 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.940 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.940 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.940 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.940 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.941 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.941 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_autoscaler", 2024-05-10T11:32:16.941 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.941 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.941 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.942 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.942 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.942 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.942 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.942 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.942 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.942 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.943 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.943 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.943 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.943 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.943 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.943 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.943 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.944 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.944 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.944 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.944 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.944 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.944 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.944 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.945 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.945 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.945 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.945 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.945 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.945 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.945 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.946 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.946 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.946 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.946 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.946 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.946 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.947 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.947 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.947 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.947 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.947 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.947 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.947 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.947 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.948 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.948 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.948 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.948 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.948 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.948 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.948 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.949 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.949 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.949 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.949 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.949 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.949 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.950 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.950 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.950 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.950 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.950 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.950 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.950 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.951 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.951 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.951 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.951 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.951 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:16.951 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:16.952 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.952 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.952 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.952 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:16.952 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.952 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.952 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.952 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.953 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.953 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.953 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.953 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.953 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.953 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.954 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.954 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.954 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.954 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.954 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.954 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.954 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.955 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.955 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.955 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.955 INFO:teuthology.orchestra.run.smithi027.stdout: "threshold": { 2024-05-10T11:32:16.955 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threshold", 2024-05-10T11:32:16.955 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:16.956 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.956 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.956 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-05-10T11:32:16.956 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1.0", 2024-05-10T11:32:16.956 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.956 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.956 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "scaling threshold", 2024-05-10T11:32:16.956 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-10T11:32:16.957 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.957 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.957 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.957 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.957 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "progress", 2024-05-10T11:32:16.958 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.958 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.958 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.958 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_pg_recovery_event": { 2024-05-10T11:32:16.958 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_pg_recovery_event", 2024-05-10T11:32:16.958 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.958 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.959 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.959 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.959 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.959 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.959 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.959 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-10T11:32:16.959 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.960 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.960 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.960 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-05-10T11:32:16.960 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-05-10T11:32:16.960 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.960 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.961 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.961 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.961 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.961 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.961 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.961 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.961 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.962 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.962 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.962 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.962 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.962 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.962 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.962 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.963 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.963 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.963 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.963 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.963 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.963 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.963 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.964 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.964 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.964 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.964 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.964 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.964 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.964 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.964 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.965 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.965 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.965 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.965 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.965 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.965 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.966 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.966 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.966 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.966 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.966 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.966 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.967 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.967 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.967 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.967 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.967 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.967 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.968 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.968 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.968 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.968 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.968 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.968 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.969 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.969 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.969 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.969 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.969 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.969 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.969 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.970 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.970 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.970 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.970 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.970 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.970 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.971 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.971 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.971 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.971 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.971 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.971 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.971 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.972 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.972 INFO:teuthology.orchestra.run.smithi027.stdout: "max_completed_events": { 2024-05-10T11:32:16.972 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_completed_events", 2024-05-10T11:32:16.972 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.972 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.972 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.973 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "50", 2024-05-10T11:32:16.973 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.973 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.973 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.973 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "number of past completed events to remember", 2024-05-10T11:32:16.973 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.973 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.973 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.974 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:16.974 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:16.974 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:16.974 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.974 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.975 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-05-10T11:32:16.975 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.975 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.975 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.975 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long the module is going to sleep", 2024-05-10T11:32:16.975 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.975 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.975 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.976 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:16.976 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:16.976 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.976 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:16.976 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.976 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:16.976 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.977 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.977 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.977 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.977 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.977 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.977 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.977 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:16.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:16.978 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus", 2024-05-10T11:32:16.978 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:16.978 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:16.978 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:16.979 INFO:teuthology.orchestra.run.smithi027.stdout: "cache": { 2024-05-10T11:32:16.979 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cache", 2024-05-10T11:32:16.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.979 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.979 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.979 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.979 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.979 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.980 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.980 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.980 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.980 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.980 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.980 INFO:teuthology.orchestra.run.smithi027.stdout: "exclude_perf_counters": { 2024-05-10T11:32:16.981 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "exclude_perf_counters", 2024-05-10T11:32:16.981 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.981 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.981 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.981 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:16.981 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.981 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.981 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.982 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-10T11:32:16.982 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-10T11:32:16.982 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.982 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.982 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:16.982 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:16.983 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.983 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.983 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.983 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.983 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.983 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.983 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.983 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.984 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.984 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.984 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.984 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.984 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.985 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.985 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.985 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.985 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.985 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.985 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:16.986 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:16.986 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.986 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.986 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.986 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.986 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.986 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.987 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.987 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.987 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.987 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.987 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.987 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:16.987 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:16.988 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.988 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.988 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.988 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:16.988 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.988 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.988 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:16.989 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:16.989 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:16.989 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:16.989 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:16.989 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:16.989 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:16.989 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:16.990 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.990 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.990 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.990 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.990 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:16.990 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:16.991 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:16.991 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.991 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:16.991 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:16.991 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.991 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.991 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.991 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.992 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.992 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.992 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.992 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools": { 2024-05-10T11:32:16.992 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools", 2024-05-10T11:32:16.992 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.993 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.993 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.993 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:16.993 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.993 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.993 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.993 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.993 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.994 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.994 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.994 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-10T11:32:16.994 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-10T11:32:16.994 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:16.995 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.995 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.995 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "300", 2024-05-10T11:32:16.995 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.995 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.995 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.995 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.995 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.996 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.996 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.996 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_interval": { 2024-05-10T11:32:16.996 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_interval", 2024-05-10T11:32:16.996 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:16.996 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.997 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.997 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15.0", 2024-05-10T11:32:16.997 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.997 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.997 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.997 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:16.997 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:16.998 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:16.998 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:16.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:16.998 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-05-10T11:32:16.998 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-05-10T11:32:16.998 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:16.998 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:16.999 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:16.999 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-05-10T11:32:16.999 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:16.999 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:16.999 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:16.999 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-10T11:32:16.999 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.000 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.000 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.000 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-05-10T11:32:17.000 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-05-10T11:32:17.000 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.000 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.000 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.001 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "9283", 2024-05-10T11:32:17.001 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.001 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.001 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.001 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-10T11:32:17.001 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.001 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.002 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.002 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.002 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.002 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.002 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.002 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.002 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "stale_cache_strategy": { 2024-05-10T11:32:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stale_cache_strategy", 2024-05-10T11:32:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "log", 2024-05-10T11:32:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-05-10T11:32:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-05-10T11:32:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "default", 2024-05-10T11:32:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "default", 2024-05-10T11:32:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "error" 2024-05-10T11:32:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-05-10T11:32:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-05-10T11:32:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-05-10T11:32:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-05-10T11:32:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-05-10T11:32:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-05-10T11:32:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_snap_create": { 2024-05-10T11:32:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_snap_create", 2024-05-10T11:32:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "mirror_snapshot_schedule": { 2024-05-10T11:32:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirror_snapshot_schedule", 2024-05-10T11:32:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "trash_purge_schedule": { 2024-05-10T11:32:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "trash_purge_schedule", 2024-05-10T11:32:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "restful", 2024-05-10T11:32:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_auth": { 2024-05-10T11:32:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_auth", 2024-05-10T11:32:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-05-10T11:32:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-05-10T11:32:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-05-10T11:32:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-05-10T11:32:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-05-10T11:32:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-05-10T11:32:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rgw", 2024-05-10T11:32:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rook", 2024-05-10T11:32:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "storage_class": { 2024-05-10T11:32:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "storage_class", 2024-05-10T11:32:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "local", 2024-05-10T11:32:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-10T11:32:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "selftest", 2024-05-10T11:32:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "roption1": { 2024-05-10T11:32:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption1", 2024-05-10T11:32:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "roption2": { 2024-05-10T11:32:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption2", 2024-05-10T11:32:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "xyz", 2024-05-10T11:32:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption1": { 2024-05-10T11:32:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption1", 2024-05-10T11:32:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption2": { 2024-05-10T11:32:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption2", 2024-05-10T11:32:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption3": { 2024-05-10T11:32:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption3", 2024-05-10T11:32:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption4": { 2024-05-10T11:32:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption4", 2024-05-10T11:32:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption5": { 2024-05-10T11:32:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption5", 2024-05-10T11:32:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption6": { 2024-05-10T11:32:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption6", 2024-05-10T11:32:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption7": { 2024-05-10T11:32:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption7", 2024-05-10T11:32:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-05-10T11:32:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "42", 2024-05-10T11:32:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: "testkey": { 2024-05-10T11:32:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testkey", 2024-05-10T11:32:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.104 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.104 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.104 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.104 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.104 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.104 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.104 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.104 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.105 INFO:teuthology.orchestra.run.smithi027.stdout: "testlkey": { 2024-05-10T11:32:17.105 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testlkey", 2024-05-10T11:32:17.105 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.105 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.105 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.105 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.105 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.106 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.106 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.106 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.106 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.106 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.106 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.106 INFO:teuthology.orchestra.run.smithi027.stdout: "testnewline": { 2024-05-10T11:32:17.107 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testnewline", 2024-05-10T11:32:17.107 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.107 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.107 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.107 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.107 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.107 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.107 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.107 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.108 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.108 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.108 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.108 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.108 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.108 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smb", 2024-05-10T11:32:17.109 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.109 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.109 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.109 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.109 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.109 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.109 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.109 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.110 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.110 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.110 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.110 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.110 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.110 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.110 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.110 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.110 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.111 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.111 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.111 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.111 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.111 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.111 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.111 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.112 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.112 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.112 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.112 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.112 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.112 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.112 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.112 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.113 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.113 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.113 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.113 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.113 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.113 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.113 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.113 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.114 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.114 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.114 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.114 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.114 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.114 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.114 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.115 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.115 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.115 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.115 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.115 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.115 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.115 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.115 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.116 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.116 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.116 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.116 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.116 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.116 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.116 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.117 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.117 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.117 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.117 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.117 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.117 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.117 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.117 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.118 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.118 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.118 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.118 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.118 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.118 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.118 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.119 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.119 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.119 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.119 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.119 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.119 INFO:teuthology.orchestra.run.smithi027.stdout: "update_orchestration": { 2024-05-10T11:32:17.119 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "update_orchestration", 2024-05-10T11:32:17.119 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.120 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.120 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.120 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.120 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.120 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.120 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.120 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-10T11:32:17.120 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.121 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.121 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.121 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.121 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.121 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-05-10T11:32:17.121 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.121 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.122 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.122 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_m_granularity": { 2024-05-10T11:32:17.122 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_m_granularity", 2024-05-10T11:32:17.122 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.122 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.122 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.122 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.122 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.123 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.123 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.123 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow minute scheduled snapshots", 2024-05-10T11:32:17.123 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.123 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.123 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.123 INFO:teuthology.orchestra.run.smithi027.stdout: "dump_on_update": { 2024-05-10T11:32:17.124 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dump_on_update", 2024-05-10T11:32:17.124 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.124 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.124 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.124 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.124 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.124 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.125 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.125 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "dump database to debug log on update", 2024-05-10T11:32:17.125 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.125 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.125 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.125 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.125 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.126 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.126 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.126 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.126 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.126 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.126 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.126 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.126 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.128 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.128 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.128 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.128 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.128 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.128 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.128 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.129 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.129 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.129 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.129 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.129 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.129 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.129 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.129 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.130 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.130 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.130 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.130 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.130 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.130 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.130 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.131 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.131 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.131 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.131 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.131 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.131 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.131 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.131 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.131 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.132 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.132 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.132 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.132 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.132 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.132 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.132 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.133 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.133 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.133 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.133 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.133 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.133 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.133 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.133 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.134 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.134 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.134 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.134 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.134 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.134 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.134 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.135 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.135 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.135 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.135 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.135 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.135 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.135 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.135 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.135 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.136 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.136 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stats", 2024-05-10T11:32:17.136 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.136 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.136 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.137 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.137 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.137 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.137 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.137 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.137 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.137 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.137 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.137 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.138 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.138 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.138 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.138 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.138 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.138 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.138 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.139 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.139 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.139 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.139 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.139 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.139 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.139 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.140 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.140 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.140 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.140 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.140 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.140 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.140 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.140 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.141 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.141 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.141 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.141 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.141 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.141 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.141 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.141 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.142 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.142 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.142 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.142 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.142 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.142 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.142 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.142 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.143 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.143 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.143 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.143 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.143 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.143 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.143 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.144 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.144 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.144 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.144 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.144 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.144 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.144 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.144 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.145 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.145 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.145 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.145 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.145 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.145 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.145 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.145 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.146 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.146 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.146 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.146 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.146 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.146 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.146 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.147 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.147 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.147 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "status", 2024-05-10T11:32:17.147 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.147 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.148 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.148 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.148 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.148 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.148 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.148 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.148 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.148 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.149 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.149 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.149 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.149 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.149 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.149 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.149 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.149 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.150 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.150 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.150 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.150 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.150 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.150 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.151 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.151 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.151 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.151 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.151 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.151 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.151 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.151 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.151 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.152 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.152 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.152 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.152 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.152 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.152 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.152 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.153 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.153 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.153 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.153 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.153 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.153 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.153 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.153 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.154 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.154 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.154 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.154 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.154 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.154 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.154 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.154 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.155 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.155 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.155 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.155 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.155 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.155 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.155 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.156 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.156 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.156 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.156 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.156 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.156 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.156 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.156 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.157 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.157 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.157 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.157 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.157 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.157 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.157 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.157 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.158 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.158 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.158 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.158 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.158 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.158 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telegraf", 2024-05-10T11:32:17.159 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.159 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.159 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.159 INFO:teuthology.orchestra.run.smithi027.stdout: "address": { 2024-05-10T11:32:17.159 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "address", 2024-05-10T11:32:17.159 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.159 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.159 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.160 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-10T11:32:17.160 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.160 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.160 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.160 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.160 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.160 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.160 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.161 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:17.161 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:17.161 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.161 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.161 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.161 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15", 2024-05-10T11:32:17.161 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.161 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.162 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.162 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.162 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.162 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.162 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.162 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.162 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.163 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.163 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.163 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.163 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.163 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.163 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.163 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.163 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.164 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.164 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.164 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.164 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.164 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.164 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.164 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.164 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.165 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.165 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.165 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.165 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.165 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.165 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.165 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.166 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.166 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.166 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.166 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.166 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.166 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.166 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.166 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.167 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.167 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.167 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.167 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.167 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.167 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.167 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.168 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.168 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.168 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.168 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.168 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.168 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.168 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.168 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.169 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.169 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.169 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.169 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.169 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.169 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.169 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.169 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.170 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.170 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.170 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.170 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.170 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.170 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.170 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.170 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.171 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.171 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.171 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.171 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.171 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.171 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.171 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.172 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.172 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.172 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.172 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.172 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.172 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.172 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.172 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telemetry", 2024-05-10T11:32:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_basic": { 2024-05-10T11:32:17.174 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_basic", 2024-05-10T11:32:17.174 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.174 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.174 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.174 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.174 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.174 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.174 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.175 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-10T11:32:17.175 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.175 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.175 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.175 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_crash": { 2024-05-10T11:32:17.175 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_crash", 2024-05-10T11:32:17.175 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.176 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.176 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.176 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.176 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.176 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.176 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.176 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-10T11:32:17.176 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.177 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.177 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.177 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_device": { 2024-05-10T11:32:17.177 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_device", 2024-05-10T11:32:17.177 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.177 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.177 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.178 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.178 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.178 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.178 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.178 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-10T11:32:17.178 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.178 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.178 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.179 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_ident": { 2024-05-10T11:32:17.179 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_ident", 2024-05-10T11:32:17.179 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.179 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.179 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.179 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.179 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.179 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.180 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.180 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-10T11:32:17.180 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.180 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.180 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.180 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_perf": { 2024-05-10T11:32:17.180 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_perf", 2024-05-10T11:32:17.181 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.181 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.181 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.181 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.181 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.181 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.181 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.181 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-10T11:32:17.181 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.182 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.182 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.182 INFO:teuthology.orchestra.run.smithi027.stdout: "contact": { 2024-05-10T11:32:17.182 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "contact", 2024-05-10T11:32:17.182 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.182 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.182 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.183 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.183 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.183 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.183 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.183 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.183 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.183 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.183 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.184 INFO:teuthology.orchestra.run.smithi027.stdout: "description": { 2024-05-10T11:32:17.184 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "description", 2024-05-10T11:32:17.184 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.184 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.184 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.184 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.184 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.184 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.185 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.185 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.185 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.185 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.185 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.185 INFO:teuthology.orchestra.run.smithi027.stdout: "device_url": { 2024-05-10T11:32:17.185 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_url", 2024-05-10T11:32:17.186 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.186 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.186 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.186 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-10T11:32:17.186 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.186 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.186 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.186 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-05-10T11:32:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-05-10T11:32:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "24", 2024-05-10T11:32:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "8", 2024-05-10T11:32:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "last_opt_revision": { 2024-05-10T11:32:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "last_opt_revision", 2024-05-10T11:32:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1", 2024-05-10T11:32:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard": { 2024-05-10T11:32:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard", 2024-05-10T11:32:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard_description": { 2024-05-10T11:32:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard_description", 2024-05-10T11:32:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.199 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.199 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.199 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.199 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.199 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.199 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.199 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.200 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.200 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.200 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.200 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.200 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.200 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.200 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.201 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.201 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.201 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.201 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.201 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.201 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.201 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.201 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.202 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.202 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.202 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.202 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.202 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.202 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.202 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.203 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.203 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.203 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.203 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.203 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.203 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.203 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.204 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.204 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.204 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.204 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.204 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.204 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.204 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.205 INFO:teuthology.orchestra.run.smithi027.stdout: "organization": { 2024-05-10T11:32:17.205 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "organization", 2024-05-10T11:32:17.205 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.205 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.205 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.205 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.205 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.205 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.206 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.206 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.206 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.206 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.206 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.206 INFO:teuthology.orchestra.run.smithi027.stdout: "proxy": { 2024-05-10T11:32:17.206 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "proxy", 2024-05-10T11:32:17.207 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.207 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.207 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.207 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.207 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.207 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.207 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.208 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.208 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.208 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.208 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.208 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.208 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.208 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.209 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.209 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.209 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.209 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.209 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.209 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.209 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.210 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.210 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.210 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.210 INFO:teuthology.orchestra.run.smithi027.stdout: "url": { 2024-05-10T11:32:17.210 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url", 2024-05-10T11:32:17.210 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.211 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.211 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.211 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-10T11:32:17.211 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.211 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.211 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.211 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.211 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.212 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.212 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.212 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.212 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.212 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "test_orchestrator", 2024-05-10T11:32:17.213 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.213 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.213 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.213 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.213 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.213 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.213 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.213 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.214 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.214 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.214 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.214 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.214 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.214 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.214 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.215 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.215 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.215 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.215 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.215 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.215 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.215 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.215 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.216 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.216 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.216 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.216 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.216 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.216 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.216 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.217 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.217 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.217 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.217 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.217 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.217 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.218 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.218 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.218 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.218 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.218 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.218 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.218 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.218 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.219 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.219 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.219 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.219 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.219 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.219 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.219 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.219 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.220 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.220 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.220 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.220 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.220 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.220 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.221 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.221 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.221 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.221 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.221 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.221 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.221 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.221 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.222 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.222 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.222 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.222 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.222 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.222 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.223 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.223 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.223 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.223 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.223 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.223 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.223 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.223 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.224 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.224 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.224 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.224 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.224 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-05-10T11:32:17.225 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.225 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.225 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.225 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.225 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.225 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.225 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.225 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.226 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.226 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.226 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.226 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.226 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.226 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.226 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.226 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.227 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.227 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.227 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.227 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.227 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.227 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.227 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.228 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.228 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.228 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.228 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.228 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.228 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.229 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.229 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.229 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.229 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.229 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.229 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.229 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.230 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.230 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.230 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.230 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.230 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.230 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.230 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.231 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.231 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.231 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.231 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.231 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.231 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.231 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.231 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.232 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.232 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.232 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.232 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.232 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.232 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.233 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.233 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.233 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.233 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.233 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.233 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.233 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.233 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.234 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.234 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.234 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.234 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.234 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_clones": { 2024-05-10T11:32:17.234 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_clones", 2024-05-10T11:32:17.234 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.235 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.235 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.235 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4", 2024-05-10T11:32:17.235 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.235 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.235 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.235 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-10T11:32:17.236 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.236 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.236 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.236 INFO:teuthology.orchestra.run.smithi027.stdout: "periodic_async_work": { 2024-05-10T11:32:17.236 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "periodic_async_work", 2024-05-10T11:32:17.236 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.237 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.237 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.237 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.237 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.237 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.237 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.237 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Periodically check for async work", 2024-05-10T11:32:17.237 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.237 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.238 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.238 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_delay": { 2024-05-10T11:32:17.238 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_delay", 2024-05-10T11:32:17.238 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.238 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.238 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.239 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.239 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.239 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.239 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.239 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-10T11:32:17.239 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.239 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.239 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.240 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_no_wait": { 2024-05-10T11:32:17.240 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_no_wait", 2024-05-10T11:32:17.240 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.240 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.240 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.240 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.240 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.241 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.241 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.241 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-10T11:32:17.241 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.241 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.241 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.241 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.242 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.242 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.242 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.242 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.242 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.242 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.242 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.243 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.243 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.243 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.243 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.243 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.243 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.243 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.244 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix", 2024-05-10T11:32:17.244 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.244 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.244 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.244 INFO:teuthology.orchestra.run.smithi027.stdout: "discovery_interval": { 2024-05-10T11:32:17.244 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "discovery_interval", 2024-05-10T11:32:17.244 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:17.245 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.245 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.245 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "100", 2024-05-10T11:32:17.245 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.245 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.245 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.245 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.245 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.246 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.246 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.246 INFO:teuthology.orchestra.run.smithi027.stdout: "identifier": { 2024-05-10T11:32:17.246 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "identifier", 2024-05-10T11:32:17.246 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.246 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.247 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.247 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.247 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.247 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.247 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.247 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.247 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.247 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.248 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.248 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:17.248 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:17.248 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.248 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.248 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.248 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:17.249 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.249 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.249 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.249 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.249 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.249 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.249 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.250 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.250 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.250 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.250 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.250 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.250 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.250 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.251 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.251 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.251 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.251 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.251 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.251 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.251 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.251 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.251 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.252 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.252 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.252 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.252 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.252 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.252 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.253 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.253 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.253 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.253 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.253 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.253 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.253 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.253 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.254 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.254 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.254 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.254 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.254 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.254 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.255 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.255 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.255 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.255 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.255 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.255 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.255 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.255 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.255 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.256 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.256 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.256 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.256 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.256 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.256 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.256 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.257 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.257 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.257 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.257 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.257 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.257 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.257 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.258 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.258 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.258 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.258 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.258 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.258 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.258 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.259 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.259 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.259 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.259 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.259 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.259 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.259 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.259 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.260 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.260 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.260 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.260 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.260 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.260 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_host": { 2024-05-10T11:32:17.261 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_host", 2024-05-10T11:32:17.261 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.261 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.261 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.261 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.261 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.261 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.261 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.262 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.262 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.262 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.262 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.262 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_port": { 2024-05-10T11:32:17.265 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_port", 2024-05-10T11:32:17.265 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.265 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.265 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.265 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10051", 2024-05-10T11:32:17.266 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.266 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.266 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.266 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.266 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.266 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.266 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.267 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_sender": { 2024-05-10T11:32:17.267 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_sender", 2024-05-10T11:32:17.267 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.267 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.267 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.267 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-10T11:32:17.267 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.267 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.268 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.268 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.268 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.268 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.268 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.268 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.268 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.268 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.269 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-10T11:32:17.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.269 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 25198, 2024-05-10T11:32:17.269 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "y", 2024-05-10T11:32:17.269 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_features": 4540701547738038271, 2024-05-10T11:32:17.269 INFO:teuthology.orchestra.run.smithi027.stdout: "available_modules": [ 2024-05-10T11:32:17.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.270 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "alerts", 2024-05-10T11:32:17.270 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.270 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.270 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.270 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:17.270 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:17.271 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.271 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.271 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.271 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:17.271 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.271 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.271 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.271 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How frequently to reexamine health status", 2024-05-10T11:32:17.272 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.272 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.272 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.272 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.272 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.272 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.272 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.273 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.273 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.273 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.273 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.273 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.273 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.273 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.273 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.274 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.274 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.274 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.274 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.274 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.274 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.274 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.275 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.275 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.275 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.275 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.275 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.275 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.275 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.276 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.276 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.276 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.276 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.276 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.276 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.276 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.277 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.277 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.277 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.277 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.277 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.277 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.277 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.277 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.278 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.278 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.278 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.278 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.278 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.278 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.278 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.279 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.279 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.279 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.279 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.279 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.279 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.279 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.279 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.280 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.280 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.280 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.280 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.280 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.280 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.280 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.281 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.281 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.281 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.281 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_destination": { 2024-05-10T11:32:17.281 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_destination", 2024-05-10T11:32:17.281 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.282 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.282 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.282 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.282 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.282 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.282 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.282 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email address to send alerts to", 2024-05-10T11:32:17.282 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.283 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.283 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.283 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_from_name": { 2024-05-10T11:32:17.283 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_from_name", 2024-05-10T11:32:17.283 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.283 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.283 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.284 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "Ceph", 2024-05-10T11:32:17.284 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.284 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.284 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.284 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email From: name", 2024-05-10T11:32:17.284 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.284 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.285 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.285 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_host": { 2024-05-10T11:32:17.285 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_host", 2024-05-10T11:32:17.285 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.286 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.286 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.286 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.286 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.286 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.287 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.287 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP server", 2024-05-10T11:32:17.287 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.287 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.287 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.287 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_password": { 2024-05-10T11:32:17.288 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_password", 2024-05-10T11:32:17.288 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.288 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.288 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.288 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.288 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.289 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.289 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.289 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Password to authenticate with", 2024-05-10T11:32:17.289 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.289 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.289 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.289 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_port": { 2024-05-10T11:32:17.290 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_port", 2024-05-10T11:32:17.290 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.290 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.290 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.290 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "465", 2024-05-10T11:32:17.290 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.290 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.290 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.291 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP port", 2024-05-10T11:32:17.291 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.291 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.291 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.291 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_sender": { 2024-05-10T11:32:17.291 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_sender", 2024-05-10T11:32:17.292 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.292 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.292 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.292 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.292 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.292 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.292 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.292 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP envelope sender", 2024-05-10T11:32:17.293 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.293 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.293 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.293 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_ssl": { 2024-05-10T11:32:17.293 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_ssl", 2024-05-10T11:32:17.293 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.293 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.294 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.294 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.294 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.294 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.294 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.294 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-10T11:32:17.294 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.295 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.295 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.295 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.295 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_user": { 2024-05-10T11:32:17.295 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_user", 2024-05-10T11:32:17.295 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.295 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.295 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.296 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.296 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.296 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.296 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.296 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "User to authenticate as", 2024-05-10T11:32:17.296 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.296 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.297 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.297 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.297 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.297 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.297 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.297 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.297 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.297 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.297 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.298 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.298 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.298 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.298 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.298 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.298 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.298 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "balancer", 2024-05-10T11:32:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: "active": { 2024-05-10T11:32:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "active", 2024-05-10T11:32:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.301 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically balance PGs across cluster", 2024-05-10T11:32:17.301 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.301 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.301 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.301 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_time": { 2024-05-10T11:32:17.302 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_time", 2024-05-10T11:32:17.302 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.302 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.302 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.302 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0000", 2024-05-10T11:32:17.303 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.303 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.303 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.303 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "beginning time of day to automatically balance", 2024-05-10T11:32:17.303 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-10T11:32:17.303 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.304 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.304 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_weekday": { 2024-05-10T11:32:17.304 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_weekday", 2024-05-10T11:32:17.304 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:17.304 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.305 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.305 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.305 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-05-10T11:32:17.305 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-05-10T11:32:17.305 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.305 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-10T11:32:17.305 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-10T11:32:17.305 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.306 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.306 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_max_iterations": { 2024-05-10T11:32:17.306 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_max_iterations", 2024-05-10T11:32:17.306 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:17.306 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.307 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.307 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "25", 2024-05-10T11:32:17.307 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-05-10T11:32:17.307 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "250", 2024-05-10T11:32:17.307 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.307 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-10T11:32:17.308 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.308 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.308 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.308 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.308 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_metrics": { 2024-05-10T11:32:17.308 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_metrics", 2024-05-10T11:32:17.308 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.308 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.309 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.309 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "pgs,objects,bytes", 2024-05-10T11:32:17.309 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.309 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.309 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.309 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-10T11:32:17.309 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-10T11:32:17.310 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.310 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.310 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_step": { 2024-05-10T11:32:17.310 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_step", 2024-05-10T11:32:17.310 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:17.311 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.311 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.311 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.5", 2024-05-10T11:32:17.311 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0.001", 2024-05-10T11:32:17.311 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "0.999", 2024-05-10T11:32:17.311 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.311 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "aggressiveness of optimization", 2024-05-10T11:32:17.311 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-10T11:32:17.312 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.312 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.312 INFO:teuthology.orchestra.run.smithi027.stdout: "end_time": { 2024-05-10T11:32:17.312 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_time", 2024-05-10T11:32:17.312 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.312 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.313 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.313 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2359", 2024-05-10T11:32:17.313 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.313 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.313 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.313 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ending time of day to automatically balance", 2024-05-10T11:32:17.313 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-10T11:32:17.314 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.314 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.314 INFO:teuthology.orchestra.run.smithi027.stdout: "end_weekday": { 2024-05-10T11:32:17.314 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_weekday", 2024-05-10T11:32:17.314 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:17.314 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.315 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.315 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.315 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-05-10T11:32:17.315 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-05-10T11:32:17.315 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.315 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-10T11:32:17.315 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-10T11:32:17.315 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.316 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.316 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.316 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.316 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.316 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.316 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.316 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.317 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.317 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.317 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.317 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.317 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.317 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.317 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.318 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.318 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.318 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.318 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.318 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.318 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.318 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.319 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.319 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.319 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.319 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.319 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.319 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.320 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.320 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.320 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.320 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.320 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.320 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.320 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.320 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.321 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.321 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.321 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.321 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.321 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.321 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.321 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.322 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.322 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.322 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.322 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.322 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.322 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.322 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.323 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.323 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.323 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.323 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.323 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.323 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.323 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.324 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.324 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.324 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.324 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.324 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.324 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.324 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.324 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.325 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.325 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.325 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.325 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.325 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.325 INFO:teuthology.orchestra.run.smithi027.stdout: "min_score": { 2024-05-10T11:32:17.325 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_score", 2024-05-10T11:32:17.326 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:17.326 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.326 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.326 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.326 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.326 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.326 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.326 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-10T11:32:17.327 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.327 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.327 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.327 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-05-10T11:32:17.327 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-05-10T11:32:17.327 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.327 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.328 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.328 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "upmap", 2024-05-10T11:32:17.328 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.328 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.328 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.328 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-compat", 2024-05-10T11:32:17.328 INFO:teuthology.orchestra.run.smithi027.stdout: "none", 2024-05-10T11:32:17.328 INFO:teuthology.orchestra.run.smithi027.stdout: "read", 2024-05-10T11:32:17.329 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap", 2024-05-10T11:32:17.329 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap-read" 2024-05-10T11:32:17.329 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.329 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Balancer mode", 2024-05-10T11:32:17.329 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.329 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.329 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.330 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ids": { 2024-05-10T11:32:17.330 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_ids", 2024-05-10T11:32:17.330 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.330 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.331 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.331 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.331 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.331 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.331 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.331 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-10T11:32:17.331 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.331 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.332 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.332 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.332 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:17.332 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:17.332 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.332 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.333 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.333 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:17.333 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.333 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.333 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.333 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-10T11:32:17.333 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.334 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.334 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.334 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.334 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.334 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.334 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.334 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.335 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.335 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.335 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.335 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.335 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.335 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.335 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.336 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.336 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.336 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_deviation": { 2024-05-10T11:32:17.336 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_deviation", 2024-05-10T11:32:17.337 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.337 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.337 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.337 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-05-10T11:32:17.337 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-05-10T11:32:17.337 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.337 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.338 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-10T11:32:17.338 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-10T11:32:17.338 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.338 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.338 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_optimizations": { 2024-05-10T11:32:17.339 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_optimizations", 2024-05-10T11:32:17.339 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:17.339 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.339 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.339 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:17.339 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.340 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.340 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.340 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-10T11:32:17.340 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.340 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.340 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.341 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.341 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.341 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm", 2024-05-10T11:32:17.342 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.342 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.342 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.342 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_down_multiplier": { 2024-05-10T11:32:17.342 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_down_multiplier", 2024-05-10T11:32:17.342 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:17.343 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.343 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.343 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-05-10T11:32:17.343 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.343 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.343 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.343 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-10T11:32:17.344 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.344 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.344 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.344 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_refresh_rate": { 2024-05-10T11:32:17.344 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_refresh_rate", 2024-05-10T11:32:17.344 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.344 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.345 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.345 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "20", 2024-05-10T11:32:17.345 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.345 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.345 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.345 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-10T11:32:17.345 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.345 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.346 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.346 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_starting_port": { 2024-05-10T11:32:17.346 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_starting_port", 2024-05-10T11:32:17.346 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.346 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.346 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.347 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4721", 2024-05-10T11:32:17.347 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.347 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.347 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.347 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-10T11:32:17.347 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.347 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.347 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.348 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_ptrace": { 2024-05-10T11:32:17.348 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_ptrace", 2024-05-10T11:32:17.348 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.348 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.348 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.348 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.348 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.349 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.349 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.349 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-10T11:32:17.349 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-10T11:32:17.349 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.349 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.350 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_interval": { 2024-05-10T11:32:17.350 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_interval", 2024-05-10T11:32:17.350 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.350 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.350 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.350 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:17.350 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.350 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.351 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.351 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-10T11:32:17.351 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.351 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.351 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.351 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_memory_target_ratio": { 2024-05-10T11:32:17.352 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_memory_target_ratio", 2024-05-10T11:32:17.352 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:17.352 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.352 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.352 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.7", 2024-05-10T11:32:17.352 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.352 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.352 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.352 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-10T11:32:17.353 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.353 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.353 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.353 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_log_destination": { 2024-05-10T11:32:17.353 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm_log_destination", 2024-05-10T11:32:17.353 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.354 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.354 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.354 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.354 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.354 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.354 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.354 INFO:teuthology.orchestra.run.smithi027.stdout: "file", 2024-05-10T11:32:17.354 INFO:teuthology.orchestra.run.smithi027.stdout: "file,syslog", 2024-05-10T11:32:17.355 INFO:teuthology.orchestra.run.smithi027.stdout: "syslog" 2024-05-10T11:32:17.355 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.355 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-10T11:32:17.355 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.355 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.355 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.355 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.355 INFO:teuthology.orchestra.run.smithi027.stdout: "cgroups_split": { 2024-05-10T11:32:17.356 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cgroups_split", 2024-05-10T11:32:17.356 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.356 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.356 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.356 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.356 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.356 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.356 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.357 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-10T11:32:17.357 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.357 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.357 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.357 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.357 INFO:teuthology.orchestra.run.smithi027.stdout: "config_checks_enabled": { 2024-05-10T11:32:17.357 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_checks_enabled", 2024-05-10T11:32:17.358 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.358 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.358 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.358 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.358 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.358 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.358 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.358 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-10T11:32:17.359 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.359 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.359 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.359 INFO:teuthology.orchestra.run.smithi027.stdout: "config_dashboard": { 2024-05-10T11:32:17.359 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_dashboard", 2024-05-10T11:32:17.359 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.360 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.360 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.360 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.360 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.360 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.360 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.360 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-10T11:32:17.360 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.360 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.361 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.361 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_alertmanager": { 2024-05-10T11:32:17.361 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_alertmanager", 2024-05-10T11:32:17.361 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.361 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.361 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.362 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-10T11:32:17.362 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.362 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.362 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.362 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-05-10T11:32:17.362 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.362 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.362 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.363 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_base": { 2024-05-10T11:32:17.363 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_base", 2024-05-10T11:32:17.363 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.363 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.363 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.363 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-10T11:32:17.364 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.364 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.364 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.364 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Container image name, without the tag", 2024-05-10T11:32:17.364 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.364 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.365 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.365 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_elasticsearch": { 2024-05-10T11:32:17.365 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_elasticsearch", 2024-05-10T11:32:17.365 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.365 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.365 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.366 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-10T11:32:17.366 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.366 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.366 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.366 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "elasticsearch container image", 2024-05-10T11:32:17.366 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.367 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.367 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.367 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.367 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_grafana": { 2024-05-10T11:32:17.367 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_grafana", 2024-05-10T11:32:17.367 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.367 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.368 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.368 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-10T11:32:17.368 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.368 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.368 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.368 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-05-10T11:32:17.368 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.369 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.369 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.369 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_haproxy": { 2024-05-10T11:32:17.369 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_haproxy", 2024-05-10T11:32:17.369 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.369 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.370 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.370 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-10T11:32:17.370 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.370 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.370 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.371 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "HAproxy container image", 2024-05-10T11:32:17.371 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.371 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.371 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.371 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_agent": { 2024-05-10T11:32:17.371 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_agent", 2024-05-10T11:32:17.372 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.372 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.372 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.372 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-10T11:32:17.372 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.372 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.373 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.373 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger agent container image", 2024-05-10T11:32:17.373 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.373 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.373 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.374 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_collector": { 2024-05-10T11:32:17.374 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_collector", 2024-05-10T11:32:17.374 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.374 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.374 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.375 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-10T11:32:17.375 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.375 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.375 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.375 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger collector container image", 2024-05-10T11:32:17.375 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.375 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.376 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.376 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.376 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_query": { 2024-05-10T11:32:17.376 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_query", 2024-05-10T11:32:17.376 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.377 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.377 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.377 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-10T11:32:17.377 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.377 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.377 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.377 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger query container image", 2024-05-10T11:32:17.377 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.378 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.378 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.378 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_keepalived": { 2024-05-10T11:32:17.378 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_keepalived", 2024-05-10T11:32:17.378 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.378 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.379 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.379 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-10T11:32:17.379 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.379 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.379 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.379 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Keepalived container image", 2024-05-10T11:32:17.379 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.380 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.380 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.380 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_loki": { 2024-05-10T11:32:17.380 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_loki", 2024-05-10T11:32:17.380 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.380 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.381 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.381 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-10T11:32:17.381 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.381 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.381 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.381 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Loki container image", 2024-05-10T11:32:17.381 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.381 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.382 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.382 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_node_exporter": { 2024-05-10T11:32:17.382 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_node_exporter", 2024-05-10T11:32:17.382 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.382 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.382 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.383 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-10T11:32:17.383 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.383 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.383 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.383 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-05-10T11:32:17.383 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.383 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.384 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.384 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_nvmeof": { 2024-05-10T11:32:17.384 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_nvmeof", 2024-05-10T11:32:17.384 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.384 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.384 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.385 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-10T11:32:17.385 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.385 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.385 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.385 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Nvme-of container image", 2024-05-10T11:32:17.385 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.385 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.386 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.386 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_prometheus": { 2024-05-10T11:32:17.386 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_prometheus", 2024-05-10T11:32:17.386 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.386 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.386 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.386 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-10T11:32:17.387 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.387 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.387 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.387 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-05-10T11:32:17.387 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.387 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.387 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.388 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_promtail": { 2024-05-10T11:32:17.388 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_promtail", 2024-05-10T11:32:17.388 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.388 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.388 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.388 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-10T11:32:17.389 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.389 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.389 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.389 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Promtail container image", 2024-05-10T11:32:17.389 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.389 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.389 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.390 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_samba": { 2024-05-10T11:32:17.390 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_samba", 2024-05-10T11:32:17.390 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.390 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.390 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.390 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-10T11:32:17.390 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.391 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.391 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.391 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Samba/SMB container image", 2024-05-10T11:32:17.391 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.391 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.391 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.392 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_snmp_gateway": { 2024-05-10T11:32:17.392 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_snmp_gateway", 2024-05-10T11:32:17.392 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.392 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.392 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.392 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-10T11:32:17.392 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.392 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.393 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.393 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SNMP Gateway container image", 2024-05-10T11:32:17.393 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.393 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.393 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.393 INFO:teuthology.orchestra.run.smithi027.stdout: "container_init": { 2024-05-10T11:32:17.393 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_init", 2024-05-10T11:32:17.394 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.394 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.394 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.394 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.394 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.394 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.394 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.395 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Run podman/docker with `--init`", 2024-05-10T11:32:17.395 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.395 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.395 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.395 INFO:teuthology.orchestra.run.smithi027.stdout: "daemon_cache_timeout": { 2024-05-10T11:32:17.395 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "daemon_cache_timeout", 2024-05-10T11:32:17.395 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.396 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.396 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.396 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:17.396 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.396 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.396 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.396 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-10T11:32:17.396 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.397 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.397 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.397 INFO:teuthology.orchestra.run.smithi027.stdout: "default_cephadm_command_timeout": { 2024-05-10T11:32:17.397 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_cephadm_command_timeout", 2024-05-10T11:32:17.397 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.397 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.398 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.398 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "900", 2024-05-10T11:32:17.398 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.398 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.398 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.398 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-10T11:32:17.399 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.399 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.399 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.399 INFO:teuthology.orchestra.run.smithi027.stdout: "default_registry": { 2024-05-10T11:32:17.399 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_registry", 2024-05-10T11:32:17.400 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.400 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.400 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.400 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io", 2024-05-10T11:32:17.400 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.400 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.400 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.400 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-10T11:32:17.401 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.401 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.401 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.401 INFO:teuthology.orchestra.run.smithi027.stdout: "device_cache_timeout": { 2024-05-10T11:32:17.401 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_cache_timeout", 2024-05-10T11:32:17.401 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.402 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.402 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.402 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1800", 2024-05-10T11:32:17.402 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.402 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.402 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.402 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache device inventory", 2024-05-10T11:32:17.403 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.403 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.403 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.403 INFO:teuthology.orchestra.run.smithi027.stdout: "device_enhanced_scan": { 2024-05-10T11:32:17.403 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_enhanced_scan", 2024-05-10T11:32:17.403 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.403 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.404 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.404 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.404 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.404 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.404 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.404 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-10T11:32:17.405 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.405 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.405 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.405 INFO:teuthology.orchestra.run.smithi027.stdout: "facts_cache_timeout": { 2024-05-10T11:32:17.405 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "facts_cache_timeout", 2024-05-10T11:32:17.406 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.406 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.406 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.406 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:17.406 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.407 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.407 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.407 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache host facts data", 2024-05-10T11:32:17.407 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.407 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.407 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.408 INFO:teuthology.orchestra.run.smithi027.stdout: "grafana_dashboards_path": { 2024-05-10T11:32:17.408 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "grafana_dashboards_path", 2024-05-10T11:32:17.408 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.408 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.408 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.409 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-10T11:32:17.409 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.409 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.409 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.409 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-10T11:32:17.409 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.409 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.410 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.410 INFO:teuthology.orchestra.run.smithi027.stdout: "host_check_interval": { 2024-05-10T11:32:17.410 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "host_check_interval", 2024-05-10T11:32:17.410 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.411 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.411 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.411 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:17.411 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.411 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.411 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.412 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to perform a host check", 2024-05-10T11:32:17.412 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.412 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.412 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.412 INFO:teuthology.orchestra.run.smithi027.stdout: "hw_monitoring": { 2024-05-10T11:32:17.413 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hw_monitoring", 2024-05-10T11:32:17.413 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.413 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.413 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.413 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.414 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.414 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.414 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.414 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-10T11:32:17.414 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.414 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.415 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.415 INFO:teuthology.orchestra.run.smithi027.stdout: "inventory_list_all": { 2024-05-10T11:32:17.415 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "inventory_list_all", 2024-05-10T11:32:17.415 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.415 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.415 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.416 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.416 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.416 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.416 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.416 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-10T11:32:17.416 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.416 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.417 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.417 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.417 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.417 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.417 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.417 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.418 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.418 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.418 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.418 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.418 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.418 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.418 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.419 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.419 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.419 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.419 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.419 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.419 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.419 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.420 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.420 INFO:teuthology.orchestra.run.smithi027.stdout: "log_refresh_metadata": { 2024-05-10T11:32:17.420 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_refresh_metadata", 2024-05-10T11:32:17.420 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.420 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.420 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.421 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.421 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.421 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.421 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.421 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-10T11:32:17.421 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.421 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.421 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.422 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.422 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.422 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.422 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.422 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-10T11:32:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-10T11:32:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-10T11:32:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.430 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.430 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-10T11:32:17.430 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.430 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.430 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-10T11:32:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-10T11:32:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "*", 2024-05-10T11:32:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-10T11:32:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "max_count_per_host": { 2024-05-10T11:32:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_count_per_host", 2024-05-10T11:32:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.434 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.434 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of daemons per service per host", 2024-05-10T11:32:17.434 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.434 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.434 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.434 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd_draining_count": { 2024-05-10T11:32:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_osd_draining_count", 2024-05-10T11:32:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-10T11:32:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "migration_current": { 2024-05-10T11:32:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "migration_current", 2024-05-10T11:32:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "internal - do not modify", 2024-05-10T11:32:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-05-10T11:32:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-05-10T11:32:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "root", 2024-05-10T11:32:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm-package", 2024-05-10T11:32:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: "root" 2024-05-10T11:32:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "mode for remote execution of cephadm", 2024-05-10T11:32:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "oob_default_addr": { 2024-05-10T11:32:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "oob_default_addr", 2024-05-10T11:32:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.442 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.442 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.442 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "169.254.1.1", 2024-05-10T11:32:17.442 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.442 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-10T11:32:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: "prometheus_alerts_path": { 2024-05-10T11:32:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus_alerts_path", 2024-05-10T11:32:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-10T11:32:17.445 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.445 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.445 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.445 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-10T11:32:17.445 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.445 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.445 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.446 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_insecure": { 2024-05-10T11:32:17.446 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_insecure", 2024-05-10T11:32:17.446 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.446 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.446 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.447 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.447 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.447 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.447 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.447 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-10T11:32:17.448 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.448 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.448 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.448 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_password": { 2024-05-10T11:32:17.448 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_password", 2024-05-10T11:32:17.449 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.449 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.449 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.449 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.449 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.449 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.449 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.450 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-10T11:32:17.450 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.450 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.450 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.450 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_url": { 2024-05-10T11:32:17.450 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_url", 2024-05-10T11:32:17.451 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.451 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.451 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.451 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.451 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.451 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.451 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.451 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-10T11:32:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_username": { 2024-05-10T11:32:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_username", 2024-05-10T11:32:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-10T11:32:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "secure_monitoring_stack": { 2024-05-10T11:32:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "secure_monitoring_stack", 2024-05-10T11:32:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-10T11:32:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: "service_discovery_port": { 2024-05-10T11:32:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "service_discovery_port", 2024-05-10T11:32:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8765", 2024-05-10T11:32:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "cephadm service discovery port", 2024-05-10T11:32:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "ssh_config_file": { 2024-05-10T11:32:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssh_config_file", 2024-05-10T11:32:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-10T11:32:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "use_agent": { 2024-05-10T11:32:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_agent", 2024-05-10T11:32:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-10T11:32:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "use_repo_digest": { 2024-05-10T11:32:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_repo_digest", 2024-05-10T11:32:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-10T11:32:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_failed_host_check": { 2024-05-10T11:32:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_failed_host_check", 2024-05-10T11:32:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if the host check fails", 2024-05-10T11:32:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_daemons": { 2024-05-10T11:32:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_daemons", 2024-05-10T11:32:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-10T11:32:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_hosts": { 2024-05-10T11:32:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_hosts", 2024-05-10T11:32:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-10T11:32:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crash", 2024-05-10T11:32:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "retain_interval": { 2024-05-10T11:32:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retain_interval", 2024-05-10T11:32:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "31536000", 2024-05-10T11:32:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-10T11:32:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_recent_interval": { 2024-05-10T11:32:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_recent_interval", 2024-05-10T11:32:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1209600", 2024-05-10T11:32:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-10T11:32:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dashboard", 2024-05-10T11:32:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-10T11:32:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-10T11:32:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_HOST": { 2024-05-10T11:32:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-10T11:32:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-10T11:32:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-10T11:32:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_ENABLED": { 2024-05-10T11:32:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_ENABLED", 2024-05-10T11:32:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-10T11:32:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-10T11:32:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "ENABLE_BROWSABLE_API": { 2024-05-10T11:32:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-10T11:32:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-10T11:32:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-10T11:32:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-10T11:32:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-10T11:32:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-10T11:32:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-10T11:32:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-10T11:32:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-10T11:32:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-10T11:32:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-10T11:32:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-10T11:32:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-10T11:32:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-10T11:32:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-10T11:32:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-10T11:32:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-10T11:32:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_PASSWORD": { 2024-05-10T11:32:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-10T11:32:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-05-10T11:32:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-10T11:32:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-10T11:32:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_URL": { 2024-05-10T11:32:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_URL", 2024-05-10T11:32:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_USERNAME": { 2024-05-10T11:32:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-10T11:32:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-05-10T11:32:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-10T11:32:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-10T11:32:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-10T11:32:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-10T11:32:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-10T11:32:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-10T11:32:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-10T11:32:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-10T11:32:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-10T11:32:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-10T11:32:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "[]", 2024-05-10T11:32:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "MULTICLUSTER_CONFIG": { 2024-05-10T11:32:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-10T11:32:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "{}", 2024-05-10T11:32:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_HOST": { 2024-05-10T11:32:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-10T11:32:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-10T11:32:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-10T11:32:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-10T11:32:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-10T11:32:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-10T11:32:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-10T11:32:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-10T11:32:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-10T11:32:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-10T11:32:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-10T11:32:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-10T11:32:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-10T11:32:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-10T11:32:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-10T11:32:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-10T11:32:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-10T11:32:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_ENABLED": { 2024-05-10T11:32:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-10T11:32:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-10T11:32:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-10T11:32:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-10T11:32:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-10T11:32:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-10T11:32:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-10T11:32:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-10T11:32:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8", 2024-05-10T11:32:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-10T11:32:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-10T11:32:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "45", 2024-05-10T11:32:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ACCESS_KEY": { 2024-05-10T11:32:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-10T11:32:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-10T11:32:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-10T11:32:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-05-10T11:32:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SECRET_KEY": { 2024-05-10T11:32:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-10T11:32:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SSL_VERIFY": { 2024-05-10T11:32:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-10T11:32:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "UNSAFE_TLS_v1_2": { 2024-05-10T11:32:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-10T11:32:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-10T11:32:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-10T11:32:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-10T11:32:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-10T11:32:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-10T11:32:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-10T11:32:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-05-10T11:32:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "cross_origin_url": { 2024-05-10T11:32:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cross_origin_url", 2024-05-10T11:32:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "crt_file": { 2024-05-10T11:32:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crt_file", 2024-05-10T11:32:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "debug": { 2024-05-10T11:32:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "debug", 2024-05-10T11:32:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable/disable debug options", 2024-05-10T11:32:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "jwt_token_ttl": { 2024-05-10T11:32:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "jwt_token_ttl", 2024-05-10T11:32:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "28800", 2024-05-10T11:32:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-05-10T11:32:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-05-10T11:32:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "motd": { 2024-05-10T11:32:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "motd", 2024-05-10T11:32:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "The message of the day", 2024-05-10T11:32:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect_resolve_ip_addr": { 2024-05-10T11:32:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "redirect_resolve_ip_addr", 2024-05-10T11:32:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.599 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.599 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.599 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.599 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.599 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-05-10T11:32:17.599 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-05-10T11:32:17.599 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.600 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.600 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.600 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-05-10T11:32:17.600 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.600 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.600 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.600 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.600 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.600 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.601 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.601 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-05-10T11:32:17.601 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-05-10T11:32:17.601 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.601 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.601 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.601 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8080", 2024-05-10T11:32:17.602 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.602 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.602 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.602 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.602 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.602 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.602 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.603 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.603 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.603 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.603 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.603 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.603 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.603 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.603 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.604 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.604 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.604 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.604 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.604 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.604 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-05-10T11:32:17.604 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-05-10T11:32:17.605 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.605 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.605 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.605 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.605 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.605 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.605 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.605 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.606 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.606 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.606 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.606 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl_server_port": { 2024-05-10T11:32:17.606 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl_server_port", 2024-05-10T11:32:17.606 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.607 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.607 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.607 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8443", 2024-05-10T11:32:17.607 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.607 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.607 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.607 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.607 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.608 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.608 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.608 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-05-10T11:32:17.608 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-05-10T11:32:17.608 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.608 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.608 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.609 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "redirect", 2024-05-10T11:32:17.609 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.609 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.609 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.609 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.609 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect" 2024-05-10T11:32:17.609 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.609 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.609 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.610 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.610 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.610 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-05-10T11:32:17.610 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-05-10T11:32:17.610 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.610 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.610 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.611 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-05-10T11:32:17.611 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-05-10T11:32:17.611 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-05-10T11:32:17.611 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.611 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.611 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.611 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.612 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.612 INFO:teuthology.orchestra.run.smithi027.stdout: "url_prefix": { 2024-05-10T11:32:17.612 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url_prefix", 2024-05-10T11:32:17.612 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.612 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.612 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.612 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.613 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.613 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.613 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.613 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.613 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.613 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.613 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.613 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.614 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.614 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-05-10T11:32:17.614 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.614 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.614 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.614 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_monitoring": { 2024-05-10T11:32:17.615 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_monitoring", 2024-05-10T11:32:17.615 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.615 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.615 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.615 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.615 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.615 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.615 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.616 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "monitor device health metrics", 2024-05-10T11:32:17.616 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.616 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.616 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.616 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.616 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.616 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.617 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.617 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.617 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.617 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.617 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.617 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.617 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.617 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.618 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.618 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.618 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.618 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.618 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.618 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.618 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.618 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.619 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.619 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.619 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.619 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.619 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.619 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.619 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.620 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.620 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.620 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.620 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.620 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.620 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.620 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.621 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.621 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.621 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.621 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.621 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.621 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.621 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.621 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.622 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.622 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.622 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.622 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.622 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.622 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.622 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.623 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.623 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.623 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.623 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.623 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.624 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.624 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.624 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.624 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.624 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.624 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.624 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.624 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.625 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.625 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.625 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.625 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.625 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.625 INFO:teuthology.orchestra.run.smithi027.stdout: "mark_out_threshold": { 2024-05-10T11:32:17.626 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mark_out_threshold", 2024-05-10T11:32:17.626 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.626 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.626 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.626 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2419200", 2024-05-10T11:32:17.626 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.626 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.626 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.627 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-10T11:32:17.627 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.627 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.627 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.627 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": { 2024-05-10T11:32:17.627 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_name", 2024-05-10T11:32:17.627 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.628 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.628 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.628 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "device_health_metrics", 2024-05-10T11:32:17.628 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.628 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.628 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.628 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-10T11:32:17.628 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.629 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.629 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.629 INFO:teuthology.orchestra.run.smithi027.stdout: "retention_period": { 2024-05-10T11:32:17.629 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retention_period", 2024-05-10T11:32:17.629 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.629 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.629 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.629 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15552000", 2024-05-10T11:32:17.630 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.630 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.630 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.630 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain device health metrics", 2024-05-10T11:32:17.630 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.630 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.630 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.631 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_frequency": { 2024-05-10T11:32:17.631 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_frequency", 2024-05-10T11:32:17.631 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.631 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.631 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.631 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-05-10T11:32:17.631 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.631 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.632 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.632 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-10T11:32:17.632 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.632 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.632 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.632 INFO:teuthology.orchestra.run.smithi027.stdout: "self_heal": { 2024-05-10T11:32:17.632 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "self_heal", 2024-05-10T11:32:17.633 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.633 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.633 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.633 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.633 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.633 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.633 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.633 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-10T11:32:17.634 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.634 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.634 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.634 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:17.634 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:17.634 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.634 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.635 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.635 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:17.635 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.635 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.635 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.635 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and check device health", 2024-05-10T11:32:17.635 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.635 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.635 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.636 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.636 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.636 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.636 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.636 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.637 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.637 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.637 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.637 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.637 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.637 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.637 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.637 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.638 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_threshold": { 2024-05-10T11:32:17.638 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_threshold", 2024-05-10T11:32:17.638 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.638 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.638 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.638 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7257600", 2024-05-10T11:32:17.638 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.638 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.639 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.639 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-10T11:32:17.639 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.639 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.639 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.639 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.639 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.640 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "diskprediction_local", 2024-05-10T11:32:17.640 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.640 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.640 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.640 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.640 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.640 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.640 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.641 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.641 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.641 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.641 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.641 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.641 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.641 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.641 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.642 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.642 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.642 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.642 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.642 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.642 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.642 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.642 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.643 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.643 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.643 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.643 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.643 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.643 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.643 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.643 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.644 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.644 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.644 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.644 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.644 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.644 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.644 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.645 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.645 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.645 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.645 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.645 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.645 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.645 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.645 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.646 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.646 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.646 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.646 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.646 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.646 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.646 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.646 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.647 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.647 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.647 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.647 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.647 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.647 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.647 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.648 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.648 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.648 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.648 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.648 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.648 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.648 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.648 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.649 INFO:teuthology.orchestra.run.smithi027.stdout: "predict_interval": { 2024-05-10T11:32:17.649 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predict_interval", 2024-05-10T11:32:17.649 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.649 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.649 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.649 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-05-10T11:32:17.649 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.650 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.650 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.650 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.650 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.650 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.650 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.650 INFO:teuthology.orchestra.run.smithi027.stdout: "predictor_model": { 2024-05-10T11:32:17.651 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predictor_model", 2024-05-10T11:32:17.651 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.651 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.651 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.651 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "prophetstor", 2024-05-10T11:32:17.651 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.651 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.651 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.651 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.652 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.652 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.652 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.652 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:17.652 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:17.652 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.653 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.653 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.653 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:17.653 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.653 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.653 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.653 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.653 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.654 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.654 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.654 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.654 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.654 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.654 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.654 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.655 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.655 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.655 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.655 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.655 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.655 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.655 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.655 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.656 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.656 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.656 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "influx", 2024-05-10T11:32:17.656 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": false, 2024-05-10T11:32:17.656 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "influxdb python module not found", 2024-05-10T11:32:17.656 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.657 INFO:teuthology.orchestra.run.smithi027.stdout: "batch_size": { 2024-05-10T11:32:17.657 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "batch_size", 2024-05-10T11:32:17.657 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.657 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.657 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.657 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5000", 2024-05-10T11:32:17.657 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.657 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.658 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.658 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-10T11:32:17.658 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.658 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.658 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.658 INFO:teuthology.orchestra.run.smithi027.stdout: "database": { 2024-05-10T11:32:17.658 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "database", 2024-05-10T11:32:17.659 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.659 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.659 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.659 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "ceph", 2024-05-10T11:32:17.659 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.659 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.659 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.659 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-10T11:32:17.660 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.660 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.660 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.660 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": { 2024-05-10T11:32:17.660 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hostname", 2024-05-10T11:32:17.660 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.660 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.661 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.661 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.661 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.661 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.661 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.661 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server hostname", 2024-05-10T11:32:17.661 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.662 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.662 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.662 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:17.662 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:17.662 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.662 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.662 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.663 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "30", 2024-05-10T11:32:17.663 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "5", 2024-05-10T11:32:17.663 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.663 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.663 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-10T11:32:17.663 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.663 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.663 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.664 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.664 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.664 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.664 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.664 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.664 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.664 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.665 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.665 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.665 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.665 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.665 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.665 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.665 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.665 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.665 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: "password": { 2024-05-10T11:32:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "password", 2024-05-10T11:32:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "password of InfluxDB server user", 2024-05-10T11:32:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: "port": { 2024-05-10T11:32:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "port", 2024-05-10T11:32:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8086", 2024-05-10T11:32:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server port", 2024-05-10T11:32:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-05-10T11:32:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-05-10T11:32:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "false", 2024-05-10T11:32:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-10T11:32:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: "threads": { 2024-05-10T11:32:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threads", 2024-05-10T11:32:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-05-10T11:32:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-05-10T11:32:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "32", 2024-05-10T11:32:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-10T11:32:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: "username": { 2024-05-10T11:32:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "username", 2024-05-10T11:32:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "username of InfluxDB server user", 2024-05-10T11:32:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.683 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.683 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.683 INFO:teuthology.orchestra.run.smithi027.stdout: "verify_ssl": { 2024-05-10T11:32:17.683 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "verify_ssl", 2024-05-10T11:32:17.683 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.683 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.683 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.684 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "true", 2024-05-10T11:32:17.684 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.684 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.684 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.684 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-10T11:32:17.684 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.684 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.684 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.685 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.685 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.685 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "insights", 2024-05-10T11:32:17.685 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.685 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.685 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.686 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.686 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.686 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.686 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.686 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.686 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.686 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.686 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.687 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.687 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.687 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.687 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.687 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.687 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.687 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.687 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.688 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.688 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.688 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.688 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.688 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.688 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.689 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.689 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.689 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.689 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.689 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.689 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.689 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.690 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.690 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.690 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.690 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.690 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.690 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.690 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.690 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.691 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.691 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.691 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.691 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.691 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.691 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.691 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.692 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.692 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.692 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.692 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.692 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.692 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.692 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.692 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.693 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.693 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.693 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.693 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.693 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.693 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.693 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.694 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.694 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.694 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.694 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.694 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.694 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.694 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.695 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.695 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.695 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.695 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.695 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.695 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.695 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.695 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.695 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.696 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.696 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.696 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.696 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.696 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.696 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.696 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "iostat", 2024-05-10T11:32:17.697 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.697 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.697 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.697 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.697 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.697 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.697 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.698 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.698 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.698 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.698 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.698 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.698 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.698 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.698 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.698 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.699 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.699 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.699 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.699 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.699 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.699 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.699 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.700 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.700 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.700 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.700 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.700 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.700 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.700 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.700 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.701 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.701 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.701 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.701 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.701 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.701 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.701 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.702 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.702 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.702 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.702 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.702 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.702 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.702 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.702 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.703 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.703 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.703 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.703 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.703 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.703 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.703 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.703 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.704 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.704 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.704 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.704 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.704 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.704 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.704 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.705 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.705 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.705 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.705 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.705 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.705 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.705 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.707 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.707 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.707 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.707 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.707 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.707 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.707 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.708 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "k8sevents", 2024-05-10T11:32:17.708 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.708 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.708 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.708 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_event_retention_days": { 2024-05-10T11:32:17.708 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ceph_event_retention_days", 2024-05-10T11:32:17.708 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.708 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.709 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.709 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7", 2024-05-10T11:32:17.709 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.709 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.709 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.709 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-10T11:32:17.709 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.709 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.710 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.710 INFO:teuthology.orchestra.run.smithi027.stdout: "config_check_secs": { 2024-05-10T11:32:17.710 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_check_secs", 2024-05-10T11:32:17.710 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.710 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.710 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.710 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:17.710 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "10", 2024-05-10T11:32:17.711 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.711 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.711 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-10T11:32:17.711 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.711 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.711 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.711 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.712 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.712 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.712 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.712 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.712 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.712 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.712 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.712 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.713 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.713 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.713 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.713 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.713 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.713 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.713 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.713 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.713 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.714 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.714 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.714 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.714 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.714 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.714 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.714 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.715 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.715 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.715 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.715 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.715 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.715 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.715 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.715 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.716 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.716 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.716 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.716 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.716 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.716 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.716 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.717 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.717 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.717 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.717 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.717 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.717 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.717 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.717 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.718 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.718 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.718 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.718 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.718 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.718 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.718 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.719 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.719 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.719 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.719 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.719 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.719 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.719 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.719 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.719 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.720 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.720 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.720 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.720 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.720 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.720 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.720 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.721 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.721 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.721 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.721 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.721 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.721 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.721 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.721 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.722 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.722 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.722 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "localpool", 2024-05-10T11:32:17.722 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.722 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.722 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.723 INFO:teuthology.orchestra.run.smithi027.stdout: "failure_domain": { 2024-05-10T11:32:17.723 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "failure_domain", 2024-05-10T11:32:17.723 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.723 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.723 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.723 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "host", 2024-05-10T11:32:17.723 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.723 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.724 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.724 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "failure domain for any created local pool", 2024-05-10T11:32:17.724 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-10T11:32:17.724 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.724 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.724 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.724 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.725 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.725 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.725 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.725 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.725 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.725 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.725 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.725 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.725 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.726 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.726 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.726 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.726 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.726 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.726 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.726 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.726 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.727 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.727 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.727 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.727 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.727 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.727 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.727 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.728 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.728 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.728 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.728 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.728 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.728 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.728 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.729 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.729 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.729 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.729 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.729 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.729 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.729 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.729 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.730 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.730 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.730 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.730 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.730 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.730 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.730 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.730 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.731 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.731 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.731 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.731 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.731 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.731 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.731 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.732 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.732 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.732 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.732 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.732 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.732 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.732 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.732 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.733 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.733 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.733 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": { 2024-05-10T11:32:17.733 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_size", 2024-05-10T11:32:17.733 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.733 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.733 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.734 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.734 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.734 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.734 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.734 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default min_size for any created local pool", 2024-05-10T11:32:17.734 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-10T11:32:17.734 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.735 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.735 INFO:teuthology.orchestra.run.smithi027.stdout: "num_rep": { 2024-05-10T11:32:17.735 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "num_rep", 2024-05-10T11:32:17.735 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.735 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.735 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.735 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3", 2024-05-10T11:32:17.735 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.736 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.736 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.736 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default replica count for any created local pool", 2024-05-10T11:32:17.736 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.736 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.736 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.736 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": { 2024-05-10T11:32:17.737 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_num", 2024-05-10T11:32:17.737 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.737 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.737 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.737 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "128", 2024-05-10T11:32:17.737 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.737 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.737 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.738 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default pg_num for any created local pool", 2024-05-10T11:32:17.738 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.738 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.738 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.738 INFO:teuthology.orchestra.run.smithi027.stdout: "prefix": { 2024-05-10T11:32:17.738 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prefix", 2024-05-10T11:32:17.738 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.739 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.739 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.739 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.739 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.739 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.739 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.739 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name prefix for any created local pool", 2024-05-10T11:32:17.739 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.739 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.740 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.740 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.740 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.740 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.740 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.740 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.740 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.741 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.741 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.741 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.741 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.741 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.741 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.741 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.742 INFO:teuthology.orchestra.run.smithi027.stdout: "subtree": { 2024-05-10T11:32:17.742 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subtree", 2024-05-10T11:32:17.742 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.742 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.742 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.742 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "rack", 2024-05-10T11:32:17.742 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.742 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.743 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.743 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-10T11:32:17.743 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-10T11:32:17.743 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.743 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.743 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.743 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.744 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mds_autoscaler", 2024-05-10T11:32:17.744 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.744 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.744 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.744 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.744 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.744 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.745 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.745 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.745 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.745 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.745 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.745 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.745 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.746 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.746 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.746 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.746 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.746 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.746 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.746 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.746 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.747 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.747 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.747 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.747 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.747 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.747 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.747 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.747 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.748 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.748 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.748 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.748 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.748 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.748 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.748 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.751 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.751 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.751 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.751 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.751 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.751 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.751 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.752 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.752 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.752 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.752 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.752 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.752 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.752 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.752 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.752 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.753 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.753 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.753 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.753 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.753 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.753 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.753 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.754 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.754 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.754 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.754 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.754 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.754 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.754 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.754 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.755 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.755 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.755 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirroring", 2024-05-10T11:32:17.755 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.755 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.755 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.756 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.756 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.756 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.756 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.756 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.756 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.756 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.756 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.757 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.757 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.757 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.757 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.757 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.757 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.757 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.757 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.758 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.758 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.758 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.758 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.758 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.758 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.758 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.759 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.759 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.759 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.759 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.759 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.759 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.759 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.759 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.760 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.760 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.760 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.760 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.760 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.760 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.760 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.761 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.761 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.761 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.761 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.761 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.761 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.761 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.761 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.762 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.762 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.762 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.762 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.762 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.762 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.762 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.763 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.763 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.763 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.763 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.763 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.763 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.763 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.763 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.764 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.764 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.764 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.764 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.764 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.764 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.764 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.765 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.765 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.765 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.765 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.765 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.765 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.765 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.765 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.766 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.766 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.766 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.766 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.766 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.766 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "nfs", 2024-05-10T11:32:17.766 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.767 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.767 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.767 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.767 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.767 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.767 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.767 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.767 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.767 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.768 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.768 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.768 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.768 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.768 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.768 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.768 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.768 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.769 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.769 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.769 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.769 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.769 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.769 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.769 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.770 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.770 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.770 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.770 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.770 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.770 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.770 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.770 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.771 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.771 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.771 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.771 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.771 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.771 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.771 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.771 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.772 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.772 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.772 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.772 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.772 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.772 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.772 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.773 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.773 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.773 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.773 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.773 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.773 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.773 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.773 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.774 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.774 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.774 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.774 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.774 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.774 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.774 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.775 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.775 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.775 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.775 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.775 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.775 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.775 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.776 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.776 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.776 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.776 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.776 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.776 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.776 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.776 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.776 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.777 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.777 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.777 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.777 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.777 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.777 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.777 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-05-10T11:32:17.778 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.778 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.778 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.778 INFO:teuthology.orchestra.run.smithi027.stdout: "fail_fs": { 2024-05-10T11:32:17.778 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "fail_fs", 2024-05-10T11:32:17.778 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.778 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.778 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.779 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.779 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.779 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.779 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.779 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-10T11:32:17.779 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.779 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.779 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.780 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.780 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.780 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.780 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.780 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.780 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.781 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.781 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.781 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.781 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.781 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.781 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.781 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.781 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.782 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.782 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.782 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.782 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.782 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.782 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.783 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.783 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.783 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.783 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.783 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.783 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.783 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.783 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.784 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.784 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.784 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.784 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.784 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.784 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.784 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.785 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.785 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.785 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.785 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.785 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.785 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.785 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.785 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.785 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.786 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.786 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.786 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.786 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.786 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.786 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.786 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.786 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.787 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.787 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.787 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.787 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.787 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.787 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.787 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.788 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.788 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.788 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.788 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.788 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.788 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.788 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.788 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator": { 2024-05-10T11:32:17.789 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-05-10T11:32:17.789 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.789 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.789 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.789 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.789 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.789 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.789 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.790 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-05-10T11:32:17.790 INFO:teuthology.orchestra.run.smithi027.stdout: "rook", 2024-05-10T11:32:17.790 INFO:teuthology.orchestra.run.smithi027.stdout: "test_orchestrator" 2024-05-10T11:32:17.790 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.790 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Orchestrator backend", 2024-05-10T11:32:17.790 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.790 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.790 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.791 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.791 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.791 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.791 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.791 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.791 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.791 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.792 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.792 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.792 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.792 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.792 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.792 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.792 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.792 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.793 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_perf_query", 2024-05-10T11:32:17.793 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.793 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.793 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.793 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.793 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.793 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.794 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.794 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.794 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.794 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.794 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.794 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.794 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.795 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.795 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.795 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.795 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.795 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.795 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.795 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.795 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.796 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.796 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.796 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.796 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.796 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.796 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.796 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.797 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.797 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.797 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.797 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.797 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.797 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.797 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.797 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.798 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.798 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.798 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.798 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.798 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.798 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.799 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.799 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.799 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.799 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.799 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.799 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.799 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.800 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.800 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.800 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.800 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.800 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.800 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.800 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.801 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.801 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.801 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.801 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.801 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.801 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.802 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.802 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.802 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.802 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.802 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.802 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.802 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.803 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.803 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.803 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.803 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.803 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.804 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.804 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.804 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.804 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.804 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.804 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.804 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.805 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.805 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.805 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.805 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_support", 2024-05-10T11:32:17.805 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.805 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.806 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.806 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.806 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.806 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.806 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.806 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.807 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.807 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.807 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.807 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.807 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.807 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.807 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.807 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.808 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.808 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.808 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.808 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.808 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.808 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.808 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.809 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.809 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.809 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.809 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.809 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.809 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.809 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.810 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.810 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.810 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.810 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.810 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.810 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.811 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.811 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.811 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.811 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.811 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.811 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.812 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.812 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.812 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.812 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.812 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.812 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.812 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.813 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.813 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.813 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.813 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.813 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.813 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.813 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.814 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.814 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.814 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.814 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.814 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.814 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.814 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.815 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.815 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.815 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.815 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.815 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.815 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.816 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.816 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.816 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.816 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.816 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.816 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.816 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.817 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.817 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.817 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.817 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.817 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.817 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.817 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.817 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.818 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_autoscaler", 2024-05-10T11:32:17.818 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.818 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.818 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.819 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.819 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.819 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.819 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.819 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.819 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.819 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.819 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.820 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.820 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.820 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.820 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.820 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.820 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.820 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.820 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.821 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.821 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.821 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.821 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.821 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.821 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.822 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.822 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.822 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.822 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.822 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.822 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.822 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.823 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.823 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.823 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.823 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.823 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.823 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.824 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.825 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.825 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.825 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.825 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.826 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.826 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.826 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.826 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.826 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.826 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.826 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.827 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.827 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.827 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.827 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.827 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.827 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.828 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.828 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.828 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.828 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.828 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.828 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.828 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.828 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.829 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.829 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.829 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.829 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.829 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.829 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:17.830 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:17.830 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.830 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.830 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.830 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:17.830 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.830 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.831 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.831 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.831 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.831 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.831 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.831 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.832 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.833 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:16 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2127414105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-10T11:32:17.833 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:16 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/506358858' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-10T11:32:17.833 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:16 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2127414105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-10T11:32:17.834 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:16 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/506358858' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-10T11:32:17.837 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.837 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.837 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.837 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.837 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.838 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.838 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.838 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.838 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.838 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.838 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.839 INFO:teuthology.orchestra.run.smithi027.stdout: "threshold": { 2024-05-10T11:32:17.839 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threshold", 2024-05-10T11:32:17.839 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:17.839 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.839 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.839 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-05-10T11:32:17.839 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1.0", 2024-05-10T11:32:17.839 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.840 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.840 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "scaling threshold", 2024-05-10T11:32:17.840 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-10T11:32:17.840 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.840 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.840 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.841 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.841 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "progress", 2024-05-10T11:32:17.841 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.841 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.841 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.842 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_pg_recovery_event": { 2024-05-10T11:32:17.842 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_pg_recovery_event", 2024-05-10T11:32:17.842 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.842 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.842 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.842 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.842 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.843 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.843 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.843 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-10T11:32:17.843 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.843 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.843 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.843 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-05-10T11:32:17.844 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-05-10T11:32:17.844 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.844 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.844 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.844 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.844 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.844 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.844 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.845 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.845 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.845 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.845 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.845 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.845 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.845 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.846 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.846 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.846 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.846 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.846 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.846 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.846 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.846 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.847 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.847 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.847 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.847 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.847 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.847 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.847 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.847 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.847 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.848 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.848 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.848 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.848 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.848 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.848 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.848 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.849 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.849 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.849 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.849 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.849 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.849 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.849 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.850 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.850 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.850 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.850 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.850 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.850 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.850 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.850 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.851 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.851 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.851 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.851 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.851 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.851 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.851 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.851 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.852 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.852 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.852 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.852 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.852 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.852 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.852 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.853 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.853 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.853 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.853 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.853 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.853 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.853 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.853 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.854 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.854 INFO:teuthology.orchestra.run.smithi027.stdout: "max_completed_events": { 2024-05-10T11:32:17.854 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_completed_events", 2024-05-10T11:32:17.854 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.854 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.854 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.854 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "50", 2024-05-10T11:32:17.855 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.855 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.855 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.855 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "number of past completed events to remember", 2024-05-10T11:32:17.855 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.855 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.855 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.856 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:17.856 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:17.856 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:17.856 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.856 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.856 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-05-10T11:32:17.856 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.856 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.856 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.857 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long the module is going to sleep", 2024-05-10T11:32:17.857 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.857 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.857 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.857 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.857 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.857 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.858 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.858 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.858 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.858 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.858 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.858 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.858 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.858 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.859 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.859 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.859 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.859 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.859 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus", 2024-05-10T11:32:17.859 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.859 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.860 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.860 INFO:teuthology.orchestra.run.smithi027.stdout: "cache": { 2024-05-10T11:32:17.860 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cache", 2024-05-10T11:32:17.860 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.860 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.860 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.861 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.861 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.861 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.861 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.861 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.861 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.861 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.862 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.862 INFO:teuthology.orchestra.run.smithi027.stdout: "exclude_perf_counters": { 2024-05-10T11:32:17.862 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "exclude_perf_counters", 2024-05-10T11:32:17.862 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.862 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.862 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.863 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.863 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.863 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.863 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.863 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-10T11:32:17.863 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-10T11:32:17.863 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.863 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.864 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.864 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.864 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.864 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.864 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.864 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.864 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.864 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.865 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.865 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.865 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.865 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.865 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.865 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.865 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.865 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.866 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.866 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.866 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.866 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.866 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.866 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.866 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.867 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.867 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.867 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.867 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.867 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.867 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.867 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.867 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.868 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.868 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.868 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.868 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.868 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.868 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.868 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.869 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.869 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.869 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.869 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.869 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.869 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.869 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.869 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.870 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.870 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.870 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.870 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.870 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.870 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.870 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.871 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.871 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.871 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.871 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.871 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.871 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.871 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.871 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.871 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.872 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.872 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.872 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.872 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.872 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools": { 2024-05-10T11:32:17.872 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools", 2024-05-10T11:32:17.872 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.873 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.873 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.873 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.873 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.873 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.873 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.873 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.874 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.874 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.874 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.874 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-10T11:32:17.874 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-10T11:32:17.874 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.874 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.875 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.875 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "300", 2024-05-10T11:32:17.875 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.875 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.875 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.875 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.875 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.875 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.876 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.876 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_interval": { 2024-05-10T11:32:17.876 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_interval", 2024-05-10T11:32:17.876 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:17.876 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.876 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.876 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15.0", 2024-05-10T11:32:17.877 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.877 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.877 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.877 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.877 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.877 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.877 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.878 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-05-10T11:32:17.878 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-05-10T11:32:17.878 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.878 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.878 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.878 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-05-10T11:32:17.878 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.878 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.879 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.879 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-10T11:32:17.879 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.879 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.879 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.879 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-05-10T11:32:17.879 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-05-10T11:32:17.880 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.880 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.880 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.880 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "9283", 2024-05-10T11:32:17.880 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.880 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.880 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.880 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-10T11:32:17.881 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.881 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.881 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.881 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.881 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.881 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.881 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.882 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.882 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.882 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.882 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.882 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.882 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.882 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.882 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.883 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.883 INFO:teuthology.orchestra.run.smithi027.stdout: "stale_cache_strategy": { 2024-05-10T11:32:17.883 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stale_cache_strategy", 2024-05-10T11:32:17.883 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.883 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.883 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.883 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "log", 2024-05-10T11:32:17.884 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.884 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.884 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.884 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.884 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.884 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.884 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.885 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-05-10T11:32:17.885 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-05-10T11:32:17.885 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.885 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.885 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.885 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "default", 2024-05-10T11:32:17.885 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.885 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.886 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.886 INFO:teuthology.orchestra.run.smithi027.stdout: "default", 2024-05-10T11:32:17.886 INFO:teuthology.orchestra.run.smithi027.stdout: "error" 2024-05-10T11:32:17.886 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.886 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.886 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.886 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.887 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.887 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-05-10T11:32:17.887 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-05-10T11:32:17.887 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.887 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.887 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.887 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-05-10T11:32:17.888 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-05-10T11:32:17.888 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-05-10T11:32:17.888 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.888 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.888 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.888 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.888 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.888 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.889 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.889 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-05-10T11:32:17.889 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.889 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.889 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.889 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.890 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.890 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.890 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.890 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.890 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.890 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.890 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.890 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.891 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.891 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.891 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.891 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.891 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.891 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.891 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.891 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.891 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.892 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.892 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.892 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.892 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.892 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.892 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.892 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.893 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.893 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.893 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.893 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.893 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.893 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.893 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.893 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.894 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.894 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.894 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.894 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.894 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.894 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.894 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.895 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.895 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.895 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.895 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.895 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.895 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.895 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.895 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.896 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.896 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.896 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.896 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.896 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.896 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.896 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.897 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.897 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.897 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.897 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.897 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.897 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.897 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.897 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.898 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.898 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.898 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.898 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_snap_create": { 2024-05-10T11:32:17.898 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_snap_create", 2024-05-10T11:32:17.898 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.899 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.899 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.899 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:17.899 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.899 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.899 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.899 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.899 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.899 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.900 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.900 INFO:teuthology.orchestra.run.smithi027.stdout: "mirror_snapshot_schedule": { 2024-05-10T11:32:17.900 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirror_snapshot_schedule", 2024-05-10T11:32:17.900 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.900 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.900 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.900 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.901 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.901 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.901 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.901 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.901 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.901 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.901 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.901 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.902 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.902 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.902 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.902 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.902 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.902 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.902 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.902 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.903 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.903 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.903 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.903 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.903 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.903 INFO:teuthology.orchestra.run.smithi027.stdout: "trash_purge_schedule": { 2024-05-10T11:32:17.903 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "trash_purge_schedule", 2024-05-10T11:32:17.903 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.904 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.904 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.904 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.904 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.904 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.904 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.904 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.904 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.904 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.905 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.905 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.905 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.905 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "restful", 2024-05-10T11:32:17.905 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.905 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.906 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.906 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_auth": { 2024-05-10T11:32:17.906 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_auth", 2024-05-10T11:32:17.906 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.906 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.906 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.906 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.906 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.906 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-05-10T11:32:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-05-10T11:32:17.908 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.908 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.908 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.908 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.908 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.908 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.908 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.908 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.909 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.909 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.909 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.909 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.909 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.909 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.909 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.910 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.910 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.910 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.910 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.910 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.910 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.910 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.910 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.911 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.911 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.911 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.911 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.911 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.911 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.911 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.911 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.912 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.912 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.912 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.912 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.912 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.912 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.912 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.913 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.913 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.913 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.913 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.913 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.913 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.913 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.913 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.914 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.914 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.914 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.914 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.914 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.914 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.914 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.914 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.915 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.915 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.915 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.915 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.915 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.915 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.915 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.915 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.916 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.916 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.916 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.916 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.916 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.916 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.916 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.917 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.917 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.917 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.917 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.917 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.917 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.917 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.917 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.918 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-05-10T11:32:17.918 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-05-10T11:32:17.918 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.918 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.918 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.918 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.919 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.919 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.919 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.919 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.919 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.919 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.919 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.920 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-05-10T11:32:17.920 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-05-10T11:32:17.920 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.920 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.920 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.920 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.920 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.920 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.921 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.921 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.921 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.921 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.921 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.921 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.921 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.922 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.922 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.922 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.922 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.922 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.922 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.922 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.923 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.923 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.923 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.923 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.923 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.923 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.924 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rgw", 2024-05-10T11:32:17.924 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.924 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.924 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.924 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.924 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.924 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.925 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.925 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.925 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.925 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.925 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.925 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.925 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.926 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.926 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.926 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.926 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.926 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.926 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.926 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.927 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.927 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.927 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.927 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.927 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.927 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.928 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.928 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.928 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.928 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.928 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.928 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.928 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.929 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.929 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.929 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.929 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.929 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.929 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.930 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.930 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.930 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.930 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.930 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.930 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.930 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.931 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.931 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.931 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.931 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.931 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.931 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.931 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.931 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.931 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.932 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.932 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.932 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.932 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.932 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.932 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.932 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.933 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.933 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.933 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.933 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.933 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.933 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.933 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.934 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.934 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.934 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.934 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.934 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.934 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.934 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.935 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.935 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.935 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.935 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.935 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.935 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.935 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.935 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.936 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rook", 2024-05-10T11:32:17.936 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.936 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.936 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.936 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.936 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.937 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.937 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.937 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.937 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.937 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.937 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.937 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.937 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.938 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.938 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.938 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.938 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.938 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.938 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.938 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.938 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "storage_class": { 2024-05-10T11:32:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "storage_class", 2024-05-10T11:32:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "local", 2024-05-10T11:32:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-10T11:32:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "selftest", 2024-05-10T11:32:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "roption1": { 2024-05-10T11:32:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption1", 2024-05-10T11:32:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "roption2": { 2024-05-10T11:32:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption2", 2024-05-10T11:32:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "xyz", 2024-05-10T11:32:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption1": { 2024-05-10T11:32:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption1", 2024-05-10T11:32:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption2": { 2024-05-10T11:32:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption2", 2024-05-10T11:32:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption3": { 2024-05-10T11:32:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption3", 2024-05-10T11:32:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption4": { 2024-05-10T11:32:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption4", 2024-05-10T11:32:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption5": { 2024-05-10T11:32:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption5", 2024-05-10T11:32:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption6": { 2024-05-10T11:32:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption6", 2024-05-10T11:32:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption7": { 2024-05-10T11:32:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption7", 2024-05-10T11:32:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-05-10T11:32:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "42", 2024-05-10T11:32:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "testkey": { 2024-05-10T11:32:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testkey", 2024-05-10T11:32:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "testlkey": { 2024-05-10T11:32:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testlkey", 2024-05-10T11:32:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "testnewline": { 2024-05-10T11:32:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testnewline", 2024-05-10T11:32:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smb", 2024-05-10T11:32:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "update_orchestration": { 2024-05-10T11:32:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "update_orchestration", 2024-05-10T11:32:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-10T11:32:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-05-10T11:32:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_m_granularity": { 2024-05-10T11:32:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_m_granularity", 2024-05-10T11:32:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow minute scheduled snapshots", 2024-05-10T11:32:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "dump_on_update": { 2024-05-10T11:32:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dump_on_update", 2024-05-10T11:32:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "dump database to debug log on update", 2024-05-10T11:32:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stats", 2024-05-10T11:32:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "status", 2024-05-10T11:32:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telegraf", 2024-05-10T11:32:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "address": { 2024-05-10T11:32:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "address", 2024-05-10T11:32:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-10T11:32:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15", 2024-05-10T11:32:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telemetry", 2024-05-10T11:32:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_basic": { 2024-05-10T11:32:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_basic", 2024-05-10T11:32:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-10T11:32:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_crash": { 2024-05-10T11:32:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_crash", 2024-05-10T11:32:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-10T11:32:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_device": { 2024-05-10T11:32:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_device", 2024-05-10T11:32:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-10T11:32:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_ident": { 2024-05-10T11:32:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_ident", 2024-05-10T11:32:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-10T11:32:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_perf": { 2024-05-10T11:32:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_perf", 2024-05-10T11:32:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-10T11:32:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "contact": { 2024-05-10T11:32:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "contact", 2024-05-10T11:32:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "description": { 2024-05-10T11:32:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "description", 2024-05-10T11:32:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "device_url": { 2024-05-10T11:32:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_url", 2024-05-10T11:32:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-10T11:32:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-05-10T11:32:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-05-10T11:32:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "24", 2024-05-10T11:32:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "8", 2024-05-10T11:32:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "last_opt_revision": { 2024-05-10T11:32:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "last_opt_revision", 2024-05-10T11:32:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1", 2024-05-10T11:32:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard": { 2024-05-10T11:32:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard", 2024-05-10T11:32:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard_description": { 2024-05-10T11:32:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard_description", 2024-05-10T11:32:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.232 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.232 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.233 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.233 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.233 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.233 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.233 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.233 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.233 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.234 INFO:teuthology.orchestra.run.smithi027.stdout: "organization": { 2024-05-10T11:32:18.234 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "organization", 2024-05-10T11:32:18.234 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.234 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.234 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.234 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.234 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.235 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.235 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.235 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.235 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.235 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.235 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.235 INFO:teuthology.orchestra.run.smithi027.stdout: "proxy": { 2024-05-10T11:32:18.236 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "proxy", 2024-05-10T11:32:18.236 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.236 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.236 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.236 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.236 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.236 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.236 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.236 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.237 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.237 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.237 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.237 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.237 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.237 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.237 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.238 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.238 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.238 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.238 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.238 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.238 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.238 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.238 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.239 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.239 INFO:teuthology.orchestra.run.smithi027.stdout: "url": { 2024-05-10T11:32:18.239 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url", 2024-05-10T11:32:18.239 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.239 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.239 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.239 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-10T11:32:18.239 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.240 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.240 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.240 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.240 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.240 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.240 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.240 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.240 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "test_orchestrator", 2024-05-10T11:32:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.242 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.242 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.242 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.242 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.242 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.242 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.242 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.242 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.244 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.244 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.244 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.244 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.244 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.244 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.244 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.245 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.245 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.245 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.245 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.245 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.245 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.245 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.245 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.246 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.246 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.246 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.246 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.246 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.246 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.246 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.246 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.247 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.247 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.247 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.247 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.247 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.247 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.247 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.247 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.247 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.248 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.248 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.248 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.248 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.248 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.248 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.248 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.249 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.249 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.249 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.249 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.249 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.249 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.249 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.249 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.250 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.250 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.250 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.250 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.250 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.250 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.250 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.250 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.251 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.251 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.251 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.251 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.251 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.251 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.251 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-05-10T11:32:18.252 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.252 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.252 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.252 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.252 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.252 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.252 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.252 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.253 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.253 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.253 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.253 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.253 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.253 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.253 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.253 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.253 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.254 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.254 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.254 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.254 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.254 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.254 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.254 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.255 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.255 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.255 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.255 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.255 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.255 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.255 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.255 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.255 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.256 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.256 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.256 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.256 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.256 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.256 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.256 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.257 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.257 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.257 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.257 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.257 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.257 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.257 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.257 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.258 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.258 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.258 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.258 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.258 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.258 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.258 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.258 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.259 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.259 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.259 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.259 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.259 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.259 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.259 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.259 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_clones": { 2024-05-10T11:32:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_clones", 2024-05-10T11:32:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4", 2024-05-10T11:32:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-10T11:32:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: "periodic_async_work": { 2024-05-10T11:32:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "periodic_async_work", 2024-05-10T11:32:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Periodically check for async work", 2024-05-10T11:32:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_delay": { 2024-05-10T11:32:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_delay", 2024-05-10T11:32:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.265 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-10T11:32:18.265 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.265 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.265 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.265 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_no_wait": { 2024-05-10T11:32:18.265 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_no_wait", 2024-05-10T11:32:18.266 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.266 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.266 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.266 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.266 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.266 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.266 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.266 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-10T11:32:18.267 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.267 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.267 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.267 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.267 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.267 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.267 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.268 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.268 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.268 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.268 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.268 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.268 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.268 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.268 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.269 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.269 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.269 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.269 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix", 2024-05-10T11:32:18.270 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.270 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.270 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.270 INFO:teuthology.orchestra.run.smithi027.stdout: "discovery_interval": { 2024-05-10T11:32:18.270 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "discovery_interval", 2024-05-10T11:32:18.271 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:18.271 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.271 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.271 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "100", 2024-05-10T11:32:18.271 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.271 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.271 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.271 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.272 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.272 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.272 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.272 INFO:teuthology.orchestra.run.smithi027.stdout: "identifier": { 2024-05-10T11:32:18.272 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "identifier", 2024-05-10T11:32:18.272 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.272 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.273 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.273 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.273 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.273 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.273 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.273 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.273 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.273 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.273 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.274 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:18.274 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:18.274 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.274 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.274 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.274 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:18.274 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.275 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.275 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.275 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.275 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.275 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.275 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.275 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.276 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.276 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.276 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.276 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.276 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.276 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.276 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.276 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.276 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.277 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.277 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.277 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.277 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.277 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.277 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.277 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.277 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.278 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.278 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.278 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.278 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.278 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.278 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.278 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.279 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.279 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.279 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.279 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.279 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.279 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.279 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.279 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.280 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.280 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.280 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.280 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.280 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.280 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.280 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.280 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.281 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.281 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.281 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.281 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.281 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.281 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.281 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.281 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.282 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.282 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.282 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.282 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.282 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.282 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.283 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.283 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.283 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.283 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.283 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.283 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.283 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.283 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.285 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.285 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.286 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.286 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.286 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.286 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.286 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.286 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.287 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.287 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.287 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.287 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.287 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.287 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.287 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.287 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.288 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_host": { 2024-05-10T11:32:18.288 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_host", 2024-05-10T11:32:18.288 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.288 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.288 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.288 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.288 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.288 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.289 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.289 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.289 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.289 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.289 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.289 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_port": { 2024-05-10T11:32:18.289 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_port", 2024-05-10T11:32:18.289 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.290 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.290 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.290 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10051", 2024-05-10T11:32:18.290 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.290 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.290 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.290 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.291 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.291 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.291 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.291 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_sender": { 2024-05-10T11:32:18.291 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_sender", 2024-05-10T11:32:18.292 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.292 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.292 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.292 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-10T11:32:18.292 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.292 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.292 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.292 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.293 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.293 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.293 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.293 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.293 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.293 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.293 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-10T11:32:18.293 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.293 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.294 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-05-10T11:32:18.294 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-05-10T11:32:18.294 INFO:teuthology.orchestra.run.smithi027.stdout: "dashboard", 2024-05-10T11:32:18.294 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-05-10T11:32:18.294 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-05-10T11:32:18.294 INFO:teuthology.orchestra.run.smithi027.stdout: "restful", 2024-05-10T11:32:18.294 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_schedule" 2024-05-10T11:32:18.295 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.295 INFO:teuthology.orchestra.run.smithi027.stdout: "available_modules": [ 2024-05-10T11:32:18.295 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.295 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "alerts", 2024-05-10T11:32:18.295 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.295 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.295 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.295 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:18.296 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:18.296 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.296 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.296 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.296 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:18.296 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.296 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.296 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.297 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How frequently to reexamine health status", 2024-05-10T11:32:18.297 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.297 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.297 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.297 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.297 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.297 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.297 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.297 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.298 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.298 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.298 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.298 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.299 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.299 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.299 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.299 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.299 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.299 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.299 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.300 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.300 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.300 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.300 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.300 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.300 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.300 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.300 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.301 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.302 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.302 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.302 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.302 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.302 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.302 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.302 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.302 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.303 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.303 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.303 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.303 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.303 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.303 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.303 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.303 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.304 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.304 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.304 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.304 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.304 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.304 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.304 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.304 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.305 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.305 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.305 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.305 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.305 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.305 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.305 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.305 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.305 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.306 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.306 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.306 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_destination": { 2024-05-10T11:32:18.306 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_destination", 2024-05-10T11:32:18.306 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.306 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.306 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.306 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.307 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.307 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.307 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.307 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email address to send alerts to", 2024-05-10T11:32:18.307 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.307 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.307 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.308 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_from_name": { 2024-05-10T11:32:18.308 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_from_name", 2024-05-10T11:32:18.308 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.308 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.308 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.308 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "Ceph", 2024-05-10T11:32:18.308 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.308 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.309 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.309 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email From: name", 2024-05-10T11:32:18.309 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.309 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.309 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.309 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_host": { 2024-05-10T11:32:18.309 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_host", 2024-05-10T11:32:18.310 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.310 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.310 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.310 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.310 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.310 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.310 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.311 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP server", 2024-05-10T11:32:18.311 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.311 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.311 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.311 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.311 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_password": { 2024-05-10T11:32:18.311 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_password", 2024-05-10T11:32:18.311 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.311 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.312 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.312 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.312 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.312 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.312 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.312 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Password to authenticate with", 2024-05-10T11:32:18.312 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.312 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.313 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.313 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_port": { 2024-05-10T11:32:18.313 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_port", 2024-05-10T11:32:18.313 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.313 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.314 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.314 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "465", 2024-05-10T11:32:18.314 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.314 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.314 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.314 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP port", 2024-05-10T11:32:18.314 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.315 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.315 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.315 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_sender": { 2024-05-10T11:32:18.315 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_sender", 2024-05-10T11:32:18.315 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.315 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.315 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.316 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.316 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.316 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.316 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.316 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP envelope sender", 2024-05-10T11:32:18.316 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.317 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.317 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.317 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.317 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_ssl": { 2024-05-10T11:32:18.317 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_ssl", 2024-05-10T11:32:18.317 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.317 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.318 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.318 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.318 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.318 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.318 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.318 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-10T11:32:18.319 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.319 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.319 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.319 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_user": { 2024-05-10T11:32:18.319 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_user", 2024-05-10T11:32:18.319 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.319 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.320 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.320 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.320 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.320 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.320 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.320 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "User to authenticate as", 2024-05-10T11:32:18.321 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.321 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.321 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.321 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.321 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.321 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.322 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.322 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.322 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.322 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.322 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.322 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.323 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.323 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.323 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.323 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.323 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.324 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.324 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.324 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.324 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "balancer", 2024-05-10T11:32:18.324 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.324 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.325 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.325 INFO:teuthology.orchestra.run.smithi027.stdout: "active": { 2024-05-10T11:32:18.325 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "active", 2024-05-10T11:32:18.325 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.325 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.326 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.326 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.326 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.326 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.326 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.326 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically balance PGs across cluster", 2024-05-10T11:32:18.327 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.327 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.327 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.327 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_time": { 2024-05-10T11:32:18.327 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_time", 2024-05-10T11:32:18.327 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.328 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.328 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.328 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0000", 2024-05-10T11:32:18.328 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.328 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.328 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.328 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "beginning time of day to automatically balance", 2024-05-10T11:32:18.329 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-10T11:32:18.329 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.329 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.329 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.329 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_weekday": { 2024-05-10T11:32:18.329 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_weekday", 2024-05-10T11:32:18.329 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:18.329 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.330 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.330 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.330 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-05-10T11:32:18.330 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-05-10T11:32:18.330 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.330 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-10T11:32:18.330 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-10T11:32:18.331 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.331 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.331 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.331 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_max_iterations": { 2024-05-10T11:32:18.331 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_max_iterations", 2024-05-10T11:32:18.331 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:18.331 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.332 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.332 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "25", 2024-05-10T11:32:18.332 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-05-10T11:32:18.332 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "250", 2024-05-10T11:32:18.332 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.332 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-10T11:32:18.332 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.332 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.333 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.333 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_metrics": { 2024-05-10T11:32:18.333 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_metrics", 2024-05-10T11:32:18.333 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.333 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.333 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.333 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "pgs,objects,bytes", 2024-05-10T11:32:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-10T11:32:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-10T11:32:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.335 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_step": { 2024-05-10T11:32:18.335 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_step", 2024-05-10T11:32:18.335 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:18.335 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.335 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.335 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.5", 2024-05-10T11:32:18.336 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0.001", 2024-05-10T11:32:18.336 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "0.999", 2024-05-10T11:32:18.336 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.336 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "aggressiveness of optimization", 2024-05-10T11:32:18.336 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-10T11:32:18.336 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.336 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.336 INFO:teuthology.orchestra.run.smithi027.stdout: "end_time": { 2024-05-10T11:32:18.337 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_time", 2024-05-10T11:32:18.337 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.337 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.337 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.337 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2359", 2024-05-10T11:32:18.337 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.337 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.338 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.338 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ending time of day to automatically balance", 2024-05-10T11:32:18.338 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-10T11:32:18.338 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.338 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.339 INFO:teuthology.orchestra.run.smithi027.stdout: "end_weekday": { 2024-05-10T11:32:18.339 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_weekday", 2024-05-10T11:32:18.339 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:18.339 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.339 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.339 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.340 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-05-10T11:32:18.340 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-05-10T11:32:18.340 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.340 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-10T11:32:18.340 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-10T11:32:18.340 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.341 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.341 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.341 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.341 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.341 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.341 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.342 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.342 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.342 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.342 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.342 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.342 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.343 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.343 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.343 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.343 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.343 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.343 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.343 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.343 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.344 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.344 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.344 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.344 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.344 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.344 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.344 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.345 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.345 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.345 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.345 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.345 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.345 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.345 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.346 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.346 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.346 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.346 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.346 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.346 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.347 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.347 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.347 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.347 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.347 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.347 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.347 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.347 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.348 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.348 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.348 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.348 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.348 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.348 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.348 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.349 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.349 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.349 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.349 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.349 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.349 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.350 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.350 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.350 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.350 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.350 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.350 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.351 INFO:teuthology.orchestra.run.smithi027.stdout: "min_score": { 2024-05-10T11:32:18.351 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_score", 2024-05-10T11:32:18.351 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:18.351 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.351 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.351 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.352 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.352 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.352 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.352 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-10T11:32:18.352 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.353 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.353 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.353 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-05-10T11:32:18.353 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-05-10T11:32:18.353 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.353 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.353 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.354 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "upmap", 2024-05-10T11:32:18.354 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.354 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.354 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.354 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-compat", 2024-05-10T11:32:18.355 INFO:teuthology.orchestra.run.smithi027.stdout: "none", 2024-05-10T11:32:18.355 INFO:teuthology.orchestra.run.smithi027.stdout: "read", 2024-05-10T11:32:18.355 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap", 2024-05-10T11:32:18.355 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap-read" 2024-05-10T11:32:18.355 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.355 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Balancer mode", 2024-05-10T11:32:18.355 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.355 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.356 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.356 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ids": { 2024-05-10T11:32:18.356 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_ids", 2024-05-10T11:32:18.357 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.357 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.357 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.357 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.357 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.357 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.357 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.357 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-10T11:32:18.358 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.358 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.358 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.359 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:18.359 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:18.359 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.359 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.359 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.359 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:18.359 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.360 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.360 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.360 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-10T11:32:18.360 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.360 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.360 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.361 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.361 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.361 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.361 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.361 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.362 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.362 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.362 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.362 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.362 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.362 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.362 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.363 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.363 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_deviation": { 2024-05-10T11:32:18.363 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_deviation", 2024-05-10T11:32:18.363 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.363 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.364 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.364 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-05-10T11:32:18.364 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-05-10T11:32:18.364 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.364 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.364 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-10T11:32:18.364 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-10T11:32:18.364 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.365 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.365 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_optimizations": { 2024-05-10T11:32:18.365 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_optimizations", 2024-05-10T11:32:18.365 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:18.365 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.365 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.365 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:18.366 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.366 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.366 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.366 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-10T11:32:18.366 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.366 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.366 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.367 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.367 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.367 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.367 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.367 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm", 2024-05-10T11:32:18.367 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.367 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.368 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.368 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_down_multiplier": { 2024-05-10T11:32:18.368 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_down_multiplier", 2024-05-10T11:32:18.368 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:18.368 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.368 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.368 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-05-10T11:32:18.368 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.369 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.369 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.369 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-10T11:32:18.369 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.369 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.369 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.369 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_refresh_rate": { 2024-05-10T11:32:18.370 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_refresh_rate", 2024-05-10T11:32:18.370 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.370 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.370 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.370 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "20", 2024-05-10T11:32:18.370 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.370 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.371 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.371 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-10T11:32:18.371 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.371 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.371 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.371 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_starting_port": { 2024-05-10T11:32:18.372 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_starting_port", 2024-05-10T11:32:18.372 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.372 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.372 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.372 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4721", 2024-05-10T11:32:18.372 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.372 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.372 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.373 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-10T11:32:18.373 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.373 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.373 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.373 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_ptrace": { 2024-05-10T11:32:18.373 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_ptrace", 2024-05-10T11:32:18.374 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.374 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.374 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.374 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.374 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.374 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.374 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.375 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-10T11:32:18.375 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-10T11:32:18.375 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.375 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.375 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.375 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_interval": { 2024-05-10T11:32:18.375 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_interval", 2024-05-10T11:32:18.376 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.376 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.376 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.376 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:18.376 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.376 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.376 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.377 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-10T11:32:18.377 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.377 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.377 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.377 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.377 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_memory_target_ratio": { 2024-05-10T11:32:18.377 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_memory_target_ratio", 2024-05-10T11:32:18.377 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:18.378 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.378 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.378 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.7", 2024-05-10T11:32:18.378 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.378 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.378 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.378 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-10T11:32:18.379 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.379 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.379 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.379 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_log_destination": { 2024-05-10T11:32:18.379 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm_log_destination", 2024-05-10T11:32:18.379 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.380 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.380 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.380 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.380 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.380 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.380 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.380 INFO:teuthology.orchestra.run.smithi027.stdout: "file", 2024-05-10T11:32:18.380 INFO:teuthology.orchestra.run.smithi027.stdout: "file,syslog", 2024-05-10T11:32:18.381 INFO:teuthology.orchestra.run.smithi027.stdout: "syslog" 2024-05-10T11:32:18.381 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.381 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-10T11:32:18.381 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.381 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.381 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.381 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.381 INFO:teuthology.orchestra.run.smithi027.stdout: "cgroups_split": { 2024-05-10T11:32:18.382 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cgroups_split", 2024-05-10T11:32:18.382 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.382 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.382 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.382 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.382 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.382 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.382 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.383 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-10T11:32:18.383 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.383 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.383 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.383 INFO:teuthology.orchestra.run.smithi027.stdout: "config_checks_enabled": { 2024-05-10T11:32:18.383 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_checks_enabled", 2024-05-10T11:32:18.383 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.384 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.384 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.384 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.384 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.384 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.384 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.384 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-10T11:32:18.384 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.384 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.385 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.385 INFO:teuthology.orchestra.run.smithi027.stdout: "config_dashboard": { 2024-05-10T11:32:18.385 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_dashboard", 2024-05-10T11:32:18.385 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.386 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.386 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.386 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.386 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.386 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.386 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.386 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-10T11:32:18.387 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.387 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.387 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.387 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_alertmanager": { 2024-05-10T11:32:18.387 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_alertmanager", 2024-05-10T11:32:18.387 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.387 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.388 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.388 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-10T11:32:18.388 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.388 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.388 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.388 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-05-10T11:32:18.388 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.388 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.389 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.389 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_base": { 2024-05-10T11:32:18.389 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_base", 2024-05-10T11:32:18.389 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.389 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.389 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.389 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-10T11:32:18.390 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.390 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.390 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.390 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Container image name, without the tag", 2024-05-10T11:32:18.390 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.390 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.390 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.390 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_elasticsearch": { 2024-05-10T11:32:18.391 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_elasticsearch", 2024-05-10T11:32:18.391 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.391 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.391 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.391 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-10T11:32:18.391 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.391 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.391 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.392 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "elasticsearch container image", 2024-05-10T11:32:18.392 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.392 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.392 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.392 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_grafana": { 2024-05-10T11:32:18.392 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_grafana", 2024-05-10T11:32:18.392 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.392 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.393 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.393 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-10T11:32:18.393 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.393 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.393 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.393 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-05-10T11:32:18.393 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.393 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.394 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.394 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_haproxy": { 2024-05-10T11:32:18.394 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_haproxy", 2024-05-10T11:32:18.394 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.394 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.394 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.394 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-10T11:32:18.394 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.395 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.395 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.395 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "HAproxy container image", 2024-05-10T11:32:18.395 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.395 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.395 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.395 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_agent": { 2024-05-10T11:32:18.396 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_agent", 2024-05-10T11:32:18.396 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.396 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.396 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.396 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-10T11:32:18.396 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.396 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.396 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.396 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger agent container image", 2024-05-10T11:32:18.397 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.397 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.397 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.397 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_collector": { 2024-05-10T11:32:18.397 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_collector", 2024-05-10T11:32:18.397 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.397 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.397 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.398 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-10T11:32:18.398 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.398 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.398 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.398 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger collector container image", 2024-05-10T11:32:18.398 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.398 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.399 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.399 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_query": { 2024-05-10T11:32:18.399 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_query", 2024-05-10T11:32:18.399 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.399 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.399 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.399 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-10T11:32:18.400 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.400 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.400 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.400 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger query container image", 2024-05-10T11:32:18.400 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.400 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.400 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.401 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_keepalived": { 2024-05-10T11:32:18.401 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_keepalived", 2024-05-10T11:32:18.401 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.401 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.401 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.401 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-10T11:32:18.401 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.401 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.402 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.402 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Keepalived container image", 2024-05-10T11:32:18.402 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.402 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.402 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.402 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_loki": { 2024-05-10T11:32:18.402 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_loki", 2024-05-10T11:32:18.402 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.403 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.403 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.403 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-10T11:32:18.403 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.403 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.403 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.403 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Loki container image", 2024-05-10T11:32:18.403 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.404 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.404 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.404 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_node_exporter": { 2024-05-10T11:32:18.404 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_node_exporter", 2024-05-10T11:32:18.404 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.404 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.404 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.405 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-10T11:32:18.405 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.405 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.405 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.405 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-05-10T11:32:18.405 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.405 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.405 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.406 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_nvmeof": { 2024-05-10T11:32:18.406 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_nvmeof", 2024-05-10T11:32:18.406 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.406 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.406 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.406 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-10T11:32:18.407 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.407 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.407 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.407 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Nvme-of container image", 2024-05-10T11:32:18.407 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.407 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.407 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.408 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_prometheus": { 2024-05-10T11:32:18.408 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_prometheus", 2024-05-10T11:32:18.408 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.408 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.408 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.408 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-10T11:32:18.408 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.409 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.409 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.409 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-05-10T11:32:18.409 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.409 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.409 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.409 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_promtail": { 2024-05-10T11:32:18.410 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_promtail", 2024-05-10T11:32:18.410 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.410 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.410 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.410 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-10T11:32:18.410 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.410 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.410 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.411 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Promtail container image", 2024-05-10T11:32:18.411 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.411 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.411 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.411 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.411 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_samba": { 2024-05-10T11:32:18.411 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_samba", 2024-05-10T11:32:18.411 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.411 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.412 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.412 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-10T11:32:18.412 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.412 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.412 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.412 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Samba/SMB container image", 2024-05-10T11:32:18.412 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.412 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.413 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.413 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_snmp_gateway": { 2024-05-10T11:32:18.413 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_snmp_gateway", 2024-05-10T11:32:18.413 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.413 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.413 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.413 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-10T11:32:18.414 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.414 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.414 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.414 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SNMP Gateway container image", 2024-05-10T11:32:18.414 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.414 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.414 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.414 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.415 INFO:teuthology.orchestra.run.smithi027.stdout: "container_init": { 2024-05-10T11:32:18.415 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_init", 2024-05-10T11:32:18.415 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.415 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.415 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.415 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.415 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.415 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.415 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.416 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Run podman/docker with `--init`", 2024-05-10T11:32:18.416 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.416 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.416 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.416 INFO:teuthology.orchestra.run.smithi027.stdout: "daemon_cache_timeout": { 2024-05-10T11:32:18.416 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "daemon_cache_timeout", 2024-05-10T11:32:18.416 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.417 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.417 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.417 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:18.417 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.417 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.417 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.417 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-10T11:32:18.417 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.418 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.418 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.418 INFO:teuthology.orchestra.run.smithi027.stdout: "default_cephadm_command_timeout": { 2024-05-10T11:32:18.418 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_cephadm_command_timeout", 2024-05-10T11:32:18.418 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.418 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.418 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.419 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "900", 2024-05-10T11:32:18.419 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.419 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.419 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.419 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-10T11:32:18.419 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.419 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.419 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.420 INFO:teuthology.orchestra.run.smithi027.stdout: "default_registry": { 2024-05-10T11:32:18.420 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_registry", 2024-05-10T11:32:18.420 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.420 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.420 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.420 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io", 2024-05-10T11:32:18.420 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.420 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.421 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.421 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-10T11:32:18.421 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.421 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.421 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.421 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.421 INFO:teuthology.orchestra.run.smithi027.stdout: "device_cache_timeout": { 2024-05-10T11:32:18.421 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_cache_timeout", 2024-05-10T11:32:18.422 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.422 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.422 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.422 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1800", 2024-05-10T11:32:18.422 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.422 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.422 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.422 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache device inventory", 2024-05-10T11:32:18.423 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.423 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.423 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.423 INFO:teuthology.orchestra.run.smithi027.stdout: "device_enhanced_scan": { 2024-05-10T11:32:18.423 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_enhanced_scan", 2024-05-10T11:32:18.423 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.424 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.424 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.424 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.424 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.424 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.424 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.424 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-10T11:32:18.424 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.425 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.425 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.425 INFO:teuthology.orchestra.run.smithi027.stdout: "facts_cache_timeout": { 2024-05-10T11:32:18.425 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "facts_cache_timeout", 2024-05-10T11:32:18.425 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.425 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.425 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.426 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:18.426 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.426 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.426 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.426 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache host facts data", 2024-05-10T11:32:18.426 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.426 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.426 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.427 INFO:teuthology.orchestra.run.smithi027.stdout: "grafana_dashboards_path": { 2024-05-10T11:32:18.427 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "grafana_dashboards_path", 2024-05-10T11:32:18.427 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.427 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.427 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.427 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-10T11:32:18.427 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.428 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.428 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.428 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-10T11:32:18.428 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.428 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.428 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.428 INFO:teuthology.orchestra.run.smithi027.stdout: "host_check_interval": { 2024-05-10T11:32:18.429 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "host_check_interval", 2024-05-10T11:32:18.429 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.429 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.429 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.429 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:18.429 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.429 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.429 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.429 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to perform a host check", 2024-05-10T11:32:18.430 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.430 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.430 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.430 INFO:teuthology.orchestra.run.smithi027.stdout: "hw_monitoring": { 2024-05-10T11:32:18.430 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hw_monitoring", 2024-05-10T11:32:18.430 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.430 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.431 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.431 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.431 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.431 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.431 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.431 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-10T11:32:18.431 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.431 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.432 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.432 INFO:teuthology.orchestra.run.smithi027.stdout: "inventory_list_all": { 2024-05-10T11:32:18.432 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "inventory_list_all", 2024-05-10T11:32:18.432 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.432 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.432 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.432 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.432 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.433 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.433 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.433 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-10T11:32:18.433 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.433 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.433 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.433 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.434 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.434 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.434 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.434 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.434 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.434 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.434 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.434 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.434 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.435 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.435 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.435 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.435 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.435 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.435 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.435 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.435 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.435 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.436 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.436 INFO:teuthology.orchestra.run.smithi027.stdout: "log_refresh_metadata": { 2024-05-10T11:32:18.436 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_refresh_metadata", 2024-05-10T11:32:18.436 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.436 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.436 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.436 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.437 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.437 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.437 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.437 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-10T11:32:18.437 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.437 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.437 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.437 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.438 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.438 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.438 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.438 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.438 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.438 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.438 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.438 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.439 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-10T11:32:18.439 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.439 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.439 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.439 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.439 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.439 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.439 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.440 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.440 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.440 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.440 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.440 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.440 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.440 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.440 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.440 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.441 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.441 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.441 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.441 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.441 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.441 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.441 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.442 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.442 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.442 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.442 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.442 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.442 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.442 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.442 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.443 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.443 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.443 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.443 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.443 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.443 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-10T11:32:18.443 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-10T11:32:18.444 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.444 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.444 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.444 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.444 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.444 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.444 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.444 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-10T11:32:18.444 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.445 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.445 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.445 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-10T11:32:18.445 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-10T11:32:18.445 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.445 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.445 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "*", 2024-05-10T11:32:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-10T11:32:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "max_count_per_host": { 2024-05-10T11:32:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_count_per_host", 2024-05-10T11:32:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of daemons per service per host", 2024-05-10T11:32:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd_draining_count": { 2024-05-10T11:32:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_osd_draining_count", 2024-05-10T11:32:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-10T11:32:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "migration_current": { 2024-05-10T11:32:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "migration_current", 2024-05-10T11:32:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "internal - do not modify", 2024-05-10T11:32:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-05-10T11:32:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-05-10T11:32:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "root", 2024-05-10T11:32:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm-package", 2024-05-10T11:32:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "root" 2024-05-10T11:32:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "mode for remote execution of cephadm", 2024-05-10T11:32:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "oob_default_addr": { 2024-05-10T11:32:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "oob_default_addr", 2024-05-10T11:32:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "169.254.1.1", 2024-05-10T11:32:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-10T11:32:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "prometheus_alerts_path": { 2024-05-10T11:32:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus_alerts_path", 2024-05-10T11:32:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-10T11:32:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-10T11:32:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_insecure": { 2024-05-10T11:32:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_insecure", 2024-05-10T11:32:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-10T11:32:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_password": { 2024-05-10T11:32:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_password", 2024-05-10T11:32:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-10T11:32:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_url": { 2024-05-10T11:32:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_url", 2024-05-10T11:32:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-10T11:32:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_username": { 2024-05-10T11:32:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_username", 2024-05-10T11:32:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-10T11:32:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "secure_monitoring_stack": { 2024-05-10T11:32:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "secure_monitoring_stack", 2024-05-10T11:32:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-10T11:32:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "service_discovery_port": { 2024-05-10T11:32:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "service_discovery_port", 2024-05-10T11:32:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8765", 2024-05-10T11:32:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "cephadm service discovery port", 2024-05-10T11:32:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "ssh_config_file": { 2024-05-10T11:32:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssh_config_file", 2024-05-10T11:32:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-10T11:32:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "use_agent": { 2024-05-10T11:32:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_agent", 2024-05-10T11:32:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-10T11:32:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "use_repo_digest": { 2024-05-10T11:32:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_repo_digest", 2024-05-10T11:32:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-10T11:32:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_failed_host_check": { 2024-05-10T11:32:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_failed_host_check", 2024-05-10T11:32:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if the host check fails", 2024-05-10T11:32:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_daemons": { 2024-05-10T11:32:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_daemons", 2024-05-10T11:32:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-10T11:32:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_hosts": { 2024-05-10T11:32:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_hosts", 2024-05-10T11:32:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-10T11:32:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crash", 2024-05-10T11:32:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "retain_interval": { 2024-05-10T11:32:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retain_interval", 2024-05-10T11:32:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "31536000", 2024-05-10T11:32:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-10T11:32:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_recent_interval": { 2024-05-10T11:32:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_recent_interval", 2024-05-10T11:32:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1209600", 2024-05-10T11:32:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-10T11:32:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dashboard", 2024-05-10T11:32:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.498 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.498 INFO:teuthology.orchestra.run.smithi027.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-10T11:32:18.498 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-10T11:32:18.498 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.498 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.499 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.499 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:18.499 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.499 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.499 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.499 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.499 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.499 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.499 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.500 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_HOST": { 2024-05-10T11:32:18.500 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-10T11:32:18.500 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.500 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.500 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.500 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.500 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.501 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.501 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.501 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.501 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.501 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.501 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-10T11:32:18.502 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-10T11:32:18.502 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.502 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.502 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.502 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.502 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.502 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.503 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.503 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.503 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.503 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.503 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.503 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_ENABLED": { 2024-05-10T11:32:18.503 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_ENABLED", 2024-05-10T11:32:18.504 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.504 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.504 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.504 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.504 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.504 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.504 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.504 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.505 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.505 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.505 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.505 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-10T11:32:18.505 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-10T11:32:18.505 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.505 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.506 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.506 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.506 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.506 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.506 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.506 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.506 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.506 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: "ENABLE_BROWSABLE_API": { 2024-05-10T11:32:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-10T11:32:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.508 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.508 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.508 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.508 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.508 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.508 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.508 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-10T11:32:18.509 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-10T11:32:18.509 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.509 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.509 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.509 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.509 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.509 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.509 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.510 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.510 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.510 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.510 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.510 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-10T11:32:18.510 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-10T11:32:18.510 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.511 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.511 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.511 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.511 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.511 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.511 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.511 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.511 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.511 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.512 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.512 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-10T11:32:18.512 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-10T11:32:18.512 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.512 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.512 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.512 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.513 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.513 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.513 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.513 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.513 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.513 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.513 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.514 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-10T11:32:18.514 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-10T11:32:18.514 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.514 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.514 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.514 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.514 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.514 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.514 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.515 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.515 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.515 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.515 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.515 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-10T11:32:18.515 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-10T11:32:18.515 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.516 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.516 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.516 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.516 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.516 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.516 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.516 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.516 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.517 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.517 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.517 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-10T11:32:18.517 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-10T11:32:18.517 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.517 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.517 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.517 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.518 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.518 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.518 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.518 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.518 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.518 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.518 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.519 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-10T11:32:18.519 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-10T11:32:18.519 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.519 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.519 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.519 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.519 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.519 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.520 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.520 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.520 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.520 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.520 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.520 INFO:teuthology.orchestra.run.smithi027.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-10T11:32:18.520 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-10T11:32:18.520 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.521 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.521 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.521 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.521 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.521 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.521 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.521 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.521 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.522 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.522 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.522 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_PASSWORD": { 2024-05-10T11:32:18.522 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-10T11:32:18.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.522 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.522 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.522 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-05-10T11:32:18.523 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.523 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.523 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.523 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.523 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.523 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.523 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.524 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-10T11:32:18.524 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-10T11:32:18.524 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.524 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.524 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.524 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.524 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.524 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.524 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.525 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.525 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.525 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.525 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.525 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_URL": { 2024-05-10T11:32:18.525 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_URL", 2024-05-10T11:32:18.525 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.526 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.526 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.526 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.526 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.526 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.526 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.526 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.526 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.527 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.527 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.527 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_USERNAME": { 2024-05-10T11:32:18.527 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-10T11:32:18.527 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.527 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.527 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.527 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-05-10T11:32:18.528 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.528 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.528 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.528 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.528 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.528 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.528 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.529 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-10T11:32:18.529 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-10T11:32:18.529 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.529 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.529 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.529 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.529 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.529 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.530 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.530 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.530 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.530 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.530 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.530 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-10T11:32:18.530 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-10T11:32:18.531 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.531 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.531 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.531 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.531 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.531 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.531 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.531 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.531 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.532 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.532 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.532 INFO:teuthology.orchestra.run.smithi027.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-10T11:32:18.532 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-10T11:32:18.532 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.532 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.532 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.533 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.533 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.533 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.533 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.533 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.533 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.533 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.533 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.534 INFO:teuthology.orchestra.run.smithi027.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-10T11:32:18.534 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-10T11:32:18.534 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.534 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.534 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.534 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.534 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.534 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.535 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.535 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.535 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.535 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.535 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.535 INFO:teuthology.orchestra.run.smithi027.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-10T11:32:18.535 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-10T11:32:18.536 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.536 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.536 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.536 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "[]", 2024-05-10T11:32:18.536 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.536 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.536 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.536 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.536 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.537 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.537 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.537 INFO:teuthology.orchestra.run.smithi027.stdout: "MULTICLUSTER_CONFIG": { 2024-05-10T11:32:18.537 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-10T11:32:18.537 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.537 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.537 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.537 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "{}", 2024-05-10T11:32:18.538 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.538 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.538 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.538 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.538 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.538 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.538 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_HOST": { 2024-05-10T11:32:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-10T11:32:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-10T11:32:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-10T11:32:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-10T11:32:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-10T11:32:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-10T11:32:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-10T11:32:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-10T11:32:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-10T11:32:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-10T11:32:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-10T11:32:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-10T11:32:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-10T11:32:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-10T11:32:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-10T11:32:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-10T11:32:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-10T11:32:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_ENABLED": { 2024-05-10T11:32:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-10T11:32:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-10T11:32:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-10T11:32:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-10T11:32:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-10T11:32:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-10T11:32:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-10T11:32:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-10T11:32:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8", 2024-05-10T11:32:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-10T11:32:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-10T11:32:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "45", 2024-05-10T11:32:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ACCESS_KEY": { 2024-05-10T11:32:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-10T11:32:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-10T11:32:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-10T11:32:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-05-10T11:32:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SECRET_KEY": { 2024-05-10T11:32:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-10T11:32:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SSL_VERIFY": { 2024-05-10T11:32:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-10T11:32:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "UNSAFE_TLS_v1_2": { 2024-05-10T11:32:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-10T11:32:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-10T11:32:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-10T11:32:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-10T11:32:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-10T11:32:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-10T11:32:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-10T11:32:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-05-10T11:32:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "cross_origin_url": { 2024-05-10T11:32:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cross_origin_url", 2024-05-10T11:32:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "crt_file": { 2024-05-10T11:32:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crt_file", 2024-05-10T11:32:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "debug": { 2024-05-10T11:32:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "debug", 2024-05-10T11:32:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable/disable debug options", 2024-05-10T11:32:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "jwt_token_ttl": { 2024-05-10T11:32:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "jwt_token_ttl", 2024-05-10T11:32:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "28800", 2024-05-10T11:32:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-05-10T11:32:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-05-10T11:32:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "motd": { 2024-05-10T11:32:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "motd", 2024-05-10T11:32:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "The message of the day", 2024-05-10T11:32:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect_resolve_ip_addr": { 2024-05-10T11:32:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "redirect_resolve_ip_addr", 2024-05-10T11:32:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-05-10T11:32:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-05-10T11:32:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-05-10T11:32:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-05-10T11:32:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-05-10T11:32:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8080", 2024-05-10T11:32:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-05-10T11:32:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-05-10T11:32:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl_server_port": { 2024-05-10T11:32:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl_server_port", 2024-05-10T11:32:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8443", 2024-05-10T11:32:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-05-10T11:32:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-05-10T11:32:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "redirect", 2024-05-10T11:32:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect" 2024-05-10T11:32:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-05-10T11:32:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-05-10T11:32:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-05-10T11:32:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-05-10T11:32:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-05-10T11:32:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "url_prefix": { 2024-05-10T11:32:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url_prefix", 2024-05-10T11:32:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-05-10T11:32:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_monitoring": { 2024-05-10T11:32:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_monitoring", 2024-05-10T11:32:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "monitor device health metrics", 2024-05-10T11:32:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "mark_out_threshold": { 2024-05-10T11:32:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mark_out_threshold", 2024-05-10T11:32:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2419200", 2024-05-10T11:32:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-10T11:32:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": { 2024-05-10T11:32:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_name", 2024-05-10T11:32:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "device_health_metrics", 2024-05-10T11:32:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-10T11:32:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "retention_period": { 2024-05-10T11:32:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retention_period", 2024-05-10T11:32:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15552000", 2024-05-10T11:32:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain device health metrics", 2024-05-10T11:32:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_frequency": { 2024-05-10T11:32:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_frequency", 2024-05-10T11:32:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-05-10T11:32:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-10T11:32:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "self_heal": { 2024-05-10T11:32:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "self_heal", 2024-05-10T11:32:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-10T11:32:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and check device health", 2024-05-10T11:32:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_threshold": { 2024-05-10T11:32:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_threshold", 2024-05-10T11:32:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7257600", 2024-05-10T11:32:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-10T11:32:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "diskprediction_local", 2024-05-10T11:32:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: "predict_interval": { 2024-05-10T11:32:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predict_interval", 2024-05-10T11:32:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-05-10T11:32:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.645 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.645 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.645 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.645 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.645 INFO:teuthology.orchestra.run.smithi027.stdout: "predictor_model": { 2024-05-10T11:32:18.645 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predictor_model", 2024-05-10T11:32:18.646 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.646 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.646 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.646 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "prophetstor", 2024-05-10T11:32:18.646 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.646 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.646 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.647 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.647 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.647 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.647 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.647 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:18.647 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:18.647 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.648 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.648 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.648 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-05-10T11:32:18.648 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.648 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.648 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.648 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.649 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.649 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.649 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.649 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.649 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.649 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.649 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.650 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.650 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.650 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.650 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.650 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.650 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.650 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.650 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.651 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.651 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.651 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.651 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "influx", 2024-05-10T11:32:18.651 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": false, 2024-05-10T11:32:18.652 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "influxdb python module not found", 2024-05-10T11:32:18.652 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.652 INFO:teuthology.orchestra.run.smithi027.stdout: "batch_size": { 2024-05-10T11:32:18.652 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "batch_size", 2024-05-10T11:32:18.652 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.652 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.652 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.652 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5000", 2024-05-10T11:32:18.653 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.653 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.653 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.653 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-10T11:32:18.653 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.653 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.653 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.654 INFO:teuthology.orchestra.run.smithi027.stdout: "database": { 2024-05-10T11:32:18.654 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "database", 2024-05-10T11:32:18.654 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.654 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.654 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.654 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "ceph", 2024-05-10T11:32:18.654 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.654 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.655 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.655 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-10T11:32:18.655 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.655 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.655 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.655 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": { 2024-05-10T11:32:18.655 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hostname", 2024-05-10T11:32:18.655 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.656 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.656 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.656 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.656 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.656 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.656 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.656 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server hostname", 2024-05-10T11:32:18.657 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.657 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.657 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.657 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:18.657 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:18.657 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.657 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.658 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.658 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "30", 2024-05-10T11:32:18.658 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "5", 2024-05-10T11:32:18.658 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.658 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.658 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-10T11:32:18.658 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.658 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.659 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.659 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.659 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.659 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.659 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.659 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.659 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.660 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.660 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.660 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.660 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.660 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.660 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.660 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.660 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.661 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.661 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.661 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.661 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.661 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.661 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.661 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.661 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.662 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.662 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.662 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.662 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.662 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.662 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.662 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.663 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.663 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.663 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.663 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.663 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.663 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.663 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.664 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.664 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.664 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.664 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.664 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.664 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.664 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.664 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.665 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.665 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.665 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.665 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.665 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.665 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.665 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.665 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.666 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.666 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.666 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.666 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.666 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.666 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.666 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.667 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.667 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.667 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.667 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.667 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.667 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.667 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.668 INFO:teuthology.orchestra.run.smithi027.stdout: "password": { 2024-05-10T11:32:18.668 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "password", 2024-05-10T11:32:18.668 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.668 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.668 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.668 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.668 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.668 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.669 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.669 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "password of InfluxDB server user", 2024-05-10T11:32:18.669 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.669 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.669 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.669 INFO:teuthology.orchestra.run.smithi027.stdout: "port": { 2024-05-10T11:32:18.669 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "port", 2024-05-10T11:32:18.670 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.670 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.670 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.670 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8086", 2024-05-10T11:32:18.670 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.670 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.670 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.670 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server port", 2024-05-10T11:32:18.671 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.671 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.671 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.671 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.671 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.671 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.671 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.672 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.672 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.672 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.672 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.672 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.672 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.672 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.672 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.673 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.673 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-05-10T11:32:18.673 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-05-10T11:32:18.673 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.673 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.673 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.673 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "false", 2024-05-10T11:32:18.674 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.674 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.674 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.674 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-10T11:32:18.674 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.674 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.674 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.675 INFO:teuthology.orchestra.run.smithi027.stdout: "threads": { 2024-05-10T11:32:18.675 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threads", 2024-05-10T11:32:18.675 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.675 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.675 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.675 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-05-10T11:32:18.675 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-05-10T11:32:18.675 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "32", 2024-05-10T11:32:18.676 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.676 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-10T11:32:18.676 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.676 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.676 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.676 INFO:teuthology.orchestra.run.smithi027.stdout: "username": { 2024-05-10T11:32:18.676 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "username", 2024-05-10T11:32:18.677 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.677 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.677 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.677 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.677 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.677 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.677 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.677 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "username of InfluxDB server user", 2024-05-10T11:32:18.678 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.678 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.678 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.678 INFO:teuthology.orchestra.run.smithi027.stdout: "verify_ssl": { 2024-05-10T11:32:18.678 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "verify_ssl", 2024-05-10T11:32:18.678 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.678 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.679 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.679 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "true", 2024-05-10T11:32:18.679 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.679 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.679 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.679 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-10T11:32:18.679 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.679 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.680 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.680 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.680 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.680 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "insights", 2024-05-10T11:32:18.680 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.680 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.680 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.681 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.681 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.681 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.681 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.681 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.681 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.681 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.682 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.682 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.682 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.682 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.682 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.682 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.682 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.682 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.683 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.683 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.683 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.683 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.683 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.683 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.683 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.683 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.684 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.684 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.684 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.684 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.684 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.684 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.684 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.684 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.684 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.685 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.685 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.685 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.685 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.685 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.685 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.685 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.685 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.686 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.686 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.686 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.686 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.686 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.686 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.686 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.686 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.687 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.687 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.687 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.687 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.687 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.687 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.687 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.688 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.688 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.688 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.688 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.688 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.688 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.688 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.688 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.688 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.689 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.689 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.689 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.689 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.689 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.689 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.689 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.690 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.690 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.690 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.690 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.690 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.690 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.690 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.690 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.691 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "iostat", 2024-05-10T11:32:18.691 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.691 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.691 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.691 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.691 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.691 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.692 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.692 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.692 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.692 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.692 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.692 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.692 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.692 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.692 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.693 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.693 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.693 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.693 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.693 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.693 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.693 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.693 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.694 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.694 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.694 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.694 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.694 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.694 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.694 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.695 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.695 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.698 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.698 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.698 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.698 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.698 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.698 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.698 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.699 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.699 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.699 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.699 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.699 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.699 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.699 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.699 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.700 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.700 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.700 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.700 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.700 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.700 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.700 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.700 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.700 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.701 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.701 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.701 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.701 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.701 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.701 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.701 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.702 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.702 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.702 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.702 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.702 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.702 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.702 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.703 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.703 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.703 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.703 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.703 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.703 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.703 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.704 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.704 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.704 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.704 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.704 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.704 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.705 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "k8sevents", 2024-05-10T11:32:18.705 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.705 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.705 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.705 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_event_retention_days": { 2024-05-10T11:32:18.705 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ceph_event_retention_days", 2024-05-10T11:32:18.705 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.705 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.706 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.706 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7", 2024-05-10T11:32:18.706 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.706 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.706 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.706 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-10T11:32:18.706 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.706 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.707 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.707 INFO:teuthology.orchestra.run.smithi027.stdout: "config_check_secs": { 2024-05-10T11:32:18.707 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_check_secs", 2024-05-10T11:32:18.707 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.707 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.707 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.707 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:18.708 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "10", 2024-05-10T11:32:18.708 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.708 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.708 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-10T11:32:18.708 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.708 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.708 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.709 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.709 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.709 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.709 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.709 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.709 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.709 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.709 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.710 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.710 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.710 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.710 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.710 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.710 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.710 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.710 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.711 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.711 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.711 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.711 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.711 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.711 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.712 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.712 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.712 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.712 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.712 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.712 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.712 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.712 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.713 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.713 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.713 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.713 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.713 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.713 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.713 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.714 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.714 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.714 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.714 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.714 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.714 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.714 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.715 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.715 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.715 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.715 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.715 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.715 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.715 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.716 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.716 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.716 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.716 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.716 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.716 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.716 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "localpool", 2024-05-10T11:32:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "failure_domain": { 2024-05-10T11:32:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "failure_domain", 2024-05-10T11:32:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "host", 2024-05-10T11:32:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "failure domain for any created local pool", 2024-05-10T11:32:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-10T11:32:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.730 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.730 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.730 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.730 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.730 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.730 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.730 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.730 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.731 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.731 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.731 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.731 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": { 2024-05-10T11:32:18.731 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_size", 2024-05-10T11:32:18.731 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.731 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.731 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.732 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.732 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.732 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.732 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.732 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default min_size for any created local pool", 2024-05-10T11:32:18.732 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-10T11:32:18.732 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.733 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.733 INFO:teuthology.orchestra.run.smithi027.stdout: "num_rep": { 2024-05-10T11:32:18.733 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "num_rep", 2024-05-10T11:32:18.733 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.733 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.733 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.733 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3", 2024-05-10T11:32:18.734 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.734 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.734 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.734 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default replica count for any created local pool", 2024-05-10T11:32:18.734 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.734 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.734 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.735 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": { 2024-05-10T11:32:18.735 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_num", 2024-05-10T11:32:18.735 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.735 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.735 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.735 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "128", 2024-05-10T11:32:18.735 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.735 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.736 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.736 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default pg_num for any created local pool", 2024-05-10T11:32:18.736 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.736 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.736 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.736 INFO:teuthology.orchestra.run.smithi027.stdout: "prefix": { 2024-05-10T11:32:18.736 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prefix", 2024-05-10T11:32:18.737 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.737 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.737 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.737 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.737 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.737 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.737 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.737 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name prefix for any created local pool", 2024-05-10T11:32:18.738 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.738 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.738 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.738 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.738 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.738 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.738 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.739 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.739 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.739 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.739 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.739 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.739 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.739 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.739 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.740 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.740 INFO:teuthology.orchestra.run.smithi027.stdout: "subtree": { 2024-05-10T11:32:18.740 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subtree", 2024-05-10T11:32:18.740 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.740 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.740 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.740 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "rack", 2024-05-10T11:32:18.741 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.741 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.741 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.741 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-10T11:32:18.741 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-10T11:32:18.741 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.741 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.741 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.742 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.742 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mds_autoscaler", 2024-05-10T11:32:18.742 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.742 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.742 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.742 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.742 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.743 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.743 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.743 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.743 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.743 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.743 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.743 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.744 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.744 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.744 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.744 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.744 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.744 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.744 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.744 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.745 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.745 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.745 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.745 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.745 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.745 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.745 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.746 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.746 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.746 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.746 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.746 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.746 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.746 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.746 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.747 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.747 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.747 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.747 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.747 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.747 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.748 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.748 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.748 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.748 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.748 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.748 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.748 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.748 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.749 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.749 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.749 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.749 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.749 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.749 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.749 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.750 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.750 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.750 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.750 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.750 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.750 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.750 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.751 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.751 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.751 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.751 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.751 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.751 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.752 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.752 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.752 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.752 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.752 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.752 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.752 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.752 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.753 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.753 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.753 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.753 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.753 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.753 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.753 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.754 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirroring", 2024-05-10T11:32:18.754 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.754 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.754 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.754 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.754 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.755 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.755 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.755 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.755 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.755 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.755 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.755 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.756 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.756 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.756 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.756 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.756 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.756 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.756 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.756 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.757 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.757 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.757 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.757 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.757 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.757 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.757 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.758 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.758 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.758 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.758 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.758 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.758 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.758 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.758 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.759 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.759 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.759 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.759 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.759 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.759 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.759 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.760 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.760 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.760 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.760 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.760 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.760 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.760 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.760 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.761 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.761 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.761 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.761 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.761 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.761 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.761 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.762 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.762 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.762 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.762 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.762 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.762 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.762 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.762 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.763 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.763 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.763 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.763 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.763 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.763 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.763 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.763 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.764 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.764 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.764 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.764 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.764 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.764 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.764 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.765 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.765 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.765 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.765 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "nfs", 2024-05-10T11:32:18.765 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.766 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.766 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.766 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.766 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.766 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.766 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.766 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.766 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.767 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.767 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.767 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.767 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.767 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.767 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.767 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.767 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.768 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.768 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.768 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.768 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.768 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.768 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.769 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.769 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.769 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.769 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.769 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.769 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.769 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.769 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.769 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.770 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.770 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.770 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.770 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.770 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.770 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.771 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.771 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.771 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.771 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.771 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.771 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.771 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.771 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.771 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.772 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.772 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.772 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.772 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.772 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.772 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.772 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.773 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.773 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.773 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.773 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.773 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.773 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.773 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.774 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.774 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.774 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.774 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.774 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.774 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.774 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.775 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.775 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.775 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.775 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.775 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.775 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.775 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.776 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.776 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.776 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.776 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.776 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.776 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.776 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-05-10T11:32:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "fail_fs": { 2024-05-10T11:32:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "fail_fs", 2024-05-10T11:32:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.779 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.779 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-10T11:32:18.779 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.779 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.779 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.779 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.779 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.780 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.780 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.780 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.780 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.780 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.780 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.780 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.781 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.781 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.781 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.781 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.781 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.781 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.781 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.781 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.782 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.782 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.782 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.782 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.782 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.782 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.783 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.783 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.783 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.783 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.783 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.783 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.783 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.783 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.784 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.784 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.784 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.784 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.784 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.784 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.784 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.785 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.785 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.785 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.785 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.785 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.785 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.785 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.785 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.786 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.786 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.786 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.786 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.786 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.786 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.786 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.787 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.787 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.787 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.787 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.787 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.787 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.787 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.788 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.788 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.788 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.788 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.788 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.788 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.789 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator": { 2024-05-10T11:32:18.789 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-05-10T11:32:18.789 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.789 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.789 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.789 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.789 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.789 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.790 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.790 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-05-10T11:32:18.790 INFO:teuthology.orchestra.run.smithi027.stdout: "rook", 2024-05-10T11:32:18.790 INFO:teuthology.orchestra.run.smithi027.stdout: "test_orchestrator" 2024-05-10T11:32:18.790 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.790 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Orchestrator backend", 2024-05-10T11:32:18.790 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.791 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.791 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.791 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.791 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.791 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.791 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.791 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.792 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.792 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.792 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.792 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.792 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.792 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.792 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.792 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.793 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.793 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.793 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_perf_query", 2024-05-10T11:32:18.793 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.793 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.794 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.794 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.794 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.794 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.794 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.794 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.794 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.794 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.795 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.795 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.795 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.795 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.795 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.795 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.795 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.796 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.796 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.796 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.796 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.796 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.796 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.796 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.797 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.797 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.797 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.797 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.797 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.797 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.797 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.797 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.798 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.798 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.798 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.798 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.798 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.798 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.798 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.799 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.799 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.799 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.799 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.799 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.799 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.799 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.799 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.800 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.800 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.800 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.800 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.800 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.800 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.800 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.800 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.801 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.801 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.801 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.801 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.801 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.801 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.802 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.802 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.802 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.802 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.802 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.802 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.802 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.802 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.803 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.803 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.803 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.803 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.803 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.803 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.803 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.804 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.804 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.804 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.804 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.804 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.804 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.804 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.804 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.805 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_support", 2024-05-10T11:32:18.805 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.805 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.805 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.805 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.805 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.806 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.806 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.806 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.806 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.806 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.806 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.806 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.806 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.807 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.807 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.807 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.807 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.807 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.807 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.807 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.807 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.808 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.808 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.808 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.808 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.808 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.808 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.809 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.809 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.809 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.809 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.809 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.809 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.809 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.809 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.809 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.810 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.810 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.810 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.810 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.810 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.810 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.811 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.811 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.811 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.811 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.811 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.811 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.811 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.811 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.812 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.812 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.812 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.812 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.812 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.812 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.813 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.813 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.813 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.813 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.813 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.813 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.813 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.813 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.814 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.814 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.814 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.814 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.814 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.814 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.815 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.815 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.815 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.815 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.815 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.815 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.815 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.815 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.816 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.816 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.816 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.816 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.816 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.816 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.817 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_autoscaler", 2024-05-10T11:32:18.817 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.817 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.817 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.817 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.817 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.817 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.817 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.817 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.818 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.818 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.818 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.818 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.818 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.818 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.818 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.818 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.819 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.819 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.819 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.819 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.819 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.819 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.819 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.819 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.820 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.820 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.820 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.820 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.820 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.820 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.820 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.820 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.821 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.821 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.821 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.821 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.821 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.821 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.821 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.822 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.822 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.822 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.822 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.822 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.822 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.822 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.822 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.822 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.823 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.823 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.823 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.823 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.823 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.823 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.823 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.823 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.824 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.824 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.824 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.824 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.824 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.824 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.824 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.824 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.825 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.825 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.825 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.825 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.825 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:18.825 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:18.825 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.825 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.826 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.826 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:18.826 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.826 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.826 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.826 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.826 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.826 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.827 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.827 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.827 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.827 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.827 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.827 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.827 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: "threshold": { 2024-05-10T11:32:18.829 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threshold", 2024-05-10T11:32:18.829 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:18.829 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.829 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.829 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-05-10T11:32:18.829 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1.0", 2024-05-10T11:32:18.829 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.829 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.830 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "scaling threshold", 2024-05-10T11:32:18.830 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-10T11:32:18.830 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.830 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.830 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.830 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.831 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "progress", 2024-05-10T11:32:18.831 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.831 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.831 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.831 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_pg_recovery_event": { 2024-05-10T11:32:18.831 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_pg_recovery_event", 2024-05-10T11:32:18.831 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.831 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.832 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.832 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.832 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.832 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.832 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.832 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-10T11:32:18.833 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.833 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.833 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.833 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-05-10T11:32:18.833 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-05-10T11:32:18.833 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.833 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.833 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.834 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.834 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.834 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.834 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.834 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.834 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.834 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.834 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.835 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.835 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.835 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.835 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.835 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.835 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.835 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.836 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.836 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.836 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.836 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.836 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.836 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.836 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.836 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.837 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.837 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.837 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.837 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.837 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.837 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.837 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.838 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.838 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.838 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.838 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.838 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.838 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.838 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.838 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.839 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.839 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.839 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.839 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.839 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.839 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.839 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.839 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.839 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.840 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.840 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.840 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.840 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.840 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.840 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.840 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.840 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.841 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.841 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.841 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.841 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.841 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.841 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.841 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.841 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.842 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.842 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.842 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.842 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.842 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.842 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.842 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.842 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.842 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.843 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.843 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.843 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.843 INFO:teuthology.orchestra.run.smithi027.stdout: "max_completed_events": { 2024-05-10T11:32:18.843 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_completed_events", 2024-05-10T11:32:18.843 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.843 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.843 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.844 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "50", 2024-05-10T11:32:18.844 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.844 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.844 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.844 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "number of past completed events to remember", 2024-05-10T11:32:18.844 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.844 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.844 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.845 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.845 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-05-10T11:32:18.845 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-05-10T11:32:18.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:18.845 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.845 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.845 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-05-10T11:32:18.845 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.846 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.846 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.846 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long the module is going to sleep", 2024-05-10T11:32:18.846 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.846 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.846 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.846 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.846 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.847 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.847 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.847 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.847 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.847 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.847 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.847 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.847 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.848 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.848 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.848 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.848 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.848 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.848 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus", 2024-05-10T11:32:18.849 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.849 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.849 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.849 INFO:teuthology.orchestra.run.smithi027.stdout: "cache": { 2024-05-10T11:32:18.849 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cache", 2024-05-10T11:32:18.849 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.849 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.849 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.850 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.850 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.850 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.850 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.850 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.850 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.850 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.850 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.851 INFO:teuthology.orchestra.run.smithi027.stdout: "exclude_perf_counters": { 2024-05-10T11:32:18.851 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "exclude_perf_counters", 2024-05-10T11:32:18.851 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.851 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.851 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.851 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.851 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.851 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.852 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.852 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-10T11:32:18.852 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-10T11:32:18.852 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.852 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.852 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.852 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.853 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.853 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.853 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.853 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.853 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.853 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.853 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.853 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.854 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.854 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.854 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.854 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.854 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.854 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.854 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.854 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.854 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.855 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.855 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.855 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.855 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.855 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.855 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.855 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.856 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.856 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.856 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.856 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.856 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.856 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.857 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.857 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.857 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.857 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.858 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.858 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.858 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.858 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.858 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.859 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.859 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.859 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.859 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.859 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.859 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.860 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.860 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.860 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.860 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.860 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.860 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.861 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.861 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.861 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.861 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.861 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.862 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.862 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.862 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.862 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.862 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.862 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.862 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.862 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.863 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools": { 2024-05-10T11:32:18.863 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools", 2024-05-10T11:32:18.863 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.863 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.863 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.863 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.863 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.864 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.864 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.864 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.864 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.864 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.864 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.864 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-10T11:32:18.865 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-10T11:32:18.865 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.865 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.865 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.865 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "300", 2024-05-10T11:32:18.865 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.865 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.865 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.865 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.866 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.866 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.866 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.866 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_interval": { 2024-05-10T11:32:18.866 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_interval", 2024-05-10T11:32:18.866 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:18.866 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.867 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.867 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15.0", 2024-05-10T11:32:18.867 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.867 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.867 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.867 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.867 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.867 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.867 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.868 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-05-10T11:32:18.868 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-05-10T11:32:18.868 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.868 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.868 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.868 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-05-10T11:32:18.868 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.869 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.869 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.869 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-10T11:32:18.869 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.869 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.869 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.869 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-05-10T11:32:18.869 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-05-10T11:32:18.870 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.870 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.870 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.870 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "9283", 2024-05-10T11:32:18.870 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.870 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.870 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.870 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-10T11:32:18.870 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.871 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.871 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.871 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.871 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.871 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.871 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.871 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.872 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.872 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.872 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.872 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.872 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.872 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.872 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.872 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.873 INFO:teuthology.orchestra.run.smithi027.stdout: "stale_cache_strategy": { 2024-05-10T11:32:18.873 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stale_cache_strategy", 2024-05-10T11:32:18.873 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.873 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.873 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.873 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "log", 2024-05-10T11:32:18.873 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.873 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.874 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.874 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.874 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.874 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.874 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.874 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-05-10T11:32:18.874 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-05-10T11:32:18.875 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.875 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.875 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.875 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "default", 2024-05-10T11:32:18.875 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.875 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.875 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.875 INFO:teuthology.orchestra.run.smithi027.stdout: "default", 2024-05-10T11:32:18.875 INFO:teuthology.orchestra.run.smithi027.stdout: "error" 2024-05-10T11:32:18.876 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.876 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.876 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.876 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.876 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.876 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-05-10T11:32:18.876 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-05-10T11:32:18.876 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.877 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.877 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.877 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-05-10T11:32:18.877 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-05-10T11:32:18.877 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-05-10T11:32:18.877 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.877 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.877 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.878 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.878 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.878 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.878 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.878 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-05-10T11:32:18.878 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.878 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.879 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.879 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.879 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.879 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.879 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.879 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.879 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.879 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.881 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.881 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.881 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.881 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.881 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.881 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.881 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.881 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.882 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.882 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.882 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.882 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.882 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.882 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.882 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.882 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.883 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.883 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.883 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.883 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.883 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.883 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.883 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.883 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.884 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.884 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.884 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.884 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.884 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.884 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.884 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.884 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.887 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.887 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.887 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.887 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_snap_create": { 2024-05-10T11:32:18.887 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_snap_create", 2024-05-10T11:32:18.887 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.887 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.888 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.888 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-05-10T11:32:18.888 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.888 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.888 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.888 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.888 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.888 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.889 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.889 INFO:teuthology.orchestra.run.smithi027.stdout: "mirror_snapshot_schedule": { 2024-05-10T11:32:18.889 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirror_snapshot_schedule", 2024-05-10T11:32:18.889 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.889 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.889 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.889 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.889 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.890 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.890 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.890 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.890 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.890 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.890 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.891 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.891 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.891 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.891 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.891 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.891 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.891 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.891 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.892 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.892 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.892 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.892 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.892 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.892 INFO:teuthology.orchestra.run.smithi027.stdout: "trash_purge_schedule": { 2024-05-10T11:32:18.892 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "trash_purge_schedule", 2024-05-10T11:32:18.893 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.893 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.893 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.893 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.893 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.893 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.893 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.893 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.894 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.894 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.894 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.894 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.894 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.894 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "restful", 2024-05-10T11:32:18.894 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.895 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.895 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.895 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_auth": { 2024-05-10T11:32:18.895 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_auth", 2024-05-10T11:32:18.895 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.895 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.895 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.895 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.896 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.896 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.896 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.896 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.896 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.896 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.896 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.896 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-05-10T11:32:18.897 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-05-10T11:32:18.897 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.897 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.897 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.897 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.897 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.897 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.897 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.898 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.898 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.898 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.898 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.898 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.899 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.899 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.899 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.899 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.899 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.899 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.900 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.900 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.900 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.900 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.900 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.900 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.900 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.900 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.901 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.901 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.901 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.901 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.901 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.901 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.901 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.902 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.902 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.902 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.902 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.902 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.902 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.902 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.902 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.903 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.903 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.903 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.903 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.903 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.903 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.904 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.904 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.904 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.904 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.904 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.904 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.904 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.904 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.905 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.905 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.905 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.905 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.905 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.905 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.905 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.905 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.906 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.906 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.906 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.906 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.906 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.906 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.906 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.907 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.907 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.907 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.907 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.907 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.907 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.907 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.908 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.908 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-05-10T11:32:18.908 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-05-10T11:32:18.908 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.908 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.908 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.908 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.909 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.909 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.909 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.909 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.909 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.909 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.909 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.909 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-05-10T11:32:18.910 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-05-10T11:32:18.910 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.910 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.910 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.910 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.910 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.910 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.910 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.426+0000 7f25cb400640 1 -- 172.21.15.27:0/506358858 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f25a80b2960 msgr2=0x7f25a80b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:18.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.426+0000 7f25cb400640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f25a80b2960 0x7f25a80b4e20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0303f0 tx=0x7f25ac030340 comp rx=0 tx=0).stop 2024-05-10T11:32:18.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.426+0000 7f25cb400640 1 -- 172.21.15.27:0/506358858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25c4104e00 msgr2=0x7f25c419b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:32:18.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.426+0000 7f25cb400640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25c4104e00 0x7f25c419b410 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f25b400eb00 tx=0x7f25b400ef00 comp rx=0 tx=0).stop 2024-05-10T11:32:18.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.426+0000 7f25cb400640 1 -- 172.21.15.27:0/506358858 shutdown_connections 2024-05-10T11:32:18.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.426+0000 7f25cb400640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f25a80b2960 0x7f25a80b4e20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:18.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.426+0000 7f25cb400640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25c419fcd0 0x7f25c41a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:18.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.426+0000 7f25cb400640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c4109ed0 0x7f25c419b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:18.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.426+0000 7f25cb400640 1 --2- 172.21.15.27:0/506358858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25c4104e00 0x7f25c419b410 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:32:18.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.426+0000 7f25cb400640 1 -- 172.21.15.27:0/506358858 >> 172.21.15.27:0/506358858 conn(0x7f25c41006a0 msgr2=0x7f25c4101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:32:18.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.426+0000 7f25cb400640 1 -- 172.21.15.27:0/506358858 shutdown_connections 2024-05-10T11:32:18.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:32:16.426+0000 7f25cb400640 1 -- 172.21.15.27:0/506358858 wait complete. 2024-05-10T11:32:18.914 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:17 smithi189 ceph-mon[34842]: pgmap v600: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 4.6 MiB/s wr, 165 op/s 2024-05-10T11:32:18.914 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:17 smithi076 ceph-mon[34152]: pgmap v600: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 4.6 MiB/s wr, 165 op/s 2024-05-10T11:32:18.914 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.915 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.915 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.915 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.915 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.915 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.915 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.915 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.916 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.916 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.916 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.916 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.916 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.916 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.916 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.916 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.917 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.917 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.917 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.917 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.917 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rgw", 2024-05-10T11:32:18.917 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.917 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.918 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.918 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.918 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.918 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.918 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.918 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.918 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.918 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.919 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.919 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.919 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.919 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.919 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.919 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.919 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.919 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.920 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.920 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.920 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.920 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.920 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.920 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.920 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.920 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.921 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.921 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.921 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.921 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.921 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.921 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.921 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.921 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.922 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.922 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.922 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.922 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.922 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.922 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.922 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.923 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.923 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.923 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.923 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.923 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.923 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.923 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.923 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.924 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.924 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.924 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.924 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.924 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.924 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.924 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.925 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.925 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.925 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.925 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.925 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.925 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.925 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.925 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.925 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.926 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.926 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.926 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.926 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.926 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.926 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.927 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.927 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.927 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.927 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.927 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.927 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.927 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.927 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.927 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.928 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.928 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.928 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.928 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.928 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.928 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rook", 2024-05-10T11:32:18.928 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.929 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.929 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.929 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.929 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.929 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.929 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.929 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.929 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.930 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.930 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.930 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.930 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.930 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.930 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.930 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.930 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.930 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.931 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.931 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.931 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.931 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.931 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.931 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.931 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.932 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.932 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.932 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.932 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.932 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.932 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.932 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.932 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.933 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.933 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.933 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.933 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.933 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.933 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.933 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.933 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.934 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.934 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.934 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.934 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.934 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.934 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.934 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.934 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.935 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.935 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.935 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.935 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.935 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.935 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.935 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.935 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.936 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.936 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.936 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.936 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.936 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.936 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.936 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.936 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.937 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.937 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.937 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.937 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.937 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.937 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.937 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.937 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.938 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.938 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.938 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.938 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.938 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.938 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.938 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.938 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.939 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.939 INFO:teuthology.orchestra.run.smithi027.stdout: "storage_class": { 2024-05-10T11:32:18.939 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "storage_class", 2024-05-10T11:32:18.939 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.939 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.939 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.939 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "local", 2024-05-10T11:32:18.940 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.940 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.940 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.940 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-10T11:32:18.940 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.940 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.940 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.940 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.940 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.941 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "selftest", 2024-05-10T11:32:18.941 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.941 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.941 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.941 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.941 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.941 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.942 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.942 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.942 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.942 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.942 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.942 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.942 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.942 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.943 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.943 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.943 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.943 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.943 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.943 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.943 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.943 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.943 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.944 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.944 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.944 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.944 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.944 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.944 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.944 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.945 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.945 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.945 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.945 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.945 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.945 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.945 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.946 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.946 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.946 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.946 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.946 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.946 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.946 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.946 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.946 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.947 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.947 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.947 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.947 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.947 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.947 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.947 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.947 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.948 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.948 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.948 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.948 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.948 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.948 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.948 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.948 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.949 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.949 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.949 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.949 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.949 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.949 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.949 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.950 INFO:teuthology.orchestra.run.smithi027.stdout: "roption1": { 2024-05-10T11:32:18.950 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption1", 2024-05-10T11:32:18.950 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.950 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.950 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.950 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.950 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.950 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.950 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.951 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.951 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.951 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.951 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.951 INFO:teuthology.orchestra.run.smithi027.stdout: "roption2": { 2024-05-10T11:32:18.951 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption2", 2024-05-10T11:32:18.951 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.952 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.952 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.952 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "xyz", 2024-05-10T11:32:18.952 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.952 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.952 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.952 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.952 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.952 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.953 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.953 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption1": { 2024-05-10T11:32:18.953 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption1", 2024-05-10T11:32:18.953 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.953 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.953 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.953 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.954 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.954 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.954 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.954 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.954 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.954 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.954 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.955 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption2": { 2024-05-10T11:32:18.955 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption2", 2024-05-10T11:32:18.955 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.955 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.955 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.955 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.955 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.955 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.955 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.956 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.956 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.956 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.956 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.956 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption3": { 2024-05-10T11:32:18.956 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption3", 2024-05-10T11:32:18.956 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-05-10T11:32:18.957 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.957 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.957 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.957 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.957 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.957 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.957 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.957 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.957 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.958 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.958 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption4": { 2024-05-10T11:32:18.958 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption4", 2024-05-10T11:32:18.958 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.958 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.958 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.958 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.959 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.959 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.959 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.959 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.959 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.959 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.959 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.960 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption5": { 2024-05-10T11:32:18.960 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption5", 2024-05-10T11:32:18.960 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.960 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.960 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.960 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.960 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.960 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.960 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.961 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.961 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.961 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.961 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.961 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption6": { 2024-05-10T11:32:18.961 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption6", 2024-05-10T11:32:18.961 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.962 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.962 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.962 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.962 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.962 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.962 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.962 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.962 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.963 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.963 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.963 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption7": { 2024-05-10T11:32:18.963 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption7", 2024-05-10T11:32:18.963 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.963 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.963 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.964 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.964 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-05-10T11:32:18.964 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "42", 2024-05-10T11:32:18.964 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.964 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.964 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.964 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.964 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.965 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.965 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.965 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.965 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.965 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.965 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.965 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.965 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.966 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.966 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.966 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.966 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.966 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.966 INFO:teuthology.orchestra.run.smithi027.stdout: "testkey": { 2024-05-10T11:32:18.966 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testkey", 2024-05-10T11:32:18.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.967 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.967 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.967 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.967 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.967 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.967 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.967 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.968 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.968 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.968 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.968 INFO:teuthology.orchestra.run.smithi027.stdout: "testlkey": { 2024-05-10T11:32:18.968 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testlkey", 2024-05-10T11:32:18.968 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.968 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.968 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.969 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.969 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.969 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.969 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.969 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.969 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.969 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.969 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.970 INFO:teuthology.orchestra.run.smithi027.stdout: "testnewline": { 2024-05-10T11:32:18.970 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testnewline", 2024-05-10T11:32:18.970 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.970 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.970 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.970 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.970 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.971 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.971 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.971 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.971 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.971 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.971 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.971 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.971 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.972 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smb", 2024-05-10T11:32:18.972 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.972 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.972 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.972 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.972 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.972 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.973 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.973 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.973 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.973 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.973 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.973 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.973 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.973 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.974 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.974 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.974 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.974 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.974 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.974 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.974 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.974 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.975 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.975 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.975 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.975 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.975 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.975 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.975 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.976 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.976 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.976 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.976 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.976 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.976 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.976 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.977 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.977 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.977 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.977 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.977 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.977 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.977 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.977 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.977 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.978 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.978 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.978 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.978 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.978 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.978 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.978 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.978 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.979 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.979 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.979 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.979 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.979 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.979 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.979 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.980 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.980 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.980 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.980 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.980 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.980 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.980 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.980 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.981 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.981 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.981 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.981 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.981 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.981 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.981 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.981 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.982 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.982 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.982 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.982 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.982 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.982 INFO:teuthology.orchestra.run.smithi027.stdout: "update_orchestration": { 2024-05-10T11:32:18.982 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "update_orchestration", 2024-05-10T11:32:18.982 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.983 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.983 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:18.983 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:18.983 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.983 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.983 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.983 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-10T11:32:18.983 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.984 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.984 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.984 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.984 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.984 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-05-10T11:32:18.984 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.984 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.985 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.985 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_m_granularity": { 2024-05-10T11:32:18.985 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_m_granularity", 2024-05-10T11:32:18.985 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.985 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.985 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.985 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.985 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.985 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.986 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.986 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow minute scheduled snapshots", 2024-05-10T11:32:18.986 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.986 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.986 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.986 INFO:teuthology.orchestra.run.smithi027.stdout: "dump_on_update": { 2024-05-10T11:32:18.986 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dump_on_update", 2024-05-10T11:32:18.987 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.987 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.987 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.987 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.987 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.987 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.987 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.987 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "dump database to debug log on update", 2024-05-10T11:32:18.987 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.988 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.988 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.988 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.988 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.988 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.988 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.988 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.988 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.989 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.989 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.989 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.989 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.989 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.989 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.989 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.989 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.989 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.990 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.990 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.990 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.990 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.990 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.990 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:18.990 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:18.991 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.991 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.991 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.991 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.991 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.991 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.991 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.991 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.991 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.992 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.992 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.992 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:18.992 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:18.992 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.992 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.992 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.993 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:18.993 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.993 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.993 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:18.993 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:18.993 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:18.993 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:18.993 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:18.994 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:18.994 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:18.994 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:18.994 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.994 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.994 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.994 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.995 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:18.995 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:18.995 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:18.995 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.995 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.995 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:18.995 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.995 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.995 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.996 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.996 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.996 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.996 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.996 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:18.996 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:18.996 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:18.997 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:18.997 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.997 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:18.997 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:18.997 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:18.997 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:18.997 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:18.997 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:18.997 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:18.998 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:18.998 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.998 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:18.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:18.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:18.998 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stats", 2024-05-10T11:32:18.998 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:18.998 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:18.998 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:18.999 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:18.999 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:18.999 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:18.999 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:18.999 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:18.999 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:18.999 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.000 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.000 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.000 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.000 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.000 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.000 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.000 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.000 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.000 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.001 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.001 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.001 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.001 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.001 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:19.001 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:19.001 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.002 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.002 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.002 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.002 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.002 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.002 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.002 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.002 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.003 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.003 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.003 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:19.003 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:19.003 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.003 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.004 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.004 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:19.004 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.004 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.004 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.004 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.004 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.004 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.004 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.005 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.005 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.005 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.005 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.005 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.005 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.005 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.005 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:19.006 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:19.006 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.006 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.006 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.006 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.006 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.006 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.007 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.007 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.007 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.007 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.007 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.007 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:19.007 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:19.007 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:19.008 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:19.008 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.008 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:19.008 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.008 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.008 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.008 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.008 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.009 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.009 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.009 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "status", 2024-05-10T11:32:19.009 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:19.010 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:19.010 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:19.010 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:19.010 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:19.010 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.010 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.010 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.010 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.011 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.011 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.011 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.011 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.011 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.011 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.011 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.011 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.011 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.012 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.012 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.012 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.012 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.012 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.012 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:19.012 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:19.012 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.013 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.013 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.013 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.013 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.013 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.013 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.013 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.013 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.014 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.014 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.014 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:19.014 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:19.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.014 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.014 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.015 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:19.015 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.015 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.015 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.015 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.015 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.015 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.015 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.015 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.016 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.016 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.016 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.016 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.016 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.016 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.016 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:19.016 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:19.017 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.017 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.017 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.017 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.017 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.017 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.017 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.017 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.018 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.018 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.018 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.018 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:19.018 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:19.018 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:19.018 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:19.019 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.019 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:19.019 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.019 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.019 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.019 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.019 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.019 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.019 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.020 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.020 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.020 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telegraf", 2024-05-10T11:32:19.020 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:19.020 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:19.020 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:19.020 INFO:teuthology.orchestra.run.smithi027.stdout: "address": { 2024-05-10T11:32:19.021 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "address", 2024-05-10T11:32:19.021 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.021 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.021 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-10T11:32:19.021 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.021 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.021 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.022 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.022 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.022 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.022 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.022 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:19.022 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:19.022 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:19.023 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.023 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.023 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15", 2024-05-10T11:32:19.023 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.023 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.023 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.023 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.023 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.023 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.024 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.024 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:19.024 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:19.024 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.024 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.024 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.024 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.025 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.025 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.025 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.025 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.025 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.025 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.025 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.025 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.025 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.026 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.026 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.026 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.026 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.026 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.026 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:19.026 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:19.026 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.027 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.027 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.027 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.027 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.027 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.027 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.027 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.027 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.028 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.028 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.028 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:19.028 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:19.028 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.028 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.028 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.028 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:19.029 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.029 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.029 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.029 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.029 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.029 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.029 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.029 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.030 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.030 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.030 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.030 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.030 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.030 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.030 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:19.031 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:19.031 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.031 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.031 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.031 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.031 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.031 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.031 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.031 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.032 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.032 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.032 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.032 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:19.032 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:19.032 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:19.032 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:19.033 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.033 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:19.033 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.033 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.033 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.033 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.033 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.033 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.034 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.034 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.034 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.034 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telemetry", 2024-05-10T11:32:19.034 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:19.034 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:19.035 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:19.035 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_basic": { 2024-05-10T11:32:19.035 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_basic", 2024-05-10T11:32:19.035 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.035 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.035 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.035 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:19.035 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.036 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.036 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.036 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-10T11:32:19.036 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.036 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.036 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.036 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_crash": { 2024-05-10T11:32:19.036 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_crash", 2024-05-10T11:32:19.037 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.037 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.037 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.037 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:19.037 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.037 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.037 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.037 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-10T11:32:19.038 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.038 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.038 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.038 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_device": { 2024-05-10T11:32:19.038 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_device", 2024-05-10T11:32:19.038 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.038 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.039 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:19.039 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.039 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.039 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.039 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-10T11:32:19.039 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.039 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.040 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.040 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_ident": { 2024-05-10T11:32:19.040 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_ident", 2024-05-10T11:32:19.040 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.040 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.040 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.040 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.040 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.041 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.041 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.041 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-10T11:32:19.041 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.041 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.041 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.041 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_perf": { 2024-05-10T11:32:19.042 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_perf", 2024-05-10T11:32:19.042 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.042 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.042 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.042 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.042 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.042 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.042 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.043 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-10T11:32:19.043 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.043 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.043 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.043 INFO:teuthology.orchestra.run.smithi027.stdout: "contact": { 2024-05-10T11:32:19.043 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "contact", 2024-05-10T11:32:19.043 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.043 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.044 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.044 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.044 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.044 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.044 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.044 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.044 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.044 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.045 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.045 INFO:teuthology.orchestra.run.smithi027.stdout: "description": { 2024-05-10T11:32:19.045 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "description", 2024-05-10T11:32:19.045 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.045 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.045 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.045 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.046 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.046 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.046 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.046 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.046 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.046 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.046 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.046 INFO:teuthology.orchestra.run.smithi027.stdout: "device_url": { 2024-05-10T11:32:19.047 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_url", 2024-05-10T11:32:19.047 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.047 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.047 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.047 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-10T11:32:19.047 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.047 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.047 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.048 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.048 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.048 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.048 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.048 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-05-10T11:32:19.048 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-05-10T11:32:19.048 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.049 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.049 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.049 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.049 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.049 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.049 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.049 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.049 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.050 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.050 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.050 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:19.050 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:19.050 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:19.050 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.050 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.051 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "24", 2024-05-10T11:32:19.051 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "8", 2024-05-10T11:32:19.051 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.051 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.051 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.051 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.051 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.051 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.052 INFO:teuthology.orchestra.run.smithi027.stdout: "last_opt_revision": { 2024-05-10T11:32:19.052 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "last_opt_revision", 2024-05-10T11:32:19.052 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:19.052 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.052 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.052 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1", 2024-05-10T11:32:19.052 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.052 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.053 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.053 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.053 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.053 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.053 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.053 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard": { 2024-05-10T11:32:19.053 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard", 2024-05-10T11:32:19.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.054 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.054 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.054 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.054 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.054 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.054 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.054 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.054 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.055 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.055 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.055 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard_description": { 2024-05-10T11:32:19.055 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard_description", 2024-05-10T11:32:19.055 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.055 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.055 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.055 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.056 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.056 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.056 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.056 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.056 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.056 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.056 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.056 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:19.057 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:19.057 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.057 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.057 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.057 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.057 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.057 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.057 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.058 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.058 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.058 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.058 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.058 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.058 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.058 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.058 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.058 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.059 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.059 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.059 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:19.059 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:19.059 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.059 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.059 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.061 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:19.061 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:19.061 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.061 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.061 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.061 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:19.061 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.061 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.061 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.062 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.062 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.062 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.062 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.062 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.062 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.062 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.062 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.063 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.063 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.063 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.063 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:19.063 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:19.063 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.063 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.063 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.064 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.064 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.064 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.064 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.064 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.064 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.064 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.064 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.065 INFO:teuthology.orchestra.run.smithi027.stdout: "organization": { 2024-05-10T11:32:19.065 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "organization", 2024-05-10T11:32:19.065 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.065 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.065 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.065 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.065 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.065 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.066 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.066 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.066 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.066 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.066 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.066 INFO:teuthology.orchestra.run.smithi027.stdout: "proxy": { 2024-05-10T11:32:19.066 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "proxy", 2024-05-10T11:32:19.066 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.067 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.067 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.067 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.067 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.067 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.067 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.067 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.067 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.067 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.068 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.068 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:19.068 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:19.068 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:19.068 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:19.068 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.068 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "url": { 2024-05-10T11:32:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url", 2024-05-10T11:32:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-10T11:32:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "test_orchestrator", 2024-05-10T11:32:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.073 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.073 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.073 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.073 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.073 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.073 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.073 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.073 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.074 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.074 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.074 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.074 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.074 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.074 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.074 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.075 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:19.075 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:19.075 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.075 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.075 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.075 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.075 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-05-10T11:32:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_clones": { 2024-05-10T11:32:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_clones", 2024-05-10T11:32:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4", 2024-05-10T11:32:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-10T11:32:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "periodic_async_work": { 2024-05-10T11:32:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "periodic_async_work", 2024-05-10T11:32:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Periodically check for async work", 2024-05-10T11:32:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_delay": { 2024-05-10T11:32:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_delay", 2024-05-10T11:32:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-10T11:32:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_no_wait": { 2024-05-10T11:32:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_no_wait", 2024-05-10T11:32:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-05-10T11:32:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-10T11:32:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix", 2024-05-10T11:32:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-05-10T11:32:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-05-10T11:32:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-05-10T11:32:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "discovery_interval": { 2024-05-10T11:32:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "discovery_interval", 2024-05-10T11:32:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-05-10T11:32:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "100", 2024-05-10T11:32:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: "identifier": { 2024-05-10T11:32:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "identifier", 2024-05-10T11:32:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-05-10T11:32:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-05-10T11:32:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-05-10T11:32:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-05-10T11:32:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-05-10T11:32:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-05-10T11:32:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-05-10T11:32:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-05-10T11:32:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-05-10T11:32:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-05-10T11:32:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-05-10T11:32:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-05-10T11:32:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-05-10T11:32:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-05-10T11:32:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-05-10T11:32:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-05-10T11:32:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-05-10T11:32:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-05-10T11:32:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-05-10T11:32:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-05-10T11:32:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-05-10T11:32:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-05-10T11:32:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-05-10T11:32:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-05-10T11:32:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-05-10T11:32:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-10T11:32:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-05-10T11:32:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_host": { 2024-05-10T11:32:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_host", 2024-05-10T11:32:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-05-10T11:32:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_port": { 2024-05-10T11:32:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_port", 2024-05-10T11:32:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-05-10T11:32:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10051", 2024-05-10T11:32:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_sender": { 2024-05-10T11:32:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_sender", 2024-05-10T11:32:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-05-10T11:32:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-05-10T11:32:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-05-10T11:32:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-10T11:32:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-05-10T11:32:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-05-10T11:32:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-05-10T11:32:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-05-10T11:32:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-05-10T11:32:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-05-10T11:32:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-05-10T11:32:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: "services": { 2024-05-10T11:32:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: "dashboard": "https://172.21.15.27:8443/" 2024-05-10T11:32:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "always_on_modules": { 2024-05-10T11:32:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "octopus": [ 2024-05-10T11:32:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-05-10T11:32:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-05-10T11:32:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-05-10T11:32:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-05-10T11:32:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-05-10T11:32:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-05-10T11:32:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-05-10T11:32:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-05-10T11:32:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-05-10T11:32:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-05-10T11:32:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "pacific": [ 2024-05-10T11:32:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-05-10T11:32:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-05-10T11:32:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-05-10T11:32:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-05-10T11:32:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-05-10T11:32:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-05-10T11:32:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-05-10T11:32:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-05-10T11:32:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-05-10T11:32:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-05-10T11:32:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "quincy": [ 2024-05-10T11:32:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-05-10T11:32:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-05-10T11:32:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-05-10T11:32:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-05-10T11:32:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-05-10T11:32:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-05-10T11:32:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-05-10T11:32:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-05-10T11:32:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-05-10T11:32:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-05-10T11:32:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "reef": [ 2024-05-10T11:32:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-05-10T11:32:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-05-10T11:32:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-05-10T11:32:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-05-10T11:32:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-05-10T11:32:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-05-10T11:32:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-05-10T11:32:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-05-10T11:32:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-05-10T11:32:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-05-10T11:32:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-10T11:32:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "squid": [ 2024-05-10T11:32:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-05-10T11:32:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-05-10T11:32:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-05-10T11:32:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-05-10T11:32:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-05-10T11:32:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-05-10T11:32:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-05-10T11:32:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-05-10T11:32:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-05-10T11:32:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-05-10T11:32:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-10T11:32:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: "last_failure_osd_epoch": 91, 2024-05-10T11:32:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: "active_clients": [ 2024-05-10T11:32:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-05-10T11:32:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-10T11:32:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-10T11:32:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-05-10T11:32:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 318364556 2024-05-10T11:32:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-10T11:32:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "libcephsqlite", 2024-05-10T11:32:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-10T11:32:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-10T11:32:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-05-10T11:32:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2004485105 2024-05-10T11:32:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-10T11:32:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-05-10T11:32:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-10T11:32:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-10T11:32:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-05-10T11:32:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2820051801 2024-05-10T11:32:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-10T11:32:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-05-10T11:32:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-10T11:32:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-10T11:32:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-05-10T11:32:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 670245521 2024-05-10T11:32:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-10T11:32:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-05-10T11:32:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-10T11:32:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-10T11:32:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-05-10T11:32:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1096956481 2024-05-10T11:32:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-10T11:32:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-05-10T11:32:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-10T11:32:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-10T11:32:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-10T11:32:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-05-10T11:32:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1031411960 2024-05-10T11:32:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-10T11:32:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-10T11:32:19.141 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:32:19.149 DEBUG:tasks.check_counter:Getting stats from a 2024-05-10T11:32:19.149 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-10T11:32:19.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:16 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2127414105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-10T11:32:19.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:16 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/506358858' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-10T11:32:19.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:17 smithi027 ceph-mon[30054]: pgmap v600: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 4.6 MiB/s wr, 165 op/s 2024-05-10T11:32:19.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:18 smithi027 ceph-mon[30054]: pgmap v601: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 4.6 MiB/s wr, 165 op/s 2024-05-10T11:32:19.151 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:18 smithi189 ceph-mon[34842]: pgmap v601: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 4.6 MiB/s wr, 165 op/s 2024-05-10T11:32:19.151 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:18 smithi076 ceph-mon[34152]: pgmap v601: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 4.6 MiB/s wr, 165 op/s 2024-05-10T11:32:19.623 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:32:21.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:20 smithi027 ceph-mon[30054]: pgmap v602: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.0 MiB/s wr, 209 op/s 2024-05-10T11:32:21.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:20 smithi076 ceph-mon[34152]: pgmap v602: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.0 MiB/s wr, 209 op/s 2024-05-10T11:32:21.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:20 smithi189 ceph-mon[34842]: pgmap v602: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.0 MiB/s wr, 209 op/s 2024-05-10T11:32:22.306 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:32:22.306 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-05-10T11:32:22.307 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 1478, 2024-05-10T11:32:22.307 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 1480, 2024-05-10T11:32:22.307 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 487733, 2024-05-10T11:32:22.307 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 470005, 2024-05-10T11:32:22.307 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 7, 2024-05-10T11:32:22.307 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 2, 2024-05-10T11:32:22.307 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.364108382, 2024-05-10T11:32:22.307 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.157023044, 2024-05-10T11:32:22.308 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.080816904, 2024-05-10T11:32:22.308 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.090750595, 2024-05-10T11:32:22.308 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:22.308 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 1478, 2024-05-10T11:32:22.308 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.186573625, 2024-05-10T11:32:22.308 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000126233 2024-05-10T11:32:22.308 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.308 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:22.308 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.309 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.309 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.309 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-10T11:32:22.309 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-05-10T11:32:22.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.309 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-05-10T11:32:22.309 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 2636, 2024-05-10T11:32:22.310 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 2935, 2024-05-10T11:32:22.310 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 12717155, 2024-05-10T11:32:22.310 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 1510429, 2024-05-10T11:32:22.310 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 7, 2024-05-10T11:32:22.310 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 5, 2024-05-10T11:32:22.310 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.529642317, 2024-05-10T11:32:22.310 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.190678041, 2024-05-10T11:32:22.310 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.150652125, 2024-05-10T11:32:22.310 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.144034876, 2024-05-10T11:32:22.311 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:22.311 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 2933, 2024-05-10T11:32:22.311 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.304392541, 2024-05-10T11:32:22.311 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000103781 2024-05-10T11:32:22.311 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.311 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:22.311 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.311 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.311 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.312 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-05-10T11:32:22.312 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 674032 2024-05-10T11:32:22.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.312 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-05-10T11:32:22.312 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 464, 2024-05-10T11:32:22.312 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 435, 2024-05-10T11:32:22.312 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 2072436, 2024-05-10T11:32:22.312 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 253077, 2024-05-10T11:32:22.313 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 5, 2024-05-10T11:32:22.313 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-05-10T11:32:22.313 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.167293581, 2024-05-10T11:32:22.313 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.064455477, 2024-05-10T11:32:22.313 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.105921870, 2024-05-10T11:32:22.313 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.018306526, 2024-05-10T11:32:22.313 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:22.313 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 434, 2024-05-10T11:32:22.313 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.057931642, 2024-05-10T11:32:22.314 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000133483 2024-05-10T11:32:22.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.314 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:22.314 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.314 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.314 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.314 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 2071088, 2024-05-10T11:32:22.315 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 252112 2024-05-10T11:32:22.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.315 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-05-10T11:32:22.315 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 1, 2024-05-10T11:32:22.315 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-05-10T11:32:22.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.315 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-MDSRank": { 2024-05-10T11:32:22.315 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-05-10T11:32:22.315 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-05-10T11:32:22.316 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 6962, 2024-05-10T11:32:22.316 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.350944807, 2024-05-10T11:32:22.316 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000050408 2024-05-10T11:32:22.316 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:22.316 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.316 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-PurgeQueue": { 2024-05-10T11:32:22.316 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-05-10T11:32:22.316 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-05-10T11:32:22.316 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.317 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.317 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.317 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:22.317 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.317 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-05-10T11:32:22.317 INFO:teuthology.orchestra.run.smithi027.stdout: "request": 0, 2024-05-10T11:32:22.317 INFO:teuthology.orchestra.run.smithi027.stdout: "reply": 0, 2024-05-10T11:32:22.317 INFO:teuthology.orchestra.run.smithi027.stdout: "reply_latency": { 2024-05-10T11:32:22.317 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.318 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.318 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.318 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.318 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_reply": 0, 2024-05-10T11:32:22.318 INFO:teuthology.orchestra.run.smithi027.stdout: "forward": 0, 2024-05-10T11:32:22.318 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_complete": 0, 2024-05-10T11:32:22.318 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_keys": 0, 2024-05-10T11:32:22.318 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_commit": 0, 2024-05-10T11:32:22.319 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_split": 0, 2024-05-10T11:32:22.319 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_merge": 0, 2024-05-10T11:32:22.319 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes": 1803, 2024-05-10T11:32:22.319 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_top": 0, 2024-05-10T11:32:22.319 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_bottom": 1803, 2024-05-10T11:32:22.319 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pin_tail": 0, 2024-05-10T11:32:22.319 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pinned": 1803, 2024-05-10T11:32:22.319 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_expired": 0, 2024-05-10T11:32:22.319 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_with_caps": 0, 2024-05-10T11:32:22.320 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": 0, 2024-05-10T11:32:22.320 INFO:teuthology.orchestra.run.smithi027.stdout: "subtrees": 5, 2024-05-10T11:32:22.320 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse": 0, 2024-05-10T11:32:22.320 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_hit": 0, 2024-05-10T11:32:22.320 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_forward": 0, 2024-05-10T11:32:22.320 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_discover": 0, 2024-05-10T11:32:22.320 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_dir_fetch": 0, 2024-05-10T11:32:22.320 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_remote_ino": 0, 2024-05-10T11:32:22.320 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_lock": 0, 2024-05-10T11:32:22.321 INFO:teuthology.orchestra.run.smithi027.stdout: "load_cent": 0, 2024-05-10T11:32:22.321 INFO:teuthology.orchestra.run.smithi027.stdout: "q": 0, 2024-05-10T11:32:22.321 INFO:teuthology.orchestra.run.smithi027.stdout: "exported": 0, 2024-05-10T11:32:22.321 INFO:teuthology.orchestra.run.smithi027.stdout: "exported_inodes": 0, 2024-05-10T11:32:22.321 INFO:teuthology.orchestra.run.smithi027.stdout: "imported": 0, 2024-05-10T11:32:22.321 INFO:teuthology.orchestra.run.smithi027.stdout: "imported_inodes": 0, 2024-05-10T11:32:22.321 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_dir_fetch": 0, 2024-05-10T11:32:22.321 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_backtrace_fetch": 0, 2024-05-10T11:32:22.321 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_peer_discover": 0, 2024-05-10T11:32:22.322 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rfiles": 0, 2024-05-10T11:32:22.322 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rbytes": 0, 2024-05-10T11:32:22.322 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rsnaps": 0, 2024-05-10T11:32:22.322 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_fetch": 0, 2024-05-10T11:32:22.322 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_set_tag": 0, 2024-05-10T11:32:22.322 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_repaired": 0, 2024-05-10T11:32:22.322 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_inotable_repaired": 0, 2024-05-10T11:32:22.322 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_inodes": 0, 2024-05-10T11:32:22.322 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_base_inodes": 0, 2024-05-10T11:32:22.323 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dirfrag_rstats": 0, 2024-05-10T11:32:22.323 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_file_inodes": 0, 2024-05-10T11:32:22.323 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_inode_file_caps": 0, 2024-05-10T11:32:22.323 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_revoke": 0, 2024-05-10T11:32:22.323 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_grant": 0, 2024-05-10T11:32:22.323 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_trunc": 0, 2024-05-10T11:32:22.323 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-10T11:32:22.323 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-10T11:32:22.324 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps": 0, 2024-05-10T11:32:22.324 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps_dirty": 0, 2024-05-10T11:32:22.324 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_cap_release": 0, 2024-05-10T11:32:22.324 INFO:teuthology.orchestra.run.smithi027.stdout: "process_request_cap_release": 0 2024-05-10T11:32:22.324 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.324 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache": { 2024-05-10T11:32:22.324 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update": 0, 2024-05-10T11:32:22.324 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update_receipt": 0, 2024-05-10T11:32:22.324 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_try_discover": 0, 2024-05-10T11:32:22.325 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_send_discover": 0, 2024-05-10T11:32:22.325 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_handle_discover": 0, 2024-05-10T11:32:22.325 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays": 0, 2024-05-10T11:32:22.325 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_delayed": 0, 2024-05-10T11:32:22.325 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_enqueuing": 0, 2024-05-10T11:32:22.325 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_created": 0, 2024-05-10T11:32:22.325 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_enqueued": 0, 2024-05-10T11:32:22.325 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_reintegrated": 0, 2024-05-10T11:32:22.325 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_migrated": 0, 2024-05-10T11:32:22.326 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_processing": 0, 2024-05-10T11:32:22.326 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_enqueued": 0, 2024-05-10T11:32:22.326 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_prioritized": 0, 2024-05-10T11:32:22.326 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_started": 0, 2024-05-10T11:32:22.326 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_completed": 0, 2024-05-10T11:32:22.326 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_quiesce_path": 0, 2024-05-10T11:32:22.326 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_quiesce_inode": 0, 2024-05-10T11:32:22.326 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_enqueue_scrub": 0, 2024-05-10T11:32:22.327 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_exportdir": 0, 2024-05-10T11:32:22.327 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_flush": 0, 2024-05-10T11:32:22.327 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragmentdir": 0, 2024-05-10T11:32:22.327 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragstats": 0, 2024-05-10T11:32:22.327 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_inodestats": 0 2024-05-10T11:32:22.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.327 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log": { 2024-05-10T11:32:22.327 INFO:teuthology.orchestra.run.smithi027.stdout: "evlrg": 0, 2024-05-10T11:32:22.327 INFO:teuthology.orchestra.run.smithi027.stdout: "evadd": 0, 2024-05-10T11:32:22.328 INFO:teuthology.orchestra.run.smithi027.stdout: "evex": 0, 2024-05-10T11:32:22.328 INFO:teuthology.orchestra.run.smithi027.stdout: "evtrm": 0, 2024-05-10T11:32:22.328 INFO:teuthology.orchestra.run.smithi027.stdout: "ev": 8414, 2024-05-10T11:32:22.328 INFO:teuthology.orchestra.run.smithi027.stdout: "evexg": 0, 2024-05-10T11:32:22.328 INFO:teuthology.orchestra.run.smithi027.stdout: "evexd": 0, 2024-05-10T11:32:22.328 INFO:teuthology.orchestra.run.smithi027.stdout: "segadd": 0, 2024-05-10T11:32:22.328 INFO:teuthology.orchestra.run.smithi027.stdout: "segex": 0, 2024-05-10T11:32:22.328 INFO:teuthology.orchestra.run.smithi027.stdout: "segtrm": 0, 2024-05-10T11:32:22.328 INFO:teuthology.orchestra.run.smithi027.stdout: "seg": 10, 2024-05-10T11:32:22.329 INFO:teuthology.orchestra.run.smithi027.stdout: "segmjr": 2, 2024-05-10T11:32:22.329 INFO:teuthology.orchestra.run.smithi027.stdout: "segexg": 0, 2024-05-10T11:32:22.329 INFO:teuthology.orchestra.run.smithi027.stdout: "segexd": 0, 2024-05-10T11:32:22.329 INFO:teuthology.orchestra.run.smithi027.stdout: "expos": 4194304, 2024-05-10T11:32:22.329 INFO:teuthology.orchestra.run.smithi027.stdout: "wrpos": 16238783, 2024-05-10T11:32:22.329 INFO:teuthology.orchestra.run.smithi027.stdout: "rdpos": 16234330, 2024-05-10T11:32:22.329 INFO:teuthology.orchestra.run.smithi027.stdout: "jlat": { 2024-05-10T11:32:22.329 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.330 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.330 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.330 INFO:teuthology.orchestra.run.smithi027.stdout: "replayed": 8414 2024-05-10T11:32:22.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.330 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_mem": { 2024-05-10T11:32:22.330 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 112, 2024-05-10T11:32:22.330 INFO:teuthology.orchestra.run.smithi027.stdout: "ino+": 1775, 2024-05-10T11:32:22.330 INFO:teuthology.orchestra.run.smithi027.stdout: "ino-": 1663, 2024-05-10T11:32:22.331 INFO:teuthology.orchestra.run.smithi027.stdout: "dir": 27, 2024-05-10T11:32:22.331 INFO:teuthology.orchestra.run.smithi027.stdout: "dir+": 27, 2024-05-10T11:32:22.331 INFO:teuthology.orchestra.run.smithi027.stdout: "dir-": 0, 2024-05-10T11:32:22.331 INFO:teuthology.orchestra.run.smithi027.stdout: "dn": 1803, 2024-05-10T11:32:22.331 INFO:teuthology.orchestra.run.smithi027.stdout: "dn+": 1803, 2024-05-10T11:32:22.331 INFO:teuthology.orchestra.run.smithi027.stdout: "dn-": 0, 2024-05-10T11:32:22.331 INFO:teuthology.orchestra.run.smithi027.stdout: "cap": 0, 2024-05-10T11:32:22.331 INFO:teuthology.orchestra.run.smithi027.stdout: "cap+": 0, 2024-05-10T11:32:22.331 INFO:teuthology.orchestra.run.smithi027.stdout: "cap-": 0, 2024-05-10T11:32:22.332 INFO:teuthology.orchestra.run.smithi027.stdout: "rss": 53900, 2024-05-10T11:32:22.332 INFO:teuthology.orchestra.run.smithi027.stdout: "heap": 215416 2024-05-10T11:32:22.332 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.332 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_server": { 2024-05-10T11:32:22.332 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_client_request": 0, 2024-05-10T11:32:22.332 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_server_request": 0, 2024-05-10T11:32:22.332 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_request": 0, 2024-05-10T11:32:22.332 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_session": 0, 2024-05-10T11:32:22.333 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_peer_request": 0, 2024-05-10T11:32:22.333 INFO:teuthology.orchestra.run.smithi027.stdout: "req_create_latency": { 2024-05-10T11:32:22.333 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.333 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.333 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.333 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getattr_latency": { 2024-05-10T11:32:22.333 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.333 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.334 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.334 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.334 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getfilelock_latency": { 2024-05-10T11:32:22.334 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.334 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.334 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.334 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.334 INFO:teuthology.orchestra.run.smithi027.stdout: "req_link_latency": { 2024-05-10T11:32:22.335 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.335 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.335 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.335 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookup_latency": { 2024-05-10T11:32:22.335 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.335 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.335 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.336 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookuphash_latency": { 2024-05-10T11:32:22.336 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.336 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.336 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.336 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupino_latency": { 2024-05-10T11:32:22.336 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.336 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.336 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.337 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupname_latency": { 2024-05-10T11:32:22.337 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.337 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.337 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.337 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupparent_latency": { 2024-05-10T11:32:22.337 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.337 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.338 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.338 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupsnap_latency": { 2024-05-10T11:32:22.338 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.338 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.338 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.338 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lssnap_latency": { 2024-05-10T11:32:22.339 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.339 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.339 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.339 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mkdir_latency": { 2024-05-10T11:32:22.340 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.340 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.340 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.340 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mknod_latency": { 2024-05-10T11:32:22.340 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.340 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.340 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.341 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mksnap_latency": { 2024-05-10T11:32:22.341 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.341 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.341 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.341 INFO:teuthology.orchestra.run.smithi027.stdout: "req_open_latency": { 2024-05-10T11:32:22.342 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.342 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.342 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.342 INFO:teuthology.orchestra.run.smithi027.stdout: "req_readdir_latency": { 2024-05-10T11:32:22.342 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.342 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.342 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.343 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rename_latency": { 2024-05-10T11:32:22.343 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.343 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.343 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.343 INFO:teuthology.orchestra.run.smithi027.stdout: "req_renamesnap_latency": { 2024-05-10T11:32:22.343 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.344 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.344 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.344 INFO:teuthology.orchestra.run.smithi027.stdout: "req_snapdiff_latency": { 2024-05-10T11:32:22.344 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.344 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.344 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.344 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmdir_latency": { 2024-05-10T11:32:22.345 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.345 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.345 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.345 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmsnap_latency": { 2024-05-10T11:32:22.345 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.345 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.345 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.346 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmxattr_latency": { 2024-05-10T11:32:22.346 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.346 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.346 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.346 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setattr_latency": { 2024-05-10T11:32:22.346 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.346 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.347 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.347 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setdirlayout_latency": { 2024-05-10T11:32:22.347 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.347 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.347 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.347 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setfilelock_latency": { 2024-05-10T11:32:22.347 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.348 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.348 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.348 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setlayout_latency": { 2024-05-10T11:32:22.348 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.348 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.348 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.349 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setxattr_latency": { 2024-05-10T11:32:22.349 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.349 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.349 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.349 INFO:teuthology.orchestra.run.smithi027.stdout: "req_symlink_latency": { 2024-05-10T11:32:22.349 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.349 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.349 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.350 INFO:teuthology.orchestra.run.smithi027.stdout: "req_unlink_latency": { 2024-05-10T11:32:22.350 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.350 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.350 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.350 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_revoke_eviction": 0, 2024-05-10T11:32:22.350 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_acquisition_throttle": 0, 2024-05-10T11:32:22.351 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getvxattr_latency": { 2024-05-10T11:32:22.351 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.351 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.351 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.351 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:22.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.351 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_sessions": { 2024-05-10T11:32:22.351 INFO:teuthology.orchestra.run.smithi027.stdout: "session_count": 2, 2024-05-10T11:32:22.351 INFO:teuthology.orchestra.run.smithi027.stdout: "session_add": 3, 2024-05-10T11:32:22.352 INFO:teuthology.orchestra.run.smithi027.stdout: "session_remove": 1, 2024-05-10T11:32:22.352 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_open": 2, 2024-05-10T11:32:22.352 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_stale": 0, 2024-05-10T11:32:22.352 INFO:teuthology.orchestra.run.smithi027.stdout: "total_load": 0, 2024-05-10T11:32:22.352 INFO:teuthology.orchestra.run.smithi027.stdout: "average_load": 0, 2024-05-10T11:32:22.352 INFO:teuthology.orchestra.run.smithi027.stdout: "avg_session_uptime": 4190, 2024-05-10T11:32:22.352 INFO:teuthology.orchestra.run.smithi027.stdout: "mdthresh_evicted": 0 2024-05-10T11:32:22.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.353 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-05-10T11:32:22.353 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-05-10T11:32:22.353 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-05-10T11:32:22.353 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-05-10T11:32:22.353 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-05-10T11:32:22.353 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-05-10T11:32:22.353 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-05-10T11:32:22.353 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-10T11:32:22.353 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-05-10T11:32:22.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-10T11:32:22.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-05-10T11:32:22.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-05-10T11:32:22.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-05-10T11:32:22.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-10T11:32:22.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-05-10T11:32:22.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-05-10T11:32:22.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-05-10T11:32:22.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-05-10T11:32:22.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-05-10T11:32:22.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-10T11:32:22.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-05-10T11:32:22.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-10T11:32:22.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-05-10T11:32:22.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-05-10T11:32:22.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-05-10T11:32:22.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-05-10T11:32:22.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-05-10T11:32:22.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-10T11:32:22.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-05-10T11:32:22.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-05-10T11:32:22.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-05-10T11:32:22.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-05-10T11:32:22.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-05-10T11:32:22.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-05-10T11:32:22.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-05-10T11:32:22.357 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-05-10T11:32:22.357 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-05-10T11:32:22.357 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 204713, 2024-05-10T11:32:22.357 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 158, 2024-05-10T11:32:22.357 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-05-10T11:32:22.357 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-05-10T11:32:22.357 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-05-10T11:32:22.357 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-05-10T11:32:22.357 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-05-10T11:32:22.358 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-05-10T11:32:22.358 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-05-10T11:32:22.358 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-05-10T11:32:22.358 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 9808, 2024-05-10T11:32:22.358 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 189, 2024-05-10T11:32:22.358 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-05-10T11:32:22.358 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-05-10T11:32:22.358 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-05-10T11:32:22.359 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-05-10T11:32:22.359 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 1600104, 2024-05-10T11:32:22.359 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 12627, 2024-05-10T11:32:22.359 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-05-10T11:32:22.359 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-05-10T11:32:22.359 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-05-10T11:32:22.359 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-05-10T11:32:22.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.359 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter": { 2024-05-10T11:32:22.360 INFO:teuthology.orchestra.run.smithi027.stdout: "op_active": 0, 2024-05-10T11:32:22.360 INFO:teuthology.orchestra.run.smithi027.stdout: "op_laggy": 0, 2024-05-10T11:32:22.360 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send": 4111, 2024-05-10T11:32:22.360 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send_bytes": 21, 2024-05-10T11:32:22.360 INFO:teuthology.orchestra.run.smithi027.stdout: "op_resend": 3, 2024-05-10T11:32:22.360 INFO:teuthology.orchestra.run.smithi027.stdout: "op_reply": 4111, 2024-05-10T11:32:22.360 INFO:teuthology.orchestra.run.smithi027.stdout: "op_latency": { 2024-05-10T11:32:22.360 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 4111, 2024-05-10T11:32:22.360 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 7.087821328, 2024-05-10T11:32:22.361 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.001724111 2024-05-10T11:32:22.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.361 INFO:teuthology.orchestra.run.smithi027.stdout: "op_inflight": 0, 2024-05-10T11:32:22.361 INFO:teuthology.orchestra.run.smithi027.stdout: "oplen_avg": { 2024-05-10T11:32:22.361 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 4111, 2024-05-10T11:32:22.361 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 4112 2024-05-10T11:32:22.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.361 INFO:teuthology.orchestra.run.smithi027.stdout: "op": 4111, 2024-05-10T11:32:22.362 INFO:teuthology.orchestra.run.smithi027.stdout: "op_r": 4111, 2024-05-10T11:32:22.362 INFO:teuthology.orchestra.run.smithi027.stdout: "op_w": 0, 2024-05-10T11:32:22.362 INFO:teuthology.orchestra.run.smithi027.stdout: "op_rmw": 0, 2024-05-10T11:32:22.362 INFO:teuthology.orchestra.run.smithi027.stdout: "op_pg": 0, 2024-05-10T11:32:22.362 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_stat": 2521, 2024-05-10T11:32:22.362 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_create": 0, 2024-05-10T11:32:22.362 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_read": 1588, 2024-05-10T11:32:22.362 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_write": 0, 2024-05-10T11:32:22.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writefull": 0, 2024-05-10T11:32:22.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writesame": 0, 2024-05-10T11:32:22.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_append": 0, 2024-05-10T11:32:22.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_zero": 0, 2024-05-10T11:32:22.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_truncate": 0, 2024-05-10T11:32:22.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_delete": 0, 2024-05-10T11:32:22.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_mapext": 0, 2024-05-10T11:32:22.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_sparse_read": 0, 2024-05-10T11:32:22.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_clonerange": 0, 2024-05-10T11:32:22.364 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_getxattr": 1, 2024-05-10T11:32:22.364 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_setxattr": 0, 2024-05-10T11:32:22.364 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_cmpxattr": 0, 2024-05-10T11:32:22.364 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_rmxattr": 0, 2024-05-10T11:32:22.364 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_resetxattrs": 0, 2024-05-10T11:32:22.364 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_call": 0, 2024-05-10T11:32:22.364 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_watch": 0, 2024-05-10T11:32:22.364 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_notify": 0, 2024-05-10T11:32:22.364 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_src_cmpxattr": 0, 2024-05-10T11:32:22.365 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls": 0, 2024-05-10T11:32:22.365 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls_filter": 0, 2024-05-10T11:32:22.365 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_other": 0, 2024-05-10T11:32:22.365 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_active": 0, 2024-05-10T11:32:22.365 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_send": 0, 2024-05-10T11:32:22.365 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_resend": 0, 2024-05-10T11:32:22.365 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_ping": 0, 2024-05-10T11:32:22.365 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_active": 0, 2024-05-10T11:32:22.366 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_send": 0, 2024-05-10T11:32:22.366 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_resend": 0, 2024-05-10T11:32:22.366 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_active": 0, 2024-05-10T11:32:22.366 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_send": 0, 2024-05-10T11:32:22.366 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_resend": 0, 2024-05-10T11:32:22.366 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_active": 0, 2024-05-10T11:32:22.366 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_send": 0, 2024-05-10T11:32:22.366 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_resend": 0, 2024-05-10T11:32:22.367 INFO:teuthology.orchestra.run.smithi027.stdout: "command_active": 0, 2024-05-10T11:32:22.367 INFO:teuthology.orchestra.run.smithi027.stdout: "command_send": 0, 2024-05-10T11:32:22.367 INFO:teuthology.orchestra.run.smithi027.stdout: "command_resend": 0, 2024-05-10T11:32:22.367 INFO:teuthology.orchestra.run.smithi027.stdout: "map_epoch": 117, 2024-05-10T11:32:22.367 INFO:teuthology.orchestra.run.smithi027.stdout: "map_full": 0, 2024-05-10T11:32:22.367 INFO:teuthology.orchestra.run.smithi027.stdout: "map_inc": 29, 2024-05-10T11:32:22.367 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_sessions": 6, 2024-05-10T11:32:22.367 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_open": 8, 2024-05-10T11:32:22.367 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_close": 2, 2024-05-10T11:32:22.368 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_laggy": 0, 2024-05-10T11:32:22.368 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_wr": 0, 2024-05-10T11:32:22.368 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_rd": 2, 2024-05-10T11:32:22.368 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_del": 0 2024-05-10T11:32:22.368 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.368 INFO:teuthology.orchestra.run.smithi027.stdout: "oft": { 2024-05-10T11:32:22.368 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_objs": 0, 2024-05-10T11:32:22.368 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_kv_pairs": 0, 2024-05-10T11:32:22.368 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_updates": 0, 2024-05-10T11:32:22.369 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_removes": 0 2024-05-10T11:32:22.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.369 INFO:teuthology.orchestra.run.smithi027.stdout: "osd-slow-ops": { 2024-05-10T11:32:22.369 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_ops_count": 0 2024-05-10T11:32:22.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.369 INFO:teuthology.orchestra.run.smithi027.stdout: "purge_queue": { 2024-05-10T11:32:22.369 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops": 0, 2024-05-10T11:32:22.369 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops_high_water": 0, 2024-05-10T11:32:22.370 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing": 0, 2024-05-10T11:32:22.370 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_high_water": 0, 2024-05-10T11:32:22.370 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executed_ops": 0, 2024-05-10T11:32:22.370 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executed": 0, 2024-05-10T11:32:22.370 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_item_in_journal": 0 2024-05-10T11:32:22.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.370 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-10T11:32:22.370 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-05-10T11:32:22.371 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-05-10T11:32:22.371 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-05-10T11:32:22.371 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 4574, 2024-05-10T11:32:22.371 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 14839118, 2024-05-10T11:32:22.371 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:22.371 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 4574, 2024-05-10T11:32:22.371 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-05-10T11:32:22.371 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-05-10T11:32:22.371 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 4574, 2024-05-10T11:32:22.372 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 14839118, 2024-05-10T11:32:22.372 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-05-10T11:32:22.372 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.372 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.372 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.372 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:22.372 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.372 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_bytes": { 2024-05-10T11:32:22.372 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-05-10T11:32:22.373 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-05-10T11:32:22.373 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-05-10T11:32:22.373 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-05-10T11:32:22.373 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-05-10T11:32:22.373 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:22.373 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:22.373 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 4111, 2024-05-10T11:32:22.373 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 12044484, 2024-05-10T11:32:22.374 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 4111, 2024-05-10T11:32:22.374 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 12044484, 2024-05-10T11:32:22.374 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-05-10T11:32:22.374 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.374 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.374 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:22.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.374 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_ops": { 2024-05-10T11:32:22.375 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-05-10T11:32:22.375 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1024, 2024-05-10T11:32:22.375 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-05-10T11:32:22.375 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-05-10T11:32:22.375 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-05-10T11:32:22.375 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:22.375 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:22.375 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 4111, 2024-05-10T11:32:22.376 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 4111, 2024-05-10T11:32:22.376 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 4111, 2024-05-10T11:32:22.376 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 4111, 2024-05-10T11:32:22.376 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-05-10T11:32:22.376 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.376 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.376 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.376 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:22.376 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.377 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle": { 2024-05-10T11:32:22.377 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-05-10T11:32:22.377 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-05-10T11:32:22.377 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-05-10T11:32:22.377 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-05-10T11:32:22.377 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-05-10T11:32:22.377 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:22.377 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:22.378 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-05-10T11:32:22.378 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-05-10T11:32:22.378 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-05-10T11:32:22.378 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-05-10T11:32:22.378 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-05-10T11:32:22.378 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.378 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.378 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.378 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:22.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:22.379 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle-0x5611fb088780": { 2024-05-10T11:32:22.379 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-05-10T11:32:22.379 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-05-10T11:32:22.379 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-05-10T11:32:22.379 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-05-10T11:32:22.379 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-05-10T11:32:22.379 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:22.379 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:22.380 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-05-10T11:32:22.380 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-05-10T11:32:22.380 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-05-10T11:32:22.380 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-05-10T11:32:22.380 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-05-10T11:32:22.380 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:22.380 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:22.380 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:22.381 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:22.381 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:22.381 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:32:23.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:22 smithi027 ceph-mon[30054]: pgmap v603: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 4.2 MiB/s wr, 140 op/s 2024-05-10T11:32:23.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:22 smithi076 ceph-mon[34152]: pgmap v603: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 4.2 MiB/s wr, 140 op/s 2024-05-10T11:32:23.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:22 smithi189 ceph-mon[34842]: pgmap v603: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 4.2 MiB/s wr, 140 op/s 2024-05-10T11:32:23.797 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-05-10T11:32:23.797 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-05-10T11:32:23.797 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-05-10T11:32:23.797 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-05-10T11:32:23.797 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-05-10T11:32:23.797 DEBUG:tasks.check_counter:Getting stats from d 2024-05-10T11:32:23.798 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph daemon mds.d perf dump 2024-05-10T11:32:24.356 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:32:25.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:24 smithi027 ceph-mon[30054]: pgmap v604: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 3.6 MiB/s wr, 121 op/s 2024-05-10T11:32:25.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:24 smithi076 ceph-mon[34152]: pgmap v604: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 3.6 MiB/s wr, 121 op/s 2024-05-10T11:32:25.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:24 smithi189 ceph-mon[34842]: pgmap v604: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 3.6 MiB/s wr, 121 op/s 2024-05-10T11:32:26.819 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:32:26.820 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-05-10T11:32:26.820 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 417, 2024-05-10T11:32:26.820 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 405, 2024-05-10T11:32:26.820 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 1916384, 2024-05-10T11:32:26.820 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 250656, 2024-05-10T11:32:26.821 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 4, 2024-05-10T11:32:26.821 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 0, 2024-05-10T11:32:26.821 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.167945200, 2024-05-10T11:32:26.821 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.065697767, 2024-05-10T11:32:26.821 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.099714055, 2024-05-10T11:32:26.821 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.018786943, 2024-05-10T11:32:26.821 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:26.821 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 404, 2024-05-10T11:32:26.822 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.055263858, 2024-05-10T11:32:26.822 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000136791 2024-05-10T11:32:26.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:26.822 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:26.822 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:26.822 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:26.822 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:26.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:26.823 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 1916384, 2024-05-10T11:32:26.823 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 250656 2024-05-10T11:32:26.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:26.823 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-05-10T11:32:26.823 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 7, 2024-05-10T11:32:26.824 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 312, 2024-05-10T11:32:26.824 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 3472, 2024-05-10T11:32:26.824 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 291216, 2024-05-10T11:32:26.824 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 6, 2024-05-10T11:32:26.824 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-05-10T11:32:26.824 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.066863934, 2024-05-10T11:32:26.824 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.048130854, 2024-05-10T11:32:26.824 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.006755576, 2024-05-10T11:32:26.825 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-10T11:32:26.825 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:26.825 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 310, 2024-05-10T11:32:26.825 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.057941147, 2024-05-10T11:32:26.825 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000186906 2024-05-10T11:32:26.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:26.825 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:26.825 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:26.826 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:26.826 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:26.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:26.826 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-05-10T11:32:26.826 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 291216 2024-05-10T11:32:26.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:26.826 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-05-10T11:32:26.827 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 0, 2024-05-10T11:32:26.827 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 2, 2024-05-10T11:32:26.827 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 0, 2024-05-10T11:32:26.827 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 0, 2024-05-10T11:32:26.827 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 2, 2024-05-10T11:32:26.827 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 1, 2024-05-10T11:32:26.827 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.001108866, 2024-05-10T11:32:26.827 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.000000000, 2024-05-10T11:32:26.828 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.000909860, 2024-05-10T11:32:26.828 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-10T11:32:26.828 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:26.828 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:26.828 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:26.828 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:26.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:26.828 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:26.829 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:26.829 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:26.829 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:26.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:26.829 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-10T11:32:26.829 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-05-10T11:32:26.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:26.829 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-05-10T11:32:26.830 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 0, 2024-05-10T11:32:26.830 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-05-10T11:32:26.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:26.830 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-05-10T11:32:26.830 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-05-10T11:32:26.830 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-05-10T11:32:26.830 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-05-10T11:32:26.830 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-05-10T11:32:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-05-10T11:32:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-05-10T11:32:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-10T11:32:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-05-10T11:32:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-10T11:32:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-05-10T11:32:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-05-10T11:32:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-05-10T11:32:26.831 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-10T11:32:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-05-10T11:32:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-05-10T11:32:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-05-10T11:32:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-05-10T11:32:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-05-10T11:32:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-10T11:32:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-05-10T11:32:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-10T11:32:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-05-10T11:32:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-05-10T11:32:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-05-10T11:32:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-05-10T11:32:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-05-10T11:32:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-10T11:32:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-05-10T11:32:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-05-10T11:32:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-05-10T11:32:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-05-10T11:32:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-05-10T11:32:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-05-10T11:32:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-05-10T11:32:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-05-10T11:32:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-05-10T11:32:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 149762, 2024-05-10T11:32:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 32, 2024-05-10T11:32:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-05-10T11:32:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-05-10T11:32:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-05-10T11:32:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-05-10T11:32:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-05-10T11:32:26.835 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-05-10T11:32:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-05-10T11:32:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-05-10T11:32:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 0, 2024-05-10T11:32:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 0, 2024-05-10T11:32:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-05-10T11:32:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-05-10T11:32:26.836 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-05-10T11:32:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-05-10T11:32:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 928, 2024-05-10T11:32:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 2, 2024-05-10T11:32:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-05-10T11:32:26.837 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-05-10T11:32:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-05-10T11:32:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-05-10T11:32:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-10T11:32:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-05-10T11:32:26.838 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-05-10T11:32:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-05-10T11:32:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 420, 2024-05-10T11:32:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 1857488, 2024-05-10T11:32:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 420, 2024-05-10T11:32:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-05-10T11:32:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-05-10T11:32:26.839 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 420, 2024-05-10T11:32:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 1857488, 2024-05-10T11:32:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-05-10T11:32:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:26.840 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:26.841 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:32:26.842 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:26 smithi027 ceph-mon[30054]: pgmap v605: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 5.8 MiB/s wr, 197 op/s 2024-05-10T11:32:27.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:26 smithi076 ceph-mon[34152]: pgmap v605: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 5.8 MiB/s wr, 197 op/s 2024-05-10T11:32:27.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:26 smithi189 ceph-mon[34842]: pgmap v605: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 5.8 MiB/s wr, 197 op/s 2024-05-10T11:32:27.951 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-05-10T11:32:27.951 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-05-10T11:32:27.951 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-05-10T11:32:27.951 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-05-10T11:32:27.952 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-05-10T11:32:27.952 DEBUG:tasks.check_counter:Getting stats from g 2024-05-10T11:32:27.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph daemon mds.g perf dump 2024-05-10T11:32:28.438 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:32:28.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:28 smithi027 ceph-mon[30054]: pgmap v606: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 4.2 MiB/s wr, 140 op/s 2024-05-10T11:32:29.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:28 smithi076 ceph-mon[34152]: pgmap v606: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 4.2 MiB/s wr, 140 op/s 2024-05-10T11:32:29.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:28 smithi189 ceph-mon[34842]: pgmap v606: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 4.2 MiB/s wr, 140 op/s 2024-05-10T11:32:30.711 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:32:30.712 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-05-10T11:32:30.712 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 413, 2024-05-10T11:32:30.712 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 401, 2024-05-10T11:32:30.712 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 1915296, 2024-05-10T11:32:30.713 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 248096, 2024-05-10T11:32:30.713 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 4, 2024-05-10T11:32:30.713 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 0, 2024-05-10T11:32:30.713 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.167648237, 2024-05-10T11:32:30.713 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.065004376, 2024-05-10T11:32:30.713 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.098826849, 2024-05-10T11:32:30.713 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.019393111, 2024-05-10T11:32:30.713 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:30.714 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 400, 2024-05-10T11:32:30.714 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.043643431, 2024-05-10T11:32:30.714 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000109108 2024-05-10T11:32:30.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:30.714 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:30.714 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:30.714 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:30.714 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 1915296, 2024-05-10T11:32:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 248096 2024-05-10T11:32:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-05-10T11:32:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 3, 2024-05-10T11:32:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 307, 2024-05-10T11:32:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 480, 2024-05-10T11:32:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 290240, 2024-05-10T11:32:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 6, 2024-05-10T11:32:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-05-10T11:32:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.058520113, 2024-05-10T11:32:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.044245406, 2024-05-10T11:32:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.005557957, 2024-05-10T11:32:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-10T11:32:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 304, 2024-05-10T11:32:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.057779348, 2024-05-10T11:32:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000190063 2024-05-10T11:32:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-10T11:32:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 290240 2024-05-10T11:32:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-05-10T11:32:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 4, 2024-05-10T11:32:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 4, 2024-05-10T11:32:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 2992, 2024-05-10T11:32:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 448, 2024-05-10T11:32:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 2, 2024-05-10T11:32:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 1, 2024-05-10T11:32:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.001810646, 2024-05-10T11:32:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.000069835, 2024-05-10T11:32:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.001666456, 2024-05-10T11:32:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-10T11:32:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 3, 2024-05-10T11:32:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.002163006, 2024-05-10T11:32:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000721002 2024-05-10T11:32:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-05-10T11:32:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 448 2024-05-10T11:32:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-05-10T11:32:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 0, 2024-05-10T11:32:30.723 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-05-10T11:32:30.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:30.723 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-05-10T11:32:30.723 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-05-10T11:32:30.723 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-05-10T11:32:30.723 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-05-10T11:32:30.723 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-05-10T11:32:30.724 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-05-10T11:32:30.724 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-05-10T11:32:30.724 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-10T11:32:30.724 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-05-10T11:32:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-10T11:32:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-05-10T11:32:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-05-10T11:32:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-05-10T11:32:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-10T11:32:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-05-10T11:32:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-05-10T11:32:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-05-10T11:32:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-05-10T11:32:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-05-10T11:32:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-10T11:32:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-05-10T11:32:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-10T11:32:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-05-10T11:32:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-05-10T11:32:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-05-10T11:32:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-05-10T11:32:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-05-10T11:32:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-10T11:32:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-05-10T11:32:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-05-10T11:32:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-05-10T11:32:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-05-10T11:32:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-05-10T11:32:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-05-10T11:32:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-05-10T11:32:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-05-10T11:32:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-05-10T11:32:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 148714, 2024-05-10T11:32:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 30, 2024-05-10T11:32:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-05-10T11:32:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-05-10T11:32:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-05-10T11:32:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-05-10T11:32:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-05-10T11:32:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-05-10T11:32:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-05-10T11:32:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-05-10T11:32:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 0, 2024-05-10T11:32:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 0, 2024-05-10T11:32:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-05-10T11:32:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-05-10T11:32:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-05-10T11:32:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-05-10T11:32:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 928, 2024-05-10T11:32:30.731 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 2, 2024-05-10T11:32:30.731 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-05-10T11:32:30.731 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-05-10T11:32:30.731 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-05-10T11:32:30.731 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-05-10T11:32:30.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:30.731 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-10T11:32:30.731 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-05-10T11:32:30.732 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-05-10T11:32:30.732 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-05-10T11:32:30.732 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 416, 2024-05-10T11:32:30.732 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 1856968, 2024-05-10T11:32:30.732 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:30.732 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 416, 2024-05-10T11:32:30.732 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-05-10T11:32:30.732 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-05-10T11:32:30.732 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 416, 2024-05-10T11:32:30.733 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 1856968, 2024-05-10T11:32:30.733 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-05-10T11:32:30.733 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:30.733 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:30.733 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:30.733 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:30.733 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:30.733 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:32:31.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:30 smithi027 ceph-mon[30054]: pgmap v607: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.2 MiB/s wr, 179 op/s 2024-05-10T11:32:31.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:30 smithi076 ceph-mon[34152]: pgmap v607: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.2 MiB/s wr, 179 op/s 2024-05-10T11:32:31.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:30 smithi189 ceph-mon[34842]: pgmap v607: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.2 MiB/s wr, 179 op/s 2024-05-10T11:32:31.709 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-05-10T11:32:31.709 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-05-10T11:32:31.709 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-05-10T11:32:31.710 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-05-10T11:32:31.710 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-05-10T11:32:31.710 DEBUG:tasks.check_counter:Getting stats from j 2024-05-10T11:32:31.711 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph daemon mds.j perf dump 2024-05-10T11:32:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:32:33.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:32 smithi027 ceph-mon[30054]: pgmap v608: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 3.9 MiB/s wr, 140 op/s 2024-05-10T11:32:33.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:32 smithi076 ceph-mon[34152]: pgmap v608: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 3.9 MiB/s wr, 140 op/s 2024-05-10T11:32:33.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:32 smithi189 ceph-mon[34842]: pgmap v608: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 3.9 MiB/s wr, 140 op/s 2024-05-10T11:32:34.619 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-10T11:32:34.619 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-05-10T11:32:34.619 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 3689, 2024-05-10T11:32:34.619 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 3690, 2024-05-10T11:32:34.619 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 63970662, 2024-05-10T11:32:34.619 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 1172791, 2024-05-10T11:32:34.620 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 8, 2024-05-10T11:32:34.620 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 4, 2024-05-10T11:32:34.620 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.804843620, 2024-05-10T11:32:34.620 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.217712033, 2024-05-10T11:32:34.620 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.317723519, 2024-05-10T11:32:34.620 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.208703530, 2024-05-10T11:32:34.620 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:34.620 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 3688, 2024-05-10T11:32:34.621 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.337524648, 2024-05-10T11:32:34.621 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000091519 2024-05-10T11:32:34.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.621 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:34.621 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.621 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.621 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.622 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-10T11:32:34.622 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-05-10T11:32:34.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.622 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-05-10T11:32:34.622 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 841, 2024-05-10T11:32:34.622 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 810, 2024-05-10T11:32:34.622 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 69279476, 2024-05-10T11:32:34.622 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 375410, 2024-05-10T11:32:34.623 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 6, 2024-05-10T11:32:34.623 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 5, 2024-05-10T11:32:34.623 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.399618576, 2024-05-10T11:32:34.623 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.065289581, 2024-05-10T11:32:34.623 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.321322482, 2024-05-10T11:32:34.623 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.045936753, 2024-05-10T11:32:34.623 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:34.623 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 810, 2024-05-10T11:32:34.624 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.121743681, 2024-05-10T11:32:34.624 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000150300 2024-05-10T11:32:34.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.624 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:34.624 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.624 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.624 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.624 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 2076800, 2024-05-10T11:32:34.625 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 258960 2024-05-10T11:32:34.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.625 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-05-10T11:32:34.625 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 327, 2024-05-10T11:32:34.625 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 637, 2024-05-10T11:32:34.625 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 59097443, 2024-05-10T11:32:34.625 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 871416, 2024-05-10T11:32:34.625 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 10, 2024-05-10T11:32:34.625 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 6, 2024-05-10T11:32:34.626 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.266689215, 2024-05-10T11:32:34.626 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.062539827, 2024-05-10T11:32:34.626 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.171553747, 2024-05-10T11:32:34.626 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.017860106, 2024-05-10T11:32:34.626 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:34.626 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 634, 2024-05-10T11:32:34.626 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.099644748, 2024-05-10T11:32:34.626 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000157168 2024-05-10T11:32:34.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.627 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:34.627 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.627 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.627 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.627 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-10T11:32:34.627 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 768384 2024-05-10T11:32:34.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.628 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-05-10T11:32:34.628 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 1, 2024-05-10T11:32:34.628 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-05-10T11:32:34.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.628 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-MDSRank": { 2024-05-10T11:32:34.628 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-05-10T11:32:34.628 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-05-10T11:32:34.629 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 7323, 2024-05-10T11:32:34.629 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.374373537, 2024-05-10T11:32:34.629 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000051122 2024-05-10T11:32:34.629 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:34.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.629 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-PurgeQueue": { 2024-05-10T11:32:34.629 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-05-10T11:32:34.629 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-05-10T11:32:34.629 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.630 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.630 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.630 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:34.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.630 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-05-10T11:32:34.630 INFO:teuthology.orchestra.run.smithi027.stdout: "request": 0, 2024-05-10T11:32:34.630 INFO:teuthology.orchestra.run.smithi027.stdout: "reply": 0, 2024-05-10T11:32:34.630 INFO:teuthology.orchestra.run.smithi027.stdout: "reply_latency": { 2024-05-10T11:32:34.631 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.631 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.631 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.631 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_reply": 0, 2024-05-10T11:32:34.631 INFO:teuthology.orchestra.run.smithi027.stdout: "forward": 0, 2024-05-10T11:32:34.631 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_complete": 0, 2024-05-10T11:32:34.631 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_keys": 0, 2024-05-10T11:32:34.631 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_commit": 0, 2024-05-10T11:32:34.632 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_split": 0, 2024-05-10T11:32:34.632 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_merge": 0, 2024-05-10T11:32:34.632 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes": 12060, 2024-05-10T11:32:34.632 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_top": 0, 2024-05-10T11:32:34.632 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_bottom": 12060, 2024-05-10T11:32:34.632 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pin_tail": 0, 2024-05-10T11:32:34.632 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pinned": 12060, 2024-05-10T11:32:34.632 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_expired": 0, 2024-05-10T11:32:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_with_caps": 0, 2024-05-10T11:32:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": 0, 2024-05-10T11:32:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: "subtrees": 6, 2024-05-10T11:32:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse": 0, 2024-05-10T11:32:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_hit": 0, 2024-05-10T11:32:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_forward": 0, 2024-05-10T11:32:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_discover": 0, 2024-05-10T11:32:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_dir_fetch": 0, 2024-05-10T11:32:34.633 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_remote_ino": 0, 2024-05-10T11:32:34.634 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_lock": 0, 2024-05-10T11:32:34.634 INFO:teuthology.orchestra.run.smithi027.stdout: "load_cent": 0, 2024-05-10T11:32:34.634 INFO:teuthology.orchestra.run.smithi027.stdout: "q": 0, 2024-05-10T11:32:34.634 INFO:teuthology.orchestra.run.smithi027.stdout: "exported": 0, 2024-05-10T11:32:34.634 INFO:teuthology.orchestra.run.smithi027.stdout: "exported_inodes": 0, 2024-05-10T11:32:34.634 INFO:teuthology.orchestra.run.smithi027.stdout: "imported": 0, 2024-05-10T11:32:34.634 INFO:teuthology.orchestra.run.smithi027.stdout: "imported_inodes": 0, 2024-05-10T11:32:34.634 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_dir_fetch": 0, 2024-05-10T11:32:34.635 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_backtrace_fetch": 0, 2024-05-10T11:32:34.635 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_peer_discover": 0, 2024-05-10T11:32:34.635 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rfiles": 0, 2024-05-10T11:32:34.635 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rbytes": 0, 2024-05-10T11:32:34.635 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rsnaps": 0, 2024-05-10T11:32:34.635 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_fetch": 0, 2024-05-10T11:32:34.635 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_set_tag": 0, 2024-05-10T11:32:34.635 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_repaired": 0, 2024-05-10T11:32:34.636 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_inotable_repaired": 0, 2024-05-10T11:32:34.636 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_inodes": 0, 2024-05-10T11:32:34.636 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_base_inodes": 0, 2024-05-10T11:32:34.636 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dirfrag_rstats": 0, 2024-05-10T11:32:34.636 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_file_inodes": 0, 2024-05-10T11:32:34.636 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_inode_file_caps": 0, 2024-05-10T11:32:34.636 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_revoke": 0, 2024-05-10T11:32:34.636 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_grant": 0, 2024-05-10T11:32:34.636 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_trunc": 0, 2024-05-10T11:32:34.637 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-10T11:32:34.637 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-10T11:32:34.637 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps": 0, 2024-05-10T11:32:34.637 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps_dirty": 0, 2024-05-10T11:32:34.637 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_cap_release": 0, 2024-05-10T11:32:34.637 INFO:teuthology.orchestra.run.smithi027.stdout: "process_request_cap_release": 0 2024-05-10T11:32:34.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.637 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache": { 2024-05-10T11:32:34.638 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update": 0, 2024-05-10T11:32:34.638 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update_receipt": 0, 2024-05-10T11:32:34.638 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_try_discover": 0, 2024-05-10T11:32:34.638 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_send_discover": 0, 2024-05-10T11:32:34.638 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_handle_discover": 0, 2024-05-10T11:32:34.638 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays": 0, 2024-05-10T11:32:34.638 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_delayed": 0, 2024-05-10T11:32:34.638 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_enqueuing": 0, 2024-05-10T11:32:34.642 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_created": 0, 2024-05-10T11:32:34.642 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_enqueued": 0, 2024-05-10T11:32:34.642 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_reintegrated": 0, 2024-05-10T11:32:34.642 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_migrated": 0, 2024-05-10T11:32:34.642 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_processing": 0, 2024-05-10T11:32:34.643 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_enqueued": 0, 2024-05-10T11:32:34.643 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_prioritized": 0, 2024-05-10T11:32:34.643 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_started": 0, 2024-05-10T11:32:34.643 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_completed": 0, 2024-05-10T11:32:34.643 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_quiesce_path": 0, 2024-05-10T11:32:34.643 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_quiesce_inode": 0, 2024-05-10T11:32:34.643 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_enqueue_scrub": 0, 2024-05-10T11:32:34.643 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_exportdir": 0, 2024-05-10T11:32:34.644 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_flush": 0, 2024-05-10T11:32:34.644 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragmentdir": 0, 2024-05-10T11:32:34.644 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragstats": 0, 2024-05-10T11:32:34.644 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_inodestats": 0 2024-05-10T11:32:34.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.644 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log": { 2024-05-10T11:32:34.644 INFO:teuthology.orchestra.run.smithi027.stdout: "evlrg": 0, 2024-05-10T11:32:34.645 INFO:teuthology.orchestra.run.smithi027.stdout: "evadd": 0, 2024-05-10T11:32:34.645 INFO:teuthology.orchestra.run.smithi027.stdout: "evex": 0, 2024-05-10T11:32:34.645 INFO:teuthology.orchestra.run.smithi027.stdout: "evtrm": 0, 2024-05-10T11:32:34.645 INFO:teuthology.orchestra.run.smithi027.stdout: "ev": 55361, 2024-05-10T11:32:34.645 INFO:teuthology.orchestra.run.smithi027.stdout: "evexg": 0, 2024-05-10T11:32:34.645 INFO:teuthology.orchestra.run.smithi027.stdout: "evexd": 0, 2024-05-10T11:32:34.645 INFO:teuthology.orchestra.run.smithi027.stdout: "segadd": 0, 2024-05-10T11:32:34.646 INFO:teuthology.orchestra.run.smithi027.stdout: "segex": 0, 2024-05-10T11:32:34.646 INFO:teuthology.orchestra.run.smithi027.stdout: "segtrm": 0, 2024-05-10T11:32:34.646 INFO:teuthology.orchestra.run.smithi027.stdout: "seg": 82, 2024-05-10T11:32:34.646 INFO:teuthology.orchestra.run.smithi027.stdout: "segmjr": 6, 2024-05-10T11:32:34.646 INFO:teuthology.orchestra.run.smithi027.stdout: "segexg": 0, 2024-05-10T11:32:34.646 INFO:teuthology.orchestra.run.smithi027.stdout: "segexd": 0, 2024-05-10T11:32:34.646 INFO:teuthology.orchestra.run.smithi027.stdout: "expos": 4194304, 2024-05-10T11:32:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: "wrpos": 193232085, 2024-05-10T11:32:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: "rdpos": 193231486, 2024-05-10T11:32:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: "jlat": { 2024-05-10T11:32:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.648 INFO:teuthology.orchestra.run.smithi027.stdout: "replayed": 55361 2024-05-10T11:32:34.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.648 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_mem": { 2024-05-10T11:32:34.648 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 478, 2024-05-10T11:32:34.648 INFO:teuthology.orchestra.run.smithi027.stdout: "ino+": 11979, 2024-05-10T11:32:34.648 INFO:teuthology.orchestra.run.smithi027.stdout: "ino-": 11501, 2024-05-10T11:32:34.648 INFO:teuthology.orchestra.run.smithi027.stdout: "dir": 46, 2024-05-10T11:32:34.648 INFO:teuthology.orchestra.run.smithi027.stdout: "dir+": 83, 2024-05-10T11:32:34.649 INFO:teuthology.orchestra.run.smithi027.stdout: "dir-": 37, 2024-05-10T11:32:34.649 INFO:teuthology.orchestra.run.smithi027.stdout: "dn": 12061, 2024-05-10T11:32:34.649 INFO:teuthology.orchestra.run.smithi027.stdout: "dn+": 12061, 2024-05-10T11:32:34.649 INFO:teuthology.orchestra.run.smithi027.stdout: "dn-": 0, 2024-05-10T11:32:34.649 INFO:teuthology.orchestra.run.smithi027.stdout: "cap": 0, 2024-05-10T11:32:34.649 INFO:teuthology.orchestra.run.smithi027.stdout: "cap+": 0, 2024-05-10T11:32:34.649 INFO:teuthology.orchestra.run.smithi027.stdout: "cap-": 0, 2024-05-10T11:32:34.650 INFO:teuthology.orchestra.run.smithi027.stdout: "rss": 65040, 2024-05-10T11:32:34.650 INFO:teuthology.orchestra.run.smithi027.stdout: "heap": 215416 2024-05-10T11:32:34.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.650 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_server": { 2024-05-10T11:32:34.650 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_client_request": 0, 2024-05-10T11:32:34.650 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_server_request": 0, 2024-05-10T11:32:34.650 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_request": 0, 2024-05-10T11:32:34.651 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_session": 0, 2024-05-10T11:32:34.651 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_peer_request": 0, 2024-05-10T11:32:34.651 INFO:teuthology.orchestra.run.smithi027.stdout: "req_create_latency": { 2024-05-10T11:32:34.651 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.651 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.651 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.652 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getattr_latency": { 2024-05-10T11:32:34.652 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.652 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.652 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.652 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getfilelock_latency": { 2024-05-10T11:32:34.653 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.653 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.653 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.653 INFO:teuthology.orchestra.run.smithi027.stdout: "req_link_latency": { 2024-05-10T11:32:34.653 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.653 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.654 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.654 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookup_latency": { 2024-05-10T11:32:34.654 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.654 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.654 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.655 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookuphash_latency": { 2024-05-10T11:32:34.655 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.655 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.655 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.655 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupino_latency": { 2024-05-10T11:32:34.655 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.655 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.656 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.656 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupname_latency": { 2024-05-10T11:32:34.656 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.656 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.656 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.657 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupparent_latency": { 2024-05-10T11:32:34.657 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.657 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.657 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.657 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupsnap_latency": { 2024-05-10T11:32:34.657 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.657 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.658 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.658 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lssnap_latency": { 2024-05-10T11:32:34.658 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.658 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.658 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.659 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mkdir_latency": { 2024-05-10T11:32:34.660 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.660 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.660 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.660 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mknod_latency": { 2024-05-10T11:32:34.660 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.660 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mksnap_latency": { 2024-05-10T11:32:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "req_open_latency": { 2024-05-10T11:32:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "req_readdir_latency": { 2024-05-10T11:32:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rename_latency": { 2024-05-10T11:32:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "req_renamesnap_latency": { 2024-05-10T11:32:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "req_snapdiff_latency": { 2024-05-10T11:32:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmdir_latency": { 2024-05-10T11:32:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmsnap_latency": { 2024-05-10T11:32:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmxattr_latency": { 2024-05-10T11:32:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setattr_latency": { 2024-05-10T11:32:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setdirlayout_latency": { 2024-05-10T11:32:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setfilelock_latency": { 2024-05-10T11:32:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setlayout_latency": { 2024-05-10T11:32:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setxattr_latency": { 2024-05-10T11:32:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "req_symlink_latency": { 2024-05-10T11:32:34.671 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.671 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.671 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.671 INFO:teuthology.orchestra.run.smithi027.stdout: "req_unlink_latency": { 2024-05-10T11:32:34.671 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.671 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.672 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.672 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_revoke_eviction": 0, 2024-05-10T11:32:34.672 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_acquisition_throttle": 0, 2024-05-10T11:32:34.672 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getvxattr_latency": { 2024-05-10T11:32:34.672 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.672 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.672 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.673 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:34.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.673 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_sessions": { 2024-05-10T11:32:34.673 INFO:teuthology.orchestra.run.smithi027.stdout: "session_count": 2, 2024-05-10T11:32:34.673 INFO:teuthology.orchestra.run.smithi027.stdout: "session_add": 4, 2024-05-10T11:32:34.673 INFO:teuthology.orchestra.run.smithi027.stdout: "session_remove": 2, 2024-05-10T11:32:34.673 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_open": 2, 2024-05-10T11:32:34.674 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_stale": 0, 2024-05-10T11:32:34.674 INFO:teuthology.orchestra.run.smithi027.stdout: "total_load": 0, 2024-05-10T11:32:34.674 INFO:teuthology.orchestra.run.smithi027.stdout: "average_load": 0, 2024-05-10T11:32:34.674 INFO:teuthology.orchestra.run.smithi027.stdout: "avg_session_uptime": 5724, 2024-05-10T11:32:34.674 INFO:teuthology.orchestra.run.smithi027.stdout: "mdthresh_evicted": 0 2024-05-10T11:32:34.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.674 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-05-10T11:32:34.674 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-05-10T11:32:34.675 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-05-10T11:32:34.675 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-05-10T11:32:34.675 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-05-10T11:32:34.675 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-05-10T11:32:34.675 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-05-10T11:32:34.675 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-10T11:32:34.675 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-05-10T11:32:34.676 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-10T11:32:34.676 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-05-10T11:32:34.676 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-05-10T11:32:34.676 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-05-10T11:32:34.676 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-10T11:32:34.676 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-05-10T11:32:34.676 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-05-10T11:32:34.676 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-05-10T11:32:34.677 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-05-10T11:32:34.677 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-05-10T11:32:34.677 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-10T11:32:34.677 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-05-10T11:32:34.677 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-10T11:32:34.677 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-05-10T11:32:34.677 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-05-10T11:32:34.677 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-05-10T11:32:34.678 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-05-10T11:32:34.678 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-05-10T11:32:34.678 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-10T11:32:34.678 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-05-10T11:32:34.678 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-05-10T11:32:34.678 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-05-10T11:32:34.678 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-05-10T11:32:34.679 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-05-10T11:32:34.679 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-05-10T11:32:34.679 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-05-10T11:32:34.679 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-05-10T11:32:34.679 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-05-10T11:32:34.679 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 260846, 2024-05-10T11:32:34.679 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 612, 2024-05-10T11:32:34.680 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-05-10T11:32:34.680 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-05-10T11:32:34.680 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-05-10T11:32:34.680 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-05-10T11:32:34.680 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-05-10T11:32:34.680 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-05-10T11:32:34.680 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-05-10T11:32:34.681 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-05-10T11:32:34.681 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 9808, 2024-05-10T11:32:34.681 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 189, 2024-05-10T11:32:34.681 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-05-10T11:32:34.681 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-05-10T11:32:34.681 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-05-10T11:32:34.681 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-05-10T11:32:34.681 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 9812344, 2024-05-10T11:32:34.682 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 69569, 2024-05-10T11:32:34.682 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-05-10T11:32:34.682 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-05-10T11:32:34.682 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-05-10T11:32:34.682 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-05-10T11:32:34.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.682 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter": { 2024-05-10T11:32:34.683 INFO:teuthology.orchestra.run.smithi027.stdout: "op_active": 0, 2024-05-10T11:32:34.683 INFO:teuthology.orchestra.run.smithi027.stdout: "op_laggy": 0, 2024-05-10T11:32:34.683 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send": 4378, 2024-05-10T11:32:34.683 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send_bytes": 21, 2024-05-10T11:32:34.683 INFO:teuthology.orchestra.run.smithi027.stdout: "op_resend": 3, 2024-05-10T11:32:34.683 INFO:teuthology.orchestra.run.smithi027.stdout: "op_reply": 4378, 2024-05-10T11:32:34.683 INFO:teuthology.orchestra.run.smithi027.stdout: "op_latency": { 2024-05-10T11:32:34.683 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 4378, 2024-05-10T11:32:34.684 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 6.082846669, 2024-05-10T11:32:34.684 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.001389412 2024-05-10T11:32:34.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.684 INFO:teuthology.orchestra.run.smithi027.stdout: "op_inflight": 0, 2024-05-10T11:32:34.684 INFO:teuthology.orchestra.run.smithi027.stdout: "oplen_avg": { 2024-05-10T11:32:34.684 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 4378, 2024-05-10T11:32:34.684 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 4379 2024-05-10T11:32:34.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.685 INFO:teuthology.orchestra.run.smithi027.stdout: "op": 4378, 2024-05-10T11:32:34.685 INFO:teuthology.orchestra.run.smithi027.stdout: "op_r": 4378, 2024-05-10T11:32:34.685 INFO:teuthology.orchestra.run.smithi027.stdout: "op_w": 0, 2024-05-10T11:32:34.685 INFO:teuthology.orchestra.run.smithi027.stdout: "op_rmw": 0, 2024-05-10T11:32:34.685 INFO:teuthology.orchestra.run.smithi027.stdout: "op_pg": 0, 2024-05-10T11:32:34.685 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_stat": 2662, 2024-05-10T11:32:34.686 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_create": 0, 2024-05-10T11:32:34.686 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_read": 1714, 2024-05-10T11:32:34.686 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_write": 0, 2024-05-10T11:32:34.686 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writefull": 0, 2024-05-10T11:32:34.686 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writesame": 0, 2024-05-10T11:32:34.686 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_append": 0, 2024-05-10T11:32:34.686 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_zero": 0, 2024-05-10T11:32:34.687 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_truncate": 0, 2024-05-10T11:32:34.687 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_delete": 0, 2024-05-10T11:32:34.687 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_mapext": 0, 2024-05-10T11:32:34.687 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_sparse_read": 0, 2024-05-10T11:32:34.687 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_clonerange": 0, 2024-05-10T11:32:34.687 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_getxattr": 1, 2024-05-10T11:32:34.687 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_setxattr": 0, 2024-05-10T11:32:34.687 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_cmpxattr": 0, 2024-05-10T11:32:34.688 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_rmxattr": 0, 2024-05-10T11:32:34.688 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_resetxattrs": 0, 2024-05-10T11:32:34.688 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_call": 0, 2024-05-10T11:32:34.688 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_watch": 0, 2024-05-10T11:32:34.688 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_notify": 0, 2024-05-10T11:32:34.688 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_src_cmpxattr": 0, 2024-05-10T11:32:34.688 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls": 0, 2024-05-10T11:32:34.688 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls_filter": 0, 2024-05-10T11:32:34.689 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_other": 0, 2024-05-10T11:32:34.689 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_active": 0, 2024-05-10T11:32:34.689 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_send": 0, 2024-05-10T11:32:34.689 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_resend": 0, 2024-05-10T11:32:34.689 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_ping": 0, 2024-05-10T11:32:34.689 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_active": 0, 2024-05-10T11:32:34.689 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_send": 0, 2024-05-10T11:32:34.690 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_resend": 0, 2024-05-10T11:32:34.690 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_active": 0, 2024-05-10T11:32:34.690 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_send": 0, 2024-05-10T11:32:34.690 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_resend": 0, 2024-05-10T11:32:34.690 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_active": 0, 2024-05-10T11:32:34.690 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_send": 0, 2024-05-10T11:32:34.690 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_resend": 0, 2024-05-10T11:32:34.690 INFO:teuthology.orchestra.run.smithi027.stdout: "command_active": 0, 2024-05-10T11:32:34.691 INFO:teuthology.orchestra.run.smithi027.stdout: "command_send": 0, 2024-05-10T11:32:34.691 INFO:teuthology.orchestra.run.smithi027.stdout: "command_resend": 0, 2024-05-10T11:32:34.691 INFO:teuthology.orchestra.run.smithi027.stdout: "map_epoch": 117, 2024-05-10T11:32:34.691 INFO:teuthology.orchestra.run.smithi027.stdout: "map_full": 0, 2024-05-10T11:32:34.691 INFO:teuthology.orchestra.run.smithi027.stdout: "map_inc": 29, 2024-05-10T11:32:34.691 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_sessions": 11, 2024-05-10T11:32:34.691 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_open": 12, 2024-05-10T11:32:34.692 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_close": 1, 2024-05-10T11:32:34.692 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_laggy": 0, 2024-05-10T11:32:34.692 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_wr": 0, 2024-05-10T11:32:34.692 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_rd": 2, 2024-05-10T11:32:34.692 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_del": 0 2024-05-10T11:32:34.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.692 INFO:teuthology.orchestra.run.smithi027.stdout: "oft": { 2024-05-10T11:32:34.692 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_objs": 0, 2024-05-10T11:32:34.693 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_kv_pairs": 0, 2024-05-10T11:32:34.693 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_updates": 0, 2024-05-10T11:32:34.693 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_removes": 0 2024-05-10T11:32:34.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.693 INFO:teuthology.orchestra.run.smithi027.stdout: "osd-slow-ops": { 2024-05-10T11:32:34.693 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_ops_count": 0 2024-05-10T11:32:34.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.693 INFO:teuthology.orchestra.run.smithi027.stdout: "purge_queue": { 2024-05-10T11:32:34.694 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops": 0, 2024-05-10T11:32:34.694 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops_high_water": 0, 2024-05-10T11:32:34.694 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing": 0, 2024-05-10T11:32:34.694 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_high_water": 0, 2024-05-10T11:32:34.694 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executed_ops": 0, 2024-05-10T11:32:34.694 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executed": 0, 2024-05-10T11:32:34.694 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_item_in_journal": 0 2024-05-10T11:32:34.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.695 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-10T11:32:34.695 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-05-10T11:32:34.695 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-05-10T11:32:34.695 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-05-10T11:32:34.695 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 4853, 2024-05-10T11:32:34.695 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 191883702, 2024-05-10T11:32:34.696 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:34.696 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 4853, 2024-05-10T11:32:34.696 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-05-10T11:32:34.696 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-05-10T11:32:34.696 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 4853, 2024-05-10T11:32:34.696 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 191883702, 2024-05-10T11:32:34.696 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-05-10T11:32:34.696 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.697 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.697 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.697 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:34.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.697 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_bytes": { 2024-05-10T11:32:34.697 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-05-10T11:32:34.697 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-05-10T11:32:34.697 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-05-10T11:32:34.698 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-05-10T11:32:34.698 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-05-10T11:32:34.698 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:34.698 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:34.698 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 4378, 2024-05-10T11:32:34.698 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 189039781, 2024-05-10T11:32:34.698 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 4378, 2024-05-10T11:32:34.699 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 189039781, 2024-05-10T11:32:34.699 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-05-10T11:32:34.699 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.699 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.699 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.699 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:34.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.699 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_ops": { 2024-05-10T11:32:34.700 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-05-10T11:32:34.700 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1024, 2024-05-10T11:32:34.700 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-05-10T11:32:34.700 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-05-10T11:32:34.700 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-05-10T11:32:34.700 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:34.700 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:34.700 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 4378, 2024-05-10T11:32:34.701 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 4378, 2024-05-10T11:32:34.701 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 4378, 2024-05-10T11:32:34.701 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 4378, 2024-05-10T11:32:34.701 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-05-10T11:32:34.701 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.701 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.701 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.702 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:34.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.702 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle": { 2024-05-10T11:32:34.702 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-05-10T11:32:34.702 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-05-10T11:32:34.702 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-05-10T11:32:34.702 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-05-10T11:32:34.702 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-05-10T11:32:34.703 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:34.703 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:34.703 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-05-10T11:32:34.703 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-05-10T11:32:34.703 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-05-10T11:32:34.703 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-05-10T11:32:34.703 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-05-10T11:32:34.704 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.704 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.704 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.704 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:34.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-10T11:32:34.704 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle-0x55b8232f8780": { 2024-05-10T11:32:34.704 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-05-10T11:32:34.704 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-05-10T11:32:34.705 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-05-10T11:32:34.705 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-05-10T11:32:34.705 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-05-10T11:32:34.705 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:34.705 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:34.705 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-05-10T11:32:34.705 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-05-10T11:32:34.705 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-05-10T11:32:34.706 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-05-10T11:32:34.706 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-05-10T11:32:34.706 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-05-10T11:32:34.706 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-05-10T11:32:34.706 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-05-10T11:32:34.706 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:34.706 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-10T11:32:34.707 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-10T11:32:35.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:34 smithi027 ceph-mon[30054]: pgmap v609: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 3.4 MiB/s wr, 127 op/s 2024-05-10T11:32:35.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:34 smithi076 ceph-mon[34152]: pgmap v609: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 3.4 MiB/s wr, 127 op/s 2024-05-10T11:32:35.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:34 smithi189 ceph-mon[34842]: pgmap v609: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 3.4 MiB/s wr, 127 op/s 2024-05-10T11:32:35.577 INFO:tasks.check_counter:Daemon mds.j mds.exported=0 2024-05-10T11:32:35.577 INFO:tasks.check_counter:Daemon mds.j mds.imported=0 2024-05-10T11:32:35.577 INFO:tasks.check_counter:Daemon mds.j mds.root_rsnaps=0 2024-05-10T11:32:35.578 INFO:tasks.check_counter:Daemon mds.j mds_server.req_mksnap_latency.avgcount=0 2024-05-10T11:32:35.578 INFO:tasks.check_counter:Daemon mds.j mds_server.req_rmsnap_latency.avgcount=0 2024-05-10T11:32:35.578 DEBUG:tasks.check_counter:Getting stats from b 2024-05-10T11:32:35.578 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph daemon mds.b perf dump 2024-05-10T11:32:35.918 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.b/config 2024-05-10T11:32:37.101 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:36 smithi076 ceph-mon[34152]: pgmap v610: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 4.3 MiB/s wr, 161 op/s 2024-05-10T11:32:37.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:36 smithi027 ceph-mon[30054]: pgmap v610: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 4.3 MiB/s wr, 161 op/s 2024-05-10T11:32:37.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:36 smithi189 ceph-mon[34842]: pgmap v610: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 4.3 MiB/s wr, 161 op/s 2024-05-10T11:32:38.475 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-05-10T11:32:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-0": { 2024-05-10T11:32:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 649137, 2024-05-10T11:32:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 1007548, 2024-05-10T11:32:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 184982233, 2024-05-10T11:32:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 1233233707, 2024-05-10T11:32:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 18, 2024-05-10T11:32:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 3, 2024-05-10T11:32:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 38.244307826, 2024-05-10T11:32:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 17.630869040, 2024-05-10T11:32:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 22.822477473, 2024-05-10T11:32:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 2.709722111, 2024-05-10T11:32:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 1007546, 2024-05-10T11:32:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 39.423413743, 2024-05-10T11:32:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000039128 2024-05-10T11:32:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 924799, 2024-05-10T11:32:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.237009848, 2024-05-10T11:32:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000256 2024-05-10T11:32:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 99520, 2024-05-10T11:32:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 33184 2024-05-10T11:32:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-1": { 2024-05-10T11:32:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 233864, 2024-05-10T11:32:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 311292, 2024-05-10T11:32:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 56554462, 2024-05-10T11:32:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 714120746, 2024-05-10T11:32:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 14, 2024-05-10T11:32:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 7, 2024-05-10T11:32:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 17.363416443, 2024-05-10T11:32:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 6.549174560, 2024-05-10T11:32:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 8.401246106, 2024-05-10T11:32:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 3.015921827, 2024-05-10T11:32:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 311291, 2024-05-10T11:32:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 10.744111851, 2024-05-10T11:32:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000034514 2024-05-10T11:32:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 394954, 2024-05-10T11:32:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.076344594, 2024-05-10T11:32:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000193 2024-05-10T11:32:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 2718832, 2024-05-10T11:32:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 479728 2024-05-10T11:32:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-2": { 2024-05-10T11:32:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 311762, 2024-05-10T11:32:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 527643, 2024-05-10T11:32:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 60542150, 2024-05-10T11:32:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 620810632, 2024-05-10T11:32:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 144, 2024-05-10T11:32:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 5, 2024-05-10T11:32:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 27.976816500, 2024-05-10T11:32:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 14.347830609, 2024-05-10T11:32:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 12.718292120, 2024-05-10T11:32:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 2.469817375, 2024-05-10T11:32:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 527641, 2024-05-10T11:32:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 14.620191623, 2024-05-10T11:32:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000027708 2024-05-10T11:32:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 742286, 2024-05-10T11:32:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.125544111, 2024-05-10T11:32:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000169 2024-05-10T11:32:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 45536, 2024-05-10T11:32:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 794336 2024-05-10T11:32:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: "cct": { 2024-05-10T11:32:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: "total_workers": 1, 2024-05-10T11:32:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: "unhealthy_workers": 0 2024-05-10T11:32:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: "finisher-MDSRank": { 2024-05-10T11:32:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: "queue_len": 0, 2024-05-10T11:32:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: "complete_latency": { 2024-05-10T11:32:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 140328, 2024-05-10T11:32:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 217.856425090, 2024-05-10T11:32:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.001552480 2024-05-10T11:32:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "finisher-PurgeQueue": { 2024-05-10T11:32:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "queue_len": 0, 2024-05-10T11:32:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "complete_latency": { 2024-05-10T11:32:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 84472, 2024-05-10T11:32:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 21.792029907, 2024-05-10T11:32:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000257979 2024-05-10T11:32:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "mds": { 2024-05-10T11:32:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "request": 306054, 2024-05-10T11:32:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "reply": 296365, 2024-05-10T11:32:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "reply_latency": { 2024-05-10T11:32:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 296365, 2024-05-10T11:32:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 2205.539040630, 2024-05-10T11:32:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.007441968 2024-05-10T11:32:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "slow_reply": 0, 2024-05-10T11:32:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "forward": 9689, 2024-05-10T11:32:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_fetch_complete": 5, 2024-05-10T11:32:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_fetch_keys": 2, 2024-05-10T11:32:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_commit": 418, 2024-05-10T11:32:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_split": 27, 2024-05-10T11:32:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_merge": 28, 2024-05-10T11:32:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes": 5812, 2024-05-10T11:32:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_top": 2416, 2024-05-10T11:32:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_bottom": 3377, 2024-05-10T11:32:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_pin_tail": 19, 2024-05-10T11:32:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_pinned": 2360, 2024-05-10T11:32:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_expired": 391, 2024-05-10T11:32:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_with_caps": 596, 2024-05-10T11:32:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "caps": 596, 2024-05-10T11:32:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "subtrees": 36, 2024-05-10T11:32:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse": 395194, 2024-05-10T11:32:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_hit": 289012, 2024-05-10T11:32:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_forward": 9674, 2024-05-10T11:32:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_discover": 5, 2024-05-10T11:32:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_dir_fetch": 0, 2024-05-10T11:32:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_remote_ino": 0, 2024-05-10T11:32:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_lock": 1711, 2024-05-10T11:32:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "load_cent": 0, 2024-05-10T11:32:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "q": 0, 2024-05-10T11:32:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "exported": 17, 2024-05-10T11:32:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "exported_inodes": 285, 2024-05-10T11:32:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "imported": 2, 2024-05-10T11:32:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "imported_inodes": 96, 2024-05-10T11:32:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "openino_dir_fetch": 0, 2024-05-10T11:32:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "openino_backtrace_fetch": 3476, 2024-05-10T11:32:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "openino_peer_discover": 0, 2024-05-10T11:32:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "root_rfiles": 699, 2024-05-10T11:32:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "root_rbytes": 217275323, 2024-05-10T11:32:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "root_rsnaps": 6, 2024-05-10T11:32:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_backtrace_fetch": 3476, 2024-05-10T11:32:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_set_tag": 0, 2024-05-10T11:32:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_backtrace_repaired": 0, 2024-05-10T11:32:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_inotable_repaired": 0, 2024-05-10T11:32:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_dir_inodes": 677, 2024-05-10T11:32:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_dir_base_inodes": 17, 2024-05-10T11:32:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_dirfrag_rstats": 660, 2024-05-10T11:32:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_file_inodes": 2799, 2024-05-10T11:32:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_inode_file_caps": 17, 2024-05-10T11:32:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_revoke": 52814, 2024-05-10T11:32:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_grant": 212235, 2024-05-10T11:32:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_trunc": 0, 2024-05-10T11:32:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-10T11:32:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-10T11:32:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_caps": 123186, 2024-05-10T11:32:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_caps_dirty": 44905, 2024-05-10T11:32:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_cap_release": 19531, 2024-05-10T11:32:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "process_request_cap_release": 189889 2024-05-10T11:32:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache": { 2024-05-10T11:32:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_update": 24, 2024-05-10T11:32:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_update_receipt": 6, 2024-05-10T11:32:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_try_discover": 4, 2024-05-10T11:32:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_send_discover": 7, 2024-05-10T11:32:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_handle_discover": 259, 2024-05-10T11:32:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "num_strays": 0, 2024-05-10T11:32:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "num_strays_delayed": 0, 2024-05-10T11:32:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "num_strays_enqueuing": 0, 2024-05-10T11:32:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_created": 44245, 2024-05-10T11:32:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_enqueued": 44245, 2024-05-10T11:32:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_reintegrated": 0, 2024-05-10T11:32:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_migrated": 0, 2024-05-10T11:32:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "num_recovering_processing": 0, 2024-05-10T11:32:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "num_recovering_enqueued": 0, 2024-05-10T11:32:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "num_recovering_prioritized": 0, 2024-05-10T11:32:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_started": 0, 2024-05-10T11:32:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_completed": 0, 2024-05-10T11:32:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_quiesce_path": 0, 2024-05-10T11:32:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_quiesce_inode": 0, 2024-05-10T11:32:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_enqueue_scrub": 17, 2024-05-10T11:32:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_exportdir": 18, 2024-05-10T11:32:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_flush": 0, 2024-05-10T11:32:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_fragmentdir": 55, 2024-05-10T11:32:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_fragstats": 0, 2024-05-10T11:32:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_inodestats": 0 2024-05-10T11:32:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log": { 2024-05-10T11:32:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "evlrg": 0, 2024-05-10T11:32:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "evadd": 238674, 2024-05-10T11:32:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "evex": 165613, 2024-05-10T11:32:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "evtrm": 159771, 2024-05-10T11:32:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "ev": 78903, 2024-05-10T11:32:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "evexg": 0, 2024-05-10T11:32:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "evexd": 5842, 2024-05-10T11:32:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "segadd": 391, 2024-05-10T11:32:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "segex": 263, 2024-05-10T11:32:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "segtrm": 253, 2024-05-10T11:32:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "seg": 138, 2024-05-10T11:32:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "segmjr": 7, 2024-05-10T11:32:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "segexg": 0, 2024-05-10T11:32:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "segexd": 10, 2024-05-10T11:32:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "expos": 893254611, 2024-05-10T11:32:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "wrpos": 1439076752, 2024-05-10T11:32:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "rdpos": 0, 2024-05-10T11:32:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "jlat": { 2024-05-10T11:32:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 79852, 2024-05-10T11:32:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 632.784388141, 2024-05-10T11:32:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.007924465 2024-05-10T11:32:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: "replayed": 0 2024-05-10T11:32:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_mem": { 2024-05-10T11:32:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 2558, 2024-05-10T11:32:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: "ino+": 47012, 2024-05-10T11:32:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: "ino-": 44454, 2024-05-10T11:32:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "dir": 215, 2024-05-10T11:32:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "dir+": 404, 2024-05-10T11:32:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "dir-": 189, 2024-05-10T11:32:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "dn": 5812, 2024-05-10T11:32:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "dn+": 85036, 2024-05-10T11:32:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "dn-": 79224, 2024-05-10T11:32:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "cap": 596, 2024-05-10T11:32:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "cap+": 46296, 2024-05-10T11:32:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "cap-": 45700, 2024-05-10T11:32:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "rss": 99396, 2024-05-10T11:32:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "heap": 231800 2024-05-10T11:32:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_server": { 2024-05-10T11:32:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "dispatch_client_request": 452589, 2024-05-10T11:32:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "dispatch_server_request": 34246, 2024-05-10T11:32:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_request": 306054, 2024-05-10T11:32:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_session": 7771, 2024-05-10T11:32:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_peer_request": 68482, 2024-05-10T11:32:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "req_create_latency": { 2024-05-10T11:32:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 44890, 2024-05-10T11:32:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 251.848369811, 2024-05-10T11:32:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.005610344 2024-05-10T11:32:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "req_getattr_latency": { 2024-05-10T11:32:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 19079, 2024-05-10T11:32:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 410.932680559, 2024-05-10T11:32:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.021538481 2024-05-10T11:32:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: "req_getfilelock_latency": { 2024-05-10T11:32:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: "req_link_latency": { 2024-05-10T11:32:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookup_latency": { 2024-05-10T11:32:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 54526, 2024-05-10T11:32:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 268.855534154, 2024-05-10T11:32:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.004930776 2024-05-10T11:32:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookuphash_latency": { 2024-05-10T11:32:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupino_latency": { 2024-05-10T11:32:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupname_latency": { 2024-05-10T11:32:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupparent_latency": { 2024-05-10T11:32:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupsnap_latency": { 2024-05-10T11:32:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lssnap_latency": { 2024-05-10T11:32:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 19, 2024-05-10T11:32:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.014276514, 2024-05-10T11:32:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000751395 2024-05-10T11:32:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "req_mkdir_latency": { 2024-05-10T11:32:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 114, 2024-05-10T11:32:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.721595435, 2024-05-10T11:32:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.006329784 2024-05-10T11:32:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: "req_mknod_latency": { 2024-05-10T11:32:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: "req_mksnap_latency": { 2024-05-10T11:32:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 19, 2024-05-10T11:32:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.341791310, 2024-05-10T11:32:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.017989016 2024-05-10T11:32:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "req_open_latency": { 2024-05-10T11:32:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 10, 2024-05-10T11:32:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.016382853, 2024-05-10T11:32:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.001638285 2024-05-10T11:32:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "req_readdir_latency": { 2024-05-10T11:32:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 119215, 2024-05-10T11:32:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 911.411737847, 2024-05-10T11:32:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.007645109 2024-05-10T11:32:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rename_latency": { 2024-05-10T11:32:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 12218, 2024-05-10T11:32:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 65.743371830, 2024-05-10T11:32:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.005380861 2024-05-10T11:32:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: "req_renamesnap_latency": { 2024-05-10T11:32:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: "req_snapdiff_latency": { 2024-05-10T11:32:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rmdir_latency": { 2024-05-10T11:32:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 1, 2024-05-10T11:32:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000925194, 2024-05-10T11:32:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000925194 2024-05-10T11:32:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rmsnap_latency": { 2024-05-10T11:32:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 13, 2024-05-10T11:32:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 3.309106468, 2024-05-10T11:32:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.254546651 2024-05-10T11:32:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rmxattr_latency": { 2024-05-10T11:32:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setattr_latency": { 2024-05-10T11:32:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 143, 2024-05-10T11:32:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 4.633096212, 2024-05-10T11:32:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.032399274 2024-05-10T11:32:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setdirlayout_latency": { 2024-05-10T11:32:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setfilelock_latency": { 2024-05-10T11:32:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 1864, 2024-05-10T11:32:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 7.258610036, 2024-05-10T11:32:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.003894104 2024-05-10T11:32:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setlayout_latency": { 2024-05-10T11:32:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setxattr_latency": { 2024-05-10T11:32:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 9, 2024-05-10T11:32:38.522 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.024776701, 2024-05-10T11:32:38.522 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.002752966 2024-05-10T11:32:38.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.522 INFO:teuthology.orchestra.run.smithi076.stdout: "req_symlink_latency": { 2024-05-10T11:32:38.522 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.522 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.522 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.525 INFO:teuthology.orchestra.run.smithi076.stdout: "req_unlink_latency": { 2024-05-10T11:32:38.525 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 44245, 2024-05-10T11:32:38.526 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 280.426785706, 2024-05-10T11:32:38.526 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.006338044 2024-05-10T11:32:38.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.526 INFO:teuthology.orchestra.run.smithi076.stdout: "cap_revoke_eviction": 0, 2024-05-10T11:32:38.526 INFO:teuthology.orchestra.run.smithi076.stdout: "cap_acquisition_throttle": 0, 2024-05-10T11:32:38.526 INFO:teuthology.orchestra.run.smithi076.stdout: "req_getvxattr_latency": { 2024-05-10T11:32:38.526 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.527 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.527 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.527 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:38.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.527 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_sessions": { 2024-05-10T11:32:38.527 INFO:teuthology.orchestra.run.smithi076.stdout: "session_count": 2, 2024-05-10T11:32:38.527 INFO:teuthology.orchestra.run.smithi076.stdout: "session_add": 6, 2024-05-10T11:32:38.527 INFO:teuthology.orchestra.run.smithi076.stdout: "session_remove": 4, 2024-05-10T11:32:38.528 INFO:teuthology.orchestra.run.smithi076.stdout: "sessions_open": 2, 2024-05-10T11:32:38.528 INFO:teuthology.orchestra.run.smithi076.stdout: "sessions_stale": 0, 2024-05-10T11:32:38.528 INFO:teuthology.orchestra.run.smithi076.stdout: "total_load": 41510, 2024-05-10T11:32:38.528 INFO:teuthology.orchestra.run.smithi076.stdout: "average_load": 20755, 2024-05-10T11:32:38.528 INFO:teuthology.orchestra.run.smithi076.stdout: "avg_session_uptime": 1141, 2024-05-10T11:32:38.529 INFO:teuthology.orchestra.run.smithi076.stdout: "mdthresh_evicted": 0 2024-05-10T11:32:38.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.529 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool": { 2024-05-10T11:32:38.529 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_bytes": 0, 2024-05-10T11:32:38.529 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_items": 0, 2024-05-10T11:32:38.529 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_bytes": 0, 2024-05-10T11:32:38.529 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_items": 0, 2024-05-10T11:32:38.529 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_bytes": 0, 2024-05-10T11:32:38.530 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_items": 0, 2024-05-10T11:32:38.530 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-10T11:32:38.530 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_items": 0, 2024-05-10T11:32:38.530 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-10T11:32:38.530 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_items": 0, 2024-05-10T11:32:38.530 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_bytes": 0, 2024-05-10T11:32:38.530 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_items": 0, 2024-05-10T11:32:38.531 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-10T11:32:38.531 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_items": 0, 2024-05-10T11:32:38.531 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_bytes": 0, 2024-05-10T11:32:38.531 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_items": 0, 2024-05-10T11:32:38.531 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_bytes": 0, 2024-05-10T11:32:38.531 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_items": 0, 2024-05-10T11:32:38.531 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-10T11:32:38.531 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_items": 0, 2024-05-10T11:32:38.532 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-10T11:32:38.532 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_items": 0, 2024-05-10T11:32:38.532 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_bytes": 0, 2024-05-10T11:32:38.532 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_items": 0, 2024-05-10T11:32:38.532 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_bytes": 0, 2024-05-10T11:32:38.532 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_items": 0, 2024-05-10T11:32:38.532 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-10T11:32:38.532 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_items": 0, 2024-05-10T11:32:38.533 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_bytes": 0, 2024-05-10T11:32:38.533 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_items": 0, 2024-05-10T11:32:38.533 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_bytes": 0, 2024-05-10T11:32:38.533 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_items": 0, 2024-05-10T11:32:38.533 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_bytes": 0, 2024-05-10T11:32:38.533 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_items": 0, 2024-05-10T11:32:38.533 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_bytes": 0, 2024-05-10T11:32:38.533 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_items": 0, 2024-05-10T11:32:38.534 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_bytes": 374430, 2024-05-10T11:32:38.534 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_items": 1369, 2024-05-10T11:32:38.534 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_bytes": 0, 2024-05-10T11:32:38.534 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_items": 0, 2024-05-10T11:32:38.534 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bytes": 0, 2024-05-10T11:32:38.534 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_items": 0, 2024-05-10T11:32:38.534 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_bytes": 0, 2024-05-10T11:32:38.535 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_items": 0, 2024-05-10T11:32:38.535 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_bytes": 0, 2024-05-10T11:32:38.535 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_items": 0, 2024-05-10T11:32:38.535 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_bytes": 9808, 2024-05-10T11:32:38.535 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_items": 189, 2024-05-10T11:32:38.535 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_bytes": 0, 2024-05-10T11:32:38.535 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_items": 0, 2024-05-10T11:32:38.535 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_bytes": 0, 2024-05-10T11:32:38.536 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_items": 0, 2024-05-10T11:32:38.536 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_bytes": 11571083, 2024-05-10T11:32:38.536 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_items": 113818, 2024-05-10T11:32:38.536 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_bytes": 0, 2024-05-10T11:32:38.536 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_items": 0, 2024-05-10T11:32:38.536 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_bytes": 0, 2024-05-10T11:32:38.536 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_items": 0 2024-05-10T11:32:38.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.537 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter": { 2024-05-10T11:32:38.537 INFO:teuthology.orchestra.run.smithi076.stdout: "op_active": 0, 2024-05-10T11:32:38.537 INFO:teuthology.orchestra.run.smithi076.stdout: "op_laggy": 0, 2024-05-10T11:32:38.537 INFO:teuthology.orchestra.run.smithi076.stdout: "op_send": 178513, 2024-05-10T11:32:38.537 INFO:teuthology.orchestra.run.smithi076.stdout: "op_send_bytes": 1449131878, 2024-05-10T11:32:38.537 INFO:teuthology.orchestra.run.smithi076.stdout: "op_resend": 37, 2024-05-10T11:32:38.537 INFO:teuthology.orchestra.run.smithi076.stdout: "op_reply": 178513, 2024-05-10T11:32:38.538 INFO:teuthology.orchestra.run.smithi076.stdout: "op_latency": { 2024-05-10T11:32:38.538 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 178513, 2024-05-10T11:32:38.538 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 1398.589558279, 2024-05-10T11:32:38.538 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.007834665 2024-05-10T11:32:38.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.538 INFO:teuthology.orchestra.run.smithi076.stdout: "op_inflight": 0, 2024-05-10T11:32:38.538 INFO:teuthology.orchestra.run.smithi076.stdout: "oplen_avg": { 2024-05-10T11:32:38.539 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 178513, 2024-05-10T11:32:38.539 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 182352 2024-05-10T11:32:38.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.539 INFO:teuthology.orchestra.run.smithi076.stdout: "op": 178513, 2024-05-10T11:32:38.539 INFO:teuthology.orchestra.run.smithi076.stdout: "op_r": 4683, 2024-05-10T11:32:38.539 INFO:teuthology.orchestra.run.smithi076.stdout: "op_w": 173830, 2024-05-10T11:32:38.539 INFO:teuthology.orchestra.run.smithi076.stdout: "op_rmw": 0, 2024-05-10T11:32:38.539 INFO:teuthology.orchestra.run.smithi076.stdout: "op_pg": 0, 2024-05-10T11:32:38.539 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_stat": 67, 2024-05-10T11:32:38.540 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_create": 995, 2024-05-10T11:32:38.540 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_read": 1183, 2024-05-10T11:32:38.540 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_write": 81018, 2024-05-10T11:32:38.540 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_writefull": 644, 2024-05-10T11:32:38.540 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_writesame": 0, 2024-05-10T11:32:38.540 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_append": 0, 2024-05-10T11:32:38.540 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_zero": 0, 2024-05-10T11:32:38.541 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_truncate": 27, 2024-05-10T11:32:38.541 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_delete": 89708, 2024-05-10T11:32:38.541 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_mapext": 0, 2024-05-10T11:32:38.541 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_sparse_read": 0, 2024-05-10T11:32:38.541 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_clonerange": 0, 2024-05-10T11:32:38.541 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_getxattr": 3500, 2024-05-10T11:32:38.541 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_setxattr": 1559, 2024-05-10T11:32:38.541 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_cmpxattr": 0, 2024-05-10T11:32:38.542 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_rmxattr": 0, 2024-05-10T11:32:38.542 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_resetxattrs": 0, 2024-05-10T11:32:38.542 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_call": 0, 2024-05-10T11:32:38.542 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_watch": 0, 2024-05-10T11:32:38.542 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_notify": 0, 2024-05-10T11:32:38.542 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_src_cmpxattr": 0, 2024-05-10T11:32:38.542 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_pgls": 0, 2024-05-10T11:32:38.542 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_pgls_filter": 0, 2024-05-10T11:32:38.543 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_other": 0, 2024-05-10T11:32:38.543 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_active": 0, 2024-05-10T11:32:38.543 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_send": 0, 2024-05-10T11:32:38.543 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_resend": 0, 2024-05-10T11:32:38.543 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_ping": 0, 2024-05-10T11:32:38.543 INFO:teuthology.orchestra.run.smithi076.stdout: "poolop_active": 0, 2024-05-10T11:32:38.543 INFO:teuthology.orchestra.run.smithi076.stdout: "poolop_send": 0, 2024-05-10T11:32:38.543 INFO:teuthology.orchestra.run.smithi076.stdout: "poolop_resend": 0, 2024-05-10T11:32:38.544 INFO:teuthology.orchestra.run.smithi076.stdout: "poolstat_active": 0, 2024-05-10T11:32:38.544 INFO:teuthology.orchestra.run.smithi076.stdout: "poolstat_send": 0, 2024-05-10T11:32:38.544 INFO:teuthology.orchestra.run.smithi076.stdout: "poolstat_resend": 0, 2024-05-10T11:32:38.544 INFO:teuthology.orchestra.run.smithi076.stdout: "statfs_active": 0, 2024-05-10T11:32:38.544 INFO:teuthology.orchestra.run.smithi076.stdout: "statfs_send": 0, 2024-05-10T11:32:38.544 INFO:teuthology.orchestra.run.smithi076.stdout: "statfs_resend": 0, 2024-05-10T11:32:38.544 INFO:teuthology.orchestra.run.smithi076.stdout: "command_active": 0, 2024-05-10T11:32:38.544 INFO:teuthology.orchestra.run.smithi076.stdout: "command_send": 0, 2024-05-10T11:32:38.545 INFO:teuthology.orchestra.run.smithi076.stdout: "command_resend": 0, 2024-05-10T11:32:38.545 INFO:teuthology.orchestra.run.smithi076.stdout: "map_epoch": 117, 2024-05-10T11:32:38.545 INFO:teuthology.orchestra.run.smithi076.stdout: "map_full": 0, 2024-05-10T11:32:38.545 INFO:teuthology.orchestra.run.smithi076.stdout: "map_inc": 38, 2024-05-10T11:32:38.545 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_sessions": 12, 2024-05-10T11:32:38.545 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_session_open": 12, 2024-05-10T11:32:38.545 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_session_close": 0, 2024-05-10T11:32:38.545 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_laggy": 0, 2024-05-10T11:32:38.546 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_wr": 2806, 2024-05-10T11:32:38.546 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_rd": 14, 2024-05-10T11:32:38.546 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_del": 831 2024-05-10T11:32:38.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.546 INFO:teuthology.orchestra.run.smithi076.stdout: "oft": { 2024-05-10T11:32:38.546 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_objs": 1, 2024-05-10T11:32:38.546 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_kv_pairs": 0, 2024-05-10T11:32:38.547 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_updates": 0, 2024-05-10T11:32:38.547 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_removes": 0 2024-05-10T11:32:38.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.547 INFO:teuthology.orchestra.run.smithi076.stdout: "osd-slow-ops": { 2024-05-10T11:32:38.547 INFO:teuthology.orchestra.run.smithi076.stdout: "slow_ops_count": 1 2024-05-10T11:32:38.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.547 INFO:teuthology.orchestra.run.smithi076.stdout: "purge_queue": { 2024-05-10T11:32:38.547 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing_ops": 0, 2024-05-10T11:32:38.548 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing_ops_high_water": 10, 2024-05-10T11:32:38.548 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing": 0, 2024-05-10T11:32:38.548 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing_high_water": 5, 2024-05-10T11:32:38.548 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executed_ops": 88490, 2024-05-10T11:32:38.548 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executed": 44245, 2024-05-10T11:32:38.548 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_item_in_journal": 0 2024-05-10T11:32:38.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.548 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-10T11:32:38.549 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:38.549 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-05-10T11:32:38.549 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:38.549 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 1194759, 2024-05-10T11:32:38.549 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 201768263, 2024-05-10T11:32:38.549 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:38.549 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 1194759, 2024-05-10T11:32:38.549 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-05-10T11:32:38.550 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-05-10T11:32:38.550 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 1194759, 2024-05-10T11:32:38.550 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 201768263, 2024-05-10T11:32:38.550 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:38.550 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.550 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.550 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.551 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:38.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.551 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-objecter_bytes": { 2024-05-10T11:32:38.551 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:38.551 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-05-10T11:32:38.551 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:38.551 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 0, 2024-05-10T11:32:38.551 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 0, 2024-05-10T11:32:38.552 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:38.552 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:38.552 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 178513, 2024-05-10T11:32:38.552 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 1453963319, 2024-05-10T11:32:38.552 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 178513, 2024-05-10T11:32:38.552 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 1453963319, 2024-05-10T11:32:38.552 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:38.552 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.552 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.553 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.553 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:38.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.553 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-objecter_ops": { 2024-05-10T11:32:38.553 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:38.553 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 1024, 2024-05-10T11:32:38.553 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:38.553 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 0, 2024-05-10T11:32:38.554 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 0, 2024-05-10T11:32:38.554 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:38.554 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:38.554 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 178513, 2024-05-10T11:32:38.554 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 178513, 2024-05-10T11:32:38.554 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 178513, 2024-05-10T11:32:38.554 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 178513, 2024-05-10T11:32:38.554 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:38.555 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.555 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.555 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.555 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:38.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.555 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-write_buf_throttle": { 2024-05-10T11:32:38.555 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:38.555 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 3758096384, 2024-05-10T11:32:38.556 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:38.556 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 44245, 2024-05-10T11:32:38.556 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 4831529, 2024-05-10T11:32:38.556 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:38.556 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 44245, 2024-05-10T11:32:38.556 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-05-10T11:32:38.556 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-05-10T11:32:38.556 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 1166, 2024-05-10T11:32:38.556 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 4831529, 2024-05-10T11:32:38.557 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:38.557 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.557 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.557 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.557 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:38.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:38.557 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-write_buf_throttle-0x55ae32046500": { 2024-05-10T11:32:38.557 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:38.558 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 3758096384, 2024-05-10T11:32:38.558 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:38.558 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 238674, 2024-05-10T11:32:38.558 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 1434882448, 2024-05-10T11:32:38.558 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:38.558 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 238674, 2024-05-10T11:32:38.558 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-05-10T11:32:38.559 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-05-10T11:32:38.559 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 79852, 2024-05-10T11:32:38.559 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 1434882448, 2024-05-10T11:32:38.559 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:38.559 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:38.559 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:38.559 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:38.559 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:38.559 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:38.560 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-05-10T11:32:39.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:38 smithi027 ceph-mon[30054]: pgmap v611: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 2.2 MiB/s wr, 85 op/s 2024-05-10T11:32:39.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:38 smithi076 ceph-mon[34152]: pgmap v611: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 2.2 MiB/s wr, 85 op/s 2024-05-10T11:32:39.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:38 smithi189 ceph-mon[34842]: pgmap v611: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 2.2 MiB/s wr, 85 op/s 2024-05-10T11:32:39.429 INFO:tasks.check_counter:Daemon mds.b mds.exported=17 2024-05-10T11:32:39.429 INFO:tasks.check_counter:Daemon mds.b mds.imported=2 2024-05-10T11:32:39.430 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-05-10T11:32:39.430 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=19 2024-05-10T11:32:39.430 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=13 2024-05-10T11:32:39.430 DEBUG:tasks.check_counter:Getting stats from e 2024-05-10T11:32:39.430 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph daemon mds.e perf dump 2024-05-10T11:32:39.701 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.b/config 2024-05-10T11:32:41.094 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:40 smithi076 ceph-mon[34152]: pgmap v612: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 2.2 MiB/s wr, 96 op/s 2024-05-10T11:32:41.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:40 smithi027 ceph-mon[30054]: pgmap v612: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 2.2 MiB/s wr, 96 op/s 2024-05-10T11:32:41.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:40 smithi189 ceph-mon[34842]: pgmap v612: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 2.2 MiB/s wr, 96 op/s 2024-05-10T11:32:41.606 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-05-10T11:32:41.606 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-0": { 2024-05-10T11:32:41.607 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 14331, 2024-05-10T11:32:41.607 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 23159, 2024-05-10T11:32:41.607 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 4657625, 2024-05-10T11:32:41.607 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 20651640, 2024-05-10T11:32:41.607 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 30, 2024-05-10T11:32:41.607 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 10, 2024-05-10T11:32:41.608 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 1.000451106, 2024-05-10T11:32:41.608 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.449651589, 2024-05-10T11:32:41.608 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.517573429, 2024-05-10T11:32:41.608 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.064203880, 2024-05-10T11:32:41.608 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:41.608 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 23153, 2024-05-10T11:32:41.608 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 1.091426711, 2024-05-10T11:32:41.608 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000047139 2024-05-10T11:32:41.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.609 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:41.609 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 18498, 2024-05-10T11:32:41.609 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.006346870, 2024-05-10T11:32:41.609 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000343 2024-05-10T11:32:41.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.609 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 3920, 2024-05-10T11:32:41.609 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 1424 2024-05-10T11:32:41.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.609 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-1": { 2024-05-10T11:32:41.610 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 251853, 2024-05-10T11:32:41.610 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 134834, 2024-05-10T11:32:41.610 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 126454798, 2024-05-10T11:32:41.610 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 25226677, 2024-05-10T11:32:41.610 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 32, 2024-05-10T11:32:41.610 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 6, 2024-05-10T11:32:41.610 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 12.029023624, 2024-05-10T11:32:41.610 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 5.268738388, 2024-05-10T11:32:41.611 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 9.109555092, 2024-05-10T11:32:41.611 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.050247666, 2024-05-10T11:32:41.611 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:41.611 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 134836, 2024-05-10T11:32:41.611 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 2.702789451, 2024-05-10T11:32:41.611 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000020045 2024-05-10T11:32:41.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.611 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:41.612 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 382286, 2024-05-10T11:32:41.612 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.053833077, 2024-05-10T11:32:41.612 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000140 2024-05-10T11:32:41.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.612 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-10T11:32:41.612 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 760976 2024-05-10T11:32:41.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.613 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-2": { 2024-05-10T11:32:41.613 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 3231, 2024-05-10T11:32:41.613 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 3232, 2024-05-10T11:32:41.613 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 2792220, 2024-05-10T11:32:41.613 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 9360380, 2024-05-10T11:32:41.613 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 12, 2024-05-10T11:32:41.613 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 7, 2024-05-10T11:32:41.614 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.286616998, 2024-05-10T11:32:41.614 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.090721131, 2024-05-10T11:32:41.614 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.100175616, 2024-05-10T11:32:41.614 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.092644825, 2024-05-10T11:32:41.614 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:41.614 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 3231, 2024-05-10T11:32:41.614 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.177378710, 2024-05-10T11:32:41.614 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000054899 2024-05-10T11:32:41.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.615 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:41.615 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 80, 2024-05-10T11:32:41.615 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000015377, 2024-05-10T11:32:41.615 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000192 2024-05-10T11:32:41.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.615 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 2090192, 2024-05-10T11:32:41.615 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 280576 2024-05-10T11:32:41.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.616 INFO:teuthology.orchestra.run.smithi076.stdout: "cct": { 2024-05-10T11:32:41.616 INFO:teuthology.orchestra.run.smithi076.stdout: "total_workers": 1, 2024-05-10T11:32:41.616 INFO:teuthology.orchestra.run.smithi076.stdout: "unhealthy_workers": 0 2024-05-10T11:32:41.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.616 INFO:teuthology.orchestra.run.smithi076.stdout: "finisher-MDSRank": { 2024-05-10T11:32:41.616 INFO:teuthology.orchestra.run.smithi076.stdout: "queue_len": 0, 2024-05-10T11:32:41.616 INFO:teuthology.orchestra.run.smithi076.stdout: "complete_latency": { 2024-05-10T11:32:41.617 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 4418, 2024-05-10T11:32:41.617 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 3.764223045, 2024-05-10T11:32:41.617 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000852019 2024-05-10T11:32:41.617 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:41.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.617 INFO:teuthology.orchestra.run.smithi076.stdout: "finisher-PurgeQueue": { 2024-05-10T11:32:41.618 INFO:teuthology.orchestra.run.smithi076.stdout: "queue_len": 0, 2024-05-10T11:32:41.618 INFO:teuthology.orchestra.run.smithi076.stdout: "complete_latency": { 2024-05-10T11:32:41.618 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 3778, 2024-05-10T11:32:41.618 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.296836196, 2024-05-10T11:32:41.618 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000078569 2024-05-10T11:32:41.618 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:41.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.619 INFO:teuthology.orchestra.run.smithi076.stdout: "mds": { 2024-05-10T11:32:41.619 INFO:teuthology.orchestra.run.smithi076.stdout: "request": 9306, 2024-05-10T11:32:41.619 INFO:teuthology.orchestra.run.smithi076.stdout: "reply": 8947, 2024-05-10T11:32:41.619 INFO:teuthology.orchestra.run.smithi076.stdout: "reply_latency": { 2024-05-10T11:32:41.619 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 8947, 2024-05-10T11:32:41.619 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 44.152111982, 2024-05-10T11:32:41.620 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.004934851 2024-05-10T11:32:41.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.620 INFO:teuthology.orchestra.run.smithi076.stdout: "slow_reply": 0, 2024-05-10T11:32:41.620 INFO:teuthology.orchestra.run.smithi076.stdout: "forward": 359, 2024-05-10T11:32:41.620 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_fetch_complete": 0, 2024-05-10T11:32:41.620 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_fetch_keys": 0, 2024-05-10T11:32:41.620 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_commit": 11, 2024-05-10T11:32:41.621 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_split": 0, 2024-05-10T11:32:41.621 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_merge": 0, 2024-05-10T11:32:41.621 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes": 176, 2024-05-10T11:32:41.621 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_top": 25, 2024-05-10T11:32:41.621 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_bottom": 151, 2024-05-10T11:32:41.621 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_pin_tail": 0, 2024-05-10T11:32:41.621 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_pinned": 139, 2024-05-10T11:32:41.621 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_expired": 0, 2024-05-10T11:32:41.622 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_with_caps": 13, 2024-05-10T11:32:41.622 INFO:teuthology.orchestra.run.smithi076.stdout: "caps": 13, 2024-05-10T11:32:41.622 INFO:teuthology.orchestra.run.smithi076.stdout: "subtrees": 6, 2024-05-10T11:32:41.622 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse": 22009, 2024-05-10T11:32:41.622 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_hit": 9586, 2024-05-10T11:32:41.622 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_forward": 359, 2024-05-10T11:32:41.623 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_discover": 17, 2024-05-10T11:32:41.623 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_dir_fetch": 0, 2024-05-10T11:32:41.623 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_remote_ino": 0, 2024-05-10T11:32:41.623 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_lock": 312, 2024-05-10T11:32:41.623 INFO:teuthology.orchestra.run.smithi076.stdout: "load_cent": 0, 2024-05-10T11:32:41.623 INFO:teuthology.orchestra.run.smithi076.stdout: "q": 0, 2024-05-10T11:32:41.624 INFO:teuthology.orchestra.run.smithi076.stdout: "exported": 0, 2024-05-10T11:32:41.624 INFO:teuthology.orchestra.run.smithi076.stdout: "exported_inodes": 0, 2024-05-10T11:32:41.624 INFO:teuthology.orchestra.run.smithi076.stdout: "imported": 3, 2024-05-10T11:32:41.624 INFO:teuthology.orchestra.run.smithi076.stdout: "imported_inodes": 17, 2024-05-10T11:32:41.624 INFO:teuthology.orchestra.run.smithi076.stdout: "openino_dir_fetch": 0, 2024-05-10T11:32:41.624 INFO:teuthology.orchestra.run.smithi076.stdout: "openino_backtrace_fetch": 71, 2024-05-10T11:32:41.624 INFO:teuthology.orchestra.run.smithi076.stdout: "openino_peer_discover": 0, 2024-05-10T11:32:41.625 INFO:teuthology.orchestra.run.smithi076.stdout: "root_rfiles": 699, 2024-05-10T11:32:41.625 INFO:teuthology.orchestra.run.smithi076.stdout: "root_rbytes": 211932931, 2024-05-10T11:32:41.625 INFO:teuthology.orchestra.run.smithi076.stdout: "root_rsnaps": 6, 2024-05-10T11:32:41.625 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_backtrace_fetch": 71, 2024-05-10T11:32:41.625 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_set_tag": 0, 2024-05-10T11:32:41.625 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_backtrace_repaired": 0, 2024-05-10T11:32:41.625 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_inotable_repaired": 0, 2024-05-10T11:32:41.626 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_dir_inodes": 0, 2024-05-10T11:32:41.626 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_dir_base_inodes": 0, 2024-05-10T11:32:41.626 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_dirfrag_rstats": 0, 2024-05-10T11:32:41.626 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_file_inodes": 71, 2024-05-10T11:32:41.626 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_inode_file_caps": 0, 2024-05-10T11:32:41.626 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_revoke": 72, 2024-05-10T11:32:41.626 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_grant": 5928, 2024-05-10T11:32:41.627 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_trunc": 0, 2024-05-10T11:32:41.627 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-10T11:32:41.627 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-10T11:32:41.627 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_caps": 980, 2024-05-10T11:32:41.627 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_caps_dirty": 398, 2024-05-10T11:32:41.627 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_cap_release": 345, 2024-05-10T11:32:41.627 INFO:teuthology.orchestra.run.smithi076.stdout: "process_request_cap_release": 9000 2024-05-10T11:32:41.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.628 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache": { 2024-05-10T11:32:41.628 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_update": 0, 2024-05-10T11:32:41.628 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_update_receipt": 132, 2024-05-10T11:32:41.628 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_try_discover": 72, 2024-05-10T11:32:41.628 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_send_discover": 133, 2024-05-10T11:32:41.628 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_handle_discover": 0, 2024-05-10T11:32:41.628 INFO:teuthology.orchestra.run.smithi076.stdout: "num_strays": 0, 2024-05-10T11:32:41.629 INFO:teuthology.orchestra.run.smithi076.stdout: "num_strays_delayed": 0, 2024-05-10T11:32:41.629 INFO:teuthology.orchestra.run.smithi076.stdout: "num_strays_enqueuing": 0, 2024-05-10T11:32:41.629 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_created": 1689, 2024-05-10T11:32:41.629 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_enqueued": 1689, 2024-05-10T11:32:41.629 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_reintegrated": 0, 2024-05-10T11:32:41.629 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_migrated": 0, 2024-05-10T11:32:41.629 INFO:teuthology.orchestra.run.smithi076.stdout: "num_recovering_processing": 0, 2024-05-10T11:32:41.629 INFO:teuthology.orchestra.run.smithi076.stdout: "num_recovering_enqueued": 0, 2024-05-10T11:32:41.630 INFO:teuthology.orchestra.run.smithi076.stdout: "num_recovering_prioritized": 0, 2024-05-10T11:32:41.630 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_started": 0, 2024-05-10T11:32:41.630 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_completed": 0, 2024-05-10T11:32:41.630 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_quiesce_path": 0, 2024-05-10T11:32:41.630 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_quiesce_inode": 0, 2024-05-10T11:32:41.630 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_enqueue_scrub": 0, 2024-05-10T11:32:41.630 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_exportdir": 0, 2024-05-10T11:32:41.631 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_flush": 0, 2024-05-10T11:32:41.631 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_fragmentdir": 0, 2024-05-10T11:32:41.631 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_fragstats": 0, 2024-05-10T11:32:41.631 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_inodestats": 0 2024-05-10T11:32:41.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.631 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log": { 2024-05-10T11:32:41.631 INFO:teuthology.orchestra.run.smithi076.stdout: "evlrg": 0, 2024-05-10T11:32:41.632 INFO:teuthology.orchestra.run.smithi076.stdout: "evadd": 8477, 2024-05-10T11:32:41.632 INFO:teuthology.orchestra.run.smithi076.stdout: "evex": 0, 2024-05-10T11:32:41.632 INFO:teuthology.orchestra.run.smithi076.stdout: "evtrm": 0, 2024-05-10T11:32:41.632 INFO:teuthology.orchestra.run.smithi076.stdout: "ev": 8477, 2024-05-10T11:32:41.632 INFO:teuthology.orchestra.run.smithi076.stdout: "evexg": 0, 2024-05-10T11:32:41.632 INFO:teuthology.orchestra.run.smithi076.stdout: "evexd": 0, 2024-05-10T11:32:41.633 INFO:teuthology.orchestra.run.smithi076.stdout: "segadd": 10, 2024-05-10T11:32:41.633 INFO:teuthology.orchestra.run.smithi076.stdout: "segex": 0, 2024-05-10T11:32:41.633 INFO:teuthology.orchestra.run.smithi076.stdout: "segtrm": 0, 2024-05-10T11:32:41.633 INFO:teuthology.orchestra.run.smithi076.stdout: "seg": 10, 2024-05-10T11:32:41.633 INFO:teuthology.orchestra.run.smithi076.stdout: "segmjr": 2, 2024-05-10T11:32:41.634 INFO:teuthology.orchestra.run.smithi076.stdout: "segexg": 0, 2024-05-10T11:32:41.634 INFO:teuthology.orchestra.run.smithi076.stdout: "segexd": 0, 2024-05-10T11:32:41.634 INFO:teuthology.orchestra.run.smithi076.stdout: "expos": 4194304, 2024-05-10T11:32:41.634 INFO:teuthology.orchestra.run.smithi076.stdout: "wrpos": 16368050, 2024-05-10T11:32:41.634 INFO:teuthology.orchestra.run.smithi076.stdout: "rdpos": 0, 2024-05-10T11:32:41.634 INFO:teuthology.orchestra.run.smithi076.stdout: "jlat": { 2024-05-10T11:32:41.635 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 2050, 2024-05-10T11:32:41.635 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 14.168217211, 2024-05-10T11:32:41.635 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.006911325 2024-05-10T11:32:41.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.635 INFO:teuthology.orchestra.run.smithi076.stdout: "replayed": 0 2024-05-10T11:32:41.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.635 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_mem": { 2024-05-10T11:32:41.635 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 118, 2024-05-10T11:32:41.636 INFO:teuthology.orchestra.run.smithi076.stdout: "ino+": 1807, 2024-05-10T11:32:41.636 INFO:teuthology.orchestra.run.smithi076.stdout: "ino-": 1689, 2024-05-10T11:32:41.636 INFO:teuthology.orchestra.run.smithi076.stdout: "dir": 74, 2024-05-10T11:32:41.636 INFO:teuthology.orchestra.run.smithi076.stdout: "dir+": 133, 2024-05-10T11:32:41.636 INFO:teuthology.orchestra.run.smithi076.stdout: "dir-": 59, 2024-05-10T11:32:41.636 INFO:teuthology.orchestra.run.smithi076.stdout: "dn": 173, 2024-05-10T11:32:41.636 INFO:teuthology.orchestra.run.smithi076.stdout: "dn+": 2548, 2024-05-10T11:32:41.636 INFO:teuthology.orchestra.run.smithi076.stdout: "dn-": 2375, 2024-05-10T11:32:41.636 INFO:teuthology.orchestra.run.smithi076.stdout: "cap": 8, 2024-05-10T11:32:41.637 INFO:teuthology.orchestra.run.smithi076.stdout: "cap+": 1752, 2024-05-10T11:32:41.637 INFO:teuthology.orchestra.run.smithi076.stdout: "cap-": 1744, 2024-05-10T11:32:41.637 INFO:teuthology.orchestra.run.smithi076.stdout: "rss": 63736, 2024-05-10T11:32:41.637 INFO:teuthology.orchestra.run.smithi076.stdout: "heap": 223608 2024-05-10T11:32:41.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.637 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_server": { 2024-05-10T11:32:41.637 INFO:teuthology.orchestra.run.smithi076.stdout: "dispatch_client_request": 18508, 2024-05-10T11:32:41.637 INFO:teuthology.orchestra.run.smithi076.stdout: "dispatch_server_request": 0, 2024-05-10T11:32:41.638 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_request": 9306, 2024-05-10T11:32:41.638 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_session": 32, 2024-05-10T11:32:41.638 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_peer_request": 3713, 2024-05-10T11:32:41.638 INFO:teuthology.orchestra.run.smithi076.stdout: "req_create_latency": { 2024-05-10T11:32:41.638 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 1692, 2024-05-10T11:32:41.638 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 25.392159896, 2024-05-10T11:32:41.638 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.015007186 2024-05-10T11:32:41.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.639 INFO:teuthology.orchestra.run.smithi076.stdout: "req_getattr_latency": { 2024-05-10T11:32:41.639 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 34, 2024-05-10T11:32:41.639 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.031906007, 2024-05-10T11:32:41.639 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000938411 2024-05-10T11:32:41.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.639 INFO:teuthology.orchestra.run.smithi076.stdout: "req_getfilelock_latency": { 2024-05-10T11:32:41.639 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.639 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.640 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.640 INFO:teuthology.orchestra.run.smithi076.stdout: "req_link_latency": { 2024-05-10T11:32:41.640 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.640 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.640 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.640 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookup_latency": { 2024-05-10T11:32:41.641 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 2165, 2024-05-10T11:32:41.641 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 2.512175641, 2024-05-10T11:32:41.641 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.001160358 2024-05-10T11:32:41.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.641 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookuphash_latency": { 2024-05-10T11:32:41.641 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.641 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.641 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.642 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupino_latency": { 2024-05-10T11:32:41.642 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.642 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.642 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.642 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupname_latency": { 2024-05-10T11:32:41.642 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.642 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.643 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.643 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupparent_latency": { 2024-05-10T11:32:41.643 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.643 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.643 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.644 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupsnap_latency": { 2024-05-10T11:32:41.644 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.644 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.644 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.644 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lssnap_latency": { 2024-05-10T11:32:41.644 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.644 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.645 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.645 INFO:teuthology.orchestra.run.smithi076.stdout: "req_mkdir_latency": { 2024-05-10T11:32:41.645 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.645 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.645 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "req_mknod_latency": { 2024-05-10T11:32:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "req_mksnap_latency": { 2024-05-10T11:32:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "req_open_latency": { 2024-05-10T11:32:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "req_readdir_latency": { 2024-05-10T11:32:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 3055, 2024-05-10T11:32:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 1.884190547, 2024-05-10T11:32:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000616756 2024-05-10T11:32:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rename_latency": { 2024-05-10T11:32:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 312, 2024-05-10T11:32:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 4.724173995, 2024-05-10T11:32:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.015141583 2024-05-10T11:32:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: "req_renamesnap_latency": { 2024-05-10T11:32:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: "req_snapdiff_latency": { 2024-05-10T11:32:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rmdir_latency": { 2024-05-10T11:32:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rmsnap_latency": { 2024-05-10T11:32:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rmxattr_latency": { 2024-05-10T11:32:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setattr_latency": { 2024-05-10T11:32:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setdirlayout_latency": { 2024-05-10T11:32:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setfilelock_latency": { 2024-05-10T11:32:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setlayout_latency": { 2024-05-10T11:32:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setxattr_latency": { 2024-05-10T11:32:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "req_symlink_latency": { 2024-05-10T11:32:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "req_unlink_latency": { 2024-05-10T11:32:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 1689, 2024-05-10T11:32:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 9.607505896, 2024-05-10T11:32:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.005688280 2024-05-10T11:32:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "cap_revoke_eviction": 0, 2024-05-10T11:32:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "cap_acquisition_throttle": 0, 2024-05-10T11:32:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: "req_getvxattr_latency": { 2024-05-10T11:32:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_sessions": { 2024-05-10T11:32:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "session_count": 2, 2024-05-10T11:32:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "session_add": 3, 2024-05-10T11:32:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "session_remove": 1, 2024-05-10T11:32:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "sessions_open": 2, 2024-05-10T11:32:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "sessions_stale": 0, 2024-05-10T11:32:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "total_load": 2717, 2024-05-10T11:32:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "average_load": 1358, 2024-05-10T11:32:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: "avg_session_uptime": 1054, 2024-05-10T11:32:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: "mdthresh_evicted": 0 2024-05-10T11:32:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool": { 2024-05-10T11:32:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_bytes": 0, 2024-05-10T11:32:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_items": 0, 2024-05-10T11:32:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_bytes": 0, 2024-05-10T11:32:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_items": 0, 2024-05-10T11:32:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_bytes": 0, 2024-05-10T11:32:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_items": 0, 2024-05-10T11:32:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-10T11:32:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_items": 0, 2024-05-10T11:32:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-10T11:32:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_items": 0, 2024-05-10T11:32:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_bytes": 0, 2024-05-10T11:32:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_items": 0, 2024-05-10T11:32:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-10T11:32:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_items": 0, 2024-05-10T11:32:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_bytes": 0, 2024-05-10T11:32:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_items": 0, 2024-05-10T11:32:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_bytes": 0, 2024-05-10T11:32:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_items": 0, 2024-05-10T11:32:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-10T11:32:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_items": 0, 2024-05-10T11:32:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-10T11:32:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_items": 0, 2024-05-10T11:32:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_bytes": 0, 2024-05-10T11:32:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_items": 0, 2024-05-10T11:32:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_bytes": 0, 2024-05-10T11:32:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_items": 0, 2024-05-10T11:32:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-10T11:32:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_items": 0, 2024-05-10T11:32:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_bytes": 0, 2024-05-10T11:32:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_items": 0, 2024-05-10T11:32:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_bytes": 0, 2024-05-10T11:32:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_items": 0, 2024-05-10T11:32:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_bytes": 0, 2024-05-10T11:32:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_items": 0, 2024-05-10T11:32:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_bytes": 0, 2024-05-10T11:32:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_items": 0, 2024-05-10T11:32:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_bytes": 351774, 2024-05-10T11:32:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_items": 244, 2024-05-10T11:32:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_bytes": 352, 2024-05-10T11:32:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_items": 4, 2024-05-10T11:32:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bytes": 0, 2024-05-10T11:32:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_items": 0, 2024-05-10T11:32:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_bytes": 0, 2024-05-10T11:32:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_items": 0, 2024-05-10T11:32:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_bytes": 0, 2024-05-10T11:32:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_items": 0, 2024-05-10T11:32:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_bytes": 9808, 2024-05-10T11:32:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_items": 189, 2024-05-10T11:32:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_bytes": 0, 2024-05-10T11:32:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_items": 0, 2024-05-10T11:32:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_bytes": 0, 2024-05-10T11:32:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_items": 0, 2024-05-10T11:32:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_bytes": 565969, 2024-05-10T11:32:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_items": 4668, 2024-05-10T11:32:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_bytes": 0, 2024-05-10T11:32:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_items": 0, 2024-05-10T11:32:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_bytes": 0, 2024-05-10T11:32:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_items": 0 2024-05-10T11:32:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter": { 2024-05-10T11:32:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "op_active": 0, 2024-05-10T11:32:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "op_laggy": 0, 2024-05-10T11:32:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "op_send": 6311, 2024-05-10T11:32:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "op_send_bytes": 12552071, 2024-05-10T11:32:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "op_resend": 31, 2024-05-10T11:32:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "op_reply": 6311, 2024-05-10T11:32:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "op_latency": { 2024-05-10T11:32:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 6311, 2024-05-10T11:32:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 56.994355216, 2024-05-10T11:32:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.009030954 2024-05-10T11:32:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "op_inflight": 0, 2024-05-10T11:32:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "oplen_avg": { 2024-05-10T11:32:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 6311, 2024-05-10T11:32:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 6650 2024-05-10T11:32:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "op": 6311, 2024-05-10T11:32:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "op_r": 278, 2024-05-10T11:32:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "op_w": 6033, 2024-05-10T11:32:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "op_rmw": 0, 2024-05-10T11:32:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "op_pg": 0, 2024-05-10T11:32:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_stat": 0, 2024-05-10T11:32:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_create": 10, 2024-05-10T11:32:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_read": 207, 2024-05-10T11:32:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_write": 2257, 2024-05-10T11:32:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_writefull": 138, 2024-05-10T11:32:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_writesame": 0, 2024-05-10T11:32:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_append": 0, 2024-05-10T11:32:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_zero": 0, 2024-05-10T11:32:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_truncate": 0, 2024-05-10T11:32:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_delete": 3392, 2024-05-10T11:32:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_mapext": 0, 2024-05-10T11:32:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_sparse_read": 0, 2024-05-10T11:32:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_clonerange": 0, 2024-05-10T11:32:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_getxattr": 71, 2024-05-10T11:32:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_setxattr": 20, 2024-05-10T11:32:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_cmpxattr": 0, 2024-05-10T11:32:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_rmxattr": 0, 2024-05-10T11:32:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_resetxattrs": 0, 2024-05-10T11:32:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_call": 0, 2024-05-10T11:32:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_watch": 0, 2024-05-10T11:32:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_notify": 0, 2024-05-10T11:32:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_src_cmpxattr": 0, 2024-05-10T11:32:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_pgls": 0, 2024-05-10T11:32:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_pgls_filter": 0, 2024-05-10T11:32:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_other": 0, 2024-05-10T11:32:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_active": 0, 2024-05-10T11:32:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_send": 0, 2024-05-10T11:32:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_resend": 0, 2024-05-10T11:32:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_ping": 0, 2024-05-10T11:32:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "poolop_active": 0, 2024-05-10T11:32:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "poolop_send": 0, 2024-05-10T11:32:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "poolop_resend": 0, 2024-05-10T11:32:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "poolstat_active": 0, 2024-05-10T11:32:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "poolstat_send": 0, 2024-05-10T11:32:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "poolstat_resend": 0, 2024-05-10T11:32:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "statfs_active": 0, 2024-05-10T11:32:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "statfs_send": 0, 2024-05-10T11:32:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "statfs_resend": 0, 2024-05-10T11:32:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "command_active": 0, 2024-05-10T11:32:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "command_send": 0, 2024-05-10T11:32:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "command_resend": 0, 2024-05-10T11:32:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "map_epoch": 117, 2024-05-10T11:32:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "map_full": 0, 2024-05-10T11:32:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "map_inc": 29, 2024-05-10T11:32:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_sessions": 12, 2024-05-10T11:32:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_session_open": 16, 2024-05-10T11:32:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_session_close": 4, 2024-05-10T11:32:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_laggy": 0, 2024-05-10T11:32:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_wr": 436, 2024-05-10T11:32:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_rd": 0, 2024-05-10T11:32:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_del": 119 2024-05-10T11:32:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "oft": { 2024-05-10T11:32:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_objs": 1, 2024-05-10T11:32:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_kv_pairs": 0, 2024-05-10T11:32:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_updates": 0, 2024-05-10T11:32:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_removes": 0 2024-05-10T11:32:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "osd-slow-ops": { 2024-05-10T11:32:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "slow_ops_count": 0 2024-05-10T11:32:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "purge_queue": { 2024-05-10T11:32:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing_ops": 0, 2024-05-10T11:32:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing_ops_high_water": 10, 2024-05-10T11:32:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing": 0, 2024-05-10T11:32:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing_high_water": 5, 2024-05-10T11:32:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executed_ops": 3378, 2024-05-10T11:32:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executed": 1689, 2024-05-10T11:32:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_item_in_journal": 0 2024-05-10T11:32:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-10T11:32:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-05-10T11:32:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 269411, 2024-05-10T11:32:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 109820690, 2024-05-10T11:32:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 269411, 2024-05-10T11:32:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-05-10T11:32:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-05-10T11:32:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 269411, 2024-05-10T11:32:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 109820690, 2024-05-10T11:32:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-objecter_bytes": { 2024-05-10T11:32:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-05-10T11:32:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 0, 2024-05-10T11:32:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 0, 2024-05-10T11:32:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 6311, 2024-05-10T11:32:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 12736252, 2024-05-10T11:32:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 6311, 2024-05-10T11:32:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 12736252, 2024-05-10T11:32:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-objecter_ops": { 2024-05-10T11:32:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 1024, 2024-05-10T11:32:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 0, 2024-05-10T11:32:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 0, 2024-05-10T11:32:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 6311, 2024-05-10T11:32:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 6311, 2024-05-10T11:32:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 6311, 2024-05-10T11:32:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 6311, 2024-05-10T11:32:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-write_buf_throttle": { 2024-05-10T11:32:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 3758096384, 2024-05-10T11:32:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 1689, 2024-05-10T11:32:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 184181, 2024-05-10T11:32:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 1689, 2024-05-10T11:32:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-05-10T11:32:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-05-10T11:32:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 207, 2024-05-10T11:32:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 184181, 2024-05-10T11:32:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-write_buf_throttle-0x55da27bf2500": { 2024-05-10T11:32:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 2368, 2024-05-10T11:32:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 3758096384, 2024-05-10T11:32:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 8477, 2024-05-10T11:32:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 12173746, 2024-05-10T11:32:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 8477, 2024-05-10T11:32:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-05-10T11:32:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-05-10T11:32:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 2050, 2024-05-10T11:32:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 12171378, 2024-05-10T11:32:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:41.692 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-05-10T11:32:42.568 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-05-10T11:32:42.569 INFO:tasks.check_counter:Daemon mds.e mds.imported=3 2024-05-10T11:32:42.569 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-05-10T11:32:42.569 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-05-10T11:32:42.569 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-05-10T11:32:42.569 DEBUG:tasks.check_counter:Getting stats from h 2024-05-10T11:32:42.570 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph daemon mds.h perf dump 2024-05-10T11:32:42.822 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.b/config 2024-05-10T11:32:43.092 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:42 smithi076 ceph-mon[34152]: pgmap v613: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.2 MiB/s wr, 58 op/s 2024-05-10T11:32:43.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:42 smithi027 ceph-mon[30054]: pgmap v613: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.2 MiB/s wr, 58 op/s 2024-05-10T11:32:43.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:42 smithi189 ceph-mon[34842]: pgmap v613: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.2 MiB/s wr, 58 op/s 2024-05-10T11:32:44.587 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-05-10T11:32:44.587 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-0": { 2024-05-10T11:32:44.588 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 20850, 2024-05-10T11:32:44.588 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 27196, 2024-05-10T11:32:44.588 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 4869115, 2024-05-10T11:32:44.588 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 75781750, 2024-05-10T11:32:44.588 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 27, 2024-05-10T11:32:44.588 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 10, 2024-05-10T11:32:44.588 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 1.955332761, 2024-05-10T11:32:44.588 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.625919611, 2024-05-10T11:32:44.589 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.567417894, 2024-05-10T11:32:44.589 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.683433083, 2024-05-10T11:32:44.589 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:44.589 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 27189, 2024-05-10T11:32:44.589 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 1.651043400, 2024-05-10T11:32:44.589 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000060724 2024-05-10T11:32:44.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.589 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:44.590 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 19166, 2024-05-10T11:32:44.590 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.004026536, 2024-05-10T11:32:44.590 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000210 2024-05-10T11:32:44.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.590 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 4384, 2024-05-10T11:32:44.590 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 14032 2024-05-10T11:32:44.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.590 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-1": { 2024-05-10T11:32:44.591 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 261426, 2024-05-10T11:32:44.591 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 163355, 2024-05-10T11:32:44.591 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 127361597, 2024-05-10T11:32:44.591 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 102670597, 2024-05-10T11:32:44.591 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 35, 2024-05-10T11:32:44.591 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 6, 2024-05-10T11:32:44.591 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 13.424515300, 2024-05-10T11:32:44.591 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 5.819951030, 2024-05-10T11:32:44.592 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 9.369276015, 2024-05-10T11:32:44.592 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.490897215, 2024-05-10T11:32:44.592 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:44.592 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 163357, 2024-05-10T11:32:44.592 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 4.418167806, 2024-05-10T11:32:44.592 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000027046 2024-05-10T11:32:44.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.592 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:44.592 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 398516, 2024-05-10T11:32:44.593 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.061557096, 2024-05-10T11:32:44.593 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000154 2024-05-10T11:32:44.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.593 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-10T11:32:44.593 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 684912 2024-05-10T11:32:44.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.593 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-2": { 2024-05-10T11:32:44.593 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 116906, 2024-05-10T11:32:44.593 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 179683, 2024-05-10T11:32:44.594 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 38150493, 2024-05-10T11:32:44.594 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 188114598, 2024-05-10T11:32:44.594 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 12, 2024-05-10T11:32:44.594 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 7, 2024-05-10T11:32:44.594 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 7.834041345, 2024-05-10T11:32:44.594 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 3.500433310, 2024-05-10T11:32:44.594 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 3.994310109, 2024-05-10T11:32:44.594 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.692352952, 2024-05-10T11:32:44.595 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:44.595 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 179683, 2024-05-10T11:32:44.595 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 8.220096780, 2024-05-10T11:32:44.595 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000045747 2024-05-10T11:32:44.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.595 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:44.595 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 143283, 2024-05-10T11:32:44.595 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.042988461, 2024-05-10T11:32:44.596 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000300 2024-05-10T11:32:44.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.596 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 2107536, 2024-05-10T11:32:44.596 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 272784 2024-05-10T11:32:44.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.596 INFO:teuthology.orchestra.run.smithi076.stdout: "cct": { 2024-05-10T11:32:44.596 INFO:teuthology.orchestra.run.smithi076.stdout: "total_workers": 1, 2024-05-10T11:32:44.596 INFO:teuthology.orchestra.run.smithi076.stdout: "unhealthy_workers": 0 2024-05-10T11:32:44.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.597 INFO:teuthology.orchestra.run.smithi076.stdout: "finisher-MDSRank": { 2024-05-10T11:32:44.597 INFO:teuthology.orchestra.run.smithi076.stdout: "queue_len": 0, 2024-05-10T11:32:44.597 INFO:teuthology.orchestra.run.smithi076.stdout: "complete_latency": { 2024-05-10T11:32:44.597 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 34484, 2024-05-10T11:32:44.597 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 40.052971252, 2024-05-10T11:32:44.597 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.001161494 2024-05-10T11:32:44.597 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:44.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.597 INFO:teuthology.orchestra.run.smithi076.stdout: "finisher-PurgeQueue": { 2024-05-10T11:32:44.598 INFO:teuthology.orchestra.run.smithi076.stdout: "queue_len": 0, 2024-05-10T11:32:44.598 INFO:teuthology.orchestra.run.smithi076.stdout: "complete_latency": { 2024-05-10T11:32:44.598 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 22230, 2024-05-10T11:32:44.598 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 2.374565605, 2024-05-10T11:32:44.598 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000106818 2024-05-10T11:32:44.598 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:44.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.598 INFO:teuthology.orchestra.run.smithi076.stdout: "mds": { 2024-05-10T11:32:44.599 INFO:teuthology.orchestra.run.smithi076.stdout: "request": 68217, 2024-05-10T11:32:44.599 INFO:teuthology.orchestra.run.smithi076.stdout: "reply": 63756, 2024-05-10T11:32:44.599 INFO:teuthology.orchestra.run.smithi076.stdout: "reply_latency": { 2024-05-10T11:32:44.599 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 63756, 2024-05-10T11:32:44.599 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 211.418264833, 2024-05-10T11:32:44.599 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.003316052 2024-05-10T11:32:44.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.600 INFO:teuthology.orchestra.run.smithi076.stdout: "slow_reply": 0, 2024-05-10T11:32:44.600 INFO:teuthology.orchestra.run.smithi076.stdout: "forward": 4461, 2024-05-10T11:32:44.600 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_fetch_complete": 0, 2024-05-10T11:32:44.600 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_fetch_keys": 0, 2024-05-10T11:32:44.600 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_commit": 56, 2024-05-10T11:32:44.600 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_split": 5, 2024-05-10T11:32:44.600 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_merge": 4, 2024-05-10T11:32:44.600 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes": 1055, 2024-05-10T11:32:44.601 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_top": 392, 2024-05-10T11:32:44.601 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_bottom": 663, 2024-05-10T11:32:44.601 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_pin_tail": 0, 2024-05-10T11:32:44.601 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_pinned": 495, 2024-05-10T11:32:44.601 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_expired": 1, 2024-05-10T11:32:44.601 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_with_caps": 59, 2024-05-10T11:32:44.601 INFO:teuthology.orchestra.run.smithi076.stdout: "caps": 59, 2024-05-10T11:32:44.601 INFO:teuthology.orchestra.run.smithi076.stdout: "subtrees": 14, 2024-05-10T11:32:44.601 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse": 101999, 2024-05-10T11:32:44.602 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_hit": 57996, 2024-05-10T11:32:44.602 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_forward": 4461, 2024-05-10T11:32:44.602 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_discover": 27, 2024-05-10T11:32:44.602 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_dir_fetch": 0, 2024-05-10T11:32:44.602 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_remote_ino": 0, 2024-05-10T11:32:44.602 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_lock": 698, 2024-05-10T11:32:44.602 INFO:teuthology.orchestra.run.smithi076.stdout: "load_cent": 0, 2024-05-10T11:32:44.602 INFO:teuthology.orchestra.run.smithi076.stdout: "q": 0, 2024-05-10T11:32:44.603 INFO:teuthology.orchestra.run.smithi076.stdout: "exported": 0, 2024-05-10T11:32:44.603 INFO:teuthology.orchestra.run.smithi076.stdout: "exported_inodes": 0, 2024-05-10T11:32:44.603 INFO:teuthology.orchestra.run.smithi076.stdout: "imported": 4, 2024-05-10T11:32:44.603 INFO:teuthology.orchestra.run.smithi076.stdout: "imported_inodes": 34, 2024-05-10T11:32:44.603 INFO:teuthology.orchestra.run.smithi076.stdout: "openino_dir_fetch": 0, 2024-05-10T11:32:44.603 INFO:teuthology.orchestra.run.smithi076.stdout: "openino_backtrace_fetch": 363, 2024-05-10T11:32:44.603 INFO:teuthology.orchestra.run.smithi076.stdout: "openino_peer_discover": 0, 2024-05-10T11:32:44.603 INFO:teuthology.orchestra.run.smithi076.stdout: "root_rfiles": 699, 2024-05-10T11:32:44.603 INFO:teuthology.orchestra.run.smithi076.stdout: "root_rbytes": 211932931, 2024-05-10T11:32:44.604 INFO:teuthology.orchestra.run.smithi076.stdout: "root_rsnaps": 6, 2024-05-10T11:32:44.604 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_backtrace_fetch": 363, 2024-05-10T11:32:44.604 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_set_tag": 0, 2024-05-10T11:32:44.604 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_backtrace_repaired": 0, 2024-05-10T11:32:44.604 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_inotable_repaired": 0, 2024-05-10T11:32:44.604 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_dir_inodes": 36, 2024-05-10T11:32:44.604 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_dir_base_inodes": 0, 2024-05-10T11:32:44.604 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_dirfrag_rstats": 36, 2024-05-10T11:32:44.604 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_file_inodes": 327, 2024-05-10T11:32:44.605 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_inode_file_caps": 0, 2024-05-10T11:32:44.605 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_revoke": 1566, 2024-05-10T11:32:44.605 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_grant": 43870, 2024-05-10T11:32:44.605 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_trunc": 0, 2024-05-10T11:32:44.605 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-10T11:32:44.605 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-10T11:32:44.605 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_caps": 18018, 2024-05-10T11:32:44.605 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_caps_dirty": 9219, 2024-05-10T11:32:44.606 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_cap_release": 4214, 2024-05-10T11:32:44.606 INFO:teuthology.orchestra.run.smithi076.stdout: "process_request_cap_release": 43504 2024-05-10T11:32:44.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.606 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache": { 2024-05-10T11:32:44.606 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_update": 9, 2024-05-10T11:32:44.606 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_update_receipt": 122, 2024-05-10T11:32:44.606 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_try_discover": 65, 2024-05-10T11:32:44.606 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_send_discover": 149, 2024-05-10T11:32:44.607 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_handle_discover": 22, 2024-05-10T11:32:44.607 INFO:teuthology.orchestra.run.smithi076.stdout: "num_strays": 0, 2024-05-10T11:32:44.607 INFO:teuthology.orchestra.run.smithi076.stdout: "num_strays_delayed": 0, 2024-05-10T11:32:44.607 INFO:teuthology.orchestra.run.smithi076.stdout: "num_strays_enqueuing": 0, 2024-05-10T11:32:44.607 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_created": 11572, 2024-05-10T11:32:44.607 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_enqueued": 11572, 2024-05-10T11:32:44.607 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_reintegrated": 0, 2024-05-10T11:32:44.607 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_migrated": 0, 2024-05-10T11:32:44.608 INFO:teuthology.orchestra.run.smithi076.stdout: "num_recovering_processing": 0, 2024-05-10T11:32:44.608 INFO:teuthology.orchestra.run.smithi076.stdout: "num_recovering_enqueued": 0, 2024-05-10T11:32:44.608 INFO:teuthology.orchestra.run.smithi076.stdout: "num_recovering_prioritized": 0, 2024-05-10T11:32:44.608 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_started": 0, 2024-05-10T11:32:44.608 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_completed": 0, 2024-05-10T11:32:44.608 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_quiesce_path": 0, 2024-05-10T11:32:44.608 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_quiesce_inode": 0, 2024-05-10T11:32:44.608 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_enqueue_scrub": 0, 2024-05-10T11:32:44.608 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_exportdir": 0, 2024-05-10T11:32:44.609 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_flush": 0, 2024-05-10T11:32:44.609 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_fragmentdir": 9, 2024-05-10T11:32:44.609 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_fragstats": 0, 2024-05-10T11:32:44.609 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_inodestats": 0 2024-05-10T11:32:44.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.609 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log": { 2024-05-10T11:32:44.609 INFO:teuthology.orchestra.run.smithi076.stdout: "evlrg": 0, 2024-05-10T11:32:44.609 INFO:teuthology.orchestra.run.smithi076.stdout: "evadd": 55512, 2024-05-10T11:32:44.610 INFO:teuthology.orchestra.run.smithi076.stdout: "evex": 0, 2024-05-10T11:32:44.610 INFO:teuthology.orchestra.run.smithi076.stdout: "evtrm": 0, 2024-05-10T11:32:44.610 INFO:teuthology.orchestra.run.smithi076.stdout: "ev": 55512, 2024-05-10T11:32:44.610 INFO:teuthology.orchestra.run.smithi076.stdout: "evexg": 0, 2024-05-10T11:32:44.610 INFO:teuthology.orchestra.run.smithi076.stdout: "evexd": 0, 2024-05-10T11:32:44.610 INFO:teuthology.orchestra.run.smithi076.stdout: "segadd": 82, 2024-05-10T11:32:44.610 INFO:teuthology.orchestra.run.smithi076.stdout: "segex": 0, 2024-05-10T11:32:44.610 INFO:teuthology.orchestra.run.smithi076.stdout: "segtrm": 0, 2024-05-10T11:32:44.611 INFO:teuthology.orchestra.run.smithi076.stdout: "seg": 82, 2024-05-10T11:32:44.611 INFO:teuthology.orchestra.run.smithi076.stdout: "segmjr": 6, 2024-05-10T11:32:44.611 INFO:teuthology.orchestra.run.smithi076.stdout: "segexg": 0, 2024-05-10T11:32:44.611 INFO:teuthology.orchestra.run.smithi076.stdout: "segexd": 0, 2024-05-10T11:32:44.611 INFO:teuthology.orchestra.run.smithi076.stdout: "expos": 4194304, 2024-05-10T11:32:44.611 INFO:teuthology.orchestra.run.smithi076.stdout: "wrpos": 193753601, 2024-05-10T11:32:44.611 INFO:teuthology.orchestra.run.smithi076.stdout: "rdpos": 0, 2024-05-10T11:32:44.611 INFO:teuthology.orchestra.run.smithi076.stdout: "jlat": { 2024-05-10T11:32:44.611 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 17409, 2024-05-10T11:32:44.612 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 100.135784592, 2024-05-10T11:32:44.612 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.005751954 2024-05-10T11:32:44.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.612 INFO:teuthology.orchestra.run.smithi076.stdout: "replayed": 0 2024-05-10T11:32:44.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.612 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_mem": { 2024-05-10T11:32:44.612 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 462, 2024-05-10T11:32:44.612 INFO:teuthology.orchestra.run.smithi076.stdout: "ino+": 12034, 2024-05-10T11:32:44.612 INFO:teuthology.orchestra.run.smithi076.stdout: "ino-": 11572, 2024-05-10T11:32:44.613 INFO:teuthology.orchestra.run.smithi076.stdout: "dir": 83, 2024-05-10T11:32:44.613 INFO:teuthology.orchestra.run.smithi076.stdout: "dir+": 188, 2024-05-10T11:32:44.613 INFO:teuthology.orchestra.run.smithi076.stdout: "dir-": 105, 2024-05-10T11:32:44.613 INFO:teuthology.orchestra.run.smithi076.stdout: "dn": 1034, 2024-05-10T11:32:44.613 INFO:teuthology.orchestra.run.smithi076.stdout: "dn+": 21407, 2024-05-10T11:32:44.613 INFO:teuthology.orchestra.run.smithi076.stdout: "dn-": 20373, 2024-05-10T11:32:44.613 INFO:teuthology.orchestra.run.smithi076.stdout: "cap": 59, 2024-05-10T11:32:44.613 INFO:teuthology.orchestra.run.smithi076.stdout: "cap+": 12767, 2024-05-10T11:32:44.614 INFO:teuthology.orchestra.run.smithi076.stdout: "cap-": 12708, 2024-05-10T11:32:44.614 INFO:teuthology.orchestra.run.smithi076.stdout: "rss": 70488, 2024-05-10T11:32:44.614 INFO:teuthology.orchestra.run.smithi076.stdout: "heap": 223608 2024-05-10T11:32:44.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.614 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_server": { 2024-05-10T11:32:44.614 INFO:teuthology.orchestra.run.smithi076.stdout: "dispatch_client_request": 93287, 2024-05-10T11:32:44.614 INFO:teuthology.orchestra.run.smithi076.stdout: "dispatch_server_request": 0, 2024-05-10T11:32:44.614 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_request": 68217, 2024-05-10T11:32:44.615 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_session": 1514, 2024-05-10T11:32:44.615 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_peer_request": 3366, 2024-05-10T11:32:44.615 INFO:teuthology.orchestra.run.smithi076.stdout: "req_create_latency": { 2024-05-10T11:32:44.615 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 11624, 2024-05-10T11:32:44.615 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 47.747355317, 2024-05-10T11:32:44.615 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.004107652 2024-05-10T11:32:44.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.616 INFO:teuthology.orchestra.run.smithi076.stdout: "req_getattr_latency": { 2024-05-10T11:32:44.616 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 2125, 2024-05-10T11:32:44.616 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 14.666616769, 2024-05-10T11:32:44.616 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.006901937 2024-05-10T11:32:44.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.616 INFO:teuthology.orchestra.run.smithi076.stdout: "req_getfilelock_latency": { 2024-05-10T11:32:44.616 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.616 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.617 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.617 INFO:teuthology.orchestra.run.smithi076.stdout: "req_link_latency": { 2024-05-10T11:32:44.617 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.617 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.617 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.617 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookup_latency": { 2024-05-10T11:32:44.618 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 16432, 2024-05-10T11:32:44.618 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 16.235495055, 2024-05-10T11:32:44.618 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000988041 2024-05-10T11:32:44.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.618 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookuphash_latency": { 2024-05-10T11:32:44.618 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.619 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.619 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.619 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupino_latency": { 2024-05-10T11:32:44.619 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.619 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.619 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.620 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupname_latency": { 2024-05-10T11:32:44.620 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.620 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.620 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.620 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupparent_latency": { 2024-05-10T11:32:44.620 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.621 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.621 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.621 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupsnap_latency": { 2024-05-10T11:32:44.621 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.621 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.622 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.622 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lssnap_latency": { 2024-05-10T11:32:44.623 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.623 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.623 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.623 INFO:teuthology.orchestra.run.smithi076.stdout: "req_mkdir_latency": { 2024-05-10T11:32:44.624 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 5, 2024-05-10T11:32:44.624 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.063375226, 2024-05-10T11:32:44.624 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.012675045 2024-05-10T11:32:44.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.624 INFO:teuthology.orchestra.run.smithi076.stdout: "req_mknod_latency": { 2024-05-10T11:32:44.624 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.624 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.624 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.625 INFO:teuthology.orchestra.run.smithi076.stdout: "req_mksnap_latency": { 2024-05-10T11:32:44.625 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.625 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.625 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.625 INFO:teuthology.orchestra.run.smithi076.stdout: "req_open_latency": { 2024-05-10T11:32:44.625 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.625 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.626 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.626 INFO:teuthology.orchestra.run.smithi076.stdout: "req_readdir_latency": { 2024-05-10T11:32:44.626 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 19448, 2024-05-10T11:32:44.626 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 82.776070270, 2024-05-10T11:32:44.626 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.004256276 2024-05-10T11:32:44.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.626 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rename_latency": { 2024-05-10T11:32:44.627 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 2214, 2024-05-10T11:32:44.627 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 6.724678116, 2024-05-10T11:32:44.627 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.003037343 2024-05-10T11:32:44.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.627 INFO:teuthology.orchestra.run.smithi076.stdout: "req_renamesnap_latency": { 2024-05-10T11:32:44.627 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.627 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.627 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.628 INFO:teuthology.orchestra.run.smithi076.stdout: "req_snapdiff_latency": { 2024-05-10T11:32:44.628 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.628 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.628 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.628 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rmdir_latency": { 2024-05-10T11:32:44.628 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.629 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.629 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.629 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rmsnap_latency": { 2024-05-10T11:32:44.629 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.629 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.629 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.629 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rmxattr_latency": { 2024-05-10T11:32:44.630 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.630 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.630 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.630 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setattr_latency": { 2024-05-10T11:32:44.630 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 12, 2024-05-10T11:32:44.630 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.130349987, 2024-05-10T11:32:44.631 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.010862498 2024-05-10T11:32:44.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.631 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setdirlayout_latency": { 2024-05-10T11:32:44.631 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.631 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.631 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.631 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setfilelock_latency": { 2024-05-10T11:32:44.632 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 324, 2024-05-10T11:32:44.632 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.291389343, 2024-05-10T11:32:44.632 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000899349 2024-05-10T11:32:44.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.632 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setlayout_latency": { 2024-05-10T11:32:44.632 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.632 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.632 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.633 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setxattr_latency": { 2024-05-10T11:32:44.633 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.633 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.633 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.633 INFO:teuthology.orchestra.run.smithi076.stdout: "req_symlink_latency": { 2024-05-10T11:32:44.633 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.634 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.634 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.634 INFO:teuthology.orchestra.run.smithi076.stdout: "req_unlink_latency": { 2024-05-10T11:32:44.634 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 11572, 2024-05-10T11:32:44.634 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 42.782934750, 2024-05-10T11:32:44.634 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.003697108 2024-05-10T11:32:44.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.635 INFO:teuthology.orchestra.run.smithi076.stdout: "cap_revoke_eviction": 0, 2024-05-10T11:32:44.635 INFO:teuthology.orchestra.run.smithi076.stdout: "cap_acquisition_throttle": 0, 2024-05-10T11:32:44.635 INFO:teuthology.orchestra.run.smithi076.stdout: "req_getvxattr_latency": { 2024-05-10T11:32:44.635 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.635 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.635 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.635 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:44.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.636 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_sessions": { 2024-05-10T11:32:44.636 INFO:teuthology.orchestra.run.smithi076.stdout: "session_count": 2, 2024-05-10T11:32:44.636 INFO:teuthology.orchestra.run.smithi076.stdout: "session_add": 4, 2024-05-10T11:32:44.636 INFO:teuthology.orchestra.run.smithi076.stdout: "session_remove": 2, 2024-05-10T11:32:44.636 INFO:teuthology.orchestra.run.smithi076.stdout: "sessions_open": 2, 2024-05-10T11:32:44.636 INFO:teuthology.orchestra.run.smithi076.stdout: "sessions_stale": 0, 2024-05-10T11:32:44.636 INFO:teuthology.orchestra.run.smithi076.stdout: "total_load": 18168, 2024-05-10T11:32:44.636 INFO:teuthology.orchestra.run.smithi076.stdout: "average_load": 9084, 2024-05-10T11:32:44.636 INFO:teuthology.orchestra.run.smithi076.stdout: "avg_session_uptime": 1078, 2024-05-10T11:32:44.637 INFO:teuthology.orchestra.run.smithi076.stdout: "mdthresh_evicted": 0 2024-05-10T11:32:44.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.637 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool": { 2024-05-10T11:32:44.637 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_bytes": 0, 2024-05-10T11:32:44.637 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_items": 0, 2024-05-10T11:32:44.637 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_bytes": 0, 2024-05-10T11:32:44.637 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_items": 0, 2024-05-10T11:32:44.637 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_bytes": 0, 2024-05-10T11:32:44.638 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_items": 0, 2024-05-10T11:32:44.638 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-10T11:32:44.638 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_items": 0, 2024-05-10T11:32:44.638 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-10T11:32:44.638 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_items": 0, 2024-05-10T11:32:44.638 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_bytes": 0, 2024-05-10T11:32:44.638 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_items": 0, 2024-05-10T11:32:44.638 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-10T11:32:44.639 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_items": 0, 2024-05-10T11:32:44.639 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_bytes": 0, 2024-05-10T11:32:44.639 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_items": 0, 2024-05-10T11:32:44.639 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_bytes": 0, 2024-05-10T11:32:44.639 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_items": 0, 2024-05-10T11:32:44.639 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-10T11:32:44.639 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_items": 0, 2024-05-10T11:32:44.639 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-10T11:32:44.639 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_items": 0, 2024-05-10T11:32:44.640 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_bytes": 0, 2024-05-10T11:32:44.640 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_items": 0, 2024-05-10T11:32:44.640 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_bytes": 0, 2024-05-10T11:32:44.640 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_items": 0, 2024-05-10T11:32:44.640 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-10T11:32:44.640 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_items": 0, 2024-05-10T11:32:44.640 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_bytes": 0, 2024-05-10T11:32:44.640 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_items": 0, 2024-05-10T11:32:44.640 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_bytes": 0, 2024-05-10T11:32:44.641 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_items": 0, 2024-05-10T11:32:44.641 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_bytes": 0, 2024-05-10T11:32:44.641 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_items": 0, 2024-05-10T11:32:44.641 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_bytes": 0, 2024-05-10T11:32:44.641 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_items": 0, 2024-05-10T11:32:44.641 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_bytes": 713111, 2024-05-10T11:32:44.641 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_items": 489, 2024-05-10T11:32:44.641 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_bytes": 0, 2024-05-10T11:32:44.642 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_items": 0, 2024-05-10T11:32:44.642 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bytes": 0, 2024-05-10T11:32:44.642 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_items": 0, 2024-05-10T11:32:44.642 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_bytes": 0, 2024-05-10T11:32:44.642 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_items": 0, 2024-05-10T11:32:44.642 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_bytes": 0, 2024-05-10T11:32:44.642 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_items": 0, 2024-05-10T11:32:44.642 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_bytes": 9808, 2024-05-10T11:32:44.643 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_items": 189, 2024-05-10T11:32:44.643 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_bytes": 0, 2024-05-10T11:32:44.643 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_items": 0, 2024-05-10T11:32:44.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_bytes": 0, 2024-05-10T11:32:44.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_items": 0, 2024-05-10T11:32:44.643 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_bytes": 2201332, 2024-05-10T11:32:44.643 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_items": 22314, 2024-05-10T11:32:44.643 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_bytes": 0, 2024-05-10T11:32:44.643 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_items": 0, 2024-05-10T11:32:44.644 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_bytes": 0, 2024-05-10T11:32:44.644 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_items": 0 2024-05-10T11:32:44.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.645 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter": { 2024-05-10T11:32:44.646 INFO:teuthology.orchestra.run.smithi076.stdout: "op_active": 0, 2024-05-10T11:32:44.646 INFO:teuthology.orchestra.run.smithi076.stdout: "op_laggy": 0, 2024-05-10T11:32:44.646 INFO:teuthology.orchestra.run.smithi076.stdout: "op_send": 42315, 2024-05-10T11:32:44.646 INFO:teuthology.orchestra.run.smithi076.stdout: "op_send_bytes": 192465142, 2024-05-10T11:32:44.646 INFO:teuthology.orchestra.run.smithi076.stdout: "op_resend": 29, 2024-05-10T11:32:44.646 INFO:teuthology.orchestra.run.smithi076.stdout: "op_reply": 42315, 2024-05-10T11:32:44.646 INFO:teuthology.orchestra.run.smithi076.stdout: "op_latency": { 2024-05-10T11:32:44.646 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 42315, 2024-05-10T11:32:44.647 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 326.492579451, 2024-05-10T11:32:44.647 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.007715764 2024-05-10T11:32:44.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.647 INFO:teuthology.orchestra.run.smithi076.stdout: "op_inflight": 0, 2024-05-10T11:32:44.647 INFO:teuthology.orchestra.run.smithi076.stdout: "oplen_avg": { 2024-05-10T11:32:44.647 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 42315, 2024-05-10T11:32:44.647 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 42890 2024-05-10T11:32:44.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.648 INFO:teuthology.orchestra.run.smithi076.stdout: "op": 42315, 2024-05-10T11:32:44.648 INFO:teuthology.orchestra.run.smithi076.stdout: "op_r": 730, 2024-05-10T11:32:44.648 INFO:teuthology.orchestra.run.smithi076.stdout: "op_w": 41585, 2024-05-10T11:32:44.648 INFO:teuthology.orchestra.run.smithi076.stdout: "op_rmw": 0, 2024-05-10T11:32:44.648 INFO:teuthology.orchestra.run.smithi076.stdout: "op_pg": 0, 2024-05-10T11:32:44.648 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_stat": 0, 2024-05-10T11:32:44.648 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_create": 10, 2024-05-10T11:32:44.648 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_read": 367, 2024-05-10T11:32:44.648 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_write": 17776, 2024-05-10T11:32:44.649 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_writefull": 208, 2024-05-10T11:32:44.649 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_writesame": 0, 2024-05-10T11:32:44.649 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_append": 0, 2024-05-10T11:32:44.649 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_zero": 0, 2024-05-10T11:32:44.649 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_truncate": 5, 2024-05-10T11:32:44.649 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_delete": 23233, 2024-05-10T11:32:44.649 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_mapext": 0, 2024-05-10T11:32:44.649 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_sparse_read": 0, 2024-05-10T11:32:44.650 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_clonerange": 0, 2024-05-10T11:32:44.650 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_getxattr": 363, 2024-05-10T11:32:44.650 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_setxattr": 20, 2024-05-10T11:32:44.650 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_cmpxattr": 0, 2024-05-10T11:32:44.650 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_rmxattr": 0, 2024-05-10T11:32:44.650 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_resetxattrs": 0, 2024-05-10T11:32:44.650 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_call": 0, 2024-05-10T11:32:44.650 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_watch": 0, 2024-05-10T11:32:44.651 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_notify": 0, 2024-05-10T11:32:44.651 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_src_cmpxattr": 0, 2024-05-10T11:32:44.651 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_pgls": 0, 2024-05-10T11:32:44.651 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_pgls_filter": 0, 2024-05-10T11:32:44.651 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_other": 0, 2024-05-10T11:32:44.651 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_active": 0, 2024-05-10T11:32:44.651 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_send": 0, 2024-05-10T11:32:44.651 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_resend": 0, 2024-05-10T11:32:44.652 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_ping": 0, 2024-05-10T11:32:44.652 INFO:teuthology.orchestra.run.smithi076.stdout: "poolop_active": 0, 2024-05-10T11:32:44.652 INFO:teuthology.orchestra.run.smithi076.stdout: "poolop_send": 0, 2024-05-10T11:32:44.652 INFO:teuthology.orchestra.run.smithi076.stdout: "poolop_resend": 0, 2024-05-10T11:32:44.652 INFO:teuthology.orchestra.run.smithi076.stdout: "poolstat_active": 0, 2024-05-10T11:32:44.652 INFO:teuthology.orchestra.run.smithi076.stdout: "poolstat_send": 0, 2024-05-10T11:32:44.652 INFO:teuthology.orchestra.run.smithi076.stdout: "poolstat_resend": 0, 2024-05-10T11:32:44.652 INFO:teuthology.orchestra.run.smithi076.stdout: "statfs_active": 0, 2024-05-10T11:32:44.652 INFO:teuthology.orchestra.run.smithi076.stdout: "statfs_send": 0, 2024-05-10T11:32:44.653 INFO:teuthology.orchestra.run.smithi076.stdout: "statfs_resend": 0, 2024-05-10T11:32:44.653 INFO:teuthology.orchestra.run.smithi076.stdout: "command_active": 0, 2024-05-10T11:32:44.653 INFO:teuthology.orchestra.run.smithi076.stdout: "command_send": 0, 2024-05-10T11:32:44.653 INFO:teuthology.orchestra.run.smithi076.stdout: "command_resend": 0, 2024-05-10T11:32:44.653 INFO:teuthology.orchestra.run.smithi076.stdout: "map_epoch": 117, 2024-05-10T11:32:44.653 INFO:teuthology.orchestra.run.smithi076.stdout: "map_full": 0, 2024-05-10T11:32:44.653 INFO:teuthology.orchestra.run.smithi076.stdout: "map_inc": 29, 2024-05-10T11:32:44.653 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_sessions": 12, 2024-05-10T11:32:44.654 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_session_open": 16, 2024-05-10T11:32:44.654 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_session_close": 4, 2024-05-10T11:32:44.654 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_laggy": 0, 2024-05-10T11:32:44.654 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_wr": 675, 2024-05-10T11:32:44.654 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_rd": 0, 2024-05-10T11:32:44.654 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_del": 233 2024-05-10T11:32:44.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.654 INFO:teuthology.orchestra.run.smithi076.stdout: "oft": { 2024-05-10T11:32:44.655 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_objs": 1, 2024-05-10T11:32:44.655 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_kv_pairs": 0, 2024-05-10T11:32:44.655 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_updates": 0, 2024-05-10T11:32:44.655 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_removes": 0 2024-05-10T11:32:44.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.655 INFO:teuthology.orchestra.run.smithi076.stdout: "osd-slow-ops": { 2024-05-10T11:32:44.655 INFO:teuthology.orchestra.run.smithi076.stdout: "slow_ops_count": 2 2024-05-10T11:32:44.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.655 INFO:teuthology.orchestra.run.smithi076.stdout: "purge_queue": { 2024-05-10T11:32:44.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing_ops": 0, 2024-05-10T11:32:44.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing_ops_high_water": 10, 2024-05-10T11:32:44.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing": 0, 2024-05-10T11:32:44.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing_high_water": 5, 2024-05-10T11:32:44.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executed_ops": 23144, 2024-05-10T11:32:44.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executed": 11572, 2024-05-10T11:32:44.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_item_in_journal": 0 2024-05-10T11:32:44.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.657 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-10T11:32:44.657 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:44.657 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-05-10T11:32:44.657 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:44.657 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 399178, 2024-05-10T11:32:44.657 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 135935410, 2024-05-10T11:32:44.657 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:44.658 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 399178, 2024-05-10T11:32:44.658 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-05-10T11:32:44.658 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-05-10T11:32:44.658 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 399178, 2024-05-10T11:32:44.658 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 135935410, 2024-05-10T11:32:44.658 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:44.658 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.658 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.659 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.659 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:44.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.659 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-objecter_bytes": { 2024-05-10T11:32:44.659 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:44.659 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-05-10T11:32:44.659 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:44.660 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 0, 2024-05-10T11:32:44.660 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 0, 2024-05-10T11:32:44.660 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:44.660 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:44.660 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 42315, 2024-05-10T11:32:44.660 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 193727138, 2024-05-10T11:32:44.660 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 42315, 2024-05-10T11:32:44.660 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 193727138, 2024-05-10T11:32:44.661 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:44.661 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.661 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.661 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.661 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:44.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.661 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-objecter_ops": { 2024-05-10T11:32:44.661 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:44.662 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 1024, 2024-05-10T11:32:44.662 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:44.662 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 0, 2024-05-10T11:32:44.662 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 0, 2024-05-10T11:32:44.662 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:44.662 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:44.662 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 42315, 2024-05-10T11:32:44.662 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 42315, 2024-05-10T11:32:44.663 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 42315, 2024-05-10T11:32:44.663 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 42315, 2024-05-10T11:32:44.663 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:44.663 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.663 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.663 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.663 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:44.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.664 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-write_buf_throttle": { 2024-05-10T11:32:44.664 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:44.664 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 3758096384, 2024-05-10T11:32:44.664 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:44.664 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 11572, 2024-05-10T11:32:44.664 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 1261996, 2024-05-10T11:32:44.664 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:44.665 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 11572, 2024-05-10T11:32:44.665 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-05-10T11:32:44.665 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-05-10T11:32:44.665 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 367, 2024-05-10T11:32:44.665 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 1261996, 2024-05-10T11:32:44.665 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:44.665 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.665 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.665 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.666 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:44.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:44.666 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-write_buf_throttle-0x55c2229ee500": { 2024-05-10T11:32:44.666 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 574, 2024-05-10T11:32:44.666 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 3758096384, 2024-05-10T11:32:44.666 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:44.666 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 55512, 2024-05-10T11:32:44.667 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 189559297, 2024-05-10T11:32:44.667 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:44.667 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 55512, 2024-05-10T11:32:44.667 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-05-10T11:32:44.667 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-05-10T11:32:44.667 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 17409, 2024-05-10T11:32:44.667 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 189558723, 2024-05-10T11:32:44.667 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:44.668 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:44.668 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:44.668 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:44.668 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:44.668 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:44.668 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-05-10T11:32:44.925 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:44 smithi076 ceph-mon[34152]: pgmap v614: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 964 KiB/s wr, 46 op/s 2024-05-10T11:32:45.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:44 smithi027 ceph-mon[30054]: pgmap v614: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 964 KiB/s wr, 46 op/s 2024-05-10T11:32:45.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:44 smithi189 ceph-mon[34842]: pgmap v614: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 964 KiB/s wr, 46 op/s 2024-05-10T11:32:45.379 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-05-10T11:32:45.379 INFO:tasks.check_counter:Daemon mds.h mds.imported=4 2024-05-10T11:32:45.379 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-05-10T11:32:45.379 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-05-10T11:32:45.379 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-05-10T11:32:45.380 DEBUG:tasks.check_counter:Getting stats from k 2024-05-10T11:32:45.380 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph daemon mds.k perf dump 2024-05-10T11:32:45.619 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.b/config 2024-05-10T11:32:47.044 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:46 smithi076 ceph-mon[34152]: pgmap v615: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1009 KiB/s wr, 55 op/s 2024-05-10T11:32:47.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:46 smithi027 ceph-mon[30054]: pgmap v615: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1009 KiB/s wr, 55 op/s 2024-05-10T11:32:47.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:46 smithi189 ceph-mon[34842]: pgmap v615: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1009 KiB/s wr, 55 op/s 2024-05-10T11:32:47.322 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-05-10T11:32:47.323 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-0": { 2024-05-10T11:32:47.323 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 773, 2024-05-10T11:32:47.323 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 742, 2024-05-10T11:32:47.323 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 6348658, 2024-05-10T11:32:47.323 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 355340, 2024-05-10T11:32:47.323 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 7, 2024-05-10T11:32:47.324 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 2, 2024-05-10T11:32:47.324 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.094163783, 2024-05-10T11:32:47.324 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.027553755, 2024-05-10T11:32:47.324 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.057818176, 2024-05-10T11:32:47.324 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.017332532, 2024-05-10T11:32:47.324 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:47.324 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 742, 2024-05-10T11:32:47.325 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.029633843, 2024-05-10T11:32:47.325 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000039937 2024-05-10T11:32:47.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.325 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:47.325 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.325 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.325 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.326 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 2078160, 2024-05-10T11:32:47.326 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 262160 2024-05-10T11:32:47.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.326 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-1": { 2024-05-10T11:32:47.326 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 1660, 2024-05-10T11:32:47.326 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 1973, 2024-05-10T11:32:47.327 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 21611960, 2024-05-10T11:32:47.327 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 1211025, 2024-05-10T11:32:47.327 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 8, 2024-05-10T11:32:47.327 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 6, 2024-05-10T11:32:47.327 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.148394046, 2024-05-10T11:32:47.327 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.043056939, 2024-05-10T11:32:47.327 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.049212151, 2024-05-10T11:32:47.327 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.042706628, 2024-05-10T11:32:47.328 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:47.328 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 1970, 2024-05-10T11:32:47.328 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.075194478, 2024-05-10T11:32:47.328 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000038169 2024-05-10T11:32:47.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.328 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:47.328 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.328 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.328 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.329 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-10T11:32:47.329 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 683776 2024-05-10T11:32:47.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.329 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-2": { 2024-05-10T11:32:47.329 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 2943, 2024-05-10T11:32:47.329 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 2945, 2024-05-10T11:32:47.329 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 21841208, 2024-05-10T11:32:47.330 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 935932, 2024-05-10T11:32:47.330 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 5, 2024-05-10T11:32:47.330 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 4, 2024-05-10T11:32:47.330 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.240074219, 2024-05-10T11:32:47.330 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.074408384, 2024-05-10T11:32:47.330 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.070057207, 2024-05-10T11:32:47.330 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.075736733, 2024-05-10T11:32:47.330 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:47.331 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 2943, 2024-05-10T11:32:47.331 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.090923744, 2024-05-10T11:32:47.331 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000030894 2024-05-10T11:32:47.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.331 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:47.331 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.331 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.331 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.332 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-10T11:32:47.332 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 0 2024-05-10T11:32:47.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.332 INFO:teuthology.orchestra.run.smithi076.stdout: "cct": { 2024-05-10T11:32:47.332 INFO:teuthology.orchestra.run.smithi076.stdout: "total_workers": 1, 2024-05-10T11:32:47.332 INFO:teuthology.orchestra.run.smithi076.stdout: "unhealthy_workers": 0 2024-05-10T11:32:47.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.332 INFO:teuthology.orchestra.run.smithi076.stdout: "finisher-MDSRank": { 2024-05-10T11:32:47.333 INFO:teuthology.orchestra.run.smithi076.stdout: "queue_len": 0, 2024-05-10T11:32:47.333 INFO:teuthology.orchestra.run.smithi076.stdout: "complete_latency": { 2024-05-10T11:32:47.333 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 8428, 2024-05-10T11:32:47.333 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.183640629, 2024-05-10T11:32:47.333 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000021789 2024-05-10T11:32:47.333 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:47.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.333 INFO:teuthology.orchestra.run.smithi076.stdout: "finisher-PurgeQueue": { 2024-05-10T11:32:47.334 INFO:teuthology.orchestra.run.smithi076.stdout: "queue_len": 0, 2024-05-10T11:32:47.334 INFO:teuthology.orchestra.run.smithi076.stdout: "complete_latency": { 2024-05-10T11:32:47.334 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.334 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.334 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.334 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:47.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.335 INFO:teuthology.orchestra.run.smithi076.stdout: "mds": { 2024-05-10T11:32:47.335 INFO:teuthology.orchestra.run.smithi076.stdout: "request": 0, 2024-05-10T11:32:47.335 INFO:teuthology.orchestra.run.smithi076.stdout: "reply": 0, 2024-05-10T11:32:47.335 INFO:teuthology.orchestra.run.smithi076.stdout: "reply_latency": { 2024-05-10T11:32:47.336 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.336 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.336 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.337 INFO:teuthology.orchestra.run.smithi076.stdout: "slow_reply": 0, 2024-05-10T11:32:47.337 INFO:teuthology.orchestra.run.smithi076.stdout: "forward": 0, 2024-05-10T11:32:47.337 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_fetch_complete": 0, 2024-05-10T11:32:47.337 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_fetch_keys": 0, 2024-05-10T11:32:47.338 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_commit": 0, 2024-05-10T11:32:47.338 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_split": 0, 2024-05-10T11:32:47.338 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_merge": 0, 2024-05-10T11:32:47.339 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes": 4196, 2024-05-10T11:32:47.339 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_top": 2, 2024-05-10T11:32:47.339 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_bottom": 4194, 2024-05-10T11:32:47.339 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_pin_tail": 0, 2024-05-10T11:32:47.340 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_pinned": 4194, 2024-05-10T11:32:47.340 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_expired": 0, 2024-05-10T11:32:47.340 INFO:teuthology.orchestra.run.smithi076.stdout: "inodes_with_caps": 0, 2024-05-10T11:32:47.341 INFO:teuthology.orchestra.run.smithi076.stdout: "caps": 0, 2024-05-10T11:32:47.341 INFO:teuthology.orchestra.run.smithi076.stdout: "subtrees": 21, 2024-05-10T11:32:47.341 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse": 0, 2024-05-10T11:32:47.341 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_hit": 0, 2024-05-10T11:32:47.342 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_forward": 0, 2024-05-10T11:32:47.342 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_discover": 0, 2024-05-10T11:32:47.342 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_dir_fetch": 0, 2024-05-10T11:32:47.342 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_remote_ino": 0, 2024-05-10T11:32:47.343 INFO:teuthology.orchestra.run.smithi076.stdout: "traverse_lock": 0, 2024-05-10T11:32:47.343 INFO:teuthology.orchestra.run.smithi076.stdout: "load_cent": 0, 2024-05-10T11:32:47.343 INFO:teuthology.orchestra.run.smithi076.stdout: "q": 0, 2024-05-10T11:32:47.343 INFO:teuthology.orchestra.run.smithi076.stdout: "exported": 0, 2024-05-10T11:32:47.344 INFO:teuthology.orchestra.run.smithi076.stdout: "exported_inodes": 0, 2024-05-10T11:32:47.344 INFO:teuthology.orchestra.run.smithi076.stdout: "imported": 0, 2024-05-10T11:32:47.344 INFO:teuthology.orchestra.run.smithi076.stdout: "imported_inodes": 0, 2024-05-10T11:32:47.345 INFO:teuthology.orchestra.run.smithi076.stdout: "openino_dir_fetch": 0, 2024-05-10T11:32:47.345 INFO:teuthology.orchestra.run.smithi076.stdout: "openino_backtrace_fetch": 0, 2024-05-10T11:32:47.345 INFO:teuthology.orchestra.run.smithi076.stdout: "openino_peer_discover": 0, 2024-05-10T11:32:47.345 INFO:teuthology.orchestra.run.smithi076.stdout: "root_rfiles": 0, 2024-05-10T11:32:47.345 INFO:teuthology.orchestra.run.smithi076.stdout: "root_rbytes": 0, 2024-05-10T11:32:47.345 INFO:teuthology.orchestra.run.smithi076.stdout: "root_rsnaps": 0, 2024-05-10T11:32:47.346 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_backtrace_fetch": 0, 2024-05-10T11:32:47.346 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_set_tag": 0, 2024-05-10T11:32:47.346 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_backtrace_repaired": 0, 2024-05-10T11:32:47.346 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_inotable_repaired": 0, 2024-05-10T11:32:47.347 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_dir_inodes": 0, 2024-05-10T11:32:47.347 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_dir_base_inodes": 0, 2024-05-10T11:32:47.347 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_dirfrag_rstats": 0, 2024-05-10T11:32:47.348 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_file_inodes": 0, 2024-05-10T11:32:47.348 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_inode_file_caps": 0, 2024-05-10T11:32:47.348 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_revoke": 0, 2024-05-10T11:32:47.348 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_grant": 0, 2024-05-10T11:32:47.349 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_trunc": 0, 2024-05-10T11:32:47.349 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-10T11:32:47.349 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-10T11:32:47.349 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_caps": 0, 2024-05-10T11:32:47.350 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_caps_dirty": 0, 2024-05-10T11:32:47.350 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_cap_release": 0, 2024-05-10T11:32:47.350 INFO:teuthology.orchestra.run.smithi076.stdout: "process_request_cap_release": 0 2024-05-10T11:32:47.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.351 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache": { 2024-05-10T11:32:47.351 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_update": 0, 2024-05-10T11:32:47.351 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_update_receipt": 0, 2024-05-10T11:32:47.351 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_try_discover": 0, 2024-05-10T11:32:47.352 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_send_discover": 0, 2024-05-10T11:32:47.352 INFO:teuthology.orchestra.run.smithi076.stdout: "dir_handle_discover": 0, 2024-05-10T11:32:47.352 INFO:teuthology.orchestra.run.smithi076.stdout: "num_strays": 0, 2024-05-10T11:32:47.353 INFO:teuthology.orchestra.run.smithi076.stdout: "num_strays_delayed": 0, 2024-05-10T11:32:47.353 INFO:teuthology.orchestra.run.smithi076.stdout: "num_strays_enqueuing": 0, 2024-05-10T11:32:47.353 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_created": 0, 2024-05-10T11:32:47.353 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_enqueued": 0, 2024-05-10T11:32:47.354 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_reintegrated": 0, 2024-05-10T11:32:47.354 INFO:teuthology.orchestra.run.smithi076.stdout: "strays_migrated": 0, 2024-05-10T11:32:47.354 INFO:teuthology.orchestra.run.smithi076.stdout: "num_recovering_processing": 0, 2024-05-10T11:32:47.354 INFO:teuthology.orchestra.run.smithi076.stdout: "num_recovering_enqueued": 0, 2024-05-10T11:32:47.355 INFO:teuthology.orchestra.run.smithi076.stdout: "num_recovering_prioritized": 0, 2024-05-10T11:32:47.355 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_started": 0, 2024-05-10T11:32:47.355 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_completed": 0, 2024-05-10T11:32:47.355 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_quiesce_path": 0, 2024-05-10T11:32:47.356 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_quiesce_inode": 0, 2024-05-10T11:32:47.356 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_enqueue_scrub": 0, 2024-05-10T11:32:47.356 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_exportdir": 0, 2024-05-10T11:32:47.356 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_flush": 0, 2024-05-10T11:32:47.357 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_fragmentdir": 0, 2024-05-10T11:32:47.357 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_fragstats": 0, 2024-05-10T11:32:47.357 INFO:teuthology.orchestra.run.smithi076.stdout: "ireq_inodestats": 0 2024-05-10T11:32:47.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.358 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log": { 2024-05-10T11:32:47.358 INFO:teuthology.orchestra.run.smithi076.stdout: "evlrg": 0, 2024-05-10T11:32:47.358 INFO:teuthology.orchestra.run.smithi076.stdout: "evadd": 0, 2024-05-10T11:32:47.359 INFO:teuthology.orchestra.run.smithi076.stdout: "evex": 0, 2024-05-10T11:32:47.359 INFO:teuthology.orchestra.run.smithi076.stdout: "evtrm": 0, 2024-05-10T11:32:47.359 INFO:teuthology.orchestra.run.smithi076.stdout: "ev": 25947, 2024-05-10T11:32:47.359 INFO:teuthology.orchestra.run.smithi076.stdout: "evexg": 0, 2024-05-10T11:32:47.359 INFO:teuthology.orchestra.run.smithi076.stdout: "evexd": 0, 2024-05-10T11:32:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "segadd": 0, 2024-05-10T11:32:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "segex": 0, 2024-05-10T11:32:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "segtrm": 0, 2024-05-10T11:32:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "seg": 36, 2024-05-10T11:32:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "segmjr": 4, 2024-05-10T11:32:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "segexg": 0, 2024-05-10T11:32:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "segexd": 0, 2024-05-10T11:32:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "expos": 4194304, 2024-05-10T11:32:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "wrpos": 50553458, 2024-05-10T11:32:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "rdpos": 50552859, 2024-05-10T11:32:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "jlat": { 2024-05-10T11:32:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: "replayed": 25947 2024-05-10T11:32:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_mem": { 2024-05-10T11:32:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: "ino": 389, 2024-05-10T11:32:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: "ino+": 4175, 2024-05-10T11:32:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "ino-": 3786, 2024-05-10T11:32:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "dir": 46, 2024-05-10T11:32:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "dir+": 49, 2024-05-10T11:32:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "dir-": 3, 2024-05-10T11:32:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "dn": 4196, 2024-05-10T11:32:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "dn+": 4229, 2024-05-10T11:32:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "dn-": 33, 2024-05-10T11:32:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "cap": 0, 2024-05-10T11:32:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "cap+": 0, 2024-05-10T11:32:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "cap-": 0, 2024-05-10T11:32:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "rss": 61652, 2024-05-10T11:32:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "heap": 215416 2024-05-10T11:32:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_server": { 2024-05-10T11:32:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "dispatch_client_request": 0, 2024-05-10T11:32:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "dispatch_server_request": 0, 2024-05-10T11:32:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_request": 0, 2024-05-10T11:32:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_client_session": 0, 2024-05-10T11:32:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "handle_peer_request": 0, 2024-05-10T11:32:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "req_create_latency": { 2024-05-10T11:32:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "req_getattr_latency": { 2024-05-10T11:32:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "req_getfilelock_latency": { 2024-05-10T11:32:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "req_link_latency": { 2024-05-10T11:32:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookup_latency": { 2024-05-10T11:32:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookuphash_latency": { 2024-05-10T11:32:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupino_latency": { 2024-05-10T11:32:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupname_latency": { 2024-05-10T11:32:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupparent_latency": { 2024-05-10T11:32:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lookupsnap_latency": { 2024-05-10T11:32:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "req_lssnap_latency": { 2024-05-10T11:32:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "req_mkdir_latency": { 2024-05-10T11:32:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "req_mknod_latency": { 2024-05-10T11:32:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "req_mksnap_latency": { 2024-05-10T11:32:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "req_open_latency": { 2024-05-10T11:32:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "req_readdir_latency": { 2024-05-10T11:32:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rename_latency": { 2024-05-10T11:32:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "req_renamesnap_latency": { 2024-05-10T11:32:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "req_snapdiff_latency": { 2024-05-10T11:32:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rmdir_latency": { 2024-05-10T11:32:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rmsnap_latency": { 2024-05-10T11:32:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "req_rmxattr_latency": { 2024-05-10T11:32:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setattr_latency": { 2024-05-10T11:32:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setdirlayout_latency": { 2024-05-10T11:32:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setfilelock_latency": { 2024-05-10T11:32:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setlayout_latency": { 2024-05-10T11:32:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "req_setxattr_latency": { 2024-05-10T11:32:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "req_symlink_latency": { 2024-05-10T11:32:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "req_unlink_latency": { 2024-05-10T11:32:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "cap_revoke_eviction": 0, 2024-05-10T11:32:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "cap_acquisition_throttle": 0, 2024-05-10T11:32:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "req_getvxattr_latency": { 2024-05-10T11:32:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_sessions": { 2024-05-10T11:32:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "session_count": 2, 2024-05-10T11:32:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "session_add": 5, 2024-05-10T11:32:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "session_remove": 3, 2024-05-10T11:32:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "sessions_open": 4, 2024-05-10T11:32:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "sessions_stale": 0, 2024-05-10T11:32:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "total_load": 0, 2024-05-10T11:32:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "average_load": 0, 2024-05-10T11:32:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "avg_session_uptime": 4888, 2024-05-10T11:32:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "mdthresh_evicted": 0 2024-05-10T11:32:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool": { 2024-05-10T11:32:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_bytes": 0, 2024-05-10T11:32:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_items": 0, 2024-05-10T11:32:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_bytes": 0, 2024-05-10T11:32:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_items": 0, 2024-05-10T11:32:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_bytes": 0, 2024-05-10T11:32:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_items": 0, 2024-05-10T11:32:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-10T11:32:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_items": 0, 2024-05-10T11:32:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-10T11:32:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_items": 0, 2024-05-10T11:32:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_bytes": 0, 2024-05-10T11:32:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_items": 0, 2024-05-10T11:32:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-10T11:32:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_items": 0, 2024-05-10T11:32:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_bytes": 0, 2024-05-10T11:32:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_items": 0, 2024-05-10T11:32:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_bytes": 0, 2024-05-10T11:32:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_items": 0, 2024-05-10T11:32:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-10T11:32:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_items": 0, 2024-05-10T11:32:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-10T11:32:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_items": 0, 2024-05-10T11:32:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_bytes": 0, 2024-05-10T11:32:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_items": 0, 2024-05-10T11:32:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_bytes": 0, 2024-05-10T11:32:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_items": 0, 2024-05-10T11:32:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-10T11:32:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_items": 0, 2024-05-10T11:32:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_bytes": 0, 2024-05-10T11:32:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_items": 0, 2024-05-10T11:32:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_bytes": 0, 2024-05-10T11:32:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_items": 0, 2024-05-10T11:32:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_bytes": 0, 2024-05-10T11:32:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_items": 0, 2024-05-10T11:32:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_bytes": 0, 2024-05-10T11:32:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_items": 0, 2024-05-10T11:32:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_bytes": 231370, 2024-05-10T11:32:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_items": 447, 2024-05-10T11:32:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_bytes": 0, 2024-05-10T11:32:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_items": 0, 2024-05-10T11:32:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bytes": 0, 2024-05-10T11:32:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_items": 0, 2024-05-10T11:32:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_bytes": 0, 2024-05-10T11:32:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_items": 0, 2024-05-10T11:32:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_bytes": 0, 2024-05-10T11:32:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_items": 0, 2024-05-10T11:32:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_bytes": 9808, 2024-05-10T11:32:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_items": 189, 2024-05-10T11:32:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_bytes": 0, 2024-05-10T11:32:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_items": 0, 2024-05-10T11:32:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_bytes": 0, 2024-05-10T11:32:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_items": 0, 2024-05-10T11:32:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_bytes": 3998355, 2024-05-10T11:32:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_items": 26102, 2024-05-10T11:32:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_bytes": 0, 2024-05-10T11:32:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_items": 0, 2024-05-10T11:32:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_bytes": 0, 2024-05-10T11:32:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_items": 0 2024-05-10T11:32:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter": { 2024-05-10T11:32:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "op_active": 0, 2024-05-10T11:32:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "op_laggy": 0, 2024-05-10T11:32:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "op_send": 4894, 2024-05-10T11:32:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "op_send_bytes": 21, 2024-05-10T11:32:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "op_resend": 3, 2024-05-10T11:32:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "op_reply": 4894, 2024-05-10T11:32:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "op_latency": { 2024-05-10T11:32:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 4894, 2024-05-10T11:32:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 5.763832950, 2024-05-10T11:32:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.001177734 2024-05-10T11:32:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "op_inflight": 0, 2024-05-10T11:32:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "oplen_avg": { 2024-05-10T11:32:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 4894, 2024-05-10T11:32:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 4895 2024-05-10T11:32:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "op": 4894, 2024-05-10T11:32:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "op_r": 4894, 2024-05-10T11:32:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "op_w": 0, 2024-05-10T11:32:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "op_rmw": 0, 2024-05-10T11:32:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "op_pg": 0, 2024-05-10T11:32:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_stat": 2713, 2024-05-10T11:32:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_create": 0, 2024-05-10T11:32:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_read": 2179, 2024-05-10T11:32:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_write": 0, 2024-05-10T11:32:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_writefull": 0, 2024-05-10T11:32:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_writesame": 0, 2024-05-10T11:32:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_append": 0, 2024-05-10T11:32:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_zero": 0, 2024-05-10T11:32:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_truncate": 0, 2024-05-10T11:32:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_delete": 0, 2024-05-10T11:32:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_mapext": 0, 2024-05-10T11:32:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_sparse_read": 0, 2024-05-10T11:32:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_clonerange": 0, 2024-05-10T11:32:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_getxattr": 1, 2024-05-10T11:32:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_setxattr": 0, 2024-05-10T11:32:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_cmpxattr": 0, 2024-05-10T11:32:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_rmxattr": 0, 2024-05-10T11:32:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_resetxattrs": 0, 2024-05-10T11:32:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_call": 0, 2024-05-10T11:32:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_watch": 0, 2024-05-10T11:32:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_notify": 0, 2024-05-10T11:32:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_src_cmpxattr": 0, 2024-05-10T11:32:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_pgls": 0, 2024-05-10T11:32:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_pgls_filter": 0, 2024-05-10T11:32:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "osdop_other": 0, 2024-05-10T11:32:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_active": 0, 2024-05-10T11:32:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_send": 0, 2024-05-10T11:32:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_resend": 0, 2024-05-10T11:32:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "linger_ping": 0, 2024-05-10T11:32:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "poolop_active": 0, 2024-05-10T11:32:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "poolop_send": 0, 2024-05-10T11:32:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "poolop_resend": 0, 2024-05-10T11:32:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "poolstat_active": 0, 2024-05-10T11:32:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "poolstat_send": 0, 2024-05-10T11:32:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "poolstat_resend": 0, 2024-05-10T11:32:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "statfs_active": 0, 2024-05-10T11:32:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "statfs_send": 0, 2024-05-10T11:32:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: "statfs_resend": 0, 2024-05-10T11:32:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: "command_active": 0, 2024-05-10T11:32:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: "command_send": 0, 2024-05-10T11:32:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: "command_resend": 0, 2024-05-10T11:32:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: "map_epoch": 117, 2024-05-10T11:32:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: "map_full": 0, 2024-05-10T11:32:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "map_inc": 29, 2024-05-10T11:32:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_sessions": 8, 2024-05-10T11:32:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_session_open": 9, 2024-05-10T11:32:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_session_close": 1, 2024-05-10T11:32:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_laggy": 0, 2024-05-10T11:32:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_wr": 0, 2024-05-10T11:32:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_rd": 2, 2024-05-10T11:32:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_del": 0 2024-05-10T11:32:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "oft": { 2024-05-10T11:32:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_objs": 0, 2024-05-10T11:32:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_kv_pairs": 0, 2024-05-10T11:32:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_updates": 0, 2024-05-10T11:32:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "omap_total_removes": 0 2024-05-10T11:32:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "osd-slow-ops": { 2024-05-10T11:32:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "slow_ops_count": 0 2024-05-10T11:32:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "purge_queue": { 2024-05-10T11:32:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing_ops": 0, 2024-05-10T11:32:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing_ops_high_water": 0, 2024-05-10T11:32:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing": 0, 2024-05-10T11:32:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executing_high_water": 0, 2024-05-10T11:32:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executed_ops": 0, 2024-05-10T11:32:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_executed": 0, 2024-05-10T11:32:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "pq_item_in_journal": 0 2024-05-10T11:32:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-10T11:32:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-05-10T11:32:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 5372, 2024-05-10T11:32:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 49291020, 2024-05-10T11:32:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 5372, 2024-05-10T11:32:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-05-10T11:32:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-05-10T11:32:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 5372, 2024-05-10T11:32:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 49291020, 2024-05-10T11:32:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-objecter_bytes": { 2024-05-10T11:32:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-05-10T11:32:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 0, 2024-05-10T11:32:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 0, 2024-05-10T11:32:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 4894, 2024-05-10T11:32:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 46368182, 2024-05-10T11:32:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 4894, 2024-05-10T11:32:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 46368182, 2024-05-10T11:32:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-objecter_ops": { 2024-05-10T11:32:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 1024, 2024-05-10T11:32:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 0, 2024-05-10T11:32:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 0, 2024-05-10T11:32:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 4894, 2024-05-10T11:32:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 4894, 2024-05-10T11:32:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 4894, 2024-05-10T11:32:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 4894, 2024-05-10T11:32:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-write_buf_throttle": { 2024-05-10T11:32:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 3758096384, 2024-05-10T11:32:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 0, 2024-05-10T11:32:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 0, 2024-05-10T11:32:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-05-10T11:32:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-05-10T11:32:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 0, 2024-05-10T11:32:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 0, 2024-05-10T11:32:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-05-10T11:32:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-write_buf_throttle-0x55c44618a6e0": { 2024-05-10T11:32:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-05-10T11:32:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 3758096384, 2024-05-10T11:32:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-05-10T11:32:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 0, 2024-05-10T11:32:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 0, 2024-05-10T11:32:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-05-10T11:32:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-05-10T11:32:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 0, 2024-05-10T11:32:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 0, 2024-05-10T11:32:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-05-10T11:32:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-05-10T11:32:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-05-10T11:32:47.423 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-05-10T11:32:47.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:47.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-05-10T11:32:47.423 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-05-10T11:32:48.318 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-05-10T11:32:48.319 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-05-10T11:32:48.319 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-05-10T11:32:48.319 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-05-10T11:32:48.319 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-05-10T11:32:48.319 DEBUG:tasks.check_counter:Getting stats from c 2024-05-10T11:32:48.319 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph daemon mds.c perf dump 2024-05-10T11:32:48.795 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:32:49.100 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:48 smithi189 ceph-mon[34842]: pgmap v616: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 102 KiB/s wr, 20 op/s 2024-05-10T11:32:49.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:48 smithi027 ceph-mon[30054]: pgmap v616: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 102 KiB/s wr, 20 op/s 2024-05-10T11:32:49.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:48 smithi076 ceph-mon[34152]: pgmap v616: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 102 KiB/s wr, 20 op/s 2024-05-10T11:32:50.729 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-05-10T11:32:50.730 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-05-10T11:32:50.730 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 246002, 2024-05-10T11:32:50.730 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 167460, 2024-05-10T11:32:50.730 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 119720997, 2024-05-10T11:32:50.730 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 51738899, 2024-05-10T11:32:50.730 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 10, 2024-05-10T11:32:50.731 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 4, 2024-05-10T11:32:50.731 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 10.373014605, 2024-05-10T11:32:50.731 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 4.008536619, 2024-05-10T11:32:50.731 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 8.133128358, 2024-05-10T11:32:50.731 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.283546705, 2024-05-10T11:32:50.731 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:50.731 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 167462, 2024-05-10T11:32:50.732 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 2.561332710, 2024-05-10T11:32:50.732 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000015295 2024-05-10T11:32:50.732 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.732 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:50.732 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 394513, 2024-05-10T11:32:50.733 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.049373167, 2024-05-10T11:32:50.733 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000125 2024-05-10T11:32:50.733 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.733 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-10T11:32:50.733 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 0 2024-05-10T11:32:50.733 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.734 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-05-10T11:32:50.734 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 79080, 2024-05-10T11:32:50.734 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 127642, 2024-05-10T11:32:50.734 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 25187012, 2024-05-10T11:32:50.734 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 119944172, 2024-05-10T11:32:50.735 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 17, 2024-05-10T11:32:50.735 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 7, 2024-05-10T11:32:50.735 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 4.956191849, 2024-05-10T11:32:50.735 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 2.320572800, 2024-05-10T11:32:50.735 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 2.679391766, 2024-05-10T11:32:50.735 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.439490816, 2024-05-10T11:32:50.736 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:50.736 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 127640, 2024-05-10T11:32:50.736 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 2.527676807, 2024-05-10T11:32:50.736 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000019803 2024-05-10T11:32:50.736 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.736 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:50.736 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 94362, 2024-05-10T11:32:50.736 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.029761378, 2024-05-10T11:32:50.737 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000315 2024-05-10T11:32:50.737 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.737 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 4992, 2024-05-10T11:32:50.737 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 257616 2024-05-10T11:32:50.737 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.737 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-05-10T11:32:50.737 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 21401, 2024-05-10T11:32:50.737 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 15307, 2024-05-10T11:32:50.738 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 7999997, 2024-05-10T11:32:50.738 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 21897120, 2024-05-10T11:32:50.738 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 42, 2024-05-10T11:32:50.738 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 10, 2024-05-10T11:32:50.738 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 1.265551548, 2024-05-10T11:32:50.738 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.439265026, 2024-05-10T11:32:50.738 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.752276201, 2024-05-10T11:32:50.738 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.207332920, 2024-05-10T11:32:50.739 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:50.739 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 15302, 2024-05-10T11:32:50.739 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.380199236, 2024-05-10T11:32:50.739 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000024846 2024-05-10T11:32:50.739 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.739 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:50.739 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 19154, 2024-05-10T11:32:50.739 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.003002942, 2024-05-10T11:32:50.740 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000156 2024-05-10T11:32:50.740 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.740 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2097728, 2024-05-10T11:32:50.740 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 790144 2024-05-10T11:32:50.740 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.740 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-05-10T11:32:50.740 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-05-10T11:32:50.740 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-05-10T11:32:50.741 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.741 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-05-10T11:32:50.741 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-10T11:32:50.741 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-10T11:32:50.741 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 22411, 2024-05-10T11:32:50.741 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 12.832749383, 2024-05-10T11:32:50.741 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000572609 2024-05-10T11:32:50.741 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:50.742 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.742 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-05-10T11:32:50.742 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-10T11:32:50.742 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-10T11:32:50.742 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 20596, 2024-05-10T11:32:50.742 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1.149672869, 2024-05-10T11:32:50.742 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000055820 2024-05-10T11:32:50.743 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:50.743 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.743 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-05-10T11:32:50.743 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 52903, 2024-05-10T11:32:50.743 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 48213, 2024-05-10T11:32:50.743 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-05-10T11:32:50.743 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 48213, 2024-05-10T11:32:50.743 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 81.331491941, 2024-05-10T11:32:50.743 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001686920 2024-05-10T11:32:50.744 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.744 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-05-10T11:32:50.744 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 4690, 2024-05-10T11:32:50.744 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-05-10T11:32:50.744 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-05-10T11:32:50.744 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 11, 2024-05-10T11:32:50.744 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-05-10T11:32:50.744 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-05-10T11:32:50.745 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 186, 2024-05-10T11:32:50.745 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 33, 2024-05-10T11:32:50.745 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 140, 2024-05-10T11:32:50.745 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 13, 2024-05-10T11:32:50.745 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 139, 2024-05-10T11:32:50.745 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 35, 2024-05-10T11:32:50.745 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 16, 2024-05-10T11:32:50.745 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 16, 2024-05-10T11:32:50.745 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 7, 2024-05-10T11:32:50.746 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 110450, 2024-05-10T11:32:50.746 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 39092, 2024-05-10T11:32:50.746 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 4690, 2024-05-10T11:32:50.746 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 20, 2024-05-10T11:32:50.746 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-05-10T11:32:50.746 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-05-10T11:32:50.746 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 139, 2024-05-10T11:32:50.746 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-05-10T11:32:50.747 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-05-10T11:32:50.747 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 1, 2024-05-10T11:32:50.747 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 62, 2024-05-10T11:32:50.747 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 4, 2024-05-10T11:32:50.747 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 113, 2024-05-10T11:32:50.747 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-05-10T11:32:50.747 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 79, 2024-05-10T11:32:50.747 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-05-10T11:32:50.748 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 699, 2024-05-10T11:32:50.748 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 217420798, 2024-05-10T11:32:50.748 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 6, 2024-05-10T11:32:50.748 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 79, 2024-05-10T11:32:50.748 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-05-10T11:32:50.748 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-05-10T11:32:50.748 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-05-10T11:32:50.748 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-05-10T11:32:50.749 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-05-10T11:32:50.749 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-05-10T11:32:50.749 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 79, 2024-05-10T11:32:50.749 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-05-10T11:32:50.749 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 503, 2024-05-10T11:32:50.749 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 30666, 2024-05-10T11:32:50.749 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-05-10T11:32:50.749 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-10T11:32:50.749 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-10T11:32:50.750 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 2214, 2024-05-10T11:32:50.750 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 582, 2024-05-10T11:32:50.750 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 539, 2024-05-10T11:32:50.750 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 47376 2024-05-10T11:32:50.750 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.750 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-05-10T11:32:50.750 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 143, 2024-05-10T11:32:50.751 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 39, 2024-05-10T11:32:50.751 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 26, 2024-05-10T11:32:50.751 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 40, 2024-05-10T11:32:50.751 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 163, 2024-05-10T11:32:50.751 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-05-10T11:32:50.751 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-05-10T11:32:50.751 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-05-10T11:32:50.751 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 9704, 2024-05-10T11:32:50.751 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 9704, 2024-05-10T11:32:50.752 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-05-10T11:32:50.752 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-05-10T11:32:50.752 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-05-10T11:32:50.752 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-05-10T11:32:50.752 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-05-10T11:32:50.752 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-05-10T11:32:50.752 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-05-10T11:32:50.752 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-05-10T11:32:50.753 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-05-10T11:32:50.753 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-05-10T11:32:50.753 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 2, 2024-05-10T11:32:50.753 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-05-10T11:32:50.753 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-05-10T11:32:50.753 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-05-10T11:32:50.753 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-05-10T11:32:50.753 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.754 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-05-10T11:32:50.754 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-05-10T11:32:50.754 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 41044, 2024-05-10T11:32:50.754 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-05-10T11:32:50.754 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-05-10T11:32:50.754 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 41044, 2024-05-10T11:32:50.754 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-05-10T11:32:50.755 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-05-10T11:32:50.755 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 50, 2024-05-10T11:32:50.755 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-05-10T11:32:50.755 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-05-10T11:32:50.755 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 50, 2024-05-10T11:32:50.755 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 5, 2024-05-10T11:32:50.755 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-05-10T11:32:50.755 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-05-10T11:32:50.755 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-05-10T11:32:50.756 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 78525343, 2024-05-10T11:32:50.756 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 0, 2024-05-10T11:32:50.756 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-05-10T11:32:50.756 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 10778, 2024-05-10T11:32:50.756 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 43.387364607, 2024-05-10T11:32:50.756 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.004025548 2024-05-10T11:32:50.756 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.756 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 0 2024-05-10T11:32:50.757 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.757 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-05-10T11:32:50.757 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 133, 2024-05-10T11:32:50.757 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 9866, 2024-05-10T11:32:50.757 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 9733, 2024-05-10T11:32:50.757 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 59, 2024-05-10T11:32:50.757 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 105, 2024-05-10T11:32:50.758 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 46, 2024-05-10T11:32:50.758 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 186, 2024-05-10T11:32:50.758 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 14929, 2024-05-10T11:32:50.758 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 14743, 2024-05-10T11:32:50.758 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 16, 2024-05-10T11:32:50.758 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 9898, 2024-05-10T11:32:50.758 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 9882, 2024-05-10T11:32:50.758 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 69128, 2024-05-10T11:32:50.759 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 223608 2024-05-10T11:32:50.759 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.759 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-05-10T11:32:50.759 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 99248, 2024-05-10T11:32:50.759 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-05-10T11:32:50.759 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 52903, 2024-05-10T11:32:50.759 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 157, 2024-05-10T11:32:50.759 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 19586, 2024-05-10T11:32:50.760 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-05-10T11:32:50.760 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 9708, 2024-05-10T11:32:50.760 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 38.284448131, 2024-05-10T11:32:50.760 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.003943597 2024-05-10T11:32:50.760 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.760 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-05-10T11:32:50.760 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 145, 2024-05-10T11:32:50.760 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.105899665, 2024-05-10T11:32:50.760 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000730342 2024-05-10T11:32:50.761 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.761 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-05-10T11:32:50.761 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.761 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.761 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.761 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.761 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-05-10T11:32:50.761 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.761 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.762 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.762 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.762 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-05-10T11:32:50.762 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 10564, 2024-05-10T11:32:50.762 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 3.665878604, 2024-05-10T11:32:50.762 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000347016 2024-05-10T11:32:50.762 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.762 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-05-10T11:32:50.763 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.763 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.763 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.763 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.763 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-05-10T11:32:50.763 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.763 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.763 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.764 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.764 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-05-10T11:32:50.764 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.764 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.764 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.764 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.764 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-05-10T11:32:50.764 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.764 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.765 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.765 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.765 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-05-10T11:32:50.765 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.765 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.765 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.765 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.766 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-05-10T11:32:50.766 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.770 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.770 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.770 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.770 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-05-10T11:32:50.771 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.771 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.771 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.771 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.771 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-05-10T11:32:50.771 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.771 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.771 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.772 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.772 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-05-10T11:32:50.772 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.772 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.772 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.772 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.773 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-05-10T11:32:50.773 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.773 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.773 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.773 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.773 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-05-10T11:32:50.773 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 17845, 2024-05-10T11:32:50.773 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 7.721532090, 2024-05-10T11:32:50.774 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000432700 2024-05-10T11:32:50.774 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.774 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-05-10T11:32:50.774 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 139, 2024-05-10T11:32:50.774 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 2.132297058, 2024-05-10T11:32:50.774 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.015340266 2024-05-10T11:32:50.774 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.774 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-05-10T11:32:50.775 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.775 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.775 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.775 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.775 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-05-10T11:32:50.775 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.775 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.775 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.775 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.776 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-05-10T11:32:50.776 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.776 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.776 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.776 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.776 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-05-10T11:32:50.776 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.776 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.777 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.777 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.777 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-05-10T11:32:50.777 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.777 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.777 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.777 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.777 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-05-10T11:32:50.777 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.778 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.778 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.778 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.778 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-05-10T11:32:50.778 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.778 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.778 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.778 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.779 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-05-10T11:32:50.779 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 108, 2024-05-10T11:32:50.779 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.073770656, 2024-05-10T11:32:50.779 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000683061 2024-05-10T11:32:50.779 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.779 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-05-10T11:32:50.779 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.779 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.779 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.780 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.780 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-05-10T11:32:50.780 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.780 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.780 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.780 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.780 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-05-10T11:32:50.780 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.781 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.781 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.781 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.781 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-05-10T11:32:50.781 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 9704, 2024-05-10T11:32:50.781 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 29.347665737, 2024-05-10T11:32:50.781 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.003024285 2024-05-10T11:32:50.781 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.781 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-05-10T11:32:50.782 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-05-10T11:32:50.782 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-05-10T11:32:50.782 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.782 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.782 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.782 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:50.782 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.782 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-05-10T11:32:50.783 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-05-10T11:32:50.783 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 2, 2024-05-10T11:32:50.783 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 1, 2024-05-10T11:32:50.783 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 1, 2024-05-10T11:32:50.783 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-05-10T11:32:50.783 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 1781, 2024-05-10T11:32:50.783 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 1781, 2024-05-10T11:32:50.783 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 3038, 2024-05-10T11:32:50.784 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-05-10T11:32:50.784 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.784 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-05-10T11:32:50.784 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-05-10T11:32:50.784 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-05-10T11:32:50.784 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-05-10T11:32:50.784 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-05-10T11:32:50.785 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-05-10T11:32:50.785 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-05-10T11:32:50.785 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-10T11:32:50.785 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-05-10T11:32:50.785 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-10T11:32:50.785 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-05-10T11:32:50.785 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-05-10T11:32:50.785 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-05-10T11:32:50.786 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-10T11:32:50.786 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-05-10T11:32:50.786 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-05-10T11:32:50.786 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-05-10T11:32:50.786 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-05-10T11:32:50.786 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-05-10T11:32:50.786 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-10T11:32:50.786 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-05-10T11:32:50.787 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-10T11:32:50.787 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-05-10T11:32:50.787 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-05-10T11:32:50.787 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-05-10T11:32:50.787 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-05-10T11:32:50.787 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-05-10T11:32:50.787 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-10T11:32:50.787 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-05-10T11:32:50.788 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-05-10T11:32:50.788 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-05-10T11:32:50.788 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-05-10T11:32:50.788 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-05-10T11:32:50.788 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-05-10T11:32:50.788 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-05-10T11:32:50.788 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-05-10T11:32:50.788 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-05-10T11:32:50.788 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 326179, 2024-05-10T11:32:50.789 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 246, 2024-05-10T11:32:50.789 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 88, 2024-05-10T11:32:50.789 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 1, 2024-05-10T11:32:50.789 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-05-10T11:32:50.789 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-05-10T11:32:50.789 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-05-10T11:32:50.789 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-05-10T11:32:50.789 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-05-10T11:32:50.790 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-05-10T11:32:50.790 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 9808, 2024-05-10T11:32:50.790 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 189, 2024-05-10T11:32:50.790 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-05-10T11:32:50.790 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-05-10T11:32:50.790 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-05-10T11:32:50.790 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-05-10T11:32:50.790 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 669966, 2024-05-10T11:32:50.791 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 4785, 2024-05-10T11:32:50.791 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-05-10T11:32:50.791 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-05-10T11:32:50.791 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-05-10T11:32:50.791 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-05-10T11:32:50.791 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.791 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-05-10T11:32:50.791 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-05-10T11:32:50.792 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-05-10T11:32:50.792 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 32991, 2024-05-10T11:32:50.792 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 76290903, 2024-05-10T11:32:50.792 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 33, 2024-05-10T11:32:50.792 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 32991, 2024-05-10T11:32:50.792 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-05-10T11:32:50.792 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 32991, 2024-05-10T11:32:50.792 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 157.776849181, 2024-05-10T11:32:50.792 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.004782420 2024-05-10T11:32:50.793 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.793 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-05-10T11:32:50.793 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-05-10T11:32:50.793 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 32991, 2024-05-10T11:32:50.793 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 34075 2024-05-10T11:32:50.793 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.793 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 32991, 2024-05-10T11:32:50.794 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 826, 2024-05-10T11:32:50.794 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 32165, 2024-05-10T11:32:50.794 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-05-10T11:32:50.794 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-05-10T11:32:50.794 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 0, 2024-05-10T11:32:50.794 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 10, 2024-05-10T11:32:50.794 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 747, 2024-05-10T11:32:50.794 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 11525, 2024-05-10T11:32:50.794 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 466, 2024-05-10T11:32:50.795 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-05-10T11:32:50.795 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-05-10T11:32:50.795 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-05-10T11:32:50.795 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-05-10T11:32:50.795 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 19437, 2024-05-10T11:32:50.795 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-05-10T11:32:50.795 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-05-10T11:32:50.795 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-05-10T11:32:50.796 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 79, 2024-05-10T11:32:50.796 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 20, 2024-05-10T11:32:50.796 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-05-10T11:32:50.796 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-05-10T11:32:50.796 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-05-10T11:32:50.796 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-05-10T11:32:50.796 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-05-10T11:32:50.796 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-05-10T11:32:50.797 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-05-10T11:32:50.797 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-05-10T11:32:50.797 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-05-10T11:32:50.797 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-05-10T11:32:50.797 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-05-10T11:32:50.797 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-05-10T11:32:50.797 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-05-10T11:32:50.797 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-05-10T11:32:50.798 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-05-10T11:32:50.798 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-05-10T11:32:50.798 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-05-10T11:32:50.798 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-05-10T11:32:50.798 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-05-10T11:32:50.798 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-05-10T11:32:50.798 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-05-10T11:32:50.798 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-05-10T11:32:50.799 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-05-10T11:32:50.799 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-05-10T11:32:50.799 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-05-10T11:32:50.799 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-05-10T11:32:50.799 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 117, 2024-05-10T11:32:50.799 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-05-10T11:32:50.799 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 29, 2024-05-10T11:32:50.799 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 12, 2024-05-10T11:32:50.800 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 12, 2024-05-10T11:32:50.800 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-05-10T11:32:50.800 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-05-10T11:32:50.800 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 1336, 2024-05-10T11:32:50.800 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 0, 2024-05-10T11:32:50.800 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 455 2024-05-10T11:32:50.800 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.800 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-05-10T11:32:50.800 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 1, 2024-05-10T11:32:50.801 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-05-10T11:32:50.801 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-05-10T11:32:50.801 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-05-10T11:32:50.801 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.801 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-05-10T11:32:50.801 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-05-10T11:32:50.801 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.801 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-05-10T11:32:50.802 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-05-10T11:32:50.802 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 10, 2024-05-10T11:32:50.802 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-05-10T11:32:50.802 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 5, 2024-05-10T11:32:50.802 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 19408, 2024-05-10T11:32:50.802 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 9704, 2024-05-10T11:32:50.802 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-05-10T11:32:50.802 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.803 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-10T11:32:50.803 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:50.803 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-10T11:32:50.803 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:50.803 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 346479, 2024-05-10T11:32:50.803 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 122702817, 2024-05-10T11:32:50.803 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:50.803 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 346479, 2024-05-10T11:32:50.803 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-10T11:32:50.804 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-10T11:32:50.804 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 346479, 2024-05-10T11:32:50.804 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 122702817, 2024-05-10T11:32:50.804 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:50.804 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.804 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.804 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.804 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:50.805 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.805 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-05-10T11:32:50.805 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:50.805 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-10T11:32:50.805 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:50.805 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-10T11:32:50.805 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-10T11:32:50.805 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:50.806 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:50.806 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 32991, 2024-05-10T11:32:50.806 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 77349039, 2024-05-10T11:32:50.806 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 32991, 2024-05-10T11:32:50.806 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 77349039, 2024-05-10T11:32:50.806 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:50.806 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.806 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.807 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.807 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:50.807 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.807 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-05-10T11:32:50.807 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:50.807 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-05-10T11:32:50.807 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:50.807 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-10T11:32:50.808 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-10T11:32:50.808 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:50.808 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:50.808 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 32991, 2024-05-10T11:32:50.808 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 32991, 2024-05-10T11:32:50.808 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 32991, 2024-05-10T11:32:50.808 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 32991, 2024-05-10T11:32:50.808 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:50.808 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.809 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.809 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.809 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:50.809 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.809 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-05-10T11:32:50.809 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:50.809 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-10T11:32:50.809 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:50.810 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 9704, 2024-05-10T11:32:50.810 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1058136, 2024-05-10T11:32:50.810 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:50.810 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 9704, 2024-05-10T11:32:50.810 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-10T11:32:50.810 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-10T11:32:50.810 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 747, 2024-05-10T11:32:50.810 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1058136, 2024-05-10T11:32:50.811 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:50.811 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.811 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.811 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.811 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:50.811 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:50.811 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x562232a92500": { 2024-05-10T11:32:50.811 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 574, 2024-05-10T11:32:50.811 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-10T11:32:50.812 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:50.812 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 41044, 2024-05-10T11:32:50.812 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 74331039, 2024-05-10T11:32:50.812 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:50.812 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 41044, 2024-05-10T11:32:50.812 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-10T11:32:50.812 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-10T11:32:50.812 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 10778, 2024-05-10T11:32:50.812 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 74330465, 2024-05-10T11:32:50.813 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:50.813 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:50.813 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:50.813 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:50.813 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:50.813 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:50.813 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-05-10T11:32:51.163 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:50 smithi189 ceph-mon[34842]: pgmap v617: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 136 KiB/s wr, 26 op/s 2024-05-10T11:32:51.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:50 smithi076 ceph-mon[34152]: pgmap v617: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 136 KiB/s wr, 26 op/s 2024-05-10T11:32:51.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:50 smithi027 ceph-mon[30054]: pgmap v617: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 136 KiB/s wr, 26 op/s 2024-05-10T11:32:51.618 INFO:tasks.check_counter:Daemon mds.c mds.exported=1 2024-05-10T11:32:51.618 INFO:tasks.check_counter:Daemon mds.c mds.imported=4 2024-05-10T11:32:51.618 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-05-10T11:32:51.618 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-05-10T11:32:51.619 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-05-10T11:32:51.619 DEBUG:tasks.check_counter:Getting stats from f 2024-05-10T11:32:51.619 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph daemon mds.f perf dump 2024-05-10T11:32:51.884 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:32:53.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:52 smithi076 ceph-mon[34152]: pgmap v618: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 16 op/s 2024-05-10T11:32:53.180 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:52 smithi189 ceph-mon[34842]: pgmap v618: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 16 op/s 2024-05-10T11:32:53.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:52 smithi027 ceph-mon[30054]: pgmap v618: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 16 op/s 2024-05-10T11:32:54.100 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-05-10T11:32:54.100 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-05-10T11:32:54.101 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 3680, 2024-05-10T11:32:54.101 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 3680, 2024-05-10T11:32:54.101 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 588270197, 2024-05-10T11:32:54.101 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 1169426, 2024-05-10T11:32:54.101 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 8, 2024-05-10T11:32:54.101 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 4, 2024-05-10T11:32:54.101 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.569458701, 2024-05-10T11:32:54.101 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.081421191, 2024-05-10T11:32:54.102 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.369769744, 2024-05-10T11:32:54.102 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.091720873, 2024-05-10T11:32:54.102 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:54.102 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 3679, 2024-05-10T11:32:54.102 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.102134338, 2024-05-10T11:32:54.102 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000027761 2024-05-10T11:32:54.102 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.103 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:54.103 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.103 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.103 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.103 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.103 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-05-10T11:32:54.103 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 448 2024-05-10T11:32:54.103 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.103 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-05-10T11:32:54.104 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 2266, 2024-05-10T11:32:54.104 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 2237, 2024-05-10T11:32:54.104 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 528342289, 2024-05-10T11:32:54.104 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 830196, 2024-05-10T11:32:54.104 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 6, 2024-05-10T11:32:54.104 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 5, 2024-05-10T11:32:54.104 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.439196665, 2024-05-10T11:32:54.104 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.057782869, 2024-05-10T11:32:54.105 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.315723223, 2024-05-10T11:32:54.105 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.066267075, 2024-05-10T11:32:54.105 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:54.105 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2236, 2024-05-10T11:32:54.105 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.063543659, 2024-05-10T11:32:54.105 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000028418 2024-05-10T11:32:54.105 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.105 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:54.106 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.106 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.106 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.106 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.106 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2074784, 2024-05-10T11:32:54.106 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 260592 2024-05-10T11:32:54.106 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.106 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-05-10T11:32:54.107 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 1516, 2024-05-10T11:32:54.107 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 1827, 2024-05-10T11:32:54.107 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 323351761, 2024-05-10T11:32:54.107 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 1246879, 2024-05-10T11:32:54.107 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 9, 2024-05-10T11:32:54.107 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 6, 2024-05-10T11:32:54.107 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.265665286, 2024-05-10T11:32:54.107 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.046923292, 2024-05-10T11:32:54.107 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.169633442, 2024-05-10T11:32:54.108 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.036461247, 2024-05-10T11:32:54.108 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:54.108 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1824, 2024-05-10T11:32:54.108 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.079575436, 2024-05-10T11:32:54.108 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000043626 2024-05-10T11:32:54.108 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.108 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:54.108 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.109 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.109 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.109 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.109 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-10T11:32:54.109 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 765744 2024-05-10T11:32:54.109 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.109 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-05-10T11:32:54.110 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-05-10T11:32:54.110 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-05-10T11:32:54.110 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.110 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-05-10T11:32:54.110 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-10T11:32:54.110 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-10T11:32:54.110 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 10691, 2024-05-10T11:32:54.111 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.302758355, 2024-05-10T11:32:54.111 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000028318 2024-05-10T11:32:54.111 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:54.111 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.111 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-05-10T11:32:54.111 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-10T11:32:54.111 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-10T11:32:54.111 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.112 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.112 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.112 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:54.112 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.112 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-05-10T11:32:54.112 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 0, 2024-05-10T11:32:54.112 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 0, 2024-05-10T11:32:54.112 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-05-10T11:32:54.112 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.113 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.113 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.113 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.113 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-05-10T11:32:54.113 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 0, 2024-05-10T11:32:54.113 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-05-10T11:32:54.113 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-05-10T11:32:54.113 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 0, 2024-05-10T11:32:54.114 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-05-10T11:32:54.114 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-05-10T11:32:54.114 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 47273, 2024-05-10T11:32:54.114 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 20451, 2024-05-10T11:32:54.114 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 26822, 2024-05-10T11:32:54.114 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-05-10T11:32:54.114 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 18058, 2024-05-10T11:32:54.114 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 0, 2024-05-10T11:32:54.115 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 0, 2024-05-10T11:32:54.115 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 0, 2024-05-10T11:32:54.115 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 31, 2024-05-10T11:32:54.115 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 0, 2024-05-10T11:32:54.115 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 0, 2024-05-10T11:32:54.115 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-05-10T11:32:54.115 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 0, 2024-05-10T11:32:54.116 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-05-10T11:32:54.116 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-05-10T11:32:54.116 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-05-10T11:32:54.116 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-05-10T11:32:54.116 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-05-10T11:32:54.116 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-05-10T11:32:54.116 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-05-10T11:32:54.116 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-05-10T11:32:54.117 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-05-10T11:32:54.117 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-05-10T11:32:54.117 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-05-10T11:32:54.117 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-05-10T11:32:54.117 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 699, 2024-05-10T11:32:54.117 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 222310885, 2024-05-10T11:32:54.117 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 6, 2024-05-10T11:32:54.117 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-05-10T11:32:54.118 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-05-10T11:32:54.118 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-05-10T11:32:54.118 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-05-10T11:32:54.118 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-05-10T11:32:54.118 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-05-10T11:32:54.118 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-05-10T11:32:54.118 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-05-10T11:32:54.118 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-05-10T11:32:54.119 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-05-10T11:32:54.119 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-05-10T11:32:54.119 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-05-10T11:32:54.119 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-10T11:32:54.119 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-10T11:32:54.119 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-05-10T11:32:54.119 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-05-10T11:32:54.119 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-05-10T11:32:54.120 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-05-10T11:32:54.120 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.120 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-05-10T11:32:54.120 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-05-10T11:32:54.120 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-05-10T11:32:54.120 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-05-10T11:32:54.120 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 0, 2024-05-10T11:32:54.120 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-05-10T11:32:54.120 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-05-10T11:32:54.121 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-05-10T11:32:54.121 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-05-10T11:32:54.121 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-05-10T11:32:54.121 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-05-10T11:32:54.121 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-05-10T11:32:54.121 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-05-10T11:32:54.121 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-05-10T11:32:54.121 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-05-10T11:32:54.122 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-05-10T11:32:54.122 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-05-10T11:32:54.122 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-05-10T11:32:54.122 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-05-10T11:32:54.122 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-05-10T11:32:54.122 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-05-10T11:32:54.122 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-05-10T11:32:54.122 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-05-10T11:32:54.123 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-05-10T11:32:54.123 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-05-10T11:32:54.123 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-05-10T11:32:54.123 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.123 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-05-10T11:32:54.123 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-05-10T11:32:54.123 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 0, 2024-05-10T11:32:54.123 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-05-10T11:32:54.124 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-05-10T11:32:54.124 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 238721, 2024-05-10T11:32:54.124 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-05-10T11:32:54.124 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-05-10T11:32:54.124 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 0, 2024-05-10T11:32:54.124 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-05-10T11:32:54.124 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-05-10T11:32:54.125 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 138, 2024-05-10T11:32:54.125 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 21, 2024-05-10T11:32:54.125 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-05-10T11:32:54.125 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-05-10T11:32:54.125 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 893254611, 2024-05-10T11:32:54.125 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 1440117600, 2024-05-10T11:32:54.125 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 1440105030, 2024-05-10T11:32:54.125 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-05-10T11:32:54.125 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.126 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.126 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.126 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.126 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 238721 2024-05-10T11:32:54.126 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.126 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-05-10T11:32:54.126 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 2550, 2024-05-10T11:32:54.126 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 46968, 2024-05-10T11:32:54.127 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 44418, 2024-05-10T11:32:54.127 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 195, 2024-05-10T11:32:54.127 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 387, 2024-05-10T11:32:54.127 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 192, 2024-05-10T11:32:54.127 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 47276, 2024-05-10T11:32:54.127 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 47539, 2024-05-10T11:32:54.127 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 263, 2024-05-10T11:32:54.127 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 0, 2024-05-10T11:32:54.128 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 0, 2024-05-10T11:32:54.128 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 0, 2024-05-10T11:32:54.128 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 106156, 2024-05-10T11:32:54.128 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 215416 2024-05-10T11:32:54.128 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.128 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-05-10T11:32:54.128 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 0, 2024-05-10T11:32:54.128 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-05-10T11:32:54.129 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 0, 2024-05-10T11:32:54.129 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 0, 2024-05-10T11:32:54.129 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 0, 2024-05-10T11:32:54.129 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-05-10T11:32:54.129 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.129 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.129 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.130 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.130 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-05-10T11:32:54.130 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.130 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.130 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.130 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.130 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-05-10T11:32:54.131 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.131 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.131 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.131 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.131 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-05-10T11:32:54.131 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.131 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.131 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.132 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.132 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-05-10T11:32:54.132 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.132 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.132 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.132 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.132 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-05-10T11:32:54.132 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.133 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.133 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.133 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.133 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-05-10T11:32:54.133 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.133 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.133 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.133 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.134 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-05-10T11:32:54.134 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.134 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.134 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.134 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.134 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-05-10T11:32:54.134 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.134 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.135 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.135 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.135 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-05-10T11:32:54.135 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.135 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.135 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.135 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.136 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-05-10T11:32:54.136 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.136 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.136 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.137 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.137 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-05-10T11:32:54.137 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.137 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.137 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.137 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.137 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-05-10T11:32:54.137 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.138 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.138 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.138 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.138 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-05-10T11:32:54.138 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.138 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.138 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.138 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.139 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-05-10T11:32:54.139 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.139 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.139 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.139 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.139 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-05-10T11:32:54.139 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.139 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.140 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.140 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.140 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-05-10T11:32:54.140 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.140 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.140 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.140 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.140 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-05-10T11:32:54.141 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.141 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.141 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.141 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.141 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-05-10T11:32:54.141 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.141 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.141 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.142 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.142 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-05-10T11:32:54.142 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.142 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.142 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.142 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.142 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-05-10T11:32:54.142 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.142 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.143 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.143 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.143 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-05-10T11:32:54.143 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.143 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.143 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.144 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.144 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-05-10T11:32:54.144 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.144 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.144 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.144 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.144 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-05-10T11:32:54.145 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.145 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.145 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.145 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.145 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-05-10T11:32:54.145 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.145 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.145 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.146 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.146 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-05-10T11:32:54.146 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.146 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.146 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.146 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.146 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-05-10T11:32:54.147 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.147 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.147 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.147 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.147 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-05-10T11:32:54.147 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.147 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.147 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.147 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.148 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-05-10T11:32:54.148 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.148 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.148 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.148 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.148 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-05-10T11:32:54.148 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-05-10T11:32:54.148 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-05-10T11:32:54.149 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.149 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.149 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.149 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:54.149 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.149 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-05-10T11:32:54.149 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 2, 2024-05-10T11:32:54.149 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 6, 2024-05-10T11:32:54.150 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 4, 2024-05-10T11:32:54.150 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 4, 2024-05-10T11:32:54.150 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-05-10T11:32:54.150 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 0, 2024-05-10T11:32:54.150 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 0, 2024-05-10T11:32:54.150 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 4905, 2024-05-10T11:32:54.150 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-05-10T11:32:54.151 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.151 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-05-10T11:32:54.151 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-05-10T11:32:54.151 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-05-10T11:32:54.151 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-05-10T11:32:54.151 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-05-10T11:32:54.151 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-05-10T11:32:54.151 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-05-10T11:32:54.152 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-10T11:32:54.152 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-05-10T11:32:54.152 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-10T11:32:54.152 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-05-10T11:32:54.152 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-05-10T11:32:54.152 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-05-10T11:32:54.152 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-10T11:32:54.152 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-05-10T11:32:54.153 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-05-10T11:32:54.153 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-05-10T11:32:54.153 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-05-10T11:32:54.153 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-05-10T11:32:54.153 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-10T11:32:54.153 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-05-10T11:32:54.153 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-10T11:32:54.153 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-05-10T11:32:54.154 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-05-10T11:32:54.154 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-05-10T11:32:54.154 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-05-10T11:32:54.154 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-05-10T11:32:54.154 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-10T11:32:54.154 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-05-10T11:32:54.154 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-05-10T11:32:54.154 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-05-10T11:32:54.155 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-05-10T11:32:54.155 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-05-10T11:32:54.155 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-05-10T11:32:54.155 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-05-10T11:32:54.155 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-05-10T11:32:54.155 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-05-10T11:32:54.155 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 360396, 2024-05-10T11:32:54.155 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 3330, 2024-05-10T11:32:54.156 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-05-10T11:32:54.156 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-05-10T11:32:54.156 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-05-10T11:32:54.156 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-05-10T11:32:54.156 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-05-10T11:32:54.156 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-05-10T11:32:54.156 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-05-10T11:32:54.156 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-05-10T11:32:54.157 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 9808, 2024-05-10T11:32:54.157 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 189, 2024-05-10T11:32:54.157 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-05-10T11:32:54.157 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-05-10T11:32:54.157 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-05-10T11:32:54.157 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-05-10T11:32:54.157 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 40423178, 2024-05-10T11:32:54.157 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 265421, 2024-05-10T11:32:54.158 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-05-10T11:32:54.158 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-05-10T11:32:54.158 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-05-10T11:32:54.158 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-05-10T11:32:54.158 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.158 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-05-10T11:32:54.158 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-05-10T11:32:54.159 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-05-10T11:32:54.159 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 6980, 2024-05-10T11:32:54.159 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 26, 2024-05-10T11:32:54.159 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 4, 2024-05-10T11:32:54.159 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 6980, 2024-05-10T11:32:54.159 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-05-10T11:32:54.159 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 6980, 2024-05-10T11:32:54.159 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 12.674749981, 2024-05-10T11:32:54.160 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001815866 2024-05-10T11:32:54.160 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.160 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-05-10T11:32:54.160 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-05-10T11:32:54.160 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 6980, 2024-05-10T11:32:54.160 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 6981 2024-05-10T11:32:54.160 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.160 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 6980, 2024-05-10T11:32:54.160 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 6980, 2024-05-10T11:32:54.161 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 0, 2024-05-10T11:32:54.161 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-05-10T11:32:54.161 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-05-10T11:32:54.161 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 4249, 2024-05-10T11:32:54.161 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 0, 2024-05-10T11:32:54.161 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 2728, 2024-05-10T11:32:54.161 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 0, 2024-05-10T11:32:54.161 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 0, 2024-05-10T11:32:54.161 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-05-10T11:32:54.162 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-05-10T11:32:54.162 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-05-10T11:32:54.162 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-05-10T11:32:54.162 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 0, 2024-05-10T11:32:54.162 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-05-10T11:32:54.162 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-05-10T11:32:54.162 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-05-10T11:32:54.162 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 2, 2024-05-10T11:32:54.163 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 0, 2024-05-10T11:32:54.163 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-05-10T11:32:54.163 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-05-10T11:32:54.163 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-05-10T11:32:54.163 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-05-10T11:32:54.163 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-05-10T11:32:54.163 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-05-10T11:32:54.164 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-05-10T11:32:54.164 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-05-10T11:32:54.164 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-05-10T11:32:54.164 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-05-10T11:32:54.164 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-05-10T11:32:54.164 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-05-10T11:32:54.164 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-05-10T11:32:54.164 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-05-10T11:32:54.165 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-05-10T11:32:54.165 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-05-10T11:32:54.165 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-05-10T11:32:54.165 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-05-10T11:32:54.165 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-05-10T11:32:54.165 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-05-10T11:32:54.165 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-05-10T11:32:54.165 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-05-10T11:32:54.165 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-05-10T11:32:54.166 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-05-10T11:32:54.166 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-05-10T11:32:54.166 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-05-10T11:32:54.166 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 117, 2024-05-10T11:32:54.166 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-05-10T11:32:54.166 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 29, 2024-05-10T11:32:54.166 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 11, 2024-05-10T11:32:54.167 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 11, 2024-05-10T11:32:54.167 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-05-10T11:32:54.167 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-05-10T11:32:54.167 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 0, 2024-05-10T11:32:54.167 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 2, 2024-05-10T11:32:54.167 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 0 2024-05-10T11:32:54.167 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.168 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-05-10T11:32:54.168 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 0, 2024-05-10T11:32:54.168 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-05-10T11:32:54.168 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-05-10T11:32:54.168 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-05-10T11:32:54.168 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.168 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-05-10T11:32:54.168 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-05-10T11:32:54.169 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.169 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-05-10T11:32:54.169 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-05-10T11:32:54.169 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-05-10T11:32:54.169 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-05-10T11:32:54.169 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-05-10T11:32:54.169 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-05-10T11:32:54.169 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-05-10T11:32:54.170 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-05-10T11:32:54.170 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.170 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-10T11:32:54.170 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:54.170 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-10T11:32:54.170 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:54.170 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 7458, 2024-05-10T11:32:54.171 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1439265759, 2024-05-10T11:32:54.171 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:54.171 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 7458, 2024-05-10T11:32:54.171 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-10T11:32:54.171 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-10T11:32:54.171 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 7458, 2024-05-10T11:32:54.171 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1439265759, 2024-05-10T11:32:54.171 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:54.171 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.172 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.172 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.172 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:54.172 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.172 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-05-10T11:32:54.172 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:54.172 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-10T11:32:54.172 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:54.173 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-10T11:32:54.173 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-10T11:32:54.173 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:54.173 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:54.173 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 6980, 2024-05-10T11:32:54.173 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 1435983963, 2024-05-10T11:32:54.173 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 6980, 2024-05-10T11:32:54.173 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1435983963, 2024-05-10T11:32:54.173 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:54.174 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.174 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.174 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.174 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:54.174 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.174 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-05-10T11:32:54.174 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:54.175 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-05-10T11:32:54.175 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:54.175 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-10T11:32:54.175 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-10T11:32:54.175 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:54.175 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:54.176 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 6980, 2024-05-10T11:32:54.176 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 6980, 2024-05-10T11:32:54.176 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 6980, 2024-05-10T11:32:54.176 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 6980, 2024-05-10T11:32:54.176 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:54.176 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.177 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.177 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.177 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:54.177 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.177 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-05-10T11:32:54.177 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:54.177 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-10T11:32:54.177 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:54.178 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-10T11:32:54.178 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-10T11:32:54.178 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:54.178 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:54.178 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-10T11:32:54.178 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-10T11:32:54.178 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-05-10T11:32:54.178 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-05-10T11:32:54.179 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:54.179 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.179 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.179 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.179 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:54.179 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:54.179 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x55d26e0ae6e0": { 2024-05-10T11:32:54.180 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:54.180 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-10T11:32:54.180 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:54.180 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-10T11:32:54.180 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-10T11:32:54.180 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:54.180 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:54.180 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-10T11:32:54.180 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-10T11:32:54.181 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-05-10T11:32:54.181 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-05-10T11:32:54.181 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:54.181 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:54.181 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:54.181 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:54.181 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:54.181 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:54.182 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-05-10T11:32:54.913 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-05-10T11:32:54.913 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-05-10T11:32:54.914 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-05-10T11:32:54.914 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-05-10T11:32:54.914 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-05-10T11:32:54.914 DEBUG:tasks.check_counter:Getting stats from i 2024-05-10T11:32:54.914 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph daemon mds.i perf dump 2024-05-10T11:32:55.156 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:32:55.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:54 smithi076 ceph-mon[34152]: pgmap v619: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 15 op/s 2024-05-10T11:32:55.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:54 smithi189 ceph-mon[34842]: pgmap v619: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 15 op/s 2024-05-10T11:32:55.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:54 smithi027 ceph-mon[30054]: pgmap v619: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 15 op/s 2024-05-10T11:32:56.840 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-05-10T11:32:56.840 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-05-10T11:32:56.840 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 236386, 2024-05-10T11:32:56.840 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 138250, 2024-05-10T11:32:56.841 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 115955858, 2024-05-10T11:32:56.841 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 48168626, 2024-05-10T11:32:56.841 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 18, 2024-05-10T11:32:56.841 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 4, 2024-05-10T11:32:56.841 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 9.718440445, 2024-05-10T11:32:56.841 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 3.751229756, 2024-05-10T11:32:56.841 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 7.947779548, 2024-05-10T11:32:56.841 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.182384957, 2024-05-10T11:32:56.842 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:56.842 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 138252, 2024-05-10T11:32:56.842 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 2.138126281, 2024-05-10T11:32:56.842 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000015465 2024-05-10T11:32:56.842 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.842 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:56.842 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 363079, 2024-05-10T11:32:56.843 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.046288373, 2024-05-10T11:32:56.843 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000127 2024-05-10T11:32:56.843 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.843 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 160, 2024-05-10T11:32:56.843 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 513712 2024-05-10T11:32:56.843 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.843 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-05-10T11:32:56.843 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 19618, 2024-05-10T11:32:56.844 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 13281, 2024-05-10T11:32:56.844 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 7603613, 2024-05-10T11:32:56.844 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 21557543, 2024-05-10T11:32:56.844 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 28, 2024-05-10T11:32:56.844 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 7, 2024-05-10T11:32:56.844 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 1.194333843, 2024-05-10T11:32:56.844 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.426449458, 2024-05-10T11:32:56.844 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.728118767, 2024-05-10T11:32:56.845 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.177923174, 2024-05-10T11:32:56.845 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:56.845 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 13281, 2024-05-10T11:32:56.845 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.338535708, 2024-05-10T11:32:56.845 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000025490 2024-05-10T11:32:56.845 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.845 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:56.845 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 19225, 2024-05-10T11:32:56.846 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.003231037, 2024-05-10T11:32:56.846 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000168 2024-05-10T11:32:56.846 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.846 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2108960, 2024-05-10T11:32:56.846 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 305376 2024-05-10T11:32:56.846 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.846 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-05-10T11:32:56.846 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 43017, 2024-05-10T11:32:56.847 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 61277, 2024-05-10T11:32:56.847 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 13637524, 2024-05-10T11:32:56.847 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 52935179, 2024-05-10T11:32:56.847 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 25, 2024-05-10T11:32:56.847 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 8, 2024-05-10T11:32:56.847 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 2.541399349, 2024-05-10T11:32:56.847 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 1.160274358, 2024-05-10T11:32:56.847 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 1.579401011, 2024-05-10T11:32:56.848 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.142203863, 2024-05-10T11:32:56.848 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:56.848 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 61270, 2024-05-10T11:32:56.848 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1.558282543, 2024-05-10T11:32:56.848 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000025433 2024-05-10T11:32:56.848 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.848 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:56.848 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 48100, 2024-05-10T11:32:56.849 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.016839603, 2024-05-10T11:32:56.849 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000350 2024-05-10T11:32:56.849 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.849 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-05-10T11:32:56.849 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 177712 2024-05-10T11:32:56.849 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.849 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-05-10T11:32:56.849 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-05-10T11:32:56.850 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-05-10T11:32:56.850 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.850 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-05-10T11:32:56.850 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-10T11:32:56.850 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-10T11:32:56.850 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 12776, 2024-05-10T11:32:56.850 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 7.186475552, 2024-05-10T11:32:56.850 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000562498 2024-05-10T11:32:56.851 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:56.851 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.851 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-05-10T11:32:56.851 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-10T11:32:56.851 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-10T11:32:56.851 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 9530, 2024-05-10T11:32:56.851 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.507676460, 2024-05-10T11:32:56.851 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000053271 2024-05-10T11:32:56.852 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:56.852 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.852 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-05-10T11:32:56.852 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 24943, 2024-05-10T11:32:56.852 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 20019, 2024-05-10T11:32:56.852 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-05-10T11:32:56.852 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 20019, 2024-05-10T11:32:56.852 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 80.528515818, 2024-05-10T11:32:56.852 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.004022604 2024-05-10T11:32:56.853 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.853 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-05-10T11:32:56.853 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 4924, 2024-05-10T11:32:56.853 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-05-10T11:32:56.853 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-05-10T11:32:56.853 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 29, 2024-05-10T11:32:56.853 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 2, 2024-05-10T11:32:56.853 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-05-10T11:32:56.854 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 514, 2024-05-10T11:32:56.854 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 162, 2024-05-10T11:32:56.854 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 302, 2024-05-10T11:32:56.854 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 50, 2024-05-10T11:32:56.854 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 283, 2024-05-10T11:32:56.854 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 30, 2024-05-10T11:32:56.854 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 26, 2024-05-10T11:32:56.855 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 27, 2024-05-10T11:32:56.855 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 22, 2024-05-10T11:32:56.855 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 48010, 2024-05-10T11:32:56.855 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 16270, 2024-05-10T11:32:56.855 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 4921, 2024-05-10T11:32:56.855 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 20, 2024-05-10T11:32:56.855 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-05-10T11:32:56.855 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-05-10T11:32:56.856 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 57, 2024-05-10T11:32:56.856 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-05-10T11:32:56.856 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-05-10T11:32:56.856 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 1, 2024-05-10T11:32:56.856 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 34, 2024-05-10T11:32:56.856 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 6, 2024-05-10T11:32:56.856 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 121, 2024-05-10T11:32:56.856 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-05-10T11:32:56.856 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 229, 2024-05-10T11:32:56.857 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-05-10T11:32:56.857 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 699, 2024-05-10T11:32:56.857 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 222310885, 2024-05-10T11:32:56.857 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 6, 2024-05-10T11:32:56.857 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 229, 2024-05-10T11:32:56.857 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-05-10T11:32:56.857 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-05-10T11:32:56.857 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-05-10T11:32:56.858 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-05-10T11:32:56.858 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-05-10T11:32:56.858 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-05-10T11:32:56.858 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 229, 2024-05-10T11:32:56.858 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-05-10T11:32:56.858 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 850, 2024-05-10T11:32:56.858 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 16056, 2024-05-10T11:32:56.858 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-05-10T11:32:56.859 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-10T11:32:56.859 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-10T11:32:56.859 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 8292, 2024-05-10T11:32:56.859 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 3865, 2024-05-10T11:32:56.859 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 1895, 2024-05-10T11:32:56.859 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 19469 2024-05-10T11:32:56.859 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.859 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-05-10T11:32:56.860 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 12, 2024-05-10T11:32:56.860 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 123, 2024-05-10T11:32:56.860 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 67, 2024-05-10T11:32:56.860 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 127, 2024-05-10T11:32:56.860 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 12, 2024-05-10T11:32:56.860 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-05-10T11:32:56.860 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-05-10T11:32:56.860 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-05-10T11:32:56.861 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 3758, 2024-05-10T11:32:56.861 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 3758, 2024-05-10T11:32:56.861 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-05-10T11:32:56.861 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-05-10T11:32:56.861 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-05-10T11:32:56.861 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-05-10T11:32:56.861 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-05-10T11:32:56.861 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-05-10T11:32:56.861 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-05-10T11:32:56.862 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-05-10T11:32:56.862 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-05-10T11:32:56.862 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-05-10T11:32:56.862 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 4, 2024-05-10T11:32:56.862 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-05-10T11:32:56.862 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 2, 2024-05-10T11:32:56.862 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-05-10T11:32:56.862 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-05-10T11:32:56.863 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.863 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-05-10T11:32:56.863 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-05-10T11:32:56.863 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 25962, 2024-05-10T11:32:56.863 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-05-10T11:32:56.863 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-05-10T11:32:56.863 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 25962, 2024-05-10T11:32:56.863 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-05-10T11:32:56.863 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-05-10T11:32:56.864 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 36, 2024-05-10T11:32:56.864 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-05-10T11:32:56.864 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-05-10T11:32:56.864 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 36, 2024-05-10T11:32:56.864 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 4, 2024-05-10T11:32:56.864 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-05-10T11:32:56.864 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-05-10T11:32:56.864 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-05-10T11:32:56.865 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 50572805, 2024-05-10T11:32:56.865 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 0, 2024-05-10T11:32:56.865 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-05-10T11:32:56.865 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 6030, 2024-05-10T11:32:56.865 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 37.696442538, 2024-05-10T11:32:56.865 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.006251483 2024-05-10T11:32:56.865 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.865 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 0 2024-05-10T11:32:56.866 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.866 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-05-10T11:32:56.866 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 409, 2024-05-10T11:32:56.866 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 4197, 2024-05-10T11:32:56.866 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 3788, 2024-05-10T11:32:56.866 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 74, 2024-05-10T11:32:56.866 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 147, 2024-05-10T11:32:56.866 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 73, 2024-05-10T11:32:56.867 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 514, 2024-05-10T11:32:56.867 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 6077, 2024-05-10T11:32:56.867 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 5563, 2024-05-10T11:32:56.867 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 27, 2024-05-10T11:32:56.867 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 4507, 2024-05-10T11:32:56.867 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 4480, 2024-05-10T11:32:56.867 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 69084, 2024-05-10T11:32:56.867 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 223608 2024-05-10T11:32:56.867 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.868 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-05-10T11:32:56.868 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 43742, 2024-05-10T11:32:56.868 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-05-10T11:32:56.868 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 24943, 2024-05-10T11:32:56.868 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 135, 2024-05-10T11:32:56.868 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 7582, 2024-05-10T11:32:56.868 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-05-10T11:32:56.868 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 3753, 2024-05-10T11:32:56.869 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 47.345432739, 2024-05-10T11:32:56.869 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.012615356 2024-05-10T11:32:56.869 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.869 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-05-10T11:32:56.869 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 40, 2024-05-10T11:32:56.869 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.056839741, 2024-05-10T11:32:56.869 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001420993 2024-05-10T11:32:56.869 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.869 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-05-10T11:32:56.870 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.870 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.870 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.870 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.870 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-05-10T11:32:56.870 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.870 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.870 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.871 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.871 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-05-10T11:32:56.871 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 4379, 2024-05-10T11:32:56.871 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1.636657164, 2024-05-10T11:32:56.871 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000373751 2024-05-10T11:32:56.871 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.871 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-05-10T11:32:56.871 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.872 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.872 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.872 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.872 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-05-10T11:32:56.872 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-05-10T11:32:56.872 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000348634, 2024-05-10T11:32:56.872 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000174317 2024-05-10T11:32:56.872 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.872 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-05-10T11:32:56.873 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.873 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.873 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.873 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.873 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-05-10T11:32:56.873 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.873 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.873 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.874 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.874 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-05-10T11:32:56.874 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.874 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.874 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.874 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.875 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-05-10T11:32:56.875 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.876 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.876 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.876 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.876 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-05-10T11:32:56.876 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.876 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.876 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.877 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.877 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-05-10T11:32:56.877 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.877 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.877 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.877 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.877 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-05-10T11:32:56.877 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.878 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.878 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.878 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.878 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-05-10T11:32:56.878 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.878 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.878 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.879 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.879 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-05-10T11:32:56.879 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 8030, 2024-05-10T11:32:56.879 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 4.534831213, 2024-05-10T11:32:56.879 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000564736 2024-05-10T11:32:56.879 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.879 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-05-10T11:32:56.879 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 57, 2024-05-10T11:32:56.880 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1.046945911, 2024-05-10T11:32:56.880 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.018367472 2024-05-10T11:32:56.880 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.880 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-05-10T11:32:56.880 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.880 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.880 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.880 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.881 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-05-10T11:32:56.881 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.881 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.881 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.881 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.881 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-05-10T11:32:56.881 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.881 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.882 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.882 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.882 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-05-10T11:32:56.882 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.882 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.882 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.882 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.882 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-05-10T11:32:56.882 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.883 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.883 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.883 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.883 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-05-10T11:32:56.883 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.883 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.883 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.884 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.884 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-05-10T11:32:56.884 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.884 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.884 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.884 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.884 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-05-10T11:32:56.884 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.885 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.885 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.885 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.885 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-05-10T11:32:56.885 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.885 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.885 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.885 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.886 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-05-10T11:32:56.886 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.886 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.886 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.886 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.886 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-05-10T11:32:56.886 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.886 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.887 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.887 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.887 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-05-10T11:32:56.887 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 3758, 2024-05-10T11:32:56.887 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 25.907460416, 2024-05-10T11:32:56.887 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.006893949 2024-05-10T11:32:56.887 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.887 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-05-10T11:32:56.888 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-05-10T11:32:56.888 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-05-10T11:32:56.888 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.888 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.888 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.888 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:56.888 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.888 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-05-10T11:32:56.889 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 2, 2024-05-10T11:32:56.889 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 5, 2024-05-10T11:32:56.889 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 3, 2024-05-10T11:32:56.889 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 2, 2024-05-10T11:32:56.889 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-05-10T11:32:56.889 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 5018, 2024-05-10T11:32:56.889 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 2509, 2024-05-10T11:32:56.889 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 1159, 2024-05-10T11:32:56.890 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-05-10T11:32:56.890 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.890 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-05-10T11:32:56.890 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-05-10T11:32:56.890 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-05-10T11:32:56.890 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-05-10T11:32:56.890 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-05-10T11:32:56.890 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-05-10T11:32:56.891 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-05-10T11:32:56.891 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-10T11:32:56.891 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-05-10T11:32:56.891 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-10T11:32:56.891 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-05-10T11:32:56.891 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-05-10T11:32:56.892 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-05-10T11:32:56.892 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-10T11:32:56.892 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-05-10T11:32:56.892 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-05-10T11:32:56.892 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-05-10T11:32:56.892 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-05-10T11:32:56.892 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-05-10T11:32:56.892 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-10T11:32:56.893 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-05-10T11:32:56.893 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-10T11:32:56.893 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-05-10T11:32:56.893 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-05-10T11:32:56.893 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-05-10T11:32:56.893 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-05-10T11:32:56.893 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-05-10T11:32:56.893 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-10T11:32:56.893 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-05-10T11:32:56.894 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-05-10T11:32:56.894 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-05-10T11:32:56.894 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-05-10T11:32:56.894 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-05-10T11:32:56.894 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-05-10T11:32:56.894 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-05-10T11:32:56.894 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-05-10T11:32:56.894 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-05-10T11:32:56.895 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 320962, 2024-05-10T11:32:56.895 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 305, 2024-05-10T11:32:56.895 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 352, 2024-05-10T11:32:56.895 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 4, 2024-05-10T11:32:56.895 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-05-10T11:32:56.895 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-05-10T11:32:56.895 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-05-10T11:32:56.895 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-05-10T11:32:56.896 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-05-10T11:32:56.896 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-05-10T11:32:56.896 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 9808, 2024-05-10T11:32:56.896 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 189, 2024-05-10T11:32:56.896 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-05-10T11:32:56.896 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-05-10T11:32:56.896 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-05-10T11:32:56.896 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-05-10T11:32:56.896 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 1497961, 2024-05-10T11:32:56.897 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 10587, 2024-05-10T11:32:56.897 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-05-10T11:32:56.897 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-05-10T11:32:56.897 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-05-10T11:32:56.897 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-05-10T11:32:56.897 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.897 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-05-10T11:32:56.897 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-05-10T11:32:56.898 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-05-10T11:32:56.898 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 16623, 2024-05-10T11:32:56.898 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 47619368, 2024-05-10T11:32:56.898 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 28, 2024-05-10T11:32:56.898 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 16623, 2024-05-10T11:32:56.898 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-05-10T11:32:56.898 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 16623, 2024-05-10T11:32:56.898 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 113.244770298, 2024-05-10T11:32:56.898 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.006812535 2024-05-10T11:32:56.899 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.899 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-05-10T11:32:56.899 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-05-10T11:32:56.899 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 16623, 2024-05-10T11:32:56.899 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 17838 2024-05-10T11:32:56.899 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.899 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 16623, 2024-05-10T11:32:56.899 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 979, 2024-05-10T11:32:56.900 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 15644, 2024-05-10T11:32:56.900 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-05-10T11:32:56.900 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-05-10T11:32:56.900 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 0, 2024-05-10T11:32:56.900 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 10, 2024-05-10T11:32:56.900 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 750, 2024-05-10T11:32:56.900 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 6780, 2024-05-10T11:32:56.900 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 458, 2024-05-10T11:32:56.900 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-05-10T11:32:56.901 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-05-10T11:32:56.901 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-05-10T11:32:56.901 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 2, 2024-05-10T11:32:56.901 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 7539, 2024-05-10T11:32:56.901 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-05-10T11:32:56.901 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-05-10T11:32:56.901 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-05-10T11:32:56.901 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 229, 2024-05-10T11:32:56.902 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 20, 2024-05-10T11:32:56.902 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-05-10T11:32:56.902 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-05-10T11:32:56.902 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-05-10T11:32:56.902 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-05-10T11:32:56.906 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-05-10T11:32:56.906 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-05-10T11:32:56.906 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-05-10T11:32:56.906 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-05-10T11:32:56.906 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-05-10T11:32:56.906 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-05-10T11:32:56.906 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-05-10T11:32:56.907 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-05-10T11:32:56.907 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-05-10T11:32:56.907 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-05-10T11:32:56.907 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-05-10T11:32:56.907 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-05-10T11:32:56.907 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-05-10T11:32:56.907 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-05-10T11:32:56.907 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-05-10T11:32:56.908 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-05-10T11:32:56.908 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-05-10T11:32:56.908 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-05-10T11:32:56.908 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-05-10T11:32:56.908 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-05-10T11:32:56.908 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-05-10T11:32:56.908 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-05-10T11:32:56.913 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 117, 2024-05-10T11:32:56.914 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-05-10T11:32:56.914 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 29, 2024-05-10T11:32:56.914 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 12, 2024-05-10T11:32:56.914 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 12, 2024-05-10T11:32:56.914 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-05-10T11:32:56.914 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-05-10T11:32:56.914 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 1628, 2024-05-10T11:32:56.915 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 0, 2024-05-10T11:32:56.915 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 422 2024-05-10T11:32:56.915 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.915 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-05-10T11:32:56.915 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 1, 2024-05-10T11:32:56.915 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-05-10T11:32:56.915 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-05-10T11:32:56.915 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-05-10T11:32:56.916 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.916 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-05-10T11:32:56.916 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-05-10T11:32:56.916 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.916 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-05-10T11:32:56.916 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-05-10T11:32:56.916 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 10, 2024-05-10T11:32:56.917 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-05-10T11:32:56.917 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 5, 2024-05-10T11:32:56.917 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 7516, 2024-05-10T11:32:56.917 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 3758, 2024-05-10T11:32:56.917 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-05-10T11:32:56.917 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.918 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-10T11:32:56.918 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:56.918 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-10T11:32:56.918 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:56.918 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 299017, 2024-05-10T11:32:56.918 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 110856645, 2024-05-10T11:32:56.918 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:56.919 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 299017, 2024-05-10T11:32:56.919 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-10T11:32:56.919 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-10T11:32:56.919 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 299017, 2024-05-10T11:32:56.919 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 110856645, 2024-05-10T11:32:56.920 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:56.920 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.920 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.920 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.920 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:56.920 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.920 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-05-10T11:32:56.920 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:56.921 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-10T11:32:56.921 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:56.921 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-10T11:32:56.921 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-10T11:32:56.921 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:56.921 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:56.921 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 16623, 2024-05-10T11:32:56.922 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 48029718, 2024-05-10T11:32:56.922 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 16623, 2024-05-10T11:32:56.922 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 48029718, 2024-05-10T11:32:56.922 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:56.922 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.922 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.922 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.922 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:56.923 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.923 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-05-10T11:32:56.923 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:56.923 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-05-10T11:32:56.923 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:56.923 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-10T11:32:56.923 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-10T11:32:56.923 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:56.924 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:56.924 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 16623, 2024-05-10T11:32:56.924 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 16623, 2024-05-10T11:32:56.924 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 16623, 2024-05-10T11:32:56.924 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 16623, 2024-05-10T11:32:56.924 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:56.924 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.924 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.925 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.925 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:56.925 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.925 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-05-10T11:32:56.925 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:56.925 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-10T11:32:56.925 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:56.926 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 3758, 2024-05-10T11:32:56.926 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 410350, 2024-05-10T11:32:56.926 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:56.926 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 3758, 2024-05-10T11:32:56.926 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-10T11:32:56.926 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-10T11:32:56.926 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 750, 2024-05-10T11:32:56.926 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 410350, 2024-05-10T11:32:56.927 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:56.927 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.927 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.927 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.927 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:56.927 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:56.927 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x5564cb768500": { 2024-05-10T11:32:56.927 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 5166, 2024-05-10T11:32:56.928 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-10T11:32:56.928 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:56.928 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 25962, 2024-05-10T11:32:56.928 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 46378501, 2024-05-10T11:32:56.928 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:56.929 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 25962, 2024-05-10T11:32:56.929 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-10T11:32:56.929 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-10T11:32:56.929 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 6030, 2024-05-10T11:32:56.929 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 46373335, 2024-05-10T11:32:56.929 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:56.929 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:56.929 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:56.930 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:56.930 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:56.930 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:56.930 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-05-10T11:32:57.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:56 smithi076 ceph-mon[34152]: pgmap v620: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 108 KiB/s wr, 21 op/s 2024-05-10T11:32:57.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:56 smithi189 ceph-mon[34842]: pgmap v620: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 108 KiB/s wr, 21 op/s 2024-05-10T11:32:57.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:56 smithi027 ceph-mon[30054]: pgmap v620: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 108 KiB/s wr, 21 op/s 2024-05-10T11:32:57.672 INFO:tasks.check_counter:Daemon mds.i mds.exported=1 2024-05-10T11:32:57.672 INFO:tasks.check_counter:Daemon mds.i mds.imported=6 2024-05-10T11:32:57.672 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-05-10T11:32:57.672 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-05-10T11:32:57.672 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-05-10T11:32:57.672 DEBUG:tasks.check_counter:Getting stats from l 2024-05-10T11:32:57.673 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph daemon mds.l perf dump 2024-05-10T11:32:57.912 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.c/config 2024-05-10T11:32:59.092 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:32:58 smithi189 ceph-mon[34842]: pgmap v621: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 64 KiB/s wr, 13 op/s 2024-05-10T11:32:59.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:32:58 smithi076 ceph-mon[34152]: pgmap v621: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 64 KiB/s wr, 13 op/s 2024-05-10T11:32:59.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:32:58 smithi027 ceph-mon[30054]: pgmap v621: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 64 KiB/s wr, 13 op/s 2024-05-10T11:32:59.656 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-05-10T11:32:59.656 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-05-10T11:32:59.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 2513, 2024-05-10T11:32:59.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 2513, 2024-05-10T11:32:59.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 17536092, 2024-05-10T11:32:59.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 798311, 2024-05-10T11:32:59.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 7, 2024-05-10T11:32:59.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 3, 2024-05-10T11:32:59.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.219071128, 2024-05-10T11:32:59.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.068436279, 2024-05-10T11:32:59.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.062673139, 2024-05-10T11:32:59.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.069428470, 2024-05-10T11:32:59.658 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:59.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2512, 2024-05-10T11:32:59.658 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.083938732, 2024-05-10T11:32:59.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000033415 2024-05-10T11:32:59.658 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.658 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:59.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.659 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.659 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-05-10T11:32:59.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 448 2024-05-10T11:32:59.659 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.659 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-05-10T11:32:59.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 2276, 2024-05-10T11:32:59.660 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 2247, 2024-05-10T11:32:59.660 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 47502109, 2024-05-10T11:32:59.660 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 835468, 2024-05-10T11:32:59.660 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 5, 2024-05-10T11:32:59.660 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 4, 2024-05-10T11:32:59.660 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.209605768, 2024-05-10T11:32:59.660 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.062619293, 2024-05-10T11:32:59.660 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.102243945, 2024-05-10T11:32:59.661 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.048146681, 2024-05-10T11:32:59.661 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:59.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2246, 2024-05-10T11:32:59.661 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.066133054, 2024-05-10T11:32:59.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000029444 2024-05-10T11:32:59.661 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.661 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:59.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.662 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.662 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.662 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.662 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2076256, 2024-05-10T11:32:59.662 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 264272 2024-05-10T11:32:59.662 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.662 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-05-10T11:32:59.663 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 391, 2024-05-10T11:32:59.663 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 708, 2024-05-10T11:32:59.663 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 12681422, 2024-05-10T11:32:59.663 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 902696, 2024-05-10T11:32:59.663 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 9, 2024-05-10T11:32:59.663 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 6, 2024-05-10T11:32:59.663 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.047936225, 2024-05-10T11:32:59.663 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.018673977, 2024-05-10T11:32:59.664 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.016363652, 2024-05-10T11:32:59.664 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.008030263, 2024-05-10T11:32:59.664 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-10T11:32:59.664 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 705, 2024-05-10T11:32:59.664 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.051777582, 2024-05-10T11:32:59.664 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000073443 2024-05-10T11:32:59.664 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.664 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-10T11:32:59.665 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.665 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.665 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.665 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.665 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-10T11:32:59.665 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 779312 2024-05-10T11:32:59.666 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.666 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-05-10T11:32:59.666 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-05-10T11:32:59.666 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-05-10T11:32:59.666 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.666 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-05-10T11:32:59.666 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-10T11:32:59.666 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-10T11:32:59.667 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 8130, 2024-05-10T11:32:59.667 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.178578458, 2024-05-10T11:32:59.667 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000021965 2024-05-10T11:32:59.667 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:59.667 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.667 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-05-10T11:32:59.667 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-10T11:32:59.668 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-10T11:32:59.668 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.668 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.668 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.668 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:59.668 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.668 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-05-10T11:32:59.668 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 0, 2024-05-10T11:32:59.669 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 0, 2024-05-10T11:32:59.669 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-05-10T11:32:59.669 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.669 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.669 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.669 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.669 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-05-10T11:32:59.670 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 0, 2024-05-10T11:32:59.670 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-05-10T11:32:59.670 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-05-10T11:32:59.670 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 0, 2024-05-10T11:32:59.670 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-05-10T11:32:59.670 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-05-10T11:32:59.670 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 9854, 2024-05-10T11:32:59.670 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 14, 2024-05-10T11:32:59.671 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 9840, 2024-05-10T11:32:59.671 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-05-10T11:32:59.671 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 9834, 2024-05-10T11:32:59.671 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 0, 2024-05-10T11:32:59.671 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 0, 2024-05-10T11:32:59.671 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 0, 2024-05-10T11:32:59.671 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 5, 2024-05-10T11:32:59.671 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 0, 2024-05-10T11:32:59.672 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 0, 2024-05-10T11:32:59.672 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-05-10T11:32:59.672 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 0, 2024-05-10T11:32:59.672 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-05-10T11:32:59.672 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-05-10T11:32:59.672 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-05-10T11:32:59.673 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-05-10T11:32:59.673 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-05-10T11:32:59.673 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-05-10T11:32:59.673 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-05-10T11:32:59.673 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-05-10T11:32:59.673 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-05-10T11:32:59.673 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-05-10T11:32:59.673 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-05-10T11:32:59.674 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-05-10T11:32:59.674 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 0, 2024-05-10T11:32:59.674 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 0, 2024-05-10T11:32:59.674 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-05-10T11:32:59.674 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-05-10T11:32:59.674 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-05-10T11:32:59.674 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-05-10T11:32:59.674 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-05-10T11:32:59.675 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-05-10T11:32:59.675 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-05-10T11:32:59.675 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-05-10T11:32:59.675 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-05-10T11:32:59.675 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-05-10T11:32:59.675 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-05-10T11:32:59.675 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-05-10T11:32:59.676 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-05-10T11:32:59.676 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-10T11:32:59.676 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-10T11:32:59.676 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-05-10T11:32:59.676 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-05-10T11:32:59.676 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-05-10T11:32:59.676 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-05-10T11:32:59.676 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.677 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-05-10T11:32:59.677 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-05-10T11:32:59.677 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-05-10T11:32:59.677 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-05-10T11:32:59.677 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 0, 2024-05-10T11:32:59.677 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-05-10T11:32:59.677 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-05-10T11:32:59.678 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-05-10T11:32:59.678 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-05-10T11:32:59.678 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-05-10T11:32:59.678 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-05-10T11:32:59.678 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-05-10T11:32:59.678 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-05-10T11:32:59.678 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-05-10T11:32:59.678 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-05-10T11:32:59.679 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-05-10T11:32:59.679 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-05-10T11:32:59.679 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-05-10T11:32:59.679 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-05-10T11:32:59.679 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-05-10T11:32:59.679 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-05-10T11:32:59.679 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-05-10T11:32:59.680 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-05-10T11:32:59.680 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-05-10T11:32:59.680 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-05-10T11:32:59.680 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-05-10T11:32:59.680 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.680 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-05-10T11:32:59.680 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-05-10T11:32:59.681 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 0, 2024-05-10T11:32:59.681 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-05-10T11:32:59.681 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-05-10T11:32:59.681 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 41044, 2024-05-10T11:32:59.681 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-05-10T11:32:59.681 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-05-10T11:32:59.681 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 0, 2024-05-10T11:32:59.681 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-05-10T11:32:59.682 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-05-10T11:32:59.682 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 50, 2024-05-10T11:32:59.682 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 5, 2024-05-10T11:32:59.682 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-05-10T11:32:59.682 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-05-10T11:32:59.682 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-05-10T11:32:59.682 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 78525343, 2024-05-10T11:32:59.682 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 78524769, 2024-05-10T11:32:59.683 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-05-10T11:32:59.683 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.683 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.683 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.683 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.683 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 41044 2024-05-10T11:32:59.683 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.684 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-05-10T11:32:59.684 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 115, 2024-05-10T11:32:59.684 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 9848, 2024-05-10T11:32:59.684 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 9733, 2024-05-10T11:32:59.684 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 28, 2024-05-10T11:32:59.684 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 31, 2024-05-10T11:32:59.684 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 3, 2024-05-10T11:32:59.684 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 9854, 2024-05-10T11:32:59.685 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 9920, 2024-05-10T11:32:59.685 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 66, 2024-05-10T11:32:59.685 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 0, 2024-05-10T11:32:59.685 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 0, 2024-05-10T11:32:59.685 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 0, 2024-05-10T11:32:59.685 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 62560, 2024-05-10T11:32:59.685 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 215416 2024-05-10T11:32:59.685 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.686 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-05-10T11:32:59.686 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 0, 2024-05-10T11:32:59.686 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-05-10T11:32:59.686 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 0, 2024-05-10T11:32:59.686 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 0, 2024-05-10T11:32:59.686 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 0, 2024-05-10T11:32:59.686 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-05-10T11:32:59.687 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.687 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.687 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.687 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.687 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-05-10T11:32:59.687 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.687 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.688 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.688 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.688 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-05-10T11:32:59.688 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.688 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.688 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.688 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.688 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-05-10T11:32:59.689 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.689 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.689 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.689 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.689 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-05-10T11:32:59.689 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.689 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.690 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.690 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.690 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-05-10T11:32:59.690 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.690 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.690 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.690 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.690 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-05-10T11:32:59.691 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.691 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.691 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.691 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.691 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-05-10T11:32:59.691 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.691 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.691 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.692 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.692 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-05-10T11:32:59.692 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.692 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.692 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.692 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.692 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-05-10T11:32:59.693 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.693 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.693 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.693 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.693 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-05-10T11:32:59.693 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.693 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.693 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.694 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.695 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-05-10T11:32:59.695 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.695 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.695 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.695 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.695 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-05-10T11:32:59.695 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.696 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.696 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.696 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.696 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-05-10T11:32:59.696 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.696 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.696 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.697 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.697 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-05-10T11:32:59.697 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.697 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.697 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.697 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.697 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-05-10T11:32:59.697 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.698 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.698 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.698 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.698 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-05-10T11:32:59.698 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.698 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.698 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.699 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.699 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-05-10T11:32:59.699 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.699 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.699 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.699 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.699 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-05-10T11:32:59.700 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.700 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.700 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.700 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.700 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-05-10T11:32:59.700 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.700 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.701 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.701 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.701 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-05-10T11:32:59.701 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.701 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.701 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.701 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.701 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-05-10T11:32:59.702 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.702 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.702 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.702 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.702 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-05-10T11:32:59.702 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.702 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.702 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.703 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.703 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-05-10T11:32:59.703 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.703 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.703 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.703 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.703 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-05-10T11:32:59.704 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.704 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.704 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.704 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.704 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-05-10T11:32:59.704 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.704 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.705 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.705 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.705 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-05-10T11:32:59.705 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.705 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.705 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.705 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.705 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-05-10T11:32:59.706 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.706 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.706 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.706 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.706 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-05-10T11:32:59.706 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.706 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.707 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.707 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.707 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-05-10T11:32:59.707 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-05-10T11:32:59.707 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-05-10T11:32:59.707 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.707 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.707 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.708 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:59.708 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.708 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-05-10T11:32:59.708 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-05-10T11:32:59.708 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 2, 2024-05-10T11:32:59.708 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 1, 2024-05-10T11:32:59.708 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 1, 2024-05-10T11:32:59.709 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-05-10T11:32:59.709 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 0, 2024-05-10T11:32:59.709 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 0, 2024-05-10T11:32:59.709 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 3046, 2024-05-10T11:32:59.709 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-05-10T11:32:59.709 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.709 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-05-10T11:32:59.709 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-05-10T11:32:59.710 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-05-10T11:32:59.710 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-05-10T11:32:59.710 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-05-10T11:32:59.710 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-05-10T11:32:59.710 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-05-10T11:32:59.710 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-10T11:32:59.710 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-05-10T11:32:59.710 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-10T11:32:59.711 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-05-10T11:32:59.711 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-05-10T11:32:59.711 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-05-10T11:32:59.711 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-10T11:32:59.711 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-05-10T11:32:59.711 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-05-10T11:32:59.711 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-05-10T11:32:59.712 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-05-10T11:32:59.712 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-05-10T11:32:59.712 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-10T11:32:59.712 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-05-10T11:32:59.712 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-10T11:32:59.712 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-05-10T11:32:59.712 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-05-10T11:32:59.712 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-05-10T11:32:59.713 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-05-10T11:32:59.713 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-05-10T11:32:59.713 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-10T11:32:59.713 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-05-10T11:32:59.713 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-05-10T11:32:59.713 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-05-10T11:32:59.713 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-05-10T11:32:59.714 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-05-10T11:32:59.714 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-05-10T11:32:59.714 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-05-10T11:32:59.714 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-05-10T11:32:59.714 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-05-10T11:32:59.714 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 227705, 2024-05-10T11:32:59.714 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 172, 2024-05-10T11:32:59.714 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-05-10T11:32:59.715 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-05-10T11:32:59.715 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-05-10T11:32:59.715 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-05-10T11:32:59.715 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-05-10T11:32:59.715 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-05-10T11:32:59.715 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-05-10T11:32:59.715 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-05-10T11:32:59.716 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 9808, 2024-05-10T11:32:59.716 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 189, 2024-05-10T11:32:59.716 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-05-10T11:32:59.716 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-05-10T11:32:59.716 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-05-10T11:32:59.716 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-05-10T11:32:59.716 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 7334121, 2024-05-10T11:32:59.716 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 42119, 2024-05-10T11:32:59.717 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-05-10T11:32:59.717 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-05-10T11:32:59.717 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-05-10T11:32:59.717 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-05-10T11:32:59.717 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.717 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-05-10T11:32:59.717 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-05-10T11:32:59.717 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-05-10T11:32:59.718 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 4693, 2024-05-10T11:32:59.718 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 21, 2024-05-10T11:32:59.718 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 3, 2024-05-10T11:32:59.718 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 4693, 2024-05-10T11:32:59.718 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-05-10T11:32:59.718 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 4693, 2024-05-10T11:32:59.718 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 3.332933535, 2024-05-10T11:32:59.718 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000710192 2024-05-10T11:32:59.719 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.719 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-05-10T11:32:59.719 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-05-10T11:32:59.719 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 4693, 2024-05-10T11:32:59.719 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 4694 2024-05-10T11:32:59.719 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.719 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 4693, 2024-05-10T11:32:59.719 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 4693, 2024-05-10T11:32:59.720 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 0, 2024-05-10T11:32:59.720 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-05-10T11:32:59.720 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-05-10T11:32:59.720 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 2598, 2024-05-10T11:32:59.720 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 0, 2024-05-10T11:32:59.720 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 2093, 2024-05-10T11:32:59.720 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 0, 2024-05-10T11:32:59.721 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 0, 2024-05-10T11:32:59.721 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-05-10T11:32:59.721 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-05-10T11:32:59.721 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-05-10T11:32:59.721 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-05-10T11:32:59.721 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 0, 2024-05-10T11:32:59.721 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-05-10T11:32:59.722 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-05-10T11:32:59.722 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-05-10T11:32:59.722 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 1, 2024-05-10T11:32:59.722 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 0, 2024-05-10T11:32:59.722 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-05-10T11:32:59.722 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-05-10T11:32:59.722 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-05-10T11:32:59.722 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-05-10T11:32:59.723 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-05-10T11:32:59.723 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-05-10T11:32:59.723 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-05-10T11:32:59.723 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-05-10T11:32:59.723 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-05-10T11:32:59.723 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-05-10T11:32:59.723 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-05-10T11:32:59.723 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-05-10T11:32:59.724 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-05-10T11:32:59.724 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-05-10T11:32:59.724 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-05-10T11:32:59.724 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-05-10T11:32:59.725 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-05-10T11:32:59.725 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-05-10T11:32:59.725 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-05-10T11:32:59.725 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-05-10T11:32:59.725 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-05-10T11:32:59.725 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-05-10T11:32:59.725 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-05-10T11:32:59.726 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-05-10T11:32:59.726 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-05-10T11:32:59.726 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-05-10T11:32:59.726 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 117, 2024-05-10T11:32:59.726 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-05-10T11:32:59.727 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 29, 2024-05-10T11:32:59.727 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 9, 2024-05-10T11:32:59.727 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 9, 2024-05-10T11:32:59.727 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-05-10T11:32:59.727 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-05-10T11:32:59.727 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 0, 2024-05-10T11:32:59.728 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 2, 2024-05-10T11:32:59.728 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 0 2024-05-10T11:32:59.728 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.728 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-05-10T11:32:59.728 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 0, 2024-05-10T11:32:59.729 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-05-10T11:32:59.729 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-05-10T11:32:59.729 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-05-10T11:32:59.729 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.729 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-05-10T11:32:59.729 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-05-10T11:32:59.730 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.730 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-05-10T11:32:59.730 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-05-10T11:32:59.730 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-05-10T11:32:59.730 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-05-10T11:32:59.730 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-05-10T11:32:59.731 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-05-10T11:32:59.731 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-05-10T11:32:59.731 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-05-10T11:32:59.731 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.731 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-10T11:32:59.731 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:59.731 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-10T11:32:59.731 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:59.732 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 5176, 2024-05-10T11:32:59.732 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 77226206, 2024-05-10T11:32:59.732 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:59.732 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 5176, 2024-05-10T11:32:59.732 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-10T11:32:59.732 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-10T11:32:59.732 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 5176, 2024-05-10T11:32:59.733 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 77226206, 2024-05-10T11:32:59.733 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:59.733 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.733 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.733 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.733 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:59.733 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.734 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-05-10T11:32:59.734 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:59.734 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-10T11:32:59.734 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:59.734 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-10T11:32:59.734 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-10T11:32:59.735 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:59.735 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:59.735 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 4693, 2024-05-10T11:32:59.735 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 74332821, 2024-05-10T11:32:59.735 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 4693, 2024-05-10T11:32:59.735 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 74332821, 2024-05-10T11:32:59.735 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:59.736 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.736 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.736 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.736 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:59.736 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.736 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-05-10T11:32:59.737 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:59.737 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-05-10T11:32:59.737 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:59.737 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-10T11:32:59.737 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-10T11:32:59.737 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:59.737 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:59.738 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 4693, 2024-05-10T11:32:59.738 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 4693, 2024-05-10T11:32:59.738 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 4693, 2024-05-10T11:32:59.738 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 4693, 2024-05-10T11:32:59.738 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:59.739 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.739 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.739 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.739 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:59.739 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.739 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-05-10T11:32:59.740 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:59.740 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-10T11:32:59.740 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:59.740 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-10T11:32:59.740 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-10T11:32:59.740 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:59.741 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:59.741 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-10T11:32:59.741 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-10T11:32:59.741 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-05-10T11:32:59.741 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-05-10T11:32:59.741 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:59.742 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.742 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.742 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.742 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:59.742 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-10T11:32:59.742 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x56294a6dc6e0": { 2024-05-10T11:32:59.743 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-10T11:32:59.743 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-10T11:32:59.743 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-10T11:32:59.743 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-10T11:32:59.743 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-10T11:32:59.743 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-10T11:32:59.743 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-10T11:32:59.744 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-10T11:32:59.744 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-10T11:32:59.744 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-05-10T11:32:59.744 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-05-10T11:32:59.744 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-10T11:32:59.744 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-10T11:32:59.745 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-10T11:32:59.745 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-10T11:32:59.745 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:59.745 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-10T11:32:59.745 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-05-10T11:33:00.482 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-05-10T11:33:00.483 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-05-10T11:33:00.483 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-05-10T11:33:00.483 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-05-10T11:33:00.483 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-05-10T11:33:00.483 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-10T11:33:00.496 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-10T11:33:00.496 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T11:33:00.496 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-10T11:33:00.562 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-10T11:33:00.562 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-10T11:33:00.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd blocklist ls 2024-05-10T11:33:01.096 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:33:01.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:00 smithi027 ceph-mon[30054]: pgmap v622: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 79 KiB/s wr, 17 op/s 2024-05-10T11:33:01.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:00 smithi076 ceph-mon[34152]: pgmap v622: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 79 KiB/s wr, 17 op/s 2024-05-10T11:33:01.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:00 smithi189 ceph-mon[34842]: pgmap v622: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 79 KiB/s wr, 17 op/s 2024-05-10T11:33:03.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:02 smithi076 ceph-mon[34152]: pgmap v623: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 56 KiB/s wr, 17 op/s 2024-05-10T11:33:03.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:02 smithi076 ceph-mon[34152]: osdmap e118: 12 total, 12 up, 12 in 2024-05-10T11:33:03.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:02 smithi027 ceph-mon[30054]: pgmap v623: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 56 KiB/s wr, 17 op/s 2024-05-10T11:33:03.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:02 smithi027 ceph-mon[30054]: osdmap e118: 12 total, 12 up, 12 in 2024-05-10T11:33:03.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:02 smithi189 ceph-mon[34842]: pgmap v623: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 56 KiB/s wr, 17 op/s 2024-05-10T11:33:03.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:02 smithi189 ceph-mon[34842]: osdmap e118: 12 total, 12 up, 12 in 2024-05-10T11:33:03.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.697+0000 7f062be00700 1 -- 172.21.15.27:0/3535214998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0624104160 msgr2=0x7f0624108150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:03.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.697+0000 7f062be00700 1 --2- 172.21.15.27:0/3535214998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0624104160 0x7f0624108150 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0610009990 tx=0x7f0610009ca0 comp rx=0 tx=0).stop 2024-05-10T11:33:03.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.697+0000 7f062be00700 1 -- 172.21.15.27:0/3535214998 shutdown_connections 2024-05-10T11:33:03.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.697+0000 7f062be00700 1 --2- 172.21.15.27:0/3535214998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0624108880 0x7f0624110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:03.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.697+0000 7f062be00700 1 --2- 172.21.15.27:0/3535214998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f06241037b0 0x7f0624103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:03.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.697+0000 7f062be00700 1 --2- 172.21.15.27:0/3535214998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0624104160 0x7f0624108150 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:03.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.697+0000 7f062be00700 1 -- 172.21.15.27:0/3535214998 >> 172.21.15.27:0/3535214998 conn(0x7f062407a530 msgr2=0x7f062407a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:03.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.697+0000 7f062be00700 1 -- 172.21.15.27:0/3535214998 shutdown_connections 2024-05-10T11:33:03.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.698+0000 7f062be00700 1 -- 172.21.15.27:0/3535214998 wait complete. 2024-05-10T11:33:03.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.699+0000 7f062be00700 1 Processor -- start 2024-05-10T11:33:03.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.699+0000 7f062be00700 1 -- start start 2024-05-10T11:33:03.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.700+0000 7f062be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f06241037b0 0x7f06241a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:03.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.700+0000 7f062be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0624104160 0x7f06241a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:03.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.700+0000 7f062be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0624108880 0x7f062419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:03.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.700+0000 7f062be00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0624113640 con 0x7f0624104160 2024-05-10T11:33:03.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.700+0000 7f062be00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f06241134c0 con 0x7f06241037b0 2024-05-10T11:33:03.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.700+0000 7f062be00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f06241137c0 con 0x7f0624108880 2024-05-10T11:33:03.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.700+0000 7f0629400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f06241037b0 0x7f06241a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:03.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.700+0000 7f0628a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0624104160 0x7f06241a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:03.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.700+0000 7f0629400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f06241037b0 0x7f06241a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:54012/0 (socket says 172.21.15.27:54012) 2024-05-10T11:33:03.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.700+0000 7f0629400700 1 -- 172.21.15.27:0/2702444782 learned_addr learned my addr 172.21.15.27:0/2702444782 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:33:03.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.700+0000 7f0629e00700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0624108880 0x7f062419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:03.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.701+0000 7f0629400700 1 -- 172.21.15.27:0/2702444782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0624108880 msgr2=0x7f062419bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:03.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.701+0000 7f0629400700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0624108880 0x7f062419bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:03.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.701+0000 7f0629400700 1 -- 172.21.15.27:0/2702444782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0624104160 msgr2=0x7f06241a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:03.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.701+0000 7f0629400700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0624104160 0x7f06241a1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:03.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.701+0000 7f0629400700 1 -- 172.21.15.27:0/2702444782 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062419c260 con 0x7f06241037b0 2024-05-10T11:33:03.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.701+0000 7f0629400700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f06241037b0 0x7f06241a1940 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f061800eb30 tx=0x7f061800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:03.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.702+0000 7f0622000700 1 -- 172.21.15.27:0/2702444782 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061800cbd0 con 0x7f06241037b0 2024-05-10T11:33:03.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.702+0000 7f062be00700 1 -- 172.21.15.27:0/2702444782 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f062419c550 con 0x7f06241037b0 2024-05-10T11:33:03.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.702+0000 7f0622000700 1 -- 172.21.15.27:0/2702444782 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06180044d0 con 0x7f06241037b0 2024-05-10T11:33:03.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.702+0000 7f0622000700 1 -- 172.21.15.27:0/2702444782 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618003df0 con 0x7f06241037b0 2024-05-10T11:33:03.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.702+0000 7f062be00700 1 -- 172.21.15.27:0/2702444782 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06241a8810 con 0x7f06241037b0 2024-05-10T11:33:03.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.703+0000 7f062be00700 1 -- 172.21.15.27:0/2702444782 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f062406a190 con 0x7f06241037b0 2024-05-10T11:33:03.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.706+0000 7f0622000700 1 -- 172.21.15.27:0/2702444782 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0618004000 con 0x7f06241037b0 2024-05-10T11:33:03.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.707+0000 7f0622000700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f060c0b2a70 0x7f060c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:03.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.707+0000 7f0622000700 1 -- 172.21.15.27:0/2702444782 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9679+0+0 (secure 0 0 0) 0x7f06180e2dc0 con 0x7f06241037b0 2024-05-10T11:33:03.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.708+0000 7f0628a00700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f060c0b2a70 0x7f060c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:03.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.708+0000 7f0628a00700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f060c0b2a70 0x7f060c0b4f30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f06100098e0 tx=0x7f0610005960 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:03.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.709+0000 7f0622000700 1 -- 172.21.15.27:0/2702444782 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f06180042f0 con 0x7f06241037b0 2024-05-10T11:33:03.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.975+0000 7f062be00700 1 -- 172.21.15.27:0/2702444782 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f062419cff0 con 0x7f06241037b0 2024-05-10T11:33:03.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.975+0000 7f0622000700 1 -- 172.21.15.27:0/2702444782 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v118) ==== 81+0+944 (secure 0 0 0) 0x7f06180aa350 con 0x7f06241037b0 2024-05-10T11:33:03.976 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1021663132 2024-05-11T11:13:13.118713+0000 2024-05-10T11:33:03.977 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/1704535643 2024-05-11T11:13:13.118713+0000 2024-05-10T11:33:03.977 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2444542866 2024-05-11T10:59:20.613831+0000 2024-05-10T11:33:03.977 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1450791252 2024-05-11T11:13:13.118713+0000 2024-05-10T11:33:03.977 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/481366655 2024-05-11T10:59:20.613831+0000 2024-05-10T11:33:03.977 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/481366655 2024-05-11T10:59:20.613831+0000 2024-05-10T11:33:03.977 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2743539323 2024-05-11T10:59:20.613831+0000 2024-05-10T11:33:03.977 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3803325728 2024-05-11T10:58:45.437819+0000 2024-05-10T11:33:03.977 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1410017298 2024-05-11T11:13:13.118713+0000 2024-05-10T11:33:03.977 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/2058505174 2024-05-11T10:58:45.437819+0000 2024-05-10T11:33:03.978 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/1704535643 2024-05-11T11:13:13.118713+0000 2024-05-10T11:33:03.978 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/2058505174 2024-05-11T10:58:45.437819+0000 2024-05-10T11:33:03.978 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2991615845 2024-05-11T11:13:13.118713+0000 2024-05-10T11:33:03.978 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1886129738 2024-05-11T10:58:45.437819+0000 2024-05-10T11:33:03.978 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1208957722 2024-05-11T10:59:20.613831+0000 2024-05-10T11:33:03.978 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4158621178 2024-05-11T10:58:45.437819+0000 2024-05-10T11:33:03.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.979+0000 7f062be00700 1 -- 172.21.15.27:0/2702444782 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f060c0b2a70 msgr2=0x7f060c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:03.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.979+0000 7f062be00700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f060c0b2a70 0x7f060c0b4f30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f06100098e0 tx=0x7f0610005960 comp rx=0 tx=0).stop 2024-05-10T11:33:03.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.979+0000 7f062be00700 1 -- 172.21.15.27:0/2702444782 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f06241037b0 msgr2=0x7f06241a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:03.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.979+0000 7f062be00700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f06241037b0 0x7f06241a1940 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f061800eb30 tx=0x7f061800ee40 comp rx=0 tx=0).stop 2024-05-10T11:33:03.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.980+0000 7f062be00700 1 -- 172.21.15.27:0/2702444782 shutdown_connections 2024-05-10T11:33:03.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.980+0000 7f062be00700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f060c0b2a70 0x7f060c0b4f30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:03.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.980+0000 7f062be00700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0624108880 0x7f062419bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:03.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.980+0000 7f062be00700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f06241037b0 0x7f06241a1940 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:03.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.980+0000 7f062be00700 1 --2- 172.21.15.27:0/2702444782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0624104160 0x7f06241a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:03.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.980+0000 7f062be00700 1 -- 172.21.15.27:0/2702444782 >> 172.21.15.27:0/2702444782 conn(0x7f062407a530 msgr2=0x7f0624101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:03.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.980+0000 7f062be00700 1 -- 172.21.15.27:0/2702444782 shutdown_connections 2024-05-10T11:33:03.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:03.980+0000 7f062be00700 1 -- 172.21.15.27:0/2702444782 wait complete. 2024-05-10T11:33:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:listed 16 entries 2024-05-10T11:33:04.969 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-10T11:33:04.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:33:04.969 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-10T11:33:05.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:04 smithi189 ceph-mon[34842]: pgmap v625: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 67 KiB/s wr, 19 op/s 2024-05-10T11:33:05.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:04 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/2702444782' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-10T11:33:05.350 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:04 smithi027 ceph-mon[30054]: pgmap v625: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 67 KiB/s wr, 19 op/s 2024-05-10T11:33:05.350 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:04 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/2702444782' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-10T11:33:05.352 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-05-10T11:33:05.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:33:05.352 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-10T11:33:05.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:04 smithi076 ceph-mon[34152]: pgmap v625: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 67 KiB/s wr, 19 op/s 2024-05-10T11:33:05.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:04 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/2702444782' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-10T11:33:05.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:33:05.439 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-05-10T11:33:05.517 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-10T11:33:05.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:33:05.518 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-10T11:33:05.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-10T11:33:05.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-05-10T11:33:05.677 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-10T11:33:05.688 INFO:tasks.cephadm:Teardown begin 2024-05-10T11:33:05.688 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-10T11:33:05.689 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:33:06.104 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:33:07.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:06 smithi189 ceph-mon[34842]: pgmap v626: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 277 KiB/s wr, 35 op/s 2024-05-10T11:33:07.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:06 smithi027 ceph-mon[30054]: pgmap v626: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 277 KiB/s wr, 35 op/s 2024-05-10T11:33:07.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:06 smithi076 ceph-mon[34152]: pgmap v626: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 277 KiB/s wr, 35 op/s 2024-05-10T11:33:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.485+0000 7f3fc4200700 1 -- 172.21.15.27:0/4121796148 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc104100 msgr2=0x7f3fbc108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.485+0000 7f3fc4200700 1 --2- 172.21.15.27:0/4121796148 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc104100 0x7f3fbc108120 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8009920 tx=0x7f3fa8009c30 comp rx=0 tx=0).stop 2024-05-10T11:33:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.485+0000 7f3fc4200700 1 -- 172.21.15.27:0/4121796148 shutdown_connections 2024-05-10T11:33:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.485+0000 7f3fc4200700 1 --2- 172.21.15.27:0/4121796148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fbc103750 0x7f3fbc103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.485+0000 7f3fc4200700 1 --2- 172.21.15.27:0/4121796148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc108850 0x7f3fbc1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.485+0000 7f3fc4200700 1 --2- 172.21.15.27:0/4121796148 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc104100 0x7f3fbc108120 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.485+0000 7f3fc4200700 1 -- 172.21.15.27:0/4121796148 >> 172.21.15.27:0/4121796148 conn(0x7f3fbc07a600 msgr2=0x7f3fbc07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.485+0000 7f3fc4200700 1 -- 172.21.15.27:0/4121796148 shutdown_connections 2024-05-10T11:33:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.485+0000 7f3fc4200700 1 -- 172.21.15.27:0/4121796148 wait complete. 2024-05-10T11:33:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.487+0000 7f3fc4200700 1 Processor -- start 2024-05-10T11:33:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.487+0000 7f3fc4200700 1 -- start start 2024-05-10T11:33:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.488+0000 7f3fc4200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fbc103750 0x7f3fbc19bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.488+0000 7f3fc4200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc104100 0x7f3fbc19c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.488+0000 7f3fc4200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc108850 0x7f3fbc1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.488+0000 7f3fc4200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc113610 con 0x7f3fbc104100 2024-05-10T11:33:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.488+0000 7f3fc4200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc113490 con 0x7f3fbc108850 2024-05-10T11:33:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.488+0000 7f3fc4200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc113790 con 0x7f3fbc103750 2024-05-10T11:33:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.489+0000 7f3fc0e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc104100 0x7f3fbc19c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.489+0000 7f3fc1800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fbc103750 0x7f3fbc19bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.489+0000 7f3fc0e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc104100 0x7f3fbc19c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53612/0 (socket says 172.21.15.27:53612) 2024-05-10T11:33:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.489+0000 7f3fc0e00700 1 -- 172.21.15.27:0/257973608 learned_addr learned my addr 172.21.15.27:0/257973608 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:33:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.489+0000 7f3fc2200700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc108850 0x7f3fbc1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.489+0000 7f3fc1800700 1 -- 172.21.15.27:0/257973608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc108850 msgr2=0x7f3fbc1a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.489+0000 7f3fc1800700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc108850 0x7f3fbc1a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.489+0000 7f3fc1800700 1 -- 172.21.15.27:0/257973608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc104100 msgr2=0x7f3fbc19c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.489+0000 7f3fc1800700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc104100 0x7f3fbc19c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.489+0000 7f3fc1800700 1 -- 172.21.15.27:0/257973608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fbc1a0df0 con 0x7f3fbc103750 2024-05-10T11:33:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.489+0000 7f3fc0e00700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc104100 0x7f3fbc19c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:33:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.489+0000 7f3fc2200700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc108850 0x7f3fbc1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:33:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.490+0000 7f3fc1800700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fbc103750 0x7f3fbc19bde0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000ef10 tx=0x7f3fb000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.490+0000 7f3fba000700 1 -- 172.21.15.27:0/257973608 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb000c9c0 con 0x7f3fbc103750 2024-05-10T11:33:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.490+0000 7f3fba000700 1 -- 172.21.15.27:0/257973608 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fb0019070 con 0x7f3fbc103750 2024-05-10T11:33:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.490+0000 7f3fba000700 1 -- 172.21.15.27:0/257973608 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0014480 con 0x7f3fbc103750 2024-05-10T11:33:08.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.490+0000 7f3fc4200700 1 -- 172.21.15.27:0/257973608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fbc1a1080 con 0x7f3fbc103750 2024-05-10T11:33:08.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.490+0000 7f3fc4200700 1 -- 172.21.15.27:0/257973608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fbc1a8960 con 0x7f3fbc103750 2024-05-10T11:33:08.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.491+0000 7f3fc4200700 1 -- 172.21.15.27:0/257973608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fbc06a190 con 0x7f3fbc103750 2024-05-10T11:33:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.494+0000 7f3fba000700 1 -- 172.21.15.27:0/257973608 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3fb0012050 con 0x7f3fbc103750 2024-05-10T11:33:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.495+0000 7f3fba000700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3fa40b2a70 0x7f3fa40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.496+0000 7f3fba000700 1 -- 172.21.15.27:0/257973608 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9679+0+0 (secure 0 0 0) 0x7f3fb00e2ff0 con 0x7f3fbc103750 2024-05-10T11:33:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.496+0000 7f3fc0e00700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3fa40b2a70 0x7f3fa40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:08.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.497+0000 7f3fc0e00700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3fa40b2a70 0x7f3fa40b4f30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8009680 tx=0x7f3fa8019040 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:08.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.497+0000 7f3fba000700 1 -- 172.21.15.27:0/257973608 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f3fb0010040 con 0x7f3fbc103750 2024-05-10T11:33:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.790+0000 7f3fc4200700 1 -- 172.21.15.27:0/257973608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fbc1a8b30 con 0x7f3fbc103750 2024-05-10T11:33:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.791+0000 7f3fba000700 1 -- 172.21.15.27:0/257973608 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11936 (secure 0 0 0) 0x7f3fb00aa4f0 con 0x7f3fbc103750 2024-05-10T11:33:08.792 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:33:08.792 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:29:51.629207+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:33:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.794+0000 7f3fc4200700 1 -- 172.21.15.27:0/257973608 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3fa40b2a70 msgr2=0x7f3fa40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.794+0000 7f3fc4200700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3fa40b2a70 0x7f3fa40b4f30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8009680 tx=0x7f3fa8019040 comp rx=0 tx=0).stop 2024-05-10T11:33:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.794+0000 7f3fc4200700 1 -- 172.21.15.27:0/257973608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fbc103750 msgr2=0x7f3fbc19bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.794+0000 7f3fc4200700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fbc103750 0x7f3fbc19bde0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000ef10 tx=0x7f3fb000c3a0 comp rx=0 tx=0).stop 2024-05-10T11:33:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.795+0000 7f3fc4200700 1 -- 172.21.15.27:0/257973608 shutdown_connections 2024-05-10T11:33:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.795+0000 7f3fc4200700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f3fa40b2a70 0x7f3fa40b4f30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.795+0000 7f3fc4200700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fbc103750 0x7f3fbc19bde0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.795+0000 7f3fc4200700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc108850 0x7f3fbc1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.795+0000 7f3fc4200700 1 --2- 172.21.15.27:0/257973608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc104100 0x7f3fbc19c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:08.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.795+0000 7f3fc4200700 1 -- 172.21.15.27:0/257973608 >> 172.21.15.27:0/257973608 conn(0x7f3fbc07a600 msgr2=0x7f3fbc101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:08.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.795+0000 7f3fc4200700 1 -- 172.21.15.27:0/257973608 shutdown_connections 2024-05-10T11:33:08.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:08.795+0000 7f3fc4200700 1 -- 172.21.15.27:0/257973608 wait complete. 2024-05-10T11:33:08.797 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 45 2024-05-10T11:33:09.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:08 smithi027 ceph-mon[30054]: pgmap v627: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 277 KiB/s wr, 35 op/s 2024-05-10T11:33:09.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:08 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/257973608' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:33:09.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:08 smithi027 ceph-mon[30054]: osdmap e119: 12 total, 12 up, 12 in 2024-05-10T11:33:09.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:08 smithi189 ceph-mon[34842]: pgmap v627: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 277 KiB/s wr, 35 op/s 2024-05-10T11:33:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:08 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/257973608' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:33:09.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:08 smithi189 ceph-mon[34842]: osdmap e119: 12 total, 12 up, 12 in 2024-05-10T11:33:09.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:08 smithi076 ceph-mon[34152]: pgmap v627: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 277 KiB/s wr, 35 op/s 2024-05-10T11:33:09.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:08 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/257973608' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:33:09.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:08 smithi076 ceph-mon[34152]: osdmap e119: 12 total, 12 up, 12 in 2024-05-10T11:33:09.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs dump --format=json 2024-05-10T11:33:10.270 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:33:11.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:10 smithi189 ceph-mon[34842]: pgmap v629: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 356 KiB/s wr, 39 op/s 2024-05-10T11:33:11.361 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:10 smithi027 ceph-mon[30054]: pgmap v629: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 356 KiB/s wr, 39 op/s 2024-05-10T11:33:11.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:10 smithi076 ceph-mon[34152]: pgmap v629: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 356 KiB/s wr, 39 op/s 2024-05-10T11:33:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.913+0000 7f6873a00700 1 -- 172.21.15.27:0/2187756181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c06b1b0 msgr2=0x7f686c06b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.913+0000 7f6873a00700 1 --2- 172.21.15.27:0/2187756181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c06b1b0 0x7f686c06b590 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f685c009990 tx=0x7f685c009ca0 comp rx=0 tx=0).stop 2024-05-10T11:33:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.914+0000 7f6873a00700 1 -- 172.21.15.27:0/2187756181 shutdown_connections 2024-05-10T11:33:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.914+0000 7f6873a00700 1 --2- 172.21.15.27:0/2187756181 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f686c10ff50 0x7f686c112340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.914+0000 7f6873a00700 1 --2- 172.21.15.27:0/2187756181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c06b1b0 0x7f686c06b590 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.914+0000 7f6873a00700 1 --2- 172.21.15.27:0/2187756181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686c06bad0 0x7f686c10fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.914+0000 7f6873a00700 1 -- 172.21.15.27:0/2187756181 >> 172.21.15.27:0/2187756181 conn(0x7f686c079350 msgr2=0x7f686c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:12.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.914+0000 7f6873a00700 1 -- 172.21.15.27:0/2187756181 shutdown_connections 2024-05-10T11:33:12.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.914+0000 7f6873a00700 1 -- 172.21.15.27:0/2187756181 wait complete. 2024-05-10T11:33:12.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.915+0000 7f6873a00700 1 Processor -- start 2024-05-10T11:33:12.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.915+0000 7f6873a00700 1 -- start start 2024-05-10T11:33:12.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.915+0000 7f6873a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686c06b1b0 0x7f686c19bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:12.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.915+0000 7f6873a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f686c06bad0 0x7f686c19c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:12.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.915+0000 7f6873a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c10ff50 0x7f686c1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:12.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.915+0000 7f6873a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f686c1158c0 con 0x7f686c10ff50 2024-05-10T11:33:12.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.915+0000 7f6873a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f686c115740 con 0x7f686c06bad0 2024-05-10T11:33:12.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.915+0000 7f6873a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f686c115a40 con 0x7f686c06b1b0 2024-05-10T11:33:12.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.916+0000 7f6863e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f686c06bad0 0x7f686c19c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:12.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.916+0000 7f6871000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686c06b1b0 0x7f686c19bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.916+0000 7f6863e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f686c06bad0 0x7f686c19c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:36790/0 (socket says 172.21.15.27:36790) 2024-05-10T11:33:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.916+0000 7f6871a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c10ff50 0x7f686c1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.916+0000 7f6863e00700 1 -- 172.21.15.27:0/1063108566 learned_addr learned my addr 172.21.15.27:0/1063108566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:33:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.916+0000 7f6863e00700 1 -- 172.21.15.27:0/1063108566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686c06b1b0 msgr2=0x7f686c19bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.916+0000 7f6863e00700 1 --2- 172.21.15.27:0/1063108566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686c06b1b0 0x7f686c19bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.916+0000 7f6863e00700 1 -- 172.21.15.27:0/1063108566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c10ff50 msgr2=0x7f686c1a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.916+0000 7f6863e00700 1 --2- 172.21.15.27:0/1063108566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c10ff50 0x7f686c1a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.916+0000 7f6863e00700 1 -- 172.21.15.27:0/1063108566 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f686c1a0de0 con 0x7f686c06bad0 2024-05-10T11:33:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.917+0000 7f6863e00700 1 --2- 172.21.15.27:0/1063108566 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f686c06bad0 0x7f686c19c310 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f685400e9c0 tx=0x7f685400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.917+0000 7f686a000700 1 -- 172.21.15.27:0/1063108566 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685400cbd0 con 0x7f686c06bad0 2024-05-10T11:33:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.917+0000 7f6873a00700 1 -- 172.21.15.27:0/1063108566 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f686c1a10d0 con 0x7f686c06bad0 2024-05-10T11:33:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.917+0000 7f686a000700 1 -- 172.21.15.27:0/1063108566 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f685400ee60 con 0x7f686c06bad0 2024-05-10T11:33:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.917+0000 7f686a000700 1 -- 172.21.15.27:0/1063108566 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68540055c0 con 0x7f686c06bad0 2024-05-10T11:33:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.917+0000 7f6873a00700 1 -- 172.21.15.27:0/1063108566 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f686c1a8950 con 0x7f686c06bad0 2024-05-10T11:33:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.918+0000 7f6873a00700 1 -- 172.21.15.27:0/1063108566 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f686c06a190 con 0x7f686c06bad0 2024-05-10T11:33:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.921+0000 7f686a000700 1 -- 172.21.15.27:0/1063108566 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f6854004750 con 0x7f686c06bad0 2024-05-10T11:33:12.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.922+0000 7f686a000700 1 --2- 172.21.15.27:0/1063108566 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f68500b29f0 0x7f68500b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:12.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.923+0000 7f686a000700 1 -- 172.21.15.27:0/1063108566 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9591+0+0 (secure 0 0 0) 0x7f68540e2d30 con 0x7f686c06bad0 2024-05-10T11:33:12.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.923+0000 7f6871000700 1 --2- 172.21.15.27:0/1063108566 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f68500b29f0 0x7f68500b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.923+0000 7f6871000700 1 --2- 172.21.15.27:0/1063108566 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f68500b29f0 0x7f68500b4eb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f685c005b00 tx=0x7f685c005a90 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:12.926+0000 7f686a000700 1 -- 172.21.15.27:0/1063108566 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f68540aa280 con 0x7f686c06bad0 2024-05-10T11:33:13.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:12 smithi027 ceph-mon[30054]: pgmap v630: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 290 KiB/s wr, 33 op/s 2024-05-10T11:33:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.224+0000 7f6873a00700 1 -- 172.21.15.27:0/1063108566 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f686c02e510 con 0x7f686c06bad0 2024-05-10T11:33:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.225+0000 7f686a000700 1 -- 172.21.15.27:0/1063108566 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11936 (secure 0 0 0) 0x7f68540a9b80 con 0x7f686c06bad0 2024-05-10T11:33:13.226 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:33:13.226 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1490955215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1490955215},{"type":"v1","addr":"172.21.15.27:6837","nonce":1490955215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3240182674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3240182674},{"type":"v1","addr":"172.21.15.27:6839","nonce":3240182674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-10T11:07:50.237787+0000","modified":"2024-05-10T11:29:51.629207+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24488,"mds_2":24484,"mds_3":24472,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14403":{"gid":14403,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/3080402649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":3080402649},{"type":"v1","addr":"172.21.15.27:6841","nonce":3080402649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2718624698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2718624698},{"type":"v1","addr":"172.21.15.189:6833","nonce":2718624698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6833/1370063583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":1370063583},{"type":"v1","addr":"172.21.15.76:6833","nonce":1370063583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/3274866857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3274866857},{"type":"v1","addr":"172.21.15.27:6835","nonce":3274866857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1387617890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1387617890},{"type":"v1","addr":"172.21.15.189:6835","nonce":1387617890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":35,"addr":"172.21.15.76:6835/3618511593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3618511593},{"type":"v1","addr":"172.21.15.76:6835","nonce":3618511593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":33,"addr":"172.21.15.189:6837/3185830081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3185830081},{"type":"v1","addr":"172.21.15.189:6837","nonce":3185830081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.189:6839/2071082194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2071082194},{"type":"v1","addr":"172.21.15.189:6839","nonce":2071082194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.76:6837/569330029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":569330029},{"type":"v1","addr":"172.21.15.76:6837","nonce":569330029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.76:6839/1778204420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1778204420},{"type":"v1","addr":"172.21.15.76:6839","nonce":1778204420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24467,24488,24472,24484,24527]},"id":1}]} 2024-05-10T11:33:13.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.229+0000 7f6873a00700 1 -- 172.21.15.27:0/1063108566 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f68500b29f0 msgr2=0x7f68500b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:13.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.229+0000 7f6873a00700 1 --2- 172.21.15.27:0/1063108566 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f68500b29f0 0x7f68500b4eb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f685c005b00 tx=0x7f685c005a90 comp rx=0 tx=0).stop 2024-05-10T11:33:13.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.229+0000 7f6873a00700 1 -- 172.21.15.27:0/1063108566 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f686c06bad0 msgr2=0x7f686c19c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:13.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.229+0000 7f6873a00700 1 --2- 172.21.15.27:0/1063108566 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f686c06bad0 0x7f686c19c310 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f685400e9c0 tx=0x7f685400ecd0 comp rx=0 tx=0).stop 2024-05-10T11:33:13.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.229+0000 7f6873a00700 1 -- 172.21.15.27:0/1063108566 shutdown_connections 2024-05-10T11:33:13.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.229+0000 7f6873a00700 1 --2- 172.21.15.27:0/1063108566 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f68500b29f0 0x7f68500b4eb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:13.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.229+0000 7f6873a00700 1 --2- 172.21.15.27:0/1063108566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686c06b1b0 0x7f686c19bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:13.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.229+0000 7f6873a00700 1 --2- 172.21.15.27:0/1063108566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c10ff50 0x7f686c1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:13.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.229+0000 7f6873a00700 1 --2- 172.21.15.27:0/1063108566 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f686c06bad0 0x7f686c19c310 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:13.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.229+0000 7f6873a00700 1 -- 172.21.15.27:0/1063108566 >> 172.21.15.27:0/1063108566 conn(0x7f686c079350 msgr2=0x7f686c111400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:13.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.229+0000 7f6873a00700 1 -- 172.21.15.27:0/1063108566 shutdown_connections 2024-05-10T11:33:13.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:13.229+0000 7f6873a00700 1 -- 172.21.15.27:0/1063108566 wait complete. 2024-05-10T11:33:13.231 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 45 2024-05-10T11:33:13.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:12 smithi189 ceph-mon[34842]: pgmap v630: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 290 KiB/s wr, 33 op/s 2024-05-10T11:33:13.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:12 smithi076 ceph-mon[34152]: pgmap v630: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 290 KiB/s wr, 33 op/s 2024-05-10T11:33:14.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:13 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1063108566' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:33:14.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:13 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:33:14.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:13 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1063108566' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:33:14.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:13 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:33:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:13 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1063108566' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-10T11:33:14.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:13 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-10T11:33:14.579 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-10T11:33:15.037 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:33:15.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:14 smithi027 ceph-mon[30054]: pgmap v631: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 285 KiB/s wr, 33 op/s 2024-05-10T11:33:15.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:14 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:33:15.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:14 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:33:15.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:14 smithi027 ceph-mon[30054]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:33:15.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:14 smithi189 ceph-mon[34842]: pgmap v631: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 285 KiB/s wr, 33 op/s 2024-05-10T11:33:15.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:14 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:33:15.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:14 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:33:15.237 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:14 smithi189 ceph-mon[34842]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:33:15.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:14 smithi076 ceph-mon[34152]: pgmap v631: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 285 KiB/s wr, 33 op/s 2024-05-10T11:33:15.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:14 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-10T11:33:15.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:14 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-10T11:33:15.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:14 smithi076 ceph-mon[34152]: from='mgr.25220 ' entity='mgr.x' 2024-05-10T11:33:17.187 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:16 smithi027 ceph-mon[30054]: pgmap v632: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-05-10T11:33:17.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:16 smithi189 ceph-mon[34842]: pgmap v632: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-05-10T11:33:17.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:16 smithi076 ceph-mon[34152]: pgmap v632: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-05-10T11:33:17.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.508+0000 7f620da00700 1 -- 172.21.15.27:0/2959463501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62081060a0 msgr2=0x7f62081125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:17.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.508+0000 7f620da00700 1 --2- 172.21.15.27:0/2959463501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62081060a0 0x7f62081125b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f620000b0a0 tx=0x7f620000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:33:17.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.509+0000 7f620da00700 1 -- 172.21.15.27:0/2959463501 shutdown_connections 2024-05-10T11:33:17.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.509+0000 7f620da00700 1 --2- 172.21.15.27:0/2959463501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62081060a0 0x7f62081125b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:17.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.509+0000 7f620da00700 1 --2- 172.21.15.27:0/2959463501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f620810b6e0 0x7f620810bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:17.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.509+0000 7f620da00700 1 --2- 172.21.15.27:0/2959463501 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62081056e0 0x7f6208105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:17.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.509+0000 7f620da00700 1 -- 172.21.15.27:0/2959463501 >> 172.21.15.27:0/2959463501 conn(0x7f62081011d0 msgr2=0x7f62081035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:17.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.509+0000 7f620da00700 1 -- 172.21.15.27:0/2959463501 shutdown_connections 2024-05-10T11:33:17.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.509+0000 7f620da00700 1 -- 172.21.15.27:0/2959463501 wait complete. 2024-05-10T11:33:17.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.510+0000 7f620da00700 1 Processor -- start 2024-05-10T11:33:17.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.511+0000 7f620da00700 1 -- start start 2024-05-10T11:33:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.511+0000 7f620da00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62081056e0 0x7f620819bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.511+0000 7f620da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62081060a0 0x7f620819c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.511+0000 7f620da00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620810b6e0 0x7f62081a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:17.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.511+0000 7f620da00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6208115930 con 0x7f62081060a0 2024-05-10T11:33:17.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.511+0000 7f620da00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f62081157b0 con 0x7f620810b6e0 2024-05-10T11:33:17.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.511+0000 7f620da00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6208115ab0 con 0x7f62081056e0 2024-05-10T11:33:17.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.511+0000 7f6206a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62081056e0 0x7f620819bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:17.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.511+0000 7f6206a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62081056e0 0x7f620819bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:36376/0 (socket says 172.21.15.27:36376) 2024-05-10T11:33:17.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.511+0000 7f6206a00700 1 -- 172.21.15.27:0/1854279565 learned_addr learned my addr 172.21.15.27:0/1854279565 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:33:17.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.511+0000 7f6206000700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62081060a0 0x7f620819c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:17.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.512+0000 7f6207400700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620810b6e0 0x7f62081a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:17.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.512+0000 7f6206a00700 1 -- 172.21.15.27:0/1854279565 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620810b6e0 msgr2=0x7f62081a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:17.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.512+0000 7f6206a00700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620810b6e0 0x7f62081a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:17.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.512+0000 7f6206a00700 1 -- 172.21.15.27:0/1854279565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62081060a0 msgr2=0x7f620819c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:17.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.512+0000 7f6206a00700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62081060a0 0x7f620819c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:17.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.512+0000 7f6206a00700 1 -- 172.21.15.27:0/1854279565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62081a0d70 con 0x7f62081056e0 2024-05-10T11:33:17.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.512+0000 7f6206a00700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62081056e0 0x7f620819bd60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f61f800d980 tx=0x7f61f800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:17.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.512+0000 7f61ff400700 1 -- 172.21.15.27:0/1854279565 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f8016070 con 0x7f62081056e0 2024-05-10T11:33:17.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.512+0000 7f61ff400700 1 -- 172.21.15.27:0/1854279565 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61f8004830 con 0x7f62081056e0 2024-05-10T11:33:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.512+0000 7f620da00700 1 -- 172.21.15.27:0/1854279565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62081a1060 con 0x7f62081056e0 2024-05-10T11:33:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.513+0000 7f61ff400700 1 -- 172.21.15.27:0/1854279565 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f80054c0 con 0x7f62081056e0 2024-05-10T11:33:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.513+0000 7f620da00700 1 -- 172.21.15.27:0/1854279565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62081a88e0 con 0x7f62081056e0 2024-05-10T11:33:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.514+0000 7f620da00700 1 -- 172.21.15.27:0/1854279565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f620806a190 con 0x7f62081056e0 2024-05-10T11:33:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.516+0000 7f61ff400700 1 -- 172.21.15.27:0/1854279565 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f61f8004220 con 0x7f62081056e0 2024-05-10T11:33:17.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.518+0000 7f61ff400700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f61ec0b2a70 0x7f61ec0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:17.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.518+0000 7f61ff400700 1 -- 172.21.15.27:0/1854279565 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9591+0+0 (secure 0 0 0) 0x7f61f80e2ce0 con 0x7f62081056e0 2024-05-10T11:33:17.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.518+0000 7f6206000700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f61ec0b2a70 0x7f61ec0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:17.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.519+0000 7f6206000700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f61ec0b2a70 0x7f61ec0b4f30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f620810be70 tx=0x7f61f400b3c0 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.524+0000 7f61ff400700 1 -- 172.21.15.27:0/1854279565 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f61f801f020 con 0x7f62081056e0 2024-05-10T11:33:17.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.762+0000 7f620da00700 1 -- 172.21.15.27:0/1854279565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f620802e510 con 0x7f62081056e0 2024-05-10T11:33:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.764+0000 7f61ff400700 1 -- 172.21.15.27:0/1854279565 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) ==== 74+0+25401 (secure 0 0 0) 0x7f61f8055e20 con 0x7f62081056e0 2024-05-10T11:33:17.766 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-10T11:33:17.766 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":119,"fsid":"063c9e18-0ebc-11ef-bc98-c7b262605968","created":"2024-05-10T10:58:11.299955+0000","modified":"2024-05-10T11:33:08.885367+0000","last_up_change":"2024-05-10T11:06:29.970346+0000","last_in_change":"2024-05-10T11:06:09.902240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-10T11:02:06.781475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-10T11:07:40.667104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-10T11:07:45.158839+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":35,"snap_epoch":118,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-10T11:07:58.869099+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":35,"snap_epoch":118,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df11dcf5-f3ad-4b7c-99e7-e26f7756278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6803","nonce":677446345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6805","nonce":677446345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6809","nonce":677446345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":677446345},{"type":"v1","addr":"172.21.15.27:6807","nonce":677446345}]},"public_addr":"172.21.15.27:6803/677446345","cluster_addr":"172.21.15.27:6805/677446345","heartbeat_back_addr":"172.21.15.27:6809/677446345","heartbeat_front_addr":"172.21.15.27:6807/677446345","state":["exists","up"]},{"osd":1,"uuid":"207c4446-366c-45dd-8496-e1234636e6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6801","nonce":98361899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6803","nonce":98361899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6807","nonce":98361899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":98361899},{"type":"v1","addr":"172.21.15.76:6805","nonce":98361899}]},"public_addr":"172.21.15.76:6801/98361899","cluster_addr":"172.21.15.76:6803/98361899","heartbeat_back_addr":"172.21.15.76:6807/98361899","heartbeat_front_addr":"172.21.15.76:6805/98361899","state":["exists","up"]},{"osd":2,"uuid":"9afd53b7-8054-4ff3-888b-de388da33ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6801","nonce":1136376247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6803","nonce":1136376247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6807","nonce":1136376247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1136376247},{"type":"v1","addr":"172.21.15.189:6805","nonce":1136376247}]},"public_addr":"172.21.15.189:6801/1136376247","cluster_addr":"172.21.15.189:6803/1136376247","heartbeat_back_addr":"172.21.15.189:6807/1136376247","heartbeat_front_addr":"172.21.15.189:6805/1136376247","state":["exists","up"]},{"osd":3,"uuid":"6e0125b3-a60e-45ea-9aca-73c16bd0c69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6811","nonce":2407454219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6813","nonce":2407454219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6817","nonce":2407454219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2407454219},{"type":"v1","addr":"172.21.15.27:6815","nonce":2407454219}]},"public_addr":"172.21.15.27:6811/2407454219","cluster_addr":"172.21.15.27:6813/2407454219","heartbeat_back_addr":"172.21.15.27:6817/2407454219","heartbeat_front_addr":"172.21.15.27:6815/2407454219","state":["exists","up"]},{"osd":4,"uuid":"22993f58-2d0f-4faa-9651-02f3ff04cf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6809","nonce":1681276823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6811","nonce":1681276823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6815","nonce":1681276823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":1681276823},{"type":"v1","addr":"172.21.15.76:6813","nonce":1681276823}]},"public_addr":"172.21.15.76:6809/1681276823","cluster_addr":"172.21.15.76:6811/1681276823","heartbeat_back_addr":"172.21.15.76:6815/1681276823","heartbeat_front_addr":"172.21.15.76:6813/1681276823","state":["exists","up"]},{"osd":5,"uuid":"d5740d97-7270-4869-b523-b6e3c15e5127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6809","nonce":1669864682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6811","nonce":1669864682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6815","nonce":1669864682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1669864682},{"type":"v1","addr":"172.21.15.189:6813","nonce":1669864682}]},"public_addr":"172.21.15.189:6809/1669864682","cluster_addr":"172.21.15.189:6811/1669864682","heartbeat_back_addr":"172.21.15.189:6815/1669864682","heartbeat_front_addr":"172.21.15.189:6813/1669864682","state":["exists","up"]},{"osd":6,"uuid":"2aa064bd-d7d2-4189-824d-59cc8d0c5e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6819","nonce":3624966800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6821","nonce":3624966800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6825","nonce":3624966800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3624966800},{"type":"v1","addr":"172.21.15.27:6823","nonce":3624966800}]},"public_addr":"172.21.15.27:6819/3624966800","cluster_addr":"172.21.15.27:6821/3624966800","heartbeat_back_addr":"172.21.15.27:6825/3624966800","heartbeat_front_addr":"172.21.15.27:6823/3624966800","state":["exists","up"]},{"osd":7,"uuid":"56a01e95-852d-4072-a5d7-50b414487dfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6817","nonce":1478795194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6819","nonce":1478795194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6823","nonce":1478795194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1478795194},{"type":"v1","addr":"172.21.15.76:6821","nonce":1478795194}]},"public_addr":"172.21.15.76:6817/1478795194","cluster_addr":"172.21.15.76:6819/1478795194","heartbeat_back_addr":"172.21.15.76:6823/1478795194","heartbeat_front_addr":"172.21.15.76:6821/1478795194","state":["exists","up"]},{"osd":8,"uuid":"28643dd0-d7e0-498b-8c59-9374a9f58239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6817","nonce":2550733386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6819","nonce":2550733386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6823","nonce":2550733386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2550733386},{"type":"v1","addr":"172.21.15.189:6821","nonce":2550733386}]},"public_addr":"172.21.15.189:6817/2550733386","cluster_addr":"172.21.15.189:6819/2550733386","heartbeat_back_addr":"172.21.15.189:6823/2550733386","heartbeat_front_addr":"172.21.15.189:6821/2550733386","state":["exists","up"]},{"osd":9,"uuid":"9f10253f-40eb-43c9-8e6f-f552d3fd736c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6827","nonce":2571330197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6829","nonce":2571330197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6833","nonce":2571330197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2571330197},{"type":"v1","addr":"172.21.15.27:6831","nonce":2571330197}]},"public_addr":"172.21.15.27:6827/2571330197","cluster_addr":"172.21.15.27:6829/2571330197","heartbeat_back_addr":"172.21.15.27:6833/2571330197","heartbeat_front_addr":"172.21.15.27:6831/2571330197","state":["exists","up"]},{"osd":10,"uuid":"4a916c16-6efc-4d99-8494-238b7cbeef63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6825","nonce":2809936234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6827","nonce":2809936234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6831","nonce":2809936234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":2809936234},{"type":"v1","addr":"172.21.15.76:6829","nonce":2809936234}]},"public_addr":"172.21.15.76:6825/2809936234","cluster_addr":"172.21.15.76:6827/2809936234","heartbeat_back_addr":"172.21.15.76:6831/2809936234","heartbeat_front_addr":"172.21.15.76:6829/2809936234","state":["exists","up"]},{"osd":11,"uuid":"ab1431f2-89cd-447b-aec8-e2dc786564d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6825","nonce":49546096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6827","nonce":49546096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6831","nonce":49546096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":49546096},{"type":"v1","addr":"172.21.15.189:6829","nonce":49546096}]},"public_addr":"172.21.15.189:6825/49546096","cluster_addr":"172.21.15.189:6827/49546096","heartbeat_back_addr":"172.21.15.189:6831/49546096","heartbeat_front_addr":"172.21.15.189:6829/49546096","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:14.195957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:01:39.580491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:03.662209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:02:38.155303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:04.337228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:03:29.863509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:06.272536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:33.018847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:04:58.259796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:05:34.558258+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:02.365095+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-10T11:06:28.164078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6},{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1021663132":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6801/1704535643":"2024-05-11T11:13:13.118713+0000","172.21.15.27:0/2444542866":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/1450791252":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6801/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:6800/481366655":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/2743539323":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/3803325728":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1410017298":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6800/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:6800/1704535643":"2024-05-11T11:13:13.118713+0000","172.21.15.27:6801/2058505174":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/2991615845":"2024-05-11T11:13:13.118713+0000","172.21.15.27:0/1886129738":"2024-05-11T10:58:45.437819+0000","172.21.15.27:0/1208957722":"2024-05-11T10:59:20.613831+0000","172.21.15.27:0/4158621178":"2024-05-11T10:58:45.437819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":22,"length":1},{"begin":35,"length":1}]},{"pool":4,"snaps":[{"begin":22,"length":1},{"begin":35,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-10T11:33:17.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.768+0000 7f620da00700 1 -- 172.21.15.27:0/1854279565 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f61ec0b2a70 msgr2=0x7f61ec0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.768+0000 7f620da00700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f61ec0b2a70 0x7f61ec0b4f30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f620810be70 tx=0x7f61f400b3c0 comp rx=0 tx=0).stop 2024-05-10T11:33:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.768+0000 7f620da00700 1 -- 172.21.15.27:0/1854279565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62081056e0 msgr2=0x7f620819bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.768+0000 7f620da00700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62081056e0 0x7f620819bd60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f61f800d980 tx=0x7f61f800dc90 comp rx=0 tx=0).stop 2024-05-10T11:33:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.768+0000 7f620da00700 1 -- 172.21.15.27:0/1854279565 shutdown_connections 2024-05-10T11:33:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.768+0000 7f620da00700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f61ec0b2a70 0x7f61ec0b4f30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.768+0000 7f620da00700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62081056e0 0x7f620819bd60 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.768+0000 7f620da00700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620810b6e0 0x7f62081a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:17.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.768+0000 7f620da00700 1 --2- 172.21.15.27:0/1854279565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62081060a0 0x7f620819c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:17.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.768+0000 7f620da00700 1 -- 172.21.15.27:0/1854279565 >> 172.21.15.27:0/1854279565 conn(0x7f62081011d0 msgr2=0x7f62081016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:17.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.769+0000 7f620da00700 1 -- 172.21.15.27:0/1854279565 shutdown_connections 2024-05-10T11:33:17.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:17.769+0000 7f620da00700 1 -- 172.21.15.27:0/1854279565 wait complete. 2024-05-10T11:33:18.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:17 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1854279565' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:33:18.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:17 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1854279565' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:33:18.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:17 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1854279565' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-10T11:33:18.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs fail cephfs 2024-05-10T11:33:19.126 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:33:19.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:18 smithi189 ceph-mon[34842]: pgmap v633: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-05-10T11:33:19.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:18 smithi027 ceph-mon[30054]: pgmap v633: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-05-10T11:33:19.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:18 smithi076 ceph-mon[34152]: pgmap v633: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-05-10T11:33:21.088 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:20 smithi027 ceph-mon[30054]: pgmap v634: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 51 KiB/s wr, 19 op/s 2024-05-10T11:33:21.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:20 smithi189 ceph-mon[34842]: pgmap v634: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 51 KiB/s wr, 19 op/s 2024-05-10T11:33:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:20 smithi076 ceph-mon[34152]: pgmap v634: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 51 KiB/s wr, 19 op/s 2024-05-10T11:33:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.631+0000 7ff005a00700 1 -- 172.21.15.27:0/650778473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff000107790 msgr2=0x7ff000107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.631+0000 7ff005a00700 1 --2- 172.21.15.27:0/650778473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff000107790 0x7ff000107c40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7feff800b0a0 tx=0x7feff800b3b0 comp rx=0 tx=0).stop 2024-05-10T11:33:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.631+0000 7ff005a00700 1 -- 172.21.15.27:0/650778473 shutdown_connections 2024-05-10T11:33:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.631+0000 7ff005a00700 1 --2- 172.21.15.27:0/650778473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff000107790 0x7ff000107c40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.631+0000 7ff005a00700 1 --2- 172.21.15.27:0/650778473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff000102380 0x7ff000102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.631+0000 7ff005a00700 1 --2- 172.21.15.27:0/650778473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000102ca0 0x7ff000107170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.631+0000 7ff005a00700 1 -- 172.21.15.27:0/650778473 >> 172.21.15.27:0/650778473 conn(0x7ff0000fddc0 msgr2=0x7ff0001001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.631+0000 7ff005a00700 1 -- 172.21.15.27:0/650778473 shutdown_connections 2024-05-10T11:33:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.632+0000 7ff005a00700 1 -- 172.21.15.27:0/650778473 wait complete. 2024-05-10T11:33:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.633+0000 7ff005a00700 1 Processor -- start 2024-05-10T11:33:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.633+0000 7ff005a00700 1 -- start start 2024-05-10T11:33:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.633+0000 7ff005a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff000102380 0x7ff0001a0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.633+0000 7ff005a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000102ca0 0x7ff0001a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.633+0000 7ff005a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff000107790 0x7ff0001a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.633+0000 7ff005a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff000119c00 con 0x7ff000102380 2024-05-10T11:33:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.633+0000 7ff005a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff000119a80 con 0x7ff000107790 2024-05-10T11:33:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.633+0000 7ff005a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff000119d80 con 0x7ff000102ca0 2024-05-10T11:33:21.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.633+0000 7fefffe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff000102380 0x7ff0001a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:21.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.634+0000 7fefff400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000102ca0 0x7ff0001a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:21.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.634+0000 7fefffe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff000102380 0x7ff0001a0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54496/0 (socket says 172.21.15.27:54496) 2024-05-10T11:33:21.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.634+0000 7fefffe00700 1 -- 172.21.15.27:0/4024138441 learned_addr learned my addr 172.21.15.27:0/4024138441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:33:21.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.634+0000 7fefff400700 1 -- 172.21.15.27:0/4024138441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff000107790 msgr2=0x7ff0001a4940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-10T11:33:21.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.634+0000 7fefff400700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff000107790 0x7ff0001a4940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:21.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.634+0000 7fefff400700 1 -- 172.21.15.27:0/4024138441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff000102380 msgr2=0x7ff0001a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:21.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.634+0000 7fefff400700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff000102380 0x7ff0001a0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:21.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.634+0000 7fefff400700 1 -- 172.21.15.27:0/4024138441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0001a5080 con 0x7ff000102ca0 2024-05-10T11:33:21.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.634+0000 7fefffe00700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff000102380 0x7ff0001a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:33:21.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.635+0000 7fefff400700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000102ca0 0x7ff0001a05b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fefec00e9c0 tx=0x7fefec00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:21.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.635+0000 7feffcc00700 1 -- 172.21.15.27:0/4024138441 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec00c9c0 con 0x7ff000102ca0 2024-05-10T11:33:21.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.635+0000 7ff005a00700 1 -- 172.21.15.27:0/4024138441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0001a5370 con 0x7ff000102ca0 2024-05-10T11:33:21.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.635+0000 7feffcc00700 1 -- 172.21.15.27:0/4024138441 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefec00cb60 con 0x7ff000102ca0 2024-05-10T11:33:21.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.635+0000 7feffcc00700 1 -- 172.21.15.27:0/4024138441 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec018680 con 0x7ff000102ca0 2024-05-10T11:33:21.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.635+0000 7ff005a00700 1 -- 172.21.15.27:0/4024138441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0001acbf0 con 0x7ff000102ca0 2024-05-10T11:33:21.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.637+0000 7ff005a00700 1 -- 172.21.15.27:0/4024138441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff00006a190 con 0x7ff000102ca0 2024-05-10T11:33:21.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.640+0000 7feffcc00700 1 -- 172.21.15.27:0/4024138441 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fefec010430 con 0x7ff000102ca0 2024-05-10T11:33:21.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.641+0000 7feffcc00700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fefe40b29f0 0x7fefe40b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:21.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.642+0000 7feffcc00700 1 -- 172.21.15.27:0/4024138441 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9591+0+0 (secure 0 0 0) 0x7fefec0e2f80 con 0x7ff000102ca0 2024-05-10T11:33:21.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.642+0000 7fefffe00700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fefe40b29f0 0x7fefe40b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:21.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.642+0000 7feffcc00700 1 -- 172.21.15.27:0/4024138441 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fefec0aa4d0 con 0x7ff000102ca0 2024-05-10T11:33:21.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.643+0000 7fefffe00700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fefe40b29f0 0x7fefe40b4eb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7feff00104e0 tx=0x7feff0007480 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:21.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.936+0000 7ff005a00700 1 -- 172.21.15.27:0/4024138441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff0001acdc0 con 0x7ff000102ca0 2024-05-10T11:33:21.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.948+0000 7feffcc00700 1 -- 172.21.15.27:0/4024138441 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) ==== 161+0+0 (secure 0 0 0) 0x7fefec0a9dd0 con 0x7ff000102ca0 2024-05-10T11:33:21.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.953+0000 7ff005a00700 1 -- 172.21.15.27:0/4024138441 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fefe40b29f0 msgr2=0x7fefe40b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:21.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.953+0000 7ff005a00700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fefe40b29f0 0x7fefe40b4eb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7feff00104e0 tx=0x7feff0007480 comp rx=0 tx=0).stop 2024-05-10T11:33:21.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.953+0000 7ff005a00700 1 -- 172.21.15.27:0/4024138441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000102ca0 msgr2=0x7ff0001a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:21.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.953+0000 7ff005a00700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000102ca0 0x7ff0001a05b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fefec00e9c0 tx=0x7fefec00ecd0 comp rx=0 tx=0).stop 2024-05-10T11:33:21.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.953+0000 7ff005a00700 1 -- 172.21.15.27:0/4024138441 shutdown_connections 2024-05-10T11:33:21.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.953+0000 7ff005a00700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fefe40b29f0 0x7fefe40b4eb0 secure :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7feff00104e0 tx=0x7feff0007480 comp rx=0 tx=0).stop 2024-05-10T11:33:21.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.953+0000 7ff005a00700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff000107790 0x7ff0001a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:21.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.953+0000 7ff005a00700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff000102380 0x7ff0001a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:21.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.953+0000 7ff005a00700 1 --2- 172.21.15.27:0/4024138441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000102ca0 0x7ff0001a05b0 secure :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fefec00e9c0 tx=0x7fefec00ecd0 comp rx=0 tx=0).stop 2024-05-10T11:33:21.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.953+0000 7ff005a00700 1 -- 172.21.15.27:0/4024138441 >> 172.21.15.27:0/4024138441 conn(0x7ff0000fddc0 msgr2=0x7ff000105140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:21.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.953+0000 7ff005a00700 1 -- 172.21.15.27:0/4024138441 shutdown_connections 2024-05-10T11:33:21.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:21.954+0000 7ff005a00700 1 -- 172.21.15.27:0/4024138441 wait complete. 2024-05-10T11:33:21.956 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-10T11:33:22.355 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:33:22 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a[71669]: ignoring --setuser ceph since I am not root 2024-05-10T11:33:22.356 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:33:22 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a[71669]: ignoring --setgroup ceph since I am not root 2024-05-10T11:33:22.356 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:33:22 smithi027 ceph-mds[71673]: -- 172.21.15.27:0/3181400371 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555f28927a40 con 0x555f288fa800 2024-05-10T11:33:22.356 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:33:22 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a[71669]: starting mds.a at 2024-05-10T11:33:22.357 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:33:22 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j[70072]: ignoring --setuser ceph since I am not root 2024-05-10T11:33:22.357 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:33:22 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j[70072]: ignoring --setgroup ceph since I am not root 2024-05-10T11:33:22.357 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:33:22 smithi027 ceph-mds[70076]: -- 172.21.15.27:0/3083159292 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560490c01a40 con 0x560490bd5000 2024-05-10T11:33:22.357 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:33:22 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j[70072]: starting mds.j at 2024-05-10T11:33:22.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:21 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a[30050]: 2024-05-10T11:33:21.938+0000 7f1fdc200700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-10T11:33:22.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:21 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/4024138441' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-10T11:33:22.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:21 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-10T11:33:22.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:21 smithi027 ceph-mon[30054]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-10T11:33:22.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:21 smithi027 ceph-mon[30054]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-10T11:33:22.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:21 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-10T11:33:22.359 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:21 smithi027 ceph-mon[30054]: osdmap e120: 12 total, 12 up, 12 in 2024-05-10T11:33:22.359 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:21 smithi027 ceph-mon[30054]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-05-10T11:33:22.425 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:33:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-h[64424]: ignoring --setuser ceph since I am not root 2024-05-10T11:33:22.426 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:33:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-h[64424]: ignoring --setgroup ceph since I am not root 2024-05-10T11:33:22.426 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:33:22 smithi076 ceph-mds[64428]: -- 172.21.15.76:0/3045771053 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5568bd861a40 con 0x5568bd834800 2024-05-10T11:33:22.426 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:33:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-h[64424]: starting mds.h at 2024-05-10T11:33:22.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:21 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/4024138441' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-10T11:33:22.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:21 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-10T11:33:22.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:21 smithi076 ceph-mon[34152]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-10T11:33:22.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:21 smithi076 ceph-mon[34152]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-10T11:33:22.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:21 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-10T11:33:22.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:21 smithi076 ceph-mon[34152]: osdmap e120: 12 total, 12 up, 12 in 2024-05-10T11:33:22.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:21 smithi076 ceph-mon[34152]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-05-10T11:33:22.428 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:33:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b[64743]: ignoring --setuser ceph since I am not root 2024-05-10T11:33:22.428 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:33:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b[64743]: ignoring --setgroup ceph since I am not root 2024-05-10T11:33:22.428 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:33:22 smithi076 ceph-mds[64747]: -- 172.21.15.76:0/3441859385 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c4e9e744e0 con 0x55c4e9e4b000 2024-05-10T11:33:22.428 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:33:22 smithi076 ceph-mds[64747]: -- 172.21.15.76:0/3441859385 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c4e9e77a40 con 0x55c4e9e4b000 2024-05-10T11:33:22.428 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:33:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b[64743]: starting mds.b at 2024-05-10T11:33:22.429 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:33:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-e[64120]: ignoring --setuser ceph since I am not root 2024-05-10T11:33:22.429 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:33:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-e[64120]: ignoring --setgroup ceph since I am not root 2024-05-10T11:33:22.429 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:33:22 smithi076 ceph-mds[64124]: -- 172.21.15.76:0/864812742 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5651a306ba40 con 0x5651a303e800 2024-05-10T11:33:22.430 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:33:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-e[64120]: starting mds.e at 2024-05-10T11:33:22.430 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:33:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k[63800]: ignoring --setuser ceph since I am not root 2024-05-10T11:33:22.430 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:33:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k[63800]: ignoring --setgroup ceph since I am not root 2024-05-10T11:33:22.430 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:33:22 smithi076 ceph-mds[63804]: -- 172.21.15.76:0/1159447079 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55595d9a5a40 con 0x55595d978800 2024-05-10T11:33:22.430 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:33:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k[63800]: starting mds.k at 2024-05-10T11:33:22.487 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:33:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f[65183]: ignoring --setuser ceph since I am not root 2024-05-10T11:33:22.487 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:33:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f[65183]: ignoring --setgroup ceph since I am not root 2024-05-10T11:33:22.488 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:33:22 smithi189 ceph-mds[65187]: -- 172.21.15.189:0/2362329513 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55621df2fa40 con 0x55621df02c00 2024-05-10T11:33:22.488 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:33:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f[65183]: starting mds.f at 2024-05-10T11:33:22.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:21 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/4024138441' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-10T11:33:22.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:21 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-10T11:33:22.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:21 smithi189 ceph-mon[34842]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-10T11:33:22.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:21 smithi189 ceph-mon[34842]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-10T11:33:22.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:21 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-10T11:33:22.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:21 smithi189 ceph-mon[34842]: osdmap e120: 12 total, 12 up, 12 in 2024-05-10T11:33:22.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:21 smithi189 ceph-mon[34842]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-05-10T11:33:22.489 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:33:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-c[65500]: ignoring --setuser ceph since I am not root 2024-05-10T11:33:22.489 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:33:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-c[65500]: ignoring --setgroup ceph since I am not root 2024-05-10T11:33:22.490 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:33:22 smithi189 ceph-mds[65504]: -- 172.21.15.189:0/968082322 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562e72421a40 con 0x562e723f4800 2024-05-10T11:33:22.490 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:33:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-c[65500]: starting mds.c at 2024-05-10T11:33:22.490 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:33:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-l[64880]: ignoring --setuser ceph since I am not root 2024-05-10T11:33:22.490 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:33:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-l[64880]: ignoring --setgroup ceph since I am not root 2024-05-10T11:33:22.490 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:33:22 smithi189 ceph-mds[64884]: -- 172.21.15.189:0/342062298 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559a9f8ada40 con 0x559a9f881000 2024-05-10T11:33:22.490 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:33:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-l[64880]: starting mds.l at 2024-05-10T11:33:22.493 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:33:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-i[65808]: ignoring --setuser ceph since I am not root 2024-05-10T11:33:22.494 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:33:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-i[65808]: ignoring --setgroup ceph since I am not root 2024-05-10T11:33:22.494 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:33:22 smithi189 ceph-mds[65812]: -- 172.21.15.189:0/3961233316 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a22a17c4e0 con 0x55a22a152800 2024-05-10T11:33:22.495 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:33:22 smithi189 ceph-mds[65812]: -- 172.21.15.189:0/3961233316 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a22a17fa40 con 0x55a22a152800 2024-05-10T11:33:22.495 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:33:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-i[65808]: starting mds.i at 2024-05-10T11:33:22.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-05-10T11:33:23.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:23 smithi027 ceph-mon[30054]: pgmap v635: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-05-10T11:33:23.389 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:33:23.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:23 smithi076 ceph-mon[34152]: pgmap v635: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-05-10T11:33:23.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:23 smithi189 ceph-mon[34842]: pgmap v635: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-05-10T11:33:24.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-10T11:33:24.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-10T11:33:24.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.189:6832/6993751,v1:172.21.15.189:6835/6993751] up:boot 2024-05-10T11:33:24.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-10T11:33:24.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.189:6834/1581927981,v1:172.21.15.189:6837/1581927981] up:boot 2024-05-10T11:33:24.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.189:6833/2625785015,v1:172.21.15.189:6836/2625785015] up:boot 2024-05-10T11:33:24.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.189:6838/2866912680,v1:172.21.15.189:6839/2866912680] up:boot 2024-05-10T11:33:24.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.76:6832/2617447746,v1:172.21.15.76:6835/2617447746] up:boot 2024-05-10T11:33:24.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.76:6833/2358929033,v1:172.21.15.76:6834/2358929033] up:boot 2024-05-10T11:33:24.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.76:6836/2674952023,v1:172.21.15.76:6838/2674952023] up:boot 2024-05-10T11:33:24.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.76:6837/4118569848,v1:172.21.15.76:6839/4118569848] up:boot 2024-05-10T11:33:24.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.27:6834/652612005,v1:172.21.15.27:6840/652612005] up:boot 2024-05-10T11:33:24.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: mds.? [v2:172.21.15.27:6835/3827629184,v1:172.21.15.27:6841/3827629184] up:boot 2024-05-10T11:33:24.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-05-10T11:33:24.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-10T11:33:24.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-10T11:33:24.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-10T11:33:24.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-10T11:33:24.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-10T11:33:24.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-10T11:33:24.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:24 smithi027 ceph-mon[30054]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-10T11:33:24.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-10T11:33:24.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-10T11:33:24.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.189:6832/6993751,v1:172.21.15.189:6835/6993751] up:boot 2024-05-10T11:33:24.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-10T11:33:24.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.189:6834/1581927981,v1:172.21.15.189:6837/1581927981] up:boot 2024-05-10T11:33:24.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.189:6833/2625785015,v1:172.21.15.189:6836/2625785015] up:boot 2024-05-10T11:33:24.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.189:6838/2866912680,v1:172.21.15.189:6839/2866912680] up:boot 2024-05-10T11:33:24.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.76:6832/2617447746,v1:172.21.15.76:6835/2617447746] up:boot 2024-05-10T11:33:24.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.76:6833/2358929033,v1:172.21.15.76:6834/2358929033] up:boot 2024-05-10T11:33:24.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.76:6836/2674952023,v1:172.21.15.76:6838/2674952023] up:boot 2024-05-10T11:33:24.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.76:6837/4118569848,v1:172.21.15.76:6839/4118569848] up:boot 2024-05-10T11:33:24.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.27:6834/652612005,v1:172.21.15.27:6840/652612005] up:boot 2024-05-10T11:33:24.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: mds.? [v2:172.21.15.27:6835/3827629184,v1:172.21.15.27:6841/3827629184] up:boot 2024-05-10T11:33:24.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-05-10T11:33:24.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-10T11:33:24.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-10T11:33:24.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-10T11:33:24.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-10T11:33:24.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-10T11:33:24.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-10T11:33:24.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:24 smithi076 ceph-mon[34152]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-10T11:33:24.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-10T11:33:24.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-10T11:33:24.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.189:6832/6993751,v1:172.21.15.189:6835/6993751] up:boot 2024-05-10T11:33:24.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-10T11:33:24.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.189:6834/1581927981,v1:172.21.15.189:6837/1581927981] up:boot 2024-05-10T11:33:24.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.189:6833/2625785015,v1:172.21.15.189:6836/2625785015] up:boot 2024-05-10T11:33:24.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.189:6838/2866912680,v1:172.21.15.189:6839/2866912680] up:boot 2024-05-10T11:33:24.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.76:6832/2617447746,v1:172.21.15.76:6835/2617447746] up:boot 2024-05-10T11:33:24.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.76:6833/2358929033,v1:172.21.15.76:6834/2358929033] up:boot 2024-05-10T11:33:24.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.76:6836/2674952023,v1:172.21.15.76:6838/2674952023] up:boot 2024-05-10T11:33:24.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.76:6837/4118569848,v1:172.21.15.76:6839/4118569848] up:boot 2024-05-10T11:33:24.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.27:6834/652612005,v1:172.21.15.27:6840/652612005] up:boot 2024-05-10T11:33:24.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: mds.? [v2:172.21.15.27:6835/3827629184,v1:172.21.15.27:6841/3827629184] up:boot 2024-05-10T11:33:24.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-05-10T11:33:24.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-10T11:33:24.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-10T11:33:24.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-10T11:33:24.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-10T11:33:24.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-10T11:33:24.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-10T11:33:24.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:24 smithi189 ceph-mon[34842]: from='mgr.25220 172.21.15.27:0/3311023808' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-10T11:33:25.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:25 smithi027 ceph-mon[30054]: pgmap v637: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-05-10T11:33:25.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:25 smithi076 ceph-mon[34152]: pgmap v637: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-05-10T11:33:25.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:25 smithi189 ceph-mon[34842]: pgmap v637: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-05-10T11:33:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.953+0000 7ff347200700 1 -- 172.21.15.27:0/2598260885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff34810da00 msgr2=0x7ff34810fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.953+0000 7ff347200700 1 --2- 172.21.15.27:0/2598260885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff34810da00 0x7ff34810fe70 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff34000b0a0 tx=0x7ff34000b3b0 comp rx=0 tx=0).stop 2024-05-10T11:33:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.953+0000 7ff347200700 1 -- 172.21.15.27:0/2598260885 shutdown_connections 2024-05-10T11:33:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.953+0000 7ff347200700 1 --2- 172.21.15.27:0/2598260885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff34810da00 0x7ff34810fe70 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.953+0000 7ff347200700 1 --2- 172.21.15.27:0/2598260885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3481045d0 0x7ff3481049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.953+0000 7ff347200700 1 --2- 172.21.15.27:0/2598260885 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff348104ef0 0x7ff34810d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.953+0000 7ff347200700 1 -- 172.21.15.27:0/2598260885 >> 172.21.15.27:0/2598260885 conn(0x7ff3480fddc0 msgr2=0x7ff3481001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.953+0000 7ff347200700 1 -- 172.21.15.27:0/2598260885 shutdown_connections 2024-05-10T11:33:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.953+0000 7ff347200700 1 -- 172.21.15.27:0/2598260885 wait complete. 2024-05-10T11:33:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.954+0000 7ff347200700 1 Processor -- start 2024-05-10T11:33:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.954+0000 7ff347200700 1 -- start start 2024-05-10T11:33:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.955+0000 7ff347200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3481045d0 0x7ff348102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.955+0000 7ff347200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff348104ef0 0x7ff348102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.955+0000 7ff347200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff34810da00 0x7ff348102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.955+0000 7ff347200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff348113600 con 0x7ff348104ef0 2024-05-10T11:33:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.955+0000 7ff347200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff348113480 con 0x7ff3481045d0 2024-05-10T11:33:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.955+0000 7ff347200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff348113780 con 0x7ff34810da00 2024-05-10T11:33:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.955+0000 7ff345e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3481045d0 0x7ff348102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.955+0000 7ff345400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff348104ef0 0x7ff348102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.955+0000 7ff345e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3481045d0 0x7ff348102230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:34080/0 (socket says 172.21.15.27:34080) 2024-05-10T11:33:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.955+0000 7ff345e00700 1 -- 172.21.15.27:0/966100361 learned_addr learned my addr 172.21.15.27:0/966100361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:33:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.955+0000 7ff346800700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff34810da00 0x7ff348102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.956+0000 7ff345e00700 1 -- 172.21.15.27:0/966100361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff34810da00 msgr2=0x7ff348102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.956+0000 7ff345e00700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff34810da00 0x7ff348102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.956+0000 7ff345e00700 1 -- 172.21.15.27:0/966100361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff348104ef0 msgr2=0x7ff348102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.956+0000 7ff345e00700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff348104ef0 0x7ff348102770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.956+0000 7ff345e00700 1 -- 172.21.15.27:0/966100361 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3481a8000 con 0x7ff3481045d0 2024-05-10T11:33:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.956+0000 7ff345400700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff348104ef0 0x7ff348102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:33:25.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.956+0000 7ff345e00700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3481045d0 0x7ff348102230 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff33800e9a0 tx=0x7ff33800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.956+0000 7ff33ea00700 1 -- 172.21.15.27:0/966100361 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33800ca10 con 0x7ff3481045d0 2024-05-10T11:33:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.956+0000 7ff33ea00700 1 -- 172.21.15.27:0/966100361 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff33800cbb0 con 0x7ff3481045d0 2024-05-10T11:33:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.956+0000 7ff347200700 1 -- 172.21.15.27:0/966100361 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3481a82f0 con 0x7ff3481045d0 2024-05-10T11:33:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.957+0000 7ff33ea00700 1 -- 172.21.15.27:0/966100361 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338018680 con 0x7ff3481045d0 2024-05-10T11:33:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.957+0000 7ff347200700 1 -- 172.21.15.27:0/966100361 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3481a8880 con 0x7ff3481045d0 2024-05-10T11:33:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.958+0000 7ff347200700 1 -- 172.21.15.27:0/966100361 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff34806a190 con 0x7ff3481045d0 2024-05-10T11:33:25.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.960+0000 7ff33ea00700 1 -- 172.21.15.27:0/966100361 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff338010430 con 0x7ff3481045d0 2024-05-10T11:33:25.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.962+0000 7ff33ea00700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff32c0b2a70 0x7ff32c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:25.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.962+0000 7ff33ea00700 1 -- 172.21.15.27:0/966100361 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9933+0+0 (secure 0 0 0) 0x7ff3380e2cf0 con 0x7ff3481045d0 2024-05-10T11:33:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.962+0000 7ff345400700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff32c0b2a70 0x7ff32c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.963+0000 7ff345400700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff32c0b2a70 0x7ff32c0b4f30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff348101990 tx=0x7ff33400b410 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:25.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:25.966+0000 7ff33ea00700 1 -- 172.21.15.27:0/966100361 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ff3380aa0f0 con 0x7ff3481045d0 2024-05-10T11:33:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:26.262+0000 7ff347200700 1 -- 172.21.15.27:0/966100361 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff34802e510 con 0x7ff3481045d0 2024-05-10T11:33:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.042+0000 7ff33ea00700 1 -- 172.21.15.27:0/966100361 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v48) ==== 104+0+0 (secure 0 0 0) 0x7ff3380a99f0 con 0x7ff3481045d0 2024-05-10T11:33:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.047+0000 7ff347200700 1 -- 172.21.15.27:0/966100361 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff32c0b2a70 msgr2=0x7ff32c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:27.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.047+0000 7ff347200700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff32c0b2a70 0x7ff32c0b4f30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff348101990 tx=0x7ff33400b410 comp rx=0 tx=0).stop 2024-05-10T11:33:27.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.047+0000 7ff347200700 1 -- 172.21.15.27:0/966100361 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3481045d0 msgr2=0x7ff348102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:27.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.047+0000 7ff347200700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3481045d0 0x7ff348102230 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff33800e9a0 tx=0x7ff33800ecb0 comp rx=0 tx=0).stop 2024-05-10T11:33:27.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.048+0000 7ff347200700 1 -- 172.21.15.27:0/966100361 shutdown_connections 2024-05-10T11:33:27.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.048+0000 7ff347200700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7ff32c0b2a70 0x7ff32c0b4f30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:27.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.048+0000 7ff347200700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff34810da00 0x7ff348102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:27.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.048+0000 7ff347200700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3481045d0 0x7ff348102230 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:27.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.048+0000 7ff347200700 1 --2- 172.21.15.27:0/966100361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff348104ef0 0x7ff348102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:27.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.048+0000 7ff347200700 1 -- 172.21.15.27:0/966100361 >> 172.21.15.27:0/966100361 conn(0x7ff3480fddc0 msgr2=0x7ff3480fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:27.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.048+0000 7ff347200700 1 -- 172.21.15.27:0/966100361 shutdown_connections 2024-05-10T11:33:27.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:27.048+0000 7ff347200700 1 -- 172.21.15.27:0/966100361 wait complete. 2024-05-10T11:33:27.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:27 smithi027 ceph-mon[30054]: pgmap v638: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 18 KiB/s wr, 9 op/s 2024-05-10T11:33:27.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:27 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/966100361' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-10T11:33:27.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:27 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-10T11:33:27.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:27 smithi076 ceph-mon[34152]: pgmap v638: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 18 KiB/s wr, 9 op/s 2024-05-10T11:33:27.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:27 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/966100361' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-10T11:33:27.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:27 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-10T11:33:27.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:27 smithi189 ceph-mon[34842]: pgmap v638: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 18 KiB/s wr, 9 op/s 2024-05-10T11:33:27.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:27 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/966100361' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-10T11:33:27.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:27 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-10T11:33:28.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-10T11:33:28.319 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:28 smithi189 ceph-mon[34842]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-10T11:33:28.319 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:28 smithi189 ceph-mon[34842]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-10T11:33:28.319 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:28 smithi189 ceph-mon[34842]: Cluster is now healthy 2024-05-10T11:33:28.319 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:28 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-10T11:33:28.319 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:28 smithi189 ceph-mon[34842]: fsmap 12 up:standby 2024-05-10T11:33:28.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:28 smithi076 ceph-mon[34152]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-10T11:33:28.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:28 smithi076 ceph-mon[34152]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-10T11:33:28.326 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:28 smithi076 ceph-mon[34152]: Cluster is now healthy 2024-05-10T11:33:28.327 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:28 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-10T11:33:28.327 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:28 smithi076 ceph-mon[34152]: fsmap 12 up:standby 2024-05-10T11:33:28.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:28 smithi027 ceph-mon[30054]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-10T11:33:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:28 smithi027 ceph-mon[30054]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-10T11:33:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:28 smithi027 ceph-mon[30054]: Cluster is now healthy 2024-05-10T11:33:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:28 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-10T11:33:28.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:28 smithi027 ceph-mon[30054]: fsmap 12 up:standby 2024-05-10T11:33:28.541 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:33:29.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:29 smithi027 ceph-mon[30054]: pgmap v639: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 18 KiB/s wr, 9 op/s 2024-05-10T11:33:29.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:29 smithi076 ceph-mon[34152]: pgmap v639: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 18 KiB/s wr, 9 op/s 2024-05-10T11:33:29.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:29 smithi189 ceph-mon[34842]: pgmap v639: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 18 KiB/s wr, 9 op/s 2024-05-10T11:33:30.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.834+0000 7fc5fa600700 1 -- 172.21.15.27:0/106722125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5f41055d0 msgr2=0x7fc5f4105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:30.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.834+0000 7fc5fa600700 1 --2- 172.21.15.27:0/106722125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5f41055d0 0x7fc5f4105a50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009990 tx=0x7fc5e0009ca0 comp rx=0 tx=0).stop 2024-05-10T11:33:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.834+0000 7fc5fa600700 1 -- 172.21.15.27:0/106722125 shutdown_connections 2024-05-10T11:33:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.834+0000 7fc5fa600700 1 --2- 172.21.15.27:0/106722125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5f4105f90 0x7fc5f41124a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.834+0000 7fc5fa600700 1 --2- 172.21.15.27:0/106722125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5f410b660 0x7fc5f410ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.834+0000 7fc5fa600700 1 --2- 172.21.15.27:0/106722125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5f41055d0 0x7fc5f4105a50 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.834+0000 7fc5fa600700 1 -- 172.21.15.27:0/106722125 >> 172.21.15.27:0/106722125 conn(0x7fc5f41010c0 msgr2=0x7fc5f41034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.835+0000 7fc5fa600700 1 -- 172.21.15.27:0/106722125 shutdown_connections 2024-05-10T11:33:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.835+0000 7fc5fa600700 1 -- 172.21.15.27:0/106722125 wait complete. 2024-05-10T11:33:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.836+0000 7fc5fa600700 1 Processor -- start 2024-05-10T11:33:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.836+0000 7fc5fa600700 1 -- start start 2024-05-10T11:33:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.836+0000 7fc5fa600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5f41055d0 0x7fc5f41a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.836+0000 7fc5fa600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5f4105f90 0x7fc5f41a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.836+0000 7fc5fa600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5f410b660 0x7fc5f419ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.836+0000 7fc5fa600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f4115800 con 0x7fc5f4105f90 2024-05-10T11:33:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.836+0000 7fc5fa600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f4115680 con 0x7fc5f41055d0 2024-05-10T11:33:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.836+0000 7fc5fa600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f4115980 con 0x7fc5f410b660 2024-05-10T11:33:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.837+0000 7fc5f9200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5f41055d0 0x7fc5f41a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.837+0000 7fc5f3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5f4105f90 0x7fc5f41a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.837+0000 7fc5f9200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5f41055d0 0x7fc5f41a18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.27:34096/0 (socket says 172.21.15.27:34096) 2024-05-10T11:33:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.837+0000 7fc5f9200700 1 -- 172.21.15.27:0/3525033703 learned_addr learned my addr 172.21.15.27:0/3525033703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:33:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.837+0000 7fc5f9c00700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5f410b660 0x7fc5f419ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.837+0000 7fc5f9200700 1 -- 172.21.15.27:0/3525033703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5f410b660 msgr2=0x7fc5f419ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.837+0000 7fc5f9200700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5f410b660 0x7fc5f419ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.837+0000 7fc5f9200700 1 -- 172.21.15.27:0/3525033703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5f4105f90 msgr2=0x7fc5f41a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.837+0000 7fc5f9200700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5f4105f90 0x7fc5f41a1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.837+0000 7fc5f9200700 1 -- 172.21.15.27:0/3525033703 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f419c1e0 con 0x7fc5f41055d0 2024-05-10T11:33:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.838+0000 7fc5f9200700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5f41055d0 0x7fc5f41a18c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800e970 tx=0x7fc5e800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.838+0000 7fc5f1600700 1 -- 172.21.15.27:0/3525033703 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e800cbd0 con 0x7fc5f41055d0 2024-05-10T11:33:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.838+0000 7fc5fa600700 1 -- 172.21.15.27:0/3525033703 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f419c4d0 con 0x7fc5f41055d0 2024-05-10T11:33:30.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.838+0000 7fc5f1600700 1 -- 172.21.15.27:0/3525033703 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5e80044d0 con 0x7fc5f41055d0 2024-05-10T11:33:30.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.838+0000 7fc5f1600700 1 -- 172.21.15.27:0/3525033703 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8003df0 con 0x7fc5f41055d0 2024-05-10T11:33:30.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.838+0000 7fc5fa600700 1 -- 172.21.15.27:0/3525033703 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5f41a8790 con 0x7fc5f41055d0 2024-05-10T11:33:30.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.839+0000 7fc5fa600700 1 -- 172.21.15.27:0/3525033703 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5f406a190 con 0x7fc5f41055d0 2024-05-10T11:33:30.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.842+0000 7fc5f1600700 1 -- 172.21.15.27:0/3525033703 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc5e8003f90 con 0x7fc5f41055d0 2024-05-10T11:33:30.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.844+0000 7fc5f1600700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc5dc0b29f0 0x7fc5dc0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:30.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.844+0000 7fc5f1600700 1 -- 172.21.15.27:0/3525033703 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9933+0+0 (secure 0 0 0) 0x7fc5e80e2ec0 con 0x7fc5f41055d0 2024-05-10T11:33:30.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.844+0000 7fc5f3e00700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc5dc0b29f0 0x7fc5dc0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:30.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.845+0000 7fc5f3e00700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc5dc0b29f0 0x7fc5dc0b4eb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc5f410ad00 tx=0x7fc5e000b260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:30.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:30.845+0000 7fc5f1600700 1 -- 172.21.15.27:0/3525033703 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fc5e80aa350 con 0x7fc5f41055d0 2024-05-10T11:33:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:31.086+0000 7fc5fa600700 1 -- 172.21.15.27:0/3525033703 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc5f419cfb0 con 0x7fc5f41055d0 2024-05-10T11:33:31.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:31 smithi027 ceph-mon[30054]: pgmap v640: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 9.6 KiB/s wr, 4 op/s 2024-05-10T11:33:31.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:31 smithi076 ceph-mon[34152]: pgmap v640: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 9.6 KiB/s wr, 4 op/s 2024-05-10T11:33:31.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:31 smithi189 ceph-mon[34842]: pgmap v640: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 9.6 KiB/s wr, 4 op/s 2024-05-10T11:33:32.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.083+0000 7fc5f1600700 1 -- 172.21.15.27:0/3525033703 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v121) ==== 181+0+0 (secure 0 0 0) 0x7fc5e80a9c50 con 0x7fc5f41055d0 2024-05-10T11:33:32.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.091+0000 7fc5fa600700 1 -- 172.21.15.27:0/3525033703 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc5dc0b29f0 msgr2=0x7fc5dc0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:32.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.091+0000 7fc5fa600700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc5dc0b29f0 0x7fc5dc0b4eb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc5f410ad00 tx=0x7fc5e000b260 comp rx=0 tx=0).stop 2024-05-10T11:33:32.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.091+0000 7fc5fa600700 1 -- 172.21.15.27:0/3525033703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5f41055d0 msgr2=0x7fc5f41a18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:32.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.091+0000 7fc5fa600700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5f41055d0 0x7fc5f41a18c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800e970 tx=0x7fc5e800ec80 comp rx=0 tx=0).stop 2024-05-10T11:33:32.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.092+0000 7fc5fa600700 1 -- 172.21.15.27:0/3525033703 shutdown_connections 2024-05-10T11:33:32.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.092+0000 7fc5fa600700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fc5dc0b29f0 0x7fc5dc0b4eb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:32.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.092+0000 7fc5fa600700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5f410b660 0x7fc5f419ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:32.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.092+0000 7fc5fa600700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5f41055d0 0x7fc5f41a18c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:32.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.092+0000 7fc5fa600700 1 --2- 172.21.15.27:0/3525033703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5f4105f90 0x7fc5f41a1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.092+0000 7fc5fa600700 1 -- 172.21.15.27:0/3525033703 >> 172.21.15.27:0/3525033703 conn(0x7fc5f41010c0 msgr2=0x7fc5f4102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.093+0000 7fc5fa600700 1 -- 172.21.15.27:0/3525033703 shutdown_connections 2024-05-10T11:33:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:32.093+0000 7fc5fa600700 1 -- 172.21.15.27:0/3525033703 wait complete. 2024-05-10T11:33:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-05-10T11:33:32.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:32 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3525033703' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:32.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:32 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:32 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3525033703' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:32.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:32 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:32.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:32 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3525033703' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:32.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:32 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:33.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-10T11:33:33.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:33 smithi027 ceph-mon[30054]: pgmap v641: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 9.6 KiB/s wr, 3 op/s 2024-05-10T11:33:33.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:33 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-10T11:33:33.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:33 smithi027 ceph-mon[30054]: osdmap e121: 12 total, 12 up, 12 in 2024-05-10T11:33:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:33 smithi076 ceph-mon[34152]: pgmap v641: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 9.6 KiB/s wr, 3 op/s 2024-05-10T11:33:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:33 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-10T11:33:33.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:33 smithi076 ceph-mon[34152]: osdmap e121: 12 total, 12 up, 12 in 2024-05-10T11:33:33.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:33 smithi189 ceph-mon[34842]: pgmap v641: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 9.6 KiB/s wr, 3 op/s 2024-05-10T11:33:33.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:33 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-10T11:33:33.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:33 smithi189 ceph-mon[34842]: osdmap e121: 12 total, 12 up, 12 in 2024-05-10T11:33:33.573 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:33:35.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:35 smithi027 ceph-mon[30054]: pgmap v643: 97 pgs: 97 active+clean; 288 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:35.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:35 smithi076 ceph-mon[34152]: pgmap v643: 97 pgs: 97 active+clean; 288 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:35.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:35 smithi189 ceph-mon[34842]: pgmap v643: 97 pgs: 97 active+clean; 288 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:36.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.030+0000 7f9d0aa00700 1 -- 172.21.15.27:0/113537065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d0406bb40 msgr2=0x7f9d04110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:36.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.030+0000 7f9d0aa00700 1 --2- 172.21.15.27:0/113537065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d0406bb40 0x7f9d04110330 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00b0a0 tx=0x7f9cfc00b3b0 comp rx=0 tx=0).stop 2024-05-10T11:33:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.030+0000 7f9d0aa00700 1 -- 172.21.15.27:0/113537065 shutdown_connections 2024-05-10T11:33:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.030+0000 7f9d0aa00700 1 --2- 172.21.15.27:0/113537065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d0406bb40 0x7f9d04110330 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.030+0000 7f9d0aa00700 1 --2- 172.21.15.27:0/113537065 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9d04109510 0x7f9d041098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.030+0000 7f9d0aa00700 1 --2- 172.21.15.27:0/113537065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0406b180 0x7f9d0406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.030+0000 7f9d0aa00700 1 -- 172.21.15.27:0/113537065 >> 172.21.15.27:0/113537065 conn(0x7f9d04079350 msgr2=0x7f9d04079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.030+0000 7f9d0aa00700 1 -- 172.21.15.27:0/113537065 shutdown_connections 2024-05-10T11:33:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.030+0000 7f9d0aa00700 1 -- 172.21.15.27:0/113537065 wait complete. 2024-05-10T11:33:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.031+0000 7f9d0aa00700 1 Processor -- start 2024-05-10T11:33:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.032+0000 7f9d0aa00700 1 -- start start 2024-05-10T11:33:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.032+0000 7f9d0aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0406b180 0x7f9d04102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.032+0000 7f9d0aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9d0406bb40 0x7f9d041029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.032+0000 7f9d0aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d04109510 0x7f9d04102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.032+0000 7f9d0aa00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9d04113620 con 0x7f9d0406b180 2024-05-10T11:33:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.032+0000 7f9d0aa00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9d041134a0 con 0x7f9d0406bb40 2024-05-10T11:33:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.032+0000 7f9d0aa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9d041137a0 con 0x7f9d04109510 2024-05-10T11:33:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.032+0000 7f9d03e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0406b180 0x7f9d04102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.032+0000 7f9d03e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0406b180 0x7f9d04102470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54270/0 (socket says 172.21.15.27:54270) 2024-05-10T11:33:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.032+0000 7f9d03e00700 1 -- 172.21.15.27:0/3918724107 learned_addr learned my addr 172.21.15.27:0/3918724107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:33:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.032+0000 7f9d03400700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9d0406bb40 0x7f9d041029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.033+0000 7f9d08a00700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d04109510 0x7f9d04102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.033+0000 7f9d03400700 1 -- 172.21.15.27:0/3918724107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d04109510 msgr2=0x7f9d04102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.033+0000 7f9d03400700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d04109510 0x7f9d04102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.033+0000 7f9d03400700 1 -- 172.21.15.27:0/3918724107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0406b180 msgr2=0x7f9d04102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.033+0000 7f9d03400700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0406b180 0x7f9d04102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.033+0000 7f9d03400700 1 -- 172.21.15.27:0/3918724107 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d041a8000 con 0x7f9d0406bb40 2024-05-10T11:33:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.033+0000 7f9d03e00700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0406b180 0x7f9d04102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-10T11:33:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.033+0000 7f9d03400700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9d0406bb40 0x7f9d041029b0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000e9f0 tx=0x7f9cf000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.034+0000 7f9d00c00700 1 -- 172.21.15.27:0/3918724107 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf000c9c0 con 0x7f9d0406bb40 2024-05-10T11:33:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.034+0000 7f9d0aa00700 1 -- 172.21.15.27:0/3918724107 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d041a82f0 con 0x7f9d0406bb40 2024-05-10T11:33:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.034+0000 7f9d00c00700 1 -- 172.21.15.27:0/3918724107 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cf000cb60 con 0x7f9d0406bb40 2024-05-10T11:33:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.034+0000 7f9d00c00700 1 -- 172.21.15.27:0/3918724107 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0018680 con 0x7f9d0406bb40 2024-05-10T11:33:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.034+0000 7f9d0aa00700 1 -- 172.21.15.27:0/3918724107 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d041a8880 con 0x7f9d0406bb40 2024-05-10T11:33:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.035+0000 7f9d0aa00700 1 -- 172.21.15.27:0/3918724107 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d0406a190 con 0x7f9d0406bb40 2024-05-10T11:33:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.038+0000 7f9d00c00700 1 -- 172.21.15.27:0/3918724107 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f9cf0010430 con 0x7f9d0406bb40 2024-05-10T11:33:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.040+0000 7f9d00c00700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9ce80b29f0 0x7f9ce80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.040+0000 7f9d00c00700 1 -- 172.21.15.27:0/3918724107 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9379+0+0 (secure 0 0 0) 0x7f9cf00e2eb0 con 0x7f9d0406bb40 2024-05-10T11:33:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.040+0000 7f9d03e00700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9ce80b29f0 0x7f9ce80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.041+0000 7f9d03e00700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9ce80b29f0 0x7f9ce80b4eb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4011550 tx=0x7f9cf4009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.041+0000 7f9d00c00700 1 -- 172.21.15.27:0/3918724107 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9cf00aa4d0 con 0x7f9d0406bb40 2024-05-10T11:33:36.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:36.282+0000 7f9d0aa00700 1 -- 172.21.15.27:0/3918724107 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9d0402e510 con 0x7f9d0406bb40 2024-05-10T11:33:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.107+0000 7f9d00c00700 1 -- 172.21.15.27:0/3918724107 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v122) ==== 169+0+0 (secure 0 0 0) 0x7f9cf00a9dd0 con 0x7f9d0406bb40 2024-05-10T11:33:37.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.113+0000 7f9d0aa00700 1 -- 172.21.15.27:0/3918724107 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9ce80b29f0 msgr2=0x7f9ce80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.113+0000 7f9d0aa00700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9ce80b29f0 0x7f9ce80b4eb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4011550 tx=0x7f9cf4009260 comp rx=0 tx=0).stop 2024-05-10T11:33:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.113+0000 7f9d0aa00700 1 -- 172.21.15.27:0/3918724107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9d0406bb40 msgr2=0x7f9d041029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.113+0000 7f9d0aa00700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9d0406bb40 0x7f9d041029b0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000e9f0 tx=0x7f9cf000ed00 comp rx=0 tx=0).stop 2024-05-10T11:33:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.114+0000 7f9d0aa00700 1 -- 172.21.15.27:0/3918724107 shutdown_connections 2024-05-10T11:33:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.114+0000 7f9d0aa00700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7f9ce80b29f0 0x7f9ce80b4eb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.114+0000 7f9d0aa00700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d04109510 0x7f9d04102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.114+0000 7f9d0aa00700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0406b180 0x7f9d04102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.114+0000 7f9d0aa00700 1 --2- 172.21.15.27:0/3918724107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9d0406bb40 0x7f9d041029b0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.114+0000 7f9d0aa00700 1 -- 172.21.15.27:0/3918724107 >> 172.21.15.27:0/3918724107 conn(0x7f9d04079350 msgr2=0x7f9d04100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.115+0000 7f9d0aa00700 1 -- 172.21.15.27:0/3918724107 shutdown_connections 2024-05-10T11:33:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:37.115+0000 7f9d0aa00700 1 -- 172.21.15.27:0/3918724107 wait complete. 2024-05-10T11:33:37.119 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-05-10T11:33:37.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:37 smithi027 ceph-mon[30054]: pgmap v644: 97 pgs: 97 active+clean; 288 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:37.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:37 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/3918724107' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:37.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:37 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:37.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:37 smithi076 ceph-mon[34152]: pgmap v644: 97 pgs: 97 active+clean; 288 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:37.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:37 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/3918724107' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:37.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:37 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:37.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:37 smithi189 ceph-mon[34842]: pgmap v644: 97 pgs: 97 active+clean; 288 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:37.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:37 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/3918724107' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:37.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:37 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:38.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b shell --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-05-10T11:33:38.368 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:38 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-10T11:33:38.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:38 smithi027 ceph-mon[30054]: osdmap e122: 12 total, 12 up, 12 in 2024-05-10T11:33:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:38 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-10T11:33:38.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:38 smithi076 ceph-mon[34152]: osdmap e122: 12 total, 12 up, 12 in 2024-05-10T11:33:38.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:38 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-10T11:33:38.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:38 smithi189 ceph-mon[34842]: osdmap e122: 12 total, 12 up, 12 in 2024-05-10T11:33:38.583 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/mon.a/config 2024-05-10T11:33:39.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:39 smithi027 ceph-mon[30054]: pgmap v646: 33 pgs: 33 active+clean; 288 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:39.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:39 smithi076 ceph-mon[34152]: pgmap v646: 33 pgs: 33 active+clean; 288 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:39.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:39 smithi189 ceph-mon[34842]: pgmap v646: 33 pgs: 33 active+clean; 288 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:41.405 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:41 smithi027 ceph-mon[30054]: pgmap v647: 33 pgs: 33 active+clean; 288 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:41.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:41 smithi076 ceph-mon[34152]: pgmap v647: 33 pgs: 33 active+clean; 288 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:41.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.485+0000 7fbc51a00700 1 -- 172.21.15.27:0/2697896872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c10d9c0 msgr2=0x7fbc4c10fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:41.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.485+0000 7fbc51a00700 1 --2- 172.21.15.27:0/2697896872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c10d9c0 0x7fbc4c10fe80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fbc4400b0a0 tx=0x7fbc4400b3b0 comp rx=0 tx=0).stop 2024-05-10T11:33:41.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.485+0000 7fbc51a00700 1 -- 172.21.15.27:0/2697896872 shutdown_connections 2024-05-10T11:33:41.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.485+0000 7fbc51a00700 1 --2- 172.21.15.27:0/2697896872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c10d9c0 0x7fbc4c10fe80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:41.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.485+0000 7fbc51a00700 1 --2- 172.21.15.27:0/2697896872 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4c104590 0x7fbc4c104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:41.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.485+0000 7fbc51a00700 1 --2- 172.21.15.27:0/2697896872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c104eb0 0x7fbc4c10d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:41.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.485+0000 7fbc51a00700 1 -- 172.21.15.27:0/2697896872 >> 172.21.15.27:0/2697896872 conn(0x7fbc4c0fddc0 msgr2=0x7fbc4c1001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:41.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.485+0000 7fbc51a00700 1 -- 172.21.15.27:0/2697896872 shutdown_connections 2024-05-10T11:33:41.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.485+0000 7fbc51a00700 1 -- 172.21.15.27:0/2697896872 wait complete. 2024-05-10T11:33:41.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:41 smithi189 ceph-mon[34842]: pgmap v647: 33 pgs: 33 active+clean; 288 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:41.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.486+0000 7fbc51a00700 1 Processor -- start 2024-05-10T11:33:41.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.486+0000 7fbc51a00700 1 -- start start 2024-05-10T11:33:41.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.487+0000 7fbc51a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c104590 0x7fbc4c1a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:41.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.487+0000 7fbc51a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c104eb0 0x7fbc4c1a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:41.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.487+0000 7fbc51a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4c10d9c0 0x7fbc4c19bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:41.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.487+0000 7fbc51a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c113610 con 0x7fbc4c104590 2024-05-10T11:33:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.487+0000 7fbc51a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c113490 con 0x7fbc4c10d9c0 2024-05-10T11:33:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.487+0000 7fbc51a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c113790 con 0x7fbc4c104eb0 2024-05-10T11:33:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.487+0000 7fbc4be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c104590 0x7fbc4c1a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.487+0000 7fbc4b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c104eb0 0x7fbc4c1a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.487+0000 7fbc51000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4c10d9c0 0x7fbc4c19bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.487+0000 7fbc4be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c104590 0x7fbc4c1a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54300/0 (socket says 172.21.15.27:54300) 2024-05-10T11:33:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.487+0000 7fbc4b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c104eb0 0x7fbc4c1a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.27:47710/0 (socket says 172.21.15.27:47710) 2024-05-10T11:33:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.487+0000 7fbc4be00700 1 -- 172.21.15.27:0/1991350989 learned_addr learned my addr 172.21.15.27:0/1991350989 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-05-10T11:33:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.488+0000 7fbc51000700 1 -- 172.21.15.27:0/1991350989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c104eb0 msgr2=0x7fbc4c1a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.488+0000 7fbc51000700 1 --2- 172.21.15.27:0/1991350989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c104eb0 0x7fbc4c1a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.488+0000 7fbc51000700 1 -- 172.21.15.27:0/1991350989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c104590 msgr2=0x7fbc4c1a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.488+0000 7fbc51000700 1 --2- 172.21.15.27:0/1991350989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c104590 0x7fbc4c1a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.488+0000 7fbc51000700 1 -- 172.21.15.27:0/1991350989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c19c210 con 0x7fbc4c10d9c0 2024-05-10T11:33:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.488+0000 7fbc51000700 1 --2- 172.21.15.27:0/1991350989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4c10d9c0 0x7fbc4c19bab0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fbc4400b070 tx=0x7fbc44004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.488+0000 7fbc48c00700 1 -- 172.21.15.27:0/1991350989 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc44021070 con 0x7fbc4c10d9c0 2024-05-10T11:33:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.489+0000 7fbc48c00700 1 -- 172.21.15.27:0/1991350989 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc44007e30 con 0x7fbc4c10d9c0 2024-05-10T11:33:41.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.489+0000 7fbc48c00700 1 -- 172.21.15.27:0/1991350989 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc440045f0 con 0x7fbc4c10d9c0 2024-05-10T11:33:41.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.489+0000 7fbc51a00700 1 -- 172.21.15.27:0/1991350989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4c19c4a0 con 0x7fbc4c10d9c0 2024-05-10T11:33:41.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.489+0000 7fbc51a00700 1 -- 172.21.15.27:0/1991350989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4c1a87c0 con 0x7fbc4c10d9c0 2024-05-10T11:33:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.490+0000 7fbc51a00700 1 -- 172.21.15.27:0/1991350989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc4c06a190 con 0x7fbc4c10d9c0 2024-05-10T11:33:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.492+0000 7fbc48c00700 1 -- 172.21.15.27:0/1991350989 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fbc44007980 con 0x7fbc4c10d9c0 2024-05-10T11:33:41.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.494+0000 7fbc48c00700 1 --2- 172.21.15.27:0/1991350989 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbc300b29f0 0x7fbc300b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-10T11:33:41.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.494+0000 7fbc48c00700 1 -- 172.21.15.27:0/1991350989 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8754+0+0 (secure 0 0 0) 0x7fbc440e3510 con 0x7fbc4c10d9c0 2024-05-10T11:33:41.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.494+0000 7fbc4be00700 1 --2- 172.21.15.27:0/1991350989 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbc300b29f0 0x7fbc300b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-10T11:33:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.495+0000 7fbc4be00700 1 --2- 172.21.15.27:0/1991350989 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbc300b29f0 0x7fbc300b4eb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c19cbf0 tx=0x7fbc3c009260 comp rx=0 tx=0).ready entity=mgr.25220 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-10T11:33:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.496+0000 7fbc48c00700 1 -- 172.21.15.27:0/1991350989 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fbc440aae30 con 0x7fbc4c10d9c0 2024-05-10T11:33:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:41.736+0000 7fbc51a00700 1 -- 172.21.15.27:0/1991350989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbc4c1a8990 con 0x7fbc4c10d9c0 2024-05-10T11:33:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.138+0000 7fbc48c00700 1 -- 172.21.15.27:0/1991350989 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v123) ==== 178+0+0 (secure 0 0 0) 0x7fbc440aa730 con 0x7fbc4c10d9c0 2024-05-10T11:33:42.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.144+0000 7fbc51a00700 1 -- 172.21.15.27:0/1991350989 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbc300b29f0 msgr2=0x7fbc300b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:42.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.144+0000 7fbc51a00700 1 --2- 172.21.15.27:0/1991350989 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbc300b29f0 0x7fbc300b4eb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c19cbf0 tx=0x7fbc3c009260 comp rx=0 tx=0).stop 2024-05-10T11:33:42.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.144+0000 7fbc51a00700 1 -- 172.21.15.27:0/1991350989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4c10d9c0 msgr2=0x7fbc4c19bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-10T11:33:42.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.144+0000 7fbc51a00700 1 --2- 172.21.15.27:0/1991350989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4c10d9c0 0x7fbc4c19bab0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fbc4400b070 tx=0x7fbc44004c80 comp rx=0 tx=0).stop 2024-05-10T11:33:42.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.145+0000 7fbc51a00700 1 -- 172.21.15.27:0/1991350989 shutdown_connections 2024-05-10T11:33:42.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.145+0000 7fbc51a00700 1 --2- 172.21.15.27:0/1991350989 >> [v2:172.21.15.27:6800/1167199542,v1:172.21.15.27:6801/1167199542] conn(0x7fbc300b29f0 0x7fbc300b4eb0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:42.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.145+0000 7fbc51a00700 1 --2- 172.21.15.27:0/1991350989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4c10d9c0 0x7fbc4c19bab0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:42.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.145+0000 7fbc51a00700 1 --2- 172.21.15.27:0/1991350989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c104590 0x7fbc4c1a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:42.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.145+0000 7fbc51a00700 1 --2- 172.21.15.27:0/1991350989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c104eb0 0x7fbc4c1a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-10T11:33:42.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.145+0000 7fbc51a00700 1 -- 172.21.15.27:0/1991350989 >> 172.21.15.27:0/1991350989 conn(0x7fbc4c0fddc0 msgr2=0x7fbc4c10fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-10T11:33:42.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.146+0000 7fbc51a00700 1 -- 172.21.15.27:0/1991350989 shutdown_connections 2024-05-10T11:33:42.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-10T11:33:42.146+0000 7fbc51a00700 1 -- 172.21.15.27:0/1991350989 wait complete. 2024-05-10T11:33:42.150 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data_ec' removed 2024-05-10T11:33:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:42 smithi076 ceph-mon[34152]: from='client.? 172.21.15.27:0/1991350989' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:42.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:42 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:42.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:42 smithi189 ceph-mon[34842]: from='client.? 172.21.15.27:0/1991350989' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:42.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:42 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:42.556 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:42 smithi027 ceph-mon[30054]: from='client.? 172.21.15.27:0/1991350989' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:42.556 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:42 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-10T11:33:43.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-10T11:33:43.097 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-10T11:33:43.128 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-10T11:33:43.161 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-10T11:33:43.162 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-10T11:33:43.217 DEBUG:teuthology.orchestra.run.smithi076:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-10T11:33:43.233 DEBUG:teuthology.orchestra.run.smithi189:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-10T11:33:43.251 INFO:tasks.cephadm:Stopping all daemons... 2024-05-10T11:33:43.251 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-10T11:33:43.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.a 2024-05-10T11:33:43.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:43 smithi027 ceph-mon[30054]: pgmap v648: 33 pgs: 33 active+clean; 288 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:43.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:43 smithi027 ceph-mon[30054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-05-10T11:33:43.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:43 smithi027 ceph-mon[30054]: osdmap e123: 12 total, 12 up, 12 in 2024-05-10T11:33:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:43 smithi076 ceph-mon[34152]: pgmap v648: 33 pgs: 33 active+clean; 288 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:43 smithi076 ceph-mon[34152]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-05-10T11:33:43.425 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:43 smithi076 ceph-mon[34152]: osdmap e123: 12 total, 12 up, 12 in 2024-05-10T11:33:43.486 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:43 smithi189 ceph-mon[34842]: pgmap v648: 33 pgs: 33 active+clean; 288 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-10T11:33:43.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:43 smithi189 ceph-mon[34842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-05-10T11:33:43.487 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:43 smithi189 ceph-mon[34842]: osdmap e123: 12 total, 12 up, 12 in 2024-05-10T11:33:43.619 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:43 smithi027 systemd[1]: Stopping Ceph mon.a for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:33:44.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:43 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a[30050]: 2024-05-10T11:33:43.882+0000 7f1fe3000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:33:44.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:43 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a[30050]: 2024-05-10T11:33:43.882+0000 7f1fe3000700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-10T11:33:44.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:44 smithi027 podman[125426]: 2024-05-10 11:33:44.038031815 +0000 UTC m=+0.320802033 container died 36f5ec0cc8e03f01a68dfa9bcb3f3cb19ae644191fc8fe46296d0323e4fbfa9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-10T11:33:44.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:44 smithi027 podman[125426]: 2024-05-10 11:33:44.640607948 +0000 UTC m=+0.923378200 container cleanup 36f5ec0cc8e03f01a68dfa9bcb3f3cb19ae644191fc8fe46296d0323e4fbfa9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-10T11:33:44.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:44 smithi027 bash[125426]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a 2024-05-10T11:33:45.353 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 10 11:33:44 smithi027 podman[125440]: 2024-05-10 11:33:44.900065021 +0000 UTC m=+0.846139462 container remove 36f5ec0cc8e03f01a68dfa9bcb3f3cb19ae644191fc8fe46296d0323e4fbfa9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-10T11:33:45.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.a.service' 2024-05-10T11:33:45.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:33:45.535 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-10T11:33:45.535 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-10T11:33:45.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.b 2024-05-10T11:33:45.876 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:45 smithi076 systemd[1]: Stopping Ceph mon.b for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:33:46.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:45 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-b[34148]: 2024-05-10T11:33:45.874+0000 7f3128b8f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:33:46.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:45 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-b[34148]: 2024-05-10T11:33:45.874+0000 7f3128b8f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-10T11:33:46.175 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:45 smithi076 podman[73212]: 2024-05-10 11:33:45.947469525 +0000 UTC m=+0.183143972 container died c141b324c977203a29f0ede9ee846bb6232dde9b390e6b312a83b50a0538c13e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-b, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-10T11:33:46.627 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:46 smithi076 podman[73212]: 2024-05-10 11:33:46.309470048 +0000 UTC m=+0.545144490 container cleanup c141b324c977203a29f0ede9ee846bb6232dde9b390e6b312a83b50a0538c13e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-05-10T11:33:46.628 INFO:journalctl@ceph.mon.b.smithi076.stdout:May 10 11:33:46 smithi076 bash[73212]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-b 2024-05-10T11:33:46.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.b.service' 2024-05-10T11:33:46.795 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:33:46.795 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-10T11:33:46.795 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-10T11:33:46.795 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.c 2024-05-10T11:33:47.090 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:46 smithi189 systemd[1]: Stopping Ceph mon.c for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:33:47.730 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:47 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-c[34828]: 2024-05-10T11:33:47.361+0000 7f368624c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:33:47.731 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:47 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-c[34828]: 2024-05-10T11:33:47.361+0000 7f368624c700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-10T11:33:47.731 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:47 smithi189 podman[75179]: 2024-05-10 11:33:47.434608852 +0000 UTC m=+0.200626580 container died 79bc0699541be7369d64ac8b54589d9c64c089258362090d90c1d9056497555d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-c, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507) 2024-05-10T11:33:47.986 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:47 smithi189 podman[75179]: 2024-05-10 11:33:47.830355264 +0000 UTC m=+0.596373009 container cleanup 79bc0699541be7369d64ac8b54589d9c64c089258362090d90c1d9056497555d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-10T11:33:47.987 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 10 11:33:47 smithi189 bash[75179]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mon-c 2024-05-10T11:33:48.218 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mon.c.service' 2024-05-10T11:33:48.266 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:33:48.267 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-10T11:33:48.267 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-10T11:33:48.267 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.x 2024-05-10T11:33:48.619 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:33:48 smithi027 systemd[1]: Stopping Ceph mgr.x for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:33:49.194 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:33:48 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x[30284]: 2024-05-10T11:33:48.893+0000 7fa2b5c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:33:49.195 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:33:48 smithi027 podman[125548]: 2024-05-10 11:33:48.987184722 +0000 UTC m=+0.250373672 container died 6a7db7f103443646d6582db1deeb2c50d544ab2c1d1adfb11df37f9eb19cca1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-10T11:33:49.773 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:33:49 smithi027 podman[125548]: 2024-05-10 11:33:49.512448928 +0000 UTC m=+0.775637895 container cleanup 6a7db7f103443646d6582db1deeb2c50d544ab2c1d1adfb11df37f9eb19cca1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2) 2024-05-10T11:33:49.774 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:33:49 smithi027 bash[125548]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x 2024-05-10T11:33:50.118 INFO:journalctl@ceph.mgr.x.smithi027.stdout:May 10 11:33:49 smithi027 podman[125562]: 2024-05-10 11:33:49.77275472 +0000 UTC m=+0.768297789 container remove 6a7db7f103443646d6582db1deeb2c50d544ab2c1d1adfb11df37f9eb19cca1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d39fa08867836ac070c47cb66c782fa559fd480b, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-10T11:33:50.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.x.service' 2024-05-10T11:33:50.392 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:33:50.392 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-10T11:33:50.393 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-10T11:33:50.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.y 2024-05-10T11:33:50.675 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:33:50 smithi076 systemd[1]: Stopping Ceph mgr.y for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:33:51.031 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:33:50 smithi076 podman[73332]: 2024-05-10 11:33:50.753458384 +0000 UTC m=+0.160193114 container died c4f46c9ba6decaf63c5b1f9570f8828592f1208d0633b8688a6239a4994d80be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-10T11:33:51.374 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:33:51 smithi076 podman[73332]: 2024-05-10 11:33:51.115376289 +0000 UTC m=+0.522111008 container cleanup c4f46c9ba6decaf63c5b1f9570f8828592f1208d0633b8688a6239a4994d80be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=) 2024-05-10T11:33:51.375 INFO:journalctl@ceph.mgr.y.smithi076.stdout:May 10 11:33:51 smithi076 bash[73332]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-y 2024-05-10T11:33:51.575 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.y.service' 2024-05-10T11:33:51.608 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:33:51.609 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-10T11:33:51.609 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-10T11:33:51.609 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.z 2024-05-10T11:33:51.939 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:33:51 smithi189 systemd[1]: Stopping Ceph mgr.z for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:33:52.235 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:33:51 smithi189 podman[75300]: 2024-05-10 11:33:51.93919918 +0000 UTC m=+0.140324535 container died 6f59165a3ad5334ead3f47d105171f522d31254f2f61c9a8d98e91d9140c0a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2) 2024-05-10T11:33:52.486 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:33:52 smithi189 podman[75300]: 2024-05-10 11:33:52.343574441 +0000 UTC m=+0.544699795 container cleanup 6f59165a3ad5334ead3f47d105171f522d31254f2f61c9a8d98e91d9140c0a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-10T11:33:52.487 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 10 11:33:52 smithi189 bash[75300]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mgr-z 2024-05-10T11:33:52.756 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mgr.z.service' 2024-05-10T11:33:52.789 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:33:52.789 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-10T11:33:52.790 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-10T11:33:52.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.0 2024-05-10T11:33:53.369 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:33:52 smithi027 systemd[1]: Stopping Ceph osd.0 for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:33:53.869 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:33:53 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0[44577]: 2024-05-10T11:33:53.416+0000 7fee01200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:33:53.869 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:33:53 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0[44577]: 2024-05-10T11:33:53.416+0000 7fee01200700 -1 osd.0 123 *** Got signal Terminated *** 2024-05-10T11:33:53.869 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:33:53 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0[44577]: 2024-05-10T11:33:53.416+0000 7fee01200700 -1 osd.0 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-10T11:33:59.368 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:33:59 smithi027 podman[125668]: 2024-05-10 11:33:59.017168167 +0000 UTC m=+5.761033092 container died d5188f7b0a675d71cb3bed6afeebce0043eaadc14829b445a2fb56d567b6373f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507) 2024-05-10T11:33:59.753 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:33:59 smithi027 podman[125668]: 2024-05-10 11:33:59.526867257 +0000 UTC m=+6.270732198 container cleanup d5188f7b0a675d71cb3bed6afeebce0043eaadc14829b445a2fb56d567b6373f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-10T11:33:59.753 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:33:59 smithi027 bash[125668]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0 2024-05-10T11:34:00.118 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:33:59 smithi027 podman[125686]: 2024-05-10 11:33:59.752940129 +0000 UTC m=+0.723254544 container remove d5188f7b0a675d71cb3bed6afeebce0043eaadc14829b445a2fb56d567b6373f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-10T11:34:00.869 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:34:00 smithi027 podman[125775]: 2024-05-10 11:34:00.471356971 +0000 UTC m=+0.037819927 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:34:00.869 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:34:00 smithi027 podman[125775]: 2024-05-10 11:34:00.595830087 +0000 UTC m=+0.162293082 container create 39ac592b81fc941cfeca8d42648c2beb29752f64ab03d12acf7a0a22c6996459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=) 2024-05-10T11:34:01.369 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:34:00 smithi027 podman[125775]: 2024-05-10 11:34:00.941777495 +0000 UTC m=+0.508240507 container init 39ac592b81fc941cfeca8d42648c2beb29752f64ab03d12acf7a0a22c6996459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-05-10T11:34:01.370 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:34:00 smithi027 podman[125775]: 2024-05-10 11:34:00.954059357 +0000 UTC m=+0.520522322 container start 39ac592b81fc941cfeca8d42648c2beb29752f64ab03d12acf7a0a22c6996459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_BRANCH=HEAD) 2024-05-10T11:34:01.370 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:34:01 smithi027 podman[125775]: 2024-05-10 11:34:01.048791902 +0000 UTC m=+0.615254937 container attach 39ac592b81fc941cfeca8d42648c2beb29752f64ab03d12acf7a0a22c6996459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-05-10T11:34:01.840 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 10 11:34:01 smithi027 podman[125775]: 2024-05-10 11:34:01.428070962 +0000 UTC m=+0.994534010 container died 39ac592b81fc941cfeca8d42648c2beb29752f64ab03d12acf7a0a22c6996459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-10T11:34:02.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.0.service' 2024-05-10T11:34:02.359 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:34:02.359 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-10T11:34:02.359 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-10T11:34:02.359 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.1 2024-05-10T11:34:02.675 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:02 smithi076 systemd[1]: Stopping Ceph osd.1 for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:34:03.425 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1[38778]: 2024-05-10T11:34:03.003+0000 7f2aa21d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:34:03.426 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1[38778]: 2024-05-10T11:34:03.003+0000 7f2aa21d0700 -1 osd.1 123 *** Got signal Terminated *** 2024-05-10T11:34:03.426 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1[38778]: 2024-05-10T11:34:03.003+0000 7f2aa21d0700 -1 osd.1 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-10T11:34:08.868 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:08 smithi076 podman[73471]: 2024-05-10 11:34:08.56646081 +0000 UTC m=+6.013090018 container died 4ba0385fa0a93627a89b5d38d61bb0818a3f3a0fd919dfc20d2652b91e7559fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-10T11:34:09.175 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:08 smithi076 podman[73471]: 2024-05-10 11:34:08.962033212 +0000 UTC m=+6.408662419 container cleanup 4ba0385fa0a93627a89b5d38d61bb0818a3f3a0fd919dfc20d2652b91e7559fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-10T11:34:09.175 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:08 smithi076 bash[73471]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1 2024-05-10T11:34:09.584 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:09 smithi076 podman[73488]: 2024-05-10 11:34:09.180207176 +0000 UTC m=+0.611451603 container remove 4ba0385fa0a93627a89b5d38d61bb0818a3f3a0fd919dfc20d2652b91e7559fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-10T11:34:09.858 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:09 smithi076 podman[73579]: 2024-05-10 11:34:09.488199021 +0000 UTC m=+0.018085500 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:34:09.858 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:09 smithi076 podman[73579]: 2024-05-10 11:34:09.606268044 +0000 UTC m=+0.136154557 container create 6b95f2b6d2bfc52fbcc108a5a5e98cf433a0aba20b56c311d79c71b32285ef0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:34:09.858 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:09 smithi076 podman[73579]: 2024-05-10 11:34:09.858835148 +0000 UTC m=+0.388721618 container init 6b95f2b6d2bfc52fbcc108a5a5e98cf433a0aba20b56c311d79c71b32285ef0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=) 2024-05-10T11:34:10.175 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:09 smithi076 podman[73579]: 2024-05-10 11:34:09.862267057 +0000 UTC m=+0.392153530 container start 6b95f2b6d2bfc52fbcc108a5a5e98cf433a0aba20b56c311d79c71b32285ef0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507) 2024-05-10T11:34:10.176 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:09 smithi076 podman[73579]: 2024-05-10 11:34:09.992295974 +0000 UTC m=+0.522182448 container attach 6b95f2b6d2bfc52fbcc108a5a5e98cf433a0aba20b56c311d79c71b32285ef0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-10T11:34:10.675 INFO:journalctl@ceph.osd.1.smithi076.stdout:May 10 11:34:10 smithi076 podman[73579]: 2024-05-10 11:34:10.388177825 +0000 UTC m=+0.918064290 container died 6b95f2b6d2bfc52fbcc108a5a5e98cf433a0aba20b56c311d79c71b32285ef0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-10T11:34:10.998 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.1.service' 2024-05-10T11:34:11.037 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:34:11.037 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-10T11:34:11.037 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-10T11:34:11.038 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.2 2024-05-10T11:34:11.447 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:11 smithi189 systemd[1]: Stopping Ceph osd.2 for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:34:11.736 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2[40006]: 2024-05-10T11:34:11.446+0000 7fdefc855700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:34:11.737 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2[40006]: 2024-05-10T11:34:11.446+0000 7fdefc855700 -1 osd.2 123 *** Got signal Terminated *** 2024-05-10T11:34:11.737 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2[40006]: 2024-05-10T11:34:11.446+0000 7fdefc855700 -1 osd.2 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-10T11:34:17.143 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:16 smithi189 podman[75425]: 2024-05-10 11:34:16.878476083 +0000 UTC m=+5.647019775 container died 8877a5d1aae92a3aab52fc0aa925bf241c57a65c8fa61e686dd6cd13c09d4857 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-10T11:34:17.487 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:17 smithi189 podman[75425]: 2024-05-10 11:34:17.2886174 +0000 UTC m=+6.057161079 container cleanup 8877a5d1aae92a3aab52fc0aa925bf241c57a65c8fa61e686dd6cd13c09d4857 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-10T11:34:17.487 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:17 smithi189 bash[75425]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2 2024-05-10T11:34:17.963 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:17 smithi189 podman[75446]: 2024-05-10 11:34:17.531845719 +0000 UTC m=+0.652784294 container remove 8877a5d1aae92a3aab52fc0aa925bf241c57a65c8fa61e686dd6cd13c09d4857 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-10T11:34:18.237 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:17 smithi189 podman[75540]: 2024-05-10 11:34:17.866962321 +0000 UTC m=+0.018207433 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:34:18.237 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:17 smithi189 podman[75540]: 2024-05-10 11:34:17.999354706 +0000 UTC m=+0.150599807 container create 232286bf42c6d076fe005af740dbafcf7be16576ec9e686c5cb1c42b691e1d48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2-deactivate, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-10T11:34:18.604 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:18 smithi189 podman[75540]: 2024-05-10 11:34:18.278130606 +0000 UTC m=+0.429375707 container init 232286bf42c6d076fe005af740dbafcf7be16576ec9e686c5cb1c42b691e1d48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-10T11:34:18.604 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:18 smithi189 podman[75540]: 2024-05-10 11:34:18.281775143 +0000 UTC m=+0.433020253 container start 232286bf42c6d076fe005af740dbafcf7be16576ec9e686c5cb1c42b691e1d48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-10T11:34:18.604 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:18 smithi189 podman[75540]: 2024-05-10 11:34:18.385413157 +0000 UTC m=+0.536658258 container attach 232286bf42c6d076fe005af740dbafcf7be16576ec9e686c5cb1c42b691e1d48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-10T11:34:18.955 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 10 11:34:18 smithi189 podman[75540]: 2024-05-10 11:34:18.603916847 +0000 UTC m=+0.755161957 container died 232286bf42c6d076fe005af740dbafcf7be16576ec9e686c5cb1c42b691e1d48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2) 2024-05-10T11:34:19.281 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.2.service' 2024-05-10T11:34:19.320 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:34:19.320 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-10T11:34:19.320 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-10T11:34:19.321 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.3 2024-05-10T11:34:19.869 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:19 smithi027 systemd[1]: Stopping Ceph osd.3 for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:34:20.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:19 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:19.798+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:20.369 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:19 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3[51229]: 2024-05-10T11:34:19.937+0000 7f5c2ee00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:34:20.369 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:19 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3[51229]: 2024-05-10T11:34:19.937+0000 7f5c2ee00700 -1 osd.3 123 *** Got signal Terminated *** 2024-05-10T11:34:20.370 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:19 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3[51229]: 2024-05-10T11:34:19.937+0000 7f5c2ee00700 -1 osd.3 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-10T11:34:21.118 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:20 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3[51229]: 2024-05-10T11:34:20.736+0000 7f5c25600700 -1 osd.3 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:56.775875+0000 front 2024-05-10T11:33:56.777076+0000 (oldest deadline 2024-05-10T11:34:20.276055+0000) 2024-05-10T11:34:21.238 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:20 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:20.848+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:21.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:21 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:21.408+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:21.736 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:21 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:21.258+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:22.118 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:21 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3[51229]: 2024-05-10T11:34:21.756+0000 7f5c25600700 -1 osd.3 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:56.775875+0000 front 2024-05-10T11:33:56.777076+0000 (oldest deadline 2024-05-10T11:34:20.276055+0000) 2024-05-10T11:34:22.221 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:21 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:21.910+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:22.221 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:21 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:21.863+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:22.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:22.220+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:22.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:22.397+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:23.118 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:22 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3[51229]: 2024-05-10T11:34:22.713+0000 7f5c25600700 -1 osd.3 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:56.775875+0000 front 2024-05-10T11:33:56.777076+0000 (oldest deadline 2024-05-10T11:34:20.276055+0000) 2024-05-10T11:34:23.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:22.945+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:23.188 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:22.869+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:23.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:23 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:23.184+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:23.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:23 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:23.368+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:24.118 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:23 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3[51229]: 2024-05-10T11:34:23.692+0000 7f5c25600700 -1 osd.3 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:56.775875+0000 front 2024-05-10T11:33:56.777076+0000 (oldest deadline 2024-05-10T11:34:20.276055+0000) 2024-05-10T11:34:24.230 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:23 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:23.942+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:24.230 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:23 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:23.830+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:24.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:24 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:24.229+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:24.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:24 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:24.351+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:25.119 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:24 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3[51229]: 2024-05-10T11:34:24.741+0000 7f5c25600700 -1 osd.3 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:56.775875+0000 front 2024-05-10T11:33:56.777076+0000 (oldest deadline 2024-05-10T11:34:20.276055+0000) 2024-05-10T11:34:25.119 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:24 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:24.965+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:25.205 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:24 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:24.859+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:25.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:24 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:24.966+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:25.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:25 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:25.206+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:25.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:25 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:25.332+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:25.869 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:25 smithi027 podman[125892]: 2024-05-10 11:34:25.595073423 +0000 UTC m=+5.805541522 container died 257e59e73faf48877c0df08d4b70c701b791a446547d8c4c5772d96298b3a695 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-10T11:34:26.188 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:25 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:25.962+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:26.188 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:26 smithi027 podman[125892]: 2024-05-10 11:34:26.16496398 +0000 UTC m=+6.375432073 container cleanup 257e59e73faf48877c0df08d4b70c701b791a446547d8c4c5772d96298b3a695 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-10T11:34:26.189 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:26 smithi027 bash[125892]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3 2024-05-10T11:34:26.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:25 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:25.938+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:26.205 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:25 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:25.876+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:26.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:26.204+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:26.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:26 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:26.290+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:26.868 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:26 smithi027 podman[125912]: 2024-05-10 11:34:26.516722022 +0000 UTC m=+0.912478776 container remove 257e59e73faf48877c0df08d4b70c701b791a446547d8c4c5772d96298b3a695 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-10T11:34:27.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:26.963+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:27.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:26.923+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:27.330 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:27 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:27.003+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:27.331 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:27 smithi027 podman[126001]: 2024-05-10 11:34:27.086086598 +0000 UTC m=+0.034814905 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:34:27.331 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:27 smithi027 podman[126001]: 2024-05-10 11:34:27.192501237 +0000 UTC m=+0.141229585 container create ddb685fa69001d3e6e43845760f6e1365722b366998dbc9f95c18b0324e9c5fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3-deactivate, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, io.buildah.version=1.33.5) 2024-05-10T11:34:27.619 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:27 smithi027 podman[126001]: 2024-05-10 11:34:27.463092646 +0000 UTC m=+0.411820994 container init ddb685fa69001d3e6e43845760f6e1365722b366998dbc9f95c18b0324e9c5fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-10T11:34:27.619 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:27 smithi027 podman[126001]: 2024-05-10 11:34:27.474791507 +0000 UTC m=+0.423519852 container start ddb685fa69001d3e6e43845760f6e1365722b366998dbc9f95c18b0324e9c5fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux ) 2024-05-10T11:34:27.619 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:27 smithi027 podman[126001]: 2024-05-10 11:34:27.570146855 +0000 UTC m=+0.518875203 container attach ddb685fa69001d3e6e43845760f6e1365722b366998dbc9f95c18b0324e9c5fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-10T11:34:27.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:27 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:27.275+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:27.736 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:27 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:27.246+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:28.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:27 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:27.969+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:28.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:28.000+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:28.238 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:28.214+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:28.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:27 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:27.983+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:28.370 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 10 11:34:27 smithi027 podman[126001]: 2024-05-10 11:34:27.961673307 +0000 UTC m=+0.910401655 container died ddb685fa69001d3e6e43845760f6e1365722b366998dbc9f95c18b0324e9c5fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-3-deactivate, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507) 2024-05-10T11:34:28.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:28 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:28.231+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:28.898 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.3.service' 2024-05-10T11:34:29.023 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:34:29.024 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-10T11:34:29.024 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-10T11:34:29.024 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.4 2024-05-10T11:34:29.118 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:29 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:29.019+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:29.340 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:29 smithi076 systemd[1]: Stopping Ceph osd.4 for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:34:29.341 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:29 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:29.230+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:29.486 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:28.999+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:29.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:29.173+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:29.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:29.033+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:29.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:29.033+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:29.604 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:29 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4[45477]: 2024-05-10T11:34:29.340+0000 7fca0b36f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:34:29.605 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:29 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4[45477]: 2024-05-10T11:34:29.340+0000 7fca0b36f700 -1 osd.4 123 *** Got signal Terminated *** 2024-05-10T11:34:29.605 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:29 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4[45477]: 2024-05-10T11:34:29.340+0000 7fca0b36f700 -1 osd.4 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-10T11:34:30.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:30.197+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:30.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:30.059+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:30.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:30.059+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:30.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:29.980+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:30.238 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:29.980+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:30.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:29 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:29.997+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:30.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:29 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:29.997+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:30.528 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:30 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:30.209+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:31.176 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:30 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4[45477]: 2024-05-10T11:34:30.899+0000 7fca07963700 -1 osd.4 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.180336+0000 front 2024-05-10T11:34:06.180362+0000 (oldest deadline 2024-05-10T11:34:30.280542+0000) 2024-05-10T11:34:31.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:31.061+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:31.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:31.061+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:31.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:31.149+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:31.238 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:30.986+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:31.238 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:30.986+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:31.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:30 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:30.972+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:31.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:30 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:30.972+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:31.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:31 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:31.199+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:31.973 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:31 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:31.682+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:32.175 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:31 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4[45477]: 2024-05-10T11:34:31.905+0000 7fca07963700 -1 osd.4 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.180336+0000 front 2024-05-10T11:34:06.180362+0000 (oldest deadline 2024-05-10T11:34:30.280542+0000) 2024-05-10T11:34:32.236 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:32.094+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:32.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:32.094+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:32.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:32.185+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:32.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:31.949+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:32.238 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:31.949+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:32.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:31 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:31.971+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:32.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:31 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:31.971+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:32.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:32 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:32.244+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:32.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:32 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:32.244+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:32.924 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:32 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:32.656+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:33.225 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:32.966+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:33.225 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:32.966+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:33.225 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:33.118+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:33.225 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:33.118+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:33.227 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:32 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4[45477]: 2024-05-10T11:34:32.934+0000 7fca07963700 -1 osd.4 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.180336+0000 front 2024-05-10T11:34:06.180362+0000 (oldest deadline 2024-05-10T11:34:30.280542+0000) 2024-05-10T11:34:33.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:32 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:32.924+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:33.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:32 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:32.924+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:33.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:33.225+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:33.592 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:33 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:33.227+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:33.592 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:33 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:33.227+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:33.901 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:33 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:33.592+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:33.963 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:33 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:33.624+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:34.175 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:33 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4[45477]: 2024-05-10T11:34:33.902+0000 7fca07963700 -1 osd.4 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.180336+0000 front 2024-05-10T11:34:06.180362+0000 (oldest deadline 2024-05-10T11:34:30.280542+0000) 2024-05-10T11:34:34.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:33 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:33.962+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:34.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:33 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:33.962+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:34.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:34 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:34.195+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:34.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:34 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:34.141+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:34.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:34 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:34.141+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:34.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:33.998+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:34.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:33.998+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:34.618 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:34 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:34.267+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:34.618 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:34 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:34.267+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:34.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:34 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:34.617+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:34.926 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:34 smithi076 podman[73689]: 2024-05-10 11:34:34.723944411 +0000 UTC m=+5.507936192 container died 44ca8387f64c790b8af083a372c589dea5253175f7da89471809bc441f1615a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:34:34.985 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:34 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:34.670+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:35.236 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:35 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:35.190+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:35.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:35 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:35.121+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:35.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:35 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:35.121+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:35.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:34 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:34.962+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:35.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:34 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:34.962+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:35.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:34 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:34.984+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:35.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:34 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:34.984+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:35.409 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:35 smithi076 podman[73689]: 2024-05-10 11:34:35.150267076 +0000 UTC m=+5.934258903 container cleanup 44ca8387f64c790b8af083a372c589dea5253175f7da89471809bc441f1615a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-10T11:34:35.409 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:35 smithi076 bash[73689]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4 2024-05-10T11:34:35.410 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:35 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:35.229+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:35.410 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:35 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:35.229+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:35.661 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:35 smithi076 podman[73720]: 2024-05-10 11:34:35.410251757 +0000 UTC m=+0.675777888 container remove 44ca8387f64c790b8af083a372c589dea5253175f7da89471809bc441f1615a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507) 2024-05-10T11:34:35.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:35 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:35.660+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:35.925 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:35 smithi076 podman[73812]: 2024-05-10 11:34:35.725253717 +0000 UTC m=+0.018332525 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:34:35.926 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:35 smithi076 podman[73812]: 2024-05-10 11:34:35.852864793 +0000 UTC m=+0.145943602 container create b4abbf2adad005e42dce8225eca25301bca8921d4751423ddff9f38e5338ae10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-10T11:34:35.969 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:35 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:35.716+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:36.230 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:36 smithi076 podman[73812]: 2024-05-10 11:34:36.097129581 +0000 UTC m=+0.390208388 container init b4abbf2adad005e42dce8225eca25301bca8921d4751423ddff9f38e5338ae10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:34:36.231 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:36 smithi076 podman[73812]: 2024-05-10 11:34:36.100135881 +0000 UTC m=+0.393214697 container start b4abbf2adad005e42dce8225eca25301bca8921d4751423ddff9f38e5338ae10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:34:36.231 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:36 smithi076 podman[73812]: 2024-05-10 11:34:36.197098872 +0000 UTC m=+0.490177708 container attach b4abbf2adad005e42dce8225eca25301bca8921d4751423ddff9f38e5338ae10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507) 2024-05-10T11:34:36.236 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:36 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:36.172+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:36.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:36 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:36.097+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:36.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:36 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:36.097+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:36.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:35 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:35.978+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:36.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:35 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:35.978+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:36.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:35 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:35.968+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:36.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:35 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:35.968+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:36.664 INFO:journalctl@ceph.osd.4.smithi076.stdout:May 10 11:34:36 smithi076 podman[73812]: 2024-05-10 11:34:36.395240652 +0000 UTC m=+0.688319464 container died b4abbf2adad005e42dce8225eca25301bca8921d4751423ddff9f38e5338ae10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240507, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-10T11:34:36.665 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:36 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:36.229+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:36.665 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:36 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:36.229+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:36.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:36 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:36.664+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:36.942 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:36 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:36.672+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:37.061 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.4.service' 2024-05-10T11:34:37.099 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:34:37.099 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-10T11:34:37.100 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-10T11:34:37.100 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.5 2024-05-10T11:34:37.236 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:37 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:37.142+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:37.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:37 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:37.141+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:37.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:37 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:37.142+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:37.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:37 smithi189 systemd[1]: Stopping Ceph osd.5 for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:34:37.239 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:36 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:36.982+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:37.239 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:36 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:36.982+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:37.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:36 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:36.942+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:37.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:36 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:36.942+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:37.425 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:37 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:37.260+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:37.425 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:37 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:37.260+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:37.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:37 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:37.390+0000 7f1a7bb5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:34:37.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:37 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:37.390+0000 7f1a7bb5e700 -1 osd.5 123 *** Got signal Terminated *** 2024-05-10T11:34:37.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:37 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:37.390+0000 7f1a7bb5e700 -1 osd.5 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-10T11:34:37.961 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:37 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:37.671+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:38.175 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:37 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:37.675+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:38.175 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:37 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:37.675+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:38.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:38 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:38.123+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:38.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:38 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:38.136+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:38.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:38 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:38.136+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:38.238 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:37 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:37.980+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:38.238 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:37 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:37.980+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:38.238 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:37 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:37.980+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:38.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:37 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:37.961+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:38.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:37 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:37.961+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:38.628 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:38 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:38.242+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:38.628 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:38 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:38.242+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:38.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:38 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:38.628+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:38.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:38 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:38.628+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:38.932 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:38 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:38.679+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:38.932 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:38 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:38.679+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:39.236 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:39 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:39.134+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:39.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:39 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:39.089+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:39.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:39 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:39.089+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:39.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:38 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:38.969+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:39.238 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:38 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:38.969+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:39.238 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:38 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:38.969+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:39.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:38 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:38.932+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:39.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:38 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:38.932+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:39.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:38 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:38.932+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:39.591 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:39 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:39.286+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:39.591 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:39 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:39.286+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:39.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:39 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:39.591+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:39.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:39 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:39.591+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:39.962 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:39 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:39.690+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:39.963 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:39 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:39.690+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:40.236 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:40 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:40.169+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:40.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:40 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:40.066+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:40.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:40 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:40.066+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:40.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:40 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:40.066+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.850865+0000 front 2024-05-10T11:34:15.850886+0000 (oldest deadline 2024-05-10T11:34:39.951027+0000) 2024-05-10T11:34:40.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:39 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:39.966+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:40.238 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:39 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:39.966+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:40.238 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:39 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:39.966+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:40.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:39 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:39.962+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:40.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:39 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:39.962+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:40.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:39 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:39.962+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:40.425 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:40 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:40.239+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:40.425 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:40 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:40.239+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:40.426 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:40 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:40.240+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:40.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:40 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:40.615+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:40.926 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:40 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:40.615+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:41.118 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:40 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:40.696+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:41.119 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:40 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:40.696+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:41.119 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:40 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:40.923+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:41.119 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:40 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:40.923+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:41.119 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:40 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:40.923+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:41.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:41 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:41.171+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:41.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:40 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:40.975+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:41.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:40 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:40.975+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:41.237 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:40 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:40.975+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:41.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:41 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:41.077+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:41.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:41 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:41.077+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:41.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:41 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:41.077+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.850865+0000 front 2024-05-10T11:34:15.850886+0000 (oldest deadline 2024-05-10T11:34:39.951027+0000) 2024-05-10T11:34:41.597 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:41 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:41.268+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:41.598 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:41 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:41.268+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:41.598 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:41 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:41.268+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:41.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:41 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:41.597+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:41.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:41 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:41.597+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:41.929 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:41 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:41.668+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:41.929 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:41 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:41.668+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:42.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:41 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:41.928+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:42.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:41 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:41.928+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:42.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:41 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:41.928+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:42.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:42.163+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:42.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:42.163+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:42.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:42.088+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.447475+0000 front 2024-05-10T11:33:55.447144+0000 (oldest deadline 2024-05-10T11:34:21.347243+0000) 2024-05-10T11:34:42.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:42.088+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.248503+0000 front 2024-05-10T11:34:04.248569+0000 (oldest deadline 2024-05-10T11:34:28.948475+0000) 2024-05-10T11:34:42.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5[46494]: 2024-05-10T11:34:42.088+0000 7f1a7414a700 -1 osd.5 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.850865+0000 front 2024-05-10T11:34:15.850886+0000 (oldest deadline 2024-05-10T11:34:39.951027+0000) 2024-05-10T11:34:42.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:42.013+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:42.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:42.013+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:42.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:42 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:42.013+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:42.570 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:42 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:42.308+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:42.570 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:42 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:42.308+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:42.570 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:42 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:42.308+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:42.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:42 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:42.570+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:42.926 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:42 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:42.570+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:42.944 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:42 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:42.690+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:42.944 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:42 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:42.690+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:43.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:42 smithi189 podman[75661]: 2024-05-10 11:34:42.87473186 +0000 UTC m=+5.581267132 container died c11f46b410640e08e5657b5263d7ee58326a440fa8792e63d9bf3b065a431b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.schema-version=1.0) 2024-05-10T11:34:43.163 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:43 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:43.049+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:43.163 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:43 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:43.049+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:43.164 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:43 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:43.049+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:43.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:42 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:42.944+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:43.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:42 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:42.944+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:43.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:42 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:42.944+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:43.486 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:43 smithi189 podman[75661]: 2024-05-10 11:34:43.308741574 +0000 UTC m=+6.015276851 container cleanup c11f46b410640e08e5657b5263d7ee58326a440fa8792e63d9bf3b065a431b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:34:43.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:43 smithi189 bash[75661]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5 2024-05-10T11:34:43.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:43 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:43.161+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:43.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:43 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:43.161+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:43.585 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:43 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:43.318+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:43.585 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:43 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:43.318+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:43.585 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:43 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:43.318+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:43.924 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:43 smithi189 podman[75678]: 2024-05-10 11:34:43.509772883 +0000 UTC m=+0.633296918 container remove c11f46b410640e08e5657b5263d7ee58326a440fa8792e63d9bf3b065a431b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5) 2024-05-10T11:34:43.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:43 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:43.585+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:43.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:43 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:43.585+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:44.118 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:43 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:43.909+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:44.119 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:43 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:43.909+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:44.119 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:43 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:43.909+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:44.119 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:43 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:43.670+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:44.119 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:43 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:43.670+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:44.196 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:43 smithi189 podman[75769]: 2024-05-10 11:34:43.827513051 +0000 UTC m=+0.018067883 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:34:44.196 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:43 smithi189 podman[75769]: 2024-05-10 11:34:43.952254034 +0000 UTC m=+0.142808865 container create eaa9ce9662d6b751186e1451a16fa1c2fb674bce43ae79a56403dc1c27c7a9ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5-deactivate, ceph=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-10T11:34:44.196 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:44 smithi189 podman[75769]: 2024-05-10 11:34:44.196303469 +0000 UTC m=+0.386858301 container init eaa9ce9662d6b751186e1451a16fa1c2fb674bce43ae79a56403dc1c27c7a9ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:34:44.196 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:44.029+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:44.196 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:44.029+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:44.197 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:44.029+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:44.197 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:44.147+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:44.197 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:44.147+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:44.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:44 smithi189 podman[75769]: 2024-05-10 11:34:44.19986153 +0000 UTC m=+0.390416364 container start eaa9ce9662d6b751186e1451a16fa1c2fb674bce43ae79a56403dc1c27c7a9ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-10T11:34:44.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:44 smithi189 podman[75769]: 2024-05-10 11:34:44.313343181 +0000 UTC m=+0.503898028 container attach eaa9ce9662d6b751186e1451a16fa1c2fb674bce43ae79a56403dc1c27c7a9ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-10T11:34:44.577 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:44 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:44.284+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:44.577 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:44 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:44.284+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:44.577 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:44 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:44.284+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:44.840 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 10 11:34:44 smithi189 podman[75769]: 2024-05-10 11:34:44.491834719 +0000 UTC m=+0.682389547 container died eaa9ce9662d6b751186e1451a16fa1c2fb674bce43ae79a56403dc1c27c7a9ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-5-deactivate, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-10T11:34:44.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:44 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:44.577+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:44.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:44 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:44.577+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:45.118 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:44 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:44.947+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:45.119 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:44 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:44.947+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:45.119 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:44 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:44.947+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:45.119 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:44 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:44.701+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:45.119 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:44 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:44.701+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:45.141 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:44.993+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:45.141 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:44.993+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:45.142 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:44 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:44.993+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:45.162 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.5.service' 2024-05-10T11:34:45.204 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:34:45.205 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-10T11:34:45.205 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-10T11:34:45.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.6 2024-05-10T11:34:45.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:45 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:45.193+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:45.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:45 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:45.193+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:45.524 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:45 smithi027 systemd[1]: Stopping Ceph osd.6 for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:34:45.589 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:45 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:45.237+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:45.589 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:45 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:45.237+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:45.589 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:45 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:45.237+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:45.776 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:45 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:45.666+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:45.776 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:45 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:45.666+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:45.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:45 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:45.589+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:45.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:45 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:45.589+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:46.118 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:45 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:45.775+0000 7f807f200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:34:46.119 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:45 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:45.775+0000 7f807f200700 -1 osd.6 123 *** Got signal Terminated *** 2024-05-10T11:34:46.119 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:45 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:45.775+0000 7f807f200700 -1 osd.6 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-10T11:34:46.119 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:45 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:45.961+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:46.119 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:45 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:45.961+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:46.119 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:45 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:45.961+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:46.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:46 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:46.244+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:46.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:46 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:46.244+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:46.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:46 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:46.043+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:46.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:46 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:46.043+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:46.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:46 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:46.043+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:46.545 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:46 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:46.279+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:46.546 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:46 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:46.279+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:46.546 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:46 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:46.279+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:46.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:46 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:46.546+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:46.926 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:46 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:46.546+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:46.995 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:46 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:46.672+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:46.996 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:46 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:46.672+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:47.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:46 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:46.994+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:47.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:46 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:46.994+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:47.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:46 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:46.994+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:47.388 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:47 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:47.237+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:47.388 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:47 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:47.237+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:47.388 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:47 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:47.041+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:47.388 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:47 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:47.041+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:47.388 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:47 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:47.041+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:47.545 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:47 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:47.259+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:47.545 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:47 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:47.259+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:47.545 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:47 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:47.259+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:47.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:47 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:47.545+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:47.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:47 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:47.545+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:47.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:47 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:47.545+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:48.012 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:47 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:47.680+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:48.013 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:47 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:47.680+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:48.013 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:47 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:47.680+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.173317+0000 front 2024-05-10T11:34:20.173509+0000 (oldest deadline 2024-05-10T11:34:47.175071+0000) 2024-05-10T11:34:48.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:48 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:48.012+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:48.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:48 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:48.012+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:48.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:48 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:48.012+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:48.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:48 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:48.012+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:34:48.403 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:48 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:48.225+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:48.403 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:48 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:48.225+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:48.404 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:48 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:48.021+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:48.404 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:48 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:48.021+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:48.404 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:48 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:48.021+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:48.577 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:48 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:48.241+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:48.578 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:48 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:48.241+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:48.578 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:48 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:48.241+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:48.578 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:48 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:48.241+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:34:48.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:48 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:48.578+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:48.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:48 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:48.578+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:48.926 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:48 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:48.578+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:49.026 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:48 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:48.689+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:49.026 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:48 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:48.689+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:49.026 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:48 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:48.689+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.173317+0000 front 2024-05-10T11:34:20.173509+0000 (oldest deadline 2024-05-10T11:34:47.175071+0000) 2024-05-10T11:34:49.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:49 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:49.023+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:49.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:49 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:49.023+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:49.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:49 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:49.023+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:49.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:49 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:49.023+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:34:49.486 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:49 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:49.058+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:49.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:49 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:49.058+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:49.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:49 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:49.058+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:49.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:49 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:49.181+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:49.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:49 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:49.181+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:49.551 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:49.289+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:49.551 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:49.289+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:49.551 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:49.289+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:49.552 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:49.289+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:34:49.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:49.551+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:49.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:49.551+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:49.926 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:49 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:49.551+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:50.033 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:49 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:49.720+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:50.033 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:49 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:49.720+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:50.033 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:49 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:49.720+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.173317+0000 front 2024-05-10T11:34:20.173509+0000 (oldest deadline 2024-05-10T11:34:47.175071+0000) 2024-05-10T11:34:50.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:50 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:50.033+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:50.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:50 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:50.033+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:50.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:50 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:50.033+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:50.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:50 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:50.033+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:34:50.486 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:50 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:50.033+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:50.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:50 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:50.033+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:50.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:50 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:50.033+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:50.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:50 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:50.208+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:50.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:50 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:50.208+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:50.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:50 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:50.208+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:34:50.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:50 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:50.327+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:50.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:50 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:50.327+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:50.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:50 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:50.327+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:50.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:50 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:50.327+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:34:50.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:50 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:50.552+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:50.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:50 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:50.552+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:50.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:50 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:50.552+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:51.075 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:50 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:50.753+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.765977+0000 front 2024-05-10T11:34:06.766080+0000 (oldest deadline 2024-05-10T11:34:30.866233+0000) 2024-05-10T11:34:51.075 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:50 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:50.753+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.769915+0000 front 2024-05-10T11:34:13.769832+0000 (oldest deadline 2024-05-10T11:34:38.470052+0000) 2024-05-10T11:34:51.075 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:50 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6[57929]: 2024-05-10T11:34:50.753+0000 7f807aa00700 -1 osd.6 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.173317+0000 front 2024-05-10T11:34:20.173509+0000 (oldest deadline 2024-05-10T11:34:47.175071+0000) 2024-05-10T11:34:51.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:51.074+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:51.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:51.074+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:51.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:51.074+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:51.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:51 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:51.074+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:34:51.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:51 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:51.080+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:51.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:51 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:51.080+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:51.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:51 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:51.080+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:51.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:51 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:51.226+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:51.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:51 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:51.226+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:51.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:51 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:51.226+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:34:51.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:51 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:51.575+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:51.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:51 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:51.575+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:51.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:51 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:51.575+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:51.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:51 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:51.363+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:51.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:51 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:51.363+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:51.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:51 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:51.363+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:51.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:51 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:51.363+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:34:51.839 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:51 smithi027 podman[126106]: 2024-05-10 11:34:51.547168742 +0000 UTC m=+5.910559959 container died b9c4302f8adf5d847f36d427dfcfaed6a485782da195e539d7c0a1ae4c5d74d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, ceph=True) 2024-05-10T11:34:52.094 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:52 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:52.041+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:52.094 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:52 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:52.041+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:52.094 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:52 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:52.041+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:52.095 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:52 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:52.041+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:34:52.095 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:52 smithi027 podman[126106]: 2024-05-10 11:34:52.094165801 +0000 UTC m=+6.457557021 container cleanup b9c4302f8adf5d847f36d427dfcfaed6a485782da195e539d7c0a1ae4c5d74d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-10T11:34:52.368 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:52 smithi027 bash[126106]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6 2024-05-10T11:34:52.369 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:52 smithi027 podman[126135]: 2024-05-10 11:34:52.337584185 +0000 UTC m=+0.780908766 container remove b9c4302f8adf5d847f36d427dfcfaed6a485782da195e539d7c0a1ae4c5d74d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-10T11:34:52.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:52 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:52.118+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:52.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:52 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:52.118+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:52.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:52 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:52.118+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:52.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:52 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:52.178+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:52.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:52 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:52.178+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:52.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:52 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:52.178+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:34:52.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:52 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:52.603+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:52.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:52 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:52.603+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:52.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:52 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:52.603+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:52.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:52 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:52.411+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:52.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:52 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:52.411+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:52.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:52 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:52.411+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:52.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:52 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:52.411+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:34:53.347 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:53 smithi027 podman[126228]: 2024-05-10 11:34:53.059286476 +0000 UTC m=+0.034094984 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:34:53.347 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:53 smithi027 podman[126228]: 2024-05-10 11:34:53.188427093 +0000 UTC m=+0.163235638 container create 71f93e63630313ff9d1bcb8ef89ab014e44a73a9edadfb6a53f85578278f7a67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, ceph=True) 2024-05-10T11:34:53.347 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:53 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:53.087+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:53.347 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:53 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:53.087+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:53.347 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:53 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:53.087+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:53.348 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:53 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:53.087+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:34:53.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:53 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:53.147+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:53.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:53 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:53.147+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:53.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:53 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:53.147+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:53.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:53 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:53.198+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:53.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:53 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:53.198+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:53.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:53 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:53.198+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:34:53.643 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:53 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:53.384+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:53.644 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:53 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:53.384+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:53.644 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:53 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:53.384+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:53.644 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:53 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:53.384+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:34:53.868 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:53 smithi027 podman[126228]: 2024-05-10 11:34:53.444947904 +0000 UTC m=+0.419756456 container init 71f93e63630313ff9d1bcb8ef89ab014e44a73a9edadfb6a53f85578278f7a67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-10T11:34:53.869 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:53 smithi027 podman[126228]: 2024-05-10 11:34:53.457664218 +0000 UTC m=+0.432472786 container start 71f93e63630313ff9d1bcb8ef89ab014e44a73a9edadfb6a53f85578278f7a67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-10T11:34:53.869 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:53 smithi027 podman[126228]: 2024-05-10 11:34:53.532296527 +0000 UTC m=+0.507105068 container attach 71f93e63630313ff9d1bcb8ef89ab014e44a73a9edadfb6a53f85578278f7a67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-10T11:34:53.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:53 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:53.643+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:53.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:53 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:53.643+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:53.925 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:53 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:53.643+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:54.356 INFO:journalctl@ceph.osd.6.smithi027.stdout:May 10 11:34:53 smithi027 podman[126228]: 2024-05-10 11:34:53.929357485 +0000 UTC m=+0.904166006 container died 71f93e63630313ff9d1bcb8ef89ab014e44a73a9edadfb6a53f85578278f7a67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-05-10T11:34:54.357 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:54 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:54.041+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:54.357 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:54 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:54.041+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:54.357 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:54 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:54.041+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:54.358 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:54 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:54.041+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:34:54.486 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:54 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:54.153+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:54.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:54 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:54.153+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:54.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:54 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:54.153+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:54.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:54 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:54.221+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:54.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:54 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:54.221+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:54.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:54 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:54.221+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:34:54.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:54 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:54.605+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:54.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:54 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:54.605+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:54.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:54 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:54.605+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:54.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:54 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:54.355+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:54.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:54 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:54.355+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:54.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:54 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:54.355+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:54.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:54 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:54.355+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:34:54.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.6.service' 2024-05-10T11:34:54.880 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:34:54.881 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-10T11:34:54.881 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-10T11:34:54.881 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.7 2024-05-10T11:34:55.175 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:54 smithi076 systemd[1]: Stopping Ceph osd.7 for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:34:55.368 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:55 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:55.080+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:55.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:55 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:55.080+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:55.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:55 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:55.080+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:55.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:55 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:55.080+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:34:55.486 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:55 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:55.183+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:55.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:55 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:55.183+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:55.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:55 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:55.183+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:55.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:55 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:55.224+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:55.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:55 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:55.224+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:55.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:55 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:55.224+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:34:55.574 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:55 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:55.186+0000 7f6a5ddad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:34:55.574 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:55 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:55.186+0000 7f6a5ddad700 -1 osd.7 123 *** Got signal Terminated *** 2024-05-10T11:34:55.574 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:55 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:55.186+0000 7f6a5ddad700 -1 osd.7 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-10T11:34:55.574 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:55 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:55.573+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:55.574 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:55 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:55.573+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:55.574 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:55 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:55.573+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:55.575 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:55 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:55.360+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:55.575 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:55 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:55.360+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:55.575 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:55 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:55.360+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:55.575 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:55 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:55.360+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:34:56.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:56 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:56.100+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:56.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:56 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:56.100+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:56.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:56 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:56.100+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:56.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:56 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:56.100+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:34:56.486 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:56 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:56.226+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:56.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:56 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:56.226+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:56.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:56 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:56.226+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:56.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:56 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:56.194+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:56.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:56 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:56.194+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:56.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:56 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:56.194+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:34:56.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:56 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:56.593+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:56.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:56 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:56.593+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:56.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:56 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:56.593+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:56.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:56 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:56.392+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:56.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:56 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:56.392+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:56.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:56 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:56.392+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:56.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:56 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:56.392+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:34:57.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:57 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:57.270+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:57.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:57 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:57.270+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:57.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:57 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:57.270+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:57.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:57 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:57.270+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.764228+0000 front 2024-05-10T11:34:31.764180+0000 (oldest deadline 2024-05-10T11:34:56.464282+0000) 2024-05-10T11:34:57.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:57 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:57.153+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:57.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:57 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:57.153+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:57.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:57 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:57.153+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:34:57.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:57 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:57.153+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:34:57.618 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:57 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:57.121+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:57.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:57 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:57.121+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:57.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:57 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:57.121+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:57.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:57 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:57.121+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:34:57.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:57 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:57.121+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:34:57.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:57 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:57.584+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:57.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:57 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:57.584+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:57.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:57 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:57.584+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:57.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:57 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:57.383+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:57.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:57 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:57.383+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:57.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:57 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:57.383+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:57.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:57 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:57.383+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:34:58.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:58 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:58.138+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:58.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:58 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:58.138+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:58.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:58 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:58.138+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:34:58.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:58 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:58.138+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:34:58.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:58 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:58.222+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:58.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:58 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:58.222+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:58.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:58 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:58.222+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:58.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:58 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:58.222+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.764228+0000 front 2024-05-10T11:34:31.764180+0000 (oldest deadline 2024-05-10T11:34:56.464282+0000) 2024-05-10T11:34:58.618 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:58 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:58.168+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:58.618 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:58 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:58.168+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:58.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:58 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:58.168+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:58.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:58 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:58.168+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:34:58.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:58 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:58.168+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:34:58.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:58 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:58.580+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:58.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:58 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:58.580+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:58.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:58 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:58.580+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:58.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:58 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:58.580+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.653635+0000 front 2024-05-10T11:34:32.653655+0000 (oldest deadline 2024-05-10T11:34:57.953791+0000) 2024-05-10T11:34:58.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:58 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:58.334+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:58.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:58 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:58.334+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:58.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:58 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:58.334+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:58.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:58 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:58.334+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:34:58.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:58 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:58.334+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:34:59.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:59 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:59.096+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:34:59.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:59 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:59.096+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:34:59.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:59 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:59.096+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:34:59.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:34:59 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:34:59.096+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:34:59.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:59 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:59.230+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:34:59.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:59 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:59.230+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:34:59.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:59 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:59.230+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:34:59.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:34:59 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:34:59.230+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.764228+0000 front 2024-05-10T11:34:31.764180+0000 (oldest deadline 2024-05-10T11:34:56.464282+0000) 2024-05-10T11:34:59.618 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:59 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:59.165+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:34:59.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:59 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:59.165+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:34:59.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:59 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:59.165+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:34:59.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:59 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:59.165+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:34:59.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:34:59 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:34:59.165+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:34:59.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:59 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:59.572+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.549109+0000 front 2024-05-10T11:34:07.549151+0000 (oldest deadline 2024-05-10T11:34:33.449311+0000) 2024-05-10T11:34:59.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:59 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:59.572+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.450085+0000 front 2024-05-10T11:34:13.450026+0000 (oldest deadline 2024-05-10T11:34:36.950123+0000) 2024-05-10T11:34:59.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:59 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:59.572+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:20.952108+0000 front 2024-05-10T11:34:20.952196+0000 (oldest deadline 2024-05-10T11:34:46.851957+0000) 2024-05-10T11:34:59.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:34:59 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7[52204]: 2024-05-10T11:34:59.572+0000 7f6a5a3a1700 -1 osd.7 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.653635+0000 front 2024-05-10T11:34:32.653655+0000 (oldest deadline 2024-05-10T11:34:57.953791+0000) 2024-05-10T11:34:59.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:59 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:59.350+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:34:59.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:59 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:59.350+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:34:59.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:59 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:59.350+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:34:59.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:59 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:59.350+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:34:59.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:34:59 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:34:59.350+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:00.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:00 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:00.047+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:00.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:00 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:00.047+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:00.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:00 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:00.047+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:00.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:00 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:00.047+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:00.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:00 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:00.232+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:35:00.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:00 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:00.233+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:35:00.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:00 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:00.233+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:35:00.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:00 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:00.233+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.764228+0000 front 2024-05-10T11:34:31.764180+0000 (oldest deadline 2024-05-10T11:34:56.464282+0000) 2024-05-10T11:35:00.618 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:00 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:00.140+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:00.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:00 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:00.140+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:00.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:00 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:00.140+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:00.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:00 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:00.140+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:00.620 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:00 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:00.140+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:00.675 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:35:00 smithi076 podman[73917]: 2024-05-10 11:35:00.602182113 +0000 UTC m=+5.536887229 container died 99ee9eb90d3a5c7606c825cf33249328bf0540df44d9871aa99fbcab4665ec65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.build-date=20240507, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-10T11:35:00.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:00 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:00.395+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:00.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:00 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:00.395+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:00.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:00 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:00.395+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:00.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:00 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:00.395+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:00.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:00 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:00.395+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:01.175 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:35:01 smithi076 podman[73917]: 2024-05-10 11:35:01.020571342 +0000 UTC m=+5.955276466 container cleanup 99ee9eb90d3a5c7606c825cf33249328bf0540df44d9871aa99fbcab4665ec65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507) 2024-05-10T11:35:01.176 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:35:01 smithi076 bash[73917]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7 2024-05-10T11:35:01.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:01 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:01.021+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:01.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:01 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:01.021+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:01.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:01 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:01.021+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:01.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:01 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:01.021+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:01.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:01 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:01.251+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:35:01.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:01 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:01.251+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:35:01.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:01 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:01.251+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:35:01.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:01 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:01.251+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.764228+0000 front 2024-05-10T11:34:31.764180+0000 (oldest deadline 2024-05-10T11:34:56.464282+0000) 2024-05-10T11:35:01.549 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:35:01 smithi076 podman[73936]: 2024-05-10 11:35:01.247336741 +0000 UTC m=+0.643519170 container remove 99ee9eb90d3a5c7606c825cf33249328bf0540df44d9871aa99fbcab4665ec65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS) 2024-05-10T11:35:01.549 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:01 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:01.442+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:01.549 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:01 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:01.442+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:01.549 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:01 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:01.442+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:01.549 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:01 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:01.442+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:01.549 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:01 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:01.442+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:01.618 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:01 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:01.158+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:01.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:01 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:01.158+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:01.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:01 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:01.158+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:01.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:01 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:01.158+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:01.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:01 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:01.158+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:01.808 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:35:01 smithi076 podman[74035]: 2024-05-10 11:35:01.454677989 +0000 UTC m=+0.021271339 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:35:01.808 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:35:01 smithi076 podman[74035]: 2024-05-10 11:35:01.581524035 +0000 UTC m=+0.148117365 container create 696f4f1ac67ab4141fa0ca50dafc876a545aec96dfe4821dc82341aa028ae26d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-10T11:35:02.120 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:35:01 smithi076 podman[74035]: 2024-05-10 11:35:01.80902431 +0000 UTC m=+0.375617639 container init 696f4f1ac67ab4141fa0ca50dafc876a545aec96dfe4821dc82341aa028ae26d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-10T11:35:02.120 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:35:01 smithi076 podman[74035]: 2024-05-10 11:35:01.812257967 +0000 UTC m=+0.378851300 container start 696f4f1ac67ab4141fa0ca50dafc876a545aec96dfe4821dc82341aa028ae26d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-10T11:35:02.120 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:35:01 smithi076 podman[74035]: 2024-05-10 11:35:01.917527733 +0000 UTC m=+0.484121069 container attach 696f4f1ac67ab4141fa0ca50dafc876a545aec96dfe4821dc82341aa028ae26d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-05-10T11:35:02.289 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:02 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:02.002+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:02.289 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:02 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:02.002+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:02.289 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:02 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:02.002+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:02.289 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:02 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:02.002+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:02.425 INFO:journalctl@ceph.osd.7.smithi076.stdout:May 10 11:35:02 smithi076 podman[74035]: 2024-05-10 11:35:02.12099739 +0000 UTC m=+0.687590721 container died 696f4f1ac67ab4141fa0ca50dafc876a545aec96dfe4821dc82341aa028ae26d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-10T11:35:02.618 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:02 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:02.195+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:02.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:02 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:02.195+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:02.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:02 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:02.195+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:02.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:02 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:02.195+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:02.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:02 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:02.195+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:02.736 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:02 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:02.289+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:35:02.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:02 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:02.289+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:35:02.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:02 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:02.289+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:35:02.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:02 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:02.289+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.764228+0000 front 2024-05-10T11:34:31.764180+0000 (oldest deadline 2024-05-10T11:34:56.464282+0000) 2024-05-10T11:35:02.755 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:02 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:02.476+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:02.755 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:02 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:02.476+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:02.756 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:02 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:02.476+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:02.756 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:02 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:02.476+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:02.756 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:02 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:02.476+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:02.787 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.7.service' 2024-05-10T11:35:02.826 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:35:02.827 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-10T11:35:02.827 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-10T11:35:02.827 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.8 2024-05-10T11:35:03.153 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:02 smithi189 systemd[1]: Stopping Ceph osd.8 for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:35:03.153 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:03.152+0000 7f294bef3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:35:03.153 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:03.152+0000 7f294bef3700 -1 osd.8 123 *** Got signal Terminated *** 2024-05-10T11:35:03.153 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:03.048+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:03.153 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:03.048+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:03.154 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:03.048+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:03.154 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:03.048+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:03.267 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:03 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:03.245+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:03.309 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:03 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:03.245+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:03.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:03.152+0000 7f294bef3700 -1 osd.8 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-10T11:35:03.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:03.329+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:35:03.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:03.329+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:35:03.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:03.329+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:35:03.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:03 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:03.329+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.764228+0000 front 2024-05-10T11:34:31.764180+0000 (oldest deadline 2024-05-10T11:34:56.464282+0000) 2024-05-10T11:35:03.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:03 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:03.245+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:03.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:03 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:03.245+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:03.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:03 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:03.245+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:03.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:03.454+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:03.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:03.454+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:03.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:03.454+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:03.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:03.454+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:03.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:03.454+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:04.316 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:04.023+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:04.316 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:04.023+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:04.317 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:04.023+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:04.317 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:04.023+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:04.623 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:04 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:04.211+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:04.623 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:04 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:04.211+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:04.623 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:04 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:04.211+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:04.624 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:04 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:04.211+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:04.624 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:04 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:04.211+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:04.675 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:04 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:04.422+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:04.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:04 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:04.422+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:04.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:04 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:04.422+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:04.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:04 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:04.422+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:04.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:04 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:04.422+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:04.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:04 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:04.422+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:04.736 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:04.316+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:35:04.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:04.316+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:35:04.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:04.316+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:35:04.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:04.316+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.764228+0000 front 2024-05-10T11:34:31.764180+0000 (oldest deadline 2024-05-10T11:34:56.464282+0000) 2024-05-10T11:35:05.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:04.983+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:05.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:04.983+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:05.238 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:04.983+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:05.238 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:04 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:04.983+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:05.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:05 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:05.194+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:05.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:05 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:05.194+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:05.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:05 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:05.194+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:05.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:05 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:05.194+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:05.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:05 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:05.194+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:05.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:05 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:05.355+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:35:05.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:05 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:05.355+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:35:05.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:05 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:05.355+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:35:05.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:05 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:05.355+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.764228+0000 front 2024-05-10T11:34:31.764180+0000 (oldest deadline 2024-05-10T11:34:56.464282+0000) 2024-05-10T11:35:05.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:05.437+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:05.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:05.437+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:05.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:05.437+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:05.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:05.437+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:05.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:05.437+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:05.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:05 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:05.437+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:06.359 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:06 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:06.003+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:06.359 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:06 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:06.003+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:06.360 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:06 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:06.003+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:06.360 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:06 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:06.003+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:06.360 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:06 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:06.003+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:06.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:06 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:06.187+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:06.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:06 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:06.187+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:06.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:06 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:06.187+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:06.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:06 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:06.187+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:06.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:06 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:06.187+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:06.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:06 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:06.187+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:42.216039+0000 front 2024-05-10T11:34:42.216211+0000 (oldest deadline 2024-05-10T11:35:05.716238+0000) 2024-05-10T11:35:06.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:06 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:06.359+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:35:06.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:06 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:06.359+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:35:06.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:06 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:06.359+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:35:06.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:06 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:06.359+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.764228+0000 front 2024-05-10T11:34:31.764180+0000 (oldest deadline 2024-05-10T11:34:56.464282+0000) 2024-05-10T11:35:06.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:06 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:06.447+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:06.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:06 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:06.447+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:06.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:06 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:06.447+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:06.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:06 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:06.447+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:06.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:06 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:06.447+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:06.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:06 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:06.447+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:07.360 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:07 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:07.033+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:07.360 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:07 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:07.033+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:07.360 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:07 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:07.033+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:07.360 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:07 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:07.033+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:07.361 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:07 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:07.033+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:07.618 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:07 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:07.140+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:07.618 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:07 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:07.140+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:07.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:07 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:07.140+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:07.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:07 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:07.140+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:07.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:07 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:07.140+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:07.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:07 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:07.140+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:42.216039+0000 front 2024-05-10T11:34:42.216211+0000 (oldest deadline 2024-05-10T11:35:05.716238+0000) 2024-05-10T11:35:07.736 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:07 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:07.359+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:57.154900+0000 front 2024-05-10T11:33:57.154794+0000 (oldest deadline 2024-05-10T11:34:19.454766+0000) 2024-05-10T11:35:07.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:07 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:07.359+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:07.458108+0000 front 2024-05-10T11:34:07.458133+0000 (oldest deadline 2024-05-10T11:34:29.758242+0000) 2024-05-10T11:35:07.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:07 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:07.359+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:16.060710+0000 front 2024-05-10T11:34:16.060732+0000 (oldest deadline 2024-05-10T11:34:37.760837+0000) 2024-05-10T11:35:07.737 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:07 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8[53483]: 2024-05-10T11:35:07.359+0000 7f29484e7700 -1 osd.8 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.764228+0000 front 2024-05-10T11:34:31.764180+0000 (oldest deadline 2024-05-10T11:34:56.464282+0000) 2024-05-10T11:35:07.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:07 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:07.480+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:07.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:07 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:07.480+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:07.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:07 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:07.480+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:07.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:07 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:07.480+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:07.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:07 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:07.480+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:07.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:07 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:07.480+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:08.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:08 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:08.011+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:08.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:08 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:08.011+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:08.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:08 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:08.011+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:08.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:08 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:08.011+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:08.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:08 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:08.011+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:08.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:08 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:08.158+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:08.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:08 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:08.158+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:08.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:08 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:08.158+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:08.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:08 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:08.158+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:08.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:08 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:08.158+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:08.620 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:08 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:08.158+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:42.216039+0000 front 2024-05-10T11:34:42.216211+0000 (oldest deadline 2024-05-10T11:35:05.716238+0000) 2024-05-10T11:35:08.912 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:08 smithi189 podman[75887]: 2024-05-10 11:35:08.568944402 +0000 UTC m=+5.545728050 container stop ba987b32ed9f9855f003d4981f7acd1811ee1df1fc90af63f7c3416df56a76a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-10T11:35:08.912 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:08 smithi189 podman[75887]: 2024-05-10 11:35:08.594535526 +0000 UTC m=+5.571319173 container died ba987b32ed9f9855f003d4981f7acd1811ee1df1fc90af63f7c3416df56a76a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-10T11:35:08.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:08 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:08.514+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:08.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:08 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:08.514+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:08.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:08 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:08.514+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:08.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:08 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:08.514+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:08.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:08 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:08.514+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:08.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:08 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:08.514+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:09.211 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:09 smithi189 podman[75887]: 2024-05-10 11:35:09.010510673 +0000 UTC m=+5.987294328 container cleanup ba987b32ed9f9855f003d4981f7acd1811ee1df1fc90af63f7c3416df56a76a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-10T11:35:09.211 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:09 smithi189 bash[75887]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8 2024-05-10T11:35:09.211 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:09.030+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:09.212 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:09.030+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:09.212 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:09.030+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:09.212 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:09.030+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:09.212 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:09.030+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:09.486 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:09 smithi189 podman[75905]: 2024-05-10 11:35:09.211697175 +0000 UTC m=+0.636829603 container remove ba987b32ed9f9855f003d4981f7acd1811ee1df1fc90af63f7c3416df56a76a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08) 2024-05-10T11:35:09.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:09 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:09.170+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:09.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:09 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:09.170+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:09.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:09 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:09.170+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:09.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:09 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:09.170+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:09.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:09 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:09.170+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:09.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:09 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:09.170+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:42.216039+0000 front 2024-05-10T11:34:42.216211+0000 (oldest deadline 2024-05-10T11:35:05.716238+0000) 2024-05-10T11:35:09.906 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:09 smithi189 podman[75996]: 2024-05-10 11:35:09.524221354 +0000 UTC m=+0.018334394 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:35:09.906 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:09 smithi189 podman[75996]: 2024-05-10 11:35:09.64616364 +0000 UTC m=+0.140276698 container create f5a7be24f3a2041abc8bf88ad2a9012ef61bc8f66c197d9ec154c5f121eecb9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-10T11:35:09.906 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:09 smithi189 podman[75996]: 2024-05-10 11:35:09.906736145 +0000 UTC m=+0.400849164 container init f5a7be24f3a2041abc8bf88ad2a9012ef61bc8f66c197d9ec154c5f121eecb9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-10T11:35:09.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:09 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:09.535+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:09.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:09 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:09.535+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:09.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:09 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:09.535+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:09.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:09 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:09.535+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:09.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:09 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:09.535+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:09.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:09 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:09.535+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:10.202 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:09 smithi189 podman[75996]: 2024-05-10 11:35:09.910164366 +0000 UTC m=+0.404277384 container start f5a7be24f3a2041abc8bf88ad2a9012ef61bc8f66c197d9ec154c5f121eecb9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-10T11:35:10.202 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:09 smithi189 podman[75996]: 2024-05-10 11:35:09.990072113 +0000 UTC m=+0.484185137 container attach f5a7be24f3a2041abc8bf88ad2a9012ef61bc8f66c197d9ec154c5f121eecb9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8-deactivate, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-10T11:35:10.202 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:09.980+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:10.202 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:09.980+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:10.203 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:09.980+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:10.203 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:09.980+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:10.203 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:09 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:09.980+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:10.478 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 10 11:35:10 smithi189 podman[75996]: 2024-05-10 11:35:10.202484017 +0000 UTC m=+0.696597048 container died f5a7be24f3a2041abc8bf88ad2a9012ef61bc8f66c197d9ec154c5f121eecb9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, ceph=True) 2024-05-10T11:35:10.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:10 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:10.202+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:10.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:10 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:10.202+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:10.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:10 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:10.202+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:10.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:10 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:10.202+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:10.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:10 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:10.202+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:10.620 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:10 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:10.202+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:42.216039+0000 front 2024-05-10T11:34:42.216211+0000 (oldest deadline 2024-05-10T11:35:05.716238+0000) 2024-05-10T11:35:10.812 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.8.service' 2024-05-10T11:35:10.850 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:35:10.850 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-10T11:35:10.851 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-10T11:35:10.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.9 2024-05-10T11:35:10.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:10 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:10.562+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:10.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:10 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:10.562+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:10.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:10 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:10.562+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:10.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:10 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:10.562+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:10.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:10 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:10.562+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:10.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:10 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:10.562+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:11.216 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:10 smithi027 systemd[1]: Stopping Ceph osd.9 for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:35:11.236 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:11.028+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:11.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:11.028+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:11.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:11.028+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:11.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:11.028+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:11.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:11.028+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:11.513 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:11.214+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:11.514 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:11.214+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:11.514 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:11.214+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:11.514 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:11.214+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:11.514 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:11.214+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:11.514 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:11.214+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:42.216039+0000 front 2024-05-10T11:34:42.216211+0000 (oldest deadline 2024-05-10T11:35:05.716238+0000) 2024-05-10T11:35:11.514 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:11.512+0000 7f282ec00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:35:11.868 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:11.512+0000 7f282ec00700 -1 osd.9 123 *** Got signal Terminated *** 2024-05-10T11:35:11.868 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:11 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:11.512+0000 7f282ec00700 -1 osd.9 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-10T11:35:11.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:11 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:11.610+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:11.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:11 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:11.610+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:11.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:11 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:11.610+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:11.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:11 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:11.610+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:11.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:11 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:11.610+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:11.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:11 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:11.610+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:12.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:11.990+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:12.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:11.990+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:12.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:11.990+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:12.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:11.990+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:12.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:11.990+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:12.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:12 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:12.172+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:12.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:12 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:12.172+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:12.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:12 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:12.173+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:12.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:12 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:12.173+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:12.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:12 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:12.173+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:12.620 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:12 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:12.173+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:42.216039+0000 front 2024-05-10T11:34:42.216211+0000 (oldest deadline 2024-05-10T11:35:05.716238+0000) 2024-05-10T11:35:12.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:12 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:12.625+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:12.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:12 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:12.625+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:12.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:12 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:12.625+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:12.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:12 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:12.625+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:12.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:12 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:12.625+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:12.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:12 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:12.625+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:13.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:12 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:12.995+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:13.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:12 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:12.995+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:13.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:12 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:12.995+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:13.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:12 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:12.995+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:13.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:12 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:12.995+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:13.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:12 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:12.995+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:13.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:13.155+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:13.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:13.155+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:13.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:13.155+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:13.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:13.155+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:13.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:13.155+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:13.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:13 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:13.155+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:42.216039+0000 front 2024-05-10T11:34:42.216211+0000 (oldest deadline 2024-05-10T11:35:05.716238+0000) 2024-05-10T11:35:13.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:13 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:13.618+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:13.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:13 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:13.618+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:13.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:13 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:13.618+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:13.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:13 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:13.618+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:13.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:13 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:13.618+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:13.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:13 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:13.618+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:14.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:14 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:14.038+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:14.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:14 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:14.038+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:14.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:14 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:14.038+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:14.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:14 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:14.038+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:14.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:14 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:14.038+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:14.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:14 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:14.038+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:14.521 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:14.119+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:14.521 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:14.119+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:14.521 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:14.119+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:14.522 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:14.119+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:14.522 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:14.119+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:14.522 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:14 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:14.119+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:42.216039+0000 front 2024-05-10T11:34:42.216211+0000 (oldest deadline 2024-05-10T11:35:05.716238+0000) 2024-05-10T11:35:14.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:14 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:14.583+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:14.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:14 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:14.583+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:14.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:14 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:14.583+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:14.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:14 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:14.583+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:14.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:14 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:14.583+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:14.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:14 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:14.583+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:14.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:14 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:14.583+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:50.443178+0000 front 2024-05-10T11:34:50.443061+0000 (oldest deadline 2024-05-10T11:35:14.542749+0000) 2024-05-10T11:35:15.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:15 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:15.009+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:15.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:15 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:15.009+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:15.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:15 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:15.009+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:15.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:15 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:15.009+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:15.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:15 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:15.009+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:15.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:15 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:15.009+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:15.548 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:15 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:15.142+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:15.549 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:15 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:15.142+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:15.549 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:15 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:15.142+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:15.549 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:15 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:15.142+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:15.549 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:15 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:15.142+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:15.549 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:15 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:15.142+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:42.216039+0000 front 2024-05-10T11:34:42.216211+0000 (oldest deadline 2024-05-10T11:35:05.716238+0000) 2024-05-10T11:35:15.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:15 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:15.555+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:15.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:15 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:15.555+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:15.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:15 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:15.555+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:15.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:15 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:15.555+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:15.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:15 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:15.555+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:15.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:15 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:15.555+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:15.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:15 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:15.555+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:50.443178+0000 front 2024-05-10T11:34:50.443061+0000 (oldest deadline 2024-05-10T11:35:14.542749+0000) 2024-05-10T11:35:16.259 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:16 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:16.127+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:54.899872+0000 front 2024-05-10T11:33:54.899805+0000 (oldest deadline 2024-05-10T11:34:24.900115+0000) 2024-05-10T11:35:16.260 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:16 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:16.127+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:04.901027+0000 front 2024-05-10T11:34:04.900927+0000 (oldest deadline 2024-05-10T11:34:29.600761+0000) 2024-05-10T11:35:16.260 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:16 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:16.127+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:13.005637+0000 front 2024-05-10T11:34:13.005712+0000 (oldest deadline 2024-05-10T11:34:38.305873+0000) 2024-05-10T11:35:16.260 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:16 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:16.127+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:19.408599+0000 front 2024-05-10T11:34:19.408707+0000 (oldest deadline 2024-05-10T11:34:47.009938+0000) 2024-05-10T11:35:16.260 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:16 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:16.127+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:31.712857+0000 front 2024-05-10T11:34:31.712749+0000 (oldest deadline 2024-05-10T11:34:57.013011+0000) 2024-05-10T11:35:16.260 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:16 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9[64824]: 2024-05-10T11:35:16.127+0000 7f282a400700 -1 osd.9 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:42.216039+0000 front 2024-05-10T11:34:42.216211+0000 (oldest deadline 2024-05-10T11:35:05.716238+0000) 2024-05-10T11:35:16.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:16 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:16.037+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:16.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:16 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:16.037+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:16.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:16 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:16.037+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:16.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:16 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:16.037+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:16.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:16 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:16.037+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:16.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:16 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:16.037+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:16.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:16 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:16.523+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:16.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:16 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:16.523+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:16.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:16 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:16.523+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:16.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:16 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:16.523+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:16.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:16 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:16.523+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:16.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:16 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:16.523+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:16.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:16 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:16.523+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:50.443178+0000 front 2024-05-10T11:34:50.443061+0000 (oldest deadline 2024-05-10T11:35:14.542749+0000) 2024-05-10T11:35:17.388 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:17 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:17.016+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:17.388 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:17 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:17.016+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:17.388 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:17 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:17.016+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:17.389 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:17 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:17.016+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:17.389 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:17 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:17.016+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:17.389 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:17 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:17.016+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:17.618 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:17 smithi027 podman[126333]: 2024-05-10 11:35:17.31357394 +0000 UTC m=+5.942371792 container died c1961132089847217f4c8b3f2b57da057684ea35905a5c88fe6be8414c661e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-05-10T11:35:17.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:17 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:17.482+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:17.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:17 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:17.482+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:17.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:17 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:17.482+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:17.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:17 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:17.482+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:17.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:17 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:17.482+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:17.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:17 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:17.482+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:17.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:17 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:17.482+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:50.443178+0000 front 2024-05-10T11:34:50.443061+0000 (oldest deadline 2024-05-10T11:35:14.542749+0000) 2024-05-10T11:35:18.118 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:17 smithi027 podman[126333]: 2024-05-10 11:35:17.738251577 +0000 UTC m=+6.367049422 container cleanup c1961132089847217f4c8b3f2b57da057684ea35905a5c88fe6be8414c661e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.vendor=CentOS) 2024-05-10T11:35:18.119 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:17 smithi027 bash[126333]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9 2024-05-10T11:35:18.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:18 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:18.005+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:18.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:18 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:18.005+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:18.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:18 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:18.005+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:18.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:18 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:18.005+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:18.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:18 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:18.005+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:18.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:18 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:18.005+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:18.618 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:18 smithi027 podman[126365]: 2024-05-10 11:35:18.14754471 +0000 UTC m=+0.820134530 container remove c1961132089847217f4c8b3f2b57da057684ea35905a5c88fe6be8414c661e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-10T11:35:18.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:18 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:18.445+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:18.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:18 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:18.445+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:18.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:18 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:18.445+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:18.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:18 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:18.445+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:18.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:18 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:18.445+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:18.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:18 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:18.445+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:18.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:18 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:18.445+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:50.443178+0000 front 2024-05-10T11:34:50.443061+0000 (oldest deadline 2024-05-10T11:35:14.542749+0000) 2024-05-10T11:35:19.241 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:18 smithi027 podman[126452]: 2024-05-10 11:35:18.884021352 +0000 UTC m=+0.035884150 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:35:19.241 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:19 smithi027 podman[126452]: 2024-05-10 11:35:19.073524792 +0000 UTC m=+0.225387618 container create 30c61d23217df7f6e98f664b8c491362d658123753eccdf603c7d29eeed77dd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:35:19.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:18 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:18.987+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:19.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:18 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:18.987+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:19.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:18 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:18.987+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:19.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:18 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:18.987+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:19.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:18 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:18.987+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:19.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:18 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:18.987+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:19.618 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:19 smithi027 podman[126452]: 2024-05-10 11:35:19.402664202 +0000 UTC m=+0.554527045 container init 30c61d23217df7f6e98f664b8c491362d658123753eccdf603c7d29eeed77dd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240507) 2024-05-10T11:35:19.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:19 smithi027 podman[126452]: 2024-05-10 11:35:19.414812633 +0000 UTC m=+0.566675462 container start 30c61d23217df7f6e98f664b8c491362d658123753eccdf603c7d29eeed77dd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-10T11:35:19.619 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:19 smithi027 podman[126452]: 2024-05-10 11:35:19.476067508 +0000 UTC m=+0.627930314 container attach 30c61d23217df7f6e98f664b8c491362d658123753eccdf603c7d29eeed77dd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:35:19.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:19 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:19.446+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:19.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:19 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:19.446+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:19.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:19 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:19.446+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:19.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:19 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:19.446+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:19.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:19 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:19.446+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:19.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:19 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:19.446+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:19.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:19 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:19.446+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:50.443178+0000 front 2024-05-10T11:34:50.443061+0000 (oldest deadline 2024-05-10T11:35:14.542749+0000) 2024-05-10T11:35:20.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:19 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:19.965+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:20.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:19 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:19.965+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:20.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:19 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:19.965+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:20.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:19 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:19.965+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:20.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:19 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:19.965+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:20.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:19 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:19.965+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:20.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:May 10 11:35:19 smithi027 podman[126452]: 2024-05-10 11:35:19.897981771 +0000 UTC m=+1.049844582 container died 30c61d23217df7f6e98f664b8c491362d658123753eccdf603c7d29eeed77dd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-9-deactivate, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-10T11:35:20.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.9.service' 2024-05-10T11:35:20.884 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:35:20.885 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-10T11:35:20.885 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-10T11:35:20.885 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.10 2024-05-10T11:35:20.910 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:20 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:20.487+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:20.910 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:20 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:20.487+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:20.910 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:20 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:20.487+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:20.911 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:20 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:20.487+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:20.911 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:20 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:20.487+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:20.911 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:20 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:20.487+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:20.911 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:20 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:20.487+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:50.443178+0000 front 2024-05-10T11:34:50.443061+0000 (oldest deadline 2024-05-10T11:35:14.542749+0000) 2024-05-10T11:35:21.175 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:20 smithi076 systemd[1]: Stopping Ceph osd.10 for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:35:21.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:20 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:20.985+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:21.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:20 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:20.985+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:21.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:20 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:20.985+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:21.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:20 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:20.985+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:21.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:20 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:20.985+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:21.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:20 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:20.985+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:21.238 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:20 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:20.985+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:21.457 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:21 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:21.199+0000 7fc2cdb1f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:35:21.457 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:21 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:21.199+0000 7fc2cdb1f700 -1 osd.10 123 *** Got signal Terminated *** 2024-05-10T11:35:21.457 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:21 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:21.199+0000 7fc2cdb1f700 -1 osd.10 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-10T11:35:21.458 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:21 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:21.457+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:21.458 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:21 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:21.457+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:21.458 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:21 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:21.457+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:21.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:21 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:21.457+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:21.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:21 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:21.457+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:21.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:21 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:21.457+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:21.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:21 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:21.457+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:50.443178+0000 front 2024-05-10T11:34:50.443061+0000 (oldest deadline 2024-05-10T11:35:14.542749+0000) 2024-05-10T11:35:22.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:21 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:21.954+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:22.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:21 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:21.954+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:22.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:21 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:21.954+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:22.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:21 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:21.954+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:22.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:21 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:21.954+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:22.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:21 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:21.954+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:22.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:21 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:21.954+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:22.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:22.453+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:22.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:22.453+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:22.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:22.453+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:22.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:22.453+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:22.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:22.453+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:22.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:22.453+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:22.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:22 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:22.453+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:50.443178+0000 front 2024-05-10T11:34:50.443061+0000 (oldest deadline 2024-05-10T11:35:14.542749+0000) 2024-05-10T11:35:23.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:22.913+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:23.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:22.913+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:23.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:22.913+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:23.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:22.913+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:23.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:22.913+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:23.238 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:22.913+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:23.238 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:22 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:22.913+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:23.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:23 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:23.498+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:23.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:23 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:23.498+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:23.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:23 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:23.498+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:23.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:23 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:23.498+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:23.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:23 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:23.498+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:23.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:23 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:23.498+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:23.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:23 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:23.498+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:50.443178+0000 front 2024-05-10T11:34:50.443061+0000 (oldest deadline 2024-05-10T11:35:14.542749+0000) 2024-05-10T11:35:24.236 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:23 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:23.953+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:24.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:23 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:23.953+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:24.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:23 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:23.953+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:24.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:23 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:23.953+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:24.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:23 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:23.953+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:24.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:23 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:23.953+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:24.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:23 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:23.953+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:24.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:24 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:24.502+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:24.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:24 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:24.502+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:24.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:24 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:24.502+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:24.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:24 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:24.502+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:24.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:24 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:24.502+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:24.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:24 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:24.502+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:24.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:24 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:24.502+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:50.443178+0000 front 2024-05-10T11:34:50.443061+0000 (oldest deadline 2024-05-10T11:35:14.542749+0000) 2024-05-10T11:35:25.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:24 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:24.997+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:25.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:24 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:24.997+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:25.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:24 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:24.997+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:25.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:24 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:24.997+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:25.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:24 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:24.997+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:25.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:24 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:24.997+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:25.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:24 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:24.997+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:25.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:25 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:25.542+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.232991+0000 front 2024-05-10T11:33:55.233073+0000 (oldest deadline 2024-05-10T11:34:20.532820+0000) 2024-05-10T11:35:25.925 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:25 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:25.542+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-05-10T11:34:06.835074+0000 front 2024-05-10T11:34:06.835036+0000 (oldest deadline 2024-05-10T11:34:32.135290+0000) 2024-05-10T11:35:25.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:25 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:25.542+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:15.036337+0000 front 2024-05-10T11:34:15.036445+0000 (oldest deadline 2024-05-10T11:34:39.736301+0000) 2024-05-10T11:35:25.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:25 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:25.542+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:22.538226+0000 front 2024-05-10T11:34:22.538273+0000 (oldest deadline 2024-05-10T11:34:47.838137+0000) 2024-05-10T11:35:25.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:25 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:25.542+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:32.439869+0000 front 2024-05-10T11:34:32.439897+0000 (oldest deadline 2024-05-10T11:34:57.740006+0000) 2024-05-10T11:35:25.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:25 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:25.542+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:37.740362+0000 front 2024-05-10T11:34:37.740344+0000 (oldest deadline 2024-05-10T11:35:03.540846+0000) 2024-05-10T11:35:25.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:25 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10[59083]: 2024-05-10T11:35:25.542+0000 7fc2c610b700 -1 osd.10 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:50.443178+0000 front 2024-05-10T11:34:50.443061+0000 (oldest deadline 2024-05-10T11:35:14.542749+0000) 2024-05-10T11:35:26.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.036+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:26.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.036+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:26.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.036+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:26.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.036+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:26.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.036+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:26.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.036+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:26.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.036+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:26.832 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:26 smithi076 podman[74142]: 2024-05-10 11:35:26.57371997 +0000 UTC m=+5.499981555 container died e668e2629b348a0f80a507d6d1ce9455681938549892cee26f80216916d08e94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-10T11:35:27.132 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:26 smithi076 podman[74142]: 2024-05-10 11:35:26.91535505 +0000 UTC m=+5.841616639 container cleanup e668e2629b348a0f80a507d6d1ce9455681938549892cee26f80216916d08e94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5) 2024-05-10T11:35:27.133 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:26 smithi076 bash[74142]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10 2024-05-10T11:35:27.425 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:27 smithi076 podman[74162]: 2024-05-10 11:35:27.133239256 +0000 UTC m=+0.554129024 container remove e668e2629b348a0f80a507d6d1ce9455681938549892cee26f80216916d08e94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-10T11:35:27.441 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.996+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:27.441 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.996+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:27.441 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.996+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:27.441 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.996+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:27.442 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.996+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:27.442 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.996+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:27.442 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:26 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:26.996+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:27.877 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:27 smithi076 podman[74248]: 2024-05-10 11:35:27.572201361 +0000 UTC m=+0.112416683 container create 272af76838f8f24461e267d377615e94303204b0eb3e6e9f6b22be29e753a1ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-10T11:35:27.878 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:27 smithi076 podman[74248]: 2024-05-10 11:35:27.477647965 +0000 UTC m=+0.017863288 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:35:27.878 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:27 smithi076 podman[74248]: 2024-05-10 11:35:27.778270608 +0000 UTC m=+0.318485936 container init 272af76838f8f24461e267d377615e94303204b0eb3e6e9f6b22be29e753a1ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507) 2024-05-10T11:35:27.878 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:27 smithi076 podman[74248]: 2024-05-10 11:35:27.781652965 +0000 UTC m=+0.321868295 container start 272af76838f8f24461e267d377615e94303204b0eb3e6e9f6b22be29e753a1ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-10T11:35:27.878 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:27 smithi076 podman[74248]: 2024-05-10 11:35:27.878119875 +0000 UTC m=+0.418335200 container attach 272af76838f8f24461e267d377615e94303204b0eb3e6e9f6b22be29e753a1ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08) 2024-05-10T11:35:28.175 INFO:journalctl@ceph.osd.10.smithi076.stdout:May 10 11:35:28 smithi076 podman[74248]: 2024-05-10 11:35:28.073037414 +0000 UTC m=+0.613252737 container died 272af76838f8f24461e267d377615e94303204b0eb3e6e9f6b22be29e753a1ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-10T11:35:28.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:28.003+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:28.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:28.003+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:28.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:28.003+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:28.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:28.003+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:28.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:28.003+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:28.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:28.003+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:28.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:28.003+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:28.597 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.10.service' 2024-05-10T11:35:28.676 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:35:28.677 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-10T11:35:28.677 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-10T11:35:28.677 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.11 2024-05-10T11:35:28.986 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:28 smithi189 systemd[1]: Stopping Ceph osd.11 for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:35:29.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:28.994+0000 7f11ce22d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:35:29.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:28.994+0000 7f11ce22d700 -1 osd.11 123 *** Got signal Terminated *** 2024-05-10T11:35:29.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:28 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:28.994+0000 7f11ce22d700 -1 osd.11 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-10T11:35:29.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:29.043+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:29.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:29.043+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:29.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:29.043+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:29.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:29.043+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:29.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:29.043+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:29.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:29.043+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:29.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:29.043+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:29.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:29 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:29.043+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6820 osd.8 since back 2024-05-10T11:35:04.300089+0000 front 2024-05-10T11:35:04.300126+0000 (oldest deadline 2024-05-10T11:35:29.000314+0000) 2024-05-10T11:35:30.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:30.059+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:30.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:30.059+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:30.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:30.059+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:30.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:30.059+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:30.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:30.059+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:30.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:30.059+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:30.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:30.059+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:30.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:30 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:30.059+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6820 osd.8 since back 2024-05-10T11:35:04.300089+0000 front 2024-05-10T11:35:04.300126+0000 (oldest deadline 2024-05-10T11:35:29.000314+0000) 2024-05-10T11:35:31.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:31.045+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:31.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:31.045+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:31.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:31.045+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:31.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:31.045+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:31.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:31.045+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:31.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:31.045+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:31.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:31.045+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:31.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:31 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:31.045+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6820 osd.8 since back 2024-05-10T11:35:04.300089+0000 front 2024-05-10T11:35:04.300126+0000 (oldest deadline 2024-05-10T11:35:29.000314+0000) 2024-05-10T11:35:32.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:32.039+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:32.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:32.039+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:32.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:32.039+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:32.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:32.039+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:32.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:32.039+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:32.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:32.039+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:32.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:32.039+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:32.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:32 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:32.039+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6820 osd.8 since back 2024-05-10T11:35:04.300089+0000 front 2024-05-10T11:35:04.300126+0000 (oldest deadline 2024-05-10T11:35:29.000314+0000) 2024-05-10T11:35:33.486 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:33.049+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-10T11:33:55.388701+0000 front 2024-05-10T11:33:55.388562+0000 (oldest deadline 2024-05-10T11:34:20.589098+0000) 2024-05-10T11:35:33.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:33.049+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-10T11:34:10.590952+0000 front 2024-05-10T11:34:10.590976+0000 (oldest deadline 2024-05-10T11:34:41.791974+0000) 2024-05-10T11:35:33.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:33.049+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-05-10T11:34:24.693180+0000 front 2024-05-10T11:34:24.693454+0000 (oldest deadline 2024-05-10T11:34:49.392920+0000) 2024-05-10T11:35:33.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:33.049+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-05-10T11:34:33.994925+0000 front 2024-05-10T11:34:33.994797+0000 (oldest deadline 2024-05-10T11:34:56.894926+0000) 2024-05-10T11:35:33.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:33.049+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-10T11:34:39.795987+0000 front 2024-05-10T11:34:39.796051+0000 (oldest deadline 2024-05-10T11:35:05.696201+0000) 2024-05-10T11:35:33.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:33.049+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-05-10T11:34:49.798049+0000 front 2024-05-10T11:34:49.797938+0000 (oldest deadline 2024-05-10T11:35:12.697716+0000) 2024-05-10T11:35:33.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:33.049+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-05-10T11:34:54.899441+0000 front 2024-05-10T11:34:54.899461+0000 (oldest deadline 2024-05-10T11:35:20.799319+0000) 2024-05-10T11:35:33.488 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:33 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11[60508]: 2024-05-10T11:35:33.049+0000 7f11c6819700 -1 osd.11 123 heartbeat_check: no reply from 172.21.15.189:6820 osd.8 since back 2024-05-10T11:35:04.300089+0000 front 2024-05-10T11:35:04.300126+0000 (oldest deadline 2024-05-10T11:35:29.000314+0000) 2024-05-10T11:35:34.737 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:34 smithi189 podman[76115]: 2024-05-10 11:35:34.476717085 +0000 UTC m=+5.598480529 container died c24dbd23f76164b76b81283da7368707d0609c3ce7f2994409f3d3f6b9b67827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux ) 2024-05-10T11:35:35.236 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:34 smithi189 podman[76115]: 2024-05-10 11:35:34.778821498 +0000 UTC m=+5.900584950 container cleanup c24dbd23f76164b76b81283da7368707d0609c3ce7f2994409f3d3f6b9b67827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-10T11:35:35.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:34 smithi189 bash[76115]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11 2024-05-10T11:35:35.237 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:35 smithi189 podman[76132]: 2024-05-10 11:35:35.004728643 +0000 UTC m=+0.525866896 container remove c24dbd23f76164b76b81283da7368707d0609c3ce7f2994409f3d3f6b9b67827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:35:35.640 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:35 smithi189 podman[76225]: 2024-05-10 11:35:35.218749389 +0000 UTC m=+0.017826000 image pull ee5e4f21ef58047cb638cccd5cfbe8f6f43412b56c94b1d7fbd9cf3cb90a53a5 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f 2024-05-10T11:35:35.640 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:35 smithi189 podman[76225]: 2024-05-10 11:35:35.338731416 +0000 UTC m=+0.137807987 container create 28c06a0e77fdef6da3e6a91b4ff68b4b88275458572480828296faf404e96aea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-10T11:35:35.640 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:35 smithi189 podman[76225]: 2024-05-10 11:35:35.557705157 +0000 UTC m=+0.356781735 container init 28c06a0e77fdef6da3e6a91b4ff68b4b88275458572480828296faf404e96aea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11-deactivate, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-10T11:35:35.641 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:35 smithi189 podman[76225]: 2024-05-10 11:35:35.560804596 +0000 UTC m=+0.359881169 container start 28c06a0e77fdef6da3e6a91b4ff68b4b88275458572480828296faf404e96aea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, ceph=True) 2024-05-10T11:35:35.641 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:35 smithi189 podman[76225]: 2024-05-10 11:35:35.640646867 +0000 UTC m=+0.439723458 container attach 28c06a0e77fdef6da3e6a91b4ff68b4b88275458572480828296faf404e96aea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-10T11:35:35.986 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 10 11:35:35 smithi189 podman[76225]: 2024-05-10 11:35:35.856176981 +0000 UTC m=+0.655253560 container died 28c06a0e77fdef6da3e6a91b4ff68b4b88275458572480828296faf404e96aea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True) 2024-05-10T11:35:36.402 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@osd.11.service' 2024-05-10T11:35:36.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:35:36.440 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-10T11:35:36.440 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-10T11:35:36.440 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.a 2024-05-10T11:35:36.868 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:35:36 smithi027 systemd[1]: Stopping Ceph mds.a for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:35:37.368 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:35:37 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a[71669]: 2024-05-10T11:35:37.037+0000 7fb24ae00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:35:37.369 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:35:37 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a[71669]: 2024-05-10T11:35:37.037+0000 7fb24ae00700 -1 mds.a *** got signal Terminated *** 2024-05-10T11:35:38.368 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:35:38 smithi027 podman[126558]: 2024-05-10 11:35:38.07838638 +0000 UTC m=+1.169668962 container died 4808f01523b20775d2f362e7c2be200d8eb57b6a27b9dd67f77228759a76540e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a, org.label-schema.build-date=20240507, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-10T11:35:38.634 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:35:38 smithi027 podman[126558]: 2024-05-10 11:35:38.609876197 +0000 UTC m=+1.701158775 container cleanup 4808f01523b20775d2f362e7c2be200d8eb57b6a27b9dd67f77228759a76540e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-10T11:35:38.634 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:35:38 smithi027 bash[126558]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a 2024-05-10T11:35:39.368 INFO:journalctl@ceph.mds.a.smithi027.stdout:May 10 11:35:39 smithi027 podman[126572]: 2024-05-10 11:35:39.007269025 +0000 UTC m=+0.907399773 container remove 4808f01523b20775d2f362e7c2be200d8eb57b6a27b9dd67f77228759a76540e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-a, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux ) 2024-05-10T11:35:39.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.a.service' 2024-05-10T11:35:39.626 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:35:39.627 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-10T11:35:39.627 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-10T11:35:39.627 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.d 2024-05-10T11:35:40.118 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:35:39 smithi027 systemd[1]: Stopping Ceph mds.d for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:35:40.619 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:35:40 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-d[70479]: 2024-05-10T11:35:40.258+0000 7f9808400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:35:40.619 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:35:40 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-d[70479]: 2024-05-10T11:35:40.258+0000 7f9808400700 -1 mds.d *** got signal Terminated *** 2024-05-10T11:35:43.851 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:35:43 smithi027 podman[126678]: 2024-05-10 11:35:43.582551613 +0000 UTC m=+3.469099917 container died b779ebe9d1e000e0055757c31671f47671642b7ddc6b9dc509f321d53b938344 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-10T11:35:44.118 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:35:44 smithi027 podman[126678]: 2024-05-10 11:35:44.059965722 +0000 UTC m=+3.946514019 container cleanup b779ebe9d1e000e0055757c31671f47671642b7ddc6b9dc509f321d53b938344 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_CLEAN=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-10T11:35:44.118 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:35:44 smithi027 bash[126678]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-d 2024-05-10T11:35:44.618 INFO:journalctl@ceph.mds.d.smithi027.stdout:May 10 11:35:44 smithi027 podman[126704]: 2024-05-10 11:35:44.253055087 +0000 UTC m=+0.655400963 container remove b779ebe9d1e000e0055757c31671f47671642b7ddc6b9dc509f321d53b938344 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-d, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True) 2024-05-10T11:35:44.779 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.d.service' 2024-05-10T11:35:44.898 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:35:44.898 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-10T11:35:44.898 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-10T11:35:44.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.g 2024-05-10T11:35:45.369 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:35:45 smithi027 systemd[1]: Stopping Ceph mds.g for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:35:45.868 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:35:45 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-g[71262]: 2024-05-10T11:35:45.537+0000 7f4cb4600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:35:45.869 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:35:45 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-g[71262]: 2024-05-10T11:35:45.537+0000 7f4cb4600700 -1 mds.g *** got signal Terminated *** 2024-05-10T11:35:47.320 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:35:46 smithi027 podman[126809]: 2024-05-10 11:35:46.88189196 +0000 UTC m=+1.465993841 container died 090e76e231458a9958d5e15083b7a88530750df2fa75d8f11f33c85dae3abe6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-g, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_CLEAN=True) 2024-05-10T11:35:47.570 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:35:47 smithi027 podman[126809]: 2024-05-10 11:35:47.539236781 +0000 UTC m=+2.123338668 container cleanup 090e76e231458a9958d5e15083b7a88530750df2fa75d8f11f33c85dae3abe6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-g, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux ) 2024-05-10T11:35:47.571 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:35:47 smithi027 bash[126809]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-g 2024-05-10T11:35:48.368 INFO:journalctl@ceph.mds.g.smithi027.stdout:May 10 11:35:47 smithi027 podman[126823]: 2024-05-10 11:35:47.898884911 +0000 UTC m=+1.004321490 container remove 090e76e231458a9958d5e15083b7a88530750df2fa75d8f11f33c85dae3abe6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-g, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-10T11:35:48.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.g.service' 2024-05-10T11:35:48.596 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:35:48.596 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-10T11:35:48.596 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-10T11:35:48.596 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.j 2024-05-10T11:35:49.118 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:35:48 smithi027 systemd[1]: Stopping Ceph mds.j for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:35:49.618 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:35:49 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j[70072]: 2024-05-10T11:35:49.215+0000 7f153aa00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:35:49.618 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:35:49 smithi027 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j[70072]: 2024-05-10T11:35:49.215+0000 7f153aa00700 -1 mds.j *** got signal Terminated *** 2024-05-10T11:35:53.564 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:35:53 smithi027 podman[126929]: 2024-05-10 11:35:53.23319637 +0000 UTC m=+4.143832770 container died 955b6fb80c9e16d5572fb8deb1d74dea120cdd7357c4a16245eb4b23ae1f48f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-10T11:35:53.868 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:35:53 smithi027 podman[126929]: 2024-05-10 11:35:53.76779904 +0000 UTC m=+4.678435485 container cleanup 955b6fb80c9e16d5572fb8deb1d74dea120cdd7357c4a16245eb4b23ae1f48f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507) 2024-05-10T11:35:53.868 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:35:53 smithi027 bash[126929]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j 2024-05-10T11:35:54.368 INFO:journalctl@ceph.mds.j.smithi027.stdout:May 10 11:35:53 smithi027 podman[126943]: 2024-05-10 11:35:53.977520931 +0000 UTC m=+0.727666549 container remove 955b6fb80c9e16d5572fb8deb1d74dea120cdd7357c4a16245eb4b23ae1f48f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-j, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-10T11:35:54.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.j.service' 2024-05-10T11:35:54.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:35:54.638 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-10T11:35:54.638 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-10T11:35:54.639 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.b 2024-05-10T11:35:54.925 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:35:54 smithi076 systemd[1]: Stopping Ceph mds.b for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:35:55.202 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:35:54 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b[64743]: 2024-05-10T11:35:54.948+0000 7f0f74f83700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:35:55.202 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:35:54 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b[64743]: 2024-05-10T11:35:54.948+0000 7f0f74f83700 -1 mds.b *** got signal Terminated *** 2024-05-10T11:35:58.348 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:35:58 smithi076 podman[74352]: 2024-05-10 11:35:58.045805879 +0000 UTC m=+3.216272200 container died 46f39929fdc34b5e297c415d7ed8da10de227b508e86027bfe5a2873a2054090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-10T11:35:58.673 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:35:58 smithi076 podman[74352]: 2024-05-10 11:35:58.423127966 +0000 UTC m=+3.593594285 container cleanup 46f39929fdc34b5e297c415d7ed8da10de227b508e86027bfe5a2873a2054090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-10T11:35:58.674 INFO:journalctl@ceph.mds.b.smithi076.stdout:May 10 11:35:58 smithi076 bash[74352]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-b 2024-05-10T11:35:58.793 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.b.service' 2024-05-10T11:35:58.828 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:35:58.828 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-10T11:35:58.829 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-10T11:35:58.829 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.e 2024-05-10T11:35:58.925 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:35:58 smithi076 systemd[1]: Stopping Ceph mds.e for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:35:59.425 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:35:59 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-e[64120]: 2024-05-10T11:35:59.143+0000 7f51967bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:35:59.426 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:35:59 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-e[64120]: 2024-05-10T11:35:59.143+0000 7f51967bb700 -1 mds.e *** got signal Terminated *** 2024-05-10T11:36:02.322 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:36:02 smithi076 podman[74487]: 2024-05-10 11:36:02.060425623 +0000 UTC m=+3.018876692 container died 59efe649ffc03a0f0982af152f601ce2df7eb5ac5ea7f258d91d6b9301d32789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:36:02.675 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:36:02 smithi076 podman[74487]: 2024-05-10 11:36:02.40169657 +0000 UTC m=+3.360147635 container cleanup 59efe649ffc03a0f0982af152f601ce2df7eb5ac5ea7f258d91d6b9301d32789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS) 2024-05-10T11:36:02.675 INFO:journalctl@ceph.mds.e.smithi076.stdout:May 10 11:36:02 smithi076 bash[74487]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-e 2024-05-10T11:36:02.792 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.e.service' 2024-05-10T11:36:02.824 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:36:02.825 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-10T11:36:02.825 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-10T11:36:02.825 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.h 2024-05-10T11:36:03.146 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:36:02 smithi076 systemd[1]: Stopping Ceph mds.h for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:36:03.425 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:36:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-h[64424]: 2024-05-10T11:36:03.146+0000 7fd8f68f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:36:03.425 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:36:03 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-h[64424]: 2024-05-10T11:36:03.146+0000 7fd8f68f2700 -1 mds.h *** got signal Terminated *** 2024-05-10T11:36:06.425 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:36:06 smithi076 podman[74607]: 2024-05-10 11:36:06.041160437 +0000 UTC m=+3.000847079 container died d0eb84fcde181738e1b7fd706a7ef9cec06d71440ec9b8589b549e54031fa6c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-h, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08) 2024-05-10T11:36:06.893 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:36:06 smithi076 podman[74607]: 2024-05-10 11:36:06.605387731 +0000 UTC m=+3.565074384 container cleanup d0eb84fcde181738e1b7fd706a7ef9cec06d71440ec9b8589b549e54031fa6c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-05-10T11:36:06.894 INFO:journalctl@ceph.mds.h.smithi076.stdout:May 10 11:36:06 smithi076 bash[74607]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-h 2024-05-10T11:36:07.031 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.h.service' 2024-05-10T11:36:07.062 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:36:07.063 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-10T11:36:07.063 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-10T11:36:07.063 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.k 2024-05-10T11:36:07.425 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:36:07 smithi076 systemd[1]: Stopping Ceph mds.k for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:36:07.426 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:36:07 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k[63800]: 2024-05-10T11:36:07.393+0000 7f36ba1fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:36:07.426 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:36:07 smithi076 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k[63800]: 2024-05-10T11:36:07.393+0000 7f36ba1fd700 -1 mds.k *** got signal Terminated *** 2024-05-10T11:36:10.425 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:36:10 smithi076 podman[74729]: 2024-05-10 11:36:10.048619083 +0000 UTC m=+2.756771244 container died f712d006514775982f7838980a5af47efe2267d12b22e7a8032f8a0aed20bc5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08) 2024-05-10T11:36:10.920 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:36:10 smithi076 podman[74729]: 2024-05-10 11:36:10.711045064 +0000 UTC m=+3.419197231 container cleanup f712d006514775982f7838980a5af47efe2267d12b22e7a8032f8a0aed20bc5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-10T11:36:10.921 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:36:10 smithi076 bash[74729]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k 2024-05-10T11:36:11.172 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.k.service' 2024-05-10T11:36:11.175 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:36:11 smithi076 podman[74743]: 2024-05-10 11:36:11.053898742 +0000 UTC m=+0.689818236 container remove f712d006514775982f7838980a5af47efe2267d12b22e7a8032f8a0aed20bc5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, ceph=True) 2024-05-10T11:36:11.176 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:36:11 smithi076 systemd[1]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.k.service: Deactivated successfully. 2024-05-10T11:36:11.176 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:36:11 smithi076 systemd[1]: Stopped Ceph mds.k for 063c9e18-0ebc-11ef-bc98-c7b262605968. 2024-05-10T11:36:11.176 INFO:journalctl@ceph.mds.k.smithi076.stdout:May 10 11:36:11 smithi076 systemd[1]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.k.service: Consumed 6.819s CPU time. 2024-05-10T11:36:11.203 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:36:11.203 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-10T11:36:11.203 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-10T11:36:11.204 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.c 2024-05-10T11:36:11.487 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:36:11 smithi189 systemd[1]: Stopping Ceph mds.c for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:36:11.753 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:36:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-c[65500]: 2024-05-10T11:36:11.508+0000 7f91ff117700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:36:11.754 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:36:11 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-c[65500]: 2024-05-10T11:36:11.508+0000 7f91ff117700 -1 mds.c *** got signal Terminated *** 2024-05-10T11:36:14.292 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:36:14 smithi189 podman[76331]: 2024-05-10 11:36:14.027617333 +0000 UTC m=+2.631153746 container died 288fb75edf64f6750ee019a23d24e8590b64d0c577d570f80acae2fe2a04204f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:36:14.650 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:36:14 smithi189 podman[76331]: 2024-05-10 11:36:14.383439719 +0000 UTC m=+2.986976189 container cleanup 288fb75edf64f6750ee019a23d24e8590b64d0c577d570f80acae2fe2a04204f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-c, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True) 2024-05-10T11:36:14.650 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 10 11:36:14 smithi189 bash[76331]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-c 2024-05-10T11:36:14.762 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.c.service' 2024-05-10T11:36:14.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:36:14.799 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-10T11:36:14.800 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-10T11:36:14.800 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.f 2024-05-10T11:36:14.986 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:36:14 smithi189 systemd[1]: Stopping Ceph mds.f for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:36:15.486 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:36:15 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f[65183]: 2024-05-10T11:36:15.120+0000 7f1e2c4d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:36:15.487 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:36:15 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f[65183]: 2024-05-10T11:36:15.120+0000 7f1e2c4d8700 -1 mds.f *** got signal Terminated *** 2024-05-10T11:36:18.336 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:36:18 smithi189 podman[76466]: 2024-05-10 11:36:18.021592346 +0000 UTC m=+3.006773313 container died ce5743ad0fa257c2fda74054a9c8768a57b28bfdc911415428af857b40c58f8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f, CEPH_POINT_RELEASE=, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-05-10T11:36:18.336 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:36:18 smithi189 podman[76466]: 2024-05-10 11:36:18.33667356 +0000 UTC m=+3.321854481 container cleanup ce5743ad0fa257c2fda74054a9c8768a57b28bfdc911415428af857b40c58f8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f, GIT_CLEAN=True, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-10T11:36:18.666 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:36:18 smithi189 bash[76466]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f 2024-05-10T11:36:18.666 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 10 11:36:18 smithi189 podman[76480]: 2024-05-10 11:36:18.546334777 +0000 UTC m=+0.518212609 container remove ce5743ad0fa257c2fda74054a9c8768a57b28bfdc911415428af857b40c58f8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-10T11:36:18.676 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.f.service' 2024-05-10T11:36:18.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:36:18.708 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-10T11:36:18.708 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-10T11:36:18.709 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.i 2024-05-10T11:36:18.986 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:36:18 smithi189 systemd[1]: Stopping Ceph mds.i for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:36:19.486 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:36:19 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-i[65808]: 2024-05-10T11:36:19.048+0000 7fcebece1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:36:19.487 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:36:19 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-i[65808]: 2024-05-10T11:36:19.048+0000 7fcebece1700 -1 mds.i *** got signal Terminated *** 2024-05-10T11:36:22.316 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:36:22 smithi189 podman[76590]: 2024-05-10 11:36:22.021467355 +0000 UTC m=+3.100083647 container died 1d25da1e43c518bca01553bb5daebba0af240792ed4d198be2233e7161e9797c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-i, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:36:22.640 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:36:22 smithi189 podman[76590]: 2024-05-10 11:36:22.381681223 +0000 UTC m=+3.460297519 container cleanup 1d25da1e43c518bca01553bb5daebba0af240792ed4d198be2233e7161e9797c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-i, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, org.label-schema.vendor=CentOS, ceph=True) 2024-05-10T11:36:22.640 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 10 11:36:22 smithi189 bash[76590]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-i 2024-05-10T11:36:22.819 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.i.service' 2024-05-10T11:36:22.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:36:22.851 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-10T11:36:22.851 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-10T11:36:22.852 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.l 2024-05-10T11:36:23.236 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:36:22 smithi189 systemd[1]: Stopping Ceph mds.l for 063c9e18-0ebc-11ef-bc98-c7b262605968... 2024-05-10T11:36:23.237 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:36:23 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-l[64880]: 2024-05-10T11:36:23.159+0000 7f2fea6c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-10T11:36:23.237 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:36:23 smithi189 ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-l[64880]: 2024-05-10T11:36:23.159+0000 7f2fea6c3700 -1 mds.l *** got signal Terminated *** 2024-05-10T11:36:26.293 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:36:26 smithi189 podman[76711]: 2024-05-10 11:36:26.032439224 +0000 UTC m=+2.965474118 container died 25ea3d564c7efc32fd95cf6a37431fb053fd3eaa175d5c187a2c9fa0fe378c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08) 2024-05-10T11:36:26.635 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:36:26 smithi189 podman[76711]: 2024-05-10 11:36:26.385007526 +0000 UTC m=+3.318042414 container cleanup 25ea3d564c7efc32fd95cf6a37431fb053fd3eaa175d5c187a2c9fa0fe378c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c77785140a1aa226ee46fbc1bec1768a7bea4786cdf8c717e77f2b0ad3d7725f, name=ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-l, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240507, RELEASE=wip-63896-from-wip-rishabh-testing-20240501.193033-d39fa08, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-10T11:36:26.635 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 10 11:36:26 smithi189 bash[76711]: ceph-063c9e18-0ebc-11ef-bc98-c7b262605968-mds-l 2024-05-10T11:36:26.798 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-063c9e18-0ebc-11ef-bc98-c7b262605968@mds.l.service' 2024-05-10T11:36:26.828 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-10T11:36:26.829 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-10T11:36:26.829 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-10T11:36:27.169 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:36:32.581 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-10T11:36:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:Deleting cluster with fsid: 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:36:35.361 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-10T11:36:35.551 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:36:37.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-10T11:36:37.994 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-10T11:36:38.025 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-10T11:36:38.051 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-10T11:36:38.054 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922/remote/smithi027/crash 2024-05-10T11:36:38.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/crash -- . 2024-05-10T11:36:38.157 INFO:teuthology.orchestra.run.smithi027.stderr:tar: /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-10T11:36:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:tar: Error is not recoverable: exiting now 2024-05-10T11:36:38.162 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922/remote/smithi076/crash 2024-05-10T11:36:38.163 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/crash -- . 2024-05-10T11:36:38.211 INFO:teuthology.orchestra.run.smithi076.stderr:tar: /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-10T11:36:38.212 INFO:teuthology.orchestra.run.smithi076.stderr:tar: Error is not recoverable: exiting now 2024-05-10T11:36:38.213 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922/remote/smithi189/crash 2024-05-10T11:36:38.214 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/crash -- . 2024-05-10T11:36:38.257 INFO:teuthology.orchestra.run.smithi189.stderr:tar: /var/lib/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-10T11:36:38.257 INFO:teuthology.orchestra.run.smithi189.stderr:tar: Error is not recoverable: exiting now 2024-05-10T11:36:38.258 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-10T11:36:38.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-10T11:36:38.375 INFO:tasks.cephadm:Compressing logs... 2024-05-10T11:36:38.375 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-10T11:36:38.380 DEBUG:teuthology.orchestra.run.smithi076:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-10T11:36:38.382 DEBUG:teuthology.orchestra.run.smithi189:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-10T11:36:38.409 INFO:teuthology.orchestra.run.smithi189.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.log 2024-05-10T11:36:38.410 INFO:teuthology.orchestra.run.smithi189.stderr:: No such file or directory 2024-05-10T11:36:38.410 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.2.log 2024-05-10T11:36:38.410 INFO:teuthology.orchestra.run.smithi189.stderr:gzip/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.log: -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.f.log 2024-05-10T11:36:38.410 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-10T11:36:38.410 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.5.log 2024-05-10T11:36:38.410 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.c.log 2024-05-10T11:36:38.411 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mon.c.log 2024-05-10T11:36:38.411 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mgr.z.log 2024-05-10T11:36:38.411 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mon.c.log: -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.i.log 2024-05-10T11:36:38.411 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-10T11:36:38.411 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.11.log 2024-05-10T11:36:38.411 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.8.log 2024-05-10T11:36:38.411 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-10T11:36:38.411 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.l.log 2024-05-10T11:36:38.411 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-10T11:36:38.424 INFO:teuthology.orchestra.run.smithi076.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-10T11:36:38.425 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.log 2024-05-10T11:36:38.425 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-10T11:36:38.426 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.7.log 2024-05-10T11:36:38.426 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.b.log 2024-05-10T11:36:38.426 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.4.log 2024-05-10T11:36:38.426 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.1.log 2024-05-10T11:36:38.429 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mgr.y.log 2024-05-10T11:36:38.429 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.h.log 2024-05-10T11:36:38.429 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.10.log 2024-05-10T11:36:38.429 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-10T11:36:38.429 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.e.log 2024-05-10T11:36:38.430 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-10T11:36:38.430 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mon.b.log 2024-05-10T11:36:38.430 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.k.log 2024-05-10T11:36:38.430 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-10T11:36:38.436 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-10T11:36:38.437 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.log 2024-05-10T11:36:38.438 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-10T11:36:38.438 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.g.log 2024-05-10T11:36:38.439 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.j.log 2024-05-10T11:36:38.441 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.3.log 2024-05-10T11:36:38.442 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.0.log 2024-05-10T11:36:38.442 INFO:teuthology.orchestra.run.smithi027.stderr: 90.3% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-10T11:36:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.3.log: 87.7% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-10T11:36:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mgr.x.log 2024-05-10T11:36:38.445 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mon.a.log 2024-05-10T11:36:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.6.log 2024-05-10T11:36:38.447 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mon.a.log: 94.4% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.g.log.gz 2024-05-10T11:36:38.447 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-10T11:36:38.449 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.6.log: -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.a.log 2024-05-10T11:36:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.cephadm.log: 84.0% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-10T11:36:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-10T11:36:38.453 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.d.log 2024-05-10T11:36:38.458 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-volume.log: /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.9.log 2024-05-10T11:36:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-10T11:36:38.466 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-05-10T11:36:38.476 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.admin.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-05-10T11:36:38.480 INFO:teuthology.orchestra.run.smithi027.stderr: 94.3% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.d.log.gz 2024-05-10T11:36:38.485 INFO:teuthology.orchestra.run.smithi027.stderr: 91.0% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-10T11:36:38.500 INFO:teuthology.orchestra.run.smithi027.stderr: 94.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-10T11:36:38.505 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-volume.log: /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 87.7% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-10T11:36:38.525 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 83.7% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-10T11:36:38.534 INFO:teuthology.orchestra.run.smithi189.stderr: 83.6% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-10T11:36:38.562 INFO:teuthology.orchestra.run.smithi076.stderr: 91.0% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-10T11:36:38.657 INFO:teuthology.orchestra.run.smithi189.stderr: 91.0% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-10T11:36:38.721 INFO:teuthology.orchestra.run.smithi189.stderr: 92.6% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mgr.z.log.gz 2024-05-10T11:36:38.722 INFO:teuthology.orchestra.run.smithi189.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-10T11:36:38.724 INFO:teuthology.orchestra.run.smithi189.stderr: 90.4% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-10T11:36:38.864 INFO:teuthology.orchestra.run.smithi076.stderr: 92.6% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mgr.y.log.gz 2024-05-10T11:36:39.065 INFO:teuthology.orchestra.run.smithi027.stderr: 90.2% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mgr.x.log.gz 2024-05-10T11:36:39.126 INFO:teuthology.orchestra.run.smithi027.stderr: 94.9% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.a.log.gz 2024-05-10T11:36:39.233 INFO:teuthology.orchestra.run.smithi076.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-10T11:36:39.237 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-10T11:36:39.451 INFO:teuthology.orchestra.run.smithi076.stderr: 90.4% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-10T11:36:40.020 INFO:teuthology.orchestra.run.smithi189.stderr: 95.7% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.l.log.gz 2024-05-10T11:36:40.520 INFO:teuthology.orchestra.run.smithi027.stderr: 90.2% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mon.a.log.gz 2024-05-10T11:36:40.833 INFO:teuthology.orchestra.run.smithi189.stderr: 92.7% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mon.c.log.gz 2024-05-10T11:36:42.800 INFO:teuthology.orchestra.run.smithi027.stderr: 95.5% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.j.log.gz 2024-05-10T11:36:52.574 INFO:teuthology.orchestra.run.smithi076.stderr: 91.6% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mon.b.log.gz 2024-05-10T11:37:00.824 INFO:teuthology.orchestra.run.smithi189.stderr: 94.4% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.i.log.gz 2024-05-10T11:37:02.228 INFO:teuthology.orchestra.run.smithi189.stderr: 94.7% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.f.log.gz 2024-05-10T11:37:11.775 INFO:teuthology.orchestra.run.smithi027.stderr: 91.4% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.0.log.gz 2024-05-10T11:37:16.021 INFO:teuthology.orchestra.run.smithi189.stderr: 91.4% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.11.log.gz 2024-05-10T11:37:19.371 INFO:teuthology.orchestra.run.smithi076.stderr: 94.8% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.k.log.gz 2024-05-10T11:37:20.025 INFO:teuthology.orchestra.run.smithi027.stderr: 91.2% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.3.log.gz 2024-05-10T11:37:20.210 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.6.log.gz 2024-05-10T11:37:22.143 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.9.log.gz 2024-05-10T11:37:22.147 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-05-10T11:37:22.147 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m43.735s 2024-05-10T11:37:22.147 INFO:teuthology.orchestra.run.smithi027.stderr:user 2m36.289s 2024-05-10T11:37:22.147 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m11.091s 2024-05-10T11:37:23.774 INFO:teuthology.orchestra.run.smithi189.stderr: 91.3% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.2.log.gz 2024-05-10T11:37:24.116 INFO:teuthology.orchestra.run.smithi189.stderr: 91.0% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.8.log.gz 2024-05-10T11:37:28.581 INFO:teuthology.orchestra.run.smithi189.stderr: 94.2% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.c.log.gz 2024-05-10T11:37:44.748 INFO:teuthology.orchestra.run.smithi189.stderr: 91.0% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.5.log.gz 2024-05-10T11:37:44.749 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-05-10T11:37:44.750 INFO:teuthology.orchestra.run.smithi189.stderr:real 1m6.353s 2024-05-10T11:37:44.750 INFO:teuthology.orchestra.run.smithi189.stderr:user 4m15.838s 2024-05-10T11:37:44.750 INFO:teuthology.orchestra.run.smithi189.stderr:sys 0m7.849s 2024-05-10T11:41:32.226 INFO:teuthology.orchestra.run.smithi076.stderr: 95.1% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.e.log.gz 2024-05-10T11:42:44.755 INFO:teuthology.orchestra.run.smithi076.stderr: 91.3% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.10.log.gz 2024-05-10T11:43:13.022 INFO:teuthology.orchestra.run.smithi076.stderr: 91.6% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.1.log.gz 2024-05-10T11:46:12.329 INFO:teuthology.orchestra.run.smithi076.stderr: 91.2% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.4.log.gz 2024-05-10T11:46:27.963 INFO:teuthology.orchestra.run.smithi076.stderr: 91.1% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-osd.7.log.gz 2024-05-10T11:49:21.669 INFO:teuthology.orchestra.run.smithi076.stderr: 93.6% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.h.log.gz 2024-05-10T11:53:24.455 INFO:teuthology.orchestra.run.smithi076.stderr: 93.2% -- replaced with /var/log/ceph/063c9e18-0ebc-11ef-bc98-c7b262605968/ceph-mds.b.log.gz 2024-05-10T11:53:24.456 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-05-10T11:53:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:real 16m46.060s 2024-05-10T11:53:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:user 7m45.250s 2024-05-10T11:53:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:sys 0m21.636s 2024-05-10T11:53:24.457 INFO:tasks.cephadm:Archiving logs... 2024-05-10T11:53:24.458 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922/remote/smithi027/log 2024-05-10T11:53:24.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-10T11:53:49.944 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/log/ceph to /home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922/remote/smithi076/log 2024-05-10T11:53:49.945 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-10T11:55:06.158 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922/remote/smithi189/log 2024-05-10T11:55:06.159 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-10T11:55:42.781 INFO:tasks.cephadm:Removing cluster... 2024-05-10T11:55:42.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 --force 2024-05-10T11:55:43.124 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:55:44.378 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 --force 2024-05-10T11:55:44.764 INFO:teuthology.orchestra.run.smithi076.stdout:Deleting cluster with fsid: 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:55:45.467 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 063c9e18-0ebc-11ef-bc98-c7b262605968 --force 2024-05-10T11:55:45.658 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: 063c9e18-0ebc-11ef-bc98-c7b262605968 2024-05-10T11:55:46.218 INFO:tasks.cephadm:Removing cephadm ... 2024-05-10T11:55:46.218 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-10T11:55:46.279 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-10T11:55:46.297 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-10T11:55:46.313 INFO:tasks.cephadm:Teardown complete 2024-05-10T11:55:46.313 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-10T11:55:46.329 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-10T11:55:46.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-10T11:55:46.337 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-10T11:55:46.340 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-10T11:55:46.503 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-10T11:55:46.514 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-10T11:55:46.515 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-10T11:55:46.523 DEBUG:teuthology.orchestra.run.smithi076:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-10T11:55:46.526 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-10T11:55:46.539 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-05-10T11:55:46.540 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-05-10T11:55:46.544 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-10T11:55:46.545 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-05-10T11:55:46.545 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv01.front.sepia.ceph.com 3 6 377 28 +1766us[+1781us] +/- 69ms 2024-05-10T11:55:46.545 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 28 -1571us[-1571us] +/- 51ms 2024-05-10T11:55:46.545 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -447us[ -431us] +/- 70ms 2024-05-10T11:55:46.545 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T11:55:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-05-10T11:55:46.580 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-10T11:55:46.580 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-05-10T11:55:46.580 INFO:teuthology.orchestra.run.smithi076.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 +1748us[+1743us] +/- 69ms 2024-05-10T11:55:46.581 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 25 -1587us[-1587us] +/- 51ms 2024-05-10T11:55:46.581 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -456us[ -462us] +/- 71ms 2024-05-10T11:55:46.581 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T11:55:46.581 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-10T11:55:46.582 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-05-10T11:55:46.582 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv01.front.sepia.ceph.com 3 6 377 21 +1838us[+1795us] +/- 53ms 2024-05-10T11:55:46.582 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 24 -1493us[-1537us] +/- 51ms 2024-05-10T11:55:46.582 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 -374us[ -418us] +/- 71ms 2024-05-10T11:55:46.582 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-10T11:55:46.584 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-10T11:55:46.594 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-10T11:55:46.595 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-10T11:55:46.605 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-10T11:55:46.850 DEBUG:teuthology.orchestra.run.smithi076:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-10T11:55:46.948 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-10T11:55:47.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-10T11:55:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T11:55:47.139 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-10T11:55:47.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T11:55:47.171 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-10T11:55:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T11:55:47.198 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-10T11:55:47.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-10T11:55:47.219 INFO:teuthology.task.internal:Duration was 4005.996120 seconds 2024-05-10T11:55:47.220 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-10T11:55:47.230 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-10T11:55:47.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-10T11:55:47.238 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-10T11:55:47.241 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-10T11:55:47.277 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-10T11:55:47.320 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-10T11:55:47.383 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-10T11:55:47.775 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-10T11:55:47.776 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-05-10T11:55:47.776 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-10T11:55:47.858 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi076.front.sepia.ceph.com 2024-05-10T11:55:47.858 DEBUG:teuthology.orchestra.run.smithi076:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-10T11:55:47.893 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-05-10T11:55:47.893 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-10T11:55:47.922 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-10T11:55:47.922 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-10T11:55:47.931 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-10T11:55:47.936 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-10T11:55:48.025 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-10T11:55:48.026 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-10T11:55:48.032 DEBUG:teuthology.orchestra.run.smithi076:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-10T11:55:48.035 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-10T11:55:49.556 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-10T11:55:49.569 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-10T11:55:49.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-10T11:55:49.655 DEBUG:teuthology.orchestra.run.smithi076:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-10T11:55:49.684 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-10T11:55:49.713 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-10T11:55:49.725 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-10T11:55:49.734 DEBUG:teuthology.orchestra.run.smithi076:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-10T11:55:49.737 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-10T11:55:49.769 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = core 2024-05-10T11:55:49.780 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-05-10T11:55:49.831 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-05-10T11:55:49.875 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-10T11:55:49.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T11:55:49.927 DEBUG:teuthology.orchestra.run.smithi076:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-10T11:55:49.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T11:55:49.943 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-10T11:55:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-10T11:55:49.961 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-10T11:55:49.977 INFO:teuthology.task.internal:Transferring archived files... 2024-05-10T11:55:49.978 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922/remote/smithi027 2024-05-10T11:55:49.979 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-10T11:55:50.123 DEBUG:teuthology.misc:Transferring archived files from smithi076:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922/remote/smithi076 2024-05-10T11:55:50.123 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-10T11:55:50.163 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922/remote/smithi189 2024-05-10T11:55:50.164 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-10T11:55:50.205 INFO:teuthology.task.internal:Removing archive directory... 2024-05-10T11:55:50.206 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-10T11:55:50.214 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-10T11:55:50.216 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-10T11:55:50.271 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-10T11:55:50.294 INFO:teuthology.task.internal:Not uploading archives. 2024-05-10T11:55:50.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-10T11:55:50.309 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-10T11:55:50.309 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-10T11:55:50.315 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-10T11:55:50.318 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-10T11:55:50.332 INFO:teuthology.orchestra.run.smithi076.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 10 11:55 /home/ubuntu/cephtest 2024-05-10T11:55:50.335 INFO:teuthology.orchestra.run.smithi189.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 10 11:55 /home/ubuntu/cephtest 2024-05-10T11:55:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 10 11:55 /home/ubuntu/cephtest 2024-05-10T11:55:50.373 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-10T11:55:50.387 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-10T11:55:50.413 INFO:teuthology.nuke:Checking targets against current locks 2024-05-10T11:55:50.441 DEBUG:teuthology.nuke:shortname: smithi027 2024-05-10T11:55:50.441 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-10T11:55:50.468 DEBUG:teuthology.nuke:shortname: smithi076 2024-05-10T11:55:50.468 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-10T11:55:50.474 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-10 10:37:51.320570', 'locked_by': 'scheduled_dparmar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO2ZeqFr+Mi+lBGNyRN7rbKcvPbOnBb/7SSjSi6s2U5//nBpwQsVcTvYwpV0Os35WOflGwLkBwQXpI8zBfUYG+U='} 2024-05-10T11:55:50.492 DEBUG:teuthology.nuke:shortname: smithi189 2024-05-10T11:55:50.493 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-10T11:55:50.500 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-10 10:37:51.317038', 'locked_by': 'scheduled_dparmar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGL9P8oSr9KcANsT+M0gFeZcBb//34wZrtiDDy36Lg63cjxFIIGT1Es+W0vTyOazAQG8uy1pCIhBQwEYQ54mkYQ='} 2024-05-10T11:55:50.516 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dparmar-2024-05-10_07:35:43-fs-wip-63896-from-wip-rishabh-testing-20240501.193033-distro-default-smithi/7700922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-10 10:37:51.318981', 'locked_by': 'scheduled_dparmar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPrg6vrjBqnla3DtydUdfBE5rN1oXn7F5V3VLnN5wlwfqevepwq1iNMwwjWJ1ukVHXWhR4gOslxjsA0HNdIDaiY='} 2024-05-10T11:55:50.531 INFO:teuthology.orchestra.console.smithi027:Power off 2024-05-10T11:55:50.531 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-10T11:55:50.554 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2024-05-10T11:55:50.555 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-10T11:55:50.574 INFO:teuthology.orchestra.console.smithi189:Power off 2024-05-10T11:55:50.574 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-10T11:55:50.592 INFO:teuthology.orchestra.console.smithi076:Power off 2024-05-10T11:55:50.592 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-10T11:55:50.602 DEBUG:teuthology.orchestra.console.smithi189:power off output: Chassis Power Control: Down/Off 2024-05-10T11:55:50.602 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-10T11:55:50.610 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-05-10T11:55:50.614 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off 2024-05-10T11:55:50.614 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-10T11:55:50.625 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-05-10T11:55:50.638 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-05-10T11:55:54.611 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-10T11:55:54.626 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-10T11:55:54.639 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-10T11:55:54.723 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-05-10T11:55:54.738 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-05-10T11:55:54.751 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-05-10T11:55:58.726 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-10T11:55:58.739 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-10T11:55:58.752 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-10T11:55:58.837 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-05-10T11:55:58.849 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is off 2024-05-10T11:55:58.862 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off 2024-05-10T11:55:58.951 INFO:teuthology.orchestra.console.smithi189:Power off completed 2024-05-10T11:55:58.963 INFO:teuthology.orchestra.console.smithi076:Power off completed 2024-05-10T11:56:02.842 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-10T11:56:02.954 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2024-05-10T11:56:03.055 INFO:teuthology.orchestra.console.smithi027:Power off completed 2024-05-10T11:56:03.162 INFO:teuthology.run:Summary data: client.0-kernel-sha1: fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/dbench}} duration: 4005.996119737625 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi027 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39fa08867836ac070c47cb66c782fa559fd480b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_dparmar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2d1f3391e3474d42902fb733914c6374 status: fail success: false 2024-05-10T11:56:03.163 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-10T11:56:03.250 INFO:teuthology.run:FAIL